Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a framework that learns the dynamic of safety-critical systems. The framework makes use of ideas from active learning, meta-learning, and reinforcement learning. As far as I understand, the framework seeks to learn an acquisition function that can evaluate the information gain from a certain action taken at a certain state. Acquisition function is a function that identifies an action the agent desires to take from a collection of unvisited states. The learning of such an acquisition function is formulated as a reinforcement learning problem, trained in a meta-learning fashion. I have the following comments:

1. The proposed method is evaluated on two problems, ADMETS and damaged aircraft.  Both problems are solved using synthetic data. While it is understandable that the framework is trained on simulated data, is there any justification with respect to the quality of the data simulator used in these two problems? Why performing well on these simulated data would be an indication of the proposed method can work reasonably well in practice?

2. Since the proposed framework is trained in a meta-learning fashion, is there any quantification on how fast/well (e.g. wrt. sample complexity) the trained policy adapting to new environments in test time?

3. While the paper takes safety during the learning of the dynamics as a critical issue, is there any justification with respect to the percentage of safety achieved by the proposed method? For example, why an 87% safety reported in Figure 5 achieved by the proposed method is a good number?

4. The key components of the proposed framework (e.g. meta-learning, reinforcement learning, active learning, safety) make sense to address the technical challenges in learning system dynamics in safety-critical domains. It is not very clear to me if the novelty of the proposed framework is appropriately highlighted. The paper can benefit from a better highlight of the novelty of the proposed method and why such novel contributions matter.
review_title: A Complex Framework that Deals with Safety-Critical Problems

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
This paper presents a meta-active learning approach to obtain an LSTM-based embedding of a dynamic system and use a chance-constrained (probabilistic safe) optimization to find optimal control configuration via applying mixed-inter linear programming (MILP) on the learned embeddings of the dynamic system. The main idea is to learn a Q-function as an acquisition function to describe the future information gain, which is the percent decrease in the error of the objective (e.g. model error) via a meta-learning strategy in which the agent interacts with the environment via distribution of altered dynamics. 

Strength: 
-  To my best knowledge, the idea that combines an LSTM embedding of dynamics with mixed-integer programming for optimization is novel meta-active learning algorithms. 
Weakness: 
- The algorithm relies on the assumption that the safety region must be hyper-ellipsoid with a known radius and the model error must come from a Gaussian distribution with known mean and variance. This makes the algorithm hard to apply for problems when the safety region is unknown or changing over time, which happens often in most real-world problems. 
- The poor writings make it hard to access the correctness of the technical details. 

Questions and comments: 
1) In Algorithm 1 line 11, should the subscript “T” be “t”? should the index “i” starting at line 7 be “t” and all the related subscript “i” be “t” instead?  
2) In Eq (1), should the subscript of U be “t:t+T” instead of “t:T”? If keeping “t:T”, the sentence above Eq (1) that states “the trajectory from [t, t+T)” should be “[t, T]”. Alternatively, the author could formally define what “t:T” means. Also, in the first paragraph of section 3.2, it’ll flow better if the authors could give some description of the second part on the right-hand side of Eq (1). 
3) Eq (2) is a bit hard to understand. What is \Delta_d^{t:2}? How is it set in practice? On the fourth line of the first paragraph on page 5, the paper states “and \bar{a} and \bar{b} are the point estimates of dynamics”, what is “\bar{a}” and “\bar{b}”?  The next sentence “d represents the d-th row and j the columns” is confusing. 
4) Eq (2), the first term under the square root has x_j^{(t)}^2, should the underscript “j” be “d’” instead? 
4) In the ADMETS experiment, the objective is defined as the L1 norm of the predicted optimal parameter and the ground truth optimal parameter;  how is the ground truth optimal parameter obtained? If the ground truth optimal parameter is already available, what’s the need for running the algorithm since the goal is to find the optimal parameters of the configurations? 
5) In the Aircraft experiment, the objective function is defined as the mean squared error of the model (see Appendix A.2), what is the model here? What is \hat{x}_i and x_i? 
6) In Appendix (A.4.2), what is the E in equation 13? 
7) Page 8, “Meta-learning for Dynamics”, line 6, “… most conducive …” should be “… most conductive …”, line 7, “While this approach provide .. “ should be “While this approach provides …” 
8) Eq (20) becomes 0 >= \prescript {o_i}  >= 1 when k_i =1, this will never be satisfied. 
 
review_title: An interesting paper that combines LSTM and mixed integer programming for meta-active learning but suffers from poor writing 

final_score: rating: 5: Marginally below acceptance threshold
review_content: #### Paper summary:

In this work, a meta active learning approach is proposed to learn the hidden dynamics of control systems, where safety is also a major concern. The main idea lies in doing meta-learning with Q-learning, meanwhile selecting safe actions by solving a mixed-integer linear programming problem. The performance of the proposed approach is verified from real datasets of deep brain stimulation by outperforming existing baselines for a significant gap in both accuracy and computational complexity.

#### Advantages:

- The paper discusses a very meaningful problem. Active learning with safety constraints is very important in many applicational domains.

- From the experiments, the proposed approach shows very significant performance in both the deep brain stimulation and the dynamics recovering tasks.

#### Disadvantages:

- From the perspective of machine learning, the proposed approach is a combination of existing techniques, thus the technical novelty is limited. 

- The paper is more about solving a specific robotics problem, which may not lie in the major focus of the ICLR conference.

Overall evaluation:

On one hand, I think the paper discusses a very meaningful problem, while on the other hand, I think ICLR may not be the most suitable venue to publish this work.

review_title: the paper is well-motivated and the proposed approach performs well empirically, while the paper is more about a specific application and somehow out of the scope of ICLR

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors proposed a meta-learning approach for active learning in the context of robot control.

What I like about this work is that there're several illustrations that make it clear which problem setup it is or what the authors meant by safety. The authors also included several competing methods, and the experiments showed promising results both in terms of several metrics, including information gain and safety. 

However, I also have some concerns about this paper that needs to be addressed.

In the experiments, there are 5 methods chosen to be compared against. However, several of them are not even mentioned before or after in introduction or related work to explain their close connections to the proposed one in more details. For example, Schrum & Gombolay 2020 and Wang et al., 2018b. And that makes me doubt whether it's a poor selection of comparisons or omission of important details in writing. 

Again in experiments, the method names described in 4.3 don't match the ones shown in figures. What's Random Bayesian in Figure 4? Why it disappeared in Figure 4(b) and 4(c)? Where's Epistemic Uncertainty and Maximizing Diversity? Similar for Figure 5, it was said that only Meta BO is excluded but where are the others? And even for Meta BO, the paper says it is "not included as it was designed for optimization tasks". But why can't the task in sec 4.2 be framed as an optimization task? What I meant is, I would like to see consistent set of methods in different tasks. Or add more tasks to show generality.

There are also inconsistency in naming. Is it BaO or Bao?

About the safety measure. It's evaluating using a point, and all parameters like minimum risk value need to be set by hand. How useful is such a safety measure? And how is it related to more common ways of measuring safety/risks, e.g. CVaR? Also the variables in Eq. (2) are not fully explained.

I'm also somewhat concerned that the authors are trying to solve too many problems in one algorithm. Safety, meta-learning, Q-learning.. But the experiments only show the results of everything combined without detailed analysis of the performance of each functionality while keeping others fixed. To some extent, the nature of the design of the method also sort of forbids modular tests. In the end, it might be just another impractical approach that incrementally adds several factors together, yet it's impossible to explain when it works or why it works.


review_title: Experiment section is somewhat problematic

Meta-review: This paper was quite contentious.  While reviewers appreciated the detailed response by the authors, and there is consensus that the paper addresses a relevant problem and contains interesting ideas, in the end there remain several concerns.  The paper provides a complex combination of techniques from active learning, meta learning and symbolic reasoning (via MILPs), and there are concerns about the clarity of the exposition.  For a paper claiming safety properties, there is also a lack of either formal theoretical analysis of well-specified safety properties, or a compelling demonstration of its effectiveness on a real system (all experiments are carried out in simulation).
Decision: Reject