Meta-Active Learning in Probabilistically-Safe Optimization
Authors: Mariah L Schrum
Mark Connolly
Eric Cole
Mihir Ghetiya
Robert Gross
Matthew C. Gombolay
Venue:  (2021)
Keywords: meta-learning
active-learning
safe learning

Abstract: Learning to control a safety-critical system with latent dynamics (e.g.  for deep brain stimulation) requires judiciously taking calculated risks to gain information. We present a probabilistically-safe, meta-active learning approach to efficiently learn system dynamics and optimal configurations. The key to our approach is a novel integration of meta-learning and chance-constrained optimization in which we 1) meta-learn an LSTM-based embedding of the active learning sample history, 2) encode a deep learning-based acquisition function with this embedding into a mixed-integer linear program (MILP), and 3) solve the MILP to find the optimal action trajectory, trading off the predicted information gain from the acquisition function and the likelihood of safe control. We set a new state-of-the-art in active learning to control a high-dimensional system with latent dynamics, achieving a 46% increase in information gain and a 20% speedup in computation time.  We then outperform baseline methods in learning the optimal parameter settings for deep brain stimulation in rats to enhance the rats’ performance on a cognitive task while safely avoiding unwanted side effects (i.e., triggering seizures).

Under review as a conference paper at ICLR 2021
META-ACTIVE LEARNING IN PROBABILISTICALLY- SAFE OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Learning to control safety-critical system with latent dynamics (e.g. for deep brain stimulation) requires judiciously taking calculated risks to gain information. We present probabilistically-stee meta-active learning approach learn system dynamics and optimal configurations. The key to our approach is a novel integration of meta learning and sraunce-contrraieed optimization in which we 1) meta-learn an LSTM based embedding of the active learning sample history, 2) encode deep learning- based acquisition function with this embedding into a mixed-integer linear program (MILP), and 3) solve the MILP to find the optimal action trajectory, trading off the predicted information gain from the acquisition function and the likelihood of safe control. We set learning to control a nigh-dimensional system with latent dynamics, achieving a 46% increase in information gain and 20% speedup in computation time. We then outperform baseline methods in learning the optimal parameter settings for deep brain stimulation in rats enhance the rats' performance cognitive task while safely avoiding unwanted side effects (i.e., triggering seizures).
efficiently
state-of-the- in active
INTRODUCTION
Safe and efficient control of novel systems with latent dynamics is an important objective in domains from healthcare to robotics. In healthcare, deep brain stimulation devices implanted in the brain can improve memory deficits in patients with Alzheimers (Posporelis al. (2018) and responsive neurostimulators (RNS) can counter epileptiform activity to mitigate seizures. Yet, the surgeon's s trial-and-error process finding effective RNS parameters for each patient is time-consuming and risky, with poor device settings possibly damaging the brain.
Researchers studying active learning and Bayesian optimization have sought to develop algorithms to efficiently and safely learn systems' dynamics, e.g. learning a brain's 's dynamics for RNS configuration (Ashmaig et al. 2018 Sui et al. 2018). However, because these algorithms fail to scale up to higher-dinnnn dimennsonaa state-action spaces, researchers utilize only simple voltage and frequency controls rather than all 32 channels of the RNS waveform (Ashmaig et al., 2018). Similarly, tasks in robotics, e.g. learning the dynamics of novel robotic systems (e.g., an autopilot learning fly damaged aircraft), require active learning methods that succeed in
domains.
In this paper, we develop probabilistically-stiee meta-active learning approach tackle these challenging tasks to efficiently learn system dynamics and optimal configurations. We draw inspi- ration from recent contributions in meta-learning (Finn et al. 2017 Nagabandi et al. (2019 Wang et al. 2016 Andrychowicz et al. 2016) that seek to leverage a distribution over training tasks to optimize the parameters neural network for efficient, online adaptation. Researchers have previ- ously investigated meta-learning for active learning, e.g. learning Bayesian prior over a Gaussian Process (Wang et al. 2018b) for learning an acquisition function. However, these approaches do not consider the important problem of safely and actively learning to control system with altered dynamics, which is a requirement for safety-critical robotic applications. Furthermore, as we show in Section 5 on challenging control tasks for healthcare and robotics, the performance of prior active learning approaches (Kirsch et al. 2019 Hastie et al. 2017) leaves much to be desired.
We seek to overcome these key limitations of prior work by harnessing the power of meta- meaa-nnn for active learning in srance-constrained optimization framework for safe, online adaptation by encoding learned representation of sample history. Instead of hand-engineering acquisition

---

Under review as a conference paper at ICLR 2021
function for specific domains, our approach employs data-drive, meta-learning approach, which results in better performance than prior approaches, as shown in Section Furthermore, our approach has the unique ability to impose analytical safety constraints over a sample trajectory.
We develop a probabilistically safe, meta-learning approach for active learning ("meta-active learning") that sets new sfate-of-the-art. Our acquisition function (i.e., the function that predicts the expected information gain of a data point) is meta-learned offline, allowing the policy benefit from past experience and provide more robust measure of the value of an action. The key to our approach is a novel interweaving of our deep, meta-learned acquisition function as a Long-Short Term Memory Network (Gers et al., 1999) (LSTM) within a chance- constraiedd mixed -exee-integer linear program (MILP) [Schrijver] 1998). y By encoding the LSTM's linear, piece-wise wise output layers into the MII.P, we directly optimize an action trajectory that best ensures the safety of the system while also maximizing the information learned about the system. In this paper, we describe our novel architecture which uniquely combines the power of learned acquisition function with chance-constrained optimization and evaluate its performance against state-of-the-art baselines in several relevant domains. To the best of our knowledge, this is the only architecture which meta-learns an acquisition function for optimization tasks and is capable of embedding this acquisition function in a ftance-constraing linear program to guarantee a minimum level of safe operation.
Contributions
The contributions of this paper as follows:
1. Meta-active learning for autonomously synthesizing an acquisition function to efficiently infer altered or unknown system dynamics and optimize system parameters.
2. Probabilistically-saff control combined with an active-learning framework through the integration of our deep learning- based acquisition function and integer linear programming.
3. State-of-the art results for safe, active learning. We achieve a 46% increase in information gain in nigh-diimensioal environment of controlling damaged aircraft, and we achieve 58% increase in information gain in our deep brain stimulation against our baselines.
2 PRELIMINARIES
In this section, we review the foundations of our work in active, meta-, and reinforcement learning. Active Learning Labelled training data is often difficult to obtain due either to tight time con- straints or lack of expert resources. Active learning attempts to address this problem by utilizing an "acquisition function" " to quantify the amount of information an unlabelled training sample, x E Du = (z,) would provide a base learner, T if that sample were given a label, y and added to labeled dataset, Dı (rrj M5) y5) i.e., DL Dr U (xx, v)- The active learning algorithm queries its acquisition function, H (Du,, Dr., TT), ), select which added to Dr. then, a label is queried (e.g., by taking an action in an environment and observing the effect) for æ. and the new labeled sample is added Dr (Muslea al., 2006 Pang et al. 2018).
Dr should be labeled and
Meta- Meta-Learning - Meta-learning approaches attempt to learn a method to quickly adapt to new tasks online. In contrast to active learning, meta-learning attempts to learn skill learning method, e.g. learning an active learning function, which can be transferable to novel tasks or scenarios. These tasks or skills are trained offline, and a common assumption is that the tasks selected at test time are drawn from the same distribution used for training (Hospedales et al., 2020).
Reinforcement Learning and Q-Learning - A Markov decision process (MDP) is a stochastic control process for decision making and can be defined by the 5-tuple (X.U, T. 7). X' represents the set of states and the set of actions. T : X' returns the probability of transitioning to state x' from state applying action, R R: x U - R is a reward function that maps a state and action to a reward, and 7 weights the discounting of future rewards. Reinforcement learning seeks to synthesize a policy, π : X + U, mapping states to actions to maximize the future expected reward. When π is the optimal policy, R., the following Bellman condition holds: QT (z,u) == E R(x,, u) + 7Q= (a', n (x))] (Sutton & Barto 2018).
x X' -> [0, 1] is the transition function that
2.1 PROBLEM SET-UP
Our work is at the unique nexus of active learning, meta-learning and deep reinforcement learning with the objective of learning the Q-function as an acquisition function to describe the expected future information gained when taking action, u, in state, x, given a of previously experienced states
2

---

Under review as a conference paper at ICLR 2021
siguerr A (0.00"11,2in
- - Prfiue
Action, (7
Dynamics Mocel (7.)
(1.0000
Gommeiee
Fully Cunnected Reba
Fully Connected
2. (.
2
Ganreciaal
Reward
[(())))
MILP
Agg A8
Statistics
10 (rig gorn) (t gn))
Encoding
State, gies
g (7
Figure 1: This figure depicts our framework. Previously collected state-action pairs are fed into an LSTM embedding layer. Mean and variance statistics are also calculated for each state and action.
The concatenated vector is fed through fully connected layer to create the Z embedding. Our Q-function consists of two, fully-connected, ReLU layers. All blue elements are neural network layers of our acquisition function; the green element is the dynamics model; and the components of the MILP are housed the yellow box.
and actions. define information gain as the percent decrease the error of the objective (e.g., decrease in model error). A formal definition is provided in the Appendix. The Q-function is trained via a meta-lcarning strategy in which an agent interacts in environments sampled from a distribution of different scenarios to distill the most effective acquisition function for active Iearning of system dynamics. In context, the state, 3, of the active learning system is given by X (Du, DL, T&), where Dr consists of all possible state-action pairs, D,, is the set state-action pairs that the agent has already experienced, and T is neural network function approximator of the transition dynamics, which is parameterized by v and updated online as samples are collected. We note that this state, X. controlling, The reward, R(), is
distinct from the state, X, of the robotic (or other) system proportional the reduction of the mean squared error (MSE) loss of T at time step, i.
SAFE META-LEARNING ARCHITECTURE
Several key components are vital for learning about an unknown system in a timely manner. First, encoded representation of the context of the new dynamics is important for determining where exploration should be focused and which actions may be beneficial for gaining information of unknown dynamics. Second, range of prior experiences in active learning should be leveraged to best inform which actions elicit the most information in a novel context within distribution of tasks. We seek develop framework with these key attributes enable sample-efficient and computationally light- active learning. An overview of our system is shown in Fig. |
3.1 META-LEARNING ALGORITHM
To infer the Q-function for an action (i.e. the acquisition function), we meta-learn over a distribution of altered dynamics as described in Algorithm [ For each episode, we sample from this distribution of altered dynamics and limit each episode to the number of time steps, M, tuned to collect enough data to accurately learn our approximate dynamics, Tw. neural network with parameters, th. We utilize Q-learning to infer the Q-function which is represented by a DQN. our training scheme we search over the action space via MILP, as described in Section 3.2 and select the action set, (7(1:T) which maximizes the Q-function while satisfying safety constraints.
The acquisition Q-function, Qs, as described in Eq. D is trained via Deep Q-Learning (Ganger et al.. 2016) with target network, Qp. During training, our Deep Q-Learning framework is augmented to account for safety. The learned acquisition function, Qo. is utilized by our policy (Eq. []. which solved via a safety-constrained MILP solver. The reward, R(O), for taking a set of actions given state is defined as the percent decrease in the MSE error of the model. T.. This Bellman loss is hackpropagated through the Q-value linear and (ReLU) output layers through the LST encoding
3

---

Under review as a conference paper at ICLR 2021
Figure 2: This figure depicts the d-dimensional hyper-ellipsoid safety. Σ, denotes the safest state for the system and the shaded region the set of all safe states, Action, Ttt), is an exploratory action, which may bring the system outside of the hyper-ellipsoid of safety. Action, z(++1), estimates that the system returns to a safe state with probability c.
layers. m| ((((++)) is the of actions, (7(++)), determined by maximizing Eq. which we describe in Section 3.2 The dynamics model, T. is retrained with each new set of state-action pairs.
Algorithm 1 Meta- Meta-lcarning for training
Algorithm 2 Meta- learning for testing
1: Randomly initialize Qa and Qs with weights 0 =
2: Initialize replay buffer, D
1: Draw test example from dis- tribution
3: for episode=1 to N do
4: Initialize Tg based on meta-learning distribution
Initialize T.
5: Collect small initial set of state-action pairs, (0)), x
3: for t=1 to M do
6: Train f. initial set
4:
Do forward pass through encoder to get Z
7:
for t=1 to do
8:
Forward pass on encoder to obtain Z
Select actions, Z((9 ac-
9:
Select (( from Eq.
cording to Eq.
10:
Execute actions (4 +N according to exploration noise, N₁
6: Execute actions, 24(t); ob-
observe states ((+1)
serve states ((++1) and reward R(())
11:
Retrain T3 on (0::), (((0:)) and observe reward R(())
7: Retrain T. on X (0:41)) (((4411)
D DU _xxx)xxx() (+,xx)))) (())
13:
Sample batch of transitions from D
Perform forward pass through encoder to obtain ((+1)
8: end for
Calculate ( = R(()) 7rQ ( ((Z(++))) 2((+1)
15:
Update Qo according (1 (270), 2(00
16:
17:
Qs - TQe + 7(1 - )o)
18:
end for
19:
end for:
3.2 MIXED-INTEGEA LINEAR PROGRAM
Our objective (Eq. [ is to maximize both the probability of the system remaining in safe configura- tion and the information gained along the finite trajectory horizon from [1, T*).
(0++)))
X1 Q (17 T) (, Qo(. 2)] + PT (( - 2. | sr] (1)
argmax ((
(U((T)) describes the expected information gained along the trajectory when the set of actions (ta--T is taken in the context of the embedding Z, and Qo(-; Z) is the expected Q-value in that can be tuned to adjust
state Z, which we discuss further in Section 3.3 A1 is a the tradeoff between safety and information gain. Pr [ ह, S 5r) defines the probability of remaining within the hyper-ellipsoid of safety. We derive lincarization of this equation based on an assumption of Gaussian dynamics which can then be solved via MILP.
Definition of Safety- We define hyper-ellipsoid of safety, parameterized by the safe state x, and radius 7 encompassing all known safe system states. In the case of an aircraft, H, would be straight and level flight and the hyper-ellipsoid would include deviations considered safe. Additionally,
4

---

Under review as a conference paper at ICLR 2021
assume that our model enror comes from Gaussian distribution with known mean and variance. This assumption allows us to impose safety constraints which can either be enforced with a probability of one, or this requirement can be relaxed to increase the potential for information gain.
By assuming that our model error originates from a Gaussian distribution, we can linearize our probability constraints described in Eq. 9 to include in the MILP (See Appendix for full derivation). Here, Φ- is the inverse cumulative distribution function for the standard normal distribution and 1 Ea denotes the probability level. represents the uncertainty in the dynamies network parameters.
represents the dth row and j represents the 5th column. A (t:++7) = z - [3x(")] where 3 defines the evolution of a state (See Appendix 1.4.1 for more detail). We compute the uncertainty, of our network via bootstrapping (Efron 2020 Franke & Nuemann, 1998), which we describe further in the Appendix. We guarantee minimum probability of safety by selecting the minimum risk value, ed, to be at or above our minimum desired probability of safety. This procedure ensure that our algorithm must select a safety constraint that exceeds this minimum safety requirement. Note: the hyper-ellipse, when linearized, is a hyper- hyper-rectangle.
nff [a(1t)) - 2.11, s 1 10-0011-0000 va),
(1)2 Σ Enason + Eefgfeften +ragitet Alrr-D),, raaad
B
Our policy takes set of information rich actions time t. potentially out of the d-dimensional hyper-ellipsoid of safety, guaranteeing with probability - that the system will return to a safe state after 7 time steps. Thus, our T-step propagation allows the system deviate from the safe region, if desired, to focus on actively learning so long as it can return to the safe region with high confidence.
3.3 AN LSTM Q-FUNCTION AS A LINEAR PROGRAM
We leverage an LSTM function approximator for Q-learning. To blend our Q-function with the MILP, we pass the LSTM output embedding, which serves as a learned representation of the sample history, through fully connected layer with ReLU activations [1 This design enables us to backprop the Bellman residual through the output layers encoded in the MILP all the way to the LSTM inputs. Thus can leverage the power of an LSTM and mathematical programming in a seamless framework Given our meta-learned acquisition function and chance-constrained optimization framework, we can now perform probabilisticalll-sate active learning. Algorithm describes how we perform our online, safe and active learning. Intuitively, our algorithm initializes new dynamics model to represent the unknown or altered dynamics, and we iteratively sample information rich, safe actions via our MILP policy, update our dynamics model, and repeat.
EXPERIMENTAL EVALUATION
We design two experiments to validate our approach's 's ability to safely and actively lear altered or unknown dynamics in real time. We compare our approach to several baseline approaches and demonstrate that our approach is superior in terms of both information gain and computation time. More details on the experimental domains can be found in the Appendix.
4.1 ASYNCHRONOUS DISTRIBUTED MICROELECTRODE THETA
Asynchronous distributed microelectrode theta stimulation (ADMETS) is cutting edge deep brain stimulation approach to treating seizure conditions that cannot be controlled via pharmacological methods. In ADMETS, neuro-stimulator is implanted in the brain to deliver continuous electrical pulses to reduce seizures. However, there is no clear mapping from parameter values to reduction in seizures that applies to all patients, as the optimal parameter settings can depend on the placement of the device, the anatomy of an individual's 's brain and other confounding factors. Further, latent subset of parameters can cause negative side-effects.
We include the mean and standard deviation of previously collected states and actions as we find the first and second moments of the data to be helpful, additional features.
Details on the hyperparameter settings including learning rates and layer sizes can he found in the Appendix along with the derivation for the lincarization of the Q-function for inclusion in the linear program.
5

---

Under review as a conference paper at ICLR 2021
(a) ADMETS Implantation.
(b) High-dinensional Domain: Courtesy: Flightgear.
Figure 3: Fig. 3a depicts surgical ADMETS device implanted in a rat brain (Ashmaig et al. 2018
(Section 4.1 Fig. depicccss the high-dimensional domain for dynamical control (Section 4.2)
As a surrogate task for evaluating how well an algorithm can find the optimal paremeters of an ADMETS device for in vivo epilepsy therapy, researchers have proposed leveraging experimental data collected from parameter sweeps of ADMETS devices in rats. In keeping with Ashmaig et al. (2018), we create simulation environments for six rats where, each ADMETS parameter setting, the cognitive function of a rat was tested (i.e., the rats ability to recall where treat was located in a maze), as measured by a "memory score." The task for safe, active learning task is to determine the ADMETS parameters (i.e., signal amplitude) in the simulation environments that maximize each rat's ss memory scores without causing unwanted side effects (c.g., seizures), which can arise when the memory score drops below zero. The reward signal utilized by meta-learner is the percent decrease error between the predicted and actual optimal parameters.
4.2 HIGH-DIMENSIONAL DOMAIN (RECOVERING DYNAMICS A DAMAGED AIRCRAFT)
Active learning algorithms can be ineffective in high-dimensional domains. As such, seek to stress-test our algorithms in just such a domain: Learning the nonlinear dynamics of a damaged aireraft online before the system enters an unrecoverable configuration (e.g., spin crashing). We base our simulation on theoretical damage models from prior work describing the full equations of motion (Watkiss) 1994 Zhang al., 2017] Ouellette 2010) within the Flightgear virtual environment. The objective of this domain is to learn the difference between the altered dynamics that result from the damage and to maintain safe flight. We consider safe flight to be our designated safe state, Ey. The aircraft takes an information rich action potentially resulting in a deviation outside of the d-dimensional hyper-ellipsole of safety. The next action is constrained to guarantee that the plane returns the hyper-ellipsoid of safety with probability c via action (t-1))
4.3 BASELINE COMPARISONS
We evaluate against the following baselines in active learning and Bayesian optimization. The acquisition function baselines, Epistemic Uncertainty (Hastie et al. 2017) and Maximizing Diversity (Schrum & Gombolay 2020) are embedded in safety framework, therefore providing head-to- head comparison between our meta-learned acquisition function and these active learning heuristics.
Epistemic Uncertainty (Hastie et al. 2017) This active learning metric selects the action that
maximizes uncertainty.
Maximizing Diversity (Schrum & Gombolay 2020 This acquisition function selects actions which maximize the difference between previously l seen states and actions.
Bayesian Optimization (BaO) (Ashmaig et al. 2018) This algorithm was developed for the ADMETS domain (Section 4.1 and is based upon a Gaussian Process model.
Meta Bayesian Optimization (Meta BO) (Wang al., 2018a) This approach meta-learns
Gaussian process prior offline over previously sampled data.
Learning Active Learning (LAL) (Konyushkova et al. 2017) This approach meta-learns an acquisition function leveraging hand-engineered features.
We benchmark against each of the above algorithms in both the ADMETS and high-dimensional domains. We do not compare to MAML (Finn et al., 2017) or Nagabandi et al. (2019) as these algorithms have notion of safety active learning.

---

Under review as a conference paper at ICLR 2021
5 RESULTS
We empirically validate that in the our meta-active learning approach outperforms baselines across both the ADMETS and high-dimensional domains in terms its ability to actively learn latent parameters and its ability to safely perform these tasks. In contrast to our approach, even though BaO, Meta BO, and LAL do not consider safety, we are able to outperform those baselines across all metrics.
Method
Information Usin
3.6
Appronch
1.88
90)
Information Gain
1.4
Ouus MARe
BaO
(108)
(5.06)
0.6
(153)
(1156)
3.0
Staastakey
0.4
Be0
1.0 (098)
(2.85))
0.81
0.2
Diverity
1,12 (158)
W08 (1,66)
0.0
Mar Divenitt
Uneetality
0:01 (211)
976 (1.19)
0.4
10
10'
Time Steps
Computation Time (Timess Worse)
(b)
(c)
Figure 4: This figure depicts our empirical validation in the ADMETS domain, benchmarking algorithm accuracy per time step (Fig. 4ab. overall (Fig. 4bb.. and vs. computation time (Fig. 4c
Active Learning- - Results from both the ADMETS and the high- sith-mimensionlll domains empirically validate that our algorithm more efficiently and safely learns the optimal control parameters (Fig. 4 and system dynamics (Fig. 5 In Fig. 5b report the mean (standard deviation) for each measure and baseline where possible. In the ADMETS domain, our model selects an action which results in 58% higher information gain than BaO and 87% higher information gain than Meta BO on average. When compared to the state-of-the-ar active learning baselines, our method performs 41 % better than Maximizing Diversity and 98% better than Uncertainty terms of information gain.
In the high-dimensional domain, we achieve a 49% improvement over Hastie et al. (2017) and 46% improvement over both (Shruum & Gombolay (2020) and Konyushkova et (2017). We point out that our approach outperforms active learning heuristics (i.c., Maximizing Diversity and Epistemic Uncertainty) when run inside our constriind framework for direct comparison. This result validates meta-learning an acquisition function is a necessary and beneficial component of our framework.
Computation Time We demonstrate that our method also outperforms previous work in terms of computation time. Across both domains, our approach not only achieves more efficient reduction model error and improvement in information gain, we are also faster than all baselines in the more challenging, high-dimensional (Fig. 5ch environment. In the lower-dimensional, ADMETS
1.0
Method
Information Gliin
Safityy
1.4
Uroomaleyy
Our Appeseuc
1.58 (.013)
87% (3.2 30)
1.2
Model
0.6
Meta
BaQ
0.01 (003)
ORE (09))
0.8
0.6
Chars
022
Uiveeity
(111)
(3.566)
0.2
Uisemmtter
Uneurtaisty
(.012)
(4.155)
0.0
Time
Computation Time (Times Worse)
(a)
(b)
(c)
Figure 5: This figure depicts our empirical validation in the high-dimensional domain benchmarking algorithm accuracy per time step (Fig. 5a), overall (Fig. [5b). and vs. computation time (Fig. 5c). Error is calculated batches of three time steps, enabling the robot to deviate from the safe region temporarily to gain information. Our algorithm outperforms baselines in both safety and information gain. We do not report full results for MetaBO in the high-dimensional domain as the high dimensional state space made this baseline computationally intractable.
7

---

Under review as a conference paper at ICLR 2021
environment (Fig. 4c BaO has slight advantage in computation time, but algorithm trades the time for a 58% information gain over BaO. Additionally, we are 68x faster than LAL and 61x faster than Meta BO, the two other meta-leariing approaches we benchmark against.
Optimal Parametar Accuracy
1.0
1.0
Model Accuracy
0.8
0.6
0.6
0.4
0.4
0.2
Ours/Active active learning
0.2
Ours/Active Learning active leaming
0.0
0
6
12
15
Time Steps
Time Steps
(a)
Figure 6: This figure shows the results of our ablation analysis. In the above plots, we set À 0, meaning there is no active learning and only safety maximized in our objective function. Fig. 6a shows the results in the ADMETS domain and Fig. 5b the results in the high-dimensional domain. In both domains, our meta-learned acquisition function is clearly an important component to achieve efficient learning.
Safety - In the high-dimensional domain, we empirically validate that, for the information results we report in Fig. 55 we can achieve an 87% probability the aircraft will return to the safe region [5a] As shown in 5b) only Maximizing Diversity (85% safe) and Epistemic Uncertainty (78% safe) allow for safety constraints be imposed. Our algorithm thus outperforms the baselines in safety as well as information gain. When we maximize for safety (i.e., À 0), our algorithm is able to achieve 99.9% safe return to the hyper-ellipsoid even without ground-truth dynamics model.
In the ADMETS domain, we find that our algorithm achieves 2.3% higher guarantee of safety compared to Maximizing Diversity. Our baseline Uncertainty achieves an equivalent safety guarantee to our algorithm, yet algorithm achieves 98% greater information gain comparatively. placing at the Pareto front in terms of safety and information gain. We outperform LAL, BaO, and MetaBo in information gain and computation time under our safety constraints though these algorithms unconstrained.
We show in 6 the results of our algorithm in both domains when maximizing only safety and removing the active learning component from our objective. Without the meta-learned active learning function, our algorithm does explore efficiently and therefore achieves lower model accuracy at each time step, demonstrating that our meta-learned acquisition function is an important component of our architecture.
5.1 DISCUSSION
Through our empirical investigation, we have demonstrated that meta-learned acquisition function operating within a chance-constrained optimization framework outperforms prior work in active learning, meta-learning, and Bayesian optimization (Hastie al. 2017 Schrum Gombolay 2020) Ashmaig et al., 2018: Konyushkova et al., 2017: Wang et al., 2018a). Specifically, we are able to simultaneously achieve an improvement in information gain via increased sample efficiency and decreased computation time. We achieve a 46% increase in information gain while still achieving a 20% speedup computation compared to active learning baselines and 60x faster computation time compared to our meta learning baseline. Our novel, deep learning architecture, demonstrates unique ability to leverage the power of feature learning across time-series data within a LSTM neural network and the utility of deep Q-learning within mathematical optimization with chance constraints to explicitly tradeoff safety and active learning. Our approach allows for minimum safety guarantees while also maximizing the information gained based learned representation of sample history.
8

---

Under review as a conference paper at ICLR 2021
6 RELATED WORK
Our work lies at the crossroads of active learning, meta-learning and safe learning. We discuss the contributions of our work and why our approach is novel in light of prior work.
Active Learning- Active learning acquisition functions provide heuristics to selecting the candidate unlabeled training data sample that, if the label were known, would provide the most information to
the model being learned (Burbidge et al. 20077] Hasenjager & Ritter 1998 Cai ct al.) 2017 Hastic
ct al., 2017). For example, in Hastie et al. (2017) the action is selected that the learner is least
certain about. In work by [Ashmagg et al. 2018), the authors utilize the acquisition function Expected Improvement (EI) to balance exploration versus exploitation to determine the optimal stimulation parameters in ADMETS.
Prior literature has also investigated on-the-fly active learning and meta-active learning (Bachman et al. 2016 Konyushkova et al. 2017). Konyushkova et al. (017) describes the algorithm Learning Active Learning (LAL). The authors present a meta-learning method for learning an acquisition function in which regressor is trained to predict the reduction in model error of candidate samples via hand engineered features. [Wang et al. (2)18a) alternatively considers a Gaussian Process based acquisition function on a distribution of tasks. They show that their method is capable of extracting structural properties of the objective function for improved data- data-efficiency. Work by Geifman & El- Yaniv (2018) attempts to actively learn the neural network architecture that is most appropriate for a given task, e.g. active learning. Pang et al. (2018) additionally proposed a method to learn an acquisition function that generalizes a variety of classification tasks. Yet, this work has only been demonstrated for classification.
method to meta- meta-train
Meta-Lea for Dynamies- Prior work has attempted address the problem of learning altered dynamics via meta-learning (Clavera et al. 2009). [elkhale et al. (2020) investigate meta-learning approach learn the altered dynamics of an aircraft carrying a payload; the authors train a neural network on prior data to predict environmental and task factors to inform how to adapt to new payloads. Finn et al. (2018) present a meta-learning approach to quickly learning control policy. In this approach, distribution over prior model parameters that are most conducive to learning the new dynamics is meta-learned offline. While this approach provides fast policies for learning new dynamics, it does not explicitly reason about sample efficiency safety.
Safe Learning- Prior work has investigated safe learning in the context safe Bayesian optimization and safe reinforcement learning. For example, (Si et al. (2015) develop the algorithm SafeOpt which balances exploration and exploitation to learn an unknown function; however, this approach has significant limiting assumptions about the underlying nature of the task. Turchetta et al. (2016) address the problem of safely exploring an MDP by defining an a priori unknown safety constraint updated during exploration, and al. (2018) utilizes Gaussian process for safely learning time series data. However, these approaches do not incorporate knowledge from prior data to increase sample efficiency, limiting their ability to choose the optimal action. [Schrum& & Gombolay (2020) attempt to overcome this problem by employing novel acquisition function, Maximizing Diversity. which is utilized to quickly learn altered system dynamics in chance constrained framework. Yet. the hand engineered acquisition function limits the capabilities of this approach.
7 CONCLUSION
In this paper we demonstrate state of the art meta-learning approach to active learning for control. By encoding the context of the dynamics via an LSTM and learning a Q-function - which we encode into a mixed-integer optimization framework - that predicts the expected information gain of taking a given action, our framework is able to efficiently and safely learn the nature of the altered dynamics. We compare our approach to baseline acquisition functions and demonstrate that ours is superior in both computation time and information gain, achieving 46% increase in information gain while still achieving a 20% speedup in computation time compared to state-of of-the-art acquisition functions and more than a 58% higher information compared to Bayesian approaches.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Marcin Andrychowicz. Misha Denil, Sergio Gómez Colmenarejo, and Matthew W Hoffmaan. Learning to learn by gradient descent by gradient descent. (Conference on Neural Information Processing Systems (Conference on Neural Information Processing Systems (Nips)): 17, 2016.
Omer Ashmaig, Mark Connolly, Robert E. Gross, and Babak Mahmoudi. Bayesian Optimization of Asynchronous Distributed Microelectrode Theta Stimulation and Spatial Memory. Proceedings of the Anmual Interaational Conference of the IEEE Engineering in Medicine and Biology Sociery, EMBS, 2018-July 1118July:268326666 2018. ISSN 1557170X. doi: 10.1 109/EMBC... 0.0001888.2280 8888110..
Philip Bachman, Alessandro Sordoni, and Adam Trischler. Learning Algorithms for Active Learning. 2016.
Suneel Belkhale, Rachel Li, Gregory Kahn, Rowan McAllister, Roberto Calandra, and Sergey Levine. Model-Based Meta-Reinffrremmen Learning for Flight with Suspended Payloads. 2020. URL
http://arxiv. www.:/ggggaaa.boe http///wwww.orr/bb/////0000-0000
11345
Robert Burbidge, Jem Rowland, and Ross D. King. Active Learning for Regression Based on
Query by Committee. Intelligent Data Engineering and Automated Learning IDEAL 2007, pp.
209-218, 2007.
doi: 10. 10.1007/978-3-540-775556
Wenbin Cai, Muhan Zhang, and Ya Zhang. Batch mode active learning for regression with expected model change. IEEE Transactions on Neural Networks and Learning Systems, 28(7):1668-168)
2017. ISSN 21622388. doi: 10.1
Ignasi Clavera, Anusha Nagabandi, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to Adapt: Meta-Learning for Hodel-Based Control. 2009.
Efron. Bootstrap Methods: Another Look at the Jackknife Author(s): B. Efron Source The Annals of Statistics Vol 7 No 1 Jan 1979 pp 1-26 Published by Institute of Mathematical Statistics Stable URL http::/wwwwissorrr//sss//////ss
7(1):1-26, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agmostic Meta-Learning for Fast Adaptation of Deep Networks. 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic Model-Agnostic Meta-Learning, (Conference on Neural Information Processing Systems (NeurIPS)), 2018
Jurgen Franke and Michael A Nuemann. Bootstrapping Neural Networks. Neural Computation, 12 (8):1929-1949, 1998.
Michael Ganger, Ethan Duryea, and Wei Hu. Double sarsa and double expected sarsa with shallow and deep learning. Journal of Data Analysis and Information Processing, 4(4):159-176, 2016.
Yonatan Geifman and Ran El- Ya-ya. Deep Active Learning with a Neural Architecture Search. (NcurIPS), 2018. URL
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with Istm. 1999.
Martina Hasenjager and Helge Ritter. Active Learning with Local Models Introduction. Neural Processing Letters, pp. 107-117, 1998.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. Model Assessment and Selection. In The Elements of Statistical Learning Data Mining, Inference, and Prediction, pp. 249-252. 2017.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-Leamnig in Neural Networks: A Survey. pp. 1-23, 2020. URL httt:://aaxiii http://axxvv...g//b////a// wwtp////////////////////// 2004. 05439
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. BatchBALD: Efficient and Diverse Batch
Acquisition for Deep Bayesian Active Learning. (NeurIPS), 2019. URL abs/ 1906. 08158
10

---

Under review as a conference paper at ICLR 2021
Ksenia Konyushkova, Sznitman Raphael, and Pascal Fua. Learning Active Learning from Data. (Conference on Neural Information Processing Systems (Nips)), 2017.
Ion Muslea, Steven Minton, and Craig A. Knoblock. Active learning with multiple views. Journal of Artificial Intelligence Research, 27:203-233 2006. ISSN 10769757. doi: 10.1613/jair.205.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Picter Abbecl, Sergey Levine, and Chelsea Finn. Learning to Adapt in Dynamics, Real-World Environments Through Meta- Reinforcement Learning. pp. 1-17, 2019.
Jeffrey A. Ouellette. Flight Dynamics and Maneuver Loads on a Commercial Aircraft with Discrete Source Damage. Thesis Master, Aerospace Engineering, 2010.
Kunkun Pang. Mingzhi Dong, Yang Wu, and Timothy Hospedales. Meta-Learning Transferable Active Learning Policies by Deep Reinforcement Learning. pp. 1-8, 2018. URL http:/7
04798
Sotirios Posporelis, Anthony S David, Keyoumars Ashkan, and Paul Shotbolt. Deep brain stimulation of the memory cireuit: improving cognition alzheimer's disease. Journal of Alzheimer's Disease, 64(():337-347, 2018.
Alexander Schrijver. Theory of linear and integer programmiing. John Wiley & Sons, 1998.
Mariah Schrum and Matthew Gommolay. When Your Robot Breaks Active Learning During Plant Failure. IEEE Robotics and Automation Letters, 5(2):438-444 2020. doi: 10. 109/LRA. 2019.2961598.
Yanan Sui, Alkis Gotovos, Joel W. Burdick, and Andreas Krause. Safe exploration for optimization with Gaussian processes. 32nd International Conference on Machine Learning ICML 2015, 2: 997-1005, 2015.
Yanan Sui, Joel Burdick, Yisong Yue, et al. Stagewise safe bayesian optimization with gaussian processes. In International Conference on Machine Learning, pp.
2018.
Richard S. Sutton and Andrew G. Barto. Reinforcementt Learning: An Introduction. The MIT Press, second edition, 2018.
html
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite Markov decision processes with Gaussian processes. Advances in Neural Information Processing Systems, (Nips): 4312-4320, 2016. ISSN 10495258.
JX Wang, Z Kurth-Nelson, D Turumala, and H Soyer. Learning reinforcement learn. pp. 1-17, 2016.
Li Wang, Evangelos A Theodorou, and Magnus Egerstedt. Safe Learning of Quadrotor Dynamics Using Barrier Certificates. 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2460-2465, 2018a.
Zi Wang, Beomjoon Kim, and Leslie Pack Kaelbling. Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior. Advances in Neural Information Processing Systems, 2018 1eBBDcmmbeee/eePPP:::000111-5550 2018b. ISSN 10495258.
Eric John Watkiss.
an unmanned aerial vehicle, 1994. URL https::/caat
nps. hat.eeee//aandlle/00555///25
Y. Zhang, C. C. de Visser, and Q. P. Chu. Aircraft Damage Identification and Classification for Database-Driven Online Flihht-Envelope Prediction. Journal of Guidance, Control, and Dynamics, 41(2):1-12, 2017. ISSN 0731-5090. doi: 10.2514/1.G00286
Christoph Zimmer, Mona Meister, and Duy Nguyen-Tuong, Safe active learning for time-series modeling with Gaussian processes. Advances in Neural Information Processing Systems, 2018 8- DecemberNNeu1P9):22300 2018. ISSN 10495258.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 DETAILS ON DOMAINS
Here describe in more detail the input and output spaces of the ADMETS and high-dimensional domains as well as other details relevant to implementation.
A.1.1 AIRCRAFT
Our simulation consists of ten states, i.e., forward velocity (u), vertical velocity (w), pitch rate (g), pitch angle (0), sideslip angle (3), roll rate (p), yaw rate (r), roll angle (1)), yaw angle (6), and altitude (Z). The control inputs are elevator (A,). thrust (A (1)), aileron (A,), and rudder (A,). Our sampling rate is 20 Hz.
A.1.2 ADMETS
The input parameter space is one-dimensional and consists of the voltage amplitude (va). The output space score quantifying memory, referred the discrimination score (dg). The objective of this domain is to maximize discrimination area, which is defined as du := dy Va. More detail on this domain and the data on which the simulation based can be found at (Ashmaig et al. (2018).
A.2 DEFINITION OF INFORMATION GAIN
We define information gain, I, at time step as the percent decrease in the error of the objective as described in Eq. 3 (t) is the error of the objective at time step t. In the high-dimensional domain, (t is defined as the mean squared error of the model, (tt) +/ Σi truth state and è is the state predicted by the model. In the ADMETS domain c(t) is defined as the L1 norm of the predicted optimal parameter and the ground truth optimal parameter (e( = ||dg - d 11)..).
(x(( (
2r(0)2 is the ground
During offline training, the ground truth can be obtained from the known model.
1(-11) c(t) (1++1)
(3)
e(t)
A.3 BASELINE ACQUISITION FUNCTIONS
A.3.1 MAXIMIZING DIVERSITY
We compare our method to the acquisition function, maximizing diversity, presented by (Schrum & Gombolay 2020). Here, (()) and (+) are states and actions that the robot has previously experienced, and f current dynamics model.
nn = arginax
Σ
u (3)
+ 3 T (x, u) =001/
(4)
UEU
A.3.2 MAXIMIZING UNCERTAINTY
This active learning metric described by (Hastie et al.) 2017) quantifies the uncertainty in the output of the model for each training example as described in Eq. Here, (u)) is the 2thh dynamics model and T is the average across models
argmax
Σ T- f()))
(5)
WEU Z
A.4 ADDITIONAL DETAILS ON MIXED-INTEGE. LINEAR PROGRAM FORMULATION
In this section we provide additional details on the lincarization of our probability constraints and objective function for integration into linear programming formulation.
12

---

Under review as a conference paper at ICLR 2021
A.4.1 DYNAMICS MODEL REPRESENTATION
In practice, in the high-dimensional domain find that T3 can be represented as single-layer perceptron (i.e. linear regression) which advantageously is computationally efficient in domains that require fast computation times. We adopt multi-layer perceptron with ReLU activations in the ADMETS domain. Our inferred dynamics therefore evolve according r. A describes the evolution of a state with no input, B the change in the state due to an action at time t, t7(), and I is the identity matrix.
X ((+++++)
(z
+ ((7(:)+))
(6)
8 =
[42
A3
A7]
(7)
AB
B
0
0
0
A
AB
0
T=
(8)
AT
B
AT-2B
AT
B
A.4.2 LINEARIZATION OF PROBABILITY CONSTRAINTS
To include our safety constraints in mixed-integer linear programming formulation, we remove the non-lincarities via conservative assumptions and other techniques. Our safety constraints are defined in Eq. and the dynamics evolve according equations g represents the t row and represents the columns.
- ed), Eo (1) + E
Lu (114 + +
- AlactT))
< <r
(9)
The following conservative assumption is made to linearize the sum of squares in Eq.
0
Σ S Σ و4glæzو
(10)
We utilize the binary decision variable, {0, 1} a probability selector variable to linearize the absolute value in Eq. 9 M represents a large positive number and I' is the point estimate of the dynamics. This lincarization technique combined with the conservative assumption Eq. I0 results in the following linear equations (Eq, I3 which can be integrated into mixed-integer linear programming formulation. the set of "probability levels", e.g. E {0.75, 0.S....) where min E defines the minimum enforced probability of safety.
-M8. क -- cp) Σ Gazüititt ((7(4) < F A
(t:+T)
Σ oa
(11)
MM. + $ 101(111111111 6p) Σ (afuūf(itt) (1(4TT < <rd A (ti+T)
Σ
od |1
(12)
Σ 8,y = 1,yde
(13)
PEE
A.4.3 VARIANCE ESTIMATION
We compute uncertainty of our network via bootstrapping. We follow the method proposed in Efron (200)) and randomly redraw bootstrap training samples with replacement from our set of training data. This technique has been verified by Franke & Nuemann (1998 to be effective method for
13

---

Under review as a conference paper at ICLR 2021
approximating the uncertainty of neural networks. The components of a are calculated according to Eq. Here the average the bootstrapped networks and x (b) a single bootstrapped network. d.j
B is the number of bootstrapped networks.
B
[Rd.j
(b) x d.j )
Od
(14)
B 1
A.4.4 LINEARIZATION OF Q-FUNCTION
Our Q-function includes non-linear relu activation function which is linearized to be included in the integer linear programming formulation. Equations T5 17 define the equations for a neural
network with relu activation. < = [(((::)]] 2 the input to the Q-function and (1) W'j the
connection between neurons and between layers 7 and 1.
QQū(::)) Ž) Σ (2) j.d
(2) og. AA D
(15)
0, Σ OWsy 0o (0),20) 44
(16)
10 = Σ ( 5
(17)
This formulation is linearized in Eq. 18 20 represents a large positive number.
h; E {0, is a binary indicator variable and M
Mk. M + Σ W (1)
94 50 Mk; + Σ w0 y
(18)
Σ m w (0) - M < (1)
0, - Σ
w (1) + MMk;
(19)
3
M - MK; (1) 0, 2 0, Vi
(20)
A.4.5 LINEARIZATION OF "RESETTING" TERM
In practice, we find that taking a final, "resetting" action time + T by adding 23 = -1 ( x, to minimize the distance between the aircraft's state and x² helps to ensure the aircraft does not loiter along the boundary of safe operation until where random perturbation could result in failure the aircraft. We linearize the resetting term in our objective function, i.e. the difference between our designated safe state, H, and the final state zr by maximizing -( + + subject to the constraint in Eq. 21 = and = are both positive continuous variables.
-T-I, +
(21)
z zt,² > 0
(22)
A.4.6 LINEARIZED OBJECTIVE
The resultant linearized objective is defined in Eq.
23 In the ADMETS domain A set to
H (11
Q
(2) w 1,, - Q0(, 2))
argmax
j
+ 22 Σ 5 dpp
d2(25 + =x))
(23)
14

---

Under review as a conference paper at ICLR 2021
B SAFETY
B.0.1 ADMETS DOMAIN
The baselines used in the ADMETS domain did not have built safety guarantees. Therefore, when comparing against these baselines, we removed the safety constraints in our algorithm to make the comparison fair.
B.0.2
DOMAIN
Safety constraints are necessary in the sigh-dimensional domain because without these constraints, the aircraft would quickly go into an unrecoverable configuration. The epistemic uncertainty Hastie iet al. (2017) and diversity [Schrum & Gombolay (2020) baselines can be integrated into our safety framework by linearizing the acquisition functions. Therefore, in our comparison of our algorithm versus the baselines, we enforced the safety constraints.
LAL Konyushkova et al. (2017), however, is not inherently safe method of active learning. To fairly compare this baseline to our method in the high-dimensional domain, we simulate the possible actions that can be selected by LAL and discard those that are not safe (i.e., the actions that take the aircraft out of the cylinder of safety). Therefore, LAL can only select an action considered safe by definition of safety.
B.1 SENSITIVITY ANALYSIS
To robustly evaluate our method compared to the baselines, we vary the hyperparameters of the approach by (Schrum Gombolay 2020) for maximizing diversity and our meta-learned function to show that our function is robust and is still superior despite changes in hyperparameters. The results of this hyperparameter sweep are shown in Table [ The hyper-parameter we vary for maximizing diversity is the number of previous training sampies that compare to. The information gain increases as the number of samples increases up to a point at which the selected sample tends to converge to the mean of the previously collected samples, causing the information gain to fall. We vary the number of hidden neurons in our Q-function as the hyper-parameter of interest as it governs the trade-off between computational speed and function approximation power.
In our approach, the addition of a hidden neuron adds an additional integer variable, resulting in an increase in computation time demonstrated in Table T However, an addition of 25 neurons only increase the computation time by 3.7% and provides a 50% increase in information gain. In comparison, (Shhrum Gombolay 2020) 's approach results in increase information gain 4% while trading off 6% loss in computation time. Likewise, increasing the number of bootstrapped models in Hastie et al. (2017)'s 's approach results in 35% increase in information gain and 37.5% increase in computation time. Therefore, in our approach we are able to gain more information without large increases computation time.
Our
Diversity
Uncertainty
Approach
(Shrrum & Gombolay
2020
Hastie et al. 2017)
Parameter Setting
30
3
2
4
Average Information
.196
.39
.29
.30
.31
.26
.26
.22
.25
.33
Gain (SD)
(.37)
(.23)
(.44)
(.30)
(.30)
(.22)
(.22)
(.45)
(.17)
(.17)
Average Computation
.12
.146
.17
.18
.19
.21
0.22
.16
.21
.26
Time (s) (SD)
(003)
(.01)
(.03)
(.04)
(.04)
(.04)
(.03)
(.04)
(.05)
(.05)
Table 1: Average information gain for our approach compared that by (Schrum & Gombolay 2020
and (Hastie et al. 2017). We vary the number of previous samples in the diversity maximization problem from one to five and the number of bootstrapped models from to four in maximizing uncertainty heuristic. We vary the number of hidden neurons in our meta-learner Q-function. We bold the setting of our algorithm that outperforms our baselines across all hyperparameter settings tested.
15

---

Under review as a conference paper at ICLR 2021
B.2 ADDITIONAL RESULTS
We present results for each damage condition in the high-dimensional domain comparing our approach the baseline acquisition functions diversity Schrum Gombolay (202)) and epistemic uncertainty
Hastie et al. (2017). Our approach outperforms the baselines for each damage condition.
Ours
Diversity
Uncertainty
Loss of Stabilizer
.371 (.18)
.207 (.11)
.189 (.09)
Wing Damage
.223 (.14)
.133 (.11)
.200 (.07)
Loss of Aileron
.380 (.12)
.215 (.07)
.312 (.005)
Table 2: We report average information gain and standard deviation for
canonical altered dynamics conditions loss vertical stabilizer, wing damage, and loss of aileron control for the best parameter settings for each approach. Our approach outperforms both heuristics.
B.3 HYPERPARAMETERS
The hyperparameters employed for each domain are listed in the Table 3 We show the learning rates for each domain which were determined via experimentation to be effective values. The size of the LSTT hidden layer and each of the two layers of the Q-function is also presented. T is the soft update coefficient for updating the target network.
ADMETS
Aircraft
Learning Rate
le-4
le-3
LSTM Hidden Layer Size
20
100
Q function Layer Size
64
50
Q function Layer 2 Size
16
30
Soft Update (7)
.001
.001
Exploration noise
Gaussian
Ornstein-Uhlenbedd
Table Hyperparameters for the ADMETS and high-dimensional domains.
16


Reviews:
[{"review_id": "wmSoys-ikPP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a meta-learning approach for active learning in the context of robot control.\n\nWhat I like about this work is that there're several illustrations that make it clear which problem setup it is or what the authors meant by safety. The authors also included several competing methods, and the experiments showed promising results both in terms of several metrics, including information gain and safety. \n\nHowever, I also have some concerns about this paper that needs to be addressed.\n\nIn the experiments, there are 5 methods chosen to be compared against. However, several of them are not even mentioned before or after in introduction or related work to explain their close connections to the proposed one in more details. For example, Schrum & Gombolay 2020 and Wang et al., 2018b. And that makes me doubt whether it's a poor selection of comparisons or omission of important details in writing. \n\nAgain in experiments, the method names described in 4.3 don't match the ones shown in figures. What's Random Bayesian in Figure 4? Why it disappeared in Figure 4(b) and 4(c)? Where's Epistemic Uncertainty and Maximizing Diversity? Similar for Figure 5, it was said that only Meta BO is excluded but where are the others? And even for Meta BO, the paper says it is \"not included as it was designed for optimization tasks\". But why can't the task in sec 4.2 be framed as an optimization task? What I meant is, I would like to see consistent set of methods in different tasks. Or add more tasks to show generality.\n\nThere are also inconsistency in naming. Is it BaO or Bao?\n\nAbout the safety measure. It's evaluating using a point, and all parameters like minimum risk value need to be set by hand. How useful is such a safety measure? And how is it related to more common ways of measuring safety/risks, e.g. CVaR? Also the variables in Eq. (2) are not fully explained.\n\nI'm also somewhat concerned that the authors are trying to solve too many problems in one algorithm. Safety, meta-learning, Q-learning.. But the experiments only show the results of everything combined without detailed analysis of the performance of each functionality while keeping others fixed. To some extent, the nature of the design of the method also sort of forbids modular tests. In the end, it might be just another impractical approach that incrementally adds several factors together, yet it's impossible to explain when it works or why it works.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TRFgf94PeS_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a framework that learns the dynamic of safety-critical systems. The framework makes use of ideas from active learning, meta-learning, and reinforcement learning. As far as I understand, the framework seeks to learn an acquisition function that can evaluate the information gain from a certain action taken at a certain state. Acquisition function is a function that identifies an action the agent desires to take from a collection of unvisited states. The learning of such an acquisition function is formulated as a reinforcement learning problem, trained in a meta-learning fashion. I have the following comments:\n\n1. The proposed method is evaluated on two problems, ADMETS and damaged aircraft.  Both problems are solved using synthetic data. While it is understandable that the framework is trained on simulated data, is there any justification with respect to the quality of the data simulator used in these two problems? Why performing well on these simulated data would be an indication of the proposed method can work reasonably well in practice?\n\n2. Since the proposed framework is trained in a meta-learning fashion, is there any quantification on how fast/well (e.g. wrt. sample complexity) the trained policy adapting to new environments in test time?\n\n3. While the paper takes safety during the learning of the dynamics as a critical issue, is there any justification with respect to the percentage of safety achieved by the proposed method? For example, why an 87% safety reported in Figure 5 achieved by the proposed method is a good number?\n\n4. The key components of the proposed framework (e.g. meta-learning, reinforcement learning, active learning, safety) make sense to address the technical challenges in learning system dynamics in safety-critical domains. It is not very clear to me if the novelty of the proposed framework is appropriately highlighted. The paper can benefit from a better highlight of the novelty of the proposed method and why such novel contributions matter.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CwbnpiFFA0o", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Paper summary:\n\nIn this work, a meta active learning approach is proposed to learn the hidden dynamics of control systems, where safety is also a major concern. The main idea lies in doing meta-learning with Q-learning, meanwhile selecting safe actions by solving a mixed-integer linear programming problem. The performance of the proposed approach is verified from real datasets of deep brain stimulation by outperforming existing baselines for a significant gap in both accuracy and computational complexity.\n\n#### Advantages:\n\n- The paper discusses a very meaningful problem. Active learning with safety constraints is very important in many applicational domains.\n\n- From the experiments, the proposed approach shows very significant performance in both the deep brain stimulation and the dynamics recovering tasks.\n\n#### Disadvantages:\n\n- From the perspective of machine learning, the proposed approach is a combination of existing techniques, thus the technical novelty is limited. \n\n- The paper is more about solving a specific robotics problem, which may not lie in the major focus of the ICLR conference.\n\nOverall evaluation:\n\nOn one hand, I think the paper discusses a very meaningful problem, while on the other hand, I think ICLR may not be the most suitable venue to publish this work.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CQl8tnESLU6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper presents a meta-active learning approach to obtain an LSTM-based embedding of a dynamic system and use a chance-constrained (probabilistic safe) optimization to find optimal control configuration via applying mixed-inter linear programming (MILP) on the learned embeddings of the dynamic system. The main idea is to learn a Q-function as an acquisition function to describe the future information gain, which is the percent decrease in the error of the objective (e.g. model error) via a meta-learning strategy in which the agent interacts with the environment via distribution of altered dynamics. \n\nStrength: \n-  To my best knowledge, the idea that combines an LSTM embedding of dynamics with mixed-integer programming for optimization is novel meta-active learning algorithms. \nWeakness: \n- The algorithm relies on the assumption that the safety region must be hyper-ellipsoid with a known radius and the model error must come from a Gaussian distribution with known mean and variance. This makes the algorithm hard to apply for problems when the safety region is unknown or changing over time, which happens often in most real-world problems. \n- The poor writings make it hard to access the correctness of the technical details. \n\nQuestions and comments: \n1) In Algorithm 1 line 11, should the subscript “T” be “t”? should the index “i” starting at line 7 be “t” and all the related subscript “i” be “t” instead?  \n2) In Eq (1), should the subscript of U be “t:t+T” instead of “t:T”? If keeping “t:T”, the sentence above Eq (1) that states “the trajectory from [t, t+T)” should be “[t, T]”. Alternatively, the author could formally define what “t:T” means. Also, in the first paragraph of section 3.2, it’ll flow better if the authors could give some description of the second part on the right-hand side of Eq (1). \n3) Eq (2) is a bit hard to understand. What is \\Delta_d^{t:2}? How is it set in practice? On the fourth line of the first paragraph on page 5, the paper states “and \\bar{a} and \\bar{b} are the point estimates of dynamics”, what is “\\bar{a}” and “\\bar{b}”?  The next sentence “d represents the d-th row and j the columns” is confusing. \n4) Eq (2), the first term under the square root has x_j^{(t)}^2, should the underscript “j” be “d’” instead? \n4) In the ADMETS experiment, the objective is defined as the L1 norm of the predicted optimal parameter and the ground truth optimal parameter;  how is the ground truth optimal parameter obtained? If the ground truth optimal parameter is already available, what’s the need for running the algorithm since the goal is to find the optimal parameters of the configurations? \n5) In the Aircraft experiment, the objective function is defined as the mean squared error of the model (see Appendix A.2), what is the model here? What is \\hat{x}_i and x_i? \n6) In Appendix (A.4.2), what is the E in equation 13? \n7) Page 8, “Meta-learning for Dynamics”, line 6, “… most conducive …” should be “… most conductive …”, line 7, “While this approach provide .. “ should be “While this approach provides …” \n8) Eq (20) becomes 0 >= \\prescript {o_i}  >= 1 when k_i =1, this will never be satisfied. \n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was quite contentious.  While reviewers appreciated the detailed response by the authors, and there is consensus that the paper addresses a relevant problem and contains interesting ideas, in the end there remain several concerns.  The paper provides a complex combination of techniques from active learning, meta learning and symbolic reasoning (via MILPs), and there are concerns about the clarity of the exposition.  For a paper claiming safety properties, there is also a lack of either formal theoretical analysis of well-specified safety properties, or a compelling demonstration of its effectiveness on a real system (all experiments are carried out in simulation).
Decision: Reject