Meta-Active Learning in Probabilistically-Safe Optimization

1 INTRODUCTION . Safe and efficient control of a novel systems with latent dynamics is an important objective in domains from healthcare to robotics . In healthcare , deep brain stimulation devices implanted in the brain can improve memory deficits in patients with Alzheimers ( Posporelis et al. , 2018 ) and responsive neurostimulators ( RNS ) can counter epileptiform activity to mitigate seizures . Yet , the surgeon ’ s trial-and-error process of finding effective RNS parameters for each patient is time-consuming and risky , with poor device settings possibly damaging the brain . Researchers studying active learning and Bayesian optimization have sought to develop algorithms to efficiently and safely learn a systems ’ dynamics , e.g . learning a brain ’ s dynamics for RNS configuration ( Ashmaig et al. , 2018 ; Sui et al. , 2018 ) . However , because these algorithms fail to scale up to higher-dimensional state-action spaces , researchers utilize only simple voltage and frequency controls rather than all 32 channels of the RNS waveform ( Ashmaig et al. , 2018 ) . Similarly , tasks in robotics , e.g . learning the dynamics of novel robotic systems ( e.g. , an autopilot learning to fly a damaged aircraft ) , require active learning methods that succeed in higher-dimensional domains . In this paper , we develop a probabilistically-safe , meta-active learning approach to tackle these challenging tasks to efficiently learn system dynamics and optimal configurations . We draw inspiration from recent contributions in meta-learning ( Finn et al. , 2017 ; Nagabandi et al. , 2019 ; Wang et al. , 2016 ; Andrychowicz et al. , 2016 ) that seek to leverage a distribution over training tasks to optimize the parameters of a neural network for efficient , online adaptation . Researchers have previously investigated meta-learning for active learning , e.g . learning a Bayesian prior over a Gaussian Process ( Wang et al. , 2018b ) for learning an acquisition function . However , these approaches do not consider the important problem of safely and actively learning to control a system with altered dynamics , which is a requirement for safety-critical robotic applications . Furthermore , as we show in Section 5 , on challenging control tasks for healthcare and robotics , the performance of prior active learning approaches ( Kirsch et al. , 2019 ; Hastie et al. , 2017 ) leaves much to be desired . We seek to overcome these key limitations of prior work by harnessing the power of meta-learning for active learning in a chance-constrained optimization framework for safe , online adaptation by encoding a learned representation of sample history . Instead of hand-engineering an acquisition function for our specific domains , our approach employs a data-drive , meta-learning approach , which results in better performance than prior approaches , as shown in Section 5 . Furthermore , our approach has the unique ability to impose analytical safety constraints over a sample trajectory . Contributions – We develop a probabilistically safe , meta-learning approach for active learning ( ” meta-active learning ” ) that sets a new state-of-the-art . Our acquisition function ( i.e. , the function that predicts the expected information gain of a data point ) is meta-learned offline , allowing the policy to benefit from past experience and provide a more robust measure of the value of an action . The key to our approach is a novel interweaving of our deep , meta-learned acquisition function as a Long-Short Term Memory Network ( Gers et al. , 1999 ) ( LSTM ) within a chance-constrained , mixed-integer linear program ( MILP ) ( Schrijver , 1998 ) . By encoding the LSTM ’ s linear , piece-wise output layers into the MILP , we directly optimize an action trajectory that best ensures the safety of the system while also maximizing the information learned about the system . In this paper , we describe our novel architecture which uniquely combines the power of a learned acquisition function with chance-constrained optimization and evaluate its performance against state-of-the-art baselines in several relevant domains . To the best of our knowledge , this is the only architecture which meta-learns an acquisition function for optimization tasks and is capable of embedding this acquisition function in a chance-constrained linear program to guarantee a minimum level of safe operation . The contributions of this paper are as follows : 1 . Meta-active learning for autonomously synthesizing an acquisition function to efficiently infer altered or unknown system dynamics and optimize system parameters . 2 . Probabilistically-safe control combined with an active-learning framework through the integration of our deep learning-based acquisition function and integer linear programming . 3 . State-of-the art results for safe , active learning . We achieve a 46 % increase in information gain in a high-dimensional environment of controlling a damaged aircraft , and we achieve a 58 % increase in information gain in our deep brain stimulation against our baselines . 2 PRELIMINARIES . In this section , we review the foundations of our work in active , meta- , and reinforcement learning . Active Learning – Labelled training data is often difficult to obtain due either to tight time constraints or lack of expert resources . Active learning attempts to address this problem by utilizing an “ acquisition function ” to quantify the amount of information an unlabelled training sample , x ∈ DU = 〈xi〉ni=1 , would provide a base learner , T̂ψ , if that sample were given a label , y and added to a labeled dataset , DL = 〈xj , yj〉mj=1 , i.e. , DL ← DL ∪ 〈x , y〉 . The active learning algorithm queries its acquisition function , H ( DU , DL , Tψ ) , to select which x ∈ DU should be labeled and added to DL ; then , a label is queried ( e.g. , by taking an action in an environment and observing the effect ) for x , and the new labeled sample is added to DL ( Muslea et al. , 2006 ; Pang et al. , 2018 ) . Meta-Learning – Meta-learning approaches attempt to learn a method to quickly adapt to new tasks online . In contrast to active learning , meta-learning attempts to learn a skill or learning method , e.g . learning an active learning function , which can be transferable to novel tasks or scenarios . These tasks or skills are trained offline , and a common assumption is that the tasks selected at test time are drawn from the same distribution used for training ( Hospedales et al. , 2020 ) . Reinforcement Learning and Q-Learning – A Markov decision process ( MDP ) is a stochastic control process for decision making and can be defined by the 5-tuple 〈X , U , T , R , γ〉 . X represents the set of states and U the set of actions . T : X × U × X ′ → [ 0 , 1 ] is the transition function that returns the probability of transitioning to state x′ from state x applying action , u. R : X × U → R is a reward function that maps a state and action to a reward , and γ weights the discounting of future rewards . Reinforcement learning seeks to synthesize a policy , π : X → U , mapping states to actions to maximize the future expected reward . When π is the optimal policy , π∗ , the following Bellman condition holds : Qπ ∗ ( x , u ) : = Ex′∼T [ R ( x , u ) + γQπ ∗ ( x′ , π∗ ( x ) ) ] ( Sutton & Barto , 2018 ) . 2.1 PROBLEM SET-UP . Our work is at the unique nexus of active learning , meta-learning and deep reinforcement learning with the objective of learning the Q-function as an acquisition function to describe the expected future information gained when taking action , u , in state , x , given a set of previously experienced states and actions . We define information gain as the percent decrease in the error of the objective ( e.g. , decrease in model error ) . A formal definition is provided in the Appendix . The Q-function is trained via a meta-learning strategy in which an agent interacts in environments sampled from a distribution of different scenarios to distill the most effective acquisition function for active learning of system dynamics . In our context , the state , X , of the active learning system is given by X : 〈DU , DL , T̂ψ〉 , where DU consists of all possible state-action pairs , DL is the set of state-action pairs that the agent has already experienced , and T̂ψ is a neural network function approximator of the transition dynamics , which is parameterized by ψ and updated online as samples are collected . We note that this state , X , is distinct from the state , X , of the robotic ( or other ) system we are controlling . The reward , R ( i ) , is proportional to the reduction of the mean squared error ( MSE ) loss of T̂ψ at time step , i . 3 SAFE META-LEARNING ARCHITECTURE . Several key components are vital for learning about an unknown system in a timely manner . First , an encoded representation of the context of the new dynamics is important for determining where exploration should be focused and which actions may be beneficial for gaining information of unknown dynamics . Second , a range of prior experiences in active learning should be leveraged to best inform which actions elicit the most information in a novel context within a distribution of tasks . We seek to develop a framework with these key attributes to enable sample-efficient and computationally light-weight active learning . An overview of our system is shown in Fig . 1 . 3.1 META-LEARNING ALGORITHM . To infer the Q-function for an action ( i.e . the acquisition function ) , we meta-learn over a distribution of altered dynamics as described in Algorithm 1 . For each episode , we sample from this distribution of altered dynamics and limit each episode to the number of time steps , M , tuned to collect enough data to accurately learn our approximate dynamics , T̂ψ , as a neural network with parameters , ψ . We utilize Q-learning to infer the Q-function which is represented by a DQN . In our training scheme we search over the action space via a MILP , as described in Section 3.2 and select the action set , ~U ( t : T ) , which maximizes the Q-function while satisfying safety constraints . The acquisition Q-function , Qθ , as described in Eq . 1 is trained via Deep Q-Learning ( Ganger et al. , 2016 ) with target network , Qφ . During training , our Deep Q-Learning framework is augmented to account for safety . The learned acquisition function , Qθ , is utilized by our policy ( Eq . 1 ) , which is solved via a safety-constrained MILP solver . The reward , R ( t ) , for taking a set of actions in a given state is defined as the percent decrease in the MSE error of the model , T̂ψ . This Bellman loss is backpropagated through the Q-value linear and ( ReLU ) output layers through the LSTM encoding layers . π ( ~Z ( t+1 ) ) is the set of actions , ~U ( t+1 ) , determined by maximizing Eq . 1 , which we describe in Section 3.2 . The dynamics model , T̂ψ , is retrained with each new set of state-action pairs . Algorithm 1 Meta-learning for training 1 : Randomly initialize Qθ and Qφ with weights θ = φ 2 : Initialize replay buffer , D 3 : for episode=1 to N do 4 : Initialize T̂ψ based on meta-learning distribution 5 : Collect small initial set of state-action pairs , U ( 0 ) , X ( 0 ) 6 : Train T̂ψ on initial set 7 : for t=1 to M do 8 : Forward pass on encoder to obtain ~Z ( t ) 9 : Select U ( t ) from Eq . 1 10 : Execute actions U ( t ) + N according to exploration noise , N ; observe states X ( t+1 ) 11 : Retrain T̂ψ on X ( 0 : t ) , U ( 0 : t ) and observe reward R ( t ) 12 : D ← D ∪ 〈U ( t ) , X ( t ) , X ( t+1 ) , R ( t ) 〉 13 : Sample a batch of transitions from D 14 : Perform forward pass through encoder to obtain ~Z ( t+1 ) 15 : Calculate y ( t ) = R ( t ) + γQφ ( π ( ~Z ( t+1 ) ) , ~Z ( t+1 ) ) 16 : Update Qθ according to ( y ( t ) −Qθ ( ~U ( t ) , ~Z ( t ) ) ) 17 : Qφ ← τQθ + τ ( 1−Qφ ) 18 : end for 19 : end for ; Algorithm 2 Metalearning for testing 1 : Draw test example from distribution 2 : Initialize T̂ψ 3 : for t=1 to M do 4 : Do forward pass through encoder to get ~Z ( t ) 5 : Select actions , U ( t ) , according to Eq . 1 6 : Execute actions , U ( t ) ; observe states X ( t+1 ) and reward R ( t ) 7 : Retrain T̂ψ on X ( 0 : t+1 ) , U ( 0 : t+1 ) 8 : end for

Summary: This paper presents a meta-active learning approach to obtain an LSTM-based embedding of a dynamic system and use a chance-constrained (probabilistic safe) optimization to find optimal control configuration via applying mixed-inter linear programming (MILP) on the learned embeddings of the dynamic system. The main idea is to learn a Q-function as an acquisition function to describe the future information gain, which is the percent decrease in the error of the objective (e.g. model error) via a meta-learning strategy in which the agent interacts with the environment via distribution of altered dynamics.