Variational Constrained Reinforcement Learning with Application to Planning at Roundabout

1 INTRODUCTION . The recent progress in model-free reinforcement learning ( RL ) ( Sutton et al. , 1992 ) has produced many interesting results in planning and control problems and proves to be effective in finding optimal policy for nonlinear stochastic systems when the dynamics are either unknown or affected by severe uncertainty ( Buşoniu et al. , 2018 ) , including complicated robotic locomotion and manipulation ( Kumar et al. , 2016 ; Xie et al. , 2019 ; Hwangbo et al. , 2019 ) . To further ensure the safety in control , applying constraints over states and actions is a natural way . Typically , a standard and well-studied formulation for reinforcement learning with constraints are the constrained Markov Decision Process ( CMDP ) framework ( Altman , 1999 ) , where the discounted sum of safety cost should be under certain bounds . As a consequence , discount parameter or bound has to be tuned to ensure safety ( Garcıa & Fernández , 2015 ) . However , in many complicated scenarios when the traffic flow changes frequently , the mathematical description of the obstacles are difficult to obtain explicitly . In this scenario , the constraints are dynamic with uncertainty , which is challenging to define proper constraint . Besides , it is still challenging to tune a cost function with such complicated constraints . Without certain predefined constraints , existed methods fail to find a feasible policy . When humans driving a car , they do not know any specific safety constraint and still perform well . For example , at the very beginning , one is always not sure about the distance between the headstock and wall or other vehicles , which may lead to some accidents . However , with more driving experience , the better awareness of safety and latent constraints will be slowly formed . In this scenario , the constraint is not formulated as any specific distance between obstacles but the feeling according to experience . One of the talents of humans is that humans heavily rely on to make safe decisions is that we could understand the latent information of complicated situation or environment and awareness , whether it is safe or dangerous ( Bubic et al. , 2010 ) . This capability is called situation awareness ( SA ) ( Endsley , 2017 ) . In RL , SA could be regarded as a model that can accurately aware current situation , 1https : //www.dropbox.com/sh/oo6zty99c6tclx1/AAA8RXynrE8K9SYpxzqBhv4Va ? dl=0 internally represent the complex dynamics and covering enough latent information like constraints . Furthermore , it should be able be embedded into RL framework to guide the agent how to safely interact with environment . This paper proposes a novel approach for encoding the measure of safety in scenarios where the explicit safety cost is not available , or the states are interfered by severe uncertainties . The contribution of this paper is in two-folds : 1 ) present a variational-based method to encode the safe and unsafe states ; 2 ) measure the level of safety in latent space with Wasserstein distance , taking the uncertainty in states into consideration . Our framework can be generally combined with various RL algorithms , improving the performance of RL algorithms in terms of safety . Finally , we evaluate our approach in roundabout task with different traffic flow and show that our approach significantly improves the success rate of baseline . 2 RELATED WORK . There are several prior works about the reinforcement learning with safety constraints have been done . Achiam et al . ( 2017 ) proposed a safety constrained policy optimization ( CPO ) approach based on the trust region method , which guarantees the constraint satisfaction with a safe initial policy . Wen & Topcu ( 2018 ) came up with a constrained cross-entropy method for finite CMDP tasks , which an effectively learn feasible policies with respect to constraint satisfaction . Chow et al . ( 2019 ) came up with Lyapunov-based approach and combine with both on-policy and offpolicy reinforcement learning algorithms , which achieves better results in terms of balancing the performance and constraint satisfaction compare with CPO . However , both of the results above need full knowledge of the constraints , and the tasks are almost static environment without uncertainty , rather than a complex dynamic case with many unpredictable obstacles , which is the focus of this paper . Situation awareness is the key to successful decision-making ( Nullmeyer et al. , 2005 ) . Recently , many works on reinforcement learning with situation awareness have been done . Teng & Tan ( 2008 ) proposed TD-FALCON methods that could integrate Context-aware Decision Support ( CaDS ) system and learning for supporting context-aware decision making , where CaDS system exploits contextual information for focused situation assessment and goal-oriented decision support . D ’ Aniello et al . ( 2014 ) proposed Context Space Theory ( CST ) to represent raw data in high-level , domainrelevant concept , namely context attribute , which could identify situations the user is involved in and considering user ’ s situated preferences . Yin et al . ( 2019 ) propose two new metrics CTSa , CTSn to assess the taxi situation , and they found a significant relationship is revealed between the taxi delay and CTSa at Level-1 , and the taxi time and CTSn at Level-2 , which provides strong reference to airport ground movements for control and management purposes . However , all of which are focused on a simple task or static environment . Most importantly , the proposed situation awareness models are all deterministic , which might lose much information in the task with the action-conditioned settings and dynamics uncertainty . 3 PROBLEM FORMULATION . Reinforcement learning with safety constraints tends to be a promising way to offer intelligence and safety simultaneously for autonomous driving . In this paper , we will focus on the decision and control tasks with safety constraints which can be modeled by constrained Markov decision process ( CMDP ) ( Altman , 1999 ) . A CMDP is a tuple , ( X , A , r , P , c , ρ ) , where X is the set of states and A is the set of actions . P ( x′|x , a ) is the transition probability function and ρ ( x ) is the starting state distribution . r ( x , a ) and c ( x , a ) are the reward the constraint function , respectively . In CMDP where the constraints c is not well defined , we aim to simultaneously find a policy π and safety cost ĉ such that max π Eτ∼π [ ∞∑ t=0 γtr ( xt , at ) ] ( 1 ) s.t . Eτ∼π [ ∞∑ t=0 γtĉ ( xt , at ) ] ≤ d̂ ( 2 ) where γ ∈ [ 0 , 1 ) is the discount factor , τ denotes a trajectory ( τ = ( x0 , a0 , s1 , ... ) ) , and τ ∼ π is shorthand for indicating that the distribution over trajectories depends on π : x0 ∼ ρ , at ∼ π ( ·|xt ) , sx+1 ∼ P ( ·|xt , at ) . d̂ is the learned safety threshold . Though the form of c is unknown , we assume that the data set composed of states labeled as safe or dangerous is available , i.e . { ( x1 , s1 ) , ( x2 , s2 ) , ( x3 , s3 ) , . . . } where s ∈ { 0 , 1 } indicates whether the state is safe or dangerous . Note that it is possible the same state x has different labels at different data points , since various actions may be taken . Based on this data set , we aim to learn a model qφ ( z|x ) to map the states to latent space and then construct a safety cost ĉ based on this latent model . 4 MAIN RESULTS . Inference the implicit constraints and awareness of a state to be safe or dangerous are the essential capability for safe decision making . To model the awareness of constraints , the key is to map from physical state space to a latent space where represents the implicit constraints . We proposed a novel method could infer the safety condition as well as the latent constraints . In this section , our approach for inference of the latent constraints will be described in detail . First , in Section 4.1 , a conditional representation model is proposed to map the states into the latent space , where the latent space of safe and dangerous states are separated from each other . Then in Section 4.2 , we employ Wasserstein distance in measuring level of safety of the given state in latent space , i.e . the constructed safety cost ĉ . 4.1 CONDITIONAL REPRESENTATION MODEL . In this part , we propose the conditional representation model ( CRM ) that is capable of mapping the state space to a latent variable space Z . For this purpose , we borrow the encoder-decoder structure from variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ; Doersch , 2016 ) , which provides an efficient approach for approximating posterior distribution of the latent variable z given an observation x . For normal VAE , the variational bound is written as log pθ ( x ) = DKL ( qφ ( z|x ) ‖pθ ( z|x ) ) + L ( x ; θ , φ ) ( 3 ) L ( x ; θ , φ ) = −DKL ( qφ ( z|x ) ‖pθ ( z ) ) + Eqφ [ log pθ ( x|z ) ] ( 4 ) where qφ ( z|x ) is the recognition model , an approximation to the intractable true posterior pθ ( z|x ) . pθ ( x|z ) is the generative conditional distribution and pθ ( z ) is an arbitrary prior distribution . DKL denotes the Kullback-Leibler ( KL ) divergence . Latent Spac Latent Space In our framework , the CRM is not only aimed at encoding , but also contains enough information for measuring how safe or unsafe the current latent is , which is not a concern in classical VAE . In equation 3 , the latent of both safe and dangerous states are sampled from the same prior distribution pθ ( z ) ( as shown in Figure 1 ( a ) ) , resulting in that distributions of latent variable for safe and dangerous states are hardly separable , which brings trouble for measuring the safety in the following step . The data set for CRM is composed of a series of data points containing both the state and its label , { ( x1 , s1 ) , ( x2 , s2 ) , ( x3 , s3 ) , . . . } . To exploit the binary classification label s in the process of latent representation learning , we rewrite the marginal likelihood in Eq . ( 3 ) as Ep ( s ) log pθ ( x|s ) = Ep ( s ) ( DKL ( qφ ( z|x ) ‖pθ ( z|x , s ) ) + L ( x ; θ , φ , s ) ) ( 5 ) L ( x ; θ , φ , s ) = −DKL ( qφ ( z|x ) ‖pθ ( z|s ) ) + Eqφ [ log pθ ( x|z , s ) ] ( 6 ) Here , the objective is to use an encoding model qφ ( z|x ) to approximate the conditional posterior pθ ( z|x , s ) , which is conditioned on both the state and its label . This is equivalent to maximizing the lower bound in Eq. ( 6 ) . Additionally , we would like the z for safe and dangerous states are separately distributed as shown in Figure 1 ( b ) , so that we are able to measure the safety of a given state xt by calculating the divergence between qφ ( zt|xt ) and the distribution of latent for dangerous states pθ ( z|s = 1 ) . Take the trajectory in Figure 1 ( b ) as example , as the divergence function D ( q ( zt|xt ) , pθ ( z|s = 1 ) ) decreases , it is intuitive that the state xt is becoming more and more unsafe . The discussion on how to exploit specific D to construct safety cost ĉ is deferred to the next subsection . Before proceeding , there are several issues to be addressed . First , different to the lower bound in VAE , the objective we aim to maximize is also conditioned on label s. If a task is highly safe or unsafe , i.e . p ( s = 1 ) or p ( s = 0 ) ≈ 0 , then the objective falls back to the form in VAE , since the rare part of data plays an insignificant role in the approximation loss . Thus it is necessary to strike a balance between the amount of safe and dangerous data . Secondly , in this paper , we use Gaussian distributions with different expectations as the prior distributions , i.e . pθ ( z|s = i ) = N ( µs=i , I ) , i ∈ { 0 , 1 } . The expectations µs=0 and µs=1 are selected to be reasonably apart from each other to ensure that the safe and dangerous states are separable in the latent space . In this paper , the encoder qφ ( z|x ) is parameterized by a Gaussian MLP ( a fully connected neural network ) and decoder pθ ( x|z , s ) parameterized by an MLP . For training of these networks , we come up with the following empirical objective function , Ep ( s ) L ( x ; θ , φ , s ) = Ep ( s ) [ −DKL ( qφ ( z|x ) ‖p ( z|s ) ) + Epθ ( x′|z , s ) , qφ ( z|x ) ‖x ′ − x‖2 ] ( 7 ) where x′ is the generated state based on the latent sampled from qφ ( z|x ) . Since both the encoder network and the prior distributions are both Gaussian , the above equation could be further written as Ep ( s ) L ( x ; θ , φ , s ) =Ep ( s ) [ ( 1 + log ( σ ( x ) 2 ) − ( µ ( x ) − µs ) 2 − σ ( x ) 2 ) + Epθ ( x′|z , s ) , qφ ( z|x ) ‖x ′ − x‖2 ] ( 8 ) where µs denotes the expectation of prior distributions . µ ( x ) and σ ( x ) represents the expectation and diagonal elements of covaraiance matrix for the Gaussian MLP . The latent of a given state x could be generated by z = µ ( x ) + σ ( x ) , ∼ N ( 0 , I ) ( 9 ) To sum up , we expected all the dangerous state maps to the arbitrary prior distribution p ( z|s = 1 ) , and all the safe state maps to the another arbitrary prior distribution p ( z|s = 0 ) , contributing to the first term of the objective function minDKL ( qφ ( z|x ) ‖pθ ( z|c ) ) . To verify the inferred distribution carrying effective information , we sample from inferred latent distribution qφ ( z|x ) then use a decoder to reconstruct the original data based on the samples , forming the second term of the objective function ‖x′ − x‖2 . Data flow between encoder and decoder and how each part is involved in the above objective function during training are shown in Figure 2 .

Summary: This paper tackles the challenge of control with safety constraints using a learned soft constraint formulation. It models data as coming from a hierarchical generative model p(s) p(z|s) p(x|z,s) where s is a safety indicator variable, z is a latent variable, and x is an observation. Then it learns a variational approximation q(z|x) to the true posterior p(z|x). By then measuring a divergence between q(z|x) and p(z|s), the authors aim to evaluate whether a state is safe or unsafe. They then combine this learned safety constraint with a constrained RL method, SSAC, to learn safe policies.