Variational Constrained Reinforcement Learning with Application to Planning at Roundabout
Authors: Yuan Tian
Minghao Han
Lixian Zhang
Wulong Liu
Jun Wang
Wei Pan
Venue:  (2020)
Keywords: Safe reinforcement learning
Autonomous driving
obstacle avoidance

Abstract: Planning at roundabout is crucial for autonomous driving in urban and rural environments. Reinforcement learning is promising not only in dealing with complicated environment but also taking safety constraints into account as a as a constrained Markov Decision Process. However, the safety constraints should be explicitly mathematically formulated while this is challenging for planning at roundabout due to unpredicted dynamic behavior of the obstacles. Therefore, to discriminate the obstacles' states as either safe or unsafe is desired which is known as situation awareness modeling. In this paper, we combine variational learning and constrained reinforcement learning to simultaneously learn a Conditional Representation Model (CRM) to encode the states into safe and unsafe distributions respectively as well as to learn the corresponding safe policy.  Our approach is evaluated in using Simulation of Urban Mobility (SUMO) traffic simulator and it can generalize to various traffic flows.

Under review as a conference paper at ICLR 2020
VARIATIONAL CONSTRAINED REINFORCEMENT LEARNING WITH APPLICATION ROUNDABOUT
TO PLANNING AT
Anonymous authors Paper under double-blind review
ABSTRACT
Planning at roundabout is crucial for autonomous driving in urban and rural environ- ments. Reinforcement learning is promising not only in dealing with complicated environment but also taking safety constraints into account as as a constrained Markov Decision Process. However, the safety constraints should be explicitly mathematically formulated while this is challenging for planning at roundabout due to unpredicted dynamic behaviour of the obstacles. Therefore, to discriminate the obstacles' states either safe or unsafe is desired which is known situation awareness modeling. In this paper, we combine variational learning and constrained reinforcement learning simultaneously learn Conditional Representation Model (CRM) to encode the states into safe and unsafe distributions respectively as well learn the corresponding safe policy. Our approach is evaluated in using Simulation of Urban Mobility (SUMO) traffic simulator and it can generalize to various traffic flows. (Anonymous code available to reproduce the experimental results and additional videos are also available
INTRODUCTION
The recent progress in model-free reinforcement learning (RL) (Sutton et al. 1992) has produced many interesting results in planning and control problems and proves to be effective finding optimal policy for nonlinear stochastic systems when the dynamics are either unknown or affected by severe uncertainty (Buşoniu al. 2018), including complicated robotic locomotion and manipulation (Kumar et al.) |016] Xie et al. 2019; [ Hwangbo et al.) 2019). To further ensure the safety in control, applying constraints over states and actions is a natural way. Typically, standard and well-studied formulation for reinforcement learning with constraints are the constrained Markov Decision Process (CMDP) framework (Altman 1999), where the discounted sum of safety cost should be under certain bounds. As consequence, discount parameter or bound has to be tuned to ensure safety (Gareia & Fernández 2015).
However, in many complicated scenarios when the traffic flow changes frequently, the mathematical description the obstacles are difficult to obtain explicitly. In this scenario, the constraints are dynamic with uncertainty, which is challenging to define proper constraint. Besides, it is still challenging to tune a cost function with such complicated constraints. Without certain predefined constraints, existed methods fail to find feasible policy.
When humans driving car, they do not know any specific safety constraint and still perform well. For example, at the very beginning, one is always not sure about the distance between the headstock and wall or other vehicles, which may lead to some accidents. However, with more driving experience, the better awareness of safety and latent constraints will be slowly formed. constraint is not formulated as any specific distance between obstacles but the feeling according to experience. One of the talents of humans is that humans heavily rely on to make safe decisions is that could understand the latent information of complicated situation or environment and awareness, whether it is safe or dangerous (Bubic et al. 2010). This capability is called situation awareness (SA) (Endsley 2017). In RL, SA could be regarded as model that can accurately aware current situation,
this scenario, the
com http:////ww....oooo.c.oooooooota
d1

---

Under review as a conference paper at ICLR 2020
internally represent the complex dynamics and covering enough latent information like constraints. Furthermore, should be able be embedded into RL framework to guide the agent how to safely interact with environment.
This paper proposes a novel approach for encoding the measure of safety in scenarios where the explicit safety cost is not available, or the states are interfered by severe uncertainties. The contribution of this paper is in two-folds: 1) present a irriationallbassen method to encode the safe and unsafe states; measure the level of safety in latent space with Wasserstein distance, taking the uncertainty in states into consideration. Our framework can be generally combined with various RL algorithms, improving the performance of RL algorithms in terms of safety. Finally, we evaluate our approach in roundabout task with different traffic flow and show that our approach significantly improves the success rate of baseline.
RELATED WORK
There are several prior works about the reinforcement learning with safety constraints have been done. |Achiam et al. (2017) proposed a safety constrained policy optimization (CPO) approach based on the trust region method, which guarantees the constraint satisfaction with safe initial policy. WWen & Topcu (2018) came up with a constrained cross-entropy method for finite CMDP tasks, which an effectively learn feasible policies with respect to constraint satisfaction. Chow et al. (2019) came up with Lyapunov-bass -based approach and combine with both on-policy and off- policy reinforcement learning algorithms, which achieves better results in terms of balancing the performance and constraint satisfaction compare with CPO. However, both of the results above need full knowledge of the constraints, and the tasks are almost static environment without uncertainty, rather than a complex dynamic case with many unpredictable obstacles, which is the focus of this paper.
Situation awareness is the key to successful decision-making Nuulmmeyer et al. 2005). Recently, many works on reinforcement learning with situation awareness have been done. [eng & Tan (2008) proposed TD-FALCON methods that could integrate Context-awarr Decision Support (CaDS) system and learning for supporting context-aware decision making, where CaDS system exploits contextual information for focused situation assessment and goal-oriented decision support. AnAellloo al. (2014) proposed Context Space Theory (CST) to represent raw data in high-level, domain- relevant concept, namely context attribute. which could identify situations the user is involved in and considering user's situated preferences. Yin et al. (2019) propose two new metrics CTSS, CTS, to assess the taxi situation, and they found significant relationship is revealed between the taxi delay and CTS, at Level-1, and the taxi time and CTS, Sn at Level-2, which provides strong reference to airport ground movements for control and management purposes. However, all of which are focused on a simple task or static environment. Most importantly, the proposed situation awareness models all deterministic, which might lose much information in the task with the action-conditioned settings and dynamics uncertainty.
3 PROBLEM FORMULATION
Reinforcement learning with safety constraints tends to be a promising way offer intelligence and safety simultaneously for autonomous driving. In this paper, we will focus on the decision and control tasks with safety constraints which can be modeled by constrained Markov decision process (CMDP) (Altman 1999). A CMDP is tuple, AX, r. P. c, P),,) where X' is the set of states and A is the set of actions. P( [x, a) is the transition probability function and p(z) is the starting state distribution. r(z, a) and c(a, a) are the reward the constraint function, respectively.
In CMDP where the constraints e not well defined, we aim to simultaneously find a policy π and safety cost è such that
max E
> (r( at)
s.t. E E
Y (0(2z, <d
(2)
t=0
2

---

Under review as a conference paper at ICLR 2020
where Y [0. 1) is the discount factor, denotes trajectory (r (Zu, du, $1,,..)), and is shorthand for indicating that the distribution over trajectories depends on æ Io ~ ar ~ (T- (z)) is the learned safety threshold. Though the form of c is unknown, we assume that the data set composed of states labeled as safe or dangerous available, i.e.
Sa+1 ~ P(- at).
(1), (I2, $2). (I', s3).... } where s E {0, indicates whether the state is safe or danger- ous. Note that it is possible the same state x has different labels at different data points, since various actions may be taken. Based on this data set, we aim to learn model (=(=))) to map the states to latent space and then construct safety cost based on this latent model.
MAIN RESULTS
Inference the implicit constraints and awareness of state to be safe dangerous the essential capability for safe decision making. To model the awareness of constraints, the key is to map from physical state space to latent space where represents the implicit constraints. We proposed a novel method could infer the safety condition as well as the latent constraints.
In this section, our approach for inference the latent constraints will be described in detail. First, in Section 4.1] conditional representation model is proposed to map the states into the latent space, where the latent space of safe and dangerous states are separated from each other. Then in Section 4.2 we employ Wasserstein distance in measuring level of safety of the given state in latent space, i.e. the constructed safety cost
4.1 CONDITIONAL REPRESENTATION MODEL
In this part, we propose the conditional representation model (CRM) that is capable of mapping the state space to a latent variable space Z. For this purpose. we borrow the encoder-decoder structure from variational autoencoder (VAE) (Kingma & Welling 2013 Doersch 2016), which provides an efficient approach for approximating posterior distribution of the latent variable = given an observation x. For normal VAE, the variational bound is written as
log po (x) DxL DxL (=(e)))))))))))))))))) L(x; (,)) L(r;0, = (gx
(3)
+ Eg [lo [o(|2]]
(4)
where (2(2))) is the recognition model, an approximation the intractable true posterior pe (2(1x). po(x)2) is the generative conditional distribution and (o(2) is an arbitrary prior distribution. DEL denotes the Kullback-Liiler (KL) divergence.
(0 (z)
Polzi's
7to +T
Zio
Po 0)
(a) VAE
(b) CRM
Figure I: a). For VAE, the green circle denotes the prior distribution. All the latent distributions from both safe and dangerous states map into compact region near each others, which has no capability distinguish the safe and dangerous stateb). For CRM, there are latent space. We hope all the dangerous state maps to the white space after inference, while normal state maps to the green space after inference. The figure shows when the trajectories drive into dangerous situation, the latent distribution will move close to white latent space from green latent space. Our method is able to separate the safe and dangerous latent space.
In our framework, the CRM not only aimed at encoding, but also contains enough information for measuring how safe unsafe the current latent is, which is not concern in classical VAE. In
3

---

Under review as a conference paper at ICLR 2020
equation 3 the latent of both safe and dangerous states sampled from the same prior distribution po(2) (as shown in Figure If)), resulting in that distributions latent variable for safe and dangerous states are hardly separable, which brings trouble for measuring the safety in the following step.
The data set for CRM is composed of a series of data points containing both the state and its label, {(21, $1), (222 82). (123, 8s),...), }.. To exploit the binary classification label in the process of latent representation learning, we rewrite the marginal likelihood in Eq. (3) as
(D E p(s) po((s))))) (P
x)) + L(x; 0, 0))) 1. llog po (x (z) s)]
(5)
0( s) (
(6)
Here, the objective to use an encoding model (0(2(z)) approximate the conditional posterior pe (2)z,,), which is conditioned on both the state and its label. This is equivalent to maximizing the lower bound in (G) Additionally, we would like the for safe and dangerous states are separately distributed as shown in Fiiuure Ijb), so that we are able to measure the safety of given state dt by calculating the divergence between go (za)) and the distribution of latent for dangerous states (zls = 1). Take the trajectory in Figure bb) as example, as the divergence function D(g(z)) Po 1)) decreases, it is intuitive that the state Te is becoming more and more unsafe. The discussion on how to exploit specific D to construct safety cost é is deferred to the next subsection.
Inference Stage
Only Used For Training
Wassersoin distance between constraints
Varistonn lower bound
W2(44)
Syor Pectistttt (adee) Hater =
Updale
Reward
State (0osenation))
Gample
Encoder
Decoder
Reconstruction
Jowrerus up
Data flow direction
Inference flow direction
Training flow direction
Po Decoder and its hyperparameter
Cast is a feedback from environment that measures how close to the constraints
Encoddr and hyperparamster
Reward is a feedback from environment that measures how good the state
Laient distribttion
Figure 2: Overview of the data flow of Conditional Representation Model for both training and inference stages.1). Training Stage: During interaction with the environment, the agent collects both dangerous and safe states. Through the encoder 4a. where & are the hyperparameters of the network (more details on the neural network architecture can be found in Appendix B) we expected all the dangerous state maps to the arbitrary prior distribution p(z|s = 1), and all the safe state maps to the another arbitrary prior distribution p(2|s 0). contributing to the first term of the objective function min DRL (4s (2) (=]x))]pp(( (z[c)). To verify the inferred distribution carrying effective information, we sample from inferred latent distribution qo (z|æ) then use a decoder to reconstruct the original data based on the samples, forming the second term of the objective function || x 2112. 2). Inference Stage: The agent uses encoder to infer latent information and formulates safety cost by Wasserstein distance. Finally.an CMDP reinforcement learning algorithm is employed to update the policy with safety cost and reward.
Before proceeding, there are several issues to be addressed. First, different to the lower bound in VAE, the objective we aim to maximize is also conditioned on label s. If task is highly safe or unsafe, i.e. p(s = 1) or ((s = 0) Rs O, then the objective falls back to the form in VAF, since the rare part of data plays insignificant role in the approximation loss. Thus it is necessary to strike balance between the amount of safe and dangerous data. Secondly, in this paper, we use Gaussian distributions with different expectations
N( T), i {0, 1). The expectations M==0 and (a=11 are selected to be reasonably apart from each other to ensure that the safe and dangerous states are separable in the latent space.
the prior distributions, i.e. po
In this paper, the encoder 9o (2(xr) is parameterized by a Gaussian MLP (a fully connected neural network) and decoder (2z222 8) parameterized by an MLP. For training of these networks, we come
4

---

Under review as a conference paper at ICLR 2020
up with the following empirical objective function,
Be(s) [x:; 0, 0) = Pp(s)] (eaa + Ees (eol))))))))) |a x01
(7)
where r' is the generated state based the latent sampled from ga (2(2))) Since both the encoder network and the prior distributions are both Gaussian, the above equation could be further written as
Ep(s) (x; 0, Φ, 8) - Epco)(( + (x)( - (1(x²) - n.)² ox))²)
+ tepp
where μt, denotes the expectation of prior distributions, p(x) and (1x) represents the expectation and diagonal elements of covaraiance matrix for the Gaussian MLP. The latent of a given state x could be generated by
= p(z) + (()) O G,E 100
To sum up, we expected all the dangerous state maps to the arbitrary prior distribution p(z|s = 1), and all the safe state maps the another arbitrary prior distribution p(z|s = 0), contributing the first term of the objective function min Dx (2((z(r) [[e))]](((((c) To verify the inferred distribution carrying effective information, we sample from inferred latent distribution qo (=|zz) then use a decoder to reconstruct the original data based on the samples, forming the second term of the objective function || x||?
Data flow between encoder and decoder and how each part is involved in the above objective function during training are shown
Figure 2
4.2 CONSTRUCTION OF SAFETY CONSTRAINTS
In this part, the measurement of divergence between the approximated posterior latent distribution (=(2/ and the prior po (=|s = 1) is specified and exploited in constructing the safety cost é in equation 2
= 1)). a number of common metrics exist, such KL divergence (Hershey & Olsen 2007) and Pensen-Shannon (JS) divergence (Lin) 199T). However, both KL and IS divergence are not suitable for measuring the divergence between distributions that are distinct from each other. Instead, we use Wasserstein distance measure the divergence between the aforementioned distributions. Wasserstein (Sstarce(Panaretos & Zemel (0199) a metric of divergence inspired by the problem of optimal mass transportation. The p- Wasserstttn distance between two distributions, p1 and on IRd is defined as:
First, for the measurement of divergence
inf W (p (pI,P2) p2) X~PLLY up2
(EII 1
(10)
where the infimum is taken over all pairs of d-dimensional random vectors X and Y marginally distributed as P1 and P2, respectively. Compared with other metries, Wasserstein distance more informative even when the distributions are far apart from each other, while KL-divergence goes to infinity and JS-divergence equals to a constant in this case. As (=(xx) and (=(=|s = 1) are both Gaussian distributions, the Wasserstein distance between them could be analytically written as:
W3 (c)()) 1)) = [|((z) - 1|=1112 tr(T diag(a(x)))
(11)
where the tr(-) denotes the trace of a matrix and diag(-) denotes forming a diagonal matrix with the given vector.
For CMDP tasks, the agent is required to keep away from the dangerous states (or constraints) beyond certain threshold. Thus, we propose a general Wasserstein-based cost as:
7
e(x) = max W W2(ga(2lpp
1,0)
(12)
where d is the threshold to be tuned. The safety cost decreases as Wasserstein distance increases and becomes zero when Wasserstein distance is larger than d. Only if the the current state achieves the threshold leee,,- there will be cost, which means there will not any influence or restriction within the threshold of sefety- as shown in Figure 3
5

---

Under review as a conference paper at ICLR 2020
60 Trenlll
Figure 3: The curve of safety cost.X-axis indicates the value of Wasserstein distance and Y-axis indicates the value of safety cost. The red dot indicates the threshold safe Wasserstein distance, under which the state is regarded as potentially dangerous.
Algorithm 1 CRM-based Reinforcement learning
Randomly initialize neural networks of reinforcement learning algorithm, and encoder qs decoder Py of CRM
Assume dangerous prior distribution μl, and safe prior distribution y for each iteration do
Sample Sy according to for each time step do
Sample o, from policy π and step forward Observe 81+11 and Fx
Evaluate qg(2(+1|5441) with encoder and calculate according to 12 Store (st, ae, Ft, Ct. St+1) for policy update
if Dead or collision, store $t-ndt+11 as dangerous state for CRM update else Store 81+1 safe state for CRM update
end for for each update step do
Update reinforcement learning policy Sample equal amount of data from safe states and dangerous states then update the CRM using gradient decent minimizing (8)
end for end for
Finally, we demonstrate how the proposed approach is incorporated with the general CMDP RL algorithms in Algorithm [ During training, the data for training CRM is collected from the agent's interaction with the environment, while CRM provides the safety cost at the same time. Once the agent dies, the last n steps state will be labeled as dangerous while others labeled as safe. For every update step, equal amount of data is sampled from the set of safe and dangerous states, then used for evaluating the gradient encoder and decoder with respect to (8. If datasets safe and dangerous states are available before the training starts, it is also desirable to pre-train CRM in advance, which general accelerates the convergence policy learning.
EXPERIMENTS
We tested algorithm the task of driving through double-lane roundabout with four exits. Figure 4/depicts a bird's sird's-eye view of the roundabout.
The challenges faced in the roundabout include a large amount of interaction between the traffic participants, the complexity other agents' driving behavior, the vast perceived uncertainty due to the geometry of the road and the continuous operation in short period time (turn to the inner track, turn to outside lane, negotiation). Most importantly, it is difficult to define the constraints and design cost function properly under dynamic traffic with uncertainty. Thus, aimed model the latent

---

Under review as a conference paper at ICLR 2020
safety constraints with CRM and employ the CMDP RL algorithm together with the learned safety cost to find a safe policy.
For the experimental environment, the roundabout scenario is simulated by the Simulation of Urban Mobility (SUMO) traffic simulator (Behrisch et al., 201T) with TraCI (Wegener et al.] 2008) for the interface. The roundabout is connected to approach roads with give-way lines at the junctions. Besides, there is a suggested speed limit that can be exceeded by the ego vehicle as well as other vehicles. The simulation parameters for the selected roundabout and the vehicle properties are presented in Table below.
Lane width
3.5m
Velocity limit
13.89m/s
Vehicle Length
5m
Max Detiratiion/eeeeeratttinnn
2.6m/s²
Max Deceleration
4.5m/
Sensor range
42m
Table 1: Simulation scenario and Vehicle parameters
The agent is to drive into the roundabout from the 1st entrance and drive out in the 4th exit as fast as possible. If there is no collision with other vehicles, it succeeds. The ego vehicle follows the policies computed the speed using the reinforcement learning algorithm, whereas other participants use the default car following model (Kraus) integrated in SUMO. Basically, the agent to learn a speed profile under the default lateral control strategy of SUMO.
Besides, to show the general applicability of our methods, we designed two scenes with different density of traffic, namely the congested traffic and regular traffic. The illustration of traffic density is shown below in Figure 4] For the congested traffic, there are 12 equally distributed vehicles every 10 seconds from different entrances driving into the roundabout, while for the normal traffic, there 12 equally spaced vehicles driving in every 30 seconds.
(a) Congested traffic
(b) Normal traffic
Figure 4: The figure depicts the bird's-eye view of the double-lane four-exit roundabout with intersection network in two different traffic situations.. The number denotes the index of different exits/entrances. The ego vehicle is in blue, while other vehicles are green and yellow, indicating vehicles are in the range of the radar. Congested traffic: around 24-30 vehicles are in the roundabout the same time. Regular traffic: around 12-15 vehicles are in the roundabout at the same time.
RESULTS
In our experiments, we aim to answer the following questions:
Does method successfully learn the latent constraints?
7

---

Under review as a conference paper at ICLR 2020
How does the trained agent perform compared with those guided by manually designed constraints?
Is our method sensitive to hyperparameters?
In this part, we evaluate the convergence of the CRM and combine it with a CMDP RL algorithm safe SAC (SSAC) (Chow al. 2019). SSAC is safety constrained variant of the original algorithm Haarnoja et al. 2018)) through Lagrangian relaxation procedure. Details of the Lagrangian-base -baen safe baselines are referred to Appendix C To make fair comparison, we also designed a very conservative cost with optimized structure and parameter, and also SSAC to train the agents. It is worth mentioning that, for image-basee based tasks, it is generally difficult to design cost, while our framework and algorithm is still applicable, which we leave for future work.
6.1 CONVERGENCE
As demonstrated in Figure 5
CRM could efficiently model the latent constraint. Besides, throughout the training, it converges with low variance even though both the model and environment randomly initialized.
100
250
3.0
Figure 5: Training of CRM. X-axis indicates the training steps in thousand and Y-axis indicates the loss of CRM. The shadowed region shows the 1-SD confidence interval over random seeds.
6.2 COMPARISON WITH BASELINE
In this part, we compare the performance between agents trained by SSAC with and without CRM. Specifically, we use success rate, i.e. the probability of leaving from the right exit without collision, as the metric of safety performance. As demonstrated in Figure 6 in both congested traffic and regular traffic scenarios, SSAC with CRM achieves better success rate and convergence speed than SSAC baseline.
05
03
05
353
630
1000
126
282
300
430
630
730
(a) Congested traffic
(b) Regular traffic
Figure 6: Success rate of agents trained by SSAC with and without CRM during training. X-axis indicates the training steps in thousand and Y-axis indicates the success rate. The shadowed region shows the I-SD confidence interval over 5 random seeds.
8

---

Under review as a conference paper at ICLR 2020
6.3 INSENSITIVITY HYPERPARAMETERS
The threshold E [0, (D(pa(=)s 0), po(=]s 1))] in 12 is a hyperparameter to be tuned. We show this part that the performance our approach is insensitive this hyperparameter. We tested 3 different threshold setting
55 5 555 5) where ub = W (poo(== 0), Po(a|s = 1)). As
ub 2ub Sub
shown in Figure 7 CRM performs stably across different settings. The Wasserstein cost is highly generalized and insensitive to hyperparameters, which is easily for implementation. To the contrary. the manually designing safety cost for complicated tasks like autonomous driving requires lots of tuning and could be very sensitive to parameters.
04
Figure 7: Success rate of agents trained by SSAC with CRM under different d. X-axis indicates the training steps in thousand and Y-axis indicates the success rate. The shadowed region shows the 1-SD confidence interval over 5 random seeds.
7 CONCLUSION
Inspired by the situation awareness of human intelligence, in this paper, we proposed a universal approach of and latent constraints representation, which could be inherently embedded into CMDP reinforcement learning framework. We believe better situation awareness is an essential steps towards creating more intelligent agents. The roundabout experiment shows that our approach can indeed help agent aware the potential danger and improve the performance. Besides, the CRM is able to generalize through different traffic flow. Our works represent an initial step in combining RL and situation modeling. Future work includes: i) optimizing the prior distribution; ii) extending to image-based task which still a open problem for CMDP; iii) exploring other kinds of situation awareness for reinforcement learning.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX) Symposium Operating Systems Design and Implementation ((OSDI) 16), pp. 265-283. 2016.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning Coomiing-Volumee Volume 70, pp. 22-31. JMLR. org, 2017.
Eitan Altman. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Michael Behrisch, Laura Bieker, Jakob Erdmann, and Daniel Krajzewicz. Sumo-simulation of urban mobility: an overview. In Proceedings of SIMUL 2011, The Third International Conference on Advances in System Simulation. ThinkMind, 2011.
Andreja Bubic, D Yves Von Cramon, and Ricarda I Schubotz. Prediction, cognition and the brain. Frontiers human neuroscience, 4:25, 2010.
Lucian Buşoniu, Tim de Bruin, Domagoj Tolić, Jens Kober, and Ivana Palunko. Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews in Control, 2018.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh, and Edgar Duenez- Guzman. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.1003/ 2019.
Giuseppe D' D'Aniello, Matteo Gaeta, Vincenzo Loia, Francesco Orciuoli, and Stefania Tomasicllo. A dialogue-based approach enhanced with situation awareness and reinforcement learning for ubiquitous access to linked data. In 20/4 International Conference on Intelligent Networking and Collaborative Systems, pp. 249-25422 256. IEEE, 2014.
Carl Doersch. Tutorial variational autoencoders. arXiv preprint arXiv: 2016
Mica R Endsley. Toward theory of situation awareness in dynamic systems. In Situational Awareness, pp. 9-42. Routledge, 2017.
Javier Gareia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 18Xiv:1812.05900. 2018.
John R Hershey and Peder Olsen. Approximating the kullback leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Pinnessinng-CASSSP'O' volume 4, pp. IV-317. IEEE, 2007.
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26))2au5872 2019.
Diederik P Kingma and Max Welling, arXiv:1312.61 2013.
Auto-encoding variational bayes.
arXiv preprint
Vikash Kumar, Abhishek Gupta, Emanuel Todorov. and Sergey Levine. Learning dexterous manipu- lation policies from experience and imitation. arXiv preprint 1mX.v:1611.0509) 2016.
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 1551115 1991.
10

---

Under review as a conference paper at ICLR 2020
Robert T Nullmeyer, David Stella, Gregg A Montijo, and Stephen W Harden. Human factors in air force flight mishaps: Implications for change. In Proceedings of the 27th Anmual Interser- vice/Industry Training, Simulation, and Education Conference, number 2260. National Training Systems Association Arlington, VA,2003 2005.
Victor M Panaretos and Yoav Zemel. Statistical aspects of wasserstein distances. Annual review of statistics and its application, 6:405 431, 2019.
Richard $ Sutton, Andrew Barto, and Ronald Williams. Reinforcement learning is direct adaptive optimal control. IEEE Control Systems Magazine, 12(2):19-22, 1992.
Teck-Hou Teng and Ah-Hwee Tan. Cognitive agents integrating rules and reinforcement learning for context-aware decision support. In 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, volume 2, pp. 318-321. IEEE, 2008.
Axel Wegener, Michat Piórkowski, Maxim Raya, Horst Hellbrück, Stefan Fischer, and Jean-Piere Hubaux. Traci: an interface for coupling road traffic and network simulators. In Proceedings of the 11th communications and networking simulation symposium, pp. 155-163. ACM, 2008.
Min Wen and Ufuk Topcu. Constrained cross-entropy method for safe reinforcement learning. In Advances in Neural Information Processing Systems, pp. 7450-7460, 2018.
Zhaoming Xie, Patrick Clary, Jeremy Dao, Pedro Morais, Jonathan Hurst, and Michiel van de Panne. Iterative reinforcement learning based design of dynamic locomotion skills for cassie. arXiv 2019.
preprint /903v:
Jianan Yin, Minghua Hu, Yuanyuan Ma, Ke Han, and Dan Chen. Airport taxi situation awareness with a macroscopic distribution network analysis. Networks and Spatial Economics, 19(3)666--995,
2019.
A EXPERIMENT SETTING
A.1 STATE AND ACTION
we assume the the ego vehicle could detect and track the a maximum of three other vehicles' position, orientation, and velocity. The parameters of the sensor field used are given in the Table Besides, the ego vehicle is able estimate its condition. Using the above information, the ego-state and observation space of the ego vehicle is defined as follows:
8 Sego =< U
(13)
where x, V. p, e indicate the 2-D position, orientation, velocity respectively. The coordinate transfor- mation that aligns the vehicle position and velocity along the vehicle axis makes the vehicle state invariant to road geometry.
The state observations of other vehicles in the range of sensor field is recorded
O =< Fz, Vi. Pa, U4
(14)
where Og refers to the radar information of vehicle i. If no vehicle is present in the range of sensor field, then the x, y are set to sensor range limits, and orientation and are set O.
Using the above and then setting that up to three vehicles could be tracked, the state of the agent at each time step can be defined the ego state and observation of the three closest vehicles follows:
Sego O2, O2, O3
(15)
As mentioned before, our agent aims to learn a speed profile under the default lateral control strategy of SUMO. Thus, the action can be defined as:
at <
(16)
where the desired velocity of ego vehicle within the range of maximum speed.
11

---

Under review as a conference paper at ICLR 2020
A.2 REWARD AND COST
The agent updates policy according to the feedback of its interaction with the environment. For our scenario, the goal is drive safely and efficiently through roundabout without any collision with other vehicles. Thus, the reward is defined as:
Tt
(17)
where the the current speed of ego vehicle, while the safety cost ((x) defined in Equation 3
B TRAINNING DETAILS
Hyperparameters
Intense Roundabout
Intense Roundabout
Minibatch size
256
256
Actor learning rate
le-4
le-4
Critic learning rate Target smoothing coefficient(r)
3e-4
3e-4
0.005
0.005
Discount(7)
0.99
0.99
Safety 7
0.5
0.5
Threshold
0.5
0.5
Table 2: SSAC Hyperparameters
For SSAC, there are two networks: the policy network and the Q network. For the policy network, fully-connectted MLP with two hidden layers of 256 units, outputting the mean and standard deviations of a Gaussian distribution. For the Q network and the Lyapunov network, we use fully-connected MLP with hidden layers of 256 units, outputting the Q value and the Lyapunov value. All the hidden layers use Relu activation function and we adopt the same invertible squashing function technique as Soft Actor-critic (SAC) Haarnoja et al. (2018) the output layer of the policy network.
Hyperparameters Minibatch size learning rate
Conditional representation model 512
0.001
Table 3: CRM Hyperparameters
For CRM, there two networks: the decoder and the encoder. For the encoder, we a fully- connected MLP with two hidden layers of 256 and 128 units repectively, outputting the mean and standard deviations of Gaussian distribution. For the decoder, two hidden layers of 128 and 256 units respectively, outputting the reconstruction of input. All the hidden layers use Sigmoid activation function.
fully-connected MLP with
The implementation of the algorithms are based on TensorFlow (Abadi et al. 2016
C LAGRANGIAN-BASED SAFE ALGORITHM
We include a Lagrangian based method as the baseline in our experiments, namely the safe soft actor-critic (SSAC). SSAC attempts to solve the following unconstrained optimization problem
min max L(T, A) E, ((,,,/))~ [ X20
H.] Q(s, fo(c,s)) + X(Qc(s. a) 2)
(18)
where Q. is the safety Q-function. By employing the same policy gradient algorithm in SAC, safe policy obtained. The value of Lagrange multipliers 3, are adjusted by the gradient ascent method with the following objectives, respectively,
J(B)= = BB (log(to)=l( + Hel
(19)
J(A) = (a.))- (s fo(c,5') - (s(ssa) a) as)(s,a)
(20)
12


Reviews:
[{"review_id": "SklUtlxBqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the challenge of control with safety constraints using a learned soft constraint formulation. It models data as coming from a hierarchical generative model p(s) p(z|s) p(x|z,s) where s is a safety indicator variable, z is a latent variable, and x is an observation. Then it learns a variational approximation q(z|x) to the true posterior p(z|x). By then measuring a divergence between q(z|x) and p(z|s), the authors aim to evaluate whether a state is safe or unsafe. They then combine this learned safety constraint with a constrained RL method, SSAC, to learn safe policies.\n\nWhile this setting is interesting and worthy of further exploration, there are sufficient issues with this work that it should not be accepted at this time. \n\nFor one thing, the motivation behind the constrained MDP formulation of reinforcement learning, such as the original SSAC paper, is to provide safety throughout training. However, this work learns a soft constraint over the course of training by using examples where the policy leads to catastrophic failures. This means that, unlike SSAC, this work does not encourage safety at all early in training. Since it collects a large number of catastrophic experiences, this method is not categorically different from e.g. training a policy with large negative rewards for crashing.\n\nFurthermore, since there is no constraint preventing crashing, a poorly-performing learned constraint might lead to better performance as measured by reward alone. Since safety during training comes with a tradeoff against rewards, as shown by SSAC, a poorly-functioning learned constraint might lead to improved reward. This work lacks an evaluation of the number of crashes to complement the rewards depicted in Figure 6.\n\nThe particulars of the method used for computing whether the safety constraint is violated are somewhat surprising. The authors use a Wasserstein distance to compute a cost as a function of the q(z|x) and p(z|s=1). They motivate this choice by the fact that the KL divergence would go to infinity for non-overlapping distributions; however, in equation (12) that would not significantly affect the computed cost. Since this divergence is calculated in the low-dimensional latent space and one of the distributions is fixed, it is also unclear that this would ever arise. It also seems that a likelihood ratio test between p(x|s=1) and p(x|s=0) would provide a more meaningful signal than simply classifying whether a point is near the \"dangerous\" prior p(z|s=1).\n\nOverall I think there are some interesting ideas inside this work, but it needs some improvements:\n1. Reframing to make the setting make more sense. If you want to compare against SSAC, you need to be minimizing the total number of catastrophic events during training. It might make sense to assume a pre-existing set of \"dangerous\" examples, e.g. labeled ahead of time by a human.\n2. Textual editing and work to make the notation more consistent, e.g. the top of page 3 uses s for states as well as the \"safe\" indicator. \n3. Significantly improved evaluation. The results here lack crucial information about the number of catastrophic events during training. I would also like to see ablations or probe experiments showing what is learned by the encoder. Furthermore, this one environment is extremely simple (one degree of freedom) and to have confidence that the method works more generally, I would like to see the method applied to richer tasks.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJl-v-KAYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a reinforcement learning model to solve a task relevant to autonomous driving, and evaluates that model in a traffic simulator called Simulation of Urban Mobility (SUMO).\n\nThe paper describes related work but the connection to the exact problem they are solving wasn’t 100% clear to me.\n\nThe description of the model was somewhat confusing to me, but my understanding is that the model does the following:\n- The dataset contains states, and each state has a label that says whether it is safe or unsafe\n- However, we don’t know the implicit constraints that determine whether a state is safe or unsafe\n- We learn a latent representation of the states, where we want the safe states and unsafe states to be separated in the latent space\n- We use a Wasserstein distance metric in the latent space to construct a safety cost function\n\nThe paper gives pseudocode for the algorithm, and experiments in a traffic simulator of the task of driving through a two-lane roundabout with four exits.\n\nI don’t see a significant enough algorithmic contribution from this paper to yield an ICLR acceptance. I think this paper would be better suited for a more application-specific conference that would be more interested in the empirical progress on a task specific to autonomous vehicles.\n\nAs a last note, the paper contains a large number of grammatical errors and typos - I’m not considering these in the score, but the paper would benefit from a close proofreading by a native English speaker.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1lF6gJTtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presented a CRM model which is VAE with separate priors for save and unsafe modes and utilize it in  RL  for roundabout planning task. \n\nPros:\n1. The motivation of the work is clear and solves an important task\n2. The approach is sensible \n\nCons:\n1. Experimental evaluation is very weak. It is only performed on one task which is on getting out of the roundabout.  \nIt also does not compare with any real-baseline methods. It is only the proposed method compare with the weaker version of the method. \n\n2. Using a mixture of Gaussian for VAE has been used in many works before, such as Nazabal, Alfredo, Pablo M. Olmos, Zoubin Ghahramani, and Isabel Valera. \"Handling incomplete heterogeneous data using vaes.\" arXiv preprint arXiv:1807.03653 (2018).\nAlso, make such prior should be equivalent to regular Gaussian prior with multi-head decoder as in figure1 (b) in VCL paper https://arxiv.org/pdf/1710.10628.pdf where s is T there. \n\nSo, this work did not discuss these approaches and the novelty of the work is also limited. \n\n3. How to set the mean of s may be critical for the performance. Analysis needed. \n\n4. The method should at least compare with conditional VAE used in the same way in RL. In this case,  the label is s and I believe that the latent space will be meaningful regarding s.\n\n5.. The second term in the right-hand-side of equation (7) has missed some constant factor. \n\n5. How figure 5 shows CRM could efficiently model the latent constraint? It only shows that the model converged. \n\n6. Writing needs to be improved. There are grammatical mistakes here and there.  \n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to add constraints to the RL problem within a variational method. The hope is to specify a safe vs non-safe states. The reviewers were not convinced that this paper makes the cut for ICLR. Moreover, there was no rebuttal from the authors, so it didn't give the reviewer a chance to reconsider their opinion. Based on the current ratings, I recommend to reject this paper. 
Decision: Reject