Learning A Risk-Aware Trajectory Planner From Demonstrations Using Logic Monitor
Authors: Xiao Li
Jonathan DeCastro
Cristian Ioan Vasile
Sertac Karaman
Daniela Rus
Venue: CoRL2021 Poster (2021)
Keywords: Learning from demonstrations
temporal logic
trajectory planning
autonomous driving
real-time verification

Abstract: Risk awareness is an important factor to consider when deploying policies on robots in the real-world. Defining the right set of risk metrics can be difficult. In this work, we use a differentiable logic monitor that keeps track of the environmental agents' behaviors and provides a risk metric that the controlled agent can incorporate during planning. We introduce LogicRiskNet, a learning structure that can be constructed from temporal logic formulas describing rules governing a safe agent's behaviors. The network's parameters can be learned from demonstration data. By using temporal logic, the network provides an interpretable architecture that can explain what risk metrics are important to the human.  We integrate LogicRiskNet in an inverse optimal control (IOC) framework and show that we can learn to generate trajectory plans that accurately mimic the expert's risk handling behaviors solely from demonstration data. We evaluate our method on a real-world driving dataset. 

Learning A Risk-Aware Trajectory Planner From Demonstrations Using Logic Monitor
Xiao Li MIT siaoli@mit.en edu
Jonathan A. DeCastro
Cristian-loan Vasile Lehigh University cvasileelehigh edu
Toyota Research Institute
j jo www.aaattccasttttr.cc.oo global
Sertac Karaman MIT
Daniela Rus MIT rus@mit.edi
sdr
Abstract: Risk awareness an important factor to consider when deploying poli- cies on robots in the real-world. Defining the right set of risk metrics can be difficult. In this work, we use a logic monitor that keeps track of the environ- mental agents' behaviors and provides risk metric that the controlled agent incorporate during planning. We introduce LogicRiskNet, learning structure that can be constructed from temporal logic formulas describing rules governing safe agent's behaviors. The network's parameters can be Icarned from demonstration data. By using temporal logic, the network provides an interpretable architec- ture that can explain what risk metrics are important to the human. We integrate LogicRiskNet in an inverse optimal control (IOC) framework and show that we can learn to generate trajectory plans that accurately mimic the expert's 's risk han- dling behaviors solely from demonstration data. We evaluate our method on a real-world driving dataset.
Keywords: Learning from demonstrations, Temporal Logic, Autonomous Driv- ing
Introduction
Imagine driver that trying to turn across traffic an unprotected intersection where there is oncoming traffic. When faced with the situation shown in Figure 1(a), the driver needs to decide when it is safe to make the turn. The driver makes this decision by evaluating how risky the nearest oncoming vehicle is in terms of the intended turning maneuver. The risk may be dependent on number of factors, including how far the vehicle is from the intersection, how fast driving and whether it has the intention to slow down. If we wish to incorporate such risk-aware decision making into an autonomous driving system, we are faced with two problems: (1) how do we accurately model the risk factors that humans take into account while driving, and (2) how do we generate plans from these risk factors. With the development of data-driven autonomous driving, we have access to a growing amount of large-scale human driving data. Large-scale data contain not only normal driving behaviors, but also what drivers these data, our goal in this work is to develop risk representation thul can expressively describe the desired/undesired behaviors of road agents and with parameters learnable from demonstration data. We aim obtain risk metrics that reflect that of the human demonstrator, and generate risk- aware trajectory plans using these metrics.
order to manage risk. Taking advantage of
Recent work on risk and uncertainty aware policy learning aim to incorporate common risk con- structs such as conditional value at risk, worst-case risk, etc., into the problem formulation either as an auxiliary loss or constraints [1, 2]. Many then use reinforcement learning (RL) to obtain the policy, The main limitation of applying RL to autonomous driving is its exploration requirement. It can be difficult to explore safely especially when we wish to deal with risky scenarios. Work on uncertainty-awar imitation learning [3] and risk-aware offline RL [4] address the exploration problem, but are limited in the types of risks they can express.
5th Conference on Robot I Learning (CoRI. 2021), London, UK.

---

Logic formuia descriting sale agent behaviors
Risk Contour
Sensing Region
11o4 (drive lane center)
Map
Risk
Logic Risk Network
Agent
Demonstatiio
Trajectory Optimization
Data
Train
Step Emvrionment
Tracked Belief State Contours
Trajectory Plan
(b)
Figure LogicRiskNat-OOC framework. (a) pictorial depiction our setup. red car the ego vehicle (controlled by our planner). The blue cars move according to the trajectories recorded in the dataset. Their trailing contours represent the tracking distributions. The red contours around the ado vehicles (neighboring present their risks (the brighter the contour the higher the risk.) (b) An illustration
vehicles controlled our inverse optimal control (IOC) procedure. outer loop generates trajectory plans according to risk map. The inner loop updates LogicRiskNet's parameters using kind of feature matching.
is important be risk-aver riskreverse in safety critical applications. often more important be averse to the right set of risks. In this paper, we want to answer the question "if we have observed our nearby road agents for while, can we develop risk model that quantifies how risky they are based on their past behaviors in the same way that humans would." We take inspiration from real- time verification [5] and formal synthesis [6] and introduce differentiable logic monitor which we refer to as LogicRiskNer as an expressive and learnable risk representation. To summarize our contributions, we
introduce differentiable logic-based risk metric and show that this metric is able to de- scribe complex, temporally-extended risk events as logic formulas and learn its parameters from human driving data;
show that can not only use LogicRiskNet to generate risk-aware trajectories, given its hierarchical monitoring properties, we can also use it to explain why an agent is risky;
demonstrate in real-world driving dataset that our method yields controllable risk-averse or, alternatively. risk risk-ngeking behaviors.
Note that, while apply our approach to the domain of autonomous driving, equally- extend our approach to other domains, such as robotics.
2 Background: Past Time Signal Temporal Logic (PISTL)
STL. [7] offers a formalism for expressing and reasoning among a rich set of rules. The rules are defined over predicates (inequalities of real-valued functions). In standard STI., the formulas forward-looking, meaning that the formulas are evaluated by looking at future trajectories. work, need to evaluate formulas using past trajectories (trajectories of road agents have tracked over the past), using this past information to compute risk of violating these rules in the future. Therefore, rules will be encoded as of past time STL (pISTL) [8] formulas with the following syntax
this
$ = :==((>) < ( V &T To,
(1)

---

where b E 12 a < b, represent finite time bounds; & is PISTL formula; p(r) < é is predicate; a E X IR" is state; p(-) IR" - IR is the predicate function, E IR a constant. (not), A (and) and V (or) are Boolean operators. F- (eventually) and G (always) are past time temporal operators. Here f(a.6) requires to be true at least once in the time interval E (-b, -a] whereas To requires
be true for all of this interval.
Let o (2ig ...; Ft, denote the state trajectory from time lo to 41 whot }= & denotes that trajectory z/foiaa satisfies & (i.e. á evaluates to true under wloih). The Boolean semantics of PtSTL are provided in Appendix A.
An example PtSTL formula is F (ro.m) (1æ) < 2) A G (n) > 3) which means that "|æz] < 2 is true for at least one / (-71, -m] and T2 3 true for all E (-T2:
Similar to STL, PLSTL is equipped with a robustness degree (or robustness for short) that quantifies the level of satisfaction of trajectory with respective to formula. A robustness function, denoted r. takes in a trajectory and a ptSTL formula and outputs a real-valued number. A robustness greater than zero signifies that the trajectory satisfies the given formula, i.e. r(zfo.t1, x) > 0 = 2 }= ø- Negative robustness implies violation of the formula. The formal definition of robustness is also given in the Appendix A.
Note that each predicate relationship p(x) < e appearing in & has parameters that can be to how humans behave. For example. the parameterized ptSTL formula Φ [romm] (aiæ + 718 < é1) A ra.ra (a22² > <2) has parameters & ((1177) E1, 022, 22). Hence, we parameterize each predicate appearing in formula o, denoting the parameterized formula of (see, e.g., 19).
3 Problem Formulation and Approach
Let x = (p. tb. denote the state of a vehicle that includes position (x-y coordinates), heading and velocity (which we later in unicycle model). We call the vehicle that is controlled by our planner the ego agent, and all other traffic participants not controlled by us ado agents. Denote Ωr as the set of trajectories with horizon T' (i.e. WT f=++ E Ωty). Suppose that the ego vehicle tracks its nearest N ado agents as well as itself via a tracking algorithm (such as a Kalman filter). This algorithm returns set of belief states at time step I, h4 - [ E xf E Ea,,
[0, N). which are concatenated into a history of length H as h = the-....... .te]. Here, xt.a, and Σ c.a. are the mean values and covariances for the ego and i-th ado vehicle states. Hy is the set of such histories (i.e. h E Hz). Define a dataset D = [Syr, Ur. Hal where HH is obtained from sensors, Ωr is the ego vehicle's ground truth future trajectory, uy = [u',
u 4+ E Ur is the corresponding sequence of controls (speed and steering). Let feature ʃ: ΩT x UT + IR be a function that maps a trajectory of states and controls to a real value (e.g. average distance to the center lane). f(Wr) (f'(W))) .... f" (W(T) R. is vector of such features. Our problem is formally defined as follows
Problem 1. Given (1) a parametric formula of that defines the behavior of a safe ado agent, (2) a feature for representing ado agent risks, (3) a set of user defined features representing behav- iors of the ego agent f. (4) demonstrations dataset D that contains human driver's decisions and environment information, find a trajectory distribution po, Q2 > [0, 1] parametrized by 0 (0 contains the ptSTL parameters & as well as feature combination weights) that best matches human behavior: i.e., minimizes the following objective functions
J1 = Epp If (wrr)) - f(wr) and J2 = Epo (008((5s)) - f (WT)
(2)
where Wi is a trajectory from the demonstration dataset. Wir is a trajectory sampled from 'p
Problem defines a feature matching problem similar to many inverse reinforcement learning (IRL) [10] and IOC [11] formulations. We design set of features f to capture driving preferences in non-risky situations, but also include a risk-based feature f capturing the risk management preferences seen in demonstrations. From the demonstration data we then learn the risk metric pa- rameters & and the weights that combine the feature values. This combination gives us a cost model that, when solved, yields driving behaviors similar to the demonstrator (in the context of our defined
3

---

features and risk models). While existing IOC approaches are able to imitate standard driving be- haviors such that the generated trajectories from the cost model are exponentially more preferred by the agent (under maximum-entropp formulations, e.g. [11].. In our case, the additional risk feature provides the capacity to generalize our model better under risky situations without hindering the ability to replicate the driving style of human demonstrators in normal scenarios.
Logic Monitor Guided Risk-Aware LID
In this section, we will describe in detail our risk-aware LfD framework. We will first introduce the LogicRiskNel a differentiable syntax tree generated from parametric PtSTL formula. The network takes as input the tracking histories ht and outputs risk assignment for each tracked vehicle based on their past behaviors. This risk assignment can be used to monitor which vehicles are risky in given scene. We then show how LogicRiskNet can incorporated into an IOC framework that is able to learn this syntax tree from demonstrations while simultancously learn cost function that specifies a risk-aware planning problem, from which a trajectory can be solved efficiently.
Our method is depicted in Figure 1(b) where there are 2 nested loops. The outer loop starts with the map and agent information which are passed through the LogicRiskNet to construct risk map. This risk map constitutes the objective function for trajectory planner. The generated trajectory serves control the ego agent in a receding horizon fashion. the inner loop, the same set optimized trajectories are also compared against human demonstrated trajectories in the same situations, and their differences serve as training signals to the LogicRiskNet.
LogicRiskNet- a differentiable parametric PtSTL risk monitor. We describe how we may con- struct PISTL risk monitor based on a learned stochastic risk measure that can be applied probabilistic description of human behaviors. We extend the definition of robustness degree (intro- duced in Section 2) encompass belief states. In order to calculate the risk of a trajectory of belief states w.r.t a PtSTL formula o, we need to modify the definition of the robustness. Recall that the basic element of Φ is predicate of the form p(x)) < c, define a(x) = c p(z) as the predicate (this is same as the robustness degree for the predicate). Since is stochastic, rather than deterministic, we can evaluate PtSTI. formulas instead using risk measure p X - R, representing expectation, mrea-variance, value-at-risk, etc. In this work, we assume the expectation risk measure, but can equally well replace this with others. We can then apply this risk measure to the robustness definition yielding af (x) p(a(x)),
X' (This treatment is similar to [13]).
Given a parametric ptSTL formula Φf, a risk measure p(-) and the tracking history hg, we can construct the robustness risk for Φf as
(h (8) = rp (hH, Ø5)
where rd (hH, Φf) is the robustness applied to the risk-based predicate aº. Once the risk predicate values are calculated each time step the trajectory, the robustness calculations for Boolean and temporal operators can be carried out. In order to tune the parameters & using demonstration data, we follow the approach in [14]. Specifically, if we replace the max and min functions in the robustness definition with softmax approximations, we can construct computation graph from Ro with the predicate parameters at the leaves and the robustness risk at the root. This makes it possible to learn & using backpropagation. The intuition behind Equation 3 is that given PtSTL. formula ø parameterized by ç, agents' past observations hg, and a risk measure p. the robustness risk R can be calculated to represent agent risk levels in terms of satisfaction of c. We provide a step-by-ster example constructing the LogicRiskNet in Appendix F.
LfD using LogicRiskNet. With LogicRiskNet R we are able to characterize risky ado vehicles as using the following risk feature:
N-1T-1 (h wT (E) = -- (5)
1
(4)
llp1. - pt. 12
t=
We further require that the risk measure coherent: [12].

---

Algorithm Trraeectoy Optimization
Inputs: initial controls uT i initial ego vehicle state x tracking history h lane informa- tion T; control effort weight 3 lane tracking weight Bo PtSTL formula parameters ç;
2: ür <- init
3: for i=1 ... do
4:
We - CoustructTrrjectorriii..
5:
f - (hH We ()) fm + f" (uir), fu JN + FN (We, Nr)
I> construct features construct objective update control sequence
6:
= gu fu BBV. fN
7: ur uir VV uur I.
8: end for
9: W. < ConstructTrajectory (ūr, x!.)
10: return Wet f
where pt and p', 1 E [0, 7) are the ego and ado vehicles' positions at time f. h is the tracking history of ado vehicle along with the history of the ego agent. Essentially, (4) describes collision feature scaled by the risk R where the inverse-squared distance may be viewed as a generalization of potential fields [15]. A safe trajectory should yield low f value. A positive R2 signifies rule abiding ado agent whereas a negative R signifies rule-violating agent. Intuitively, we use determine how risky ado agent is, and by solving for trajectories that minimize f we can control the ego agent to stay away from risky ado agents. Our potential- fielld formulation of risk acts approximate the behavior of drivers in reacting to potential collisions but do not over-react if collision is not imminent.
the past observations
To imitate risk-free driving behavior, we additionally introduce a control effort feature and nominal trajectory feature for the ego vehicle as follows
T-1
T-1 Σ t=0
("(())
Σ ||'0| where u = =v, (ulW.. Nr) 1=
ll' PX11?
(5)
where P'y E NT are points on the nominal trajectory. Nominal trajectories can be the human ego agent's trajectories (during training) or trajectories on the target lane center (during deployment). These features encourage the ego agent to follow trajectory (and speed) profile with smooth control efforts. Given features f /", S.V. we solve for trajectory plan wr by minimizing ʃ 3u f + 3N fN using gradient descent. Algorithm describes this trajectory optimization process.
In Algorithm 1, line 4 describes the process of using a unicycle model construct trajectory from control sequence. This procedure allows to generate kinematically feasible trajectories compared to direct trajectory optimization. In caleulating the risk feature fd, we use a constant velocity and heading model estimate of ado agents' future trajectories P',, t E jo, T 1). During training, the nominal trajectory Nr is the ego vehicle's ' ground truth future trajectory from the dataset. During
deployment, this nominal trajectory is taken from points the target center lane which can be obtained from route planning.
Algorithm describes how the parameters from the LogicRiskNet along with the objective function weights are learned from demonstrations using maximum entropy model of uncertainty.
All entities with superscript are taken from the demonstrations. Line 4 calculates the sum of features obtained from trajectories from the human driver averaged over the dataset. Lines 6-10 calculates the same but with trajectories obtained from Algorithm 1. Line 12 updates the parameters such that A , and A Ass match.
Experimental Results
We train and evaluate our method on the NuScenes dataset. NuScenes [16] is a dataset for au- tonomous driving based in Boston and Singapore. It contains 850 scenes each 20s long, containing 23 object classes and HD semantic maps with 11 annotated layers. We chose this particular dataset for the rich semantics it provides which is well suited for rule definitions. We will use 650 scenes
5

---

for training and 200 scenes for validation. Details in experiment setup, implementation and hyper- parameters used are provided in the Appendix C.
Rules used. We use the following two rules to descrite behaviors safe ado agent. Our goal is generate ego plans which avoid ado agents deemed risky in the sense that they violate these rules.
Φ GH. (driveNearLane A keepSateCaaiisFFooEEgg)))
(6)
2 = G Go, H) (egoInIntersection - (farPromlntersecction V /drveSlooly))
where H. is the tracking horizon (we use H 4). Φ1 describes that safe ado vehicle should "always drive near the center lane and keep a safe distance from the ego vehicle." d2 expresses that intersections "always if the ego vehicle is in the intersection implies that safe ado vehicle should either be far from the intersection or drive slowly." The final rule ø = Φ1 A Φ2 takes the form of conjunction of set of sub-rules. Predicate and parameter definitions can be found in Appendix B.
Methods evaluation. We evaluate our method and comparison cases in terms optimality (time to reach goal position) and safety (minimum distance to nearby vehicles). Within the dataset, will set the human ego vehicle's s start and end positions as the initial and goal positions. Optimality is measured the time travel from the initial position within distance to the goal position averaged over the validation set. Safety measured as the minimum distance to nearby ado vehicles in scene averaged over the validation set. We perform two experiments: 1) during evaluation, we control the ego vehicle with our learned planner, and the ado vehicles move according to the trajectories recorded in the dataset and time is synchronized; and 2) re-run the evaluation in more realistic setting using ado vehicles that react to the ego according to an intelligent driver model (see Appendix G). each of our evaluations, the ego vehicle can navigate an environment with realistic human or reactive ado vehicles.
We use four methods for comparison. LogicRiskNet-IOC refers to the proposed method; Human refers the human driver in the dataset; BC refers to a behavior cloning agent; and TrajOpt refers to trajectory optimization agent. Implementation details of BC and TrajOpt agents can be found in Appendix C.
Results and discussions. LogicRiskNet does not only output the final risk measure but can also output the risk of all intermediate sub-formulas without needing additional computation. This allows us explain the reason of its decisions. Figure 2 shows example of unprotected right turn at an intersection and the vehicle that we are monitoring is highlighted by the yellow dash circle in Figures 2(a) and 2(c). Here are showing two consecutive steps in the scenes. Looking at the top plot in Figure 2(b) we can see that this vehicle is labelled as risky (a negative risk value denotes
Algorithm 2 LogicRiskNet Guided LiD
1: Inputs: parameteric PtSTL formula oe parameter learning rate y0; demonstration dataset
2: <- ginit g" &- Iu. snst BV BN
3: for i=l do
D ΣPD (so Ing,wah 15) + n (ut) + pM (w). Nº4)
4:
A
5:
Sj. 00
6:
for
j=l ... [D] do
7:
We, fm FN TrajectoryOptiiiization
init æd. h N4.38 3u BN,e) üT
8:
Sio+ = fo + fm j.
9:
end for
10:
A Ajo S
11:
-== (A A Ajo)2
12:
(E. 8", 3N t (5, nr B.) - ~ gu, zN L
13:
end for
14: return E, fu 3N

---

R
R
Snivsslowlyy
DivveSlooly
R Poruttoiteooo
R Feoonlattnerii
Then
60
Figure 2: LogicRiskNet monitor results. (a) Monitoring the highlighted ado vehicle at time 1.5 sec. (b) Risk traces for @ and selected sub-formulas. (c) Monitoring at time 2 sec. (d) Corresponding risk traces.
Table 1: Performance comparison results
Time to goal
Safety distance
Goal
min
mean
max
90th
min
mean
max
90th
BC
13.21
19.98
20.0
20.0
0.34
4.22
10.86
8.66
37%
TrajOpt
16.92
18.50
20.0
19.76
3.89
6.37
12.43
8.34
87%
Riskl RiskLogiiNNe-IOCC
15.31
17.35
20.0
19.0
2.72
3.98
9.01
6.12
93%
Human
19.0
19.75
20.0
20.0
3.03
4.62
9.45
6.84
100%
violation of the safety rule σ.) To gain more insights why this is the case, we can look the risk values of sub-formulas. The middle and bottom plot shows that this vehicle is neither driving slowly nor far away from the intersection. Figure 2(d) shows that after one time-step the risk for the monitored vehicle has increased (risk value becomes more negative), and the cause is not because the vehicle has sped up but because it has driven closer to the center of the intersection. The structure of the entire LogicRiskNet and the monitoring traces for all sub-formulas are shown in Appendix D.
Table shows comparison results in terms of the time it takes to reach the goal and the minimum distance to neighboring vehicles during this navigation. An aggressive agent will achieve a low time- to-goal (drive faster) well small safety distance. The %Reach goal measures the percentage of scenes in the validation set the corresponding agent is able to reach the goal state. Failure is called if the agent does not reach the goal in the duration of the scene. All results are averaged over scenes in the validation set (each scene is 20 seconds). From the table we can observe that LogicRiskNet- IOC achieves the most similar behavior Human with the former slightly more aggressive. In comparison, TrajOpt is much more conservative. This is because it treats all agents as the same collision object that is needs to avoid whereas LogicRiskNer-IOC agent will place less emphasis on low risk agents (i.e. vehicles that are stopped). This gives the latter more free space to plan and also behaves more human-like (as humans place different amount of risk on vehicles based on their past behaviors). BC agent has wide span of behaviors but mainly suffers from distribution shift [17] shown by its low success rate at reaching the goal. While we use non-reactive agents this evaluation, in Appendix G we also extend it to more realistic simulated environment.
Once the network parameters & and the objective parameters are learned, we obtain a system that behaves similar to the human demonstrator. In practice, this may not be satisfactory and the user may wish to further fine tune the system. We conduct a study to see how varying the magnitude of the risk feature in Equation (4) changes the behavior of the ego agent. In Figure 3, the x-axis represents a risk coefficient multiplied to f on line 6 in Algorithm 1. We run Algorithm 1 across all scenes the validation set and record the time-to-goal and safety distance. Figure 3 shows that with increasing importance placed on the risk term, the ego vehicle becomes more conservative (driving slower and keeping further distance ado vehicles). This finding allows the user to control the ego vehicle to strike a desired balance between risk-averse and risk- behavior.

---

Salegg destanc
Risk confiiiinn.
(a)
(b)
Figure 3: Controlling risk management behaviors by tuning risk feature coefficients. Both Time-to- goal (b) safety distance increases with the risk coefficient indicating that the ego vehicle becomes more conservative as we tune up this coefficient.
6 Related Work
Risk and uncertainty aware policy learning has been well-studied in the context of reinforcement learning. There have been several comprehensive surveys this topic; [1] provide general survey, while [2] provide greater focus on autonomous driving (AD). In general, common risk metrics such conditional value at risk, meca-variance, worst-case analysis, etc. have been used either an auxiliary loss in the objective function or as set of constraints. In many robotic and driving tasks, however, is not readily applicable given its exploration requirement. Imitation learning and offline RL helps with addressing this problem. Specifically, in [3], kernelized movement primitives are used to estimate uncertainties in the demonstrations
finding optimal gains in controller. The authors of [4] use offline distributional RL to learn policy that is averse to conditional value at risk. Our work, in contrast, aims to learn from demonstrations learn a human's notion risk. does not require exploration beyond the demonstrations, yet it is able to generalize well to new scenarios, and has the capacity learn rare situations such as rule violations and near collisions.
Temporal logic (TL)-guided policy learning is an area that we take have taken inspiration from. In this area, TL often used to specify the ego agent's desired high-level behavior and used to generate rewards. The authors of [18, 19, 20] provide surveys of recent work the use of TL in RL. The exploration problem still exists in these methods. To address these challenges, the authors of [21, 22] learn finite state automata from demonstration and use them to guide planning with the value iteration network which avoids exploration. can sometimes be tedious to manually design TL formula that yields satisfying behaviors. Work has been done to make components of the formula learnable from data. In [23], the authors propose learning linear temporal logic (LTL.) formulas from demonstrations. Given the close relationship between TI. and automaton[244 the authors of [25] propose method that learns reward machines (an automata-like reward presentation) from demonstrations. A shortcoming of these methods is that the LTL that they use operates propositions (binary variables with values true or false). Unlike our STL-based approach, these approaches require discrete state and action spaces and require the demonstrations themselves to have the same discrete representations. We provide additional discussions on related work in risk- aware formal synthesis in Appendix H.
7 Conclusion
In this work, we introduced the LogicRiskNet- an expressive and differentiable risk representation based on temporal logic descriptions risky agent behaviors. We show that with demonstration dataset we are able to learn risk parameters that results in trajectories similar to the demonstrator. We also show that given the structure of LogicRiskNet we can monitor and explain why an agent is risky without additional computation. Given our choice of the IOC framework and the interpretability of our method, users are able tune the objective function post-training to obtain the desired level of risk-averse behaviors.

---

Acknowledgments
This work is supported by the Toyota Research Institute (TRI). This article solely reflects the opin- ions and conclusions of its authors and not TRI, Toyaota, or any other Toyota entity. Their support is gratefully acknowledged. We also thank Nvidia for providing computation resources.
References
[T] Garcia and F. Fernández. A comprehensive survey safe reinforcement learning. Mach. Learn. Res., 16:1437-1480, 2015.
[2] Z. yu Zhu and H. Zhao. A survey of deep and il for autonomous driving policy learning. ArXiv, 011221000009999 2021.
[3] J. Silvério, Y. Huang. F. J. Abu-Dakka, L. Rozo, and D. Caldwell. Uncertainty-awart imitation learning using kernelized movement primitives. 2019 IEEE/SS International Conference on Intelligent Robots and Systems (IROS), pages 90-97, 2019.
[4] N. A. Urp'i, S. Curi, and A. Krause. Risk-averse offline reinforcement learning. ArXiv, abs/2102.05371, 2021.
[5] J. V. Deshmukh, A. Donzé. S. Ghosh, X. Jin, G. Juniwal, and S. Seshia. Robust online moni- toring of signal temporal logic. ArXiv, abs/1506.08234, 2015.
[6] C. Belta, B. Yordanov, and E. A. Gol. Formal methods for discrete-time dynamical systems. 2017.
[7] A. Donzé and O. Maler. Robust satisfaction of temporal logic over real-valued signals. In FORMATS, 2010.
[8] E. A. Gol. Efficient online monitoring and formula synthesis with past stl. 2018 5th International Conference on Control, Decision and Information Technologies (CoDIT), pages 916 921, 2018.
[9] E. Asarin, A. Donzé, O. Maler, and D. Nickovic. Parametric identification of temporal proper- ties. In RV, 2011.
[10] G. Neu and C. Szepesvari. Apprenticeship learning using inverse reinforcement learning and gradient methods. In UAI, 2007.
[11] M. Kuderer, S. Gulati, and W. Burgard. Learning driving styles for autonomous vehicles from demonstration. 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 2641-26466 2015.
[12] L. Lindemann, N. Matni, and G. Pappas. Stl robustness risk over discrete-time stochastic processes, ArXiv, Abs/2104.01503. 2021.
[13] Safaoui, L. Lindemann, D. Dimarogonas, I. Shames, and Summers. Control design for risk-based signal temporal logic specifications. IEEE Control Systems Letters, 4:1000-1005, 2020.
[14] K. Lunng N. Aréchiga, and M. Pavone. Back-propagation through signal tempo- ral logic specifications: Infusing logical structure into gradient-based methods. In Workshop on Algorithmic Foundations of Robotics, 2020.
[15] A. Pierson, Schwarting, S. Karaman, and D. Rus. Navigating congested environments with risk level sets. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5712-5719, 2018. doi: 119/....../.......0 8466677
[16] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijhom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903.1 1027, 2019.

---

[17] F. Codevilla, E. Santana, A. M. López, and A. Gaidon. Exploring the limitations of behav- ior cloning for autonomous driving. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9328-9337, 2019.
[18] H.-C. Liao. A survey of reinforcement learning with temporal logic rewards.
[19] Q. Gao. Deep reinforcement learning with temporal logie specifications. 2018.
[20] C.-J. Heiker. Temporal logic specifications in reinforcement learning. 2021.
[21] B. Araki, K. Vodrahalli, Leech, C. Vasile, M. Donahue, and D. Rus. Learning to plan with logical automata. In Robotics: Science and Systems, 2019.
[22] B. Araki, K. Vodrahalli, T. Leech, C. Vasile, M. Donahue, and D. Rus. Deep bayesian non- parametric learning of rules and plans from demonstrations with learned automaton prior. In AAAI, 2020.
[23] D. Kasenberg and M. Scheutz. Interpretable apprenticeship learning with temporal logic spec- ifications. 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pages 4914 4921, 2017.
[24] C. Baier and J. Katoen. Principles of model checking. 2008
[25] R. T. Icarte, E. Waldie, T. Q. Klassen, R. Valenzano, M. P. Castro, and A. Mcllraith, Learning reward machines for partially observable reinforcement learning. In NeurIPS, 2019.
10


Reviews:
[{"review_id": "p9gkH0Epa9q", "reviewer": "Reviewer_c3zR", "summary": "This paper proposes a framework to learn a motion planner (a parametrized distribution of trajectories) that takes into account the risk ok other agents. The proposed approach is to design a risk monitor that can process previous trajectories and compute a risk score to each of the ado vehicles. The monitor is then used to learn the motion planner. The proposed framework is then evaluated using the NuScenes dataset. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The motivation behind using risk as a factor in motion planning is interesting.\n- The use of ptSTL leads to a formal framework to reason about and quantify risk\n\nWeaknesses:\n- The paper lacks several technical details which makes it hard to follow. In particular, the discussion on LogicRiskNet in Section 4 is vague. Examples are \"If we use softmax approximations .... we can construct a computation graph ....\". It is not clear what the authors are referring to in this paragraph. A more detailed construction is needed to understand the constructions that are verbosely described in this Section.\n- One hidden assumption is that previous behavior of ado cars is a good predictor of their risk in the future. The paper failed to explain why this is indeed the case. \n- The evaluation section is based on simple hand-picked risk rules. It is not clear how the performance of the proposed framework is affected by the choice of these rules. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The idea of using ptSTL to capture risk and design a risk monitor is interesting. Nevertheless, the paper is lacking deep technical discussion about several of the key components of the framework.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- The authors should provide an empirical evidence of why historical data provide a good predictor of the future risk. The authors may use the same dataset used in evaluation or any other dataset to provide such evidence. "}}, {"review_id": "5DHiW9ypOgU", "reviewer": "Reviewer_3SPR", "summary": "This paper introduced a risk-aware motion planner for autonomous vehicles that learns from expert demonstrations. The learning structure was called LogicRiskNet. In particular, the learned trajectories are guided by temporal logic rules. In addition to evaluating the risks of surrounding vehicles, the learned structure is able to explain why a surrounding vehicle is in a risky situation. Finally, the developed approach was evaluated using a dataset called NuScenes, which was selected due to its rich semantics. The proposed method was compared with a few baselines, including naive behavior cloning, and optimization methods. Results showed that the developed approach produced high success rate, and shorter time to navigate to the goal. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main concern is that autonomous driving is a highly interactive domain, but the paper used prerecorded data to simulate the interactions. The differentiable logic-based risk metric is fine, because it doesn't involve any changes to the world (simulated or not). It's becoming problematic when evaluating the proposed control method (risk-adverse and risk-seeking behaviors). It does not make sense to evaluate the performance of a vehicle assuming the surrounding vehicles are not responsive at all. This issue was briefly mentioned in the \"Methods of evaluation\" paragraph, but it's pointing to future work to address this issue. It's unclear why simulators like CARLA was not used in the paper. \n\nThe other concern is that Sections 2 and 3 were not presented in a way that's friendly to readers who are not very familiar with the relevant mathematics. The reviewer knows basic knowledge representation of AI, and has some experience about LTL, but still find it difficult to follow the symbols and equations. It's good that there are transition sentences that summarize the technical details, which helped the reviewer follow the whole paper. Still, it will be good if the paper can be presented in a way that's mathematically less heavy. \n\nThere were only two formulas used in the evaluation, though the developed approach is supposed to be generally applicable to logic formulas. Also, the explanation part of the work was not evaluated. \n\nA few minor issues: \n\n - Figures show up before their references in the body of the paper. For instance, Figure 2 is on page 6, but it was not mentioned until page 7. Figure 1 is on page 2, and its first mention is on page 4. \n\n - Font size in figures is very small, and the embedded text is hardly readable. \n\n - Please explain what \"ado agent\" means the first time it's used. The reviewer couldn't figure it out until reading Figure 1. \n\n - The paper has many sentence statements that start with \"The authors of [X]...\" In general, a reference is to support a statement, but should not be a statement itself. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The logic-guided risk evaluation part of the paper (LogicRiskNet) is well motivated, and makes a lot of sense. But the evaluation part has a control component, and was evaluated using a prerecorded dataset, which is questionable and not convincing. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "2-aBOMxYau6", "reviewer": "Reviewer_Qizq", "summary": "In this paper, the authors introduce LogicRiskNet, a form of differentiable temporal logic. \nThe parameters of LogicRiskNet can be learned from a set of demonstrations.\n\nLogicRiskNet incorporates a measure of the risk connected to the agent's behavior. Furthermore, the logic encoded by LogicRiskNetwork, allows to interpret and explain the agent's choices. \n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "NOTE: as I declared in the \"Reviewer Expertise\", my knowledge in the area is limited (I do not have direct experience with temporal logic, and with inverse reinforcement learning or control). \nI think that I understand the general idea of the work and the logic that connects each step. In this sense, I think that the paper is well written. I am however uncertain about the usage of the terminology \"risk\", and it seems that the algorithm is applicable to a small set of tasks, and this is not reflected in the abstract. \n\n__Strenghts__\n\nThe paper presents a method to mimic the behavior of a demonstrator based on his risk awareness. \nThe model provided relies on logic and therefore is interpretable. Furthermore, the model is made differentiable by replacing classic $\\max$ and $\\min$ with their soft version. The differentiability allows the model to be trained with classic gradient-based techniques. \n\n__Weakness__\n\nI am not sure that I understood correctly some parts on of the paper. Risk, according to my knowledge, can be seen as a modification of an original distribution $p$, so to give more \"probability\" to rare \"bad\" events (risk-averse) or rare \"good\" events (risk-seeking). Common measures are \n- Value at risk (VaR) (returns the value at a given quantile)\n- Average value at risk (AVaR) (returns the average of the distribution from 0 to x quantile)\n- Entropic value at risk (EVaR) (returns the average under a modified distribution $q$ such that $\\max_q \\mathbb{E}[X]$ with $D_{\\text{KL}}(q||p) <= const$).\n\nEquation 4 presents a sort of \"potential field\", and it is related, if I understand correctly, to the distance of two vehicles. The risk measure presented, does not seem to be really a risk measure. Perhaps the name is deceiving. \nDoes this \"risk\" measure apply therefore only to autonomous driving or to more general problems?\n\n\n\nIn general, the paper seems to be focused only on the particular case described in the example in Figure 1. From the abstract, it seems that the approach is far more general than the autonomous driving example, and autonomous driving has only been used to test it. However, from the paper, it seems that the algorithm is appositely designed for autonomous driving.\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In general, the paper seems reasonable and well done. However, the writing is a bit deceiving.\nIt is not clear what the risk measure is. And it is not clear if the method is specifically thought of for autonomous driving. \nIf so, I'd recommend the authors specify that more clearly in the title and perhaps in the title too.\n\nUPDATE\n==\n\nI want to thank the authors for their response. I appreciated the changes in the text. In my opinion, the exposition can be still enhanced, and I still think that the authors could have explained better how to generalize their approach, (i.e., lines 53 and 54 tell us that you can generalize, but not how you do that).\nIn general, my evaluation of the paper is positive, and I keep my score unchanged. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Poor: Limited knowledge of the area"}, "extra_text": {"issues": "As already remarked, I believe that the authors must better clarify \n- what is the risk measure considered (and why it is called risk measure). Does it only apply to autonomous driving or to more general set-up?\n- how could one generalize the algorithm to a wider scenario? Is it possible? (for me it is fair that the algorithm focuses only on autonomous driving, but then the motivation of the work should be better specified)."}}, {"review_id": "034QvoXZJen", "reviewer": "Reviewer_gdH3", "summary": "This work presents an architecture for learning to predict risk associated with other agents in a multi-agent setting (with focus on autonomous driving) in order to produce a risk metric to be used in planning. The architecture is based on a differentiable approximation of a temporal logic description of risks. The method takes a set of logic formulas which describe safe behaviour in terms of features, and is trained via imitation learning to produce risk management behaviour similar to an expert demonstrator. The learned risk model is interpretable. The method is compared against the human expert and 2 other approaches in an autonomous driving simulation, and is closer in performance to the expert driver than the existing approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Novel and interesting approach\n- Applicable to a number of important robotics problems\n- Results show a good improvement compared to existing methods\n\nWeaknesses:\n- Some parts of explanation could be made clearer", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The quality of the approach and results appear to be good. Presentation clarity is mostly good, with a few areas where clarity could be improved which should be revisited.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "- Line 17 - turning across traffic rather than turning left may be clearer for international readers\n- Line 38 - should be ‘risk-averse’, not ‘risk-adverse’; same on line 39, 50, 238\n- Would be good to briefly introduce 'ego'/'ado' terminology as this doesn't appear to be commonly used in other work\n- Possible inconsistent use of \\omega}_T between equation 2 and equation 3 - is the same symbol representing expert trajectories in one equation vs agent generated trajectories in the other?"}}]

Author rebuttal:

Meta-review: In this paper the authors, by leveraging tools from temporal logic, design a framework to learn risk-aware trajectory planners. The framework is evaluated on a real-world driving dataset.

In general, the reviewers agree that the proposed idea is novel and useful, and I concur with this assessment. However, they also point out some weaknesses, including some lack of details in the exposition of the technical approach and a somewhat weak experimental evaluation (e.g., the experiments only consider two, rather simple logical rules).  The authors should carefully address the reviewers' concerns in their rebuttal.

UPDATE POST DISCUSSION PHASE: I would like to thank the authors for their comments and clarifications during the discussion phase. The reviewers generally agree that this paper presents an interesting idea and provides a useful contribution. On the other hand, some aspects of this paper still need to be improved - in particular, in terms of its exposition clarity (for example, the authors should better discuss to what extent they leverage the methodology presented in [14]). The decision is to accept this paper, but the authors should carefully address all reviewers' comments in the final version of this paper, with a key focus on improving exposition clarity.
Decision: Accept (Poster)