Constrained Hierarchical Deep Reinforcement Learning with Differentiable Formal Specifications
Authors: Zikang Xiong
Joe Eappen
Ahmed H Qureshi
Suresh Jagannathan
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Reinforcement Learning
Differentiable Formal Specification Language
Robot Navigation
Robot Planning and Control

Abstract: Formal logic specifications are a useful tool to describe desired agent behavior and have been explored as a means to shape rewards in Deep Reinforcement Learning (DRL) systems over a variety of problems and domains. Prior work, however, has failed to consider the possibility of making these specifications differentiable, which would yield a more informative signal of the objective via the specification gradient. This paper examines precisely such an approach by exploring a Lagrangian method to constrain policy updates using a differentiable style of temporal logic specifications that associates logic formulae with real-valued quantitative semantics. This constrained learning mechanism is then used in a hierarchical setting where a high-level specification-guided neural network path planner works with a low-level control policy to navigate through planned waypoints. The effectiveness of our approach is demonstrated over four robot dynamics with five different types of Linear Temporal Logic (LTL) specifications. Our demo videos are collected at https://sites.google.com/view/schrl.

Under review as a conference paper at ICLR 2023
CONSTRAINED HIERARCHICAL DEEP REINFORCE- MENT LEARNING WITH DIFFERENTIABLE FORMAL SPECIFICATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Formal logic specifications are useful tool to describe desired agent behavior and have been explored as means to shape rewards in Deep Reinforcement Learn- ing (DRL) over a variety of problems and domains. Prior reward-shaping work, however, has failed to consider the possibility of making these specifications dif- ferentiable, which would yield a more informative signal of the objective via the specification gradient. This paper examines precisely such an approach by explor- ing a Lagrangian method to constrain policy updates using differentiable temporal logic specifications capable of associating logic formulae with real-valued quan- titative semantics. This constrained learning mechanism is then used in a hierar- chical setting where a high-level specification-guided neural network path planner works with a low-level control policy to navigate through planned waypoints. The effectiveness of our approach demonstrated over four robot configurations with five different types of Signal Temporal Logic (STL. specifications. Our demo videos are collected in httths://sstts. wow.:eeessggogll.eoooo/eeeseesee
INTRODUCTION
Specifying tasks with precise and expressive temporal logic formal specifications has a long his- tory (Pnueli [1777] Kloetzer Belta, 20088) Wongpiromsarn al.] [012] Chaudhuri et al.) [021)) but integrating these techniques into modern learning-based systems has been limited by the non- differentiability of the formulas used to construct these specifications. In the context Deep Rein- forcement I Learning (DRL). line of recent work (Li et al. 2017 Hasanbeig et al., 2019b] Jothimu- rugan et al. 2019 Icarte et al. 2022 tries to circumvent this difficulty by turning Linear Temporal Logic (LTL) specifications into reward functions used to train control policies for the specified tasks. The quantitative semantics introduced yields real-valued information about the task that can then be used by reinforcement learning agents via policy gradient methods. However, the sample complex- ity of such policy gradient approaches limits the scalability of these algorithms, especially when comes to extracting reward functions from complex specifications (Yang et al. 2022). Moreover, these techniques do not consider how to effectively leverage the differentiability of the quantita- tive semantics associated with these specifications to yield more accurate gradient than the policy gradient estimated from the LTL reward and samples.
Interestingly, as show in this paper, this differentiability property can indeed be leveraged to meaningfully constrain policy updates. Previous approaches (Schulman et al. 2013) 2017] Achiam al.) 2017) constrain policy updates using KL-divergence and safety surrogate functions. For ex- ample, Achiam et al. (Schullan et al. (2017) use Lagrangian methods for this purpose. Based on the same Lagrangian methods, we consider how to constrain policy updates with differentiable formal specifications (Leung al.] [020 2022) equipped with rich quantified semantics, expressed in the language of Signal Temporal Logic (Mae) Nickovic 2004). This semantics gives us the ability to specify various tasks with logic formulas and realize them within hierarchical reinforcement learning framework.
Instead of burdening single policy to satisfy formal specifications and achieve control tasks simul- tancously (Li et al., (2077) Hasanbeig et al., [2019b Jothimurugan et al., 2019), we choose to learn hierarchical policy. Hierarchical policies have proven be effective in DRL with complex tasks

---

Under review as a conference paper at ICLR 2023
(Sutton et al. 1999; Nachum et al. 2018 Jothimurugan al. 2021 carte al. 2022) In contrast to previous DRL techniques integrated with LTL (Jothimurugan et al. 2021 carte et al. 2022). however, we replace multiple low-level options (Sutton et al., 1999) with single goal-condttioned policy (Schaul et al., 2015 Nachum et al.) 2018). A high-level planning policy, constrained by a formal specification, provides sequence of goals to guide a low-level control policy satisfy this specification. Additionally, because we wish for a learned policy to satisfy tasks as fast as possible, the high-level policy and low-level policy are jointly trained for both the satisfaction rate and the number of steps required to complete the objective. Finally, we also show novel applications of a neural-ODD policy as a high-level policy and integrate neural network-baser predicate functions as part of our specification framework.
Our contributions are as follows. (1) We propose a programmable hierarchical reinforcement learn- ing framework constrained by differentiable STL specifications, which avoids the sample complex- ity challenges of previous reward shaping work, and scales to benchmarks with high dimensional environments. (2) We show that the joint training of the high-level and low-level policy in this hierarchical framework provides better performance than training these components individually. (3) We demonstrate that our framework can be easily extended with neural predicates for complex specifications, such as irregular geometric obstacles that would be difficult to specify using purely symbolic primitives.
2 BACKGROUND
Coal-Conditioned DRL Schaul et al. (2015) shows that training a single policy conditioned by
multiple goals using only one neural network feasible and can act as universal option model Yao et al. 2014). Given a goal g and an agent observation og, the action = T(o (99 is predicted by the goal-condition policy <(; | g). Intuitively, a leads an agent to get "closer" to the goal g. Iteratively calling policy π (o: g) in a loop will finally lead an agent to reach goal g.
Two-layer Hierarchical DRL Combining a high-level planning policy with low-level control policy can often expand the range of problems solvable by DRL algorithms (Florensa et al.) 2017] Shu et al. 2018). Given a high-level planning policy x² G - G mapping all historical goals to the next goal, low-level policy x' O, G - A maps the observation conditioned by goal to action space A. At time step t, supposing that the low-level policy is toward i-th goal, the action is In this work, the low-level policy n¹
computed with a, n' (orlg,) where 9i T" [[go...... ..... ........................ .]). is also called the control policy, and the high-level policy is identical to the planning policy.
Lagrangian Methods in Constrained DRL Lagrangian methods solve constrained maximization problems. For a real vector x, consider the
problem:
max f(x) s.t. h((x)
This can be expressed as an unconstrained problem with the Lagrange multiplier X. Let C(x, A) f(x) + Xh(x).
(x" x+) )++ arg min max C(x, X x
which can be solved by iteratively updating the primal variable x and dual variable with gradients (Stooke et al. 2020). The here acts as "dynamic" penalty coefficients for updating real vector x. Lagrangian methods are widely used in the policy gradient update of many popular constrained DRL. algorithms (Schulman et al. 2017. Achiam et al.) 2017). Note that to compute the gradient through the constraint function hx). the function h must be differentiable. Since we want to constrain training with formal specifications, we, therefore, introduce a differentiable formal specification language in the following sections.
TLTL Syntax and Operator Semantics
The syntax of TLTIL contains both first-order logic oper- ators A (and). - (not), V (or), + (implies), etc., and temporal operators O (next), Qja,8) (eventually), [a.6] (globally), Uf, (until). The initial time a and end time b "truncate" a path. For example, la.6] qualifies property that globally holds during time and b. The syntax of TLTL is recursively defined via the following grammar:
T 1 P
ó A loblovo b + + Oo (a1,1 Paabio & Maa
2

---

Under review as a conference paper at ICLR 2023
Given a path and state space S, TLTL formula <(7) S" + B maps path to Boolean value. Discrete Boolean values, however, make computing a continuous gradient infeasible. More specifically, need quantitative semantics to enable differentiable specifications; adapt Signal Temporal Logic (STL.) (Maler & Nickovic 2004) that equips TLTI. specifications with quantitative semantics for this purpose.
Quantitative Semantics The full definition of the quantitative semantics is provided in Ap- pendix We introduce four of the semantic rules here relevant to describe backpropagation through specifications. p(T, Φ) denotes real-valued function evaluated on path T and represents the quantita- tive semantics of specification Φ) In this logic, we require that a true predicate (T) always has pos- itive value k. which we write as ((Taa:)) T) k. Every user-defined (or neural network predicate) P must also be accompanied by its quantitative interpretation p(s, P) in the real-value domain; here, the return value is greater than 0, denotes that the predicate satisfied. For example, a reaching- task predicate Prach can be defined as p(s, Prench) :- -||s all + where is state, y is goal, and is positive threshold; state is close to goal g. p(s, Preach) ) will positive. In Sec. [3. we further show that neural network can be learned quantitative measurement of a predicate. For the and (A) operator, p P(Taa))) ø A ()) = min(p(Tass) Φ), ([Tao) +)). The quantitative semantics for temporal operator (a.66] (Eventually) defined as ((Ta.)),, (a.ojo) max P(T(a))
Backpropagation through Specifications The STL semantics defines a differentiable quantitative measurement for first-order logical and temporal operators, which allows gradi- ents to backpropagate through them (Leung) et al. 2020 2022). Considering a coverage specification containing two goals A, B with goal-reaching predicates OA ΦB as illustrated in Fig. the specification is:
A
B
Pcoerr 01o,TI A Op,T
(1)
Figure 1: Demonstration of incorporating differ- entiable specifications and neural predicates to satisfy a Coverage specification that requires an agent visit two goals (A and B) while avoiding a complex obstacle (the "duck"))
The quantitative semantics predicates Φ and dn. as well as A and 10 are detailed in Sec. 2 We consider path with
goal-reaching
three waypoints (T 2). Given the initial path [90. 91, 92], Pcover can be evaluated as
P (To.2) (Toovr)) min max [E|U,2]
.A)) max P(T Φ2)
ΦA), LE(0,2]
All the operators in this semantics, including min, max, p( Φ.) and slicing (i- :D. differentiable and can be easily compiled to a computational graph with any auto-differentiation framework (e.g.. PyTorch, JAX). Hence, we can differentiate through Eq. (2). Step II and III in Fig. conceptually shows how backpropagation generates path satisfying Pcover. A step-by-ster example illustrating the entire updating process is provided in Appendix
APPROACH
3.1
NEURAL PREDICATES
The specification grammar allows specifications to use predicates whose interpretation is given by complex quantitative functions such as a neural network. For example, we can learn a neural network predicate for collision-avoidance tasks with neural Signed Distance Functions (SDF) (Park et al.) 2019). A neural SDF fa(p, obj) represents the closest distance from a given point p to the surface of the object obj. When is inside the object, the sign of fa is negative. The neural predicate fa can be seamlessly incorporated into STL specifications - when a waypoint is inside the object, fa is negative (evaluated as 1); when waypoint is outside the object, fa is evaluated example, given a predicate of the duck obstacle shown in Fig. [ datck = fa(p, duck), we can write collision-ane avoiianen-vviddnce specification Rduck. Step in Fig. shows that increasing the value of
T. For
3

---

Under review as a conference paper at ICLR 2023
dduck with gradient ascent tweaks the waypoint to be outside of this duck obstacle. Details about the neural SDF are provided in Appendix
The predicate has been interpreted as quantitative measurement of given state (i.c., function
P: + RR. The reward function is also exactly a function mapping from a state to a reward value.
Thus, techniques such as inverse reinforcement learning Abbeel & Ng (2004) ; Ziebart et al. (2008)
can be applied to learn a differentiable predicate without explicitly labeled data. pha et al. 2019)
provides a framework that makes the structure in STL also learnable. Our primary focus is not on
investigating the diversity and application of neural predicates. We point out that our framework can be easily extended with differentiable functions generated using any relevant method, and leave details about the specific techniques that should be used in future work.
3.2 CONSTRAINED PATII PLANNING POLICY
Sec. 2 describes how gradients backpropagate through formal specification and how to use them to refine waypoints. However, directly tweaking the waypoint for different initial paths is com- putationally expensive for real-time deployment and cannot leverage learning techniques to yield better-quality plans. To alleviate these issues, introduce path path-planning network.
Path-Planning Network We employ a learnable recurrent neural-ODE planning neural network to generate planning paths as sequences of waypoints. Neural ODEs (Chen al. 2018) can back- propagate with long paths effectively. The choice of a Recurrent Neural Network (RNN) is because temporal logic tasks time-dependent. An example task may be to visit points Pl. P2. P3 sequen- tially. Without memorizing the visited points, a neural network cannot decide the next point that is to be reached. From a sampled initial position go. we run the neural ODE with fixed timestep (i.e.. Euler ODE with timestep 1) and generate goal path (i.c., waypoints) [90- 91.... [T]] Given a Gated Recurrent Unit cell (GRU) (Cho al. 2014), and hidden state h;,
gi+1 = g: + GRU(g, h4)
The idea here is that the GRU cell predicts the change of waypoint instead of directly predicting the next waypoint. An illustration figure for this structure is provided in the Appendix (Fig 10b).
Sperification-Consttaaiee Policy Gradient Because the GRU-ODE suitable for path planning, we choose it as our high-level plamning policy. For computing policy gradients, we modify the GRU- ODE as a stochastic policy. The Gaussian high-level policy is x² N ot), where gr is the mean and is the variance. gt is predicted by the GRU-ODE shown in Eq. (3). and is predicted by linear transform from the hidden state ht. Consider the Lagrangian methods Sec. and objective function maximizing the expected reward under constraint:
C(0, A)
E T-aa
+ A(e (
(4)
where e > 0 is positive number. The introduction of is because we want p(T, Φ) to be positive (evaluated as T) when L converges optimally. While optimizing with gradient ascent, we first update the high-level policy parameters 0 with the specifiiationn-conttaannee policy gradient:
(Velog log T²(T))²(7) AVop(T.0)) - )
VC(0,))
where Vo log (T)) is the standard policy gradient, and P(T, o) is a quantitative specification. We can compute Vop(T, Φ) because is sampled from π.. In practice, this requires that is sampled with reparameterization (Kingma & Welling 2013). After updating the primal variable 0, we update the dual variable À with gradient: VxL(0,
p(T, o) in descent direction.
The reward (T) contains two parts: the quantitative specification p(T, Φ) and the control reward (T) introduced later. We define r"(r)= p(T, Φ) + (T). Note that not only use p(T, Φ) as part of the constraint function, but also consider it as part of reward ,² A larger P(T, Φ) means that the specification is "closer" to being satisfied. (7) = Σ r is cumulative of control reward r when following path A larger value for r' (r) means the generated path is easier to be followed by the low-level control policy. The involvement of r' (T() lets the planning policy also consider the capability of the low-level control policy.
4

---

Under review as a conference paper at ICLR 2023
3.3 CONTROL POLICY
The low-level control policy (0x g is goal-conditioned policy, where ÿ is sampled from high- level policy z N² (o 9) trained with PPO (Schulman et al. 2017). and the control reward rf [ - l2 و1 rell2. where 2; is the position of robot at time t. r will be positive if and only a robot gets closer to the planned goal 9 in one timestep. We train the control policy and planning policy jointly in DRL loop. The detailed algorithm is in Appendix
4 EXPERIMENTS
We present experimental details related to robot dynamics, environments, and specifications in Sec. 4.1 The remaining subsections aim to answer the following guestions: Q1. How does our approach compare to other DRL approaches with LTL specifications? Q2. What is the performance of the policies trained with our algorithm? Q3. What do we gain by jointly training the plan and control policy in hierarchical structure? Q4. What if we replace the high-level policy with path planner? QS. Why do we need the GRU-ODE as the planning policy network? Q6. Why is the specification-constaaine policy gradient better than the policy gradient with reward shaping?
4.1 DYNAMICS, ENVIRONMENTS, AND SPECIFICATIONS
Dynamics and Environments All the robots we evaluated are shown
in Fig 2 Drone is collected from PyBullet Drone (Panerati et al. 2021 Point, Car and Doggo are collected from safety gym (Achiam & Amodei 2019). Their corresponding state and action space dimensions, denoted as (state dim, action dim), from (a)-(d) are (18, 18), (14,2), (26,2), and
(a) Drone
(b) Point
12), resp.
We provide the illustrative figures for the environments (i.e., plane, ob- stacles) in Appendix G The Drone robot operates in an environment that contains the duck obstacle described carlier, while the other robots oper- ate in an environment containing walls. The specification of the duck and walls are represented neural SDF in our task specifications.
(c) Car
(d) Doggo
Figure four Robots
Specifications We consider five types of specifications. They are Se-
quence, Coverage, Branch, Loop, and Signal. Sequence tasks require robot to reach goals in given order. In contrast, the Coverage tasks only require a robot to reach all goals without consider- ing the order in which they are visited. Branch tasks are defined in terms of conditional behaviors. For example, visiting A then also visit B: if visiting C then also visit D. Loop tasks ask robots to repeatedly visit set of goals, while Signal tasks ask robots to repeatedly visit set of goals until a certain condition is satisfied. All tasks also require robots to avoid obstacles modeled with neural network predicates. The formal definition of these five task types is provided in Appendix G
4.2 TRAINING PERFORMANCE (Q1)
We compare the training performance of our algorithm, Specification Constrained Hierarchical Re- inforcement Learning (SCHRL), with previous works, including CRM and DHRM from Icarte al.
2022), and TLTL (Li et al. 2017). Approaches such as those described in Jothimurugan et al. (2019) 2021 cannot directly handle Loop and Signal tasks, so we cannot meaningfully compare their ap- proach with ours in our experiments. HTLTI is a hierarchical version of TL.TI, where a high-level planning policy is trained with a TLTI. reward. The difference between this algorithm and ours the objective function of high-level policy, where remove the Lagrange item A(e p(T, Φ) in Eq. (4). To the best of our knowledge, HTLTL has not been discussed in any previous work; it is introduced purely an ablation experiment to assess the effectiveness of our algorithm.
CRM and TLTL are non- non-hieran ain-hierrcchial algorithms. They need to control a robot and satisfy a task with single controller. The state dimensions of our robots are at least 14 (on Point) and up to 58 (on Doggo); these continuous control tasks pose pressure on the control side, and CRM and TLTL fail to generate reasonable controller other algorithms with the same amount of samples all
5

---

Under review as a conference paper at ICLR 2023
SCHRL
HTLTL
DHRM
CRM
TLTL
doggo-seq
doggo-cover
doggo-branch
doggo-loop
doggo-signal
Figure 3: The quantitative specification scores (y-axis) against the simulation steps (x-axis) during training. We trained every algorithm with 5 different random seeds. The complete steps for Doggo 1 x 10 Fig. 12 given in the Appendix provides the results of other robots. The tasks from left to right are Sequence, Coverage, Branch, Loop, and Signal, respectively. Any y-axis value above these green dash lines is greater than 0 and means specification is satisfied (e.g., in coverage task, the generated path gets closed enough to all the goals).
the tasks as shown in Fig. and Fig. 12 ever, it requires every transition in a reward machine to learn an option policy which is extremely challenging specification and control complexity increase. It also cannot reuse the existing option policy for different transitions. For example, if two transitions in the reward machine have the same go-right requirement, DHRM will learn two separate policies. For Doggo with a more complex control task, the DHRM cannot learn reasonable policy to achieve the tasks in given steps (see Fig. 3). but can get higher scores than CRM and TLTL on Sequence and Coverage tasks on Point with simpler control tasks (see Fig. 12 Appendix H
the Appendix H DHRM is hierarchical policy. How-
The goal-conditioned policies applied our HTLTL and SCHRL algorithms avoid building multiple policies. More importantly, on the control level, all sampled data is used to train single goal- conditioned control policy instead being distributed to different policies like DHRM. As a result, SCHRL and HTLTL perform better than DHRM given the same number of samples. The difference between SCHRL and HTLTL is their use of specification constraints. The training results in Fig. 3 show that specification constraints significantly improve training both w.r.t the number of samples required and the best score it can achieve. Moreover, we noticed that HTLTL could reach higher rewards when compared with DHRM, CRM, and TLTL, which lends support to our of hierarchical RL structure with goal conditoodt control policy and GRU-ODE planning policy.
4.3 SATISFYING RATE AND STEPS TO REACH (Q2)
The performance of trained policies is measured with two metrics. Firstly, we care about whether robot can achieve a specified task. There may be many reasons that cause a robot fail task. For example, a robot can fall down, fail to avoid obstacles, or be misled by badly planned path. Our policies demonstrate higher-than-95% satisfaction rates on all the tasks the Drone, Point, Car, and Doggo, shown in Table 1 and Table 3 in Appendix
We also care about how fast robot can achieve task. We summarize the number of steps needed to achieve Doggo task in Table Table Appendix ggiees details for the other robots in our study. Note that "Loop" requires a robot to keep looping between some goals, so we report the number of steps when it cumulatively achieves the given goals ten times. The data in Table T is the average (before +) and standard deviation (after +) of five policies trained with different random seeds.
Table 1: Performance of Doggo Tasks
Tasks
Seq
Cover
Branch
Loop
Signal
Satisfaction Rate
0.98 + 0.02
0.96 0.02
0.97 0.01
0.95 0.02
0.97+ 0.01
Stops to Reach
454.89 + 88.84
545.95 + 129.76
433.20 # 100.47
1431.13 + 333.19
1523.00 + 278.10
The performance results in Table T are achieved by different components in our algorithm. ablation study in Sec. 4.4 that the absence of these components decreases performance.
Our

---

Under review as a conference paper at ICLR 2023
4.4 ABLATION STUDIES
We conducted ablation studies on Doggo (the robot with the highest state/action dimensions) to justify our design choices. The data in Fig. and Fig. are summarized from 5 groups of 100 simulations. All the Steps to Reach are normalized within each task.
4.4.1 HIERARCHICAL POLICIES ABLATION (Q3, Q4)
We trained the planning and control policy jointly in each epoch, as detailed in Appendix when learning the goal-conditioned control policy. the control policy is specialized the goals generated from the planning policy; when learning the planning policy, the control policy's s reward is also considered in its reward function (as detailed in Sec. (.2). To demonstrate the benefits of joint training. we replace the control policy with general goal condiioond control policy, and replace the planning policy with a gradient- based path planner, separately.
SCHRL
General Controller
Gradient Planner
0.75
0.25
0.00
0.00
dogge-ag
diggoecoerr degge-branch
deggo-loop deggoosignal
daggowseq
deggoecoowr
doggedemenh
doggoshop
doggn-signal
Figure 4: Policies Ablation
General Goal-Conditioned Control Policy A general goal-conditioned policy can be obtained by training the policy with randomly sampled goals and the same reward function introduced in Sec. 3.3 The general goal-conditioned policy is used replace the control policy an SCHRL hierarchical policy; Fig. shows comparative results. Although all general controllers (gold bars) have only a slightly lower satisfaction rate than SCHRL. (red bar), they require notably higher number of steps to reach. This is expected because general goal-conditioned policies do not adapt to the goals generated by a planning policy. For example, when sharp turn happens in the planned path, general goal-conditioned policy reacts more slowly than the policy learned by SCHRL. Note that on the Signal task, the difference steps to reach can be as high as 24%.
Gradient Planner gradient planner can backpropagate through a differentiable specification, refine the waypoints with a gradient as shown in Fig and generate path. We replaced the planning policy with the gradient planner an SCHRL hierarchical policy, and show the comparative results in Fig. On task Sequence, Coverage, and Branch, the original SCHRL policy performs better in terms of both satisfaction rate and steps to reach. The largest difference in satisfaction rate (66%) the Coverage task. On the Sequence task, the SCHRL policy requires 29% fewer steps average to achieve a task. On tasks Loop and Signal, the planning policy and the gradient planner perform similarly, and both have around 95% satisfaction and similar steps reach counts.
Sequence, Coverage, and Branch give flexibility to choose waypoint (cyan circles in Fig 5a and Fig. (bb) between two goals. For example, Sequence task may ask a robot to reach A then B, but before reaching B. the robot can reach an additional waypoint in the middle. This mid- dle waypoint can help a robot to cir- cumvent an obstacle that intersects a straight-line path between two goals.
(a) Planning Policy
(b) Gradient Planner
Figure Planning Paths
However, as the specification only re- quires all the waypoints be outside the region of an obstacle, gradient planner only optimizing on the specification will not know how to place these middle waypoints appropriately. An example is shown in Fig. 5b where these middle waypoints are placed randomly. In contrast, because

---

Under review as a conference paper at ICLR 2023
planning policy also considers the control reward as optimization objective, badly placed way- point will result in the control policy failing to reach goal and being assigned a bad reward. Hence, planning policy can learn to place waypoints shown in Fig. Sa Because the path generated by the planning policy is shorter and yet does not intersect with obstacles, this path is easier to follow and guide robot finish the task faster. In addition to the satisfaction rate and the steps to reach, we also note that a gradient planner needs additional computation time for different initial- ization positions, while a well-trained planning policy can plan faster with forward propagation. A comparsion on planning time is provided in Appendix
4.4.2 PLANNING NETWORK ABLATION (Q5)
Our planning policy is built upon a GRU-ODE, which has benefits both in terms of memory and gradient backpropagation.
Fig. we show that GRU-ODE outperforms both Multilayer Percep- tron (MLP) and GRU policy in almost every case. For example, on Sequence task, the GRU-ODE outperforms MLP and GRU 65% and 51% the satisfaction rate and 29% and 18% on the steps to reach, respectively.
GRU-ODE
MLP
GRU
Satisfaction Rate
Steps to Reach
1.00
1.00
0.75
0.50
0.50
0.25
doggo-5eg s04
doggo-cover
branc-branch
60gg0-1o0p
doggo-signal
d0ggo-904
dopg0-cover
doggs-branch
doggo-loop
dogg0 signal
Figure 6: Planning Network Ablation. The GRU policy on Loop task and the MLP policy on Signal task failed to reach all the time, so their data is absent from the plot.
One exception that the MLP the Loop task works equally well as GRU-ODE. This is because solving the Loop tasks does not re- quire memory about the past. For example, in Fig. to generate path that satisfies the Loop constraint, when we know one goal is located the upper-right green circle, policy can always predict the next goal as the lower-right green circle to build one edge of square loop path with whatever history has. However, this not the case for other tasks such as Sequence, where agent needs to reach then reach B. If the robot reaches first, it still needs go to A. Without memory of the past, an MLP policy does not have the ability to make a decision in this case. This explains why works badly other tasks (e.g., 0% satisfaction rate Signal task). Although the GRU policy has a memory of the past, longer and more complex paths still pose training challenges. For example, the GRU policy has a low satisfaction rate on the Loop and Signal tasks, both of which have long paths. Details of these five tasks are provided in Appendix G
Figure Loop task
4.4.3 CONSTRAINED POLICY GRADIENT ABLATION (Q6)
Previous reward-shaping work updates policy with sample- based Policy Gradient (PG). We both the PG (eq. minus A(e p(T, Φ)) and our Constrained Policy Gradient (CPG) (eq.
Table 2: PG vs CPG Ablation
times on all the high-level path plan- ning tasks within 10000 gradient updates. The gradient updates will stop if over 95% of planned paths satisfy the
Tasks
CPG
Seq
9800 + (1 (1 (5)
640 102 (5
Cover
N/A (0/ 5)
784 (5)
STL specification. In each column, the number before and after + is the mean and standard deviation gradient up- dates, resp. N/A means in 5 runs, the algorithm failed to train a planning policy with over 95% satisfaction rate. The 2 numbers in the parentheses the runs successfully trained by policy and the total number of runs, resp. In all the 5 tasks evaluated, constrained policy gradient (with the Lagrangian term) results in 10x fewer gradient steps and can always successfully train path planning policy.
Branch
9700 200 (2 5)
720 120 (2/ (5/5)
Loop
N/A (0/
(5/5)
Signal
N/A (0/5)
1500 (5/))
8

---

Under review as a conference paper at ICLR 2023
5 RELATED WORK AND CONTRIBUTION
Related Work Integrating symbolie techniques with deep learning has led to recent interest in different learning domains (Manhaeve et all 2018 Chaudhuri et al. 2021 Badreddine et al. 2022).
L.TI. is a formal logic initially designed for system verification tasks (Pnueli 1977). Because LTT.
can encode complex system behaviors in a precise manner, a line of work (Fainekos et al.] 2005
Kloetzer & Belta, 2008 Kress-Gazit et al. 2009; Wongpiromsarn et al., 2012) considers its use in specifying various planning and control tasks. More recently, as advances in reinforcement learning have demonstrated its ability to solve challenging planning and control problems in an unknown, stochastic environment, a line of work has considered how to combine LTL.-specffied tasks with
reinforcement learning. However, this approaches (Fu & Topeu 2014 Li et al. 2017 Hasanbeig
et al.
Jiang et al.) 2020; Bozkurt et al. formulate this integration in terms discrete rewards. However, to the best of our knowledge, they have not considered applying STL (LTL equipped with quantified semantics) specifications in a constrained setting (Schulman et al.] 2015 Achiam et al. 2017), which can be solved with the wradient-based Lagrangian methods (Stooke et al.] (020). Lagrangian methods require differentiable constraint functions. Although differentiable STL specifications are discussed Leung et al. (2020 2022 they do not apply differentiable STL in a DRL setting.
2018
2019a b 2020
2022: Jothimurugan et al.. 2019
2020
Xu et al. 2020 learte et al. 2022] Zhang & Kan 2022
of either continuous
Jothimurugan et al.
(202T) y Icarte et al. 2022) notice that hierarchical structure provides benefits when learning from LTL specifications. However, they require learning multiple policies as options (Sutton et al., 1999). Moreover, one policy in Dothimurugan et al. (202I); Icarte et al. (2022) is only used for one state transition on the automaton built from the specification, which makes such an approach problematic as specifications become more complex. In contrast, we select a different hierarchical structure based on a goal-conditioned control policy and path planning policy (Levy) jet al., 2017: Nachum et al.] 2018: Zhang et al. 2020); such an approach has been shown to be feasible theoretically (Schaul et al. 2015). Only one control policy is required in this setting, and all data samples can be used to train this single control policy.
Our work is also related to constrained reinforcement learning (Schulman et al.) 2015 Achiam (ct al. 2017). programmable reinforcement learning (Andre & Russell| 2002 Yang et al.) 2021 Qiu & Zhu 2021), probabilistic temporal logic constraint (Hasanbeig al. 2019b] Jansen et al.
2020 and gradient-based robot planning and control (Ratliff et al. 2009; Campana et al. 2016
Leung et al.) [2200] Dawson & Fan 2022 Unlike previous reinforcement learning algorithms that usually focus on safety constraints, we extend our constraints to be more expressive using formal logic. Programmable reinforcement learning also specifies the behaviors of reinforcement learning agent in programmable way (Andre & Russell [2002] Yang et al.) (222];; for example, Qiu & Zhu (202T) learns specifications with an idea inspired by neural architecture search. However, (Andre & Russell (2002); Yang et al. (202T) also fall into the category of reward engineering with formal specifications. Qiu & Zhu (2021) design a domain-specifie language without loops to automatically compose policies and thus cannot handle some of our tasks, such as Loop. Hasanbeig al. (2019b) Jansen et al. (200 synthesize controllers and shields with probabilistic guarantees for satisfying LTL specifications under uncertainty. The uncertainty is embedded in the environment (e.g., the uncertainty in the structure of the workspace, and the agent's actions). Such uncertainty also exists in our environments manifesting stochastic dynamics and policies. Modeling these probabilis- tic behaviors and providing probabilistic guarantees of satisfying LTL. specifiicits in hierarchical settings is interesting direction for future work. Gradient-based motion planning (Ratlif' et al., 2009 Campana et al. 2016 Leung et al. 2020: Dawson & Fan, 2022) also leverages backpropaga- tion to generate plans. However, they typically only work when the dynamies (i.c., simulator) are known, differentiable and deterministie. Lastly, our high-level planning policy is related to plan- ning networks (Qurehhi et al. 2019), which supports fast online planning. However, learning such a planning neural network in a DRL loop, which benefits both the planning policy and control policy, has not been discussed in previous work.
Conclusions In this paper, we show how to leverage the differentiability of STL specifications to constrain policy updates in hierarchical reinforcement learning framework, which leads to framework co-optimizing the planning and control policies with challenging system dynamics and LTL tasks. We demonstrate that approach outperforms other DRL techniques equipped with LTL specifications when the LTL specification is used only for reward shaping. We also justified the design choices of our approach with detailed ablation studies.
9

---

Under review as a conference paper at ICLR 2023
REPLICATION STATEMENT
Our code is available in hutp::/gthbbb.ommmaroooyyoooooen
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Joshua Achiam and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. In Online, 2019.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
David Andre and Stuart Russell. State abstraction for programmable reinforcement learning agents. In Aaaifiaii, pp. 119-125, 2002.
Samy Badreddine, Artur d' Avila Garcez, Luciano Serafini, and Michael Spranger. Logic tensor net- works. Artificial Intelligence, 303: 2022. ISSN 0004-3702. doi: https://doi.or//10000/ jaartint. 103649. URL https: //ww:///wwwww/// a wcw.ncediiiicccccco..o.co.c rect. com coom//ccennc/articlle. pi ///////043722200000000
Alper Kamil Bozkurt, Wang, Michael Zavlanos, and Miroslav Pajic. Control synthesis from linear temporal logic specifications using model-free reinforcement learning. In 2020 IEEE In- ternational Conference on Robotics and Automation (ICRA), pp. 10349-10355. IEEE, 2020.
Mylène Campana, Florent Lamiraux, and Yean-Paul Laumond. A gradient- path optimization method for motion planning. Advanced Robotics, 11117-18):112611111 2016.
Swarat Chaudhuri, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando Solar-Lezama Lezama, and Yisong Yue. Neurosymbolic programming. Found. Trends Program. Lang., 7(3):158-24) 2021. doi: 1021561//500000044. URL https::/doi.
561 /5116550000049
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar, Oc- tober 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-11999 URL https://aclantholt hotpaaattiiooooooo//14111177 org/D14-1179
Charles Dawson and Chuchu Fan. Robust counterexample-guue optimization for planning from differentiable temporal logic. arXiv preprint arXiv:2203.02033. 2022.
Georgios E Fainekos, Hadas Kress Gazit, and George J Pappas. Temporal logic motion planning for mobile robots. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation, pp. 2020-2025. IEEE, 2005.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical re- inforcement learning. In International Conference Learning Representations, 2017. URL
ltttss://ppeneeviee net/ www.moddddlmoaaaaa....
Jie Fu and Ufuk Topeu. Probably approximately correct mdp learning and control with temporal logic constraints. arXiv preprint urXiv:1404.7077 2014.
Yann Gilpin, Vince Kurtz, and Hai Lin. A smooth robustness measure of signal temporal logic for symbolic control. IEEE Control Systems Letters, 5(1):241-246, 2021. doi: 10.1109/LCSYS. 2020.3001875.
10

---

Under review as a conference paper at ICLR 2023
Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Logically-conssrainee re- inforcement learning. arXiv preprint arXiv:/801.08099 2018.
Mohammadhosein Hasanbeig. Alessandro Abate, and Daniel Kroening. Certified reinforcement learning with logic guidance. arXiv preprini arXiv:1902.00770 2019a.
Mohammadhosein Hasanbeig, Yiannis Kantaros, Alessandro Abate, Daniel Kroening, George Pap- pas, and Insup Lee. Reinforcement learning for temporal logic control synthesis with probabilis- tic satisfaction guarantees. In 2019 IEEE 58th Conference on Decision and Control (CDC), pp- 5338 5343. IEEE, 2019b.
Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Cautious reinforcement learning with logical constraints. arXiv preprint 1rXiv:2002.12156, 2020.
Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate. LCRL:: Certified policy synthesis via logically-constrainee reinforcement learning. In International Conference on Quan- fitative Evaluation of SysTems, Springer, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano. and Sheila A Mcllraith. Reward ma- chines: Exploiting reward function structure reinforcement learning. Journal of Artificial telligence Research, 15:173-208 2022.
Nils Jansen, Bettina Könighofer, Sebastian Junges, AC Serban, and Roderick Bloem. Safe rein- forcement learning using probabilistic shields. 2020.
Susmit Jha, Ashish Tiwari, Sanjit A Seshia, Tuhin Sahai, and Natarajan Shankar. Telex: learning signal temporal logic from positive examples using tightness metric. Formal Methods in System Design, (4(3))364-338) 2019.
Yuqian Jiang, Sudarshanan Bharadwaj. Bo Wu, Rishi Shah, Ufuk Topcu, and Peter Stone. Temporal- logic-based reward shaping for continuing learning tasks. arXiv preprint arXiv:2007.01-998 2020.
Kishor Jothimurugan, Rajeev Alur, and Osbert Bastani. A composable specification language for reinforcement learning tasks. Advances in Neural Information Processing Systems, 32, 2019.
Kishor Jothimurugan, Suguman Bansal, Osbert Bastani, and Rajeev Alur. Compositional reinforce- ment learning from logical specifications. Advances in Neural Information Processing Systems, 1110026-100339 2021.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXhv:1312.6114, 2013.
arXiv preprint
Marius Kloetzer and Calin Belta. A fully automated framework for control of linear systems from temporal logic specifications. IEEE Transactions on Automatic Control, 53(1):287-297, 2008.
doi: 11.1 00.1109/AC.2207.99495533
Hadas Kress-Gazit Georgios E. Fainekos, and George J. Pappas. Temporal--ogic-basse reactive mission and motion planning. IEEE Transactions on Robotics, 25(6):1370-131,, 2009. doi:
(0110//TR0.2009203000
K. Leung, N. Aréchiga, and M. Pavone. Backpropagattion through signal temporal logic specifica- tions: Infusing logical structure into gradient-based methods. Int. Journal of Robotics Research, 2022.
Karen Leung, Nikos Aréchiga, and Marco Pavone. prop-pprongtttion through signal temporal logic specifications: Infusing logical structure into gradient-based methods. In International Workshop on the Algorithmic Foundations of Robotics, pp. 432-449. Springer, 2020.
Andrew Levy, Robert Platt, and Kate Saenko. 1712 12, 2017.
Hierarchical actor-critic. arXiv preprint
11

---

Under review as a conference paper at ICLR 2023
Xiao Li, Cristian- Vasile, and Calin Belta. Reinforcement learning with temporal logic rewards. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3834- 3839. IEEE, 2017.
Oded Maler and Dejan Nickovic. Monitoring temporal properties of continuous signals. In Formal Techniques, Modelling and Analysis of Timed and Tole--Tolerant Systems, pp. 152-166. Springer, 2004.
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Decpproblog: Neural probabilistic logic programming. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018, URL https: httpps://proceeeeeesee dc5a637ed5e62cc becb http:////ppoeeeee...eeeeee. www.pnsp......i......... http://.....aiiii.cc./////eee//a cttpp//p/////1///iieee////////
Paper. pdf
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
Jacopo Panerati, Hehui Zheng, SiQi Zhou, James Xu, Amanda Prorok, and Angela P. Schoellig- Learning to fly-a gym environment with pybullet physics for reinforcement learning of multi- agent quadcopter control. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
Amir Pnueli. The temporal logic of programs. In 18th Annual Symposium on Foundations of Computer Science (sfes 1977), pp. 46-57, 1977. doi: 10. 10.9/09S 1977..
Wenjie Qiu and He Zhu. Programmatic reinforcement learning without oracles. In International Conference on Learning Representations, 2021.
Ahmed H Qureshi, Anthony Simeonov, Mayur J Bency, and Michael C Yip. Motion planning networks. In 2019 International Conference Robotics and Automation (ICRA), pp. 2118-2124 IEEE, 2019.
Nathan Ratliff, Matt Zucker, J Andrew Bagnell, and Siddhartha Srinivasa. Chomp: Gradient op- timization techniques for efficient motion planning. In 2009 IEEE International Conference on Robotics and Aulomation, pp. 489-494. IEEE, 2009.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0633 2017.
Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In International Conference on Learning Representations, 2018. URL htt.:http////c htttps:://pperrevieee
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9144 PMLR, 2020.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
12

---

Under review as a conference paper at ICLR 2023
Tichakorn Wongpiromsam, Ufuk Topeu, and Richard M. Murray. Receding horizon temporal logic planning. IEEE Transactions on Automatic Control, 57(1))28171 2830, 2012. doi: 10. 10.1109/TAC.
2012.2195811.
Zhe Xu, Ivan Gavran, Yousef Ahmad, Rupak Majumdar, Daniel Neider, Ufuk Topcu, and Bo Wu.
Joint inference of reward machines and policies for reinforcement learning. Proceedings of the
International Conference Automated Planning and Scheduling, 30(1):590- (91590--98, Jun. 2020.
URL httttts:://ossaaaal www.aaaaa.iiginnnaaaeeeeeeeeeeee index. http:/pppp///////a/aiiiceeee
6756
Cambridge Yang. Michael L. Littman, and Michael Carbin. On the (in)tractability of reinforcement learning for LTL objectives. In Luc De Raedt (ed.), Proceedings of the Thirty-Firss International
Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 3650 3658. ijcai.org. 2022. doi: (0024963/ijcaii2222/550 URL https: http::://oooo.ooo dww.ooo...gg..ooo 24963/ jcaai/ccca.... wtt.:/222..202225000...
Yichen Yang. Jeevana Priya Inala, Osbert Bastani, Yewen Pu, Armando Solar-Lezama, and Martin Rinard. Program synthesis guided reinforcement learning for partially observed environments. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 29669-29683. Curran Asso- ciates, Inc., 2021. URL htttps: newrisss.eeurpp.. cttp://rr.ssscccppaee//222221112 cctpapper//22///1lele Fnee22275504111110500022222eeee2
pdf
Hengshuai Yao, Csaba Szepesvari, Richard S Sutton, Joseph Modayil, and Shalabh Bhatnagar. Universal option models. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran As- sociates, Inc., 2014. URL /tR/httss///prrceedngggsssu neurips cc/ppaper/2211///1eee. 99947ffa077cc333c4400004442eeee
pdf
Hao Zhang and Zhen Kan. Temporal logic guided meta q-learning of multiple tasks. IEEE Robotics and Automation Letters, 7(3):8194-8201, 2022. doi: 10. R..11099AA..2222.1888888
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency-constrained subgoals in hierarchical reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In-
formation Processing Systems, volume 33, pp. 21579-21590. Curran Associates, Inc., 2020. URL / new.iisss wttprripp..pppppp...eeee cctppaper//22///11eee. B5338177053333000676666666--888-
pdf
Brian D Ziebart, Andrew I. Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
APPENDIX
A HIERARCHICAL LEARNING ALGORITHM
We summarize our learning algorithm in Algo. Line 3 to 7 sample data with control policy conditioned by the planned path т. Line 8 to 9 is the policy updates introduced in Sec. 3.2 and
Sec. 3.3 respectively.
We point out that control and planning policies adapt to each other during training. The reward n used for the planning policy contains cumulative reward r'(r) of the control policy, enabling the planning policy to generate paths that are easier to follow. Meanwhile, the control policy is conditioned by the planning path, specializing it to the planning policy, allowing it to reach goals faster.
B TLTL SEMANTICS
The TLTL grammar rule is recursively defined as
T= 1 P - -olb eb Oo (a111 1aa,io Maapp
13

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Apeciffiation-Constaiiie Hierarchical Reinforcement Learning
Notions: Spec i Planning Policy n². Control Policy x', Rollout Buffer B, Sequence [-]. Observation o, Control Reward r", Planned Path 7, Initial Position go-
B +- 0; for i ,,2,,..,N, Ni do
for j = 1. N2 do
9u + Samplelnitial():
T +- ForwardPath 90)
// T is waypoint sequence. // Control policy follows т.
[o). [r*] = Rollout (otiooutttTTT)))) ;); B.add(r, [r]]);
Update Plan .B):
// Sec. 3.2
UpdatcConttoolPoleeye ( B):
// Sec. 3.3
10
B+00 0;
Φ
Φ
Φ
O agy Next
0 [a. Eventually
Globally
Until
Figure 8: Temporal Operator Semantics.
The semantics of first-order logic operators (,,A,V,=) is trivially defined. We summarize the semantics of temporal operators (O, Oja, D| 2(a,t) in Fig. All paths in Fig. start at time and end at time 6. Next (O)) means that logical formula holds at next state in path: Eventually (0 a. Φ) says that & will satisfied at least once; Globally (a),,y))) enforces the requirement that will always be satisfied; Until (o Ma,,6 asserts that always holds until t holds.
Additionally, we allow specifications to contain based predicates P 5 - [T, 1) that map state to a Boolean value. For example, we can express a safe predicate (safe(s) = + s & S, that holds precisely when does not belong to the set of unsafe states S.,. We combine these predicates freely in other specifications; for example, the following specification:
(Dia.sh a. Peufe) A (O(a,)) ((veeeee
Pivveness)
(5)
captures the requirement that an agent always stays in a safe region but eventually reaches a goal state (Puveeness).
C FULL SIGNAL TEMPORAL LOGIC SEMANTICS
A specification's quantitative semantics (i.c., STL) maps a path to a real value. signature is:
Sm R. We provide the full semanties below:
14

---

Under review as a conference paper at ICLR 2023
P(Taab), T)
=k
(k: ( 0)
p(Ta,b), P)
=F(Ta)
P(Tast)) -))
-((Ta:b) Ø)
P(Tass)) Φ A 46)
= =min(p(TT::b) 6), ; pl Tasb), vt)
P(Taatb), 6 VV 2b)
max(p((Teob) Φ), p (T:b) 15)
((T(abb), sb = vb)
maax(((Taa):: ) Ta:b). +))
((Ta:))) O$)
=P(Ta+100): (1)
p(Tast)) (0.00))
min p[T ())
(E|a.0]
a(Tabb), (1,b)10)
max (TT):)) ())
rE(a.b]
a(T Φ (1a
max rE(a.0)
min
((T v), min p(Ta,,7 .))
rela.t-)
T is defined as positive value k. According to the definition of P(Ta,)), 7)), the 1. is defined as -k. Value undetined in this semantics. For a predicate P, we associate with differentiable function F(s) S + R. This function can be any differentiable function that maps state to a real value. For example, one can define it as the negative distance to a goal plus a threshold or represent it with neural network in Sec. 3.1 The other rules define how compose different predicates in numerical ways. For example, the () (globally) operator is interpreted (until) is non-trivial. It comes from an equivalent formula of U(,,6/=
interpreted as a min operator between two predicates; the min operator
finite path. The semantics of U(a
& Ua,oje 01a,6) (16 A AD (1,0-110)
where f is time when 1/ holds.
All the operators in this semantics, including min, max, predicate function F(-,6) index ([-),, and slicing (i: : =D), are differentiable and can be easily compiled to a computational graph with any nuto-differrntiition framework (e.g., PyTorch, JAX).
The min and max can the semantics can soften as min and maz:
min so] ('min(() (v) of
mar = soʃ (oftmar x
where
sof softmin(y) e Σ, e-vi
softmar(v) e
e
Tha et al. (2019) y Leung et al. 2020
2022
Gilpin et al. 1021 demonstrated that such soften thick
benihts gradient- based STL path planning. Experiments about using soft STL (with soft min and max) and hard STL (with general min and max) as Lagrangian terms is provided in the Appendix.
D BACKPROPAGATE, THROUGH A COVERAGE SPECIFICATION
We diseuss how the gradient backpropagates a coverage specification example with two goals A, B specified by é A O n, resp:
Ócover A10,110 A O1rr100
15

---

Under review as a conference paper at ICLR 2023
The goal-reaching predicates Ω4 and Φg defined p(s, 6.) -||s اlو c, where is state, g is the position of either A or B, and is positive threshold. We consider path with three waypoints (T' = 2). Peover can be evaluated as
(T0.2) (cower) min max ((T(:2); PA); max ((T),2) (n) rE(0.2]
(E(0.2]
The yellow dot in Fig. is the initial waypoint and can be any valid initial state. The other two green points are planning waypoints. Because we want the path to satisfy the specification, we maximize the p (T0.2)- (bover) using its gradient. Fig. 9 shows how the gradient backpropagates through the Eq. 2 Before presenting the details, we point out that the gradient of min and max is only a gradient masking operator. For example, given æ1 22, min(x1,,2) will not backpropagate any gradient to the variable 22.
A
B
A
B
A
(a) Initial
(b) Evaluate
(c) Evaluate A and Update
A
DB
A
B
A2 <
DB
(d) Evaluate +
(e) Evaluate A and Update
(f) Final
Figure 9: Gradient backpropagation through a coverage specification. (a) shows the initial path and the distance to A and B. We do not modify the initial waypoint. It is detached from the gradient computation. (b) 0jo.21 (eventually)
max operator over a path. It will select the two short distances (larger negative distance, marked with check logos) to backpropagate. (c) A is evaluated as a min operator. Thus, selects the longer distance (smaller negative distance, marked with check logo) backpropagate. Now, tweak waypoint 1 to B with gradient. This figure simplifies the gradient updating process to one step for illustration purposes. In practice, the update can be more tortuous, and waypoint may take several steps to reach B. (d) and (e) are similar to (b) and (c), respectively, but they update on waypoint (f) shows the final planned path. This path covers both A and B.
E DETAILS ABOUT NEURAL SDF
The neural SDF fits an SDF fa(p, obj) with neural network. A neural SDF fa(p, obj) computes the closest distance from given point p the surface of object obj. When p is inside the object, the sign of fa is negative. One can easily get SDF with an object mesh (Park et al. 2019). However, the SDF gotten this way usually con-differentiable. Instead, we sample dataset of position- distance pairs (p, d) with such an SDF and train differentiable neural network to predicate the closest distance with a position p.
Fig. 10a is the contour plot of neural SDF This neural SDF is built from the duck obstacle in (e)-(f) Fig. [I More precisely, it is the contour plot for a cross-section of this 3D duck when y 0. The purple outline represents value O, with values inward decreasing monotonically. This neural SDF can naturally represent the differentiable function F the STL semantics in Appendix C For example, when want to define collision-avoidande predicate Φsafe on this duck obstacle. if a waypoint is inside this obstacle, the &safe should be interpreted as a negative value under STL semantics (i.e., 1)). Similarly, outside waypoint should have a positive value (i.e., T). Moreover,
16

---

Under review as a conference paper at ICLR 2023
GRU-ODE
Path
Ag
hv
hg
d.g
dt
dt
(a) The contour of duck neural SDF
(h) GRU-ODE
Figure 10: (a) The purple outline represents value O with values inward decreasing monotonically. (b) The initial point 9u is given. Feeding it to the GRU cell generates the time derivative dgo is generated by adding the dos and go. Iterating this procedure generate a path [9o. 91, ga].
dt
when maximizing an STL formula with gradients, the gradients will "push" waypoint inside the obstacle to the outside, as shown in Fig.
F DETAILS ABOUT GRU-ODE
We consider a neural ODE with a fixed timestep (i.e., Euler ODE with timestep 1). Given a GRU cell, goal 9s and hidden state h4, the next goal gi+1 Ii + GRU(g, h2). Fig. 10b illustrates the forward computation of the GRU-ODE. Note that
9 = 9o + GRU(go ho) + + GRU(g:-1, h4-1)
making it easy to backpropagate from 9i to any previous forward of GRU. This idea is similar to the ResNet (He al. 2016). which alleviates gradient vanishing in long-term backpropagation.
G TASKS AND SPECIFICATIONS
Car /cover
(b) Car Branch
(c) Car Loop
(d) Car Signal
(c) Drone Seq./Cover.
(f) Drone Branch
(g) Drone Loop
(h) Drone Signal
Figure 11: Illustaaioon for All Tasks. The Point and Doggo have the same setting with Car on all the tasks. The robots (Point, Car and Doggo) in Fig. Tlb (Branch) are randomly initialized in the bottom-left room; all the other tasks in the 4-room environment randomly initialize robots (Point, Car and Doggo) in the bottom-right room. All the Drone tasks randomly initialize the Drone box space with lower-bound (-5, 4, 3) and upper-bound 5,)
The Point and Doggo have the setting with Car on all the tasks.
17

---

Under review as a conference paper at ICLR 2023
The Sequential task requires robot to visit sequence of goals in order.
(6)
where f and Φr is goal spec. This formula means eventually reaching all the goals in specified time slots.
In Fig. Ila car is required to visit the upper-right (do). upper-left(11). bottom-left(dy), and bottom-right( blue spheres sequentially. The time slot assigned to Φn. Φ1, 62. and da are [1, 2]. [3,4]. 6). and 87, respectively. The time here represents the index of waypoint in path. In other words. in path with a length 8, the first or the second waypoint in this path should reach the upper-right blue spheres. Similarly, the Drone Sequence also requires the robot to visit 4 blue balls in Fig. Ie in the slots [1, 2]. [3, 4), [5, 6]. and (7.8).
As introduced in Sec. 2 the Coverage task's formula is
A QU.TT
The Coverage task shares the same map with the Sequence task. For Point, Car, and Doggo, they use the same map in Fig. [Ta] and Drone uses the map in [ie However, in the Coverage task, robot is only asked to visit the four spheres, but discards the order. In Eq. the T is assigned for all the robots.
Branch task can be seen as a collection if ... then clusters, e.g., if visiting then visiting B, if visiting C' then visiting D. We encode this type of task as
V (0p,mo A An, (Ø: = (1.r))))),
where N is even. This formula says eventually satisfying dhe and globally db implies that Φ2 will be satisfied eventually. The Φ; and Φ2, has the same color in Fig. IIbaadd and Fig, Tif This specification defines that robot must visit one goal, and it will visit the other one with the same color eventually. For all the robots, t = 2, T = 3 and N 6 Eq. ( All the Point, Car, and Doggo use the same map in Fig. 11b
The Loop task asks a robot to loop among some waypoints. It can be written as
(V
N
Vonn
oz
1=0
This formula says that globally, any of Φ; should be satisfied, and o; will not be satisfied next time. A waypoint in path will also stay in one of the giving goals specified by Φ., and the next waypoint will not stay here for the next time. In Fig. [Tc] the robot will loop among the four red spheres, and the Drone will loop among the three blue balls Fig. Tig The TLTL only supports finite paths, so we have to set a terminated time (T in Eq. (9). For all the robots, T 20.
Signal task a loop task with until operators:
N (V
N
(o UULTNNTI
(10)
This means looping until ΦN+1 satisfied and eventually visits the position specified by dv+22. For all tasks, the @x++1 is a counter predicate; it is evaluated as (11) until the 10th waypoint in path, which means robot needs to visit the position specified by Φo. .o....,4, 10 times, and eventually visits the position specified by @N+2. In experiments, T = 11 for all the tasks. In Fig. [ld robot will loop among the four corner spheres and visit these spheres ten times, and eventually visit the center sphere. In Fig. 1Th the Drone will keep visiting three pink balls ten times and eventually visit the highest yellow one.
All these formulas will be appended with obstacle-avoidane cluster [1 Pobs For the Drone tasks, the dabs is a duck neural SDF introduced in See 3.1 For the Point, Car, and Doggo tasks, the Jobs is neural SDF of all the walls.
18

---

Under review as a conference paper at ICLR 2023
H MORE TRAINING CURVES
We provide the training curves for Drone, Point, and Car in this section. All the data collected with training with five different random seeds.
SCHRL
HTLTL
DHRM
CRM
TLTL
drone-seq
drone-cover
drone-branch
drone-loop
drone-signal
point-seq
point-cover
cont-bbanch
point-loop
point-signal
car-seq
car-cover
car- car-branch
car-loop
car-signal
Figure 12: The quantitative specification scores (y-axis) against the simulation steps (x-axis) during training.
The SCHRL outperforms the other algorithms for robot dynamics and tasks. In the given steps, the DHRM learns better policies than non-hierarchical CRM and LTL on the Sequence and Coverage tasks on Point.
I ADDITIONAL POLICY PERFORMANCE
We report the SCHRL policy performance of Drone, Point, and Car this section. The data collected with 5 groups of 100 times simulations.
Table 3: Satisfaction Rates Drone, Point, and Car
Tasks
Seg
Cover
Branch
Loop
Signal
Drone
0.96 + 0.01
1.00 = 0.00
0.97 + 0.01
1.00 = 0.00
1.00 0.00
1.00 0.00
1.00 =
1.00
Car
0.98 1 0.00
0.96 = 0.01
0.95 1 0.01
0.94 0.01
0.95 = 0.01
The trained SCHRL policies generated decent satisfaction rates all the dynamics and tasks. The worst case appeared in the Loop task of Car, where it has 95% satisfaction rate on average. The Point robot almost can always satisfy a task expecting the Signal task with 99% satisfaction rate on average.
Table 4: Achieving Steps of Drone, Point, and Car
Sog
Cover
Branch
Loop
Signal
Drone
1453.36 373.71
1165.97 131.42
761.66 153.76
2342.76 + 213.68
2859. 220.96
Point
601.93 18.70
669.93 17.51
546.96
34.00
1861.21 126.20
1912.04 174.05
Car
$47.47-977.
592.56 + 153.10
508.48 + 136.58
1611.68 + 387.99
1802.66 66 26.79
19

---

Under review as a conference paper at ICLR 2023
The steps to achieve tasks summarized in Table 4 All the Loop and Signal tasks have longer planned paths. They are with higher achieving steps than the Sequence, Cover, and Branch for each robot.
PLANNING TIME
One benefit we mentioned in Sec. 4.4. is planning time. Compared with Leung et al.) 22020 2022
Gilpin et al. 2021 our approach requires less planning time when deployed online.
Table 5: Planning Time (in sec.) Comparision
Tasks
Planning Net
MIP
Gradient
Sey
0.0011222
0.096137
0.842844
Cover
0.001770
0,218563
3.484746
Branch
0.0014437
0,078942
7.204325
Loop
7.423811
Signal
0.002383
7,853121
2.670637
The planning policy is neural network that generates planning paths from randomly sampled initial states. A forward propagation in milliseconds will produce a path with MIP (Gilpin et al. [02TT) or gradient techniques (Leung al. 2020 2022 for several seconds to solve. We quantity the planning time in Table 5
The time in Table 5 is the average time (in sec.) over 100 runs. Our planning policy is at least 55X faster than MIP planners and 489X faster than gradient-based planners.
K GOAL-CONDI POLICY AND OPTIONS
Previous work learns one control policy (option) for each transition in the automaton built with an LTL specification. For example, a sequence task with four goals will require four policies.
Table 6: Samples Required until Convergence for Point
Goal Conditioned Policy
Option
Options
4 Options
Number Samples
2.93++5
2.01++5
4.02c+5
8.0e+55
Our work replaces these options with a more flexible goal-conditioned control policy. Unlike using an option to reach one goal, the goal-conditioned policy learns to reach any given goal. Because we train only one control policy, all the samples we during training are used for training this policy, while the samples in the options framework are distributed to train multiple policies. Over 5 runs, the average number of samples required to train policies reaching the convergence reward on the Point robot is in Table G
L SOFT AND HARD STL LAGRANGIAN TERMS
Table 7: Soft and Hard STL Constraints
Soltt STL
Hard STL
Seg
268.20 80.45
710.20 193.76
Cover
1090.60 a 428.56
1035.00 a 370.58
Branch
49.00 24.23
43.20 9,74
Loop
119.20 + 23.07
112.60 + 35.57
Signal
289.79
$88.00 128.79
We provided smooth STL Leung et al. 2020 2022
Tha al. (009 ; Gilpin et al. 2021 with soft
min and max in our implementation. In a constrained learning context, STL is used as part of a loss function. To explain the role of soft and hard STL in this context, conducted ablation study
20

---

Under review as a conference paper at ICLR 2023
showing the efficiency of soft and hard STL loss. We trained our planning policy network with both soft and hard STL loss, and stop training when the policy could generate 95% of paths satisfying the STL specification. Each cell in Table is the result of 5 runs. The mean and stand deviation are before and after +, resp. The results in Table are mixed. Soft STL. loss performs significantly better task seq and signal, but slightly worse the cover, branch, and loop tasks on average. Intuitively, choosing soft and hard STL is analogous to choosing Lg or Loo loss (L2 loss based on
MSE is "smooth" while Loo loss based on max is "hard").
21


Reviews:
[{"review_id": "wxIXnIRvOe", "reviewer": "Reviewer_urxX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to constrained deep reinforcement learning using signal temporal logic as a constraint language. The formulas defining the constraints are converted to algebraic constraints using the quantitative semantics of STL, which, in turn are incorporated as Lagrange terms in a dual ascent algorithm. These concepts are explored within the space of hierarchical RL. The results are evaluated on benchmark domains.", "summary_of_the_review": "I think that this paper is likely to be of interest to a non-trivial number of people. The work is technically reasonable and the ideas are quite nice. ", "strength_and_weaknesses": "Weaknesses:\n1. The general novelty of the approach is not that high, as it combines building blocks that are well understood (there is more to be said on this in strengths section). \n2. This paper is written in a way that is honestly kind of boring and doesn't sell the approach particularly well. 70% of this paper is background and everything interesting is kind of compressed at the end in tables or in the appendix. In some sense, it feels like the goal of this paper is to push benchmarks, not to convince the reader that they could be solving new and interesting problems that can be made more tractable by injecting domain knowledge in the form of constraints. If I were to rewrite this paper, I would put the highlight on concrete problems that are intractable using standard DRL, for example, a focus could be an illustration of how a formal specification provides a principled way out of the nightmare that is reward shaping, in particular with a focus on the additional constructs enabled by STL.\n\nStrengths:\n1. I think the problem tackled here is really important and generally underappreciated (see above).\n2. While none of the building blocks are particularly novel, it has to be noted that they come together rather well in this paper. The proposed solution is rather elegant without any obviously awkward bits and I could see this as an actual tool that one would productively use for tackling a problem where the policy lends itself to an STL specification.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "q3f0df36wyA", "reviewer": "Reviewer_Ea5U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper treats the problem of constrained deep reinforcement learning, where constraints are given in the form of signal temporal logic (STL). The paper proposes a (to my knowledge) novel technique for using Lagrangian methods with STL. They evaluate on several domains and illustrate the efficacy of their approach.", "summary_of_the_review": "My primary concern with this paper is the lack of comparison with other differentiable STL methods that have popped up in the literature. While I believe this approach is well motivated despite this, an evaluation or discussion would help set this paper in context.", "strength_and_weaknesses": "# Strength\n\nThe developed algorithm provides a well-motivated construction of a constrained policy gradient algorithm by applying lagrangian methods to a STL formula's quantitative semantics. The result seems technically sound and (in my opinion) superior to other methods I have seen in this area.\n\n# Weakness\n\nThat said, I am suspect of the novelty of using a differentiable STL formula to guide controller synthesis. While I am not deeply involved in this area, I know of at least two works [1],[2] with a quick google scholar search revealing many related articles. I view the comparison with these existing methods.\n\nIf the authors could clarify why these are either not appropriate baselines or provide a comparison, I would be willing to increase my score.\n\nFinally, as with all such methods, it seems that the \"units\" of the STL formula are important for determining the slope of the constraint function. An analysis of changing the STL formula to have the same qualitative semantics, but different quantitative semantics would have been appreciated, i.e., by changing the predicates. The result would have a different gradient landscape.\n\n[1] Pant, Yash Vardhan, Houssam Abbas, and Rahul Mangharam. \"Smooth operator: Control using the smooth robustness of temporal logic.\" 2017 IEEE Conference on Control Technology and Applications (CCTA). IEEE, 2017.\n[2] Leung, Karen, Nikos Aréchiga, and Marco Pavone. \"Back-propagation through signal temporal logic specifications: Infusing logical structure into gradient-based methods.\" International Workshop on the Algorithmic Foundations of Robotics. Springer, Cham, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_vfYboarzzl", "reviewer": "Reviewer_KU3V", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper concerns hierarchical reinforcement learning (RL) in safety critical scenarios. The critical aspect is captured via the inclusion of temporal logic constraints, in fact (a variant of) LTL. The key feature of this paper is that a quantitative notion of LTL is defined and used, to get a differentiable logic as opposed to the more binary nature of standard LTL (at a state: formula is satisfied or not). The RL procedure is defined in a hierarchical way, where high-level planning is complemented by low-level execution of a controller. The authors provide all necessary definitions, explain their variant of a quantitative notion, and provide an extensive evaluation of their approach on rather challenging environments. ", "summary_of_the_review": "Good paper, related work/novelty can be better argued. \n", "strength_and_weaknesses": "+ The paper is easy to understand\n+ The key concept is clear and important, aiming to bringing learning under/with/via temporal constraints differentiable and thereby more scaling\n+ The experimental evaluation is extensive and convincing\n- The connection to probabilistic temporal logic constraints is unclear\n- At parts, the paper could be better polished (mostly in terms of grammar)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2chQsIMiS3", "reviewer": "Reviewer_Pa7M", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper builds on recent work in formal methods and ML community on using temporal logic specifications to shape rewards for deep reinforcement learning. The main claim to novelty is the use of differentiable representation of temporal logic specifications. A secondary claim is hierarchical setting where path planner works with low-level control policy. ", "summary_of_the_review": "The paper appears to identify some of the most important needs for making DRL work in practice - hierarchical policies, and the use of temporal logic specification. It seems to be ignoring literature over the last few years on a related topic and makes claims to novelty that are not justified. Further, the presentation of the approach is vague and most of the paper is just re-summarizing known concepts from formal methods and control. Perhaps, presenting this work as taking ideas from different sub-areas and adapting them to DRL would have been much better. But in that case, the paper needs to justify the extra challenges in adapting to DRL and use more involved examples (beyond simple planning/control examples) to show the benefit. ", "strength_and_weaknesses": "Strengths:\n\n- The paper addresses an important problem of using formal/logical specifications for reward shaping in RL\n\nWeaknesses:\n\n- The paper makes a number of claims to novelty that are not accurate when one takes the vast literature on controller synthesis and learning in formal methods, robotics and control literature. The abstraction mentions that \"Prior work, however, has failed to consider the possibility of making these specifications differentiable, which would yield a more informative signal of the objective via the specification gradient.\" This fundamental claim to novelty is inaccurate. Smooth representations of formal specifications (in non-RL) setting have been widely investigated earlier. See for examples: http://robotics.cs.rutgers.edu/wafr2020/wp-content/uploads/sites/7/2020/05/WAFR_2020_FV_55.pdf,  https://link.springer.com/article/10.1007/s10703-019-00332-1 and https://ieeexplore.ieee.org/document/9114883 . When there exists a systematic compilation approach to achieve smoothness, it is important to compare the new approach for differentiability with these existing techniques. The paper needs to provide some justification based on empirical runtime or optimization effectiveness. The reviewer recognizes that this is a new context - DRL, but what is the added challenge in DRL when it comes to having smooth specifications. \n\n- The idea of using a Lagrangian for regularization with constraints is quite common, and it is not clear if the use of temporal logic constraints creates any new challenge.\n\n- The integration of neural network predicates with logic is also widely used. CLIP system from 90s and its recent resurgence in the form of Logic Tensor Networks (LTN): https://www.sciencedirect.com/science/article/abs/pii/S0004370221002009 is an example. Several other mixtures of deep learning and logic use it, including https://proceedings.neurips.cc/paper/2018/hash/dc5d637ed5e62c36ecb73b654b05ba2a-Abstract.html . If the use of this idea in DRL (as against supervised learning applications of LTN) would have involved some special challenge, this claim to novelty would have been justified. At least discussion of these and acknowledgement, the novelty is in adapting a well-known idea to the new setting of DRL would be a more accurate representation. \n\n- It is not clear why the paper views hierarchical decomposition as a novelty. Doesn't the use of temporal logic specification in this paper (and all previous papers doing so and cited in this paper) also implicitly or explicitly end up being hierarchical because temporal logic specifications are compositional? Section 3 does not elaborate on this claim to hierarchy. One is left to assume that the paper is referring to the standard practice of having two steps - planning and control. \n\n- The section on differentiable specification is rather vague, and it is unclear what is the algorithm here (making it difficult to compare conceptually with the literature mentioned above). \n -- The paper points to the neural signed distance function as an example of a differentiable predicate, and then just states that \"our approach .. is not limited to neural SDFs\". What other neural predicates can be learned? \n -- The mention of IRL and claim that they can learn differentiable predicate in this context is confusing. The entire claim in the paper is of the use of temporal logic specifications and hierarchical planning/policy generation, why would IRL to learn reward functions be relevant here? The discussion goes on to say the paper is \"not on investigating the diversity and application of neural predicates.\" which may well be true but one needs more than one example of neural predicates. Also, how these would form a differentiable temporal logic is left vague.\n -- The use of max/min for globally/future in Equation 2/3 are just the standard quantitative semantics of temporal logic that is widely known. If the entire claim to differentiability rests on using this well-known quantitative semantics, then it is alarming because it is widely known that this semantics (due to max/min) is in practice not very smooth (and is the motivation of the literature mentioned in the review above). \n\n- Can authors discuss the similarity and differences in their use of neural-ODE and the TrajODE https://www.ijcai.org/proceedings/2021/0207.pdf  (both conceptually and empirically)? \n\n- The experimental evaluation uses very simple environments, and all of them are essentially path planning examples. Doing an ablation study, where existing smooth representation of temporal logic are considered, would help better explain the value of the proposed approach. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject