Constrained Reinforcement Learning for Safety-Critical Tasks via Scenario-Based Programming
Authors: Davide Corsi
Raz Yerushalmi
Guy Amir
Alessandro Farinelli
David Harel
Guy Katz
Venue: Submitted to ICLR 2023 (2023)
Keywords: Constrained Reinforcement Learning
Scenario Based Programming
Safety
Robotic Navigation

Abstract: Deep reinforcement learning (DRL) has achieved groundbreaking successes in various applications, including robotics. A natural consequence is the adoption of this paradigm for safety-critical tasks, where human safety and expensive hardware can be involved. In this context, it is crucial to optimize the performance of DRL-based agents while providing guarantees about their behavior. This paper presents a novel technique for incorporating domain-expert knowledge into a constrained DRL training loop. Our technique exploits the scenario-based programming paradigm, designed to specify such knowledge in a simple and intuitive way. While our approach can be considered general purpose, we validated our method by performing experiments on a synthetic set of benchmark environments, and the popular robotic mapless navigation problem, in simulation and on the actual platform. Our results demonstrate that using our approach to leverage expert knowledge dramatically improves the safety and performance of the agent.

Under review as a conference paper at ICLR 2023
CONSTRAINED REINFORCEMENT LEARNING FOR SAFETY-CRITICAL TASKS VIA SCENARIO-BASED
PROGRAMMING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep reinforcement learning (DRL) has achieved groundbreaking successes various applications, including robotics. A natural consequence is the adoption of this paradigm for safety-critical tasks, where human safety and expensive hard- ware be involved. In this context, it is crucial to optimize the performance DRL-based agents while providing guarantees about their behavior. This pa- per presents novel technique for incorporating domain-expert knowledge into constrained DRL training loop. Our technique exploits the scenario-based pro- gramming paradigm, designed specify such knowledge in simple and intu- itive way. While our approach can be considered general purpose, we validated our method by performing experiments on synthetic set of benchmark environ- ments, and the popular robotic mapless navigation problem, in simulation and on the actual platform. Our results demonstrate that using our approach to leverage expert knowledge dramatically improves the safety and performance of the agent.
INTRODUCTION
In recent years, deep neural networks (DNNs) have achieved state-of-the-art results in large va- riety of tasks, including image recognition (Dul 2018), game playing (Mnih et al.) (013), protein folding (Jumper et al. 2021), and more. In particular, deep reinforcement learning (DRL) (Sut-) ton & Barto 2018) has emerged as popular paradigm for training DNNs that perform complex tasks through continuous interaction with their environment. Indeed, DRL models have proven remarkably useful in robotic control tasks, such as navigation (Kulhánek et al., (019) and manipula- tion (Nguyen & La 2019 Corsi et al. 2021). where they often outperform classical algorithms (Zhu & Zhang 2021). The success of DRL-based systems has naturally led to their integration as control policies in safety-critich tasks, such as autonomous driving (Sallab et al. 2017). surgical assis- tance (Pore al.) 2202I), flight control (Koch et al. 2019). and more. Consequently, the learning community has been seeking to create DRL-based controllers that simultaneously demonstrate high performance and high reliability; i.e., are able to perform their primary tasks while adhering some prescribed properties, such as safety and robustness.
An emerging family of approaches for achieving these two goals, known as constrained DRL (Achiam et al., (017).. attempts to simultancously optimize two functions: the reward, which encodes the main objective of the task; and the cost, which represents the safety constraints. Cur- rent state-of of-the-art art algorithms include IPO (Liu et al. 2020). SOS (Marchesini et al., 2021b), CPO (Achiam et al., 2017). and Lagrangian approaches (Ray al., 2019). Despite their success in some applications, these methods generally suffer from significant setbacks: (i) there is no uni- form and numan-readahhe way of defining the required safety constraints; (ii) it is unclear how to encode these constraints as signal for the training algorithm; and (iii) there is clear method for balancing cost and reward during training, and thus there is risk of producing sub-optimal policies.
In this paper, we present novel approach for addressing these challenges, by enabling users to encode constraints into the DRL training loop in simple yet powerful way. Our approach generates policies that strictly adhere these user-defined constraints without compromising performance. We achieve this by extending and integrating two approaches: the Lagrangian- algorithm (Ray et al. 2019) for DRI. training, and the scenario-based programming (SBP) (Damm & Harel 2001

---

Under review as a conference paper at ICLR 2023
Harel et al. 2012b) framework for encoding user-defined constraints. Scenario-baz programming is software engineering paradigm intended to allow engineers to create a complex system in a way that is aligned with how humans perceive that system. A scenario-based program is comprised of scenarios, each of which describes a single desirable (or undesirable) hehavior of the system at hand; and these scenarios are then combined to simultaneously, in order produce cohesive system behavior. We show how such scenarios can be used to directly incorporate subject-matter expert (SME) knowledge into the training process, thus forcing the resulting agent's behavior to abide various safety, efficiency and predictability requirements.
In order to demonstrate the usefulness of our approach to unfety-critical tasks, we used it to train a policy for performing mapless navigation (Zhang et al., [017]] Tai et al.) (2017) for robotics by the Robotis Turrtehot3 platform. While common DRIL-triiiin techniques were shown to give rise to high-performance policies for this task (Marchesini & Farinelli, 2020), these policies are often unsafe, inefficient, or unpredictable, thus dramatically limiting their potential deployment in real- world systems (Marchesini et al. 2021abb). Our experiments demonstrate that, by using our novel approach and injecting subject-matter expert knowledge into the training process, we are able to generate trustworthy policies that are both safe and high performance.
To have a complete assessment of the resulting behaviors, we performed a formal verification analy- sis, following methods such as with (Katz et al. 2017 Liu et al., (219)), of various predefined safety properties that proved that our approach generates safe agents to deploy in any environment,
2 BACKGROUND
Deep Reinforcement Learning. Deep reinforcement learning LLi 2017) is specific paradigm for training deep neural networks (Goodfellow et al., 2016). In DRI, the training objective is to find a policy that maximizes the expected cumulative discounted reward RG = E E, y' rz]. where E [o, 1] is the discount factor, a hyperparameter that controls the impact of past decisions on the total expected reward. The policy, denoted as wa. is probability distribution that depends on the parameters A of the DNN, which maps an observed environment state s to an action a. Proximal policy optimization (PPO) is a state-of-the-ar DRL algorithm for producing To (Schulman et al.) 2017]. A key characteristic of PPO is that it limits the gradient step size between two consecutive policy updates during training, to avoid changes that can drastically modify mg (Schulman et al.) 2015).
In mission-critical tasks, the concept of optimality often goes beyond the maximization a reward, and also involves "hard" safety constraints that the agent must respect. A constrained markov deci- sion process (CMDP) is an alternative framework for sequential decision making, which includes an additional signal: the cost function, defined as C. S x A - I. whose expected values must remain below a given threshold d E R. CMDP can support multiple cost functions and their thresholds, denoted by {Ck) and ) respectively. The set of valid policies for a CMDP is defined
Ic = (mo E II: Vk, (TO) S d
where Jc, (ma) is the expected sum of the t cost function over the trajectory and de is the corre- sponding threshold. Intuitively. the objective is to find a policy function that respects the constraints (i.e.. is valid) and which also maximizes the expected reward (i.e., optimal). A natural way to en- code constraints in a classical optimization problem is by using Lagrange multipliers. Specifically, in DRI., a possible approach is to transform the constrained problem into the corresponding dual unconstrained version (Liu et al. 2020 Achiam et al. 2017). The optimization problem can then be encoded as follows:
J(Ø) = min max [(mo, A) = min max Ja(To) - Σ X
Aa ( - dx)
(2)
A20
K
Crucially, the optimization of the function (0) can be carried out by applying any policy gradient algorithm, common implementation is based PPO (Ray et al. 2019).
Scenario- -eenario-Based Programming. cannario-based programming (SBP) (Damm& & Harel 2001 Harel & Marelly 2003) paradigm designed to facilitate the development of reactive systems, by al- lowing engineers to program a system in a way that is close to how it is perceived by humans
The supplementary material includes the appendices. The code will be released upon publication.
2

---

Under review as a conference paper at ICLR 2023
with focus inter-object, system-wied behaviors. In SBP, system is composed of scenarios, each describing a single, desired or undesired behavioral aspect of the system; and these scenarios are then executed in unison cohesive system.
An execution of a scenario-base (SB) program is formalized as a discrete sequence of events. At each time-step, the scenarios synchronize with each other to determine the next event be triggered. Each scenario declares events that it requests and events that it blocks, corresponding to desirable and undesirable (forbidden) behaviors from its perspective; and also events that it passively wails- for. After making these declarations, the scenarios are temporarily suspended, and an event-selection mechanism triggers a single event that was requested by at least one scenario and blocked by none. Scenarios that requested or waited for the triggered event wake up. perform local actions, and then synchronize again; and the process repeated ad infinitum. The resulting execution thus complies with the requirements and constraints of each of the individual scenarios (Harel & Marelly 2003 Harel et al. 20126). For a formal definition of SBP, see (Harel et al. 2012b).
Although SBP is implemented in many high-level languages, it is often convenient to think of sce- narios as transition systems, where each state corresponds to a synchronization point, and each edge corresponds to event that could be triggered. Fig. [ uses that representation to depict simple SB program that controls the temperature and water-level in a water tank (borrowed from (Harel et al.) 2012a)). The scenarios add hot water and add cold water repeatedly wait for WATER LOw event, and then request three times the event Add HOT or Add COLD, respectively. Since these six events may be triggered in any order by the event selection mechanism, new scenario stability is added to keep the water temperature stable, achieved by alternately blocking Add HOT and Add COLD events. The resulting execution trace shown in the event log.
add hot water
Vostt For: WATER
WATER
Request: Add HOT
Add
Request Add HOT
Adl HOT
Requestt Add HOT
GOLD
HOT COLL HOT
add cold water
For WATER
WAI LOAY
AIl COLD
AMU COLD
Add COLD
Adi COLD
Add HOT
Add COLD
Well For: Add HOT Blocked:
Wall For: Add COLD Blocked Add HOT
stabiiity
COLD
scenario-based program for controlling a water tank. The small black circle indicates the initial state. Figure is inspired by the work Harel et al. (2012a)
Figure 1:
SBP is an attractive choice for the incorporation of domain-specific knowledge into a DRL agent training process, due to being formal, fully executable and support of incremental development (Gordon et al. 2012 Alexandron et al.) 2014). Moreover, the language it uses enables domain- specific experts to directly express their requirements specifications as an SB program.
3 EXPRESSING DRL CONSTRAINTS USING SCENARIOS
Mapless Navigation. We explain and demonstrate our proposed technique using the mapless nav- igation problem, in which robot is required to reach a given target efficiently while avoiding collision with obstacles. Unlike in classical planning. the robot is not given a map of its surrounding environment and can rely only on local observations e.g.. from lidar sensors or cameras. Thus, a successful agent needs to be able to adjust its strategy dynamically, as it progresses towards its target. Mapless navigation has been studied extensively and is considered difficult to solve. Specif- ically, the local nature of the problem renders learning a successful policy extremely challenging and hard to solve using classical algorithms (Pfeiffer et al.) 2018). Prior work has shown DRL ap- proaches to be among the most successful for tackling this task, often outperforming hand-crafted algorithms MMarchesini & Farinelli 2020).
3

---

Under review as a conference paper at ICLR 2023
Figure 2: The Robotis Turtlebot3 platform.
As a platform for our study, used the Robotis Turtlebot 3 platform (Turtlebot, for short; Fig. 2). which is widely used in the community (Nandkumar et al. [021]] (Amstess & Slaets 2019). The Turtlebot capable of horizontal navigation and equipped with lidar sensors for detecting nearby obstacles. In order to train DRL policies for controlling the Turtlebot, we built a simulator based on the Unity3D engine (Juliani al., 2018), which is compatible with the Robotic Operat- ing System (ROS) (Quigley et al.) 2009) and allows fast transfer to the actual platform (sim-to- real (Zhao et al., 2020). We used a hybrid reward function, which includes discrete component for the terminal states ("collision", or "reached target"), and a continuous component for the non- terminal states. Formally:
terminal states
R (disl:-1 - dist.) 7-8
(3)
otherwise
Where dist, is the distance from the target at time k; 7 is a normalization factor; and | is a penalty, intended to encourage the robot to reach the target quickly (in our experiments, empirically set and / = 0.001). Additionally, in terminal states, we increase the reward by the target is reached, or decrease it by in case of collision. For our DNN topology, we used an architecture that was shown to be successful in a similar setting (Marchesini & Farinelli, 2020): (i) an input layer of nine neurons, including seven for the lidar scans and two for the polar coordinates of the target; (ii) two fully-connected hidden layers of 32 neurons each; and (iii) an output layer of three neurons for the discrete actions (i.e., move FORNARD, turn LEFT, and turn RIGHT). Section 4 we provide details about the training algorithm we used. Using the reward defined in Eq. 3 we were able to train agents that achieved high performance - i.e., obtained a success rate of approximately 95%, where "success" means that the robot reached its target without colliding into walls or obstacles.
Analyzing the trained agents further, we observed that even DRL agents that achieved a high suc- cess rate may demonstrate highly undesirable behavior in different scenarios. One such behavior is a sequence of lack-and-forth turns, that causes the robot to waste time and energy. Another unde- sirable behavior is when the agent makes lengthy sequence of right turns instead of a much shorter sequence of left turns (or vice versa), wasting time and energy. A third undesirable behavior that we observed is that the agent might decide not to move forward towards target that is directly ahead, even when the path is clear. Our goal was thus to use our approach to remove these undesirable behaviors.
Rule-Based Approach. Following the approach of (Yerushalmi et al., 2022), we integrated scenario-basee program into the DRL training process, in order to remove the aforemmnnioned unde- sirable behaviors. More concretely, mentioned undesirable behaviors we observed. To accomplish this, we created a mapping between each possible action o E (Move FORWARD, Turn LEFT, Turn RIGHT) of the DRL agent and a dedicated event Ca, E (SBP MoveForward, SBP TurnLeft, SBP TirnRight) within the scenario- based program. These events allow the various scenarios to keep track and react to the agent's actions. Similarly to (Yerushalmi et al. 2022), refer to these Ca, events as external events, in- dicating that they can only be triggered when requested from outside the SB program proper. By convention, we assume that after each triggering of a single, external event, the scenario-based pro- gram executes a sequence of internal events (a super-step (Yerushalmi et al. 2022)), )) until it returns to steady-state and then waits for another external event.
created specific scenarios to rule out each of the three afore-
The novelty of our approach, compared to (Yerushalmi et al. 2022), is in the strategy by which scenarios to affect the training process. Specifically, we define the DRL cost function to
4

---

Under review as a conference paper at ICLR 2023
LEVT HIGHT
LESTE HIGHT
NONE
PIRVIDD
INTEVENT
Mlackess
PORDARDD LEPT, RIGSTT
NOCE
FORMARDD, ERT
LEFT and
AWV EXENIT (urgee divertoon
sesight
ROBWARD RIGHT
FORWARTL RIGHT
ROBMARD
Mladkss
EEFT, REGGT
(a) avoid back and forth rotation
avoid turns larger than 180
avoid nurning when clear
Figure 3: A visualization of the three scenarios. Figure (b) refers the Left turns part only. 'Wait For' and 'Blocked' in the state-blob indicates events that the scenario waits for blocks, respectively. The events SBP_M wow.MMooorrrrr FORWARD, LEFT, RIGHT.
SBP_Turnieft and SBP.TurnRight are represented respectively, by
correspond to violations of scenario constraints by the DRL agent. Whenever the agent selects action that mapped to blocked SBP event, we increase the cost. This approach is described further in Section 4 and constitutes general method for injecting explicit constraints (expressed, e.g., by scenarios) directly into the policy optimization process.
Example: Constraint Scenarios. Considering again our Turtlebot mapless navigation case study, we created scenarios for discouraging the three undesirable behaviors we had previously observed. The scenarios, defined in Python, presented in Appendix A and are visualized in Fig. 3 using
an amalgamation of Statecharts and SBP graphical notation languages (Harel 1987: Marron et al, 2018).
Scenario avoid back-and-forth rotation (Fig. 3(a) seeks to prevent in-place, back-and-forth turns by the robot, to conserve time and energy.
Scenario avoid turns larger than 180 (Fig. 3(b) seeks to prevent left turns in angles that are greater than 180", to conserve time and energy (the right-turn case symmetrical). A forward slash indi- cates an action that is performed when a transition is taken; square brackets denote guard conditions, and $k and SleftCounter are variables. Each turn rotates the robot by 30 and so we set k
Scenario avoid turning when clear (Fig. 3(c) seeks to force the agent to move towards the target when it is ahead, and there a clear path to it. This is performed by blocking any turn actions when this situation occurs. Triggered events carry data, which can be referenced by guard conditions.
4 USING SCENARIOS IN DRL TRAINING
Even after defining constraints as an SB program, obtaining differentiable function for the training process is not straightforward. We propose to use the following binary (indicator) function this end:
ok (* a, 8t+1) ) = I (the tuple (st, a, 8141) is a blocked state in the SB program, by the tt rule)
Intuitively, summing the values of the different C₁ 's over the training episode yields the exact number of violations to the respective tth rule during the full trajectory; those are the values we aim to minimize; moreover, following the intuition of Roy et al. (202T). ), this value, if normalized over the number of steps, can be seen as probability of having a violation. This value can be treated as a cost function, the corresponding objective function defined as follows: Jc. = En c(s,, a,, 81.8))) for trajectory of N steps. This value is dependent on the action policy and is therefore differentiable on the parameters 6 of the policy through the policy gradient theorem.
Optimized Lagrangian-PPe
Section 2 we proposed to relax the Lagrangian constrained opti- mization problem into unconstrained, min-max version thereof. Taking the gradient of Equation
5

---

Under review as a conference paper at ICLR 2023
and some algebraic manipulation, derive the following two simultaneous problems:
VVotA) - AeJocs
Ak. V C(T = - (Jcc (T) -m) (4)
In closed form, the Lagrangian dual problem would produce exact results. However, when applied using a numerical method like gradient descent, it has shown strong instability and the proclivity to optimize only the cost, limiting the exploration and resulting in poorly-performin agent (Achiam let al. 2017). To overcome these problems, we introduce three key optimizations that proved crucial obtaining the results present the next section.
Reward Multiplier: the standard update rule for the policy in a Lagrangian method given in Equation 9 However, as mentioned above, it often fails to maximize the reward. To overcome this failure, we introduce new parameter a, which we term reward multiplier, such that 2 Σx An. This parameter is used multiplier for the reward objective:
Vol(n, Vo(a Jn(x)) - EAJJce (T))
(5)
K
2. Lambda Bounds and Normalization: Theoretically, the only constraint on the Lagrangian multipliers is that they are non-negative. However, when solving numerically, the value of Å can increase quickly during the early stages of the training, causing the optimizer to focus primarily on the cost functions (Fq. 4 potentially not pushing the policy towards a high performance reward-wise. To overcome this, introduced dynamic constraints the multipliers (including the reward multiplier a),, such that Σ X + a = 1. In order also enforce the previously mentioned upper bound for a, we clipped the values of the multipliers such that Σx K ds I 2. Formally, we perform the following normalization over all the multipliers:
d
Vk, AE
a a=
Σ AE K
(6)
2(Ex K Ax)
3. Algorithmic Implementation: the primary objective of the previously introduced optimiza- tions is balance the learning between the reward and the constraints. To further stabilize the training, we introduce additional, minor improvements to the algorithm: (i) lambda initialization: we initialize all the Lagrangian multipliers with zero to guarantee a focus on the reward optimization during the early stages of the training (consequently, following Eq. 6 a = 1); (ii) lambda learning rate: to guarantee a smoother update the Lagrangian multipliers, scale this parameter to 10% of the learning rate used for the policy update; and (iii) delayed start: we enable the update of the multipliers only when the success rate is above 60% during the last 100 episodes. Intuitively, this delays the optimization of the cost functions until a minimum performance threshold is reached.
5 EVALUATION
Setup. We performed training distributed cluster of HP EliteDesk machines, running 3.00 GHz, with 32 GB RAM. We collected data from more than 100 seeds for each algorithm, reporting the mean and standard deviation for each learning curve, following the guidelines of Colas et al. (2019). For training purposes, we built realistic simulator based on the Unity3D engine (Jliaai al. 2018). Next, we evaluated the performance of the trained models using a physical Robotis Turtlebot3 robot (Fig. 2 and confirmed that it behaved similarly to the behavior observed in our simulations.
Results. Fig. depicts a comparison between policies trained with standard end-to-end PPO (Schulman et al. 2017) (the baseline), and those trained using our constrained method with the injection of rules. In Figs. a)) and d). show results of policies trained with just avoid aack-and-forth rotation added as a constraint. Fig, 4a) shows that the success rate of the base- line stabilizes around 87%, while the success rate of our improved policies stabilizes around

---

Under review as a conference paper at ICLR 2023
95%. Fig. 4 d) then compares the frequency of undesired behavior occurrences between the base- line, at about 13 per episode, and our policies, where the frequency diminishes almost completely.
Next, for Fig. 4bb) show results of policies trained with all three of our added rules; note that the success rate for these policies stabilizes around 95%, compared to 87% for the baseline.
Finally, in Figs. 4c), (e) and (f), we compare the frequency of the occurrence of undesired behaviors between the baseline and the policies trained with all rules active. Using the baseline, the frequency of the three behaviors is about 13, 3, and 17 per episode. The undesired behaviors are removed almost completely for the policies trained with our additional rules and method.
We note that the undesired behavior addressed by the rule avoid turns larger than 180 is quite rare in general; and so the statistics reported in Fig. ()) were collected over the final 100 episodes of training.
Bacelin
0.-
(a) Success rate
(b) success rate
avoid turns larger than 180
32-
Daselia
Dase
15-
20003
30000
48003
30000
4ptidd
4p apaddo
(d) avoid back-and-fortee forth rotation
(e) avoid back-and-forth rolation
(f) avoid turning when clear
Figure 4: A comparison between the baseline policies to policies trained using our approach. The black dotted line states the threshold (dx) we considered for the 5th rule.
The results clearly show that our method is able to train agents that respect the given constraints, without damaging the main training objective- - the success rate. Moreover, it also highlights the scalability of our method, i.e. performing well when single or multiple rules are applied. Review- ing Fig bb), comparing the baseline's 's success rate with our method's success rate when all rules are applied together with all the optimizations presented in Section 4 shows clear advantage. Excit- ingly, our approach even led to an improved success rate, suggesting that the contribution of expert knowledge can drive the training to better policies. This showcases the importance of enabling expert-knowledge contributions, compared to end-to-end approaches.
b).
Formal Verification and Safety Guarantees. To further prove the effectiveness of our method, we show results of DNN verification engine (Katz et al., 2019) assessing the reliability of our trained models. DNN verification is sound and complete method for checking whether DNN model displays unwanted behavior, over all possible inputs. We trained two batches of 60 models each: one batch of models trained by the "baseline" training algorithm, and batch with models trained by our approach. The results indicate that while almost none of the baseline models upheld the properties (i.e., there was always at least one input for which they violated the properties), more than 80% of our improved models satisfied the properties for all possible inputs. A full summary of these results appears in Table of Appendix B
Additional Environment. To further validate our approach, we applied our algorithm to an ad- ditional environment beyond the one used in Section [ In particular, we drastically increased the number of obstacles, which is one of the main challenges in mapless navigation. Fig, [5 shows the difference between the two training environments.
7

---

Under review as a conference paper at ICLR 2023
(a) basic environment
(b) additional environment
Figure 5: comparison between the basic training environment that used for the main analysis (Sec. and the additional environment used this Section.
UPPO
33
PM
opicade
apacaae
(a) success
(b) avoid back-and-forth rotation
(c) avoid turning when clear
Figure 6: Experiments in additional, more complex, environment. The figure shows comparison of our
Lagrangian PPO (LPPO) and baseline (PPO).
Fig. shoows the results of our experiments in more complex environment. The results indicate that our approach scales very well, even when the number of obstacles larger. Clearly, with respect to the results of Sec. the algorithm requires more iterations to converge; however, the success rate satisfies the requirements, keeping the number of violations to the rule below the given threshold.
RELATED WORK
To the best of our knowledge, this is the first work that combines scemario-based programming into the training of constrained deep reinforcement learning system. In Yerushalmi et al.) (2022). the authors proposed integration between SBP and DRL using reward-shaping approach that penalizes the agent when rules violated, with unconstrained optimization method. Our approach, based on constrained optimization, provides many advantages compared to the mentioned work, which results in nigh-performing agents and fewer rule violations. We provide an extensive comparison between the two approaches below.
DiEE Ic E3
paea fic 8.
2.0
UFFO
pardly
pardly
spacdh
spaads
spbods
(a) success rate
(b) avoid back-and-forth rotation
(c) avoid turns larger than 180"
approach, denoted by LPPO, with those achieved by (Yerushalmi al. 2022). denoted by penalty and its value: fixed penalty 0.1, 1.0. or L.5: graph (a) measures the success rates with all three scenario-based rules. The results of using the penalty approach with a penalty value of 0.1 are practically the same as using our approach. However, using penalty approach with penalty values of 1.0 and 1.5 results in poor performance; graph (b) measures the frequency of violations to the avoid fack-and-forth rotation rule. The results of using the penalty approach with penalty values 0.1 are similar to ours. When using penally approach with a penalty value 1.5. violations diminishes completely - however, also the performance, as mentioned above; graph (c) measures the frequency of viola- tions to the avoid turns larger than 180 rule. The results of using the penalty approach with penalty values of 1.0 and 1.5 practically the same
Figure 7: The graphs compare results achieved
8

---

Under review as a conference paper at ICLR 2023
Fig. 7 compares the results of our approach and those of Yerushalmi et al. (2022) As shown in Fig. using their approach, a low penalty value allows the agent to reach high perf-prrfrmaceee reward-wist but fails to minimize the cost (e.g., the number of rule violations). In contrast, a high penalty value reduces the agent's rule violations but fails to reach adequate performance in terms of the reward function. Our approach is shown here reach similar performances as the best of (Yerushalmi al.) 2022), using a penalty value of 0.1, and reducing the agent's 's rule violations as the best of it, using penalty value of 1.0 or 1.5.
Our approach adopts a constraint-driven DRL framework that differentiates between optimizing the main reward and minimizing the costs. This differentiation presents significant advantages, including:
Allows the setting of constraint thresholds independently for each rule/property and the handling of multiple such constraints in the same way, unlike methods such as (Yerushalmi et al. 2022) that only allow a global minimization zero of the total cost.
Separates reward maximization from cost minimization, simplifying the reward engineer- ing task.
Automatically balances the focus of the training, between the different cost elements and the reward, by learning the values of the different multipliers for each cost factor.
Introduces novel numerical optimizations to the training phase, resulting in a more stable algorithm with higher cumulative reward (as shown in Appendix C on a synthetic set of benchmarking environments).
In a recent work on constrained reinforcement learning (Roy et al. 2021 the authors advocate an optimized version of Lagrangian-PPE PPO. They propose a different approach to balance the constraints and the return, based on the softmax activation function and without imposing bounds on the values for the multipliers. Moreover, their work focuses on game development, a different domain from our focus, which presents very different challenges, e.g., safety and efficiency not considered crucial requirements. In addition, they do not encode constraints using a framework geared for this purpose, such as SBP.
Limitations. Our method suffers from various limitations. First, it does not completely guarantee that the resulting policies are safe. For example, as shown Table of Appendix B even though
the number of formally safe models is significant, it is not absolute. Second, the scalability of the method needs be investigated. We showed in this work that the algorithm can easily handle one to three constraints, in addition to the main objective. We leave to future work the analysis of performance when the number of constraints increases further. Third, we noticed some performance deterioration after about 10,000 episodes shown in Fig. 4 (a) and (b). We believe that the performance deterioration is related to the activation of the cost multipliers, especially as the performance was recovered afterward. We plan investigate that further in the future.
7 CONCLUSION
This paper presents a novel and generic approach for incorporating subject-matter-expert knowl- edge directly into the DRL learning process, allowing to achieve user-defined safety properties and behavioral requirements. We show how to encode the desired behavior as constraints for the DRL al- gorithm and improve a state-of-the-art algorithm with various optimizations. Importantly, we define properties comprehensibly, leveraging scenario-based programming to encode them into the train- ing loop. We apply method to real-world robotic problem, namely mapless navigation, and show that our method can produce policies that respect all the constraints without adversely affect- ing the main objective of the optimization. We further demonstrate the effectiveness of our method by providing formal guarantees, using DNN verification, about the safety of trained policies.
Moving forward, we plan to extend our work to different environments including navigation in more complex domains (c.g., air and water). Another key challenge for the future is to inject rules aiming to encode behaviours in cooperative (or competitive) multi-agent environment.

---

Under review as a conference paper at ICLR 2023
REFERENCES
J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained Policy Optimization. In Proc. 34th Int. Conf. on Machine Learning (ICML), pp. 22-31, 2017.
G. Alexandron, M. Armoni, M. Gordon, and Harel. Scenario-Base Basaad Programming: Reducing the Cognitive Load, Fostering Abstract Thinking. In Proc 36th Int. Conf. on Software Engineering (ICSE), pp. 311-320, 2014.
G. Amir, M. Schapira, and G. Katz. Towards Scalable Verification of Deep Reinforcement Learning. In Proc. 21st Int. Conf. on Formal Methods in Compuler-Aided Design (FMCAD), pp. 193-203, 2021.
G. Amir, D. Corsi, R. Yerushalmi, L. Marzari, D. Harel, A. Farinelli, and G. Katz. Verifying Learning-Basee Based Robotic Navigation Systems, 2022. Technical Report. https://arxis orq/ ab///bb///a 2205. 13536
R. Amsters and P. Slaets. Turtlebot as a Robotics Education Platform. In Proc. 10th Int. Conf. on Robotics in Education (RiE), pp- 170-181, 2019.
E. Bacci, M. Giacobbe, and D. Parker. Verifying Reinforcement Learning Up to Infinity. In Proc. 30th Int. Joint COnf. on Artificial Intelligence (IJCAI), 2021.
C. Colas, O. Sigaud, and Oudeyer. A hitchhiker's guide to statistical comparisons of reinforcement learning algorithms. arXiv preprint 194i::990400699 06979, 2019.
D. Corsi, E. Marchesini, and A. Farinelli. Formal Verification of Neural Networks for Safety- Critical Tasks in Deep Reinforcement Learning. In Proc. 37th Conf. on Uncertainty in Artificial Intelligence (UAI). pp. 333-343 2021.
W. Damm and Harel. LSCs: Breathing Life into Message Sequence Charts. Journal on Formal Methods in System Design (FMSD), 19(1):45 80, 2001.
J. Du. Understanding of object detection based on cnn family and yolo. In Journal of Physics: Conference Series, 2018
Eliyahu, Y. Kazak, G. Katz, and M. Schapira. Verifying Baaming-Augmentdd Systems. In Proc. Conf. of the ACM Special Interest Group Data Communication the Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM), pp. 305- 318, 2021.
T. Gehr, M. Mirman, D. Drachsler-Cohen E. Tsankov, S. Chaudhuri, and M. Vechev. A12: Safety and Robustness Certification of Neural Networks with Abstract Interpretation. In Proc. 39th IEEE Symposium on Security and Privacy (S&P), 2018.
I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT Press, 2016.
M. Gordon, A. Marron, and O. Meerbaum-Salam Spaghetti for the Main Course? Observations on the Naturalness of Scenario-Basse Programming, In Proc. 17th ACM Annual Conf. on Innovation and Technology in Computer Science Education (ITCSE), pp. 198-203 2012.
Gronauer.
Bullet safety gym, 2021.
Bu But Seet-Saffee Gey
D. Harel and R. Marelly. Come, Let's Play: Scenario-Based Programming using LSCs and the Play-Engine, volume 1. Springer Science Business Media, 2003.
D. Harel, G. Katz, A. Marron, and G. Weiss, Non-Intrusive Repair of Reactive Programs. In Proc. 17th IEEE Int. Conf. Engineering of Complex Computer Systems (ICECCS), pp. 3-12, 2012a.
D. Harel, A. Marron, and G. Weiss. Behavioral Programming. Communications of the ACM (CACM), 5507):90-100,, 2012b.
David Harel. Statecharts: A visual formalism for complex systems. Science of computer program- ming, 8(3):231-274, 1987.
10

---

Under review as a conference paper at ICLR 2023
David Harel, Assaf Marron, and Gera Weiss. Programming coordinated behavior in java. In ECOOP 2010 - Object- Oriented Programming, Pp. 250-274,22 2010.
X. Huang, M. Kwiatkowska, S. Wang, and M. Wu. Safety Verification of Deep Neural Networks. In Proc. 29th Int. Conf. on Computer Aided Verification (CAV). pp. 3-29, 2017.
A. Juliani, V. Berges, E. Teng, A. Cohen, J. Harper, C. Elion, C. Goy, Y. Gao, H. Henry, M. Mattar, et al. Unity: A General Platform for Intelligent Agents, 2018. Technical Report. https://
arxiv. www.org/aaaa httg//bb//// maabs/1809.00 02627
J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Židck, A. Potapenko, et al. Highly accurate protein structure prediction with al- phafold. Nature, 2021.
G. Katz, C. Barrett, D. Dill, K. Julian, and M. Kochenderfer, Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. In Proc. 29th Int. Conf. on Computer Aided Verification (CAV), pp. 97-117, 2017.
G. Katz, D. Huang, D. Iheling, Julian, C. Lazarus, R. Lim, P. Shah, S. Thakoor, H. Wu, A. Zeljić, D. Dill, M. Kochenderfer, and C. Barrett. The Marabou Framework for Verification and Analysis of Deep Neural Networks. In Proc. 31st Int. Conf. on Computer Aided Verification (CAV), pp. 443-452, 2019.
W. Koch, R. Mancuso, R. West, and A. Bestavros. Reinforcement learning for uav attitude control. ACM Transactions on Cyber-Physical Systems, 2019.
Kulhánek, Derner, T. De Bruin, and Babuška. Vision-based navigation using deep reinforce- ment learning. In 2019 European Conference on Mobile Robots (ECMR), 2019.
Y. Li. Deep Reinforcement Learning: An Overview, 2017. Technical Report. http://arxiv. org//bbs/ hog/abs/1701.0777
C. Liu, T. Amon, C. Lazarus, C. Barrett, and M. Kochenderfer. Algorithms for Verifying Deep Neural Networks, 2019, Technical Report.
Y. Liu, J. Ding, and X. Liu. Ipo: Interior-Point Policy Optimization under Constraints, In Proc. 34th AAAI Conf. on Artificial Intelligence (AAAI), pp. 4940 4947, 2020.
Z. Lyu, C. Ko, Z. Kong, N. Wong, D. Lin, and L. Daniel. Fastened Crown: Tightened Neural Network Robustness Certificates. In Proc. 34th AAAI Conf. on Artificial Intelligence (AAAI), pp- 5037-5044 5044, 2020.
E. Marchesini and A. Farinelli, Discrete Deep Reinforcement Learning for Mapless Navigation. In Proc. IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 10688-10694, 2020.
E. Marchesini, D. Corsi, and A. Farinelli. Benchmarking Safe Deep Reinforcement Learning in Aquatic Navigation. In Proc. IEEE/RSJ Int. Conf on Intelligent Robots and Systems (IROS), 2021a.
E. Marchesini, D. Corsi, and A. Farinelli. Exploring Safer Behaviors for Deep Reinforcement Learn- ing. In Proc. 35th AAAI Conf. on Artificial Intelligence (AAAI), 2021b.
Assaf Marron, Yotam Hacohen, David Harel, Andreas Mülder, and Axel Terfloth. Embedding scenario-based based modeling in statecharts. In MODELS workshops, pp. 443-452, 2018.
V. Mnih, K. Kavukeuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with Deep Reinforcement Learning, 2013. Technical Report. https::/aaras ozq/abs//312.1 5602
C. Nandkumar, P. Shukla, and V. Varma. Simulation of Indoor Localization and Navigation of Turtlebot 3 using Real Time Object Detection. In Proc. Int. Conf. on Disruptive Technologies for Mulii-Discipinary Research and Applications (CENTCON), 2021.
11

---

Under review as a conference paper at ICLR 2023
H. Nguyen and H. La. Review of deep reinforcement learning for robot manipulation. In 2019 Third IEEE International Conference on Robotic Computing (IRC), 2019. 2019.
M. Pfeiffer, S. Shukla, M. Turchetta, C. Cadena, A. Krause, R. Siegwart, and J. Nieto. Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Mapless Navigation by Leveraging Prior Demonstrations. IEEE Robotics and Automation Letters, 3(4):4423-4430, 2018.
A. Pore, D. Corsi, E. Marchesini, D. Dall' Alba, A. Casals, A. Farinelli, and P. Fiorini. Safe Re- inforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic- Assisted Surgery. In Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), pp- 4025 4025-4031, 2021.
M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, et al. Ros: an open-source robot operating system. In ICRA workshop open source software, 2009.
A. Ray, J. Achiam, and D. Amodei. Benchmarking Safe Exploration in Deep Reinforcement Learn- ing, 2019. Technical Report. https://cdme http::/ccc..ooppaaaacmo///eeeeep
pdE.
J. Roy, R. Girgis, J. Romoff, P. Bacon, and C. Pal. Direct Behavior Specifiatrpeeiiiiicciiiiiiioonnn via Constrained Re- inforcement Learning, 2021, Technical
A. Sallab, M. Abdou, E. Perot, and S. Yogamani. Deep reinforcement learning framework for autonomous driving. Electronic Imaging, 2017.
Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms, 2017. Technical http://ppp///r......oo////////1. 06347
R. Sutton and A. Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
L. Tai, G. Paolo, and Liu. Virtual-to-Real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation. In Proc. IEEE/RSJ Ini. Conf. on Intelligent Robots and Systems (IROS), pp. 31-36, 2017.
S. Wang, K. Pei, J. Whitehouse, J. Yang, and Jana. Formal Security Analysis of Neural Networks using Symbolic Intervals. In Proc. 27th USENIX Security Symposium, pp. 1599-1614, 2018.
Yaacov. BPPy: Behavioral Programming in Python, 2020.
hutps://github.com/TThii
BGU/BPPy.
Raz Yerushalmi, Guy Amir, Achiya Blyasaf, David Harel, Guy Katz, and Assaf Marron. Scenario- Assisted Deep Reinforcement Learning. In Proc. 10th Int. Conf. on Model-Driven Engineering and Software Development (MODELSWARD), pp. 310-319, 2022.
Zhang, J. Springenberg, Boedecker, and W. Burgard. Deep reinforcement learning with succes- sor features for navigation across similar environments. In 20/7 IEEE/RSI International Confer- ence on Intelligent Robots and Systems (IROS), 2017.
W. Zhao, Queralta, and T. Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: survey. In 2020 IEEE Symposium Series Computational Intelligence (SSCI), 2020.
K. Zhu and T. Zhang. Deep reinforcement learning based mobile robot navigation: A review. Ts- inghua Science and Technology, 2021.
12


Reviews:
[{"review_id": "taW9bI9-i64", "reviewer": "Reviewer_ENWz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this paper introduce a new approach to safe (or constrained) reinforcement learning (RL) that leverages a concept called scenario-based programming. In essence, this programming paradigm is a means to inject domain knowledge to the RL loop, by running such a (set of) program(s) in parallel with the RL agent, as some form of external controller. A nice feature of these scenario-based programs is that they can capture a set of potential environments, and is therefore a good way to capture robustness of RL. The authors introduce these programs, describe in detail their embedding into RL agents, and introduce a dedicated ‘blocking’ mechanism that avoids actions not adhering to the program specification and rescales the reward/cost function accordingly, to avoid future violations. The evaluation is done on a turtlebot environment with unity as simulation engine. \n\n--- after rebuttal ---\nSince the authors did not reply to my questions, I recommend rejection for this submission\n---\n", "summary_of_the_review": "Good paper, related work/novelty can be better argued, some further experiments could strengthen the message.", "strength_and_weaknesses": "*Strengths*\n- The paper is very well written and easily accessible.\n- The concept is nice and effective.\n- The type of domain knowledge via scenario-based programming is easily accessible for users, other than many other approaches that employ constrained RL. \n- The evaluation is done on state-of-the-art environments.\n- A formal verification step is additionally added to assess the quality of the trained network.\n\nWeaknesses\n- Some questions remain open regarding novelty and further evaluation. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eGRwDA0iHL", "reviewer": "Reviewer_XYtz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a new scenario-based programming approach where one can include rule-based constraints to safe-RL problems. By observing the suboptimality of trained PPO agents in a mapless navigation task, they come up with three rules that when implemented as additional constraints during the policy optimization, help remove the unwanted behaviors from the agent policy. The approach is compared to vanilla PPO that optimizes the safe-RL objective in the two experimental setups. The authors separately include a comparison to a similar scenario-based constrained RL approach, where they show improved behavior.", "summary_of_the_review": "Overall I think the paper should be rejected: the contribution of the paper is quite minor, and the evaluations are not extensively done. Even if the experiments were significantly improved, I still think that the paper does not offer a significant method.", "strength_and_weaknesses": "* As a strength, I think that the idea to use scenario-based programming to inject rules easily into the optimization of safe-RL agents is a good idea, and the authors have picked a good application.\n\n* However, I think that the contribution is quite minor. The approach can be subsumed, at least the author's formulation, within reward-shaping in general, as they cast the rules into constraints that are optimized together with the reward function. I think the introduced approach is not a new methodology but rather a successful reward shaping/modeling scenario for a particular experimental case.\n\n* It is not clear why the introduced approach should be the winning one (compared to other e.g. reward shaping, approaches). An alternative formulation could have shaped the reward to include costs on the control actions, thereby effectively preventing some of these undesirable behaviour (if not all). Or the action space could have been defined appropriately (to avoid back and forth rotation). Or an MPC-like approach could have been followed to filter-out turns larger than 180 degrees for example. These are not discussed. \n\n* As a related note, there are no comparisons to recent constrained/safe RL algorithms, only to a vanilla PPO approach. It is not motivated why only their approach can be used to train agents that satisfy the three rules they created. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GXw9SGnrds", "reviewer": "Reviewer_7ji2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a way of defining the constraints in constrained RL via Scenario-based Programming. They demonstrate the approach on the problem of mapless navigation in simulation as well as on a real Turtlebot.", "summary_of_the_review": "The problem of how to design reward/constraint functions is relevant, but the proposed approach is rather heuristic and not empirically well motivated (does it actually help people?), and the experiments are very simple. ", "strength_and_weaknesses": "Strengths: \n- The problem is interesting as defining the reward function and constraints can be difficult. \n- The authors test their approach on a real Turtlebot robot\n\nWeaknesses:\n- Scenario-based programming seems like yet another high-level state machine / visual programming language. As with all such syntactic representations for manually encoding rules, it is difficult to motivate why this representation is better than any other. Some kind of user study might have helped provide some evidence here.\n- The proposed approach appears both fairly trivial and heuristic, (e.g. the reward multipliers, another hyperparameter).\n- The RL baseline with the three failure scenarios seem suspiciously bad for such a simple environment. Imposing a constraint (make it a much harder problem) to keep the agent from turning back and fourth also seems like a complicated micromanagement of what might be some problem with the problem setup (state, reward). I am not convinced that the proposed approach is the best way to fix this.\n\nThe paper also contains some questionable statements like:\n- \"Specifically, the local nature of the problem renders learning a successful policy extremely challenging and hard to solve using classical algorithms (Pfeiffer et al., 2018)\" - At a glance, I did not find any such strong claim in the referenced paper. It's a bit unclear what you mean by \"local\" here, but the mapless navigation problem used in this paper does not seem that difficult. A turtlebot can stop nearly instantly. You could just path to towards the goal with A* and replan on an incrementally built map. While there isn't any one algorithm that is *standard* for this problem (known goal, partially unknown environment), there is a lot of work on informative path planning and motion planning in unknown environments, if you need something more sophisticated than that. \n\n- \"Prior work has shown DRL approaches to be among the most successful for tackling this task, often outperforming hand-crafted\nalgorithms (Marchesini & Farinelli, 2020).\" - That paper just appears to compare one RL-based formulation to another. It is not clear what you mean by \"hand-crafted\" algorithms, but I can't find anything about applicable planning algorithms.\n\n- Eq.2: I think you mixed up the subscripts since you talk about maximizing reward?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject