Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations
Authors: William Sharpless
Dylan Hirsch
Sander Tonkens
Nikhil Uday Shinde
Sylvia Lee Herbert
Venue: ICLR 2026 Poster (2026)
Keywords: Hamilton-Jacobi Analysis
Dual Satisfaction
Safe Reinforcement Learning
Decomposition
Primary area: reinforcement learning
TL;DR: Novel value function decomposition theorems for dual-satisfaction RL, yielding new performant algorithm

Abstract: Hard constraints in reinforcement learning (RL) often degrade policy performance. Lagrangian methods offer a way to blend objectives with constraints, but require intricate reward engineering and parameter tuning. In this work, we extend recent advances that connect Hamilton-Jacobi (HJ) equations with RL to propose two novel value functions for dual-objective satisfaction. Namely, we address: 1) the Reach-Always-Avoid (RAA) problem – of achieving distinct reward and penalty thresholds – and 2) the Reach-Reach (RR) problem – of achieving thresholds of two distinct rewards. In contrast with temporal logic approaches, which typically involve representing an automaton, we derive explicit, tractable Bellman forms in this context via decomposition. Specifically, we prove that the RAA and RR problems may be rewritten as compositions of previously studied HJ-RL problems. We leverage our analysis to propose a variation of Proximal Policy Optimization (DO-HJ-PPO), and demonstrate that it produces distinct behaviors from previous approaches, out-competing a number of baselines in success, safety and speed across a range of tasks for safe-arrival and multi-target achievement.

Published as a conference paper at ICLR 2026
DUAL-OBJECTIVE REINFORCEMENT LEARNING WITH NOVEL HAMILTO--JACBBI -BELLMAN FORMULATIONS
William Sharpless", Dylan Hirsch', Sander Tonkens, Nikhil Shinde, Sylvia Herbert University of California, San Diego, "qqual contribution, site
ABSTRACT
Hard constraints in reinforcement learning (RL) often degrade policy performance. Lagrangian methods offer a way to blend objectíves with constraints, but require intricate reward engineering and parameter tuning. In this work, we extend recent advances that connect familton-Jacobi-eee
(HJB) equations with RL to pro- pose two novel value functions for dual-objective satisfaction. Namely, we address: 1) the Beach-Always-Avvii (RAA) problem of achieving distinct reward and penalty thresholds - and 2) the Reach-Reach (RR) problem of achieving thresh- olds of two distinct rewards. In contrast with temporal logic approaches, which typically involve representing an automaton, we derive explicit, tractable Bellman forms in this context via decomposition. Specifically, and RR problems may be rewritten as compositions of previously studied HJ-RL problems. We leverage our analysis propose a variation of Proximal Policy Optimization (DOHJ-PPO), and demonstrate that it produces distinct behaviors from previous approaches, out-competing a number of baselines in success, safety and speed across a range of tasks for safc-arrival and multi-targe achievement.
prove that the RAA
INTRODUCTION
The development of special Bellman equations from the Hamilton-Jacobi (HJ) perspective of dynamic programming (DP) has illustrated novel route safety and target-achievement in reinforcement learning (RL) Fisac et al. (0019;; Hsu et al. (202T). In comparison with the canonical RL discounted- sum cost and corresponding additive DP update, these equations, namely the Safety Bellman Equation (SBE) and Reach-Avoid Bellman Equation (RABE), propagate the minimum (worst) penalty and maximum (best) reward, yielding a value function defined by the outlying performance of a trajectory. In mission-critical applications, where avoiding failure is a necessary condition, these equations have proved invaluable the field of safe control Mitchell et al. 2005): Ames al. (0016). By focusing on extremal values rather than discounted sums, the HJ-RL equations induce behaviors that act with respect to the best or worst outcomes in time-optimal fashions, performing far more safely than Lagrangian methods Ganai et al. (2023); [So et al. (2024). Accordingly, these updates yield policies with significantly improved performance in target-achievement and abstacle-avoidance tasks over long horizons Yu et al. (2022a b), relevant to fundamental and practical problems in many domains.
In this work, we advance the existing HJ-RL formulations by generalizing them to compositional problems. To date, the HJ-RL Bellman equations are limited to three operations: Reach (R), wherein the agent seeks to reach a goal (achieve a reward threshold), Avoid (A), wherein the agent seeks to avoid obstacle (avoid a penalty threshold), and Reach-Avoid (RA), where the agent avoids obstacles until reaching the goal. In this light, we extend the HJ-RL Bellman equations to two complementary problems concerned with dual-satisfaction, namely the Reach-Reach (RR) problem for reaching two goals and the Reach-Alwy Sett--Awwyss-AA Avoid (RAA) problem for continuing to avoid hazards after reaching goal, demonstrated Figure D We prove that the RAA and RR have a fundamental structure such that their Bellman equations may be decomposed into combinations of SBEs and RABES. From this theory, we devise DOHJ-PPO, novel algorithm for learning the RAA and RR values which bootstraps concurrently solved decompositions for coupling on on-policy PPO roll-outs. Notably, this allows one to automatically learn to satisfy dual-objective tasks, for example, in the RAA, the F16 learns to fly into the desired airspace without crashing afterward (Figure top middle-left), and in the RR case, the Hopper learns to jump into a target without diving so it may then achieve the second target (Figure [[] bottom left). The RAA and RR problems are distinct from both standard sum-of-rewe values and the simpler HJ-RL formulations, providing new perspectives and performant tools for constrained decision-making.

---

Published as a conference paper at ICLR 2026
Hopper- -RAA
F16-RAA
SafetyGyn-RAA
Hopper-RR
F16-RR RR
Safetyčym
fcheetah BR
Figure I: Depiction the Reach-Always-Avoid (RAA) and Reach-Reach (RR) Tasks. In RAA tasks, the aero-level set of the rewards (goals) and penalties (obstacles) are depicted in green and red respectively, while in the RR problem, the zero-level set of the two rewards (two goals) are depicted in green and blue. The RAA value is defined by the minimum of the minimum penalty and maximum reward, inducing the agents to enter the goals
time without ever entering the obstacles. The RR value is defined by the minimum of the maximum rewards, inducing the agents to enter both goals at some time.
Our contributions include:
We introduce novel value functions corresponding to the RAA and RR problems.
2. We prove that these value functions and their optimal policies can be decomposed into reach, avoid, and reach-avoid value functions (Theorems) and 2
3. We demonstrate the nature of the RAA and RR values and their optimal policies simple
grid-world example with deep Q-learning (DQN) (Figure 2
4. We propose DOHJ--PP to solve these value functions, which bootstraps concurrently solved decompositions for effectively coupling the on-policy rollouts (Section 7.2).
5. In continuous control tasks, showcase that with little to tuning, DOHJ-PPO is more successful, safer and faster than Lagrangian and existing HJ-RI. baselines (Figure
RELATED WORKS
This work involves aspects of safety (e-g. hazard avoidance), liveness (e-g. goal reaching), and balancing competing objectives. We summarize the relevant related works here.
Constrained and Multi-Objective RL. Constrained Markov decision processes (CMPDs) maximize the expected sum of discounted rewards subject to expected sum of discounted costs, instantaneous safety violation function remaining below a set threshold threshold/Altmani Altmman (202T): Achiam et al. effective way incorporate state constraints into RL problems, and the efficient and accurate solution of the underlying optimization problem has been extensively researched, first by Lagrangian methods and later by array of more sophisticated techniques (Stokk et al. (220):; Li et al. (2024): CCen et al. )) Miryoosefi and Jin (202T): Yang
(2017a); Wachi and Sui 2020). CMDPs
al. (2020). Multi-objective RL is an approach designing policies that obtain Pareto-optimal expected sums of discounted vector-valued rewards Wiering et al. 20014); Van and Nowé Nowe((014)) (2014); Cai
al. 2023), including by deep-Q and other deep learning techniques Mossalam et al. (2016) Abels
et al. 2019): Yang et al. (2019). By contrast, this work explicitly balances rewards and penalties in way that does not require specifying a Lagrange multiplier or similar hyperparameter. Moreover, our work treats goal-reaching and hazard-avoidance as hard constraints, and the learned value function has a direct interpretation terms of the constraint satisfaction.
Goal-Conditioned RI (GCRL). GCRL simultaneously learns optimal policies for range of different (but typically related) tasks Liu et al. (2022); ) Plappert et al. (018)): Ren et al. (019); Ma et al. (2022): Campero et al. (2020); Trott et al. (2019); Eysenbach et al. 2022): : Ma et al. (2022): Campero et al. (2020). In GCRL, states are augmented with information on the current goal. While these goals in their simplest form mostly independent, some work extends GCRL to sophisticated composite tasks Chane- Sane et al. (202I). Our work primarily focuses on composing specific learned tasks rather than learning general tasks simultaneously.
2

---

Published as a conference paper at ICLR 2026
Linear Temporal Logic (LTL), Automatic State Augmentation, Automatons, and Generalized Objective Functions. Many works have been explored that merge LTL and RL, canonically focused on Non-Markovian Reward Decision Processes (NMRDPs) Bacchus et al. (199)). Here, the reward gained at each time step may depend on the previous state history. Many of these works convert these NMRDPs MDPs via state augmentation [Bacchus al. (1997); Thiebaux et al. (2006); Camaacoo ct al. (2021): Icarte et al. (2018); Camacho et al. (2019). Often the augmented states are taken to be products between an ordinary state and an automaton state, where the automaton is used to determine "where" in the LTL specification an agent currently is. Other works using RL for LTL tasks involve MDP verification Brázdil et al. al.] (2014), hybrid systems theory Cohen al. (2023), GCRL with complex LTL tasks Qiu et al. (2023), almost-sure objective satisfaction Sadigh et al. (2014), incorporating (un)timed specifications Hamilton et al. (2022), and using truncated LTL Li al. (017)). While the problems we attempt to solve (e.g. reaching multiple goals) can be thought of as specific instantiations of LTL specifications, our approach to solving these problems is fundamentally different from those in this line of work. Our state augmentation and subsequent decomposition of the problem are performed in a specific manner to leverage new HJ-based methods on the subproblems. Through our specific choice of state augmentation, policy in theory (and approximately so in practice) despite the non-NMRDP setup. There is also significant literature on generalized objective functions in RL Wang al. (220)); Cui and Yu (2023): ;
still prove that
achieve an optimal
Tang et al. (2025). but these works are either not able to or are not tailored to simultancously handle multiple rewards/penalties in the context of safe optimal control, which is where our decompositional approach becomes useful. On the other hand, works that do try to handle multiple rewards and penalties (including by decomposition) still use discounted swwentted-----oot-
objectives van Seijen
et al.. (2017); Pitis (2023): Lin et al. (2020).
Hamilton-Jacobi (HJ) Methods. HJ is dynamic programming based framework for solving reach, avoid, and reach-avoid tasks Mitchell et al. (2005); [ Fisac et al. (2015). The value functions used in HJ have the advantage of directly specifying desired behavior, so that a positive value corresponds to task achievement and negative value corresponds to task failure. Recent works use RL to find corresponding optimal policies by leveraging the unconventional Bellman updates associated with these value functions So al. (002); ; Hsu al. (202T); Fisac et al. (2019). We build on these works by extending these advancements to more complex tasks, superficially mirroring the progression from MDPs NMRDPs in the LTL-RL literature. Additional works merge HJ and RL, but do not concern
themselves with such composite tasks Ganai et al. (2023): Yu et al. (002a)) Zhu et al. (2024).
3 PROBLEM DEFINITION
Consider a Markov decision process (MDP) M = (S,A, f) consisting of finite state and action spaces S and A, and unknown discrete dynamics f that define the deterministic transition 8c+1 = [st, Let an agent interact with the MDP by selecting action with policy : S - A to yield a state trajectory s7. i.c. $7+1 = (57, , (s?)).
In this work, we consider the Reach-Always-Aee both involve the composition of two objectives, which are each specified terms of the best reward and worst penalty encountered over time. In the RAA problem, let r. S - R represent a reward to be maximized and a penalty to be minimized. We will let = -p for mathematical convenience, hence, our aim minimize the largest-over-time (worst) penalty p becomes the aim to maximize the amallest-ovvr-tiee q. In the RR problem, let 71. S + R be two distinct rewards to be maximized. The agent's overall objective to maximize the worst-case outcome between the best-over-time reward and worst-over-time penalty (in RAA) and the two best-over-time rewards (in RR), i.e.
(RAA) and Reach-Reach (RR) problems, which
s.t. maximize (w.I.. T))
min { max; r(77), min; {(s7)}
(RAA)
s = f (ST (s?)),
so = s,
maximize (w.r.t. T) s.t.
min { maxx, ri ()))), max; r2()7)
(RR)
s 1 = f (s7 (s7)),
si
As the names suggest, these optimization problems are inspired by but not limited to - tasks involving goal reaching and hazard avoidance. More specifically, the RAA problem is motivated by
3

---

Published as a conference paper at ICLR 2026
RA Value & Rollout
RAA Value & Rollout
Value Rollout
RR Value & Rollout
RA
V
RAA
VR
V
Figure 2: DQN Erid-World Demonstration of the RAA & RR Problems. We compare novel formulations with previous HJ-R formulations (RA & R) a simple grid world problem with DQN. The zero-level sets of q (hazards) are highlighted in red, those of r (goals) in blue, and trajectories in black (starting at the dot). In both models, the agents actions are limited to (left, right, straight) and the system flows upwards over time.
task which an agent wishes to both reach a goal 9 and perennially avoid a hazard H (even after reaches the goal). The RR problem is motivated by task in which an agent wishes to reach two goals, 91 and 92. either order. While these problems are thematically distinet, they are mathematically complementary (differing by single max/min operation), and hence tackle them together.
The values for any policy in these problems then take the forms VRAA and VRR.
VRA = min { mmax r(s7), min min) and V min mmxx (s7)), max (
One may observe that these values are fundamentally different from the infinite-sum value commonly employed in RL /Sutoon and Barto (2018), and do not accrue over the trajectory but, rather, are determined by certain points. Moreover, while each return considers two objectives, these objectives are combined in worst-case fashion ensure dual-sarie saull-saisfaction Although many of the related works discussed approach similar tasks (e.g. goal reaching and hazard avoidance) via traditional sum-of- discounted-rewarde formulations, these novel value functions have a more direct interpretation in the following sense: if is positive (only) within G and q is positive (only) inside H. RRA (s) will be positive if and only if the RAA task will be accomplished by the policy x. Similarly T1 and r2 are positive within G1 and G2, respectively, VRR (s) will be positive if and only if the RR task will be accomplished by the policy π.
REACHABILITY AND AVOIDABILITY IN RL
The reach VR avoid VÄ, and reach-avoid VRA values, respectively defined by
V (s) ms7) (T( = min (s) max min (((7 min g(s Sst
have been previously studied Fisac et al. (2019) leading to the derivation of special Bellman equations. To put these value functions in context, assume the goal G is the set of states for which r(s) is positive and hazard H. is the set of states for which g(s) is non-positive. See Figure 2 for simple grid-world demonstration comparing the RAA and RR values with the previously existing RA and R values. Then Vi"" VT and VRA are positive if and only if π causes the agent to eventually reach G, always avoid H. and to reach G without hitting H. prior to the reach time, respectively. The Reach-Aver Bellman Equation (RABE), for example, takes the form Hsu et al. (202T)
ViRA (s) = min max max VRA (f(s, a()))))r))))))))))))))))))))aa (9())
aEA
and is associated with optimal policy (s) (without the need for state augmentation, see the appendix). This formulation does naturally induce contraction, but may be discounted to induce contraction by defining ViAa (2) implicitly via
VV (s) (1 7) min (r(),, (())) min max max Vi (f(s, a)) r(s))
a(s)).
aEA
for each 7 [0, 1). A fundamental result (Proposition in Hsu et al. 2021 ) is that lim (s) (s).s).
4

---

Published as a conference paper at ICLR 2026
Reach-Reach
Reach-Always-Avoid
BB
Figure 3: Examples where Non-Augmented Policy is Flawed. In both MDPs, consider agent with no memory. (Left) For a deterministic policy based current state, the agent can only achieve one target (RR), as this policy must associate the middle state with either of two possible actions. (Right) The RAA case slightly more complex. Assume the robot will make sure to avoid the fire at all costs (which is easily done from the current state). It would also prefer to not encounter the cone hazard, but will do so if needed to achieve the target. From its current state the robot cannot determine whether pursue the target by crossing the move to right. The correct decision depends on state history, specifically on whether the robot has already reached the target state or not (e.g. imagine the initial state is on the target state).
These prior value functions and corresponding Bellman equations have proven powerful for these simple weach/void/reach-ane -ariid problem formulations. In this work, we generalize the aforementioned results to the broader class involving VRAA (assure no penalty after the reward threshold achieved) and VRR (achieve multiple rewards optimally). Through this generalization, we are able to train an agent to accomplish more complex tasks with noteworthy performance.
THE NEED FOR AUGMENTING STATES WITH HISTORICAL INFORMATION
We here discuss small but important detail regarding the problem formulation. The value funetions we introduce may appear similar to the simpler HJ-RL value functions discussed in the previous section; however, in these new formulations the goal of choosing policy क S A is inherently flawed without state augmentation. In considering multiple objectives over an infinite horizon, situations arise in which the optimal action depends on more than the current state, but rather the history the trajectory. This complication is not unique to our problem formulation, but also occurs for NMDPs (see the Related Works section). To those unfamiliar with NMDPs, this at first may seem like paradox as the MDP is by definition Markov, but the problem occurs not due to the state-transition dynamics but the nature the reward. An example clarifying the issue shown in Figure 3
To allow the agent to use relevant aspects of its history, we will henceforth consider augmentation of the MDP with auxiliary variables. A theoretical result in the next section states that this choice of augmentation is sufficient in that no additional information will be able to improve performance under the optimal policy. Note that the state augmentation is needed because of the use HJR -tyle optimization objectives (rather than discounted suwwod-rewards). The point of the state augmentation not to make the rewards and penalties Markovian (indeed, they are already Markovian as they are deterministic functions of the current state).
5.1 AUGMENTATION OF THE RAA PROBLEM
We consider an augmentation of the MDP defined by M = (3, !') consisting of augmented states = S x y x Z and the same actions A. For any initial state let the augmented states be initialized as y r(s) and q(s), and let the transition of M be defined by
x = f (s, (sT ve ve, 27))
Vē+1 = max {r (s7+1) ..E
2 = min {q (si+1) 22
such that yt and ze track the best reward and worst penalty up to any point. Hence, the policy for M given by मे S + A may now consider information regarding the history of the trajectory.
By definition, the RAA value for M,
VRA (s) = min { max r(s?), min min {
is equivalent to that of M except that it allows for policy स which has access to historical information. We seek to find ग that maximizes this value.
5

---

Published as a conference paper at ICLR 2026
5.2 AUGMENTATION OF THE RR PROBLEM
For the Reach- Reach problem, we augment the system similarly, except that is updated using max operation instead of min:
si+1 - f (sf . (0) 20 (5 =()) Again, by definition,
Vi+1 = max {ri (6(+1) vi a[ max fr2 (oi+1) F
vi (8) min maxri(s}) maxrr(s)))
The RR problem is again to find an augmented policy म which maximizes this value.
6
OPTIMAL POLICIES FOR RAA AND RR BY VALUE DECOMPOSITION
We now discuss our first theoretical contributions. We refer the reader to the appendix for the proofs of the theorems.
6.1 DECOMPOSITION OF RAA INTO AVOID AND REACH-AVOID PROBLEMS
Our main theoretical result for the RAA problem shows that we can solve this problem by first solving the avoid problem corresponding to the penalty g(s) obtain the optimal value function (s) and then solving a reach-avoid problem with the negated penalty function g(s) and a modified reward function (s).
Theorem 1. Let VRAA (8) == maxe V RAA (s). For all initial states E S.
(s) = max max min FRRA (s7) min (s rSt
(I)
where FRAA(S) = min (r(s), (+)), with
VX(6) max min (8
This decomposition is significant, as methods customized to solving avoid and reach-avoid problems were recently explored in Fisac al. (2019); ; Hsu et al. (202T): [So et al. (2024); So and Fan (2023).
allowing us to effectively solve the optimization problem defining V (s) as well as the optimization problem that defines the right-hand hand-side T
Corollary 1. The value function VRAA satisfies the Bellman equation
max VRAA (f(s, FR)M))) ).(s)) AEAA
VRAA (+) min max
Readers familiar with temporal logic (TL. may be interested in how these decompositions relate decompositions of predicates in TL.. We discuss the distinction between these two classes of decompositions in Sec. of the Appendix, and how the TL predicate algebra is insufficient for safe optimal control.
6.2 DRCOMPOSITION THE RR PROBLEM INTO THREE REACH PROBLEMS
Our main result for the RR problem shows that we can solve this problem by first solving two reach problems corresponding to the rewards rı (s) and r2(s) to obtain reach value functions Vg (s) and VR, (s), respectively. We then solve third reach problem with modified reward FRR (s).
Theorem 2. Let (s) = max= V (s). For all initial states S.
V = max max FRR (s"")
(2)
where (RR(S) = max (min (s), VE, (s)) min (ra(s), V2 (+)}}. with Vg, (s) = max max rn (87), Vž, (s) = max max 122 (s
*
Corollary 2. The value function V&R satisfies the Bellman equation
VVR (s) = max
max V&R (f(s, a)); PRR aEA

---

Published as a conference paper at ICLR 2026
6.3 OPTIMALITY OF THE AUGMENTED PROBLEMS
We previously motivated the choice to consider augmented MDP M over the original MDP in the context of the RAA and RR problems. this section, we justify our particular choice of augmentation. Indeed, the following theoretical result shows that further augmenting the states with additional historical information cannot improve performance under the optimal policy.
Theorem 3. Let 8 E Then
max V (+) I max VRAA(S) (8) max min max (max)(s)): min g($t) }
00,01
and
max V (s) S max V max min m (st), max r2(
40,41
where Se+11 = [(t,, az) and
The terms on the right of the lines above reflect the best possible sequence of actions to solve the RAA or RR problem, and the theorem states that the optimal augmented policy achieves that value, represented by the middle terms. This value will generally be less than equal to the outcome from using a non-augmented policy, represented by the terms on the left.
DOH.J-PPO: SOLVING RAA AND RR WITH RL
In the previous sections, we demonstrated that the RAA and RR problems can be solved through decomposition of the values into formulations amenable to existing RL methods. However, we make a few assumptions in the derivation that would limit performance and generalization, namely, the determinism of the values well access to the decomposed values (by solving them beforehand). In this section, we propose relaxations to the RR and RAA theory and devise a custom variant of Proximal Policy Optimization, DOHJJ-PO, solve this broader class of problems, and demonstrate its performance.
7.1 STOCHASTIC REACH-AVOID BELLMAN EQUATION
well known that the most performative RI. methods allow for stochastic learning. In So et al. (2024), the Stochastic Reachability Bellman Equation (SRBE) is described for Reach problems and used to design specialized PPO algorithm. We first generalize this notion to a Stochastic Reach-Avoid Bellman Equation (SRABE), Using Thcorems I andd] 2 the SRBE and SRABE offer the necessary tools for designing PPO variant for solving the RR and RAA problems.
By anology to the SRBE, the SRABE is given by
(s) = Egaa
min
max
{ (VRAA (f(s, ((s,, (R) {
(SRABE)
More rigorously, we actually consider the discounted SRABE, which is contractive, and the corre- sponding quality function below in the limit 7 - (as in Hsu et al. (2021).
ii (s) (1 7)) (--7)mmi min (FRAa (s). {(5)} + Baar min {
max (ir (f(s, FRAn(S)) {sss)}]
Q (s) = (1 - min (FRAA (()) mimin min max
for RAA (f(s, FRAA(s)) a(s))
Theoretically speaking, the use of the SRABE is justified by Theorem in the Appendix. With this action-value function we then follow So et al. to derive the corresponding policy gradient result with an augmented version of the dynamics; for details, see Prop. in the Appendix. The PPO advantage function is then given by Âğ^ = QRAA Vras Schulman et al. (207)). Although, this approximation may be poor in highly noisy settings, show this approach yields conservative estimates of the value with stochastic policies (Appendix sec. D). and validate it empirically with stochastic dynamics in Sec. 8.3
7

---

Published as a conference paper at ICLR 2026
7.2 ALGORITHM
We introduce DOH.J-PPO for solving the RAA and RR problems, which integrates the SRABE and SRBE via three minimal modifications PPO Schulman et al. (2017) (see appendix for more).
Additional actor and crities are introduced to represent the decomposed objectives. Per Theo- T one may know that the RAA and RR values are given by composition of the simpler R, A and RA values. Therefore, we learn these decompositions with their own networks and integrate them into the composed actor and critic training. namely via the GAE and target with the special RAA and RR reward functions in Theorems[and]
The composed actor and critic are learned concurrently to the decomposed actor and critics by bootstrapping the current values. Rather than learning the decomposed and composed rep- resentations sequentially, DOHJ-PPP bootstraps to learn them simultancously. Namely, at each iteration, rollout trajectories for composed and decomposed updates with each actor. In the update of the composed representation specifically, the decomposed values are inferred from the current decomposed critic(s) along the composed trajectories. This design choice allows us to couple the on-policy learning of PPO in the following way.
Trajectories for training the decomposed actor and critic(s) initialized with states sampled from the composed trajectories, which we refer to as coupled resets. While it is possible to estimate the decomposed objectives independently-icee prior solving the composed task-this approach might lead to inaccurate or irrelevant value estimates in on-policy settings. For example, in the RAA problem, the avoid decomposition will solely prioritize avoiding penalties and, hence, might converge to an optimal strategy within a
region, misaligned with the overall task.
EXPERIMENTS
8.1 DQN DEMONSTRATION
We begin by demonstrating the utility of our theoretical results (Theorems I and 2 through simple 2D grid-world experiment using DQN (Figure 2 In this environment, the agent can move left, right, or remain stationary. while drifting upward at constant rate. Throughout, reward regions shown in blue and penalty regions in red. On the left, we compare the optimal value functions learned under the classic Reach-Avoid (RA) formulation with those from the Reach-Always-Avoid (RAA) setting. In the RAA scenario, trajectories successfully avoid the obstacle but may terminate in regions from which future collisions are inevitable, as there is no incentive to consider what happens after reaching the minimum reward threshold. In contrast, under the RAA formulation, where the objective involves maximizing cumulative reward while accounting for future penalties (as per Theorem [T) the agent learns to reach the target while remaining in safe regions thereafter. On the right, we consider similar environment without obstacles but with two distinct targets. Here, the Beach-Reach (RR) formulation induces trajectories that visit both targets, unlike simple reach tasks in which the agent halts after reaching a single goal. These qualitative results highlight the behavioral distinctions induced by the RAA and RR objectives compared to their simpler counterparts. Additional algorithmic and experimental details are provided in the Apendix.
8.2 CONTINUOUS CONTROL TASKS WITH DOHJ-PPO
To evaluate the method under more complex and less structured conditions. we extend our analysis to continuous control settings. Specifically, we consider RAA and RR tasks in the Hopper, 6, Sa fef and Hal fCheetah environments, depicted in Figure [ In the RAA tasks, the penalty function generally characterizes regions of states where the agent (or its body parts) is intended to avoid, while the reward characterizes regions of states where the agent is intended to reach.
As baselines, we compare DOHJ-PPO against a variety of classes of RL algorithms. We include several augmented Lagrangian methods which transform constraints (either for reaching both or always avoiding) into mixed objectives, such as Constrained PPO (CPPO) Achiam et al. (2017b). PPO- LAG LAGRRa et al. (2019), P2BPO Dey et al. (2024), and LOGBAR Zhang al. (2024). Additionally, we include three HJ-L baselines designed for the previous R and RA problems, RESPO Ganai et al.] (2023). RCPPO So al. (2024) and RA Hss al. (021)) Lastly, we also include a few methods
8

---

Published as a conference paper at ICLR 2026
Hopper-RAA -RAA
F16-RAA
Safetyčya-RMA
HalfCheetah-RRA
Partial Succesa
R
SUCCESS
Hopper-RR
F16-RR
SafetyGym- RR
HalfCheetah--R
Partinl Success
RY
SUCCESS
247
564/90
EOUPE
R1 a11R2
Figure 4: Success (-+) and Partial Success (->) in RAA and RR Tasks for DOHJ-PPO and Baselines, We evaluate DOHIJ-PP in black against baselines 1,000 trajectories in the Hopper, F16, SafetyGym and HalfCheet thhheetah environments. In the first and third row, the Partial Success percentage each algorithm given, defined by the number of trajectories to achieve one objective (reaching or always-avoiding in the RAA, reaching either in the RR). In the second and fourth rows, SUCCESS pereentage is given, defined by the number of trajectories to achieve both objectives. Most baselines achieve partial success, however, few achieve total success as the environment hecomes more difficult, underscoring the difficulty of balancing objectives in RI..
based approaches in STL/LTL-RL and MORL, including decomposed STL (D-STL) PPO, sparse-reward STL PPO (SPARSE) and a MORL-based PPO. All algorithms are trained on random initial conditions and then evaluated new random initial conditions within distribution. To quantify performance of the dual-objective tasks, we measure (1) the percent of trajectories which achieve at least both tasks successfully, (2) the percent of trajectories which achieve at least one of the tasks (dubbed partial success), and (3) the mean steps in each trajectory until success.
Empirically, find that our method performs the top-level, achieving first or second place among all tasks and environments (Figure In fact, for the multi-target (RR) or safe-achievement (RAA) as dynamies become more complex, our algorithm increasingly dominates the 10 state-of-the-ar baselines (c.g. the aa)eetheaa)), Note, that almost all algorithms can achieve partial success at a high rate in each dual-objective task, highlighting the difficulty of mixed or competing objectives, particularly with discounted-sum rewards. Moreover, DOHI-PPO is the sole performant algorithm in both RAA and RR tasks, displaying the fastest achievement times across tasks (see appendix).
These results underscore the challenging nature of composing multiple satisfaction objectives using traditional baselines with discounted-sum rewards. In contrast, DOHJ-PPO provides a direct and robust solution to handling these complex tasks, with little to tuning. Our algorithm enjoys these benefits because the structure of the novel Bellman updates, which propagate the extreme (maximum and minimum) values opposed to the short-term average (discounted-summ) values.
8.3 COMPARISON IN STOCHASTIC DYNAMICS
To design an algorithm robust to randomness, DOHJ-PPO employs the SRBE and SRABE discussed in Sec. 7. place of their analogous deterministic forms. This choice equates to an approximation of the decompositional results (Thms. and 2 that interchanges the extrema and expectation operators. The empirical results in Fig. justify this approximation with stochastic policies, however, this noise is introduced for exploration and ultimately attenuated in training. To interrogate the behavior of DOHJ-PPO with stochastic dynamics, of the Ha Halffhaa wCheehhhtte ah dynamics for both RAA and RR tasks. Note, only the velocities and angular velocities of the agent are perturbed protect contact physics. We compare our algorithm against the
inject affine Gaussian noise into the evolution

---

Published as a conference paper at ICLR 2026
Deterministic Dynamics
4(0.0.52)
MO0, 1.0%)
((0,2.0%)
200%
R0A SUICKESS
DON
SPARS
DDng
DCTL
D00 SUCCEES
SPARSE
LOGBAR
20%
Spcch
Figure 5: Success (1) in the Ha FCheetah RAA and RR Tasks with Increasingly Stochastic Dynamies. We plot the learning trajectories of DOHJJ-PP in black and the top baselines for the Hal #Cheetah environment with an affine Gaussian noise added to the dynamics. Task achievement (success) is given by the percentage of 256 trajectories that either reach the target and always-avoid the obstacles or reach both targets (corespouding to VRAA > 0 and VRR > 0). Each column corresponds to a different scale of noise null, low (0.5), moderate (1.) and high (2.) which is added to the velocities and angular velocities of the Ha ICneet dynamics. In the RAA task, DOHJ-PPO outperforms all baselines up the highest noise settings where all algorithms perform equivalently poorly. the RR task, DOHJ-PPO outperforms all algorithms significantly. summary, this ablation demonstrates the robustness of DOHU-PPO to certain stochasticity in the dynamics and the validity of the SRBE and SRABE approximations.
top three baselines in each task along a scale of standard deviations of low (0.5), moderate (1.) and high (2.) quantity, plotting the maximum learning curve over three seeds in Fig.
In this ablation, we find that the proposed approach, using the novel Bellman equations with stochastic relaxations, offers a significant performance improvement RAA task, DOHJ-PPO dominates the top performing baselines with a 8%%22% peak-performance gap between it and the second best algorithm (and is the fastest to peak-performance) for moderate noise levels, beyond which all algorithms perform equally poorly. In the RR case, we find an even starker result, with all but one experiment demonstrating a >30% improvement in reak-performanee even in the high-noise regimes, with an exception of the moderate noise case where DOHJ-PPO still performs >15% than the best baseline, DSTL. Interestingly. DOHJ-PPO is slower than DSTL to peak performance, but performs twice well at best in three of the four settings. These results demonstrate that despite certain highly-noisy dynamics DOHJ-PPP PPO is competitive at worst and optimal in majority. See Appendix Sec. D for further analysis and discussion of the usage of the SRBE and SRABE approximations.
in the face of significant noise. In the
CONCLUSIONS
In this work, we introduced two novel Bellman formulations for new problems (RAA and RR) which generalize those considered in several recent publications. We derive decomposition results to break them into simpler Bellman equations, which can then be composed to obtain the corresponding value functions and optimal policies. We use these results to design DOHJ-PPO, which shows to be the most performant and balanced algorithm in safe-arrival and multi- achievement. DOHJ-PPO employs the stochastic relaxations of the simpler Bellman equations (the SRBE and SRABE), for which we offer rigorous justification and empirical validation in the case of stochastic policies. As expectation and extrema operations do commute, more work is needed provide guarantees under stochastic dynamics. Nonetheless, we demonstrate through an artificial ablation that DOHJ- PPO can be successful the face of certain dynamic randomness. With regard more complex objectives, it appears one might employ our results to iteratively decompose layered objectives corresponding to temporal logic specifications into graph of Bellman values. However, doing so would require deriving generalized decomposition principles for nontrivial compositions of logical operations. Moreover, a practical algorithm for solving the decomposed graph of values might benefit from a more efficient representation, mechanisms to guarantee convergence, and heuristics to improve sampling efficiency, but we leave this to future work. By solving the RAA and RR values, this work provides road-map to extend complex Bellman formulations, via decomposing Levee problems into lower-level ones, establishing a foundation for nuanced tasks in real- world environments and safe RL.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 11-20. PMLR, 2019.
Joshua Achiam, David Held, Aviv Tamar, and Abbeel. Constrained policy optimization. ICML, 155.17000.02822222 30 May 2017a.
Joshua Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 201 7b.
Eitan Altman. Constrained Markov decision processes: Stochastic modeling. Routledge, Boca Raton, 13 December 2021.
Aaron Ames, Xiangru Xu, Jessy Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions Automatic Control, 62(8): 3861-3876, 2016.
Bacchus, Craig Boutilier, and Adam J Grove. Rewarding behaviors. In Proceedings of the National Conference Artificial Intelligence. pages 1160-1167. cs.toronto.edu, 4 August 1996.
Fahiem Bacchus, Craig Boutilier, and Adam Grove. Structured solution methods for non- Mon-Markovian decision processes. In AAAI/AAA, pages 112-117, 1997.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016. URL
0154011440
Tomá's Brázdil, Krishnendu Chatterjee, Martin Chmelík, Vojtéch Forejt. Jan Kietínský, Marta Kwiatkowska, David Parker, and Mateusz Ujma. Verification of Markov decision processes using learning algorithms. arXiv [cs. 10 February 2014.
Xin-Qiang Cai, Pushi Zhang, Li Zhao, Jiang Bian, Masashi Sugiyama, and Ashley Llorens. Dis- tributional Paretol oujti-objeectve reinforcement learning. Neural Inf Process Syst, 36: 15593-15613, 2023.
Alberto Camacho, Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A Mcllraith. LTL and beyond: Formal languages for reward function specification in reinforcement learning. In Proceedings of the Twenty-Eighih International Joint Conference Artificial Intelligence, pages 6065-6073, California, August 2019. International Joint Conferences on Artificial Intelligence Organization.
Alberto Camacho, Oscar Chen, Scott Sanner, and Sheila Mcllraith. Non-Markovian rewards expressed in LTL: Guiding search via reward shaping. Proceedings of the International Symposium on Combinatorial Search, 8(1):159-16)) 160, September 2021.
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with AMIGo: Adversarially motivated intrinsic goals. arXiv [cs.LG], 22 June 2020.
Elliot Chane-Sane C Schmid, and Laptev. Goal-conditioned reinforcement learning with imagined subgoals. ICML, 1001110000001111300-01 1440, July 2021.
Yi Chen, Jing Dong, and Zhaoran Wang. A primal-dual approach constrained Markov decision processes. arXiv [math. OC), 26 January 2021.
Max H Cohen, Zachary Serlin, Kevin Leahy, and Calin Belta. Temporal logic guided safe model- based reinforcement learning: A hybrid systems approach. Nonlinear Anal. Hybrid Syst., 47 (101295): February 2023.
Wei Cui and Wei Yu. Reinforcement learning with con-cumulative objective. IEEE Transactions Machine Learning in Communications and Networking, 1:124-137, 2023, doi: 109
2023.3285543.
11

---

Published as a conference paper at ICLR 2026
Sumanta Dey, Pallab Dasgupta, and Soumyajit Dey. P2bpo: Permeable penalty Barrier-based policy optimization for safe rl. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages
2024.
Benjamin Eysenbach, Tianjun Zhang, Salakhutdinov, and S Levine. Contrastive learning as goal-conditioned reinforcement learning. Neural Inf Process Syst. .bb.22206.077668:5550--35566 15 June 2022.
Jaime F Fisac, Mo Chen, Claire Tomlin, and S Shankar Sastry. Reach-avoid problems with time- varying dynamics, targets and constraints. In Hybrid Systems: Computation and Control. ACM, 2015.
Jaime F Fisac, Neil F Lugovoy, Vicenç Subies-Royo, Shromona Ghosh, and Claire J Tomllin. Bridging comiltonnjacobl safety analysis and reinforcement learning. 2019 International Conference on Robolics and Automation (ICRA), pages 8550-8556. IEEE, 2019.
Milan Ganai, Chiaki Hirayama, Ya-Chien Chang, and Sicun Gao. Learning stabilization control from observations by learning lyapunov-like proxy models. 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
Nathaniel Hamilton, Preston Robinette, and Taylor T Johnson. Training agents to satisfy timed and untimed signal temporal logic specifications with reinforcement learning. In Software Engineering and Formal Methods, Lecture notes in computer science, pages 190-206. Springer International Publishing, Cham, 2022.
Kai-Chieh Hsu, Vicenç Rubies-Royo, Claire J. Tomlin, and Jaime F. Fisac. Safety and liveness guarantees through reach-avoid reinforcement learning. In Proceedings of Robotics: Science and Systems, Held Virtually, July 2021. doi: 10.15607/RSS.200 202.S.000.....1..7....
Rodrigo Toro Icarte, Toryn Q Klassen, R Valenzano, and Sheila A Mcllraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. ICML, 80:2112- 2121, July 2018.
Tianjiao Li, Ziwei Guan, Shaofeng Zou, Tengyu Xu, Yingbin Liang, and Guanghui Lan. Faster algorithm and sharper analysis for constrained Markov decision process. Oper. Res. Lett., 54 (171007)1071077 May 2024.
Xiao Li, Cristian- Vasile, and Calin Belta. Reinforcement learning with temporal logic rewards. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3834 3839. IEEE, September 2017.
Zichuan Lin, Derek Yang. Li Zhao, Tao Qin, Guangwen Yang, and Yan Liu. Rd"2: Reward decomposition with representation decomposition. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11298 1 1308. Curran Associates, Inc., 2020.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arXiv [cs.Al], 20 January 2022.
Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned rein- forcement learning via f-advantage regression. In S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho, and A Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 310-323. Curran Associates, Inc., 2022.
Sobhan Miryoosefi and Chi Jin. A simple reward-free approach to constrained reinforcement learning. ICML,, abss2217..0226::55555 166666169866666 12 July 2021.
Ian M Mitchell, Alexandre M Bayen, and Claire J Tomlin. A time-dependent hamilton -actobi formulation of reachable sets for continuous dynamic games. IEEE Transactions on automatic control, 50(7):947-95) 2005.
Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning, arXiv [cs.AI), 9 October 2016.
12

---

Published as a conference paper at ICLR 2026
Silviu Pitis. Consistent aggregation of objectives with diverse time preferences requires markovian rewards. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob MeGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv [cs.LG], 26 February 2018.
Wenjie Qiu, Wensen Mao, and He Zhu. Instructing goal-conditioned reinforcement learning agents with temporal logic objectives. Neural Inf Process Syst, 36:39147-39175. 2023.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning, arXiv preprint w'Xi:1910.017088, 7(1):2, 2019.
Zhizhou Ren, Kefan Dong, Yuanshuo Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Neural Inf Process Syst, 32: 12:11344-11 04644-3374, June 2019.
Dorsa Sadigh, Eric S Kim, Samuel Coogan, S Shankar Sastry, and Sanjit A Seshia. A learning based approach to control synthesis of Markov decision processes for linear temporal logic specifications. In 53rd IEEE Conference on Decision and Control, pages 1091-1096. IEEE, December 2014.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017. URL http::/arxiv. httpaat..i...o..aaaa/a
optimization algorithms. CoRR,
1707.06347
Oswin So and Chuchu Fan. Solving stabilize-avoid optimal control via epigraph form and deep reinforcement learning. arXiv [Os, 23 May 2023.
Oswin So, Cheng Ge, and Chuchu Fan. Solving minimum-cost reach avoid using reinforcement learning. In The eighth Annual Conference on Neural Information Processing Systems, 2024. URL httpstts::// http::://oeeeeeeeeeeee http://wwwww..teetoeteeeeeee.ee (2no0gg002221
Adam Stooke, Joshua Achiam, and P Abbeel. Responsive safety in reinforcement learning by PID lagrangian methods. ICML, 119:9133-9143, 8 July 2020.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford
Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Yuting Tang, Yivan Zhang, Johannes Ackermann, Yu-Jie Zhang, Soichiro Nishimori, and Masashi Sugiyama. Recursive reward aggregation. In Reinforcement Learning Conference, 2025.
S Thiebaux, C Gretton, J Slaney, D Price, and F Kabanza. Decision-theoretic planning with non- Markovian rewards. Artif. Intell. Res., 25:17-74, 29 January 2006.
Alexander R Trott, Stephan Zheng, Caiming Xiong, and R Socher. Keeping your distance: Solving sparse reward tasks using aaf--balancin shaped rewards. Neural Inf Process Syst, abs/1911.01417, 4 November 2019.
Moffaert Van and Nowé. Multi-objective reinforcement learning using sets of Pareto dominating policies. The Journal of Machine Leurning Research, 15(1):348333522 2014.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Harm van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Barnes, and Jeffrey Tsang. Hybrid reward architecture for reinforcement learning. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 3398,5408, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained Markov decision processes. ICML,, 119:9797-9806, 12 July 2020.
13

---

Published as a conference paper at ICLR 2026
Ruosong Wang, Peilin Zhong. Simon S Du, Russ R Salakhutdinov, and Lin Yang. Planning with general objective functions: Going beyond total rewards. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14486-14497. Curran Associates, Inc., 2020.
Marco A Wiering, Maikel Withagen, and Madalina M Drugan. Model-based multi-objective reinforce- ment learning. In 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 1 1-6. IEEE, December 2014.
R Yang. X Sun, and K Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. In Advances in Neural Information Processing Systems. proceed- ings.neurips.ce, 2019.
Tsung- Yen Yang. Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection- based constrained policy optimization. arXiv [cs.LG), 7 October 2020.
T'ung-Yee
Dongjie Yu, Haitong Ma. Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. International Conference on Machine Learning, pages 25636-25655 PMLR, 2022a.
Dongjie Yu, Wenjun Zou, Yujie Yang, Haitong Ma, Shengbo Eben Li, Jingliang Duan, and Jianyu Chen. Safe model-based reinforcement learning with uncertainty-awart reachability certificate. arXiv preprint wrXiv:2210.0055. 2022b.
Baohe Zhang, Yuan Zhang. Lilli Frison, Thomas Brox, and Joschka Bödecker. Constrained reinforce- ment learning with smoothed log barrier function. arXiv preprint arXiv:2403. /4508, 2024.
Kai Zhu, Fengbo Lan, Wenbo Zhao, and Tao Zhang. Safe agett reinforcement learning via approximate samilton-jacoke reachability. J. Intell. Robot. Syst., 111(1), 30 December 2024.
10 ETHICS STATEMENT
This project was conducted and completed on entirely responsible and ethical grounds, and meets the highest standard of the ICLR Code of Ethics. We believe the rigor, investigation and communication not only upholds scientific ideals whilst avoiding societal harm, but advances machine learning for the betterment of all society, namely by improving learning to be hyper-tunin tuning, and thus better for the planet and human race. Moreover, the work fundamentally improves the safety and reliability of reinforcement learning, and thus greatly improves society in which machine learning is heavily integrated. Above all, the work is honest, noting limitations and caveats, while depicting the strengths believe make this work invaluable for the field.
performant with much less
REPRODUCIBILITY STATEMENT
All theorems, algorithms and parameters for this work are totally explained in this paper (partially in the appendix) in what the authors believe to be clear and understandable form. All theorems have been proven in detail, including all necessary lemmas, propositions and references authors believe is intelligible. The algorithm proposed in the work is explained clearly in the main text and written lyne-by-line the appendix, along with all hyper-paramett The code for this work has been inherited from another group with security clearance and we are awaiting their response to publicize it, but will do so as soon as possible as we are committed to fair and open resources without bias or discrimination.
manner the
for each environment.
14

---

Published as a conference paper at ICLR 2026
Appendix
Contents
Achievement Speed Results from DOHIJ-PP Experiments
S
Proof of RAA Main Theorem
16
Proof of RR Main Theorem.
24
Proof of Optimality Theorem
24
The SRABE and its Policy Gradient
24
The DOHJ-PPO Algorithm
DDQN Demonstration
Baselines
Details of RAA & RR Experiments: Hopper
35
Details of RAA & RR Experiments: F16
36
J
Details of RAA & RR Experiments: SafetyGym
37
K Details of RAA & RR Experiments: HalfCheetah
3
Contrast with Decompositions in Temporal Logic
3
M
Broader Impacts
Acknowledgments
ACHIEVEMENT SPEED RESULTS FROM DOHJ-PPO EXPERIMENTS
Here present additional results for RAA and RR problems solved with DOHJ-PPO. In both settings, DOHJ-PPO out-performs or matches the best of baselines with less tuming and faster arrival. Notably as the difficulty of the problem increases the gap increases significantly with DOHJ-PPO remaining the sole algorithm that can achieve the task in reasonable time and in both RAA and RR categories.
Hopper-RAs
F16-RAA
Sa SafetyGym- MefetyyGyRRAAA RAA
HalfCheetah- RAA
(90945
STEPS TO SUCCESS
(508
Hopper-RR
F16-RR
SafetyGym-RR
Hal fCheet
Figure 6: Steps to Success (+-) in RAA and RR Tasks for DOHJ--PO and Baselines For the same 1000 trajectories in Figure g we quantify here the number of steps until achievement of both tasks: reaching without crash afterward in the RAA. reaching both goal in the RR. DOHJ-PPO is not only competitive but consistently achieves the dual-objective problems in the fewest number of steps.
15

---

<<9 T, t t2 T. Published as a conference paper at ICLR 2026
PROOF NOTATION
Throughout the theoretical sections of this supplement, we use the following notation.
We let N (0, ..... be the set of whole numbers.
We let A be the set of maps from N to A. other words, A is the set of sequences of actions the agent choose. Given a1, a2 A. and E N, we let (a1, a2]- be the element of A. for which
a1 (t) ag -7)
[ai, a2)
Similarly, given E A and a E A, we let (a, al be the element of A for which
t= 0, t 1.
(t) a(t 1) 1)
Additionally, given A and E N. we let a|, be the element of A for which
(t) = a(t + e N.
The [, operation corresponds to concatenating two action sequences (using only the 0th to (r - 1)at elements of the first sequence), the [; operation corresponds to prepending action an action sequence, and the +| operation corresponds to removing the 0 to (r 1)* elements of an action sequence.
We let II be the set of policies т : S - A. Given s E S and x E II, we let 5 + be the solution of the evolution equation
5. (1 + 1) = ( (E))
for which c,- (0)- In other words, e the state trajectory over time when the agent begins at state s and follows policy æ.
We will also "overload" this trajectory notation for signals rather than policies: given a A. let <a N + S be the solution of the evolution equation
< (t + 1) = f (t), a(t))
for which (0) In other words, 5 (:) is the state trajectory time when the agent begins state 8 and follows action sequence a.
A PROOF OF RAA MAIN THEOREM
We first define the value functions, VĀ RA Vt RAA : + by (x) = max min q (5. (T)), well rEN
VRA (s) max max min well TEN
FRAA (5T min (ET (ET(k))
RST
V (8) max min max (E)(
min 4 (5. (57 (k))
well
TEN
KEN
where FRAA is in Theorem I
We next define the value functions, U ÜRA URA S + R, which maximize over action sequences rather than policies:
(x) = max min 7 (T)) aEA TEN
(s) max max min aeA TEN
FRAA (E)(T)) min y (K)
KST
URA (s) max min WEA
maxx (E)(T) min g (E. (m))
TEN
REB
Observe that for each E S
v 2 (s), (s) 2 (s),
(s) 2 (s).
We now prove series of lemmas that will be useful in the proof of the main theorem.
16

---

Published as a conference paper at ICLR 2026
Lemma 1. There is II such that
x = min q (47 (T))
reN
for all s E S.
Proof. Choose such that
(s) E arg max o (f(s,a)) V.
aeA
Fix E Note that for each T E N,
A (5 (T+ (V(
max (E (()a))
aEA
max max min "EA
(cl.aa] (16 + 1)
max maxmin "EA ag KEN
max min 9 aE KEN
c ((
z max ming g (x))) ac KEN
ca
e (c. (T))
follows by induction that oA (T)
2 2x (5. (0)) for all E N. so that
(s) min q (5" (r)) 2 min v (T) ox (c (0) (s).
reht
Corollary 3. For all s S, we have (A(s) = (8).
Lemma 2. There is a π E such that
G (s) = max min
FRAa (55 min q (5% (r))
TEN
RST
for all S.
Proof. First, let us note that in this proof we will use the standard conventions that
max -o and min +00.
We next introduce some notation. First, for convenience, we set 2² = üRA and V VRA- Given
E S and E A, we write
FRAN (E*(T) min (5. (1²())) }
(a) max min TEN
RST
Similarly, given e S and II, we write
(*())
max min TEN
FRAA (TT(T)) min (57 (x))))
A5T
Then
(*(+)
max max min sell TEN
TRAA (5. (T)) ming g (5T (R))
mmax V (*)s),
KST
well
and
max max min (FRAA (8.) (T)) min y (5 (k)) aEA TEN
max v (s). aeA
KST
It is immediate that (s) 2 (*(s) for each S, so it suffices to show the reverse inequality. Toward this end, it suffices to show that there a π E II for which (*(s) = U (s) for each s E S. Indeed, in this case, V* 2 V (s) o² (s).
17

---

Published as a conference paper at ICLR 2026
We now construct the desired policy Let ag t00, Su 2, and 0g : -
-00.
We recursively define a E R, St S S, and 07 : S + R U {oo} fort = 1.2.... by
max (RAA(S), max of (f(s, .(0)). aEA
a++1 = max min S
(3)
min max (RAM(()), max (f(s.a)) .9(5) = act1 } (4)
S411 = St U
8 S S SSe
aEd
(s)
SE St:
(()
04++
Se+1 \ St:
(5)
00
s Se+1
From
4
follows that
So S Si C S2 5
(6)
which together with
3
shows that
00 2 ai 02
Also, whenever S \ S, non-empty, the set being appended to St in (4) is non-empty so
Us. S.
(8)
t=0
For each E S, let (s)) be the smallest E N for which s E S... We choose the policy π E of interest by insisting
t (s) E arg max o ( Vs E S.
(9)
aEA
In the remainder of the proof, show that (*((s) 8² (s) for each S by induction. Let n N and suppose the following induction assumptions hold:
(*(s)= = = o 2 On 00(s') an Vs' S Sn.
Vs E Sn.
(10)
(11)
Note that the above hold trivially when since So Ø and on +00. Fix some particular y E Sa+1 and some 2 E S Sn-1. We must show that
(T( n² (v) ( 2 aa+1,
(12)
x" (2) Ont1-
(13)
In this case, induction then shows that (=((s) (8) for all E Sr. Since this union is equal to S by the desired result then follows.
To show
12
13
we first demonstrate the following three claims.
1. Let x E S and E A be such that fax w) Sn and ((x) 2 an+1 We claim x E Sa+1-
We can assume & Sn, for otherwise the claim follows immediately from (6) Since fax w) E Sn, we have v (f(x, w)) 2 an by (IO). Thus
max (x)A(T), max of ((( ((a)) aeA
ant1 min
min [max(FRAA((( an an+1}
aa+11
where the first inequality follows from 3 and the equality follows from 77) Thus
Ou+1 min max (RAA(T)), max of (f(r,a)) 0000)
aEA
the claim follows from 44
18

---

Published as a conference paper at ICLR 2026
2. Let x E Su+1 Sp, and w A be such that far w) Sn. We claim that
v=(x) = v²(x) ap+1.
(14)
To show this claim, will make use of the dynamic programming principle
(s) min max (FRAN(S), ul (f(s)a(0))) .a()), S,E,,
from which follows that
V (s) min {max (s)) V (f(s, (()s)) ((5) s
(15)
and
0*(() min max FRAn(S), max ([(s,a)) .(0)) AsES.
(16)
aEA
Since a E Su+1 Sn. then o(x) n + by definition of o, so ;())
arg max 2n (f(z)a) by Thus
" [f(x,,())))
maxe (f(x,
(17)
aEA
But then
" (f(xz (x))) 2 o (f(x, On 0n+1 -0,
where the second inequality comes from (TO), the third comes from 77. and the final inequality comes from (3) (S'S Sn, is non-empty because x E S.)) Thus f(x (x)) E S, by (5). It then follows from (10) that
v (f(x, n (x))) - (S(z, ((x))
= (()))
(18)
Now, observe that for all E Sn and A ES Sn
o 0'(s) = (s) 2 an 2 v² -00 =0
(19)
where the first equality and inequality and the final equality from (5). Moreover, f(z, a) E S, for at least one a (in particular
from 100 the second inequality is from (I).
w). Letting A' = {a A [ a) E S.), it follows from (9 that
max v² (f(x,a)) = max v (f(x, = max v, (f(z, a)) max e (f(x, (20) aEA
aed'
aeA
aeA
From
20
we have
V" (f(z, maxv" (f(z, max o (f(x,
(21)
aEA
aEA
Now observe that
V'
(x) min mmax (FRAa(z), V (f(z, (x)))) x(x))
0 (a:) min
max (R)A(E),
max o" (f(x, a)) .2(x))
aEA
On+1
min
max
(x) max of (f a)) .((x)
aEA
where the first equation is from (15). the second from (16), and the third from But then (14) follows from the above equations together with (2)
3. Let x: S Sn. We claim that u² (x:) an+1. Suppose otherwise. Then we can choose
a E A and 7 E N such that
FRAA (T)) min g (E2(K)) > antl. RST
min
(22)
It follows that (T) E Sn, for otherwise
Out1 min (F
19

---

Published as a conference paper at ICLR 2026
by 33. creating contradiction.
So x & Sn and (T) E Sn. indicating that there some {O....,T T - 1] such that (0) $ Sn and f (0), a())) ( +1 Sp. Moreover, q (E) an+1 by (22)
It follows from claim 1 that (0) S.+1.
But then it follows from claim 2 that o" (52 an+1- However,
o" (0) 2 min
TRAa ( (T - -0) min
(Eero) (k)
"MST-8"
min
FRAA (52 - + 0))
min a (5 + +0)) nST
-min
FRAa (E)(
min
(5))
We (8.0+++
On+1.
giving the desired contradiction.
Having established these claims, we return to proving (22 and 13 hold. In fact, 13 follows
immediately from claim 3, so we actually only need to show (22
y E Sa, then from (5) and TOP. we have that (v) = 0 (y) r (v)
6 (V), and from
and (10) we also have that u (t) an an+1- Together these establish when y E Sn-
So suppose y E Sa+1 \ Sn. First, observe that 1 (v) an+1 by
There are now two
possibilities. If there is some E A for which f(y, E Sp, then 12 follows from claim 2. If instead, f(, a) & Sn for each a E A, then max o (f(y, a)) xc by (or = by definition of 17). Thus ant1 min (V), (V)) by (4, so
2 min (FRAA (v), {()) aat1 2 vy)),
where the final inequality follows from claim This completes the proof.
Corollary 4. For all s S. we have (s) ÜRA (s).
Lemma 3. Let F - R. Then
supsup sup F ((a,,')),7) rENa'EA
supsup F (a, 7) TEA
(23)
Proof. We proceed by showing both inequalities corresponding to 23 hold.
Given any E A and E N, we have supares' F a(a,a'),, r) 2 F (a. Taking the
suprema over a E A and
N on both sides this inequality gives the desired result.
Given any a E A and 7 E N, we have
sup a'EA'
([a, a a()r:)) sup F (a"", ,T,,
a"EA
that the result follows from taking the suprema over e A and on both sides of this inequality.
Lemma 4. For each S,
URAA (8) (s).
20

---

Published as a conference paper at ICLR 2026
Proof. For each S, we have
R (s) = max max min aε TEN
ming FRAA (E2(7) ming (5. (x:))
(24)
max max min AE TEN
(E)(T) , (E)(T) ming ( (())x))
(25)
max max min DE TEN
r (ET())) max min
ca' (E)())(())
min y (5. (x)))
EA C'EN
RST
max max min aEA TEN
(E)(T))
max min E'EA EN
( d (T + +()))
min q (x:)
NSt
max max max min DEA TEN E'EA
r (T)(
min K'EN
و ( + k3)
min q (5% (x))
max max max min aEA rEN E'EA
r ( (()
min N'EN
(5. (r + n'))
min
(s!a,a()(())) (x)
(26)
max max min DE TEN
r (T)(T)
min E'EN
(5" ( + R²))
min (5)(k)
(27)
KST
max max min aεA
r (E)(T))
min
(5 (R))
TEN
KEN
max min AEA
m (T))
min
(5 (x))
TEN
KEN
(s),
where the equality between 24) and 25 follows from Corollary and where the equality between
126 and (7 follows from Lemma 3
Before the next lemma, we need to introduce two last pieces of notation. First, we let II be the set of augmented policies मं S x y x Z A. where
y {r(s) S} and Z = {g(s) s SEE SE.}.
Next, given E S, E v. E 2, and ग E II, we let the solution of the evolution
E N - S, -h N - V. and C,- : N + Z, be
( 1) (,0,
7 1) max (1(((()))))
() ({++1)) min (9 ((
for which (0) = s, T (0) = r(s). and (0)= q()).
Lemma S. There is a गे E such that
( min
max (E (T).
min
TEN (5. (T))
(28)
TEN
for all 8 E S.
Proof. By Lemmas and together with Corollary we can choose n, N,0 II such that
RA (s) max min { (ET(())) v2 (CT(T) ming (5)))) Vs E
TEN
(s) = min q (T)
Vs S.
TER
We introduce some useful notation we will use throughout the rest of the proof. For each s E S, let [s] + = f(s, (s)), w/] max ([)]+)), [=] min(z,g ([s)+)).
21

---

Published as a conference paper at ICLR 2026
We define augmented policy by
T(s)) g(,, = (8(s) ((ss)
min [=] (f((s))) 2 2
otherwise.
Now fix some & E S. For all E N. set 8, = (t), 3t = (t) = ma r(ā,). and & = 5. (t) = gin,gi (ā,), and also set 2 E3 (t). vs r( and 23 mi (2;)).
First, assume that t is such that min [o [ 22 ([+,)))} < min &. (3t)) In this case. i (Te, Ne, 2,) = (Tt), so that
min ux (a.) = m ox (-+-1))
by our choice of Since H. is non-decreasing in 4, thus have
(i, 2,, v2 (++)} S min Se+1, x (+z+1)).
Next, assume that is such that min [Te I. [ ( })} 2 min (ve, 2e, o In this case, we have that म (3t, De, 2)) = π (Tt),. so
min(fr Ze, x ( < min [. [ Vx ( |+)) }} min 2+1, vA (f++1))
It thus follows from these two cases that min(N. &, v ( is non-decreasing in f. Let T= min {i E N min(( [+] ex ((+]+t) < min 2, {
There are again two cases:
(T < 00) In this case, (āo,, Dt. 2e) = (T+) for t < T. Then Te = =1;, Dt = yi and ã = 2 for all
t T. It follows that [āz]* = 2 [0d = Fe and [Z] = 2441 for all t S T. Thus
by definition T.
min (Vi+1 24 oA ( 2 min 1i (xt) Vt <
and
min (vi+1 21 A (z]+1)} } < min (Vž कî, Vx (+))}.
But since yf is non non-decraasing and min &f ex (zi)} is non-increasing in t, it follows that min 2f. of (x:)} must achieve its maximal value at the smallest t for which it strictly decreases from t to t + i.e
min (UT, žτ, o (TT)) = min (vT, 27, ex (x+))
= max min (ve zi v (x;))
reN
max fr (x.) 2. vx (x;)}
reN
= (s).
where the final equality follows from our choice of म. Since min Ze, (xx))) is non- decreasing in t, then
min 2
min 24, vx 2 min žr, (āT)) = (s) WE T.
Thus
(x) 2 min
max LEN
(7x) min q (ix) lim min (Ve, 2.} 2 TRA = TRAA (+); CEN
t-
where the final equality follows from Lemma Thus the proof is complete in this case.
(T 00) In this case, (0o,, Dt, 2t) (5x) for all E N. Then x; Vt vi- and E, 2f for all t E N. Also [Tz]+ 2i+1. [7 vi+1. and [a 2441 for all t N. Thus by definition T,
min ( o (1t+1)} 2 min (vi 20 (20) W.N..
22

---

Published as a conference paper at ICLR 2026
Let T' arg mN min foi, 22 Vx (zi)). Then min (UT,, ET, V/ (Ār:)} min fo9,, 27,, v2 (x%-) max min (ve =/ oA (x:)) CeN
2 max min (r (2]) 2 2 (z]))
TEN
(s).
The rest of the proof the follows the same as the previous case with '' replaced by T".
Corollary 5. For all s S. we have (A) URAA(S) (s).
Proof of Theorem 7 Theorem I is now direct consequence of the previous corollary together with
Corollary aand Lemma 4
A.1 A DIRECT DERIVATION OF THE RAA BELLMAN EQUATION
Here, we offer a direct derivation for the RAA Bellman equation. Note, this derivation does not guarantee that the resulting Bellman equation is unique, and just for inuition for the rigor above.
URAA (s) =
max min
max (E(0......))
r(x20
(T)),
min NE IE.
((x (N:)
min g(x (R)) (1)))
min (s), max min max rE(0,1.....
r(x (xx0 (T)): (1.2......
4g.01....
min
((8,, max min max ag.03....
r(s): TE (1.2....
max
rxx (x (7))
min
((x50 (k))
(1.2......)
(29)
where x.
x.0-01- is the system trajectory starting from state under the sequence of actions ag a1...... Using the identity min max(a, c} = max min(a, c}, min(b,c]
URAA (s)
min q()), max max ag.01....
min
r(s), min q(x20.01... KE(1.2.... } ((x)0.00<(x)))
(w))
min
min RE (1,2...
((x max r(x2oratm =E(1,,... 1
(7))
min (s))) max min r(s),
max a1.01 K (1,2
min
((x00.11..(())
max min
min (1..1.
max r(x (xx20.tmm(77) =E(1.2....)
00.01
min
(s), max
min r(b), max
min 00.01 RE (1,2.....
g(x ((x (w)) max max maxxRRAA((ss,a)
g(x (())) max oi (fAA((()
min ((s), max
min q(s), rs)), max min ag.01... (E
min (ss,, max min r(s), max min ( (k)) max URAA (f(s, ag.01.... RE (0.
where in the penultimate step we used the identity minfa, max c)} minfa, max(minfa, b), cH}. Noticing that v (s()) = max.o. min g(x (x)), we
have
()) = min ((ss), max min (r(s), ox max (fA,)(())))))
(30)
This completes the derivation of the RAA Bellman equation.
Lastly, we may note that if define F(s) min(r(s), vz (s)}. the above becomes the RA Bellman equation,
( = min {a(s), max { ((()), max UÑA (f(s,)))
(31)
23

---

Published as a conference paper at ICLR 2026
B PROOF OF RR MAIN THEOREM
We first define the value functions,
RI. ViO VR. VRR S - R by
Vii (s) max/i (T)),
hell
R2 (8) max max r2 (T),
nell TEN
R (s) max max FRR (C. (T))
hEll TEN
VRR (s) max min
max (E
max (E (7)) TEN
mEll
TEN
We next define the value functions, UKII: 'R2: RR. URR S + R, which maximize over action sequences rather than policies:
RI (8) max max T1 (T),
acA TEN
R2 (8) max max 72 (T)(
0R (s) max max FRR (T)(
aEA TEN
UR (s) max min
max 11 (T)(
max ((((r))
REA
TEN
TEN
where FRR is as in Theorem
Observe that for each E S,
uk (s) 2 Vii (s),
(s)(s) 2 (s),
(s) 2 (s),
( 2 (s).
We now prove series of lemmas that will be useful in the proof of the main theorem. Lemma 6. There are T1, n2 E TI such that
U (s) max r (E (T)) and (s2(s)
max 72 (53 (T))
TEN
TEN
for all s S.
Proof. We will just prove the result for (s) since the other result follows identically. For each 8 E S, let T. be the smallest element of N for which
max (T.)) U
aeA
Moreover, for each E S, let a, be such that
Ti (5 (T.)) = uki (8).
Let π E II be given by क (s) = a, (0). It suffices to show that
(E (T,)) ui (s)
(32)
for all S, for in this case, have
( (s) 2 max (ET) (T)) 2 r (ET (T.)) (s)
Vs S.
rEN
We show
32 holds for each S by induction r,. First, suppose that S is such that T, 0.
Then
(5 (Tx)) = ri (s) r (E. (Tx)) = u (s).
For the induction step, let n e N and suppose that
( (Tx)) ( (8) vs S such that 7, n.
24

---

Published as a conference paper at ICLR 2026
Now fix some E S such that 72
1. Notice that
Mki
(x) 2 UKi (S (x, (x)))
5 ((())()))()))
maxni
aeA
ach
n
(amm(())
(ch (n 1))
n
r ( (Tx))
URi (x);
so that uii (S ( (x))) = (R)(() and [s (a)) S n. suffices show
[x)x n,
(33)
for then, by the induction assumption, we have
ii (E (T=)) = ri (f)))))t)))(())))) = Uki (S (I)) = "RR (2)
To show
33
assume instead that
[())x((()) < n.
But
URI (x) maxri
(52 (T((, (e)) + 1))
aeh
( (T)() + +1)
2r1
= r (4 (T(()(())
vii (I
(1),
so that
Uei (I) max ri (Et (T(()(()) + 1)) aEA
and thus
S (ol) + < -++1
giving our desired contradiction.
Corollary 6. For all s S, we have V Viki (s) and (x) (x)
Lemma 7. There is Tl such that
(s)
max FRR (5% (T))
TEN
for all S.
Proof. This lemma follows by precisely the same proof as the previous lemma, with Tı, 2 and m replaced with FRR, og and æ respectively.
Corollary 7, For all s E S, we have Viz (s) = (s).
Lemma 8. Let E1 + and (2 N - RR. Then
sup max { min (T()), sup Ca(T + r') TEN
min (uu (((T (((T+T)) +2)))
PEN
-min
sup &i (T), sup Ce(TT) (TEN
TEN
Proof. We proceed by showing both inequalities corresponding to the above equality hold.
25

---

Published as a conference paper at ICLR 2026
<<
Observe that
sup max TENl
min
(1(T),
sup l2(T + ] min MEN
sup Gi(T + + (TER
sup sup 62(T + tr7] min TEN TEN
I max
min
sup sup (T)), TEN
sup G1(T + r sup
sup (2(T) JEN
(TeN TEN r'EN
min
sup TEN
(I)), (e(r)
TEN
(2) Fix > 0. Choose 71, E N such that (m))) > supfEN (i(T) - & and (2(12) 2
suPreN (e(T) - 8. Without loss of generality, we can assume T1 S 72. Then
min sup Ci(T + +a))))())) } T'EN
sup max min (I(T), sup C2(T + +1)) T'EN
TEN
sup
(I(T), sup Se(T +7) T'N
reNl
min
(((m)) sup G2( + N'EN
min
(((
min
sup (T)) - sup (2(T) - E]
(rEN
reN
min
(up (e(T) -E.
[ren
TEN
But since >> was arbitrary, the desired inequality follows.
Lemma 9, For each E S,
(s) U (s).
26

---

Published as a conference paper at ICLR 2026
Proof. For each S,
2R 8) max max 'RR (T)( AEA TEN
(34)
max max max {min (Ti (T)( tik (T)) min toh
(35)
agA TER
max max max acA TEN
min
T1 (C2(T)
max max a'EA T'EN
co' (+))
min
max max A'EA T'EN
(((())(1)
12 (E.
max max max acA
min
T1 (T))
max max T'ER
(c!aal)
- }))} }
TEN
(T 72 (5.
min
max A'EA T'EN
maxra (ل!= - + イ))
max max max max DEA TEN A'EA
min
r1 (ET(
T'EN
(s(ara)) (F +イリ) 2. (E5(T))
min
maxri T'EN
r1 (()aa()((T)
mmx 12 (
max max max max DEA TEN A'CA
min
TEN
min
max T'EN
(T +')) دي (((=
(36)
max max max
min
r1 (T)) maxrs (E +
aEA
TEN
N'NN
min
maxri T'EN
(5 (T + 7)) (5
(37)
= max min
max
(C)(
max 12 (E)(T) TEN
(38)
aEk
TEN
(8),
where the equality between 34 and 35 follows from Corollary 6 the equality between 36 and 37 follows from Lemma 3, and the equality between 37 and 38 follows from Lemma g
Before the next lemma, we need to introduce two last pieces of notation. First, we let II be the set of augmented policies ग S x y x
+ A, as in the previous section, but where
y = (s)
s E S} and z (r((s) i SE.).
Next, given E S, y y,2E E Z, and ग II, we let Eo : - S. के. IN - V. and C5- NN- - Z, be the solution of the evolution
( 1) (00
77 + 1) 1)= max (n ((++1)))
2 (t [[t++)) = max
for which (0) = =, (0) = Ti (s). and -0)
Lemma 10. There is मे E II such that
(s) = min
max
r1 (5%
max r2 (E. (T))
TEN
TEN
for all 8 E S.
27

---

Published as a conference paper at ICLR 2026
Proof. By Lemmas 6 and 7
together with Corollary 6 we can choose n, 81 02 II such that
(s)
max/i (5.
ss S,
rEN
(s)
Vs E S max max {min {ri (5. (7)) , viz (T))) min (r2 (E) dh (T))) TEN
max
(T))
reN
(s(s)
Vs e S.
Define TT
i(s))
max [u, < (s)))
7(8,8;2)
(1(s)
max (U, 2 (5(s) and S 2,
(2(s)
max 2 (8(()) and > 2.
Now fix some E S. For all E N. set
(4), Tr (1) max (8+) and E -0: (1)
max+g: (2(T)), and also set x = c (t). It suffices to show
RR () < min
max FEN
(i+). max r2 (2,)
(39)
TEN
since the reverse inequality is immediate. We proceed in three steps.
1. We claim there exists a / E N such that max (Ta( (2())) 2 (Tt). Suppose otherwise. Then (Te, De, 2e) (T+)) so that कe = =a for E Thus
max (r1( max (Bx)
tEN
BEN
(s)
max {min {ri (æ+) uka (x+)) min {r2 (x+) uki (x))
TEN
mmx max {min {ri (z+). uRa (++)) min (r2 (æ+), uii (++)
TEN
< max max (e.)
TEN
providing the desired contradiction.
2. Let T be the smallest element of N for which
max (Ti(#r), (2(()))))(((()))) (=(rr) 2 (*T),
which must exist by the previous step, and let T" be the smallest element of N for which
max {min {ri (zf²), ukz (+î.)} min (r2 (xir), via (2+r (s), which must exist by our choice of n. We claim 7" 2 T.
Suppose otherwise. Since 3, = x for S T. then in particular ØT = T.., so that
max {min {ri (ØT)) uRz (5r }} min (r2 (Fr)), oki (FT }}} 0R (s).
But then
max(ri (iT) 12 (āT)) 2 i (āT))
By our choice of T. we then have T T", creating contradiction.
3. It follows from the previous step that
(iT) (x = (s).
By our choice of T. there are two cases: r1 (ārr) 2 0 (Tr) and r²())) 2 (TT). We assume the first case and prove the desired result, with case two following identically. To reach contradiction, assume
r2 (2
0 (TT)
V.
28

---

Published as a conference paper at ICLR 2026
But then (0t, yr, 2,) = 02(2,) for all f 22 T, so up (ãr) m2x < (0r) S
(s(s Thus r2 (2 S (22 S " (ār)) (g(s) It follows that
max {min {ri (z Uiz (atr)) min {r2 (+p²) URi ) < (s),
contradicting our choice T
Thus r2 (T,) 2 (āT)) (s) for some / E N and also ri (āT) 2 (g((T) = ())(s,, so
that 39 must hold by Lemma
Corollary 8. For all 8 E S, we have (s, ri), (2))) =
Proof of Theorem The proof of this theorem immediately follows from the previous corollary together with Corollary 7 and Lemma
C PROOF OF OPTIMALITY THEOREM
Proof of Theorem 3 The inequalities in both lines of the theorem follow from the fact that for each II, we can denne corresponding augmented policy म E II by
न (s. = (s) Vs E S. 2
VRR for each S. Note that in general, we cannot define a corresponding policy for each augmented policy, so the reverse inequality does not generally hold (see Figure 3 for intuition regarding this fact).
in which case AA (s) VAAAA (s) and VRR (s) (s)
The equalities in both lines of the theorem are simply restatements of Lemma and Lemma 9
D THE SRABE AND ITS POLICY GRADIENT
We first justify the SRABE from a theoretical perspective. For each E (0, 1) and stochastic policy S + ((4) (with ((4) the probability simplex on A), let V... RA RA S - IR be the (unique)
solutions of the Bellman equations
RÁ (s) = (1 7) min (r(s)) + 7Ea~x min
max
T. RA
(f(s, a)) (f(s,a)), r(s)) .9(s)
RÁ (RA (s) = (1 7) min {r(s), {())} + min
max
max ViA (f(s, r(s) 99(s)
RE.A
respectively, where r: S + R and q: + R. Note that the above Bellman equations are indeed y-contractive.
Theorem 4. Let E 1). Given any π : S + A(A), we have
V7. <
V7*
RA
RA
Moreover, there exists n² SS - A(A) such that
V
V
Proof. Let + ((A). Define the Bellman operators B7, B:, RS + R$ (where IRS is the set of all maps u: S - R)by by
[s]((s)== = (1 - 7) min (r(s), {(})} + Basa [min {max (u (f(s a)) ((s))
max v (f(s, a)) r(s) ((s)) ||. aEA
B]]d(()) = (1 - 7) min {r(s), {s))} + min max
respectively. For each E IRS we have BT < B:wl Then V. = B37 < B: and ViAA" ([(x))
29

---

Published as a conference paper at ICLR 2026
Since Bº is contraction and also monotonic operator (B _ Bt[w] when S w), it follows
from the comparison principle for Bellman operators that V
I
1/7+*
RA
"RA"
Now let n² S A be such that (s) is supported on arg mrg v(((s,))) " Then Von." B7 RA = RA and RA = Ba[V so that RA VA
To understand the significance of the above theorem, recall that when solving RA problems (as is needed during our solution of the RAA problem) we are interested in estimating V in the limit RA
7 - 1 (see Proposition 3 in Fisac et al. 2019)). The above theorem tells us that to obtain VRA we can search for a (possibly stochastic) policy æ that maximizes V7, Doing so allows us to use the PPO adaptation described in the DOHJ-PPO algorithm for finding RA value functions. Analogous
results hold for the and A subproblems.
D.1 POLICY GRADIENT
By analogy the SRBE, the SRABE is given by
(s) = Egwa min
max
{ (RRAA (f(ss
FRAA(S) .9(8)] (()())))))
(SRABE)
The corresponding action-value function
Quan(s, a) min { (max {VRAa (f(s, FRAa(S)) (F(,,a)) a(s))
We define modification of the dynamics f involving an absorbing state Soo as follows:
[f(s,a)
q (f(s,a)) < (s) < FRAa (f()
((
800
otherwise.
We then have the following proposition:
Proposition 1. For each s S and every 0 R'p, we have
ViTe (s) Eg'ndd (s) [ ( in
where (s) is the stationary distribution of the Markov Chain with transition function
P(s's)) Σ [((ass) =]],
aEA
with the bracketed term equal to the proposition inside is true and 0 otherwise.
Following Hsu et al. 2021) then define the discounted value and action-value functions with y E 1).
Proof Sketch of Proposition [ (adapted from So al. (2024)). We here closely follow the proof of Theorem in So et al. (2024). which itself modifies the proofs of the Policy Gradient Theorems in Chapter 13.2 and 13.6 Sutton and Barto (2018). We only make the minimal modifications required to
30

---

Published as a conference paper at ICLR 2026
adapt the PPO algorithm developed previously for the SRBE to on for the SRABE.
Vo Voke (s) =Vo
Σ mo(als)
Σ aEd
+ t Vo min {max { (Veis (f(s,a)) ( ((()))
Σ aEA
+ (T(ass) [q(s)
VRAA (f(s,a)) FRAN(S) VOURAN (f(s,)))
(40)
Σ
Pr(s T)
(41)
ES
Voto(als') To(als")
Σ
Σ Pr(s + s,,,TT)
Σ
To(als')
Qxe QRāA(S'aa)
Ee
aEd
Σ
Σ
Pr(s + k)
Bontu(o")
Veln QRAa
SES
K==0
By'nd' (5) BawTo(s))
(
where the equality between 40) and comes from rolling out the term V RAA a)) (see Chapter 13.2 in Sutton and Barto 2018) for details), and where Pr(s + s k. π is the probability that under the policy म, the system in state s' at time given that it is in state at time 0.
Note, Proposition D is vital to updating the actor in Algorithm
E THE DOHJ-PPO ALGORITHM
In this section, outline the details of our Actor-Critic algorithm DOHJ-PPO beyond the details given in Algorithm[]
In Algorithm| 1 the Bellman update Q", i] differs for the RAA task and RR task, and the [?[ differs between the reach, avoid, and reach-avoid tasks.
E.1 THE SPECIAL BELLMAN UPDATES AND THE CORRESPONDING GAES
Akin to previous HJ-RL policy algorithms. namely RCPO Yu et al. (022b)). RESPO Ganai et al. (2023) and RCPPO So et al. (2024), DOHI-PPO fundamentally depends on the discounted HJ Bellman updates Fisac et al. 2019). To solve the RAA and RR problems with the special rewards
defined in Theorems & 2 DOHI-PPO utilizes the Reach, Avoid and Reach-Avoid Bellman updates, given by
B rl(s. a) (1 - 7)((s) + max (r(s),Q(s,,))
(42)
B (s) (1-7)q(8) + min (1(s). Q(s,) a)}.
(43)
B (s. (1-7)mil min (r(s). min (q(s), max (r(s), o)).)
(44)
To improve our algorithm, we incorporate the Generalized Advantage Estimate corresponding to these Bellman equations in the updates of the Actors. As outlined in Section A of et al. (2024), the GAE may be defined with reduction function corresponding to the appropriate Bellman function which will be applied over a trajectory roll-out. We generalize the Reach GAE definition given in So et al. (202)) propose Reach-Avoid GAE (the Avoid GAE is simply the flip of the Reach GAE)
31

---

Published as a conference paper at ICLR 2026
Algorithm 1: DOHJ-PPO (Actor-Critic)
Require:
Composed and Decomposed Actor parameters 0 and 8, Composed and Decomposed Critic parameters w and wis GAE X, learning rate 3x and discount factor 7. Let B amd B represent the Bellman update and decomposed Bellman update for the users choice of problem (RR or RAA).
1: Define Composed Actor and Critic Q
2: Define Decomposed Actor(s) and Critic(s) Q
3: for = ......... do
4: for f = 0 to T T-11 do
5: Sample trajectories for T. {ōe, ae, $e+1]
6: Define {sx) with Decomposed Critics (s,s) (Theorems I&
7:
Composed Critic update:
V (Q(m))BB(@.
8:
Compute Bellman-Gre -AE A with (Standard) update Composed Actor Decomposed Critic update(s):
9:
10:
w レ s - 3x V Qi(T)) (Q((1))-710111)) 17(21((1))
11:
Compute Bellman-GAE A with B (Standard) update Decomposed Actor(s)
12:
13:
end for
14:
end for
15:
return parameter 0,
as all will be used in DOHJ-PPO algorithm for either RAA or RR problems. Consider reduction function @ (m) R" + R, defined by
RA
RA (11, T2, F3.
(n-1) x2n+1) = RA (I1, RA (x3...
12m+1)),
(45)
RRA (x, = (1 -17) 7) min {x, y) + + min {y, max (x, 2}}
(46)
The k-step Reach-Avoid Bellman advantage A ARA is then given by, A 1/2 (8) (b ((s))), ... 1)) 1D
(kk)
(((+)))
(47)
We may then define the Reach-Avoid GAE hs as the X-weighted sum over the advantage functions
1
(s)
x
(48)
which may be approximated over any finite trajectory sample. See So et al. (0024) for further details.
E.2 MODIHICATIONS FROM STANDARD PPO
To address the RAA and RR problems, DOHJ-PPO introduces several key modifications to the standard PPO framework Schulman et al. (0177)
Additional actor and critic networks introduced to represent the decomposed objectives. Rather than learning the decomposed objectives separately from the composed objective, DOHI-PPO optimizes all objectives simultaneously. This design choice is motivated by two primary factors: (i) simplicity and minor computational speed-up, and (ii) coupling between the decomposed and composed objectives during learning.
The decomposed trajectories are initialized using states sampled from the composed trajectory, refer to coupled resets.
32

---

Published as a conference paper at ICLR 2026
While it is possible to estimate the decomposed objectives independently- i..., prior to solving the composed task-this approach might lead to inaccurate or irrelevant value estimates in on-policy settings. For example, in the RAA problem, the decomposed objective may prioritize avoiding penalties, while the composed task requires reaching reward region without incurring penalties. In such a case, decomposed policy trained in isolation might converge an optimal strategy within a reward-irrelevant region, misaligned with the overall task. Empirically, we observe that omitting coupled resets causes DOHJ-PPO perform no better than standard baselines such CPPO, whereas their inclusion significantly improves performance.
The special RAA and RR rewards are defined using the decomposed critic values and updated using their corresponding Bellman equations. This procedure is directly derived from our theoretical results (Theorems 1 and 2) which establish the validity of using modified rewards within the respective RA and R Bellman frameworks. These rewards are used to compute the composed critic target as well as the actor's GAE. In Algorithm this process is reflected in the critic and actor updates corresponding the composed objective.
[
F DDQN DEMONSTRATION
As described in the paper, we demonstrate the novel RAA and RR problems in 2D Q-learning problem where the value function may observed easily. We juxtapose these solitons with those of the previously studied RA and R problems which consider more simple objectives. To solve all values, we employ the standard Double-Deep Q learning approach (DDQN) Van Hasselt et al. (2016) with only the special Bellman updates.
F.1 GRIL-WORLD ENVIRONMENT
The environment is taken from Hsu et al. (2021) and consists of two dimensions, s = (x, y), and three actions, {left, straight, right). which allow the agent to maneuver through the space. The deterministic dynamics of the environment are defined by constant upward flow such that,
(I--1: (141))
a; = left
f((2,, y:).
=
(I,, Fi41)
a, = straight
(49)
(1i+1, Vi+1)
0; right
and if the agent reaches the boundary of the space, defined by x 2 [2]. S 2 and y 2 10, the trajectory is terminated. The 2D space is divided into 80 x 120 cells which the agent traverses through.
In the RA and RAA experiments, the reward function is defined as the negative signed-distance function to a box with dimensions (Ies Ve, w. h) = (0, 4.5, 2, 1.5), and thus is negative iff the agent is outside of the box. The penalty function q is defined as the minimum of three (positive) signed distance functions for boxes defined at (Ie; Ve: w, h)) = =5+0775, 3, 1, 1) and (Ies De, w, h)) = (0, 6, 2.5, 1), and thus is positive iff the agent is outside of all boxes.
In the R and RR experiments, one or two rewards are used. In the R experiment, the reward function r is defined as the maximum of two negative segned-distacce function of boxes with dimensions (Z., Ne, w. h) 11125 O, 0.5, 2), and thus is negative iff the agent is outside of both boxes. In the RR experiment, the rewards r1 and r'2 are defined as the negative signed distance functions of same two boxes independently. and thus are positive if the agent is in one box or the other respectively.
F.2 DDQN DETAILS
As per our theoretical results in Theorems I and 2 we may now perform DDQN to solve the RAA
and RR problems with solely the previously studied Bellman updates for the RA Hsu et al. (202l) and R problems Fisac et al. (2019). We compare these solutions with those corresponding to the
RA and R problems without the special RAA and RR targets, and hence solve the previously posed problems. For all experiments, we employ the same adapted algorithm as in Hsu et al. (202I) with modification of the hyper-parameters given in Table T.
33

---

Published as a conference paper at ICLR 2026
Table 1: Hyperparameters for DDQN Grid World
DDQN hyperparameters
Values
Network Architecture
MI.P
Numbers of Hidden Layers
2
Units per Hidden Layer
100, 20
Hidden Layer Activation Function
tanh
Optimizer
Adam
Discount factor
0.9999
Learning rate
le-3
Replay Buffer Size
le5 transitions
Replay Batch Size
100
Train-Collect Interval
10
Max Updates
4c6
G BASELINES
In both RAA and RR problems, we employ Constrained PPO (CPPO) Achiam et al. (2017a) as the major baseline algorithm was not designed to minimize its constraints necessarily but may do so in attempting to satisfy them. As novel direction in RL, few algorithms have been designed optimize max/min accumulated costs and thus CPPO serves as the best proxy. Below we also include a naively decomposed STL algorithm offer some insight into direct approaches optimizing the max/min accumulated reward.
handle secondary objectives which are reformulated constraints. The
G.1 CPPO BASELINES
Although CPPO formulations do not directly consider dual-objective optimization, the secondary objective in RAA (avoid penalty) or overall objective in RR (reach both rewards) may be transformed into constraints to be satisfied of surrogate problem. For the RAA problem, this may be defined as
oc
max ET
Σn (aax)(s})) ESt
S.t.
min (s7) N O.
(50)
For the RR problem, one might propose that the fairest comparison would be to formulate the surrogate problem in the same fashion, with achievement of both costs as a constraint, such that
oo
eSt max ri (ap))((7)) max (2(7)) }
max, E
Σ
min
s.t. min { maxx ri (+7), max max r2(
2 0,
(51)
which we define as variant (CPPO-vI). Empirically, however, we found this formulation to be the poorest by far, perhaps due to the abundance of the non-smooth combinations. We thus also compare with more naive formulations which relax the outer minimizations to summation in the reward
max, Er
Σ r mmxx (17)) + max raxxr2ss)) (2())) E'tt
s.t.
min max (1(57) max r2xx)) 2 0.
r'St
(52)
which we define variant 2 (CPPOv2), and additionally, in the constraint
mi + max m2xx)(($7) max, Ex Σ
s.t. maxri(s7) + max 0, (53)
e'tt
which we define as variant 3 (CPPOv3). This last approach, although naive and seemingly unfair, vastly outperforms the other variants in the RR problem.
G.2 STL BASELINES
In contrast with constrained optimization, one might also incorporate the STL methods, which in the current context simply decompose and optimize the independent objectives. For the RAA problem,
34

---

Published as a conference paper at ICLR 2026
the standard RA solution serves trivial STL baseline since may attempt to continuously attempt to reach the solution while avoiding the obstacle. In the RR case, we define decomposed STL bascline (DSTL) which naively solves both R problems, and selects the one with lower value to achieve first.
H DETAILS OF RAA & RR EXPERIMENTS; HOPPER
Table 2: Hyperparameters for Hopper Learning
Hyperparameters for DOHJ-PPO
Values
Network Architecture
MLP
Units per Hidden Layer
256
Numbers of Hidden Layers
2
Hidden Layer Activation Function
tanh
Entropy coefficient
Linear Decay 1e-2 0
Optimizer
Adam
Discount factor
Linear Anneal 0.995 0.999
GAE lambda parameter
0,95
Clip Ratio
0.2
Actor Learning rate
Linear Decay 3e-4 - 0
Reward/Cost Critic Learning rate
Linear Decay 3e-4 0
Number of Environments
128
Number of Steps
400
Total Timesteps (RAA)
50M
Total Timesteps (RR)
50M
Scan Steps
4
Update Epochs
10
Number of Minibatches
32
Add'l Hyperparameters for CPPO
Kp
1
Kr
le-4
KD
The Hopper environment is taken from Gym Brockman al. (2016) and So al. (2024). In both RAA and RR problems, we define rewards and penalties based on the position of the Hopper head, which we denote as (x,, y) in this section.
In the RAA task, the reward defined as
(54) to incentive the Hopper reach its head to the position at (z. (2, 1.4). The penalty is defined as the minimum of signed distance functions to a ceiling obstacle at (1, wall obstacles at x > and x 0 and floor obstacle at y 0.5. In order safely arrive high reward (and always avoid the obstacles), the Hopper thus must pass under the ceiling and not dive or fall over in the achievement of the target, as is the natural behavior.
r(x y) = VI - 2
1.4 - 0.1
In the RR task, the first reward is defined again as
T1 (z. 3) = v | - + 1.4 0.1
(55)
incentive the Hopper to reach its head the position at (z, y) (2, 1.4), and the second reward as
r2 3) = / x - 0 +
1.4] - 0.1
(56)
to incentive the Hopper reach its head to the position at (z, y) (0, 1.4). In order achieve both rewards, the Hopper must thus hop both forwards and backwards without crashing or diving.
In all experiments, the Hopper is initialized in the default standing posture at random [0, 2] so as to learn a position-agnostic policy. The DOHJ-PPO parameters used to train these problems can be found in Table
35

---

Published as a conference paper at ICLR 2026
1 DETAILS OF RAA& & RR EXPERIMENTS: F
The F16 environment taken from et al. (024), including F16 fighter jet with dimensional observation. The jet is limited to a flight corridor with up 2000 relative position north P2 1200 relative altitude (æ and =500 relative position east (TP
In the RAA task, the reward is defined as
r(z, y) = ZPN 1500
(57)
to incentivize the F16 to fly through the geofence defined by the vertical slice at 1500 relative position north. The penalty is defined the minimum of signed distance functions geofence (wall) obstacles at ZPN > 2000 and |gpEl > 500 and a floor obstacle at æ < 0. order to safely arrive at high reward (and always avoid the obstacles), the F16 thus must fly through the target geofence and then evade crashing into the wall directly in front of it.
In the RR task, the rewards are defined
rI(ZPN, 5
VIZEN - 1250 + + 850] - 30
(58)
and
12(EPN, V
IxPN 1250 1y 350
30
(59)
to incentive the F16 to reach both low and high-altitude horizontal cylinders. order to achieve both rewards, the F16 must thus aggressively pitch, roll and yaw between the targets.
In all experiments, the F16 is initialized with position EPN E [250, 750), R2 E [300, 900), E [-250, 250] and velocity in E [200, 450). Additionally, the roll, pitch, and yaw initialized with 1. /16 16 to simulate a variety of approaches to the flight corridor. Further details can be found in
Soet al. (2024). The DOHJ-PPO parameters used train these problems can be found in Table 3
Table 3: Hyperparameters F16 Learning
Hyperparameters for DOHJ-PPO
Values
Network Architecture
MLP
Units per Hidden Layer
256
Numbers of Hidden Layers
2
Hidden Layer Activation Function
tanh
Entropy coefficient
Linear Decay 1e-2 0
Optimizer
Adam
Discount factor y
Linear Anneal 0.995 - 0.999
GAE lambda parameter
0.95
Clip Ratio
0.2
Actor Learning rate
Linear Decay 1e-3 0
Reward/Cost Critic Learning rate
Linear Decay 1e-3 + 0
Number of Environments
256
Number of Steps
200
Total Timesteps (RAA)
50M
Total Timesteps (RR)
100M
Scan Steps
10
Update Epochs
10
Number of Minibatches
64
Add'l Hyperparameters for CPPO
Kp
1
K
le-4
K
36

---

Published as a conference paper at ICLR 2026
J DETAILS OF RAA & RR EXPERIMENTS; SAFETYGYM
The SafetyGym environment taken from the SafetyGym Point Environment Brockman et al. (2016),
reimplemented in So et al. (2024). In both RAA and RR problems, we define rewards and penalties based on the position of the Point position, which we denote as (z, y) in this section.
In the RAA task, the reward is defined as
r(z,y) = V [x]] + lyd 0.3
(60)
incentive the Point reach to the position penalty y is defined as the minimum of signed distance functions to wall ob- stacles at [x| (.6,, In order to safely arrive at high reward (and always avoid the obstacles), the Point thus must drive through the obstacles and without leaving the box crashing after arriving.
(z.y)
(.). The
and y 2 3/, and a scatter of eight random obstacles 0.2), (-0,,--)))), (-0.3, 0.8), (-1.3, -1.33)
In the RR task, the first reward is defined again as
r1 (v. y) v x 2.5 y/ 2.5 0.3
(61)
to incentive the Point to drive to the position (x, y) (2.5, 2.5). and the second reward as
(2 v) v x + 2.5 + y/ + 2.5 - -0.
(62)
to incentive the Point to drive to the position at (x, y) (-2.5, 225). In order to achieve both rewards, the Point must thus drive both targets.
In all experiments, the Point is initialized at a random (z, v) E [-2, 2) so as to learn position- agnostic policy. The DOHU-PPO parameters used to train these problems be found in Table
Table 4: Hyperparameters for SafetyGym Learning
Hyperparameters for DOHJ-PPO
Values
Network Architecture
MLP
Units per Hidden Layer
256
Numbers of Hidden Layers
2
Hidden Layer Activation Function
tanh
Entropy coefficient Optimizer
Linear Decay 5e-3 - 0
Adam
Discount factor ty
Linear Anneal 0.995 >> 0.9995
GAE lambda parameter
0.95
Clip Ratio
0.2
Actor Learning rate
Linear Decay 3e-4 - 0
Reward/Cost Critic Learning rate
Linear Decay 3e-4 + 0
Number of Environments
128
Number of Steps
400
Total Timesteps (RAA)
50M
Total Timesteps (RR)
100M
Scan Steps
Update Epochs
10
Number of Minibatches
32
Add'l Hyperparameters for CPPO
Kp
K₁
le-4
KD
K DETAILS OF RAA & RR EXPERIMENTS: HALFCHEETAR
The HalfChectah environment is taken from Gym Brockman et al. (016),, reimplemented in So et al. (024).. In both RAA and RR problems, define rewards based the HalfCheetah head position,
37

---

Published as a conference paper at ICLR 2026
which we denote as (x. y) in this section, and define the penalties based the position of all HalfCheetah body positions (back, neck, head, thighs, shins, and feet).
In the RAA task, the reward is defined as
r(z, v)
5.5| 0.25
(63)
to incentive the HalfCheetah reach its head to the cylinder centered at 5.5. The penalty defined as the minimum signed distance functions to a floor obstacles at (0. 4.5) and (0, 6.5) which have height of 0.05 and width 0.5, and back wall obstacle at 00.9. In order safely
arrive at high reward (and always avoid the obstacles), the HalfCheetah thus must jump over the front
floor obstacle (at 4.5) and avoid crashing into, either by hopping over standing, the back floor obstacle (at r = 6.5).
In the RR task, the first reward is defined again as
T1 (xr v) = v Ia
0.1
(64)
to incentive the HalfCheetah to reach its head to the position at (2,3) = (5, 1), and the second reward
as
r2 y)
0.1
(65)
incentive the HalfCheetah to reach its head the position (xr, y) = (-5.1), ) In order to achieve both rewards, the HalfCheetah must thus gallop both forwards and backwards.
In all experiments, the HalfCheetah initialized in the default standing posture at random æ [0.5, 1.5] so as to learn position-agnostic policy. The DOHJ-PPO parameters used to train these problems can be found in Table
Table 5: Hyperparameters for HalfCheetah Learning
Hyperparameters for DOHJ-PPO
Values
Network Architecture
MLP
Units per Hidden Layer
256
Numbers of Hidden Layers
2
Hidden Layer Activation Function
tanh
Entropy coefficient Optimizer
Linear Decay 5e-3 + 0
Adam
Discount factor y
Linear Anneal 0.995 - 0.9995
GAE lambda parameter
0.95
Clip Ratio
0.2
Actor Learning rate
Linear Decay 3e-4 - 0
Reward/Cost Critic Learning rate
Linear Decay 3e-4 + 0
Number of Environments
128
Number of Steps
400
Total Timesteps (RAA)
100M
Total Timesteps (RR)
150M
Scan Steps
Update Epochs
10
Number of Minibatches
32
Add'l Hyperparameters for CPPO
Kp
100
K₁
le-4
Kp
L CONTRAST WITH DECOMPOSITIONS IN TEMPORAL LOGIC
Temporal Logic (TL) predicates, including those for reach (eventually :p), avoid (always not e)), and reach-avoid (not v until (p) problems, enjoy useful algebraic properties. These properties provide convenient way to decompose complex TL. predicates into simpler predicates. Unfortunately, the
38

---

Published as a conference paper at ICLR 2026
decomposition of these predicates does not generally translate to valid decompositions of the optimal value functions in HJR.
More explicitly, it is indeed possible to phrase HJR problem formulations, such as RR and RAA problems, using the language of quantitative semantics from the TL literature. In particular, the standard HJR problem is find the sequence of actions that maximizes some objective function, and this objective function can generally be written as a quantitative semantic corresponding to the specification of the desired system behavior (however, this is not explicitly written in TL notation in the HJR literature). That said, the optimal value function decomposition results for the RR and RAA problems are distinct from the decomposition of the corresponding quantitative semantics. In other words, our decomposition results are statements about the optimal control associated with the quantitative semantic, not the quantitative semantic itself.
This subtle distinction can be clarified by the following Dounter-examplee where valid decomposi- tion the quantitative semantic (on the TL side) for the RR and RAA problem does not translate to valid decomposition of the optimal value functions (on the HJR side) for the RAA problem. We will use F as the eventually operator, as the always operator, and A/V- as logical and/or/not.
L. RAA CASE
Consider RAA problem where my friend is holding piñata which I would like to break with bat. We can always decompose the RAA quantitative semantic into R and quantitative semantics. However, suppose there is some state of the system from which I can either eventually hit the piñata or always avoid hitting my friend, but not both. In this case, the optimal value function for the R and problems will both be non-negative, even though I cannot actually achieve the RAA task from this state.
To make this argument explicit, define the atomic predicates &p and represent hitting the piñata and hitting my friend, respectively:
x t)
} = r (x(()) 2 0, ト 3b p(x(t) 2 0.
xx
t)
Given a predicate μr, let Pa. be the corresponding quantitative semantic. Thus, polx, t] = r(x(t))
and ps p=|xx t] = p(x(()). The quantitative semantics for the R, A, and RAA problems are then, respectively:
pR|x, 4) == pr = max rxx(T)),
(66)
T2t
pa[x, == :B:::=PG-<wx pG-
-min- -p(((T)) 726
(67)
XR E == P(FF)A(G ,)|x, = min
maxr (xax)(((T)), min
min -((((())
(68)
72
721
As mentioned earlier, the optimal value functions for the R, A, and RAA problems then be written in terms of the quantitative semantics, i.e.
(s) = max PR [x 0),
(s(s) = max PA [x 0),
V = max PRAA [x,, 0).
where x is the trajectory of the bat corresponding to the initial state and policy But here is the key point: although it always true that
PRAA 1) = min minn(en(xx.e (ax
in general we only have that
39

---

Published as a conference paper at ICLR 2026
ViA (s) (s)( (X(s))
Indeed this inequality is sometimes strict. Again, consider the case where there is some state of the system from which I either eventually hit the piñata always avoid hitting my friend, but cannot do both. Then
RAA (s) < 0 S min (V))
In other words, the algebra that applies to the TL predicates translates algebra the TL quantitative semantics, but it does not generally translate to an algebra on the optimal value functions. This is why our decomposition required thorough justification.
L.2
RR CASE
Next, suppose that to two downstream islands a wide river that flows from north to south. The islands are at the same latitude, but one is to the west and one is to the east. Suppose the boat starts far enough upstream that can reach either island, but the current too strong to move from one island to the other. Then the boat can satisfy the R task for the west island (by ignoring the east island) and it can satisfy the task for the east island (by ignoring the west island), but cannot satisfy the RR task. The decomposition of the quantitative semanties for the RR problem into the minimum of the quantitative semantics for the two R problems will then not translate to valid decomposition of the value functions.
solving an RR problem for small robotic boat that must deliver supplies
More explicitly, with ri and r2 the signed distance functions either island, let
(x, F = n (x(1)) 2 0, (x,4) () F (x(())
(69)
(70)
Thus, Pep xx| f] (x(()) and p (x(().) The quantitative semantics for reaching the west island, reaching the east island, and reaching both islands are then respectively,
pRI[x, t] = pF t] = max (x(T)),
(71)
720
PR t] pFv(x,t] t| mmxx)(x((T)),
(72)
+2t
p == PLF,)) min ma max 12 (x(+))
(73)
72
12t
As before, the optimal value functions for the two R problems and the RR problems can then be written terms of the quantitative semantics, i.e.
Vii (s) = max PRu [
(74)
(s)) = max |R2 [x 0],
(75)
(R)(s) max PRR X 0J.
(76)
But we still have the analogous issue: although it is always true that
pRR[x, E] min(PRI (x, t), PRZ (x, t]}.
in general we only have that
(5) < min (s), VR2 (s)).
40

---

Published as a conference paper at ICLR 2026
When begin from some state of the system from which can eventually reach the east island or I can eventually reach the west island, but not both, then the inequality is striet:
(R) < & min((Vi(s)) (s)
M BROADER IMPACTS
This paper touches on advancing fundamental methods for Reinforcement Learning, In particular, this work falls into the class of methods designed for Safe Reinforcement Learning. Methods in this class are primarily intended to prevent undesirable behaviors in virtual or cyber-physical systems, such as preventing crashes involving self-driving vehicles potentially even unacceptable speech among chatbots. It is an unfortunate truth that safe learning methods can be repurposed for unintended use cases, such as prevent malicious agent from being captured, but the authors do not foresee the balance of potential beneficial and malicious applications of this method to be any greater than other typical methods in Safe Reinforcement Learning.
N ACKNOWLEDGMENTS
This section has been redacted for the purpose of anonymous review.
41


Reviews:
[{"review_id": "geamxQWIIc", "reviewer": "Reviewer_FUvU", "summary": "This paper investigates two dual-objective satisfactory problems in RL: the reach–reach problem and the reach–always–avoid problem. These problems extend the previously studied reach, avoid, and reach–avoid problems analyzed from the Hamilton–Jacobi perspective of dynamic programming. The authors derive the corresponding Bellman equations for both problems and demonstrate that each can be decomposed into compositions of the previously studied HJ-based reach, avoid, and reach–avoid problems. Furthermore, the paper extends the framework to handle stochastic policies and provides a formulation for implementing it using PPO. Experiments conducted across several environments validate the effectiveness of the proposed method.", "questions": "Please refer to the weaknesses above.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The RR and RAA problems are two interesting dual-objective satisfactory problems, and the paper reveals their unique structures that require specialized treatment.\n\n- The paper presents a solid theoretical analysis by deriving the Bellman equations for the RR and RAA problems and elucidating their connections to the previously studied reach, avoid, and reach–avoid problems.\n\n- Empirical results across multiple environments show that the proposed PPO-based method outperforms baseline approaches.", "weaknesses": "I do not have major concerns about the paper (possibly due to limited familiarity with the related literature). Some minor points are as follows:\n- The paper considers a deterministic transition function. It is unclear whether the proposed method can be extended to the stochastic transition case.\n- For stochastic policies, it remains uncertain whether the proposed PPO-based update can guarantee convergence to an optimal policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AM8luvOdNK", "reviewer": "Reviewer_Cdby", "summary": "The paper defines two dual-objective, extremal value functions in reach-avoid optimal control problems: \"**reach and always avoid**\" (RAA) and \"**reach and reach**\" (RR),  and shows each can be decomposed into known HJ-RL primitives of Reach (R), Avoid (A), Reach-Avoid (RA), yielding explicit Bellman targets and corresponding algorithms that train a composed actor–critic together with additional critics/actors for the sub-objectives.\nThe paper argues history is necessary and introduces a minimal augmentation (running max/min) with a sufficiency result.\nThe proposed value function decompositions was combined with deep Q-learning (DQN) and proximal policy optimization (PPO) and tested in a simple grid-world and modified continuous control environments (Hopper, F16, SafetyGym, and HalfCheetah).", "questions": "Minor issues\n- The introduction employs strong intensifiers (e.g., “performing far more safely”) without quantitative context. This reads like forward-referencing results.\n- Apostrophes for plurals. \"SBE’s and RABE’s\" $\\to$ SBEs and RABEs (no apostrophes for plural acronyms)\n- l.049: user?\n- l.050: \"Figure 1, top middle-left\" is awkward. Consider using subcaptions (e.g., Figure 1a).\n- l.050-051: Consider avoiding F16 and Hopper, because readers may not know them if they are not familiar with these environments.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Originality and significance**: moderate. The problems studied in this paper are classical in logic terms, but the decomposition is novel and practically attractive.\n- **Quality and correctness**: the reductions using min/max identities and distributivity/commutativity under deterministic transition and policy assumptions (relaxed later) seem sound and reasonable for the stated setting. Proofs were not carefully checked.\n- **Clarity**: This paper is clear, easy to follow, and contextualized well. Section 2 related works is comprehensive yet concise, very useful for readers. The most relevant works (e.g., `Fisac et al. (2019)`, `Hsu et al. (2021)`) were cited throughout and explained well. Figure 1 is a bit confusing (Figure 2 is more understandable for me) and dense, but it illustrates the problem setting to some extent.\n- **Compositional formulation**: converting RAA and RR to R, A, and RA can be helpful for compositionality and generalization.", "weaknesses": "## Formal relation to temporal logic\n\nI believe the value function decompositions are based on some algebra/logic implicitly.\nIt would help readers if the paper states the specs more explicitly, potentially using temporal logic, even if the proposed method does not explicitly use automata.\n\nMy feeling is that this paper is just doing quantitative semantics for temporal modal operators $F$ **f**inally, $G$ **g**lobally/always, and $U$ **u**ntil.\n\nLet $r, p, q$ be temporal propositions.\nThen,\n- R is $F r$ \"finally reach r\"\n- A is $G \\lnot p$ \"always avoid (not reach) p\"\n- RA is $\\lnot p U r$ \"avoid p until reach r\"\n- RAA is $F r \\land G \\lnot p$ \"(finally reach r) and (always avoid p)\"\n- RR is $F p \\land F q$ \"(finally reach p) and (finally reach q)\"\n\nThe first three have quantitative semantics:\n- R: $F r \\leftrightarrow \\max_t r_t$\n- A: $G \\lnot p \\leftrightarrow \\min_t -p_t$\n- RA: $\\lnot p U r \\leftrightarrow \\max_t \\min\\\\{r_t, \\min_{t' \\leq t} -p_{t'}\\\\}$\nwhich were shown in Section 4.\n\nIt is possible to prove\n- RAA: $F r \\land G \\lnot p \\equiv (\\lnot p U r) \\land (G \\lnot p)$  = \"(avoid p until reach r) and (always avoid p)\" (?)\n- RR: $F p \\land F q \\equiv F((p \\land F q) \\lor (q \\land F p))$ \"finally ((reach p and finally reach q) or (reach q and finally reach p))\"\n\nThe LHSs have quantitative semantics given in Section 3:\n- RAA: $\\min\\\\{\\max_t r_t, \\min_t -p_t\\\\}$\n- RR: $\\min\\\\{\\max_t p_t, \\max_t q_t\\\\}$\n\nThe RHSs have quantitative semantics given in Section 6. (small gaps)\n\nThe point is, there exists a more principled approach to derive these value functions and Bellman equations than the lengthy *10+ page proofs* in the appendix, which are more error-prone.\n\nThe author stated that \"*While the problems we attempt to solve (e.g. reaching multiple goals) can be thought of as specific instantiations of LTL specifications, our approach to solving these problems is fundamentally different from those in this line of work.*,\" but being \"fundamentally different\" doesn't mean it's better, more efficient, or more generalizable.\nThere's no evidence that LTL cannot express RAA or RR.\nEven if it's true, there alternatives such as metric temporal logic (MTL) or signal temporal logic (STL).\nI'm not an expert in temporal logic, but the concept of **monitoring** is very relevant to the state augmentation issue.\nThe author is encouraged to further connect this paper with the existing logic literature;\nif they decide not to do so, at least justify the \"min/max algebra\" approach by spelling out the axioms and operations (commutativity, associativity, distributivity, idempotence, absorption, monotonicity, etc.).\n\n## Complexity: why RA?\n\nThe second issue is pertinent to the first one.\nThis paper decided to convert RAA and RR in to R, A, and RA to \"leverage new HJ-based methods on the subproblems\", because these HJ-RL \"primitives\" have been previously studied.\nHowever, after scrutinizing Sections 3, 4 and 6, it seems to me this approach unnecessarily complexifies the problem?\nWhy not directly derive the value functions and Bellman equations from the specifications in Section 3?\nEspecially, the use of RA in RAA seems strange.\nWhy not just R and A?\n\n\"Focusing on extremal values rather than discounted sums\", \"directly specifying desired behavior\", and composing task specs, values, and policies is a nice direction.\nHowever, the RL community has developed several methods for general learning objectives `Wang et al. (2020)`, `Cui & Yu (2023)`, `Tang et al. (2025)`.\nPlease compare this work with them, and explain why the proposed method is preferred for the stated tasks.\n\n## MDP without rewards and need for augmented states\n\nFigure 2 and Section 5 might be misleading because each node seems to be considered as a state, and whether a node has been reached isn't part of the state.\n\"Non-augmented\" MDPs/policies are flawed because this paper is actually using a \"**MDP without rewards**\" setting: the MDP $M = (S, A, f)$ is a triple and only consists of the transition function, and the reward function is not on the environment side but designed by the agent. \nAlthough I believe this setting is indeed more natural and should be used, but since the more widely used formulation considers the reward function as a part of MDP, it's incorrect to consider a node as a state, otherwise the reward function is not well defined.\n\n\"Node not reached/with an item\" and \"node reached/without an item\" should be two distinct states;\n\"Reaching an unvisited node/collecting an item\" and \"reaching a previously visited node\" should generate different observations/rewards (e.g., locations + a bunch of flags).\nThe flaw is not due to the limitations of \"memoryless\" policies; it's because of the wrongly chosen states.\n\n## Stochastic reach-avoid\n\nThis paper extended reach-avoid problem formulations using min/max to the stochastic setting in Section 7.1.\nHowever, the value function and Bellman equation (SRABE) were given without sufficient justification.\n\nCaveat: *expected maximum* and *maximum expectation* are two different objectives, and consequently their value functions are also different.\nOne may argue that both are meaningful objectives, but the corresponding value functions and Bellman equations should not be *defined* but *derived* from the objective.\nThe error of interchanging max and expectation in `Quah & Quek (2006)` was noticed by `Gottipati et al. (2020)` and `Cui & Yu (2023)` and some fixes were proposed.\nThe author is encouraged to explicitly state the learning objective in Section 5 like what they did in Section 3 and confirm the Bellman equation truly corresponds to this objective.\n\n## References\n\n- Kian Hong Quah and Chai Quek. **Maximum reward reinforcement learning: A non-cumulative reward criterion.** Expert Systems with Applications, 31(2):351–359, 2006. https://doi.org/10.1016/j.eswa.2005.09.054.\n- Sai Krishna Gottipati, Yashaswi Pathak, Rohan Nuttall, Raviteja Chunduru, Ahmed Touati, Sriram Ganapathi Subramanian, Matthew E Taylor, and Sarath Chandar. **Maximum reward formulation in reinforcement learning.** arXiv preprint, 2020. https://arxiv.org/abs/2010.03744.\n- Ruosong Wang, Peilin Zhong, Simon S Du, Russ R Salakhutdinov, and Lin Yang. **Planning with general objective functions: Going beyond total rewards.** In Neural Information Processing Systems, 2020. https://proceedings.neurips.cc/paper/2020/hash/a6a767bbb2e3513233f942e0ff24272c-Abstract.html\n- Wei Cui and Wei Yu. **Reinforcement learning with non-cumulative objective.** IEEE Transactions on Machine Learning in Communications and Networking, 1:124–137, 2023. https://doi.org/10.1109/TMLCN.2023.3285543.\n- Yuting Tang, Yivan Zhang, Johannes Ackermann, Yu-Jie Zhang, Soichiro Nishimori, Masashi Sugiyama. **Recursive Reward Aggregation.** In Reinforcement Learning Conference, 2025. https://openreview.net/forum?id=13lUcKpWy8", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8XSqI3ARGC", "reviewer": "Reviewer_BuMr", "summary": "The paper extends Hamilton–Jacobi-style RL beyond the classic Reach (R), Avoid (A), and Reach–Avoid (RA) objectives to two harder, compositional objectives: Reach–Always–Avoid (RAA) (reach a goal and keep avoiding hazards forever afterward) and Reach–Reach (RR) (reach two distinct goals, in any order). The key theoretical result is that both RAA and RR can be decomposed into already-solved HJ-RL subproblems (avoid + reach–avoid for RAA; three reach problems for RR), yielding explicit Bellman forms (Thm. 1–2). Because optimal actions depend on history (e.g. “have I already reached goal 1?”), the paper introduces a specific state augmentation that tracks max-reward and min-penalty over time and shows this augmentation is sufficient (Thm. 3). On top of this, they build DO-HJ-PPO, an on-policy method that learns the decomposed critics alongside the composed RAA/RR critic and couples rollouts so subproblems stay relevant. Experiments on gridworld, Hopper, F-16, SafetyGym, and HalfCheetah show higher “both-objectives achieved” rates and faster completion than Lagrangian and prior HJ-RL baselines.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The decompositions are stated cleanly, proved in the appendix, and tied directly to implementable Bellman operators; the need for history and the exact augmentation are justified, not hand-waved.\n2. Many safe / task-spec RL problems really are “reach X and stay safe” or “reach X then Y”; having explicit Bellman forms and a working PPO variant for those is useful, especially since baselines mostly get only “partial success.”", "weaknesses": "1. Assumptions are narrow: Main theory is for deterministic, finite MDPs, yet real tasks are stochastic/continuous, but the paper only gives a heuristic stochastic variant (SRABE) without matching guarantees. A short discussion of what breaks in stochastic dynamics is needed.\n2. State augmentation cost: The proposed augmentation grows the state with running max/min signals; for higher-dim tasks (real robots, multi-goal specs) this could be heavy, and the paper doesn’t study scalability.\n3. The paper lacks sample-complexity or contraction analysis for the composed operators. Prior HJ-RL work leans on discounted versions to get contraction, and here the composed PPO version is justified mostly by analogy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8CKbK80cwD", "reviewer": "Reviewer_jprC", "summary": "This paper proposes a novel RL framework that integrates Hamilton-Jacobi equations from optimal control theory to handle dual-objective tasks involving safety and goal achievement. \nThe authors introduce two new value formulations: Reach-Always-Avoid, which requires reaching a goal while maintaining safety thereafter, and Reach-Reach, which requires achieving two distinct goals sequentially. \nThese value functions are defined via min-max extremal operators, enabling direct reasoning about best- and worst-case outcomes.\nThe authors show that the Reach-Always-Avoid and Reach-Reach problems can be decomposed into simpler subproblems (Reach, Avoid, and Reach-Avoid), each corresponding to known HJ-Bellman equations. \nTo make these formulations tractable, they augment the MDP state with historical extrema to restore the Markov property and propose an RL algorithm DO-HJ-PPO, a variant of PPO that jointly learns decomposed and composed actor-critic pairs. \nThe algorithm employs a Reach-Avoid GAE to approximate the extremal Bellman differences.\nExperiments on discrete and continuous control tasks demonstrate that DO-HJ-PPO outperforms baselines.", "questions": "1. The paper extends the HJ-Bellman framework to stochastic and learned environments, but most derivations assume deterministic dynamics. How robust is the theoretical foundation when applied to stochastic MDPs?\n\n2. The RAA and RR formulations are proven decomposable, but the algorithm jointly learns all sub-critics with bootstrapped targets.\nIs there any theoretical guarantee that this joint training converges to the same fixed point as the original HJ decomposition?\n\n3. The continuous-control tasks use simplified geometric rewards.\nWould DO-HJ-PPO still outperform baselines if trained with the dense, physically meaningful rewards used in standard MuJoCo benchmarks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper builds on HJ optimal control theory and connects it elegantly to RL. The proposed formulations (Reach-Always-Avoid and Reach-Reach) are mathematically rigorous and clearly grounded in control-theoretic Bellman principles.\n\n2. The theoretical decomposition of Reach-Always-Avoid and Reach-Reach problems into subproblems is clean and provable, allowing modular reuse of existing HJ-RL methods.\n\n3. By directly modeling safety and goal satisfaction in the value function, the approach avoids the delicate tuning required by Lagrangian or constrained RL methods.\n\n4. Experiments span both discrete and continuous control tasks. The method consistently outperforms strong baselines, showing robustness and good generalization.", "weaknesses": "1. While the theoretical model is elegant, many experiments (e.g., Hopper-RAA, Hopper-RR) use geometrically simple rewards that do not fully reflect the difficulty of real-world control problems. Thus, the results may overstate performance in realistic scenarios.\n\n2. The algorithm involves multiple coupled actor-critic networks (for decomposed and composed objectives), which can significantly increase computational and memory costs. The paper lacks ablation studies or discussion on scalability or efficiency.\n\n3. The baselines (e.g., CPPO, RESPO) are standard but may not be fully optimized under the same hyperparameter or reward conditions, making the fairness of some comparisons uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)