Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learning
Authors: Oswin So
Eric Yang Yu
Songyuan Zhang
Matthew Cleaveland
Mitchell Black
Chuchu Fan
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Optimal Control
Deep Reachability
Primary area: reinforcement learning
TL;DR: We propose Feasibility-Guided Exploration (FGE), a method that simultaneously identifies safe initial conditions and solves the optimal control problem over these conditions using reinforcement learning.

Abstract: Recent advances in deep reinforcement learning (RL) have achieved strong results on high-dimensional control tasks, but applying RL to reachability problems raises a fundamental mismatch: reachability seeks to maximize the set of states from which a system remains safe indefinitely, while RL optimizes expected returns over a user-specified distribution. This mismatch can result in policies that perform poorly on low-probability states that are still within the safe set. A natural alternative is to frame the problem as a robust optimization over a set of initial conditions that specify the initial state, dynamics and safe set, but whether this problem has a solution depends on the feasibility of the specified set, which is unknown a priori. We propose Feasibility-Guided Exploration (FGE), a method that simultaneously identifies a subset of feasible initial conditions under which a safe policy exists, and learns a policy to solve the reachability problem over this set of initial conditions. Empirical results demonstrate that FGE learns policies with over 50% more coverage than the best existing method for challenging initial conditions across tasks in the MuJoCo simulator and the Kinetix simulator with pixel observations.

Published as a conference paper at ICLR 2026
SOLVING PARAMETER-ROBUST AVOID PROBLEMS WITH UNKNOWN FEASIBILITY USING REINFORCEMENT LEARNING
Oswin So Eric Yang Yu"1 Songyuan Zhang'
Matthew Cleaveland Mitchell Black²
Chuchu Fan-
DDpartment of Acronautics and Astronautics, MIT 2MIT Lincoln Laboratory
(oswinse, eyyu) } @mit.edu
ABSTRACT
arXiv:2602.15817 11881. [cs.LG] 17 Feb 2026
Recent advances in deep reinforcement learning (RL) have achieved strong results on high-dimensional control tasks, but applying RL to optimal safe controller synthesis raises a fundamental mismatch: optimal safe controller synthesis seeks to maximize the set of states from which a system remains safe indefinitely, while RL optimizes expected returns yield policies that perform poorly on low-probability states still within the safe set. A natural alternative is to frame the problem as a robust optimization over of initial conditions that specify the initial state, dynamics and safe set, but whether this problem has solution depends on the feasibility of the specified set, which is unknown priori. We propose Feasibility-Guide Exploration (FGE), method that simultaneously identifies subset of feasible initial conditions under which a safe policy exists, and learns policy to solve the optimal control problem over this set of initial conditions. Empirical results demonstrate that FGE learns policies with over 50% more coverage than the best existing method for challenging initial conditions across tasks in the MuJoCo simulator. The project page can be found at https://oswinsoxxy//feg
a user-specified distribution. This mismatch
INTRODUCTION
Hamilton-Jacobi (HJ)-based optimal control analysis a powerful tool for obtaining control policies and reasoning about the largest set of initial conditions from which a dynamical system can satisfy safety constraints for all time (Lygeros et al. 1998). However, classical methods for solving HJ-based optimal control analysis use grid- solvers that require state discretization and hence suffer from the curse of dimensionality, rendering them intractable for systems with high-dimensional nonlinear dynamics (Mitchell et al. 2005). Recent advances in deep reinforcement learning (RL) have demonstrated remarkable success in approximating optimal control solutions in high dimensions
(Fisac al. 2019; Hsu et al.) 202 1 Bansal & Tomlin [021] Ganai et al., 2023 Hsu et al., [023]
Nguyen et al. 2024 So et al. 2024: Ganai et al.] 2024). these approaches, one typically poses an
optimization problem whose exact solution is the optimal control solution for single initial condition, then uses RL to learn a policy that solves this problem over some user-defined distribution of initial conditions. However, this sampling-based formulation introduces fundamental misalignment: the RL objective minimizes expected cost under a potentially narrow initial distribution rather than worst-case safety over the entire set of interest. As result, learned policies may perform well frequently sampled states according to the user-specified distribution, but fail catastrophically in low-density regions that are still within the operating region and could be made safe.
A natural remedy is to recast the HJ-based optimal control problem as a robust optimization problem: Given a set of initial conditions of interest in the form of set of initial states (and more generally, a set of parameters that specify the initial state, dynamics and safcty specification), instead of arbitrarily introducing a distribution over this set and taking an expectation, one minimizes the cost under the worst-case initial condition within this set (Pinto et al., 2017). This perspective aligns directly with the original goal of worst-case safety. However, it hinges on being able to specify a candidate set of initial conditions that are feasible (i.e., can be made safe), which in itself is nontrivial task, finding
"These authors contributed equally to this work. DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.

---

Published as a conference paper at ICLR 2026
Base
Explore
Best Response
Rehearsal
FGE
1.0
Dersiy
Mallor of
Cersity
3.5
Figure I: FGE modifies the parameter sampling distribution. FGE modifies the base distribution over the parameters by introducing an explore distribution to improve policy performance parameters that have not been observed to be safe so far, and a rehearsal distribution obtained via sampling-based based approximate best response to train poorly performing parameters that were previously solved. We combine all three distributions to obtain the final distribution for FGE that balances the objectives of maximizing safety rate gain and minimizing safety rate loss.
the set of feasible initial conditions is precisely the goal of HJ-based optimal control analysis and computationally infeasible to solve exactly (Mitchell et al., 2007). An infeasible robust problem renders all policies equally optimal (as all yield the same worst-case objective), which does not solve the original safety problem.
In this paper, we address this gap by introducing the parameter-robus avoid problem with unknown feasibility. Our key idea is to simultaneously (i) identify large subset parameters where safety is achievable, and (ii) learn policy satisfying the safety constraints on this identified subset (Fig. T
Our main contributions are as follows:
We formalize the novel robust safe optimal control objective that jointly selects a maximal feasible parameter region and safe policy.
We propose practical framework that interleaves (a) feasibility-guidee set expansion via constraint- driven exploration and (b) robust policy optimization over the current feasible set using techniques from saddle-point finding and online learning.
On suite of high-dimensional avoid problems in the MuJoCo simulator, our approach learns policies that yield significantly larger safe sets than prior methods.
2 RELATED WORK
Saddle-point Problems Solving a robust safe optimal control problem requires the solution of minimax optimization problem, which results in saddle-point problem the equivalent game has Nash equilibrium (Orabona 2019). Recent works have shown connections between saddle-poin problems and online learning when players play with sublinear regret (Orabonal 2019). These methods have been used in RL to solve for game theoretic equilibria in large-scale games such as poker (Brown et al., 2019) and Diplomacy (Gray et al.) 2020). The parameter-robust avoid problem with unknown feasibility differs from existing works in saddle-pon point finding in that the domain for one of the players is an optimization variable and is constrained to be subset of the feasible space that is not known a priori and must be solved for as well.
Robust Reinforcement Learning A significant line of research has sought to improve the robustness of policies learned via reinforcement learning. Domain randomization (Tobin al. (017). where policies are trained on randomized environments, is a popular approach due to its simplicity and effectiveness. Risk-aware RL methods that try to optimize over tail distributions of uncertainties (Yang et al. [2221] Stachowicz & Levine, 2024) are another approach to improving robustness. Most directly related is RARL (Pinto et al., 2017), which solves a robust RL problem by training an adversarial disturbance policy. This implicitly assumes feasibility under the worst-case disturbance. Our approach does not assume feasibility under the worst-case. Instead, we learn policy that robust the largest possible subset of the parameter space.
Curriculum Learning and Unsupervised Environment Design Our method of adjusting the initial state distribution to maximize the safe set is reminiscent of curriculum learning in RL, which also adjusts the initial state distribution, albeit to improve learning efficiency (Narvekar et al. 2020). Methods such VDS (Zhang et al.) (2200b) and PLR (Jiang et al. (0211) apply different heuristics to prioritize initial states that can provide more informative learning signals. Closely related is unsupervised environment design (UED), which tackles producing distribution over environments
2

---

Published as a conference paper at ICLR 2026
that "supports the continued learning of a partieular policy" Dennis et al.
2020 Many UED
methods choose environments that maximize regret, differing in how regret is estimated and how the selection occurs (Dennis et al. 2020 Jiang et al.) 202 Lanier et al. 2022 "arker-Holdee et al. 2022, Rutherford et al. 2024). While our method also focuses on the initial state distribution, we specifiially tackle parameter- robust avoid problems rather than general RI. tasks, enabling us to better exploit structure, such as with feasibility classifier.
Safe Reinforcement Learning There is large body of work on safe RL that adds safety constraints to the original unconstrained MDP objective, whether in the form of constrained MDPs (Altman 1999) or almost-sure (Sootla et al., 2022) or state-wise (Wachi al.) 2018) safety. This results in a constrained optimization problem where the goal is to optimize the original task objective while satisfying safety constraint that often conflicts with the task objective. The main challenge in safe RL often lies in how to generalize RI. algorithms that perform unconstrained optimization to the constrained setting, and is usually solved with techniques from constrained optimization such as primal-dual optimization with Lagrange multipliers (Ray et al. 2019 Stooke et al. 2020 Li et al. 202T), interior-point methods (Liu et al. 2020), projection based methods (Yang et al. 2020; Xu al. ).21), epigraph reformulations (So & Fan 2023; Zhang al. 2025). or penalty method (Sootla et al. 2022; Tasse et al.; 2023). Our work differs in that we only have a safety objective and no
task objective and focus on maximizing the set of parameters that are safe. Consequently, the usual
concerns in safe RI. of balancing task performance and safety do not arise in our setting.
3 PROBLEM FORMULATION
We consider deterministic discrete-timee dynamics with parameters @ e
8 fo(sk, a), 80 = so()).
(1)
where 8% S R the state time step N2o, A S R the control, fo : S is the dynamics function under 0, Su θ - S maps from the parameter space to the initial state, and S, A denote state and action spaces respectively.
We first introduce the parameter-robust avoid problem, which we illustrate with an example. Consider the case of autonomous driving, where we wish to find a safe policy. Here, safety means avoiding entering unsafe states such as collisions or losing grip when the road is wet or icy. However, instead of only considering a single weather condition, we wish for the policy to be safe for all weather conditions, including the most adverse ones such as heavy rain snowfall. Mathematically, we want to find a policy To E M == { To S x e A) } such that for all weather conditions (i.e., parameters) O O, including the most unsafe ones, the system starting from state sf) remains outside a set of unsafe (failure) states Fo S S for all time I Compared to the standard HJ avoid problem Mitchell et al. 2005), this allows the dynamics fo and the unsafe set Fo vary with 0.
Let hg S > R be a safety function and the unsafe set Fo == {s ho(s) > 0} be its strict zero- superlevel set, i.e., the set of all unsafe states under hg. Our problem can be transformed into the standard avoid problem using an augmented state space $ = S x Q where of the augmented dynamics is fixed. Then, our problem is equivalent to the following HJ optimal safe control problem:
Vreach (8: 0) == min max he (sx). s.t. 8k+1 (sk: Te (sx)))
my E.M K20
where the zero-sublevel set of the safety value function Vreach 5 - R contains all state- nate-parameter pairs that are feasible, i.c., there exists a policy that renders the system safe for all time (Mitchell ict al. 2005). Let S, := { (O) 0) | 0 E θ denote the set of all initial state-parameter pairs. all such initial state-paramest the zero-sublevel set Sute == { (so 0) Vreach (So; 6) < 0 }. then the original parameter-roie rrobuss avoid problem is feasible and the minimizer T of (2) also solves the original problem (Section A. Note that using state augmentation allows us to treat both initial states and parameters as initial augmented states without loss of generality.
pairs are feasible according to the value function Vreach. i.e., So C Sunte for
FFquivleenll, this is contextual MDP Hallla al. 2015 with deterministic dynamics and an almost-sure safety RI. objective (Sootla et al. 2022) where the unsafe states should be avoided with probability 1. NNot that this is not the expected cumulative sum of rewards and hence is not the traditional RL. value function. However, does satisfy similar properties such the safety Bellman equation (Fisac al. 2019.
3

---

Published as a conference paper at ICLR 2026
Although some existing works (Fisac et al. 2019) directly solve ( enables the use existing RL algorithms and avoids the need to come up with new optimizers for
different approach that
the max-over-time objective (Veviurko et al., 2024 Let To.so min { k 2 0 ho(s)) > 0 } be the
first hitting time of Fo. and define the value function
{0.1] as
To.
((0: 0) max (TT,, 0), ((TO,0) 0)
=
Σ K=
[hh((s)) > 0}, 8k-1 fo(, TT(()))
Then, the zero-sublevel sets of Veach is equivalent to the zero level set of V (see Section A.2 and solving one solves for the other. Now, the sum-over-time objective (3) matches the objective used in standard Markov Decision Processes (MDPs) and can be solved using off-the-shelf methods.
Standard deep RL assumes the initial state is drawn from a distribution p. but we need to solve V for all initial states (8o. 0) E Sg. A naive choice such uniform p over Sy can be problematic. For example, in autonomous driving, staying safe on icy roads at high speeds is harder than under normal conditions; if weather sampled uniformly, an RL algorithm will learn more slowly and thus perform worse in snowstorms than sunny days. We illustrate this on toy MDP in Section B
One idea to instead solve robust problem over the state space in a similar spirit to a static version of robust reinforcement learning (Morimoto & Dova 2005 Pinto et al. (017). i.e. Parameter Space
Phase Space
To.ep
max M. mingge
(ho > 0}, (4)
1.5
Avo
k=
0.2
where adversary learns the worst-case param- eter for the current policy during training, and the policy æ always plays against its most chal- lenging parameter. However, if some 9 makes the objective nonzero for every Te, then every Ty attains the same suboptimal value of 1. In driving task, for example, the adversary could choose an extreme snowstorm and a high initial velocity that guarantees loss of traction. Since no policy can avoid failure in this setting, the op- timal response is effectively to do nothing. Yet, training only such impossible scenarios pre- vents the policy from learning to handle easier, feasible parameters (e.g.. good weather).
0.5
0.9
0.5
0.0
-1
0.5
1.0
4p
5
Figure 2: Adaptive Cruise Control Example. We illustrate an example using an autonomous driv- ing scenario. The parameters 0 [ā, mo] model varying max acceleration ā (e.g.. from weather) and initial relative velocity Ang- We want avoid crashing into the car in front (Ap S 0). (Left) We plot trajectories for different and (Right) visual- ize the feasible parameter set O in green.
This issue can be solved by restricting the maximization to a set of feasible parameters O' O, but this set is not known a priori. We thus propose the sarameter-robust avoid problem with unknown feasibility as the following constrained optimization problem
To.eo
max O' s.t. max min erce
((ho(s)) > 0} 0, su su((), sk+1 fo(ss. Te(s))) (5)
T. Hee!
Unlike the robust setting, we relax the (potentially infeasible) requirement that the entire parameter space feasible and instead look for the largest feasible subset O C O. We illustrate this using an autonomous driving example (Fig. 2}. where the parameters 0 [ā, vo] control the max acceleration a (e.g., from weather) and initial relative velocity Avo. and safety is defined as not crashing into the in front (Ap I 0). Only subset of the full parameter space is feasible, and solving (5) equates to identifying this subset and finding a policy that is safe under such feasible parameters. In the next section, we propose practical algorithm to solve (5) that simultaneously identifies the feasible set O* and learns a policy π that solves the safe optimal control problem over this set.
4 FEASIBILITY-GUDDE EXPLORATION
In this section, we discuss three key challenges in solving the parameter-robust avoid problem with unknown feasibility (5) and how we propose to address them.
4

---

Published as a conference paper at ICLR 2026
Positive labels only
1.00
xnollous
Ours
9,16- 1|8)
Probabilitty
0.75
in GB)
q 188)
1.0
Leamned class sa 1(0)
0.5
0.50
Pinne (4)
0.25
X" 1103
0.0
0.00
0.0
0.2
0.4
0.6
0.8
0.2
0.4 0.6
0.8
0.2
0.4
0.6
0.5
0
Figure 3: (Left) We only have positive labels for feasibility from the set of parameters that were measured to be safe Dj. Since we don't have negative labels, standard supervised learning cannot be used. (Center) We address this by defining a target distribution Pmix via mixing positive labels from PD, (0) with noisy labels from p", controlled by a. (Right) Approximating Pmix (f 1 with 2o (f ( using rollout samples and Dj samples yields feasibility classifier with no false positives and a controllable false-negative rate with a and (Theorem
4.1 ESTIMATING THE FEASIBLE PARAMETER SET O*
We first tackle the challenge of estimating a feasible set of parameters O' C Or. This important: if O' includes infeasible parameters, the robust constraint in (5) becomes unsatisfiable and the policy degenerates. However, feasibility labels are asymmetric: under deterministic dynamics, any parameter observed to be safe is feasible, while an unsafe observation may simply reflect suboptimal policy, Because only positive labels are reliable, standard supervised learning cannot be applied.
Our goal is to learn conservative classifier: it should never mark an infeasible parameter as feasible (zero false positives), while keeping false negatives low. To do this, we exploit the label asymmetry by training on two data sources: (i) parameters observed to be safe, which provide trustworthy positive labels, and (ii) on-policy samples, which may include false negatives but reveal the boundary of o Mixing these sources yields a classifier that stays conservative despite noisy negative labels. We do this using a data distribution Pmix that upweights reliable positives and downweights potentially noisy negatives. As shown next, this yields classifier with zero false-positives and controllable false-negative rate.
Formally, let 1 denote the event that the system remains safe for all time, and let =f 1(0 E O*} } be true probability that 0 is feasible. We approximate O by the set of empirically safe parameters D-- { Be ( 0) }} C O., and detine PD, (0) as the uniform distribution over D,. Let p² (f 1 0) be the safety probability of A under policy æ, and let p(t) be an arbitrary sampling distribution over O For a mixing coefficient a E (0, 1), we form the mixture (see Fig.
Pmix (f. 0) a (f 0)PD, (0) + (1 aapp (f ())()).
(6)
which combines positive samples from Dj and samples with noisy labels from p. We approximate Pmix with go θ - [0, 1] by solving the following variational inference problem
min DKL (Pmix(f. (f, || qv (f(0) (m)x(()) (0))
whose optimal solution is the conditional
Puix (f = ap'(f (f ())p,(()) (0) + (1 (1- (f()0())) ())(
(8)
app, (0) + (1 (()
Thresholding yields the classifier s00) = (f
0) 2 B} for 3 E (0, 1). If the parameter 0 E c is infeasible, then (f = 1|Ø) O, ensuring perfect classification of infeasible parameters. If the parameter E DD C O* is measured to be feasible, then
== 1 - p(0) E Dj Puix (f = 110) -1-(111'(1---1111111 (1-p"(f=110)))
(9)
a (D)
so Ç controls the attemuation of false negatives (derivation in Section A.6p. When is optimal, p (f 1 1 0) = 1 and the error vanishes. Even for suboptimal ग,, we can reduce < either by taking a + 1 (making Punx approach p²) or by choosing & to have have low density on O*. Theorem formalizes conditions under which qy- is accurate on O". Moreover, by choosing p = qu (flj 0), we can provide provide guarantees classifier accuracy that are independent of the policy (Proposition B
In practice, we take p to be the rollout distribution of under म. so that pT ())()) corresponds to the joint distribution induced by on-policy rollouts. Since we minimize the forward KI., samples
5

---

Published as a conference paper at ICLR 2026
from Pmix suffice to optimize the objective; these are obtained simply by mixing on on-policy samples with positive examples from Dj-
Remark 1 (Comparison Density Modeling Approaches). Prior work models feasibility by esti-
mating a density over Dj (Kang et al. [022] Castaneda et al. (2023), and require choosing a density
threshold which is less principled; we compare with density methods in Section 5.2
4.2 FEASIBLE ROBUST OPTIMIZATION WITH SADDLE-POIN FINDING
With an estimate of the feasible parameter set O* from Section we now address the robust optimization problem in (5). While RARL (Pinto et all 2017) can be used to solve this maximin problem, its Gaussian policy adversary makes it difficult to constraint its output to feasible parameters, and such adversarial dynamics are often unstable in practice (Zhang et al. 2020a
Instead, we adopt saddle-point methods from online learning (Orabonal 2019). which allow the adversary to be explicitly restricted to O and provide theoretical convergence guarantees under suitable conditions. We retain the alter- nating structure between policy updates and se- lecting worst-case parameters, but compute the adversary's s action via best-response over the ap- proximated O Sublinear regret for the policy is needed for convergence, so we use follow- hhe-regularizee-leadee (FTRL) with quadratic regularizer vb which keeps successive policies close (Orabona 2019), stabilizing the policy up- dates and avoiding oscillations as the worst-case 0 changes. A quick comparison with gradient descent ascent (GDA, Benaam & Hirsch 1999),
0.4
PGF
0.2
-02-
-0.4
0.0
1000
Iteration
Figure 4: (GDA) vs FGE on a Bilinear Game. On max minge[-1.1] n for n, GDA fails to converge in last-iterate, while FGE using (12) converges the saddle point at the origin. Both converge in average iterate (see Section D. We plot the average for FGE.
Gradient Descent Ascent
whose iterates RARL approximates with Deep RL, shows the stability from FTRL improves convergence (Fig. 4.. The iterates then take the form
1 == arg min (T
(10)
T++1 == arggaa max whe (
(.)
(11)
Then, arguments from (Orabona 2019 Ch. 12) show that 111) has werage-iterate convergence to the saddle-point under suitable assumptions (see Section C). We use deep RL. to approximate (TT) by
treating the sum as expectation and linearizing J around to obtain (see Section A.5 for details):
Te+1 = + 1/6 V
Σ 1=1
J(T 02) = + n V 0)]. Do. = Uniform((0.V.... (12)
where Do., a. is the discrete uniform distribution over the set of maximizers (the rehearsal buffer). Using techniques from (Farina al. 2020). a Monte-Carks approximation of the expectation in (12) still results in sublinear regret with high probability for π and hence still converges to the saddle-poim with high probability. Since (12) performs gradient ascent J, implement it policy update step and apply any on-policy RI algorithm, such as PPO, to the equivalent RI. problem.
To.e0
(13)
Tr-1 arg max Ea-Do.
Σ
(hb( > 03}. 0c+1 arg min J o(0),, 000) E ((0)
We use the negative indicator as the reward function. Since the exact minimizer is intractable, we sample a batch of 0 ~ p(- 0 E O*) use neural network estimator of to select an approximate worst-case f141. While the assumptions required for theoretical convergence such convex-concavit and access to the exact best-response oracles may not be satisfied for the problems we consider, we still observe empirical benefits like improved stability, as shown in our results. These theoretical results provide insight on why our method works well empirically, but theoretically proving convergence is not feasible under current mathematical frameworks and is still an open challenge.
4.3 EXPANDING THE FEASIBLE SET ESTIMATE
remaining challenge is how to expand the measured feasible set Dj so that it better approximates the true feasible region O*. We can only control false negatives for parameters within Dj, and Dj

---

Published as a conference paper at ICLR 2026
Base Distribution
Exploration
Robust RL
Feasibility Learning
P(B)
Reset Distribution
Explorn Distributioo
Oir-Pllccy
Add sefe
Dataset
Rejoction Sample
PPO
using,
Rollout Labels
Explore
Rehearsal
Base
Safe
Buffer of warst-case
parameters
PT (8) 1.0
Approximate Best- Response
Policy Classilier (f ())
Pmix
arginin py (1-1 (0) P
Sampled Approx Best Best-Sesporice
Feasibility Classilier
0.5 Train By (f (8) to match
Pmix 6)
0.0
Figure 5: Feasibility-Guider Exploration (FGE). Starting from on-policy RL algorithm, adapt the initial state distribution as a mixture of the base (M), exploration (a) and rehearsal (*) distributions. The exploration component expands the feasible set and the rehearsal buffer targets feasible parameters that the current policy underperforms. After each episode, newly discovered feasible parameters are added the dataset D,. A mixture of samples from Dj and the episode (x.x) forms Pmix (j. ()) used to train the feasibility classifier qg- The fcasibility classifier qy guides rejection sampling for the explore distribution. We also train a policy-conditione classifier p" to predict x' 's performance given 0 enabling approximate best-response selection of worst-case feasible (*) for rehearsal. We use the ACC example and plot parameter space all plots (see Fig.
only expands when the policy is safe for parameters during rollouts. This creates negative feedback loop: a weak policy fails many parameters, keeping D, small, which in turn restricts the training distribution and prevents improvement on harder parameters.
To break this cycle, we introduce an explicit exploration mechanism that proactively seeks parameters whose feasibility is still uncertain. Using the feasibility classifier Φ, we draw candidate parameters that are currently estimated to be infeasible by rejection sampling 0 ~ p(- 6(0) = 0). This encourages the policy improve on parameters outside the known feasible region, allowing the algorithm to discover new safe parameters and expand DD and thus improve the classifier over time.
4.4 FEASIBILITY-GUDEE) EXPLORATION
We present Feasibility- Guided Exploration (FGE; Fig. Algorithm 2
Each iteration, update the policy using 113 instantiated with PPO, to solve the parameter-rohust avoid subproblem under the current feasible set estimate θ*. During on-policy rollouts, we augment the rehearsal buffer Dø, by mixing in the exploration distribution (Section 4.33) and the base distribution. The resulting mixture preserves the base distribution's 's support while concentrating density on difficult yet feasible parameters (from the rehearsal buffer) and on currently infeasible ones (from the explore distribution). Next. we perform the best-response step in (13), where we use a learned policy classifier instead of J (Section D.3.2). Finally, we train the feasible classifier by optimizing qy to match data drawn from both the measured feasible set Dj and on-policy rollouts generated by the current policy π.
EXPERIMENTS
We compare FGE against baselines from the following categories. Robust RL: Domain randomization (DR) (Tobin et al., 2017). ,, the most popular method for robustifying RL policies. We also compare against RARL (Pinto al. 2017) representative work robust RL. Curriculum Learning:
7

---

Published as a conference paper at ICLR 2026
ToyLevels
Dubins
Hopper
HalfCheetah
FixedWing
Figure 6: Benchmark Tasks. Unsafe regions to be avoided are shown in red for cach environment.
Fraction of with
rate ((1)
1.0
FGE
Safety Rate (1)
Coverage Gain va DR (f)
Coverage Loss vs DR
VDS
PLR
VDS
PAIRED
PLR.
RARI
0.2
FARR
PARED
FABR
0.0
SPL
ACCEL
0.8
0.9
SF1
Safety Rate (T)
0.75
(a) Performance profile
(b) IQM over all tasks
Figure 7: FGE has the highest safety rates. (Left) Performance profile (Agarwal et al. 2021 At every threshold, FGE has the most runs with higher safety rate than the threshold, dominating all other baseline methods. (Right) IQM over all tasks. FGE achieves the highest safety rate by rendering the most hard parameters safe without losing coverage on easy parameters.
VDS (Zhang et al. 2020b) and PLR (Jiang et al. 2021b) are two curriculum learning methods that do not require expert knowledge in designing curriculum. UED: PAIRED (Dennis al. 22020). FARR (Lanier et al. 2022), ACCEL (Parker-Holder et al. 2022) and SFL (Rutherford er al. 2024) all recent methods from UED that have been shown outperform DR. All RI. algorithms
instantiated with the PPO algorithm (Schulman et al. (0117) and optimize for (33) but with a different distribution over the parameters 0 that each method defines (see Section G.2 for full details).
We evaluate our method on a suite of tasks consisting of an illustrative example, two MuJoCo environments (Todorov et 2012), driving scenario, and a nigh-dimensional exed-wing aircraft simulator (So & Fan [2023 Heidlauf et al. (2088) (Fig. (6). We also test on a lunar lander from (Mathhews et 2025) with RGB image-basee basge-based observations. Details can be found Section G.3
Benchmarks.
Evaluation Metrics. As not all parameters 0 are feasible, the highest achievable safety rate for each environment depends on the size of the feasible set for that environment. enable fair comparison across environments, we measure the safety rate conditioned on the parameter being feasible, where we conservatively estimate that a parameter is feasible if any method can make safe. Additionally, to understand where the differences arise, we also define coverage gain to measure how many "hard" parameters a method can discover as feasible, and coverage loss to measure how many "casy" parameters method cannot discover feasible, where use the performance of DR proxy for parameter difficulty. Details on these metrics are in Section G.
5.1
RESULTS
Following evaluation recommendations from Agarwal al. (202T), we plot the performance profile in Fig. 7a FGE is the highest curve, indicating that it consistently achieves high safety rate across tasks. Next, to examine differences between methods, we plot safery rate, coverage gain, and coverage
loss (Fig.) 7. where FGE achieves the greatest coverage gain relative to DR across all environments. Training curves Section D.2 show that FGE remains stable during training.
Next, perform case studies understand why FGE achieves the highest coverage gains.
FGE achieves gains in coverage by allocating more samples to rare and difficult parameters. We first analyze Dubins (Fig. g) and examine challenging parameters where the ego must squeeze through a tight gap to remain safe. In particular, we find that parameters where the ego car must overtake extremely rare (bottom left pixel in Fig. S. Comparing FGE and VDS, observe that FGE allocates vastly more samples to this condition than VDS. Consequently, FGE remains safe on this challenging overtake condition while VDS crashes.
8

---

Published as a conference paper at ICLR 2026
FGE
0:10
0:10
VDS
Figure 8: FGE renders challenging initial conditions safe by allocating more samples to them. (left) We analyze the Dubins environment and find two types of challenging initial conditions where the ego car must squeeze through a tight gap to remain safe. particular, the overtake condition is extremely rare. (right) FGE allocates vastly more samples this condition than VDS. (center) Consequently, FGE is able to solve this challenging overtake condition while VDS crashes.
FGE
VDS
1.0:
1.0
Feasibility
Count ((te)
Classifier
0.5
0.5
0.5
Reset Distribution
500
position (9)
X pppssiion (8)
x positiio (8)
Figure 9: Rejection Sampling Enables Efficient Exploration in FGE. The explore distribution in
FGE effectively allocates samples to regions that have not been solved yet. By the middle plot, FGE has concentrated third of its samples on the gap between the two platforms, eventually solving this region by the end of training. In contrast, VDS fails to remain safe in this region.
Reset Distribution
FGE
DR
VDS
PLR
RARL
200
3000
Count
000
200
2000
100
100
100
100
1000
position
pppoition
position
posiiiion
posiiion
Figure 10: Heuristics used by existing methods do not concentrate samples as efficiently as FGE. Only FGE is able to effectively concentrate its samples in the difficult to solve gap region on Hopper.
The explore distribution in FGE efficiently samples and difficult parameters via rejection sampling. A similar story holds on Hopper. Here, the small gap between platforms requires recovery maneuver avoid falling through. We visualize the learned feasibility classifier across training iterations (Fig. 99) Early in training, the robot quickly learns to stay safe on flat ground, and the classifier & marks this region feasible. The explore distribution stops sampling from this solved region and eventually concentrates third of samples on the gap once all other regions are solved. As Dubins, this enables FGE to stay safe the gap while VDS does not.
Heuristics used by existing methods fail to concentrate samples as efficiently FGE. To test our hypothesis that FGE's coverage gains arise from its sampling distribution, we compare reset distributions Hopper (Fig. 10). marking the gap region with dashed vertical lines. Although methods such as VDS and PLR bias sampling towards certain regions, they do not concentrate samples within the gap as well FGE. UED methods (PAIRED, FARR, ACCEL, SFL) also fail to produce efficient distributions despite their regret- based motivations, largely due to large regret approximation errors (Rutherford al. 2024). We compare with UED in more detail in Section
FGE scales to high-dimensional observations and parameters. We now examine whether FGE scales to digh-dimen observations and parameters by testing Lander2D and LanderşD, both

---

Published as a conference paper at ICLR 2026
(a) Safety Rate (T)
(b) Coverage Gain vs DR (1)
[c) Coverage Loss vs DR (4)
(d) Unique Coverage (1)
Landertted
03
70
02
10
04
Figure 11: FGE scales to high-dimensional observations (Lander2D) and high-dimensional parameter spaces (Lander9D). FGE maintains the highest safety rates even tasks with RGB image observations (Lander2D, Lander9D) and with 9D parameter spaces (Lander9D).
FGE w/o Seh,
8000
2000
Coverage Gain DR (1) Coverage Loss DR (4)
Sufety Rate
FGE - exploration
FGF
0.31
5.00
0.5
reheorsal
0.58
1000
2000
3000
4000
5000
0.00
0.25
0.50
0.75
1:00
0.00
0.05
0.10
0.15
0.20
Step
Figure 12: Removing either exploration or rehearsal distributions significantly hurts performance on Dubins.
Figure 13: Removing the rehearsal buffer destabilizes training on ToyLevels.
of which use RGB image observations (Fig. TTI.
We compare against DR and the two best-performing baseline methods (VDS, PLR). FGE continues to achieving the highest safety rates.
5.2 ABLATION STUDIES
Having observed that FGE's gains in coverage come from the effectiveness of the explore distribution, we next answer the following research questions to better understand the components of FGE: (Q1) Is the explore distribution responsible for the strong coverage gain of FGE? (Q2) How does the rehearsal buffer affect the performance? (Q3) How important is the feasibility classifier?
(Q1) Exploration is essential for high safety rates. Removing the explore distribution from FGE significantly reduces coverage (Fig. 12 Thus, the explore buffer is essential to FGE's 's performance.
(Q2) The rehearsal buffer is critical for stability. Removing the rehearsal buffer leads to smaller drop in coverage gain, but higher coverage loss (Fig. [2] as the training becomes unstable (Fig. 13).
(Q3) The proposed classifier using mixture distribu- tions performs better than density-based methods. We compare our proposed approach to learning classifier & from Section 4.1 to learned density model using pla- nar flows (PF, Rezende & Mohamed, 2015) and Neural Spline Flows (NSF. Durkan et al. 2019) (Fig. 14) Using a classifier significantly outperforms both PF and NSF, with NSF still achieving higher coverage than the next best-perforning baseline (PLR).
Coverage Gain vs DR (T)
PLR (2nd best)
Classifier
0.61
0.40
NSF
0.20
Planar
0.00
0.25
0.50
0.75
1.00
Figure 14:
Replacing the feasibility classifier with a density-based classifier hurts performance on Dubins.
6 CONCLUSION
We have proposed a novel approach to solving parameter-robust avoid problems with unknown feasibility via simultaneous feasible-set estimation and saddle-point finding and demonstrated the strong performance of our method compared to prior approaches in solving a multitude of challenging benchmark problems, where our method learns policies that achieve significantly larger feasible sets.
Limitations Our proposed method approximates the best-response oracles of the saddlc-point finding algorithm, thus the theoretical convergence guarantees may not hold. This common UED works (e.g., by assuming access to a best-response oracle (Dennis et al., 2020)). These theoretical results provide insight on why our method works well empirically, but theoretically proving convergence is not feasible under current mathematical frameworks and is still an open challenge. Moreover, our framework assumes deterministic dynamics, which enables determining feasibility from a single safe rollout. Handling safety and feasibility for stochastic dynamics via approaches such as chance constraints is more complex for our proposed method compared to techniques such as domain randomization and is a promising future direction.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
All proofs of theoretical results are included in Section A Implementation details and hyperparameters of all algorithms included in Section G supplementary materials.
We also include the source code of our algorithm in the
ACKNOWLEDGEMENT
This material is based upon work supported by the Under Secretary of Defense for Research and Any opinions,
Engineering under Air Force Contract No. FA8702-15-2 15-000000 findings, conclusions or recommendations expressed in this material are those of the author(s) and do necessarily reflect the views of the Under Secretary of Defense for Research and Engineering. This work was also supported in part by grant from Amazon.
2025 Massachusetts Institute of Technology.
Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice. U.S. Govermment rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 4 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.

---

Published as a conference paper at ICLR 2026
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural information processing systems, 02223344-93222 2021.
Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
Somil Bansal and Claire J Tomlin. Deepreach: A deep learning approach to high-dimensional reachability. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1817-1824. IEEE, 2021.
Michel Benaim and Morris W Hirsch. Mixed equilibria and dynamical systems arising from fictitious play in perturbed games. Games and Economic Behavior, (9(1-))336-72 1999.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-31 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 01440, 2016.
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret mini- mization. In International conference on machine learning, pp. 793-802. PMLR, 2019.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 1rXiv:1810.128%0 2018.
Fernando Castaneda, Haruki Nishimura, Rowan Thomas McAllister, Koushil Sreenath, and Adrien
Gaidon. In-distribution barrier functions: Self-supervised policy filters that avoid out-of-distribution states. In Learning for Dynamics and Control Conference, pp. 286-299. PMLR, 2023.
Étienne de Montbrun and Jérôme Renault. Optimistic gradient descent ascent in zero-sum and general-sum bilinear games. arXiv preprint arXiv:2208.033083 2022.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:1 13049-13061,, 2020.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios, Neural spline flows. Advances in neural information processing systems, 32, 2019.
Gabriele Farina, Christian Kroer, and Tuomas Sandholm. Stochastic regret minimization in extensive- form games. In International Conference Machine Learning, pp. 3018-300100 3028. PMLR, 2020.
Jaime Fisac, Neil F Lugovoy, Vicenç Rubies-Royo Shromona Ghosh, and Claire Tomlin. Bridging hamilton-jacohb safety analysis and reinforcement learning. In 20/9 International Conference on Robotics and Automation (ICRA), pp. 5550-8556, IEEE, 2019.
Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems, 36:69764-69797, 2023.
Milan Ganai, Sicun Gao, and Sylvia Herbert. Hamilton jacobi reachability in reinforcement learning: survey. IEEE Open Journal of Control Systems, 2024.
Jonathan Gray, Adam Lerer, Anton Bakhtin, and Noam Brown. Human-level performance in no-press diplomacy via equilibrium search. arXiv preprint arXiv:2010.02923 2020.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint #502 2015.
Peter Heidlauf, Alexander Collins, Michael Bolender, and Stanley Bak. Verification challenges in f-16 ground collision avoidance and other automated maneuvers. ARCH@ ADHS, pp. 208-217, 2018.
12

---

Published as a conference paper at ICLR 2026
Kai-Chieh Hsu, Vicenç Lubies-Royo Claire J Tomlin, and Jaime F Fisac. Safety and liveness guarantees through reach-avoid reinforcement learning. arXiv preprint arXiv:2112.12288, 2021.
Kai-Chieh Hsu, Duy Phuong Nguyen, and Jaime Fernandez Fisac. Isaacs: Iterative soft adversarial actor-critic for safety. In Learning for Dynamics and Control Conference, pp. 90-103. PMLR, 2023.
Minqi Jiang, Michael Dennis, Jack Mookerr, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances Neural Information Processing Systems, 34:1884 1897, 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4940-4950. PMLR, 2021b.
Katie Kang, Paula Gradu, Jason Choi, Michael Janner, Claire Tomlin, and Sergey Levine. Lya- punov density models: Constraining distribution shift in learning-based control. In International Conference on Machine Learning, pp. 10708-10733. PMLR, 2022.
J. B. Lanier, Stephen McAleer, Pierre Baldi, and Roy Fox. Feasible Adversarial Reinforcment Learning for Underspecified Environments, October 2022. URL URL httpttt::/axx...op//p/ppo// 2207. 09597 urXiv:2207.09597 [cs].
Jingqi Li, David Fridovich-Keie Keil, Somayeh Sojoudi, and Claire J Tomlin. Augmented lagrangian method instantaneously constrained reinforcement learning problems. 202/ 60th IEEE Conference Decision and Control (CDC), pp. 2982-2989. IEEE, 2021.
Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-pont point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 4940-4947, 2020.
John Lygeros, Claire Tomlin, and Shankar Sastry. On controller synthesis for nonlinear hybrid systems. In Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No. 98C1136171,, volume 2, pp. 2101-2106 IEEE, 1998.
Michael Matthews, Michael Beukman, Chris Lu, and Jakob Nicolaus Foerster. Kinetix: Investigating the training of general agents through open-ended physics-based control tasks. In The Thirteenth International Conference Learning Representations, 2025, URL htttts::///peeeeee net/ Eorum id= 2o-orummid--2xxcdzzreeeeee
lan M Mitchell, Alexandre M Bayen, and Claire J Tomlin. A time-dependent hamilton-jacobi formulation of reachable sets for continuous dynamic games. IEEE Transactions automatic control, 50(7):947-957, 2005.
Ian M Mitchell et al. A toolbox of level set methods. UBC Department of Computer Science Technical Report 2R-2007-11, 1:6, 2007.
Eugene A Morelli. Global nonlinear aerodynamic modeling using multivariate orthogonal functions. Journal of Aircraft, 32(2):270-277, 1995.
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335-359, 2005.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, (188)):1550, 2020.
Duy P Nguyen, Kai-Chich Hsu, Wenhao Yu, Jie Tan, and Jaime F Fisac. Gameplay filters: Robust zero-shot safety through adversarial imagination. arXiv preprint arXiv:2405. 00846, 2024.
Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv: 1912. 2019.
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In International Conference Machine Learning, pp. 17473-17498. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2026
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial rein- forcement learning. In International conference on machine learning, pp. 2817-2826. PMLR, 2017.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint 1rXiv:1910.01708, 7(1):2, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. International conference on machine learning, pp. 1530-1538. PMLR, 2015.
Alexander Rutherford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, and Jakob Foerster. No regrets: Investigating and improving regret approximations for curriculum discovery. Advances in Neural Information Processing Systems, 37: 37:16071-16101, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017
Oswin So and Chuchu Fan. Solving stabilize-avoid optimal control via epigraph form and deep reinforcement learning. arXiv preprint arXiv:2305. 2023.
Oswin So, Cheng Ge, and Chuchu Fan. Solving minimum-cost reach avoid using reinforcement learning. Advances in Neural Information Processing Systems, 777:30511-3988. 2024.
Aivar Sootla, Alexander I Cowen-Rivers Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning, pp. 20423-20443, PMLR, 2022.
Kyle Stachowicz and Sergey Levine. Racer: Epistemic risk-sensitive rl enables fast driving with fewer crashes. arXiv preprint arXiv:2405.04714, 2024.
Brian L. Stevens and Frank L. Lewis. Aircraft Control and Simulation. Aircrafi Engineering and Aeroxpace Technology. 76(5), January 2004. ISSN 0000-22677.. doi: 10.1 1..100//eet.22000 1004412776eae 001. URL // /doii .aaiorre orgg /10g//00..0 1et.11108aaaaa. 227.00......6..... 001
Publisher: Emer-
ald Group Publishing Limited.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. International Conference on Muchine Learning, pp. 9133-9143, PMLR, 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
Geraud Nangue Tasse Tamlin Love, Mark Nemecek, Steven James, and Benjamin Rosman. Rosarl: Reward-only safe reinforcement learning. arXiv preprint arXiv:2306. 00003, 2023.
Team RV. Team formation training guide, 2002. URthhttss://wwwr www. www.lpilllsooooooooooooooooooooe org/
http:///w..totooooooooe//tttttt. 30.
pdf Accessed: 2010-09-
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Grigorii Veviurko, Wendelin Böhmer, and Mathijs De Weerdt. To the max: reinventing reward in reinforcement learning. arXiy preprint arXiv:2402.0/ 2024.
Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. Safe exploration and optimization of constrained mdps using gaussian processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
14

---

Published as a conference paper at ICLR 2026
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Cipo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pP- 11480-1149) 1191. PMLR, 2021.
Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matchijs TJ Spaan. Wesac: Worst-case soft actor critic for safety-constrained reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10639-10646, 2021.
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Projection-basee constrained policy optimization. In International Conference Learning Representations, 2020.
net forum? iidrmmmerrrreeeess
Kaiqing Zhang, Bin Hu, and Tamer Basar. On the stability and convergence of robust adversarial reinforcement learning: case study on linear quadratic systems. Advances in Neural Information Processing Systems, 33:22056-22068, 2020a.
Songyuan Zhang, Oswin So, Mitchell Black, Zachary Serlin, and Chuchu Fan. Solving Multi-Agent Safe Optimal Control with Distributed Epigraph Form MARL. In Proceedings of Robotics: Science and Systems, LosAngeles, CA, USA, June 2025. doi: 10. 1005577/888
Yunzhi Zhang, Picter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement, Advances in Neural Information Processing Systems, 33:7648-7659) 2020b.
15

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1 RELATIONSHIP BETWEEN THE PARAMETER PARAMETER-ROBUST ROBUST AVOID PROBLEM AND THE STANDARD REACHABILITY PROBLEM
We define the parameter-robust avoid problem as the following constraint satisfaction problem
min Mo EM
(14)
s.t.
(*())) 00 4k 2 0 sk+1 fu(Sk: R To (st)), $u = (0) 40 E 6
(15)
4 0
(16)
(17)
Define the augmented state Consider the reachability problem 2 i.e.
8, and the augmented dynamics f(3, a) = [f(s, a, 0).0]
Vreach (8o: 0)
min max (88), TyEM k20
8k+1 fo(sk, (0()))
Let So (zo(0),0) E }
denote the set of all initial state-parameter pairs, and let S (so, () Vreach (8o; 0) I 0 denote the vero-sublevel set of the value function Vreach- Then, the two problems are equivalent in the following sense.
Proposition 1. So S, then the parameter-robust avoid problem is feasible.
Proof. We prove via contradiction. Suppose that the parameter-robust avoid problem infeasible. Then, for all policies Te E M, there exists some k 2 and some O such that he(sk) > However, since (za(0).0) E S. (2) implies that there exists some т such that ha(8)) S 0 for all k O. This a contradiction. Therefore, the parameter-robust avoid problem feasible with policy T--
A.2 THE ZERO-SUBLEVEL SETS OF VEEACH IS EQUAL TO ZERO LEVEL SET V
Lemma 1. The zero-sublevel set of the value functions Vreach is equal to the zero level set of V, i.e., (seach(s; s 0 (s, ((s = =}.
{ (s, 0)
(18)
Proof. We show via double inclusion.
Consider su and 0 such that Vreach 0) < 0. Then, there exists control policy m such that ho(#k) S 0 for all 2 0, and consequently, ho(e))) = 0 for all 2 0. Therefore, (s0; 0) Ele I 0.
Now, consider su and such that V(so: 8) = Then, Σ 1no = 0, implying that ho(()) < for & 2 0. Therefore, Vrench (o; 0) min, max ho (8())) 0.
A.3 UPPER BOUNDING THE FALSE NEGATIVE RATE
We show that the learned classifier qu- for a given threshold can be accurate given conditions on a and the density ratio 5" /p over feasible parameters.
Theorem 1. Let qu- denote the solution and define Ç: θ o0)
((0) 1-a p(0)
(19)
a pD,(0) (0)
Then, for feasible parameter 0g E DD we have that gos- (f = (r) 2 B for all 0g where
pº (f 2 3 (10))=)1 ()00))-
(20)
For infeasible parameter 0g E C- qu- will always correctly classify it with qo- (f = (a)) = 0.
16

---

Published as a conference paper at ICLR 2026
Proof. For any infeasible parameter 0 O",, we have that PD, (f 1 p"((f= 1 0. Hence,
Pmix (f = 10) = 0.
For feasible parameter 0 E Dj. we have that (f - 1 Hence,
app, (0) + (1 (0))0)
Pmix (f 1|0)
(21)
app, (1-a)
For this to be greater than B, this implies that
apD + (1 - a)pp (f 2 110) > B (apD, (0) + (1 (1-())(((0) a)p(O))
(22)
(app (0) + (1 aa)(a))))))))))))))))))))))))))) apD, (0)
(f 1 2
(23)
(1 - a)))p(
a Pb, (0)
= 3 - (1- - (8
(24)
p(0)
= 3 - (1 - ())(())
(25)
A.4 BENAVIOR OF THE CLASSIFIER UNDER REJECTION SAMPLING
We now consider the case where we take p as qu (0lf = 0), i.e.,
p()) = qu 00 2o = (o0f==00)pp0)
(26)
gx
This couples the classifier and the parameter sampling distribution. For 9 E Dj S O' If we hold the policy π fixed and we iteratively update the classifier qe and p, we obtain the following iterates:
()) = (O1 = =0) = ( (f 010)p(0)
(27)
q (f =
(1 (1)
(+ = 0/0) p='(f===- = 0f app, (0) + (1 (1)1))
(28)
We then have the following result about the behavior of these iterates. Proposition 2. The fixed-point always exists, and given by
(f 010) - a PD, PD, (0)p" (f (f==)f=) 0)) p= (f 010)p(0)d (=0) Ee-r(o) 1v" (f = 0|0)]
g (f = 010) = .max (f.
(29)
p(0)
where
p" p=(f = 0)=00)-- = / p"
(30)
(1
Proof. Note that app, (o (0, 1), so the error rate of the classifier is always upper-
bounded by the policy-inducee error rate:
(k+ = 010) (f 010).
(31)
Moreover, note that 0 is absorbing for the iterates: if (f) 010) 0. then (k0) 0 and thus (f++1) 0) 0.
For A such that pz (f 0|@) 0, we analyze the fixed points of the iterates. The fixed point (ex)
satisfies
(1 a) g000) (1=00)))))))
a)
(x=) = 010) = p -f
(32)
app, (0) + (1 (==00)00)
(F=0)
Or.
(f)) (f 0(9) (Opo, (0) + (1 - a) (00 (f 0(0)p(0)
p(0)
g< 0)
(1 a)p 010)
g( = 0)
=0
(33)
17

---

Published as a conference paper at ICLR 2026
In other words, at the fixed point, either (f = 0|0)
0 or
((00) 010)p(0) (1
of
p())
upp,(0) + (0) a)
q (f - 0) (1 - a)p (f - 010)
= 0.
(34)
(f 0)
Assume the latter. Then,
1
(f= [0f=000 0|0) = (f
(35)
pp, (0)
+ T-a pla)(()
= -f=(f==00)
(36)
pry (f=
+ 1-a of (f=0 p)p()
Rearranging,
a pd, (0)g - 0)
(f = 010) 11 - a (f = 01))p(0)
= 010)
(37)
PD, (0)g (f = =0)
- (f)) (0l0
pf (f 010);
(38)
i-a a
((0)
PD, (0)g/00) (00) 0)
-
ا (xx (f = 00) - p² (f - 0
a
p(0)
(39)
Substituting (f) 010) into (000 (f 0). we get
(n)) (f 0) / 00) (f = ()))d
(40)
/
PD, (0) (f = =0)
D" p-(ff = 010)
p(0)d0,
(41)
- a
p(0)
/ rT((f = (10)p(0)d0
I-a c (00 00) / PD, (0)
(42)
p" (f = 0)
a
(00) = 0).
(43)
1
Hence,
(f = 0) = p" (f = 0) a
(f)) = 0),
(44)
1
(<) (f= = 0) (1 a)p² (f=1) 0).
(45)
Substituting back, we get the fixed point
PD, (0)p (f = 0) p(0)
(f= 010) = p=
(46)
Since (f 010) 2 0, we have that
(f))(( = 010)= = .mx 010) - PD, (0)p(()
P0 = 0)) )))
(47)
p(0)
Consequently, we can use this to bound classifier accuracy even under the worst-case policy n where p² (f = 010) = 1:
Proposition 3. Under the worst-case policy where pd (f = 0|0) 1, for desired error e E (0. 1),
p(0)
a z (1 -1-61 (D)
- (f = 010) S c.
(48)
18

---

Published as a conference paper at ICLR 2026
Proof. If (l(()) 0/0) is zero, then it is trivially satisfied. Suppose it is not. Then,
(000 0 = =1 PD, (0)
(49)
p(0)
ス - k) p(0)
(50)
PD, (0)
A.5 DERIVATION OF
Using quadratic proximal regularizer ()(()) 2n || Tell2 and linearized losses J(x) = (TT,) + (Tr)) (T - ne). then taking the gradient and setting it to zero, we obtain
t ne) + 1
vJ(m)= =0
(51)
T= +
V((TT)
(52)
A.6 DERIVATION OF 9
= Pmix (f = 1|0) = - (1-p"(f=110) 110) (1
a PD, PD.
0 E Di
(53)
a p(0)
For E Dr, starting from the definition of Pmmx (f = 1 in (
Pmix (f = 110) = ap" = 1 0 p) (0) + - af ())(()
(54)
app, (0) + (1 (1-a)p(00)
oopD, (0) + (1 a)w" 000"(f ())p(0))
(55)
app, + (1-a)p(0)
(56)
19

---

Published as a conference paper at ICLR 2026
OR
aR
0RR
aR
R
R 0
R 0
R 0
$1
SH
aL
aL
aL
R
R
Figure 15: Chain MDP
B FURTHER ANALYSIS OF CHAIN MDP
B.I (SINGLE) CHAIN MDP ANALYSIS
We first define the single-chain MDP of length H, which consisting of H states $1,..., SH, two actions at, and nn and initial state 81. For any state i, taking the action az terminates the episode and returns reward of -1, while taking the action moves to the next state 8,-1. The final state H a terminal state, which returns a reward of 0 and terminates the episode. We visualize the single-chain MDP in Fig. 15
first analyze the single-chain MDP. For simplicity, we parameterize the policy by a sealar x E R, where we take action ug with probability and action az with probability 1 n
The single-trajectory REINFORCE. update law
V V=) = Σ V log (agl)s)
(57)
where de, St, Ry random variables. Let T denote the (random) time step where the episode terminates by taking or if < H, and let = denote the episode terminating by reaching the terminal state 85.
We also have log, Taa = ag) = à Also,
and V log, x(a = aL) = 1-
TT- 1,
R
T=
(58)
T
Note that the entire trajectory is captured by the random-variable T. Hence, it sufficient compute the gradient conditioned on each value of T', then take the expectation over T", Let g(T) denote the gradient conditioned T.
For T < H, we have
g(T)
Σ
V log (a az) + Va log R2)
(59)
T-1 T-t + (- (--=-) (- (---
Σ
(-
(60)
π
y
(61)
1- π
IfT = H, then g 0 since the return is O.
Finally, looking at the distribution T' we have
P(T = f) = --'1(11 x), 1 < H,
(62)
P(T H) =
(63)
20

---

Published as a conference paper at ICLR 2026
Gradient Variance
4
4
6
10
12
14
16
18
Chain Length H
Figure 16: Variance of the single-sampl gradient estimator for the chain MDP.
Hence,
H-1 Blg(T)) = Σ = ((T(t))((), Varlg(T)] = Σ -
H-1
(64)
Plotting the variance numerically in Fig. 16 for 0.5 and y 0.99, we see that the variance increases with H.
B.2 MULTI-CHAIN MDP ANALYSIS
We now consider the multi-chain MDP, which consists of P independent chains length H; for i 1...., P. The parameter 8 f1,..., P) controls which chain's 's initial state to start from. See Fig. 17 for a visualization.
Here, assume the policy is parametrized by æ for i = I,..., i.e., each chain has separate policy. Suppose we sample from the uniform distribution over (1,..., P)- variance of single-chain MDP with horizon H. Then, the variance of the resulting multi-sample monte-carlo REINFORCE gradient estimator will be scaled down equally across all / chains, i.c., each of the chains gets budget of N samples, Given that larger variances of the gradient estimator theoretically slow down the convergence rate of SGD by reducing the maximum learning rate that can be used Bottou al. (2018), this means that the policy for the chain with the longest horizon H; converges the slowest as it has the largest variance.
Let ((H) denote the
obtain variance of V YHH,)/N for the ith chain.
Instead, we sample from distribution proportional to the gradient variance, i.e.,
p(0)) V(H:).
(65)
prioritizing chains with larger gradient variances, then the variance will be equal for all chains, with the worst chain converging faster than the previous case.
21

---

Published as a conference paper at ICLR 2026
RR
0R
0 = 1
(1)
R = 0
R
(1)
$1
H1
RR
0=2 2
R =
R
(2)
$1
Hi
OR
OR
0 ==P
(P)
R
R = 0
(P)
81
HH
Figure 17: Multi Chain MDP
C CONVERGENCE GUARANTEES TO SADDLE POINTS
We present proof based on Orabona (2019) that a combination of best-response for the player and follow-the the-regularized reguuerrzeedleaderr for the player results in convergence the saddle-poing colll--ointtoee under suitable set of assumptions. For analysis, we flip the sign on J to better match the convention from the online learning literature (Orabona 2019) and consider the minimization objective.
Consider the following algorithm:
Algorithm Saddle-point finding with best-response and follow gellow-tberrguulaiizd--eede
Require: 1A1 .M
1: for (= -= 1,..., do
2: Set 0, E arg maxage (p::0)
3: Set 1+41 = (1) + arg min MEM Etzr Ju, 0.)
4: end forreturn 0
where we define the average iterates fy and Por as
1
PT
=
Pt:
(66)
T
t=
1
0r
O,-
(67)
T
1=1
We now make the following assumptions:
Assumption 1. J A:Mxx - R. is convex in the first argument and concave in the second argument. Moreover, for any 0, (-,0)) is L-Lipschitz, i.e.,
[J(1100) 0) (12,, 0)1 < Ellun Pall-
(68)
We then have the following result.
Lemma (Adapted from Orabona (0119). Let J : : x θ - R. be convex in the first argument and concave in the second argument. Then, we have
< sup J (ūr, 0) min s 0r) < T Regret (1(T) #E0
(69)
#EM
22

---

Published as a conference paper at ICLR 2026
for any p4p E arg min Ju, 0), where
Regret (1)) = Σ J(n, 01)
Σ [1, 0c).
(70)
Proof. By definition of Regret M for any μ E M,
T
1 T
Jmn, 01)
Σ
J(n 01) = T
Regret M (n)
(71)
T
T
Moreover, since 0, is the best-response to pt, for any E O, we have that J(,. 0) S Ju,, 0i) for any Hence,
Σ M( 0)
1 1
Σ Jun, 06) S0.
(72)
Summing 71 and 722 then yields
1 Σ ((, (egret(((1)
(pc. 0)
(73)
Now, Jensen's inequality gives that, for any 0 E O and μ M.
T
Hür, 0) J(, S
Mme 0)
1
Σ
J(,
(74)
T
Combining this with (3) then gives
J(ör, 0) - J(4 07) - Regreti (n).
(75)
Choosing /z 14 arg min [1, 0r) and taking suprema 0 θ then gives (69 where the non-negativity of the duality gap follows from supace J(l 0) [ 0') 2 inf J (11, 0* for all y' and 0'.
Consequently, if we can show that the regret of the y player is sublinear, Lemma 2 implies that the average iterates (ūr, 0r) converge zero duality gap, i.e., saddle point of the game. We show that this holds next in our case where uses a
strategy.
Lemma 3. Let 1p MM- + B be closed, m-strongly convex function, and define the sequence of regularizers the as
ve(r) = 1 (v(()) min v5(2)),
(76)
T--1
with the non-increasing sequence it defined as
aym
M LVt
1 1= I....T
(77)
Then, the regret for the μ player is sublinear, i.e., Regret&MMul = O(VT).
(78)
Proof. Under Assumption I the conditions for (Orabona 2019 ((()) = [u, (t) which yields the result.
Corollary 7.7) are satisfied with
23

---

Published as a conference paper at ICLR 2026
Combining these two then gives us the desired convergence result.
Corollary The average iterates (pr, 0r) converge saddle point of the game.
Proof. By Lemma 3 the regret for μ is sublinear. Invoking Lemma 2 then gives us the convergence of the average iterates (rr, saddle point of the game.
D ADDITIONAL RESULTS AND ABLATIONS
In this section, include additional results and ablation studies that are not in the main paper due space constraints.
D.I PER-ENVIRONMENT COVERAGE PLOTS
While Fig. provides summary of the performance of each method across all environments, present a per-environment breakdown of the performance of each method in Fig. [8] The conclusion remains the same, with FGE yielding significant increases in coverage gain compared the next best baseline method. Interestingly, the next best method varies a lot across the different environments: RARL achieves the second best coverage gain in three out of the five environments, but consistently achieves the worst coverage loss against DR.
D.2 TRAINING CURVES
We plot the training curves for all environments in Fig. 19
D.3
ADDITIONAL. ABLATION STUDIES
We next perform additional ablation studies to understand the impact of the design choices of FGE that were covered in the main paper.
D.3.1 DOES A MORE TRADITIONAL EXPLORATION TECHNIQUE SUCH AS RND SOLVE OUR PROBLEM?
Coverage Gain (1)
Coverage Loss (1)
Dubine
PUB
EXL
RMB3
BA
4.00
..80
0.50
0.5
0.50
075
-0.55
-0.0
-025
0.00
Hoppar
03
DR
0.00
0.50
0.5
0.50
075
-0.75
-0.50 -0.5
Figure 20: RND ablation.
No, RND is beneficial but does not replace FGE's gains. In Fig. 20 we add RND Burda al. 2018) intrinsic reward to DR. We observe slight improvements on coverage gain for DR relative to its Ron-RND counterpart. However, adding RND to DR neither outperforms FGE the next pest-performig baseline. Thus, we conclude that exploration techniques such as RND cannot replace FGE in solving the parameter-robus avoid problem with unknown feasibility.
24

---

Published as a conference paper at ICLR 2026
(a) Safety Rate (1)
(b) Coverage Gain vs DR (1)
(c) Coverage Loss vs DR
(d) Unique Coverage (1)
ToyLoves
1000
FASR
8.00
0.90
0.0
0.2
1.0
30
2/0
08
0.0
1.0
10
Dubns
PAZ/
ADCEL
0.00
Dubinc Mubleeee
FOR
5.00
0:00
RCCEL
Hopper
PATL
ADCEL
0.0
0.2
0.4
0.0
1.0
3.0
0.2
0.4
0.5
0.0
3.0
6.2
84
0.
0.
3.0
0.0
0.2
0.4
08
58
10
HoPChedahh
2.07
3.03
0.0
0.2
0.1
0.00
0.0
02
0.4
0.6
0.8
1.0
86
08
1.0
1000
ReedWing
1X50
ACCEL
0.0
322
Figure 18: Coverage Comparison. (a) FGE achieves near 100% coverage across all environments, and consistently achieves the highest coverage in the harder environments. (b) FGE achieves the largest coverage increases relative to DR across all environments, substantially outperforming all other baselines. Other methods can expand the feasible set of DR, but underperform FGE by 40% to 90% on the harder environments. (c) Most baselines do not lose coverage when compared to DR with the exception of RARL due to instability during training. ACCEL also performs poorly due to the need for each rollout to last for the maximum episode steps, reducing the number of total updates steps taken. (d) With the exception Leylevels, almost all the unique safe parameters occur due to FGE, suggesting that FGE consistently renders parameters feasible that other method can.
25

---

Published as a conference paper at ICLR 2026
Coverage Gain vs DR (1)
Coverage Loss vs DR (4)
0.6
ToyLevels
0.0
1.0
Dubine
Hopper
0.4
0.5
163
163
1.0
HellCheeteh
0.5
163
0.75
FiedWing
0.50
0.5
0.25
0.00
10
50
Iterations
160
Iterations
1ed
FGE
RAAL
Figure 19: Training Curves.
D.3.2 DOES USING A VALUE FUNCTION OVER A CLASSIFIE MAKE A DIFFERENCE?
PLR (2nd best)
Dubins
Classifier
0.61
0.57
Value function
0.0
0.5
1.0
Figure 21: Value vs classifier for feasibility on Dubins.
Both classifier and value function performed similarly, but should still matter in practice. We compare using our policy classifier with an RL. value function where if s E Ag, V(s) > 0, else Vss) = We empirically find on Dubins that using either method does not matter much. However, we hypothesize that using the RL value function as a policy classifier should incur a few issues in practice. For instance, if an unsafe state has a long termination sequence, the cumulative discounted value of that state become arbitrarily small. This may lead to numerical imprecision that is expected with any learning- based approximation of the value function, potentially leading to incorrect classifications.
26

---

Published as a conference paper at ICLR 2026
D.3.3
How SENSITIVE IS OUR CLASSIFIER THE MIX FRACTION a?
Coverage Gain vs DR (T)
Coverage Loss vs DR (1)
.PLR
0.61
-0.01
.00
0.56
-0.1
Dubins
.25
0.61
00.11
.50
0,70
-0.02
.75
0.59
-0.01
.99
0.0
0.2
0.4
0.6
0.8
1.0
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
Figure 22: a sensitivity ablation.
It's fairly robust. Seeing how sensitive NSF's performance is choice of threshold from Sec- tion 5.2 we conduct sensitivity test on the mix fraction a for our classifier. From Fig. 22 we observe that a 0.75 performed the best on Dubins. However, from our experiments we did notice that the performance with respect to a differed by task. In addition, FGE still outperforms the next best-performing baseline, PLR, for all values of a, indicating that our proposed method's gains are robust to these hyperparameters. For consistency, we default to 0.5 across all experiments.
D.4 SAFE REINFORCEMENT LEARNING COMPARISONS
FGE is orthogonal safe RL methods since it only defines the initial state distribution. Therefore, FGE can be easily paired with any on-policy method, including those in safe RL, to improve safety. We illustrate this by defining safe RL task ToyLevels (Section G.3) where the policy must minimize cumulative discounted sum of control input magnitudes subject to safety constraints, i.c..
min Σ ([ (-)
(79)
s.t. max max [[I,
(80)
We pair two safe RL algorithms, PPO-lagrangian (PPO-L) and SauteRL Sootla et al. (2022)), with FGE and the domain randomization baseline (DR). Since both algorithms have hyperparameter (lagrange multiplier for PPO-L, unsafe penalty for SauteRL), we run a hyperparameter scan for each algorithm and plot the reward coverage of each in Fig. 23
27

---

Published as a conference paper at ICLR 2026
ToyLevels
PPO-L
SauteRL
Reward (1)
-2
-3
PPO + FGE
PPO + FGE *
-4
PPO + DR
* PPO + DR
0.80
0.85
0.90
0.95
1.00
0.80
0.85
0.90
0.95
1.00
Safety Rate (t)
Safety Rate (f)
+ DR
+ FGE
Figure 23: Safe RL Pareto Frontier.
We observe that FGE greatly expands the pareto frontier compared to DR in both PPO-L and SauteRL. In addition, all safe RL variants (circles) achieve higher reward than the non-safe RL baselines (stars) as expected, since the non-safe RL methods do not consider the reward. This shows that FGE can be effectively used to complement existing safe RI. methods to improve safety over wider set of parameters.
E RELATIONSHIP WITH UNSUPERVISED ENVIRONMENT DESIGN (UED)
Many methods from UED aim to choose parameters that have high regret (Dennis al. 2020 Lanier
et al. 2022 Parker-Holder et al. 2022 Rutherford et al. 2024), with regret for parameter defined
as
Regret(ms, 0) = J ( 0) max J ( 0).
(81)
The challenge with 88 is that the second term involves optimization problem and is thus intractable. Consequently, UED methods propose different methods of approximating this term, which we briefly discuss below.
PAIRED (Dennis et al. 2020): In PAIRED, regret is approximated by training an antagonist policy that aims to maximize the regret. Specifically, separate PPO policy is used, with per-timestep rewards replaced by per-episode rewards computed as the difference between the return of næ and the antagonist policy. However, one issue is that the antagonist policy will also initially struggle for parameters 0 that are hard but feasible, resulting in both policies being unsafe and resulting in regret of O. This does not yield any learning signal to the antagonist policy and results in a bad approximation of the regret.
FARR (Lanier al., (022): FARR approximates the regret directly by using PPO to solve for the optimal policy मे for each parameter 0 for which the regret is evaluated on. This is computationally expensive for tasks where converging requires a large number of environment interactions. Moreover, in cases where the parameter 0 is very challenging, it is possible that PPO returns suboptimal solution, especially if learning to be safe for 0 normally requires some type of curriculum learning or generalization of knowledge from easier parameters. Hence, in most realistic scenarios where there are constraints on computational resources, the quality of the regret approximation directly trades off with the number of parameters that can be evaluated.
ACCEL (Parker-Holder et al. 2022): ACCEL approximates the regret using the positive value loss (Rutherford et al. 2024). However, noted by Rutherford et al. (2024), this unable reliably
28

---

Published as a conference paper at ICLR 2026
identify the frontier of learning and often prioritizes parameters that already solved. The cause hypothesized by Rutherford et al. (2024) to stem from inaccurate value estimation. Moreover, in contrast to typical PPO implementations that only have short rollout length before updating the policy and value function, every rollout of ACCEL has the maximum episode length f This requirement drastically reduces the number of policy updates that happen for fixed budget of environment interactions.
SFL (Rutherford al., 2024): Instead of using the regret, SFL computes learnability metric, defined as p(1 - p) for the success rate p. However, this means that the learnability is set to 0 for parameters that are currently not successful, which is problematic for difficult parameters that are still feasible.
Interestingly, FGE can be interpreted from the lens of UED's regret analysis. Since tackle reachability problems with deterministic dynamics, we know the value of the problematic second term in (81) for any parameters / that were observed be safe any point during training. Consequently, the feasibility classifier we learn in Section [.1 can be viewed as privileged approximation of the second term in (8T). Specifically, we take to be 0 for parameters predicted to feasible by
classifier à and I otherwise. Hence, FGE's best-response procedure in Section 4.2 can be seen as form of sample-basee regret maximization over
F FEASIBILITY GUIDED EXPLORATION PSEUDOCODE
We provide the pscudocode for FGE in Algorithm
Algorithm 2 Feasibility-Guided Exploration (FGE)
Initialize the rehearsal buffer Dæ.u = (
2: Initialize the feasible buffer DD- =
3: for t= 0, 2.... do
4: Collect set of trajectories using the current policy but with mixture of reset distributions
(Algorithm 3)
5:
Update the policy and value function with any on-policy RL algorithm PPO) PPO)
6:
for each completed trajectory do
7:
if Trajectory is feasible then
8:
Add the parameter 0 to Dj if it is not already in Dj end if end for
9:
10:
Train the feasibility classifier qu (fl0) on a mix of Dj and the most recent rollouts
12:
Use the most recent rollouts to update the policy classifier (f(0)
13:
Compute the best-response 0* = arg minge P. (f)
14:
Update the rehearsal buffer with Do Do, )0*)
15:
end for
13322296111
cd009f8c01
examples/maze_p1r.pp
29

---

Published as a conference paper at ICLR 2026
Algorithm 3 Reset Distribution for FGE
Require: Pbase the probability of sampling 9 from the base distribution p(0)
Require: Pexplores the probability of sampling 8 from the explore distribution
Require: Prehearse: the probability of sampling 0 uniformly from the rehearsal buffer Dn,
Require: Prehearse + Phase + Pexplore
Let u 1))
2: 0 S < Pbase then
3: return 0 p(0)
4: else if 0 I Phase < Pexplore then
5: return I (f 1 < B)
6: else
7:
return 0 Do.t uniformly
8: end if
G EXPERIMENT DETAILS
G.1 DEFINITION OF METRICS
In this subsection, describe how the coverage metrics plot computed.
Let 5m be a binary random variable that if method m safe for the (random) parameter 0 and 0 otherwise. When computing the metrics, assume 0 ~ p(0) is sampled from the base distribution. For example, given a random 0, SDR = DR is safe for and otherwise.
Let aBR denote the binary random variable that is if DR is safe for 0 for any seed.
Also, let Sany denote the binary random variable that is 1 if, given random 0, any method (including all ablations), over any seed, can keep 9 safe.
We compute the Safety Rate, Coverage Gain DR, Coverage Loss vs DR and Unique Coverage as conditional probabilities, which we describe in detail next.
Safety Rate with 5_ = 1.
This is the normal safety rate, conditioned on if parameter is estimated to be feasible
SafetyRate,, p((m
9any 1).
(82)
other words, there exists at least one method where parameter safe, then for sure feasible and we include it in the feasible set.
Coverage Gain vs DR This coverage metric tracks the increase in coverage over for some method over DR. Let CovGain, denote the coverage gain DR for method Then,
CovGain = p(sm = 1 Sayy 1, 1 = 0).
(83)
In other words, we consider the parameters that are safe by method and are safe by at least one method, but DR is unable to make safe for any seed. Higher coverage gains are better. A value of 1 means that 9. Sany. i.e., any 0 that safe by least one method also safe by m. value of means that does not render any additional @ safe compared to DR, union over all seeds.
Coverage Loss DR This coverage metric tracks the loss in coverage over for some method m compared to DR. Let CovLoss, denote the coverage loss DR for method m. Then,
CovLoss = =s = 0| = 1).
(84)
In other words, consider the parameters 0 that are safe by DR for least seed, but is unsafe for method m. A value of means that $ seed DR are unsafe under method m. Lower coverage losses are better. value of means that any that is safe for DR for at least seed, method is also safe.
- Sm. = 0, i.e., all that are safe by at least one
30

---

Published as a conference paper at ICLR 2026
Unique Coverage This coverage metric tracks how well method render parameters safe that no other method can render safe. For a given plot, let M denote the set of methods we are plotting. Let UniqueCov, denote the unique coverage for method m. Then,
Smn - =0)
max UniqueCov = 5m my E.M,MMm
(85)
UniqueCov m UniqueCov
UniqueCov m
(86)
Σ
A value of means that the set of safe parameters for method m is superset of the safe set for all other methods. value of means that it is subset of the union of the safe sets for all other methods.
For all plots, we plot the error bars with the 50% percentile error, i.e. the black bar represents the .25 and .75 quantiles over seeds.
G.2 DETAILS ON BASELINE METHODS
In this subsection, we provide more details on how we implement each of the baseline methods used in the experiment section.
All methods solve
13) and use the same indicator function reward function defined in 13 i.e.,
no (#(8) - (hp(s)) > 0),
(87)
and terminate when unsafe (i.c., he 0), differing only in the reset distribution over the parameters. Specifically, let p(0) denote the reset distribution such that the (13) is written as
Too
(1h0(8)) > 03}
max Eg-p
(88)
k=0
Then, each baseline method only changes the choice of p. which can potentially change over training.
DR (Tbii al., 2017) We take p to be the uniform distribution over θ
VDS (Zhang et al. 2020b) We train ensemble of RL value functions, and take p to be the probability distribution proportional to the standard deviation, or "disagreement", among the value function predictions.
PLR (Jing et al. 2021b) We take the 0 to be the "level" in (Jiang et al. 2021b) and use their
scheme for choosing the level as p.
RARL (Pinto et al., 2017) While the adversary in RARL outputs action each timestep, construct variant where the adversary solves a one-step RL problem and only outputs the parameter 0 at the initial timestep.
UED baselines (PAIRED (Dnnis et al. 2020), FARR (Lanier et al. 2022), ACCEL Parker-Holder et al. 2022), SFL (Rutherford al. 2024)) We follow their setup exactly our problem maps exactly to the PAIRED problem, with our parameters matching up with the UED "environment" or "level" B.
G.3 ENVIRONMENT DETAILS
We provide more details for each of the environments visualized in Fig. Leyeev
In the ToyLevels environment (Fig. 24), the agent spawns the top of the map and falls down at a constant rate. The state space S composed of the coordinate the agent. The control space A {0, 2) corresponds to set of three controls: shift left by constant shift right by
31

---

Published as a conference paper at ICLR 2026
Lasy
Hard
Inf.
Figure 24: ToyLevels.
Figure 25: Dubins.
or do not shift. The goal of the agent is to reach the green region at the bottom of the map while avoiding the red regions. We split the spawn states the top of the map into three levels: Easy, Hard, and Infeasible. The easy region is spacious, and the agent is likely to reach the bottom even with suboptimal controls. The infeasible region impossible to reach the bottom because there an obstacle in the way. The hard region is narrow and contains a disturbance region that pushes the agent toward the right at a constant rate. We consider this hard because the agent must learn to move to the left of the disturbance, then output constant sequence of "shift left" actions in order to reach the bottom safely.
We define & E R as the x-position of the agent's 's initial position, and set the base parameter distribution p(0) such that the agent resets uniformly in the easy region 90%, hard region 0.01%, and infeasible region 9.99% of the time. Intuitively, the hard region represents set of initial conditions that extremely rare and requires more samples to solve, and the learned policy is slightly different from that requires for other initial states.
Dubins. The Dubins environment (Fig. 25 requires an ego car to drive around a two-lane circular track. There are two other cars driving at constant angular velocities, and the ego car must avoid colliding into either of them. The static parameter space E IR is defined by the other cars' angular velocities. We set the base parameter distribution p(0) to be uniform in
11
L22 40
Given the environment circular track's inner radius Tinner and lane width Whaee the outer radius is computed
Touter
Tinner
2Wlnne + 2e
where is small offset applied between the vehicle and each lane boundary. Both ego and other cars are modeled as discs with radius Vear The ego car is initialized on the inner lane on the right side of the track, and other from the left side of the track.
are initialized at staggered angular positions across different lanes starting
The state space S C 1 is vector representing the x-coordinate, y-coordinate, heading, and linear velocity. The control space (Asteery Aace E A C R² is the steering and acceleration. Angular velocity of each agent computed
2v
Astee
(Tinner + Touter 3/2
and the ego velocity is updated
f- clip(1 + assce dt, Unnin Vmaxx)
Hopper. The Hopper uses the MuJoCo mjx simulator Todorov et al. (0012) and uses the Hopper model from the Deepmind control suite Tassa et al. (2018) (Fig. 26 For this environment. the state space S C R consists of the joint positions and velocities, while the control space A B controls the torques for each of the four joints. We increase the gear ratios from [30, 40, 30, 10) to [30, 50, 50, 100) to enable the Hopper robot perform more acrobatic maneuvers such backflips.
32

---

Published as a conference paper at ICLR 2026
Figure 26: Hopper.
Figure 27: Cheetah.
Note that this is different from the hopper in OpenAl gym Brockman et al. (2016) as it has four controllable joints instead of three and has nose.
We additionally add two floating platforms (in blue in Fig. 26). Safety is defined as the z-position of the hopper staying above height of 0.9, as well as all parts of the hopper except the foot (c.g.. thigh, pelvis, torso, nose) not colliding with the two floating platforms.
The parameter
controls the initial x p-soitioo of the hopper.
Cheetah. In Cheetah also uses the MuJoCo mjx simulator Todorov et al. (2012) and uses the Cheetah model from the Deepmind control suite Tassa al. (2018). In this environment, the cheetah must keep its head (visualized with a white circle in its head) between the red boundaries (Fig.27). It sits on a floating platform (grey) that moves up and down regular intervals. the platform moves down far enough, the cheetah must stand on its hind legs to remain safe, as shown in the Fig. 27
For this environment, the state space S 1 consists of the joint positions and velocities, while the control space A C R controls the torques for each of the six joints.
The parameter
controls the force applied to the moving platform.
FixedWing. In FixedWing, the goal is to perform the extended-trail exercise used to practice formation flying for fixed-wing airplanes (Team RV (2002) (Fig. 28). Specifically, there are two aircraft named lead and chase. We let the lead aircraft fly in a circular trajectory. The goal of the chase aircraft, which is to be controlled, is to stay within extended-trail cone, which defined with the following specifications:
The distance between the airplanes should be within 500 ft and 1000 ft.
The chase airplane should be between 30 to 45 degrees the lead aircraft, measured in the lead aircraft's frame
These specifications result in the extended-trail cone taking the shape of the outer shell of frustrum.
We use the jax model of the aireraft from So & Fan (2023) Heidlauf et (2018) which is based off Morelli (995);; Stevens & Lewis (2004). The state space S S R composing of the state-space of the two aircraft each with dimension IR and the guidance controller of the lead aircraft with dimension R The control space A S R consists of the stick commands for the chase aircraft. Importantly, the throttle for chase aircraft is set to a higher value (0.4) than the lead aircraft (0.28), which will result in the chase aircraft overtaking (and exiting the extended-trail cone) if the additional energy is not handled properly.
Lander2D. In Lander2D, the goal is for the lander in green avoid colliding with the red circular obstacles, leaving the boundaries of the screen or running out of fuel, and is implemented using Kinetix (Matthews et al. 2025). Consequently, the only way for the lander to survive is land on the blue platform and stay there without falling off. The parameter [To. yol E R describes the initial position of the lander.
We use RGB images for the observation space with a dimension of 64 x 64 x and apply a frame stacking of 3. Moreover, include single scalar observation of the remaining fuel.
33

---

Published as a conference paper at ICLR 2026
Figure 28: FixedWing.
Figure 29: Lander2D.
Figure 30: Lander9D.
34

---

Published as a conference paper at ICLR 2026
Table
Common hyperparameters across all tasks.
Method
Hyperparameter
Value
Neural Network Size
[256, 256]
Activation Function
tanh
All
NN Optimizer # envs (per collect)
Adam
1024
# rollout steps
30
Pbase
0.02
Pexplore
0.88
FGE
Preheasse
0.1
Classifier mix fraction a
0.5
Classifier threshold 3
0.5
PLR
Prioritization
rank
Temperature 3
0.1
Staleness Coefficient p
0.1
VDS
# ensemble
3
RARL
# policy iterations per round
50
# adversary iterations per round
50
PAIRED
# adversary parameters sampled per iteration
32
FARR
# PPO steps for best response computation
50,000
# Parameters added per iteration
ACCEL
Prioritization
rank
Temperature 3
Staleness Coefficient P
0.1
Top K
4
Mutate Scale
0.04
SFL
Update Period T
50
Buffer Size K
100
Sample Ratio
0,9
Lander9D. Lander9D is harder version of Lander2D, where the parameter 0 [wo, yo, fo. platform position, platform width, obsO,. obsO,, obsl 1 E Rº9 describe the posi- tions of the lander, the blue platform and the red obstacles. Everything else remains the same in Lander2D.
G.4 HYPERPARAMETERS
Task-independent hyperparameters can be found in Table I and task-dependent hyperparameters can be found in Table 2
G.S COMPUTE RESOURCES
All training was performed on desktop machine with Intel i7-13700KF CPU and a NVIDIA GeForce RTX 4090 graphics card.
35

---

Published as a conference paper at ICLR 2026
Table 2: Task-specific hyperparameters for each method.
Method
Hyperparameter
ToyLevels
Dubins
Hopper
Cheetah
FixedWing
Policy learning rate
4e-4
5e-5
4e-4
4e-4
4e-4
Value learning rate
le-3
8e-4
8e-4
8e-4
8e-4
# PPO epochs
5000
50000
50000
50000
50000
All
Truncate timesteps
199
1000
1000
1000
1000
Batch size
30
30
15
30
30
Entropy coefficient
le-3
5e-3
2e-3
le-4
le-4
H SENSITIVITY OF SADDLE-POIN FINDING TO VIOLATED ASSUMPTIONS
The assumptions required for the convergence- www.errgencc-guuranteees in Section 4.2 hold, namely a convex- concave objective function and exact best-response oracles, do not hold in our empirical experiments in Section [] Nevertheless, we still see empirical benefits from using such a scheme.
A rigorous theoretical treatment of saddle-pond point finding in the noncomvex-nonconcawe setting is out of scope for this work. However, as a preliminary investigation into this phenomenon, we investigate empirically how sensitive the convergence is when we break the necessary assumptions.
To do so, we consider the following minimax optimization problem
min EE #E(-1.1)
max J(,
(89)
where the function J:R. RR- + R is defined
J(x, 0) (000==(((()) 1(,(1, 0) 0),
(90)
h(T, 1296x + 36x² 216x20 2592
(91)
In this example, the solution to (89) is given by values of (x, 0) inside the red region in Fig. 3la For any value within this region, all values of give J(T, 0) = 0. Since J 0) E {0, 1}. this means that this is the optimal value, i.c., no other choice of results in a lower value of maxøe[-1.1] ( 8)..
Note that J is not convex conccve, and thus violates Assumption Furthermore, we violate the assumption of best-response oracles by approximating the best-response with random samples (as done FGE for the maximization), and control the approximation error by controlling the number of random samples.
Using these approximations, we run the iterates defined by 1II and plot the results in Fig. 31 but without using the regularizer vb. First, despite the function not being convex-concave, the saddle-point finding algorithm still manages to converge to the saddle-point and never diverges no matter how coarse the best-response oracle approximation is. Second, expected, the number of iterations required to converge decreases as the best-response oracle becomes more accurate.
We leave proper theoretical treatment of whether there do exist relaxed assumptions (c.g., nonconvex- nonconcave and inexact best-reponse oracle) as future work.
36

---

Published as a conference paper at ICLR 2026
J(n, 8)
1.00
1.0
0.75
10
Iterations to converge
-0.25
104
0.50
0075
1.00
-1.0
-0.5
0.0
0.5
100 200
300
500
Number samples for best-response oracle
(a) The function to be optimized. Note that is not convex-concave. The saddle-point is located within the red rectangle: for any value of inside this region, all values of 0 result in [m,, 0) O, which is lowest value
(b) Number of iterations to converge as function of the number of samples used for the sampling- based best-response oracle approximation. Note that the iterates never diverged during the experi- ments.
Numerical experiment toy example to investigate the sensitivity of the saddle-point finding iterates to violated assumptions.
Figure 31:
I EXPERIMENT ON BILINEAR GAMES
We investigate whether using finite-sample approximation of the expectation in (2 can empirically still leads convergence scheme on convex-concave problem by testing on the following bilinear game:
max min π 8
(92)
We compare gradient descent-ascent, with exact expectations and (12) with single-sample
approximation the expectation, and plot the last-iterate and average-iterate errors in ı in (32).
Note that gradient descent-ascent does not converge, is well known in the literature (de Montbrun
Renault 2022). Moreover, as guaranteed by Section C the average-iterates of (I2) converge. While using single-sample approximation does slow down the convergence rate, it does seem to still converge empirically. Investigating this theoretically
of scope for the current work.
GDA
10-4
FSE Det
PCE Rand
Error in
10-3
10"
Error in average
10-1
10
10-3
10-4
20000
Figure 32: Last-iterate and average-iterate errors bilinear game.
37


Reviews:
[{"review_id": "CL5SXJM7vD", "reviewer": "Reviewer_CPF8", "summary": "This work proposes a method (called FGE) for solving an initial-states-robust avoid (reachability) problem when feasibility of the initial states set is unknown. To achieve this, it jointly learns (a) a classifier that identifies which parameters (defining initial states) admit a safety-preserving policy and (b) trains a robust policy over that discovered feasible subset by alternating approximate saddle-point / best-response updates and a replay/rehearsal scheme. FGE adapts the reset (initial-state) distribution as a mixture of a base distribution, an explore distribution (via rejection sampling against the feasibility classifier) to expand the feasible set, and a rehearsal buffer to stabilize robust optimisation. The paper then provides theoretical guarantees on the classifier and empirical evidence across several MuJoCo and vehicle/aircraft tasks that FGE yields substantially larger safe (feasible) initial state sets than baselines.", "questions": "Please see the weaknesses above. Mainly:\n\n1. How is the reward function combined with the safety constraints in each environment? If not, how do the current baselines \"see\" the safety constraints?\n2. Could the authors provide reward and/or success-rate plots to show how the improved initial states distributions affect the safety constraints and return maximization (which is the main motivation for this work)?\n3. Why were safe RL baselines (e.g., PPO-Lagrangian, CPO, Sauté-RL, ROSARL) omitted from the comparison? Would including them alter the conclusions?\n4. How does FGE scale to high-dimensional observation spaces (e.g., RGB pixel inputs)?\n5. In the theoretical formulation, both dynamics and policy depend on $\\theta$, but the experiments vary only the start distribution. What happens when environment dynamics change with $\\theta$?\n6. Please fix minor presentation issues throughout (e.g., undefined variables in Eq. 1 and incorrect state variables in definitions).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Theoretical grounding:** The paper gives a clear problem formulation, shows equivalence between its indicator-style objective and a reachability formulation (Lemma 1), and analyzes properties of the learned feasibility classifier (Theorem 1), which underpin the empirical design choices\n- **Clear, practical algorithmic pipeline:** FGE is presented as an algorithm that can be dropped on top of any on-policy method (PPO in the experiments), combining feasibility learning, rejection-sampling exploration, and a rehearsal buffer. This is an appealing and implementable recipe. Algorithmic pseudocode and reset-distribution mechanics are provided. \n- **Several empirical evaluations and useful metrics:** Experiments cover multiple domains (ToyLevels, Dubins, Hopper, HalfCheetah, FixedWing) and a broad set of baselines. The coverage metrics and derived diagnostics (Coverage Gain vs DR, Coverage Loss vs DR, etc) are insightful, especially the Coverage Gain vs DR plots that highlight FGE’s ability to find rare hard initial states. Several ablations (e.g. explore / rehearsal / classifier choices) are also provided and are informative.\n- **Significance for safe RL:** The paper tackles an important, well-motivated gap between reachability/robust objectives and the sampling of initial states in RL: with unknown feasibility of those initial states. The proposed formalization may be a useful addition to the safe-RL literature.", "weaknesses": "- **Notation and readability:** \n  - The abstract and introduction are hard to read because of vague sentences. For example, there is repeated use of terms like \"initial conditions\" and \"initial parameters\" throughout without defining what they mean in the context of this paper (this is only done in the next section). I initially thought \"initial conditions\" meant the initial conditions of an optimisation process, and not the distribution/set of initial states. Similarly, I thought initial parameters meant the initial parameters of the policy neural network, but instead it meant some parameters that define the deterministic dynamics function and specific initial state for an episode.\n  - The problem formulation was also extremely confusing. The paper uses notations from control theory instead of RL (despite the RL focus), and uses the usual RL notations for something else. For example, A and S are not action and state spaces but instead avoid states and safe initial states respectively. Also the value function is not the RL value function (it is not an expectation of cumulative rewards and does not necessarily satisfy the Bellman equations). This just makes readability extremely more difficult than it has to be as the paper progresses. It would have been much clearer if the authors just used standard RL notations and terminology. For example the problem setting seems to be essentially a deterministic contextual MDP with an almost-surely safe RL objective (the agent needs to avoid unsafe states with probability 1).\n  - Minor: Lots of typos and mathematical impreciseness. E.g.\n    - Inconsistent use of = and := (both are being used for definitions). The general convention is using = to claim equality and := to define a function. \n    - Equation 1 $f_{\\theta}$ and $x_0$ are undefined (explicitly say what they are). \n    - Lines 130 and 144 $V(x_0;\\theta)$ should be $V(x;\\theta)$. \n    - $T_{\\theta}$ should depend on $x$.\n    - Line 1348 \"that that\".\n\n- **Scope limitations (method assumptions):** \n  - The approach is only applicable to episodic deterministic environments where the agent have access to the reset function (can chose the initial state of each episode). \n  - The paper defines the dynamics and policy as being conditioned on the parameter $\\theta$ (which is fixed thoughout an episode), but the experiments appear to only change the start sates with $\\theta$. It is unclear what happens when the environment dynamics also change with $\\theta$, as in most contextual MDPs.\n  - It is also unclear how FGE scales to high-dimensional observations such as RGB pixels (the experiments seem to use state vectors). The feasibility classifier and rejection sampling may become challenging in very high-dim $\\theta$ or when observations are high-dimensional. \n\n- **Theory–experiment alignment:** The theoretical setup conditions (convexity/concavity, access to best-response oracles, deterministic dynamics) needed for convergence claims are not satisfied in the empirical setups; while the paper acknowledges this, it should more explicitly describe the gap and, if possible, include analyses showing sensitivity to the violated assumptions (some ablations exist but more explicit discussion is needed).\n\n- **Experimental reporting gaps (Major):**  \n  - The paper does not show standard reward / success/failure training curves in the main results (only coverage metrics are emphasized). For a safe-RL paper, plots showing rewards and success/failure rates (or explicit returns vs constraint violations) would help assess whether policies genuinely optimize task objectives while avoiding unsafe states. A natural concern regarding the proposed method is that the oversampling of hard initial states may compromise policy learning for the actual task.\n  - The paper also does not clearly report the reward functions used in each environment (how goal reward and avoid/safety labels are combined). Given the task descriptions, it seems like it is not just something as simple as the negative of the indicator function in Equation 3. Because the RL objective and evaluation hinge on both avoiding unsafe regions and achieving task goals, omission of explicit reward definitions (and corresponding training plots) makes it hard to judge whether FGE is improving safety, task performance, or both (which is their main motivation/claim for improving the initial states distribution).\n\n- **Baselines and comparisons (Major):**\n  - The choice of baselines leans heavily on robust RL, curriculum, and UED methods (instantiated with PPO). However, no standard safe-RL baselines are used nor cited (such as the ones in the OmniSafe benchmark, like PPO-lagrangian and CPO). More importantly, it is also unclear why the authors did not compare against works that are also focused on maximally safe policies (the setting of this paper), like Saute-RL and ROSARL. \n  - This is important because (a) FGE explicitly uses the safety label / safety function to train its feasibility classifier and to guide resets  (information that many baselines may not use) and (b) the paper's stated goal is to produce safe policies. The absence of safe-RL baselines and citations to directly related safe-RL literature weakens the claims substantially.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EoPPeYMKaQ", "reviewer": "Reviewer_6h75", "summary": "The paper studies a special case of a safety problem in the context of decision making, that the authors call `parameter-robust avoid` problem. The goal is to identify a subset of parameters/initial conditions, which lead to the existence of a safe policy. More specifically, a safe policy is the policy that avoids an unsafe set for *all* parameter values. \n\nThe authors first reformulate the problem as a maximisation of the subset of the parameters subject to constraints: first hitting time of the avoid set and dynamics constraints. Then the authors provide a two-stage algorithm common for robust parameter optimization problems, where they optimize over a safe policy and maximise the support of the safe parameters.", "questions": "* I find it confusing how the authors define reachability. I looked in Tomlin et al 2000, and they have a more common definition of reachability: the set of states that the dynamical system can reach with a control policy. Safety is typically not a requirement for reachability definition. I would also point out that Tomlin et al 2000 study discrete-event systems and their safety definition is not directly applicable to any dynamical system. I suggest simply formally providing a definition of the authors want to study, even if these concepts are inspired by Tomlin et al 2000. \n* Could you explain intuitively Section 4.2\n* How is policy optimization performed? It seems it’s just a gradient descent algorithm. Did the authors use PPO, Reinforce or something custom here?\n* Line 470. This is a bit confusing, I read it first as your method outperforms NSF just marginally. But this doesn’t seem to be the case! I’d recommend a more direct language here: your approach outperforms significantly both NSF and PLR.\n* Small issues:\n    * Lemma 1. Technically, in Lemma 1 equation 4 is not describing a sublevel set for V(x, \\theta) but a level set. \n    * Eq 8. `\\Psi` is not defined", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is solving a complex problem with an interesting solution, and good results\n* There’s a good ablation study\n* The exposition of results is great! I wish more papers would present the results this way. There’s a thesis of what algorithm does better than competitors and the supporting evidence.", "weaknesses": "* The paper’s flow can be improved as it is not easy to understand the problem and the contribution at the first read. While it’s not easy to write a complicated contribution in an easy way, some steps could be taken:\n     * Provide a concrete example of the problem - why is it important? \n     * Try to avoid a bottom up approach to explain the solution as much as possible. When there’s a lot of steps to get from point A to point B, the reader may lose the thread of these explanations. For instance, \n          - I think formulation (6) can be explained intuitively without Lemmas and Theorems, which will give the reader an idea of what’s going to happen. Then explaining how we get there, will give the reader the context (if the reader is interested).\n          - The same is true for Section 4.1. In this case, for example, the regularizer `\\psi` is not even defined when introduced, and it’s not entirely clear how this algorithm would work. Is there iteration, stopping conditions, do we sample different `\\theta` on every step? \n* It would be good to discuss  the problem definition in lines 121-124 in a bit more detail. What are the assumptions on \\Theta, h, f. Why is this problem hard?\n* Why mention saddle-point convergence if the convergence guarantees are very restrictive? The authors mention that actually, but it all becomes confusing fast. I’d suggest having this discussion in conclusions/limitations. \n* What’s the main idea of the parameter set estimation algorithm? Overall, I found this section hard to understand.\n* It is not clear what the RL formulation(s) of the problem is. What formulations were used for baselines?\n* I don’t quite understand what coverage is. It seems this metric depends on the algorithms, which is not easy to understand. Do we have access to the ground truth safe parameters? Is it possible to estimate the ground truth set of safe parameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WKQy5LV3Fi", "reviewer": "Reviewer_kNnA", "summary": "The paper proposes an exploration method for identifying a subset of feasible initial conditions in the parameter-robust avoid problem. The core component is a conservative feasibility classifier that incrementally expands the estimated feasible region from collected samples. The policy is updated by approximately solving a robust policy optimization problem and then executed to gather new data for further classifier refinement. Across MuJoCo and other control benchmarks, FGE achieves solid safe coverage compared to standard baselines.", "questions": "- How do you justify using Theorem 1 although $\\pi$ and $\\rho$ are continuously updated during training? Have you considered an online learning or bounded drift variant of the guarantee?\n\n- Did you observe blind spots caused by false positives in practice? Could uncertainty-based or boundary-based focused sampling (e.g., entropy or ensemble variance) alleviate this issue?\n\n- How sensitive is performance to occasional mislabeling from accidental successes? Have you tried multi-rollout or confidence-based labeling?\n\n- Have you tested FGE in higher-dimensional parameter spaces (at least in 5D)? What are the computational or sample-efficiency challenges as the parameter dimension grows?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- It's interesting how the paper connects feasible-set estimation with robust policy optimization. The formulation naturally follows from the parameter-robust avoid problem and builds a nice bridge between reachability analysis and safe RL. \n\n- The proposed method is empirically validated on several benchmarks, showing consistent improvements in safe coverage. Theoretical claims are somewhat idealized but conceptually sound (see Weaknesses for details).\n\n- Overall, the paper offers a useful approach for safe exploration that explicitly tackles feasibility estimation, even if its scalability to high-dimensional parameter spaces remains unexplored.", "weaknesses": "- Theorem 1’s guarantees (zero false positives and controllable false negatives) hold only for a fixed policy $\\pi$. But, FGE continuously updates $\\pi$ using PPO, making the rollout distribution $\\rho$ and the conditional success probability non-stationary. This violates the theorem’s assumptions and undermines the claimed guarantee. \n\n- Since FGE only samples parameters classified as infeasible, any false positive (infeasible but predicted feasible) is never revisited. This can cause the algorithm to miss meaningful parts of the parameter space and under-cover the true feasible set.\n\n- If a single rollout happens to succeed by chance (e.g., due to disturbances or randomness), that parameter is permanently added to $D_f$. These accidental successes can cause the estimated feasible set to become larger than it actually is, especially near boundaries where the system is sensitive to small variations.\n\n- All experiments use low-dimensional parameter spaces. Performance in higher-dimensional or stochastic settings remains untested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)