Safe Continuous-time Multi-Agent Reinforcement Learning via Epigraph Form
Authors: Xuefeng Wang
Lei Zhang
Henglin Pu
Husheng Li
Ahmed H Qureshi
Venue: ICLR 2026 Poster (2026)
Keywords: Continuous-time
safe multi-agent reinforcement learning
epigraph
Primary area: reinforcement learning
TL;DR: We propose an epigraph-based CT-MARL framework that coping with continuous-time constrainted MDP problems.

Abstract: Multi-agent reinforcement learning (MARL) has made significant progress in recent years, but most algorithms still rely on a discrete-time Markov Decision Process (MDP) with fixed decision intervals. This formulation is often ill-suited for complex multi-agent dynamics, particularly in high-frequency or irregular time-interval settings, leading to degraded performance and motivating the development of continuous-time MARL (CT-MARL). Existing CT-MARL methods are mainly built on Hamilton–Jacobi–Bellman (HJB) equations. However, they rarely account for safety constraints such as collision penalties, since these introduce discontinuities that make HJB-based learning difficult. To address this challenge, we propose a continuous-time constrained MDP (CT-CMDP) formulation and a novel MARL framework that transforms discrete MDPs into CT-CMDPs via an epigraph-based reformulation. We then solve this by proposing a novel physics-informed neural network (PINN)-based actor–critic method that enables stable and efficient optimization in continuous time. We evaluate our approach on continuous-time safe multi-particle environments (MPE) and safe multi-agent MuJoCo benchmarks. Results demonstrate smoother value approximations, more stable training, and improved performance over safe MARL baselines, validating the effectiveness and robustness of our method. Code is available at https://github.com/Wangxuefeng1024/Safe-Continuous-time-Multi-Agent-Reinforcement-Learning-via-Epigraph-Form.

Published as a conference paper at ICLR 2026
SAFE CONTINUOUS-TIME MULTI-AGENT REINFORCE- MENT LEARNING VIA EPIGRAPH FORM
Xuefeng Wang'; Zhang', :englin Pu', Husheng Li Ahmed H. Qureshi" Purdue University
ABSTRACT
Multi-agent reinforcement learning (MARL) has made significant progress in re- cent years, but most algorithms still rely on a discrete-time Markov Decision Pro- cess (MDP) with fixed decision intervals. This formulation is often ill-suited for complex multi-agent dynamics, particularly in high-frequency or irregular time-interval settings, leading to degraded performance and motivating the de- velopment of continuous-ting time MARL (CT-MARL). Existing CT-MARL methods mainly built Hamilton Jaconn-Jaaa-BBBae -eellin (HJB) equations. However, they rarely account for safety constraints such as collision penalties, since these intro- duce discontinuities that make HJB-based learning difficult. To address this chal- lenge, we propose a contimuous-time constrained MDP (CT-CMDP) formulation and novel MARL framework that transforms discrete MDPs into CT-CMDPs via an epigraph- based reformulation. We then solve this by proposing a novel physics-informe infirmedd neural network (PINN)-based actor-critic method that enables stable and efficient optimization in continuous time. We evaluate our approach on continuous-time safe multi muntti-artticle environments (MPE) and safe multi-agent MuJoCo benchmarks. Results demonstrate smoother value approximations, more stable training, and improved performance over safe MARL baselines, validating the effectiveness and robustness of our method. Code is available at this link.
INTRODUCTION
MARL has achieved remarkable success in diverse domains, ranging from strategic games (Samvelyan et al., 2019; Vinyals et al., 2019), multi-robot coordination (Haydari & Yilmaz, 2020; Kuyer et al., 2008), and wireless communication (Wang et al., 2023). These advances demonstrate the potential of MARL as powerful framework for solving complex cooperative and competitive decision-making problems. Despite these achievements, most existing MARL algorithms are for- mulated in discrete time and fundamentally rely on the Bellman equation (Bellman, 1966). This formulation often assumes fixed time intervals between decision steps, which is adequate in set- tings where the decisions naturally occur at uniform time intervals. However, this assumption not well-suited for complex high-frequency domains such as autonomous driving (Kiran et al., 2021; Chen et al., 2021), financial trading (Shavandi Khedmati, 2022), where continuous-time time control. In such cases, discrete-time RL often struggles to learn accurate policy (Doya, 2000; Mukherjee & Liu), as fixed-step discretization fails to represent non-uniform temporal dynamics, resulting in degraded performance and unstable learning (Tallec al., 2019; Park al., 2021; De Asis & Sutton, 2024). These limitations highlight the necessity of developing an alterna- tive framework beyond discrete-time Bellman equations, which is compatible with CT-MARL.
requires
Recent studies (Wang et al., 2025) have explored the HJB equations to solve CT-MARL problems. The HJB can be viewed the continuous-time analogue of the Bellman recursion, where the value function is characterized as the viscosity solution of nonlinear Partial Differential Equation (PDE) (Shilova al.. 2024). In practice. PINNs have emerged as common approach to approximate HJB solutions: they train neural networks to minimize HJB PDE residuals and leverage gradient- consistent signals for policy improvement (Mukherjee Liu; Meng al., 2024). This formulation eliminates the need for fixed time discretization and enables MARL to operate in continuous-time
Equaal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2026
domains. However, in safety CT-MARL settings, state constraints (e.g., when they are treated as collision penalties) introduce value discontinuities, making it difficult for -JB-based PINNs to ap- proximate the value functions accurately (Zhang et al., 2024).
To address these challenges, we first cast safe CT-MARL as CT-CMDP with explicit state con- straints. We then introduce revised epigraph reformulation that augments the system with aux- iliary state 2, transforming the discontinuous constrained values into a continuous form suitable for PDB-based learning. On top of this reformulation, we adopt an actor-critic framework to learn val- ues and policies under continuous-tine state constraints. Specifically, we improve epigraph-based training by integrating the inner and outer optimization into a unified scheme. At each rollout, we compute the optimal auxiliary state z" and uses it directly for training, while keeping all networks s-independent. This design avoids the noise of random z sampling, yields more accurate policy updates, and eliminates costly root-finding at execution.
Our main contributions are summarized as follows. (1) To the best of our knowledge, this is the first work to explicitly incorporate state constraints into the formulation of CT-MARL. We introduce an epigraph-based reformulation to bounds discounted cumulative cost and state constraints within a unified objective, effectively transforming discontinuous values into continuous ones. (2) We design an improved epigraph training scheme that integrates inner and outer optimization, providing more stable learning signals and removing the need for costly root-finding algorithms. (3) We prove the existence and uniqueness of viscosity solutions for epigraph-based HJB PDEs, providing theoretical
support for our method. Extensive experiments adapted continuous-time safe MPE and multi-
agent MuJoCo benchmarks further demonstrate that our approach consistently outperforms current safe MARL methods.
2 RELATED WORK
2.1 CONTINUOUS-TIME REINFORCEMENT LEARNING
Discrete-ting reinforcement learning (DTRL) often performs poorly in continuous-ting environ- ments, particularly when decision intervals irregular (Tallec et al., 2019; Park et al., 2021; De Asis & Sutton, 2024). Consequently, continuous-time reinforeement learning (CTRL) has re- ceived growing attention 2021; Wang et al., 2020; Bradtke& & Duff, 1994; Jia & Zhou, 2022a;b). Most existing studies focus on the single-agent setting, proposing various approaches for value function approximation (Mukherjee & Liu; Wallace & Si, 2023; Lee & Sutton, 2021). For example, Mukherjee & Liu employ PINNs to approximate the value function and guide a PPO-based policy update, while Jia & Zhou (2022b) address stochastic dynamics through Martingale loss designed for stochastic differential equations. In contrast, research on CT-MARI. remains limited. Prior works (Luviano & Yu, 2017; Jiang al., 2023) have considered multi-agent problems in continuous time, but largely in application-specie contexts rather than as general-purpose algorithms. The study in Wang et al. (2025) represents the first systematic attempt to design CT-MARL methods, combining PINNs with value gradient itera- tion to improve value approximation and performance. However, these approaches still inherit the limitations of PINNs that they can only approximate smooth value functions and therefore neglect safety constraints.
more suitable framework for such problems (Doya, 2000; Yildiz al.,
2.2 MULTI-AGENT SYSTEMS WITH SAFETY CONCERNS
Multi-agent scenarios often raise critical safety concerns, and directly learning under combined reward and safety signals poses significant challenges. A number of studies have explored safe MARL frameworks to address these issues (Gu et al., 2023; ElSayed-Aly et al., 2021: Gu et al., 2024; Shalev-Shwartz et al., 2016). For instance, Chow et (2018) employ primal-dus methods to enforce safety constraints, while Althoff et al. (2019) adopt trust-region approach. Gu et al. (2021) introduce MACPO and MAPPO-Lagrange, which provide theoretical guarantees for both monotonic reward improvement and safety constraint satisfaction. In addition, Zhang et al. (2025b) leverage epigraph forms to formulate multi-agent safe optimal control problems, improving stability during training. However, these approaches are primarily developed in discrete-time settings, which limits their ability to capture continuous dynamics. Some efforts have incorporated safety into continuous-time multi-agent systems (e.g., Tayal et al. (2025)), but they assume fully known system
2

---

Published as a conference paper at ICLR 2026
dynamics and rely optimal control algorithms, significantly restricting applicability. In more realistic scenarios, where dynamics are only partially known or highly complex, such methods fail to provide practical solutions.
Existing methods remain limited in handling discontinuities and safety constraints in CT-MARL. Discrete-time safe MARL algorithms provide theoretical guarantees but do not naturally extend to continuous dynamics, while continuous- timmm approaches struggle with discontinuous value func- tions. To address these challenges, propose an epigraph-based reformulation that unifies safety constraints and standard cost functions within single objective, enabling principled and stable learnnng in CT-MARL.
METHODOLOGY
In this section, we present our epigraph-basee PINN actor-critic iteration (EPI) for solving CT- MARL with state constraints. 1) We first formalize the learning problem as CT-CMDP. Sec- ondly, 2) we reformulate the CT-CMDP using an epigraph form. By introducing an auxiliary state z to augment system states, this reformulation converts discontinuous value functions into con- tinuous ones. Building this reformulation. 3) we develop actor-critic learning architecture that aligns with the epigraph inner-outer optimization scheme. Specifically, the outer optimiza- tion computes the optimal auxiliary state 2" along the rollout, ensuring that the critic captures the tightest feasible trade-off between return and safety. Based on this, the inner optimization trains the critic using PINNs, which jointly update the return and constraint networks together with = to approximate the epigraph-based value function. This stabilized critic then serves as the foundation for actor training: we derive an advantage function consistent with the epigraph-bascl HJB PDEs, which provides the key learning signal for policy improvement.
3.1 PROBLEM FORMULATION
3.1.1 CONTINUOUS TIME CONSTRAINED MARKOV DECISION PROCESS
We consider a CT-CMDP problem, formally defined by the tuple
M= (x, (4,) J, f, (4) (tx (talk20,7).
(1)
RRT is the joint control space for N agents. The system evolves according to time-invariant nonlinear dynamics z(t) f(((t), u(t) with x00)= = xp. where f X' U + X. Each agent applies a decentralized policy T, X x [0, 0o) + Ur and the joint policy is denoted as (#1...., N. All agents share the non-negative cost function / Σ 1==4 4. where 4 x U; - IR is the independent cost function of agent i. The system is further subject to state-dependent safety constraints specified by a function c: X' R. with the feasible set defined as = {x E X c(x) < 0). Control actions are updated at irregular decision times (4.)220 with strictly positive intervals the discount factor. Throughout the paper, we assume that U4; is compact and convex, I and e are Lipschitz continuous, and I, is Lipschitz continuous and bounded. The joint objective is to minimize the cumulative cost under joint control input u (ui...., uN) subject to state constraints c(x)
where X' S R the global state space, and 4 Ui x ... UN S. R
12+1 L2. E (0, 1] is
w(x) = min I 4 7- ((x(T),((T)) dr well
s.t. c(x(T)) S0 2
3.1.2 EPIGRAPH REFORMULATION
The value becomes discontinuous (Altarovici al., 2013) when state constraints are violated in Eq. 2, which hinders the convergence of +JBBbaseed PINN training. To address this, we leverage an epigraph reformulation that converts value in Eq. 2 into continuous representation.
Definition (Epigraph Reformulation). We introduce an auxiliary state variable 2(t) E R to refor- mulate Eq. 2 using the epigraph forms. Here, follows the dynamic &(t) -I(x(t), u(t)) In y z(1). Therefore, the auxiliary value function is defined as
max c(x(T)), 1.00 7- ((-(() (((((()),u(())en) 2) - [E]t,,o)]
V(r, 2) = min max
(3)
wEll
3

---

Published as a conference paper at ICLR 2026
Lemma 3.1 (Value Equivalence). Suppose the assumptions in Sec. 3.1.1 hold. For all (t, [0, 0o) x X x I, the constrained value v and auxiliary value V are related by
((x) min{= E IR (z, 2) < 0}.
(4)
Here, the sub-zero level set of auxiliary value V becomes the epigraph of the constrained value v. The proof is listed in Appendix A.1.
Lemma 3.2 (Optimality Condition). For all (t, =) E [0, 00) & R, consider small enough h > 0, the auxiliary value function V satisfies
V(z, =) min max
max [E([,,]+))]
(5)
The proof is listed in Appendix A.2.
Theorem 3.3 (Epigraph-based HJB PDE). Let V x R - be the auxiliary value function
defined in Eq. 3. Then V is the unique viscosity solution of the following HJB PDE for all (t,z, [0, 00) x X'
max
max c(z) - V(I, z). min H( V V=v =0, TE(t,0)
(6)
wel
where H(x, 2, V.V, V) is Hamiltonian and satisfies = V V. f(x V.V. u) + y.
and optimal control u² arg min H. The derivation proof is provided in Appendix A.3.
3.2 EPIGRAPH LEARNING FRAMEWORK
Critic Learning
Oouge
Qtata
Ava
Data Collection
wae (y) (77(--)--
fac oae
Innor Optimization
Laurning
Figure Overview of the proposed epigraph-based CT-MARL framework. The pipeline begins with data collection, where individual agent rollouts are aggregated into centralized rollout X' for the training: the outer optimization computes optimal z² to balance discounted cumulative cost and safety constraints; the inner optimization corresponds to critic learning, where return networks (z)) and constraint value networks Vo (x) are optimized jointly with the optimal auxiliary state ="; and actor learning leverages the advantage function to improve policies.
As illustrated in Fig. 1, our framework integrates the epigraph- based inner-outer optimization (Zhang et al., 2025b) into the actor-critic paradigm. The outer loop updates 2" along the rollout by solving Eq. 7, ensuring that the critic trained with the minimal that simultancously satisfies both costs and safety constraints.
min s.t. min max supc(((TT), f. (==(( -2 50.
r2t
In the inner loop, the critic is trained as follows: the return and constraint value networks (VT and & (x)) are optimized using 2n to approximate the auxiliary value function (xx This stabilized critic subsequently supplies the learning signals for decentralized actors, which map lo- cal observations to continuous-time policies under the standard centralized training decentralized execution setup (Foerster et al., 2018; Lowe et al., 2017). We next describe the revised outer opti- mization in detail, focusing solving the optimal auxiliary state 2" that trades off discounted cost against safety violations without costly root-finding algorithms (So & Fan, 2023; So et al., 2024; Zhang al., 2025b).
4

---

Published as a conference paper at ICLR 2026
3.2.1 REVISED OUTER OPTIMIZATION
We seek the minimal such that the epigraph-based value V remains non-positive, as defined in Eq. 4. Using the return and constraint value network learned by the critic, the optimal auxiliary state can be found by solving for the minimal feasible solution
min E IR max (V2 V (x), (x) 2) =
(8)
where return value network Vi (x) that approximates the discounted cumulative cost Je (/-+((((()) TT)) dr, and constraint value network V (zo)((r)) represents the violation for worst- case future constraints sup/2: sup,). c(x(T)).
In previous epigraph formulations (Tayal et al., 2025; Zhang et al., 2025b), the outer problem is solved during the execution phase: is sampled along the rollouts during training, and x" is com- puted at execution time via root-finding (Stoer et al., 1980). This design has two drawbacks in CT-MARL: (1) the random sampling of introduces nonstationary noise that destabilizes the up- dates of actor and critic and further leads poor convergence; (2) execution, root-finding must be performed at every step, which is computationally expensive and often incompatible with real-time requirements. In contrast, we design the return and constraint value networks as functions of the states solely. We then integrate the outer optimization into actor-critic training: for each episode, 2" is computed using the current learned value V along the predicted rollout. The actor is then trained against a z-independent critic, yielding a z-independent policy π(x). This design ensures stable actor training, and enables real-time deployment by eliminating the need for root-finding during execution. Since the critic's value networks are 2-independent, the outer optimization is simplified scalar search for z", which adds negligible cost model training.
3.2.2 INNER OPTIMIZATION WITH CRITIC LEARNING
The inner optimization is responsible for updating the PINN-based critic networks. Given task- dependent range [zming 2max]. : the outer optimization computes 2", which then clipped to this range (i.e., z" + min [mnummxx""" 2 Zmin zmax)) before being used to train the critic module. The critic consists of two value networks: a return value network (x), and a constraint value network veons()). (z)). Together with the computed 2 from Eq. 8, these define the composite epigraph-based value function
2zz,²) max Vgoue(z)) (z), Viret (z)
To ensure stable and accurate training, we employ three complementary losses.
(i) Residual Loss. We use PINN architecture (Mukherjee Liu) to approximate the value function governed by apigraph-based HJB PDEs, and introduce a residual loss that penalizes violations of the corresponding PDES
CReidual = max (c(x) - V. min V f(x, u) 0, V 1(x,1((,0, u) + In |
(10)
NEll
(ii) Target Loss. In standard PINNs, a boundary loss is combined with the PDE residual to ap- proximate PDE solutions (Cai et al., 2021; Raissi et al., 2019). In the infinite-horizon setting, however, no boundary condition is available, and training the critic only on residuals is insuffi- cient: optimization may converge, but incorrect PDE solutions (Wang al., 2022). To address this, we add a rollout-based larget loss that measures the discrepancy between the epigraph- -pased value approximation with a numerical target defined by Eq. 3. For each episode, the current value V generates closed-loop trajectory (a(T), u(T) T=ti
- from this trajectory we construct the tar- get (ag 2) max (maxx c(x(r)), Je - ((x(T)) u(T)) dr 2*} and minimize the squared error
Target
Vigr (x,)))
x²) - max (oous(z) (x), (ie((x) z"
(1I)
(iii) Value Gradient Iterations. Standard PINN training in multi-agent settings often struggles to approximate accurate value functions, primarily because the learned value gradients are inaccurate or unstable (Wang et al., 2025; Zhang et al., 2024). The VGI techniques (Eberhard et al., 2025; Wang et al., 2025) are designed to enhance the quality of learned value gradients. In our frame- work, accurate gradients V. Vxx) are crucial for precise value approximations, which in turn affect
5

---

Published as a conference paper at ICLR 2026
actor learning and ultimately determine the quality of the resulting policies. To establish the the- oretical basis of this module, we follow Theorem 3.4 in Bokanowski et al. (2021) and Theorem in Hermosilla & Zidani (2023). Here, u, is the optimal control imput in this paper.
Vn, Vxr,) ve (x(x)))(,,, we) + (1 x(==))c(z;)) + A sctar V (F(+ V2, I ue),
(12)
where the x(x2) 1f 1 (2.) 24 (x))((2)) As shown in Eq. 12, the value gradient satisfies a recursive relation coupling the local cost gradient with the backpropagated dynamics term. The overall critic objective is weighted sum of the three losses as
Ccritic Ares RResidull AegrCTarget + Avgi
(13)
where the weights (Ares: Augt Augi) are selected to keep the losses on comparable scales and are determined via grid search.
3.2.3 ACTOR LEARNING
After introducing the inner-outer optimization for critic learning, we turn to the actor learning. We first define the epigraph-bases Q-function, which used for deriving policy update rules.
Definition 2 (Epigraph-bare bassed Q-function). Following the definition in (So & Fan, 2023), for any state-action pair (Ire, 44) and auxiliary state 2t, the epigraph-based Q-function is defined
Q(xe; 25, ue) = max (c(=z), 7 VV((*tth }
(14)
where 244hh and 24th the states and optimal auxiliary state at + h. respectively. h is short time interval.
Lemma 3.4 (Epigraph-bard based advantage function). The epigraph- advantage function
A(rx, =i, ue) = Q 25, we) V (Ie,
(15)
equivalent epigraph-based HJB PDE when h->0
A(r,, =7; u.) = max (axx))()) V (ir,, =)), Va, V [(x,, ue) 0z2 V 1(x,, u.) + Inn V). (16)
In practice, evaluating the epigraph-base basedd advantage in Eq. 16 requires knowledge of the true dy- namics (2, u) and cost function [[2, u). Since these quantities generally unknown in model-free reinforcement learning, we replace them with neural networks that are jointly trained alongside the actor. The derivation of the epigraph-based advantage function is listed at Appendix A.4.
Dynamics and Cost Networks. To assist with the policy training, we employ two neural networks: dynamics network L& (2r, u, A,) that predicts the next state x' given the current state-action pair, and cost network ls (x, u, Ae) that estimates the instantancous stage cost. Both models are trained via supervised regression using observed transitions (z, u, x', ()) from the environment. Specifically, the training losses are
LE) [[fe(x, u, A.) - x'112 (oma( 11, u, A.) 1(x, w)]l
(17)
where x' is the observed next state and ((z, u) is the empirical cost signal. Equivalently, the dynamics learning can be interpreted as approximating the continuous-time derivative dynamics (Sg u, A.) A/
Actor Update with Learned Models. By substituting V(a, 2"), fs and Lo into the epigraph advan- tage expression Eq. 16, we obtain differentiable surrogate
Ao z². u) = max V. V.V fe 0. 16 In V}.
(18)
The actor (u( 2) updated by minimizing the expected surrogate ad advantage
Lucar ( Eg~:Na. Ao 2", ])
(19)
where Xy is the sampled data along the rollout.
decentralized-exeeutte structure: each agent's 's actor Ty (o, At) takes its local observation o as input, while the training signal is derived from the state The overall training pipeline is summarized in Algorithm in Appendix B.
Specifically, adopt caitralized-tranning

---

Published as a conference paper at ICLR 2026
Cumulative Reward
Cooperative Navigation
Cumulative Reward
Gooperative Predator Prey
Cumulative Reward
Corridor
-100
-100
-400
-200
-300
-600
-400
-300
-800
-500
400
-1000
5000
10000
5000
10000
5000
10000
Episode
Episode
Episode
Cumulative Reward
Formation
Cumulative Reward
Line
Cumulative Reward
Target
-100
-50
-200
-100
-200
-150
-400
-300
-200
-400
-600
-250
5000
10000
5000
10000
5000
10000
Episode
Episode
Episode
EPI
EPPO
SAC-Lag
PPO-Lag
MACPO
CBF
Figure 2: Overall results for adapted MPE environments.
4 EXPERIMENTAL RESULTS
We organize our empirical study around the following research questions: Q1. How well does our method balance discounted cumulative cost and constraint satisfaction compared state-of-the- art baselines? Q2. How does the different loss component in critic learning contribute to stable training and accurate value approximations? Q3. How does performance change when training with versus without the epigraph reformulation? Q4. How sensitive is the epigraph formulation to the choice of the auxiliary variable during training? Q5. How robust is the method under stochastic disturbances, and how does performance degrade under model-mismant noise? Q6. How does the performance change under different discretization resolutions At?
4.1 BENCHMARKS AND BASELINES.
To evaluate approach under continuous-time environments with safety constraints,
sider two adapted benchmarks: the safe continuous-time MPE (Lowe et al., 2017: Wang et al., 2025) and continuous-time Safe MA-MuJoCo (Gu al., 2023; Wang et al., 2025). In MPE, we design several scenarios including Corridor, Formation, Line, Target, Simple Spread, and Cooperative Predator-Prey. These tasks typically place agents in environments with obsta- cles and require them to avoid both collisions with obstacles and collisions with other agents while navigating or pursuing their objectives. MuJoCo, we adapt several scenarios such as Half Cheetah and Ant into continuous-time versions and introduce randomly placed walls ob- stacles. The agents must coordinate to move forward efficiently while avoiding crashing into walls, ensuring that the learned policies account for both locomotion and safety considerations. Lastly, we design a di- dactic example based constrained coupled oscilla- tor, which admits analyt- ical ground-truth solution for both value functions and ac- tions. This example provides transparent testbed di- rectly validate the correctness of our learned critics against exact solutions. Full details of the agent setups, metrics, state and action spaces. and cost specifications provided in the Appendix C. We compare approach EPI with MACPO (Gu et al., 2021), MAPPO-Lag (Gu et al., 2021), SAC-Lag (Haarnoja al., 2018), 10, EPPO (Zhang al., 2025b) and CBF (Zhang et al., 2025a). The first three represent
Conctrannt Wonteant Visstion Rate
30
Corridor
Cooperative Predator Prey
Cooperative Navigation
10
10
28
10-
2
0
0
14
Cost
Cost
Cost
EPI
+
EPPO
SAC-Lag
PPO-Lag
MACPO
+
CBF
Figure 3: Performance of constraints and cost over MPE settings.

---

Published as a conference paper at ICLR 2026
Cumulative Reward
HalfCheetah- 2x3
Cumulative Reward
HalfCheetah-22
Cumulative Reward
HalfCheetah-6xl
20-
20
0
-20
-20
2500 5000
7500 10000 Episode
2500
5000 Episode
7500 10000
2500
7500 10000 Episode
5000
Cumulative Reward
Ant-2x4
Cumulative Reward
Ant-2x4d
Cumulative Reward
Ant-8x1
20
-20
-20
-10
-20
2500 5000 7500 10000 Episode
2500 5000 Episode
7500
10000
2500
5000 Episode
7500 10000
EPI
EPPO
SAC-Lag
PPO-Lag
MACPO
CBF
Figure 4: Overall results for adapted multi-agent MuJoCo environments.
the most widely used families of safe MARL algorithms: trust-region based methods (MACPO) and Lagrangian based methods (MAPPO-Lag, SAC-Lag). covering both on-policy and off-policy learning. We also include EPPO as an epigraph- based baseline that follows the traditional epigraph optimization framework. We additionally include control barrier function (CBF) baseline, which enforces safety through model-based barrier certificates and is commonly used in safe multi-agent control. Although these algorithms were originally developed in the discrete-time setting, we adapt them to continuous time by equipping their crities with the same PDE residual loss used in method. Since the performance gap between discrete-time time and continuous-tim algorithms has al- ready been well studied (Tallec et al., 2019; De Asis & Sutton, 2024), our basclines focus only isolating the effect different safety mechanisms (trust-region, Lagrangian, or epigraph).
Cumulative Reward
Cooperative Navigation
Cumulative Reward
-50
Cooperative Predator Prey
Cumulative Reward
-100
Corridor
-100
-000
-150
-200
-300
-400
-200
-250
500
-50
-300
5000
10000
5000
10000
5000
10000
Episode
Episode
Episode
Cumulative Reward
Formation
Cumulative Reward
Line
Cumulative Reward
Target
-100
-50
-200
-100
-300
-150
-400
-600
-200
5000
10000
5000
10000
5000
10000
Episode
Episode
Episode
EPI
w/o Residual
w/o VGI
w/o Target
Figure 5: Ablation study of different loss terms in critic network over MPE.
4.2 RESULTS ANALYSIS
In this section, we present systematic analysis of the results, addressing each research ques- tion in turn. Q1. Our method consistently outperforms all baselines across both adapted MPE and MuJoCo environments in Fig. and Fig. 4. We adopt the same reward design commonly used in prior safe MARL works such as MACPO (Gu et al., 2021). Specifically, the reward the combination of the task cost provided by the environment (e.g., distance to the target
8

---

Published as a conference paper at ICLR 2026
in MPE) and the safety penalty provided by the environment (e.g.. collision penalty between agents or with obstacles), as detailed in Appendix C, which directly reflects performance un- der both objectives. In Fig. 3 and 6, each point corresponds to the average performance of one algorithm, with horizontal and vertical bars denoting standard deviations. Since the goal minimize both cost and constraint violations, the lower-left corner of each panel repre- sents the desirable region. These results show that our algorithm EPI achieves nearly the low- cost and constraint violation in every scenarios. Specifically, EPPO often remains stuck suboptimal solutions because randomly samples the auxiliary state instead of using x" for model training, introducing noise that disrupts policy updates and prevents stable convergence. MACPO enforces constraints through a hard trust-region style update, which yields strong violation rejection but tends to be overly conserva-
HalfChestah-2kk
HalfChectah-6x1
Ant-244
Constraint Viatetion
10
15
tive. SAC-Lag and MAPPO- Lag rely Lagrangian laxation, which is known suffer from instability when balancing objectives under tight safety require- ments (Zhang et al., 2025b). CBF achieves reasonable relies on the gradient of a learned barrier function Bxx)). approximation errors in this component can distort the effective safe set and degrade the overall performance.
0.5
1.0
1.0
1.5
2.0
1.0
1.5
2.0
Cost
Cost
Cost
FPI
F990
SAC-Fag
PPO-lag
MACPO
Figure 6: Performance of constraints and cost over MuJoCo settings.
insutraint-violation levels but tends be conservative. The CBF condition
Q2. The ablation results in Fig. 5 clearly demonstrate the importance of each loss com- ponent in critic learning. method compared with its ablation variants across representative continuous-tima MPE tasks. Removing the target loss or the VGI loss significantly degrades performance, whereas removing the residual loss has only a mi- effect. This difference stems from the fact that, unlike existing HJ-based PINN methods (Zhang et al., 2024; Tayal et al., 2025; Cai et al., 2021) that address finite- horizon problems with boundary conditions, our framework targets the infinite-horizon setting where no such boundary conditions available. In this case, the target loss serves as an anchor to stabilize value approx- imations, ensuring that value function Vxx) does not drift arbitrarily, while the VGI loss enforces consistency of the learned value gra-
presents
cumulative reward performance our full
3.5
Fit
High Residual Loss Weight
Target Weight
3.0
Average Distance
2.5
2.0
1.5
0.5
Formation
Line
Target
Figure 7: Weighted loss performance.
dients, which are crucial for both accurate value approximations and policy improvement. In con- trast, the HJB residual loss mainly regularizes the PDE structure, but its role becomes less critical once the value gradients are optimized by VGI. As result, the removal of VGI has a severe im- pact, since inaceurate value gradients directly harm both critic accuracy and actor updates, while the residual loss contributes less critically to overall training stability.
The grouped bars in Fig. report the average distance to the target (lower better) for three MPE tasks (Formation, Line, and Target) under different loss weightings. The balanced setting (EPI) attains the smallest distance in all tasks and shows the tightest variability. Over-emphasizing any single component degrades performance: increasing the target loss weight is particularly harmful on Line (large increase in distance), while overweighting (x20) the residual or the VGI loss also worsens results relative to EPI, though lesser extent. These ablations support using the balanced weighting adopted by EPI.
Q3. We generate one trajectory using EPI and collect the visited states. On these same states, compare the value and policy from three methods: EPI, Ground Truth via the LQR method (details

---

Published as a conference paper at ICLR 2026
in the Appendix C), and ablation without the epigraph reformulation, where the state constraint is
treated as a collision penalty added to the cost function I, making the value function discontinuous.
For Ground Truth, the value is com- puted as the dis- counted cumulative cost. While for the EPI and ablation without the epigraph form, the value is predicted through the trained value network. EPI closely tracks the Ground Truth in both value and actions for both agents, indicating accurate value ap- proximation and stable control policies. In contrast, the ablation without the epigraph form exhibits severely mis-scaled value predictions (we plot it after a X scaling share the same y-axis) and noticeably unstable actions, which in practice are more likely to violate constraints because the discontinuous value function is not addressed by the epigraph form. The poor performance of the ablation without epigraph stems from the discontinuity of the value function when state constraints are directly encoded as hard penalties. Such discontinuities are notoriously difficult to approximate with neural networks, leading to severely mis-scaled value predictions and unstable gradients for policy updates. By contrast, the epigraph reformulation converts the discontinuous penalty into a continuous and smooth upper-bound optimization, which stabilizes critic learning and yields reliable policies.
Value Comparison
Agent1 Action Comparison
Agent2 Action Comparison
0.10
Action (Agent1)
0.5
Action (Agent2)
0.50
wi Eargush 6201
0.25
Value
0.00
-0.5
0.00
-1.0
Ground Truth Epigraph
-0.50
Ground Truch
Rogrash
100
20
30
20
20
Step
Step
Step
Figure 8: Performance with and without epigraph reformulation.
Q4. To better understand how model performance depends on we test two MPE tasks (Formation and Line) under different values of z. Specifically, we train the EPI model with 0.22max, 2",2" 0.2 2" 0.52max) Fig. 9 reports the
{2+ 0.52max: x² sults, where the æ-axis indicates cost and the y-axis denotes the constraint violation rate. Compared with the opti- mal auxiliary state z", us- ing a suboptimal shifts the trade-off between cost and constraint satisfaction, often resulting in either much higher violation rates or larger costs. Specif- ically, a smaller z (e-g.. 2" -0.22max 2² -0.5mmax) significantly increases the violation rate while only slightly reducing cost. Getting back to the epigraph form max[V (z), Virat =], a smaller makes yret(222 2) - z lager than V (x)) so the return term dominates in the epigraph form. As a result, the optimization prioritizes reward improvement while neglecting constraint satisfac- tion, leading frequent violations. In constrast, when larger than (e.g., x* + 0.2 x" + 0.52max), the term /ret( (z, - z becomes smaller than dominate in the epigraph form. This forces the critic and actor to emphasize constraint satisfaction, which reduces violations but increases cost.
Constraint Violation Rate
Formation
Line
100
100
80-
80-
60-
60
20-
8.6
Cost
Cost
0.5
0max
0.2
Figure 9: Sensitivity test of different z choices.
& (r)ns(r), making constraint value
5 CONCLUSION
In this paper, we propose an epigraph-based framework for CT-MARL that addresses the chal- lenges of balancing reward maximization with constraint satisfaction. By reformulating the prob- lem through the epigraph forms, we introduced an inner-outer optimization procedure that enables stable critic learning and effective policy updates. Our design further integrates different losses in critic learning, including target, residual, and VGI losses, to anchor value approximations and improve gradient accuracy in the infinitehhorizon setting. Through extensive experiments in both adapted MPE and MuJoCo benchmarks, we demonstrated that our method consistently outperforms state-of-the- -hrttrrrrt baselines in terms of both cost reduction and constraint satisfaction.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This material is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-24-1-0233. Any opinions, findings, and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force.
ETHICS STATEMENT
This work focuses Decision-making for the continuous-time constrained MDP problems. All experiments are conducted entirely in simulation and do not involve human subjects or personal data.
REFERENCES
Albert Altarovici, Olivier Bokanowski, and Hasnaa Zidani. A general Hamilton-Jacobs framework for non-linear state-constrained control problems. ESAIM: Control, Optimisation and Calculus of Variations, 19(2)337-357, 2013.
Matthias Althoff, Andrea Giusti, Stefan Liu, and Aaron Pereira. Effortless creation of safe robots from modules through pelg--roggaminn and self-verification. Science Robotics, 4(31): caaw 1924, 2019.
Richard Bellman. Dynamic programming. science, 153(3731):34 37, 1966.
Olivier Bokanowski, Anya Désilles, and Hasnaa Zidani. Relationship between maximum princi- ple and dynamic programming in presence of intermediate and final state constraints. ESAIM: Control, Optimisation and Calculus of Variations, 27:91, 2021.
Steven Bradtke and Michael Duff. Reinforcement learning methods for continuous-time markov decision problems. Advances in neural information processing systems, 7, 1994.
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics- informed neural networks (PINNs) for fluid mechanics: A review. Acta Mechanica Sinica, 37
(12):1777-1738. 2021.
Jianyu Chen, Shengbo Eben Li, and Masayoshi Tomizuka. Interpretable end-to-end urban tonomous driving with latent deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 23(6):5068-507) 2021.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained rein- forcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167); 1-51, 2018
Kris De Asis and Richard S Sutton. An idiosynerasy of time-discretization in reinforcement learning. arXiv preprint 24Xiv:2406./443 /495 2024.
Kenji Doya. Reinforcement learning in continuous time and space. Neural computation, 12(1): 219-245, 2000.
Onno Eberhard, Claire Vernade, and Michael Muehlebach. A Pontryagin perspective on reinforce- ment learning. In 7th Annual Learning for Dynamics & Control Conference, pp. 233-244, PMLR, 2025.
Ingy Elyyyed-Aly, Suda Bharadwaj, Christopher Amato, Rüdiger Ehlers, Ufuk Topcu, and Lu Feng. Safe multi-agent reinforcement learning via shielding. arXiv preprint arXiv:2101. 2021.
Lawrence C Evans. Partial differential equations, volume 19. American mathematical society, 2022.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.

---

Published as a conference paper at ICLR 2026
Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang. Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint ar arXhv:2110.02793, 2021.
Shangding Gu, Jakub Grudzien Kuba, Yuanpei Chen, Yali Du, Long Yang, Alois Knoll, and Yaodong Yang. Safe multi-agent reinforcement learning for multi-robot control. Artificial In- telligence, 319:103905, 2023.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning. pp. 1861-1870. Pmir, 2018.
Ammar Haydari and Yasin Yılmaz. Deep reinforcement learning for intelligent transportation sys- tems: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(1):11-32, 2020.
Cristopher Hermosilla and Hasnaa Zidani. Relationship between the maximum principle and dy- namic programming for minimax problems. Applied Mathematics & Optimization, 87(2)33,, 2023.
Yanwei Jia and Xun Yu Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research, 23(275):1-50, 2022a.
Yanwei Jia and Xun Yu Zhou. Policy evaluation and temporal-different learning in continuous time and space: A martingale approach. Journal of Machine Learning Research, 23(154):1-55, 2022b.
Yi Jiang, Weinan Gao, Jin Wu, Tianyou Chai, and Frank L Lewis. Reinforcement learning and cooperative Hoo output regulation of linear continuous-tin multi-agent systems. Automatica, 148: 1 10768, 2023.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: survey. IEEE transactions on intelligent transportation systems, 236):4909-4926, 2021.
Lior Kuyer, Shimon Whiteson, Bram Bakker, and Nikos Vlassis. Multiagent reinforcement learning for urban traffic control using coordination graphs. In Machine Learning and Knowledge Dis- covery in Databases: European Conference, ECML PKDD 2008, Antwerp. Belgium, September 15-19, 2008, Proceedings, Part I 19, pp. 656-671. Springer, 2008.
Donggun Lee. Berkeley, 2022.
autonomy under uncertainty. PhD thesis, University of California,
Jaeyoung Lee and Richard S Sutton. Policy iterations for reinforcement learning problems in con- tinuous time and space-fundamen fundamental theory and methods. Automatica, 126: 109421, 2021.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competttve environments. Advances in neural informa- tion processing systems, 30, 2017.
David Luviano and Wen Yu. Continuous-time path planning for multi-agents with fuzzy reinforce- ment learning. Journal of Intelligent & Fuzzy Systems, 33(1):491-501, 2017.
Yiming Meng, Ruikun Zhou, Amartya Mukherjee, Maxwell Fitzsimmons, Christopher Song, and Jun Liu. Physics-informe neural network policy iteration: Algorithms, convergence, and verifi- cation. arXiv preprint arXiv:2402.1011 2024.
Amartya Mukherjee and Jun Liu. Bridging Physics-Infeee Informed Neural Networks with Reinforcement Learning: Hamilton- Jacobi Bellman Proximal Policy Optimization (HJBPPO). In ICML Work- shop on New Frontiers in Learning, Control, and Dynamical Systems.
12

---

Published as a conference paper at ICLR 2026
Seohong Park, Jackyeom Kim, and Gunhee Kim. Time discretization- innpiiiiattionnnniiaanttt safe action repetition for policy gradient methods. Advances in Neural Information Processing Systems, 34:267-211 2021.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics- neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.0400 04043, 2019.
Shai Shalev-She Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03329, 2016.
Ali Shavandi and Majid Khedmati. A multi-agent deep reinforcement learning framework for algo- rithmic trading in financial markets. Expert Systems with Applications, 208: 18124, 2022.
Alena Shilova, Thomas Delliaux, Philippe Preux, and Bruno Raffin. Learning HJB viscosity solu- tions with PINNs for continuous-time reinforcement learning. In ICMI. 2024 Workshop: Foun- dations of Reinforcement Learning and Control-Conmection and Perspectives, 2024.
Oswin So and Chuchu Fan. Solving stabilize-avoid optimal control via epigraph form and deep reinforcement learning. arXiv preprint arXiv:2305.14154 2023.
Oswin So, Cheng Ge, and Chuchu Fan. Solving minimum-cost reach avoid using reinforcement learning, Advances in Neural Information Processing Systems, 37:30951-30984, 2024.
Josef Stoer, Roland Bulirsch, R Bartels, Walter Gautschi, and Christoph Witzgall. Introduction to numerical analysis, volume 1993. Springer, 1980.
Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust to time discretization. In International Conference Machine Learning, pp. 6096-6104. PMLR, 2019.
Manan Tayal, Aditya Singh, Shishir Kolathaya, and Somil Bansal. A physics-informed ma- chine learning framework for safe and optimal control of autonomous systems. arXiv preprint arXiv:2502.11057 1055, 2025.
Oriol Vinyals, Igor Babuschkin, Wojeiech Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft using multi-agent reinforcement learning. nature, 575(7789):5003344 2019.
Brent A Wallace and Jennie Si. Continuous-ting reinforcement learning control: A review of theo- retical results, insights on performance, and needs for new designs. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research, 21(198): 1-34, 2020.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:1 10068, 2022.
Xuefeng Wang, Xinran Li, Jiawei Shao, and Jun Zhang Ac2c: Adaptively controlled two-hop communication for multi multi-agent reinforcement learning. arXiv preprint arXiv:2302. 2023.
Xuefeng Wang, Lei Zhang, Henglin Pu, Ahmed H Qureshi, and Husheng Li. Continuous-time value iteration for multi-agent reinforcement learning. arXiv preprint ar Xiv::5509009155, 2025.
Cagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Continuous-time model- reinforce- ment learning. International Conference on Machine Learning, pp. 12009-12018 PMLR, 2021.
13

---

Published as a conference paper at ICLR 2026
Lei Zhang, Mukesh Ghimire, Wenlong Zhang, Zhe Xu, and Yi Ren. Value approximation for two- player general-sum differential games with state constraints. IEEE Transactions on Robotics, 40: 4631-4649, 2024.
Songyuan Zhang, Oswin So, Mitchell Black, and Chuchu Fan. Discrete gcbf proximal policy opti- mization for multi-agent safe optimal control. arXiv preprint arXiv:2502.03640, 2025a.
Songyuan Zhang, Oswin So, Mitchell Black, Zachary Serlin, and Chuchu Fan. Solving multi- agent safe optimal control with distributed epigraph form marl. arXiv preprint arXiv:2504. /5425, 2025b.
14

---

Published as a conference paper at ICLR 2026
A MATHEMATICAL PROOF
A.1 LEMMA 3.1: EQUIVALENCE OF TWO VALUE FUNCTIONS
Proof. Following proofs in (Lee, 2022; Zhang et al., 2024)), Eq. 4 implies the following equivalence
v(x)
=
V(2, 2) S
To prove the above relation, we first start from =(x) - 2 S 0, which implies that there exists a joint control input u 2 such that
((x()),, u(T)) - S
with c(x(r)) & 0 for Vr 2 t. Thus, there will exist a joint control u such that V(z, 2) S 0.
Second, when V(z, z) s 0 and c(x(T)) I 0 for VT 2 t hold, it implies that there exists E U such that
((x(T), u(r)) S O,
which concludes v(x) O. Therefore, the Lemma 3.1 is proved.
A.2
LEMMA 3.2: OPTIMALITY CONDITION
Proof. Following proofs in (Lee, 2022; Zhang et al., 2024; Evans, 2022), given all (t,x, E 00) x X x R and select enough small h >> 0. There exist two different joint control inputs (u1()), (2)- E 2 such that
[W](T), [1, M], u(()) = (u2(T),
(wa(t)
TE (t + 0o).
Then we have the following transformation for Eq. 3
v(a min
max an W2E
max [E((,t+
c(x(T)),
c(x(7)),
rE((+h,o0)
u((()) =(t))
1th >--)((=(T)) (((())))(()))
min max aiell
max te(c,t+)
c(x(T)) min max ugell
max 7E((th,00)
c(I(T)),
1+h
(--(((=(T) u(T)) (=(t+ h)
14h
min max uiell
max c(x(T)), min max TE(C(++)
max
c(x(T))
wstll
rE(l+h,00)
?" ((x(T), - =(1++))) h1)}}
2+h
min max wiEll
max c(x(T)); rE((.rtt)
min maxt wgEll
max
c(<(T)),
re(t n,))
(
=(t+ h))
min max Eiell
max c(x(T)), > (V((( h), + hi) (r((cr+h)
c(x(T)) 7 (V(((1 + 1), 2(1 + +n))
min max wEll [E(t,t++)
max
Therefore, the Lemma 3.2 is proved.
15

---

Published as a conference paper at ICLR 2026
A.3 THEOREM 3.3: EPIGRAPH-BASER HJB PDE
Proof. Following proofs in (Lee, 2022; Zhang et al., 2024; Evans, 2022), given all all (L,z, 2) E [0, 00) X I with small horizon Al 0, we apply Lemma 3.2 and Taylor expansion derive the spigraph-based HJB PDE as follows
V(x, 2) = min max NEW/
rEj,,AA max c(<(T)), /*V( (t)
2u min max c(ar) (1 + (n 2) + V.V. /(x, 0.V. 1(x, u), +(A NEW/
= -max -max(c( (1+ (1 In At) min (V 2) + V u)t 0,V 1(x, U At At + o()i)) wel
Subtracting V(z, 2) from both sides of above equality, dividing by At. and letting At 0 yields the following HJB PDE, where V(x, 2) is the optimal solution to such PDE.
maxfcla c(xx - V(z, 2), min V V f(z, u) - 0,V [[x, u) + Iu vi] 0. well
Here H V.V. f(z, u) 0,V 1(z, u) Iny V Hamiltonian and optimal control arg min H.
Next we prove that V(z, 2) is the unique viscosity solution to the epigraph- HJB PDE using the contradiction technique. First, for U E C (X' x R) such that V - U has local maximum at (ro, zu) E X IR and (V (20, zu) = 0, we will prove
max (((oo) 20), min well
[zU 20) fo)-f(aaa f(zo: u) -x0 20)o u) -Un--UUAa 20)1 2 o.
Suppose the above inequality is not correct. We consider that there exists 0 > 0 and i E l such that
c(x) - U Ux0, =0) S -0
VaU- f(z, 0,U 12x ū) Iny U -0
for all points (z, 2) sufficiently close to (.o, z0): ||=()) - zoll + (z(s) zol for small enough h > O, where E [fo, ln + hi] Under the assumptions in Sec. 3, and given state trajectories x and
evolved from the initial conditions x To and 2g according to the corresponding dynamics, the following inequality holds
c(>(s)) - U(o, =0) -00 V=U((z()) -
0.0 0,U(x(s), ((x(s), + In U(x(s),;(s)) < --0
Since V- U has a local maximum at (xo, 20). we can have that
min
(>V(x(( - 2(1 + h)) - V( 20)]
well
min
- U(o, =0]] [>" + h), =(1o = + h))
well
min (VV,U((((t)))))... ((o)) f(z(to), u) a, a,U(c(to): ((=(to), u) + In (((tU(ttt))))t.)))))))))))))())) 2(t))))
well
< 0h
We know that Lemma 2 implies
V(ro, 20) min max max
c(x(s)), >'V(o(to 1), + hi))
well
(sr
By subtracting U Uxo, 20) on both side, we have
(V (x0 =0) min max ((x(())) U(z0, 20), > V (a(to + h), =(o + +h) U (xxo z0) well
Since (V U) 2 0 holds such that V(zo, 20) = U Uzn, 20), then will have that
min max ((x(s)) V (zo, zo), <VV(o(to + #1) + h)) V (zo, 20)} welf
min max(0, 0h) 0, well
16

---

Published as a conference paper at ICLR 2026
which has contradiction with (V (20. 20) 0. Thus prove that
max maxxc((x)) (((xo)-U((a)) 20), min wEll
V VxU 20) fzo. u (20, 20). uo) (Zu 20)1 2 0.
Second, for U C (X
R) such that V U has local minimum at (0) X x R and
(V) 20) = O, we will prove
max (((x)) zo), min wEll
V (zo. 20) (2o. uo)- (20, 20). +o + (20, 20)1 5o.
The definition of auxiliary value (ax, 2) shows that
(E(,,00) max c(a(T)), [." (
(xx
min max wEll
(((o)) { (((())
min max
well
for all u U. By subtracting U (x 20) both sides, we have
(V - (xu, 20) 2 max(c((o) Uzz,, ( 1-=' u)dt - 20 Uxxo 21))
The rest of the proof is to show
min
V (=0, =0) f u) 02U (mo: 20) 1x u) + Iny U(ro, 20)] s O.
wEll
Suppose the above inequality is not correct. We consider that there exists 0 > 0 such that
min
Vn 2) [(x, u) (zU(z, 2) 1(x, u) + In Uxx =)] 2 0, uell
for all points (xr, 2) sufficiently close to (No, 20): || zoll + | 0ol h for small enough h 0, where 8 E [fo, to + h.. Given state trajectories x and 2 that evolve from the initial conditions x = 20 and 2g under the corresponding dynamics with any control 24, where
i(s) = arg min { (=)((((s,,(())) f(x(ss) (zU(x(s), (o)))))))) ((x(s),
Geul
+ 7 (z)()),())))
Then we have the following condition that holds
(xU(((()) (s)) f(x(s), ū) - (zU((z(s) ((x(s), ū) + In U(x(s), 2 0. Consider V U has local minimum at (zo, z0), we will have that
min well [>^V(z(tto - h), =(t h)) - V =o)]
2 min Well
[>AU(x((o + 1)) h)) - U(xo, 20]]
min
(z(())))) ((x(to), a)+ +my- (((((to,,((t)))
well
20h
Based on this derivation, we finally have that
"iin"VVatl + h), =(1o h)) 2 V 20) + 0h V V( 20).
aelf
However, we know that Lemma 3.2 implies that
min min ~ (z((o - h),, z(to + h)) < ((zo, ael/
which is a contradiction. Thus, we prove that
max maxffc(ra= (((r)) U(zo 20), min [V, 20) fz) -0) (Ug, 20).1(20, to) =)]] s
well
Hence, we prove that Vaz, 2) is the viscosity solution to the epigraph- based HJB PDE. The unique- ness follows Theorem of Chapter 10 in Evans (2022).
17

---

Published as a conference paper at ICLR 2026
A.4 ADVANTAGE FUNCTION
We define the Q Ze, u4) (((H) over a short time interval h and compute
Q ub) - ((I(, 2)) = max(d(x), 2+ 2))
max(c(x,) (Zz In 11 (Ft. 21) V., V- f(z,, 4))
- -0,, 100, u44 (I =2) o(h)}
max(c(xx) V(z, 2). (V., V f(z,, u)) az (x,, ue) + Im V)h}
We divide h on both sides of the above equation and let h + 0 to compute the advantage function
as
Q( 24, uz) ((z22
A(ri, Zz, Uz)
lim h-0
= =max(c(ar) ((rt, Van. V f(z,, u.) - 0,, V. [x,, uz) + Iny. V}
A.5 CONVERGENCE OF EPIGRAPI VALUE FUNCTION
Consider the augmented state (æ, 2) with state constraint c(x) and non -on-negative cost ((z, u). Define the discounted Sigraph-Belllmn operator over short step At
(TV) (1 -1- (()-Z)))((z)) + A min { well max {c(xx), ((a4ssi =+ti))3
for V X R + R bounded. Then the value iteration V TV2 converges uniformly to the unique fixed point of T.
Proof. (i) Contraction. For any e(x;) and bounded functions V, W. we have the following condition satisfying the contraction.
max(c((=)) V(I+AI, 4+4))} max(c(x1). (2444)2--)))
(2444122445 W (2444)4444)
W
(ii) Existence and uniqueness. By Banach's fixed-por point theorem, T admits unique fixed point V, and for value iteration V = TVV. we have that
||V - Vlloo S ~0 Vl + 0,
(iii) Approximate evaluation. If each iteration uses an approximate operator T satisfying T| oo S then
lim sup V Vllao S gst
k++00
B TRAINING ALGORITHMS
In this part, we provide additional details on the overall algorithmic pipeline and clarify the key implementation choices.
18

---

Published as a conference paper at ICLR 2026
Algorithm EEpigraph-Based Continuous-Time MARL
1: Initialize actor ии, return critic Vaa, constraint critic Veons dynamics network Se, reward net- work 4,. and local rollout R.
2: for 1...... 7 do
3:
Collect one rollout:
4:
I < env.reset()
5:
for k: = 1,.... K do
6:
sample arbitrary decision time t ~ ~ T for each agent L...., do
7:
8:
04 To, xr)
9:
end for
10:
jeint action (#1, ux)
11:
(z',r) f- env.step(u)
12:
append (z, u, r, x²) to local rollout R
13:
+-+a x¹
14:
end for
15:
Outer optimization: epigraph update
16:
find =¹ = [n E IR max (Veons (x), Viet (x, 2) =) 0}
17:
Dynamics and Cost Model learning on R
18:
update ç. oo as per the Eq. 17.
19:
Inner optimization given zt: Critic update on XR
20:
update v & by losses Lcons, Lret, LHJB and LvGI as per the Eq. 11, Eq. 10 and Eq. 12.
21:
Actor update for each agent
22:
for i = 1...., N do
compute (zx u, 2*) for all (z, (,, XR and update the 9 as the Eq. 18.
24: end for
end for
C ENVIRONMENTAL SETTINGS
We provide detailed descriptions of all benchmark environments used in our experiments. For each scenario, we list the number of agents, the number of obstacles, the safety constraints imposed, and the specific task objective with metrics.
Metrics. We report two primary metrics-ont one reward-style training score that aggregates task cost and constraint penalty, and one violation rate measured over held-out rollouts. (1) Cumulative penalty / reward-style training score. In many standard environments (e.g., MPE and multi-agent MuJoCo), the task reward often consists of two independent components: (i) task term such as distance-to-targge or velocity tracking, and (ii) safety penalty that is activated only when constraint- relevant events occur (e.g., collisions proximity violations). This design is also used in prior safe MARL methods such as MACPO and Lagrangian baselines (Gu et al., 2021).
For clarity of notation, we write the task cost as &, (derived from the negative reward the task term) and denote the constraint penalty as 2 0. The environment therefore provides composite instantaneous cost
v E, + Kit
which simply aggregates the task objective and the constraint penalty already defined in the envi- ronment. For a trajectory with horizon T(r), we define the total episode cost as
T(r)-1
J(r)
Σ V/
S(7):=
where S(7) the cumulative reward used for performance plots.
(2) Violation rate (evaluation). Given Neval episodes (we use Neval 100 by default), define the episode-level violation indicator
v(7) 1 0},
19

---

Published as a conference paper at ICLR 2026
Le., episode is counted as violating if it ever incurs positive state-constraint penalty.' The violation rate is then
Viol. Rate
v(r.).
Neval
C.I SAFE MPE
In the MPE, setup the details follows: Action. Continuous 2-D acceleration for x and y axis. Reward and costs. Each agent is assigned a per-agent target J.. The dense goal reward is
goal
x1; gille 12
A discrete collision cost with obstacles other agents applies:
10, aggett overlap 0,
c
(t)
otherwise.
We also record continuous proximity/penetration cost (not added into the dense goal reward):
[206,
8 0 (overlap)
((rs + r.) - -11=; l xoll), ((6)
0.55
5 M 0
OEO
where Ti, To are radius (sizes).
Difference from the original discrete-time MPE. The standard MPE environment uses fixed and discrete integration step At, where each simulation step updates the agent states according to Pt+1 Pt At and 1++1 U2 Åt with fixed time increment. In contrast, continuous- time MPE adapts the physical integration step to an arbitrary At provided by the learning algorithm. The state evolution follows
f(t) m
p(t) u(t),
+(t)
damping v(f),
and is numerically integrated via
f At,
p p-pto. At.
v
m
using the user-specified At. For clarity, the update used in the original environment is step(F)
p - p 0.1 (fixed 0.1), = v
F 0.1
(fixed as 0.1).
Our continuous-time version introduces
step continuous(F.A. At):
p= p=p+ v. At (depend the input At).
== セーシナ At (depend on the input At).
so that the state update depends directly on the argument Al rather than fixed constant.
This scenario contains agents with large corridor walls. Agents must avoid collisions with the corridor walls and with each other while navigating from their starting positions to reach the assigned target locations on the opposite side.
Corridor.
If is an indicator of hard violations, this coincides with "any violation." If is continuous hinge, the same criterion H;
20

---

Published as a conference paper at ICLR 2026
Formation. This scenario also involves 3 agents and 2 obstacles. The agents required bypass obstacles and then coordinate to form a triangular formation at the designated region, under the constraint of avoiding collisions with both obstacles and other agents.
Line. In this task, 3 agents operate in an environment with obstacles. After avoiding the obstacles, the agents must position themselves form straight line. The safety constraints enforce that agent collides with obstacles or with other agents during navigation.
Target. This scenario uses 2 agents with obstacle placed in the environment. Each agent is assigned a fixed target position, and the agents must navigate to their respective goals while avoiding collisions with the obstacle and with each other.
Cooperative Navigation. This is a cooperative navigation task with 3 agents and no obstacles. The agents must spread out cover multiple target landmarks while avoiding collisions among themselves, Specifically, the agents' goals are the one closest to them rather than fixed ones.
Cooperative Predator-Prey.
This task includes 3 controllable predator agents and prey that moves randomly. There are no obstacles, but predators must avoid colliding with each other. The predators' objective is to coordinate their movements to capture the prey.
C.2 SAFE MULTI-AGENT MuJoCo
Half Cheetah. We adapt the Half Cheetah environment into three multi-agent variants: Half Cheetah-2x3, Half Cheetah-3x2, and Half Cheetah-6x1. In each case, the body is partitioned into joints agents with different grouping configurations. For example, Half Cheetah-3x2 is three agents with 2 moving joints for each agent. Randomly placed walls are introduced into the environment, requiring the agents not only to coordinate efficient forward locomotion but also to avoid collisions with obstacles.
Reward. r = Trun -++1-1
Safety cost. A binary proximity cost to the wall:
4 = 1{ [wall - Tagen < <9} E {0,
Observation augments the usual state with wall velocity, wall force proxy, and clipped distance to the wall; the environment also recolors the wall when unsafe.
Difference from the original MuJoCo environment. In standard MuJoCo control tasks, the sim- ulation uses fixed micro time step 0.01 (each frame takes 0.01), and each environment step cor- responds fixed number of internal physics frames (e.g., frame_ske fixed control interval At = 0.05. Our continuous-timg MuJoCo variant removes this fixed control interval. For any desired At, we execute
5), resulting in
do_s imulation( N), N At
0.01
i.e, the number of internal physics frames is chosen dynamically according to the requested in- tegration step. Thus the effective control interval is no longer fixed but fully determined by At, enabling
continuous time rollouts. The reward terms (forward velocity, control cost, contact cost) are normalized by the actual At, ensuring consistency across different temporal resolutions. The original update is
step(u):
N N 5, do do.simulateuu.N)
Our continuous-time version becomes
step_cont ctpp.cconiinousuuu. At) A1/000, do-simulate(u,N)
Ant. We construct four multi-agent variants of the Ant: Ant-2x4, Ant-4x2, Ant-8x1, and Ant-2x4d. In all cases, the body is controlled by joints agents arranged in different groupings across the legs. As with Half Cheetah, walls are introduced as obstacles, and the agents must coordinate locomotion while ensuring safety by avoiding collisions with these obstacles. The reward is set the same the Half Cheetah.
Safety shaping. Identical piccewise-slant corridor: compute you from (xr, y) and define
ob
(1yaal 1.8).
21

---

Published as a conference paper at ICLR 2026
C.3 CONSTRAINED COUPLED OSCILLATOR ENVIRONMENT
We consider a two-agent coupled spring- system. The state and control are
r= [x1
01
22
02
==
[u
u2
Each agent i E {1,2} controls one mass with continuous time dynamics
áे Us: k x4
b + ue.
with spring constant = 1.0 and damping coefficient b = 0.5. Stacking the states gives t Ax Bu with
0
0
0
г 07
b
0
0
0
A
B
0
1
0
0
k
b
0
1
Control limits and discretization.
Actions normalized ü; |-1, 1] and mapped to physical inputs by u, Umax Uz with Umax 10 (component-wir) wise box constraint).
+++
1 +
kx. but + ut)
2 ++1 x + 20 At.
for horizon of N 30 steps. Stage cost. The per-step quadratic cost is {(x, u) o + z
+ de (x1 x2)² 2 (u] + uz),
with coupling strength A 2.0 and control penalty 8 0.01. Equivalently, E(z, "Q= + Ru where
+ A
-Ae
0
0
0
Q =
R = 812.
A
0
1
+
Ao
0
0
For training we use a shaped reward
1
(wx, uz).
30
Hard state constraint. We impose ordering constraint between the two positions,
31 S x2 + 0.02,
and record an additional penalty
Pt - 1f F2 0.02},
returned alongside Tt.
Smooth violation signal (for logging). We also log smooth surrogate of the constraint violation,
((x) 20(s (#1 - Ø2 + 0.02)) - 1, a(2) Ite-5' $ = =220,
which maps to (-1,1) )) and grows monotonically with the amount of violation.
Unconstrained LQR. The continuous-time algebraic Riccati equation (CARE)
P PA PBR L P + 0
is solved for the unique positive semidefinite matrix P. The unconstrained optimal linear feedback
K R-BBTP
(LQR(*) Krr.
22

---

Published as a conference paper at ICLR 2026
Hard state constraint and CBF condition. We impose the safety constraint
11 12 - 0.02 S
= h(x) = 0.02 - (x1 x2) 2 0.
Let Vh(x) 1-1 0 0j' A (first-order) control barrier function (CBF) condition enforces
forward invariance of the safe set C = {x : h(x) 2 0} by requiring
u = Vh(x) (Ax + Bu) 2 -ah((x))
with user-chosen class-K parameter O. Defining
a(z) B
b(x) Vh(x)* Ax a ER,
the CBF condition Eq. C.3 is the single affine-in-u half-space constraint
a(x) Tu 2 b(x).
Closed-form safety projection. To obtain safe control with minimal distortion from ULQR, solve the weighted projection
min MER?
(u tu--uaa ULQU) T W (u ULQR) s.t. a(x) T a 2 b(x),
with W = R (" ("R-metri""; Euclidean W = I is also possible). Because Eq. C.3 has a single linear constraint, admits closed form:
a ULQU b, u (2) (1Qu(x) + W- otherwise, with t
uLQU(=),
b b- a ULQU
TW LTWlla
Finally we saturate to the actuator limits umax > 0: (GT(x) clip(u" "nu"""( (x)) Umaax Umax)
D ADDITIONAL ENVIRONMENTAL RESULTS
D.1 VISIUAL TRAJECTORIES
The trajectory demonstrations in Fig. 10 highlight clear behavioral differences across algorithms in Formation scenario. Our proposed method EPI learns smooth trajectories that avoid obstacles while consistently reaching the target, demonstrating both constraint satisfaction and goal achievement. In contrast, EPPO occasionally captures the avoidance behavior but often gets stuck at suboptimal solutions. This is because during training, its randomized sampling of the auxiliary state 2 prevents stable policy convergence in continuous tion, the learned policy lacks accurate control signals. On the other hand, MACPO, which enforces hard constraints via trust-region style update, tends to overestimate the obstacle region. As a re- sult, agents often exhibit overly conservative Schaviors-suut violations-raun rather than efficiently pursuing their targets. Together, these comparisons confirm that EPI achieves the most balanced and effective behavior among the three approaches.
settings; even if outer optimization is applied at execu-
retreating toward corners to avoid
D.2 PERFORMANCE UNDER STOCHASTIC SETTINGS
To evaluate robustness under stochastic dynamics, we perturb the continuous-time transition model as Zt+At = -f [zr, u,) At + Eg (0, o² I)). in Fig. We consider three noise magnitudes: Low Noise: o² 0.1 Mid Noise: σ² = 0.5 and High Noise: o² = 1.0. We observe that No Noise and Low Noise yield similar identical cost and constraint-violation behavior across all three tasks. Because the PINN-basee value approximation are inherently robust to small local perturbations, as long as the injected disturbance is within moderate range, the learned dynamics model, cost model, and value gradients remain accurate. In contrast, Mid Noise and High Noise introduce much larger deviations in the state propagation. These disturbances accumulate over time, causing the PINN to receive significantly deviated training signals. Since method does not incorporate explicit uncertainty modeling or stochastic HJB formulations, the serious noise directly degrades the learned critic and value gradients, eventually leading to unstable or even failed policies.
23

---

Published as a conference paper at ICLR 2026
EPI-1
(b) EPI-2
(c) EPI-3
(d) EPI-4
(e) EPI-5
(1) EPPO-1
g EPPO-2
(h) EPPO-3
ii EPPO-4
EPPO-5
(k) MACPO-1
(I) MACPO-2
(m) MACPO-3
(n) MACPO-4
(o) MACPO-5
Figure 10: Trajectory demonstrations (key frames) across methods in Formation. Row EPI re-
sults, Row 2: EPPO results, Row 3: MACPO results.
Constraint Violation Rate
Corridor
Cooperative Predator-Prey
Cooperative Navigation
60
20
40
10
10
20
1
2
1
2
3
+
No Noise
Low Noise
Mid Noise
High Noise
Figure 11: Performance under different noise levels.
Average Distance to Target
Ac = 0.03
Al 0.1
1.50
1.25
1.00
0.50
0.25
0.00
Formation
Line
Target
Figure 12: Average distance to the target under different Al.
24

---

Published as a conference paper at ICLR 2026
D.3 EFFECT OF THE DISCRETIZATION INTERVAL.
Figure 12 evaluates how the choice of discretization interval A/ affects the performance of EPI. For each fixed At, roll out complete trajectories using the learned policy and measure the average distance to the target over the entire trajectory. Across all three scenarios, we observe a consistent trend: the averrage distance to the target increases as At becomes larger. This behavior is expected in continuous- time control. When At is small, the temporal resolution is high and the policy is up- dated frequently, allowing the learned value gradients to provide fine-grained control corrections. In contrast, larger At leads to coarser control updates, reducing the precision of the policy's response to the evolving system dynamics. Moreover, both the HJB residual and the VGI update rely on local differential information. As At grows, the mismatch between the continuous-time formulation and the discrete rollout increases, which in turn amplifies approximation errors in the learned value gra- dients. These errors accumulate along the trajectory and result in the observed degradation in task accuracy.
D.4 TRAJECTORY OF =" THROUGH THE TRAINING
Episocie 1000
Episade 12000
Episode 23000
52-
1.0-
5.0
0.5
4.8
Retur
Retur
Ratum
Iharch
Branch
Borch
Conadraint
Constrant
10
20
30
60
50
10
20
30
40
50
10
20
30
40
59
Time Stop
Time Stop
Time Stop
Figure 13: 2n trajectory through the training in the target.
Figure 13 illustrates the evolution of the optimal epigraph variable =f and the active branch (return vs. constraint) at three representative stages of training. In early training (Episode 1000), the policy frequently visits infeasible states, causing Veons (21) O and forcing the epigraph to select the constraint branch; consequently z remains at the clipped upper bound Zmax- By mid training (Episode 12000), the critic starts to maintain Venns(*:)) < for part of the trajectory, producing intermittent switching and a decreasing 2f. In late training (Episode 23000), the trajectory remains feasible. the return branch consistently selected, and 2 decreases smoothly along the rollout. These behaviors align with the expected epigraph semantics: infeasible states produce 2max while improved policies yield stable return -ommnnddm gradually decreasing 27"
D.5
COMPARE EPI WITH TRADITIONAL. EPIGRAPH METHOD
Formation
Line
Target
Cumulative Reward
Cumulative Reward
Cumulative Reward
100
-00
-100
-300
-200
-150
-400
300
-500
-600
-250
2500
7500 10000 Episode
5000
2500
7500 Episode
5000
10000
2500
5000 Episode
7500
10000
Original Epigraph
Figure 14: Performance of EPI and traditional epigraph under MPE settings.
Figure 14 compares our z-independent epigraph formulation (EPI) with the traditional 2-dependent epigraph used in EPPO-like methods on the FORMATION, LINE, and TARGET tasks. In the tradi- tional design, scalar is randomly sampled at the initial state of each episode and then propagated
25

---

Published as a conference paper at ICLR 2026
through its auxiliary dynamies, that both critic and actor conditioned this randomly chosen epigraph level. As shown in Fig. 14, converges to a lower cumulative reward, and exhibits substan- tially larger variance across seeds. In contrast, EPI learns z-independent critics (Vrons(z) V (=)) and computes =" via one-dimensional search during training, while the actor depends only on the physical state This removes the nonstationary noise introduced by random sampling: for fixed r, the policy gradient under EPI is unique, whereas in the traditional epigraph it fluctuates with the sampled z even when the critic has already converged. In continuous-time settings this issue am- plified, since small changes in 2 shift the switching time between the constraint and return branches and thereby alter the entire rollout.
D.6 COMPARISON BETWEEN EPI AND DISCRETE-TING BASELINES
1.5
Distance to
1.0
0.5
0.0
Target
OT-MAPRO-LAG
Figure Performance of EPI and discrete-time baselines under MPE settings.
To validate the performance of traditional discrete-time based methods in continuous-time settings, the Fig 15 compares EPI with two discrete-time baselines (DT-CBF and DT-MAPPO-LAG) on the Line and Targer tasks in the continuous-time MPE environment. All baselines adapted to the discrete-ting timee setting by removing their residual-loss components. Apart from this modification, all implementation details follow their original published versions (Zhang al., 2025a). Across both tasks, EPI consistently achieves lower mean distance to the target and smaller variance, demonstrat- ing the performance gain from the modules that designed for the continuous-time settings.
Table Hyperparameter settings used.
Parameter
Value
Episode length for MPE
Episode length for MuJoCo
100
Episode length for Didactic
50
Total number of episode for MPE
30000
Total number of episode for MuloCo
30000
Total number of episode for Didactic
3000
z range for MPF
0-10
z range for MuJoCo Z range for Didactic Discount factor 7
0-5
0-2
0.99
Actor learning rate
0.0001
Critic (Return) learning rate
0.001
Critic (Constraint)learning rate
0.001
Dynamics model learning rate
0.001
Reward model learning rate
0.001
Exploration steps
1000
Model save interval
1000
Random seed
113-120
26

---

Published as a conference paper at ICLR 2026
E HYPERPARAMETERS AND NEURAL NETWORK STRUCTURES
Experiments were conducted on hardware comprising an Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz, four NVIDIA A5000 GPUs and eight NVIDIA A6000 GPUs. This setup ensures the computational efficiency and precision required for the demanding simulations involved in multi- agent reinforcement learning and safety evaluations.
Table I lists the defaults used in all experiments. Episode lengths are chosen so that single rollout covers full interaction cycle (50 steps for MPE and the didactic environment, 100 for MuJoCo). We train for 30000 episodes in MPE and MuloCo and for 3000 episodes in the didactic setting, reflecting simulator cost and convergence speed. The z range controls epigraph sampling for the VGI updates and is set wider in MPE (0-10), moderate in MuloCo (0-5), and narrow in the didactic task (0-2). The actor uses a conservative learning rate (le-4) for stable policy updates; the critics and the dynamics/reward models use le-3 to accelerate value/model fitting. Training is warm-started with 1000 exploration steps, checkpoints are saved every 1000 episodes, and reported results are averaged over seeds 113-120.
Table 2: Summary of neural network architectures used in framework.
Network
Input Dimension
Architecture Activation
Return Value Network
State (d)
FC(128) FC(128) - FC(1), ReLU or Tanh
Constraint Value Network
State (d)
FC(128) + FC(128) FFC(1), ReLU or Tanh
Dynamics Network
State Joint Action + na)
FC(128) FC(128) FC(d). ReLU
Reward Network PolicyNet
State Joint Action + na)
FC(128) FC(128) FC(1). ReLUU
Observation Time Interval (o
FC(128) FC(128) FC(64) FC(o). ReLU
Table summarizes the five multilayer perceptrons used in our framework. Two scalar critics map the state E Rdd to the return value and the constraint value, each with two hidden layers of width 128 and ReLU or Tanh activations. The dynamics and reward models take the concate- nated state-action input (z, tive/increment and scalar reward; both use two 128-width hidden layers with ReLU. The policy network consumes the observation o R augmented with scalar time-interval feature At to condition actions on continuous- time step size, and produces an a-dimensional action through 128-128 64 hidden stack with ReLU.
Idd and output, respectively, d-dimensional state deriva-
Notation: d = state dimension, number of agents, so the joint action has dimension na. The value heads output scalars; the dynamics head outputs 18d; the policy head outputs Ra. Action squashing or clipping to environment bounds (if used) is applied after the final linear layer.
observation dimension, per-agent action dimension,
F THE USE OF LARGE LANGUAGE MODELS (LLMs)
employed LLMs as a writing assistant to polish the paper.
27


Reviews:
[{"review_id": "jUMesT4JoN", "reviewer": "Reviewer_VtHr", "summary": "The paper formulates safe CT--MARL as a continuous-time CMDP and introduces an epigraph variable $z$ to smooth discontinuities, deriving an epigraph-HJB and training a PINN-based actor--critic with a revised inner/outer optimization that computes $z^\\*$ during rollouts. Results are on safe CT-MPE and CT-MA-MuJoCo with ablations.", "questions": "1. When $V_{\\text{cons}}(x)>0$ along a rollout, Eq.(8) has no feasible $z$. What exact rule do you use for $z^\\*$ in this case, and why does the resulting objective still correspond to the epigraph--HJB target used in Lem.3.1/Thm.3.3? \n\n2. How is $\\partial_z\\tilde V$ computed in Eq.~(10) with $z$-independent critics? Do you use a straight-through $-{\\bf 1}$ on the return branch or a smooth approximation to $\\max$? Any stability tricks near the switching set? \n\n3. Given Eqs.~(16)--(17), can you bound the gap between model-PDE residuals and true residuals, or provide stress tests (dynamics mismatch) linking model error to violation rates?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Addresses a real gap: explicit continuous-time safe MARL (state constraints) with a principled epigraph--HJB route and viscosity-solution framing. \n\n2. Clean derivations: Lem.3.1/3.2 and Thm. 3.3 tightly connect DPP to the PDE with the $\\ln\\gamma$ term.\n\n3. Practical training design:  $z^\\*$ computed during training; $z$-independent critics simplify deployment. \n\n4. Broad evaluation \\& ablations:  consistent gains on CT-MPE/MA-MuJoCo; ablations clarify each loss term’s effect (target/VGI $>$ residual).", "weaknesses": "1. Feasibility of the outer problem: Eq.(8) requires $\\max\\{V _{\\text{cons}}(x),V _{\\text{ret}}(x)-z\\}\\le 0$. If $V _{\\text{cons}}(x)>0$, no $z$ is feasible; the paper clips $z^\\*$ to $[z _{\\min},z _{\\max}]$ (Sec.~3.2.2), which does not restore feasibility or preserve the theory.\n\n2. Model-bias vs. safety: residual/advantage (Eqs.~10,17) use learned $f_\\xi,l_\\phi$; PDE guarantees hold for true dynamics but not for the surrogate, and no robustness link to violation rates is provided. \n\n3. “Continuous-time” evaluation clarity: Alg.~1 samples arbitrary decision times and the policy conditions on $\\Delta t$, but there is no systematic $\\Delta t$-sweep or irregular-step stress test in the main results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W5bZSYzPZM", "reviewer": "Reviewer_GrMo", "summary": "This paper proposes a novel framework for safe continuous-time multi-agent reinforcement learning (CT-MARL) using epigraph forms. The authors address the challenge of incorporating safety constraints into CT-MARL, which often leads to discontinuities in value functions that are difficult to handle with traditional methods. They introduce an epigraph-based reformulation that converts discontinuous value functions into continuous ones by augmenting the system with an auxiliary state variable z. This reformulation enables the use of Physics-Informed Neural Networks (PINNs) for stable and efficient optimization in continuous time. The authors prove the existence and uniqueness of viscosity solutions for the epigraph-based HJB PDEs, providing theoretical support for their method. Extensive experiments on continuous-time safe multi-particle environments (MPE) and multi-agent MuJoCo benchmarks demonstrate that their approach outperforms existing safe MARL baselines in terms of both cost reduction and constraint satisfaction. This work offers a robust and effective solution for safe CT-MARL by leveraging epigraph forms and PINNs to handle discontinuities and ensure stable learning in continuous time.", "questions": "Please refer to the “Weakness” section for related questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation behind this work is interesting and necessary.\n2. The theoretical proofs provided in this work are comprehensive.", "weaknesses": "1. In the section “Related work”, what are the differences between this work and [1]? I feel that this work merely extends the ideas of [1] to the continuous-time scenario. I suggest that the authors restate the contributions of this paper.\n2. In the section “Methodology”: ① The figure in Fig1 is incorrect. According to the description of Algorithm 1, at least three steps are not independent but form an integrated loop. An arrow from Learning to rollout should be added. ② I don't quite understand how equation (7) is transformed into equations (8) and (9). Some additional explanations may be needed here. ③ In the “INNER OPTIMIZATION WITH CRITIC LEARNING” section, are the three loss functions updated independently or jointly? If they are updated jointly, how are their weights defined?\n3. In the section “Experiments”: ① In 4.1 Benchmarks and baselines, the authors mention “In MuJoCo, we adapt several scenarios such as HalfCheetah and Ant into continuous-time versions and introduce randomly placed walls as obstacles.” I don't know how the continuous time is designed here. I think it is not necessary to spend too much space describing this environment. Instead, the differences between the original version and the continuous-time version should be explained. ② “We design the reward as the summation of the minus cost and constraints listed in Appendix C, which directly reflects performance under both objectives.” I don't understand why the reward is designed in this way. Why not design it as a similar independent reward and cost like MACPO? Since this paper mainly focuses on RL in continuous time, it may be helpful to highlight the contributions of this paper by separating the reward and cost. ③ Section 4.3 lists the impact of the choice of z on performance changes, but lacks a visualization of how the z value changes over time. I think it would be useful to add such an experiment to show that the changes in z do indeed affect the final decision.\n\n**Reference**\n\n[1] Zhang, Songyuan, et al. \"Solving Multi-Agent Safe Optimal Control with Distributed Epigraph Form MARL.\" arXiv preprint arXiv:2504.15425 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GnBb5cRcX9", "reviewer": "Reviewer_qGRY", "summary": "This paper proposes EPI, a method for solving safe continuous time MARL problems posed as a constrained optimal control problem. An epigraph reformulation is used to handle the state constraints, then a PINN is used to learn the continuous-time value function, where two additional losses are used to improve the PINN training. Empirical results on the MPE and Safe MA-MuJoCo environments demonstrate that the proposed EPI outperforms continuous-time variants of existing safe MARL methods.", "questions": "- Line 139: The paper writes that each agent applies a decentralized policy, but each policy seems to take the entire state space as input. Later on in Line 211, the authors write that the decentralized actors “map local observations”, which is confusing.\n- Why do the networks take ∆t as input? Does ∆t remain constant or does it change?\n- What are the weights for the different terms for the PINN learning?\n- How sensitive is the method to the weighting of these terms?\n- What does “overweighting” mean for the ablation in Figure 9? How much are the weights increased by?\n- Algorithm 1, line 16: I’m assuming that z^* is solved for for every x in the rollout?\n- How are the benchmarks “adapted”? The MPE and Safe MA-MuJoCo benchmarks are originally for discrete time.\n- How does the proposed method compare to (discrete) CBF-based methods for safety in MARL, such as [A]? Though it is a discrete-time method, it should still apply to continuous-time environments, similar to how the other discrete-time methods have been adapted.\n\n[A]: Zhang, Songyuan, et al. \"Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control.\" ICLR 2025.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- To the best of my knowledge, the use of the epigraph form for continuous-time safe MARL is novel.\n- The experimental results on MPE show that EPI has strong performance compared to existing methods. On MuJoCo EPI, it is not yet clear how the performance in terms of cost and constraint compare to existing methods.", "weaknesses": "- Many details in the experiments section are not very clear (see questions)\n- It is not clear which parts of the proposed method are claimed as novel contributions and which ones are taken from existing methods. For example, are the concepts of the target loss and value gradients claimed to be novel, or were they proposed in previous works and then adapted to the current setting?\n- The experiments on multi-agent MuJoCo seem to only compare the reward (Figure 2). It is not clear how the different methods compare in terms of constraint satisfaction. Something similar to Figure 4 for the multi-agent MuJoCo setting would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "60SsxMYNlu", "reviewer": "Reviewer_cjAT", "summary": "This paper considers the continuous-time multi-agent reinforcement learning (CT-MARL) problem with safety constraints. To address the problem, the paper presents a framework that transforms discrete MDPs into CT-CMDPs via an epigraph-based reformulation. The problem is then solved with a PINN-based actor-critic method. Empirical results show that the proposed method performs better in continuous-time safe multi-particle environments and safe multi-agent MuJoCo environments.", "questions": "1. How are the decision times $t_k$ selected? Are they given, or does the policy need to decide?\n\n1. Considering the motivation of the proposed method, why does the value become discontinuous when state constraints are violated? Why can this hinder the convergence? Why can the epigraph formulation address this issue? \n\n1. Why does the random sampling of $z$ introduce nonstationary noise and lead to poor convergence? Is this a drawback only in the continuous-time setting? It would be great if the authors could show this theoretically or empirically, or both. \n\n1. It seems that the value function $\\tilde V$ and the advantage function $A$ are conditioned on $z^\\*$, which, to my understanding, can change a lot during training. However, the learned policies are not conditioned on $z^*$. Therefore, it seems that the policy learning target changes constantly during training. Will this cause unstable training?\n\n1. Why $f_\\xi$ and $l_\\phi$ do not have $\\Delta_t$ as input in Equation (16)?\n\n1. Which part of the proposed method is designed specifically for *multi-agent* systems?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The problem considered is interesting. \n\n2. The proposed method shows strong empirical results. \n\n3. Each part of the loss function is well ablated.", "weaknesses": "1. The proposed method is not very well motivated and includes some unclear parts. Please refer to Questions. \n\n1. The metric used in the experiment section can be questionable. The paper \"designs the reward as the summation of the minus cost and constraints\", which can be sensitive to the relative scales of the cost and the constraints. I think figures like Figure 4 are clearer than Figure 3. Or the authors can also show the training curves of the cost and the constraints separately. \n\n1. It seems that all the baselines chosen in this paper are designed for discrete-time MDPs. It would be better if some continuous-time methods could be included as baselines. \n\n1. Minor: Some acronyms are not defined clearly in the paper, e.g., PINN, PDE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)