Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning
Authors: Xuefeng Wang
Lei Zhang
Henglin Pu
Ahmed H Qureshi
Husheng Li
Venue: ICLR 2026 Poster (2026)
Keywords: Continuous-time
multi-agent reinforcement learning
physics-informed neural networks
Primary area: reinforcement learning
TL;DR: This paper leverages physics-informed neural networks combined with value gradient iteration to deal with continuous-time multi-agent reinforcement learning problems.

Abstract: Existing reinforcement learning (RL) methods struggle with complex dynamical systems that demand interactions at high frequencies or irregular time intervals. Continuous-time RL (CTRL) has emerged as a promising alternative by replacing discrete-time Bellman recursion with differentiable value functions defined as viscosity solutions of the Hamilton–Jacobi–Bellman (HJB) equation. While CTRL has shown promise, its applications have been largely limited to the single-agent domain. This limitation stems from two key challenges: (i) conventional methods for solving HJB equations suffer from the curse of dimensionality (CoD), making them intractable in high-dimensional systems; and (ii) even with learning-based approaches to alleviate the CoD, accurately approximating centralized value functions in multi-agent settings remains difficult, which in turn destabilizes policy training. In this paper, we propose a CT-MARL framework that uses physics-informed neural networks (PINNs) to approximate HJB-based value functions at scale. To ensure the value is consistent with its differential structure, we align value learning with value-gradient learning by introducing a Value Gradient Iteration (VGI) module that iteratively refines value gradients along trajectories. This improves gradient accuracy, in turn yielding more precise value approximations and stronger policy learning. We evaluate our method using continuous‑time variants of standard benchmarks, including multi‑agent particle environment (MPE) and multi‑agent MuJoCo. Our results demonstrate that our approach consistently outperforms existing continuous‑time RL baselines and scales to complex cooperative multi-agent dynamics. Code is available at https://github.com/Wangxuefeng1024/Continuous-Time-Value-Iteration-for-Multi-Agent-Reinforcement-Learning.git.

Published as a conference paper at ICLR 2026
CONTINUOUS-TIME VALUE ITERATION FOR MULTI- AGENT REINFORCEMENT LEARNING
Xuefeng Wang'; Zhang': Henglin , Ahmed H. Qureshi', Husheng Li Purdue University
ABSTRACT
Existing reinforcement learning (RL) methods struggle with complex dynamical systems that demand interactions at high frequencies or irregular time intervals. Continuous-time RL (CTRL) has emerged as promising alternative by replacing discrete-ting time Bellman recursion with differentiable value functions defined viscosity solutions of the Hamilton-Jacoh -eallmm (HJB) equation. While CTRL has shown promise, its applications have been largely limited to the single-agent (i) conventional methods
domain. This limitation stems from two key challenges: for solving HJB equations suffer from the curse of dimensionality (CoD), making them intractable in high-dimensional systems; and (ii) even with learning-based ap- proaches to alleviate the CoD, accurately approximating centralized value functions in multi-agent settings remains difficult, which in turn destabilizes policy training. In this paper. we propose a CT-MARL framework that uses physics-informed neural networks (PINNs) to approximate HJB-basee value functions at scale. To ensure the value is consistent with its differential structure, we align value learn- ing with value- gradient learning by introducing Value Gradient Iteration (VGI) module that iteratively refines value gradients along trajectories. This improves gradient accuracy, in turn yielding more precise value approximations and stronger policy learning. We evaluate our method using continuous- time variants of standard benchmarks, including multi-agent particle environment (MPE) and multi-agen MuJoCo. Our results demonstrate that our approach consistently outperforms ex- isting continuous-time RL baselines and scales to complex cooperative multi-agent dynamics. Code is available at this link.
INTRODUCTION
RL has achieved remarkable success in a range of discrete-time single- and multi-agent interaction tasks, including robotic manipulation (Brunke al., 2022), strategy games (Vinyals al., 2019), wireless communications (Feriani & Hossain, 2021; Wang et al., 2023), and traffic coordination (Hay- dari & Yılmaz, 2020). However, many real-world domains are inherently continuous timm and operate at high or irregular decision frequencies, such as continuous-time state estimation in robotics (Talbot al., 2025), autonomous driving (Brechtel al., 2014), and market trading (Shavandi & Khedmati, 2022). Despite this, most existing RL methods formulate decision-making in discrete-time, where Bellman updates are computed
fixed time interval. Such discrete-time RL (DTRL) approximates continuous-time process by imposing fixed discretization step, which introduces two inherent limitations. First, when the timestep is coarse, the resulting controller becomes non-smooth and leads to suboptimal or unstable behavior (Doya, 2000). Second, when the timestep is fine, the number of states and iteration steps become large, which enlarges not only memory storage but also many learning trials (Doya, 2000). In addition, the time step At approaches to 0, the Bellman operator can become ill-conditioned: the temporal-difference objective functions may be dominated by ap- proximation noise, leading the Bellman updates to become unstable and have a large variance (Wang et al., 2020; Shilova et al., 2024). These limitations indicate that performance may critically depend on the chosen discretization method, thereby motivating continuous-tin RL (CTRL) methods that avoid Rubanova al., 2019).
and directly learn value functions in continuous time (Doya, 2000;
"quual contribution.
Corresponning author.

---

Published as a conference paper at ICLR 2026
08
Grund Thuth CLAMNN
Autian (agent
action (Agent 3)
676
100
Rewerd
10
DTAGAL
600
0.3
02
15
20
35
20
20
TE Timeslap
Timestop
Timestop
Timestap
(a) Value Approximation
(b) Reward Approximation
(c) Agent Action
(d) Agent Action
The performance of our CT-MARL and DT-MARL is compared a continuous-ting -time, two-agent coupled oscillator task. In the discrete-time setting, DT-MARL trained with MADDPG can achieve near-optimal performance. However, when transferred to the continuous-time domain, MADDPG suffers from significant bias and error, resulting in poor approximations. In contrast, CT-MARL yields smoother actions, higher rewards, and more accurate value approximations, closely aligning with the analytical LQR ground truth.
Figure 1:
Unlike Bellman operator-based DTRL (Bellman, 1966), CTRL leverages HJB Partial Differential Equation (PDEs) to compute differential value functions (Mukherjee & Liu; Shilova et al., 2024). However, solving HJB PDEs through conventional approaches (e.g.. dynamic programming or level set method (Osher et al., 2004)) suffers from CoD in nigh-dimensional dynamical systems (Bellman al., 1965), especially where the computational complexity grows exponentially with the state di- mension in multi-agent systems. PINNs have emerged as a powerful tool to circumvent CoD (Weinan et al., 2021), and offer convergence guarantees for problems with smooth solutions (Shin et al., 2020; Ito et al., 2021). To approximate the solutions of HJB PDEs, PINNs translate the underlying physics law (e.g., PDEs) along with boundary conditions into the loss functions to refine networks.
Yet existing CTRL methods focus almost exclusively on single-agent settings and do not extend naturally to MARL. In agent- domains, each agent must interact with the environment while simultancously adapting to other learning agents, introducing severe non-stationarity, making value learning substantially harder (Yu et al., 2022). Even with PINNs, accurately learning value functions under the centralized training decentralized execution (CTDE) framework remains challenging: PDE-based residual losses alone often produce biased or inaccurate value gradients, where coupled dynamics amplify approximation errors (Zhang et al., 2024b). These inaccuracies propagate through the PDE residuals, degrade value learning, and ultimately deteriorate policy optimization. To address this limitation, we propose a novel learning approach that combines PINN and VGI optimize the value learning. The PINN component ensures the value approximations satisfy the HJB PDEs, while VGI iteratively propagates and refines the value gradient approximations along the sampled trajectories. To better understand the limitations of DTRL, and advantages of our CTRL. algorithms for multi-agent scenarios, we present a didactic case as shown in Fig. 1. this simple continuous-ting control task, DT-MARL fails to accurately approximate the true value functions, leading to incorrect control actions, particularly for agent 2. In contrast, our CT-MARL algorithm closely follows the ground-truth trajectory, maintains high returns, and generates accurate control actions for both agents.
Our work makes the following contributions. (1) We leverage PINNs to approximate differential value functions and apply them to solve cooperative continuous-ting multi- agent reinforcement learning problems within deterministic systems, which have rarely been explored by previous studies. (2) We introduce novel value-gradien iteration term that dynamically refines the approximations of value gradients during training. This setting improves the computational accuracy of the value gradients, accelerates learning convergence, and leads highly accurate value approximations, enabling efficient policy learning. (3) We create continuous-time versions of two standard MARL benchmarks, the continuous-timg MPE and continuous -ime multi-agen MuJoCo. The results demonstrate that our method consistently surpasses other current CTRL baselines, highlighting the advantages of precise aalue-gradient learning in high-dimensional multi-agent systems.
2 RELATED WORK
2.
CONTINUOUS-TIME REINFOR LEARNING
CTRL has received increasing attention in recent years, however, most existing works focus on the single-agent settings. These studies aim to optimize policies in continuous time domains, avoiding time discretization and yielding more accurate control actions in robotics and navigation. For instance,
2

---

Published as a conference paper at ICLR 2026
Bian & Jiang (2021) proposes a continuous-time value iteration algorithm for solving HJB equations without the need for a stabilizing initial policy, while Faradonbeh & Faradonbeh (2023) introduces an HJB-based actor-critic network with theoretical guarantees for the lifinite-horizon case. Similarly, Lee & Sutton (2021) develops two policy iteration algorithms that compute comprehensive solutions to HJB equations. In contrast, Jia & Zhou (2022b) introduces temporal -ifferee learning based algorithm to deal with continuous-time problems via discretization. Building on this line, Jia & Zhou (2022a; 2023) uses rigorous algorithms and strong theoretical foundations study single-agent CTRL with stochastic dynamics. Other approaches include Yang et al. (2021), which develops robust actor-crithe framework for nonlinear systems with unmodeled dynamies; Shilova al. (2024), which leverages PINNs with e-scheduling to approximate value functions and empirically outperforms discrete-time RI baselines; and Yildiz et al. (2021), which integrates neural ODEs with Bayesian inference to model uncertainty in state evolution and proposes continuous-time actor-critic algorithm that mitigates challenges such as Q-function vanishing and poor discretization. In contrast, CT-MARL remains relatively underexplored compared to the substantial studies for single-agent settings. For example, Luviano & Yu (2017) solves the multi-agent pathfinding problem using fuzzy Q-learning, while Jiang et al. (2023) proposes model-based value iteration algorithm tailored for continuous-time multi-agent systems. Beyond these examples, only a limited number of studies have addressed CT-MARL, highlighting the importance of contributions.
2.2 SOLVING HJB EQUATIONS VIA PINNS
In single-agent optimal control or multi-agent cooperative settings, value functions are characterized as the viscosity solutions to HJB equations (Crandall & Lions, 1983), which are the first-order nonlinear parabolic PDEs. However, solving HJB equations with conventional numerical methods is computationally intractable in high-din sigh--imensional settings due to CoD (Osher et al., 2004; Osher & Shu, 1991). Recent studies show that PINNS can mitigate CoD by leveraging their Monte Carlo nature when PDE solutions are smooth (Weinan et al., 2021). PINNS approximate value functions using trainable neural networks by minimizing PDE-driven loss functions, including boundary residuals (Han & Long, 2020; Han et al., 2018). PDE residuals (Bansal & Tomlin, 2021; Zhang al., 2024a;b), and supervised data derived from numerical solvers (Nakamura- Zimmerer Nakmeura-Zimmerrr al., 2021). Notably, recent studies demonstrate that integrating HJB-based PINNs with Proximal Policy Optimization (PPO) leads improved performance over standard PPO in continuous-tinn single- agent MuJoCo environments (Mukherjee & Liu). However, solving CT-MARL problems through the integration of PINNs and RL remains an open and unexplored area of research.
3 METHODOLOGY
In this section, we present our Value Iteration via PINN (VIP) framework for solving CT-MARL problems. 1) We first formulate the problem setting as the cooperative multi-agent deterministic system in continuous time. By adopting CTDE paradigm, 2) we approximate the H.JB-based value function using PINN equipped with residual and anchor losses. To further enhance the stability and accuracy of both the value and its gradient, we incorporate additional VGI loss that iteratively refines both value and value-gradient accuracy along sampled trajectories. Building upon this critic, 3) we derive a continuous-time advantage function directly from the HJB residual to guide decentralized policy updates, which enables each agent to update its actor in a manner consistent with continuous-timg optimality conditions.
3.1 PROBLEM FORMULATION
In this paper, we focus multi-agent cooperative settings. Following the continuous-time control system framework (Yildiz et al., 2021; Lee & Sutton, 2021), we formulate the continuous-time multi-agent problem as tuple
M = ( (x, (14 1 N. f.r, (takxoo.p.
(I)
where X' C R is the state space and 24 = U4 x ... x Ux C R represents the joint action space of N agents. The global state and control input are represented by X and E U. Agent interactions occur over an infinite time horizon. The multi-agent system evolves according to time-invariant nonlinear dynamics defined by श f(x u), where f X' l - X' is the global dynamics function.
3

---

Published as a conference paper at ICLR 2026
We define π + u as the decentralized joint policy ( Tx). All agents share global reward r : X' x R. p E (0, 1] is the discount factor or time-horizon scaling parameter. Unlike standard formulations that assume a fixed time step, consider a strictly increasing sequence of decision times (4x J220 with variable gaps & = 1x+11 4 > 0. In this paper, we assume that 24 is compact and convex; f is Lipschitz continuous; is Lipschitz continuous and bounded.
Definition 1 (Value Function of Multi-agent Systems). Given = (Ui..... uN) as joint control input, the optimal global value function is defined as
V(x) = max NEM
e
3.2 HJB AND POLICY LEARNING
For CT-MARL problems, we build HJB PDEs rather than discrete-time Bellman equations (Bell- man, 1966), which are ill-suited to continuous-timm settings. In this subsection, we explain how the HJB equations are leveraged to solve the CT-MARL problems. Specifically, network Ve parametrized by set of weights and biases 0, and describe how the global value function V6 trained and how each agent's policy тo; updated, such that the overall procedure serves as a continuous-ting time analogue of actor-critic policy iteration. The convergence of HJB-bases PINNs for learning Vo has been established in Meng et al. (2024), providing theoretical support for our approach.
define a value
3.2. CRITIC LEARNING WITH HJB
The optimal value function is fully represented by the HJB equation. This PDE encodes the fun- damental relationship between reward, system dynamics, and value evolution, and serves as the theoretical backbone for our value learning method. The following lemma states this connection formally in the multi-agent cooperative setting.
Lemma 3.1 (HJB for Multi-agent Systems). For all X, the value function V(z) is the optimal solution to satisfy the following HJB PDEs
-pV(x) + f u²) +r(z, =0
where optimal control input = arg max H(I, V (z)). The Hamiltonian H is defined as H V. (=) u) + +r(z,
The proof is attached in Appendix A.I.
To approximate differentiable value functions, we solve the HJB PDEs in Eq. 3. Since conventional numerical methods become intractable beyond six state dimensions (Bui et al., 2022), instead employ PINNs, which approximate value functions by minimizing PDE residuals. Specifically, we define the HJB PDE residuals as follows. Here, 2e is the optimal control input in this paper.
Ra(x)) = ( + Vz, Vo(x:) frz, u.) + r(2,, uz).
(4)
and minimize the residual loss Lres (ax)),, towards zero during model refinement.
3.2.2 POLICY LEARNING
While analytical optimal control laws be derived in some cases by maximizing the Hamilto- nian (Nakamura-Zimmert et al., 2021; Shilova et al., 2024), such closed-form solutions are not available in complex multi-agent systems like MPE or MuJoCo. To overcome this challenge. an actor network to generate control inputs, replacing the need for analytical controls in the critic network used to approximate value functions. The actor and critic networks are refined iteratively until convergence, enabling the actor network to approximate optimal control policies. During train- ing, we compute a continuous-time advantage function derived by residual Ra(x:). This advantage function is used for policy gradient, where each agent's decentralized policy na, (w: xx) is optimized to maximize long-term return.
use
Lemma 3.2 (Instantancous Advantage). Assume the one-step funuctio over a short interval 8t 0 be
Q u+) r(w,, ux) öt pól (FE48t).
4

---

Published as a conference paper at ICLR 2026
Then the instantancous advantage satisfies Ax ue) -VV(x)) Ve, Vo) f(0 (((,))))
(6)
The proof is attached in Appendix A.2. With the critic's instantaneous advantage
Az un) = - -Vo((z)) + v. Vo(zz) f(z, un) + +r(,,
(7)
we update each agent's policy network Te (u, 24) in a decentralized fashion. For agent minimize the negative expected advantage under the joint policy
Lp. = - (Ie: Uz) log Tá. (us xz);
(8)
Here, v = (us, u-z) denotes the joint action, where un ~y Tá is sampled from agent i's policy, and u-; represents the actions of all other agents, sampled w-; ~ T
Since our actor is trained by using the advantage function, it important to ensure that this update direction leads to policy improvement. The following lemma formalizes this property and shows that gradient step on our actor loss yields value-increasing policy update.
Lemma 3.3 (Policy Improvement), Let vold be the current joint policy and Tnew the updated policy after gradient step the actor loss Lp with sufficiently small step size. Then
QUeen (x+,, uz) 2 Q Taid (I'r ut).
The proof can be found in Appendix A.3.
3.3 VALUE GRADIENT ITERATION MODULE
The performance of continuous-tin time control policies depends explicitly on the accuracy of the value, which in turn depends not only the precision of its own approximation but also on the correctness the value gradient V, (z). Recent studies have demonstrated that the accuracy of the value directly affects the learned control policies (Zhang et al., 2024b). However, enforcing the HJB PDE through residual loss alone does not guarantee accurate value gradients. Because in -ineeniienni multi-agen agent systems, small gradient errors be significantly amplified by coupled dynamics, leading to inaccurate policy gradients. To address this limitation, we introduce the VGI module, which explicitly propagates value gradients along sampled trajectories and refines the gradient estimates in a self-consistent manner. By iteratively updating V, Vxr) using the local dynamics and the current value approximation, VGI provides sirectorr-aligneee correction signal that complements the global PDE constraint, resulting in substantially more accurate value-gradient learning.
Definition 2 (VGI Gradient Estimator). Given small time step At, the VGI estimator of the value gradient at (Ie, u,) is defined by
V V2,V V2, + e -pAi I uz) V Vetaa V(I++A))-
(10)
The VGI target in Eq. 10 can be interpreted as a one-step unrolling of the Bellman equations in the space of gradients. The first term captures the instantaneous contribution of the local reward gradient, while the second term propagates the downstream value information through the Jacobian of the system dynamics. This construction resembles a -ini-discretized version of the value gradient flow and provides a practical surrogate for supervised gradient learning in the absence of ground-truth derivatives. The derivation process is posted at Appendix A.4. To justify why the VGI refinement is stable and converges to promising gradient estimate, we analyze its update rule as fixed-point iteration.
Theorem 3.4 (Convergence of VGI). Let G IRd Idd be defined as
G(C) Vz, r(r,, U4 At + ppptppttaa J u+) TC,
(11)
and assume the dynamics [ u.) I| is bounded. Then G is a contraction, and the sequence ((k-1) G(((k)) converges unique fixed point Ç* Rdd.
5

---

Published as a conference paper at ICLR 2026
The proof is detailed in Appendix A.5
Rather than introducing a separate network to predict value gradients, we directly compute the automatic derivative of the shared PINN critic (z)).. This gradient is then trained to match the VGI-generated target defined in Eq. 10. Specifically, we minimize the mean squared error between the computed and target gradients
Sgi
V Vo(x+) 8ell2:
(12)
where 9: ut) a fu " Vi.-sr (Z(+ Here, r0 uy) denotes reward model and Jú (14, uz) represents dynamics model, where e and t are respective network parameters.
3.4 IMPLEMENTATION DETAILS
While the previous sections introduced our continuous-time actor-critic framework and the VGI module for value- gaalieggaddent consistency, several practical considerations are essential to make the overall method operational and effective.
3.4.1 DYNAMICS MODEL AND REWARD MODEL
In a continuous- time setting, the true dynamics are given by ă = f(z, u). but directly Iearning f via #+ as a supervision target is pretty unstable in practice. Instead, we adopt a discrete-time model-based approach (Sutton, 1991; Hafner et al., 2019) that train neural network S (3t,, u.)
At
to predict the next state T1 via
Laya [5 uz) - xe+
(13)
After learning fu. we recover the continuous time derivative by finite differences fa(u,
At
Similarly, we fit reward network ( uz) to the observed instantancous reward
Lew [ u.) rz
(14)
Both fo and rs are trained jointly, enabling us to compute the VGI module's target.
3.4.2 ANCHOR LOss FOR CRITIC NETWORK
In addition the HJB residual loss, we incorporate TD-style anchor loss improve both the stability and accuracy of value learning. While the residual loss enforces the correctness of the value gradient, does not constrain the value of V(x). Terminal-condition losses can provide such supervision, but they often rely access to well-defined terminal targets, which may be unavailable complex continuous control environments such as MuJoCo. In these cases, the anchor loss offers an additional source of value landscape, helping the critic produce reasonable value approximations even when terminal rewards are sparse, delayed, or difficult to specify. We define one-step return
Rg r(z,, un At + -PAt (I++
(15)
The anchor loss then enforces the value network to match these returns
Lanchur Vo(xz)) Rs ||
(16)
The overall critic objective combines all four losses
Luntal
Lies
+
Aunchor Lanchar
Xg Levgi
(17)
HIB residual
TD anchor
consistency
Here Aunchors Ag are tunable weights balancing PDE residuals, malue-bootstra anchoring, and gradient consistency. In practice, we jointly train the reward model, dynamics model, and value network using the data from the current trajectory. The detailed training process is listed in Appendix A.6.

---

Published as a conference paper at ICLR 2026
Cumulative Reward
Ant
Cumulative Reward
Walker
Cumulative Reward
HalfCheetah
20
20
0
-4
2500 5000 7500 10000
2500 5000 Episode
7500 10000
5000 7500 10000 Episode
Episode
Cumulative Reward
Ant 2x4d
Cumulaatve Reward
Swimmer
Cumulative Reward
Ant 4x2
20
20
20
10
01
-10-
2500 5000 7500 10000 Episode
2500
5000 Episode
7500
10000
0
2500 5000
7500 10000 Episode
VIP
VIP w/o VGI
DPI
IPI
ODE
HJBPPO
Figure 2: Performance across continuous-time multi-agent MuJoCo settings. The y-axis shows the mean cumulative reward.
EXPERIMENTAL RESULTS
We evaluate our Value Iteration via PINN (VIP) method on two continuous-time multi-agent benchmarks: MPE (Lowe et al., 2017) and multi-agent MuloCo (Peng et al., 2021). In addition, we design a didactic benchmark, coupled oscillator (see details in Appendix B.1), to analyze value gradient approximations. This case study is easy to follow and enables the numerical computation of true values and their gradients, which provides a clear and interpretable setting to validate the effectiveness of VIP. Our experiments
designed to answer the following four key questions: (1) Overall efficacy: Does the proposed VIP model outperform existing continuous-time RI. baselines in these environments? (2) VGI ablation: How much does the VGI module contribute to final performance and training stability? (3) PINN design choice: How does activation function choice and loss weighting the critic network affect the performance of VIP? (4) Time discretization impact: How well do discrete-time and continuous-time methods perform under arbitrary or unfixed time intervals? (5) Comparison berween VIP and discrete- time baselines: Could discrete-tine -imee baselines perform comparable at continuous-ting settings?
Ground
VIP
VIP w/o
1.10
1.10
1.10
0.86
0.24
1.05
0.220
0.85
9 1.00
0.22
1.00
0.218
1.00
0.84
0.20
0.216
0.83
0.95
0.95
0.214
0.95
0.82
0.18
0.81
1.000
0.90
0.0000
d1
d1
V.V of Ground Truth
V.V VIP
V,V
1.10
1.10
0.1100
1.10
0.1075
0.0900
1.05
0.1050
0.0875
0.1025
0.1000
0.95
0.0825
0.95
0.95
0.0975
0.0800
0.0950
0.90
8.90
1.00
0.90
1.00
1.05
0.90
1.00
Figure 3: V. and V contour using VIP w/ VGI and w/o VGI dj-dg frame.
7

---

Published as a conference paper at ICLR 2026
Benchmarks. We evaluate method against competitive baselines across eight experimental settings involving up to six agents and 113 state dimensions. We extend the MPE framework to continuous-timg formulation by using suriable-step Euler integration scheme, where the time interval At is sampled from a predefined range at each step. Experiments are conducted on the cooperative navigation and predator prey environments. Similarly, adapt MuloCo to continuous- time settings and evaluate on suite of multi-agent locomotion tasks, including ant (2 X 4, 2 x 4d, 4 2), walker, swimmer, and cheetah (6 1). Further implementation details of these benchmarks are provided in Appendix B.2 and B.3. Lastly, we introduce simple yet illustrative couples oscillater environment to highlight the behavior of exact value functions, value gradients, and the relative performance of different methods under controlled setting.
4.1 BASELINE METHODS
To evaluate CT-MARL framework VIP, we compare against four continuous-ting policy iteration baselines and include an ablated variant our method without VGI: CT-MBRL (ODE) (Yildiz et al., 2021): continuous-time model-based RL approach that learns system dynamics via Bayesian neural ODEs. This method uses an actor-critic frame- work to approximate state-value functions and derive the continuous-time optimal policies. Differential Policy Iteration (DPI) (Lee & Sutton, 2021): a
Accumulated Roward
Cooperative Predator Prey
Cooperative Navigation
model-based method with differential policy iterations that alternates between (i) solving the HIB PDEs to approxi- mate continuous-time value functions and (ii) updating the policy by following the instantaneous gradient of value approximations. Integral Policy Iteration (IPI) (Lee & Sutton, 2021): partially model-free approach with integral policy iterations that reformulates the value function as a continuous integral, avoiding explicit differentiation during policy improvement. HJBPPO (Mukherjee & Liu): recent method that employs a LINN-based critic to approximate the HJB residual and leverages standard PPO-style policy optimization scheme to guide agent learning. In our experimental settings, we discretize the integral, roll out trajectories accumulate rewards, and fit a policy minimize the resultant value functions. Ablation (w/o VGI): an ablated version of VIP without VGI, which isolates the efficacy of value gradient refinement. In addition, to evaluate how VIP compares with DTRL methods, we incorporate three widely used baselines in experiments: MATD3 (Zhan et al., 2021), MAPPO (Yu et al., 2022), and MADDPG (Lowe et al., 2017).
-20
500
-0000
2000
4000
8000
10000
-1500g
2000
10000
Epizode
Episade
VIP wwo VaI
DPI
IP1
ODE
VIP
HJBPPO
Figure 4: Performance across continuous-time MPE settings. The y-axis shows the mean cumulative reward.
4.2 RESULTS ANALYSIS
Model Performance. We evaluate all RL methods in MPE and MuJoCo environments using five random seeds and report the mean cumulative reward curves in Fig. 2 and 4. The results show that VIP with VGI consistently converges fastest and achieves the highest final return across all tasks, which empirically validates the efficacy of integrating PINNs with RL. As traditional time-dependent HJB equations are PDEs with a single boundary condition at terminal time, PINN may struggle to backpropagate the correct physics information when relying solely boundary values, often resulting in poor value approximations (Krishnapriyan et al., 2021). This limitation becomes even more severe in the infinite-horizon setting, where the HJB formulation is iime-independent and terminal losses are no longer available. To address this limitation, we incorporate the anchor and VGI loss terms in Eq. 17, which capture the landscape of value and its gradient so that PINNs converge to the true values. Ablation results further confirm the importance of VGI: removing VGI leads to significantly lower cumulative rewards across all experiments. This observation is consistent with the conclusion in the previous studies (Zhang et al., 2024a), which further strengthens that accurate value gradient approximation is crucial for effective PINN training and policy improvement. To further demonstrate the importance of VGI, we revisit the didactic example, generate 400 rollouts from sampled initial states, and compute the average value using models with and without VGI.
8

---

Published as a conference paper at ICLR 2026
Ant 2x4
Walker
Cheetah
Accumulsted Reward
-2
2
-20
1000
2000
3000 Episode
4000
5000
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Episode
Episode
Swimmer
Cooperative Predator Pray
Cooperative Navigation
Accumulated Reward
20
-10
250
10
-15
-20
-700
-10
-25
-1000
1000
2000 Episode
3000
4000
5000
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Episode
Episode
VIP via ReLU
VIP via Tanh Tanh (Ours)
Figure 5: VIP performance with ReLU and Tanh activation functions in MuJoCo and MPE settings.
As shown in Fig. 3, the value contour projected onto the d1-d2 frame reveals that the model with VGI closely matches the ground truth in both struc-
Min Distance to Pray
3
ture and scale, while the model without VGI pro-
1 2
duces significantly biased approximations. These results confirm that adding VGI necessary improve the accuracy of value approximations. A detailed comparison of the corresponding value gradients is also illustrated in Fig. 3. Furthermore, we evaluate the sensitivity of each loss term for the critic network by measuring the minimum distance to prey in Fig. 6, which highlights the critical role of PINN in refining value and policy networks.
w/o Residual
w/o Anchor
wio vGi
vIP
Figure Ablation study of different loss terms
in critic network for cooperative predator prey.
Choice of PINN Design. We first examine the impact of activation function choice when in- corporating PINN-based losses into critic network refinement. Specifically, train VIP using ReLU and Tanh activations in both MuJoCo and MPE environments and report the accumulated rewards in Fig. and Fig. 11 (Appendix C.2). The results show that the VIP with Tanh consis- tently achieves higher accumulated rewards than the one with ReLU across all tasks. This ex- periment indicates the importance of activation function choice when using PINN-based losses to refine the critic network and has a consistent conclusion with the previous studies (Bansal Tomlin, 2021: Zhang et al., 2024b). The good performance of Tanh can be attributed to its smoothness and differentiabiiltyy which are particularly important when using PINNs to solve PDEs. PINNs often use fully- connected network ar- chitectures and rely on
Accumulated Reward
Cosperative Navigation
Cooperative Prey
-10
nuto-diffeernniatioon to compute value gradi- ents for PDEs. The PINN residuals, includ- ing PDEs, are further optimized using gradi- ent descent. Smooth ac- tivation functions like Tanh support stable and accurate gradient flow throughout training, enabling more effective value approximations. In con- trast, VIP with ReLU often encounters aero-gradient regions during backpropagation, which re- sults in gradient explosion for deeper network architectures degrades the learning of value functions due to insufficient nonlinearity. Therefore, the choice of a smooth activation function like Tanh is better suited for physics-informed learning, thereby ensuring more accurate approx-
-300
-20
-1000
-1500
1000
2000
3000
4000
5000
-40
1000
2000
3000
4000
5000
Episode
Episode
VIP
Revised Residual Weight
Revised Anchor Weight
Figure 7: VIP performance with different weight settings in critic losses.

---

Published as a conference paper at ICLR 2026
imations of the value functions. We also investigate the impact of weight parameters for PINN loss terms during VIP training. Specifically, we evaluate three configurations in Eq. 17: 1) bal- anced weights for all loss terms; 2) large weight for anchor loss while keeping the others balanced; 3) a large weight for HJB residual with balanced weights for the remaining terms. As shown in Fig. 7, the best accumulated reward performance is achieved when all loss terms are properly balanced. Imbalanced weight settings yield stiffiness in the training dynamics of PINNS (Wang al., 2021), which makes the dominant loss term (with the largest weights) converge faster than the others (Wang et al., 2022). In our experiments, such an imbalance causes VIP only to satisfy the PDEs residuals or anchor loss during training, ultimately leading to poor value approximations.
Time discretization impact. Lastly, we evaluate the robustness of VIP and well-trained MADDPG model (Lowe et al., 2017) by generating rollouts with varying time intervals in the didactic environment and computing the average return across these rollouts. Notably, in the original discrete-time training setting with a 4-dimensional state and 1-dimensional action space, MADDPG can already achieve near-optimal performance. Fig. 8 illustrates that VIP maintains a nearly constant return across different time intervals, whereas MADDPG's performance degrades signifi- cantly as the interval increases. This result highlights the advantage of VIP in continuous- time multi- scenarios. DTRL algorithms' comparison. Figure compares VIP with three widely used discrete-
Average Return
-10-
-15-
-20-
-25
VIP (CT-MARLI
(DT-MARL)
-30
0.05
0.06
0.07
0.06
0.09
0.10
Time Interval
Figure 8: CT-MARL and DT-MARL perfor- mance under different time intervals.
Cumulaaive Reward
Ant 2x4d
Ant 4x2
Half Chestah
Swimmer
20
10
-10
-20
-20
-15
-10
-20-
2500 5000 Episode
7500
10000
2500 5000 Episode
7500
10000
2500
5000 Episode
7500 10000
2500 5000
7500 10000 Episode
MATD3
MADDPG
MAPPO
VIP
Figure 9: Comparison between VIP and DTRL baselines.
time MARL baselines (MATD3, MADDPG, MAPPO) across several multi-agent MuJoCo tasks. We observe that all discrete-ting -imetee-tmmeeee methods' performance degrades substantially in tasks such as Ant and HalfCheetah. This degradation not surprising. Discrete-time Bellman backups implicitly assume fixed and sufficiently small time steps, and their value estimates become biased or unstable when the underlying system evolves in continuous time with arbitrary or -rrenment-deeendentt time intervals. Moreover, large integration steps amplify approximation errors in both the critic update and the target network, leading to inaccurate temporal-differenc objective functions and unstable actor gradients, which aligns well with the observation in other works (Tallec et al., 2019; Park al., 2021; De Asis & Sutton, 2024).
CONCLUSION
We propose novel approach that integrates PINNs into the actor-critic framework solve CT-MARL problems. Specifically, we approximate value functions using HJB- PINNs and introduce VGI to improve value approximations, thus mitigating the adverse impact of inaccurate value approximations on policy learning. We validate our VIP across continuous-time variants of MPE and MuloCo environments and empirically demonstrate that VIP converges faster and achieves higher accumulated reward compared to baselines of SOTA. Furthermore, we investigate the importance of activation function choice and loss term weighting for VIP performance. In summary, our proposed VIP offers promising approach to solve CT-MARL problems. Since our current formulation is based on the HJB equation, we focus on cooperative settings; extending VIP to HJl-based formulations that handle competitive mixed-m motivee scenarios is a compelling direction for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Science Foundation under Grant No. 2418106.
ETHICS STATEMENT
This work focuses on decision decision-making under continuous- multi-agent systems. All experiments conducted entirely in simulation and do not involve human subjects personal data.
REFERENCES
Somil Bansal and Claire J Tomlin. DeepReach: A deep learning approach to high-dimensional reachability. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1817-1824. IEEE, 2021.
Richard Bellman. Dynamic programming. science, 153(3731):34 1966.
Richard Bellman, Robert Kalaba, et al. Dynamic programming and modern control theory, volume 81, Citeseer, 1965.
Tao Bian and Zhong- Ping Jiang. Reinforcement learning and adaptive optimal control for continuous- time nonlinear systems: A value iteration approach. IEEE transactions on neural networks and learning systems, 33(7):2781-2799 2021.
Sebastian Brechtel, Tobias Gindele, and Rüdiger Dillmann. Probabilistic decision-making under certainty for autonomous driving using continuous pomdps. In / 17th international IEEE conference on intelligent transportation systems (ITSC). pp. 392-399. IEEE, 2014.
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela Schoellig. Safe learning in robotics: From learning- -baend control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5(1):41 1-444, 2022.
Minh Bui, George Giovanis, Mo Chen, and Arrvindh Shriraman. Optimizeddp: An efficient, user- friendly library for optimal control and dynamic programming. arXiv preprint arXiv:2204.05520, 2022.
Michael Crandall and Pierre-Louis Lions. Viscosity solutions of Hamilton-Jacobi equations.
Transactions of the American mathematical society, 277(1)11-42, 1983.
Kris De Asis and Richard Sutton. idiosyncrasy time-discretization in reinforcement learning.
arXiv preprint arXiv:2406. 02044066.14555. 14951, 2024.
Kenji Doya. Reinforcement learning in continuous time and space. Neural computation, 12(1):
219-245, 2000.
Mohamad Kazem Shirani Faradonbeh and Mohamad Sadegh Shirani Faradonbeh. Online reinforce- ment learning in stochastic continuous-time time systems. In The Thirty Sixth Annual Conference Learning Theory, pp. 612-656. PMLR, 2023.
Amal Feriani and Ekram Hossain. Single and multi-agent deep reinforcement learning for ai-enabled wireless networks: A tutorial. IEEE Communications Surveys & Tutorials, 23(2):12261225 1252, 2021.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp.
PMLR, 2019.
Jiequn Han and Jihao Long. Convergence of the deep bsde method for coupled fbsdes. Probability Uncertainty and Quantitative Risk, 5(1):1-33, 2020.
Jiequn Han, Aruuff Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34)850058811) 2018.

---

Published as a conference paper at ICLR 2026
Ammar Haydari and Yasin Yilmaz. Deep reinforcement learning for intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(1): 13(1):11322, 2020.
Kazufumi Ito, Christoph Reisinger, and Yufei Zhang. A neural network- based policy iteration algorithm with global -upuurrlln convergence for stochastic games domains. Foundations of Computational Mathematics, 21(2):331-374 2021.
Yanwei Jia and Xun Yu Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research, 23(275):1-50 2022a.
Yanwei Jia and Xun Yu Zhou. Policy evaluation and cumporal-differenn learning in continuous time and space: A martingale approach. Journal of Machine Learning Research, 23(15)): 2022b.
Yanwei Jia and Xun Yu Zhou, q-learning in continuous time, Journal of Machine Learning Research, 24(161):1-60 61, 2023.
Yi Jiang, Weinan Gao, Jin Wu, Tianyou Chai, and Frank L Lewis. Reinforcement learning and cooperative Hoo output regulation of linear continuous multi-agent systems. Automatica, 148:110768, 2023.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informe neural networks. Advances in neural information processing systems, 34:26548 265660, 2021.
Jacyoung Lee and Richard S Sutton. Policy iterations for reinforcement learning problems in continuous time and fundameanna theory and methods. Automatica, 126:109421, 2021.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Picter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettivy environments. Advances in neural information processing systems, 30, 2017.
David Luviano and Wen Yu. Continuous-time path planning for multi-agents with fuzzy reinforcement learning. Journal of Intelligent Fuzzy Systems, 33(1): (3(1):491-501 2017.
Yiming Meng, Ruikun Zhou, Amartya Mukherjee, Maxwell Fitzsimmons, Christopher Song, and Jun Liu. Physics-informed neural network policy itcration: Algorithms, convergence, and verification. arXiv preprint arXiv:2402. /0/ 19, 2024.
Amartya Mukherjee and Jun Liu. Bridging Physics- Informed Neural Networks with Reinforcement Learning: Hamilton-Jacohe Jacobi-Bellma Proximal Policy Optimization (HJBPPO). In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems.
Tenavi Nakamura-Zimmeraa Qi Gong, and Wei Kang. Adaptive deep learning for digh-dimensional Hamilton Jacioonn--aii----a Belllln equations. SIAM Journal on Scientific Computing, 43(2):A1221
2021.
Stanley Osher and Wani Shu. High-order essentially nonoscillatory schemes for Hamilton-Jacoa equations. SIAM Journal on numerical analysis, 28(4))907-92 1991.
Stanley Osher, Ronald Fedkiw, and K Piechor. Level set methods and dynamic implicit surfaces. Appl. Mech. Rev., (7(3):155B155 2004.
Seohong Park, Jackyeom Kim, and Gunhee Kim. Time iettretizaaiinnnnvriann safe action repetition for policy gradient methods. Advances in Neural Information Processing Systems, 34:267-279, 2021.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34: 12008-1222 2021.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irreuulrrly-smppled time series. Advances in neural information processing systems, 32, 2019.
Ali Shavandi and Majid Khedmati. A multi-agen agent deep reinforcement learning framework for algorithmic trading in financial markets. Expert Systems with Applications, 208: 18124, 2022.
12

---

Published as a conference paper at ICLR 2026
Alena Shilova, Thomas Delliaux, Philippe Preux, and Bruno Raffin. Learning HJB viscosity solutions with PINNs for continuous-ting reinforcement learning. In ICML 2024 Workshop: Foundations of Reinforcement Learning and
and Perspectives, 2024.
Yeonjong Shin, Jéröme Darbon, and George Em Karniadakis. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs. Communications in Computational Physics, (855):2442-20772 2020.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 264):160- 1991.
William Talbot, Julian Nubert, Turcan Tuna, Cesar Cadena, Frederike Dümbgen, Jesus Tordesillas, Timothy D Barfoot, and Marco Hutter. Continuous-timg state estimation methods in robotics: A survey. IEEE' Transactions Robotics, 2025.
Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust time discretization. In International Conference on Machine Learning, pp. 6096-6104, PMLR, 2019.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamneki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. nature, 575(7782):3500 354, 2019.
Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research, 11999)) 2020.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informee neural networks. SIAM Journal on Scientific Computing. 43(5): A3001, 2021.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110768 2022.
Xuefeng Wang. Xinran Li, Jiawei Shao, and Jun Zhang. Ac2c: Adaptively controlled two-hop communication for multi-agent reinforcement learning. arXiv preprint arXiv:2302.125115 2023.
E Weinan, Jiequn Han, and Arnulf Jentzen. Algorithms for solving high dimensional PDEs: from
nonlinear Monte Carlo to machine learning. Nonlinearity, 35(1):278, 2021.
Yongliang Yang. Weinan Gao, Hamidreza Modares, and Cheng-Zhong Xu. Robust actor-critic learning for continuous-tin time nonlinear systems with unmodeled dynamics. IEEE Transactions Fuzzy Systems, 3066)2101-2212, 2021.
Cagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Continuous-time model-based reinforce- ment learning. In International Conference on Machine Learning, pp. 12009-12018. PMLR, 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances in neurul information processing systems, 35:24611 01:22411-24666 1-24624, 2022.
Mengying Zhan, Jinchao Chen, Chenglie Du, and Yuxin Duan. Twin delayed multi- agent deep deterministic policy gradient. In 202/ IEEE International Conference on Progress in Informatics and Computing (PIC), pp. 48-52. IEEE, 2021.
Lei Zhang, Mukesh Ghimire, Zhe Xu, Wenlong Zhang, and Yi Ren. Pontryagin neural operator for solving general-sum differential games with parametric state constraints. In 6th Annual Learning for Dynamics& & Control Conference, pp. 1728 1740. PMLR, 2024a.
Lei Zhang, Mukesh Ghimire, Wenlong Zhang, Zhe Xu, and Yi Ren. Value approximation for two-player general-sum differential games with state constraints. IEEE Transactions Robotics, 2024b.
13

---

Published as a conference paper at ICLR 2026
A MATHEMATICAL PROOF
A.1 PROOF OF LEMMA 3.1
Proof. Given all X' with small horizon At > O, apply Taylor expansion to the definition of (1)) in Eq. 2 to derive the HJB PDE as follows
(x)
max wEll
e-plr- u)dr
++A
e --(t--r((((T), u(T)) max max uEll
((x())
max wEll
l+At
--p(T- ((T))))))()))))) u(T)) +mxx max
r-peetSt-rrs(ss (t) At))ds
max wEll
wEll
I+A
e - ((x u(T)) pae
e-p(o-)) + A0), u(s + At)
-max WEU
max uell
++
e C-PT-0 w(T))
( (x(T))
max
max uEll
J+A
max ut e påt
PAi max
((((++Att)
well
"EM
(E) (1 påt o(t)) max (z) f(z, u) o(A)) well
wEl
By canceling out ((x) on both sides of the above equality, we obtain that
-pV(x)& + max (Vxx(V)V(x)) [(x u) w)))+ wEll
Dividing by At and letting At + O, we have that
-pV(x) + max (x) u) + r(r, u)) 0.
well
Therefore, the V(x) is the optimal solution to the following HJB PDEs:
u²) + r u²) = 0,
Here optimal control imput =² arg mex H(x, V, V,V(z)) where H is the Hamiltonian defined as H (=)) f(x, u) + r(2, u..
A.2 PROOF OF LEMMA 3.2
Proof. Recall the one-step Q-function over a short interval öt > 0, where u is the optimal control input
Q un) r(x,, u2) öt -éé (11+66))
For small ôt we have the first-order Taylor expansion in state:
V (x148)) (zx, Va, Vxx,) f(x,, uz) öt o(ôt).
Similarly, e- pót
pôt + o6t)).
Plugging both expansions into Q u) gives
Q(Ie, u4) = r(ze, 44) 8t + (1 - p 5t + o(5t)) Vex V(x.)" f(x,, uz) St + o(8t)
r((r, u.) 81 + (x::) + V V(::) [ ((x) u.) oV (xx)) o(80).
Subtract V(xx) and discard the higher-order term:
Q(xi, ux) V (xx) (x,V V., V(x.) f(z, u.) r(xe, u)]ot o(ōt).
Dividing by 8t and letting 5t - 0 yields the instantaneous advantage density
Q(x,, un) ((x)
A(x,, wx) lim
(xx) Va, (xx) f(x,, ue) + r(xe: u.)
86
This completes the proof.
14

---

Published as a conference paper at ICLR 2026
A.3 PROOF OF LEMMA 3.3
Policy Improvement via State-Action Value Function. We consider the standard policy improvement step, where the new policy obtained by maximizing the state-action value function
Noew (It) = arg max (Iz, ue). aEMd
where the one-step Q-function with small öt > 0 is defined as:
(z uz) = r((1,, ue öt + pót V (I(+61)
r u4) öt pôt By' [Q ul(+6))]
Since assume the goal state function stays invariant, only define the Q S0 e r, Because the new policy new yields equal or higher value in expectation
EuNtto 1Q (21, we) 2 Euvotas [QTH (x2, ue
Then we can have
@Tin Tto 5t + -ot -P waa 1Q (Itz: Ut, )])
S rto öt + ephbt
pôt (Eu., ~iww 1Q The (Itr Ut, U1,
Tto öt + pplr,e, öt - E ppàl (Bugrntaa TQ (Iiz, uk,)))
rto ôt pll 5t - c (Eus, ~om TQ ( (Itz, Ue2)l)
Too ōt + E eplrr - e pler St e ppôe (Bug 1QTaa (Its: Uc, )1)
e ridt
A.4 DERIVATION OF DEFINITION
Proof. We consider the value function defined in Eq. 2 and follow the Proof of Lemma 3.1 to write the dynamic programming principle of (zx) as
V(xx) r(r,, de At t e - V
where is the optimal control input. Taking the gradient with respect to on both sides using the chain rule
Vx, (z,)) V ra, wz) + e e f uz) T Vansa (I++ae).
which matches the estimator proposed in Definition 2.
A.5 PROOF OF THEOREM 3.4
Proof. From the definition of VGI in Definition 2, applying first-order Euler step gives
E++Ar = + f(z,, u) u. o(At).
This yields the first-order VGI approximation
V2, V(r)) V2, +e -- V (1t, (Er, V Vasaa ((e+ts)). We can rewrite this approximation as an affine map
(=G(C)
where
b V r(s,, U/ At, A= ppi V (Z,, (.)) ( VaV(a.
15

---

Published as a conference paper at ICLR 2026
In Section 3.1, the joint dynamics function f(z u) is assumed time- iome-invariant, which implies its Jacobian v is also indeieeeeenden.. Consequently, the matrix A is time-invariant, and the update map can be written a single contraction map
G(() + As,
rather than family of time-indexed maps Gt. Assuming the dynamics have bounded Jacobian,
V v=f(xx, ut)l _ Ly.
obtain the bound
|| I e (1 -
Because we study high-frequency settings, At is chosen sufficiently small, which makes L, + O. so that 3 < 1. For any Ç1, Ç2 E R
[[G(G) G((2))l = || (2) s Bll C2ll,
Hence, G' is a contraction. Banach's s exeed-point theorem guarantees a unique fixed point (* and linear convergence ||<(a) - - _ 3 (1]((0) 1* Therefore, the value-gradient iteration converges,
completing the proof.
A.6 TRAINING ALGORITHM
We present the training algorithm for our proposed approach, Value Iteration via PINN (VIP), as follows
Algorithm 1 Value Iteration via PINN (VIP)
1: Init: value net Ve, policy nets (mw, J dynamics for reward ie
2: for I .......7 T do
3: Collect one rollout:
4: + env.reset()
5: for k = 1, K do
6:
sample decision time for each agent 1 N do
b is arbitrary time
7:
8:
U: ~ Tw, (u: x)
9:
end for set joint action u = (uI,.... uN) (z',r) +- env.step(u)
10:
11:
12:
append (z, u, r, x') to local rollout R
13:
+ =!
14:
end for
Dynamics and Reward Model learning on update v. 6 as per the Eq. 13 and 14.
16:
Critic update on R compute all losses Lres Lanchor, Lugi by Eq. 17.
18:
+ f ay Vo(...)
20:
Actor update for each agent for = 1,...,
.....,NN N do compute A(z, u) for all (z, u) R
22:
23: W/ + wi V., B(aa)ERR A(x, u) log Tw,
Tw, (u x)]) by Eq. 8.
end for
25:
end for
B ENVIRONMENTAL SETTINGS
B. COUPLED OSCILLATOR
We evaluate a two-agent coupled spring-damper system. Each agent E (1,2) controls one mass in pair of identical oscillators with linear coupling. The continuous-tima dynamics are
d Ui;
= 1,2,
(18)
0; = k
bez +
16

---

Published as a conference paper at ICLR 2026
where
Ij and v; are the position and velocity of mass
k = 1.0 is the spring constant, and 6 = 0.5 is the damping coefficient;
u, E -- Umax, Umax] is the control force applied by agent i, with Umax 10.
At each step the joint reward is
- (2) 22 A (z) x2)²
{uu n²)
with coupling strength > = 2.0 and control penalty = 0.01. We normalize by constant factor (here 1/10) that E [-1,0].
For the coupled oscillator with linear dynamics
á = Az + B
can compute the exact hnfinite-horizon LQR solution
1. Solve the continuous algebraic Riccati equation (CARE)
P PBR FBBRBB B P+Q=0
for the symmetric matrix P R
2. Form the optimal state-feedback gain
R--BBP.
3. The optimal control law is
u u'(x) = - K x,
u = x
where K; is the i-th row of K.
4. The corresponding optimal value function is the quadratic form
(x) Px,
whose exact gradient is
V V=)* 2P..
We use u (x), (*(x), and V (x) as ground truth targets when evaluating the precision of the policy, the error of the value function, and the consistency of the gradient.
B.2 CONTINUOUS-TINE MPE
We build on the standard MPE of Lowe et al. (2017), which simulates N holonomic agents in 2D world with simple pairwise interactions. In the original MPE each control step advances the physics by fixed time-step Afrxed 0.1 To evaluate our continuous-time framework under irregular sampling, we modify simulator so that each step the integration interval drawn randomly,
At; ~ Uniform( Atmin Atmax).
The underlying dynamics, observation and action spaces, reward functions, and task definitions remain exactly as in the original MPE. For the cooperative predator prry environment, we only control the predators (3 agents) action capture the prey (1 agent). While the prey is set up with random actions.
B.3 CONTINUOUS- TIME MULTI-AGENT MuJoCo
For nigh-dimensional control we adapt the discrete-ting time Multi-Agent MuJoCo suite (e.g. cooperative locomotion, quadruped rendezvous). By default MuloCo uses an internal physics integrator with base time-step of 0.01 s and repeats each action for Kixed frames, yielding an effective control interval Afixed s. We instead sample the number of frames per control step,
Kg
Uniform Integer(1 9),
so that each step advances by
At KR x 0.01 [0.01, 0.09]s
at random. All other aspects of the environment (observations, reward structure, termination condi- tions) kept identical to the original multi-agent MuJoCo tasks.
17

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENTAL RESULTS
Experiments were conducted on hardware comprising an Intel(R) Xcon(R) Gold 6254 CPU @ 3. 3.10GHz and four NVIDIA A5000 GPUs. This setup ensures the computational efficiency and precision required for the demanding simulations involved multi-agent reinforcement learning and safety evaluations.
C.1 VALUE GRADIENT COMPARISON
2005
Valee
2015
332
Gnalen
Timepp
Timesep
Gnaden
Tineeep
(a) Ground Truth Gradient
(b) VPI Gradient w/ VGI
(c) VPI Gradient w/o VGI
Figure 10: Value gradient comparison between using of VPI module.
To further demonstrate the effectiveness of VIP, we sample the same trajectory in the coupled oscillator environment and compute the value gradient from the analytical LQR solution, from VIP equipped with the VGI module (ours), and from VIP trained without VGI, respectively, Fig. 10 presents resulting 3-D surfaces. The surface panel (b) preserves the principal ridges and valleys of the ground truth, showing that the network recovers the correct geometric structure of V; its absolute error remains below 0.02 across almost all timesteps and gradient dimensions. In contrast, the surface in panel (c) is noticeably distorted: several peaks flattened, troughs misplaced, and the absolute error frequently exceeds 0.08. This comparison confirms that the VIP module critical for aligning the learned gradients with the analytical solution, therehy reducing bias and stabilising the HJB residual.
C.2 RELU VS TANH AT ANT x 4d AND ANT 4 x 2.
Fig. 11 compares VIP's learning curves when the policy network uses ReLU Tanh activations on Ant 2 x 4d and Ant x 2. Across both tasks the Tanh implementation converges faster and
Ant 2x4d
Ant 4x2
Accumulated Reward
20
20
0
-20
-20
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
Episode
Episode
VIP via ReLU
VIP via Tanh (Ours)
Figure 11: VIP performance with ReLU and Tanh Ant 2 4d and Ant x 2.
18

---

Published as a conference paper at ICLR 2026
attains higher plateau reward, whereas the ReLU version peaks earlier and then undergoes mild performance decay. The observation aligns with the earlier monh-versus-Reee analysis reported in the main paper Fig. 5: smoother activation functions mitigate gradient saturation and promote more stable policy updates. The additional evidence from the two Ant variants therefore reinforces our previous claim that Tanh is better suited for value-gradiees propagation in VIP.
C.3 STOCHASTIC NOISE & MODEL MISMATCH
Ave Distance to Target
2-
Predator Prey
Cooperative Navigation
No Noise
Low Noise
Mid Noise
High Noise
Figure 12: Performance under different noise levels.
Figure 12 evaluates the robustness of VIP under different levels stochastic perturbation injected directly into the dynamics, according to
T++A: f(z,, uc) At + Ec. (0
where we consider three noise scales: o² 0.1 (Low), 0.5 (Mid), and 1.0 (High). With low noise, VIP maintains strong performance across both Predator Prey and Cooperative Navigation, indicating that the learned continuous tinttuuuus--meeeeeo value function and its gradients remain stable under mild stochasticity. As the noise level increases, performance degrades noticeably. which is expected: our method is designed for deterministic continuous-timg systems, and stochastic settings fundamentally alter the associated HJB equations (leading to stochastic HJB or HJB -okelllla Since VIP does not incorporate additional components for modeling diffusion terms uncertainty, the degradation at high noise levels is consistent with the thcoretical scope of our formulation.
forms).
C.4 SENSITIVITY CHECK FOR At
-10
VIP (CT-MARL)
Average Return
-15
-20-
-25
-30
0.06 0.08 0.10 0.12 0.14 0.16 0.18 0.20
Figure 13: Average return under different At in coupled oscillator
For sufficiently small step sizes, VIP remains stable and achieves strong return, consistent with the contraction guarantee. As A/ increases, the first-order discretization of the continuous-time dynamics
19

---

Published as a conference paper at ICLR 2026
becomes less accurate, and the local linearization used in VGI and PDE no longer matches the true state evolution. Once At exceeds a task task deepedent threshold (c.g., At z 0.15 in the Coupled Oscillator). approximation eπrоrs accumulate across iterations and the performance begins to degrade. For very large At, the contraction conditions are effectively violated and the value-gradient update longer maintain stability, leading to a sharp drop in performance. This behavior matches theoretical interpretation of VGI: the bounded-Jacobing and small-At requirements are sufficient conditions for contraction, the method remains robust over a wide range of practical step sizes, but the performance will degrade sharply if the At far from the effective region of At.
Table 1: Hyperparameter settings used in all experiments.
Parameter
Value
Episode length
50
Replay buffer size
10a
Discount factor p
0.95
Soft update rate
0,001
Actor learning rate
0.0001
Critic learning rate
0,001
Dynamics model learning rate
0.001
Reward model learning rate
0.001
Exploration steps
1000
Model save interval
1000
Random seed
111-120
D HYPER- PARAMETERS
As Table shows, the exploration steps are used to delay the decay of the exploration rate: during the first 1000 steps, the exploration schedule remains fixed to encourage initial exploration. The soft update rate controls the target network update in the critic and value estimation, where adopt target network with an exponential moving average to stabilize bootstrapped training. This technique helps suppress oscillations in value learning and leads more accurate estimation of long-horizon returns.
Table 2:
Summary of neural network architectures used in our framework.
Network
Input Dimension
Architecture and Activation
Value Network Dynamics Network
State (d)
FC(128) - FC(128) FC(1), ReLU Tanh
State Joint Action (d + na)
FC(128) - FC(128) FC(()), ReLU
Reward Network
State Joint Action na)
FC(128) - FC(128) FC(1), ReLU
PolicyNet
Observation Time Interval (o + 1)
FC(128) + +F((188) FC(64) - FC(a), ReLU
Table 2 summarizes the architectures of the neural networks used in our VIP framework. All networks are implemented FC layers with hidden size 128 unless otherwise specified. The value network takes the concatenation of the global state and time (d) as input and outputs scalar value. In our implementation, we use the Tanh activation function for the value network, as it provides smoother and more stable gradient propagation, which is critical for PINN-based value approximation. To validate this choice, we conducted an ablation study comparing Tanh and ReLU activations in the previous section.
E THE USE OF LARGE LANGUAGE MODELS (LLMs)
We use LLMs writing assistant to polish/revise the paper.
20


Reviews:
[{"review_id": "5tA1jq3DWp", "reviewer": "Reviewer_5Xhd", "summary": "This paper extends prior work in continuous-time reinforcement learning (CTRL) from single-agent settings to cooperative multi-agent reinforcement learning (MARL). The authors focus on overcoming the challenges of large state dimensionality and accurate value approximation in continuous domains.\nTheir proposed framework combines Physics-Informed Neural Networks (PINNs) with a new Value Gradient Iteration (VGI) module.\n\n- PINNs are used to approximate the viscosity solution of the Hamilton–Jacobi–Bellman (HJB) equation in continuous time, addressing the curse of dimensionality without explicit discretization.\n\n- The VGI module refines value gradients through a recursive, single-step Bellman-like update, which improves stability and accuracy in policy learning.\n\n\nThe paper presents both theoretical analysis (via a contraction-based convergence argument) and empirical validation on continuous-time adaptations of the Multi-Agent Particle Environment (MPE) and Multi-Agent MuJoCo benchmarks.", "questions": "1. Can you provide empirical or analytical evidence that the method provides benefits specifically because of its multi-agent nature (e.g., coordination effects), rather than just being a strong centralized continuous control solver?\n2. Computational Cost: Could you report the relative compute cost (training time, memory, convergence rate) of VIP compared to DPI, IPI, or ODE baselines?\n3. Stochastic Dynamics: Would your framework still hold if small stochastic noise were introduced in the dynamics (i.e., moving toward stochastic differential equations)?\nIf not, how sensitive is the PINN–VGI coupling to such noise?\n4. Value Gradient Iteration Stability: The VGI recursion resembles a single-step gradient rollout. Does it ever introduce instability or oscillation during training? Are there hyperparameters controlling its contraction rate?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Scalability to high-dimensional systems: The integration of PINNs with VGI addresses the curse of dimensionality in continuous-time settings, enabling effective learning in systems with large state spaces. To my knowledge, VGI module is a novel contribution \n- Benchmarking contribution: The authors extend two widely-used discrete MARL benchmarks—Multi-Agent Particle Environment (MPE) and Multi-Agent MuJoCo—to continuous-time settings.", "weaknesses": "I have previously reviewed this paper for another conference, and while the authors addressed some feedback in this version, several substantive issues remain unaddressed, which I list here:\n\n- **Scope is limited to Cooperative MARL & lacks comparison with relevant baselines.** Although framed as a CT-MARL approach, the method applies only to cooperative continuous-time MARL scenarios. It does not handle competitive or mixed settings, where agents have conflicting objectives, which would require solving Hamilton–Jacobi–Isaacs (HJI) equations rather than HJB equations. This limits the generality of the contribution—especially because many real-world multi-agent systems involve competition or partial cooperation. Moreover, in the cooperative case, the learning problem effectively reduces to solving a single centralized control problem with a large joint state space. This raises the question: Is the proposed method genuinely multi-agent in nature, or simply a high-dimensional continuous control solver using a centralized critic? Thus, in my understanding, the high dimensionality of the state could also be an issue in CT-RL. This raises the need for a more thorough comparison with single-agent CT-RL methods.\n\n- **Incremental Theoretical Contribution.** The convergence result (Theorem 3.4) relies on a standard contraction mapping argument under bounded Jacobians and time-invariant dynamics. While mathematically sound, it does not introduce new theoretical insight specific to MARL or continuous-time value iteration. The proof does not incorporate any properties unique to cooperative or decentralized MARL (such as non-stationarity, joint-policy coupling, or inter-agent coordination). Thus, it serves more as a consistency check, rather than a result illuminating how the number of agents or their interactions affect stability or learning dynamics.\n\n- **Motivation and Framing**. The motivation is overall clear—discrete-time MARL algorithms degrade under variable time steps—but the experimental evidence for this claim remains limited. Figure 1 provides a useful visual demonstration, but it’s unclear whether the performance drop of discrete methods (e.g., MADDPG) reflects fundamental limitations of discrete-time updates or simply under-tuned baselines; see next comment. This concern was raised by multiple NeurIPS reviewers and remains only partially addressed.\n\n- **Methodological Comment**. It would strengthen the empirical evaluation if the proposed method were also tested on standard discrete-time benchmarks where the baseline algorithms (e.g., MAPPO, MADDPG) are known to perform well and reproducibly, ideally reproducing reported results from the original papers. This would help disentangle whether the observed advantages stem from the continuous-time formulation itself or from known optimization and reproducibility issues in discrete MARL. Since the proposed approach is theoretically a continuous-time generalization of Bellman updates, it should in principle also perform competitively in discrete settings when the time step is small.\n\n- **Limited Experiments.** In summary, the empirical comparison still omits several key baselines and comparative analyses: no competitive or stochastic control baselines, no detailed compute or resource comparison (training time, memory usage), discrete-time baselines (MADDPG, MAPPO, MATD3) are still missing or not well-tuned for variable time steps.\n\nOverall, the authors should better articulate what is new in the multi-agent context and/or appropriately compare the proposed method.\n\n\n## Minor Comments\n\n\n- In 3.4.2, they still mention the use of terminal-condition losses, while they don’t explicitly exist in equation (17) anymore, which makes sense if the paper now addresses the infinite horizon setup that makes it not ideal to compute such a loss. This should be clarified—either by explicitly removing the term from the text or explaining how those losses were absorbed into the loss formulation.\n- The introduction should clearly state that the method applies only to deterministic multi-agent systems.\n- a brief note distinguishing the smooth vs. viscosity case would help prevent confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "74KoiNTpbV", "reviewer": "Reviewer_jF5y", "summary": "This paper proposes a Continuous-Time Mutli-Agent RL algorithm based on Physics-Informed Neural Networks (PINN). The algorithm, called Value Iteration via PINN (VIP), employs Value Gradient Iteration to estimate the gradient of the value function represented as HJB PDEs. A model of the environment and a reward model are trained alongside policy and value function, and are used for estimating the gradients for value function and expected advantage with respect to the critic and policy parameters respectively. Value gradient iteration, a method for refining value function gradient estimates, is introduced and used for computing an additional target for the value function. The algorithm is evaluated empirically and shown to be competitive with a diverse set of recently introduced methods. Finally, an ablation assesses the effect of individual design choices.", "questions": "- The results on the walker environment show a significantly higher cumulative reward for VIP when compared to the baselines. Moreover, the cumulative reward drops significantly with further training. Can you explain this?\n- For the cooperative navigation task many of the compared algorithms show a very large variance which implies that for some seeds the performance is much better. Can you elaborate on that?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper has a great buildup with excellent motivation including a succinct discussion of related work. Problem formulation and other preliminaries are presented cleary. The central theoretical claims all come with a proof. The value gradient iteration (VGI) method introduced in this paper may be of broad interest for CT-RL in general. Utilizing the differential nature of PINNs in order to train them on more refined targets that include the estimated gradient is a great application of this kind of model. A diverse set of environments is used for evaluation and the experimental results look promising. The effectiveness of VGI is shown empirically on a toy example.\n\nOverall, the paper was very pleasant to read and the contribution to the ICLR community is likely very signficant.", "weaknesses": "No discrete-time RL baseline was included in the experiments. Y-axis scaling in some of the plots make them difficult to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c8chGByUEC", "reviewer": "Reviewer_qN3L", "summary": "This paper proposes a **Continuous-Time Multi-Agent Reinforcement Learning (CT-MARL)** framework, termed **VIP (Value Iteration via PINN)**, which formulates MARL in the continuous-time domain.\nThe authors argue that conventional discrete-time RL (DTRL) methods struggle when applied to systems requiring high-frequency decision-making or irregular time intervals. To address this, VIP replaces discrete Bellman recursion with differentiable value functions that satisfy the **Hamilton–Jacobi–Bellman (HJB)** equations.\n\nKey contributions include:\n\n1. Leveraging **Physics-Informed Neural Networks (PINNs)** to approximate differential value functions governed by HJB PDEs.\n2. Introducing a **Value Gradient Iteration (VGI)** module to refine gradient estimation and stabilize value learning.\n3. Evaluating VIP on continuous-time versions of MPE and multi-agent MuJoCo environments, demonstrating improved sample efficiency and performance over existing continuous-time baselines (e.g., ODE-based RL, HJBPPO, DPI, and IPI).", "questions": "To make the paper stronger and clearer, the following points should be addressed:\n\n1. **Motivation of continuous-time formulation:**\n   Why is the continuous-time formulation superior to a fine-grained discrete approximation?\n   Can you clarify the distinction between CTMC and continuous-action control tasks?\n\n2. **Connection to model-based MARL:**\n   Since VIP involves explicit modeling of ( f_\\psi ) and ( r_\\phi ), how does it differ from model-based methods such as MA-MBRL or MA-MPC?\n   A comparative discussion (or experiment) is needed.\n\n3. **Experimental coverage:**\n   Can you include or at least discuss results against mainstream MARL baselines (e.g., SMPE, Revisiting Off-policy MARL, MAPPO)?\n   This would better contextualize the practical relevance of your approach.\n\n4. **Algorithmic intuition:**\n   Could you provide an intuitive explanation or figure illustrating how the VGI module refines the gradient flow and why it is necessary for training stability?\n\n5. **Scalability and computation:**\n   What is the computational overhead of solving the PINN-based HJB equations in multi-agent systems?\n   Can this method scale to more than 10 agents or to real-time control?\n\n6. **Clarify novelty:**\n   Many parts of the method (PINN + HJB + actor-critic) exist in prior continuous-time single-agent works.\n   Please clearly highlight what is *new* in your multi-agent extension, both conceptually and technically.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* **Timely and ambitious topic:**\n  The paper tackles the underexplored area of continuous-time MARL, aiming to bridge PDE-based control theory and deep RL.\n\n* **Interesting methodological combination:**\n  Integrating PINNs into an actor–critic framework and adding a value-gradient refinement mechanism (VGI) reflects a creative blend of ideas from model-based RL and physics-informed learning.\n\n* **Comprehensive derivations:**\n  Theoretical background and mathematical formulations (e.g., derivations of HJB, VGI, convergence proofs) are detailed and rigorous.\n\n* **Empirical effort:**\n  Experiments on both MPE and multi-agent MuJoCo show that VIP achieves improved training stability and faster convergence than existing continuous-time baselines.\n\n* **Potential significance:**\n  The exploration of continuous-time MARL could inspire future research directions that better align RL with real-world physical systems and continuous control.", "weaknesses": "Despite its conceptual ambition, the paper faces several **foundational and empirical limitations**, which prevent it from reaching ICLR acceptance standards.\n\n### (1) Motivation and conceptual clarity\n\nThe motivation for formulating MARL in continuous time is **not well-justified**.\nWhile the authors mention irregular time intervals as a motivation, the distinction between *continuous-time Markov chains (CTMC)*, *continuous control with continuous action spaces*, and *discrete-time approximations* is **not clearly articulated**.\nIt remains unclear **why a CTMC-based formulation is fundamentally necessary**, and what benefits it provides over well-established discrete-time methods with sufficiently small time steps. The experimental tasks are sufficient using the discrete-time methods enough.\nThis conceptual gap makes the overall significance of the work questionable.\n\n### (2) Insufficient discussion on relation to model-based MARL\n\nThe proposed VIP framework involves explicit learning of **a dynamics model** ( f_\\psi(x, u) ) and a **reward model** ( r_\\phi(x, u) ), followed by value and policy optimization — characteristics more aligned with **model-based RL** rather than model-free methods.\nHowever, the paper **does not compare or discuss** how VIP relates to representative **model-based multi-agent RL** or **model predictive control (MPC)** approaches (e.g., MA-MBRL, MA-MPC, or recent differential game solvers).\nThis omission weakens the positioning of the work.\n\n### (3) Limited baseline comparisons\n\nExperimental baselines are limited to other continuous-time algorithms (e.g., HJBPPO, DPI, IPI).\nHowever, for a fair evaluation in MARL, it is crucial to include **widely recognized baselines** such as **SMPE**[1], **Revisiting Off-policy MARL**[2], and **RACE**[3], which remain standard for multi-agent benchmarks like MPE and MuJoCo.\nWithout such comparisons, it is difficult to assess whether improvements stem from the continuous-time formulation or from differences in model complexity.\n\n[1]SMPE: Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration. ICML 2025\n\n[2]Revisiting Cooperative Off-Policy Multi-Agent Reinforcement Learning. ICML 2025\n\n[3]RACE: Improve Multi-Agent Reinforcement Learning with Representation Asymmetry and Collaborative Evolution. ICML 2023\n\n\n\n### (4) Lack of intuitive explanation and algorithmic clarity\n\nThe paper is mathematically dense but lacks intuitive explanation.\nFor instance, the **motivation and role of VGI** are underexplained in the main text, and there is little visualization or step-by-step discussion of the **training procedure**, **computational cost**, or **failure cases**.\nSimilarly, the algorithm section lacks a high-level intuition connecting the PDE-based reasoning with practical RL optimization.\nReaders unfamiliar with HJB theory or PINNs may find it hard to grasp the conceptual novelty.\n\n### (5) Writing and presentation quality\n\nAlthough mathematically complete, the paper suffers from **dense notation, unclear transitions, and insufficient narrative coherence**.\nSections often read like technical reports rather than structured scientific arguments.\nThe connection between the theoretical motivation and empirical evidence needs stronger integration.\n\nWhile the paper presents an interesting integration of **PINNs** and **continuous-time MARL**, it lacks clear motivation, comprehensive baseline coverage, and intuitive exposition.\nThe conceptual distinction between continuous-time and discrete-time MARL remains ambiguous, and the empirical validation does not convincingly demonstrate broad significance.\nSubstantial revisions — including stronger justification, clearer writing, and broader experimental evaluation — are necessary before this work could be considered for publication at ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zOptAK5usf", "reviewer": "Reviewer_b9mm", "summary": "This paper proposes a continuous-time multi-agent reinforcement learning (CT-MARL) framework called VIP, which leverages PINNs to approximate value functions as solutions to HJB. \nTo address inaccuracies in value gradients that can destabilize policy learning in multi-agent settings, the authors introduce a Value Gradient Iteration (VGI) module that iteratively refines gradients along trajectories, improving value approximation and overall performance.\nOverall, the paper extends continuous-time RL to multi-agent cooperative scenarios, demonstrating good performance over baselines while mitigating the curse of dimensionality.", "questions": "- How sensitive is VGI to bias/variance in dynamics and rewards? Have you tried perturbing these models or training with deliberately misspecified dynamics to see VIP’s robustness envelope?\n\n- How does VIP behave with very small/large variable $\\Delta t$? Any adaptive-step control in the algorithm?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The integration of PINNs with a VGI module is a fresh approach to CT-MARL, addressing key challenges like the curse of dimensionality and gradient inaccuracies in high-dimensional multi-agent systems.\n\n- Provides solid mathematical foundations, including lemmas and theorems with proofs.\n\n- Demonstrates genuine advantage of CT training under varying $\\Delta t$ and nice stress test against a competitive DT agent.", "weaknesses": "- VGI targets rely on accurate dynamics and reward models, if they have a little difference, the estimation may differs a lot and fails.\n\n- VGI convergence relies on bounded Jacobians and small $\\Delta t$; it’s not fully clear how sensitive VIP is when these conditions are loosened\n\n- Discrete-time MARL baselines are excluded from the main comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)