Negotiated Reasoning: On Provably Addressing Relative Over-Generalization
Authors: Junjie Sheng
Yantian Wang
Bo Jin
Hongyuan Zha
Jun Wang
Wenhao Li
Xiangfeng Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Agent Reinforcement Learning
Relative Over-Generalization
Stein variational gradient descent
Primary area: reinforcement learning

Abstract: We focus on the relative over-generalization (RO) issue in fully cooperative multi-agent reinforcement learning (MARL). Existing methods show that endowing agents with reasoning can help mitigate RO empirically, but there is little theoretical insight. We first prove that RO is avoided when agents satisfy a consistent reasoning requirement. We then propose a new negotiated reasoning framework connecting reasoning and RO with theoretical guarantees. Based on it, we develop an algorithm called Stein variational negotiated reasoning (SVNR), which uses Stein variational gradient descent to form a negotiation policy that provably bypasses RO under maximum-entropy policy iteration. SVNR is further parameterized with neural networks for computational efficiency. Experiments demonstrate that SVNR significantly outperforms baselines on RO-challenged tasks, including Multi-Agent Particle World and MaMuJoCo, confirming its advantage in achieving better cooperation.

Published as a conference paper at ICLR 2026
NEGOTIATED REASONING: ON PROVABLY ADDRESS- ING RELATIVE
Junjie Sheng', Yantian Wang', Bo Jin', Hongyuan Zha", Jun Wang',
Wenhao Li Xiangfeng Wang
1. School of Computer Science and Technology, Tongji University
2. School of Data Science, The Chinese University of Hong Kong, Shenzhen
School of Computer Science and Technology. East China Normal University (ECNU)
4. Key Laboratory of Mathematics and Engineering Applications (MoE), ECNU Shenzhen Loop Area Institute (SLAI)
(whlá, biinj, wyt2022 @togggji edu, cn, chahy@cuhk.edu (jarvisestu, jwangecs, x xfwange cs} ecnu. edu.cr en
ABSTRACT
Over-generalization is thomy issue in cognitive science, where people may be- come overly cautious due to past experiences. Agents in multi-agent reinforcement learning (MARL) also have been found suffering relative oxer-generalliation (RO) as people do and stuck to sub-optimal cooperation. Recent methods have shown that assigning reasoning ability agent can mitigate RO algorithmically and empirically, but there has been a lack of theoretical understanding of RO, let alone designing provably RO-free methods. This paper first proves that RO can be avoided when the MARL method satisfies a consistent reasoning requirement under certain conditions. Then we introduce a novel reasoning framework, called negotiated reasoning, that first builds the connection between reasoning and RO with theoretical justifications. After that, we propose an instantiated algorithm, Stein variational negotiated reasoning (SVNR), which uses Stein variational gra- dient descent to derive a negotiation policy that provably avoids RO in MARL under maximum entropy policy iteration. The method further parameterized with neural networks for amortized learning, making computation efficient. Numerical experiments on many RO-challenged environments demonstrate the superiority and efficiency of SVNR compared to state-of-the of-tte-rttttttttt methods in addressing RO.
INTRODUCTION
Multi-agent reinforcement learning (MARL) has been successfully applied in multiplayer games (Rashid et al., 2019; Kurach et al., 2020; Li et al., 2026), robotics (Ding et al., 2020), and traffic control (Calvo & Dusparic, 2018). This paper addresses relative ver-generralzztionn (RO), critical pathology in fully cooperative MARL. settings where agents pursue team-optimal outcomes. RO is analogous to over-generalization in cognitive science (Rand et al., 2014; Laufer et al., 2016), where limited experiences lead broad, often inaccurate generalizations as in the "once bitten, twice shy" idiom, where a person bitten by snake develops fear of rope-like objects. This cognitive phenomenon has been documented across language acquisition (Gershkoff-Sune Stowe al., 2006), social learning (Rand et al., 2014), and decision-maie makinn (Laufer et al., 2016).
In MARL, relative over-generaliig werrgeneralizzatin (RO) poses a significant challenge optimal coopera- tion (Palmer, 2020), as agents overfit their policies to others' exploration behaviors. This is evident in Particle Gather, where particles aiming to reach landmark sy synchronously become rvek-averse after experiencing penalties from uncoordinated visits, causing methods like (Lowe et al., 2017; Wei et al., 2018; Wen et al., 2019) to converge to suboptimal strategies (see (6). Two major approaches address RO: credit assignment methods (evolving from early lenient learning (Panait et al., 2006b; Wei & Luke, 2016; Palmer et al., 2017) to sophisticated value decomposition (Li et al., 2021; Peng et al., 2021; Zhang et al., 2021; Gupta et al., 2021; Siu et al., 2021; Huang et al., 2022; Kang et al., 2022; Yang et al., 2022; Shi et al., 2024; Hu & Ying, 2024) and shaped values (Wan et al., 2022; Shi & Peng, 2022; Zhao al., 2023; Li al., 2024a; Toquebiau al., 2024)), and reasoning-endower

---

Published as a conference paper at ICLR 2026
methods (Wen al., 2019; Ma et al., 2022: Tian et al., 2019; Wei al., 2018) that adopt an ego-agent perspective, equipping agents with capabilities to model others' behavior like recursive reasoning in (Wen et al., 2019) inspired by human cognition (Von Der Osten et al., 2017). Despite empirical successes, both approaches lack solid theoretical foundations. Some works prove algorithm conver- gence (Peng et al., 2021; Li et al., 2024a; Hu & Ying, 2024) optimality in matrix games (Wan et al., 2022), but none formally define RO. This raises two key questions: (1) can RO be provably avoided? and if it can, (2) how to design method that provably addresses RO?
This paper answers the first question with theoretical justifications and introduces new concepts to analyze Relative Over-genett generalization (RO) in Multi-Agent Reinforcement Learning (MARL). The current RO is defined on empirical converged joint policy, which makes it difficult to analyze MARL methods before training. To address this issue, we introduce Perceived Relative Over-generalizatiin (PRO) and Executed Relative Dver-generalization (ERO), which define RO for each joint policy update and policy execution, respectively. The RO is guaranteed to be addressed when ERO is avoided at convergence. With the basis, we prove that RO can be provably avoided when the MARL method satisfies a consistent reasoning condition at convergence. This condition requires each agent to model the behaviors of others consistently with their updated//xxcuted behaviors.
For the second question, we propose a novel negotiated reasoning framework that satisfies the consistent reasoning condition, inspired by human negotiation processes (Kim, 1996; Carnevale & Lawler, 1986) and graphical model message-passing inference (Pearl, 1988). Our framework enables explicit reasoning through negotiation policies during training and decision-making based on negotiated agreements. We prove that agents achieve consistent reasoning when they reach action selection agreements through appropriate negotiation, and introduce Stein Variational Negotiated Reasoning (SVNR), which derives negotiation policies via Stein variational gradient descent and employs strict nested negotiation structure. With maximum entropy policy iteration, SVNR provably achieves consistent reasoning and optimal cooperation at convergence under mild conditions. We further parameterize SVNR with neural networks and implement amortized learning to address computational complexity, distilling negotiation dynamics into network updates and approximating multiple negotiation rounds with single forward passes. Experiments in challenging differential games, particle world and multi-agent MuJoCo environments demonstrate SVNR's superiority in addressing RO compared to state-of-the-eee reasoning methods.
The main contributions are threefold: 1) We confirm the existence of provably addressing relative over- geerrennraalzzionn (RO) methods; 2) We propose a novel framework called negotiated reasoning (NR) and specify the Stein variational NR method, which is the first MARL method that can provably address relative over-generalization (RO); 3) We propose a practical implementation of SVNR that demonstrates superior performance in achieving global optimal cooperation in RO-challenged tasks.
Remark 1. Our work adopts an autoregressive conditional policy factorization. This approach has been used in several multi-agent policy factorization works (Ding et al., 2022; Wang et al., 2023a; Fu et al., 2022; Ye et al., 2022; Li et al., 2024b) and decision-making foundation models (Wen et al., 2022), supporting SVNR's effectiveness. Unlike these works, which address general multi-agent cooperative tasks, we focus specifically on the RO problem. The autoregressive policy factorization (the strictly nested negotiation set in SVNR) just one optimal form. The optimal negotiation set covers broader range of factorizations. Moreover, in contrast to some previous negative results on autoregressive policy factorization-sutt
as the inability to leverage other agents' optimal actions (Ding et al., 2022), sensitivity to the autoregressive order (Li et al., 2024b), and the requirement for centralized execution (Fu et al., 2022)-we we provide a theoretical proof of the optimality of any strictly nested negotiation set. We also achieve decentralized execution through amortized negotiation mechanism.
Remark 2. In addition to autoregressive policy factorization, another class of methods in MARL sequentially updates agents' local, independent policies (Wang al., 2023b; Kuba et al., 2022; Feng et al., 2023; Zhang et al., 2024). These methods are closely related to SVNR, though they mainly address non-stationarity rather than RO. Unlike autoregressive factorization, where agents exchange current policies in the negotiation process, sequential update methods convey the impact of one agent's 's policy update on the environment and subsequent agents.
2

---

Published as a conference paper at ICLR 2026
2 RELATIVE TIEERGENERALIZATION
This section defines RO under CTDE MARL contexts. Specifically, we propose two concepts, perceived RO (PRO) and executed RO (ERO), that distinguish different RO in CTDE. Then, we bridge the two concepts to RO and prove that RO can be avoided when PRO and ERO are addressed under mild conditions. Prior to introducing formal definitions, we first establish the problem formulation and associated mathematical notation.
Cooperative Stochastic Game Cooperative Stochastic Game (CSG) is commonly used model cooperation in multi-agent systems (Petrosjan, 2006). It is defined by tuple (S, ( R, 7), where N is the number of agents; S the state space; Ut represents the action space for agent with 24 = x, 24 representing the joint action space; P(s') 8, u) representing the probability that environment transit to s' when taking joint action u at state 8; R. : S x u + R is the team reward' function; E [0, 1] is the discount factor. The goal for the CSG is to find policies
) that make accumulative reward the highest. The T. : :S - U1 maps the state to agent i's action and the objective of CSG can be formulated maxx El 12 0c 7 (RR un), where u, is sampled from the policies ui (T 8z).
Multi-Agent Reinforcement Learning. MARL methods are popular for solving the cooperative stochastic game. This paper considers the mainstream of MARL schemes: centralized training decentralized execution (CTDE). Each agent é holds an execution policy Fz("" s) to make execution in decentralization manner and a perceived joint policy (u s) to do centralized training. The perceived joint policy can be factorized as मं TiPi, where π; is the individual policy and p; is the perceived opponent policy. Following MaxEnt MARL (Tian et al., 2019; Wen et al., 2019; Wei et al., 2018), each agent i optimizes its policy by minimizing the KL-divergence between perceived joint policy and the induced optimal joint policy: min., DK (#t (T) where a is the factor that balances the reward and entropy. The n. is induced by the Boltzmann optimal policy:
(u s) == exp (a (Qion (s1, u,) - Vooit (s:)))
(I)
where Qin Vosie denote optimal, soft state-action and state value function, respectively (Haarnoja et al., 2017). After that, each agent i obtains decentralized execution policy as ", 8) I f;;du " and the utility of the decentralized execution is: U ;= Be R(s;, uz), where #. == II. मे is the executed joint policy, and Bn is the state-action marginals of the trajectory distribution induced by R.
Relative vver-generalization is a critical game pathology in MARL. It occurs when agents prefer a sub-optimal Nash Equilibrium over an optimal Nash Equilibrium because each agent's s individual policy in the sub-optimal equilibrium has a higher utility when paired with arbitrary policies from opponents (Wei et al., 2018). This definition assumes MARL methods directly select the joint policy from multiple Nash Equilibriums while these methods make a comparison between the current joint policy and updated joint policy for each updating. Thus extend RO by considering each update. Besides that, the current CTDE scheme in MARL motivates us to decompose RO to perceived relative weerggeneralization (PRO) in the training phase and executed relative geerraanerliization (ERO) in the execution phase. First, we define the ERO, which extends RO at each execution step and identifies whether the optimal cooperation is disturbed due to not knowing the behaviors of opponents.
Definition 2.1 (Executed Relative Over- Ever-generalization) Agent i suffers executed relative over- generalization if and only if the utility of executed joint policy can be improved by letting agents know others' actions: max,, (u '1 #7(m"1=)) > Ill, #3 (3((()) where of arg min., (TT.::||TT) is the i's optimal policy with P. and मं. = p:du- is the executed policy for each agent i.
It is straightforward that agents do not suffer from RO if all agents are free from ERO at convergence. Besides that, agents also suffer from RO during their training phase, and we further propose the definition of Perceived Relative Over-generalization.
Definition 2.2 (Perceived Relative Over-geneeralization). Agents suffer perceived relative over- generalization iff. there exists an agent i whose optimal perceived joint policy can be closer to the optimal joint policy when knowing the optimal opponent policy: min,, D >
'The utility, reward and payoff not distinguished.
3

---

Published as a conference paper at ICLR 2026
min,; DKL (TTTT (u (()))))) where Ta is the optimal joint policy with entropy factor a, and Ta (u == Jut T du' is the optimal opponent policy.
The perceived optimal joint policy for each agent is equal to the optimal joint policy for the case that the agents are free from PRO. When each agent i reasons others' behaviors consistent with their optimal policy P = T (u ") in the training phase, others' exploration will not impact the agent's policy updating and the PRO is avoided. If PRO is avoided and -> 0, all agents execute deterministically. the agent's execution will not be impacted by others' exploration stochastic in the execution phase, and ERO is avoided. These conditions are denoted consistent reasoning, and we define them below.
Definition 2.3 (Consistent Reasoning). Agents meet consistent reasoning if and only if all agents reason others' behaviors consistent with their optimal policy P; = T ()" in the training phase and reason others' behaviors consistent with their executed actions during execution.
When the requirement is met at convergence, agents are free from ERO, and they do not suffer from RO. Existing reasoning methods are unable to reach consistent reasoning. We take Figure I as an example to better illustrate how these methods suffer from PRO and ERO respectively. It is a single-stage, cooperative game and contains two agents "A" and "B". The action space of each agent is {0, 1). In Figure (Left), MADDPG (Lowe et al., 2017) usually suffers from PRO due to agents reason others through their historical behaviors. For agent A, if (0) = (1( 0,5, it will obtain # (1, 0) = which is sub-optimal. MASQL (Wei ct al., 2018) usually suffers from ERO in Figure 1 (Right). If H (1, 0) # (0, 1) 0.5 and
B
B
0
3
2
Figure 1: The PRO and ERO payoff functions examples.
(1, 0)- = 4'g (0, 1) = 0.5, then PRO is avoided. However when making decentralized execution based on म' for each agent, (1,, #0, = # #1, # #00 1) 0.25, which are sub-optimal and suffer from ERO.
Intuitive Interpretations of Theoretical Concepts. To connect the formal definitions with practice, we summarize PRO and ERO via variational inference and distributional factorization; a full version provided in Appendix E.8. (1) PRO as Variational Bias. In MaxEnt MARL, agent minimizes DKa. (x,((T;; where p is the perceived opponent policy. If pt deviates from the optimal conditional T";. the objective becomes biased, so optimization steers toward a safe local optimum rather than the cooperative global one. Thus, PRO is a training-time estimation error caused by imperfect beliefs about others. This mirrors misspecified variational families: the learner optimizes a KL against a target distribution filtered through wrong conditional, so the fixed point is self-consistent with Ps but not globally optimal. (2) ERO as Factorization Loss. Even if training converges to an optimal joint policy न. decentralized execution requires a product form मे (u) II, T. (u4). When is correlated or multimodal, projecting it onto independent marginals incurs factorization loss, yielding miscoordination and lower return. Geometrically, is the closest point in the independence manifold, so correlation structure in ी is flattened, removing the signals needed to pick common mode. (3) Consistent Reasoning as Closing the Loop. Consistent reasoning enforces both training consistency (p; + = and execution consistency (negotiation collapses to single agreed mode so (w) = II. (u,)) as a + 0), aligning planned and executed actions. One distribution is optimized, shared, and executed, turning centralized optimality into decentralized realizability.
NEGOTIATED REASONING FRAMEWORK
Inspired by the critical role of negotiation for consistent reasoning social cooperation, we introduce negotiation in the reasoning process to avoid PRO and ERO with theoretical justifications and propose a novel reasoning framework, NR. In NR, agents take M particles {u A to represent the initial perceived joint policy distribution p(u²)= == 1 M Suro for a state s. Moreover, each agent i holds a negotiation (i.e., perturb) policy fi(u; uc,, 8) that updates its action when knowing the C,'s s action selection. Here C₁ C I....,N is the negotiated set for agent i, which determines whom to negotiate, ft := [J...... f(K) where f is the negotiation policy of agent i in iteration k and K is the number of negotiation rounds which is often large enough. Then every k- 1-1)) & N. < M, < K. Such agent i makes negotiated reasoning as u. = (u: 8, u.c.
M
P.A
4

---

Published as a conference paper at ICLR 2026
negotiation process can be interpreted agents starting from initial action beliefs and negotiating with each other based on their negotiation policies. When f.r converges to an identity map for each agent, the perceived joint policy converges to steady perceived joint policy (i.e., agreement): limg-> p(u* 8) := के Σx=1 Sue (u) x (u 8), Yu E U. Negotiated reasoning avoids PRO when it meets certain conditions.
is crucial to distinguish this framework from communication based MARL methods that exchange messages to resolve partial observability (i.e., approximating global state). In contrast, Negotiated Reasoning operates on the probability measure space. The "negotiation" is a functional gradient descent process in Reproducing Kernel Hilbert Space (RKHS) that aligns the joint policy distribution with the global value landscape. This addresses equilibrium selection rather than state estimation.
Theorem 3.1 (PRO-free Negotiated Reasoning). For any environment state s where the optimal joint policy is defined as Ta consider each agent i takes negotiated reasoning defined on compact action space U they are PRO-free with K steps negotiated reasoning if lims-+ p(u |s) = Yu Eu.
x' (u"
|s),
This motivates to learn negotiation policy fi satisfying the following conditions:
(.k-1) lim fl s, uc un kー R-
s) = x (" s). Vi u, E u, f.k-1 lim p(u*
The first condition requires the negotiation policies converge the identity map, and the second one requires the perceived joint policy to be identical to the optimal joint policy when the negotiation
policy converges. We will specify the negotiated policy learning in the following two sections.
As for ERO-free in decentralized execution, we prove that setting म; = u. D.K with annealing a + ensures ERO-free in decentralized execution (see proof in Appendix E.2).
Theorem 3.2 (ERO-free Negotiated Reasoning). For any environment state when agents are PRO-free with K reasoning steps, they achieve ERO-free free with annealing a + 0 if each agent i sample action म; = =
0.K
When all the conditions are met, it is straightforward that consistent reasoning obtained. Up to this point, we have established theoretical connection between reasoning and RO. The next step is to design a negotiation policy that satisfies the condition in equation 2 and integrate this negotiated reasoning into existing multi-agent reinforcement learning.
4 STEIN VARIATIONAL NEGOTIATED REASONING
After building the theoretical relationship between reasoning and RO, this section proposes Stein variational NR, SVNR, under the NR framework, which is the first MARL method that provably addresses RO. We first derive the negotiation policy based on Stein variational gradient descent which obtains PRO-free negotiated reasoning. Then we devise the policy iteration method of SVNR and prove that it addresses PRO and ERO. Finally, we propose a practical implementation by parameterizing SVNR with neural networks and amortizing the learning procedure.
4.1 LEARNING THE NEGOTIATION POLICY
To learn the negotiation policy that converges to an identity map and lets perceived joint policy converges to the optimal joint policy as in equation 2, we start by building the relationship between negotiation policy and perceived joint policy. Decomposing KL divergence from the perceived joint policy, we have DxL. (p(u ()[[T((( (u s)) = D (p(u () + D (p(u:
8, u- ()))))))))l 8, u- ()).))(.::)). It states that the KI. divergence between perceived and optimal joint policy can be minimized by
min p(u,/a,00))
DKL (p(u,) s, u- u.((p(u (ux =-))p(u
when fixing other agents' action selections (update only one agent's 's action). This motivates us to design negotiation policy that minimizes the equation 3. One of the most popular ways solve
5

---

Published as a conference paper at ICLR 2026
equation 3 is (MP)SVGD (see Appendix A) which can naturally fit the updating of the single agent's action while fixing others'. Specifically, it adopts the following scheme, i.e.,
fi(u:] u.., 8) : uf + edi (uc,)', Vi S N,
to update the joint policy distribution. The is the learning rate, and Φ; is the transformation direction in vector-valued reproducing kernel Hilbert space. Then the optimal & has a closed form solution for equation 3 when restricting @illx. I 1 and e + O:
of (uc,) = Ey=plk; (uc, vc.) V.. log (y: (c.))).)))))))))) + V A, SCA6))..
(5)
The o² provides the steepest direction optimize the KL divergence. This iterative update process is mathematically grounded in the transport of probability measures via Stein variational gradient flow. We provide a detailed theoretical interpretation of this negotiation process and its visualization in Appendix L. The Appendix D. shows the details of the derivation.
To further ensure the identity map convergence and let the converged perceived joint policy identical to the optimal joint policy, the design of {C,) plays key role as seen in graphical inference problems (Pearl, 1988; Zhuo al., 2018). Benefiting from the centralized training, we can de- sign C; without considering communication limitations. When {C:}-= is strictly nested (e.g., fl 2) for all i), negotiated reasoning equation with equation converges and the agree- ment is identical to optimal joint policy (i.e.. satisfies PRO-free conditions equation 2) as proved in Appendix E.4. We denote the negotiated reasoning with (MP)SVGD and strict nested negotiation set as Stein variational negotiated reasoning. While strict nesting guarantees exact representability, relax- ing this constraint leads to a bounded approximation error characterized by Information Projection, as detailed in Appendix E.9.
4.2 MAXIMUM ENTROPY POLICY ITERATION
In the previous section, we assumed that the optimal joint policy is known in advance. However, agents have to iteratively learn Q, and V functions estimate the optimal joint policy and update their sampling policy accordingly in practice. This section establishes SVNR on the maximum entropy policy iteration and shows the convergence to the optimal joint policy theoretically. Concretely, we first define the soft bellman operator as
T= u.) : -BB.. [V (s
(6)
where (e:) = Es [Q(s,, u.) - a log ((; s.)] Each round of iteration usually consists of joint policy evaluation and joint policy improvement, where joint policy evaluation aims to evaluate the policy performance with Q and joint policy improvement updates each agent's 's policy accordingly. As for the joint policy evaluation, obtain the following theorem.
Lemma 4.1 (Joint Policy Evaluation). For a mapping Q" S x u + IR with U < co, define the Q++1 = Q* where the P the soft bellman operator, then converges to the joint soft Q-function of Ā as k + co.
Following equation 1 and equation 5, the मे is updated as:
1 M M
+(u) = lim K-K
Σ
Seen (u), u.t. = - 1 + E00 (uc, u, 1, vi N
t=1
(7)
1 # exp (Q,u,, 8), V(uc,,))
where Q(u;, uc, 8) ( ac; =uc uu Q(ū s), Vi(uc,, Fagr ac, @ū 8), and of take स instead of xt" construct the SVGD direction. Then can obtain the following joint policy improvement lemma:
Lemma 4.2 (Policy Improvement). When the negotiation policies are strictly nested, given the current perceived joint policy as ît update it based on the equation 7 and obtain the new perceived joint policy #'. The Q*' (St, uz) 2 Q* (st, u:) with [ co.
Following Lemma 4.1 and Lemma 4.2, we can establish the following SVNR policy iteration theorem and our proposed coordinated policy iteration method accordingly.

---

Published as a conference paper at ICLR 2026
Theorem 4.3 (SVNR Policy Iteration). When the individual policies satisfy the strict nested require- ment, considering repeated apply the joint policy evaluation and joint policy improvement on the perceived joint policy it, then will converge to that makes Q (8t, u.) 2 Q* Ut): Vfr II, (st, u.) E U, [u] 00.
While the analysis assumes discrete action spaces utilize standard fixed-point theorems, the theoretical results extend to continuous domains through measure-theorethe unification. Furthermore, the core Negotiated Reasoning mechanism (via SVGD) natively designed for continuous spaces. We provide the detailed continuous formulation and justification in Appendix K. Based on the Theorem 4.3, we can obtain the convergence of SVNR policy iteration the optimal joint policy. Further, taking Theorem 3.2, we can obtain ERO-free executed joint policy म by annealing to small enough number.
However, empirically, the SVNR policy iteration assumes knowing the word model and encounters high computation and storage complexity due to 1) inefficient policy representation: SVNR policy iteration represents the joint policy with particles that scale poorly state-action space; 2) intractable optimization: During learning, the soft bellman operator takes expectations on both the state and joint policy distribution, which is intractable in realistic settings. To this end, propose practical implementation for SVNR.
A PRACTICAL IMPLEMENTATION OF SVNR
(-lts, My, us)
(-1us, No,
sta ma(-/au, us)
xx( (1,,
MPSVSD
5 (1
CPF Training
(1)
(1
pd
(5()
(16))
Dec Testing
Figure 2: The practical SVNR in the 3-agents system. SVNR adopts nested negotiated reasoning and adopts amortized MPSVGD output the actions. The amortized MPSVGD distills the multi rounds negotiated reasoning dynamic by well-established neural networks. The "Dec Testing" (rightmost part) illustrates how the proposed SVNR executes in decentralized manner.
To address the inefficient policy representation and intractable optimization issues, this section adopts neural networks to parameterize the policies and learn them with the proposed amortized optimization. To gain efficient action sampling, we propose Amortized MPSVGD. It aims to adopt neural networks to perform variational distillation of the negotiation equilibrium. Rather than cloning the iterative negotiation trajectory, the network aims to approximate the steady negotiation result (the fixed point) in (u) directly via neural network inference. Formally, each agent holds stochastic mapping function w² = =fa ( that maps initial noises (i.e., gaussian noises) to its action distribution. The ç is the noise drawn by agent i. We denote the induced joint distribution p& (uls, E) : I fi. (uils' s). The goal of the proposed amortized MPSVGD method is to find 2h* that satisfies:
arg min KL (p& (-1 (((.)))(((1))) (
(8)
A straightforward way to learn is to iterate the equation 7 procedure until convergence and to establish the neural networks [V1.... Φx) which can fit the agreement. However, the equation requires many rounds of updating, and this motivates us to introduce an incremental update scheme. For each agent i, its policy parameter 1h is updated by moving along its SVGD's gradient in order to approach the target joint policy. Sampling joint actions u.. ....; u." from p and assuming we
7

---

Published as a conference paper at ICLR 2026
perturb agent i's action un f Sc, s) in appropriate direction A f (E/: fc, s), the induced KL divergence in equation can further be reduced. MPSVGD provides the most greedy direction as
AS" (:; (; = Euuop [u((uc (: (x)))) Vu: Vu:Q" (81, u')l + a V uf (u,,, (: 8t))
where a; is the agent i's temperature term, O is the neural network paramter of central critic, and K; is the agent i's kernel function in MPSVGD. We then set (J,(0.).) Af (Feng al., 2017).
du.
Further, the gradient in MPSVGD can be backpropagated the mapping network Φs i.c.
05. (E:
Eg (E:
(10)
Onb.
Therefore, any gradient-based methods can optimize the parameters 1b; The detailed derivations of equation 9 and equation 10 are shown in Appendix D. With this Amortized MPSVGD mapping function, neural network inference can directly sample joint actions. Crucially, by optimizing via this incremental scheme, the network f learns to distill the multi-step negotiation dynamics into the function weights. Consequently, single forward pass (K = 1) becomes sufficient to approximate the equilibrium distribution during inference, avoiding expensive inner-loop optimization.
Furthermore, we consider the intractable evaluation step as in equation 6. Inspired by soft Q- learning (Haarnoja et al., 2017), we can transform the fixed point iteration to the stochastic optimiza- tion on minimizing the ll QlI- Specifically, the importance sampling is adopted approximate the value function and minimize the bellman error:
Onew arg ming: Eagyuss 1ND
1-D 2 (8(+1) Q8 Q8'((,,,))) (St, u))²
(11)
where v (st) == a log E Eu' pp-le( [exp (1Q2 (s,, w'))]. We summarize the proposed SVNR in Figure with pseudocode in Appendix B. While the practical implementation introduces approxima- tion errors compared to the exact soft Bellman operator used in our theoretical analysis, we provide a formal error analysis in Appendix 10, showing that the performance loss bounded.
SVNR assumes nested negotiation during training. which aligns with the widely adopted CTDE paradigm. This assumption enables agents leverage global information for improved coordination while training, yet critically, SVNR operates in a fully decentralized, communication-fres manner during execution. Other assumptions in our analysis (e.g., stationarity, bounded rewards) are standard in MARL literature and necessary for theoretical rigor without imposing impractical constraints.
EXPERIMENTS
We take two differential games (Two Modalities and Max of Three (Panait al., 2006a)) and the Particle Gather (Mordatch & Abbeel, 2018)) as our initial testbeds. We then scale to com- plex continuous-control domains in MaMuJoCo (Peng et al., 2021). Baselines include popular MARL methods and reasoning-basee approaches that target RO RO-MADDPG (Lowe et al., 2017), MASQL (Wei et al., 2018). PR2 (Wen et al.. 2019), ROMMEO (Tian et al., 2019), and MMQ (Zhu et al., 2024)-as well as strong mallee-dcomposittoo/ccttooo--eee
general baselines in MaMuJoCo,
i.e., MAPPO (Yu et al., 2022), QMIX (Rashid et al., 2020), and FACMAC (Peng et al., 2021). To ensure rigorous evaluation, we employ identical network backbones and fixed entropy annealing schedules across all Maximum Entropy methods, isolating the performance gains attributed to the reasoning mechanism. A detailed analysis of computational trade-offs, theoretical justification for compute costs, and hyperparameter protocols is provided in Appendix H.5.
Note that our primary contribution is the theoretical development of an RO-free solution for MARL. We validate these claims using benchmarks that provide sufficient complexity, including MaMuloCo, while maintaining tractability. For two differential games and the Particle Gather, report aggregate test performance in Table and defer its quantitative analysis to Appendix G. General-purpose baselines are also reported in Appendix G.
(1) The Differential Game (DG). DG is a flexible and wide-adopted framework to design a chal- lenging stateless MARL environment. We consider a three-agents case. Each agent shares
8

---

Published as a conference paper at ICLR 2026
common dieensinnninn bounded continuous action space of [-10, 10]. Their rewards are shared
and determined by their joint action under the reward function r(11, U2. Ug) = max(g] 92). where
a1, 02, 03 are actions of agents respectively, 91 = 0.8 [- 21 #1 ("2+5) and 92 = h2 x
Ma -aام C1
+2-
(1.1) PRO-C PRO-Challenged DG. Setting c = C2 results in two-modality. which raises the difficulty for agents obtain the optimal perceived joint policy and thus is PRO-challenged environment. We set 7, 22 -3, C1 C2 10 in the differential game to construct the Two Modalities scenario as the PRO-Challenged scenario. There exists two points (-5, -5,3) and (7, 7, -3) that have the highest, 10, utility. Thus the optimal perceived joint policy should capture the two modalities. However, when agents do not know the optimal opponent policy. they usually tend to converge to one single modality, and PRO happens. We train each method with 5000 episodes and visualize their converged perceived joint policies by sampling. As shown Figure 3, our SVNR captures the two modalities of the game while other baselines converge to the single modality policy.
h2 = 1.0, $2 2, I2 7,
(a) SVNR (Ours) (h) MADDPG
(c) MASQL
(d) PR2
(c) ROMMEO
(f) MMQ
Figure 3: The converged perceived joint policy visualization Two Modalities scenario. The optimal perceived joint policy should capture both modalities, and only SVNR captures the modalities.
(1.2) ERO- ERO-Challenged DG. We consider a difficult scenario for continuous MARL, Max of Three, which extended from the Max of Two (Tian et al., 2019; Wei al., 2018; Wen et al., 2019). Specifically, we set the 122 1,222 values for 82, we can flexibly control how the ERO affects the agents. The smaller the 82, the smaller the coverage of 92. and the more severe the ERO issue. We examine different methods under different 82, i.e., 82 1.5, 32 2.0 and 82 = 3.0 and 5000 episodes are used for all cases.
1/2 7, 22 -4, = 0, C2 = 10. By setting different
Skps
Sups
Episudes
82
3.0
(b) 82 2.0
(c) 82 1.5
(d) Particle Gather
Figure 4: Influence of different coverage factors on the training curves of (a-c) our method and different baselines the Max Three. (d) shows the training curves the Particle Gather scenario. The solid lines and shadow areas denote the mean and variance of the instantaneous rewards with 5 different seeds. With the larger $2, the agents encounter a higher impact relative over-generalization and the proposed SVNR achieves the optimal solution in all settings.
(a) SVNR (Ours) (b) MADDPG
(c) MASQL
(d) PR2
(e) ROMMEO
(1) MMQ
Figure 5: The sampled joint actions of (a) SVNR and (b-d) representative baselines under the settings of Figure (4c) from 3000 training timesteps. Each point represents a joint action taken by the agents at the corresponding timestep. and different colors represent the levels of rewards.
Although the formulation the game is relatively simple, it poses great difficulty to gradient- based algorithms in almost all the joint action space. The gradient points to sub-optimal solution.

---

Published as a conference paper at ICLR 2026
As shown in Figure 4, the MADDPG algorithm falls into the local optimum (i.e., the reward is 0) under all settings. MASQL, PR2, and ROMMEO can only jump out of the local optimum under the relatively simple setting (i.e., $2 3.0) with significant variance, while SVNR can steadily converge to the global optimum while jumping out of the local optimum under all settings.
To better understand the learning behavior in the MAX OF THREE, we visualize the learning dynamic under $2 = 1.5 in Figure 5. Each point represents a joint action taken by the agents from 1 to 3000 steps. Different colors represent the levels of instantaneous rewards. During
1500 steps. SVNR agents have a significant visitation probability on the local optima (the left side at Figure Sa). They visit the global optima more frequently at 1500 to 3000 steps while exploring the other area. With the learning process kept on, SVNR converges to the 10 step reward as shown in Figure 4c. Other baselines are concentrated near the local optimum.
(2) Particle Gather.
This game is built with Multi-Agent Particle World (Lowe et al., 2017). There are 2 particles in a continuous physical world. Each particle controlled by agents, the z-agent and the y-agent, which control the particle's movement together. When 2 particles reach fixed landmark, 4 agents are rewarded with 5 together. Moreover, if only one particle reaches the landmark, all the agents are penalized by 22 Otherwise, there is no instantaneous reward (i.e., 4 agents are rewarded by 0) that will be feedback to all agents. This iterated continuous game lasts for 25 timesteps. The goal of all agents is to maximize the individual expected cumulative reward for 25 timesteps. This scenario is difficult because without knowing others' actions, the best choice for all the agents will be to get far away from the landmark, making the optimal policy (reach the landmark simultaneously) hard to obtain. All methods are trained for 5000 episodes, which consists of 25 timesteps, with tuned hyperparameters, and the learning curves are shown in Figure 4d. It shows that all baselines converge to the worst solution except for PR2 and MADDPG falling into the local optimum. SVNR still steadily converges to the global optimum while jumping out of the local optimum.
(3) Multi-Agent MuJoCo (MaMuJoCo). We further evaluate SVNR on 4 MaMuJoCo environments which convert classic single-agent MuJoCo tasks into fully cooperative, multi-agent settings via physically meaningful partitions of the action space. In all 4 environments, agents receive the same shared reward as the underlying single-agent task, and episodes terminate/truncate simultaneously for all agents under the same conditions as the single-agent versions. Table I summarizes test returns across the four MaMuJoCo tasks. SVNR achieves the highest returns in all scenarios, with especially large margins over MAPPO/QMIXIEACMAC and consistent improvements over PR2/ROMMEO. These results, combined with those on differential games and Particle Gather, indicate that negotiated reasoning yields robust coordination benefits from diw-dimeniional, RO-dominant settings to high- dimensional continuous control with physically meaningful agent partitions.
Table 1: MaMuJoCo test performance. SVNR achieves the highest returns across all four tasks.
Malhods / Secmarios
HallCheelal--
TachChectb--1p Ip
Ami-2x4
Walker2F-233
SVNR (Ours)
6853 & 212
423 1 89
536 31
1678 1 275
112 125
105
620 #
MASQL
56 + 65
-90 + 86
225 + 34
332 + 18
PR2
5002 1
381 1 11
3511.8 58
1922 1 70
ROMMEO
5008 127
236
1300
MMQ
184 + 16
524 +
116 + 53
487 + 72
MAPPO
D0BT 1177
15:135 133
57 135
672 1 59
QMLX
5260 018
213 4
FACMAC
8210 1 584
131 72
398 1 36
536 1. 205
(4) Ablation Studies. Full protocols and tables are deferred to Appendix H. Varying the SVGD particle count M on MaMuJoCo (from 16 to 64) shows a broad performance plateau, with a practical sweet spot at M E {32,440 Training time scales approximately linearly in M. Scaling the number of agents from 2 (MaMuJoCo) to 3 (Max of Three) and 4 (Particle Gather) preserves near-constant normalized performance with only modest increases in wall-clock cost, indicating that amortized negotiation maintains coordination quality as team size grows. Finally, on Particle Gather, strict nested negotiation yields the best returns, but partially nested DAGS recover most of the performance at lower cost. Aggressively sparse peer sampling (1-2 peers per agent) remains viable when compute is tight, with performance degradation consistent with the theoretical approximation gap analyzed in Appendix E.9. Together, these results suggest SVNR offers a favorable scales to mall-medium teams, and is robust to reasonable deviations from strict negotiation topology. We further provide a comprehensive theoretical analysis and empirical ablation study on the sensitivity of the temperature parameter and its annealing schedule in Appendix H.4.
-ufuracy-ffficincyyy trade-off,
10

---

Published as a conference paper at ICLR 2026
Acknowledgements Junjie Sheng is an independent researcher interning Tongji University. Wenhao Li acknowledges support from the National Natural Science Foundation of China (NSFC) under Grant 62406270 and the STCSM Shanghai Rising-Stoe Star Program under Grant 24YF2748800. Hongyuan Zha acknowledges support from the Shenzhen Stability Science Program 2023 and the NSFC under Grant 72495131. Dr. Jun Wang was supported, in part, by grant from Ant Research. Xiangfeng Wang acknowledges support from the NSFC under Grant 62231019 and SHEITC under Grant D02-GZZ-RGZNN-BBX--ON
Ethics Statement. Our work on negotiated reasoning for addressing relative over- werr-eneraaiiztion in multi-agent reinforcement learning has several potential socictal impacts. On the positive side, by developing methods that provably address relative over-generalization, we contribute the reliability and effectiveness of cooperative multi-agent systems. This advancement could benefit applications such coordinated robotics for search and rescue operations, traffic management systems where improved cooperation could reduce congestion, and resource allocation in distributed systems like power grids and supply chains. However, while our work focuses on cooperative settings, techniques that improve multi- coordination could potentially be adapted for adversarial purposes. such as coordinated automated attacks in cybersecurity contexts or applications in competitive rather than cooperative scenarios. To promote responsible use, we recommend continuing research cooper- ative MARL benchmarks that address socially beneficial problems, establishing ethical guidelines for deployment, and developing interpretability methods that can help understand the negotiation processes. Our primary focus on theoretical understanding limits immediate risks, but ongoing ethical discussion about increasingly capable multi-agent systems remains essential as the field advances.
Reproducibility Statement We are committed to enabling the reproducibility of our results to the best of our ability. In the paper, we provide detailed descriptions of the experimental setup, including implementation details, hyperparameters, and prompt designs, as well as data generation steps in Section 6, Appendix F, G and H. Our approach builds upon several open-source projects, and we have included links to the relevant code repositories for transparency and ease of reference. We document key elements necessary for reproducing our findings, such as training procedures, evaluation metrics, and the use of multiple random seeds. While we have taken significant steps to ensure that the methodology is clear and replicable, variations in software environments, hardware configurations, other external factors may affect exact reproducibility. Nonetheless, believe the provided information should allow others to replicate our findings or apply similar approaches with reasonable accuracy,
REFERENCES
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-95, 2018
Robert J Aumann. Subjectivity and correlation in randomized strategies. Journal of mathematical Economics, 1(1):67-96, 1974.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper--paameee optimization. In NeurlPS, 2011.
Michael Bowling. Convergence and no-regret in multi-agent learning. In NeurlPS, 2004.
George W. Brown. Iterative solution of games by fictitious play. Activity Analysis of Production and Allocation, 13(1):374, 1951.
Jeancarlo Arguello Calvo and Ivana Dusparic. Heterogeneous multi-agent deep reinforcement learning for traffic lights control. In AICS, 2018
Peter JD Carnevale and Edward Lawler. Time pressure and the development of integrative agree- ments in bilateral negotiations. Journal of Conflict Resolution, 30(4):636-659) 1986.
Guohui Ding, Joewie J Koh, Kelly Merckaert, Bram Vanderborght, Marco M Nicotra, Christoffer Heckman, Alessandro Roncone, and Lijun Chen. Distributed reinforcement learning for cooperative multi-robot object manipulation. In AAMAS, 2020.

---

Published as a conference paper at ICLR 2026
Ziluo Ding, Kefan Su, Weixin Hong, Liwen Zhu, Tiejun Huang, and Zongqing Lu. Multi-agent sequential decision-making via communication, arXiv preprint arXiv:2209.12713. 2022.
Lang Feng, Dong Xing, Junru Zhang, and Gang Pan. Fp3o: Enabling proximal policy optimization in multi-agent cooperation with parameter-sharing versatility, arXiv preprint arXiv:2310.05053, 2023.
Yihao Feng, Dilin Wang, and Qiang Liu. Learning to draw samples with amortized stein variational gradient descent. In UAI, 2017.
Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning, In ICML, 2022.
Andrew Gelman and T. P. Speed. Characterizing joint probability distribution by conditionals.
Journal of the Royal Statistical Society. Series B (Methodological), 55(1):185-188, 1993.
Lisa Sershkoff-Stower Brenda Connell, and Linda Smith. Priming overgeneraliiations in two-and four-year-old children. Journal of Child Language, 33(3)4611-86, 2006.
Amy Greenwald and Keith Hall. Correlated-g learning. In ICML,,000
Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin Böhmer, and Shimon Whiteson. Uneven: Universal value exploration for multi-agent reinforcement learning. In ICML, 2021.
Tuomas Haarmoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In ICML, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.
Junling Hu and Michael P Wellman. Nash q-learning for general-sum stochastic games. Journal of machine learning research, 4(Nov):1039-1069, 2003.
Rui Hu and Lei Ying. Multi-agent optimistic soft lleariin: A A. with global convergence guarantee, 2024. URL TV.
net / forum? id=de33G5
Wenhan Huang, Kai Li, Kun Shao, Tianze Zhou, Matthew Taylor, Jun Luo, Dongge Wang, Hangyu Mao, Jianye Hao, Jun Wang, et al. Multiagent q-learning with sub-team coordination. In NeurIPS, 2022.
Yipeng Kang, Tonghan Wang. Qianlan Yang, Xiaoran Wu, and Chongjie Zhang. Non-linear coordi- nation graphs. In NeurIPS, 2022.
Jeong-Yor Yoo Kim. Cheap talk and reputation in repeated pretrial negotiation. The RAND Journal of Economics, pp. 787-802, 1996.
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation multi-agent reinforcement learning. In ICIRR, 2022.
Karol Kurach, Anton Raichuk, Piotr Stanczyk. Michal Zajkac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, al. Google research football: A novel reinforcement learning environment. In AAAI, 2020.
Offir Laufer, David Israeli, and Rony Paz. Behavioral and neural mechanisms of overgeneralization in anxiety. Current Biology, 7636)22 2016.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. arXiv preprini arXiv: 0088000000099 2018
Chao Li, Yupeng Zhang, Jianqi Wang, Yujing Hu, Shaokang Dong, Wenbin Li, Tangjie Lv, Changjie Fan, and Yang Gao. Optimistic value instructors for cooperative multi-agent reinforcement learning. In AAAI, 2024a.
12

---

Published as a conference paper at ICLR 2026
Sheng Li, Jayesh K Gupta, Peter Morales, Ross Allen, and Mykel J Kochenderfer. Deep implicit coordination graphs for multi-agent reinforcement learning. In AAMAS, 2021.
Wenhao Li, Xiangfeng Wang, Bo Jin, Jingyi Lu, and Hongyuan Zha. Learning roles with emergent social value orientations. IEEE Transactions on Puttern Analysis and Machine Intelligence, 48(2): 1657-1673, 2026.
Zhiyuan Li, Wenshuai Zhao, Lijun Wu, and Joni Pajarinen. Backpropagation through agents. In AAAI, 2024b.
Michael Littman. Friend-or-Foe q-leaming in general-sum games. In /CML, 2001.
Michael L. Littman. Markov games as framework for multi-agent reinforcement learning. In ICML, 1994.
Qiang Liu. Stein variational gradient descent as gradient flow. In NeurlPS, 2017.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. NeurIPS, 2016.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compeitty environments. In NeurlPS, 2017.
Xiaobai Ma, David Isele. Jayesh K Gupta, Kikuo Fujimura, and Mykel J Kochenderfer. Recursive reasoning graph for multi-agent reinforcement learning. In AAAI, 2022.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In AAAI, 2018.
Gregory Palmer. Independent learning approaches: Overcoming multi-agent learning pathologies in leam-games. The University of Liverpool (United Kingdom), 2020.
Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. Lenient multi-agent deep reinforcement learning. arXiv preprint arXiv: 04402, 2017.
Liviu Panait, Sean Luke, and R Paul Wiegand. Biasing coevolutionary search for optimal multiagent behaviors. IEEE Transactions Evolutionary Computation, 10(6):629-644 2006a.
Liviu Panait, Keith Sullivan, and Sean Luke. Lenient learners cooperative multiagent systems. In AAMAS, 2006b.
Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan kaufmann, 1988.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexanrre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. In NeurIPS, 2021.
Leon A Petrosjan. Cooperative stochastic games. Advances in Dynamic Games: Applications to Economics, Management Science, Engineering, and Environmental Management, pp. 139-145, 2006.
David G Rand, Alexander Peysakhovich, Gordon T Kraft-Todd, George E Newman, Owen Wurzbacher, Martin A Nowak, and Joshua D Greene. Social heuristics shape intuitive coop- eration. Nature Communications, 5(1):1-12, 2014.
Tabish Rashid, Philip Torr, Gregory Farquhar, Chia-Man Hung. Tim Rudner, Nantas Nardelli, Shimon Whiteson, Christian Schroeder de Witt, Jakob Foerster, and Mikayel Samvelyan. The starcraft multi-agent challenge. In AAMAS, 2019.
Tabish Rashid, Mikayel Samvelyan. Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, (178):11-51, 2020.
13

---

Published as a conference paper at ICLR 2026
Lin Shi and Bei Peng. Curriculum learning for relative overgeneralization. arXiv preprint arXiv:2212.02733. 2022.
Yuchen Shi, Shihong Duan, Cheng Xu, Ran Wang. Fangwen Ye, and Chau Yuen. Dynamic deep factor graph for multi-agen reinforcement learning. arXiv preprint arXiv:2405.05542, 2024.
Chapman Siu, Jason Traish, and Richard Yi Da Xu. Dynamic coordination graph for cooperative multi-agent reinforcement learning. In ACML, 2021.
Zheng Tian, Ying Wen, Zhichen Gong, Faiz Punakkath, Shihao Zou, and Jun Wang. A regularized opponent model with maximum entropy objective. In IJCAI, 2019.
Maxime Toquebiau, Nicolas Bredeche, Faiz Benamar, and Jae-Yun Jun. Joint intrinsic motiva- tion for coordinated exploration in multi-agent deep reinforcement learning. arXiv preprint arXiv:2402..3397.. 2024.
Friedrich Burkhard Von Der Osten, Michael Kirley, and Tim Miller. The minds of many: Opponent modeling in stochastic game. In IJCAI, 2017.
Lipeng Wan, Zeyang Liu, Xingyu Chen, Han Wang, and Xuguang Lan. Greedy-based value representation for optimal coordination in multi-agent reinforcement learning. ICML, 2022.
Jiangxing Wang, Deheng Ye, and Zongqing Lu. More centralized training, still decentralized execution: Multi-agent conditional policy factorization. In ICLR, 2023a.
Xihuai Wang, Zheng Tian, Ziyu Wan, Ying Wen, Jun Wang, and Weinan Zhang. Order matters:
Aggent-b-aagen policy optimization. In ICLR, 2023b.
Ermo Wei and Sean Luke. Lenient learning in independent-learner stochastic cooperative games. The Journal of Machine Learning Research, 17(1):2914-2955, 2016.
Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft Q-leariing In AAAI Spring Symposium Series, 2018.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is sequence modeling problem. In NeurlPS, 2022.
Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning, In ICLR, 2019.
Qianlan Yang, Weijun Dong, Zhizhou Ren, Jianhao Wang. Tonghan Wang, and Chongjie Zhang. Self-organized polynomial coordination graphs. ICML, 2022.
Jianing Ye, Chenghao Li, Jianhao Wang, and Chongjie Zhang. Towards global optimality in coopera- tive marl with the transformation and distillation framework. arXiv preprint arXiv:2207.11 11143, 2022.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang. Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. In NeurlPS, 2022.
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In ICML, 2021.
Wenjing Zhang, Wei Zhang, Wenqing Hu, and Yifan Wang. B2mapo: A batch-by-batch multi-agent policy optimization balance performance and efficiency. arXiv preprint arXiv:2407.15077, 2024.
Wenshuai Zhao, Yi Zhao, Zhiyuan Li, Juho Kannala, and Joni Pajarinen. Optimistic multi-agent policy gradient for cooperative tasks. arXiv preprint arXiv:2311.01953. 2023.
Ting Zhu, Yue Jin, Jeremie Houssineau, and Giovanni Montana. Mitigating relative over- generalization multi-agent reinforcement learning. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: /pwp//veeevveeeeeeeeeeeeeeeeeeee wwtee//wwwweeeeeeeeeeeeeeeeeeeee tttet//fooomiiidosssste.. 3qU.
Jingwei Zhuo Chang Liu, Jiaxin Shi, Jun Zhu, Ning Chen, and Bo Zhang. Message passing stein variational gradient descent. ICML, 2018
14

---

Published as a conference paper at ICLR 2026
Supplementary Material
Table of Contents
A Stein Variational Gradient Descent
16
B Algorithm Pseudocode
16
Missing Theorems
16
Missing Derivations
D.I Derivation of Equation 12
17
D.2
Derivation of Equation 9
17
E Missing Proofs
18
E.1 Proof for Theorem 3.1
18
E.2 Proof for Theorem 3.2
18
E.3 Proof for Theorem C.I
18
Proof for ERO-free property of SVNR
..........
Proof for Lemma
E.6
Proof for Lemma 4.2
19
E.7 Proof for Theorem 4.3
19
E.8 Full Intuitive Interpretations of Theoretical Concepts
19
E.9
Theoretical Analysis of Relaxed Negotiation Topologics
20
E.10
Error Analysis of Practical Implementation
20
F
More Details
21
Fl1
Environment Details
21
F2
Implementation Details
21
F33
Hyperparameter Selection: Negotiation Rounds (K) and Particle Count (M)
22
G Missing Results
22
G.1 Analysis of Table 2
22
G.2 Additional General
General-Purposs Purpose MARL Basclines on RO Challhhnneged Tasks
23
Missing Ablation Studies
23
H.1 Sensitivity Particle Count M MaMuloCo
23
H.2 Scaling with the Number of Agents
24
H.3 Robustness to Communication/Negottaato Topologies
25
H.4 Sensitivity Analysis of Temperature Parameter ex Computational Analysis and Fair Comparison Protocol
26
H.5
27
More Related Work
29
I.1
Negotiated Reasoning vs. Communication-basse MARL
29
1.2
Relation Opponent Modeling (OM)
30
J Theoretical Grounding of Decentralized Execution via Common Randomness
30
K Theoretical Analysis in Continuous Action Spaces
31
K.1 Measure-Theoretic Unification of the Soft Bellman Operator
31
K.2 Native Continuity of Negotiated Reasoning.....
31
K.3 Bridging the Gap via Particle Approximation
31
L Interpretability of Negotiated Reasoning
32
15

---

Published as a conference paper at ICLR 2026
L.1 Mathematical Interpretation of Rounds and Agreement
32
L.2 Visualizing the Convergence of Measure
32
M Limitations and Future Work
32
M.1 Extended Theoretical Analysis on Partial Observability
33
A STEIN VARIATIONAL GRADIENT DESCENT
Stein Variational Gradient Descent (SVGD) (Liu & Wang, 2016) is a popular Bayesian inference method that sequentially transforms particles to approximate target distributions. Considering target distribution p(æ) where z E X' C. RD, SVGD constructs g(x) from some initial distribution
(x(x)) := M Σi=1 (z:), where is the Dirac delta function, {z li=1 particles at initial, and M is the number of particles. Then it transforms particles with transform function f() = x + cσ(x) where is the step size and X RD is the transform direction. To be tractable and flexible, is restricted to a vector-valuse reproducing Kernel Hilbert space (RKHS) HD = Ho x ... x Hy and Ho is scalar-valued RKHS of kernel k(..)) which is positive definite and in the Stein class of p (e.g., RBF kernel k( y) = exp(-le- x //1// (2h))). According to Stein theory, the steepest direction that minimizing DKL (lp) is
x" (x) = Eynu V, + V va, y)l.
(12)
while is small enough. Update particles based on g.lkk +- xf + e (xf until x (x) O. SVGD ensures = p when the iteration ends and k(x,y) y) is strictly positive definite (Liu Wang, 2016).
MPSVGD (Zhuo et al., 2018) is scalable variant of SVGD that considers the target distribution that can be compactly described by a probabilistic graphical model (PGM). It leverages the conditional independence structure in PGM and transforms the original high-dimensional problem into a set of local problems. Concretely, PGM p(z) be factorized p(x) I[Fe bp (2F) where F C (1..... D} is the index set and Fr = [.a]aer. Then the Markov blanket for d is Fa FU(F:FF d}} Md] and it tells the conditional dependence that p(za| x=a) p(aa æra). MPSVGD updates each dimension with TĄ Ft + eœa(#s.) where S. = {d] UTa a and da E Ha. The Ha is associated with the local kernel ka Xs. x Xs. - R and
(x) = Eys ~ [aa (FS.. Us.. Vy, log P(ya yr.) + V ka us.)]
With enough rounds of updating, the particles converge the target distribution p(r)).
B ALGORITHM PSEUDOCODE
As shown in Algorithm SVNR adopts amortized MPSVGD with a centralized critic learn the policy for each agent. Each agent holds its conditional policy (a (cc, s) with { strict nested set. In the execution stage, agents utilize common randomness to coordinate: they initialize actions using synchronized pseudo-random number generator (PRNG) seeds to generate correlated noise {E,, {c, without active communication. The action of agent is generated by fa, (Ex: &c, s) based on these synchronized noises and local state s. This mechanism leverages the noise as correlation device (Aumann, 1974) rather than a communication channel (see Appendix for theoretical details). After interacting with the environment, all agents sample experiences and aggregate them into the replay memory. Further, based on equation and equation 11, each agent's policy can be updated in the learning phase.
C MISSING THEOREMS
Theorem C.1 (Nested factorization requirement). For a policy factorization method that adopts local policies { (ui uc,).... Nu((ux ucx)) } represent the joint policy Ty((u), it can achieve full joint policy representation capacity if and only if there exists a permutation a of [N) that satisfies
1,+1. ,NN c [ WW-(Coo)))s
[V
Vi.
16

---

Published as a conference paper at ICLR 2026
Algorithm 1 SVNR: Stein Variational Negotiated Reasoning
Initial policy fa, for every agent i; centralized critic Qo: coordination edges C; empty replay buffer P; kernel function K; for agent particle numbers K; target critic Q" : Q². while not convergence do
Input:
Collect Experiences:
Generate synchronized noise & <N I) via common seeds (no communication); Ec. and observe the next state 8', reward Add new experiences into the replay buffer, i.e., D t- DU a,,,,r,,')] Sample Experiences: Sample from the buffer, i.e, {(s, a. r. s'),... ~D D.
Compute action for state s, i.e., u, 6 fWi & s) for each agent
Execute the joint action a = (.),...,,,
LM for state s' and update based on Update Value Functions: For each agent i, sample {a Ji=1
Equation 11.
Update Policies:
Sample noise signals for agent at state i.e. & EV(O., I), v I..., M and generate
k joint actions for state $t, i.e., uf <- f (Ef: $), Ae 1,..., M;
Calculate Afø. based on equation for each agent i, the gradient th; by equation and update yh, using ADAM. if time to update then
Update target parameters: 0 - 0.
end if' end while
For simplicity we denote as C = (C,,..., Ov} e CNested and the CNested is called Nested Coordination Spuce.
The proof of Theorem C.I can be found in Appendix E.3, The above theorem urges us to decompose the joint policy into conditional policies that satisfy the nested requirement. ROMMEO takes C: = -i, Vi, which satisfies our nested factorization requirement and achieves the full capacity.
D MISSING DERIVATIONS
D.1 DERIVATION OF EQUATION 12
Derivation. As proved in the MPSVGD (Zhuo al., 2018), for a graphical model p(z) x Im p(2 )c,). let z = T(x) = [x1...., T, øN] with T; : 24 + 2; + cá's (x). E H4 where Hi is a Reproducing kernel Hilbert Space (RKHS) associated with the local kernel k; &:X: x X R, we have
K. (q1r) (pp) = K.KL (ymn) (z: 2o:) a (2c,) [p 2oz) a (zc:)),
and the solution for min [ovll, 51 K. (1zz)l1)) is of/ |1 where
(x) = (k: yc,) ) V,, logp(u: yc.) + V,, (xc, yc.)
Under mild conditions as states in the MPSVGD (Zhuo et al., 2018), the convergence condition 6, (x) if and only g(x,(=c;) = (I) Take pd and exp (Q") as q and p respectively, then
Aft (t: = Euupe (ki(us. (g.(:)) Q" (st, u') larn + V Vn Ps, (+:
where S. U.
D.2 DERIVATION OF EQUATION
Derivation. One direct way to update the parameter Φi is to obtain by running MPSVGD until convergence and update o;
[+1 +- arg min
1l (Et; - =11
d,
17

---

Published as a conference paper at ICLR 2026
To gain computationally efficient approximation, perform gradient descent step
of + of + c. c. Aff (5: 05.² (5:
dΦ;
with small step size
E
MISSING PROOFS
E.1 PROOF FOR THEOREM 3.1
In NR framework, each agent holds n, = p(u (s) if lim p(u² (s) + x" ("e 8), then
min DKL (ri(pi(u"" [)][[=]))))))))))))))){)))))){ min D Dx (fa (u s)))[=aa)
fi
J.
Thus it is PRO-free after K reasoning rounds.
E.2 PROOF FOR THEEOREM 3.2
+ O, then = approaches to the maximum utility
U max U
0,
due to Quoit = U + E, w0) (- 8z)). For PRO-free agents in NR, plu |
T (w 5) and a O, take म; u,
O.K
then
max U II m/ = U=
Thus they are ERO-free.
E.3 PROOF FOR THEOREM C.1
Proof. The conditional theorem (Gelman Speed, 1993) proves that the { mc:).... N (uN ucx)) uniquely determines the joint policy if and only if the C E CNested For any joint policy ıT, we can obtain
J Iu(u)dupu VIsiSN. S (((uducc
(u (c.)
When the C E CNested the conditional policies uniquely determine the joint policy. Then for arbitrary joint policy, we can represent as the nested conditional policies, and Theorem C.I gets proved.
E.4 PROOF FOR ERO-FREE PROPERTY SVNR
We first prove the strict nested negotiation makes SVNR converge (i.e., the first condition in equa- tion 2). Without loss of generalization, we take C₁ {I....,i) . for every agent i. For agent
G = (1) and the equation 4 degenerate to the SVGD, which has been proved weakly converged to target distribution (u1) in (Liu, 2017):
lim S (ui 8, uc, 1,k ( k
ui u1
Lk-1 WSM M
lim x x 8), k-
Vu Ed
Then with agent 1 converged, agent 2's update degenerate to the SVGD and converges to the target conditional distributions. Iteratively, we can obtain:
lim f (w: 8, u 1k-1) = u
1K-1
vI M,
k-*K
(14)
lim p(u = n" (ult s, L.- Yuf EU, k
18

---

Published as a conference paper at ICLR 2026
Thus prove its convergence.
According to Appendix E.3, the (strict) nested conditional policies can adopted to represent arbitrary joint policy and when the conditional policies uniquely determine the joint policy. Then with equation 14, we have
lim p(u * () = x"(2" s), Vu E EU,is N. k
and thus the SVNR is PRO-frec.
E.5 PROOF FOR LEMMA 4.1
Proof. We refer the readers to the SQL (Haarnoja et al., 2017)'s Appendix A.2.
E.6 PROOF FOR LEMMA 4.2
Proof. Following the Proof E.4, with rounds of SVNR negotiation,
M
#' lim R
Σ
(w), cxp - Q( uc,, ((cc,)))
(15)
M
Then the policy improvement can be proved as in Appendix A.1 of (Haarnoja et al., 2017).
E.7 PROOF FOR THEOREM 4.3
With the Theorem C.I, Lemma 4.1 and Lemma 4.2, our convergence to the optimal joint policy can be similarly proved the SQI.(Haarnoja al., 2017)'s Appendix A.2.
E.8 FULL INTUITIVE INTERPRETATIONS OF THEORETICAL CONCEPTS
To bridge the gap between the formal definitions and their practical implications, analyze PRO and ERO through the lenses of variational inference and distributional factorization. (1) PRO as Variational Bias. In the standard MaxEnt framework, agent i optimizes its policy ni by minimizing the KL-divergence DKL (TiPi [Ta.. where is the perceived opponent policy. PRO arises when o deviates from the true optimal conditional distribution of the opponent (T" Mathematically, this introduces a biased variational objective. Even if agent optimizes perfectly against Pi, the resulting gradient points toward a local optimum (safety) rather than the global optimum (cooperation) because the "belief" pt incorporates the opponent's s exploration noise or historical sub-optimality. PRO is fundamentally training-time estimation error, akin to "shadow boxing" against clumsy opponent; the agent learns to be overly cautious, effectively "learning" to avoid the risk required for optimal cooperation. (2) ERO as Factorization Loss. Even the training phase converges to an optimal joint policy distribution á (where PRO is solved), decentralized execution imposes structural constraint: the executed policy must be the product of independent marginals, मे = I T (us). ERO occurs when the optimal joint distribution क is highly corrllated multimodal. In such cases, the projection i onto the space of independent product distributions results in a significant factorization loss. The support of II, क; inevitably covers leading to a lower expected return than the joint policy s. ERO is an execution-time coordination failure, representing "broken telephone" effect. Even if all agents know the optimal plan in theory, the lack of a mechanism to synchronize their specific random samples at runtime causes them to act incoherently, breaking the optimal joint structure. (3) Consistent Reasoning as Closing the Loop. We define consistent reasoning as the fixed-point condition where two requirements are met simultaneously: (1) Training Consistency, where p; + --, (the variational bias vanishes); and (2) Execution Consistency, where the negotiation mechanism collapses the multimodal joint distribution into a specific mode (agreement) such that 7 (u) z II, TT(u,) as a - 0. This ensures that the planned joint action during the reasoning phase aligns perfectly with the executed action. The negotiation process acts as a ppre-commitmen" device, ensuring that agents not only identify the optimal peak in the reward landscape but also agree converge to the same peak together.
of the state-action space with low utility (miscoordinatioon,
19

---

Published as a conference paper at ICLR 2026
E.9 THEORETICAL ANALYSIS OF RELAXED NEGOTIATION TOPOLOGIES
While Theorem C.I and E.4 rely on strict nesting to guarantee the exact representability of any arbitrary joint policy x², the behavior of partial DAGS and sparse peer sampling can be formally characterized through the lens of Variational Inference and Information Projection.
Information Projection & Approximation Gap. Mathematically, SVNR optimizes the negotiation policy to minimize the KL-divergence ( m²) (Eq. 3).
When the coordination set {C,} satisfies the nested property (Theorem C.I), the family of representable distributions lested is sufficiently expressive to contain T.. Thus, the minimum divergence is zero.
Strict Nesting:
Partial DAGs/Sparse Topologies: Restricting the negotiation set to subset C₁ C C2 restricts the
variational family to a sparser manifold, denoted sggaee In this case, the SVNR update dynamics (Eq. 5 and 9) drive the policy to the Information Projection (I-Projection) of the optimal policy onto this restricted family:
Tisparse
arg min ( (TT +
(16)
Consequently, the performance gap is theoretically bounded by the residual divergence determined by the conditional independencies forced by the graph topology. Specifically, the omitted edges correspond agent pairs with low mutual information in the optimal equilibrium (i.e., weak coupling), the approximation gap D x (f || x;)) remains small. This explains why the degradation observed in experiments is smooth rather than catastrophic: the method finds the optimal approximation allowed by the communication constraints.
E. ERROR ANALYSIS OF PRACTICAL IMPLEMENTATION
In Section 4, we established the convergence of SVNR using the exact soft Bellman operator. However, the practical implementation in Section 5 relies on function approximation (neural networks) for both the critic and the policy. Here, we formally characterize the error introduced by this approximation.
Let T denote the exact soft Bellman operator and II be the space of representable policies. In the practical algorithm (SVNR), we perform an approximate policy iteration. We can decompose error into two distinct terms:
Value Approximation Error (EQ): Instead of computing the exact fixed point Q" T=Q*, we minimize the Bellman residual using a function approximator Qa. This introduces an error bounded by:
Eq
= 1Q 1Q T T Qelloo,
(17)
This error stems from the limited representational capacity of the neural network and the sinite-sample estimation of the expectation E., (s))..
2. Policy Projection Error (Ex): the thcoretical derivation, the policy update is the exact energy- based projection knew exp(Q(ss)) (s)))... J. In our practical implementation (Amortized SVGD), the parameterized policy gg. is updated to minimize the KL-divergence DK ( [/ew). The error here is characterized by the Kernelized Stein Diserepancy (KSD). Specifically, if the update terminates when the norm of the Stein variational gradient bounded by 8, then the resulting distribution approximates the target within an error margin Ex, which vanishes as the number of particles M + oo and the function class of & becomes sufficiently expressive.
Error Propagation: Let Êtotol,k SQ.A + εr, be the combined error at iteration k. Following standard results in Approximate Dynamic Programming (Bertsekas & Tsitsiklis, 1996; Munos, 2005), the propagation these errors through the iterative process is bounded by the discount factor ry The asymptotic performance loss is bounded by:
Cy sup [|
lim I|Q* - @T**lea S (1-7)² A=
(18)
where C' is constant related to the concentrability coefficient of the distribution shift.
Conclusion: The shift from model- based to critic-based implementation transforms the exact con- traction mapping into an approximate one. Crucially, unlike heuristic approximations, the SVNR
20

---

Published as a conference paper at ICLR 2026
error E is structurally controlled: the of SVGD ensures that the policy update direction aligns with the steepest descent on the KL divergence in the RKHS. Thus, the practical algorithm preserves the theoretical monotonicity property up the combined approximation error margin.
F MORE DETAILS
F.1 ENVIRONMENT DETAILS
HalfCheetah-2x3, Partitioning "2x3" splits the half-cheetah into two agents, each controlling three hinge joints: Agent 0 and Agent cach have action space Box (3.)) with joint groups (bthigh, bshin, bfoot) and (fthigh, fshin, ffoot), respectively". Observations support "qpos" and "qvel" categories. All agents observe the position/velocity of the cheetah's tip. All agents receive the same reward as Gymnasium's 's HalfCheetah.
HalfCheetah-Ipl. This environment contains two half-cheetahs coupled by elas- tic tendon, partitioned into two agents ("Ip1"), each controlling six joints. Agent controls (bfoot 0, bshin0, bthig ffoot fshin0, fthigh0); Agent 1 con- trols (bfoot1, bshinl, bthighh, ffoot1, fshinl, fthighh), with action spaces Box(-1,1, (6. )).² Supported observation categories include "qpos". "qvel", the tendon Ja- cobian J"') and tendon length/velocity ("ten length,ten velocity"). All agents receive the average reward of each cheetah. Episodes end as in Gymnasium's HalfCheetah.
Ant-2x4. Partitioning "2x4" groups the ant's front legs into one agent and the back legs into the other. Each agent controls four joints with action space Bo (4, (4,)), corresponding to (hip1, ank 1en, hip2, ankle2) for the front and (hip3, ankle3, hi 14,,,, ankle4, for the back*. Observation categories include "qpos". "qvel", and "efre_ext" by default vl. Global nodes refer to the torso ("root"). All agents receive the same reward Gymnasium's Ant.
Walker2d-2x3 Partitioning "2x3" isolates the right and left legs into two agents. Each agent has a 3D action space Box(-1, 1, (3, )): the right leg controls (foot-joint, leg-joint, thigh.joint), and the left leg controls poot_leff.joirte leg.leet..joiii
thigh.left.joiin servation categories support "qpos" and "qvel". Each agent additionally observes the walker's top. All agents receive the same Walker2D reward.
Ob-
F.2 IMPLEMENTATION DETAILS
For SVNR, we take the negotiation set: C; = (1....,,, Vi. For all experiments, we use the TPE Sampler (Bergstra et al., 2011) select the learning rates, particle numbers, and the entropy coefficient a based on the maximum mean reward in 50 trails. The learning rate and initial a are finetuned in [10 4 10-11 and [10-] 10). and particle numbers are finetuned in an integer space from 16 to 64. Other hyperparameters follow the ROMMEO The optimizer is ADAM, and the sizes of the replay buffer and batch are 10 and 512. k(x, x²) exp(-1/h1) - x' (l2), bandwidth med? / log n. where med is the median of the pairwise distance between the current points {sx ) as suggested in amortized SVGD (Feng et al., 2017). To gain exploration in the early stage, we anneal a based on a a' + exp(-0.) x max(steps 10, 0)) 500 methods in most of the scenarios where a' is the initial a. The only exception is that we anneal a to when we investigate the PRO for all methods.
hhttps://robotinn farama oning-2x---rroot-aaa--eeee
httpp:://iithbbbccmm/mmmmee
rommeo
21

---

Published as a conference paper at ICLR 2026
F.3 HYPERPARAMETER SELECTION: NEGOTIATION ROUNDS (K) AND PARTICLE COUNT (M)
The selection of the negotiation rounds K and particle count M is grounded in the theoretical properties of Stein Variational Gradient Descent (SVGD) and specifie amortization strategy.
Negotiation Rounds (K), From a theoretical standpoint, Theorem 3.1 requires K + oo for the iterative particle updates c T'u((= to converge to the fixed point where the Stein discrepancy is zero. However, in our practical implementation (Section 5, Algorithm 1), we set K 1 for all tasks. This structural advantage of Amortized MPSVGD. Instead of maintaining set of particles that must be iteratively updated K times via the kernel interaction term at every decision step, we parameterize the policy neural sampler = (E: The optimization objective in Equation 8 minimizes the KL divergence. By updating via the chain rule and the Stein variational gradient (Eq. 9 & 10), the neural network distills the multi-step negotiation dynamies into the weights of the function fe. Mathematicallly, the network fū learns approximate the limit of the functional composition of the Stein operator, i.e., (E) s limg- T* (E). Consequently, during both training inference and execution, a single forward pass (K = 1) is sufficient to generate samples that approximate the equilibrium distribution.
Particle Count (M). The choice of M governs the fidelity the empirical measure approximation to the true posterior. M balances the approximation error (which scales with convergence rate related to 1/ /M) against the computational complexity of the Stein gradient (which is (MM2) due to pairwise kernel computations).
Theoretical Lower Bound: M must be sufficient to support the modes of the target distribution. For a multimodal objective (like the "Two Modalities" differential game in Section 6). M must be large enough such that the initial particles cover the basins of attraction for all significant modes; otherwise, the deterministic update dynamics may collapse into a subset of local optima.
Practical Guidance: In our extensive ablation studies (Appendix H.1), we observed a performance plateau where increasing M beyond a certain threshold yields diminishing returns in reducing the Stein discrepancy. We found that M E [32, 40] is the effective range for tested environments. This range provides sufficient particle density to estimate the score function V log x accurately via the kernel density estimate while maintaining low wall-clock training time.
G MISSING RESULTS
G.1 ANALYSIS OF TABLE 2
Table 2: Test performances. The proposed SVNR achieves the highest returns in all scenarios.
Max Of Three (82 (82= 3.0)
Max Of Three
Max Of Three
Methods Scenarios
Particle Gather
(s2 2.0)
(s2 1.5)
SVNR (Ours)
9.60 0.30
9.64=0.11
9.71=0.20
4.76 0.20
MADDPG
2.08 i 4.63
-0.66 = 0.67
-0.64
0.00 1: 0.00
MASQL
8.92 + 0.37
-0.58 0.24
0.12
-0.54 + 0.20
PR2
4.76 1 3.64
-0.64 = 0.45
0..1
0.00 +:: 0.02
ROMMEO
6.14 +: 4.82
1.59 1 5.03
-0.9
-0.87 1 0..2
MMQ
9.54 + 0.13
1.63 + 2.51
-0.07
+ 0.04
- -0.75 + 0.00
Table 2 reports test-time returns for the Max of Three differential game across three coverage factors $2 {3.0, 2.0, 1.5) and for Particle Gather. We summarize three salient observations:
1) Robustness to narrowing basins (Max of Three). As the coverage factor decreases (s, = 3.0 +
1.5), the global optimum becomes harder to reach due to sharper reward basins and stronger gradients toward suboptimal regions (i.e., exacerbated ERO). SVNR maintains near-optimal returns across all settings (9.60 0.30, 9.64 + 0.17, 9.71 + 0.20), while baselines degrade sharply: MADDPG hovers around 0 negative returns (2.08 + 4.63, -0.66 + 0.67, -0.64 + 0.43), and reasoning methods that partially mitigate RO at 3.0 (e.g., MASQL 8.92 0.37, MMQ 9.54 collapse when $2 narrows (MASQL: -0.58 = 0.24 at 2.0, -0.34 = 0.12 at 1.5; MMQ: 1.63 = 2.51 at 2.0, -0.07 + 0.04 at 1.5). PR2 and ROMMEO exhibit high variance (c.g., PR2 4.76 = 3.64 at 3.0) and
similarly deteriorate 82 decreases (PR2: -0.64 + 0.45 2.0, -0.29 + 0.10 1.5; ROMMEO:
22

---

Published as a conference paper at ICLR 2026
1.59 + 5.03 2.0, -0.59 = 0.25 at 1.5). These trends are consistent with negotiated reasoning preventing both PRO during policy updates and ERO during execution.
2) Consistency and low variance. SVNR's standard deviations remain small in all Max of Three settings (at most =0.30), indicating stable convergence. By contrast, several baselines show large variances (e.g.. ROMMEO +4.82 $2 3.0), reflecting sensitivity to exploration- -induucdo- miscoor- dination and order effects.
3) Coordinated arrival in Particle Gather. SVNR attains the highest return in Particle Gather (4.76 + 0.20), where agents must synchronize arrivals to avoid penalties. PR2 and MADDPG remain near zero (0.00++0.0% and 0.000. and MASQL/ROMMEON
negative (e.g.. ROMMEO -0.87 = 0.22), indicating failure to establish reliable joint timing under decentralized execution. These outcomes align with our theoretical guarantees: once PRO avoided and + 0, negotiated reasoning removes ERO at execution.
Overall, the numerical evidence in Table complements the figure-bare based analyses in the main text: SVNR consistently achieves optimal or near-optimal cooperation where RO-prone baselines either collapse or exhibit high variance as coordination becomes more brittle.
G.2 ADDITIONAL SANERAL-PURPORE MARL BASELINES ON RO- CHALLENGED TASKS
To further clarify 'vNR's position in the broader MARL landscape, benchmark strong general- purpose methods (MAPPO, QMIX, FACMAC) on the RO-challenged tasks (Max of Three and Particle Gather). Results in Table 3 show that, despite their strong performance in many cooperative domains, these methods struggle to cope with the PRO/ERO pathologies intrinsic to RO-heavy settings, often converging suboptimal equilibria.
Table 3: Additional ceneral-purpose MARL baselines on RO-challenged tasks. SVNR achieves the highest returns across all settings.
Max Of Three
Max OF Three
Max Of Three
Methods Scenarios
Particle Gather
(#2 3.0)
(82 2.0)
(s2 1.5)
SVNR (Ours)
9.60 0.30
9.64 0.17
9.71 + 0.20
4.76 = 0.20
MADDPG
2.08 + 4.63
-0.66 + 0.67
-0.64 0.43
0.00 0.00
MAPPO
2,87 +: 0.12
0.662 1
-0.68 0.33
-0.00 0.02
QMIX
2.15125.6
0.42 0.56
-0.39 0.32
0.00 1:
FACMAC
2.67 + 1.42
0.51 + 0.52
-0.45 0.28
0.00 0.00
Discussion. In Max of Three, general-purpose methods achieve low or negative returns even at 82=3.0 (e.g., MAPPO 2.87 0.1 QMIX 2.15 2.58, FACMAC 2.67 + 1.42) and degrade further as narrows (e.g., MAPPO -0.68 + 0.33 at 1.5), consistent with their lack of explicit mechanisms to prevent PRO during updates or ERO at execution. near-zero negative returns (e.g.. QMIX 0.02), reflecting difficulty in achieving synchronized arrivals under decentralized execution. By contrast, SVNR maintains near-optimal returns across all RO RO-challenged settings, reinforeing theoretical claim that negotiated reasoning achieves consistent reasoning (PRO-free) and, with a > 0, avoids ERO at execution.
Particle Gather, these methods converge to
H MISSING ABLATION STUDIES
This appendix presents ablations on axes central the practicality: (1) particle count M (SVGD particles used in negotiation), (2) scaling with the number of agents. and (3) robustness to non-strict communicationnnegotttooo topologies. Unless otherwise stated, SVNR uses the same network archi- tectures and training budgets as in the main experiments, with Adam optimizers and identical replay and target update schedules. Wall-clock training time is reported as minutes per 10 environment steps and depends on hardware. Here we give measurements on 1 NVIDIA A100 (80GB).
H.1 SENSITIVITY TO PARTICLE COUNT M ON MAMUJoCo
Protocol. We vary the SVGD partiele count M E {16, 24, 32, 40, 48, 56, 64) on 4 MaMuJoCo tasks. For each setting, run 5 random seeds. We report mean + std test returns and average
23

---

Published as a conference paper at ICLR 2026
wall-clock minutes per 10t environment steps. We also report normalized average performance R.(M) where (MM is the mean return on task t with
score across tasks, NAP((M) - Eier (3(3
M particles, and 32 is the reference setting used in our main experiments. A value NAP(M) ss 1 indicates performance comparable to M-=2..
Results. Across all four tasks, performance is flat in the range M E [24, 48), with mild peak around M=40, and slightly lower returns for very small (M=16) or larger (M=64) particle counts. Training time scales near-linearly with M. These trends suggest a practical sweet spot at M E {32, for best accuracy aceuracy-efficienc trade-off.
Table SVNR particle-count ablation on MaMuJoCo (5 seeds per setting). Mean + std returns and training time. NAP normalized average performance across tasks (vs M- Time Index normalizes minutes per 10 steps by the M-32 setting.
Minutes per 10" steps (Time Index)
HalfCheetah--2x
CoupledHallCheetahh// Ipl
Ant-2x4
Walker2d-2x3
NAP(M)
16
8798 = 240
102 = 95
521 = 38
1601 = 311
0.962
0.73
24
8842 + 220
418 + 90
531 = 34
1650 + 290
0.983
0.85
32
8891 1: 210
429 1: 88
538 1 31
1687 1: 271
1.000
1.00
40
8920 205
435
540
1702 + 268
1.007
1.14
8887
215
431
537
1689 =
1.01
1.27
225
420
1665 1.
8820 235
412
1642 + 289
0.977
1.50
Takeaways. (i) SVNR robust to the choice of M in a broad range; (ii) M E [32, 40] offers a good balance of performance and compute; (iii) training time scales close linearly with M, which matches our design-time complexity analysis.
H.2 SCALING WITH THE NUMBER OF AGENTS
Protocol. We examine how the number of agents affects both performance and wall-clock efficiency when varying M (16, 24, 32, 40, 48, 56, 64): (i) two-agent tasks: the four MaMuJoCo tasks; (ii) three-agent task: Max of Three; (iii) four-agent task: Particle Gather. We define:
:PI
1 M
Σ
(M( TET Re R3()
NTI
Σ MEM
T(M) T(32)*
T
M
where M the particle set, are tasks in the regime, (M) the mean return on task at M, and T(M) minutes per 10th steps (averaged over the relevant tasks). Thus NPIx denotes performance comparable to M-32, and NTI> indicates higher compute cost than M=32.
Results. Performance degrades mildly the number of agents increases, while training time grows sublinearly-to-linee (reflecting both additional policies and negotiation). practice, M E [32, 40] keeps NPI close to across 2 agents with acceptable NTI.
Table 5: Agent-count scaling summary across particle counts M E {16,.... 64). NPI normalized performance index; NTI normalized time index (both relative to M332).
#Agents
Tasks included HaliChheetah-2x:
NPI (mean)
NTI (mean)
2
CoupledHaiKCCeeetah- 1pl, Ant-2x4, Walker2d-2x3
0.995
1.00
3
Max Three
0.989
1.09
4 Particle Gather
0.976
1.22
Takeaways. SVNR maintains near-constant normalized performance agents scale, with modest increases in training time. This suggests the amortized negotiation and correlated sampling scheme effective at containing both PRO and ERO across agent counts with manageable compute.
24

---

Published as a conference paper at ICLR 2026
H.3 ROBUSTNESS TO COMMUNICATIION/NEGTTAAIII TOPOLOGIES
H.3.1 THE FULL SET AND NULL SET
There are two rypical C & CNestod i.e. full negotiation and strict nested negotiation. Our SVNR adopts the nested decomposition that C AJ. We design SVNR-F, which adopts C; -i to show whether making conditions on more agents can improve the performance. Moreover, we also devise SVNR-M as another bascline which is the proposed SVNR adopt C₁ (}. This can be useful to show the importance of let C₁ E Nested We take the experiments on the Max of Three and Particle Gather for further analysis.
Moa
Episoies
(a) 82 3.0
(b) 2.0
(c) 1.5
(d) Particle Gather
Figure 6: Influence of different coverage factors on the training curves of (a-c) our method and different baselines in the Max Of Three. (d) shows the training curves in the Particle Gather scenario. The solid lines and shadow areas denote the mean and variance of the instantaneous rewards with 5 different seeds. With the larger $2, the agents encounter higher impact of relative over-generalization, and the proposed SVNR achieves the optimal solution in all settings.
As shown in Figure 6, both the SVNR and SVNR-F outperform the SVNR-M under = 1.5, 2.0, 3.0 in the Max of Three scenario, which indicates the necessity of taking other agents' noises into consideration. We also visualize their joint actions from to 3000 steps under $2 = 1.5 as shown in Figure 7. Both SVNR and SVNR-F find the optimal solutions, while SVNR-M suffers from RO and is stuck in the sub-optimal areas.
(a) SVNR (Ours)
(b) SVNR-F
(c) SVNR-M
Figure 7: The 1000 sampled joint actions of all methods in the setting of 1.5 in the Max of Three scenario. Each point represents a joint action taken by the agents at specific timestep. and different colors represent the levels of instantaneous rewards. All joint actions are sampled every 3 timestep from 1 to 3000 timesteps the training phase.
Experiments Particle Gather show similar results to those shown in Figure 6d. As shown in the figure, both the SVNR and SVNR-F outperform the SVNR-M in the Particle Gather scenario, which indicates the necessity of taking other agents' noises into consideration again.
H.3.2 MORE STOCHASTIC SETS
Protocol. Particle Gather (four agents) enables testing richer communication//eegottaion graphs. We compare: (i) each agent randomly samples I peer to communicate with per step; (ii) each agent samples peers per step; (iii) a random, partially nested directed acyclic graph (DAG) over the four agents (acyclicity enforeed per step; edges resampled every K environment steps to reduce bias). We report mean + std test returns over seeds and minutes per 10 steps. For reference, we include the strictly nested topology used in theory.
25

---

Published as a conference paper at ICLR 2026
Results. SVNR is robust non-strict topologies: performance degrades gracefully with sparser communication, while wall-clock time improves due to reduced messaging and fewer cross-terms in SVGD updates. Partially nested DAGs recover most of the strictly nested performance with non-trivial reduction in compute.
Table 6: Topology ablation Particle Gather (4 agents, 5 seeds).
Topology
Avg edges/agent
Return (mean = std)
Strictly nested (SVNR default)
3.0
4.62 0.34
Random, partially nested DAG
Ri 2.5
4.33 0.24
Sample-2 peers (per step)
2.0
1.08 0.31
Sample- peer (per step)
1.0
2.37 0.37
Takeaways. Strict nestedness gives the best returns, aligning with our theory on full repre- sentability; (ii) random, partially nested DAGs retain most benefits at lower cost, confirming the Information Projection analysis (Appendix E.9) where the topology preserves sufficient conditional dependencies to capture the bulk of coordination information; (iii) aggressive sparsification (1 peer) remains viable but yields larger variance and lower returns consistent with a wider approximation gap in the variational family.
H.4 SENSITIVITY ANALYSIS OF TEMPERATURE PARAMETER a
The temperature parameter a plays a dual role in the SVNR framework: theoretically, bridges the gap between the stochastic explorative policy and the deterministic optimal execution (as discussed in Theorem 3.2); algorithmically. governs the optimization landscape smoothing. Here, we provide theoretical analysis of why SVNR is robust within bounded range of a, followed by comprehensive ablation experiments.
H.4.1 THEORETICAL ANALYSIS: a AS A HOMOTOPY PARAMETER
Mathematically, the sensitivity to a can be analyzed through the lens of homotopy continuation methods.
The Role of Final a (Approximation Error). Recall that the optimal joint policy is induced by the Boltzmann distribution T. (u|s} x @ u)).
As a + 0: The distribution converges to a Dirac delta function centered at the global maximum: lima-00 T3 (u|s) ō(u u.). This is the condition required for strictly ERO-free execution (Theorem 3.2).
For finite 0: The executed policy retains stochasticity. Let AQ(u) Q u*) Q(s, u) be the sub-optimality gap. The probability of sampling a sub-optimal action u' decays exponentially: P(u') exp( AQ("))
The performance loss (regret) due to non-zero final a'iinal is bounded. If an is small relative to the reward gap of the local optima (the "energy barrier"), the probability mass concentrates effectively on the global optimum. Therefore, precise tuning of ffial is not required, provided final & min AQ(u).
The Role of Annealing Schedule (Optimization Landscape). The annealing process functions continuation method. At high a (early training), the energy landscape E(u) = (u)) ) is smoothed. The Stein Variational Gradient Descent (SVGD) partieles experience gradient field dominated by the entropy term, V log π x - + entropy, allowing particles to traverse potential barriers.
of SVGD provides higher robustness than standard single-point MCMC. Since we maintain a set of interacting particles {u(},M ) with repulsive kernel force E, Vk(u,, u), the particles naturally resist collapsing into local optima too early, making SVNR less sensitive to the annealing rate than standard Soft Q-Learning.
Crucially,
26

---

Published as a conference paper at ICLR 2026
H.4.2 EMPIRICAL SENSITIVITY ANALYSIS
To validate this theory, conducted extensive ablations on the Max of Three (52 1.5) environment, which is highly sensitive to RO. All results are averaged over seeds.
A. Sensitivity to Final a finat). We fixed the annealing schedule (decaying over 50% of total steps) but varied the target floor value finel. As shown in Table 7, performance is stable for any a ffinal E [0, 0.1]. The method is brittle; does not require a to be exactly zero, only sufficiently small to suppress noise below the coordination threshold.
Table 7: Sensitivity to Final a in Max of Three (82 = 1.5).
Final a
Mean Return
Std Dev
Conv. Rate
Interpretation
1.0
6.82
2.15
20%
Too High: Distribution too diffuse (ERO).
0.1
9.15
0.45
100%
Acceptable: Mass concentrates on optimum.
0.01
9.71
0.20
100 %
Optimal: Approximates Dirac delta.
0.001
9.68
0.22
100%
Optimal: Diminishing returns.
0.0
9.65
0.25
100%
Hard Max: Equivalent to greedy execution.
B. Sensitivity Annealing Schedule. We fixed Osstart 1.0 and final 0.01, varying the decay function over the total training steps T. Results are shown in Table 8.
Table Sensitivity to Annealing Schedule Max of Three (82 = 1.5).
Schedule Type
Decay Duration
Return
Std Dev
Analysis
Instant
0% (Fixed 0.01)
-0.65
0.12
Failure: Trapped in local optima.
Fast Linear
10% of T
4.20
4.80
Unstable: "Quenching" causes collapse.
Medium Linear
30% T
9.62
0.28
Robust.
Slow Linear
80% of T
9.73
0.15
Robust: Best stability.
Exponential
0.9995
9.69
0.19
Robust: Smooth transition works well.
The results confirm that while annealing is necessary ("Instant" schedule fails, validating our PRO theory), there exists a wide safe region. Any schedule spanning 30% to 80% of training yields optimal results. The repulsive mechanism in SVGD significantly widens the safe hyperparameter basin compared to standard baselines.
H.5 COMPUTATIONAL ANALYSIS AND FAIR COMPARISON PROTOCOL
In this section, we provide a rigorous breakdown of our fair comparison protocols, including hyper- parameter tuning, entropy schedules, and theoretical justification for the computational trade-offs inherent to SVNR.
H.5.1 THEORETICAL JUSTIHICATION: COMPUTATIONAL COST vs. CONVERGENCE GEOMETRY
A key consideration for SVNR is characterizing what the additional computational complexity achieves compared to standard baselines. While standard policy gradient methods (e.g.. MADDPG) rely gradients in the Euclidean parameter space (O(1) complexity per update), SVNR approximates a gradient flow in the space of probability distributions.
Mathematically, let P(U) be the space of joint policy distributions. Standard updates 0k+1 <- 0g + (V) follow steepest descent in a Euclidean metric. However, this geometry often ill- suited for the non-convex landscape of RO RO-challenged games, where the "valleys" of sub-optimal Nash equilibria are steep and difficult to escape.
SVNR, via the Stein Variational Gradient Descent (SVGD) mechanism, approximates the Wasserstein gradient flow of the KL divergence functional (()) = (xI(pl(=)) The update direction o* the Reproducing Kernel Hilbert Space (RKHS) HD is given the Stein operator:
Φ (u) En'n [k(u', u) V. log "" (u') + Vark(u', w)).
(19)
27

---

Published as a conference paper at ICLR 2026
Evaluating this kernelized update introduces computational complexity of (M2)) (where M is the number of particles). However, this cost yields a descent direction optimal in terms of the Stein Fisher Information. Crucially, the convergence rate is governed by the Stein Poincaré inequality. Unlike standard gradients that vanish at any local optimum (including sub-optimal RO points), the particle interaction term Vw=k(u', u) acts repulsive force, preventing the distribution from collapsing into a single sub-optimal mode. Therefore, although the wall-clock time per step is higher for SVNR, the sample complexity to escape RO significantly lower. The compute budget thus utilized to approximate the optimal transport map from the initial belief to the optimal equilibrium.
H.5.2 HYPERPARAMETER TUNING AND SEARCH SPACES
To ensure fairness, we utilized the Tree-structured Parzen Estimator (TPE) sampler for all methods (SVNR and baselines) with an identical budget of 50 trials per environment. All methods utilized the same network architecture backbone (3-layer MLP with ReLU activations) to ensure that differences in representational capacity did not influence the results. We optimized the search spaces detailed in Table 9.
Table 9: Hyperparameter Search Spaces for TPE Tuning.
Hyperparameter
Search Space
Distribution
Learning Rate (1)
[I 1 x 10="]
Log-uniford
Batch Size (B)
{256. 512, 1024] 2.001,0.011 {1, 10, 100]
Categorical
Polyak Averaging (T)
Uniform
Reward Scaling
Categorical Categorical
Hidden Units (MLP)
{64, 128, 256}
SVNR Specific
Particle Count (M)
{16, 64)
Integer Uniform
Baseline Specific (PR2, ROMMEO)
Recursive Steps (k)
{1.....3) 3)
Integer Uniform
H.5.3 IDENTICAL ENTROPY SCHEDULES
Entropy schedules are critical in MaxEnt MARL, higher a promotes exploration that inciden- tally mitigate RO. To isolate the contribution of the negotiated reasoning mechanism, we employed #*jdentical, fixed a annealing schedules** * for all MaxEnt-bar based methods (SVNR, MASQL, PR2. ROMMEO, MMQ). The schedule used was:
t To
0g
Grend + (Aistant Orend) X exp
(20)
where O'start 1.0, Clead 0.01, and the decay rate Ta was fixed for all agents given environment. This ensures that SVNR's ability to capture multi-modal optima stems from the Stein variational updates, not from artificially inflated entropy.
H.5.4 WALL-CLOCK TIME VS. PERFORMANCE ANALYSIS
We provide a comparison of training time (on single NVIDIA A GPU) versus final performance on the Ant 4 (MaMuJoCo) task in Table 10.
While SVNR incurs higher wall-clock time (~2.2xx compared to simple baselines like MADDPG due to particle processing, it is comparable to other reasoning methods (PR2, ROMMEO), Crucially, SVNR provides the highest "Return per GPU-Hour" because the PRO-free updates prevent the optimization trajectory from oscillating between sub-optimal equilibria, effectively "short-cireuiting the learning process in RO-challenged landscapes where faster baselines fail to converge to the global optimum.
28

---

Published as a conference paper at ICLR 2026
Table 10: Compute Efficiency and Performance Comparison on Ant-2x4.
Method
Params ([(@I)
Time (hrs)
Rel. Time
Final Return
Convergence Step
SVNR (Ours)
~1.2
4.8
1.0x (Ref)
536 31
~1.5M
MADDPG
00.8
2.1
0.44x
108 = 26
Failed (Local Opt)
MASQL
~0.8M
2.3
0.48x
225 34
~2.8M
PR2
1.55M
5.2
1,08x
354 = 58
~2.0
ROMMEO
1.444
4.9
1.02x
424 = 60
~1.8M
MAPPO
0.9M
1.8
0.38x
87 + 135
Failed
I MORE RELATED WORK
Opponent Modeling Our work also has a connection with opponent modeling (Albrecht & Stone, 2018) (OM), which involves modeling the behavior of others. The traditional OM methods only model opponent's behavior based on their history, assuming they play stationary policies (Littman, 2001; Brown, 1951). There are two main limitations to these methods. The first one is that these methods tend to work with predefined targets of opponents. Fictitious play (Brown, 1951), friend- or-foe q (Littman, 2001), and many OM methods (Hu & Wellman, 2003; Greenwald & Hall, 2003; Littman, 1994) make a strong assumption on opponent policies which makes them unsuitable for current MARL where opponents change their policies with learning (Wen et al., 2019). The other limitation is that agents require the Nash equilibrium to update their Q function during training (e.g.. Nash Q learning (Hu & Wellman, 2003) and Wolf models( 2004)). These limitations make hard to apply traditional OM methods to MARL. Compared to the traditional OM methods, our methods do not have these limitations. Besides, some popular OM methods have been proposed: gaasoning-endower methods (Wen et al., 2019; Tian al., 2019), and we have summarized them in the previous subsection.
Probabilistic inference for (MA)RL Formulating RI, problems as prohabilistic inference problems has shown substantial results in obtaining maximum entropy exploration (Haarnoja et al., 2017; 2018; Levine, 2018) and allows a number of inference methods to be adopted. These methods embed the problem into a graphical model by modeling the relations among states, actions, next states, and indicators of optimality. Then the optimal policy can be recovered by making inferences on the graphical model. For example, Soft Q-learning (Haarnoja et al., 2017) expresses the optimal policy via a Boltzmann distribution and adopts amortized SVGD (Feng et al., 2017) to make approximate sampling on the target distribution. Different RL problems, the MARL problem involves a number of agents interacting with each other which makes it non-trivial to make extensions from single agent RL reformulations. MASQL (Wei et al., 2018), ROMMEO (Tian et al., 2019), and PR2 (Wen et al., 2019) let each agent model the relations among states, its actions, the actions of its opponents, next states, and indicators of optimality. Each agent expresses the optimal joint policy via Boltzmann distribution and derives its individual policy and opponent policy accordingly. However, the opponent policy of the agent is not guaranteed to be consistent with the individual policies of opponents. Compared with these methods, the agent in our SVNR perceives opponent policy as consistent with the individual policies of opponents by K-Step negotiation during training.
1.1 NEGOTIATED REASONING VS. COMMUNICATION-BASE MARL
While "negotiation" and "communication" may appear semantically similar, they operate on funda- mentally different mathematical objects in our framework.
Communication addresses Partial Observability. In standard communication--asee MARL (e.g.. TarMAC, BicNet), the objective is to approximate the sufficient statistics of the full global state Mathematically, let O, be the observation space and M be the message space. Communication learns state-dependent mapping μ x,O; M such that the policy Tz(u, (u: O m.-)) approximates the centralized policy x (wals). Crucially, the "message" m is a random variable dependent on the state, i..., V
Negotiated Reasoning addresses Equilibrium Selection via Variational Inference. In contrast, SVNR is an optimization process defined on the probability measure space P(Y).. It constructs flow
29

---

Published as a conference paper at ICLR 2026
of measures f9k ) 1k=o driven by functional gradient descent to minimize the KL-divergence functional
[(g) = Dxz (xz(qTTT)) The "negotiation" the transformation T(u) = u + cΦ(u), where & is the
steepest descent direction in the RKHS Hx. governed by the Stein operator A..:
p (u) E hu)]] Eu-qn V log (u))((u)) Vh(u)]-
(21)
Here, agents exchange gradient information ., Q) and action particles during training to align the joint distribution with the global value landscape. This process changes the optimization landscape to avoid suboptimal local optima (RO), rather than aggregating state observations.
A critical distinction lies in the execution phase. Our method is communication-free in the standard MARL sense (i.c., no state-dependent message passing).
From game-theoretis perspective, the "shared noise" in our Amortized SVNR serves correlation device (Aumann, 1974), not a communication channel.
Standard Nash Equilibrium assumes independent mixing: =(uls) = II, (u,ls) This restricts
agents from coordinating specific optimal joint actions in multimodal landscapes (as seen in our "Two Modalities" experiment).
Correlated Equilibrium (Ours): Agents condition strategies on a public signal &, such that T(uls) II. (u, (,, ())
In our framework, is ex-ante common randomness (e.g., g., synchronized PRNG seed). It satisfies the independence condition & I s. This distinguishes it from communication messages m, where m f(s).
1.2 RELATION TO OPPONENT MODELING (OM)
Our work connects to Opponent Modeling (OM) but differs fundamentally objective.
OM is Predictive. Traditional OM is predictive task (typically regression density estimation) where agent estimates parameters 0_; to approximate P(u-ils, ( via Maximum Likeli- hood Estimation (MLE): ming Ep - log (o())]()) This approach often leads to Relative Over- generalization (RO) because agents optimize against the current (potentially suboptimal) behavior of others.
Negotiated Reasoning is Prescriptive. SVNR provides consistent reasoning framework. We do not merely predict what opponents will do based on history. Instead, we solve for a fixed point where every agent's reasoning consistent with the optimal joint distribution:
lim (u( (u) pi(u_:==/ T (1,
(22)
This satisfies the Consistent Reasoning condition (Definition 2.3), which standard OM fails to guarantee during the exploration phase.
J THEORETICAL GROUNDING OF DECENTRALIZED EXECUTION VIA COMMON RANDOMNESS
In this section. we clarify the theoretical nature of the shared noise utilized in SVNR's 's execution phase and distinguish it from communication.
Correlated Equilibrium vs. Communication. From game-theoreti, perspective, the shared noise & serves as a correlation device (Aumann, 1974), not a communication channel.
Standard Nash Equilibrium (NE); Assumes independent action mixing, <uls)) II, (u.ls) This independence often limits agents to suboptimal outcomes in cooperative games (e.g., miscoor- dination in the "Chicken" game).
Correlated Equilibrium (CE); Allows agents to condition their strategies on a public signal E, such that (uls) II. (u(ul ())(
30

---

Published as a conference paper at ICLR 2026
In SVNR, the sharing of occurs ex-ante. In the literature of Contract Theory and Mechanism Design, this is akin to agents agreeing on a "convention" or a random seed prior to the game to coordinate on specific equilibrium. This is fundamentally distinct from communication in MARL, which is typically defined as the transmission of private observations O;, beliefs, or state-dependent information during execution resolve partial observability. Our method does not transmit state-dependent information; Ranudo- Number Generator (PRNG) seed (common randomness)
it utilizes a synchronized break symmetries and coordinate exploration/exccction without sandwidth cost.
Amortized Inference Implementation. Practically, our Amortized MPSVGD distills the iterative negotiation process into function fv. (Es: .).
Training: Agents explicitly negotiate via the particle updates to find the optimal joint distribution.
Execution: Agents sample actions using the learned policy. The "sharing" of & is implemented simply by synchronizing random seeds among neighbors. This allows agents implicitly coor- dinate their sampling from the joint distribution gø i) without exchanging messages about the state s.
Therefore, SVNR achieves decentralized execution in the sense that data transfer occurs between agents during the decision-making step t.
K THEORETICAL ANALYSIS IN CONTINUOUS ACTION SPACES
While our convergence analysis in Section 4.2 assumes finite action spaces for notational simplicity, implementation of SVNR operates in continuous domains. This appendix clarifies the theoretical consistency between the finite-space analysis and the continuous-space implementation, grounded in measure- measure-theoretic unification and the geometry of Reproducing Kernel Hilbert Spaces (RKHS).
K.1 MEASURE- UNIFICATION OF THE SOFT BELLMAN OPERATOR
The theoretical gap between discrete and continuous analysis is notational rather than structural. The Soft Bellman operator used in our proof's relies on the soft value function. In continuous action spaces U C Rd, this generalizes naturally by replacing the counting measure with the Lebesgue measure. The value function becomes:
V(s) = a log / exp (Q(3,u)
dp(u).
(23)
Provided that Q is bounded and measurable (ensuring the integral exists), the properties of mono- tonicity and contraction (in the _ norm) required for Lemma 4. and Theorem 4.3 hold for the continuous operator just they do for the discrete Consequently, the policy iteration guarantees extend to continuous function spaces under these mild regularity conditions.
K.2 NATIVE CONTINUITY OF NEGOTIATED REASONING
Crucially, the core novelty of work-the Negotiated Reasoning mechanism-is theoretically stronger in continuous spaces.
Our negotiation process (Eq, 4, 5, 12) utilizes Stein Variational Gradient Descent. The theoretical guarantees of SVGD, specifically the Stein Identity and the steepest descent direction in the RKHS 7 are derived explicitly for continuous, differentiable probability densities supported Ed (Liu & Wang. 2016).
SVGD Theory:
The negotiation update u + u u eδ* (u) approximates a gradient flow in the space of probability measures under the Kullhack-Leibler divergence metric. This geometric interpretation relies on the differentiable structure of the continuous action space, which is absent in the discrete setting.
Gradient Flows:
K.3 BRIDGING THE GAP VIA PARTICLE APPROXIMATION
Our method operates in hybrid theoretical regime bridged by particle approximation:
31

---

Published as a conference paper at ICLR 2026
1. Policy Iteration (Global Convergence): As established in Section K.1, the global convergence properties hold in continuous spaces via measure theory.
Negotiated Reasoning (Local Update): As established in Section K.2, the update mechanism is natively continuous.
The "gap" is bridged by our Amortized MPSVGD (Section 5), which uses a finite set of particles {ue}r= f==1 to approximate the continuous posterior. This serves Monte Carlo approximation of the integrals defined the soft value function, which asymptotically exact as M - oo by the Law of Large Numbers. Thus, the finite-particle implementation is consistent approximation of the continuous theoretical framework.
L INTERPRETABILITY OF NEGOTIATED REASONING
In this section, we elaborate on the transparency of the negotiation process within SVNR. The concept of "negotiation" in our framework is mathematically grounded in the iterative transport of probability measures via the Stein variational gradient flow, rather than a heuristic communication protocol. This perspective allows to interpret the learning dynamics through the lens of Amortized Variational Inference.
L.1 MATHEMATICAL INTERPRETATION OF ROUNDS AND AGREEMENT
Theoretically, the negotiation corresponds the functional gradient descent in the Reproducing Kernll Hilbert Space (RKHS).
The rounds K represent the discrete steps taken to transport the initial particle distribution qu toward the target posterior (the optimal joint policy) via the transform T(u) u + cØ" (u). In our Amortized SVNR (Section 5), we distill this multi-step transport dynamic into parameterized function f Consequently, the explicit "round count" collapses into the complexity of the learned mapping, where the network learns to approximate the cumulative effect of the transport.
Negotiation Rounds (K):
The "agreement" is mathematically defined as the system reaching the fixed point where the Stein Discrepancy approaches zero, i.e., [upo(App())) xs O, where Ap is the Stein operator. This implies that the empirical measure of the agents" joint policy matches the optimal Boltzmann distribution.
Agreement:
L.2 VISUALIZING THE CONVERGENCE OF MEASURE
The dynamics of negotiation are explicitly visualized as the evolution of the joint policy's support our experimental results.
Evolution of Support (Figure 5): Figure 5 illustrates the transport of the joint action measure over training steps. Initially (steps 1-1500), the probability mass is distributed over sub-optimal modes (local Nash Equilibria). As the amortized policy fg minimizes the KL-divergence, we observe the concentration measure shifting from the local optimum to the global optimum (steps 1500-3000). This trajectory visually represents the "negotiation" resolving the Perceived Relative over-generalization (PRO) by reshaping the energy landscape of the policy and transporting particles the high-probability regions of the target distribution.
Topological Comparison (Figure 7): By comparing SVNR with SVNR-M (no negotiation) in Figure we isolate the effect of the conditional dependency structure (the nested sets C.). Figure 7(a) versus Figure 7(c) demonstrates that without the Stein transport (negotiation), the joint distribution remains trapped in a sub-optimal mode. The "negotiation" is interpretable as the correction vector applied to the joint distribution that aligns the agents' conditional policies, ensuring the joint support covers the global optimum.
M LIMITATIONS AND FUTURE WORK
This section discusses the limitations and outlines directions for future research.
32

---

Published as a conference paper at ICLR 2026
Computational Overhead and Scalability. The communication complexity of our negotiation pro- cess during training is O(N), where N is the number of aggetss-commann conissaomaaaaeee to standard centralized training methods. While this does not affect execution efficiency (as communication is required during testing), scaling to environments with many agents or high-dimensional state/action spaces may require balancing RO-free guarantees with computational efficiency. Our current implemen- tation uses automated hyperparameter tuning via TPE Sampler to optimize learning rates, entropy coefficients, and particle numbers, providing reliable default configurations across various settings.
Theoretical Assumptions. Our framework assumes nested negotiationloomuniiation during train- ing, consistent with the Centralized Training with Decentralized Execution (CTDE) paradigm widely used in MARL. This allows agents to leverage global information for improved coordination during training while maintaining fully decentralized, communication-fire execution. Other assumptions (e.g., stationarity, bounded rewards) are standard in MARL literature and necessary for theoretical rigor without imposing impractical constraints.
Environmental Complexity. Our validation focuses on standard benchmark environments with sufficient complexity to verify our theoretical claims while maintaining tractability. Extending our approach to more complex, high-dimensional domains represents an important future direction, which will likely require additional architectural innovations to preserve our RO-free guarantees while maintaining computational efficiency.
Partial Observability. The current implementation leverages the Centralized Training with De- centralized Execution (CTDE) paradigm to address partial observability. As demonstrated in the MaMuloCo experiments, our method effectively projects global guidance onto local policies during training. However, explicitly incorporating recurrent architectures (e.g.. Transformers or LSTMs) to better encode long-horizon sequential observations within the negotiation policies remains a promising direction for handling complex POMDPs with severe memory dependencies.
While addressing these limitations is beyond the scope of this paper, they represent valuable avenues for future research that could significantly broaden the upplicability of our RO-free MARL approach.
M.1 EXTENDED THEORETICAL ANALYSIS ON PARTIAL OBSERVABILITY
In this section, provide deeper theoretical analysis regarding the applicability of Stein Variational Negotiated Reasoning (SVNR) to Partially Observable Stochastic Games (POSGs) and the feasibility of fully decentralized training.
M.1.1 SVNR IN PARTIALLY OBSERVABLE STOCHASTIC GAMES
While the main text formulates the problem using global states s for clarity, SVNR naturally extends to POSGs through the lens of projected variational inference. In POSG, agent i observes local history TÜ E Tr while the global state s (or joint history T) available only during centralized training.
The objective of Maximum Entropy MARL this setting to learn a joint policy π rul(r) that minimizes the KL-divergence with the energy-based optimal policy induced by the global Q-function Q(T,u):
min DEL (M(
|| z exp (2)(T.u))
(24)
In SVNR, the negotiation policy is parameterized by amortized neural networks foy (w, Im, &,, fo;) which condition only on local information T;. The update rule in our Amortized MPSVGD (Equation 9 and 10) performs projection of the global gradient onto the local parameter space. The gradient for the local policy parameters 1/2 is:
aJ Oy.
0. (E:
x Ex (S:
(25)
Oyh
33

---

Published as a conference paper at ICLR 2026
Here, Af, (E:T) ) is the Stein gradient computed using the global critic (full observability), repre-
senting the optimal direction in the functional space. The term 0f 0f (E:%)) is the Jacobian of the local policy given local history.
04
This update effectively solves the following projection problem:
2 arg min E+ [KL (9global-ITT) TT. (-1T,)]]-
(26)
By updating d via the chain rule, the agent learns a local policy Te. Tg.(-1r) (-1r:) that is the best possible approximation (in terms of KL-divergence of the globally optimal negotiated outcome, conditioned on its limited view т₁. This theoretical formulation explains the strong empirical performance of SVNR partially observed benchmarks like MaMuJoCo (Table 1).
M.1.2 FEASIBILITY OF FULLY DECENTRALIZED TRAINING
Although our implementation utilizes a centralized critic Qu, s) for sample efficiency, the SVNR framework theoretically compatible with fully decentralized training, provided the global utility function admits a factorizable structure.
Consider a scenario where the global Q-function decomposes according to a factor graph (e.g., pairwise Markov Random Field) consistent with the agent topology:
Qwaa (u, 8) EQc(ue $c),
(27)
<EC
where e represents a local clique of agents (e.g., neighbors) and Qe is a local utility function. The core component of method, the Stein variational update direction for agent is given by:
o (u) Eurg [u, V Quoual (u, s) + V (u, )]
(28)
Due to the linearity of the gradient operator, the score function term decomposes locally:
Vv, Qroral (u, 5) = Σ Vo,Qo(u,
(29)
cifCc
This implies that agent i does not need to query global critic. Instead, it only requires the gradients of the local utility functions from the cliques it belongs to. If we employ a decomposable kernel k(n, n²) = I1 ky (ug, vz)), the expectation term also factorizes.
Consequently, Algorithm can reformulated as a Distributed Stein Variational Gradient Descent (DSVGD) algorithm. In this variant, the "negotiation" during training occurs via gradient message passing between neighbors rather than querying a central oracle, extending the applicability of SVNR to scenarios where centralized training is not feasible.
34


Reviews:
[{"review_id": "LiVAh0dMsp", "reviewer": "Reviewer_biXy", "summary": "The paper reframes relative over-generalisation (RO) in cooperative MARL into two operational notions: perceived RO (PRO) during policy updates and executed RO (ERO) during decentralised execution. It argues that if agents reason consistently about teammates (during training and at test), RO can be avoided. To operationalise this, it proposes Negotiated Reasoning and an instantiation, SVNR, which uses (message-passing) SVGD to negotiate joint actions, embedded in a maximum-entropy policy-iteration loop, with a practical amortised neural version for speed. Theory shows PRO-free negotiation under a strictly nested conditional factorisation and (stated) finite action spaces; experiments on RO-heavy games and multi-agent MuJoCo show strong empirical gains.", "questions": "- Continuous actions: which parts extend beyond finite $|U|$?\n- Do agents share any variables at test time? If not, please fix the pseudocode; if yes, how is this still communication-free?\n- How sensitive are results to final $\\alpha$ and the annealing schedule across tasks?\n- Please provide wall-clock/GPU hours and tuning budgets/ranges for all baselines, and confirm identical exploration/α schedules where applicable.\n- Beyond strictly nested, what guarantees (or empirical scaling laws) hold for partial DAGs/peer sampling?\n- Can you characterise the error introduced when replacing the model-based policy-iteration view with the critic-based practical algorithm?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear, useful split of RO into PRO vs ERO, making the pathology diagnosable during training and execution.\n- Principled negotiation mechanism tied to MaxEnt policy iteration (not just a heuristic add-on)\n- Stated convergence/guarantee story under a strictly nested factorisation \n- Amortised implementation to distil many negotiation steps into one forward pass; practical ablations on particles, team size, topology.\n- Strong empirical wins on PRO/ERO-challenged settings and competitive continuous-control benchmarks.", "weaknesses": "- Core theorems are written for finite action spaces, while key experiments use continuous actions; the theory–practice gap should be tightened or clearly scoped.\n- The paper states communication-free execution, yet the pseudocode shares noise variables between neighbours at test time- I think this needs unambiguous clarification.\n- Guidance on schedules/sensitivity is thin and could affect stability/robustness.\n- The idealised policy-iteration description initially assumes a known model, then switches to critic learning; the implications of this shift aren’t fully analysed.\n- Compute/tuning parity vs baselines isn’t fully tabulated \n- Scope limits: assumes CTDE, full observability; robustness under partial observability is left open.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wmt1Fem8s1", "reviewer": "Reviewer_85jW", "summary": "The paper tackles the problem of relative over-generalization (RO) in cooperative multi-agent reinforcement learning. The authors point out that under CTDE settings, RO arises in two stages: during training (when agents form a perceived joint policy) and during execution (when they act without seeing others’ final actions). They formalize these as Perceived RO (PRO) and Executed RO (ERO), and show that if all agents achieve consistent reasoning—reasoning about others in a way consistent with their optimal or executed policies—then RO can be avoided. To realize this condition, they propose a Negotiated Reasoning (NR) framework where agents iteratively update joint action beliefs through structured “negotiation.” They instantiate this idea as Stein Variational Negotiated Reasoning (SVNR), which leverages SVGD updates and a nested negotiation structure. A neural amortized version enables efficient decentralized execution. Experiments on several MARL benchmarks demonstrate that SVNR avoids sub-optimal equilibria and performs better than prior reasoning-based methods.", "questions": "1. Generalization to partial observability: The consistent reasoning definition assumes full observability of state. Do you foresee PRO/ERO or the theoretical guarantees extending to POMDP settings?\n\n2. Practical guidance: For someone implementing SVNR, how should they select the number of particles 𝑀 and negotiation rounds K? Is there a heuristic or convergence indicator?\n\n3. Interpretability: Can you visualize or quantify the evolution of “agreement” or “negotiation” during training to make the proposed reasoning process more transparent?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Well-motivated and conceptually clear. The paper tackles a long-standing issue in cooperative MARL—relative over-generalization (RO)—from a fresh theoretical angle. The decomposition into Perceived RO (PRO) and Executed RO (ERO) is a intuitive way to separate the effects of exploration during training and coordination failures during execution. This framing alone makes the paper stand out conceptually.\n\n2. Theoretical depth. The authors do not stop at defining RO but go on to establish a sufficient condition—consistent reasoning—under which RO provably disappears. The reasoning flow (formalization → condition → constructive algorithm) is solid and self-contained, which is rare in the MARL literature where many “theoretical” claims are hand-wavy.\n\n3. Negotiated Reasoning framework is novel. The idea of embedding a negotiation mechanism among agents—modeled via particle-based updates and linked to SVGD—is original and technically well-grounded. It provides a new way to understand coordination as an iterative reasoning process rather than just communication or credit assignment.\n\n4. Empirical validation matches the theory. Experiments across both simple and complex cooperative environments (Differential Games, Particle Gather, MaMuJoCo) demonstrate consistent gains.", "weaknesses": "1. Assumptions may be restrictive. The theoretical results rely on strictly nested negotiation sets and the maximum-entropy policy iteration framework. These are strong assumptions, and it’s unclear how sensitive the algorithm is to relaxing them. More empirical discussion of non-strict or sparse negotiation topologies would strengthen the practical claim.\n\n2. Scalability and computational overhead are under-discussed. While the amortized neural version helps, the original SVNR involves particle-based updates and iterative negotiation steps. The paper lacks a quantitative analysis of training cost, memory footprint, or runtime.\n\n3. Limited connection to broader MARL literature.\nThe paper positions itself mainly against “reasoning-based” methods, but doesn’t fully clarify how its negotiation differs in spirit from other opponent-modeling or communication-based approaches. A clearer conceptual contrast would make the contribution easier to situate.\n\n4. Experimental evaluation is strong in breadth but shallow in analysis.\nWhile performance improvements are clear, the paper could provide more interpretability: e.g., what negotiation dynamics emerge, how many rounds are effectively used, or how PRO/ERO evolve during training. Without this, the reader must take the “negotiation” story largely on faith.\n\n5. Writing density. Some proofs and definitions could use more intuitive explanation or visual support (especially PRO/ERO and the consistent reasoning condition). The theoretical sections are mathematically heavy, which may alienate non-specialist readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WJVMHDkggh", "reviewer": "Reviewer_xop6", "summary": "This work proposes a “negotiated reasoning” with an SVGD-based algorithm that proves RO-free convergence under strict conditions. The authors validate the approach on Differential Games, Particle Gather, and MaMuJoCo, improving on returns versus baseline approaches.", "questions": "* Clarify which theorems still hold in continuous action spaces without discretization.\n* Quantify sensitivity to $\\alpha$-annealing with an ablation.\n* Tighten the link between Theorem 3.2’s $\\alpha \\rightarrow 0$ requirement and continuous-control results.\n* Will the authors release their code?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Clean conceptual split (PRO vs. ERO) and a “consistent reasoning” criterion that ties the pathology to modeling assumptions.\n* A (mostly) coherent theory: SVNR via nested negotiation + MaxEnt iteration.\n* Comprehensive experiments show that ERO consistently improves over reasoning baselines and mainstream MARL on MaMuJoCo.", "weaknesses": "* Guarantees rely on strong assumptions, namely annealing $\\alpha$ to 0, and at times finite action spaces, which do not match continuous-control practice.\n* Early theory assumes access to the optimal joint policy, then uses an estimator. The gap between proof conditions and the amortized neural implementation is not fully bridged.\n* Decentralized execution relies on amortization faithfully reproducing multi-round negotiation, a strong limiting assumption in practice.\n* Sample efficiency and robustness are underexplored. At the minimum I would expect training curves for the method versus the baseline approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)