Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning
Authors: Yonghyeon Jo
Sunwoo Lee
Seungyul Han
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Agent Reinforcement Learning
Value Decomposition
Centralized Training with Decentralized Execution
Exploration
Primary area: reinforcement learning
TL;DR: S2Q stores values of multiple subactions, enabling efficient adjustment when the optimality of value function shifts via exploration.

Abstract: Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on a single optimal action and struggle to adapt when the underlying value function shifts during training, often converging to suboptimal policies. To address this limitation, we propose Successive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables $Q^{\text{tot}}$ to adjust quickly to the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms, demonstrating improved adaptability and overall performance. Our code is available at https://github.com/hyeon1996/S2Q.

Published as a conference paper at ICLR 2026
RETAINING SUBOPTIMAL ACTIONS TO FOLLOW SHIFTING OPTIMA IN MULTI-AGENT REINFORCEMENT LEARNING
Yonghyeon Jo, Sunwoo Lee, Seungyul Han
Graduate School of Artificial Intelligence
Ulsan National Institute of Science and Technology (UNIST)
Ulsan, South Korea 44919
(yonghyeonjo, surwoo netwoolees lee, syhan (eriannnunnst.c.mmmm ac. kr
ABSTRACT
Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on single optimal action and struggle to adapt when the underlying value function shifts during training, of- ten converging to suboptimal policies. To address this limitation, propose Sue- cessive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and en- ables Q to adjust quickly the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL al- gorithms, demonstrating improved adaptability and overall performance. code is available at https//githb..om//yymn 1996/S2..
INTRODUCTION
Multi-agent reinforcement learning (MARL) has emerged as powerful and versatile framework for solving complex sequential decision-making problems involving multiple interacting agents (Canese) eet al., 2021; Li et al. 2022; Wen et al. 2022). Its applicability spans wide range of domains, including robotic control, autonomous driving and traffic management, intelligent manufacturing as well as strategic and competitive environments such real-time strategy games (Canese al. 20 al. [2022] Orr & Dutta 2023] Shalev-Shwartz et al., 2016 Bahrpeyma & Reichelt] [2022] Vinyals et al. 2019). Within the centralized training with decentralized execution (CTDE) paradigm, where agents can leverage centralized information during training while acting independently at credit assignment methods (Sunehag et 2017 Rashid 2020a Wang et al. 2020a) have proven highly effective across many MARL benchmarks. ct al.
execution time, value-decomposition-bo
Nevertheless, conventional approaches face fundamental challenges due to the requirements of de- centralized exccution. In particular, QMIX (Rashid et al., 2020a), one of the most representative (IGM) condition (Son al., 2019) through
CTDE methods, enforces the undividual-Globl- monotonicity constraint, ensuring that the joint action-value function increases monotonically with respect to the utility of each agent. Although this design enables competitive performance across diverse tasks, it also restricts the expressiveness of the joint action-value function, limiting its ability to represent complex interactions among agents. Several lines work have attempted to overcome this limitation. For instance, RRahid et al. (2020b) introduces an unconstrained target network to improve the fidelity of optimal- estimation, while Zhou et al. (2022) incorporates auxiliary information to mitigate representational bottlenecks under partial observability. More recent studies reformulate the value function into forms that are more amenable to decomposition (Shen et al.) al. 2024). further improving stability and performance.
2022
Despite these advances, existing methods still struggle when the optimality of the value function shifts during training. This challenge arises not only from the representational rigidity imposed by the monotonicity constraint but also from the heavy reliance on e-greedy policy in the typically
"Corresonndence Scungyul Han.

---

Published as a conference paper at ICLR 2026
large joint action spaces of MARL. To address these issues, we propose Successive Sub-value Q- learning (S2Q), a novel MARL framework that successively leams multiple sub-value functions, each designed to identify distinct suboptimal action. Leveraging these sub-value functions, S2Q replaces naive c-greedy policy with a Softmax-based strategy that encourages persistent visitation around promising joint actions. This allows S2Q to accurately detect shifts in the optimal action and rapidly adapt its policy, thereby avoiding premature convergence to suboptimal solutions. We further provide theoretical and empirical analyses to validate the effectiveness of our approach, demonstrating that S2Q achieves substantial improvements over other recent MARL methods on challenging benchmarks, including the StarCraft Multi-Agent Challenge (Samvelyan et al. 2019) and Google Research Football (Kurach et al. 2020).
PRELIMINARIES
2.1 DECENTRALIZED POMDPS
In cooperative multi-agent RL, the problem is formulated as Decentralized Partially Observable Markov Decision Process (Dec-POMDP), represented by the tuple G = AS P,r,O,O,, O,N, N),,, where S denotes the state space, A the action space of each agent, P the transition probability function, the reward function, O the observation function, O the observation space, N the number of agents, and y the discount factor et al., 2016). At each timestep t, the environment resides in a global state E S, while agent receives a local observation o, := O(s,) E O. Based on its trajectory history TỤ == (ob, ab. o....... )f), agent selects an action aj E according to decentralized policy n². After receiving the joint action environment transitions to the next state $1+1 ~ P(- st. st,aa) ax) and the reward Tg. := r(t,, ar). The
(a)...., a.") from all agents, the
overall objective of MARL is to maximize the expected discounted return, E E = rrr)).
2.2
VALUE DECOMPOSITION UNDER CTDE
Under the Centralized Training with Decentralized Execution (CTDE) paradigm, agents act inde- pendently during execution while utilizing global state information during training. Among the most prominent methods in this setting (ashidashid et al. 2020a). which employs a monotonic mix- ing function to enforce the
(IGM) condition, ensuring that maximizing in- dividual utility Q' cannot reduce the joint value Q"1 Although such alignment is beneficial, the monotonicity constraint limits the expressiveness of Q often hindering effective minimization of the temporal-differenn (TD) loss. To address this limitation, Weighted QMIX (WQMIX)(Rashid ct al. 2020b) introduces an auxiliary unconstrained joint action-value function Q":
(( ((t.
(St. Te, a.) --)) Y/- r. Qinrg (St+1, Te+1, a(41),
where T. -7+ is the joint history, Qing the target, and of arg max (1r,, the individual target action. WQMIX updates Qiat toward the target yt by adaptively weighting the TD error with w(st, ),), where w(+;, a.) = 1 if Qta((sr, Tr, ar) < yr and wot,, at) = w. < otherwise, thereby prioritizing updates underestimated actions.
2.3 COMMUNICATION IN DEC-POMDPS
In Dec-POMDPS, partial observability frequently necessitates inter-agent communication to enable effective coordination (Sukhbaatar et al., 2016). A common approach is for each agent to transmit message m E M, where M denotes the set of all possible messages and m is the message received by agent at time f. These messages are typically incorporated into each agent's 's individual utility function Q' (T/, af, mi), allowing agents to leverage shared information to improve decision- making. For example, MASIA (Guan et al.) 2022) employs encoder decoder architecture that extracts a latent representation 2, from the joint observation and reconstructs the global state s, so that agents can communicate message given by functions of 2, Most communication-basee MARL approaches therefore require message exchange during both training and evaluation. In contrast, we consider a more flexible setting where communication is used only when necessary: evaluation proceed without message passing in environments that do not require it, while in communication- critical tasks, agents still exchange latent representations coordinate effectively.
2

---

Published as a conference paper at ICLR 2026
3 RELATED WORKS
3.1 CTDE METHODS IN MARL
Within the CTDE paradigm, value decomposition methods decompose the joint action- value into individual utilities, ensuring scalable MARI. (Sunehag et al.
2017 Rashid et al. 2020a Yang et al.) 2020). Building this foundation, recent works suggest
expressive identity representation 2023a), or assigning roles based on agent trajectories (Wang et al. 2020b Zeng et al. 2023). Another line of work extends exploration strategies studied in the single-agen agent setting (Tang et al.
Naderializadeh et al., 2020, Zang et al. |023] Liu et al.
2017 Burda et al. 2018: Han & Sung
b0 to the multi-agent domain under the CTDE paradigm, introducing centralized latent vari-
ables or coordination signals to capture richer joint behaviors (Mahajan et al. 2019 Li et al. [02T]] Jo al. 2024). parallel, several works focus on making real-world deployment more feasible via improving the sample efficiency, (Yang et al. 2024 Qin et al. 2024) and robustness of MARI. algorithms (Yuan et al., 2023] Lee et al. 20 Finally, another line research extends the actor- critic framework to the multi-agent setting by employing centralized critics with decentralized actors
Lowe et al. 2017
Foerster et al. 2018 Iqbal Sha 2019 Su et al. 2021
3.2
OVERCOMING THE MONOTONICITY CONSTRAINT
While ensuring tractability, QMIX's s monotonicity constraint significantly limits representational capacity, motivating active research to overcome this restriction. WQMIX (Rashid et al. (020b) mitigates these limitations through weighted projections and non-monotonic targets. ad addition, some methods leverage centralized critic with inherent constraints (Wang al. 2020d Zhang et al., 2021; Peng et al. (021), while recent works pursue more flexible factorizations of the joint action-value function (Son al. 2019 Wang al. 2000al Wan al. [021 Li et al. 2023b 2024). or incorporating alternative learning objectives (Zhou et al. 2022 Hu et al. 2023] Zhou et al. 2023
Li al. 2023a Liu et 2023b). A representative example of such an objective is risk sensitivity,
accounting for return variance (Qiu et al. [2021; Shen et al. 2023 Chen et al. 2024).
3.3 COMMUNICATION IN MARL
Communication has emerged as a critical mechanism in MARL for enabling coordination under partial observability. A range of works focus on the design of communication protocols, specifying how and when agents should exchange messages (Foerster et al. 2016: Sukhbaatar et al., [016 Liu et al. 2020; Hu al., 2024). Others investigate the type of information that should be shared to ensure that communication is informative (Li & Zhang 2023 Shao et al., 2023). Another group aims improve communication efficiency by reducing redundancy or compressing messages (Wang al. 2019 Guan al. 2022). Finally, selective communication strategies aim to determine which messages to send or whom to communicate with, thereby reducing unnecessary communication
overhead (Das et al. 2019 Yuan al. 2022 Zhu et al. 2024 Sun et al. 2024).
METHODOLOGY
4.1
MOTIVATION: OVERCOMING DYNAMIC OPTIMALITY SHIFTS IN MARL
Oridinal Value
Shifted Value
Original Value
Shifted Value
Value
Shifted
A
N
N
A
8
-12
-12
A
6
-12
-12
A
.11
11.1
11.11
A
115
11
-1.3
A
8.0
-66
4.66
A
6.0
-55
-6.5
-12
7
0
-2
7
0
B
-11.1
6.8
0.6
.113
6.8
03
B
-1.
16
1.
B
-6.5
-55
-6.5
C
C
6e) Payoff waue vayinn
(0) CMML 91"
(6) WOMIK goe
Original Value
Siited Value
Original ValLe
Shitted Vilue
Original Value
S-itted Value
N
8
56
-5.
A
3.
5/
5/
A
0.0
3.0
4.0
A
0.1
0.1
0.1
A
0.1
-.1
0.1
A
6.0
0.0
0.0
-56
-5.7
-5.7
3.7
57
5.7
0.0
B
57
5200K-2
get
S2C/K=2 0T
S2QIK= Q1"
Figure 1: Fundamental Limitations of value decomposition algorithms. (a): The actual payoff of the matrix game, (b), Training result of QMIX and WQMIX. (d): Training results of S2Q when K 2, where K is the hyperparameter controlling number of sub-networks use.
3

---

Published as a conference paper at ICLR 2026
In this section, we revisit key limitation of value decomposition under the CTDE paradigm. QMIX is known to struggle in representing the true global value due to the IGM constraint, and several methods have been proposed to mitigate this issue (Rashid al. (020b). Most focus on learning an unconstrained value function (e.g.. WQMIX) or enhancing state-awareness for better inference of the global optimum. Yet, core challenge remains: by focusing solely on a single optimal joint action, these methods still fail to recover the global optimum when the value function changes dynamically.
To illustrate this issue, we consider the payoff matrix game in Fig. D consisting of a single state with two agents, each selecting actions from A, B, C. The payoff matrix specifies the joint value for each (a),a²). In Fig. Ja. left), the optimal joint action is (A, A) with value 8, while (B, B) and (C, C.) are suboptimal with values 7 and 6. After training the algorithms to convergence, we modify the payoff as shown in Fig. 1 a. right), where the optimum shifts to (C, C) with value 8, and (A, A) and (B, B) drop to 6 and 7. This setup mirrors realistic MARI, scenarios in which exploration updates value estimates and causes the optimal action to change. Figures (c))-(c) show the learning behavior of QMIX (Rashid et al. 2020a) and WQMIX (Rashid al.. (020b) before and after the payoff change. We use WQMIX as a representative example since the methods relying on enhanced
are expected to behave similarly to WQMIX, as the matrix game involves only a single state. The results reveal that QMIX fails in both cases because its monotonic mixing network cannot capture the non-monotonic value structure, while WQMIX uses an unconstrained target Q" and a weighted TD objective but still fails to adapt when the optimum shifts to (C, C)- These findings expose a key weakness, as existing methods often converge suboptimal solutions because they are unable to track a moving optimum.
We attribute this failure to the fact that conventional methods do not explicitly track suboptimal ac- tions. Once information about alternative high-value modes is discarded, the learner cannot adapt when those actions later become optimal. To overcome this limitation, we propose Successive Sub- value Q-learnngg (S2Q), novel MARL framework that successively learns sub-value functions Q 1 I.... K, which share the same architecture as Q but are each dedicated to capturing a distinct suboptimal action. When the optimal action changes, S2Q can immediately leverage the cor- responding sub-value function and guide Q to adapt. As shown in Fig. (') with K = 2, under the original payoff matrix Q == Qti learns the optimal (A, A), while Q1 and Q7th capture (B, B) (second-optimal) and (C, C') (third-optimal) After the payoff is modified, Qu identifies (C, C) as optimal, enabling Qet to rapidly pivot to the correct solution. Beyond adaptability, the maintained sub-valte value functions also support more effective exploration than standard e-greddy by actively sam- pling alternative high-value modes. To further illustrate S2Q's s exploration behavior, Appendix D presents heatmaps of joint action probabilities showing that S2Q identifies and prioritizes valuable suboptimal actions and consequently adapts quickly when the value landscape shifts. Through this design, S2Q is expected to achieve faster convergence than existing CTDE methods, demonstrating its efficiency across diverse MARL environments. The next section details the successive learning scheme of the proposed S2Q.
4.2 SUCCESSIVE SUB-VALUE Q-LEARNING FOR RETAINING SUBOPTIMAL ACTIONS
Suppress ('(9).,nde..)
Suppress (D0ann))
Suppress (1))
Suppress (D0)
a22
a 1-1
Q8
1 Mixing Network F
9
Mixing Network
Mixing Network -%
Mixing Network -5p
0%
O5
Figure 2: Illustration of S2Q framework. Each subnetwork Qeubik transmits Ah.. set of optimal actions according to all previous subnetworks. Qsud,k+1 learns the unrestricted target Q* while suppressing the Q-values of actions included in A².
We now present the detailed formulation of S2Q, introduced in Section 4.1 S2Q trains multiple sub-value functions to follow Q" as in WQMIX, but with one key idea: the first sub-value function identifies the maximum joint action under the IGM condition, and the next sub-value function sup- presses its value in the learning objective. This makes the action no longer the maximum, allowing the next sub-value function to learn the highest remaining joint action under IGM, effectively the
4

---

Published as a conference paper at ICLR 2026
next suboptimal action of Q". Repeating this process yields a sequence of high- nigue candidates, with each sub-value function specializing in a different region of the action space.
Specifically, S2Q employs sequence of sub-value functions Q l 0, ........ K to successively track optimal and suboptimal actions. By definition, we set Q8 := Qua, where QtI is trained to select the true optimal action, while each Q% aims to capture the k-th suboptimal action by suppressing those identified by previous sub-value functions. All sub-value functions share the same mixture architecture as QMIX and satisfy the IGM condition. Specifically, each Q consists of independent individual utilities Qi and the joint greedy action a = arg max Qj (TÈ. corresponds to the k-th suboptimal joint action.
-,,,)) with aj =
To optimize each Q we build upon the TD-lcarning objective of WQMIX by leveraging the unrestricted Q" trained via equation[Tante incorporating suppression term for previously identified suboptimal actions. The resulting objective is defined as:
E(e [ ( ar) @Qith (St, a.) ax)= (y+ - all(aa E Ak-1,)) max(Quin (8t, Tı, at). CJ) 2
where WE is WQMIX-based weighting function for each k. detailed in Appendix 3/+ Te + (0iin((441 (SL+1, T141, 11)+ ) is the target from Q" in equation [ I is the indicator function, B with A- = B denotes the set of previously identified suboptimal actions. The factor a controls how strongly the values of actions in Ax- are suppressed. and max(-, C*) with positive constant C' > 0 is applied to properly handle potentially negative values of Qiarg. When Varg is positive, we can simply use Qiarg instead of max max(Qi (Qaag. C) in equation 2 for implementation. From the proposed successive Q-learning, each sub-value func- tion Q can select the k-th suboptimal joint action of a given Q" exactly, as guaranteed by the following theorem, provided that the suppression constant a is sufficiently large.
is the replay buffer, and Ar, = {ai. ai....... a
Theorem 4.1. Let (st. Te, ax) and 1
K k=o be the joint action-valus function and sub-value and equaatinn 2 respectively, and let (aū....., ak,))
functions obtained by minimizing equation I denote the K K+1 successive suboptimal joint actions of Q" for given St and Tr at timestep t s.t.
Q" (st. Te, Te,,T,,80 2..)) 2 ... (01, Tı, ak,)) 2 (01, Tı, at), Va, & "eū.....,,k... If the reward function r is bounded and suppression factor a is sufficiently large, then,
a arg max Quub (st, Te, at) wk E {0, K}.
(3)
Proof) Proof of Theorem 4.1 is provided in Appendix
According to Theorem 4.1, it is guaranteed that, with bounded rewards and a sufficiently large sup- pression factor a, the successive learning procedure maintains Q ...,, Qush where Q71h represents the global optimum and the remaining sub-value functions capture successive suboptimal actions. By explicitly tracking the high-value joint actions, S2Q allows Q" to effectively update its values and ensures that Q can promptly adapt when the optimal action changes. Fig. 2 illlsstrates this process, where each Qia suppresses previously identified actions before selecting the next best candidate. Importantly, although this procedure is presented in the context of WQMIX, it is general and can be applied to any CTDE method by replacing Q' with Q"x Consequently, S2Q can follow changes in the value landscape more closely and adapt to new optima faster than conventional approaches.
4.3
COORDINATED EXECUTION VIA COMMUNICATION DURING TRAINING
As described above, S2Q continuously tracks suboptimal actions, but for these actions to contribute effectively, Q" must be trained toward global convergence. However, prior MARL methods typically rely on agent-wise e-greedy exploration, under which the probability of joint exploration decreases exponentially with the number of agents, causing most joint actions to remain near the current op- timum. To overcome this limitation, we explicitly execute tracked suboptimal actions with priority determined by a Softmax distribution P, over their Q" values, thereby ensuring more frequent visits, enabling Q" to find better optimal and suboptimal actions, defined as
P. = (Pot .. Pk.) == Softmax QQ" (st, Tı, aT...T....... " Tr, a/T)
(4)
5

---

Published as a conference paper at ICLR 2026
Policy
Algorithm 1 S2Q Framework
Train (comm.)
k of c E-greedy (ereedy(())
1: Initialize Q", [Q k==
"k= (E, D)
Test comm.) a greedy((.)
2: for each training iteration do
3: for each environment step do
4: if training then
Value
Encoder-Decee Decoder
$2Q Learning
5: Obtain 24 E(T()
&
P -
Softmax( (ab.)) ('(ak:))
6:
Obtain Pr, St = (zt)
Recon
7:
Sample k ~ P.
ai:t
ag:
D
Sample a ~ e-greedy(Q
t
Q
DuD
Q
else
10:
Sample a. ~ greedy(Q end if
Zt
Oh (5,
11:
Mixing Network
12:
end for
E
13:
Compute target
(
14:
Update Q", Quub via Equation I
15:
Update E, D
Figure 3: Overall framework of S2Q
16:
end for
where T is temperature parameter. The behavior policy first samples k Pi, and then executes actions from Q according to e-greedy rule, i.c., a; ~ e-greedy(Qi.. This design first selects based Q and then explores around the corresponding suboptimal action, and empirically show in Appendix thatt S2Q visits a wider range spaces than conventional c-greedy exploration.
Exact computation of Pt, however, requires access to global information that decentralized agents cannot directly observe. More importantly, since all agents must select the same sub-value func- tion index in order to execute @ consistently, coordination among agents is necessary. To solve these challenges, we introduce communication during training, allowing agents to estimate Pr jointly and synchronize their choice Q For this purpose. we adopt an encoder-decoder ar- chitecture for general representation learning, following prior communication methods (Guan et al.) Wang et al. 2019). Specifically, encoder maps local histories т; into latent representation E(T.). and the decoder D reconstructs both the global state and an approximate distribution (št, P.) D(2.). Agents then synchronize the same sampled from P, instead of P, and execute the corresponding 1 ensuring consistent and accurate use sub-value functions.
2022
At test time, by contrast, communication is required in the default setup of S2Q. Since @abh= = alone suffices to produce the greedy optimal action aj.:> evaluation remains fully decen-
Q0th = tralized and communication-fies This provides practical advantage cation methods that rely on message passing even during evaluation. Nevertheless, in environments where communication indispensable for task success, also consider a variant, denoted S2Q- Comm, in which the latent is provided to each Q during both training and execution. Through this practical design, S2Q integrates successive sub-value functions with Softmax-bases behavior policy, enabling Qeot to adapt quickly when the optimal action changes. The overall S2Q framework illustrated in Fig. 3 where each local history Tę is processed along the S2Q Learning module, and the Encoder- Decoder module to generate the softmax probabilities Pt, and its prediction P,, During training, agents use P; to synchronize their selection of the sub-value function Q while evaluation relies solely the local outputs. The training procedure is summarized in Algorithm and further details on loss functions and implementation are provided in Appendix
conventional communi-
5 EXPERIMENTS
In this section, we evaluate S2Q on two widely used MARL benchmarks: the StarCraft Multi- Agent Challenge (SMAC) (Samvelyan et al. 2019), which involves micromanagement tasks in StarCraft II. and Google Research Football (GRF) (urach et al. 2020), which features coopera- tive like scenarios with opponent team including goalkeeper. As shown in Fig. aa), consider the SMAC-Harde suite, consisting of one hard map (5m vs 6m) and five super hard maps (MMM2, 2 7m 30m, corridor, 6h 8z, 385z vs 3s 6z), all of which require high degree

---

Published as a conference paper at ICLR 2026
(a) SMAC-Hard+: Corridor
(b) GRF: academy 3 3.vs.22
(c) SMAC-Comm: 1o. 2r vr.4r
Figure 4: Experiment environments
of coordination among agents. Fig. 4 b) illustrates the GRF setups, including academy_3. ___v__________ and academy_4 _e_veemy___vvseee _______, where the number of agents and team formations are varied. In addition, Fig. ()) depicts the SMAC-Comm suite, which explicitly requires agent communication for suc- cessful task completion, including 1o.10b.vs_lz 10 2r_vs_4r, 5z-vs_lul. and bane_y .vs hM. Further details of the environmental setup are provided in Appendix E For all reported result tables and plots, we present the mean and standard deviation across five random seeds.
5.1 PERFORMANCE ANALYSIS
For performance evaluation, we consider communication free tasks from SMAC-Hard+ and GRF, along with communication-demandiii scenarios from SMAC-Comm, with appropriate baselines for each setting to ensure fair comparison. In the SMAC-Hard+ and GRF benchmarks, we compare S2Q with QMIX and methods addressing the limitations of monotonic value decomposition, includ- ing WQMIX which leverages Q" for more accurate value estimation; DOP (Wang et al. 2020dh and FOP (Zhang al., 2021), which extend the cetor-critic paradigm to promote better global coordination; PAC (Zhou et al. (022) and RiskQ (Shen et al.] (2023). which integrate counterfac- tual prediction or risk-aware objectives; and MARR (Yang et al.] 2024), which improves sample efficiency through reset mechanisms. We also include MASIA (Guan et al. (022),, which recon- struets global state information from local observations, to evaluate the effect of communication. For the SMAC-Come Comm scenarios, where information exchange among agents is critical under par- tial observability, we adopt S2Q-Comm, variant that provides the latent 2 to each agent's 's utility Q'(T', 2,,0),)))) af), and compare it with QMIX and QMIX-Comm, variant of QMIX augmented with learned latent information produced by an encoder decoder structure that reconstructs the state, FullCom, which augments QMIX with full observation sharing, and recent communication-focusee MARL methods such as MASIA, NDQ (Wang et al.. 2019), and CAMA (Shao et al., 2023), which enhance communication via information-theoreii regularization or complementary attention, as well as MAIC (Yuan et al. 2022) and T2MAC (Sun et al. 2024), which employ selective and targeted communication for improved coordination efficiency. All algorithms are implemented us- ing the authors' official code, and S2Q is trained with the best rest-performing hyperparameter (K 2, 0.1). Since the Q-values quickly turn positive in environments we consider, we adopt Qüarg instead of max (aax(Qiarg: C') equation for implementation, as mentioned in Section 4.2 Additional experimental details, including the hyperparameter setup of S2Q are provided in Appendix
SMAC-Hard+ and GRF: Fig. 5 compares performance on SMAC-Hard+ and GRF. S2Q con- sistently outperforms existing baselines, achieving faster convergence and higher asymptotic re- turns. This advantage is most evident in exploration-intensiiee scenarios such as 6h_vs_8z and 3s5z_vs_3 33 6z, where conventional methods adapt slowly to the optimal joint action shifts, as shown in the payoff matrix experiment Section 4.1 By continually tracking suboptimal actions with successive sub-value functions, S2Q allows Q to rapidly adjust toward the new optimum, enabling more efficient exploration and quicker convergence, even in the challenging GRF tasks.
SMAC-Comm: Fig. 6 results for communication-critee tasks. 32Q-Comm performs com- scenario, but demonstrates substantial gains in
parably to baselines in the simpler more challenging settings such 5z_vs_lul and bane_vs _M,,, where tight coordination and real-time adaptation are required. Synchronized sub-value function selection through the learned
Throughout our experiments, we adopt OW-QMIX variant of WQMIX (Rashid al. 2020h

---

Published as a conference paper at ICLR 2026
Tost Win Rate (50)
Test Winh Patt- 196)
Ros Winn Ruata (N6
Timesteps
Timesteps
TimostepG
Timesteps (M)
SMMC-Harde: mmm_
SMAC-Harde: MMM2
SMAC-Hardk: 27m_y__00m
SMAC-Harde: Corrior
Test Win Rate (5)
Test Win Pate-
X040
55
50
Ya
Timesteps 42
Timesteps
Timestops
Timestaps
SWNC-ttrr=: 8
SMAC-Haarde
GRF: arademym_syv
GRF: academym__.p
Figure 5: Performance comparison: Average test win rates in the SMAC-Hard+ and GRF tasks
Naa Win Ratt 143
VAS FATS 250
1JEE CORE
Text. Ratt
NEO
Timesteps
50
[MI
50 Timesteps
Timest
2r
1MACmComm::1
Figure 6: Performance comparison: Average test win rates in the SMAC-Comm tasks
latent =, enables agents consistently exploit informative suboptimal actions, leading more effi- cient cooperation. addition, comparison with OMII--Comm shows that these improvements arise from S2Q's s successive sub-value learning, rather than relying solely on information sharing. These findings suggest that S2Q-Comm can serve as general approach for leveraging communication efficiently, providing promising direction for scaling larger teams and more complex partially observable environments.
Together, these experiments show that by continually tracking and exploiting suboptimal actions, S2Q achieves faster convergence and more efficient learning than state-of-the-art MARL methods. We further evaluate S2Q on additional stochastic environments. including SMACv2 (Ellis et al. 2023], which randomizes team compositions, and SMAC-Hard (Deng et al. 2024). with mixed scripted opponents. For clarity, we refer the latter benchmark SMAC- (Mixture Oppo- nent) to distinguish it from the SMAC-Ha rd+ environment. Despite the increased stochasticity, S2Q consistently outperforms all baselines, suggesting that our method can effectively address stochas- tic dynamics meaningful extent. Furthermore, we extend S2Q to other CTDE methods such as VDN (Sunchag et al. 2017) and QPLEX (Wang et al.) 2020a), observing similarly consistent performance gains. Detailed results for these additional experiments are provided in Appendix
5.2 TRAJECTORY ANALYSIS
To investigate the training dynamics of S2Q (K 2). we analyze the distribution of agents' ac- tions in the 6h vs 8z scenario, shown in Fig. a)). Fig. b)) reports the evolution of hit rate and win rate over training. Since SMAC tasks require agents to defeat the opposing team, the dominant actions are move and hit. Early in training, agents tend to prioritize survival, favoring move and exhibiting low hit rates. However, training progresses, the hit rate gradually increases, leading to higher win rates, indicating that the optimal policy shifts from evasive behavior towards more aggressive strategies where hit dominates. Fig, ( illustrates how S2Q adapts to this shift in op- timality by leveraging its successive sub-value functions. Initially, move considered the optimal action (a),), while hit actions (a a²,)) tracked suboptimal. As training proceeds and Q"
8

---

Published as a conference paper at ICLR 2026
Ally
Enemy
Move
Hit
Enemy
=
Hit
( - 2
Hit
Enemy
Hit
Ally
Move
Move
Move
Move
(a)
Exploration (2M)
Optimal (5M)
6h vs Bz
80
a
TEST WIN RATE
HIT RATE
Value (%)
4
(((iz)
(*
(4i.2
(*)
(ai.2)
66 6
490
4.545
10 1
10.2
80
Timestons (10%)
06
02
Ag 02
ai
01
0g
02
06
02
d6
02
(b)
Figure 7: Training behavior of S2Q. Left: training step 2M. Right: training step 5M
recognizes that hit yields higher returns, S2Q gradually increases the execution frequency of hit via the Softmax based behavior policy. This enables Q5 to promptly adjust the newly optimal ac- tion, resulting in an increase in hit rate and ultimately higher win rates. These results provide clear illustration of how S2Q tracks suboptimal actions and uses them to adapt efficiently when the opti- mal action changes. Additional trajectory analysis for the SMAC-Comm, SMAC-Hard (Mixture Opponent). , and GRF environments are also provided in Appendix H.4 While detailed analyses are provided in Appendix H.4, we observe a consistent trend across environments: agents frequently begin by selecting locally greedy actions conditioned on their initial states, but gradually transition toward the truc optimal actions training progresses.
5.3 ABLATION STUDY
In this section, we present ablation studies evaluating the contributions of S2Q's core components and hyperparameters. Component analysis is averaged across all scenarios in SMAC-Hard+, while hyperparameter analysis is conducted on 6h_vs_8z, where S2Q's benefits are most evident. Addi- tional results, including additional hyperparameter analysis in MMM2 and academy 4.. 4.vs. which exhibit trends consistent with those observed in 6h_vs.8 ablations on the suppression constant a, the weighting factor we for WQMIX-base based TD learning, and computational complexity, are reported in Appendix H to validate the effectiveness and generality of S2Q across diverse settings.
Table 1: Component evaluation of S2Q on SMAC-Hard+ tasks.
K-2 (OUB)))
(OURS)
Sact win Pate (%)
Test Win Rate (%6)
T-000
Tw0
Method
Avg. Win Rate(%)
S2Q
73.43 5.29
S2Q_oracle
77.47 4.32
S2Q independent
46.22 8.20
03 30 13 20 30
30
25 20 25 20
$2Q_no.wTDD
70.59 + 4.78
SMAC-Hard+: 6h,
Timesteps (M)
Timesteps (M)
8r
S2Q no soft so
55.17 + 6.71
(a)
(b)
S2Q_random
48.05 9.37
Figure 8: Hyperparameter analysis. Effect of (a) the number of sub-networks K and (b) Softmax temperature T.
QMIX
43.94
Component evaluation: evaluate the contribution of each component S2Q, we design five ablations: S2Q oracle uses the true softmax distribution P, instead of the estimated P,; S2Q independent samples l independently, removing coordinated execution; S2Q no wTD

---

Published as a conference paper at ICLR 2026
moves the weighted TD learning by replacing Q" with h6th; S2Q_nq soft eliminates Softmax- -baedd execution, always acting according to Qh and randoo uniformly samples rather than using Softmax probabilities. Tablerreports the average win rate across six SMAC-Hard Hard+ scenarios.
The results show that 'S2Q oracle' serves as a reference upper bound, since it can access to the true softmax distribution P and therefore achieves higher performance than S2Q. In contrast, .22_independdnt exhibits a substantial performance drop relative to S2Q because it cannot ac- curately estimate the selection probabilities and consequently fails to maintain proper synchroniza- tion across agents. Together, these ablations clearly demonstrate the critical role of accurately es- timating the softmax probabilities tion. 'S2Q no wTD' maintains strong performance, indicating that successive sub-value learning and its execution are more critical to performance than the auxiliary weighting scheme. In con- trast, 'S2Q_no_soff suffers a noticeable drop in win rate, showing that simply retaining suboptimal actions without prioritizing their execution is insufficient. 'S2Q_random' further degrades perfor- mance, as ignoring the relative importance of different sub-value functions dilutes learning. These findings highlight that the proposed successive sub-value learning and the Softmax-guider execution indispensable for enabling S2Q to achieve rapid convergence toward high-quality joint policies.
enabling effective coordinated sub-value selection. In addi-
Number of sub-value functions K: Controls the number of sub-value functions to learn. Fig. S(a) shows performance for K E [0, 2, 3]. The results demonstrate that even with relatively small values of K, S2Q can effectively address diverse suboptimal actions in complex environments with large joint action spaces, such SMAC, yielding rapid convergence and high final win rates. In particular, K = 2 achieves the best performance, reaching nearly 70% win rate, On the other hand, fails capture meaningful suboptimal actions, thereby limiting the effectiveness
setting K of exploration. Conversely, larger value, such as K = 3. may introduce excessive variance and destabilize learning. These findings indicate that a moderate number of candidate sub-networks achieves a favorable trade-off between diversity and stability, maximizing the effectiveness of S2Q.
Softmax temperature T: The temperature T regulates the sharpness of the Softmax distribution P.. Fig. ()) shows the performance for T E (0.01, 0.1, 0.2, 1.0). The results indicate that T = 0.1 yields the best balance between convergence speed and final win rate. A very small temperature (T' = 0.01) produces overly deterministic behavior early in training, limiting exploration and slow- ing progress toward the optimum. In contrast, a high temperature (T 1.0) promotes excessive exploration, delaying convergence and occasionally leading to suboptimal plateaus. Interestingly, T = 0.2 also performs competitively, suggesting that S2Q is relatively robust to moderate changes in temperature, as long as sampling remains focused enough to prioritize promising sub-values.
LIMITATIONS
While S2Q shows strong performance across diverse MARL environments, using multiple sub-value functions increases computation and memory requirements. However, our analysis in Appendix HL.5 shows that this overhead is moderate relative to the performance gains, making the trade-off favor- able in most practical settings where faster convergence and higher final performance are critical. In addition, S2Q relies on Softmax-based selection strategy with a temperature parameter controlling the perration-eplliiaation balance. Nevertheless, we find that S2Q is not highly sensitive to this parameter, and a reasonable value can be chosen with minimal tuning effort, allowing S2Q remain practical and easily deployable across wide range of environments.
7 CONCLUSION
In this work, we addressed the fundamental challenge posed by monotonic value decomposition in MARL and introduced S2Q, a novel framework based on successive Q-learning. By sequentially learning multiple sub-value functions, S2Q continuously tracks high-value alternative actions and rapidly adapts when the optimal joint action shifts, leading to more effective exploration and faster convergence. This design enables efficient coordination without requiring explicit communication or access to global state information. Extensive experiments on challenging MARL benchmarks, including SMAC and GRF, demonstrate that S2Q consistently avoids suboptimal solutions induced by monotonicity constraints and achieves substantial improvements over state-of-the the-art methods. These results position S2Q as a practical and general framework for advancing cooperative MARI..
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported partly by the Institute of Information & Communications Technology Plan- ning Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-11220466 Development of Core Technologies for Task-oriented Reinforcement Learning for Commercializa- tion of Autonomous Drones), (No. 2S-2025-25442822 AI Star-Fellowship Program (UNIST)), and (No. RS-2020-11201 136, Artificial Intelligence Graduate School Support (UNIST)), and partly by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2025--3523199 LLM-Based Multi-Agent Reinforcement Learning for End-to-End Large Autonomous Swarm Control).
ETHICS STATEMENT
This work focuses on algorithmic improvements in cooperative multi-agent reinforcement learning (MARL) under the CTDE paradigm. Our study uses publicly available simulation environments, including SMAC (Samvelyan
al. 2019), Google Research Football (Kurach et al.) 2020), and SMACv2 (Ellis et al. 2023), which do not involve human subjects, personal data, sensitive in- formation. Our work is motivated by cooperative and coordination tasks, and does not explicitly encourage or evaluate harmful uses. We adhere to the ICLR Code of Ethics and affirm that all re- search was conducted alignment with principles of fairness, transparency, and academic integrity.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide detailed description of S2Q in Section 4.3 with implemen-
tation details in Appendix C, Benchmarks including SMAC (Samvelyan et al. 2019) GRF (Kurach et al. 2020), and SMACv2 (Ellis et al. (023) are publicly available, with descriptions and code links Appendix E An anonymized code repository containing the full implementation is submitted as
suppiementary material where configurations are detailed in Appendix F Finally, considered base- lines and their repositories listed Appendix F.3 Together, these resources collectively enable independent reproduction of our results reported in this paper.
REFERENCES
Fouad Bahrpeyma and Dirk Reichelt. A review of the applications of multi-agent reinforcement learning in smart factories. Frontiers in Robotics and AI, 9:1027340, 2022.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 0118810.12888 12894, 2018.
Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò. Multi-agent reinforcement learning: A review of challenges and applica- tions. Applied Sciences, 11(11):4948, 2021.
Jingdi Chen, Hanhan Zhou, Yongsheng Mei, Carlee Joe-Wong, Gina C Adam, Nathaniel Bastian, and Tian Lan. Rgmdt: Return-gap-minimizing decision tree extraction in non-euclidean metric space. Advances in Neural Information Processing Systems,
2024.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on machine learning, pp. 1548-1546. PMLR, 2019.
Yue Deng, Yan Yu, Weiyu Ma, Zirui Wang, Wenhui Zhu, Jian Zhao, and Yin Zhang. Smac-hard: Enabling mixed opponent strategy script and self-play smac. arXiv preprint 241. 2024.
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi- agent reinforcement learning. Advances in Neural Information Processing Systems, 36:37567- 37593, 2023.
11

---

Published as a conference paper at ICLR 2026
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances neural information processing systems, 29, 2016.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Cong Guan, Feng Chen, Lei Yuan, Chenghe Wang, Hao Yin, Zongzhang Zhang, and Yang Yu. Efficient multi-agent communication via self-supervised information aggregation. Advances in Neural Information Processing Systems, 15200-11033,, 2022.
Seungyul Han and Youngchul Sung. Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration. In International conference on machine learning, pp. 4018-4029. PMLR, 2021a.
Seungyul Han and Youngchul Sung. A max-min entropy framework for reinforcement learning. Advances in Neural Information Processing Systems, 34:25732-2517 2021b.
Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih-wei Liao. Rethinking the imple- mentation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2102.03479, 2021.
Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Learning multi-agent communication from graph modeling perspective. arXiv preprint arXiv:2405.085 085500 2024.
Tianmeng Hu, Biao Luo, Chunhua Yang, and Tingwen Huang. Mo-mix: Multi-objective multi-agent cooperative decision-making with deep reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45((0): 2023.
Shariq Iqbal and Fei Sha. Actor-attentionn-cittec for multi-agent reinforcement learning. In Interna- tional conference on machine learning, pp. 2961-2970. PMLR, 2019.
Yonghyeon Jo, Sunwoo Lee, Junghyuk Yeom, and Seungyul Han. Fox: Formation-aware explo- ration in multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38. pp. 12985-12994, 2024.
Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michat Zajkac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 4501 45100, 2020.
Sunwoo Lee, Jaebak Hwang, Yonghyeon Jo, and Seungyul Han. Wolfpack adversarial attack for robust multi-agent reinforcement learning. arXiv preprint arXiv:2502.02844 2025.
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Cel- ebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information 2021.
Processing Systems, 34:3991-4000
Chuming Li, Jie Liu, Yinmin Zhang, Yuhong Wei, Yazhe Niu, Yaodong Yang. Yu Liu, and Wanli Ouyang. Ace: Cooperative multi-agent q-learning with bidirectional deiiinedeeeddnccy. In Pro- ceedings of the AAAI conference artificial intelligence, volume 37, pp. 8536-8544 2023a.
Huiqun Li, Hanhan Zhou, Yifei Zou, Dongxiao Yu, and Tian Lan. Concaveq: Non-monotonic value function factorization via concave representations in deep mueti- reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17461-1746) 2024.
Pengyi Li, Jianye Hao, Hongyao Tang, Yan Zheng, and Xian Fu. Race: improve multi-agent rein- forcement learning with representation asymmetry and collaborative evolution. In International Conference Machine Learning, pp. 19490-19503, PMI.R, 2023b.
12

---

Published as a conference paper at ICLR 2026
Tianxu Li, Kun Zhu, Nguyen Cong Luong, Dusit Niyato, Qihui Wu, Yang Zhang, and Bing Chen. Applications of mulli-agen reinforcement learning in future internet: A comprehensive survey. IEEE Communications Surveys & Tutorials, (4()::240-1229, 2022.
Xinran Li and Jun Zhang. Context-aware communication for multi-agent reinforcement learning.
arXiv preprint rrXiv22312.15000 2023.
Shunyu Liu, Yihe Zhou, Jie Song. Tongya Zheng, Kaixuan Chen, Tongtian Zhu, Zunlei Feng, and Mingli Song. Contrastive identity-aware learning for multi-agent value decomposition. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11595-11603, 2023a.
Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agen game abstraction via graph attention neural network. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 721 1211-72 2020.
Zichuan Liu, Yuanyang Zhu, and Chunlin Chen. 2 q: Neural attention additive model for inter- pretable multi-agent q-learinng. In International Conference on Machine Learning, pp. 22539- 22558, PMLR, 2023b.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed pooperatve--competin environments. Advances in neural informa- tion processing systems, 30, 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in neural information processing systems, 32, 2019.
Navid Naderializadeh, Fan H Hung, Sean Soleyman, and Deepak Khosla. Graph convolutional value decomposition in multi-agent reinforcement learning. arXiv preprini 004.000000.04440,, 2020.
Frans A Oliehoek, Christopher Amato, al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
James Orr and Ayan Dutta. Multi-agent deep reinforcement learning for multi-robot applications: A survey, Sensors, 23(7):3625, 2023.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexande Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gra- dients. Advances in Neural Information Processing Systems, 1221822--22221 2021.
Haoyuan Qin, Chennan Ma, Deng Deng, Zhengzhu Liu, Songzhu Mei, Xinwang Liu, Cheng Wang, and Siqi Shen. The dormant neuron phenomenon in multi-agent reinforcement learning value factorization. Advances in Neural Information Processing Systems, 37:35727-35759, 2024.
Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang, Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. Rmix: Learning risk-sensitive policies for cooperative reinforcement learning agents. Advances in Neural Information Processing Systems, 14:23049-23066 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi multi-agent reinforcement learning. Journal of Machine Learning Research, 21
2020a.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi agent reinforcement learning. Advances in neural information processing systems, 11200199-1211,,, 2020b.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv:
2019.
Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv: 2016.
Jianzhun Shao, Hongchang Zhang, Yun Qu, Chang Liu, Shuncheng He, Yuhang Jiang, and Xi- angyang Ji. Complementary attention for multi-agent reinforcement learning. In International conference on machine learning, pp. 30776-30793, PMI.R 2023.
13

---

Published as a conference paper at ICLR 2026
Siqi Shen, Mengwei Qiu, Jun Liu, Weiquan Liu, Yongquan Fu, Xinwang Liu, and Cheng Wang. Resq: A residual q tunction-based approach for multi-agent reinforcement learning value factor- ization. Advances in Neural Information Processing Systems, 35:5471-54833 2022.
Siqi Shen, Chennan Ma, Chao Li, Weiquan Liu, Yongquan Fu, Songzhu Mei, Xinwang Liu, and Cheng Wang. Riskq: risk-sensitive multi-agent reinforcement learning value factorization. Ad- vances Neural Information Processing Systems, 36:34791-34825, 2023.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- fional conference machine learning, pp. 5887-5896. PMLR, 2019.
Jianyu Su, Stephen Adams, and Peter Beling. Value-decomposition multi-agent actor-critics. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11352-11360, 2021.
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropaga- tion. Advances in neural information processing systems, 29, 2016.
Chuxiong Sun, Zehua Zang, Jiabao Li, Jiangmeng Li, Xiao Xu, Rui Wang, and Changwen Zheng. T2mac: Targeted and trusted multi-agent communication through selective engagement and evidence-driven integration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15154-15163, 2024.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning, arXiv preprint arXiv: 770.002996, 2017.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schul- man, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances neural information processing systems, 30, 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350-354 2019.
Lipeng Wan, Zeyang Liu, Xingyu Chen, Han Wang, and Xuguang Lan. Greedy-basee value representation for optimal coordination in multi-agent reinforcement learning. arXiv preprint arX14:2112.00444. 2021.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-larniing arXiv preprint arXiv:2008. 01062, 2020a.
Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decompos- able value functions via communication minimization. arXiv preprint arXiv:1910005366, 2019.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003.08039, 2020b.
Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng. Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020c.
Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang, Dop: Off-policy multi-agent decomposed policy gradients. In International conference on learning representa- tions, 2020d.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang- Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Infor- mation Processing Systems, 35: 012116609-15222 2022.
Yaodong Yang, Jianye Hao, Ben Liao, Kun Shao, Guangyong Chen, Wulong Liu, and Hongyao Tang. Qatten: A general framework for cooperative multiagent reinforcement learning. arXiv preprint arXiv:2002.0339: 03939, 2020.
14

---

Published as a conference paper at ICLR 2026
Yaodong Yang, Guangyong Chen, Jianye Hao, and Pheng-Ann Heng. Sample-effciiet multia- gent reinforcement learning with reset replay. In Ruslan Salakhutdinov, Zico Kolter, Kather- ine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Pro- ceedings of the 41st International Conference on Machine Learning, volume 235 of Proceed- ings of Machine Learning Research, pp. 55961-55975. PMLR, 21-27 Jul 2024. URL https: //proceedings. mix.
mlr. www.s................... prress/v230 Dtt../////////33/a22222..2 html
Lei Yuan, Jianhao Wang, Fuxiang Zhang. Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang. Multi-agent incentive communication via decentralized teammate modeling. In
Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 1466-9477, 2022.
Lei Yuan, Ziqian Zhang, Ke Xue, Hao Yin. Feng Chen, Cong Guan, Lihe Li. Chao Qian, and Yang Yu. Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attack- ers. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 11753-
11762, 2023.
Yifan Zang, Jinmin He, Kai Li, Haobo Fu, Qiang Fu, Junliang Xing, and Jian Cheng. Automatic grouping for efficient cooperative multi-agent reinforcement learning. Advances in neural infor- mation processing systems, 36:461 451-4121, 2023.
Xianghua Zeng, Hao Peng, and Angsheng Li. Effective and stable role-based multi-agent collabo- ration by structural information principles. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11772-117888 2023.
Tianhao Zhang. Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entroppy multi-agent reinforcement learning. In International conference on machine learning. pp. 12491-12500, PMLR, 2021.
Hanhan Zhou, Tian Lan, and Vaneet Aggarwal. Pac: Assisted value factorization with counterfactual predictions in multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35:15757-/ 2022.
Hanhan Zhou, Tian Lan, and Vaneet Aggarwal. Value functions factorization with latent state in- formation sharing in decentralized multi-agent policy gradients. IEEE Transactions Emerging Topics in Computational Intelligence, 7(5):3351-136 2023.
Tianyu Zhu, Xinli Shi, Xiangping Xu, Jie Gui, and Jinde Cao. Hypercomm: Hypergraph-based communication in multi-agent reinforcement learning. Neural Networks, 178:106432, 2024.
15

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
In preparing this paper, we made limited use of large language models (LLMs) exclusively for pol- ishing grammar and improving the clarity of our writing. No part of the research process, including problem formulation, method design, experimentation, analysis, or interpretation, was conducted using LLMs. Their role was strictly restricted to editorial assistance, ensuring the presentation of our work is clear and readable.
B MISSING PROOF
Theorem B.1 (Successive Q-leariing)- Let (s), ai) and {Q K k=t be the joint action-value and equation] 2 respectively,
function and sub-value functions obtained by minimizing equation [ and let (a0.1..... ",0,,...,,,K,, ..., denote the K+1 successive suboptimal joint actions of Q" for given 8t and Ts at timestep U.S.A. S.I.
Q" (St: Tr, 2... 2.. If the reward function r is bounded and suppression.facee factor is sufficiently large, then,
ak,) T,, 2 (st, Tr, ar), Va, (a ak,h.
a arg max Q (st, a.) 4k E {O,..., K).
(B.1)
Proof. We consider the minimization of the joint value loss equation ! together with the proposed sub-value losses equation 2 At convergence, the Bellman optimality equation implies that Q" satis-
fies Yt T2 + 1Q Q. Using this relation, each sub-value function Q satisfies
Q (s): a,) max (*t Tr, at), C) if a, Ak-1, (s): (T.:
@i (Sts Tr, ax)
otherwise,
for every (st, Te, a4) pairs.
For given (st, Te) pair, if the suppression factor satisfies ('e Tı, Tr, (s1, Ti, ak,
à. E max(Q" T,, a,),)) 2 20,
(B.2)
then Quab To, āz) Qaab (st, Tr, ake) Vā, E Ak-1,t-
(e, max(Q" (5c: Te, Tr, C)
s
(5 Te, a(,
From the definition of ah Quth(s TI, ā.)
(8 Tz)
<
(T,
Q (st, Tr, ai.)) Vā, E A\ so we can conclude
a arg max Q (st, Tı, ar).
(B.3)
Thus, if we take the supremum over all state observaisn histories and the maximum over all k e O,..., K. the suppression factor must satisfy
(st, Tr, ā.) - Q" T at.)
a M sup max max S1.7i ke (0, .K A.E.A
20
max(Q"(<x Te, āz), C)
and this supremum is finite because the bounded-reward assumption implies that Q* also bounded, which ensures that equation B.3 holds for all (st, T:) pairs and for all k E (0..... K). This completes the proof of the theorem.
C IMPLEMENTATION DETAILS
In this section, we detail the implementation of the proposed S2Q method. First, we present the encoder-decoder architecture described in Section 4.3 together with its associated loss function in Appendix C.T. Then, we introduce the parameterization for the practical implementation of S2Q and the corresponding total loss function in Appendix C.2 Furthermore, details on re-defined weight- ing function Wi. which enables S2Q to more effectively shift Q towards the optimal policy, are provided in Appendix C.3
16

---

Published as a conference paper at ICLR 2026
C.I TRAINING THE ENCODER- DECDDDE STRUCTURE
As introduced in Section 4.3 exact computation of Pt requires access to the global state s. There- fore, we proposed the approximation of P. based on encoder-decoder architecture. For practical implementation, we parameterize the encoder-decoder structure with neural network parameter yi and represent it Eg and Dv. The encoder Es utilizes an rnn layer, and maps the concatena- tion of all agents' local histories of into latent representation 2t (TE).) Then, the decoder Dy reconstructs both the global state and an approximate distribution (Śt. P.) = (z)))). Then, the encoder-decoder architecture is trained to approximate p based on a cross-entropy (CE) loss CE(PK pk), while the reconstruction mean square error (MSE) loss MSE(s, 8) encourages 2 to capture essential state information, thereby improving the prediction accuracy of p The overall loss of the encoder-decoder architecture Liaten( (1) defined as:
Liatem (1) CB(PK PK) MSH(s,
(C.1)
C.2 PRACTICAL IMPLEMENTATION OF S2Q
For practical implementation of S2Q, we parameterize both the monotonic sub-value functions " ..... h and the unrestricted value function Q' using neural network parameter 8. As de- scribed in Section [.2] WQMIX (Rashid et al. 2020b) trains the unrestricted value function Q" via temporal-differente (TD) loss in equation which we rewrite for the parameteriszed Q-function
Lq- (0) ((s (st,,,,, T,, y yt = +0 2/+1), (C.2)
where Tr (T) (Tt....,,7)) 7 is the joint history, Q the target network, and at arg max Q' the individual target action. Then, WQMIX further updates Q"x toward the target ye by adaptively weighting the TD eгrоr using w(st, at), defined as w(s,, at) = 1 if Q" Tr, a,) y/ and w(s;, at) = we otherwise, thereby prioritizing updates on actions that are underestimated.
Following this design, we train Qi. and then redefine the mean squared errors of the sub-value functions equation 2 as the successive loss Ssuccessive (6), expressed as:
Lanccessive (0)
Be a.) wo((s. an) (Q (8t,, ar)
k=
(v+ alaas E Ak-1.) (7,
(C.3)
where Wi is weighting function for each & inspired by WQMIX, which we detail in Appendix C.3 Il is the indicator function, B is the replay buffer, and Ar,t = (a ... a with A_ denotes the set of previously identified suboptimal actions. The factor a controls how strongly the values of actions in A4-1. 1. suppressed. Combined with the encoder-decoder loss Ltatent (()) introduced in Appendix C. the final loss function of $2Q is derived as:
Ls2o(0, v) = Lsuccessive (0) + Lq- (0) + Liasent (w)
(C.4)
To further stabilize optimal action selection, we adopt simple sampling strategy that balances sta- bility and exploration. Specifically, with probability p = 0.5. the sub-network index is fixed to = for the entire episode, ensuring consistent learning of the global optimum. Otherwise, & is sampled at each timestep from P,, promote exploration of suboptimal actions. This simple yet effective mechanism provides both reliable convergence toward the optimum and sufficient exploration diver- sity. Algorithm C.1 summarizes the full S2Q learning process.
17

---

Published as a conference paper at ICLR 2026
Algorithm C.1 Successive Sub-value Q-learning (S2Q) Framework
1: Initialize unrestricted value network Q3 and its target network Ug. sub-value networks
(Qa 00 encoder-decoder (Fip, Dτ), and replay buffer B
2: for each training iteration do
3: With probability p = 0.5, fix sub-value index h = 0 for the entire episode
4:
for each environment step / do
5:
training then
6:
Encode trajectories т; into latent variable 2, E (T.)
7:
Decode 2 to obtain categorical distribution and reconstructed state: Pt, & = (z) if episode is fixed to & 0 then
8:
9:
Set =
else
11:
Sample k ~ P.
end if
13:
Select joint action a e-greedy(Q
else
15:
Select joint action a greedy(Q)))
end if
17:
Execute joint action a,, observe reward Tr and next state 8 Store transition (8t, Te; Tt: 81+1) in replay buffer
19:
end for
20:
Compute TD target y = r (Q
21:
Update [Quth (0.kkk= and Q7 via Lsuccessive and LQ- in equations C.3
C.2
22:
Update encoder and decoder (F,, D) via Liatem in equation C.1
23:
end for
24:
Periodically update target network Q7
C.3 WEIGHITING FUNCTION IN WEIGITTE TD LEARNING
As explained in Section 2.2 WQMIX updates Q toward the target y/ by adaptively weighting the TD error using w(s,, a.). Building WQMIX, S2Q introduces weighting function wk. defined
1,
if Q* Tr, at) 2 maxaçEAR: Q7 Te, a;), ==
(. as)
1, if Quuu(((s Tr, ax) < Yt ail(a; E Ab-1a) Ving (St, Te, ar) k k= l,...,, K
otherwise.
(C.5)
Our design of wi motivated by the following rationale. For k 0, the weighting rule ensures that optimality consistently propagated into Q == Q allowing S2Q to rely solely on Q during evaluation without requiring communication. For k > 1, the rule closely resembles the WQMIX weighting scheme, but instead of directly comparing against the TD target yt. it suppresses the values of previously identified suboptimal actions before applying the comparison, thereby enabling the successive extraction of alternative high-value actions. In all other cases, the factor we acts as down-weightee weighting term to moderate updates. A sensitivity analysis of w, provided Appendix H.2
18

---

Published as a conference paper at ICLR 2026
D SOFTMAX-BASED EXPLORATION BEHAVIOR OF S2Q
Heatmap Joint Action Distribution (%)
Hcotmop Joint Action Distribution (%)
Heatmap of joint Action Distribution
100
62.73
1.63
1.23
00.97
1.43
93.
1.64
1.57
Agent Actiors at
Agent Actioon el
50
Percomare
1.48
23.19
0.55
1:59
10:97
0.24
1,60
0.03
0.03
Agent
0.55
8:50
1.43
1:54
1.60
02
0.03
C
A
Agant 2 Actions a²
Agent 2 Actions a²
Agint Artions a²
(a)
(c)
Figure D.1: Heatmap joint action distribution under three settings: (a) S2Q (K 2, 1.0), (b) S2Q (K 2, = 0.5), (c) per-agent e-greedy
In this section, we analyze the exploration behavior of S2Q compared to conventional per-agent e-greedy strategies. To visualize the exploration behavior of each method, we revisit the 2-agent, 3- action matrix game introduced in Section 4. where the optimal joint action is (A, A) with value 8, and the suboptimal actions (B, B) and (C.C) have values and respectively. Fig. D.1 illustrates the joint action probabilities under three different settings: (a) S2Q with K = 2 and 7 = 1, (b) S2Q with
2 and T 0.5, and (c) conventional per-agent e-greedy exploration.
Figure D. demonstrates that S2Q successfully identifies the optimal action while still executing
suboptimal actions with meaningful frequency, even at modest temperature of 0.5. At higher temperature of T = 1, the optimal action remains the most frequeetly executed, while exploration across suboptimal actions maximized. In contrast, conventional per-agent e-greedy exploration concentrates almost exclusively conventional methods that rely on a single optimal action and independent <-gredyy sampling: they struggle to adapt when the value function changes dynamically. By contrast, S2Q adapts smoothly to changes in the optimal action, based on its prioritized Softmax-based exploration over tracked suboptimal actions. This broader exploration provides richer training signals for Q", facilitating faster convergence toward globally optimal solutions.
the optimal action (A, A). This highlights key limitation of
19

---

Published as a conference paper at ICLR 2026
E ENVIRONMENT DETAILS
In this section, we provide detailed description of the environments and scenarios used for evalu- ating the proposed S2Q. We consider two main benchmarks: the StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al. 2019) and the Google Research Football (GRF) environment (Kurach) et al. 2020). As mentioned in Section [ we further categorize SMAC scenarios based on their complexity and communication requirements. Specifically, we distinguish between SMAC-Hard+, which presents challenging coordination tasks with a larger number of units and complex combat strategies, and SMAC-Comm, which emphasizes tasks requiring explicit inter-agent communication for effective execution. Appendices E.I
E.2 E.3
provide detailed descriptions of the SMAC-Hards
GRF, and RMAC-Comin scenarios discussed in Section 5 In Appendices E.4 and E.5 we detail SMACv2 (Ellis et al. 2023) and SMAC-Hard (Mixture Opponent) (Deng al. conduct additional experiments to further demonstrate the effectiveness of S2Q. The performance results for SMACv2 and SMAC-Hard (Mixture Opponent) are provided in Appendices
2024). where
G. and G.2
respectively.
E.1
THE STARCRAFT MULTI-AGENT CHALLENGE (SMAC)-HARD+
Ally
Enemy
Ally
Ally
Enemy
(a) 5m_vs_6m
(b) MMM2
(c) 27m
Enemy
Enemy
Ally
Ally
Ally
Enemy
(d) Corridor
(c) 6h vs 8z
(1) 3s52 vs 3s6z
Figure E.1: Visualizations of SMAC-Hard+ scenarios
SMAC (Samvelyan et al., 2019) is widely used benchmark designed to test cooperative Multi- Agent Reinforcement Learning (MARL) algorithms in complex, decentralized settings. Built on top of the StarCraft II game engine, SMAC presents a series of tactical combat scenarios, where team of Al-controlled allies faces off against enemies run by a built-in script. Each scenario differs in terms of terrain, unit types, and strategic difficulty, requiring agents master sophisticated combat tactics such as focus fire, kiting, and exploiting environmental features. A match concludes either the time limit expires. We provide visualizations of SMACC-Hrd++
when one side is climinated scenarios in Fig. E. and summarize the details of each considered episodes in Table E. followed by a detailed description of the SMAC benchmark.
State space: Global state aggregates detailed information from all entities on the battlefield. For allied units, this includes their positions, health, cooldowns, shields if applicable, and unit types. Enemy data is similar, except it omits the cooldown stats. In addition, the most recent action taken by each agent is recorded as a one-hot encoded vector.
Observation space: In SMAC, agents' observations are restricted to allies and enemies within sight range of 9 units. Specifically, an agent's observation vector is composed of four distinct seg- ments. First, movement capabilities encoded across four directions (up, down, left, right). Sec- ond, data about visible enemies includes their relative positions, distances, health, shield values, unit type, and whether they currently be targeted. Third, ally-related information mirrors the enemy format but excludes the agent itself. Finally, self-features reflect the observing agent's 's own condition:its current health, shield level (if any), and unit classification.
20

---

Published as a conference paper at ICLR 2026
Action space: Agents in SMAC operate using set of discrete actions. These include movement in the four primary directions (north, south, east, west), the ability to attack enemies within a range of 6 units, and special abilities limited certain units such Medivacs, Agents also issue stop command or perform a no-op action, though the latter reserved for units that have been eliminated.
Reward function: SMAC incorporates shaped reward function composed of three main elements: damage inflicted on enemy units, elimination of those units, and overall victory in the scenario. The reward is formally defined as:
R Σ
AHealth(e)
Σ @Enemiee
I(Health(e) =0) -Reaaddeaah I[win)) Reward (E.1)
#Genemies
Here, (Halthh(e) represents the decrease health enemy unit during a given timestep, and IT(-)
is an indicator function. Rewarddeath and Reward are set to 10 and 200, respectively.
Table E.1: Detailed descriptions of SMAC-Hard+ scenarios
Map
Ally Units
Enemy Units
State Dimension
Obs Dimension
Num. Actions
5m 6m
5 Marines
6 Marines
98
55
12
1 Medivac,
IMedivac,
MMM2
2 Marauders,
3 Marauders,
322
176
18
7 Marines
8 Marines
27m_vs.30m
27 Marines
30 Marines
1170
285
36
Corridor
6 Zealots
24 Zerglings
282
156
30
Bs52.vs..35 3s62
Stalkers,
Stalkers, 6 Zealots
230
136
15
5 Zealots
6b_ys_87
6 Hydralisks
8 Zcalots
140
78
14
E.2 GOOGLE RESEARCH FOOTBALL (GRF)
Ally
Opponent
Ball
(a) academy_33 vs.2
(b) academy _4_v__3
Figure E.2: Visualizations of GRF scenarios
Kurach et al. (2020) offers a multi-agent soccer environment where each player is controlled by an autonomous agent. The game models realistic ball physics, player motion, and interaction mechanics such tackling and passing. Teams must coordinate to achieve scoring opportunities while competing against opponents driven by scripted behaviors. From GRF scenarios, we consider academy 3.vv vs_i_wi w-wwiiiittt keeper r and academy 4_vv 2.with www.wwihheeeeeee keeper scenarios, which abbreviate as academy 33vv..2.. 2 and academy 4 4_vs_3 for brevity. Fig. E2 illustrates the initial positions of the entities on the field, while Table E.2 summarizes the considered scenarios.
State space: The global state s consists of all player positions and velocities, as well as ball position and velocity. The data for both ally and opposing teams to same.
21

---

Published as a conference paper at ICLR 2026
Observation space: The observation for agent includes local information about the ego player, nearby teammates, opponents, and ball-related features, all expressed relative to the agent's att' current frame.
Action space: The discrete action space of GRF covers movement in eight directions, sliding, pass- ing, shooting, sprinting, and standing still, all of which necessary in order to achieve scoring opportunities.
Reward function GRF provides two primary reward settings: Scoring and Checkpoint. Scoring function rewards agents with a +1 reward for scoring a goal and a -1 penalty for conceding one. While the Checkpoint function provides additional intermideate rewards. For example agents may receive rewards for successful passes or defensive actions. In our experiments, we follow the more sparse Scoring function, for more challenging scenarios.
Table E.2: Detailed description of GRF scenarios
Scenario
Ally
Opponent
State Dim
Obs Dim
Action Dim
goalkeeper,
academy.3.vs.2 central midfield
26
26
19
1 center back
goalkeeper, 34
academy.4.v 4.vssm central midfield
34
19
2 center back
E.3 SMAC-COMM
(a) 1o. 1o.2r_vs.4r
(b) 1o_10b_vs.lr
(c) Sz.vs_lul
(d) bane_vs.hM
Figure E.3: Visualizations SMAC-Comm scenarios
SMAC-Comm shares the same state, observation, action space, and reward function as the previ- ously introduced SMAC-Hard+, but its tasks are specifically designed to emphasize communica- tion. We illustrate the scenarios from SMAC-Comm considered in our experiments in Fig. E.3 and summarize them as Table
Table E.3: Detailed description of SMAC-Comm scenarios
Map
Ally Units
Enemy Units
State Dimension
Obs Dimension
Num. Actions
lo 2r vs 4r
1 Overseer,
4 Reapers
68
49
10
2 Roaches
1o_10b_vs.Ir
Overseer,
Roach
148
84
7
10 Banclings
1u
5 Zealots
Ultralisk
63
35
7
bane vs hM
3 Banelings
Hydralisk,
52
35
8
Medivac
22

---

Published as a conference paper at ICLR 2026
E.4 SMACV2
vs.5
(b) zerg vs.
(c) protoss 5 vs
Figure E,4: Visualizations of SMACv2 scenarios
SMACv2 (Ellis et al. 2023) extends the original SMAC Samvelyan et al. 2019) benchmark to pro- vide a more rigorous testbed for evaluating generalizable cooperative multi-agent learning. While the state, observation, and action spaces, well as the reward function, remain similar SMAC, several key modifications distinguish SMACv2. First, instead of fixed map configurations, SMACv2 introduces randomized initializations (e.g., unit positions, health, numbers, and attributes), which prevent agents from overfitting to static scenarios. Second, it emphasizes greater unit diversity and asymmetric matchups, requiring more sophisticated tactical coordination among agents. Third, the difficulty scaling mechanism is refined: rather than varying only the number or placement of en- emy units, environmental factors are also randomized, creating a natural train-test distribution gap. In our experiments, we consider the terran ..n.... vs 5, zerg 5 vs 5, protoss 5 vs 5 scenar- ios. Visualizations for each scenario are provided in Fig. and the scenario-specific details are summarized in Table E.4
Table E.4: Detailed description of SMACv2 scenarios. Probabilities indicate the sampling distribu- tion for randomized compositions.
Map
Ally Units (prob.)
Enemy Units (prob.)
State Dim
Obs Dim
Num. Actions
Marine (0.45)
Marine (0.45)
1erran_5_vv.3
Marauder (0.45)
Marauder (0.45)
120
82
Medivac (0.1)
Medivac (0.1)
Zergling (0.45)
Zergling (0.45)
5_v__
Hydralisk (0.45)
Hydralist (0.45)
120
82
11
Baneling 10.
Baneling(O.1)
Zealot (0.45)
Zcalot (0.45)
5_vs_5
Stalker (0.45)
Stalker (0.45)
130
92
11
Colossus (0.1)
Colossus (0.1)
E.5 SMAC-HARD (MIXTURE OPPONENT)
SMAC-Hard (Mixture Opponent) (Deng et al. 2024 shares the same state, observation, action space, and reward function the previously introduced SMAC-Hard+, but incorporates additional stochasticity by providing multiple opponent strategies. To distinguish it from the tasks from orig- inal SMAC-Hard+, we attach Hard to each scenario to denote tasks from SMAC-Hard (Mixture Opponent). We summarize the scenarios we consider in our experiments as Table E.3
Table E.5: Detailed description of SMAC-Hard (Mixture Opponent) scenarios
Map
Ally Units
Enemy Units
State Dimension
Obs Dimension
Num. Actions
Hard
2 Stalkers,
Stalkers Zealots
120
80
11
Zealots
3s. vs.vs.52. Hard
3 Stalkers,
Zealots
68
48
11
5m_vs.6m
Hard Hard
Marines
Marines
98
12
23

---

Published as a conference paper at ICLR 2026
F EXPERIMENTAL DETAILS
In this section, provide the experimental details of the proposed S2Q. All experiments provided in this paper are conducted on a GPU server equipped with an NVIDIA GeForce RTX 3090 GPU and Intel Xeon Gold 6348 (2.60GHz, 28 cores) processor, running Ubuntu 20.04. In Appendix F.1 we introduce popular CTDE baselines, including VDN, QMIX. and QPLEX. While we follow the original implementations and loss scaling of the prior baseline CTDE algorithms for shared hyper- parameters, we focus our parameter search on the hyperparameters newly introduced in S2Q. This hyperparameter setup is provided in Appendix F.2 Finally, Appendix F3 details various MARL. methods that address the limitations of monotonic value decomposition, well as communication- focused MARL algorithms.
F.1 DETAILS OF CONSIDERED CTDE BASELINES
VDN (Sunehag al., 2017) is a cooperative MARL method based on value factorization, decom- posing the joint action-valus function into sum of individual agent value functions, enabling cen- tralized training with decentralized execution. The global Q-value is defined as:
(st, Σ (T
where individual Q' depends only on the agent's own trajectory o/ and the chosen action of.
flexible, non-linear function for combining individ- ual agent utilities into joint action-value. To ensure that maximizing each agent's local Q' aligns with maximizing the global objective, QMIX enforces monotonicity constraint between the joint value and the individual utilities, formalized as:
QMIX (Rashid et al. 2020a) introduces
aQ aQ Vi,
QPLEX (Wang et al. 2020a) advances the expressiveness of value factorization by introducing du- plex dueling architecture while still satisfying the IGM Silviidull-Gooal--aaa property. QPLEX decomposes the Q-values into advantage functions and formulates IGM as:
arg max, Al(rt, (a)),
arg max (TO(TT
arg M A (TN .aa)
where a and A' represent the joint and individual advantage functions, respectively. The joint Q-value is then constructed
N (T) Σ (T, (A) -
with the weighting factors (T a) generated via multi-head attention mechanism, enhancing the flexibility of the model.
Throughout our experiments, results for VDN, QMIX, and QPLEX are based the implementa- tions provided in PyMARL2 (Hu et al. 2021). an soun-s MARL framework that includes im- plementations of diverse algorithms along with various improvements, such TD(A), larger batch sizes, and the Adam optimizer. The code is available at: hattps://githnn com/ hi j pymar12
24

---

Published as a conference paper at ICLR 2026
F.2 HYPERPARAMETER SETUP
In our experiments, while we adopt the default hyperparameter configuration from PyMARL2 Hu al., 2021) for settings shared across CTDE methods, we foeus identifying effective values for the hyperparameters introduced in S2Q. The best parameters for each scenario are summarized in Table F2 According to Table F.2, across variety of tasks, K consistently achieve the best performance. While the weighting factor w exhibits some task-specific variation, values of 0.75 or 0.9 generally perform well. These results demonstrate the robustness and flexibility of S2Q, achieving high win rates and rapid convergence across diverse scenarios.
T 0.1, and
1.0
Table F.1: Common Q-learning Hyperparameters
Hyperparameter
Value
Decay Value
1.0 - 0.05
Anneal Time
100000
Target Update Interval
200
Discount Factor
0.99
Buffer Size
5000
Batch Size
128
Learning Rate
0.001
Optimizer
Adam
Optimizer Alpha
0.99
Optimizer Eps
le-5
Gradient Clip Norm
10.0
Num GRU Layers
I
RNN Hidden State Dim
64
Double
True
Table F.2: Scenario-specifie hyperparameter setup of S2Q
Scenario
T
Cx
W
SMAC-Hard+
5m_vs_6m
0.1
1.0
0.9
MMM2
2
0.1
1.0
0.9
27m_vs.3Cm
0.1
1.0
0.9
corridor
2
0.1
1.0
0.75
6h.vs.82
0.1
1.0
0.9
3s52_v3_3s62
2
0.1
1.0
0.75
GRF
academy_3 _3__s__________
0.1
1.0
0.75
scademy-4.ve
0.1
1.0
0.75
SMAC-Comm
1o 2x vs 4r 1o 10b vs
2
0.1
1.0
0.9
0.1
1.0
5z_vs_lul
2
0.1
1.0
0.9
bane-vs.hM
0.1
1.0
0.9
SMACv2
terran.5.vs.5
2
0.1
1.0
0.75
zerg.5.vs.5
2
0.1
1.0
0.75
grotoss_5.vs.5
0.1
1.0
0.75
SMAC-Hard (Mixture Opponent)
5m_vs_6m_Hard
2
0.1
1.0
0.9
2s3z.Hard
0.1
1.0
0.9
2..v-522Harr
2
0.1
1.0
0.9
25

---

Published as a conference paper at ICLR 2026
F.3 DESCRIPTION OF OTHER MARL METHODS FOR COMPARISON
QMIX (Rashid et al. 2020a): introduces mixing network that combines per-agent value functions into a joint action-value under a monotonicity constraint, ensuring consistency between centralized training and decentralized execution. We base our implementation of QMIX on the following repos- itory: htt.h:
WQMIX (Rashid et al. 2020b); extends QMIX by introducing weighted projections in the mixing process, alleviating the representational limitations of monotonicity and enabling more accurate learning of optimal joint action-values. We base our implementation of WQMIX on the following repository: httpss://// / com/c.o//.. (kzzz/pymarl2
RiskQ (Shen et al., 2023): introduces a quantile-basee value factorization that models joint re- turn distributions as weighted mixtures of per-agent utilities, satisfying the risk-sensitive IGM prin- ciple and enabling coordination under uncertainty. The official code can be found at: https:
7/github. com/ xmu r1- 1a11-3dv/RB 3dv/Riakk
PAC (Zhou et al. 2022): leverages counterfactual predictions of optimal joint actions to provide
assistive information value factorization, using a novel counterfactual loss and variational en- coding to improve coordination under partial observability. The official code can be found at: https://github.. htw.:/uub...cm///aaaaaaaaeee
MARL
FOP Zhang et al. 2021): factorizes the optimal joint policy in mexximmm-entropp MARL into indi- vidual actor-critic policies, with theoretical guarantees convergence to the global optimum. The official code can be found at: hhttps://gttii
sea ovefilel
DOP WWngg al.) (020d): integrates value function decomposition into multi-agent actor-critic
methods, enabling efficient off-policy learning while addressing credit assignment and centralized- decentralized mismatch, with guarantees of convergence. The official code can be found https:
//githab.com/Tonggiaaaaaat htt////oonnnnnn//o/oo/a
com/
MARR (Yang et al.) 2024): improves sample efficiency in MARL by introducing reset strat- egy and data augmentation, enabling high-replay training in parallel environments with fewer en- vironment interactions. The official code can be found at: htttps:/ hwt.:/u..oo.c//aaaaaaeeeeeeeeeee cww.CNDOT///////////
ICML24-MARR
MASIA Guan et al. 2022): enables efficient multi-agent communication by aggregat- ing received messages into compact, task-relevant representations using permutation-
invariant encoder and self-supervised objectives, improving coordination and decision-
making. The official code can be found at: https://github. com/ chenf-ai/
Multi AgenttCoomuun Commumiicaiionnnnnoo Contiiinnonssddrinng Representtttttttr-aaaaaiiiin hrt.:pooooo..toieeeeeeeeeeeeeoea
NDQ Waag et al. 2019): combines value function factorization with communication minimiza- tion, enabling agents act independently most of time while selectively exchanging messages using information regularizers to improve coordination. The official code can be found at:
https://github. wwt...ooooo..aagggggooooooo NDQ
MAIC (Yuan et al. 2022): enables agents to generate targeted incentive messages that directly influence teammates' value functions, promoting efficient explicit coordination while remaining compatible with different value function factorization methods. The official code can be found at:
https://github.com
ctt. http::///gtthubb..commaasssssss
CAMA (Shao al. 2023): complementary attention enhance high-contribution enti-
ties and compress con-conribbttion ones, addressing distracted attention and limited observabil- ity, thereby improving coordination in cooperative MARL. The official code can be found at:
https://githuu: cwwpuub..ooothuuruttee
T2MAC (Sun al. 2024): enables agents communicate selectively with trusted partners and integrate information at the evidence level, improving coordination and communication efficiency in com/ ZangZehua/
cooperative MARL. The official code can be found at:
T2MAC
26

---

Published as a conference paper at ICLR 2026
G ADDITIONAL EXPERIMENT RESULTS
In this section, we present additional experiment results to further demonstrate the effectiveness of our proposed S2Q method. Appendice G. G..2 we provide additional performance evaluation on SMACv2 (Ellis al. 2023 and SMAC-Hard (Mixture Opponent) (Deng et al. 2024), while extend S2Q to other representative CTDE baselines, VDN and QPLEX, in Appendix G.3
G.1 PERFORMANCE COMPARISON IN SMACV2
120
GM4
Test Win Rate (9%)
Test Winn Rate (%)
WOO
Timesteps (M)
Timesteps SMACv2: 2erg_5_vs_5
Timesteps (M)
SMAC22: terran____nn_ _____
SMACv2: protoss_5_v ___oos____v__5______________5
Figure G.I: Performance comparison: Average test win rates in the SMACv2 tasks
From Fig. which shows the performance in SMACv2 scenarios, we can observe that S2Q demonstrates superior performance over the MARL baselines across all scenarios, achieving both superior performance and faster convergence. Notably, its advantage is most pronounced in the zerq-5_vs.5 and arotoss-__ve varying ally and enemy team compositions makes the environment particularly challenging. In this setting, S2Q demonstrates its ability to effectively track the values of suboptimal actions, thereby enabling efficient exploration and guiding the Qio towards the optimal policy.
scenarios, where the high degree of stochasticity induced by
G.2 PERFORMANCE COMPARISON IN SMAC-HARE (MIXTURE OPPONENT)
Test Win Rate (9)
Test Win Rate (96)
10
Timesteps (M)
Timesteps (M)
Timesteps (M)
SMAC-Hard: 3s_ vs_ Hard
SMAC-Hard: Hard
SMAC-Hard: So--Hard:Sm_ve Seairdd:5______6m 6m_ Hard
Figure G.2: Performance comparison: Average test win rates in the SMAC-Hard (Mixture Opponent) tasks
Fig. illustrates the performance on SMAC-Hard (Mixture Opponent) scenarios, including Bmm_v_6m_Harre 2s3z_Hard, and 3s_vs_5z_Hard, As mentioned in Appendix E.3 we append "Hard" to each scenario name to distinguish from the the scenarios from the SMAC-Hard+ bench- mark. According to the results, S2Q consistently outperforms all MARL baselines across every considered task, demonstrating its superiority. Notably, in the 35_vs.52. agents must coordinate precise hit-and-run maneuvers against opponents that exhibit diverse tacti- cal behaviors, S2Q demonstrates outstanding adaptability. This suggests that S2Q captures subtle opponent-condittoned value structures that conventional MARL baselines fail exploit.
scenario, where
27

---

Published as a conference paper at ICLR 2026
G.3 GENERALITY OF S2Q ACROSS CTDE METHODS
Although our primary discussion of S2Q has been in the context of WQMIX, the proposed proce- dure is general and readily extends other CTDE methods, such VDN (Sunehag et al. 2017) and QPLEX (Wang et al., 2020a). By replacing the joint action-value function Q with Qo S2Q can be integrated into these coloppdccomposition baselines without modification. Across all scenarios, we observed significant performance improvements over the corresponding baselines. Consistent with the results obtained in WQMIX, the performance gains were most evident in the challenging 6h_vs_8 environment, which requires extensive exploration. These results demonstrate the practi- cality and robustness of S2Q, highlighting its ability to follow changes the value landscape more closely and adapt to new optima faster than conventional approaches.
Table G.1: Performance comparison SMACv2 environments
Scenario
VDN
VDN+S2Q
QPLEX
QPLEX+S2Q
5m_vs.6m
62.47 = 11.12
68.25 = 4.78
58.18 = 3.52
61.86 + 2.65
MMM2
0.00
14.12 + 2.43
50.78 10.63
=8.03-8.227
27m. 77.vv... 30m
13.75 5.33
61.35 + 2.33
47.34 + 8.26
63.18
Corridor
56.25 26.14
70.49 + 13.37
48.62 + 24.02
64.68 + 18.51
6h_vs_82
8,94 4.31
48.03 9.37
3.44 1.13
41.32 6.23
3s5z vs 3s6z
47.66 + 19.66
49.82 = 10.31
43.19 + 17.02
48.73 - 10.46
28

---

Published as a conference paper at ICLR 2026
H ADDITIONAL ABLATION STUDY
In this section, we provide a deeper analysis on S2Q. In Appendix H.1 we provide further analysis on the hyperparameters' K and T' in additional tasks, while in Appendix H.2 provide additional ab- lation study on the suppression constant a and weighting factor We. Furthermore, in Appendix [H.) we analyze the role of auxiliary signals in scenarios where communication pendix H.4 provide additional examples of shift in value function along with trajectory analysis in SMAC-Comm environment. Finally, Appendix H.S presents computational complexity analysis on S2Q, to show that the computational overhead from leveraging multiple sub-value functions remains moderate, compared to the significant performance gain of S2Q.
critical, and Ap-
H.1 HYPERPARAMETER ANALYSIS IN ADDITIONAL TASKS
100
Test (1)
Kid
Jest Winn Ralte- (86)
T-d.21
TH6.2
Best Vitn Dulte (80
Cod
Best Mitn Raite (90
Tud
Timesteps (M]
Timestops (M)
Timestaps (M)
Timestaps (M]
SMAC Harde: MM/M2
Hardh
Hande: MMMM2
GRP: academy 4 vs
GRF: academy v.
Figure H.1: Hyperparameter analysis. From left right: (1) effect of K in MMM, (2) effect of temperature in MMM2, (3) effect in academyy444 - 4.v...v.. 3.,,,, and (4) effect of T in academy -_vs.
To further emphasize the generality of S2Q beyond the settings examined in Section 5.3 addi- tionally evaluate the effects of the number sub-value networks K and the Softmax temperature T in two more challenging environments: MMM2 scenario from SMAC-Hard+ and academy 4 scenario from GRF
Number of sub-value functions K: As in earlier results, K = 2 achieves the best overall perfor- mance in both MMM2 and academy_4 _4_vs_3, offering the strongest balance between convergence speed and final win rate. In contrast, K 0 fails to provide meaningful exploratory diversity, while larger values such as K = 3 introduce increased variance and mild instability. These observations further support that moderate number of candidate sub-networks yields favorable trade-off be- tween diversity and stability.
Softmax temperature T:: moderate temperature (e.g., T 0.1) provides the most effective balance between focused sampling and sufficient exploration. Very low temperature values (T 0.01) result in overly deterministic behavior early training, suppressing useful exploration, while higher temperatures (e.g.. T 1.0) promote excessive exploration and slow convergence, Notably, = 0.2 performs competitively, indicating that S2Q remains robust to reasonable variations in temperature as long sampling remains sufficiently concentrated promising sub-values.
Across both environments, we observe trends that closely mirror those reported earlier for 6h-vs_8z, demonstrating that the behavior of S2Q is consistent and robust actoss domains. Over- all, the additional experiments on SMAC-Hard+ and GRF confirm that S2Q maintains stable and consistent performance characteristies across diverse and challenging multi-agent environments, inforcing its generality and robustness.
29

---

Published as a conference paper at ICLR 2026
H.2
ABLATION STUDY ON ADDITIONAL HYPERPARAMETERS
(OURSI
0.9 (OURS)
Test Win Rate (%)
0=0.1
4;
0.5
lest Win Rate
--2.0
40-
0. 05
1.5
15 20
25
30 35
4. 43 50
3.5 0.0
1. 2.0
Timesteps (M)
Timesteps (M)
SMAC-Hard+: 6h_vs_8z
SMAC-Hard+: 6h_vs__6_
(h)
Figure H.2: Hyperparameter analysis. Effect of (a) suppression constant a and (b) weight factor we-
Suppression constant Fig.
evaluates the effect of the suppression constant
[0.1,0.5, 1.0, 1.5, 2.0). which determines how strongly the values of previously identified actions reduced and thereby controls how far sampled suboptimal actions lie from the current optimum. The results indicate that a = 1.0 yields the best performance: this setting encourages exploration of meaningful, moderately distant sub-actions that are informative for updating the value landscape. A smaller value, = 0.5, also performs well since it biases sampling toward nearby sub-actions that remain relevant the optimum. In contrast, very small value such suppress the previously selected optimal actions, preventing the discovery of meaningful suboptimal actions and resulting
0.1 fails to sufficiently
significant degradation in performance. On the other hand, larger values (a = 1.5,, 2.0) substantially hurt performance: excessive suppression forces sampling of actions that are very distant from the optimum, producing updates that are less informative (and often mislead- ing) for tracking the true optimal policy, which destabilizes learning and slows convergence. These findings suggest that a moderate suppression level is necessary to balance exploration breadth with the relevance of sampled sub-actions.
Weighting factor we: Fig. H.2 b) reports the effect of varying the weighting factor we. which de- termines how TD-error is scaled by reinforeing corrective signals that reduce underestimation. The results show that w. 0.9 yields the highest win rates, striking a balance hetween effective error correction and stable value learning. slightly smaller value, w. 0.75, also performs well, indi- cating robustness to moderate relaxations. However, more aggressive reductions such as = 0.5 and w. 0.25 significantly weaken the propagation of TD-error, leading to information loss and degraded performance. These findings highlight that maintaining relatively high ensures that informative TD signals are consistently transmitted, while still guiding the value functions towards the optimal values.
30

---

Published as a conference paper at ICLR 2026
H.3
EFFECT OF AUXILIARY SUPERVISION
136
S20
Test WWin Rate (96)
Te.st Win (%)
VIJHX
DMIX
SI50
S20 CORN
80
(MIX-COMM
WOJHI.COOM
RI50 CEIMA
30
05
1.0
63
05 3.0 15 20 25
4.0
50 C5 15 15 24 40 5.0
Timesteps (M)
Timesteps (M)
Timesteps (M)
SMAC-Hards: MMM2
SMAC-Hards: 6e
GRF: academy 4_
Figure H.3: Performance comparison: Average test win rates in the SMAC-Hard+ and GRF task for MARL algorithms and their communication variant
To demonstrate that the performance gains S2Q arise from the proposed successive valvvalue learning, rather than the auxiliary latent information, we conduct additional experiments on SMAC- Hard+ scenarios, including MMM2 and ch_vs_8z) and academy .4. 1_vs_3 scenario from GRE In these environments, we equipped all non-communication baselines with the same encoder-decoder architecture used S2Q-Comm, and denote these variants by appending "-Comm". This setup en- sures that every algorithm receives identical latent signal derived from reconstructing the global state, thereby isolating the effect of the auxiliary representations from the effect of S2Q's ' successive sub-value learning.
The results, presented in H.3. H.33show that injecting these latent- auxiliary signals has minimal influence on performance success. In some cases, performance even degrades due to the latent introducing additional noise. Similar trends appear in the GRF academy -__s_3 environment, where communication is not the primary bottleneck. Notably, S2Q-Comm consistently outperforms all baselines, including their ". Comm" variants. These findings confirm that the advantage of S2Q-Comm does not originate from the auxiliary state-reconstruction signal or the latent variable itself. Instead, the gains stem from S2Q's core mechanisms.
the considered tasks where communication not critical for task
H.4
SHIFT VALUE FUNCTION AND ADDITIONAL TRAJECTORY ANALYSIS
Exploration (1M)
Shift
Optimal (10M)
06
ag
ag
aj
aj
aj
Shift
a2
42
a2
(ak44)
(K()))
(aa
Pk
Q'(a
(.(%)
$30.9
6.2 6.3
22.
60.0
7.5 7.2
93.6
4.4
6.1
4.2
4.7
a6
aj
az
ag
aj
a2
an a
a2
ag
a
42
a
aj
a2
ag aj a
Figure H.4: Value function shift in Sm_vs_6m_Hard)
In addition to the motivation discussed in Section 4.1 and the trajectory analysis presented in Section 5.2 we provide further examples of shifts in the value function. Specifically, we ex- amine value-function changes in SMAC-Hard (Mixture Opponent) 5m.vs-6m_Hard and GRF scademy-4_vv... with results depicted Fig. and and H.5 In 5m_vs_6m_Hase agents initially focus greedy attacks to secure easy eliminations, but the value function subsequently learns that repositioning to form favorable formations yields higher long-term returns, causing the optimal ac- tion shift from attack to move.
Similarly, in academy vs 3, agents first attempt low-success long shots, but eventually discover that approaching the goal passing increases scoring probability, shifting the optimal action from
31

---

Published as a conference paper at ICLR 2026
Exploration (2M)
Shift
Optimal (5M)
a²
a
4g
4n
K
a1
d1
a2
Shift
Q' (Aic30
(h())
(k
@'
p
1.9 1.7
A89.1
48
0.83 0.84 0.96
62.0
0.24 0.
30.
0.15
9.3
ag aj
a
ag ai
Go ai 02
ao aj
a
ao
aj a
ao aj
a
Figure H.5: Value function shift in academy 4 vs 3
shoot to pass. These examples further highlight S2Q's capability to track and adapt to evolving optimal behaviors across diverse environments.
Ally
Enemy
Move
Hit
Ally
Move
Enemy
Move
Hit
Move
(a)
Exploration (0.3M)
Shift
Optimal (2M)
Tesil 1Vt axli
V0h 741
(((4))
(Gia))
(((a))
24
10
7.0
78.2
10
Gg
0g a
Gg di
(b)
(C)
Figure H.6: Training behavior of S2Q. Left: training step 0.3M. Right: training step = 2M
In addition, we further investigate the training dynamics of S2Q (K = 2), we analyze the 5z lul scenario from SMAC-Comm, where the agents must learn coordinate based on the latent information 2. 6h vs 8z. Unlike scenarios that reward aggressive engagement, the optimal strategy in $z_vs_lul is to lead to heavy losses due to the enemy's superior strength. As shown in Fig H. the action distri- bution reveals that agents initially favor attack, reflecting local optimum that prioritizes immediate engagement. Consequently, the attack ratio increases in the early stages of training. However, as training progresses, S2Q enables agents to recognize the risks of premature aggression, resulting in gradual decline in attack frequency and a corresponding rise in the use of move. Fig(H.6| illustrates how this shift is accompanied by improvements in survival time and overall win rate.
and group with the distant ally, while direct attack actions often
Fig. H.6 c) highlights how S2Q adapts to this changing notion of optimality. Early in training, attack tracked as the dominant action (a),), while move considered suboptimal. As the value landscape evolves, Q* reassigns higher returns move. prompting S2Q to increase its execution frequency through Softmax-basee behavior policy. This dynamic reallocation allows Q to align with the true optimal strategy, thereby reducing reliance local optima. These findings emphasize that even in environments requiring implicit coordination and communication, such as effectively leverages its successive sub-valu component to explore, track, and ultimately converge to the optimal policy.
S2Q
32

---

Published as a conference paper at ICLR 2026
H.S COMPUTATIONAL COMPLEXITY
S2Q introduces additional computational overhead due to its use of multiple sub-value functions and the encoder-decoder architecture for coordinating k-selection. To evaluate this overhead and demonstrate that it remains minimal relative to performance benefits, we compare S2Q and QMIX in controlled experiments on an NVIDIA GeForce RTX 3090 GPU and an Intel Xeon Gold 6348 (2.60GHz, 28 cores) processor running Ubuntu 20.04 with Py PyTocch. In MMM2 and 27m vs 30m scenarios where QMIX achieves competitive performance, we measure the time required to reach 50% win rate. Table H.T summarizes the results. According to Table H.I S2Q incurs only moderate increase computation, approximately 8.4% for MMM2 and 3.5% for 2 2mm.vs300m More importantly due to faster learning enabled by successive sub-value function tracking, S2Q reaches the target win rate of 50% significantly faster than QMIX, demonstrating that the modest overhead is outweighed by the improved convergence speed and overall performance.
Table H.I: Comparison of training time between S2Q (for different K) and QMIX.
Scenario
Metric
S2Q (K=1)
S2Q (K=2)
S2Q (K=3)
QMIX
Time 1M
1 122.0 min
129.3 min
136.5 min
119.3 min
MMM2
at 50% win
1.48M
1,28M
1.67M
2.16M
Time at 50% win
180.6 min
165.5 min
228.0 min
257.7 min
Time 1M
241.0 min
245.1 min
258.0 min
236.8 min
27m_vs_30m
T at 50% win
1.39M
1.15M
1.57M
1.86M
Time at 50% win
335.0 min
281.9 min
405.1 min
440.4 min
In addition to time-wise comparison, also evaluate GPU memory consumption quantify the overhead introduced by maintaining multiple sub-value networks. Table H.2 reports peak GPÚ us- age for S2Q with different values of K and for QMIX. According the results, memory usage increases moderately with K due to the additional sub-networks, but the overall overhead remains reasonable: in MMM2, S2Q with K 2 requires only 248 MB more memory than QMIX. and even K = 3 remains within a 19% increase. A similar pattern holds in the more computationally de- manding 27m_vs 3mm scenario, where S2Q(K 2) uses 336 MB more memory than QMIX. The incremental cost from K 2 to K 3 is also modest, considering the added representational capacity. Importantly, this memory overhead scales predictably and linearly with K.
Combined with the substantial reduction in training time reported in Table H.] these results demon- strate that S2Q achieves significant improvements in learning efficiency while incurring only limited computational and memory overhead. This confirms that the successive sub-value learning frame- work offers a favorable trade-off between memory usage and performance gains, even in large-scale SMAC scenarios.
Table H.2: Comparison of memory usage between S2Q (for different K) and QMIX.
Scenario
Metric
S2Q (K=1)
S2Q (K=2)
S2Q (K=3)
QMIX
MMM2
Perk GPU usage (MB)
1967.0 MB
2087.0 MB
2188.0 MB
1839.0 MB
2 7m vs 30m
Perk GPU usage (MB)
3665.0 MB
3761.0 MB
4076.0 MB
3425.0 MB
33


Reviews:
[{"review_id": "RghlyVWgIY", "reviewer": "Reviewer_nTYN", "summary": "This paper proposes Successive Sub-value Q-learning (S2Q), a framework that successively learns multiple subvalue functions to retain information about alternative high-value actions. By incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables Qtot to adjust quickly when the optimal action changes. Experimental results show that S2Q outperforms other recent MARL methods on the StarCraft II Multi-Agent Challenges and Google Research Football.", "questions": "Figure 1 demonstrates changes in the payoff matrix. Could you provide concrete examples of how the value function dynamically evolves in SMAC-Hard or GRF environments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Dynamic value functions\n\n    S2Q overcomes the limitation that conventional methods do not explicitly track suboptimal actions. When the optimal action changes, S2Q can immediately leverage the corresponding sub-value function and guide Q^{tot} to adapt. \n\n- Introducing communication during training\n\n    S2Q explicitly executes tracked suboptimal actions with priority determined by a Softmax distribution P_t over their Q^{∗} values, thereby enabling exploration of a wider range of spaces than conventional ϵ-greedy exploration.", "weaknesses": "- Old Benchmarks \n\n    The StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) is an old benchmark. It is advised to report the experimental results on the recently proposed SMAC-Hard benchmark [1].\n\n    [1] SMAC-Hard: Enabling Mixed Opponent Strategy Script and Self-play on SMAC, arXiv:2412.17707.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HyLE3eNef9", "reviewer": "Reviewer_3XJX", "summary": "The paper proposes Successive Sub-value Q-learning (S2Q) for cooperative MARL under CTDE. Instead of committing to a single estimated-optimal joint action, S2Q learns a sequence of monotonic, value-decomposition subnetworks that each focuses on a different high-value mode by suppressing actions selected by earlier subnetworks in its TD objective. To synchronize the index k during training, the authors use an encoder–decoder that reconstructs the global state and approximates the Softmax distribution. Experiments on SMAC Hard+, GRF, SMAC-Comm, and SMACv2 report higher win rates and faster convergence than strong baselines. Ablations suggest the Softmax selection and successive learning are key. Compute overhead is reported as modest while reaching 50% win rate substantially faster than QMIX.", "questions": "1. Can you formalize guarantees (even under simplified assumptions) that successive suppression + weighting (B.5) yields distinct top-k action sets, or at least that $Q_0^{sub}$ is not harmed by the auxiliary subnetworks? A proposition about bias introduced by the suppression term would be valuable.\n\n2. What happens if you remove communication during training and have agents sample k independently, or conversely provide baselines with a matched auxiliary (state reconstruction)? Please include a “no-comm” S2Q variant and, if possible, an oracle $P_t$ variant to bound gains.\n\n3. You show $K=2,T=0.1$ works broadly (Tab. E.2). Could you include per-map sweeps (or at least MMM2 and academy 4v3) to show consistency and to better justify the choice of $K$?\n\n4. Since the paper cites “MARL as sequence modeling”, could you comment on (or include) a representative sequence-model baseline and discuss compatibility?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The toy matrix-game illustrates why retaining information about nearby high-value actions can help when the optimum shifts. S2Q operationalizes this via a suppressed TD objective and Softmax-guided behavior policy. The algorithmic presentation (Alg. 1; eqs. (B.2–B.5)) is easy to follow. \n\n2. Results span SMAC Hard+, GRF, SMAC-Comm (with a “-Comm” variant), and SMACv2, showing consistent gains and faster learning, not only final win rates. The compute table quantifies overhead.\n\n3. Removing Softmax selection or randomizing $k$ materially hurts performance. Analysis of $K$ and temperature $T$ is informative. Sensitivity of suppression $\\alpha$ and weighting $w_c$ appears in the appendix.\n\n4. Default evaluation needs no communication relying on $Q_0^{sub}$, which is attractive compared to methods that require message passing at inference.", "weaknesses": "1. The paper claims theoretical/empirical analyses, but no formal result is provided to justify that minimizing the modified TD with the suppression term reliably extracts distinct top-k modes under the IGM constraint or preserves contraction/stability properties. A small lemma would strengthen the case.\n\n2. S2Q learns an encoder–decoder to approximate $P_t$ and reconstruct $s$, which provides additional supervision such as cross-entropy and reconstruction. Several non-communication baselines do not leverage comparable auxiliary signals, raising comparability questions. A variant without the encoder–decoder, or with matched auxiliaries for baselines would calibrate the gain attributable to successive sub-values and auxiliary training.\n\n3. While $K$ and $T$ are studied, they’re analyzed mainly on 6h vs 8z. It would be helpful to report these sweeps across more tasks (e.g., MMM2, GRF academy 4v3) and include a “no-comm” ablation that samples $k$ independently to show the necessity of synchronization, and an oracle $P_t$ using the true Softmax to bound performance. \n\n4. In Eq. (2)/App. B.2, the target for previously selected actions is reduced by $\\alpha Q^*$, which can change the scale and sign of TD targets. There is no analysis of potential instability or bias this introduces, particularly with function approximation. Clarifying normalization or clipping strategies, and why the scheme does not collapse would help. In addition, this method shares the similar idea with [1], it would be better to compare with this baseline.\n\n[1] Wan, L., Liu, Z., Chen, X., Wang, H., & Lan, X. (2021). Greedy-based value representation for optimal coordination in multi-agent reinforcement learning. arXiv preprint arXiv:2112.04454.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5yHyhOTrVe", "reviewer": "Reviewer_xg2N", "summary": "This paper proposes a method to overcome value function shifts during training in CTDE MARL, named S2Q. The proposed method learns a set of several sub-value functions, where each aims to identify different suboptimal actions.", "questions": "1. Could the authors describe another more practical scenario aside from the provided matrix games where the optimal action shift might happen? For example in the experimented environments such as SMAC, when could such shift happen?\n2. i find interesting the sudden boost around 1M timesteps in figure 6 for the task bane_vs_hM; have the authors explored why this is seen only in this specific environment, instead of a more linearly increasing curve?\n3. how can you find the best k? is there any guarantees (theoretical maybe) that for higher k the variance and learning instability will increase (as mentioned in lines 454-455)?\n4. considering the sub value functions are learned in a sequential manner, how does the proposed method relate to the higher order q-learning approach presented in [1]?\n\n[1] https://arxiv.org/abs/2304.13383", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well written and provides a very extensive set of experiments and ablations. The results are consistently strong across very diverse environments. While apparently simple, i find clever the idea to surpress the optimal actions in the calculations of subsequent value functions and the performances show big improvements in a range of tasks.", "weaknesses": "The authors could have provided a deeper analysis of the scalability of the proposed method, since it requires sequential computations using sub-networks. I.e, since there is a mixing for each Q, up until what point can k scale? In the communication encoder-decoder module in figure 3, the authors could have provided a better description of the architecture of these modules. \n\nPlease find below some more specific questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o48FEII87c", "reviewer": "Reviewer_44v8", "summary": "This paper proposes S2Q, a new framework for cooperative MARL under CTDE settings. This paper retains and learns from suboptimal actions so that the system can better adapt when the optimal joint policy changes during the training. Instead of relying on single best action, S2Q successively trains multiple sub-value functions that each caputure distinct high-value joint actions. By combining them through softmax-based behaviour policy, the method encourages persistent exploration and rapid adjustment to new optima. Experiments on SMAC, SMAC-Comm, and GRF shows outperformance to baseline algorithm.", "questions": "Based on the weaknesses described above, I have some questions for the authors.\n\n1. Could the authors provide a more formal justification or theoretical insight into why maintaining multiple sub-value functions helps track shifting optima more effectively?\n2. Are there any guarantees on the convergence or stability of S2Q, particularly when sub-value suppression is applied iteratively? A sketch of theoretical grounding could greatly strengthen the contribution.\n3. Could the authors provide a direct comparison or at least a discussion to clarify whether S2Q’s performance gains come from retaining suboptimal modes or simply from having additional representational capacity?\n4. The paper mentions that the computational overhead is “moderate.” Could the authors quantify this or training time ratio relative to QMIX or WQMIX?\n5. How does the overhead scale with the number of sub-value networks K and the number of agents N? This would help assess the method’s practicality in larger-scale MARL tasks.\n6. Have the authors considered testing in deliberately non-stationary environments, such as mixed multiple opponent strategies in SMAC-Hard environment,to validate S2Q’s adaptability?\n7. In SMAC-Comm results, how much improvement arises from the encoder–decoder communication mechanism versus the S2Q framework itself?\n8. The paper positions S2Q as addressing the issue of “shifting optima.” Could the authors better relate this to other known issues such as non-stationarity, policy co-adaptation, or value landscape drift in MARL? Clarifying this link may help highlight the conceptual novelty and general applicability of S2Q.\n\nIf all the concerns are well-addressed, I would consider raising the score. Thank you.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality: The paper presents an original idea of retaining suboptimal actions through the proposed S2Q framework. This introduces a new perspective in value decomposition MARL by modeling multiple high-value modes instead of a single optimal action, addressing a long-standing issue of adaptability under shifting optima.\n2. Quality: The technical formulation is solid and well-motivated, supported by a clear derivation of objectives and loss functions. The experimental setup is comprehensive, evaluating the method on multiple challenging benchmarks and performing ablations to validate each component.\n3. Clarity: The paper is clearly written, with well-organized sections, intuitive figures, and a motivating example (the payoff matrix) that effectively conveys the main idea. The methodology and algorithm are clearly explained and reproducible, with good visual aids to guide understanding.\n4. Significance: The contribution is significant for cooperative MARL. It addresses a key limitation in existing value decomposition approaches, poor adaptability when optimal actions shift. The framework’s generality and empirical robustness indicate potential impact on future MARL research.", "weaknesses": "1. Limited theoretical depth. While the paper provides a solid conceptual and empirical contribution, the theoretical analysis of why successive sub-value learning guarantees adaptability or improved convergence remains shallow. The paper lacks formal results on convergence properties, stability, or value approximation bounds for S2Q, which would strengthen its scientific aspect.\n2. Incremental novelty relative to prior frameworks. Although the idea of retaining suboptimal actions is interesting, the implementation may be viewed as an extension of WQMIX with auxiliary sub-networks and a Softmax exploration mechanism. The paper could better clarify how S2Q fundamentally differs from or improves upon multi-head or ensemble-based exploration methods beyond heuristic layering.\n3. Computational efficiency discussion. The paper acknowledges increased computation and memory overhead but only qualitatively. A quantitative analysis comparing GPU memory, or training wall-clock time to QMIX/WQMIX would make the practicality argument more convincing, especially for scaling to large agent teams.\n4. Clarity of communication variant. The role of the encoder–decoder communication module is somewhat underexplained. It is unclear how much of the improvement in SMAC-Comm tasks comes from the communication architecture itself versus the S2Q mechanism. A clearer disentanglement or comparison to standalone communication baselines would improve interpretability.\n5. Missing experiment details. Although the codebases are described in the Appendix individually, it seems that the performance in Figure 5 of the QMIX algorithm originated from pymarl2 does not achieve the performance stated in RIIT[1]. A detailed training parameters or any modifications are better to be discussed.\n\n[1] Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)