Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Observation Delays
Authors: Songchen Fu
Siang Chen
Shaojing Zhao
Letian Bai
Hong Liang
Ta Li
Yonghong Yan
Venue: NeurIPS 2025 poster ()
Keywords: Deep Reinforcement Learning
Signal Delay
Multi-agent System
Primary area: reinforcement_learning
TL;DR: This paper formalizes the problem of delayed observation in multi-agent systems and proposes a solution framework, achieving superior performance on classic benchmarks with delayed observation.

Abstract: In real-world multi-agent systems (MASs), observation delays are ubiquitous, preventing agents from making decisions based on the environment's true state. An individual agent's local observation typically comprises multiple components from other agents or dynamic entities within the environment. These discrete observation components with varying delay characteristics pose significant challenges for multi-agent reinforcement learning (MARL). In this paper, we first formulate the decentralized stochastic individual delay partially observable Markov decision process (DSID-POMDP) by extending the standard Dec-POMDP. We then propose the Rainbow Delay Compensation (RDC), a MARL training framework for addressing stochastic individual delays, along with recommended implementations for its constituent modules. We implement the DSID-POMDP's observation generation pattern using standard MARL benchmarks, including MPE and SMAC. Experiments demonstrate that baseline MARL methods suffer severe performance degradation under fixed and unfixed delays. The RDC-enhanced approach mitigates this issue, remarkably achieving ideal delay-free performance in certain delay scenarios while maintaining generalizability. Our work provides a novel perspective on multi-agent delayed observation problems and offers an effective solution framework. The source code is available at https://github.com/linkjoker1006/RDC-pymarl.

Rainbow Delay Compensation: A Multi-Agent Reinforcement Learning Framework for Mitigating Delayed Observation
Songchen Fu 1.2+, Siang Chen Shaoojnn Zhao! Letian Bai Hong Liang
Ta Li Li Yonghong Yan 1.2
l Laboratory of Speech and Intelligent Information Processing, Institute of Acoustics, CAS University of Chinese Academy of Sciences
3 Department of Electronic Engineering, Tsinghua University
fusongchen@hecl. ioa. en, lita@hecl. ioa. ac.
Abstract
In real-world multi-agent systems (MASs), observation delays are ubiquitous, preventing agents from making decisions based on the environment's true state. An individual agent's 's local observation typically comprises multiple components from other agents or dynamic entities within the environment. These discrete observation components with varying delay characteristics pose significant chal- lenges for multi-agent reinforcement learning (MARL).. In this paper, we first formulate the decentralized stochastic individual delay partially observable Markov decision process (DSID-POMDP) by extending the standard Dec-POMDP. We then propose the Rainbow Delay Compensation (RDC), a MARL training frame- work for addressing stochastic individual delays, along with recommended im- plementations for its constituent modules. We implement the DSID-POMDP's observation generation pattern using standard MARL benchmarks, including MPE and SMAC. Experiments demonstrate that baseline MARL methods suf- fer severe performance degradation under fixed and unfixed delays. The RDC- enhanced approach mitigates this issue, remarkably achieving ideal delay-free performance in certain delay scenarios while maintaining generalizability Our work provides novel perspective multi-agent delayed observation problems and offers an effective solution framework. The source code is available at htttg///giihubb.ccm//aiiooooese
Introduction
Multi- reinforcement learning (MARL) has been widely applied in various domains such as multiplayer games [28] [9], robot control [3 [4], agent communication [5, 40], and quantitative trading (37). However, beyond inherent challenges in MARL, including non-stationarity, partial ob- the observation delay problem has often been overlooked. From signal transmission in biological systems [II] to communication in large-scale swarms [29] delay issues are ubiquitous
servability, credit assignment, and the curse of dimensionality [38 12].
weal-world scenarios and typically have detrimental effects most systems. Due to the coupled influences of the environment, allied agents, and other agents (opponents or targets), multi-agent systems (MASs) exhibit more prevalent and complex observation delay situations than single-agent systems.
Early studies on system delay problems were primarily rooted control theory 23), where solu- tions relied heavily on fixed transition models-an assumption often violated in complex MASs (25
TThese authors contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The introduction of augmented state spaces 2 32 marked a pivotal shift, enabling reinforcement learning methods to handle deterministic delays through model-based state estimation these approaches advanced single-aged aagett systems, their extension to multi-agent settings remained su- perficial, typically limited to fixed delay scenarios 33 21 Recent progress in delayed-obserrvtiin Markov decision processes (DOMDPs) [44] formalized stochastic delay modeling, yet existing work concentrates overwhelmingly on single-agent domains. Multi-agent solutions [36, (35 established theoretical foundations and algorithmic innovations at the levels of communication and feedback. Yet, a critical gap remains: the fundamental challenge stochastic partial observability in MASs remains unresolved. This oversight is particularly significant given the inherent asynchrony and network-induced uncertainties world multi-agent applications.
While
In MARL, the evolving policies of other agents render the learning environment inherently non- stationary. Stochastic observation delays exacerbate this challenge, as agents must not only cope with non-stationarity but also account for the uncertainty introduced by varying delays. These delays also exacerbate the credit assignment problem: misaligned observations and rewards make it more challenging to associate actions with outcomes, thereby hindering effective policy optimization. Fundamentally, unfixed delays violate the Markov assumption by providing agents with inaccurate perceptions of the current state. Moreover, unfixed delays have greater impact than fixed ones. With fixed delays, agents can adapt and implicitly predict others' observations over time, forming a kind of cognitive inertia. However, under stochastic delays, such predictions become unreliable, making harder for actor networks to compensate. These facts all indicate that the problem of unfixed delayed observations in MARL is in urgent need of being addressed.
Previous studies have primarily focused on delay issues in single-agent systems, whereas research on MARL mainly consists of simple extensions of single-agent delay theories. However, delays in multi-agent environments are not constant, and different observation components may experience varying delays for each agent. Our work provides deeper understanding of delayed observation MASs and proposes general algorithmic framework to address the associated challenges. This means practitioners can tailor the framework by selecting and integrating suitable algorithms for different scenarios, achieving optimal performance. The main contributions of this paper are as follows:
We define the decentralized stochastic individual delay POMDP (DSID-POMDP), providing a universal mathematical model for MASs with delayed observation.
We propose the Rainbow Delay Compensation (RDC) training framework, which miti- gates the impact of delayed observation by reconstructing delay-free observation, utilizing curriculum learning, and leveraging knowledge distillation.
Based on DSID-POMDP, we innovatively introduce two compensator operation modes (Echo and Flash) and implement two models based on Transformer and Gated Recurrent Unit (GRU) networks.
We integrate two classic MARL algorithms, VDN 30 and QMIX 27 into the RDC framework. The proposed method is tested common MARL benchmarks, demon- strating significant performance improvements under fixed and unfixed delay conditions, approaching near delay-free performance levels.
2
Related Works
Recent studies have explored Deep Reinforcement Learning (DRL. approaches to address delay issues in single-agent systems. Walsh et al. [32] introduced the constant delay Markov decision process (CDMDP), which extends action sequences to incorporate fixed delays in observation and reward. However, the resulting state space expansion suffers from exponential growth, limiting the feasibility of pure state-based solutions. To overcome this, they proposed Model-Base Simulation (MBS) for discrete environments and Model Parameter Approximation (MPA) for continuous envi- ronments, pioneering model- based methods for delay-free state estimation. Firoiu et al. [9] employed environment prediction models to reduce performance degradation from fixed action delays in gaming scenarios. Bouteiller et al. 4 developed a partial trajectory resampling method to resolve credit assignment challenges in stochastic delay environments. Liotet et (20 implemented imitation learning to align delayed agents with expert action distributions, but only for fixed delays. Wang et al.

---

combined state augmentation and state prediction with delay-reconciled training for separate actor-critic optimization, demonstrating significant improvements in stochastic delay scenarios.
34
Significant progress has also been made in addressing the challenges of delay in MARL. Chen et al. [6] developed the Delay-Aware Multi-Agent Reinforcement Learning (DAMARL) framework, which mitigates fixed observation delays through centralized training with auxiliary information. Subsequent work by Yuan al. (36) introduced TimeNet to dynamically optimize agents' waiting time for delayed communications, thereby enhancing collaborative efficiency. For reward delay scenarios, Zhang et al. 39 proposed Delay-Adaptive Multi-Agent V-Learning (DAMAVL) with proven convergence under finite and infinite delay conditions. Practical applications have shown promise, as demonstrated by Liu et al. [21]'s 's successful implementation of DAMARL in cooperative adaptive cruise control (CACC) systems. While Wang et al. [33 advanced the field by predicting action effects through state prediction. Still, current approaches remain limited to fixed delay scenarios and typically overlook the asynchronicity of delayed observations. It is worth noting that research on robust MARL [[5] also aims to address the challenges of non-ideal observations. Unlike the observation lag caused by delays, this line of work focuses more inaccuracies in observations resulting from system errors adversarial attacks.
3 Preliminaries
3.1 Decentralized Partially Observable Markov Decision Process
decentralized partially observable Markov decision process (Dec-POMDPP a model designed for coordination and decision-making in MASs. In this framework, the POMDP introduces an " "observation" variable, enabling decision-makers to observe only a portion of the system state at each time step, building upon the original MDP. Dec-POMDP extends this concept to multi- agent scenarios by incorporating additional joint variables. can be formally defined as a 7-tuple (I, A, Z, P, R, O), where I represents finite set of agents. S denotes the set of state systems encompassing all possible environment states. AA and Z are the action and observation spaces for each agent, respectively, with A A, and Z Z, P, R, and O are defined as P(s' (s, a), R(r')s |s. a), and O(oss), representing the probabilities of transitioning to state s', receiving reward r', and obtaining observation after executing the joint action in state This formulation captures the decentralized nature of decision-making in MASs, where agents must coordinate their actions based on partial observation of the environment.
3.2 Decentralized Stochastic Individual Delay-Partially Observable Markov Decision Process
While delayed observation MDPs rep- resent special case of POMDPs, we maintain a clear distinction be- tween delayed and partial observation in this paper. In typical multi-agent environments, each agent's 's observa- tion comprises three components: (1) self-state information (typically with minimal delays due to internal trans- mission), (2) other agents' states (ob- tained through perception or com- munication), and (3) environmental states. Since delays in observing other agents and environmental states fol- low similar principles, we collectively term these observation sources as "en- tities" for clarity. The observation de- lays from other entities are typically positively correlated with their rela- tive distances. Inspired by this phenomenon, we can model the possible delay values of different entities in agent,'s 's observation multiple user-defined probability distributions, not just those
Agents
Agent,
Agent,
Di
Di
other-agent state abservations
observations
salf-statee
Or
Agent
cavironmest-rie observations
Rm
Ara
An
Farioy
Easity
Emilay
ASID-POMDE'S
Figure 1: A simple example of extended state and delayed observation in DSID-POMDP. The left side describes the components of agent, 's observation and annotates their de- lay value distributions. The matrix in the upper right corner shows the contents of ((1) in the extended state.
3

---

related to distance. Therefore, we propose the decentralized stochastic individual delay POMDP (DSID-POMDP).
Definition 1. DSID-POMDP =
Ro, Op) augments a Dec-POMDP =
(I,S, A, 2, R, O), such that
1. Ip = IUJ where J is the set environment entities,
2. X 8 where T' denotes the maximum possible delay value,
Ap A,
4. Zp = Z.
D D(agent,,, entity, x), iIjje ID.
6. Px(x'Ix xx a) =sss'|s, a) IT-= 0(5((-1) 5(-44))), ((((()++1) Ex, (x( R)s,a), Ob(z|x) 1liez IIiEIIIIET, Ei p(d = =))(( )000
The DSID-POMDP established under the condition that the information exposed by each entity in the system only contains information from itself. The new element, individual delay distribution D' represents the distribution of delay values for entityj in the observation of agent,. natural and
intuitive constraint is that the delay value de must satisfy the condition: di < min(d1-11 - 1,
To incorporate delayed states, the state of the DSID-POMDP is extended to include the delay-free state and the previous T states, that is, x = (-(T) ((T++)) .... S(1). s}. To maintain consistency in notation, we also represent the delay-free state as (()) In the observation function, p(d' t)
represents the probability that the delay value dd equals i, and s/ -t) represents the state information of entity, from the perspective of agent; in state ((z)- Figure provides more intuitive explanation of individual delay distributions and the observation function.
3.3 Classic MARL Algorithms
The operation of the RDC framework relies on baseline algorithms. The classification of MARL algorithms follows similar pattern to DRL, where they can be categorized into value-based and policy-based methods according to their underlying principles. After comprehensively considering algorithm performance and characteristics, we employ value-based algorithms in our framework, VDN (30 and QMIX 277, which demonstrate excellent performance in discrete action space tasks. VDN extends DQN 24 in straightforward manner by decomposing the team value function using a linear factorization approach. During training, VDN samples batches from the replay buffer and updates parameters by minimizing the TD error, with its loss function defined as:
(0) [B[[r max Q .,;00) 0-) Quoral (s, u, 0))
where 0- represents the parameters of the target network. The target network periodically copies (hard update) gradually weights (soft update) the parameters 0 of the evaluation network. És denotes the expectation over a finite batch of samples.
Building upon VDN, QMIX uses a mixing network to aggregate the local Q-values of individual agents into a centralized global Q-value, while satisfying the monotonicity constraint: This ensures consistency between the global and local Q-values. This modification endows QMIX with a parameterized critic, whereas VDN relies solely on a simple summation operation. This distinction serves to validate RDC's 's adaptability to different RL algorithm frameworks.
8Qual 2 0, Vi. aQ
4 Methods
In this section, we present the architectural details of RDC. As illustrated in Figure 2 the framework extends conventional MARL algorithms by incorporating four key components: a compensator module, 2) delay- reconciled critic, 3) the curriculum learning for actors, and 4) the policy knowledge distillation. The arrows different colors and styles represent the data flows various stages. Blue arrows belong to the teacher model, and red arrows belong to the student model. Solid arrows denote the training phase, while dashed arrows represent the inference phase. The data flows involved in the inference phase can be used during the training phase. While we describe several viable

---

delay-free
delay-free
Critic
Environment
Critie
delayed
delayed
obs
obs
Memory
Delay Filter
Memory
compensated obs
compensated obs
Compensator
eval
teacher student
Compensator
Teacher Agent
action distribution and value evaluation
Student Agent
Figure 2: The internal structure of the DA-MARL framework.
implementations and corresponding algorithms, these do not represent an exhaustive enumeration of possibilities within this framework. The framework maintains broad compatibility most mainstream actor-critic algorithms can seamlessly adapt to RDC. At the same time, other MARL approaches can typically be accommodated by selectively removing specific framework components. The compensator may employ any sequence intiictionccaaable architecture, and both the curriculum learning and knowledge distillation mechanisms can be customized by researchers based on particular policy networks and task requirements.
4.1 Observation Delay Occurrence and Compensation Process in Multi-agent System
Due to the mutually independent ac- tion delays among agents in MASs, and each agent's actions can influence the global state, the delay equivalence theorem (34] cannot hold. This paper focuses exclusively on delayed obser- vation and assumes that agents only transmit their own information exter- nally-an assumption consistent with most multi-agent environment config- urations in the research community. Therefore, from the viewpoint of a sin- gle agent, information updates from different entities are relatively inde- pendent of each other. This principle leads to the phenomenon where other parts of the observation do not belong to the same time step. Figure 3 illustrates simple example of delayed observation. The numbers indicate the time step from which each part of the agent's current observation originates smaller numbers correspond older information. The left side demonstrates the information updates in the observations of agent, over system clock steps 0 to 3. In contrast, the right side shows the corresponding compensation process of observations without delay. The observation from the second agent at step 3 still contains information from step O, requiring the most compensation steps for this portion.
Delay Occurrence
Delay Compensation
step
cmp step
step
step
3
2
Mtep
emp step
step
cmpp step
3-tep delny correggonds to 3-step comprnvation
Figure 3: A simplified example illustrating MASs" delay occurrence and compensation process.
4.2 Delay Compensator
Based on the occurrence process of observation delay, we intuitively design two modes of the delay compensator-Flatt and Echo-to reconstruct delay-free observation. Flash performs a simple compensation using available information and directly outputs the reconstructed result. It implicitly accounts for the issue of varying delays within observation through its internal model design. Echo, an autoregressive model, incrementally outputs the next-step information based on known data at each compensation step. Under the masking layer's 's control, the T-th output of Echo yields the final reconstructed observation. Figure (5) illlustrasss the workflows of the two compensators in parallel. Theoretically speaking, Flash offers faster reconstruction with lower resource consumption, making it suitable for scenarios with slight delay variations and high requirements for decision-making speed. Echo's s operation mode fully complies with the ideal delay compensation process shown in Figure 3 and can adapt to variable delay values and unknown delay patterns. We believe that as
5

---

agent policies iterate and update, the data distribution will change, potentially causing significant performance fluctuations in the compensators. Thus, both compensators are trained synchronously with reinforcement learning, meaning the RDC framework maintains a complete online training process.
Sufficient and effective input data enables the compensator achieve better performance. Based on the definition of DSID-POMDP, we naturally extend the current observation into an observation sequence that includes history ob- servations. Furthermore, drawing on existing augmentation methods [ 34 33], we empiri- cally incorporate action sequences from the past T timesteps. Unlike previous approaches, the delay steps in this paper are represented as vector whose length corresponds to the number of other entities included in the observation. We implement the two types of compensators using GRU [ and Transformer [31] networks, which excel at processing sequential data. Figure 4 illustrates the extended observation input forms when different model structures are employed. Fixed inputs refer to the pre-concatenated se- quence data fed into the model during the first input step. In contrast, sequential inputs repre- sent the incrementally augmented input informa- tion during the model's s autoregressive process. For Echo, convert the delay value vector into a binary (0 or 1) vector to enhance consistency between history and autoregressive inputs.
GRU
Transformer
History inpuls
Zuer
Auua
D&r
Zan
D.
Zet
Axt
D₁1
D,
7
Aa
D,'
ontputs
2,
Au
D,"
Figure 4: Inputs of compensators with different modes and networks. The figure only illustrates the input formation of single agent, timestep 1, and for clarity, we omit the subscript i. The subscript denotes the timestep in the environment iteration, while the superscript indicates both the k- th output of Echo and the (k+1)- input. Inputs of Flash exclude the yellow- gellow-highlghheed components.
Additionally, we design dual-head residual compensator to handle different data types in observa- tions. We employ the cross-entropy loss function for the classification task, while utilizing the mean squared error for the regression task. The residual output accuracy of the compensator. The loss functions for the two compensators are as follows:
further improve the reconstruction
Liasst (0) = LCE(I", I(ZT_GT Z)) + LMSE(FF) (F", F(ZT.GT 7)):
T
Cechho()) = T
Σ
CCE (T I(2 z*-1)) + LMSE(FF F( )))1
where Is = - zーー1 with I(.) and F(-) representing the extraction of integer-type and float-type contents from the target, respectively. Z denotes the observation obtained by agent, at time t (omitted in the formula). 2 denotes the ground truth after l compensation steps of the delayed observation. denotes the model parameters of the compensator.
4.3 Delay-reconeiled Critic and Curriculum Learning Actor
Wang et al. 34 first introduced the decon-eecddeiled concept to address single-agent delayed obser- vation problems. Their key insight was that feeding the critic with delay-free global states during centralized training could mitigate the impact of delays. Since the critic's 's involvement is unnecessary during model inference, the delay-reconciled critic can seamlessly integrate with the centralized training with decentralized execution (CTDE) paradigm.
In addition to receiving compensated observation, the actor may exhibit poor convergence when facing complex scenarios. This occurs because the online training of the compensator also requires time, and the compensator's outputs during the early training stages may significantly deviate from those observed under delay-free conditions. Empirically, the exploration phase in early reinforcement learning training typically plays a crucial role, even though the reward values may not show noticeable improvement during this period. However, unlike the critic, which only operates during training, the final model must ultimately rely on realistically available observation as input. Curriculum learning 3] provides solution to this challenge. During the initial training phase, provide the actor with

---

delay-free observation (i.c., the compensator's ground truth) and gradually reduce the probability of using delay-free observation as training progresses, until the actor relies entirely on compensated observation. A linear annealing strategy is used in our experiments, though more sophisticated or adaptive annealing approaches are also permissible. For less complex tasks (e.g., MPE), actor curriculum learning is not strictly necessary.
4.4 Knowledge Distillation
Despite all design and optimization efforts, discrepancy between compensated observation and delay-free ground truth inevitably persists. However. agents' policies in identical scenarios can be objectively evaluated through quantitative metrics. When training with delayed obser- vation, guidance from nigh-performance policy model can steer the model toward more accurate policy optimiza- tion or value estimation, thereby accelerating convergence. Motivated by this insight, we incorporate knowledge dis- tillation [16] into the RDC framework. While numerous distillation approaches exist, we experimentally identify an effective methodology. Before training the target model in high-delay cnvironments, we first train a teacher model under low-delay conditions. As anticipated, the teacher model achieves performance closer to ideal delay-free training than independently trained student models in high-delay scenarios. During student model training, we feed the teacher model with compensated observation and employ it to guide both the hidden representations and output decisions of the student actor and critic. The corresponding loss function is formulated as:
delxyed obs
Flash
compensated
Echo
mask layer
T T'teps
Compensator
Figure 5: Workflow of Flash and Echo.
(O.) Ler(actione action.) + 31 LMSE(Qr; Q.) &2LMs 0g),
(4)
(1,.) (0.) + (1 (1--1)1)))) C..)
(5)
where 31 and 32 denote the weighting factors within the knowledge distillation loss function, and determines the relative weighting between the knowledge distillation loss and the reinforcement learning loss.
We do not apply knowledge distillation the compensator or load the teacher's compensator during student training. This design choice stems from two key considerations: First, in online learning scenarios, the compensator's performance should evolve alongside policy improvements, and directly transferring the teacher's 's compensator knowledge through distillation may not necessarily benefit student policy training. Second, this approach allows for more explicit demonstration of the effects of pure policy guidance, which we elaborate on in the experimental section. Following a similar principle curriculum learning, we implement an identical annealing strategy for the knowledge distillation process.
Experiments
Scenario Selection: We select two of the most popular multi-agent re- inforcement learning environments for our experiments-MPE122 and SMAC 228. Overall, SMAC tasks present greater challenges than MPE tasks due their more complex state and action spaces. We chose simple- tag (TAG), simple-spree (SPREAD), and sinple-reference (REFERENCE)
SPREAD
REFERENCE
2kp
200
$2.0
Bate-DX
MPE and three progressively harder scenarios on SMAC: 3s_vs_52.
Figure 6: Training performance on MPE.
5m_vs_____ 5m vs_ 6m, and 6h_ vs_ 87. Both benchmarks are discrete environments, so we incorporated value function factorization algorithms into the RDC framework. The reward value obtained per task in
7

---

MPE is the sole evaluation metric. While SMAC also provides specific reward values, the win rate is a more critical metric since the objective achieve victory.
Experimental Scheme: We reformu- late the delayed observation problem in MASs using DSID-POMDP and ex- tend commonly used benchmarks. Consequently, we implement previ- ous solutions within the RDC frame- work, such as the delay-reconciled critic and augmented observation in- put, to enable more equitable compar- isons. For baseline RI. algorithms, we select code-level optimized FT-QMIX and FT-VDN 17 that demonstrate
We (06
Wr SALA
Figure 7: Training performance SMAC.
significant improvements over vanilla QMIX and VDN, whose performance on discrete tasks has been repeatedly validated [88 [0]. To simplify the presentation, the following abbreviations will be used in the experimental results section: Oracle, Base, Echo, Flash, DR (delay-reconciled critic), H (history input), C (curriculum learning actor), and KD (knowledge distillation). Only the Oracle is trained using the baseline algorithm in delay-free environment, obtaining the ideal performance that represents the algorithm's 's capability, unaffected by delays. In the main text, all presented results based using FT-QMIT as the baseline algorithm and implementing the compensator with Transformer networks.
Test Setting: For every 10,000 training steps, we conduct test with either 64 or 32 episodes and record model performance. After model training, we perform extensive testing under fixed and unfixed delay conditions, running 1,280 episodes for each setting.
5.1
Training
The training performances of FT- QMIX under delayed observation (Base) in Figure 6 and Figure 7 re- veal severe performance degradation across all six scenarios, particularly in the 5m vs_ 6m and 6h vs 8z tasks, where win rates approach zero. Our ablation studies employing currieu- lum learning (Base+C) and delay- reconeiled training (Base+DR) with- out compensation mechanisms show that in complex scenarios, neither cur- riculum learning alone nor simply pro-
TAG
SPASAD
RERE4ENCE
-15
-25
Ouko
Deay
Doly
Figure 8: Performance under fixed delay on MPE.
viding delay-free states to the critic can satisfactorily counteract the detrimental effects of delayed observation on the learning process. These findings suggest that delayed observation has fundamen- tal impact on both initial convergence and overall policy optimization.
The RDC-enhanced models exhibit significantly faster convergence in most scenarios, demonstrating that while the knowledge distillation pro- cess requires sequential training of teacher and student models, incurs minimal additional training overhead. This efficiency advantage arises from accelerated training in low-delay con- ditions, where the student model re- quires fewer than one-third of the original training iterations. Notably, the enhanced models achieve perfor-
TAIE TTO
UNT Taic D0
Figure 9: Performance under fixed delay on SMAC.
8

---

mance that matches or slightly exceeds that of the delay-free Oracle. The marginal improvement is attributable to additional training steps, as evidenced by Oracle's ongoing performance gains the end of training in both SPREAD and REFERENCE scenarios.
5.2 Performances on different delay settings
Fixed delay testing enables more precise observation of progressive performance degradation and more accurate assessment of model generalization. As shown in Figure and Figure the Oracle method exhibits substantial performance deterioration as delays increase. The performance drop in the Base and Base+DR methods at delays of 0-2 reveals their inability generalize to unseen delay conditions during testing, despite successful convergence during training. In contrast, RDC-enhanced models maintain superior performance across all scenarios, demonstrating particularly robust delay adaptation in SPREAD and REFERENCE tasks where reward stability persists despite increasing delays. The model's generalizability on SMAC is significantly lower than on MPE, which indicates that the impact of delay is more pronounced in complex scenarios.
We evaluate model performance un- der two unfixed delay conditions: in-distribution (within trained delay ranges) and half-out-of-distribbuie (novel delay ranges). Unless other- wise specified, the random delays in this paper follow uniform distribu- tion within a given range. The per- formance under different delay distri- butions will be discussed later. As shown in Figure I0 RDC-enhanced models with Transformer- pensators demonstrate only marginal performance degradation in out-of-distribution tests while maintaining near-Oracle performance (green dashed line). Ablation studies reveal the contribution of each module: Flash+DR underper- forms due to the limited imput information, while Flash+H+DR shows significant improvement by incorporating history observations. Echo's autoregressive design provides richer input information, resulting in smaller gains from historical data. Knowledge distillation using low-delay teacher model proves effective, though we excluded Flash+DR from this approach due to its poor baseline performance.
TAG
SPROND
RETERENCE
CoCa
Figure 10: Performance under unfixed delay MPE.
com-
To demonstrate that the RDC frame- work can adapt to different delay distributions, we evaluate the perfor- mance of models trained under uni- form delay distribution when tested on other distributions. As shown in Figure I1 tribution, the normal distribution, and the Poisson distribution as additional evaluation options. In these tests, the delay values are constrained to either the range of 3-9 6-12. The bino- mial distribution, which characterizes the number of successes in n independent trials, naturally fits our delay settings after simple shift. Although the values generated by the Poisson distribution are also discrete, they require truncation due the distribution's unbounded support. For the normal distribution, truncation is typically fol- lowed by rounding due to its continuous nature. Considering the properties of different distributions, we adopt the following parameter settings in our experiments: Binomial(6, 0.5) and Binomial(9, 0.5); P6isson(6) and Poisson(9): Normal(6, 2) and Normal(9, 2). The results demonstrate that the model generalizes effectively across various delay distributions. This result consistent with our expectation, as the compensation process for delayed observations in the RDC framework does not rely on any prior knowledge of the delay distribution. Since different distributions entail different frequencies of high and low delay values, minor performance fluctuations are understandable.
150
we select the binomial dis-
Delp
Datay Runge
Figure 11: Performance under different delay distributions.

---

5.3 Additional results and analysis
How does the performance of Flash compare with that of Echo? The re- sults show that as the delay magni- tude increases, Flash typically demon- strates more pronounced performance degradation than Echo, particularly when handling out-of-distribution de- lays. As illustrated in Figure [2] maintaining identical input bistory se- quence lengths and training configu- rations is essential for achieving opti- mal performance with the Flash com- pensator. This suggests an overfitting
SPRDAD
-22
-20
--0-
Scterbac delp
Echalled Saiyy
Figure 12: Performance under different history lengths.
compensation pattern, confirming our earlier concern about Flash. Although this limitation in gen- eralizability and flexibility is undesirable, we must emphasize its significant advantages in training efficiency and inference speed, which can be decisive factors in specific scenarios. In TAG scenario with fixed delay of 6, the compensator inference times for Echo and Flash are approximately 0.02 seconds and 0.004 seconds, respectively.
Why does our method outperform the delay-free Oracle baseline across mul- tiple tests? We align the number of en- viromment steps required for training across all algorithms. As a result, the RDCCenhanced method, which lever- ages knowledge distillation, obtains additional guidance from the teacher model within the same training hori- zon. To validate this explanation, conduct an experiment on MPE where the RDC-enhanced models remained unchanged, while Oracle, Base, and Base-DR are trained for an additional 10 million steps, which is the same number of steps used to pretrain the low-delay teacher model. The results in Figure 13 confirm our hypothesis: in the zero-delay setting, the Oracle performance surpasses that of the RDC-enhanced method, consistent with its role as the ideal upper bound. In delayed settings, however, the non-RDDC-nnhaneed methods still suffer from significant performance degradation.
-11
-25
-40
Dato
Dulky
Figure 13: Performance under longer training steps.
All experimental results are provided in Appendix D including comprehensive ablation studies and evaluations of different baseline algorithms and compensator network architectures. These experimental results not only validate the compatibility of the RDC framework with algorithms beyond the actor-crite cittorrriticcpiiii paradigm but also highlight that the choice of baseline algorithm and compensator architecture plays a crucial role in achieving superior performance.
6 Conclusion
In this paper, we propose the RDC framework and demonstrate the effectiveness of its constituent modules in addressing delayed observation in MASs. The compensator, as the core component of the framework, directly attempts reconstruct delay-free observation. The curriculum learning actor and delay-aware critic provide higher-quality data, while knowledge distillation from a low-delay teacher offers policy guidance during model training. The integrated algorithm, which incorporates these modules, achieves outstanding performance with fixed and unfixed delays comparable to those in delay-free environments. Our future research will focus on designing more effective compensator architectures and knowledge distillation techniques to enhance the model's 's generalizability under varying delays in complex scenarios, as well as developing theoretical frameworks with weaker assumptions. Overall, our research not only presents a multi-agent reinforcement learning algo- rithm capable of combating delayed observation but also provides a practical training framework, establishing solid foundation for future studies.
10

---

Acknowledgements
This research is supported by the Oriented Project Independently Deployed by the Institute of Acoustics, Chinese Academy of Sciences (MBDX202402).
References
[1] Zvi Artstein. Linear systems with delayed controls: A reduction. IEEE Transactions
Automatic control, 27(4):869-879 1982.
[2] James L Bander and Chelsea C White III. Markov decision processes with noise-corrupted and delayed state observations. Journal of the Operational Research Society, 50(6):660 668, 1999.
[3] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pages 41 -48, 2009.
[4] Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Binas. Reinforcement learning with random delays. In International conference on learning represen- tations, 2020.
[5] Marwa Chafii, Salmane Naoumi, Reda Alami, Ebtesam Almazrouei, Mehdi Bennis, and Merouane Debbah. Emergent communication in multi-agent reinforcement learning for future wireless networks. IEEE Internet of Things Magazine, 6(4):18 ((4):18224 24, 2023.
[6] Baiming Chen, Mengdi Xu, Zuxin Liu, Liang Li, and Ding Zhao. Delay-aware multi- agent reinforcement learning for cooperative and competitive environments. arXiv preprint arXiv:2005.05441, 2020.
[7] Baiming Chen, Mengdi Xu, Liang Li, and Ding Zhao. Delay- model-basee reinforcement learning for continuous control. Neurocomputing, 450:1 1289 2021.
[8] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder- decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
[9] Vlad Firoiu, Tina Ju, and Josh Tenenbaum. At human speed: Deep reinforcement learning with action delay. arXiv preprint arXiv:1810.00
2018.
[10] Songchen Fu, Shaojing Zhao, Ta Li, and Yonghong Yan. Qtypemix: Enhancing multi-agent cooperative strategies through heterogeneous and homogeneous value decomposition. Neural Networks, 184:107093, 2025.
[11] Mareus Gerwig, Karim Hajjar, Albena Dimitrova, Matthias Maschke, Florian P Kolb, Markus Frings, Alfred F Thilmann, Michael Forsting, Hans Christoph Diener, and Dagmar Timmann. Timing of conditioned cycblink responses is impaired cerebellar patients. Journal of Neuro- science, 25(15):3919-3931 2005.
[12] Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55(2):895-943 2022.
[13] Shangding Gu, Jakub Grudzien Kuba, Yuanpei Chen, Yali Du, Long Yang. Alois Knoll, and Yaodong Yang. Safe multi-agent reinforcement learning for multi-robot control. Artificial Intelligence, 319:103905, 2023.
[14] Shangding Gu, Dianye Huang, Muning Wen, Guang Chen, and Alois Knoll. Safe multiagent learning with soft constrained policy optimization in real robot control. IEEE Transactions Industrial Informatics, 2024.
[15] Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. arXiv preprint arXiv:2307.16212. 2023.
11

---

[16] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1503.02333, 2015.
[17] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih-wei Liao. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. arXiv preprini arXiv22102.03479, 2021.
[18] HAO Jianye, Xiaotian Hao, Hangyu Mao, Weixun Wang, Yaodong Yang, Dong Li, Yan Zheng. and Zhen Wang. Boosting multiagent reinforcement learning via permutation invariant and permutation equivariant networks. In The Eleventh International Conference Learning Representations, 2022.
[19] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34(04), pages 4501-4510, 2020.
[20] Pierre Liotet, Davide Maran, Lorenzo Bisi, and Marcello Restelli. Delayed reinforcement learning by imitation. In International conference on machine learning, pages 13528-13556 PMLR, 2022.
[21] Jiaqi Liu, Ziran Wang, Peng Hang, and Jian Sun. Delay-aware multi-agent reinforcement learning for cooperative adaptive cruise control with model-based stability enhancement. arXiv preprint MxXiv:240.116690 2024.
[22] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-commptiivee environments. Neural Information Processing Systems (NIPS), 2017.
[23] Miroslav R Matausek and AD Micic. On the modified smith predictor for controlling process with an integrator and long dead-time. IEEE Transactions on Automatic Control, 44(8):1603- 1606, 1999.
[24] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
[25] Silviu-Iulian Niculescu. Delay effects on stability: a robust control approach, volume 269. Springer, 2003.
[26] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint arXiv:2006.07866, 2020.
[27] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21 21(17))11-51, 2020.
(28) Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nan- tas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The stareraft multi-agent challenge. arXiv preprint arXiv: 1902.04043, 2019.
[29] Andrew C Singer, Jill Nelson, and Suleyman Kozat. Signal processing for underwater acoustic communications. IEEE Communications Magazine, 47(1):90-96, 2009.
[30] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv: 7006 05296. 2017.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
12

---

[32] Thomas Walsh, Ali Nouri, Lihong Li, and Michael L Littman. Learning and planning in environments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18:83-105, 2009.
[33] Fanshuo Wang, Hui Zhang, and Ya Zhang. Resolving action delay: Multi-agent reinforcement learning based on state prediction. In Chinese Intelligent Systems Conference, pages 552-563. Springer, 2024.
[34] Wei Wang, Dongqi Han, Xufang Luo, and Dongsheng Li. Addressing signal delay in deep reinforcement learning, In The Twelfth International Conference on Learning Representations, 2023.
[35] Yunchang Yang, Han Zhong, Tianhao Wu, Bin Liu, Liwei Wang, and Simon Du. A reduction- based framework for sequential decision making with delayed feedback. Advances in Neural Information Processing Systems, 36:46362 46389, 2023.
[36] Tingting Yuan, Hwei-Ming Chung, Jie Yuan, and Xiaoming Fu. Dacom: Learning delay-aware communication for multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37(10), pages 11763-11771, 2023.
[37] Hengxi Zhang. Zhendong Shi, Yuanquan Hu, Wenbo Ding, Ercan E Kuruoğlu, and Xiao-Ping Zhang. Optimizing trading strategies in quantitative markets using multi-agent reinforcement learning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 136-140. IEEE, 2024.
[38] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pages 321-384, 2021.
[39] Yuyang Zhang, Runyu Zhang, Yuantao Gu, and Na Multi-agent reinforcement learning with reward delays. In Learning for Dynamics and Control Conference, pages 692-704. PMLR, 2023.
[40] Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent deep reinforcement learning with communication. Autonomous Agents and Mulri-Agent Systems, 38(1):4, 2024.
13

---

Technical Appendices and Supplementary Material
A Proofs
Here prove the correctness of the state transition function and observation function expressions defined in DSID-POMDP. First, the state x contains T history states:
x {8, 8(1))) ((T))-
Expand the expression of the new state transition function:
PD (x' x, a) PD (s', (-1):..., s'
((-T).
Observing these two sequences reveals the transition process of history states s' (-t) = ((-++)) which consequenly be decomposed into current state transitions and history state transitions. Due to the Markov property, the oldest state in the original state can be directly discarded.
PD (x'xx, a) = Pp (s', 5'(-1)
8(-2),
T
= =P(s'ss 8, a)
II 8 (((()))8((1+))
1=1
In other words, when the transition of the historical sequence is determined, the state transition function of DSID-POMDP becomes identical to that of POMDP. A fundamental assumption for the validity of the observation function is that observations from different entities are mutually independent. While this assumption holds most scenarios, it may not be satisfied in specific cases, such as in single-channel communication systems where signals from different entities can interfere. For agent, the observation delay value corresponding to entily, is 6, can be expressed as 20 (-1). Therefore, the probability of this agent obtaining observation d from entity entity; is:
OD (2 (x) = Ep(a -)) (() -0000"=2'0000 =0)ss-n)
where O (o = وd (()) denotes the probability of obtaining observation 2 given state 8(1)) in the original POMDP. Due to the independence assumption, this definition can be extended to joint observation through factorization:
T Σ
Od (z|x) = II
II
pdd 8) (3") -s 0 2d = =()))(-)))-
VET JETj
B Scenario Introduction
simple-tag: This is competitive predator-pre peryy simulation where good agents must evade slower but aggressive adversaries. The good agents are faster but incur penalty for each collision with an adversary, while adversaries are rewarded for successfully hitting them. The terrain includes static obstacles that block movement. To prevent good agents from escaping indefinitely, they are penalized for exiting the designated area based on predefined boundary function. By default, the scenario starts with one good agent, three adversaries, and two obstacles, creating dynamic balance of pursuit and evasion. In our experiments, the policy of good agents is fixed as a pre-trained model with MADDPG [22), consistent with Papoudakis et al. (26). The algorithm only controls the adversaries, transforming the adversarial environment into a cooperative onc.
simple-spread: In this cooperative multi-agent scenario, N agents (default: 3) must learn to cover landmarks efficiently while avoiding collisions. The agents are collectively rewarded based on how well they cover all landmarks, measured by the sum of the minimum distances between each landmark and its closest agent. Agents must minimize this global distance metric to maximize their shared reward. To prevent reckless behavior, each agent is individually penalized for colliding with other agents. The trade-off between global coverage and collision avoidance adjustable the local_ ratio parameter, enabling fine-tuned control over agent coordination strategies.
14

---

Table 1: Mcan Environment parameters.
MPE
SMAC
Parameter
Value
Parameter
Value
limit agent obs last action
25
difficulty
7
obs, agent ohs last action state last, action
False
True
state. www.ttemeeepeeeeee numberr conic_fov
False
False
simple-reference:
This scenario involves two cooperative agents navigating toward three uniquely colored landmarks, each with hidden target assignment. Crucially, an agent's 's target landmark is only known by the other agent, requiring real-time communication to resolve uncertainty. Both agents act as speakers and listeners, exchanging information to infer each other's goals while navigating.
Unlike MPE, the objective in all SMAC scenarios is to achieve victory, with the sole approach being the complete elimination of enemy forces. The scenario names explicitly indicate the force compositions both allied and enemy units-for example, 6h vs 8z denotes six Hydralisks battling cight Zealots. This environment emphasizes short-term coordination among agents and the effective utilization of unit-specific advantages.
SMAC:
C Implementation Details
C.1 Delayed Observation Implementation
(a)
(b)
Figure 14: Applications of SSID-POMDP on SMAC's 6h vs 8z and MPE's simple-tag.
We develop delay filters for SMAC and MPE to simulate realistic perception and communication delays, implementing four distinct modes: no delay (none), fixed delay (f), partially fixed delay (pf), and unfixed delay (uf). The fixed delay mode sets fixed delay value and applies it to all observations. The partially unfixed delay mode uses fixed delay values but does not necessarily delay all observation contents. The unfixed delay mode introduces randomness in both delay values and which contents get delayed. While delay filters support different distributions or distance-based delay calculations between entities and agents, our experiments use uniform distributions for all random variables, without loss of generality. The delay implementation mechanism primarily maintains history observation records for each agent and retrieves information from past observations at the current timestep according to specific delay policies.
We classify agent observation into four categories based on their delay characteristics: movement features (real-time updated information), enemy features (environmental entity data excluding allies), ally features (information from allied agents), and self features (agent's state). In practice, naming these features not crucial-our primary distinction lies their delay characteristics. For example, in SMAC's s 6h th__s__ 87 scenario, observations include movement capabilities (4 dimensions), 8 enemy states (each with 6 dimensions), ally statuses (each with 5 dimensions), and self-attributes (1 dimension). Our implementation delays only enemy and ally features, while maintaining real-time
15

---

updates for movement and self features. This reflects real-world conditions, where agents have immediate access to their own states but experience delayed environmental perception.
The system initializes the observation history at the start of each episode and continuously records delay-free observations. It retrieves historical data based configured delay parameters when generating delayed observations. For example, in MPE's simple-tag scenario, predators may track evaders using positions delayed by 2 timesteps, impairing pursuit efficiency. Similarly, in SMAC, troops exhibit response lags to enemy movements. The system maintains temporal consistency by ensuring delayed observations are always drawn from valid history states while preserving the original environment dynamics. This implementation enables systematic investigation of various delay patterns (fixed, unfixed, or distance-dependee multi-agent decision makinng process. The modular design allows for the flexible configuration of delay parameters without modifying the core environment logic.
while maintaining the integrity of the underlying
Additionally, the delay filter enforces temporal consistency constraints, ensuring that information observed step cannot be older than that step When the delay-reconciled critic is not employed, the global state is formed by concatenating observations from all agents. By implementing delay filters in existing simulation environments, establish a reliable and flexible experimental platform for investigating the impact of delayed observation on the performance of MASs.
C.2 Compensator Implementation
Z
Awis
0
2
Ao
1 11
2
A
2
2/8
A
3
2
3
2
3
(a)
(b)
2g
Awis
0 0
2
Ao
11000111
Zg
A
2
2
7
A
3
Zz'
A
z
3 3
22
A
3
2,²
3
2,3
3338893 3333
(c)
Figure 15: Inputs and outputs of compensators. (a): Flash without history inputs. (b): Flash with history inputs. (c): Echo without history inputs. (d): Echo with history inputs.
We utilize a deep learning-based compensator to mitigate the effects of observation delay in MASs, and employ GRU-based and Transformer-bant architectures to predict delay-free observation from historical data. The compensator implementation consists of input sequence construction, label generation, and mask generation. The compensator processes input sequences constructed from past observations and actions, with optional T-step historical context (padded when insufficient). Figure [5 follows the example from Figure Both Flash+H and Echo+H models acquire additional information. Notably, Echo's s autoregressive inference aligns closely with historical sequence processing, which helps the model understand the relationships between delay and observation. The figure visually represents delay value sequences as their corresponding actual environment timestep sequences. In practical implementation, we input delay value sequences for Flash and binary (0 1) sequences indicating the presence of delay for Echo.
16

---

Table 2: Mean Training parameters.
MPE
SMAC
Parameter
Value
Parameter
Lmax
Sc6/1e7
Lmax
6e6
nepisude
lest, nepisode
batch sire
32
batch, size
128
spsilon_anncal_ting anncal_time
5o4
epsilon anncal time
1eS/5e6
True
standardise rewards
False
actor model
GRU
actor model
GRU
target_ cests__udate_mmrrrree intervl
200
taggtt update interval
200
mixing_embedd.e Lim_eeebeddin
32
mising_embedd embed dim
32
hypemet, embed
hyperne, embed
actor hidden dim
64
actor wee__hidddddiie _imm
64
td Ia_aabda
0.6
1d Jambda
0.6/0.3
i[Jearming_rate
le-3
rl Llleanning-ao rate
le-3
compensator model
GRU/Transformer
compensator model
GRU/Transfomec
compensator, hidden_dimn
compensator hidden
compensatorr mode
Nome/Faash/Ecco
compensator mode compensator learning rate
Nome/Flash/Echo
compensator learning rate
le:3
delay_type
unfixed
delay_type
unfixed
delay_value
6
delay_ valuu
delay_ _ccupe
3
delay_ sccupe use history
use history
True/False
True/Fale
history Jength
history_ delay reconciled curriculum, start value curriculum, end_valus curriculum, start value
delay reconciled
True/False
True/False
curiculumm start, value
curriuuuumcnoe ciii__nd___l value
0
curriculum step curriculum ete_step distillation start value distillation,
le6
1e6
3e6
curriculum, end
4e6
1/0
distillation start value
1/0
distillation end value
distillation start step distillation stop
2e6/366
distillation start value distillation _nn__stepp step
2c6
4c6/766
4e6
We adopt a supervised learning framework where the label generation module creates ground truth from stored delay-free observations, using ideal delay values reference. The mask generation module prevents error propagation Echo by masking previously compensated content. Our hybrid loss function combines mean squared error (MSE) for continuous features (e.g., positions) with weighted cross-entropy (CE) for discrete features (e.g., unit status), balancing their numerical scales. To enhance training efficiency and stability, we implement teacher forcing initially using real observation as next-step input with 100% probability, gradually decreasing to 0% during training to help the model learn to rely on its predictions. As shown in Table the teacher forcing training mode demonstrates no significant advantage compared to not using teacher forcing. Therefore, we disable this option by default during training.
C,3 Training Details
In this section, we present the hyperparameter settings for different tasks. Table and Table detail the key environmental and training parameters, respectively. While ensuring algorithm performance, we maintained consistency in hyperparameters as much as possible. All actor networks use the GRU architecture, while QMIX's critic employs a two-layer hypernetwork. The GRU compensator consists of one GRU layer and three linear layers, whereas the Transformer compensator uses only one pair of encoder-decoder layers and supports pure encoder structure. The replay buffer size is fixed at 5000, with batch sizes set to 32 for MPE and 128 for SMAC. During training initialization, = =1 and linearly anncals to = 0.05. The default training step for MPE is set to 5c6. However, we observed second significant performance surge around 2e6 steps REFERENCE, with performance still maintaining an upward trend at 5e6 steps. Consequently, we extended the training steps to lc7 to provide sufficient convergence time for each algorithm. Accordingly, the starting and ending steps for knowledge distillation were also increased.
We do not entirely disregard the curriculum learning option in MPE scenarios. However, early experi- ments revealed that this technique proved ineffective for MPE, while being indispensable for SMAC. This result demonstrates that curriculum learning actors can significantly mitigate convergence issues caused by random seeds when handling complex tasks. In reinforcement learning, the problem of model non-conveeeence due to the improper timing of performance ascent is prevalent. Careful
17

---

observation of convergence curves reveals that performance ascent typically does not occur during the initial training phase SMAC tasks. In contrast, in MPE tasks, performance improvement closely follows the start of training. This timing difference may explain why curriculum learning actors succeed remarkably in SMAC scenarios.
The teacher model is trained for 1c7 steps across all tasks with delay setting of 1-3. We attempted directly employ the Oracle as the teacher model, where, during student model training, the Oracle receives delay-free observation and provides immediate guidance. This approach proved unsuccessful, likely due to the inherent discrepancy between delay-free observation and compensated observation. Specifically, when the student model receives compensated observation, decisions based on delay-free free observation might objectively superior but could disrupt the student model's judgment. To address this issue, we experiment with low-delay teacher model, achieving exceptional performance. We encourage other researchers explore further variations within our framework.
D Supplementary Results
Table 3: Performance comparison with teacher forcing enabled and disabled.
Fixed Delay Value
12
Rchoo Echo
190.9
26.0
175.8 + 29.7
168.8 23.4
161.7 +: 25.4
158.5 + 25.9 155.5 25.4
185.9 23.4
175.1
30.3
176.9 29.9
167.3 21.0
SPREAD
Echo
-34.2 +
-34.1 1.9
-34.2 2.0
-3.33 2.2
33.4 2.1
Echo
-3366
= 2.2
-31.3
-35.4
1.7
Table 4: Fixed delay results (rewards) of MPE with baseline algorithm FT-QMIX.
Fixed Delay Value
12
DR
H
KD
TAG
Oracle
213.4 33.5
136.2
23.0
84.5
16.9
70.1
66.1
13.3
Base
110.2 17.7
135.2
22.5
125.4
22.8
112.6
101.5
19.6
135. 26.7
156.9
23.2
142.3
22.3
120.9
20.8
111.7
17.5
Flash
176.6 27.7
177.5 = 31.2
150.4 = 22.5
132.8 21.6
127.8
23.7
Flash
188.0 26.6
180.4
24.7
166.7 29.0
168.5 28.2
149.9
26.3
Flash
213.7 30.8
194.8
27.5
182.5 1: 32.0
169.8 1 23.9
150.9
24.1
Fcho
194.2 20.0
187.6
29.0
173.8 - 22.6
160.7 30.1
163.7
27.5
Echo
175.1
176.9
167.3
21.0
155.5
25.4
Escho
206.5 31.0
28.5 200.8
197.9
184.7 + 32.4
175.6 + 22.9
165.2
28.0
209 8 32.3
25.1
182.9 9 36.6
176.2 + 24.8
160.5 29.5
H
KD
SPREAD
Oracle
33.1 2.0
-37.6
48.8 = 2.0
59.9 2.9
67.2
Base
10.4 2.2
-381
10.3
2.4
-14.2 2.2
-16.5
Base
-11.1 2.4
-392
41.0
: 2.5
-44.7 2.0
-49.2
Flash
37.7 1.7
337.8 22 23
39.3
1.9
-00.1 2.1
-40.0
2.3
Flash
-33.9 1.8
-.0
-3.1
-34.4
-3.8
Flash
16
-3241
2.3
$332222
-5.4
36.3
2.6
36.2
36.8
33.4
Echo
v
2.2
31.3
1.5
35.411
36.6
Eicho
32.2 1.6
33.0 -20-233
33.3
33
22
33.9
1.9
Echo
31.8
1.9
32.0
32.3
1.9
33.1
2.1
33.2
2.2
Oracle
17.6 1.2
22.5 122
-31.
1.1
38.2 1.5
43.7 1.9
Base
29.5
28.6
30.3 1 1.7
34.3
36.8
Base
-28.9
-26
-27.7
1.7
-0.9
34.7
2.1
-34.8
34.9
35.1
Flash
4
-17.1
-174
-17.3 1.2
-8.0
20.0 1.2
Flash
13
-17.0+13
-16.7
1.7.6111
-9.6 1.7
Echo
1.4
-180=14
-.
1..00
19.11
Echo
4
-18.4
-18.5 15
11.1
18.7
20.2
1.4
Echo
20.6
-20.0
-1.
1.1
2
Echo
4
16..7
-16.911
116.9
-17.6+12
-18.714
18

---

Table 5: Unfixed delay test results (rewards) of MPE with baseline algorithm FT-QMIX.
Delay Range
SPREAD
REFERENCE
Oracle
15.2
-45.9
-56.4
-28.3
36.1 1
127.00 =
113.6
19.9
-39.6
2.5
29..
-33.2
Base
144.6 1: 21.1
132.4 19.7
-40.5
-4..0
-
-30.4
Flish
161.5 + 29.4
151.4
2.3
-30.2
2.0
-35.2
165.4
-34.5
-18.2
Flash
188.
31.1
30.9
-32.1
-33.0
-17.0
-17.2
Fcho
175.1 25.0
165.5 24.5
-36.7 14.7222
-66.9 2.2
18.8 -18.6
-18.9 1.3
Echo
177.1
168.8 27.4
-34.2120
-35
1.2
-18.9 1.2
180.0
2.2
22.3
21.1
-31.41
-32.5
-17.0
-17.4
Table 6: Fixed delay test results of SMAC (win rates) with baseline algorithm FT-QMIX.
Delay
DR
KD
35_ vs_v__5
Oracle
99.7 =
61.6 = 9.3
26.6 7.9
2.4
0.1
Base
82.9 6.2
79.3
8.1
84.9
6.1
69.9
65.0 9.1
Base
97.0 2.6
95.5
95.1
3.6
93.8
4.5
88.4
4.8
Base
97.0 2.7
98.0
2.0
92.3 1
74.8
8.3
33.0 10.5
Base
92.1
288
5.9
11.5
4.5 3.4
Flash
95.0 4.3
96.6 = 3.0
9..4
78.1 6.2
37.7
10.2
Flash
43.5
7.9
F Flash
99.8
99.3
99.4
94.9 3.4
82.9 5.4
Eeha
96.3 2.8
97.1
2.7
3.8
83.6
6.3
24.7 8.5
Ficha
98.5
98.8 1
9.8
84.2
6.7 6.7
60.9 8.0
Echo
99.8 0.8
99.8 + 0.7
99.8 0.8
97.1 1 2.7
80.0 8.1
DR
KD
5mLys_66
Oracle
84.8
0.0
0.0 0.0
0.0 0.0
0.0 0.0
Base
0.5 +
0.1 1.0
0.9 1.6
0.5 0.3
0.1 0.5
Base
7.3 1,3+00
15.9
12.1 5.4
0.4
0.1 0.5
Base
58.1 9.0
80.2
7.1
57.2 10.8
20.5 8.1
3.0 2.8
Flush
83.2 6.4
80.6170
L.
59.9 19.99..55
36.6
Flash
4.4
81.9
6.4
43.9
4.844.5
Flash
6.7
6.
7.7
9.8
Eibo
819 7.2
7.1
7.3
58.4 10.2
11.7 8.3
73.4
7.7
Echo
86.3 = 6.5
85.2
79.9 7.7
65.2 8.9
51.8 8.6
DR
KD
6h_vs_8z
Oracle
92.0 4.0
0.5 1.2
0.0 +
0.0
0.0 0.0
Base
0.6 0.6:11 1.2
1.0
0.7 1 1.5
0.1
0.0 0.0
Base
4.1.3.1
3.8 1.81.0
2.4
2.6
1.4 1.8
0.7 1.5
Base
3.1
4.8
2.7
0.4
0.2
Base
5..8
70.6
12.7 6.1
2.7
0.3
Flash
10.9
0.5
Flash
8.1
7.6
7.9
5.9
1.2
Flah
90.6
8161611
8.7
12.6
3.4 3.4
Echo
8.2
82.0
7.3
9.66
11.2 5.6
2.9 3.1
Fibha
86.7 6.2
8.2
19.1 6.8
5.1
0.6 1.2
Echo
94.1
6.3
57.6 10.7
6.7
6.2
Table 7: Unfixed delay test results (win rates) of SMAC with baseline algorithm FT-QMIX.
DR KD
3t_
Ornele
62.2 1. 7.8
16.2 6.4
0.0
0.0
2.9 3.2
0.0
1.2
3.1
3.7
2.0
6.5
1.3
Base
98.1 1.7
64.8
6.2
4.1 3.2
2.3
0.9 1.4
Base
97.0 4: 2.6
74.7 1 7.4
79.0
45.4 8.5
68.6 11.8
7.2 5.4
Flash
97.2 2.9
60.6
63.7 6.6
73.5
73.8
22.3
Flush
99.7 0.9
2.6
150170
76.67..
80.7 7.1
28.9 7.6
Echo
2.0
95.5
7.8
66.9
78.8 7.1
27.5 9.4
Echo
0.7
84.5
76.0
45.2 10.9
19

---

Our experimental results are primarily presented through figures and tables, with detailed explana- tions. We first conduct extensive experiments on MPE, followed by selective validation on SMAC. The MPE ablation studies compare two baseline algorithms: FT-QMIX and FT-VDN. As shown in Figure [6]and and Figure 17 FT-VDN demonstrates slightly inferior performance to FT-QMIX across fixed and unfixed delay settings, particularly on REFERENCE. Notably, the RDC-enhanced FT-VDN maintains reasonable delay resistance despite lacking a critic network, confirming the framework's s compatibility with non-actor-critic approaches while preserving baseline performance characteristics. Architecture comparisons (Figure 16 19p reveal that GRU-based compensators generally underper- form Transformer variants, except 5m 6m. For brevity, Table 19 the less competitive FT-VDN and GRU-based results. In SMAC experiments. we employ curriculum learning for the actor to ensure convergence, discussed previously.
Table 8: Fixed delay test results of SMAC (rewards) with baseline algorithm FT-QMIX.
Fixed Delay Value
DR
H
KD
35 3s_vs_ Sz
Oraele
0.1
21.3
19.0 0.7
0.6
9.6
Base
0.5
2..
21.5
2.5
21.4
2..
21.8
Base
21.7
2.2
22.1
2.5
17.7
Base
0.2
0.2
21.
0.7
13.6
Flash
0.2
0.2
0.3
2.
19.3
Flash
2.2
2.14
2.2
21.8
19.7 0.6
Flash
213
Echo
2.2
2.2
21.
2..
17.9
0.8
20.5
Feho
21.0 0.1
0.1
2.12
21.3 0.1
21.3 1 0.4
DR
KD
5mLvs_6m
0.1
0.7
3.4
Base
0.2
0.2
8.3
8.3
8.0
Base
0.3
9.1004
9.2
8.0
8.4
Base
0.7
1.7
0.6
0.3
6.6
Base
1..0
15.8
12.0088.
9.4
Flash
13.7
Flash
18.5
18.1
1..
1..4
9.9
0.5
1.7
0.6
1.7
Echo
0.7
17.8
1.7
15.8
14.3
Fcho
0.7
1..6
16.3
1.88
11.9 11.9
Echo
0.7
0.5
0.8
0.9
Oraele
19.6
1.1
0.2
8.7 0.2
8
Base
0.3
0.2
11.0
Base
1.30
14.0 0.3
13.6
13.2
12.9 0.3
Base
0.3
11.903
Base
4
0.6
1..
1..5
12.8
11.5
Flash
1.3
1..5
16.0
1..3
11.9 0.3
Flash
0.3
0.56
1..
1..6
12.4
Flash
19
19.1
1..9
14.
13.0
Echo
1..
Echo
1.3
1..6
15.1
1..4
12.0 0.3
Echo
0.2
Table 9: Unfixed delay test results (rewards) SMAC with baseline algorithm FT-QMIX.
Delay Range
0-6
3-9
0-6
3.9
0-6
3.9
C. DR H KD
Sin, VS vs.
Gh.
Oracle
21.5 2: 0.4
17.2
5.5 1.5.1..3 0.2
3.4 0.1
11.9 1 0.4
9.0 0.2
Base
2: 0.4
21.4
1
12.9 4: 0.3
12.4
Base
21.7
0.2
22.1 0.4
10.7
0.9
9.1 D.4
12.3 0.3
11.4
Base
21.5 4: 0.2
21330..
17.9 0.8
14.6
1.64000 0.8
18.5
0.6
13.7
0.5
Flash
21.4
0.2
2.2
0.7
17.2
0.6
1.5
14.6
0.6
Flash
Flush
21.1
0.1
21.54
18.4 0.7
17.7 0.7
1811
15.9
0.5
Echo
2.1
21.
1..0
0.9
19.0
15.8 0.7
Echo
17.5
//-
20

---

TAG
SPREAD
240
-32
-20
200
Reward
180
Reward
Reward
-25
160
120
-40
-35
100
12
12
Delay
Delay
Delay
QMIXITF Echo)
QMIXITF Flash)
VONITF Echo)
VONITF Flash)
QMIXIGRU Echo)
QMIXIGRU Flash)
Figure 16: Performance comparison of RnCeenhadced algorithms with different baseline methods and compensator network architectures under fixed delay settings on MPE.
TAG
SPREAD
REFERENCE
240
-18
200
Reward
180
Reward
Reward
-25
160
140
-30
-42
3/9
Delay Range
Delay Range
Delay Range
QMIXITF Echo)
QMIXITF Flash)
VONITF Echo)
VONITF Flash)
QMIXIGRU Echo)
QMIX/GRU Flash)
Figure 17: Performance comparison of RDC-enhanced algorithms with different baseline methods and compensator network architectures under unfixed delay settings MPE.
3s. vs 5z
5m vs. 6m
6h_ vs 8z
80
Win Rate (96)
Win (%)
Win Rate (%))
60
60-
20
20-
20-
5
Delay
Delay
Delay
QMIXITF Echo)
QMIXITF Flash)
QMIXIGRU Echo)
QMIXIGRU Flash)
Figure 18: Performance comparison of RDC-enhanced algorithms with different compensator network architectures under fixed delay settings on SMAC.
21

---

VS _2
100
100
100
80
Win Rate (%6
Win Raie (%)
Win Rate (%)
40
40
40
20
Delay Range
Delay Range
Delay Range
QMIXITF Flash)
QMIXIGRU Echo)
Figure 19: Performance comparison of RDC-enhanced algorithms with different compensator network architectures under unfixed delay settings on SMAC.
TAG
SPREAD
REFERENCE
0.10
0.06
0.10
0.04
Value
0.06
0.06
0.03
0.04
0.02
0.01
0.02
0.00
Step
Step
Step
Flashi
Base
Figure 20: Observation loss curves on MPE with the baseline algorithm FT-QMIX and Transformer compensators.
2.00
x10
3s_vs_5z
2.00
x10
3m_vs_6m
1.0
x10-
6h_vs
1.75
175
1.50
1.50
Value
1.25
Value
Value
0.6
1.00
1.00
0.2
Step
1c6
Step
leB
Step
1eb
EDD
EchoeC-DR
CeH+DR
Flash+C+ DR
Figure 21: Observation loss curves on SMAC with the baseline algorithm FT-QMIX and Transformer compensators.
22

---

Figure 20 and Figure 21 iillsttrt the variation of compensator loss values with increasing training steps in partial experiments. In SMAC, the observation errors of Base consistently range between 0.1 and 0.2. To distinguish the performance of other algorithms clearly, we omitted the results of Base. The compensator provides substantial improvement in observation accuracy. Flash achieves signifi- cantly lower compensation errors when utilizing history inputs. On REFERENCE and 5m_vs 6m scenarios, Echo exhibits substantially higher compensation errors than Flash, a conclusion similarly reflected in non-fixed delay test results. Notably, algorithms employing knowledge distillation achieve superior performance without significant improvement in the compensator, suggesting that addressing delayed observation problems requires simultaneous consideration of both observation compensation and policy formation.
Our experiments are conducted on servers with NVIDIA A30 GPUs (approximately 24GB VRAM). The CPU model is generally unimportant as long as it supports more than eight threads we use both Intel(R) Xeon(R) Gold 6242R CPU @ 3. 3.10GHZ and Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz. For the baseline algorithm FT-QMIX with RDC enhancement implemented via Transformer compensator, Echo requires an average of 23.8 and 58.3 hours for training MPE and SMAC, respectively, while Flash needs 10.8 and 34.5 hours correspondingly. These time measurements include the teacher model's training duration.
E Limitations and Impacts
This paper first theoretically defines DSID-POMDP and proposes the RDC training framework based on the formation process of delayed observation within it. We then discuss the limitations of the proposed method in terms of theory, experimental results, and computational overhead.
A crucial assumption underlying DSID-POMDP is that each entity in the system only exposes its state to others. This means that every observation from other entities received by agent; contains independent and unique information. This assumption fails in environments where agents relay information through hopping. because when entity, information is delayed or missing, entityy might carry its information. While DSID- POMDP could be extended to accommodate such scenarios, we maintain that this would not represent the optimal approach for defining this class of problems.
On the experimental front, both Echo and Flash demonstrated poor generalizability on SMAC. As the delay value increased, the win rate declined rapidly. We believe this does not imply that agents inherently cannot achieve victory under such delayed conditions. The current compensator only processes information from the agent's 's own perspective and does not account for the influence of other agents on the environment. Better compensator designs and training techniques should effectively mitigate this issue.
The increase in resource overhead with the growing number of agents is a common challenge faced by MARL algorithms. In our experiments, we adopt the practice of having different agents share a single neural network, setup that is widely used under the CTDE framework. Therefore, although each agent has its own policy network and compensator in terms of structure, the network parameters of all the networks are actually the same during computation. The input to an agent is essentially the observation data, and as the number of agents and entities in the environment increases, the dimensionality of the input will inevitably increase. We believe that this linear increase in dimensionality is acceptable some extent. When the increase in observation dimensionality significantly impacts the model's inference speed, additional feature extractors may be needed to compress the information. In conclusion, the increase in the number of agents and entities in the environment will not result in corresponding increase in the number of networks, but rather an increase in the dimensionality of observations.
Despite the limitations above, this paper presents novel perspective delayed observation in multi-agent systems and provides a MARL-based solution. Researchers can utilize the RDC training framework in conjunction with baseline algorithms to address application problems across various domains, resulting in a positive societal impact. Our work carries no negative societal implications.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Both abstract and introduction demonstrate the contributions and scope of the paper accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Section 3 Section 5 and Section 6 we briefly discuss the limitations of
the current work and the assumptions of proposed theory. Appendix E provides more comprchensive explanation.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
24

---

Answer: [Yes]
Justification: We provide a complete proof the correctness of the defined DSID-POMDP in Appendix A and explicitly state the full set of assumptions following the definition.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our code is developed based the widely-used community project pymarl. Section 5 and Appendix (Ctixii(Cettii all hyperparameter settings and environmental configu- rations that may affect the experimental results. Additionally, we provide implementation specifics of delayed observations in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
25

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide anonymous link code in the abstract and have uploaded compressed file as supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppccc/pubccc/uuucceee Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section and Appendix C we provide detailed descriptions of the experi- mental setup and hyperparameter configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experimental results this paper include error bars or 95% confidence intervals. To improve visual clarity and conciseness, we apply smoothing to some curves in the figures without compromising the accuracy of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information about the computational resources required for the experiments in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc http:////eeeeeeecccppp////cceeee ittee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: We discuss the socictal impacts of the proposed work in the abstract, conclusion, and Appendix
Guidelines:
27

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the original papers of codes and benchmarks.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide details of new assets in the appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
30


Reviews:
[{"review_id": "qBSfWLMEOL", "reviewer": "Reviewer_Cv3z", "summary": "The authors propose Rainbow Delay Compensation (RDC), a Multi-Agent RL (MARL) framework that aims to address non-fixed, stochastic observation delays in Multi-Agent Systems (MAS). The authors present a modified version of Dec-POMDPS to handle this - decentralized stochastic individual delay partially observable Markov decision process (DSID-POMDP), which adds delay distributions to different agents' observations/entities. They implement these delays in SMAC and MPE, and show that standard value decompositions can struggle with these delays, while their proposed algorithm can better handle these delays.", "questions": "1. Why is addressing non-fixed, stochastic observation delay a critical problem in MARL beyond the general statement that \"delay issues are ubiquitous\"?\n2. Could you clarify why the RDC-enhanced methods appear to perform better than the delay-free \"Oracle\" baseline during training?\n3. Based on Figure 10, it appears the performance differences between several of the top RDC variants are not statistically significant. Does this suggest that the most complex configurations of RDC are not always necessary?", "limitations": "Apart from the mentioned limitations, the complexity of the full RDC method may pose challenges in practical applications requiring careful hyperparameter tuning and implementation, and likely requires significant additional computational resources.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nProblem:\n- Although there is existing work on delays in single-agent RL [1] and MARL with fixed delays [2], this appears to be one of the first works to model non-fixed, stochastic observation delays in MARL.\n\nMethod:\n- The DSID-POMDP Formalism appears to be a useful contribution, as it provides a way to model non-fixed observation delays in MARL.\n\n## Weaknesses\n\nProblem:\n- The motivation for considering observation delays in MARL could be more clearly articulated. From the introduction, it is not specifically clear why this is an important problem to address. The paper could benefit from a more detailed discussion of the challenges posed by observation delays in MARL and why an important problem to address (beyond saying \"delay issues are ubiquitous\"). \n\nEvaluation & Experiments:\n- The are known deficiencies in the SMAC v1 environments [1], These environments can be solved by simply observing the timesteps, which means this might not be a useful scenario to test observation delays, as agents have been shown to ignore observations in these environments. I would recommend considering the SMAC v2 environments [1], which have addressed these issues.\n- It is unclear why the oracle (no observation delays) baseline performs worse than the proposed method with delays in Figure 7. The explanation in lines 355-357 is not clear. This is unexpected and should be classified better.\n- SMAC experiments are typically run for 10 million timesteps, so it is unclear why Figure 7 stops at 6 million. \n\nMethod: \n- The method proposed has many components (a compensator, a delay-reconciled critic, the curriculum learning for actors, policy distillation), which makes it difficult to understand how each component contributes to the overall performance. Although the paper does provide some ablation studies, it would need to include variants such as the full RDC and no compensator or no distillation to better understand the impact of each component. \n- Following up from the previous point, the proposed method and framing was designed for non-fixed observation delays, but from figure 10, it seems the different components do not have a statistically significant impact in performance. \n\nRelated Work: \n- This work appears closely related to work in robust MARL, e.g. robustness to noise in observations (such as [3]). This field of MARL should be discussed.  \n\nReferences:\n\n1] Wang W, Han D, Luo X, Li D. Addressing signal delay in deep reinforcement learning. In The Twelfth International Conference on Learning Representations 2023 Oct.\n\n2] Chen B, Xu M, Liu Z, Li L, Zhao D. Delay-aware multi-agent reinforcement learning for cooperative and competitive environments. arXiv preprint arXiv:2005.05441. 2020 May 11.\n\n3] He S, Han S, Su S, Han S, Zou S, Miao F. Robust multi-agent reinforcement learning with state uncertainty. arXiv preprint arXiv:2307.16212. 2023 Jul 30.", "was_revised": true, "final_justification": "I believe this work is interesting and relevant, but the fact that the proposed method is trained for 10m more steps than the baselines feels like an unfair empirical advantage. \n\nI have seen and appreciate the author's new results on training for a similar training time. Unfortunately, I still feel the training steps concern is not fully addressed. Some of the new results help, but also some of it doesn't make sense, e.g. REFERENCE-Base scores move between -28.5 (16 m) and 29.2 (17m) and then -29.6 (similarly with REFERENCE-Base+DR). Although I can accept, maybe it is a mistype, e.g. not including the negative. Furthermore, this is only included on MPE environments, not SMAC, e.g. not in Fig. 7 results (I know doing experiments would take some more training time and might be tough to do over the rebuttal period). \n\nOverall, I do feel this work presents some interesting ideas, but my concerns are not fully addressed, therefore I still lean to a borderline reject.", "rebuttal": "&emsp;&emsp;Thank you very much for your careful reading of our paper and for raising many critical questions. We sincerely appreciate your valuable feedback and are committed to making the necessary revisions in the final submission. After reviewing your comments, we have summarized the identified weaknesses and questions, and we respond to each point in detail below. We genuinely hope that our responses will help resolve your concerns and that you will reconsider the value of our work. Should any further questions arise, we will address them promptly. All the data provided below were obtained using the same testing settings as described in the paper.\n\n**Weakness 1.1 & Question 1: Why is addressing non-fixed, stochastic observation delay a critical problem in MARL?**\n\n&emsp;&emsp;Your question is indeed important. In MARL, the evolving policies of other agents render the learning environment inherently non-stationary [1, 2]. Stochastic observation delays exacerbate this challenge, as agents must not only cope with non-stationarity but also account for the uncertainty introduced by varying delays. These delays also exacerbate the credit assignment problem [1]: misaligned observations and rewards make it more challenging to associate actions with outcomes, thereby hindering effective policy optimization. More fundamentally, stochastic delays violate the Markov assumption by providing agents with inaccurate perceptions of the current state. Moreover, stochastic delays have a greater impact than fixed ones. With fixed delays, agents can adapt and implicitly predict others’ observations over time, forming a kind of cognitive inertia. However, under stochastic delays, such predictions become unreliable, making it harder for actor networks to compensate. We address the importance of handling stochastic delays from four perspectives and will include a dedicated discussion in the Introduction of the final version.\n\n**Weakness 2.2 & Question 2: Could you clarify why the RDC-enhanced methods appear to perform better than the delay-free \"Oracle\" baseline during training?**\n\n&emsp;&emsp;We apologize for the confusion. This phenomenon is addressed in lines 358–362 of the paper. We believe the RDC-enhanced method outperforms Oracle in the no-delay scenario because it undergoes more training. As noted in line 679, each low-delay teacher model is trained for 1e7 steps before the student model continues training under knowledge distillation for the same number of steps. However, when comparing training curves and test results, we used the same number of steps for both Oracle and RDC student models, which inadvertently favored the latter. In theory, if the Oracle had fully converged, this discrepancy would not arise. However, it is unrealistic to assume convergence within a fixed training budget in practice. To balance performance and training cost, we limited training to what we considered sufficient to support our experimental conclusions.\n\n&emsp;&emsp;As shown in the paper, the RDC-enhanced method outperforms the no-delay Oracle, particularly in the SPREAD and REFERENCE scenarios. In both cases, the Oracle’s training curves were still trending upward at the cutoff point, which supports our statement: “*as evidenced by Oracle's ongoing performance gains at the end of training in both SPREAD and REFERENCE scenarios.*” To further address your concern, we provide Oracle's no-delay performance at different training steps for these scenarios. As expected, the Oracle outperformed our method with fewer training steps. This result further validates our previous conclusion. We will clarify this point more explicitly in the final version to help readers better interpret this seemingly \"unnatural\" result.\n||||||||\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Training Steps|5e6|6e6|7e6|8e6|9e6|**1e7**|  \n|SPREAD-Oracle|-33.1$\\pm$2.0|-32.0$\\pm$1.9|-31.1$\\pm$1.7|-29.2$\\pm$1.6|-29.8$\\pm$1.6|**-28.8$\\pm$2.0**|\n|Training Steps|1e7|1.1e7|1.2e |1.3e7|1.4e7|**1.5e7**|  \n|REFERENCE-Oracle|-17.6$\\pm$1.2|-17.1$\\pm$1.5|-17.3$\\pm$1.4|-17.0$\\pm$1.3|-16.6$\\pm$1.4|**-16.5$\\pm$1.2**|\n\n**Weakness 3.2 & Question 3: Does the lack of significant difference in Figure 10 suggest that the most complex RDC configurations may be unnecessary?**\n\n&emsp;&emsp;We appreciate your observation. Modularity is indeed a key principle of the RDC framework. In simpler scenarios, the presence or absence of certain modules may have limited impacts. For example, we did not apply curriculum learning to the actor in MPE tasks. As shown in Figure 10, history inputs have minimal effect in the TAG and REFERENCE scenarios, possibly indicating a weaker reliance on historical information in these cases. However, this does not diminish the value of the RDC framework. The influence of each module depends heavily on the task's underlying dynamics and is often difficult to assess before training is complete. The framework is designed to offer a flexible set of modular features, allowing users to easily enable or disable components through configuration.\n\n**Weakness 2.1: Why choose SMAC for evaluating observation delays instead of the improved SMACv2, which addresses known deficiencies?**\n\n&emsp;&emsp;SMACv2 [3] highlights two main issues with SMAC: (1) its environments are overly deterministic, enabling algorithms to \"memorize\" rather than \"learn\"; and (2) it lacks meaningful partial observability. Regarding the first point, we do not believe this undermines the value of using SMAC for MARL research. Notable studies such as QMIX and MAPPO achieved strong results on SMAC, validating it as a useful benchmark for assessing algorithm effectiveness. Moreover, our results suggest that SMAC does not completely ignore observational input—if it did, the yellow curves in Figures 6–9 would have matched ideal performance, which they clearly do not. As for the second point, our work focuses on handling observation delays, not on partial observability. Whether the agent's field of view is too large does not impact our conclusions.\n\n&emsp;&emsp;Additionally, SMACv2 significantly increases computational cost due to its longer training duration, and memory usage scales rapidly with the number of agents. While we acknowledge that SMACv2 is a more challenging benchmark, our aim is not to improve absolute performance but to enhance robustness to delayed observations. SMAC allows us to test this effectively while being more resource-efficient.\n\n\n**Weakness 2.3: SMAC experiments are typically run for 10 million timesteps, so it is unclear why Figure 7 stops at 6 million.**\n\n&emsp;&emsp;In all three SMAC scenarios of our paper, 6e6 timesteps are sufficient for model convergence. This choice helps minimize unnecessary computational overhead, allowing us to allocate more resources to the extensive ablation studies conducted in this work. Below, we report the performance of the Oracle model under varying training durations in the no-delay setting. While sparse sampling introduces some variance, the full convergence curves clearly show that the models converge by 6e6 steps across all scenarios. We will include these convergence curves and the rationale for selecting 6e6 timesteps in the revised version of the paper.\n|||||||||  \n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Training Steps|**6e6**|7e6|8e6|9e6|1e7|  \n|3s_vs_5z-Oracle|**99.7$\\pm$1.2**|99.8$\\pm$0.8|99.3$\\pm$1.3|99.9$\\pm$0.5|99.8$\\pm$0.8|\n|5m_vs_6m-Oracle|**84.8$\\pm$6.7**|84.8$\\pm$7.0|85.2$\\pm$6.6|84.4$\\pm$5.0|83.8$\\pm$6.9|\n|6h_vs_8z-Oracle|**92.0$\\pm$4.0**|90.4$\\pm$5.1|90.2$\\pm$6.1|93.4$\\pm$3.6|93.0$\\pm$4.6|\n\n\n**Weakness 3.1: It needs to include variants such as the full RDC and no compensator or no distillation to better understand the impact of each component.**\n\n&emsp;&emsp;Due to the page limitation, only the most important results are presented in the main text. The complete ablation study results, along with additional analyses, are provided in Appendix D. These include the results for the full RDC, as well as the no compensator and no distillation settings, as you mentioned. We believe our ablation study is already quite thorough, covering nearly all reasonable combinations of component usage.\n\n**Weakness 4.1: This work appears closely related to work in robust MARL, e.g. robustness to noise in observations (such as [4]). This field of MARL should be discussed.**\n\n&emsp;&emsp; You raise a very insightful point. Both our work and studies on robust MARL [4] address the challenge of non-ideal observations. However, they target fundamentally different issues. From a problem perspective, delayed observations concern when information is received—observations are accurate but arrive late. In contrast, robust MARL focuses on what is observed, often dealing with inaccuracies due to sensor noise or adversarial interference. Methodologically, our approach proactively reconstructs no-delay observations from delayed inputs. Robust MARL methods typically adapt passively to observation imperfections without attempting to reconstruct them. In summary, while both lines of research involve imperfect observations, they address distinct and non-overlapping challenges. We will incorporate relevant developments in this area into the Related Works section of our revised submission.\n\n**Reference**\n\n[1] Gronauer S, Diepold K. Multi-agent deep reinforcement learning: a survey[J]. Artificial Intelligence Review, 2022, 55(2): 895-943.\n\n[2] Zhang K, Yang Z, Başar T. Multi-agent reinforcement learning: A selective overview of theories and algorithms[J]. Handbook of reinforcement learning and control, 2021: 321-384.\n\n[3] Ellis B, Cook J, Moalla S, et al. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning[J]. Advances in Neural Information Processing Systems, 2023, 36: 37567-37593.\n\n[4] He S, Han S, Su S, et al. Robust multi-agent reinforcement learning with state uncertainty[J]. arXiv preprint arXiv:2307.16212, 2023."}, {"review_id": "R2w7u17511", "reviewer": "Reviewer_vxnb", "summary": "This paper addresses the challenge of observation delays in multi-agent systems. The authors extend the conventional Dec-POMDP framework into a decentralized stochastic individual delay partially observable Markov decision process (DSID-POMDP), which mathematically models multi-agent systems with observation delays. Within this framework, they introduce a technique called rainbow delay compensation (RDC) to handle stochastic individual delays. The proposed method demonstrates notable performance compared to the baseline algorithms under both fixed and unfixed delay settings.", "questions": "#### **Maximum delay**\n\n- Is it natural to assume that the existence of a maximum delay? In other words, how is the boundedness of the delay guaranteed?\n\n- In real world, since the distances between agents are not consistent, it seems likely that each agent may experience different maximum delay. Is the maximum delay applied uniformly to all agents?\n\n- In some cases, information may fail to reach the agent (e.g., packet loss). I wonder whether the proposed algorithm can still work properly under such scenarios.\n\n---\n\n#### **Delay distribution**\n\n- In the case of stochastic delays, the performance can vary significantly depending on the delay distribution (e.g., uniform or Poisson). The authors should clarify which delay distribution was used in the experiments and, if possible, briefly evaluate the performance under different distributions.\n\n- The proposed method exclusively focuses on addressing observation delay. However, in real-world environments, various types of delays—such as reward, execution, and inference delays—can occur simultaneously. It remains unclear whether the proposed approach would be effective when these different types of delays occur in combination.\n\n---\n\n####  **Performance of oracle**\n\n- The authors said that the oracle model represents delay-free performance. However, in figures 8 and 9, its performance appears to degrade as the delay increases. This discrepancy should be clarified to avoid unnecessary confusion.\n\n- I am curious how the proposed model outperforms the delay-free performance (oracle). If the proposed method truly performs effective delay compensation, shouldn’t its performance asymptotically converge to the delay-free performance? If the proposed method consistently outperforms the oracle, it may suggest that the improvement stems not purely from effective delay compensation, but rather from other techniques that enhance overall learning performance. It would be great to clarify this point, either through further explanation or additional experiments to support their claim.", "limitations": "The proposed method is limited to environments with observation delay only. Therefore, the reviewer cannot guarantee that it will perform equally well in the presence of other types of delays. Moreover, it is also unclear whether the method would maintain its performance in environments where multiple types of delays occur in combination.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "#### **Strengths**\n\n- The paper proposes a mathematical framework that extends delay-related problems previously discussed mainly in single-agent systems to multi-agent systems. As real-world applications of reinforcement learning (RL) continue to grow, handling delays in RL has become an increasingly relevant and practical issue. In this context, the paper focuses on an important and timely problem.\n\n#### **Weaknesses**\n\n- In real-world environments, multiple types of delays—such as observation, reward, execution, and inference delays—can occur simultaneously. The proposed method focuses exclusively on observation delay, raising concerns about whether it can still perform effectively in environments where various types of delays occur in combination.\n\n- The proposed approach, termed RDC, feels like an extension of existing techniques originally proposed for single-agent systems [1, 2, 3]. Because of this, the work may lack strong novelty.\n\n---\n\n[1] Firoiu, V., Ju, T., & Tenenbaum, J. (2018). At human speed: Deep reinforcement learning with action delay. arXiv preprint arXiv:1810.07286.\n\n[2] Wu, Q., Zhan, S. S., Wang, Y., Wang, Y., Lin, C. W., Lv, C., … & Huang, C. (2024). Variational delayed policy optimization. Advances in neural information processing systems, 37, 54330-54356.\n\n[3] Wang, W., Han, D., Luo, X., & Li, D. (2023, October). Addressing signal delay in deep reinforcement learning. In The Twelfth International Conference on Learning Representations.", "was_revised": true, "final_justification": "I have reviewed the responses, and will raise my score to 4.", "rebuttal": "&emsp;&emsp;We sincerely appreciate your thoughtful and constructive comments. From your review, we believe you have recognized the value of our work. We have carefully summarized the questions and weaknesses you raised and addressed each of them in detail below. We hope that our responses address your concerns and further highlight the strengths of our study. If possible, we will refine the relevant aspects in the final version based on your feedback. Please don't hesitate to reach out if you have any further questions—we would be happy to address them. All the data provided below follows the same testing settings as described in the main text.\n\n**Question 1.1: Is it natural to assume that the existence of a maximum delay? In other words, how is the boundedness of the delay guaranteed?**\n\n&emsp;&emsp;This is a very reasonable question and one we also considered during our research. We believe that assuming a maximum delay is practical in most real-world applications, as nearly all systems impose an upper bound on tolerable delays. Information exceeding this threshold is typically discarded, which not only ensures bounded delays but also prevents unnecessary resource consumption. As for infinite delays, they can effectively be treated as packet loss.\n\n&emsp;&emsp;You may also be concerned about how the maximum delay is determined. In most practical scenarios, delay fluctuations are relatively stable, allowing users to set a reasonable threshold to avoid excessive data loss. However, in cases where delays are highly variable and hard to estimate, our method may not be suitable.\n\n**Question 1.2: Is the maximum delay applied uniformly to all agents?**\n\n&emsp;&emsp;This is a very practical and relevant question. The RDC framework is specifically designed to handle scenarios where different entities experience different delay distributions, so we do not require a uniform maximum delay across all agents. Instead, it only needs the maximum delay that the system as a whole can tolerate. Importantly, setting this maximum delay does not reduce computational efficiency for agents with shorter delays. In our experiments, we chose, without loss of generality, to use a unified maximum delay across all entities to ensure the clarity and interpretability of the results.\n\n**Question 1.3: Can the proposed algorithm still work in the presence of packet loss?**\n\n&emsp;&emsp;Yes, in such cases, the algorithm still functions as expected. As discussed earlier, information that fails to reach an agent can be treated as having exceeded the maximum delay threshold or as effectively lost. Our experimental environment already includes this mechanism. For example, in step 7, agent A receives data from agent B, which originated from step 5, implying a 2-step delay. In step 8, if the delay exceeds 3, agent A simply uses the most recent available information as the current observation. In other words, the algorithm handles packet loss by consistently using the latest available data. That said, such events are relatively rare during training. In scenarios where packet loss is frequent or complete, the RDC framework would not be applicable, as it is not designed to recover information from absence. Nor is that the intended purpose of our method.\n\n**Question 2.1: Which delay distribution was used in the experiments? How does performance vary under different distributions?**\n\n&emsp;&emsp;Different delay distributions can indeed affect model performance. As stated in Appendix C, we used a uniform delay distribution in our experiments and provided the rationale. We apologize for not emphasizing this point more clearly, which may have led to your misunderstanding. Notably, this highlights a strength of the RDC-enhanced method: it does not require prior knowledge of the delay distribution, nor does it include a learning process to fit specific distributions. The compensator simply restores delayed information regardless of its distribution.\n\n&emsp;&emsp;Theoretically, different delay distributions affect performance by changing the likelihood of certain delay values occurring at different time steps. To further address your concern, we evaluated model performance under several common distributions with delay ranges of 3–9. All models were trained under uniform delay conditions. The results show that performance fluctuations remain within a controllable range. When a distribution's values align well with the delay range the model handles effectively, performance may improve relative to the uniform case. Conversely, if delay values fall outside the model's optimal range, performance may decline slightly.\n||||||  \n|:---:|:---:|:---:|:---:|:---:|\n|Distribution Types|Uniform|Bernoulli(0.5)|Normal(6, 2)|Poisson(6)|\n|TAG-RDC(Echo)|189.2$\\pm$22.3|184.1$\\pm$32.4|189.8$\\pm$29.8|191.3$\\pm$24.8|\n|SPREAD-RDC(Echo)|-31.4$\\pm$1.9|-31.8$\\pm$2.0|-32.0$\\pm$1.6|-32.1$\\pm$1.8|\n|REFERENCE-RDC(Echo)|-17.0$\\pm$1.2|-17.1$\\pm$1.3|-17.0$\\pm$1.3|-16.8$\\pm$1.2|\n\n**Weakness 1 & Question 2.2: Can the proposed method handle combinations of different delays, as seen in real-world scenarios?**\n\n&emsp;&emsp;Thank you for this insightful question. We acknowledge that different types of delays often coexist in real-world systems, but they typically fall under distinct research objectives. In the early stages of our work, we reviewed existing studies on observation delays [1], action delays [2], and reward delays [3]. Prior research [4] has shown that these delay types can be treated equivalently in single-agent settings, and we initially attempted to extend this idea to MARL. However, the results were not encouraging. In multi-agent systems, especially with the observation delays we focus on—more reflective of real-world settings—such equivalence breaks down. Action and inference delays from multiple agents jointly influence environment transitions and are tightly coupled with agents' delayed observations. Without strong independence assumptions, it is not possible to prove that different delay types are equivalent. Therefore, our method specifically targets observation delays in multi-agent settings and does not guarantee high performance when other delay types are present.\n\n**Question 3.1: Why does the oracle’s performance degrade with increasing delay, despite being described as delay-free performance?**\n\n&emsp;&emsp;The Oracle model is trained using a baseline algorithm (e.g., FT-QMIX) in a no-delay environment, and its performance under 0-delay testing reflects the ideal performance that delay compensation methods aim to achieve. While we considered plotting the Oracle's 0-delay performance as a horizontal line to serve as a reference baseline, we found this would not clearly illustrate how performance degrades under delay. We apologize for any confusion this may have caused and hope this explanation clarifies our design choice.\n\n**Question 3.2: Why does the proposed method outperform the delay-free oracle? Shouldn’t it at most converge to the oracle if it only compensates for delays?**\n\n&emsp;&emsp;We provided an explanation for this phenomenon in lines 358–362 of the paper. First, regardless of algorithmic improvements, the performance after delay compensation should theoretically be bounded above by the performance of the no-delay Oracle. In our paper, the observation that the RDC-enhanced method outperforms the ideal Oracle in some scenarios is attributed to the implicit effect of additional training steps. Although the student model undergoes the same amount of training as the Oracle, the extra guidance it receives from the teacher model through knowledge distillation is difficult to quantify. Moreover, another key factor is that the Oracle did not fully converge. In the two scenarios where this phenomenon is most evident—SPREAD and REFERENCE—the Oracle's training curves still show a slight upward trend toward the end. We provide Oracle’s performance with additional training steps in the table below. As shown, the Oracle eventually surpasses the performance of our method with continued training. However, to maintain consistency across all plots in the main text, we have to limit the Oracle's results to the corresponding number of environment steps. We will include a more detailed discussion of this phenomenon in the revised submission.\n||||||||\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Training Steps|5e6|6e6|7e6|8e6|9e6|**1e7**|  \n|SPREAD-Oracle|-33.1$\\pm$2.0|-32.0$\\pm$1.9|-31.1$\\pm$1.7|-29.2$\\pm$1.6|-29.8$\\pm$1.6|**-28.8$\\pm$2.0**|\n|Training Steps|1e7|1.1e7|1.2e |1.3e7|1.4e7|**1.5e7**|  \n|REFERENCE-Oracle|-17.6$\\pm$1.2|-17.1$\\pm$1.5|-17.3$\\pm$1.4|-17.0$\\pm$1.3|-16.6$\\pm$1.4|**-16.5$\\pm$1.2**|\n\n**Weakness 2: The proposed approach feels like an extension of existing techniques originally proposed for single-agent systems.**\n\n&emsp;&emsp;Our core contribution lies in identifying the previously overlooked mechanism of observation delay in multi-agent systems. Based on this insight, we designed a more accurate and robust compensator architecture. In addition, the use of various curriculum learning strategies also effectively addresses challenges encountered during the training process.\n\n**Reference**\n\n[1] Chen B, Xu M, Liu Z, et al. Delay-aware multi-agent reinforcement learning for cooperative and competitive environments[J]. arXiv preprint arXiv:2005.05441, 2020.\n\n[2] Wang F, Zhang H, Zhang Y. Resolving action delay: Multi-agent reinforcement learning based on state prediction[C]//Chinese Intelligent Systems Conference. Singapore: Springer Nature Singapore, 2024: 552-563.\n\n[3] Zhang Y, Zhang R, Gu Y, et al. Multi-agent reinforcement learning with reward delays[C]//Learning for Dynamics and Control Conference. PMLR, 2023: 692-704.\n\n[4] Wang W, Han D, Luo X, et al. Addressing signal delay in deep reinforcement learning[C]//The Twelfth International Conference on Learning Representations. 2023."}, {"review_id": "qISu1QkRLO", "reviewer": "Reviewer_Q4bC", "summary": "This paper tackles a pressing challenge in MARL: handling heterogeneous and stochastic observation delays. The authors introduce both a novel problem formulation and a robust solution framework, making the following core contributions:\n\n1. The authors formalize the setting of stochastic, component-wise observation delays through the Decentralized Stochastic Individual Delay Partially Observable Markov Decision Process (DSID-POMDP). This extends the classical Dec-POMDP framework to realistically capture delay dynamics where different parts of an agent’s observation may arrive at varying times.\n\n2. To address these complex delays, the authors propose RDC, a modular and general-purpose framework comprising four key components:\n(a) Compensator: Learns to reconstruct delay-free observations using past delayed observations and actions. Two designs are presented: Flash (one-shot reconstruction) and Echo (autoregressive modeling).\n(b) Delay-Reconciled Critic: Incorporates delay-free ground-truth states during centralized training to improve stability and learning quality.\n(c) Actor Curriculum: Gradually shifts the actor's input from ground-truth to reconstructed observations to ease the transition and stabilize early training phases.\n(d) KD: Leverages a teacher policy trained in a low-delay environment to guide the student in the high-delay setting.\n\n3. Through comprehensive experiments on established MARL environments like MPE and SMAC, under both deterministic and stochastic delay settings, the authors show that traditional MARL methods degrade significantly in delayed settings. In contrast, RDC-based models consistently reduce the impact of delays and, in several cases, approach the performance of an idealized oracle with perfect, delay-free observations.", "questions": "1. Following up on W1, what is your primary hypothesis for the compensator's reduced effectiveness on SMAC compared to MPE as delay increases? Does this suggest a need for more advanced sequence prediction models, or perhaps incorporating some form of shared environmental model into the compensation process?\n\n2. In your knowledge distillation experiments, you used a teacher trained on a low-delay setting (delay 1-3) rather than a zero-delay Oracle. You mention that using the Oracle directly as a teacher was unsuccessful. Could you please elaborate on this negative result? What was the failure mode (e.g., training instability, policy divergence)? This is a very interesting practical finding.\n\n3. Have you considered providing the actual delay values $d^{ij}$ as an additional input to the compensator during training, if they were available? Do you think this would simplify the learning problem and improve performance, or is it better for the model to infer delays from the data itself?", "limitations": "Please refer to the section on Weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper tackles a problem that is both highly relevant to real-world applications (e.g., robotics, communication networks) and relatively under-explored in the MARL literature. The focus on stochastic, individual component delays is a crucial step beyond prior work that often assumes fixed or uniform delays for the entire observation vector. This finer-grained problem definition is much more realistic.\n\n2. The introduction of the DSID-POMDP is a standout contribution. It provides a clean and formal mathematical foundation for this complex delay problem. The formulation is intuitive and appears correct, serving as a solid theoretical underpinning for the algorithmic solutions that follow.\n\n3. The RDC framework is a well-designed, multi-faceted solution where each component addresses a specific challenge.\n\n4. The paper's motivation is clear, the methods are explained logically, and the figures (especially Figures 2 and 3) are highly effective at illustrating the framework and the core problem. The authors have also provided the source code for reproducibility.\n\n5. Experimental benchmarking is good, and includes necessary ablations and generalizations.\n\n\nWeaknesses:\n\n1. Generalization Gap on More Complex Tasks: While the results are impressive, there is a noticeable performance gap on the more complex SMAC scenarios compared to MPE. As shown in Figure 9, while RDC significantly improves upon the baselines, the win rate still drops sharply with increasing delay (e.g., in 6h_vs_8z). The paper acknowledges this in Appendix E, but a deeper discussion in the main text about why this might be the case would be beneficial. Is the compensator's task simply too hard with high-dimensional observation/action spaces? Are small prediction errors more catastrophic in strategically brittle scenarios?\n\n2. The DSID-POMDP formalism and the RDC framework implicitly assume that the information from each entity $j$ is unique. As the authors correctly point out in the Limitations (Appendix E), this assumption would be violated in scenarios with information relay (e.g., agent-$k$'s observation contains information about agent-$j$). While extending the model is noted as future work, this is a key assumption that limits the applicability of the current theory to a specific class of MAS.\n\n3. The current approach trains a separate compensator for each agent. While this is standard in MARL, the input to the compensator includes a history of observations from multiple entities (allies and enemies). The paper could be strengthened by a discussion of the computational and sample complexity as the number of agents and entities grows. How does the input dimension and sequence length scale, and what are the potential bottlenecks for applying RDC to systems with dozens or hundreds of agents?", "was_revised": true, "final_justification": "I have reviewed the responses from the authors, and will stick to my original score.", "rebuttal": "&emsp;&emsp;We are honored that you took the time to carefully review our work and provided positive feedback. To better address your questions, we have summarized the weaknesses and questions you pointed out and responded to each of them in detail. We hope this will help clarify any confusion. Thank you once again for your recognition of our work, and we warmly welcome any further questions or suggestions you may have.\n\n**Weakness 1 & Question 1: Following up on W1, what is your primary hypothesis for the compensator's reduced effectiveness on SMAC compared to MPE as delay increases? Does this suggest a need for more advanced sequence prediction models, or perhaps incorporating some form of shared environmental model into the compensation process?**\n\n&emsp;&emsp;You are absolutely right — SMAC tasks indeed have a more complex state space compared to MPE, and under such conditions, even the same level of error can lead to more severe consequences. However, there are many tasks whose complexity far exceeds that of SMAC, which is one of the key motivations behind our proposal of the RDC framework. Rather than aiming for absolute performance superiority, our goal is to provide a structural guideline for future research in this field. The modular design of the RDC framework also allows researchers to easily replace components with more advanced sequence prediction models or world models. When dealing with more complex tasks, such improved models will indeed become essential for effectively handling delayed observations.\n\n**Question 2: In your knowledge distillation experiments, you used a teacher trained on a low-delay setting (delay 1-3) rather than a zero-delay Oracle. You mention that using the Oracle directly as a teacher was unsuccessful. Could you please elaborate on this negative result? What was the failure mode (e.g., training instability, policy divergence)? This is a very interesting practical finding.**\n\n&emsp;&emsp;Thank you for taking the time to carefully read our paper. In our initial plan, we did use Oracle as the teacher model for the delay strategy. However, during actual training, we observed that the model's performance began to decline as knowledge distillation started. In subsequent exploration, we were pleasantly surprised to find that using a low-delay teacher model yielded excellent results. Therefore, we believe that the policy discrepancy is the main reason for the failure of the Oracle teacher model. The experimental results show that the policy differences between the no-delay Oracle strategy and the RDC-enhanced method under delayed observations are significant in the early stages of training. During the knowledge distillation process, the student strategy gradually changes. The larger policy discrepancies may lead to the collapse of the student strategy.\n\n**Question 3: Have you considered providing the actual delay values d^ij as an additional input to the compensator during training, if they were available? Do you think this would simplify the learning problem and improve performance, or is it better for the model to infer delays from the data itself?**\n\n&emsp;&emsp;Your question is very precise. In fact, the actual delay value is part of the compensator's input. We believe this setup can be applied to most multi-agent systems, as delayed information can be accompanied by a timestamp, allowing the receiver to compute the delay value. From a model perspective, this is essential information for completing the prediction. Without the delay value input, we cannot inform the model of the amount of compensation required for different pieces of information. Perhaps in a system with fixed delays, this method could be feasible.\n\n**Weakness 2: The DSID-POMDP formalism and the RDC framework implicitly assume that the information from each entity j is unique. As the authors correctly point out in the Limitations (Appendix E), this assumption would be violated in scenarios with information relay (e.g., agent-k's observation contains information about agent-j). While extending the model is noted as future work, this is a key assumption that limits the applicability of the current theory to a specific class of MAS.**\n\n&emsp;&emsp;We have acknowledged this limitation in our paper. The current theoretical framework does not yet extend to multi-agent systems that involve information forwarding or similar mechanisms. It is foreseeable that, in such scenarios, the observations received by each agent will become more complex. However, this does not make the problem intractable. With appropriate information filtering and simple probabilistic aggregation, it may be possible to extend the theory accordingly. Moreover, the RDC-enhanced method may remain applicable in these cases.\n\n**Weakness 3: The current approach trains a separate compensator for each agent. While this is standard in MARL, the input to the compensator includes a history of observations from multiple entities (allies and enemies). The paper could be strengthened by a discussion of the computational and sample complexity as the number of agents and entities grows. How does the input dimension and sequence length scale, and what are the potential bottlenecks for applying RDC to systems with dozens or hundreds of agents?**\n\n&emsp;&emsp;This is actually a common issue faced by multi-agent reinforcement learning (MARL) algorithms. However, it is essential to clarify that in our experiments, we employed a shared network setup, which is commonly used in the Centralized Training with Decentralized Execution (CTDE) framework for MARL algorithms. Therefore, although each agent has its own compensator in terms of structure, the network parameters of all the compensators are actually the same during computation. The input to the compensator is essentially the observation data, and as the number of agents and entities in the environment increases, the dimensionality of the input will inevitably increase. We believe that this linear increase in dimensionality is acceptable to some extent. When the increase in observation dimensionality significantly impacts the model’s inference speed, additional feature extractors may be needed to compress the information. In conclusion, the increase in the number of agents and entities in the environment will not result in a corresponding increase in the number of compensators, but rather an increase in the dimensionality of observations. If possible, we will add a discussion on this issue in the final submission."}]

Author rebuttal:

Meta-review: The authors propose a framework for MARL with observation delays. The reviewers appreciated that this could be a step towards making MARL more applicable in real world scenarios, though also noted that it addresses just one of many possible issues. (I'm don't think that's a fatal criticism if the basic contribution is still worthwhile.) 

To address the delay issue, the authors separately train a compensation mechanism, which seems to help. This is good, but there was some concern among at least one reviewer on how the training time for the compensator should be counted. This is not an unreasonable concern because the use of an extra mechanism with its own training regime does change things, and there could be reasonable disagreements about how his training should be reported and counted.

Perhaps the best thing is to aim to be as clear and explicit as possible to avoid misunderstandings. Overall, I am leaning positive on this paper, with the hope that the authors will take the back and forth with reviewer C3vz into account in any revised version to minimize any possibility of confusion/frustration/disappointment in how future readers will interpret the results.
Decision: Accept (poster)