Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
Authors: Whiyoung Jung
Sunghoon Hong
Deunsol Yoon
Kanghoon Lee
Woohyung Lim
Venue: ICML 2025 poster (2025)
Keywords: Multi-Agent Reinforcement Learning
Asynchronous Multi-Agent Reinforcement Learning
MacDec-POMDP
Primary area: reinforcement_learning->multiagent
TL;DR: We propose ACAC, an algorithm designed to enhance learning efficiency in asynchronous MARL by eliminating padding, leading to faster convergence and improved performance.

Abstract: Multi-Agent Reinforcement Learning (MARL) struggles with coordination in sparse reward environments. Macro-actions —sequences of actions executed as single decisions— facilitate long-term planning but introduce asynchrony, complicating Centralized Training with Decentralized Execution (CTDE). Existing CTDE methods use padding to handle asynchrony, risking misaligned asynchronous experiences and spurious correlations. We propose the Agent-Centric Actor-Critic (ACAC) algorithm to manage asynchrony without padding. ACAC uses agent-centric encoders for independent trajectory processing, with an attention-based aggregation module integrating these histories into a centralized critic for improved temporal abstractions. The proposed structure is trained via a PPO-based algorithm with a modified Generalized Advantage Estimation for asynchronous environments. Experiments show ACAC accelerates convergence and enhances performance over baselines in complex MARL tasks.

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement
Learning
Whiyoung Jung"
!
Sunghoon Hong"
Deunsol Yoon
Kanghoon Lee
Woohyung Lim
Abstract
Overcooked
Multi-Agent Reinforcement Learning (MARL. struggles with coordination sparse reward envi- ronments. Macro-actions sequeecces of actions executed single decisions facilitate long- term planning but introduce asynchrony, compli- cating Centralized Training with Decentralized Execution (CTDE). Existing CTDE methods use padding to handle asynchrony, risking misaligned asynchronous experiences and spurious correla- tions. We propose the Agent-Centric Actor-Critic (ACAC) algorithm to manage asynchrony with- out padding. ACAC uses agent-centric encoders for independent trajectory processing, with an attention-based aggregation module integrating these histories into centralized critic for im- proved temporal abstractions. The proposed struc- is trained via PPO-based algorithm with modified Generalized Advantage Estimation for asynchronous environments. Experiments show ACAC accelerates convergence and enhances per- formance over baselines complex MARL tasks.
200
Return
AGAC (macro)
100
(micro)
4
8
12
16 20
Environment steps (1M)
Figure I. (Left) An example of Overcooked problem. (Right) Com-
parison of the learning efficiency when using micro-actions (dot- ted) versus macro-actions (solid).
Go to Tomato
Go to Knife 1
Chop
Go to Lettuce
Go to Knife 2
Chop
Onion
Go Plate
Timesteps
|
Observation & Decision Time
Macro-action
Figure 2. An example trajectory of asynchronous decisions.
treat a well-defined sequence of actions' -referred as macro-action-as as single high-level decision (Xiao et al., 2020a;b). By selecting macro-actions, agents adopt long- term strategies that significantly enhance learning efficiency compared to relying solely on low-level )actroo) An illustrative example of such an environment is the Over- cooked problem, shown in the left side Figure 1. this setting, macro-actions like Go TO TOMATO each comprise sequence of micro-actions aimed completing sub- task. As depicted in the right side of Figure 1, using these macro-actions can significantly enhance learning efficiency.
1. Introduction
Multi-Agent Reinforcement Learning (MARL) has achieved significant progress (Omidshafiei et al., 2017; Foerster et al., 2018; Lowe et al., 2017; Rashid et al., 2018; Yu et al., 2022: Kuba et al., 2022), enabling distributed agents to learn coordinated strategies efficiently. Despite these advances, many real-world tasks remain notoriously challenging due to sparse rewards, where feedback signals are infrequent and less informative. In such environments, agents struggle to infer how their individual actions influence collective performance, often requiring extensive exploration.
However, employing macro-actions also causes the agents' action-selection selection timesteps to become asynchronous, as shown in Figure 2. this figure, for instance, when the blue agent selects Go TO TOMATO, spends four timesteps exe- cuting the macro-action before finally acquiring the tomato. After finishing macro-action, each agent receives new
One promising approach to address sparse rewards is to
'quual contribution 'LG Al Research, Scoul, Republic Korea. Correspondence to: Woohyung Lim <w.lim@ cw.lim@lgresearch..i.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Macro-actions not fixed sequences of micro-actions; Section 2.3 further details.

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
observation and chooses its macro-action accordingly. Because macro-action durations vary across agents, the act times when they complete their actions-and thus when they select new ones differ as well. This variability in execution times underlies the asynchronous nature of the MARL environment.
V/
1
Centralized Training with Decentralized Execution (CTDE) widely adopted framework for efficient learning in syn- chronous MARL environments. While CTDE can also be considered enhance learning efficiency in asynchronous MARL settings, its straightforward application becomes infeasible. In asynchronous environments, observation and macro-action information are available only for subsets of agents any given timestep. Consequently, existing CTDE methods designed for synchronous MARL cannot be di- rectly applied. To address this limitation, current approaches (Xiao et al., 2020a;b) typically pad the missing information for agents during centralized training using their most re- cent observations. However, as highlighted by (Liang et al., 2022), this padding technique can result in the abstraction of misleading information, potentially undermining the ef- fectiveness of the learning process.
(a) With original macro-ohservations
V/
V/2
V/
V/
Orkii
x
pedded
pacded
2
To address this challenge, propose Agent- Centric Actor-Critic (ACAC) algorithm. ACAC leverages agent- centric history encoders to process each agent's 's trajectory individually, followed by an attention-based mechanism that integrates these per-agent histories into a centralized critic. This design eliminates the redundancy inherent in padding-basee methods, leading to more accurate temporal abstractions, faster learning, and higher-quality policies. To train the centralized critic and decentralized actors, we intro- duce a PPO-based algorithm incorporating a modified GAE suited for asynchronous settings. We evaluate ACAC on sev- eral macro-action. based multi-agent benchmarks, where ex- perimental results show that it not only accelerates learning convergence under sparse rewards but also achieves higher final returns than conventional padding- based approaches.
podied
pudled
(b) With padded macro-observations
Figure Centralized critics for the example in Figure
act synchronously by selecting and executing their actions simultaneously at each discrete time step. The objective to maximize shared cumulative reward.
2.2. Asynchronous Multi-Agent RL Settings
In asynchronous settings, agents operate asynchronously by selecting and executing macro-actions that may vary in duration, enabling flexible and scalable coordination among agents. This framework can be modeled as a Macro-Action Decentralized Partially Observable Markov Decision Pro- (MacDec-POMDP) (Amato et al., 2014).
2. Background
2.1. Synchronous Multi-Agent RL Settings
Cooperative Multi-Agent Reinforcement Learning is gen- erally formulated as Decentralized Partially Observable Markov Decision Process (Dec-POMDP) (Bernstein al., 2002: Omidshafiei et al., 2015). Dec-POMDP formally defined tuple D (I, A, Ω2 T, R, O,r), where I represents the set of N agents, S denotes the environmental state space. and A= = X AA' is the joint action space of the agents. Ω = x f f is the joint observation space, T spec- ifies the state transition probabilities, and is the global reward function. O defines the observation function, and the discount factor. In this synchronous setting, all agents
A MacDec-POMDP is defined as a tuple D' (D. Here, .M = XsezM' represents the joint macro-action space, where M is the macro-action space for each agent i, and m² E M denotes a macro-action chosen by agent i. Similarly, < = XiEIς" denotes the joint macro-observation space, where is the macro-observation space for each agent x' E E(' represents the macro cosecrotbseeviioon of agent i, and ă E Ç is the aggregated joint macro-observation across all agents. For the full notation for the MacDec-POMDP can be found Appendix B.
In a MacDec-POMMD, each agent selects a macro-action m
2

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
based its current macro-observation z². Once macro- action m' is chosen, the sequence of micro-actions that constitute it is executed in order. Upon completion of these micro-actions, the macro-action m' terminates. At this point, the agent receives both a new macro-observation "eex and select a new macro-action mexi based on the up- dated macro-observation >neex-- This cycle continues until the episode concludes. The objective of MacDee-POMMDD is identical to that of Dec-POMDP,, but it incorporates macro- observations and macro-actions to facilitate higher-level decision-making and improve learning efficiency.
Specifically, the padding-based method does not distinguish whether the newly obtained information is identical to pre- vious information merely reuses past macro-observations, As a result, both the presence of new information and the duration information derived from previous data become inaccurate, thereby reducing the accuracy of history abstrac- tion. This lack of distinction compronises the centralized critic's ability to accurately interpret the agents' historical antronactions and macro-observatiios, further diminishing the overall learning effectiveness in asynchronous MARL environments.
2.3. Macro-Actions in MacDec-POMDP
3. Method
Within MacDec-POMDD macro-actions are modeled as Decision Process framework
3.1. Review of CTDE Framework for Asyne. MARL
options in the (Sutton et al., 1999). Each macro-action is defined as m = (Am, Im Bm)) where Tm is an intra-option policy over micro-actions given a history of micro-observations, Im the initiation set from which the macro-action can start, and im is the termination condition, giving the prob- ability that the macro-action ends given the current micro- observation history. The design of these intra-option policies and termination conditions enables inter-agent interaction and adaptation. This careful formulation allows cooperative behaviors to emerge naturally within the MacDec-POMDP framework.
Centralized Critic. A centralized critic in the CTDE frame- work is usually trained using the combined histories of all agents. At each timestep f, the observation history of the i-th agent is denoted as:
251 = 2 o is available u E
where 25g2 represents the sequence of all observations of agent up to timestep f. The critic takes as input the set of all agents' histories and predict the value timestep t as:
Volăg)) where ăge= = f2
In asynchronous settings, not all agents can obtain macro- observations z at every timestep t. To address this, missing macro-observations replaced with the latest valid macro- observations, 2i,taees when constructing the history:
2.4. Extending Sync. MARL to Async. MARL
Centralized Training with Decentralized Execution (CTDE) a widely adopted framework for efficient learning in syn- chronous MARL environments. CTDE leverages central- ized critic to estimate the joint value function for all agents, facilitating coordinated policy development. While CTDE can also be considered to enhance learning efficiency in asynchronous MARL settings, its straightforward applica- tion becomes infeasible.
( ( 2f)
if 2f is available,
25:
(= :..ae)) if at unavailable.
This mechanism ensures that the input to the critic &ge remains consistent across timesteps, even when macro- observations missing. By leveraging the latest valid macro-observations,, the centralized critic adapts effectively to asynchronous environments while maintaining compati- bility with the training process.
In asynchronous environments, macro-observation infor- mation available only for subsets of agents any given timestep. Consequently, existing CTDE methods designed for synchronous MARL. cannot be directly applied because the centralized critic requires joint macro-observations of varying sizes, as illustrated in Figure 3 (a). To address this limitation, current approaches (Xiao et al., 2020a;b) typ- ically pad the missing information for agents using their most recent macro-observations, as shown in Figure (b). This padding strategy represents the simplest way to apply synchronous MARL methods to asynchronous CTDE with- out considering specialized algorithmic or network structure modifications. However, as highlighted by (Liang et al., 2022), this padding technique can lead to the abstraction of misleading information, potentially undermining the effec- tiveness of the learning process.
Decentralized Actor. agent independently receives its local -osrr-obbeevvttion and selects macro-action based on its current local macro- observation and past history. Unlike the critic, there is structural difference between synchronous and asyn- chronous settings, as each agent operates independently. The macro-action m for the i-th agent at timestep is sam- pled as follows:
In decentralized execution, each
mi TT. 2g)))
Actor-Critic Algorithm for Asynchronous MARL. We employ an actor-critic algorithm to train decentralized ac- tors, each parameterized by 0', alongside a centralized critic
3

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
V
Macro-action
Macro-value
MLP
Agent Cantric
Agent Centric
A
Decentralized Actor
Centralized Critic
A
Avetage Pooiin
New Hidden State
Latest Hidden State
Latest Hidden State
CRU
Update
hiddnn state
Update
MLP
Sell-Attetion
done.
Chapping
Gaing to Onion
What's
Time
62
A
A
Embudding
Agent-Centric
Encoder
Enooder
New observation
Executing Macro-actions
Figure 5. Illustration of the agent-centric actor-critic.
These agent-centric histories are then aggregated using an attention based module, resulting in more accurate joint history representation for value estimation. By focusing encoding each agent's 's history before combining them, approach effectively handles the lack of macro-observations typical in asynchronous MARL and mitigates the limitations of padding- based methods.
Figure 4, Overall structure of the agent-centric centralized critic: (Left) Agent-centric encoder, and (Right) Centralized critic inte- grating agent-centric histories.
parameterized by th. The training procedure is outlined follows:
[(1)= = F, { 7" rr 'm - (ESS))
)
27
A Centric Encoders. Each agent's history can be abstracted from its macro-observations using a recurrent network. However, in asynchronous MARL, is essential to account for the time elapsed between consecutive macro- observations. Without this duration information, identical consecutive macro-observations lack context about whether the interval between them is one timestep or ten, which can lead to unstable or inaccurate learning of each agent's history due to varying durations.
((A') EE, llog Th. (ml Ac)
where r sampled trajectories using the current actors and Tr. denotes the reward at timestep t. The advantage function
A, is defined as:
frest
A: =
Σ tr + gome (āStaa)) Vis (Zge). Nom.
To address this, propose agent-centric encoder that integrates both the macro-observatte 2/ and the correspond- ing timestep p. = t of each macro-observation. In the proposed encoder, the timestep information pt is embed- ded using sinusoidal positional encoding (Vaswani et al., 2017). This embedded timestep is then concatenated with the macro--bberveee
where frex is the next timestep at which new macro- observation for any agent becomes available.
3.2. Agent-Centric Centralized Critic
The combined information pro- cessed through multilayer perceptron (MIP) and gated recurrent unit (GRU) to effectively abstract each agent's s history.
In synchronous MARL, joint history abstraction is typically achieved by collecting joint observations at each timestep. However, we argue that this method does not align well with asynchronous MARL settings, where only a subset of agents may provide observations at any given timestep. Although padding can be employed to fill missing observations, this approach often introduces redundant information and leads to inaccuracies in the joint history representation.
By incorporating timestep information På alongside macro- observation 2f, the agent-centric encoder ensures that each agent's history is learned accurately and consistently, even in asynchronous MARL settings. The left side Figure illustrates the structure of the agent-centric encoder.
To address these issues, we propose method that first con- structs each agent's history on a per-agent basis and then integrates these agent-specific histories into a joint represen- tation. Specifically, we introduce an agent-centric history encoder to capture each agent's macro-observation history.
Centralized Critic. The proposed agent-centric centralized critic consists of two main components: an agent-centric encoder that abstracts each agent's 's history and an attention- based aggregation module that combines these abstracted histories.

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
The components and overall structure of the agent-centric centralized critic are illustrated in Figure 4. The centralized critic at timestep is computed as follows:
macro-observations and their corresponding timesteps up to timestep f. Here, A denotes the advantage estimate timestep t using Generalized Advantage Estimation (GAE)
(Schulman et al., 2016) with a discounting factor X, and c is the clipping ratio.
h En (2st)
h4 = { h4. .i.................. ... h²]
For the centralized critic, the loss function is defined as:
Ve (Ite) = fo (Aggr-Module, (ire)
E. ( [( (
Vo (ho)))))
where p/z {pr z is available, S I is the timestep information of macro-observations of agent until timestep t.
where RG is the return timestep & computed by using GAE. We also adopt value normalization technique, named PopArt (Hessel et al., 2019), which was used in MAPPO (Yu et al., 2022).
Figure 5 provides an example of its input and output at timestep t of the agent-centric centralized critic and actor. In this example, the blue agent (on the left) can obtain a new macro-observation and is able select new macro- action. In contrast, the green (in the middle) and the red (on the right) agents cannot obtain new macro-observation be- cause their previous macro-action is still in progress. In such cases, agents that have acquired a new macro-observation input their macro-observations into their respective agent- centric encoders to generate updated histories. Conversely, agents that have not obtained new macro-observation use their existing history input to the aggregation module. This approach allows the centralized critic to accurately estimate the value by leveraging the most recent and rele- vant historical information from each agent, even in asyn- chronous MARL settings. We provide detailed comparison between padding-based methods and ACAC Appendix C.
3.4. Generalized Advantage Estimation (GAE) for Asynchronous MARL
Generalized Advantage Estimation (GAE) (Schulman et al., 2016) is common technique for estimating advantage func- tions, balancing the high variance of empirical returns with the high bias of temporal difference estimates through a hyperparameter X. Specifically, À = 0 relies on single-step temporal difference (TD) estimates, while À uses the full empirical return minus the baseline. Intermediate values of À provide controlled balance between these extremes.
In the original GAE, X-discounting is typically applied at the phicro-timester level, reflecting the fixed intervals between micro-actions in synchronous environments. However, ap- plying this standard formulation directly to asynchronous MARL environments, particularly those employing tem- poral abstraction via macro-actions (c.g., in MacDec- POMDP setting), presents a significant challenge. The in- tervals between consecutive macro- caecro-bbervvtions can vary substantially. This makes traditional micro-timestep-base discounting unsuitable because it doesn't 't align with the varying durations of incro-actions. Specifically, as a macro- action becomes longer, this discounting method applies greater discount to its future rewards, thereby diminishing the perceived importance of that macro-action decision.
3.3. Agent-Centric Actor-Critic Algorithm
We introduce the Agent-Centric Actor-Critic (ACAC), a PPO-based MARL algorithm that utilizes an agent-centric centralized critic. In this section, we provide a comprehen- sive description of the ACAC algorithm.
First, leveraging the agent-centric encoder described previ- ously, we construct each agent's 's actor, denoted as Ty. (- ht). Each actor is trained using a loss function based on the es- timated advantage, similar to the actor loss employed in MAPPO (Yu et al., 2022). The loss function is defined as:
Proposed GAE for Asynchronous Settings. To address the challenges of asynchronous environments, we propose a modified GAE approach. Our key insight lies in shift- ing the X-discounting from the micro-timestep level the macro-tinee ticeo-t level. This means that future TD errors are discounted based on the number of macro-action decision steps taken, rather than the timesteps elapsed.
(((') = E, [min (surr,, surre)]
surr; A
surt2 clip (pi(0"), - +e) A
where the importance sampling ratio is given by
Consider an example where macro-observations (and con- sequently, value estimates) are obtained at micro-timesteps
p2(00)= = The h!) Ty. (mi ht)
"To clarify the distinction, throughout the remainder of this section, we use macro-timestep. on the other hand, is defined the count of micro- timesteps at which any agent obtains a new macro-ohservation.
interchngeeably with timesteps. A
representing the ratio of the current policy mg. to the old policy vy. for selecting macro-action mi given the
5

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
(oo, 1(1). ((2). 4(3). ... 0,2,5, Here the index in the parenthesis, k represents macro-timestep corresponding to the micro-timestep ((z). The multi-step TD errors at these micro-steps are defined as:
method emphasizes both the significance of future rewards and the strategic importance of each macro-action choice. This approach directly contributed to the strong performance observed with ACAC in the asynchronous MacDec-POMDD POMDP environment.
oo = (ro + m11) + + V/1) - Vi0o
This formulation preserves the key properties of the original GAE: when = 0, reduces the single-step temporal difference (()) and when À 1, becomes the empirical return minus the baseline. Hence, À continues to govern the bias-variance trade-off in advantage estimation as the original GAE does. For a more detailed derivation, please refer to Appendix D.
bi (r2 yra + (22a)) +7 + Viez Vlay
8i = (1s) + +Viiss - Vi
01 =
7 re)) + +. Vigg
Given these TD errors, we have two primary choices for de- signing the advantage function, differing in their application of X-discounting:
4. Related Work
Macro-action-based MARL, or Hierarchical MARL, has been actively studied in the field of multi-agent sys- tems (Nachum et al., 2019; Yang et al., 2020; Wang et al., 2021a:b). Most works focus on learning macro-actions (options or skills) and typically enforce fixed durations, operating in synchronous settings, In contrast, our work ad- dresses scenarios with pre-defined macro- aactios of varying durations, focusing on training high-level policies for asyn- chronous MARL, thereby broadening MARL's applicability to more realistic environments.
Micro-level discounting: Discounts future TD errors based on the actual micro-timesteps elapsed between decision points. This is standard in many GAE imple- mentations.
A A,miero
61
hord dsay
(0)
+ ((000) - δi +...
This can be generally written as:
Asynchronous MARL has recently gained attention, with several methods proposed to address specific tasks such as cooperative charging sensor networks (Chen et al., 2021; Liang et al.. 2022), bus holding control (Wang & Sun, 2021), warehouses logistics (Yoshitake & Abbeel, 2023; Krnjaic et al., 2024), petrochemical scheduling optimization (Hong et al., 2024). The studies most closely related to our work aim to address the more general problem of asynchronous MARL by defining it within the framework of MacDec- POMDPs (Amato et al., 2014) and conducting research based on this formulation.
A A.iiero
Σ (ks) -(0) 110)) 8l00 k=
Discounts future TD errors based on the sequence of macro-action decision points. Each transition between decision points (e.g., 4() to k()) counts as a single macro-timestep, regardless of the actual micro-timen timesstimesseps..
Macro-level discounting:
A A macro
ò1 + 21 (as (0o) dies (oo)
Xiao et al. (2020a) presented two deep Q-networks (DQNs) (Mnih et al., 2013) based approaches that learn macro-action-valur functions in decentralized manner and centralized manner. Xiao et al. (2020b) proposed DQN based approach for learning in CTDE setting. Xiao et al. (2022) extended the previous macro-action based DQN methods an actor-critic method for asynchronous MARL. Yu et al. (2023) modeled multi-robot cooperative explo- ration, where agents can communicate with each other while performing actions, as an asynchronous MARL problem and applied MAPPO to address it.
(0)
+ 2 -lor Sucs +...
This can be generally written as:
A. macro
(al0 koy) 1 8100
k=
The key distinction lies in the exponent of it is the macro- timestep & for macro-level discounting, whereas for micro- level discounting, it is the micro-timestep (())-
Previous works have applied conventional actor-critic ar- chitectures from synchronous MARL by padding missing observations, which often introduces redundancy and in- accuracies in joint history representation. In contrast, we propose an agent-centric history encoding approach that
For the ACAC algorithm, we adopt the macro-level dis- counting approach. This choice was driven by its ability to effectively handle the variable intervals inherent in our asynchronous setting. By aligning the discounting with the critical decision points associated with macro-actions, this
6

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
Overcooked-Rand Since the original Overcooked environ- ments have randomness and only limited uncertainty, introduce a more complex setting where object and agent positions are randomized each episode. This introduces randomness and significantly increases uncertainty, prevent- ing agents from memorizing fixed action sequence and allowing evaluation of their generalization ability. To ensure all configurations remain solvable, we adjust any random placements that would make the scenario infeasible.
(a) BoxPushing
(b) Map A
(c) Map B
(d) Map C
Figure The collection of environments. (a) BoxPushing environ- ment with grid size 10, and (b-d) Overcooked environments with different maps.
5.1. Evaluation
We compare our proposed method, ACAC, with two padding- based baselines: naive independent actor centralized eritic (Mac-NIACC) and independent actor individual centralized critic (MIC-IAICC) (Xiao et al., 2022). The main distinc- tion between these baselines is whether they use a single centralized critic (Mac-NIACC) or a individual critic for each agent (Mac-IAICC), Both methods employ an actor- critic-based CTDE framework with policy gradients. We also evaluate ACAC-Vanilla, variant of ACAC that shares the same architecture but does not use the PPO-style loss function, instead adopting the same loss function the two baselines.
aggregates individual agent histories through attention- based module, providing more accurate joint history rep- resentation without relying on padding.
5. Experiment
We evaluate our method on two collections of MacDec- POMDP environments: BoxPushing and Overcooked (Xiao al., 2020b; 2022).
BoxPushing. The BoxPushing environment, a benchmark for MacDec-POMDP, involves two agents (blue and green) cooperating to push large box to a yellow goal area. It features three maps varying sizes, including 10 x 10 grid shown in Figure 6 (a). Agents observe one of five possi- ble conditions for the cell ahead (e.g., EMPTY, BOUNDARY, BIG BOX) and can take actions like PUSH, MOVE-TO- BOX, or TURN, While small box can be pushed alone, pushing large box requires both agents; otherwise, attempting to push it alone incurs a penalty, Agents receive small reward for delivering a small box to the goal and a large reward for a large box. The episode ends as soon as any box reaches the goal, emphasizing the need for coordination to maximize rewards and avoid suboptimal strategies.
The baselines labeled with the suffix micro are evalu- ated in environments without macro-actions (i.e., standard synchronous MARL). MAPPOnicro denotes the original MAPPO algorithm applied to the picro-action-based envi- ronment. Each experiment and we report the mean and standard error of returns." The detailed hyperparameters can be found in Appendix F.
with five random seeds,
5.2. Comparison with Baselines
Figure 7 shows that ACAC effectively handles various environments, including BoxPushing, Overcooked, and Overcooked-Ree Rand.
Overcooked. Adapted from Gym-Cooking (Wu et al., 2021), Overcooked environment involves three agents work- ing together to prepare and deliver salads (e.g., tomato, onion). Vegetables must be chopped, plated, and delivered to a designated location (marked by yellow star). The environment consists of three as shown in Figure 6 (b-d). Agents observe 5 x 5 grid around themselves and execute macro-actions like GO TO TOMATO, CHOP, and DELIVER. Macro-actions terminate upon completion or the goal is unreachable (e.g., in map B. separated regions prevent reaching a tomato). Agents receive rewards for chopping and delivering correct salads but incur penalties for incorrect deliveries and time delays. They are required to collaborate swiftly and efficiently to successfully prepare and deliver the salad.
Micro-action baselines (labeled with mi cro) generally fail to reach optimal performance, likely due to the sparse re- ward structure requiring longer-term planning better handled by macro-actions. Among these micro-action baselines, ACACmicro and MAPPO both PPO-based, perform relatively well in Overcooked, emphasizing the importance the underlying PPO algorithm.
aspect
grid maps (A, B. C),
Compared to other macro-action baselines, ACAC con- verges more quickly and more reliably optimal perfor- mance in all environments. For instance, in Overcooked, ACAC surpasses other baselines both in the speed of con- vergence and the stability of returns. This improvement stems from ACAC's novel architectural design, which avoids padding-related issues. Even in the more challenging
'We local macro-observations for training agents, while Xiao et al. (2022) allow access to ground truth state.
'We report undiscounted returns, whereas Xiao et al. (2022) reported discounted returns.

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
ACAC
ACAC-Vanilla VaAA-Vanlll
Mac-NIACC
Mac-IAICC
ACACmiar
NIACCniore
IAICCmkcr
MAPPOmicro
BoxPushing-6x6
BoxPushing-Bal
BoxPushing-10k:
Return
200
200
200
0.3
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
Overcooked-A
Overcooked-B -B
Overcooked-C
200
200
200
Return
100
100
100
0
0
16
20
12
16
20
Environment steps (1M)
Environment steps (1M)
Envirenment steps (1M)
Overcooked Rand-A A
Overcooked Rand-B
Overcooked
200
200
200
Retum
100
100
100
0-
16
24
32
40
16
40
8
16
24
32
40
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
Figure 7. Training curves on (upper) BoxPushing, (middle) Overcooked, and (lower) Overcooked-Rand environments. The maximum training environment step depends the difficulty of each environment.
Overcooked- Rand, ACAC maintains superior performance, underscoring its generalization capabilities.
Since Overcooked-Large requires much longer planning horizons, most micro-action- upproaches fail to learn effectively. As result, we limited our comparison here to macro- action-ction-oo action-based models.
(CAC-Vanilly outperforms most baselines, as its architec- ture effectively mitigates the padding issues in Mac-NIACC and Mac-IAICC. However, it trains slower than ACAC, highlighting the benefits of a PPO-based objective.
As shown in Figure 8, the performance gap between ACAC and other baselines is even more pronounced in Overcooked- Large. In nearly all tasks, ACAC achieves the optimal return, while the existing baselines often fail to learn effectively. These difficulties are likely attributable prolonged macro- actions that require extensive padding in conventional archi- tectures, making it harder to learn reliable value estimates.
These overall results indicate that our approach more effec- tively handles history abstraction and structural reasoning across agent histories in pothesize that the padding in Mac-NIACE and Mac-IAICC complicates value function learning, reducing their overall performance compared to ACAC.
problems. We hy-
5.4. Ablation Studies
5.3. Further Comparison in More Challenging Scenario
Effect of Padding in History Encoders. To further as- the impact of eliminating padding in the joint his- tory encoder, we introduce ACAC-Duplicate an ab- lated version of ACAC that duplicates macro-observations as in Mac-IAICC. More specifically, ACAC-Duplicate re- tains the same overall architecture and algorithm as ACAC but differs in how it updates the agent-centric history en- coder each timestep. In ACAC, only newly received macro-observations used to update agent's encoder
To conduct more rigorous evaluation, we introduce Overcooked-Large. a substantially harder environment. We increase the map size from 7 x 7 to 11 x 11, raise the num- ber of agents from 3 to 6, and increase the number of tools (e.g., knives and plates) from 2 to 4, thus expanding both the joint macro-observation and macro-action spaces. These changes require significantly more complex cooperation.
8

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
As Figure 9 illustrates, our results consistently show that macro-level discounting in GAE significantly outperforms its micro-level counterpart across various environments, in- cluding BoxPushing, Overcooked, and Overcooked-Ram We believe this is because macro-level discounting better emphasizes the value of future rewards and the strategic importance of each macro-action choice. This strongly sug- gests that macro-level X-discounting a superior approach for GAE in asynchronous MARL.
ACAC
Mac-NIACC
Mac-IAICC
Overcocked-1 arge-A
Overcocked-Largee
Overcoolved C
200
1.00
100
20 40
65
1o 130
20 46
nn 150
6n no 100 Environmant staps (IM)
Envirnmenssoap (1M)
Environment stops (IM)
Overcooledd Large-Rand-A-
Overcooked-Largg-.ee
OverrackeddLagg.-aond
200
200
For a complete set of ablation studies and additional results, please see Appendix E.
100
100
-20
6. Conclusion
60 Enstrormanttiiss 1iM)
Ensiremmen staps (1M)
Enviromment staps 11N)
In this work, we introduced the Agent-Centric Actor- Critic (ACAC) algorithm address the challenges of asynchronous multi-agent reinforcement learning (MARL) within the Cen- tralized Training with Decentralized Execution framework. ACAC's architecture integrates two key components: I) an agent-centric history encoder that captures each agent's macro-observatinn history along with the specific timestep at which their macro-action begins, and 2) an aggregation module that integrates these individual, agent-specific his- tories into a centralized critic. This design allows ACAC to effectively address the inefficiencies inherent in padding- based methods. To train this novel architecture, we further proposed a PPO-based algorithm and adapted the standard Generalized Advantage Estimation (GAE)-original) con- ceived for synchronous settings- to handle the varying du- rations of macro-actions in asynchronous MARI..
Figure& 8. Training curves on Overcooked-Large environments.
ACAC
AGAC- Dupllcupleeeeee
ACAC (micro-level GAE)
BoxPushing
Overcooked
Overcpoked-Rand
300
200
200
100
0,2
Envtrorment st606 (1M)
Envirorment Stees LiNi
EEndronmenttto Steps (1M)
Figure 9. An ablation study examining the effects of removing padding and implementing macro-level X-discounting in GAE. We report average return over all environments for each collections.
Our experimental results on macro-action-based multi-agent benchmarks demonstrate that ACAC consistently outper- forms existing approaches, proving particularly effective in scenarios that demand complex coordination and robust gen- eralization. By directly addressing padding-related issues through its agent-centric design and a PPO-based objec- tive, ACAC achieves both accelerated training convergence and superior overall performance. Moreover, comprehen- sive ablation studies confirmed that eliminating duplicated macro-obsed snacro-observations and using the modified GAE is crucial for preventing suboptimal outcomes, thereby underscoring ACAC's efficacy in asynchronous multi aalti-agent environments.
for value computation. By contrast, ACAC- ACAC-Duplicate up- dates every agent's encoder each timestep using that agent's most recent macro-observaiion--heeeee newly re- ceived or padded from a previous menestep-therelee dupli- cating macro -asrroobbsevvttions in the learning process. This setup allows us to directly compare how the presence (ACAC-Duplicate) or absence (ACAC) of duplicate macro- observations influences both learning speed and final perfor- mance.
In Figure 9, ACAC consistently achieves optimal perfor- mance, whereas ACAC-Duplicate converges more slowly or even settles into suboptimal solutions. We hypothesize that repetitive and potentially confusing training signals re- sulting from padded macro macro-observations hinder the learning of an accurate value function, highlighting the advantage of ACAC's design, which avoids such padding.
While ACAC excels in macro-action-baser MARL, extend- ing its capabilities continuous action spaces is key future direction. We believe ACAC is adaptable for continuous control. However, primary challenge is the current lack of asynchronous MARL benchmarks that combine continu- ous actions with defined macro-aa aacrooactioos. Developing such benchmarks is crucial future work. Successfully adapting ACAC for these settings would significantly broaden its ap- plicability in dynamic fields like robotics and autonomous driving, enhancing its versatility.
Effect of Macro-level X-discounting in GAE. We now examine the choice of X-discounting in GAE within asyn- chronous MARL. We compare the performance of GAE using the micro-level X-discounting (referred to as ACAC (mi cro-level GAE)) against our proposed macro- level X-discounting.
9

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
Impact Statement
Kuba, J. G., Chen, R., Wen, M., Wen, Y., Sun, F., Wang, J., and Yang, Y. Trust region policy optimisation in multi- agent reinforcement learning. In International Confer- ence on Learning Representations (ICLR), 2022.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Liang, Y., Wu, H., and Wang. H. ASM-PPO: Asynchronous and scalable multi-agent PPO for cooperative charging. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 798-806, 2022.
References
Amato, C., Konidaris, G., and Kaelbling, L. P. Planning with macro-actions in decentralized POMDPs. In Inter- national Conference on Autonomous Agents and Multi- agent Systems (AAMAS), pp. 1273-1280, 2014.
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mor- datch, Multi-agent actor-critic for mixed cooperative- competitive environments. In Advances in Neural Infor- mation Processing Systems (NeurIPS), volume 30, 2017.
Amato, C., Konidaris, G., Kaelbling, L. P., and How, J. P. Modeling and planning with macro-actions decentral- ized POMDPs. Journal of Artificial Intelligence Research, 64:817-859 859, 2019.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A.. Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing Atari with deep reinforcement learning. arXiv preprint arXiv: Iet::3/122555 2013.
Bernstein, D. S., Givan, R., Immerman, N., and Zilberstein, S. The complexity of decentralized control of markov decision processes. Mathematics of Operations Research, 27(4))819 840, 2002.
Nachum, O., Ahn, M., Ponte, H., Gu, S. S., and Kumar, V. Multi-agent manipulation via locomotion using hierarchi- cal Sim2Real. In Conference on Robot Learning (CORL), pp. 1 110-121, 2019. 2019.
Chen, Y., Wu, H., Liang, Y., and Lai, G. VarRLenMARI. A framework of variable-length time-step multi-agent re- inforcement learning for cooperative charging in sensor networks. In IEEE International Conference on Sens- ing, Communication, and Networking (SECON), pp. 1-9, 2021.
Omidshafiei, S., agha-Mohammadi, A.-A., Amato, C., and How, J. P. Decentralized control of partially observ- able markov decision processes using belief space macro- actions. In IEEE International Conference on Robotics and Automation (ICRA), pp. 5962-5969, IEEE, 2015.
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence, 2018.
Omidshafiei, S., Pazis, J., Amato, C., How, J. P., and Vian, J. Deep decentralized multi-tase multi-agent reinforcement learning under partial observability. In International Con- ference on Machine Learning (ICML), pp. 2681-2690. PMLR, 2017.
Hessel, M., Soyer, H., Espeholt, L., Czarnecki, W., Schmitt, S., and Van Hasselt, H. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Confer- on Artificial Intelligence, volume 33, pp. 3796-3803, 2019.
Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J., and Whiteson, S. QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learn- ing (ICML), pp. 4295-429500 PMLR, 2018.
Hong, S., Yoon, D., Jung, W. Lee, J., Yoo, H., Ham, J., Jung, S., Moon, C., Jung, Y., Lee, K., Lim, W., Jeon, S., Lee, M., Hong, S., Lee, J., Jang, H., Kwak, C., Park, J., Kang. C., and Kim, Naphtha cracking center scheduling optimization using multi-agent reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2024.
Schulman, J., Moritz, P., Levine. S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. In International Conference Learning Representations (ICLR), 2016.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707:00777737 2017.
Kmnaic, A., Steleac, R. D., Thomas, J. D., Papoudakis, G., Schäfer, L., To, A. W., Lao, K., Cubuktepe, M., Haley, M., Börsting, P., and Albrecht, S. V. Scalable multi-agent re- inforcement learning for warehouse logistics with robotic and human co-workers. JEFE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pp- 677-684, 2024.
Sutton, R. S., Precup, D., and Singh, S. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):
181-211, 1999.
10

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, I.., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems (NeurlPS), 2017.
Yu, C., Yang, X., Gao, J., Chen, J., Li, Y., Liu, J., Xiang, Y., Huang, R., Yang, H., Wu, Y., and Wang, Y. Asyn- chronous multi-agent reinforcement learning for efficient real-time multi-robot cooperative exploration. In Interna- tional Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 1107-1115, 2023.
Wang, J. and Sun, L. Reducing bus bunching with asyn- chronous multi-agent reinforcement learning. In Interna- tional Joint Conference on Artificial Intelligence (ICCAI), pp. 426 433, 2021.
Wang, R., Kew, J. C.,, Lee, D., Lee, T.-W., Zhang. T., Ichter, B., Tan, J., and Faust, A. Model-based reinforcement learning for decentralized multiagent rendezvous. In Conference Robot Learning (CORL), pp. 711-725, 2021a.
Wang, T., Gupta, T., Mahajan, A., Peng, B., Whiteson, S., and Zhang, C. RODE: Learning roles to decompose multi- agent tasks. In International Conference Learning Representations (ICLR), 2021b.
Wu, S. A., Wang, R. E.,, Evans, J. A., Tenenbaum, B., Parkes, D. C.,, and Kleiman- -leiman-Weiner, M. Too many cooks: Coordinating multi-agent collaboration through inverse planning. Topics in Cognitive Science, 2021.
Xiao, Y., Hoffman, J., and Amato, C. Macro-action based deep multi-agent reinforcement learning. In Conference Robot Learning (CORL), pp. 1146-1161, 2020a.
Xiao, Y., Hoffman, J., Xia, T., and Amato, C. Learning multi-robot decentralized macro-action- policies via centralized Q-net. IEEE International Conference on Robotics and Automation (ICRA), pp. 10695-10701, 2020b.
Xiao, Y., Tan, W., and Amato, C. Asynchronous actor-critic for multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
Yang, J., Borovikov, I., and Zha, H. Hierarchical cooperative multi-agent reinforcement learning with skill discovery. In International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pp. 1566-1574, 2020.
Yoshitake, H. and Abbeel, P. The impact of overall optimiza- tion on warehouse automation. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp- 1621-1628, 2023.
Yu, C., Velu, A., Vinitsky, E.,, Gao, J., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of PPO in cooperative multi-agent games. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, pp. 24611 16611-24624 2022.

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
A. Detailed Explanation of the Overcooked Example in Figure 2
Go to Tomato
Go to Knife
Chop
Go Lettuce
Chop
Go Onion
Go Plate
Timesteps
Observation Decision Time
Overcooked Problem
(b) Asynchronous Decisions with Macro-Actions
Figure 10. (a) Example of an Overcooked problem. (b) Example trajectory of asynchronous decisions.
This section provides a detailed explanation of Figure 2 introduced in the main text. The same figure is reproduced here as Figure 10.
Figure 10 (a) illustrates Overcooked environment where three agents collaborate efficiently prepare and deliver salads (e.g., tomato, onion). In this environment, tasks such chopping vegetables, plating them, and delivering them designated delivery point (yellow star) need to be performed. The agents' macro-actions include Go TO TOMATO, CHOP, DELIVER, and more. Once macro-action is selected, the agent executes a sequence of micro-actions (e.g., navigating along a path) to reach the target location and complete the assigned task.
Figure (b) demonstrates an example of agent behavior using macro-actions, with their actions depicted across different timesteps. Below step-by-step breakdown of the agents' behavior:
Timestep O:
Each agent selects a macro-action based on its current macro-observation:
Agent selects Go TO TOMATO.
Agent selects Go TO LETTUCE.
Agent 3 selects Go TO ONION.
Timestep 2:
Agent completes its Go TO LETTUCE macro-action and receives a new macro-observation. Based on this, Agent 2 selects a new macro-action: Go TO KNIFE 2.
Timestep
Agent completes its Go TO TOMATO macro-action and receives macro-observatioo. Based this, Agent 1 selects new macro-action: Go TO KNIFE 1.
Timestep 7:
Agent completes its Go TO ONION macro-action and receives new macro-observation. Based on this, Agent 3 selects a new macro-action: Go TO PLATE 1.
Timestep 8:
Agent completes its GO TO KNIFF macro-action and receives a new macro-observation. Based on this, Agent 2 selects a new macro-action: CHОP.
Timestep 9:
Agent completes its Go TO KNIFE macro-action and receives a new macro-observation. Based on this, Agent selects a new macro-action: СпООР
Timestep 11:
Agent completes its Go TO PLATE macro-action and receives macro-observation. Based this, Agent 3 selects its next macro-action.
This process demonstrates the asynchronous makinng dynamies in the Overcooked environment.
12

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
B. Formal Definition of MacDec-POMDP
The Macro-Action Decentralized Partially Observable Markov Decision Process (MacDec-POMDP) (Amato et al., 2019) incorporates the options framework (Sutton al., 1999) into the Decentralized Partially Observable Markov Decision Process (Dec-POMDP) by defining set of macro-actions (or options) for each agent.
Followed by the previous work (Xiao et al., 2022), (I,S, M, C, T, R. O, 7), where I = (1....,N) N) is set of indices of agents, S is the state space, A Iliez A' is the joint micro-action space, M = Iliez M' is the joint macro-action space, f = Ilie S2' is the joint micro-observation space, < = Ilez Ç* is the joint macro-observation space, T(s'ls, a) is the state transition probability, R is the reward function shared over all agents, O a), o E Ω is the joint observation probability, ( macro-observation probability, and y is the discounting factor. Note that this paper uses "micro-action" instead of "prinitive-action" for intuitive understanding.
MacDec-POMDP is represented as tuple
Ç is the joint
Each macro-action is tuple m (Bmm Im' Fm)) MM composed of a termination condition B.. Hi - [0, 1), initiation set Imd C Hmmec and low-level policy Tmi Hinic > A', where Hmic (or H is the micro(or macro)-action- observation history space.
The objective MacDcc-POMDP1 is then to find a joint high-level policy = Ilir v that maximizes the following expected discounted return from an initial state So for given low-level policies, i.e., macro-actions:
V arg max E 1E R (8t, at) s0, V).
B.1. Example of Macro-Actions in the Overcooked Environment
To illustrate how inter-agent interaction is handled in practice, consider the Overcooked environment:
Intra-option policy: A macro-action like Go TO TOMATO may involve path planning. If another agent is blocking the path, the intra-option policy handles this by having the agent wait until the path is clear, then continue. This behavior is naturally encoded into the policy without requiring the macro-action to terminate.
Termination condition: If the task goal becomes invalid for instance, another agent picks up the tomato- the macro- action terminates automatically according to its predefined termination condition. These behaviors are not considered "interruptions" in the conventional sense. Rather, they are expected outcomes under the macro-action's design. The inter-agent interaction and adaptation are achieved through well-defined intra-option policies and termination conditions, enabling cooperative behaviors to naturally emerge within the framework.
13

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
C. Comparison of the Procedures for Padding-Based Methods and ACAC
In this section, we provide a detailed explanation of how two different opproaches-Puudding--e methods and our proposed ACAC-operate within an Asynchronous MARL setting. To illustrate these procedures, we refer to the example in Figure 10. Both methods follow a CTDE (Centralized Training and Decentralized Execution) framework under (MacDec-)) JPOMDP setting, meaning they rely macro-observation histories. In practice, these histories are typically (MacDec-PPOMD processed by recurrent networks (e-g., RNNs or LSTMs) in both the decentralized actors and the centralized critic to combine past and current macro-observali effectively.
<-0
--1
1-5
<-3
1-4
L-6
6-7
2
1-
(-11
Decentralied
Aciar
Decentraizes Actor
Decentrazzed Actor 3
Centralazd Crilcc
Cttc
Cree
Cnto
Figure 11. Full CTDE procedure of padding-basee based methods for the example in Figure 10
C.1. Procedure of Padding-Basse Methods
In the padding-basee based approach, each agent's 's decentralized actor selects macro-action solely based on its own macro- observation. The actor is only active at timesteps when new macro-observation arrives (i.e., whenever new macro-action can be chosen). At each such timestep, the agent updates its actor's 's hidden state to incorporate the newly received macro-observation.
Meanwhile, the centralized critic takes as input the concatenation of agents' macro-observations, along with the concatenated history of these macro-observations, estimate the joint value. When any agent obtains new macro- observation, a joint macro-observattoon must be constructed compute the value for that timestep. For agents that do not receive a new observation at that timestep. the critic uses each agent's 's most recent macro-observation. The centralized critic then updates its hidden state with this joint macro-observation and, based on the updated hidden state, provides value estimate.
Figure illustrates how the decentralized actors and the centralized critic operate under the Padding approach in the example from Figure
14

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
C.2. Procedure of ACAC
<-3
1--4
Z-
<-
Decantraizzdd Aciar
MUP
MJ
MIP
Agwit OVeR Enooder
m
Decentra
Actor
MUP
VL
Ag
Agont-Darts:
Agarn-Cartic
Asam-Cordt
s -
18
m
m5
Desensallzss
Astor
Agend Denti
Ageii
Agent Canks
?
VIL
MELP
MLP
MPP
MLP
MILP
VL.2
Appripation
Appregalion
Apprepation
Fogrepen mouten
Appregation oe
Appreggi
Apprepation
Centralieed Criic
X
X3
Figure 12. Full CTDE procedure of ACAC for the example in Figure 10
In ACAC, the decentralized actor functions similarly to that in padding- based methods, with one key difference: in addition receiving the macro-observation as input, also receives timestep at which the macro-observation was obtained. This provides information about the duration between consecutive macro-observations, allowing for richer history representation.
Structurally and operationally, ACAC's centralized critic differs significantly from padding- based methods. In the padding- based approach, all agents' macro-observatto must be available at each timestep to compute the value; when no new cocervobservvtiion is obtained by particular agent, its most recent observation is reused (i.e., padded). By contrast, the centralized critic in ACAC is designed to accept only new macro- coservobbsrrvtions as input. More concretely, each agent's ' macro-observations are processed by an agent-centric history encoder, which updates that agent's hidden state, At timestep
15

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
when agent receives macro-observation, it updates its hidden state through its agent-centric history encoder. The centralized critic then aggregates each agent's most recently updated hidden state-whether from the current a previous timestep-Via aggregation module to estimate the value. This design enables the critic to combine agent-specific history representations without padding stale observations from other agents.
Figure 12 illustrates the structure and operation of ACAC's decentralized actor and centralized critic for the example in Figure 10.
16

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
D. GAE Derivation
D.I. Motivation and Objectives of GAE
Generalized Advantage Estimation (GAE) (Schulman et al., 2016) is a widely used technique in policy gradient methods such as PPO (Schulman et al., 2017), primarily because helps strike a balance between the high variance of empirical returns and the high bias of temporal difference (TD) estimates. GAE introduces hyperparameter À that adjusts this bias-variance trade-off. Specifically, when empirical returns minus the current value estimate. Setting À to intermediate value between 0 and allows for smooth trade-off between these two extremes.
GAE relies solely on TD estimates, while when À 1, it uses the
D.2. Illustrating GAE in Asynchronous MARL
In asynchronous MARL, intervals between consecutive macro-observations can vary. This irregularity renders the standard GAE formula (originally designed for synchronous MARL) inapplicable. To address this issue, we propose modified GAE procedure that accounts for the asynchronous nature of MARL.
Rather than presenting the formula immediately, we begin with a concrete example illustrating how to compute GAE an asynchronous setting (see Figure 10). In this example, new macro-observations occur at timesteps 0, 2, 4, 7,8, 9, 11. From the perspective of macro-action decision points, the temporal differences (TDs) can be computed as follows:
Timestep O:
6u = fro + 7n + V22 - Vo
Timestep 2:
82 = {r2 + 733) + 22 - V/2
Timestep 4:
64 = [r + 778 + +2r + +,,
V4
Timestep 7:
87 = (r>} + +VVs - 1/
Timestep 8:
8g = {rs} 7 - Vs
Timestep 9:
dg = {re + mooo 2 Vo
By taking the dddicouutedd sum of these TDs, obtain various advantage estimates at timestep 0. For instance:
1 do
Vo {ro + m +
(2)
A du + +
Vo {ro + 7r + + + +
(3)
1 5u 2 + 700 '81
(4)
Vo (ro + 721 + 2r
2r + 7 + + -
(5)
46th 8n + + 2 1 ~
(6)
1/ (ro + w1 + 3
re 9
17] V88
(7)
20 1000 Vo
rr
(8)
Next, we apply an exponential weighting with parameter X, analogous to the standard GAE approach, to produce the final
17

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
advantage estimate:
A = (1- (1 (2) + +A40 + + + +1 + +...)
(9)
+ = (1 1) 2) + 2 +
+ ,/ + 7 dds + 2 + +/ +..)
(10)
- -[661++++ +
(11)
+...) +...
= (1 - +1 11 01 7
(12)
50 2 A2
+ 8 + 7 +...
(13)
Note that this version of GAE, derived for the asynchronous case, differs from the standard GAE used in synchronous MARL. The main adaptation is in how the TDs are accumulated across irregular macro-observation intervals accurately reflect asynchronous dynamics.
D.3. Generalized Asynchronous GAE Formulation
Building on the above example, we now present generalized formulation for asynchronous GAE. First, define the timesteps at which any agent obtains new macro-observation. These timesteps be defined recursively:
I l(++1) = min fu u 3ga + Estay: u u> 1(0) for & 2 0,
[(0)
(14)
(15)
where 2gμ denotes the histories of all agents up to timestep u. Intuitively, 1(++1) is the first timestep after l(x) at which any new macro-observation arises. Given these macro-observable timesteps ((x). we define the temporal difference at each ((k) as
1(++111
-Viss
Σ
y jurer + a/a two Vi
(16)
(=)(
Next, for 1(K), we define the advantage estimates
A!T) using dias as follows:
A =
10-1)-1
-Vin +
Σ
zu xu-lwra } + -latnn Vigen
(17)
a=k()
1 = dumm + + 40 d1 Sla
1(--2-11
g xuttwra } + 2/u022 -a0 Venre
Viis +
(18)
x=k(
A dmm + - kw (W3
diatet + -aa 84
1----1
-VVa +
Σ
zu -u + g/eatar (2)) Vin
(19)
a=/a)
A(()) = dims + g/ 2latt
Brai p1o4t))-10 8 + p/a+m) -100 8l m)
4m-m--1
=-Vay + yamker + - lw0 Visams
(20)
18

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
We then compute an exponentially weighted sum of these terms with parameter (analogous the original GAB);
(A) GAR,
1) (A()) + +AA, A (3) +
(21)
2) 2) (diEs) +dfomes + - (44) 61r411 / + X2rores + (4)) dieen + 2/(022) -10) de } + t...)
1) (u ++++++++++ x²+ ...] + + (w0 {A + x2+ 113 +...}
(22)
+ (4422) 100) 81ce (22 + 2a 2 +.... .......+...)
(1 - - 0lees + - - di + g/(0t2) -( 1(a) 22
I-X di be + t...)
(23)
= din + + N 1o + x/ (1)) X2 (2c( +...
(24)
= (mm) 1(0) At (m))
(25)
m=
Notably, this asynchronous GAE formulation still satisfies the original GAR objectives. When À = 0, reduces to using only temporal differences:
A
where t = L(y.. When À = 1, relies on the empirical returns minus the current value function:
A G
=
Σ
7 km) 140 (((m)
(26)
m=k
Ó100) *l(cte) -w Aur + A/ato -kw de +
(27)
(((t2)))
Viin
Σ
gu yu-kessu } gla- too Vienns
(28)
k=((
121-1-
+ 2/11111 4100
IV +
Σ
gu- ta 1o-ka ghac 1ha1))1-111 Vice +...
x=
7a sa-uur - Viaa
(29)
0C
-u- u
Tn } Vi.
(30)
19

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
E. Additional Results
E.1. Full Results on Effect Removing Padding and Implementing Macro-level X-discounting for GAE
ACAC
ACAC-Duplicate
ACAC (micro-level GAE)
BoxPushing-6x6
BoxPushing-8xx
BoxPushing-
300
300
Return
200
200
200
100
100
100
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Enviranment steps (1M)
Environment steps (1M)
Enviranment steps (1M)
Overcooked-A
Overcooked-B
Overcooked-C
200
200
200
Return
100
100
100
4
8
12
16
20
8
12
16
20
0
4
12
16
20
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
Overcooked-Rand-A
Overcooked-Rande
Overcooked-Rand-C
200
200
Return
100
100
100
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
Figure 13. Ablation study examining the effects of removing padding and implementing macro-level X-discounting for GAE
20

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
E.2. Further Sensitivity Analysis and Ablation Study
ACAC (==0.005)
ACAC (s=0.010)
ACAC (s=0.015)
Overcooked-A
Overcooked-B
Overcooked-C
200
200
200
Return
100
100
100
0
8
12
16
20
8
12
16
20
0
4
12
16
20
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
Rverco
Overcooked-Rand-B
Overcooked-RRaddC
200
200
200
Return
100
100
100
0
8
16
24
32
40
16
24
40
8
16
24
32
40
Environment steps (1M)
Environment steps (1M)
Enviranment steps (1M)
Figure 14. Sensitivity analysis of the effect of the clipping ratio
ACAC
ACAC (No TE)
ACAC (No SA)
Overcooked-A
Overcooked-B
Overcooked-C
200
200
200
Return
100
100
0
4
8
12
16
20
8
12
16
20
0
12
16
20
Enviranment steps (1M)
Environment steps (1M)
Enviranment steps (1M)
Beercooked-Randae
Overcooked-Rande
Overcooked-Rand-C
200
200
200
Return
100
100
100
8
16
24
32
40
8
16
24
32
40
0
8
16
24
32
40
Environment steps (IM)
Environment steps (1M)
Enviranment steps (1M)
Figure 15. Ablation study on the effects of time embedding (TE) and self-attention (SA)
21

---

Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning
F. Hyper-parameters
The hyper-parameter used for our proposed method, ACAC, is described in Table 1. We mostly follow experimental setting of (Xiao al., 2022). while we used the better value for some hyper- ppeerpparamters..
Table I. Common hyper-paramette used across the environment collections
Hyper-parameter
BoxPushing
Overcooked
Overcooked- -and
Overeooked-Large( Large(-Rand)
Total training timesteps
500K
20M
40M
100M
MLP Layer Size (Actor)
[32, 32]
[32, 32]
[128, 64]
RNN Layer Size (Actor)
32
32
64
MLP Layer Size (Critic)
[32, 32]
[128, 64]
RNN Layer Size (Critic)
32
64
Discount factor y
0.98
0.99
Episodes per train
16
8
Episodes per target critic update
64
32
Clipping ratio
0.05
0,01
Epochs
2
5
Episodes per train
16
8
Max episode length
100
200
Learning Rate (Actor)
3e-4
Learning Rate (Critic)
3e-4
Minibatch size
8
GAE A
0.95
G. Computational Resources for Experiments
We used AMD EPYC 7453 28-Core Processor and A10 for our experiments, and the running time of the proposed ACAC typically ranges from approximately 24 hours to 150 hours on Overcooked environments and from approximately 30 minutes to hours on BoxPushing environments.
H. Implementation and License Information
The implementation code for this work can be found at https::///// /ttt::///ithhb Rewecchh-e-Reeseeccc/eeeeeee implementation code is based on the original implementation code for the previous work (Xiao et al., 2022), but there is no license information in the repository at https: / thub. www.yuchennaaeeeeeo wwwycchenx//Macreeee x/MacroMART. We use the transformer implementation from Hugging Face, which is licensed under the Apache 2.0 License. The gym-cooking environment (Wu et al., 2021) at https: //g/ttuttt com/ rosewang2008 gym cookiigg and its macro-action version (Xiao et al., 2022) at https: //gitttte thub. com/We aeeehaooan//mmmmacrooooooo the proposed method and baseline algorithms, licensed under the MIT license.
The
which were used for the evaluation of
22


Reviews:
[{"review_id": "vzBspyFeZz", "reviewer": "Reviewer_FJpH", "summary": "This paper tackles the challenges encountered in asynchronous multi-agent reinforcement learning (MARL) arising from the use of macro-actions with varying durations. In traditional Centralized Training with Decentralized Execution (CTDE) frameworks, a padding technique is often used to fill in missing macro-observations. However, such padding can introduce redundancy and misleading correlations in the history representation. To address this, the authors propose the Agent-Centric Actor-Critic (ACAC) algorithm. ACAC utilizes individual history encoders for each agent to process its own macro-observation trajectory, and an attention-based aggregation module to integrate these histories into a centralized critic. Additionally, the algorithm incorporates a modified Proximal Policy Optimization (PPO) objective and an adapted version of Generalized Advantage Estimation (GAE) suitable for asynchronous settings. The experimental results, conducted on several benchmark tasks including BoxPushing, Overcooked, Overcooked-Rand, and the more challenging Overcooked-Large, demonstrate that ACAC converges faster and achieves higher returns compared to padding-based baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your valuable comments are much appreciated. In response, we aim to clarify our contributions and address the points of concern.\n\n### **[Q1] Qualitative Analysis Request**\n\nThank you for the insightful suggestion to correlate attention scores with behavior.\nThis is an excellent direction for future work to gain deeper understanding.\n\n&nbsp;\n\n### **[Q2] Hyperparameter Sensitivity**\n\nWe conducted experiments to analyze hyperparameter sensitivity.\nDue to space limitations in this response, we have detailed these results in our separate response provided to reviewer RGMo (Sensitivity Analysis on Hyperparameter).\nWe kindly request your understanding and refer you to that specific comment for the detailed results.\n    \n&nbsp;\n\n### **[Q3,W3] Continuous Action Space**\nWe believe ACAC is structurally suitable for continuous actions with standard actor-critic modifications.\nThe main challenge is the current lack of asynchronous MARL benchmarks with continuous action spaces and defined macro-actions.\nDefining these requires careful effort, preventing results within the rebuttal period, but it remains important future work.\n\n&nbsp;\n\n### **[Q4,W1] Analysis of Modified GAE**\nSimilar to the original GAE, our proposed method does not guarantee convergence in general.\nHowever, just as the original GAE converges when $\\lambda$ equals 0 or 1, our proposed GAE also inherently converges in these specific cases.\nThe rationale behind using macro-action-based $\\lambda$ discounting in GAE within ACAC is detailed in our response to reviewer Cf7Q (Analysis of Modified GAE); please refer to our comments there for further clarification.\n\n&nbsp;\n\n### **[W2] Computational Complexity of ACAC Structure**\n    \nDefining N=#agents, K=#observations per agent, Z=obs dim, H=hidden dim:\n    \n- Actor: Complexity is nearly identical, aside from ACAC's minor time embedding overhead.\n- Critic:\n    - History Encoding\n        - Mac-IAICC: Encodes the concatenated joint history (size NZ) whenever any agent gets a new observation (up to NK times total). This leads to a worst-case complexity of O(N²KZ).\n        - ACAC: Processes each agent's observation (size Z) individually through its dedicated encoder. Across all agents and observations (NK total), the complexity is O(NKZ). This scales better than Mac-IAICC's encoding by a factor of N.\n        - Thus, ACAC's history encoding scales linearly with N (O(NKZ)), offering better scalability than the quadratic O(N²KZ) of joint encoding, especially for larger N.\n    - Value Estimation\n        - Mac-IAICC: Typically uses an MLP processing aggregated hidden features (size NH). Computed up to NK times, giving O(N²KH) total complexity.\n        - ACAC: Employs an attention mechanism over N agent representations (size H), requiring O(N²H) computation per step where an observation arrives. In the worst case (NK steps), this totals O(N³KH) complexity.\n        - While a simpler MLP aggregation (as in Mac-IAICC) offers one potential alternative for scalability, potentially trading off some performance, another promising direction for future work is to explore the integration of efficient attention mechanisms [1] that reduce complexity (e.g., to linear O(NH) per step) while aiming to retain the expressive capacity for modeling inter-agent dependencies.\n- Practical Runtime: Despite theoretical differences, the observed runtimes for ACAC and Mac-IAICC were similar in our experiments (N=3, 6). This suggests ACAC's more efficient history encoding helps offset the computational cost of the attention module in practice for these agent populations.\n\n\n[1] Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020). Linformer: Self-Attention with Linear Complexity. arXiv preprint arXiv:2006.04768.", "claims_and_evidence": "The paper makes several key claims:\n1. Problem Identification: The conventional padding method in asynchronous MARL introduces redundant and inaccurate information into the joint history representation, thus impairing the effectiveness of the centralized critic. \n2. Methodological Contribution: By employing agent-centric history encoders and an attention-based aggregation mechanism, ACAC can directly utilize the latest available macro-observations without resorting to padding. \n3. Performance Improvement: The proposed ACAC, when combined with a modified PPO objective and an adapted asynchronous GAE, outperforms existing padding-based approaches in terms of convergence speed, stability, and final performance. \nThe experimental results, including learning curves, ablation studies (e.g., comparison with ACAC-Duplicate), and evaluations across various environments, provide clear and convincing evidence supporting these claims. In particular, the superior performance in complex scenarios like Overcooked-Large reinforces the paper’s assertions about the benefits of the proposed method.", "methods_and_evaluation_criteria": "The methodological design and evaluation criteria in the paper are well-motivated and appropriate:\n1. Method Design: The authors identify the core issue in asynchronous settings—the irregularity in receiving new macro-observations—and address it by designing an agent-centric encoder that processes each agent’s macro-observation along with its associated timestamp. This enriched representation allows for a more accurate reconstruction of the agents’ histories. The subsequent attention-based aggregation module enables the centralized critic to combine these individual histories without the need for padding.\n2. Evaluation Criteria: The paper employs a diverse set of benchmark environments that vary in complexity (from BoxPushing to Overcooked variants). The experiments are conducted using multiple random seeds and include metrics such as convergence speed and final returns. Additionally, ablation studies are performed to isolate the effects of padding versus non-padding, providing further empirical support for the proposed approach.", "theoretical_claims": "The theoretical contributions primarily focus on the derivation of an adapted GAE for asynchronous MARL:\n1. Theoretical Derivation: The authors provide a detailed derivation of a modified GAE formulation that accounts for irregular intervals between macro-observations. By introducing a variable that captures these intervals, the new formulation properly adjusts the temporal difference errors and advantage estimates.\n2. Assessment: The derivation is logically sound and recovers the standard GAE formulation as a special case when the intervals are uniform. While the derivation is complex, the underlying assumptions and steps are clearly articulated. It would be beneficial for the final version to further clarify any underlying assumptions about the distribution of the time intervals and boundary conditions to ensure readers fully grasp the scope of the theoretical results.", "experimental_designs_or_analyses": "The experimental design and analysis are robust and comprehensive:\n1. Experimental Setup: The paper evaluates ACAC across multiple environments (BoxPushing, Overcooked, Overcooked-Rand, and Overcooked-Large) that effectively capture the challenges of asynchronous MARL. The inclusion of both standard and randomized scenarios demonstrates the method’s generalization capabilities.\n2. Data Analysis: Learning curves, accompanied by mean and standard error statistics, clearly illustrate the performance improvements of ACAC over baseline methods. The ablation studies, particularly the comparison between ACAC and a variant that duplicates macro-observations (ACAC-Duplicate), effectively show the detrimental effects of padding in the learning process.\n3. Recommendations for Improvement: Future work might explore the extension of ACAC to continuous action spaces and larger-scale multi-agent systems. Additionally, a discussion on hyperparameter sensitivity and computational efficiency would further enhance the experimental analysis.", "relation_to_broader_scientific_literature": "The key contributions of the paper are well-situated within the broader literature:\n1. Macro-actions and Hierarchical RL: The paper builds on the established ideas of macro-actions and the options framework (e.g., Sutton et al., 1999) as well as hierarchical reinforcement learning methods. It extends these ideas by addressing the challenges of asynchrony, a topic that has been explored in works such as Amato et al. (2014, 2019) and Xiao et al. (2020, 2022).\n2. CTDE and Multi-Agent Coordination: The work leverages the Centralized Training with Decentralized Execution (CTDE) framework—common in multi-agent RL—to propose an innovative agent-centric design. This is related to recent advances in methods like MAPPO (Yu et al., 2022) and other actor-critic variants that tackle coordination issues in multi-agent settings.\n3. Modified GAE: The adaptation of Generalized Advantage Estimation for asynchronous settings ties back to foundational work by Schulman et al. (2016), but the modification to account for variable macro-observation intervals is novel and addresses a gap in existing methods.\n4. Attention Mechanisms: The use of attention-based aggregation to combine agent histories connects with broader trends in using attention (e.g., Vaswani et al., 2017) for handling sequence data, which is increasingly common in multi-agent communication and coordination research.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. Originality: The agent-centric approach to avoid padding issues in asynchronous MARL is innovative and addresses a well-known challenge in the field.\n2. Theoretical Rigor: The derivation of the modified GAE for asynchronous settings is detailed and, overall, mathematically sound.\n3. Empirical Validation: The experiments are comprehensive, spanning several benchmark environments and including ablation studies that clearly demonstrate the benefits of the proposed method.\n\nWeaknesses:\n1. Complexity of Theoretical Derivations: Some parts of the derivation (especially the asynchronous GAE) are complex and may benefit from additional intuitive explanations or visual aids.\n2. Scalability and Computational Overhead: While the experiments are convincing, more discussion on the computational cost and scalability of the attention-based aggregation, especially in larger agent populations, would be helpful.\n3. Extension to Continuous Actions: The paper acknowledges that extending ACAC to continuous action spaces is an open challenge. More discussion on potential approaches or anticipated difficulties could strengthen the contribution.", "other_comments_or_suggestions": "1. Supplementary Material Organization: Consider reorganizing the supplementary material for clearer navigation. For instance, clearly labeling each appendix section with a brief overview of its contents could help readers.\n2. Minor Typos and Clarity: I noticed a few minor typos and formatting issues in the text; a careful proofreading would enhance readability.\n3. Visualization of Encoder Dynamics: Including visualizations of the agent-centric encoder’s hidden state evolution or attention weight distributions could provide deeper insights into how the aggregation module improves history representation.", "questions_for_authors": "1. Can the authors provide qualitative analyses (e.g., visualizations of attention weights or hidden state dynamics) that illustrate how the agent-centric encoder and aggregation module enhance the representation of asynchronous histories compared to the padding-based approach? A detailed response and accompanying visualizations would help clarify the internal workings of the proposed model and further justify the design choices, potentially increasing the paper's impact.\n2. Have the authors conducted sensitivity analyses on key hyperparameters such as the clipping ratio in PPO and the λ parameter in the modified GAE? How robust is the performance of ACAC across different settings? Insights into hyperparameter robustness would strengthen the empirical evidence and provide practical guidance for applying the method in various settings.\n3. Could the authors elaborate on the potential challenges and necessary modifications for extending ACAC to continuous action spaces? Are there any preliminary experiments or theoretical insights in this direction? A clearer discussion on this topic would help situate the current work within broader applications and guide future research efforts.\n4. What assumptions, if any, are made regarding the distribution or variability of the time intervals between macro-observations? Could extreme variability in these intervals affect the validity of the modified GAE formulation? Clarifying these assumptions would help determine the generality of the method and its applicability to diverse asynchronous environments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qzS0RKR359", "reviewer": "Reviewer_Cf7Q", "summary": "This paper proposes the Agent-Centric Actor-Critic (ACAC) algorithm to address asynchronous multi-agent reinforcement learning (MARL) in sparse-reward environments with macro-actions. The key innovation lies in replacing padding-based centralized critics with agent-centric history encoders and attention-based aggregation, which avoids spurious correlations from padded observations. ACAC introduces a modified GAE for asynchronous settings and demonstrates superior performance on macro-action benchmarks like BoxPushing and Overcooked, showing faster convergence and higher returns compared to baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your feedback and would like to offer a detailed explanation of our contributions while addressing the concerns raised.\n\n### **[W1] Analysis of Modified GAE**\nIt is known that the original GAE does not guarantee convergence in general.\nHowever, specific boundary cases are clearly defined: when $\\lambda$=0, GAE reduces to the TD-error; when $\\lambda$=1, it simplifies precisely to the empirical return.\nSimilarly, the proposed GAE in our paper maintains this important property—when $\\lambda$=1, it equals the empirical return, and when $\\lambda$=0, it becomes the multi-step TD-error between consecutive decision points ($\\delta_{l(k)}$ in our formulation, where $l(k)$ is the $k$-th timestep a new observation becomes available for any agent).\n\nIn MacDec-POMDP settings featuring temporal abstraction via macro-actions, there can be two choices for designing the advantage function based on how $\\lambda$-discounting is applied:\n- 1. Applying $\\lambda$-discounting at the **micro-timestep level**: This approach discounts future TD errors based on the number of primitive timesteps elapsed.\n- 2. Applying $\\lambda$-discounting at the **macro-timestep level**: This approach discounts future TD errors based on the number of macro-action decision steps taken.\n- Let's illustrate using an example where macro-observations (and value estimates $V_t$) are obtained at timesteps 0, 2, 5 and 6:\n    - Rewards: $r_0, r_1, r_2, r_3, r_4, r_5, r_6, \\dots$\n    - Values: $V_0, -, V_2, -, -, V_5, V_6, \\dots$\n    - Multi-step TD errors between decision points:\n        - $\\delta_0 = r_0 + \\gamma r_1 + \\gamma^2 V_2 - V_0$\n        - $\\delta_2 = r_2 + \\gamma r_3 + \\gamma^2 r_4 + \\gamma^3 V_5 - V_2$\n        - $\\delta_5 = r_5 + \\gamma V_6 - V_5$\n    - Below are the resulting advantage calculations at t=0 for each approach:    \n        - (1) Micro-level $\\lambda$-discounting : $A^{\\lambda, \\text{micro}}_0 := \\delta_0 + (\\lambda^{2} \\gamma^{2}) \\delta_2 + (\\lambda^{5} \\gamma^{5}) \\delta_5 + \\dots$\n        - (2) Macro-level $\\lambda$-discounting : $A^{\\lambda, \\text{macro}}_0 := \\delta_0 + (\\lambda \\gamma^{2}) \\delta_2 + (\\lambda^{2} \\gamma^{5}) \\delta_5 + \\dots$\nWe adopted the second approach (macro-level $\\lambda$-discounting) for ACAC, as it emphasize both the significance of future rewards and the critical decision points associated with macro-actions. As in Table 2 of response for RGMo (Sensitivity Analysis on Hyperparameter), this approach effectively handles the variable intervals in our asynchronous setting and contributes to the strong performance observed.\n\n&nbsp;\n\n### **[W2] Real-world Applicability**\nThank you for raising this important point regarding the evaluation environments.\nWe agree that verifying applicability beyond grid-world tasks is crucial.\nWhile settings requiring asynchronous coordination are common in the real world, the field of Asynchronous MARL, especially involving macro-actions, is still emerging.\nConsequently, there is currently a limited set of established benchmarks available, most of which are grid-based environments like BoxPushing and Overcooked.\nWe aimed for a comprehensive evaluation within the current scope of the field and thus tested our ACAC method on these known and publicly available benchmarks specifically designed for macro-action-based asynchronous MARL.\nWe recognize the importance of demonstrating applicability in more realistic scenarios.\nExtending our work to real-world problems is a key direction for our future research.", "claims_and_evidence": "The superiority of ACAC over padding-based methods is supported by experiments. The ablation study (ACAC-Duplicate) convincingly demonstrates padding’s negative impact.", "methods_and_evaluation_criteria": "The agent-centric encoder and attention-based critic are well-justified for asynchronous settings. The modified GAE adapts standard GAE to variable intervals logically. Benchmarks selected are appropriate for demonstrating algorithm efficacy.", "theoretical_claims": "The theoretical foundations are basically sound.", "experimental_designs_or_analyses": "The experiments are thoughtfully designed and robust in demonstrating the method's strengths. Broader scenario testing would better illustrate the adaptability of the techniques.", "relation_to_broader_scientific_literature": "The paper contextualizes its work within macro-action MARL and CTDE frameworks, citing key works (Xiao et al., 2020a; Amato et al., 2019).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Paper Strength\nACAC introduces agent-centric history encoders with timestep embeddings and an attention-based critic, effectively addressing the misalignment caused by padding in asynchronous MARL. This architecture is novel and well-motivated.\nExtensive experiments on diverse benchmarks (including randomized and large-scale variants) demonstrate ACAC’s advantages in convergence speed, final performance, and generalization. Ablation studies (e.g., ACAC-Duplicate) confirm the necessity of avoiding padding.The modified GAE formulation for asynchronous MARL is theoretically justified and empirically validated, addressing irregular macro-observation intervals.\nThe paper clearly identifies limitations of padding-based methods (e.g., spurious correlations) and provides a structured comparison of ACAC’s workflow versus baselines\nPaper Weakness\nWhile the modified GAE is motivated empirically, its convergence properties or bias-variance trade-off in asynchronous settings lack formal analysis.\nAll experiments focus on grid-world tasks (BoxPushing, Overcooked). Real-world applicability (e.g., continuous control) remains unverified.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "please see the weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hz2c50qbV5", "reviewer": "Reviewer_RGMo", "summary": "This paper introduces Agent-Centric Actor-Critic (ACAC), a novel algorithm designed for asynchronous multi-agent reinforcement learning (MARL) in environments with sparse rewards and varying macro-action durations. Each agent's trajectory is processed independently, capturing the history of macro-observations along with their timesteps, enabling accurate temporal abstraction without padding. Meanwhile, an attention mechanism integrates the individual agent histories into a centralized critic, allowing for more accurate value estimation in asynchronous settings. This paper adapt the modified GAE to handle irregular intervals between macro-observations, ensuring effective policy optimization in asynchronous environments. This paper evaluates ACAC on several macro-action-based benchmarks, such as BoxPushing and Overcooked, demonstrating that ACAC achieves faster convergence and higher final returns compared to baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your thoughtful remarks and would like to provide clarification on our contributions and address the raised concerns.\n\n&nbsp;\n\n### **[Q1,W1] Analysis of Modified GAE**\nSimilar to the original GAE, our proposed method does not guarantee convergence in general.\nHowever, just as the original GAE converges when $\\lambda$ equals 0 or 1, our proposed GAE also inherently converges in these specific cases.\nThe rationale behind using macro-action-based $\\lambda$-discounting in GAE within ACAC is detailed in our response to reviewer Cf7Q (Analysis of Modified GAE); please refer to our comments there for further clarification.\n\n&nbsp;\n\n### **[Q2] Information Loss Comparison**\nGiven an episode obtained using macro-actions, both ACAC and padding-based methods theoretically acquire the same amount of information.\nHowever, the padding-based approach continuously uses information from observations that are not actually collected,\nmaking it difficult to distinguish between cases where no new information is obtained and those where identical information from a previous observation is repeated.\nThis can result in information distortion.\nTo eliminate such distortion without losing information, ACAC employs an agent-centric encoder that utilizes only each agent's available information for history encoding.\nThis structure allows ACAC to more accurately estimate joint histories compared to padding-based methods, thereby achieving superior performance.\n\n&nbsp;\n\n### **[Q3,W3] Evaluations on the SMAC**\nWe agree with your point that evaluating our approach in environments like SMAC could enhance the persuasiveness of our results.\nIndeed, several studies have explored hierarchical approaches in SMAC; however, most of them have focused on synchronous scenarios where macro-actions have a fixed, identical duration for all agents.\nOur work, in contrast, specifically addresses scenarios involving asynchronous macro-actions, where each macro-action has varying durations.\nConsequently, conducting performance comparisons in synchronous SMAC environments would not meaningfully reflect the contributions of our method.\n\nTo compare ACAC and the baseline methods effectively in SMAC under asynchronous settings, we would need to explicitly define macro-actions with varying durations.\nUnfortunately, it is not feasible to develop and test such an asynchronous version of SMAC within the limited time frame available for this rebuttal.\nNevertheless, we are actively working on developing an asynchronous variant of SMAC.\nWe strongly anticipate that ACAC will exhibit superior performance over existing baselines, such as Mac-IAICC and Mac-NIACC, once evaluated in this asynchronous SMAC environment.\n\n&nbsp;\n\n### **[Q4] Influence by Opponent Policies**\nThank you very much for suggesting an evaluation in environments like SMAC-Hard, where multiple opponent policies are mixed.\nSimilar to our earlier explanation regarding SMAC, we believe conducting evaluations in such environments would be meaningful only within an asynchronous version of SMAC.\nThis, however, requires developing a dedicated asynchronous SMAC environment.\nAdditionally, effectively responding to various opponent strategies, as seen in SMAC-Hard, would necessitate developing a module explicitly designed for opponent strategy inference.\nWhile this is beyond the current scope of our research, which primarily targets asynchronous MARL scenarios, \nwe anticipate that ACAC's per-agent history encoder could be extended or enhanced to perform opponent strategy inference.\nThis capability would potentially enable the model to adapt effectively to different opponent strategies, representing an exciting and promising direction for future research.\n\n&nbsp;\n\n### **[W2] Sensitivity Analysis on Hyperparameter**\nWe agree your suggestion, so we have performed sensitivity analysis on hyperparameter, clipping ratio and GAE $\\lambda$(micro-level vs macro-level $\\lambda$-discounting), across Overcooked and Overcooked-Rand environments.\nSince the complex Overcooked-Rand-B map showed the clearest and most significant performance differences between configurations,\nclearly highlighting component impacts, we focus on these results here for brevity.\nFull results will be included in the revised manuscript.\nWe report the average final performances and standard errors over five random seeds.\n\n**Table 1**. Sensitivity analysis on hyperparameter: clipping ratio (ε).\n\n| ACAC ( ε=0.01) | ε=0.005 | ε=0.015 |\n|---------------------|----------------|------------|\n| 212.42 ± 0.64      | 188.73 ± 14.99 | 209.62 ± 3.33 |\n\n**Table 2**. GAE $\\lambda$ comparison.\n| ACAC | GAE with Micro-level $\\lambda$-discounting |\n|------------|------------|\n| 212.42 ± 0.64  |132.98 ± 19.02  |\n\nThe results shows that performance is robust to clipping ratio variations; ACAC's macro-action based GAE discounting outperforms the original timestep-based discounting.", "claims_and_evidence": "Yes. The paper provides theoretical and empirical evidence for its main claims. This paper claims that ACAC generalizes well to more complex and randomized environments. This evidence is strong, but additional experiments with larger complexity, such as SMAC, could further validate the generalization capabilities.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are well-designed and appropriate for the problem. The choice of benchmarks, baseline comparisons, and ablation studies supports the validity of the approach.", "theoretical_claims": "The paper does not present formal theoretical proofs in the traditional sense, but it does provide detailed derivations and justifications for key algorithmic components. In the modified GAE section and agent-centric history encoder and centralized critic section, it would be beneficial to show that the modified GAE preserves the convergence guarantees of the original GAE or to discuss any potential limitations introduced by the asynchronous adaptation.", "experimental_designs_or_analyses": "The paper presents a well-designed set of experiments, Overcooked and BoxPushing, to evaluate the proposed Agent-Centric Actor-Critic (ACAC) algorithm. The baselines such as Mac-NIACC, Mac-IAICC, and MAPPO are appropriate. However, I believe a much harder environment such as SMAC will be more beneficial.", "relation_to_broader_scientific_literature": "The paper's contributions are related to MARL and HRL.\nBy addressing the challenges of asynchronous MARL with macro-actions, avoiding the pitfalls of padding-based methods, and adapting GAE for asynchronous settings, the paper extends the state-of-the-art in the field. \nThe use of attention mechanisms and rigorous empirical validation further strengthens the paper's contributions.", "essential_references_not_discussed": "I believe the references are sufficient.", "other_strengths_and_weaknesses": "* Strengths:\n1. The paper introduces a novel approach to handling asynchronous multi-agent reinforcement learning (MARL) by avoiding the common padding technique, which is a significant departure from existing methods.\n2. The proposed Agent-Centric Actor-Critic (ACAC) algorithm has the potential to significantly impact real-world applications where agents operate asynchronously.\n3. The adaptation of Generalized Advantage Estimation (GAE) to asynchronous settings is an original contribution that addresses a gap in the literature.\n\n* Weaknesses:\n1. While the paper provides detailed derivations for the modified GAE, it lacks formal theoretical guarantees or convergence proofs for the proposed methods.\n2. The paper could benefit from a sensitivity analysis to show how the performance of ACAC varies with different hyperparameter settings.\n3. While the chosen benchmarks are appropriate, including additional environments could further validate the generalizability of ACAC, such as SMAC and its variants.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. Can the authors provide formal theoretical guarantees or convergence proofs for the proposed Agent-Centric Actor-Critic (ACAC) algorithm, particularly for the modified GAE in asynchronous settings?\n2. How does the agent-centric approach compare to padding-based methods in terms of information loss or distortion?\n3. Can the authors provide evaluations on the SMAC environment? Choose some representative tasks if possible.\n4. Will the ACAC be influenced by mixed opponent policies? I suggest conducting an experiment on SMAC-HARD to test whether ACAC is free from opponent strategy. Choose some representative tasks if possible.\n\nIf you respond to these questions and address these concerns, I'll be willing to raise the score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6kgC5JWuW", "reviewer": "Reviewer_SHS8", "summary": "This paper proposes an Agent-Centric Actor-Critic framework for asynchronous multi-agent reinforcement learning, which includes a module that addresses asynchrony without relying on padding. The proposed module incorporates agent-centric history encoders for independent trajectory processing and an attention-based centralized critic for integrating agent information. Experimental results demonstrate that ACAC outperforms existing methods in macro-action benchmarks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and would like to clarify our contributions and address concerns as follows:\n\n### **[Q1] Parameter Comparison**\nWe agree that comparing parameters is important, so we compared the total number of parameters between ACAC and Mac-IAICC. The results are as follows:\n- Overcooked/Rand: ACAC (320k) > Mac-IAICC (241k) \n- Overcooked-Large/Rand: ACAC (874k) < Mac-IAICC (963k)\n\nCrucially, ACAC significantly outperforms Mac-IAICC in Overcooked-Large despite having *fewer* parameters.\nThis strongly indicates ACAC's performance gains stem from its architectural design (more effective processing of asynchronous information), not just model capacity.\n\n&nbsp;\n\n### **[Q2] Use of RNN in baselines**\nYes, all macro baselines utilize RNNs as ACAC.\n\n&nbsp;\n\n### **[W1] Practicality of Pre-defined Macro-Actions**\nOur framework doesn't assume uninterrupted macro-actions.\nIn the environments used (e.g., Overcooked), actions *can* be interrupted by other agents (e.g., blocking).\nWhen this occurs, the action terminates early, allowing the agent to immediately choose a new macro-action based on the situation.\nThis provides reactive adaptation to dynamic interactions.\nWhile the environment handles these interruptions, our paper focuses on **ACAC's ability to handle the resulting asynchrony** in decision points for effective learning in the CTDE framework.\n\n&nbsp;\n\n### **[W2] Padding Issue & Critic Novelty**\nWe appreciate the perspective on the padding problem.\nWhile padding is simple, our paper argues and results show its impact is significant in asynchronous MARL, especially complex tasks.\nIt can cause misleading temporal information(padding obscures the actual time elapsed between an agent's valid observations, hiding crucial duration information), inaccurate history abstraction(reusing past observations via padding makes it hard to distinguish new vs repeated stale information, leading to inaccurate joint history representations), and hinder credit assignment. Our experiments (Figs 7, 8) consistently show ACAC (no padding) outperforms padding-based methods, especially in complex settings (Overcooked-Large), suggesting addressing padding is crucial, not minor. This motivated ACAC's design to handle asynchronous inputs directly.\n\n&nbsp;\n\n### **[W3] Component Contribution Verification**\nWe agree ablation studies are crucial, so we have performed ablations for time embedding (\"No TE\") and self-attention (\"No SA\", using MLP) across Overcooked and Overcooked-Rand environments.\nSince the complex Overcooked-Rand-B map showed the clearest and most significant performance differences between configurations,\nclearly highlighting component impacts, we focus on these results here for brevity.\nFull results will be included in the revised manuscript.\nWe report the average final performances and standard errors over five random seeds.\n\n**Table 3**. Ablation study on time embedding (TE) and self-attention (SA).\n|ACAC | No TE | No SA |\n|--------|--------|--------|\n|212.42 ± 0.64 | 174.65 ± 18.62 | 136.69 ± 19.44 |\n\nThese results show that removing either the TE or SA component severely degrades both stability and the final performance,\nconfirming that each component is vital to ACAC's effectiveness.\n\nRegarding the PPO/GAE ablation, directly applying PPO to Mac-IAICC (N critics) is non-trivial.\nHowever, our ACAC vs. ACAC-Vanilla comparison (Sec 5.2, Fig 7) isolates the PPO objective's impact.\nThey share the same architecture; only the optimization differs (PPO vs. simpler policy gradient).\nACAC's superior performance demonstrates the benefit of our PPO-based objective.\n\n&nbsp;\n\n### **[Experimental Designs 3] Validity of Ablation Comparison**\nThis experiment aimed to isolate the negative impact of processing padded/duplicated information, which is difficult within standard padding methods.\nACAC-Duplicate uses the ACAC architecture, including time embedding and per-agent history encoders, but adopts the padding method's history update rule (update on any agent's new observation).\nThis creates a valid intermediate comparison, effectively showing the performance degradation caused by processing duplicated information within our agent-centric framework.\n\n&nbsp;\n\n### **[Experimental Designs 4] Lack of Implementation Details**\nKey architectural details are in Sec 3/Fig 4, with hyperparameters in Appendix F/Table 1.\nPlease feel free to ask if further hyperparameter details are needed.\nWe will release the full source code publicly for reproducibility.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs, particularly the ablation studies, raise several concerns.\n\n1. The ACAC method includes additional encoders and a transformer module, which significantly increases the number of parameters. It is crucial to determine whether the observed performance improvements are attributable to the additional parameters.\n\n2. The effectiveness of each modification in the proposed method has not been sufficiently verified through ablation studies. For instance, the paper integrates timestep information—an ablation comparing ACAC with and without timestep information, and previous methods with this timestep information, would help clarify its impact. Similarly, applying the modified GAE in a PPO version of Mac-IAICC would help demonstrate its contribution.\n\n3. The current ablation study is somewhat unclear. For example, the authors appear to examine the negative effect of duplicated observations, but the ACAC-Duplicate condition does not accurately reflect previous methods, as they do not involve encoders or timestep updates. This might only indicate that the encoder mechanism, when poorly applied, leads to negative effects.\n\n4. The appendix lacks implementation details for the proposed module, which would be beneficial for replicating and understanding the method.", "relation_to_broader_scientific_literature": "This paper makes a contribution to improving the performance of asynchronous MARL by addressing the padding problem inherent in existing methods. The ACAC method may inspire further advancements in asynchronous MARL algorithms.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper is well-written and structured, making it easy to follow.\n\n- Illustrative figures and graphs effectively enhance understanding of the setting and methodology.\n\n- The motivation for the work is clear, and the proposed method is concisely presented.\n\nWeaknesses:\n\n- A key concern is the practical applicability of the setting in MARL. The algorithm requires individually pre-defined macro-actions, which may not be realistic in a multi-agent environment, where the execution of one agent’s macro-action can be disrupted by other agents. For example, one agent’s macro-action may be hindered by others, causing the action to be incomplete or interrupted. The current macro-action framework does not explicitly account for the interactions between agents, which is critical in highly coordinated tasks where agents’ actions are interdependent.\n\n- The padding problem in asynchronous MARL seems relatively minor, and the proposed modification to the critic input does not appear to be significantly novel.\n\n- It remains unclear why the proposed method outperforms existing ones. The performance gains could stem from the additional parameters or the incorporation of time embeddings (as discussed in the experimental part above). Without isolating these factors, it is difficult to determine the source of the improvements. Furthermore, the use of self-attention seems unnecessary in this context.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. What is the total parameters in ACAC compared to the other baselines?\n\n2. Do the other baselines also incorporate RNNs in their value function?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes  an agent-centric actor-critic algorithm that targets the challenge of asynchronous decision-making in multi-agent reinforcement learning by eliminating padding through independent trajectory encoding and attention-based aggregation, resulting in improved learning efficiency and convergence.


The paper demonstrates multiple strengths:

- Methodological novelty and strong motivation: The paper presents a well-justified and innovative solution to asynchronous MARL by avoiding spurious correlations introduced by padding (Reviewer FJpH, Reviewer Cf7Q, Reviewer RGMo).

- Sound theoretical insights: The derivation of a modified GAE formulation for irregular macro-action intervals is logical and recovers standard GAE as a special case (Reviewer FJpH, Reviewer Cf7Q).

- Comprehensive empirical validation: ACAC outperforms strong baselines across diverse macro-action benchmarks with robust ablations and sensitivity analyses (Reviewer FJpH, Reviewer Cf7Q, Reviewer RGMo).

- Clear presentation and accessibility: The paper is well-written, with helpful figures and well-organized supplementary materials, enhancing readability and reproducibility (Reviewer SHS8, Reviewer FJpH).

- Scalability and practicality: The authors demonstrate that ACAC has favorable complexity and runtime characteristics compared to existing methods, even in larger agent populations (Reviewer FJpH).

Prior to the rebuttal, common concerns raised by reviewers included the lack of evaluation in highly coordinated or continuous control environments and the potential impact of added model complexity (Reviewer SHS8, Reviewer RGMo, Reviewer Cf7Q), as well as the unclear contribution of individual components such as timestep embeddings and attention modules (Reviewer SHS8), and limited discussion on convergence guarantees for the modified GAE (Reviewer RGMo, Reviewer Cf7Q). These concerns were addressed through detailed clarifications, new ablation studies, architectural comparisons, and explanations of the macro-action framework’s adaptability to agent interaction.

Therefore, AC recommend acceptance of this paper.
Decision: Accept (poster)