OvercookedV2: Rethinking Overcooked for Zero-Shot Coordination
Authors: Tobias Gessler
Tin Dizdarevic
Ani Calinescu
Benjamin Ellis
Andrei Lupu
Jakob Nicolaus Foerster
Venue: ICLR 2025 Poster (2025)
Keywords: multi-agent reinforcement learning
reinforcement learning
multi-agent systems
zero-shot coordination
overcooked
human-AI coordination
Primary area: datasets and benchmarks

Abstract: AI agents hold the potential to transform everyday life by helping humans achieve their goals.
To do this successfully, agents need to be able to coordinate with novel partners without prior interaction, a setting known as zero-shot coordination (ZSC).
Overcooked has become one of the most popular benchmarks for evaluating coordination capabilities of AI agents and learning algorithms.
In this work, we investigate the origins of ZSC challenges in Overcooked.
We introduce a state augmentation mechanism which mixes states that might be encountered when paired with unknown partners into the training distribution, reducing the out-of-distribution challenge associated with ZSC.
We show that independently trained agents under this algorithm coordinate successfully in Overcooked.
Our results suggest that ZSC failure can largely be attributed to poor state coverage under self-play rather than more sophisticated coordination challenges. The Overcooked environment is therefore not suitable as a ZSC benchmark.
To address these shortcomings, we introduce OvercookedV2, a new version of the benchmark, which includes asymmetric information and stochasticity, facilitating the creation of interesting ZSC scenarios.
To validate OvercookedV2, we conduct experiments demonstrating that mere exhaustive state coverage is insufficient to coordinate well. Finally, we use OvercookedV2 to build a new range of coordination challenges, including ones that require test time protocol formation, and we demonstrate the need for new coordination algorithms that can adapt online.
We hope that OvercookedV2 will help benchmark the next generation of ZSC algorithms and advance collaboration between AI agents and humans.

Published as a conference paper at ICLR 2025
OVERCOOKEDV2: RETHINKING OVERCOOKED FOR ZERO-SHOT COORDINATION
Tobias Gessler* Benjamin Ellis
Tin Dizdarevic
Anisoara Calinescu Jakob N. Foerster
Andrei Lupu
FLAIR, University of Oxford
ABSTRACT
Al agents hold the potential transform everyday life by helping humans achieve their goals. To do this successfully, agents need to be able to coordinate with novel partners without prior interaction, setting known as zero-shot coordina- tion (ZSC). Overcooked has become one of the most popular benchmarks for evaluating coordination capabilities of Al agents and learning algorithms. In this work, we investigate the origins of 7SC challenges in Overcooked. We introduce a state-augmentation mechanism that mixes states that might be encountered when paired with unknown partners into the training distribution, reducing the out-of- distribution challenge associated with ZSC. Our results show that ZSC failures can largely be attributed to poor state- caaeecooeragee rather than more sophisticated coordi- nation challenges. The Overcooked environment is therefore not suitable as ZSC benchmark. To address these shortcomings, we introduce Overcooked a new version of the benchmark, which includes asymmetric information and stochas- ticity, facilitating the creation of interesting ZSC scenarios. To validate Over- Vo0 we demonstrate that mere exhaustive state coverage is insufficient to coordinate well. Finally, we use OvercookedV2 to build new range of coordi- nation challenges, including ones that require test-time protocol formation, and we demonstrate the need for new coordination algorithms that can adapt online. We hope that OvercookedV2 will help benchmark the next generation of ZSC algorithms and advance collaboration between agents and humans.
INTRODUCTION
A1 agents promising revolutionise our daily lives, from digital assistants (Bai et al.) 2022 Team 2024) to advanced robotics (Leal et al. [223]] Ahn et al. 2024). To achieve their promise. Al agents must be able to collaborate with humans and help them achieve their goals (Kapoor al.) 2024 Doing this successfully requires making decisions where the optimal behaviour depends on the actions of other agents, including humans. Unlike humans, who naturally navigate these situations using theory of mind to reason about the mental states of others (Premack & Woodruff] 1978), current Al agents struggle to adapt their behaviours. Building agents that can coordinate well with humans is therefore one of the fundamental quests of AI (Gupta et al. 2024).
Cooperative multi-agent reinforcement learning (MARL) holds promise for addressing coordination problems but has thus far struggled to handle the complexity of the real world. Most research has focused on learning policies for entire teams, referred to as self-play (SP) (Gupta et al,) 2017). Methods such QMIX (Rashid et al.) 2018) and COMA (Foerster al.) (017) have achieved strong performance on the popular cooperative benchmark SMAC (Samvelyan et al. 2019). While SP agents perform well when interacting with each other, they often struggle when paired with unknown partners since they rely heavily on conventions formed during training. With the success of MARL algorithms, there has been a growing interest in building agents that can coordinate with novel partners. One of the main settings this work has focused on is zero-shot coordination (ZSC).
The goal for ZSC, as introduced by Hu (2021b), is to construct an algorithm that allows agents to be trained independently yet coordinate effectively at test time. This approach serves as proxy
"Correspondence to tob as wtt.e..essleeee............. tobb Available JaxMARI: httpt::t///....... http:::////ittuu.coomm PLAI ROX JaXMARI.

---

Published as a conference paper at ICLR 2025
for the independent reasoning processes of humans (Hu et al. in cross-play (XP), where independently trained models are paired together. Most ZSC research has evaluated on the Hanabi benchmark (Bard et al. 2020), that focuses reasoning about the beliefs and intentions of other players. This faces the risk of overfitting to the environment Gorsane et al. (2022), and Hanabi's complexity makes training computationally expensive, limits agent inter- pretability, and is disconnected to real-world tasks. In contrast, the Overcooked benchmark (Carroll al. 2020), where agents prepare dishes in a restaurant setting, provides easier visualisation and interpretation of agent behaviours in task inspired by the real world. It has been widely used to study Auman-Al coordination (Strouse al. 2022; Zhao et al., 2022 Yu et al., (023). Despite the benchmark being fully observable, coordination with humans has shown to be challenging.
2 ZSC is commonly evaluated
In this work, we analyse the origin of 7SC difficulties, as measured by the SP-XP gap, in Over- cooked. We show that the SP-XP gap is mostly caused by insufficient state-coverage rather than coordination challenges. For example, agents might adopt slightly different movement patterns or timing, causing their partners to encounter unfamiliar states and fail entirely. By employing a state augmentation algorithm during training, we nearly close the SP-XP gap across all layouts. Our al- gorithm exposes agents to states they might encounter when paired with unknown partners during training, thereby eliminating the out-of-distriuttion generalisation challenge. State-coverage is one aspect that makes ZSC challenging; however, ZSC may also require learning grounded communica- tion strategies or test-time protocols. Thus, solving state-coverage alone is not equivalent to solving ZSC. As a result, Overcooked is inadequate for benchmarking ZSC algorithms.
To address these shortcomings. we introduce novel environment, OvercookedV2. that re- quires agents to coordinate for high returns. The environment is implemented as part of the popular JaxMARL framework (Rutherford al., 2023). We introduce meaningful par- tial observability by restricting the agents' ob- servations and introducing hidden information sources to ensure that agents cannot simply in- fer unknown information without interpreting the behaviour of others (Ellis et al. [003 Foer- ster et al.] 2019). Moreover, we implement dis- tinct communication channels and configurable
Agent View Radius (Blue Agent)
Cooking Pot
Agent
Ingredient Pile
Button Indicator
Recipe Indicator
Plate Pile
Delivery Station
Figure 1: Overview of an OvercookedV2 layout with multiple ingredients, a dynamic recipe indi- cator, and an agent view radius of one cell.
stochasticity, allowing us to create interesting coordination challenges. This is a significant advan- tage over Hanabi, which offers only one complex evaluation setting. We provide three handcrafted classes of layouts with specific coordination challenges, along with a user interface for custom lay- out creation. First, we introduce a temporally extended version of the toy Cat-Dog coordination problem introduced by Hu et al. (2021a). We then present class of layouts where agents must form protocols at test time to succeed at ZSC, i.e., scenarios where not possible to coordinate via a fixed policy. Finally, we implement layouts where agents have to rely on the "demoing" of recipes.
To validate OvercookedV2, we demonstrate that the new coordination scenarios cannot be solved by mere state-coverage alone. Moreover, while our results show that 7SC methods such other- play (Hu et al. 2021b) can improve ZSC performance, all methods struggle with many of the new scenarios, particularly those requiring test-time adaptation. This challenge requires fundamentally new methods capable of adaptive coordination, highlighting an open challenge for ZSC. We hope that OvercookedV2 will help benchmark the next generation of ZSC algorithms and aid in the de- velopment of AI systems that can effectively coordinate with both humans and other agents.
RELATED WORK
MARL Environments. In multi-agent setting agents are required either to collaborate or compete with each other. Agents must communicate with each other, develop specialised behaviours, and be robust to the failure of other agents (Gronauer & Diepold 2022). Standardised environments have facilitated great progress in RL (Bellemare et al. 2013:; Brockman al.] (016). For cooperative MARL, the SMMAC/Samellyy et al. (2019). based on StarCraft II, has become popular. Its limited stochasticity and partial observability led to the development of SMACv2 (Ellis et al., 2023). For
2

---

Published as a conference paper at ICLR 2025
evaluating human-Al cooperation and ZSC, the community primarily used the Hanabi (Bard al. 2020) and Overcooked (Carroll et al. (0200) benchmarks. Most ZSC algorithms have been evaluated in the Hanabi environment (Hu et al. 2021b Lupu al. 2021). which focuses reasoning about the beliefs and intentions of other players. However, the complexity of Hanabi makes it difficult to gain insights into agent behaviours. Additionally, the holistic nature of Hanabi makes training computationally expensive and limits the ability to focus on isolated coordination problems.
Overcooked. The Overcooked benchmark, introduced by Carroll al. (2020), is based on the pop- ular video game Overcooked. Players control chefs in a kitchen setting, with the goal of delivering dishes. To deliver a dish, players must work together to complete several high-level tasks. The benchmark attempts to make strategy coordination difficult, in addition to the motion coordination challenge. Players are placed in a grid grid-world environment containing interactive objects such as cooking pots, onion dispensers, plate dispensers, delivery stations, and empty counters. Players can navigate the environment and interact with these objects. They can perform sequences of interac- tions to prepare, cook, and deliver dishes. The goal is to deliver a dish which rewards all players equally and removes the dish from the game. The Overcooked benchmark has been widely used to evaluate human-Al coordination algorithms (Strouse et al., 2022 Zhao et al. 2022; Yu al. 2023) Recent work has explored language model-based agents in Overcooked (Liu et al. 2024 Tan et all, 2024 Li al., 2024). However, since the benchmark is fully observable without asymmetric infor- mation agents might not need to communicate. Moreover, Knott et al. (2021) show that RI, agents in Overcooked lack state diversity, which raises questions of their influence on ZSC failures.
3 BACKGROUND
Dec-POMDPs. A Dec-POMDP is defined as tuple (n, A,TT,,,,,,, (Olihhock et al.y 2008). Here, n E N denotes the number of agents, S the state space. A is the action space for each agent, and O represents an agent's observation space. The transition probability func- tion T S x A" + ((S) dictates the evolution of the environment, the observation function O S A + A(O") determines what agents can perceive, and R : SSX A + A(R) is the reward function. Finally, E 10 is the discount factor. At every timestep t, each agent selects an action oj E A. The environment then transitions from state 8 S to 8141 E S according to T(s;, a) with probability P(s++1|s at), where a, ................................ ..a)) represents the joint action. A reward rg E IR is drawn based on R(s,, az), and each agents receives an observation 0/_1 with the joint observation Oc+1 = (0.41 of41 sampled from O( at). The primary objective is to learn policies n; (O' x A)" + ((A) for each agent i, which maximises the expected cumulative discounted reward EETT T 7"" -rr]
Partial Observability and Stochasticity. Partial observability and stochasticity are required for a problem to be truly multi-agent. EElis et al. (2023) show that when the initial states and transition dynamics of a Dec-POMDP are deterministic, open-loop policies suffice. This means there exists an optimal joint policy where the only required information is the timestep and agent ID, effeee tively reducing the problem to a predetermined sequence of actions. Open-loop policies render the observation function-and any partial observability it -irreleeenrre.. Partial observability only becomes meaningful when the environment is sufficiently stochastic to require closed-loop policies. However, even in stochastic environments, not all partial observability significant, for example, when it can be inferred from available observations. In fully observable settings, each agent can in- dependently compute all optimal joint policies based solely on the current state, rather than relying on the cetion-obseevaatie so, they will only fail to coordinate if they choose joint policies that are immediately incompatible.
history. It can then enact its part of one such joint policy. If both agents do
Zero-Shot Coordination. Self-play is the predominant approach for learning in Dec-POMDPs, where agents are trained and evaluated together. For n agents, with æ denoting the joint policy and the i-th agent's policy component, self-play optimises л²* arg max, [[T] #2, TT). Complex Dec-POMDPs often have multiple maxima, leading to distinct optimal policies that depend on arbitrary conventions which often fail when paired with independently trained agents. However, in many real-world scenarios, agents are required to cooperate with unknown partners or humans at test time. Hu et al. (2021b) formalised the ZSC problem, where the goal is to develop algorithms that enable agents to be trained independently yet coordinate effectively at test time. This can be seen proxy for the independent reasoning processes of humans (Hu et al. 2021a and explicitly
3

---

Published as a conference paper at ICLR 2025
10.0
10.3
10.33
1.00
10.03
3.50
10
10.00
d 20
1.00
10.00
50.000
5.0
2.5
Allce
2.00
10.02
1.20
23.00
50.31
0.0
0/4
10.98
10.09
2.20
10
9.40
2.5
30.00
19.93
10
9.48
$0.98
10.
2N
20.33
30.03
7.5
2.00
3.40
2.20
3.44
0:00
2.00
3.00
2.00
10.0
0
2
3
4
6
7
8
BR
Alice
Beb
Bob
Figure 2: Button Game Cooperative guessing game with many equivalent communication ac- tions. Alice (left) observes a pet, either a cat or dog, and Bob (right) must guess the pet. Al- ice must press one of N buttons, activating one of 2/ light bulbs. The bulb's parity encodes the pet's identity, which Bob observes before guess- ing the pet. The game does not require coor- dination as Bob can always guess correctly by looking at the parity of the bulb. Correct/incor- rect guesses are rewarded with +/-10 points.
Figure Cross-play matrix for 10 independent SP agents and best response (BR) to uniform random agent (U). The game admits a globally optimal strategy for Bob. SP agents succeed in self-play but fail to coordinate with novel part- ners since they overfit specific buttons and fail to generalise. The BR agent achieves perfect scores in all pairings. This demonstrates that ap- parent coordination issues can in some cases be entirely explained by lack of state coverage.
rules out reliance on arbitrary conventions as optimal solutions. A common approach to evaluate ZSC algorithms cross-play, where independently trained models are paired together. We refer to the difference in performance between self-play and cross-play as the SP-XP-gap. If different training runs of the same method struggle to coordinate, is unlikely that these agents will succeed with other unknown agents, let alone with human players. Therefore, cross- cross-play serves as a cheap and effective proxy for assessing ZSC performance.
4 MOTIVATION: GENERALISATION ACROSS STATES VS COORDINATION
The SP-XP gap observed in ZSC can be caused by either suboptimality, brittle optimal policies, fundamentally incompatible conventions (what refer Sub-optimality results from training runs being unstable or converging to local optima and is not a coordination-speciff issue. Making sure that agents generalise across states is generally goal in RL, but particularly relevant in ZSC, where independently trained agents can adopt similar strate- gies but with arbitrary variations in execution, such as different movement patterns or timing. While these differences are per-se irrelevant, they can cause partners to encounter unfamiliar states, poten- tially leading task failure if agents lack robustness. The Cat-Dog toy problem introduced by IHu et al. (202 illustrates a genuine coordination challenge where successful ZSC requires learning a grounded communication strategy which incompatible with optimal SP policies. In contrast, consider the Button Game (Figure 2 Appendix A.I)) adaptation of the Cat-Dog problem. There is no need for Alice and Bob to establish specific communication convention, as the information about the pet's identity is always transmitted to Bob, regardless of which button Alice presses. The game does not pose an actual coordination challenge, as agents can achieve a perfect score with unknown partners by generalisation across states alone.
"true coordination challenge").
Experiments in Toy Environment. We train 10 independent agent pairs in self-play using in- dependent Q-learning, as well BR uniform random agent for Alice. The code for this experiment is implemented in a Jupyter notebook and is available for viewing and execution in Google Colat The results are illustrated Figure 3 Despite the lack of coordination challenge, agents still fail in a ZSC setting as they overfit to specific button choices and fail in unseen states.
httpss:/coll reseaaaabbeeeeeeeeeee resear www.aaa..brreheeeeeeeeeee
ccm i (5w003 MMgU5

---

Published as a conference paper at ICLR 2025
Table 1: Results of SP and XP scores of 10 PPO agents with distinct seeds in self-play and state- augmented SP for all layouts across 500 episodes (mean + standard deviation). The XP score is reported as the mean of all cross-play pairings. The gap represents the difference between SP and XP scores. The SP-XP gap is significantly reduced by training in the State- hateeAugeetted setting, demonstrating that much of was attributable to poor state coverage of SP policies.
Standard
State-Augment
Layout
SP
XP
Gap
SP
XP
Gap (A)
Cramped Room
259 +
257 +
2
259
255 +
4 (2)
Asymm. Advantages
278 +
278 +
0
277 + 1
276 +
(1)
Coordination Ring
214 + 8
96 72
118
269 + 44
198 + 50
71 (-47)
Forced Coord.
199 +
138 70
61
200 2
193 21
8 (-53)
Counter Circuit
163 - 14
59 2 57
104
184 + 15
140 38
44 (-60)
LIMITATIONS OF OVERCOOKED FOR ZSC
Training RL agents that can coordinate with humans in Overcooked has repeatedly shown to be difficult (Strouse et al. 2022 Yang al., 2022 Yu et al. 2023). Carroll et al. (2020) provide web interface that allows users to interact with trained deep RL agents. Interacting with the provided RL agents, including self-play, copulation-based training, and human-aware PPO agents, reveals significant limitations in their capabilities. Even in the simplest layout, cramped room-where all steps required prepare onion soup can be completed by both players and which primarily presents basic motion coordination challenge-the agents struggle in certain scenarios. When human players introduce variability, such as placing items like plates on multiple counters, the RL agents fail to make meaningful progress. Although all necessary steps prepare additional onion soups remain within the agent's 's capabilities, they often exhibit unproductive behaviours. For instance, they remain stationary in corners engage in repetitive, non non-goal-oriented actions such as repeatedly placing and picking up an onion from a counter. This observation led us to investigate the origins of the SP-XP gap in Overcooked, which analyse in this section.
5.1 EXPERIMENTAL SETUP
To study the origin of the SP-XP gap, we train ten independent PPO dde Witt et al., 2020: Schulman et al. 2017) agents in standard self-play and in self-play with augmented starting states for each episode. Specifically, we propose an iterative algorithm to collect these starting states. Before training we rollout episodes for each policy pairing (mi; TTj and collect the resulting trajectories T,- From these trajectories, we select every tenth state and store them in a state buffer B. We then train each policy independently in self-play, with the starting state of each episode sampled uniformly from B. This process is repeated for ten iterations during training with each policy being trained for the same total number of timesteps as in the standard settings. This algorithm exposes agents to states during training, that might have otherwise caused agents to go off distribution during cross- play. However, it is important to note that we still train exclusively in self-play; at no point are two different policies trained together, therefore preventing the formation of shared conventions during training. We then evaluate the cross-play performance of all pairings within each of the two populations. Agent implementation details and an overview of the aate-auumentttion algorithm are provided in Appendix A.2
5.2 RESULTS AND DISCUSSION
Our results show that agents trained in the state-augmented setting perform well across all layouts in cross-play (Table T). In the Cramped Room and Asymmetric Advantages layouts, even agents trained via standard self-play resulting in an almost negligible SP-XP gap. However, in all other layouts, standard self-play agents show a significant SP-XP gap when paired with unknown partners. By training with augmented
collaborate effectively with all other independently trained agents,
httpp::/hummncomppaat
.gi thub. www.vo/oooooooeddeeeeee
5

---

Published as a conference paper at ICLR 2025
Standard
State-Augmented
250
259.00
24
63:12
103.00
0.66
3.00
33.00
$38.00
6:90
159.70
86.00
123.00
96.72
134.86
$58.54
25
130.92
125-96
255.28
-20
29
176.0
17.20
8.16
76.92
1.91
0.40
25.00
12477
119.4
11220
18140
67.92
114.00
127.4
8324
106.64
124.76
122.44
124.8
16
136.04
136.00
3.24
27.14
3.
66.11
14911
67.56
172.66
96.04
336,03
371.08
131.76
77.35
174.64
-10
Policy
130.24
352.00
7:58
133.00
133.09
3.52
59.16
2.64
21.00
24 36
6.64
7.22
177 04
132 92
27-46
37.60
274.00
153
$41.32
122.00
135
135.35
159.03
243.28
152.64
134.36
162.66
100
27.99
26
03.62
49.40
26.90
17:02
1.44
26.36
127.36
1992-
131.01
128.2
196.4
153.1
335.34
166
166.20
50
256.488
150.4
27.16
21.94
102.02
76.80
23.92
22.13
31
114.20
195-4-
132.5
72.14
298.0
226.16
73.00
187.20
196.66
101.6
137.00
2
3
6
7
7
0
Policy 2
Policy 2
Figure 4: Cross-play matrix for the standard and state-augmented settings in the Counter Circuit layout. Ten agents were independently trained for each setting; each cell (i. j) of the matrix rep- resents the average score across 500 episodes played by the ith and jth agents. Agents trained in the standard setting achieve high SP scores (diagonal), but most XP pairings perform poorly or fail completely. the state-augmented setting, SP scores are slightly higher than in the standard setting, and XP scores are dramatically improved, with no total failures.
states, this gap is almost entirely closed in the Forced Coordination layout. Moreover, in the Co- ordination Ring and Counter Circuit layouts, state-augmented agents also perform well across all cross-play pairings, with no complete failures. Training curves and cross-play matrices for all lay- outs are provided in Appendix A
Results. In the cramped room and asymmetric advantages layouts, all agents achieve high SP and XP scores, averaging around 260 and 280 respectively. This indicates that there are few high payoff strategies, therefore agents do not encounter unknown situations in XP pairings, which would make them fail. In the Coordination Ring layout, standard agents achieve SP scores of 200-220, but exhibit a significant SP-XP gap. This likely due to two equally viable strategies that lead to off-distribution failures in cross-play. Agents trained with starting state augmentation improve, achieving an average XP score of 198, with no total failures. In the Forced Coordination layout, standard agents perform well in self-play but some cross-play pairings fail entirely. Starting state augmentation closes the SP- XP gap, with only an eight-point difference. Lastly, in the Counter Circuit layout standard self-play agents perform well, reaching an average total reward of 163 as illustrated in Figure In cross pross-play, some pairings maintain these high scores, but many achieve only low scores, and some fail entirely. Stgme-a agents outperform standard agents, achieving better SP scores and significantly higher average XP score 140, demonstrating that the SP-XP gap can be effectively closed by sufficient state-coverage. This layout is larger compared to the others, inereasing the number of possible states that agents might encounter when paired with unknown partners.
Discussion. The SP-XP gap is largely attributable poor coverage, rather than policy incon- sistency. By augmenting starting states during training, we reduce the out-of-distribution challenge, leading to significant improvements in cross-play performance across all layouts. In some layouts, even self-play-trainge trained agents collaborated effectively with independently trained agents. This is likely due limited number of high-return strategies in those layouts. Most Overcooked layouts have straightforward solutions with few alternative conventions, making ZSC performance more about state-coverage than coordination. Additionally, the fully observable nature of the benchmark does not require agents to communicate asymmetric information. When agents are sufficiently gen- eral, they can effectively solve the game effectively with unknown partners. Our findings show that Overcooked is not particularly hard coordination challenge in ZSC. Instead, the main difficulty lies in state-cover coveercove suggesting the need for benchmarks that emphasise coordination for high rewards rather than generalisation alone.

---

Published as a conference paper at ICLR 2025
6 OVERCOOKEDV2
As discussed in the previous section, the original Overcooked environment has significant short- comings. To create a new version of the environment where agents must coordinate to achieve high returns, we propose three major changes: limiting agent observations with view radius, introduc- ing asymmetric information, and randomising starting positions and directions. By implementing these changes, we introduce meaningful partial observability and increase the stochasticity of the environment, addressing the issues identified earlier. The main objective of the game remains un- changed: agents must prepare and deliver dishes, earning rewards for each successful delivery. For more details on the game Overcooked, refer to Section2] Our novel environment OvercookedV2 is implemented as part of the JaxMARL framework (Rutherford al. (023), a library that combines JAX implementations of popular MARL environments and algorithms. Implementation details and additional features OvercookedV2 are available in Appendix B
6.1 ENVIRONMENT DYNAMIC'S
Partial Observability. To introduce partial observability, limit each agent's observations to a configurable radius around their current position. However, this alone insufficient to achieve meaningful partial observability. To create scenarios where agents must communicate to maximise rewards, the environment must also include asymmetric information. We therefore extend the en- vironment to feature a configurable number of ingredients, each with its own ingredient dispenser. Now, the environment supports multiple distinct recipes, where each recipe consists of multiset of three ingredients. We extend the game by adding a recipe to the global state, denoted as R. At the start of each episode, recipe R is sampled from set of possible recipes R. with an option to resample after each successful delivery. Asymmetric information is introduced in a layout through recipe indicator block, which displays the recipe that needs to be cooked and delivered. This indi- cator is only visible to agents within its view radius. A reward is given if the delivered dish matches the indicator, with an optional parameter to impose negative reward for incorrect deliveries.
Our environment introduces additional sources of stochasticity, most notably through the recipe sampling process, detailed in the previous paragraph. The probability P(R) of lecting a recipe from R. is equal for all recipes. We also provide varying levels of stochasticity for agent initialisation. In the default case, agents are placed predetermined positions, all facing upwards. as in the original environment. Alternatively, starting positions and orientations can be randomised, sampled uniformly from all reachable positions for each agent. This is particularly relevant in layouts with separate rooms that restrict agents' access to certain items and information. All interactions with objects, such as pots and counters, remain deterministic.
Stochasticity.
Communication Channels. To collaborate effectively in the presence of asymmetric information, agents must communicate. In Overcooked, agents can exchange arbitrary signals by placing items on counters within another agent's view radius or by observing each other's movements. Arbitrary communication allows agents to exchange signals that hold no inherent meaning and can be used to establish communication protocols. In contrast, grounded communication provides agents with verifiable information, independent of the actor's intentions. To enable grounded communication, introduce new block that reveals the recipe when agent interacts with it, remaining visible for a configurable number of timesteps. Pressing the button incurs a configurable negative reward. Additionally, we implement an option provide agents with signal for successful delivery.
6.2 LAYOUTS
The grid-world nature of Overcooked allows the environment to be configured with different layouts to investigate different aspects of coordination. To design a new layout, one can simply provide an ASCII string that describes the layout configuration, along with an optional list of possible recipes. A detailed explanation and example for the interface is provided in Appendix B.5.2 We provide pre-configured layouts for all scenarios present in the original Overcooked benchmark and adapted versions of these layouts with multiple ingredients. Additionally, we introduce three classes of handcrafted layout configurations that pose interesting coordination challenges. Our layouts feature non-obvious ZSC solutions, which cannot be solved effectively with arbitrary conventions. Each class is described in the following, and overview of the layouts is shown in Figure
7

---

Published as a conference paper at ICLR 2025
Figure 5: Handcrafted coordination challenges. We provide three classes of layouts, each with simple (top row) and a more complex (bottom row) configuration. From the left to right: Grounded Coordination Simple/Ring layouts present a temporally extended version of the toy Cat-Dog coordi- nation problem introduced by Hu et al. (202Ta). Test-Time Protocol Formation Simple/Wide layouts require agents to form protocols at test-time, based on feedback they receive after delivery. Demo Cook Simple/Wide layouts require agents to rely the meaning of the other agents' actions.
Grounded Coordination. We temporally extend the Cat-Dog coordination problem (Hu et al, 2021a) into layout. Only one agent can observe to recipe indicator and the other one must prepare dishes. The agent observing the indicator can opt to use arbitrary signals ingredients counters or performing certain movements in the view radius of the other agent. Moreover, we include a button indicator as grounded communication channel. where the agent can press button and, for cost, reveal the current recipe. For agents to succeed in ZSC they need to rely on grounded communication and ideally form protocols at test-time. In self-play agents would adopt arbitrary conventions cireumvent paying the cost of pressing the button, which will fail in ZSC setting. We expect OBL to perform well by learning to communicate using the button indicator.
form of placing
Test-Time Protocol Formation. These layouts again include asymmetric information, requiring agents to communicate. While no grounded coordination channel is available in this layout, agents receive feedback when delivery successful. To play with unknown partners, agents can the grounded delivery feedback to form arbitrary protocols at test time. Again, self-play agents will establish arbitrary conventions during training to solve the communication effectively. Instead of learning fixed conventions during training, agents have to adapt and coordinate on communication protocols test time succeed at ZSC these layouts. We also expect OBL struggle this layout, agents do not have grounded way to communicate. The formation of protocols at test time requires novel ZSC methods capable adapting online to other agents.
Demo Cook. In the demo cook layouts, agents operate under asymmetric information with no grounded communication or feedback. The layouts require implicit communication through actions, as the choice of ingredient placed in the pot conveys information about the recipe. For ZSC, agents must interpret these actions and optimally use the common understanding to form test-time proto- cols. Communication through implicit actions is common for humans; however, might present significant challenge for Al agents.
OVERCOOKED V2 EXPERIMENTS
In this section, we present the results of several experiments conducted the OvercookedV2 envi- ronment. We introduce new model architecture designed for learning in the complex, partially ob- servable coordination scenarios created by OvercookedV2 and demonstrate its strong performance. We then compare the ZSC performance of various algorithms across different layouts in our environment. Our results show that OvercookedV2 presents hard coordination challenges and that good 7SC performance merely issue of state coverage.
8

---

Published as a conference paper at ICLR 2025
Table Results SP and XP scores for seeds with different methods in multiple layouts across 500 episodes (mean + standard deviation). The XP score is reported as the mean of all possible pairings. The Gap represents the difference between SP and XP. Both self-play and state augmented agents perform well in SP, but score dramatically lower in XP for most layouts. Other-Plas SP scores are slightly lower than self-play agents scores, however XP scores improve most layouts. All methods struggle in cross-play demonstrating that the new scenarios provide hard ZSC challenge.
Layout
Method
SP
XP
Gap
Self-Play
154 - 10
-55 124
210 - 51
Grounded Coord. Simple
State-Augmented
156 - 29
-17 115
176 - 69
Other-Play
122 + 24
4 73
116 + 47
Fictitious Co-Play
64 + 58
46 = 47
22 -2 45
Self-Play
164 10
12 75
175 + 28
Grounded Coord. Ring
State-Augmente
165 +
-16
183 26
Other-Play
121 + 36
-9 21
131 + 37
Fictitious Co-Play
86
6 46
79 29
Self-Play
145
-81 99
220 + 26
Test Time Simple
State-Augmented
161 18
555 103
230 130+0 53
Other-Play
221377
-3 51
131 + 50
Fictitious Co-Play
35::44
6 29
25 47
Self-Play
194
-30 96
220 31
Test Time Wide
State-Augmented
137 + 82
-12 72
142 + 81
Other-Play
175
-11 42
193 - 61
Fictitious Co-Play
95 56
23 40
68 = 50
Self-Play
172 19
36 82
138 + 48
Demo Cook Simple
State-Augmented
170 16
112 50
54 + 36
Other-Play
14
47 48
96 19
Fictitious Co-Play
22 48
1 25
25 15 48
Self-Play
195 +
40 61
140 + 13
Demo Cook Wide
State- Autte-Augeeteed
192
60 = 60
138 + 23
Other-Play
162 + 23
62 53
104 + 45
Fictitious Co-Play
45 43
13 = 28
31 34
Experiment Setup. We find that models using a similar architecture as proposed by Carroll et al. 2020) did not learn effectively under the additional complexity in OvercookedV2. We modify the network mainly by adding three 1x1 convolutions to transform the sparse observations of the environment. In addition, add recurrent layer to the network. Furthermore, other-play a ZSC algorithm to prevent coordinated symmetry breaking. A limitation of other-play is that the set of symmetries must be known beforehand. This is particularly relevant in Overcooked, where we cannot define universal set of symmetries for the entire environment: instead, symmetries depend on the specific layout. our layouts we permute the recipe ingredients while keeping all other ingredients fixed. Moreover, we evaluate the population based ZSC algorithm FCP (Strousc) al. 2022), where we learn an agent as the best response their past checkpoints. Implementation details are available in Appendix C.1
implement
population of self-play agents and
Results Adapted Layouts. Agents trained with our new architecture achieve high scores partially observable versions of all original overcooked layouts. They efficiently use of both pots present in the layouts to cook multiple dishes simultaneously; a behaviour that previous policies struggled evaluate agents in adapted version of these layouts with multiple ingredients. Agents are able to maintain strong performance these layouts even while conditioning on the recipe indicator. Detailed results can be found in Appendix
with. Furthermore,
Results Coordination Challenges. All methods struggle to perform well in XP across all layouts, shown in Table 2 In the Grounded Coordination layouts, self-play and state- autteetteented agents achieve similar SP scores. However, in cross-play, -lay, state-augmente agents show an improvement for the simpler version of the layout, whereas in the ring layout, XP scores are similar. In the Test-Time

---

Published as a conference paper at ICLR 2025
Protocol layouts, agents also experience slight improvements with state augmentation. Other-play agents achieve slightly lower SP scores but generally improve in XP across most layouts. In the Demo Cook layouts, agents perform better in XP compared to the other layouts, though the SP-XP gap remains significant. FCP SP scores are generally lower than those of self-play agents. However, even in cross-play, FCP agents achieve positive average score, though these scores remain much lower than the SP scores of self-play agents. None of the methods successfully close this gap.
Discussion. When inspecting the animation of episodes it is clear that self-play agents rely on arbitrary conventions such as placing an ingredient on the middle counter to communicate. When paired with an unknown partner these conventions do not hold, and agents exhibit unproductive behaviours such as stalling completely or constantly delivering an incorrect recipe. We provide a collection of episode visualizations to demonstrate these behaviours" While other-play helps improve ZSC performance in most layouts, the scores are still far worse than in self-play. This is particularly evident the Test-Time Protocol layouts, where agents experience large SP-XP gap. We also observe that FCP agents score significantly lower than self-play agents, even in SP. This could be due to SP eval being out of distribution for FCP agents. Moreover, we use relatively small population size, FCP could benefit from larger populations. Overall, OvercookedV2 presents a challenging ZSC environment that none of the evaluated methods solve successfully.
8 CONCLUSION
Our work revealed significant, largely unexplored, limitations of the Overcooked benchmark, which has been widely used to benchmark human-Al coordination. The finding that Overcooked does not present a meaningful ZSC challenge raises concerns about previous methods evaluated solely in this environment. Evaluation of ZSC algorithms in single environment, i.c., Hanabi, leading to community-wide overfitting to this environment, is a common issue in MARL research (Gorsane jet al., 2022). To address this, introduced OvercookedV2, providing more challenging and flexible coordination benchmark. The benchmark, despite its complexity, cannot represent all ZSC challenges; evaluation should therefore not be limited to this benchmark. We provide an initial set of layouts featuring handcrafted coordination scenarios and welcome contributions from the commu- nity OvercookedV2. We anticipate that OvercookedV2 can help benchmark the next generation of ZSC algorithms and aid in the development of collaborative AI systems that can coordinate with both humans and other Al agents.
ACKNOWLEDGEMENTS
AC acknowledges funding from UKRI AI World Leading Researcher Fellowship (Grant EP/W002949/1) and from a JPMC Research Award. JF is partially funded by the UKI grant EP/Y028481/1 (originally selected for funding by the ERC). JF is also supported by the JPMC Research Award and the Amazon Research Award.
REFERENCES
Michael Ahn, Debidatta Dwibedi, Chelsea Finn, Montse Gonzalez Arenas, Keerthana Gopalakr- ishnan, Karol Hausman, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Sean Kirmani, Is- abel Leal, Edward Lee, Sergey Levine, Yao Lu, Isabel Leal, Sharath Maddineni, Kanishka Rao, Dorsa Sadigh, Pannag Sanketi, Pierre Sermanet, Quan Vuong, Stefan Welker, Fei Xia, Ted Xiao, Peng Xu, Steve Xu, and Zhuo Xu. AutoRT: Embodied Foundation Models for Large Scale Or- /200l/24411.m
chestration of Robotic Agents, July 2024. URL 1rXiv:2401.1293 [cs].
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization, July 2016. URL htttp://arxin http:///xxvvv..g//aaa/ (0ooo//600000550 arXiv:1607.064561 [cs, stat].
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson
Link to collection of OvercookedV2 episode visualizations: wwt.s///ww//fff lpK4
https: pttps://drive... ge..geee...oocoomm
trive/folders wwwpss/stPPPPPrPrr
10

---

Published as a conference paper at ICLR 2025
Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hemandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, April 2022. URL.ttt. /tttt:////xiii..t////////tt wrg. abs 2204 05862 arXiv:2204.00 0552222202222222 [es].
Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, lain Dunning, Shibl Mourad, Hugo Larochelle, Mare G. Bellemare, and Michael Bowling. The Hanabi Challenge: A New Frontier for AI Research. Artificial Intelligence, 280: 103216, March 2020. ISSN 00043702. doi: 10. 00506 urXiv1192.000500 [cs, stat].
URL http:://axiiv www.//tww....aaaaoooooaoooeoeaaa absgg//b//190000
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learn- ing Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelli- gence Research, 47:253-279, June 2013. ISSN 1076-9757. doi: 10.1613/jair.391.. URL
http://arxiv. org/ http://bb/1220.. 4708 arXiv:1207.4700 [cs].
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Me Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//git cttpuu.coomeggg www//ooool//aaa...o.ooo
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym, June 2016. htttt:://axwi..rrg/////
01540 arXiv:1606.013 540 [cs].
Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. On the Utility of Learning about Humans for Human-AI Coordination, January 2020. htRLtttp://aaet
orq /ab////b////// 1910. C5789 arXiv:1910.05789 [cs, stat].
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling, December 2014. URL http:
//
/111/ 3555 #rXiv:1412.3555 [es].
Brandon Cui, Andrei Lupu, Samuel Sokota, Hengyuan Hu, David Wu, and Jakob N Foerster. ADVERSARIAL DIVERSITY IN HANABI. 2023.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Independent Learning All You Need in the StarCraft Multi-Agent Challenge?, November 2020. URL http:: orgg abs/abss/aa0 /0tp./21....0 09533 arXiv:201 1.025333 [cs].
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine, Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design, February 2021. URL [cs].
02096 arXiv:2012.02096
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob N. Foerster, and Shimon Whiteson. SMACv2: An Improved Benchmark for Coopera- tive Multi-Agent Reinforeement Learning, October 2023. URL 2212. 07489 arXiv:2212.07489 [es].
wwwp:iivvvrrraoooomo abs/
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual Multi-Agent Policy Gradients, December 2017. URL TRLLhtpp: httt::///t/xi/// abs/1705. 08926
arXiv: 100i::100.000 705.08926 [cs].
Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, Iain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian Action Decoder for Deep Multi-Agent Re- inforcement Learning. In Proceedings of the 36th International Conference on Machine Learn ing. pp. 1199-99991111 PMLR. May 2019. URL httt:tt:::// httttp:://pxoceeedngg.. mir. press/v97/ Foerster1 9a. html
ISSN: 2640-3498.

---

Published as a conference paper at ICLR 2025
Rihab Gorsane, Omayma Mahjoub, Ruan de Kock, Roland Dubb, Siddarth Singh, and Amu Preto- rius. Towards a Standardised Performance Evaluation Protocol for Cooperative MARL, Septem- ber 2022. URL httt::tptaaatee httptt///aaxxi...ooo org.:/ii..rrr.ooooooo.o /2009/2229002 1100099..88888 10485 arXiv:2209.0048 [cs].
Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: survey. Ar- tificial Intelligence Review, 55(2):895-943, February 2022. ISSN 0269-2821, 1573-7462.
doi: 10.1007/11442--02--999966 s 610462-021-09998
URL https: /wt.iki....///mm//// springerrngeere cwm/10.1000
Jayesh K. Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative Multi-agent Con- trol Using Deep Reinforcement Learning. In Gita Sukthankar and Juan A. Rodriguez-
Aguilar (eds.), Autonomous Agents and Multiagent Systems, volume 10642, pp. 66-83. Springer International Publishing, Cham, 2017. ISBN 978-3-319-716811 URL http:://ttt/kk...... spri com.ccoo////ommmmm..ooo. http/10.11077. Series Title: Lecture Notes in Computer Science.
978-3-319-71682.
4. doi: 000007997788311--16888777 978 3- 710---1-18-----
Ritwik Gupta, Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The Shift from Models to Compound AI Systems, 2024. URL http: /bwir: berkeley eddu/ blog/2024 0/24//2//2 02/02/2/////// http////////////////oo/oooo///o/ compoundddaddd ai sysssoouudd---sstteeeeee Publication Title: The Berkeley Artificial
Intelligence Research Blog.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. URL http: /ttt:://axiiv. wwwpxi..oggaab//11...c arXiv:1512.03385 [es].
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023, URL http://githhb. wwwp:ococc////ccccoooo
Joey Hejna and Dorsa Sadigh. Few-Shot Preference Learning for Human-in-the-Loop RL, Decem- ber 2022. httttttt///axx..oto
03363 arXiv:2212.03363 [cs].
Hengyuan Hu, Adam Lerer, Brandon Cui, David Wu, Luis Pineda, Noam Brown, and Jakob Fo- erster. Off-Belief Learning, August 2021a. URL http:/ttp///ttit/ orgg/b/////...333. 04000 arXiv:2103.04000 [es].
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-Play" for Zero-Shot Co- ordination, May 2021b. URL http::://aax....rrg////.//// http://..rrgaas///00333.3 02979 arXiv:2003.02979 [cs].
Forrest N. landola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size, November 2016. URL httLL/tpp//aaa....or//////////// [es].
arXiv:1602.073600
Dayal Singh Kalra and Maissam Barkeshli. Why Warmup the Learning Rate? Underlying Mech- anisms and Improvements, June 2024. URL http:://aaxi...rrr///////200.0.0 arXiv:2406.09405 [cond-mat, stat).
Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter, July 2024. URL http://aaxxx..rr///b////0/// arXiv:2407.01502 [cs].
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017. URL http::ttpa//ii..cooo org/ abs /1112. 6980. arXiv:1412.6990 [cs].
Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, A. D. Dragan, and Ro- hin Shah. Evaluating the Robustness of Collaborative Agents, January 2021. URL http:
//arxiv.co //arxiv..rrg/as//200000 05507 Nit::1011005500 [cs].
12

---

Published as a conference paper at ICLR 2025
Isabel Leal, Krzysztof Choromanski, Deepali Jain, Avinava Dubey, Jake Varley, Michael Ryoo, Yao Lu, Frederick Liu, Vikas Sindhwani, Quan Vuong, Tamas Sarlos, Ken Oslund, Karol Hausman, and Kanishka Rao. SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust At- tention, December 2023. URL [es].
wttp:/wwwww..g/////////33/ 01 990 arXiv:2312.01990
Dapeng Li, Hang Dong, Lu Wang, Bo Qiao, Si Qin, Qingwei Lin, Dongmei Zhang, Qi Zhang, Zhi- wei Xu, Bin Zhang, and Guoliang Fan. Verco: Learning Coordinated Verbal Communication for Multi-agent Reinforcement Learning, April 2024. URL
http://aaxxx...or///////20000
17780 arXiv:2404. 17780 [cs].
Jijia Liu, Chao Yu, Jiaxuan Gao, Yuging Xie, Qingmin Liao, Yi Wu, and Yu Wang. LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination, January 2024.
//arxiv.orq abs 2312.11221 15224 arXiv:2312.1522 [cs].
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory Diversity for Zero- Shot Coordination. In Proceedings of the 38th International Conference on Machine Learning, pp. 7204-7213, PMLR, July 2021. URL lupu21a. html
wwt.mmlr.pprrre mlt.ppress/119/
ISSN: 2640-3498.
Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos Vlassis. Optimal and Approximate Q-value Functions for Decentralized POMDPs. Journal of Artificial Intelligence Research, 32:289-353
May 2008. ISSN 1076-9757. doi: 10. 10.1613/jair2444. URL http:://arxi..org/eee. 1111 0062
arXiv: 11Xiv:1111.0062 [es].
David Premack and Guy Woodruff. Does the chimpanzee have theory of mind?
Behavioral and Brain Sciences, 1(4):515-526, December 1978. ISSN 1469-
1825, 0140-525X. doi: 10.1017500405255000661222 /tw.
URL https: bttps::/www/ www.
art oat.a////ossttccciirraaeeeeeeeea www.rriiiinnnineessseeeeeee sciences/ 13980229985011001799922222222220
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi- Agent Reinforcement Learning, June 2018. Uttp:tt:///art org/ abwaabas/10000 /1003.1113..0 11485 arXiv:1803.I 1485 [cs, stat].
Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ing- varsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whiteson, Bruno Lacerda, Nick Hawes, Tim Rocktaschel, Chris Lu, and Jakob Nicolaus Foerster. Jax- MARL: Multi-Agent RL Environments in JAX, December 2023. abs/2311. 10090 arXiv:2311.1000 [cs].
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr. Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge, December 2019. URL htttp://arxiv:
bogg/bs// 192. 04043 arXiv:1902.04043 [cs, stat].
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Pol- icy Optimization Algorithms, August 2017. arXiv:1707.06344 [cs].
D.J. Strouse, Kevin R. McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collabbrattn with Humans without Human Data, January 2022. URL http::://aaaxi..ogr//b/////// 08176 1xXi022110008111 [cs].
Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True Knowl- edge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learning, March 2024. URL http:://aaat atg//bs//241.. [es].
arXiv:2401.1413
13

---

Published as a conference paper at ICLR 2025
Gemini Team. Gemini: A Family of Highly Capable Multimodal Models, June 2024. URL http:
Mesut Yang, Micah Carroll, and Anca Dragan. Optimal Behavior Prior: Data-Efficient Human Models for Improved Human-AI Collaboration, November 2022. StRp::///aaxii
abs 2211. 01602 2XXi:2211.01660 [cs].
Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang. Yu Wang, and Yi Wu. Learn- ing Zero-Shot Cooperation with Humans, Assuming Humans Are Biased, February 2023. URL
http://arxiv. wrw.ooggg//........ /202 01 605 arXiv:2302.01605 [es].
Rui Zhao, Jinming Song, Yufeng Yuan, Hu Haifeng, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei. Maximum Entropy Population-Based Training for Zero-Shot Human-Al Coordination, June 11701 arXiv:2112.11700 [es].
2022. URL wrwiv. wttp///g////aoooo/oooo
14

---

Published as a conference paper at ICLR 2025
Algorithm State-Augmented Self-Play Algorithm
1: Input: Number of policies P, number of iterations I, number of rollouts R
2: Initialise each policy R; for l,...,
3: for each iteration & = 1 to / do
4: Initialise an empty state buffer B
5: for each pair of policies (m,, TTy) do
6: for each rollout r = to do Collect trajectory T, for the policy pair (m,, mT) Store every tenth state from Tr in the buffer B end for end for
7:
8:
9:
10:
for each policy x; do
12:
Configure the environment to select random state from the buffer B as the initial state on each reset
13:
Perform self-play for N timesteps 14: end for
15:
end for
A ADDITIONAL MATERIAL FOR LIMITATIONS OF OVERCOOKED
A.I BUTTON GAME
In the Button Game, illustrated Figure 2 Alice observes pet, which can be either a cat or a dog, and Bob must guess the identity of the pet. We define pet_bit bit = 0 if the pet is a cat and 1 if it is dog. After observing the pet, Alice selects an action by pressing one of N available buttons. Pressing button activates one of 2NN light bulbs that Bob can observe. The light bulb at position 2a pel will light up, where a is the button Alice pressed. Bob must then guess whether Alice saw cat or dog. Note that there is need for Alice and Bob to establish specific communication convention. The parity of the light bulb encodes the pet's identity regardless of which button Alice presses. Both agents receive reward of +10 points for correct guess and -10 points for incorrect guess.
A.2 OVERCOOKED EXPERIMENTAL SETUP
We perform all experiments in the JaxMARL implementation of the OvercookedV2 environment. exppe r iments
Experiment code available at https::/t//t btttp:://itthu: com/ We configure the environment to be fully observable and match the environment dynamics of the original implementations. The observations differ to the original implementation since we have ded- icated layers for newly introduced objects such as the recipe indicator. However, these layers in the observation grid will simply be zero when objects are not used the layout. For more details on environment implementations refer to Section
We train agents using independent PPO, with parameters shared among all agents. The training runs parallelised across multiple environment instances simultancously. Our network architecture inspired by previous work Overcooked by Carroll et al. (2020). We employ a feed-forward network, parameterised by three 2D convolutional layers with 32 channels each, and kernel sizes of 5x5, 3x3, and 3x3, respectively. We pad our inputs using zeros and employ ReLU as the activation function after each convolutional layer. The output of the convolutional encoder is flattened and passed through a dense layer with 64 neurons. We apply LayerNorm (Ba et al. 2016) to this output, which is then fed into another dense layer with 64 neurons. The final output used in the actor- network to parameterise categorical distribution and by the critic to produce value estimate through a projection layer. We use the Adam optimiser (Kingma & Ba (017) 7 to update the network parameters. Additionally, we apply learning rate warmup at the beginning of training, followed by a cosine learning rate decay schedule. This helps to stabilise training and improve performance (Kalra& & Barkeshli, 2024). The same hyperparameters are used for both the standard and state- augmented settings; an overview is provided in Appendix 3 Our experiments were conducted on server equipped with 8 NVIDIA A40 GPUs with 48GB of memory and AMD EPYC 7513 32-Cort Core Processor. The models were trained using JAX (Bradbury et al. 2018) and FLAX (Heek
al. 2023
15

---

Published as a conference paper at ICLR 2025
A.3 ADDITIONAL FIGURES
This appendix provides supplementary figures that offer deeper insights into the experiments and support results discussed in the main text. We first provide additional figures for the experiments conducted for analysing the limitations of Overcooked and then the experiments conducted in Over- cookedV2.
Standard
State- Peate-Augmente
250
254.48
25765
25136
230.44
201.00
253
2:3
236 20
256.00
257.50
253
201
255-00
234
225-42
355.00
253.22
255
2-1.66
205.66
20014
746.0
257
255-8
256
297.24
256.0
200
15.24
258.0
296.4
257-44
235
254.1
354
337
234
314.44
052.42
346.11
256.00
238.2
25131
23/10
23632
25.166
20124
239
237
25.000
239
259.12
210.07
256.1
230-22
236.44
25044
150
Policy
5
250.2
22.40
250.00
253.00
23380
230
40
223
203.04
237.04
231.90
20:00
253.72
238
257:48
250.32
257.72
230.00
200.24
25
100
26.56
257
216
26
256
334-34
255
20044
356.7
232
223.6
35424
201.00
254-1
236
357.01
206-32
201.66
254.0
24//
2AR
24
131.6
20100
235
297
200.92
292.0
25832
20752
132
20h-11
298.9
200/11
136-22
238.44
250
2
3
6
7
3
7
0
Policy 2
Policy 2
Figure 6: Cross-play matrix for the standard and state-aummented settings in the Cramped Room layout. Ten agents were independently trained for each setting; each cell (i, j) of the matrix repre- sents the average score 500 episodes played by the ith and jth agents. Agents in both settings achieve nearly identical scores in self-play and cross-play, resulting in an SP-XP gap that is nearly zero.
Standard
State-Augmented
300
378.28
20026
239
37754
276.00
275,08
376.55
276.00
20.36
2082 20
2.024
274
207
20
275
27
27712
27188
275
276
271.5
27
27262
27..0
27344
27578
250
200
m
27:.44
27838 21688
27124
271326
213
2.5
2/5
279.27
277.2
2/3.00
27120
275-22
277.32
271A4
275.55
27632
214.28
21354
2130
Policy 1
150
5
27700
206.2
271.00
227
276
275
377.03
273.22
30.44
27778
77.16
274.88
208
277
774
27624
27784
771-
07764
277
977.88
276.4
27532
77606
100
2/./2 2004
21814
3/538
271.20
27724
279 R8
2/3
279.78
277.8
4/1
67/27
279
27h
39
274.25
2/5.27
27632
274.72
215.5
21AM
0
2
3
4
6
7
9
0
2
3
4
6
7
8
0
Policy 2
Policy 2
Figure 7: Cross-play matrix for the standard and state-augmented settings in the Asymmetric Ad- vantages layout. Ten agents were independently trained for each setting; each cell (i, j) of the matrix represents the average score across 500 episodes played by the ith and jth agents. Agents in both settings achieve nearly identical scores in self-play and cross- cross-play, resulting in an SP-XP gap that is nearly zero.
16

---

Published as a conference paper at ICLR 2025
Standard
State-Augmented
350
+338.32
00
33
146-86
203.00 04
25.03
$1.00
79.18
77.40
307.55
201.0
20133
10.00
144
234.33
25135
120.40
210.00
378.66
300
146
17.41
27.01
11
1:16
25.40
162
17388
218
177,4
178.66
191.6
118.76
208.46
216.0
250
147.34
20.56
220
124.22
10.00
142
132
164.2
228
206.3
169-66
215.46
218.66
200
Policy
350.92
3256
190.7
187.00
32.44
20.43
199.60
16.20
96.20
17.42
20.40
213.56
223.22
57259
136.96
336-02
396.0
232.40
211.40
10755
150
100
142.4
$7-11
219
218
213.1
121.4
630.66
64.72
256.04
130.88
76.12
172288
228
214
219.56
-20872
215.0
32752
227.52
238.22
322.63
118.00
226.00
50
2
3
6
7
7
0
Policy 2
Policy 2
Figure 8: Cross-play matrix for the standard and state-augmented settings in the Coordination Ring layout. Ten agents were independently trained for each setting: each cell (i. j) of the matrix repre- sents the average score across 500 episodes played by the ith and jth agents. All standard self-play agents achieve similar SP scores of around 200-220 points. While some pairings perform well in cross-play as self-play, many fail to deliver effectively, resulting in only few successful deliv- eries. In contrast, in the state-augmented setting, half of the agents achieve similar self-play scores, with some even discovering more efficient strategies, yielding scores around 300 points. Impor- tantly, all policy pairings in cross-play perform well, with no complete failures.
Standard
State State-Augmented
250
177.04
1.7
175
179.01
6.70
-20
2.41
355
3.44
1.5
037.07
307
299
138
235.24
299.00
230
-15
Policy
0.78
0.58
359
80
1.28
303.00
0.44
159
123
227.00
4.40
1.72
500
527 69
120
200.72
350.00
350.44
330
200.12
205.00
174.74
166.4
100
6
86
45.94
894.77
8.9
368.04
893.0
109
0.34
2.48
1.9
1.9
298.19
187.8
299.00
19112
399
199
235:44
199.99
399.21
139
115.10
50
1.34
0.20
27.84
0.22
37.82
0.00
26.00
96.38
:33.22
$32.64
372.00
372.2
572.55
130.40
273.85
373.30
575.83
376.22
175.72
201.40
2
3
6
7
9
0
2
3
6
7
Policy 2
Palicy 2
Figure 9: Cross-play matrix for the standard and state-augmented settings the Forced Coordina- tion layout. Ten agents were independently trained for each setting; each cell (i. j) of the matrix represents the average score across 500 episodes played by the ith and jth agents. All agents achieve similar self-play scores of around 200 points. While some pairings perform equally well in cross- play as in self-play, others fail entirely, with some not completing even single delivery on average. In contrast, agents in the state-augmented setting also achieve similar self-play scores, and all policy pairings in cross-play perform well, with no complete failures.
17

---

Published as a conference paper at ICLR 2025
Plated
Cooked
Ingredient 1
Ingredient
Ingredient 3
1 bit 1 bit
2 bits
bits
2 bits
Bit-encoding scheme for ingredients, plates and dishes. The first bit indicates whether the item is cooked (0: Not Cooked, I: Cooked), and the second bit indicates whether plated (0: Not Plated, I: Plated). Each ingredient is represented by 2 bits, encoding O to 3 units of the ingredient (00: O units, 01: unit, 10: 2 units, 11: 3 units).
Figure 10:
B OVERCOOKED V2 ENVIRONMENT DETAILS
In this section provide additional details for the environment dynamics, implementation and features of Overcooked While we reuse some of the interaction logic from the existing Over- cooked implementation in JaxMARL, the need for multiple ingredients and a configurable number of players requires new state representation and significant reworking of the interaction logic. In the following, we detail the dynamics of the game, explain the structure of observations, and show the compatibility to the original Overcooked environment. Finally, we provide more details for the provided layouts as well as a guide for the layout creation interface.
B. STATE REPRESENTATIONS
The state of the game consists of the game grid G, time t, recipe R, and agents A {ar, (01,02,.... (U2,...,ap)) J. One of the main challenges is allowing for a dynamic number ingredients for each cell of the grid and in agent inventories. Since JAX requires static shapes for arrays, developed a bit-encoding scheme to represent ingredients, as illustrated in Figure IO In this scheme, the first bits serve flags indicate whether an item has been cooked plated. The subse- quent bits represent the quantity of each ingredient in the item. We allocate two bits per ingredient, which is sufficient to encode up to three instances of the same ingredient. Given that our recipes are fixed at three ingredients, this encoding can represent all possible combinations of (partial) dishes. This encoding scheme is applied all representations of ingredients throughout the game.
The game grid is represented as a three-dimensional grid with dimensions width x height x 3. The first layer contains static objects, which include: empty, wall, goal, pot, recipe indicator, button recipe indicator, plate pile, and ingredient piles. The second layer holds the encoded ingredients at each cell. The third layer stores additional information such as pot and button timers. Agents are represented by their position, direction, and inventory.
B.2 ENVIRONMENT DYNAMICS
One step in the environment consists of three sub-steps, as illustrated Figure 11 tempt move all agents their new positions, possible. Initially, the new positions are computed without considering collisions. If an agent attempts to move to location occupied by an object, the move action is executed; the position remains unchanged, although the agent's direction is updated regardless of whether movement occurs. After computing all movements, we resolve col- lisions iteratively. During each iteration, we identify all collisions on the grid, mark the involved agents, and reset their positions to their previous locations. This procedure continues until no fur- ther collisions are detected. Iterative resolution is necessary because, with more than two agents, resolving collision might cause another collision with different agent. Finally, ensure that no pair of agents has swapped positions, as this is not a valid move; any agents that have swapped positions reset to their original locations.
First, we at-
Next, we process all agent interactions sequentially, starting with the first agent. an interaction is not allowed, the action results in no-op. The outcome of agent's interaction depends both the agent's 's inventory and the object the agent is facing. For example, an agent is facing an ingredient dispenser and its inventory is empty, the agent picks up unit of the ingredient. For
18

---

Published as a conference paper at ICLR 2025
Movements
Interactions
Environment
Move Agents
Increment Time
Process Interactions
Handle Collisions
Cook Pots
Compute Rewards
Prevent Swapping
Update Indicators
Illustration of the three sub-steps within a single environment step. First, agents are moved their new positions. Then, interactions with the environment are processed. Finally, the global environment state is updated.
Figure 11:
each correct delivery, agents receive a reward of 20 points. If negative rewards are enabled, agents receive -20 points for an incorrect delivery. Furthermore, we provide shaped rewards similar those in the original Overcooked environment. Rewards in Overcooked can be sparse since preparing and delivering dish requires many steps. placing ingredient into the pot, picking up plate while dish is cooking, and picking up dish. However, with the addition multiple recipes, these actions may not always be beneficial, for example adding ingredients the pot that do not match the current recipe R. We only give these rewards when the actions are aligned with the current recipe.
provide shaped rewards for specific actions, such as
Finally, we perform updates required each timestep that unrelated to agent interactions. This includes incrementing the global timer f, updating cooking pots, and decrementing button timers. The timers for pots and button recipe indicators, stored in the extra info layer, are decremented by one. We provide a parameter to configure whether agents must interact with a pot to start the cooking process; if this parameter not set, cooking starts automatically when the pot is full. Once pot finishes cooking, the cooked flag is added to the ingredients in the pot.
B.3 OBSERVATIONS
The observations are structured as width x height num num__mmaa www_channels tensor. If a view radius is configured, agents only observe the partial state within this radius, with any areas outside the game grid being padded with zeros. Most the layers for each cell are one-hot encoded. Each cell the grid includes the following components: one-hot encoding of static objects (e.g.. wall, pot, etc.), and an ingredient encoding for items on counters and in pots. Agents are represented by an indicator of their position, one-hot hot encoding of the direction they are facing at their respective cell, and an encoding of their inventory. Separate channels are used to encode all other agents and the agent receiving the observation. We also include separate layer for encoding the ingredients in recipe indicators. All ingredients are encoded using the same scheme: the first two channels indicate whether the item is cooked and/or plated, while the following channels represent the number of instances of each ingredient present in the cell. This encoding scheme is consistently applied across all ingredient layers in the observation.
B.4 EPISODE VISUALISATION
We provide a visualisation engine that renders game state into an image. The visualisation logic has been adapted from the JaxMARL implementttnnnnnn for the original Overcooked environment. The implementation has been extended to support the visualisation of all new objects, as well multiple ingredients and recipes. Additionally, we have rewritten the implementation to enable JIT compi- lation of the entire rendering pipeline, which significantly speeds up the rendering process. This optimisation allows for vectorised mapping over entire state sequences and even across batches of sequences. We offer user-friendly interface render entire episodes into GIFs for easy inspection.
19

---

Published as a conference paper at ICLR 2025
Figure 12: Illustration of three different layouts adapted from the original Overcooked environment to include multiple recipes. From left to right: Cramped Room V2, Asymmetric Advantages with recipe indicator the right and Two Rooms forced coordination layout.
B.5 LAYOUTS
In this subsection we provide additional details for the layouts included with Overcooked Fur- thermore, we introduce the layout creation interface in detail, which allows researchers to build layouts simply by specifying an ASCI string and a set of possible recipes.
B.5.1 ADAPTED LAYOUTS.
We provide pre-configured layouts for all five the layouts introduced by the original Overcooked benchmark: Cramped Room, Asymmetric Advantages, Coordination Ring. Forced Coordination, and Counter Circuit. By setting no agent view radius, possible to replicate the environment dynamics of the original benchmark. Additionally, multiple recipes, some of which are minor adaptations the original Overcooked environments, as illustrated in Figure T2 All these layouts feature two ingredients onions (yellow) and broccoli (green) - and allow for all possible recipe combinations, resulting in a total of four distinct recipes. We provide variant of the Cramped Room layout, which has the same room structure as the original but adds a recipe indicator and an additional ingredient. Additionally, we offer adaptations of the Asymmetric Advantages layout, which maintain the original challenge but now include multiple recipes. We provide different variants of this layout, with the recipe indicator positioned on the left, centre, right. partially observable setting, one agent must wait for the other to 'demo' the recipe. We also introduce layout featuring two rooms where agents must collaborate to cook dishes. this layout, the agent on the right has access to ingredients, plates, recipe indicator, and the delivery station, while the agent on the left is the only one with pot to cook. This setup forces agents to work together, similar to the Forced Coordination layout.
offer several simple layouts that include
B.5.2 LAYOUT CREATION INTERFACE
In addition, we offer an easy-to-use interface for researchers create their own custom layouts presented in Figure [3 To design custom layout, one simply provide an ASCII string that describes the environment configuration, along with an optional list possible recipes. a list of recipes is not explicitly provided, list of all possible combinations of ingredients available in the layout used by default. The agent view radius can be configured independently of the layout; however, layouts may be designed with particular radius in mind to create specific challenges.
20

---

Published as a conference paper at ICLR 2025
Define the layout string overcookedv2_demo
Char
Description
R
W
Wall (counter)
НВWXW
A
Agent
Define possible recipes recipes
X
Delivery station
P
B
Plate (bowl) pile
10
10,0,11, 10,1,11
Pot location
R
Recipe indicator
12
L
Button recipe indicator
13
Create the layout
0-9
Ingredient pile
14
layout Layout. from_string
(space)
Empty cell
15
overcockedv2 demo,
bessible_recipss-ne
Example code block using the string to layout interface in the Overcooked environ- ment. This example defines the layout shown in Figure I The table on the right explains the symbols used in the layout grid.
Figure 13:
21

---

Published as a conference paper at ICLR 2025
C ADITIONAL MATERIAL FOR OVERCOOKED V2 EXPERIMENTS
C.I EXPERIMENTAL SETUP
In this section we outline the setting in which we conducted our experiments in OvercookedV2. We conducted all experiments in partially observable setting with agent view radius of two cells. Additionally, the starting positions of all agents were randomised at the beginning of each episode. The environment was configured to give negative rewards if the agents delivered the wrong dish. This setup prevents strategies in which the agent with access to all the necessary objects ingredients, plates, pot, and delivery station - ignores coordination attempts from the other agent and alternately delivers the two possible recipes. With negative rewards, this strategy would yield an expected total return of zero.
C.1.1 TRAINING
Initially, we attempted to train policies in the complex layouts of our new enviromment using archi- tectures similar to those employed by Carroll et al. (2020). However, we found that these policies did not learn good policies in the new scenarios. Even in fully observable settings, agents struggled led to perform optimally when conditioned on the recipe indicator. This led us to explore different archi- tectures, drawing inspiration from ResNet (He et al. 2015) and SqueezeNet (Iandola et al.) 2016). Ultimately, we discovered that incorporating three layers of 1xl convolutions with 128, 128, and channels respectively at the beginning of the network significantly improved performance. These 1x1 convolutions are highly efficient and effective since the observations in our environment are sparse. Adding these layers allows us to transform each cell of the observation grid and reduce its dimensionality before applying the more computationally expensive 3x3 convolutions. Following the 1xl convolutions, we use three 2D convolutional layers with 16, 32, and 32 channels, each with a 3x3 kernel size, and apply zero-padding to the inputs. We use ReLU as the activation function after each convolutional layer. The output from the convolutional encoder is flattened and passed through a dense layer with 128 neurons, followed by LayerNorm (Ba et al. 2016).
Moreover, given that we operate in partially observable environment, we added memory compo- nent to our network architecture. Without this, agents would be unable to condition on their AOH, which is necessary, for example, memorise the current recipe displayed block. To achieve this, we added a GRU (Chung et al. 2014) with a hidden size of 128, with the hidden state initialised to zeroes and reset at the termination of each episode. The output from the GRU is then passed through another dense layer with 128 neurons, which was used by the actor- network to parameterise categorical distribution and by the critic to produce value estimate via projection layer. Experiments with more complex architectures did not yield substantial perfor- mance improvements and considerably increased training times. We trained our agents using PPO in the same setting outlined in Appendix A.2
the recipe indicator
C.1.2 OTHER-PLAY
We implement other-play in Overcooked top of our PPO implementation. Other-play relies on the knowledge of a class of symmetries within the environment. In Overcooked, these symmetries can include action symmetries or ingredient permutations. For instance, in layout with horizontal symmetry, the actions and down could be considered symmetries. Similarly, ingredient permu- tations, or possibly permutations of subsets of the available ingredients. can form a symmetry class as defined by other-play. A limitation of other-play is that the set of symmetries must be known beforehand. This is particularly relevant in Overcooked, where we cannot define a universal set of symmetries for the entire environment; instead, symmetries depend on the specific layout.
In the layouts where we apply other-play in this section, we have two possible recipes, each consist- ing of three instances of the same ingredient type. We permute these two ingredients while keeping all other ingredients fixed. At the start of each episode, an ingredient permutation & is randomly drawn from the set of symmetries & for each agent. This permutation is then applied to each agent's 's observation, permuting all observation encodings of ingredients, dishes, and ingredient dispensers. We implemented these (partial) ingredient permutations part of the environment code and store the permutation mappings for each agent as part of the game state. When evaluating ZSC perfor- mance, all agents receive the non-permuted observations.
22

---

Published as a conference paper at ICLR 2025
250
Mlen Espondda Etaluts
100
200-
150
200
Exhande
100
50
1000
8.0
070
8.0
Envronment Stap
Envronment Stap
167
Enviroment Stap
(a) Cramped Room
(b) Asymmetric Advantages
(c) Coordination Ring
Moan Ep Retur
Moan Epioods Rotum
1.50
150
1200
100
53
10
0.6.
8.0
0.4
1.5
Enviromment
Environment
(d) Forced Coordination
(e) Counter Circuit
Figure 14: Training results on partially observable versions the original Overcooked layouts with random starting positions in OvercookedV2. Plots show the mean and standard deviation across 10 seeds.
C.1.3 FICTITIOUS CO-PLAY
Fictitious co-play is an algorithm that allows training robust agents capable of adapting to different skill levels. For each FCP agent we train a population of eight self-play agents and collect three checkpoints during training, with the final checkpoint representing the agents' final state. These checkpoints were then used to train a common best response. The parameters of the collected checkpoints are frozen, and only the FCP agent is trained. The batch of checkpoint partners used for training in each iteration was chosen at random. Additionally, the agent controlled by the FCP policy in the environment was selected randomly in each iteration. Therefore, the FCP policy needs to be robust enough to handle all agents' perspectives.
C.2 ADAPTED LAYOUTS
Original Layouts. The evaluation of our model in partially observable version of the layouts from the original Óvercooked benchmark, illustrated in Figure 2T shows strong performance. The hyper- parameters for all layouts available in Appendix D.2. Our agents trained in self-play achieve similar or even superior results compared to policies trained the fully observable versions of the original environment (Carooll al.) 2020). In the Asymmetric Advantages and Coordination Rings layouts, all trained policies make use of both pots present in the layout to cook multiple dishes si- multaneously; behavior that previous policies struggled with. Even in the Counter Cireuit layout, agents successfully deliver dishes, despite the layout heing significantly larger than others. The in- creased size presents greater challenge for agents operating in a partially observable environment. Furthermore, random position initialization can place agents far from the ingredients at the start, requiring them be robust in navigating such situations. These layouts effectively demonstrate the capabilities of our policies in a partially observable setting; however, they do not yet present truly challenging coordination scenarios. Most of the layouts are relatively small, and except for the Asymmetric Advantages layout, agents are located in the same room, which diminishes the im- pact of partial observability. To create more interesting and challenging coordination scenarios, essential to introduce asymmetric information.
Adapted Ingre--ng Layouts. We extend two of these layouts by introducing an additional ingredient and a recipe indicator. This modification requires our policies to take into account the recipe of the current episode to effectively deliver dishes. Our results, illustrated in Figure [5] show that in both adapted layouts, agents successfully condition the recipe indicator. However, certain recipe combinations yield slightly lower rewards than others, which may be due to slight differences in the distance between the respective ingredients and pots. In the Cramped Room V2
23

---

Published as a conference paper at ICLR 2025
Mean Episode Return
200
200
150
150
Recipe
Onions
100
100
Onions, Broccoli
Onion, Broccolis
3 Broccolis
8.0
0.2
0.6 Environment Step
0.4
1.0
8.0
0.2
0.4
0.6
0.8
Environment Step
1e7
(a) Cramped Room V2
Mean Episode Return
400
400
300
300
Recipe
200
200
3 Onions
Onions, Broccoli
100
100
Onion, Broccolis
Broccolis
8.0
0.5
1.0
1.5
2.0 1e7
8.0
0.5
1.0
1.5
2.0
Environment Step
Environment Step
1e7
(b) Asymmetric Advantages V2 with recipe indicator the right side
Figure 15: Training results on adapted Overcooked layouts in OvercookedV2. Both layouts feature two distinet ingredients (onions and broccoli) and total of four possible recipes. The plots on the left display the mean total reward during training, while the plots on the right filter results to show only episodes where respective recipe was selected. Each plot shows the mean and standard deviation across 10 seeds.
layout, both agents are located in the same room and can observe the recipe indicator is within their view radius. In the Asymmetric Advantages V2 layout, only the agent on the right can observe the indicator, requiring the other agent to wait for the recipe to be 'demoed" before delivering the correct dishes. These scenarios are more complex than the original benchmark layouts, but the coordination challenges remain limited, and there are straightforward solutions, such as 'demoing' the recipe, that work well ZSC setting. In the following sections, we will explore classes of layouts that present more complex coordination challenges with non-obvious solutions.
24

---

Published as a conference paper at ICLR 2025
D HYPERPARAMETERS AND TRAINING CURVES
This appendix provides the hyperparameters used in our experiments.
D.1 LIMITATIONS OF OVERCOOKED
D.1.1 HYPERPARAMETERS
Parameter
Value
ACTIVATION
relu
CLIP_ER
0.2
MIN_FEATURES
32
ENT_CCEF
0.04
DIM SIZE
64
GAE LAMBDA
0.95
GAMMA
0.99
LR
0.0004
LR WARMUP
0.05
MAX GRAD_NORK
0.5
NUM ENVS
NUM MINIBATCHES
16
STM_STEES
256
REW_SHAP ING HORIZON
ING_ORIZONN
5000000
False
TOTAI_ TIMESTES
10000000
UPDATE_ER EPOCHS VO_C
0.5
Table 3: Hyperparameters for the layouts: Cramped Room, Asymmetric advantages, Coordination Ring, Forced Coordination and Counter Circuit.
25

---

Published as a conference paper at ICLR 2025
D.1.2 TRAINING RESULTS
In this subsection we present training curves for both the standard and state-augmented settings across all layouts.
Standard
State-Augmented
Mean Episode Return
200
200
100
100
8.0
0.2
0.4
0.6
0.8
1.0
8.0
0.2
0.4 Environment Step
0.8
1.0
Environment Step
1e7
1e7
Figure 16: Training results for the limitations experiment in the Cramped Room layout. The plot on the left displays the mean total reward during training in the standard setting, while the plot on the right shows the state augmented setting. Each plot shows the mean and standard deviation across 10 seeds.
Standard
State-Augmented
Mean Episode Return
200
200
100
100
8.0
0.2
0.4
0.6
0.8
1.0
8.0
0.2
0.4
0.8
1.0
Environment Step
1e7
Environment Step
1e7
Figure 17: Training results for the limitations experiment in the Asymmetric Advanteges layout. The plot the left displays the mean total reward during training in the standard setting, while the plot on the right shows the state augmented setting. Each plot shows the mean and standard deviation across 10 seeds.
Standard
State-Augmented
Mean Episode Return
300
200
100
100
8.0
0.2
0.4
0.6
1.0
8.0
0.2
0.4
0.8
1.0
Environment Step
1e7
Environment Step
1e7
Figure 18: Training results for the limitations experiment in the Coordination Ring layout. The plot on the left displays the mean total reward during training in the standard setting, while the plot the right shows state augmented setting. Each plot shows the mean and standard deviation across 10 seeds.
26

---

Published as a conference paper at ICLR 2025
Standard
State-Augmented
Mean Episode Retur
200
200
150
150
100
50
8.0
0.2
0.4
0.6
1.0
8.0
0.2
0.4
0.6
0.8
1.0
Environment Step
Environment Step
1e7
Figure 19: Training results for the limitations experiment in the Forced Coordination layout. The plot on the left displays the mean total reward during training in the standard setting, while the plot on the right shows the state augmented setting. Each plot shows the mean and standard deviation across 10 seeds.
Standard
State-Augmented
Mean Episode Return
200
200
150
150
100
100
8.0
0.2
0.4
0.6
1.0
8.0
0.2
0.4
0.6
0.8
1.0
Environment Step
Environment Step
Figure 20: Training results for the limitations experiment in the Counter Circuit layout. The plot the left displays the mean total reward during training in the standard setting, while the plot on the right shows the state augmented setting. Each plot shows the mean and standard deviation across 10 seeds.
27

---

Published as a conference paper at ICLR 2025
D.2 EXPERIMENTS IN OVERCOOKEDV2
D.2.1 ADAPTED LAYOUTS
Parameter
Value
ACTIVATION
relu
ANNEAL_
True
CLIP EPS
_LIP_EPS
0.2
CNN LIN_FEATURES
32
ENT ENT_CCEF
0.01
FC FC_DIM SIZE
128
CAE LAMBDA
0.95
GAMMA
0.99
GRU_HIDDEN_DIM
128
0,0005
LR WARMUP
0.05
MAX GRAD NORM
0.25
MINIBATCH_SIZA
2048
256
NUM MINIBATCHES
64
STE_STEPS
256
REW SHAP ING HORIZON
5000000
SCALE_ BBFE__CLP___EE
False
TOTAL TOTAI__TIMSSTI TIMESTEPS
10000000
UPDATE
IPDATE_EPOCHS EPOCHS
VF_COEF
0.5
Table 4: Hyperparameters for layouts: Cramped Room, Asymmetric advantages, Coordination Ring, Forced Coordination and Counter Circuit.
Parameter
Cramped Room V2
Asymm. Advantages V2
ACTIVATION
relu True
relu
ANNEAL
True
CLIP EPS
0.2
0.2
CNN FEATURES
32
32
ENT ENT_CORF
0.01
0.01
FC FC_DIM_STT SIZE
128
128
GAE LAMBDA
0.95
0.95
GAMMA
0.99
0.99
GRU_HIDDEN_DIM
128
128
LR
0.0005
0.0005
LR. WARMUP
0.001
0.001
MAX_ GRAAGAAAARRAR
0.25
0.25
NUM ENVS
256
256
NUM MINIBATCHES
64
64
NUM_STEPS STEPS
256
256
REW_SHAP
REW SHAP ING HORIZON
5000000
10000000
SCALE_CLIP CLIP_EPS
False
False
TOTAL_ TIMESTEPS
TOTAL_
10000000
20000000
UPDATE_ VF VF_COEF
4
4
0.5
0.5
Table 5: Hyperparameters for the layouts Cramped Room V2 and Asymmetric Advantages V2.
28

---

Published as a conference paper at ICLR 2025
D.2.2 GROUNDED COORDINATION
Environment Parameter
Value
"gent_view_ste view_size layout negative_rewards random_agent_positiion
agent_
2
grounded coord [impll//ring)
True
True
True
ENV_NAM
overcooked_ v2
Table 6: Environment configuration for the Grounded Coordination layouts.
Parameter
Self-Play State-Augmented
Other-Play Fictitious Co-Play
ACTIVATION
relu
relu
relu
relu
ANNEAL_LR
True
True
True
True
CLIP EPS
0.2
0.2
0.2
0.2
CNN FEATURES
32
32
32
ENT ENT_COER COEF
0.01
0.01
0.02
0.04
FC_DIM_SIDD
128
128
128
128
CAE LAMBDA
0.95
0.95
0.95
0.9
GAMMA
0.99
0.99
0.99
0.99
BRU_HIDDEN_DIE
128
128
128
128
LR
0.00025
0.00025
0.00025
0.0007
WARMUP
0.05
0.05
0.05
0.05
MAX GRAD_NORR
0.25
0.25
0.25
0.25
NUM ENVS
256
256
64
256
NUM MINIBATCHES
64
64
64
64
NUM_STEPS
256
256
256
256
REW_S WWP_SHAPIIN ING _ORIIZORIOO
15000000
15000000
25000000
15000000
SCALE _CALE_CLIP EPS
False
False
False
False
TOTAL TIMESTEPS
30000000
30000000
50000000
30000000
UPDATE_EPOCHS VF VF_COEF
4
0.5
0.5
0.5
0.5
Table 7: Hyperparameters for Self-Play, State-Augmented Other-Play and Fictitious Co-Play in the Grounded Coordination layouts layouts.
29

---

Published as a conference paper at ICLR 2025
D.2.3 TEST-TIME PROTOCOL FORMATION
Enviromment Parameter
Value
agent _entt__veew___ _iew_sizeeeeeeeee_eeeee
2
Indicate_guccesffll delivery layout
True
test time fiine_(simplerrigg
negat.ve_rrewads
True
random_agent_posittiee
True
semplee_reeipe__oo_deeleeeee
True overcooked_ v2
Table 8: Environment configuration for the Test Time Protocol Formation layouts.
Parameter
Self-Play State-Augmented
Other-Play Fictitious Co-Play
ACTIVATION
relu
relu
relu
relu
ANNEAL LR
True
True
True
True
CLIP _PS
0.2
0.2
0.2
0.2
CNN_ FEATURES
32
32
32
32
ENT
0.01
0.01
0.02
0.04
FC_DIM SIZE
128
128
128
128
GAE LAMBDA GAE_LAMBDA
0.95
0.95
0.95
0.9
GAMMA
0.99
0.99
0.99
0.99
GRU HIDDEN DIM
128
128
128
128
0.00025
0.00025
0.00025
0.0007
LR WARMUP
0.05
0.05
0.05
0.05
MAX GRAD_NORM NORM
0.25
0.25
0.25
0.25
NUM ENVS
256
256
64
256
NUM MINBAACHHES
64
64
64
NUM SSEESSSSS
256
256
256
256
REW SOM_SHAPING SHAPINGGHORIZON
HORIZON
15000000
15000000
25000000
15000000
SCALE CLIP EPS
False
False
False
False
TIMESTES
30000000
30000000
50000000
30000000
UPDATE_ UPDATE_EPOCHS EPOCHS
4
4
4
VF CECE
0.5
0.5
0.5
0.5
Table Hyperparameters for Self- Pelf-Play, AugtenAugented, Other-Play and Fictitious Co-Play in the Test Time Protocol layouts.
30

---

Published as a conference paper at ICLR 2025
D.2.4 DEMO CoOk
Environment Parameter
Value
sist__vee__siie layout
2
test time (simple/wide)
negative_rewards
True
random_agent__poitionn
True
jmmplee_recipe__o__deiieeee
True
ANV_
overcooked_ v2
Table 10: Environment configuration for the Demo Cook layouts.
Parameter
Self-Play
State-Augmented
Other-Play Fictitious Co-Play
ACTIVATION
relu
relu
relu
relu
ANNEAL_LR
True
True
True
True
CLIP EPS
0.2
0.2
0.2
0.2
CNN FEATURES
32
32
32
ENT ENT_COER COEF
0.01
0.01
0.02
0.04
FC_DIM_SIDD
128
128
128
128
CAE LAMBDA
0.95
0.95
0.95
0.9
GAMMA
0.99
0.99
0.99
0.99
BRU_HIDDEN_DIE
128
128
128
128
LR
0.00025
0.00025
0.00025
0.0007
WARMUP
0.05
0.05
0.05
0.05
MAX GRAD_RORR
0.25
0.25
0.25
0.25
NUM ENVS
256
256
64
256
NUM MINIBATCHES
64
64
64
64
NUM_STEPS
256
256
256
256
REW_S WWW_SHAPIIN ING _ORIIZORIOON
15000000
15000000
25000000
15000000
SCALE SCALE_CLIP EPS
False
False
False
False
TOTAL TIMESTEPS
30000000
30000000
50000000
30000000
UPDATE_EPOCHS VF VF_COEF
4
0.5
0.5
0.5
0.5
Table II: Hyperparameters for Self-Play, State-Augmented, Other-Play and Fictitious Co-Play in the Demo Cook layouts.
31

---

Published as a conference paper at ICLR 2025
D.3
FCP TRAINING CURVES
100-
Moan Spicode Raturn
1000
60
80
Epissoe
60
60
68
70
2.5
1.3
2.0
2.5
3.0
0.
0.5
2-3
2.3
2.0
255
3.0
03
2.3
2.3
2.0
Step
(a) Grounded Coordination Simple
(b) Grounded Coordination Ring
(c) Test Time Simple
60
500
Eptodd
1.0
Neah op
Ephodd
50
50
83
8:
80
Envitoiment Stop
Environment Stop
167
Environment Stop
(d) Test Time Wide
(c) Demo Cook Simple
(f) Demo Cook Wide
Figure 21: Training results for the FCP experiment in Overcooked V2 layouts. Each plot shows the
mean and standard deviation across 10 seeds.
32

---

Published as a conference paper at ICLR 2025
E
FUTURE WORK
Future work should focus on evaluating additional ZSC methods OvercookedV2. Our experi- ments demonstrated that state-of-the-art ZSC methods struggle with the complex coordination chal- lenges present in OvercookedV2. While algorithms such as off-belief learning (Hu et al.) 202 aa might offer improvements some scenarios, fundamental challenges remain, particularly for test- time adaptation. Humans can adapt their strategies on the fly and coordinate with unknown partners from very few interactions, whereas RL notoriously sample- sample-inefficient (Hejna & Sadigh 2022). Humans use their theory of mind to quickly reason about their partners and establish common con- ventions. Developing agents with this capability is required for effective human-Al coordination and should be key focus for future ZSC research.
We plan to expand OvercookedV2 with new, challenging scenarios that test even more aspects of coordination. Specifically, we intend to design more layouts that pose complex coordination chal- lenges. By introducing harder ZSC scenarios, we aim to reveal further gaps in existing approaches and drive additional research in this domain. Moreover, the grid-world nature of OvercookedV2 and its simple environment creation interface provide an ideal platform for exploring UED (Dennis et al. 2021). UED would enable the automatic generation of increasingly challenging coordination scenarios, creating useful distribution of training scenarios. This also opens new avenues for investigating the generalisation capabilities of ZSC methods, particularly cooperating with novel partners in previously unseen layouts.
Lastly, while our primary focus in this work has been ZSC, our findings also apply ad-hoc set- tings, where the original Overcooked benchmark has been commonly used. The generalisation challenge in ad-hoc settings is even greater due to the increased variability in skill levels and strate- gies compared to Al agents trained with the same method. For this reason, Overcooked was able to provide challenging benchmark for ad-hoc coordination. However, the additional complexity that allows to create scenarios requiring true coordination in OvercookedV2 will also be valuable for benchmarking the next generation of ad-hoc coordination methods. Furthermore, the partial ob- servability in OvercookedV2 allows methods such ADVERSITY to train diverse population of "reasonable" agents which could be used for evaluation (Cui et al., 2023). We already provide an interactive interface for humans to interact with on benchmarking human-Al coordination in OvercookedV2.
environment, and future work should focus
33


Reviews:
[{"review_id": "rOKP4MZJYX", "reviewer": "Reviewer_XSHM", "summary": "A key challenge in zero-shot coordination (ZSC) is how to communicate with an agent that has never been seen before. Typical algorithms such as self-play settle on arbitrary communication protocols, which no longer work when playing with a never-before-seen agent that doesn’t have the same communication protocol. Instead, agents would ideally learn grounded communication protocols.\n\nSo far, ZSC algorithms have been tested primarily in Hanabi and Overcooked. However, in Overcooked, there is no need for grounded communication – indeed, many failures in Overcooked can be eliminated by simply increasing the diversity of states encountered during training – so it does not serve as a good test of the grounded communication aspect of ZSC.\n\nTo address this, the authors extend Overcooked to add features that induce partial observability, stochasticity, and asymmetric information across agents, and use these features to design six new handcrafted coordination challenges, which require grounded communication to perform well in the ZSC setting. Experiments demonstrate that the environment is challenging for existing algorithms, making it a good benchmark to hill climb on.", "questions": "## Questions on the general message\n\n* Is it accurate to say that the focus of the paper is on grounded communication, rather than coordination more generally? (If not, why not?)\n* Can you say more about the intended solutions for each of the handcrafted challenges?\n* Do you think humans would solve these environments as intended, if they were given only the rules of the game and some time to familiarize themselves with the environment? (It would be interesting to do a human study to check.)\n\n## Questions about the handcrafted layouts\n\nHere are my current guesses about how the layouts work – are any of these incorrect?\n* The white circle with red background is a recipe indicator block, the red circle is a button recipe indicator, and the green square is a delivery location. (Please add a legend to Figure 5 for future readers.)\n* Agents are only provided a signal for successful delivery in the Test-Time Protocol Formation setups.\n* The yellow (onion) and green (broccoli) ingredients are to be actually used in recipes, while the blue ingredient is available to enable arbitrary communication strategies.\n\nWhat is the view radius for each of the environments? This matters to interpret the environments – for example, a view radius of 1 in Demo Cook would imply that the agents must communicate via counter placements or pot interactions, whereas a view radius of 2 or more would allow the agents to communicate via their movements as well.\n\n(Incidentally, I would recommend the authors also consider communication through movement as a form of grounded communication – this has been studied under the term “legibility”, see e.g. [Dragan et al, 2013](https://www.ri.cmu.edu/pub_files/2013/3/legiilitypredictabilityIEEE.pdf).)\n\n(Ideally the paper would also specify other hyperparameters such as the sizes of various positive and negative rewards, but they are not as crucial.)\n\n## Other notes\n\n* Please add some more clarification about the Button game somewhere (appendix is fine). I had to read the Colab to understand how it worked. (In particular, it is quite unintuitive that when Alice presses a button, it does not turn on a fixed light bulb, but rather which light bulb it turns on depends on whether the pet is a cat or a dog. Consider reskinning the environment to be more intuitive, e.g. maybe when Alice presses a button, it removes one of N screens on Bob’s side of the room, and behind the screen is a picture of the pet.)\n\n## Overall take\n\nI’ve recommended a Reject (3), but I want to note that I do like a lot of the work and think that it is reasonably close to a point where I’d want to accept it. Mostly what is needed is a significant rewrite, focusing more narrowly on grounded communication in particular, with less emphasis on state augmentation, and significantly more exposition of the handcrafted challenges. It will likely also require collecting some more evidence about the utility of the environment for grounded communication (most notably a qualitative analysis of failure modes, and having OBL as a baseline). This evidence may reveal issues with the handcrafted layouts, but if so I expect the authors will be able to design new layouts that fix the issues.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "* As the paper notes, grounded communication for ZSC is an interesting area of research, and there is only one benchmark for it currently (Hanabi). Evaluating solely on Hanabi runs the risk of overfitting to the environment.\n* The authors don’t note this, but I especially like that OvercookedV2 requires that grounded communication be done alongside other skills such as low-level motion planning (unlike Hanabi which is almost purely a communication challenge).\n* I _think_ at least the simple versions of the handcrafted coordination challenges make sense as tests of different types of grounded communication, though I found it hard to evaluate due to missing details in the paper.", "weaknesses": "## State robustness is a part of ZSC\n\nThe authors argue that OvercookedV1 is a poor fit for testing ZSC because a lot of the failures are due to poor state robustness. However, I would argue that state robustness *is* a key challenge in ZSC. When playing a collaborative game with a novel partner, the partner’s actions may move into novel, off-distribution states; to be good at ZSC an agent must be robust to this. Algorithms like self-play naturally fail to produce state robustness because they exploit a single collaborative strategy that only visits a narrow part of state space; this is why algorithms like [Fictitious Co-Play](https://arxiv.org/abs/2110.08176) improve results – by increasing the diversity of partners, we increase the diversity of states that the agent is trained on, making it better at ZSC.\n\nBased on Section 4, it seems that the authors are interested only in “fundamentally incompatible conventions”, giving this the name a “true coordination challenge”. The intended solution seems to be to base conventions off of actions that are meaningfully grounded in the environment dynamics, which I would call “grounded communication”, following [Hu et al](https://proceedings.mlr.press/v139/hu21c.html). This is a fine focus, but it should not be confused with ZSC – it is a particular subskill that is required to do well at ZSC.\n\nRelatedly, lines 141-152 discuss how “Partial observability and stochasticity are required for a problem to be truly multi-agent” because “there is an optimal joint policy where the only information required is the timestep and agent ID, which essentially boils down to a pre-determined sequence of actions”. While this is all technically accurate, this is irrelevant to the ZSC problem, where the whole point is that the agents do not get to coordinate ahead of time on an optimal joint policy. In fact, the [Overcooked video game](https://store.steampowered.com/app/448510/Overcooked/) that the benchmark is based on is itself (usually) fully observable, but nonetheless poses significant ZSC challenges, as anyone who has played it can attest.\n\nI would recommend that the authors rewrite the paper to focus on the grounded communication problem, rather than ZSC more generally. The contributions of the paper make much more sense interpreted this way – indeed increased state robustness should not make much of a difference to grounded communication. Elsewhere in this review I have treated the paper as though it were discussing the problem of grounded communication rather than ZSC more broadly.\n\n## Missing details\n\nOne of the key contributions of this work is the six new handcrafted coordination challenges, illustrated in Figure 5. However the paper provides fairly limited details about these layouts (including in the appendices, though perhaps I missed the details), making it very difficult to understand what the challenge in these environments is. I think I’ve inferred it based on the prose, but the paper should specify it directly (see also the subheading “Questions about the handcrafted layouts” under “Questions”). I would recommend that the authors put much more detail into Section 6.3, particularly discussing the intended solution to each of the layouts.\n\n## Why is OvercookedV2 difficult?\n\nThe authors argue that OvercookedV2 is a good benchmark to test grounded communication. In support of this, they show that state augmentation alone is insufficient to close the gap between self-play and cross-play in OvercookedV2.\n\nFirstly, this isn’t quite as obvious as the paper suggests. According to Table 1, for original Overcooked on Cramped Room and Asymmetric Advantages, there is approximately no gap to begin with, on Coordination Ring and Counter Circuit, the gap is approximately halved, and on Forced Coordination the gap is mostly eliminated. So statements like “By employing a state augmentation algorithm during training, we nearly close the SP-XP gap across all layouts” (line 67) are overclaims. In OvercookedV2, state augmentation doesn’t change the gap on three environments, but does substantially reduce the gap on the other three environments. So while state augmentation is certainly less effective for OvercookedV2, it isn’t ineffective.\n\nHowever, the more important objection is that this doesn’t convince me that OvercookedV2 is limited by the ability to do grounded communication. OvercookedV2 adds significantly more complexity to the environment, resulting in a greatly expanded effective state space to be robust to. In particular, with partial observability, optimal policies now depend on _belief states_, which is an exponentially larger space than regular state space. It is plausible that state augmentation was enough to address state robustness in the relatively limited OvercookedV1 environments, but could not address state robustness in OvercookedV2.\n\nTo address this, the authors could conduct a qualitative analysis of the failures exhibited during cross-play in OvercookedV2. Do we see failures of grounded communication, or failures of robustness? Ideally the authors would extend the existing Overcooked demo linked in footnote 2 with their environments and policies, so readers could observe these failures themselves.\n\nThe authors could also investigate how the effectiveness of the state augmentation method changes with complexity of the environment, to predict how much the OvercookedV2 environments “should” be helped by state augmentation if they didn’t have grounded communication challenges, and compare that with actual performance. However, this is probably more effort than it is worth.\n\n## Baselines\n\nI believe that the current state of the art algorithm for grounded communication is Off Belief Learning (OBL), from which the authors get their Button game. The experimental results on OvercookedV2 should include OBL as well – it seems quite plausible that OBL succeeds where State Augmentation and Other-Play did not, particularly since Overcooked is a poor fit for the “symmetry-breaking” approach employed by Other-Play. (Though if the primary challenge for OvercookedV2 is still state robustness, as would be my guess, then OBL probably wouldn’t make much of a difference.)\n\nAnother relevant algorithm is [Fictitious Co-Play](https://arxiv.org/abs/2110.08176), though unlike OBL this is not targeted at grounded communication, and so is reasonable to exclude.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fpz9Mbij5q", "reviewer": "Reviewer_Fs42", "summary": "The study of Zero-shot Human-AI coordination is important within the MARL community. Prior work has focused on the benchmark Overcooked, and shown how naive methods such as self-play fail because they form arbitrary conventions that don't generalize. However, this work shows that by merely increasing the state coverage self-play agents are exposed to, they can close the SP-XP gap. As such, the authors conclude the original 5 layouts are not sufficient coordination challenges. They then introduce a new benchmark which extends Overcooked by introducing asymmetric information and stochasticity, and show that state coverage is insufficient to coordinate on this new setting.", "questions": "The \"environment creation via ASCII text\" was listed as a contribution. How does this differ from the same functionality the original JaxMARL codebase already included beyond the additional object predicates from your new layout? \n\nSimilarly, you listed there was an interactive interface for humans to play on the new environment. JaxMARL included this for Overcooked as well, so how is your interface different? If it's easy to benchmark human-AI coordination would you be able to show preliminary results on the new benchmark?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a compelling idea and it was very nice to see how for self-play agents, state coverage on the original 5 layouts essentially solved the generalization gap. The authors did a nice job explaining how their new environment introduces novel challenges to the Overcooked literature through communication and partial observability, as well as limitations of self-play on these new layouts.", "weaknesses": "It would've been nice for all evaluations of the state-augmented agents on the default layouts to see how well they play with humans. If state coverage is really the key to removing the SP-XP gap, then we should see performance with humans be higher than conventional SP method playing with humans. Otherwise, it is not clear that state-coverage alone is getting at the ZSC capabilities methods like population-based training tries to learn. A similarly insightful analysis would be comparing how well the state-augmented agents on the default layouts play with other algorithms (SP, MEP, E3T, etc.) that are SOTA for ZSC on overcooked. If state-augmented agents can perform at SOTA levels or close to that, it would be more compelling evidence that the issue with generalization on the original problems was just state coverage. \n\nFor the new environments, it would be compelling to see baselines such as MEP or E3T, which are some of the most successful algorithms for ZSC in Overcooked (the former being population based, the latter being self-play based). MEP and other population-based methods in particular try to simulate strategy diversity (by generating diversity in trajectories of states visited), so if these methods fail on your new benchmark it would be especially compelling as a new challenge for the MARL community to try to tackle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JI6c4PBhm9", "reviewer": "Reviewer_GLBf", "summary": "This paper discusses limitations of the Overcooked benchmark, a widely adopted benchmark for zero shot coordination (ZSC), and proposes a new benchmark named OvercookedV2. The newly proposed benchmark better tests agents' coordinaiton skills and is less affected by the state coverage during agents' training (which significantly influences the test results on the Overcooked). In this way, OvercookedV2 mostly focuses on coordination between agents rather than generalization of the learned policy.", "questions": "1. In the paper, you mentioned that in the Overcooked, ZSC failure can be attributed poor state coverage rather than more sophisticated coordination challenges. In this way, the Overcooked mainly evaluates policy generalization rather than coordination. So I wonder what is your definition of 'coordination'? Because I think generalizing to new partners is almost equal to ZSC.\n\n2. Could you include more ZSC baselines in the paper. Other-play relies on pre-known symmetries in the environment so it may not work in the new environment. But there are also many ZSC methods that I believe are completely data-driven [1], [2], [3], [4].\n\n[1] Strouse, D. J., et al. \"Collaborating with humans without human data.\" Advances in Neural Information Processing Systems 34 (2021): 14502-14515.\n\n[2] Zhao, Rui, et al. \"Maximum entropy population-based training for zero-shot human-ai coordination.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 5. 2023.\n\n[3] Lou, Xingzhou, et al. \"Pecan: Leveraging policy ensemble for context-aware zero-shot human-ai coordination.\" arXiv preprint arXiv:2301.06387 (2023).\n\n[4] Yan, Xue, et al. \"An efficient end-to-end training approach for zero-shot human-AI coordination.\" Advances in Neural Information Processing Systems 36 (2024).\n\n3. I wonder whether you plan to integrate human-AI coordination into OvercookedV2, since another important advantage of the Overcooked is its support for human-AI interaction.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Zero shot coordination (ZSC) is an important topic\n\n2. The Overcooked, the most popular benchmark for ZSC, is out-dated. It is even fair to say that challenges in this environment have already been solved. The proposed OvercookedV2 poses new challenges for studies on coordination (not only ZSC but also more general studies on agent coordination).\n\n3. Thorough study on drawbacks of the Overcooked is given.", "weaknesses": "1. Insufficient evaluations of current ZSC methods on the new environment.\n\n2. Some grammatical mistakes, e.g. 'where agents independently trained agents can adopt similar strategies but with arbitrary variations in execution'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)