The Overcooked Generalisation Challenge

## Abstract

Abstract We introduce the Overcooked Generalisation Challenge (OGC) – the first benchmark to study agents’ zero-shot cooperation abilities when faced with novel partners and levels in the Overcooked-AI environment.
This perspective starkly contrasts a large body of previous work that has trained and evaluated cooperating agents only on the same level, failing to capture generalisation abilities required for real-world human-AI cooperation.
Our challenge interfaces with state-of-the-art dual curriculum design (DCD) methods to generate auto-curricula for training general agents in Overcooked.
It is the first cooperative multi-agent environment specially designed for DCD methods and, consequently, the first benchmarked with state-of-the-art methods.
It is fully GPU-accelerated, built on the DCD benchmark suite minimax , and freely available under an open-source license: https://git.hcics.simtech.uni-stuttgart.de/public-projects/OGC .
We show that current DCD algorithms struggle to produce useful policies in this novel challenge, even if combined with recent network architectures that were designed for scalability and generalisability.
The OGC pushes the boundaries of real-world human-AI cooperation by enabling the research community to study the impact of generalisation on cooperating agents.

## 1 Introduction

Developing intelligent agents capable of collaborating with humans remains a key challenge in artificial intelligence (AI) research [].
Computational agents that team up with humans [, ] to work towards common goals and achieve joint objectives promise to vastly expand human abilities [] and enable numerous applications, e.g. in
human-robot interaction [].
Recent years have seen considerable advances in understanding human cooperative behaviour [, ], computational modelling of cooperation [, , , ] and human behaviour [], as well as advances in the development of computational methods for human-AI cooperation [, , , , ].
In parallel, several benchmarks have been proposed to foster the development and evaluation of these methods [, , , ].
In particular, the Overcooked-AI environment [], used for evaluating zero-shot human-AI coordination, has gained significant traction and became a key benchmark in the field
[, , , , , , ].

Figure: Figure 1: Coordination challenges in the Overcooked-AI Coordination Ring layout.
Refer to caption: x1.png

A common thread among all these benchmarks is their focus on evaluating the cooperative abilities of agents in-distribution, meaning they are tested in the same environment in which they were trained [, , , , ].
This approach is severely limited:
First, it restricts the range of strategies that a learning agent can encounter.
Consider the Coordination Ring layout from Overcooked-AI shown in Figure 1 as an example.
While the layout requires agents to adapt to the partners’ preferred walking directions, the amount of interaction is mostly controlled by a few simple choices: going left or right and which objects to pass over the counter.
Previous work has demonstrated that an agent can effectively cooperate in such a layout by being trained with a relatively small, diverse population of partners [, , ].
Similarly to how reinforcement learning (RL) agents can remember thousands of levels(^1^11Note that the Overcooked community often refers to a concrete kitchen in the environment as a layout, while the UED community refers to instances of an under-specified environment a a level. These terms refer to a similar idea but stem from different communities. We use these terms interchangeably when discussing specific environment configurations in Overcooked.) in generalisation benchmarks [] and overfit to their training environments [], these agents are likely to overfit to their partner population, as well as the respective layouts. Second, training and testing agents on the same layout does not represent real-world human-AI collaboration.
This would require generally capable artificial agents that can successfully collaborate in different physical configurations, including those they have not been trained on.

To address these limitations, we introduce the Overcooked Generalisation Challenge (OGC) – a zero-shot cooperation benchmark that challenges agents to cooperate in novel layouts and with unknown agents.
To obtain a suitable training distribution of levels, we apply techniques introduced in research on generalisation to the cooperative multi-agent setting, including procedural content generation [], unsupervised environment design (UED) [], and dual curriculum design (DCD) [], while we provide hand-designed testing levels.
Additionally, to assess zero-shot cooperation on testing levels, we provide populations of diverse testing agents for them, similar to how previous works have evaluated standard zero-shot cooperation [, , , ].
To the best of our knowledge our work is the first to combine DCD techniques with a cooperative multi-agent RL environment and thus bridges the gap between two previously unrelated research areas;
it tests the impact of generalisation on human-AI coordination and the ability of DCD algorithms to design optimal auto-curricula for cooperating agents.
Furthermore, since historically UED benchmark environments have been often based on simple navigation tasks [, , , , ], our OGC also can be used to evaluate and compare DCD algorithms in a setting that requires additional interaction with objects and a partner agent for achieving a joint goal.
We benchmark several DCD algorithms and network architectures on our challenge and find that only PAIRED [], together with a policy that incorporates a soft Mixture-of-Experts (SoftMoE) module [], has some success at generalising to the testing levels and outperforms competitive baselines, including robust PLR [, ] and ACCEL [].
This is surprising as previous research has highlighted the performance benefits of robust PLR. Taken together, our contribution is three-fold:

- 1.
We introduce the Overcooked Generalisation Challenge – a novel benchmark challenge in which agents are asked to cooperate with novel partners in previously unseen layouts.
- 2.
We provide OvercookedUED – an open-source environment that can be used with state-of-the-art DCD algorithms and that is integrated into minimax [], taking full advantage of the hardware acceleration provided by JAX.
- 3.
We benchmark our environment by training agents with common DCD algorithms [, , ]
and show that current DCD algorithms struggle on the challenge even if we employ recent network architectures [, ]. Furthermore, we assess zero-shot cooperation performance with a population of diverse partners to link zero-shot cooperation and generalisation and show that as policies become more generally capable, they achieve better zero-shot cooperation.

## 2 Related work

### 2.1 Generalisation in reinforcement learning

It is well known that RL agents fail to generalise to new environments [, , , , ].
Consequently, several benchmarks have been proposed to evaluate the generalisation capabilities of RL agents [, , ].
To study this generalisation gap, hand-picked [] or procedurally generated [, , ] sets of training and testing levels have been used.
These works have shown that RL agents can memorise large numbers of levels during training [] even if techniques to increase generalisation are applied [].
They have also shown that regularisation can improve generalisation [, ].
While many of these works have studied individual agents, generalisation has also been studied in multi-agent settings [, , ].
A common finding is that agents must experience sufficiently diverse training data to generalise at all []. One established approach uses domain randomisation [, DR].
This approach has been used to successfully detect cars [], achieve indoor fly [], agile locomotion [] among others [, ]. It has been shown, however, that DR produces many uninformative samples through this random sampling procedure [].
This can lead to the learner’s inability to generalise to challenging configurations of these parameters [].

### 2.2 Unsupervised environment design

**Table 1: Overview over unsupervised environment design and procedurally generated environment benchmarks.✓denotes present, - absent, and ? unknown (not open-source). Closed-source projects are in gray to highlight that we can not check them ourselves. Obs. is shorthand for observations.**
| Name | Multi-agent | Zero-shot coop. | GPU accelerated | Open Source | Partial obs. | Vector obs. | Img. obs. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| XLand [] | (✓) | - | ? | - | ✓ | ? | ✓ |
| XLand 2.0 [] | (✓) | - | ? | - | ✓ | ? | ✓ |
| LaserTag [] | ✓ | - | - | - | ✓ | - | ✓ |
| MultiCarRacing [] | ✓ | - | - | - | ✓ | - | ✓ |
| CoinRun [] | - | - | - | ✓ | ✓ | - | ✓ |
| CoinRun-Platforms [] | - | - | - | ✓ | ✓ | - | ✓ |
| ProcGen [] | - | - | - | ✓ | ✓ | - | ✓ |
| 2D Mazes [, ] | - | - | - | ✓ | ✓ | - | ✓ |
| CarRacing [] | - | - | - | ✓ | ✓ | - | ✓ |
| Bipedal Walker [, , ] | - | - | - | ✓ | ✓ | ✓ | - |
| AMaze [] | - | - | ✓ | ✓ | ✓ | - | ✓ |
| XLand-MiniGrid [] | - | - | ✓ | ✓ | ✓ | ✓ | ✓ |
| Craftax [] | - | - | ✓ | ✓ | ✓ | - | ✓ |
| OvercookedUED (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

More recently, generalisation in RL has been viewed through the lens of unsupervised environment design [, , , UED] that aims to combat the disadvantages of DR by introducing generated auto-curricula [].
UED tackles the issue of creating a useful distribution of training tasks and environments by generating increasingly complex environments that facilitate continued agent learning.
It does so by adapting the free parameters of an under-specified environment to the capabilities of the learning agent.
The UED methods discussed here fall under the class of Dual Curriculum Design [, DCD] algorithms.
DCD requires three parts: 1) a level generator, 2) a curator and 3) a learning agent.
The level generator provides new training levels while the curator picks the appropriate training levels from new and past ones.
Some popular UED algorithms based on Prioritised Level Replay [, PLR], including robust PLR [, PLR^⟂], parallel PLR [, PLR^⟂,∥], MAESTRO [] and ReMiDi [], combine a random generator with a capable curator.
Other methods like (population) PAIRED [], ACCEL [], and parallel ACCEL [, ACCEL^∥] are environment generators or editors that do not curate.
Methods like Replay-Enhanced PAIRED [, REPAIRED] combine advantages from both approaches.
While the development of these DCD methods has been steady, they have mostly been explored in simple environments, see Table 1:

##### Platform games

Platform games are popular for studying generalisation in RL.
In these games, agents move from left to right in a simple 2D world while dodging enemies and obstacles to reach a certain goal.
Examples include CoinRun(-Platforms) [] and the Sonic benchmark [].
Solvable platform games are easy to generate procedurally and perform relatively well.
However, they are often limited by their action and interaction possibilities and the kind of strategies required to solve them, i.e. they often only require going from left to right and dodging objects.

##### Mazes

Both the original work on quantifying generalisation [, ] as well as more recent work on DCD algorithms [, , , , , ] make use of mazes.
Like platform games, mazes can be generated easily while ensuring they are solvable.
They have thus become a common benchmark for DCD algorithms and are part of recently released DCD benchmarks [, ].
While they are usually used to compare algorithms,
they are limited to a single agent, with limited options to interact with the environment and other agents.
This becomes apparent when considering that current algorithms can solve most mazes in only a few ten thousand gradient updates [].

##### Bipedal walker

Bipedal walker-based UED environments [] have seen their fair share of adaption [, ].
They combine similar advantages and disadvantages as the maze environments, i.e. they are easy to manipulate and make solvable, with a continuous control task and dense rewards.

##### Other

Other notable UED environments include CarRacing [], a race car environment in which an agent is tasked with following evolving sets of racing tracks, XLand [, ], a closed-source multi-task universe for generating single- and multi-agent tasks and environments, XLand-MiniGrid, an open-source variant of XLand [] based on MiniGrid [], LaserTag [, ], a competitive 2D laser tag game, and MultiCarRacing [, ], which extends CarRacing to multiple competing agents.
CarRacing and XLand-MiniGrid share properties similar to those of earlier examples of UED environments.
XLand, LaserTag, and MultiCarRacing are the only environments with multi-agent aspects, but they are not open-source.
XLand neither focuses on the multi-agent setting nor zero-shot coordination and its impact on human-AI cooperation.
LaserTag and MultiCarRacing are most closely related to our work as they have been used to study multi-agent UED in the competitive setting in [].
Opposed to [], we study the cooperative setting with different game-theoretic dynamics [].

##### GPU-accelerated implementations

Since DCD algorithms require significant runtime, vectorised and GPU-accelerated Jax-based [] implementations of these have become popular within the (multi-agent) RL [, , , , , , , , , ] and the UED/DCD [, ] communities.
These greatly improve the running speed of experiments but also require special environment implementations,
compare [].

### 2.3 Human-AI cooperation in Overcooked

Overcooked-AI [] has emerged as one of the most important benchmarks for human-AI cooperation [, , , , , , , , , , ].
It has been used for zero-shot cooperation [, , ], language model-based cooperative agents [, ], as well as human modelling in cooperation [].
The environment is fully cooperative and involves two agents in cooking and delivering a soup of onions to earn a combined reward.
Commonly, research in Overcooked focuses on training and evaluating agents on one of several layouts that test different aspects of cooperation.
While the original work researches onion-only layouts [], later works have included additional ingredients to study biases and preferences in cooperating agents [].
For this reason, works can be hard to compare, and the amount of variation in layouts has only grown, see for instance [, , , , , ].
Overcooked is only one recent example of work on cooperative multi-agent reinforcement learning (MARL) environments
including matrix penalty games [, ], multi-agent particle-world environments [, ], StarCraft [], Hanabi [], and Google Research Football [].
While each of these works has its own merit, Overcooked is unique in its focus on researching and testing cooperative zero-shot human-AI cooperation and, thus, is the only one applicable to our research interest.
Most closely related to our work is [] in which the authors use procedurally generated Overcooked layouts to evaluate human-robot interaction.
However, they have not trained a general Overcooked agent or evaluated RL agents in human-AI cooperation.
Our work is the first to explore the impact of cross-level generalisation for zero-shot cooperation and the first to offer the necessary tools to do so.

## 3 Preliminaries

The cooperative multi-agent UED setting is formalised as a decentralised under-specified partially observable Markov decision process (Dec-UPOMDP) with shared rewards.
Such a Dec-UPOMDP is defined as $\mathcal{M}=\langle\mathcal{N},A,\Omega,\Theta,\mathcal{S}^{\mathcal{M}},
\mathcal{T}^{\mathcal{M}},O^{\mathcal{M}},\mathcal{R}^{\mathcal{M}},\gamma\rangle$
in which
$\mathcal{N}$ is the set of agents with cardinality $n$, $\Omega$ is a set of observations and $\mathcal{S}^{\mathcal{M}}$ constitutes the set of true states in the environment.
Partial observations $o^{i}\in\Omega$ are obtained using the observation function $O:\mathcal{S}\times\mathcal{N}\rightarrow\Omega$ by agent $i\in\mathcal{N}$. Following [], a level $\mathcal{M}_{\theta}$ is defined as a fully-specified environment given some parameters $\theta\in\Theta$.
In it, agents each pick an action $a_{i}\in A$ simultaneously to produce a joint action $\mathbf{a}=(a_{1},\dots,a_{n})$ and observe a shared immediate reward $R(s,\mathbf{a})$.
Then,
the environment transitions to the next state according to a transition function $\mathcal{T}:\mathcal{S}\times\mathcal{A}^{1}\times...\times\mathcal{A}^{n}
\times\Theta\rightarrow\Delta(\mathcal{S})$ where $\Delta(\mathcal{S})$ refers to the space of distributions over $\mathcal{S}$.
$\gamma\in[0,1)$ specifies the discount factor.
Agents learn a policy $\pi$.
The joint policy $\boldsymbol{\pi}$ together with the discounted return $R_{t}=\sum^{\infty}_{i=0}\gamma^{i}r_{t+1}$ induce a joint action value function $Q^{\boldsymbol{\pi}}=\mathbb{E}_{s_{t+1:\infty},\mathbf{a}_{t+1:\infty}}[R_{t}
|s_{t},\mathbf{a}_{t}]$. The definition extends a Dec-POMDP [, ] with the free parameters of the environment $\Theta$, analogously to previous works [, , ].
Our definition differs from [] in terms of the shared rewards and general-sum nature.
Within our Dec-UPOMDP, we perform UED to train a policy over a distribution of fully specified environments that enable optimal learning.
This is facilitated by obtaining an environment policy $\Lambda$ [] that specifies a sequence of environment parameters $\Theta^{T}$ for the given policy that is to be trained.
How $\Lambda$ is obtained depends on the DCD method. For example, in OvercookedUED, $\Theta$ represents the possible positions of walls, pots, serving spots, agent starting locations, and onion and bowl piles which is adjusted by $\Lambda$ over the duration of training.

## 4 The Overcooked Generalisation Challenge

Figure: Figure 2: An overview of the Overcooked Generalisation Challenge and how it is typically used in a DCD algorithm. It features components for teacher-based methods like PAIRED [] via a UED environment and edit-based methods like ACCEL [] via mutator functions of existing layouts.
Refer to caption: x2.png

To evaluate the impact of generalisation on cooperative agents and, ultimately, human-AI collaboration, we introduce the Overcooked Generalisation Challenge (OGC, see Figure 2).
It extends previous work by evaluating the cooperative abilities out-of-distribution, i.e., not only with novel partner agents but also on previously unseen testing levels.
The OGC uses procedural content generation to generate training levels.
As levels are generated on the fly, training against a diverse population of experts on each layout for zero-shot capabilities is practically infeasible.
This setup allows the training and testing on diverse levels. It is, therefore, more closely aligned with real-world human-AI collaboration that is not limited to one specific physical environment.
In contrast to existing UED environments that are limited to a single agent and based on simple navigation tasks, the OGC focuses on the interaction of multiple agents in a complex, cooperative task.
Compared to single-agent environments, multi-agent environments are inherently more complex because the agents interact with each other, as well as with
the physical environment.
More specifically, in the OGC, two agents are tasked with cooking a soup together in the five original layouts of Overcooked-AI [] (see Figure 3),
but without having encountered them during training.
The original five layouts have been designed to test and explore different kinds of cooperation and thus form suitable out-of-distribution test levels.
UED expands the range of possible layouts and behaviours, further challenging layout generation and agents’ generalisability.
Specifically, an environment designer (or teacher agent) must account for agents collaborating when trying to generate layouts that are at the forefront of the abilities of the student agent.
Such teacher agents interact with the challenge by designing layouts either from scratch through interacting with OvercookedUED - a novel environment for creating Overcooked levels - or by alternating existing layouts through the Overcooked mutator.
This enables the challenge to be used by a diverse set of algorithms.

### 4.1 Components of the challenge

The OGC comprises several components that enable its integration with DCD algorithms (see Figure 2).
Most importantly, it features an Overcooked environment capable of running different levels fast and in parallel, as well as components to design
layouts and zero-shot cooperation testing populations.

##### Overcooked

Figure: Figure 3: We study the five evaluation layouts proposed in []: Cramped Room, Asymmetric Advantages, Coordination Ring, Forced Coordination, and Counter Circuit.
Refer to caption: x3.png

OGC builds on the Overcooked environment.
We adapted the version introduced in the JaxMARL project [], keeping most features consistent with the original implementation.
This includes action and observation spaces, i.e. the set of actions is given by {left, right, up, down, interact, stay} and observations are encoded as a stack of 26 boolean masks. We extend the JaxMARL implementation by adding back features from the original work [], i.e. reward shaping
and the option for receiving partial observations based on hand-crafted features.
Additionally, we parallelise different levels during rollouts by using padding and one-hot encoding of the positions of environment elements.
While this facilitates fast parallel rollouts, it requires the introduction of a maximum height $h$ and width $w$.

##### OvercookedUED

As a full UED environment, OvercookedUED generates random layouts to be used by algorithms that do not specify a designer, such as PLR.
As for [], OvercookedUED does not check whether a random layout is solvable and leaves the task of identifying suitable training layouts to the DCD method.
A random layout always features one or two onion piles, bowl piles, pots and serving locations, and two agents.
The number of walls placed is configurable, but the environment requires a border wall on the edge.
For algorithms that make use of a designer to create layouts (PAIRED, etc.), OvercookedUED provides a UED environment (see Figure 2).
This environment allows a designer policy
to take design steps to parameterise the underspecified MDP. The action space of the teacher agent consists of the total number of cells in the $h\times w$ grid.
The first design steps place walls on the grid, later steps place objects, piles, and agents.
In case of a conflict, elements are placed randomly on
free cells.

##### Overcooked mutator

Some DCD algorithms (ACCEL, etc.) rely on alternating existing layouts by mutating them.
OvercookedUED supports layout mutation through five basic operations: (1) converting a random wall to a free space and vice versa, (2) moving goals, (3) pots, (4) plate piles, and (5) onion piles.
Given a layout, our mutator randomly samples $n$ operations and applies them.

##### Implementation

**Table 2: Mean steps-per-second given varying degrees of parallelisation measured by taking 1000 steps in all parallel environments with randomly sampled actions.**
| # Parallel Envs | 1 | 32 | 256 | 1024 |
| --- | --- | --- | --- | --- |
| AMaze | $264$ | $8,141$ | $67,282$ | $264,142$ |
| OvercookedUED | $151$ | $4,921$ | $40,011$ | $156,696$ |

The OGC is implemented in Jax and integrated into minimax and can be tested with all available DCD algorithms present in the project.
We present the steps-per-seconds on our setup given varying degrees of parallelism in Table 2 and compare it to their GPU-accelerated maze environment AMaze.
OvercookedUED is a more fully featured environment and needs to take steps with two agents, resolving interactions and collisions
and consequently achieves fewer steps per second on average but is sufficiently fast.
In our work, the implementation achieved
up to $20,000$ steps per second.

### 4.2 Evaluation

Our work evaluates agents in two ways.
First, by comparing performance on out-of-distribution Overcooked layouts, we can asses the capabilities of a DCD algorithm to generate generally capable agents and, second, by using a population of hold-out FCP agents, we can asses zero-shot cooperation on the evaluation layouts.

##### Zero-shot cooperation

Overcooked was originally designed for building tools for human-AI cooperation.
Within it the use of zero-shot cooperation with a diverse population
has become a proxy for assessing the abilities of an agent to coordinate with humans.
Similarly, we provide trained fictitious co-play [, FCP] populations with 24 agents per layout.
Additionally, we include tools for evaluating agents against these populations.
FCP allows testing against diverse agents with different
preferences and skills.

##### Metrics

We use three evaluation metrics in our challenge:
1) mean episode reward and 2) mean environment solved rate in the evaluation layouts, similar to previous work [], and 3) performance with the FCP population as described above.
The environment is solved if an agent pair delivers more than one soup.
This ensures that random agents usually do not solve an environment and distinguishes agents that make no or few deliveries from ones that do so consistently across many episodes.

## 5 Benchmarking the challenge

Figure: Figure 4: Sample levels generated by the different methods after $15,000$ (Middle) and $30,000$ (End) epochs. Even after considerable training, none of the methods can guarantee the generation of solvable layouts (Middle-row leftmost and rightmost).
Refer to caption: x4.png

We benchmark the challenge with several DCD algorithms and network architectures.
We aim to set a performance baseline for future works and show what evaluations are doable with this benchmark.
To this end, we first show how difficult it is to play novel layouts in Overcooked and then how these agents might be used to evaluate zero-shot cooperation.
All baselines use MAPPO [] as the learning algorithm and are trained via Centralised Training Decentralised Execution [].
As for DCD algorithms, we compare the performance of DR
, PLR^⟂,∥, Pop. PAIRED and ACCEL^∥.
We chose these methods as they have better theoretical guarantees (PLR^⟂ vs PLR), better runtime performance (ACCEL^∥ and PLR^∥), or because we found them to perform better empirically (Pop. PAIRED vs PAIRED).
We excluded POET [] in this analysis as it outputs specialists rather than generalists, which we require [].
Additionally, we excluded MAESTRO [] as it is based in prioritised fictitious self-play [, ] that is not easily adaptable to the cooperative setting [].
As in [], if not stated otherwise, we train in $32$ parallel environments and stop after $30,000$ outer training loops, amounting to just under $400$ million steps in the environment.
Hyperparameters were picked after a grid search over reasonable values and all parameters are provided in
Appendix A.4.
Our default neural network architecture consists of a convolutional encoder with a recurrent neural network with a LSTM []. It is picked for its good performance in previous work [] (see Appendix A.5 for details).
Using these parameters, we verified that agents also overfit to their level in Overcooked by evaluating agents trained on a single layout on all layouts (cf. Appendix A.6.1).

##### Zero-shot generalisation

**Table 3: Mean episode reward for the different methods averaged over the respective testing layouts. The best result is shown in bold. We report aggregate statistics over three random seeds.**
| Method | CNN-LSTM | SoftMoE-LSTM | CNN-S5 |
| --- | --- | --- | --- |
| DR | $0.46\pm 0.16$ | $5.22\pm 7.19$ | $0.00\pm 0.00$ |
| PLR^⟂,∥ | $0.17\pm 0.06$ | $0.91\pm 0.71$ | $0.12\pm 0.15$ |
| Pop. PAIRED | $0.19\pm 0.09$ | $\mathbf{13.34\pm 5.70}$ | $0.24\pm 0.19$ |
| ACCEL^∥ | $0.20\pm 0.14$ | $0.67\pm 0.60$ | $0.28\pm 0.26$ |

In addition to our default network architecture, we explore the use of SoftMoE [], which have recently been identified for their potential for enabling scaling and generalisation, and S5 layers [] due to the strong results of structured state-space models [] in meta reinforcement learning [].
SoftMoE modules replace the penultimate layer after the feature extractor and S5 layers the LSTM in all experiments.
The results of these comparisons are shown in Table 3.
As can be seen from the table, the OGC is challenging.
Compared to commonly used single-agent Maze environments (such as AMaze, compare []), all DCD methods struggle to obtain good results.
Overall, the better performance of SoftMoE-LSTM, combined with Pop, is notable. PAIRED outperforms all other models significantly $0.01<p<0.05$ using a one-sided paired t-test.
This is also shown in the mean solved rate where it reaches $14.6\pm 7.7\%$, while all other models have solved rate of mostly $0\%$ (cf. Appendix A.6.2).
While this model performs better on average, layouts differ greatly in their difficulty.
Our best-performing model reaches modest performance in Asymmetric Advantages and Cramped Room while mostly failing in the others with no other model achieving noteworthy results.
Recall that the environment features more moving parts that must be placed correctly to facilitate learning.
This makes it hard for approaches like DR to find optimal placements by pure chance, as reflected in the results.
The full results are in the Appendix A.6.3.

##### Zero-shot cooperation

Figure: Figure 5: Zero-shot coordination results of the SoftMoE-LSTM policiy paired with an FCP population trained on the respective layout. We report the mean episode reward and standard error.
Refer to caption: x5.png

Ultimately, we want the OGC to connect generalisation and zero-shot coordination. To that end, we propose to use the included population of FCP agents (see Appendix A.6.4 for details) to establish how general cooperative agents can coordinate with a diverse set of policies.
We present preliminary results in Figure 5.
As performance on out-of-distribution levels rises, agents become more competent at zero-shot cooperation.
PAIRED always outperforms baselines (cf. Appendix A.6.5).
Although even PAIRED policies often perform only slightly better than random baselines, which signifies the challenges of our benchmark.
This is also evidenced by the kinds of levels these methods generate (Figure 4), as they tend to pivot towards generating open spaces that ease cooperation but are noteably different from evaluation layouts.

## 6 Discussion

### 6.1 DCD methods and OvercookedUED

Previous work [] has found that PLR^⟂ tends to outperform the other here-tested algorithms in navigation-based tasks.
Our more challenging environment suggests that this might not always be the case.
In our preliminary analysis, PAIRED outperformed other DCD methods.
Compared to mazes, car racing, or walker environments with fewer moving pieces, Overcooked layouts are more complex to design, requiring the designer to place multiple objects in relation to each other and the agents. This requires a capable generator
and suggests that simple navigation-based environments
used to benchmark DCD in UED algorithms do not allow full performance evaluation.
As such, OvercookedUED can be an important part of evaluating DCD algorithms.
We envision that general Overcooked agents should be evaluated in scenarios that are difficult for self-play agents using our benchmark.
These include zero-shot cooperation with strongly-biased agents []
in Coordination Ring
(see Section 1)
and Asymmetric Advantages as described in [] and for which we provide the tools.

### 6.2 Limitations

Despite its many advantages, our challenge and evaluations also have two limitations.
First, we artificially restricted the maximum size of the layouts to allow the environment to be both fully observable as in [] and parseable by CNN-based feature encoders.
Future work should focus on more natural representations of the whole scene, e.g. using graphs or item embeddings.
While we included a partial observation that could theoretically be computed independently of size, similar to the vector-based observation used for behaviour cloning agents in [], batching across layouts in OvercookedUED still requires the layouts to be scaled to the same height and width.
Second, while our challenge allows us to study zero-shot coordination via generalising across layouts, reasoning about other agents [, , , , ] might be equally important to achieve zero-shot cooperation capabilities on unknown layouts.
This is plausible given that humans can reason about the mental states of other agents via Theory of Mind [], as well as the physical configuration of the space in which they operate.
Future work could thus explore reasoning about other agents in previously unexplored environments.

## 7 Conclusion

We have presented the Overcooked Generalisation Challenge (OGC) –
a generalisation challenge focusing on (zero-shot) cooperation in MARL in out-of-distribution test levels.
Our challenge is the first unsupervised design MARL environment and is significantly more challenging than previous environments commonly used in UED and DCD research.
In addition to using the challenge in UED research, we have shown how the OGC can be used in future research on human-AI collaboration as a zero-shot cooperation benchmark for general agents.
That is, our challenge establishes a link between generalisation and zero-shot coordination.
Our work is the first to provide the research community with the tools to train and evaluate agents capable of coordinating in previously unknown physical spaces and with novel partners.

## Acknowledgements

The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting C. Ruhdorfer.
A. Penzkofer was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 2075 – 390740016.
The authors would like to especially thank L. Shi for the numerous insightful discussions.

## Appendix A Appendix / supplemental material

### A.1 Accessibility of the benchmark

We make our challenge available under the Apache License 2.0 via a code repository: [https://git.hcics.simtech.uni-stuttgart.de/public-projects/OGC](https://git.hcics.simtech.uni-stuttgart.de/public-projects/OGC).
Our environment is built on top of the existing minimax project (accessible under Apache License 2.0 via [https://github.com/facebookresearch/minimax](https://github.com/facebookresearch/minimax)) and is thus accessible to researchers that are already familiar with the project.
minimax is extensively documented, fast, and supports multi-device training.
For all details, including a full description of the advantages of minimax, we kindly refer the reader to the accompanying publication [].
Our Overcooked adaption is extended from the one in JaxMARL also accessible under Apache License 2.0 via [https://github.com/FLAIROx/JaxMARL](https://github.com/FLAIROx/JaxMARL).
Our code includes extensive documentation and examples for how it may be used.
Additionally, our code is written in a modular fashion and other multi-agent environments can be integrated with the runners thanks to the careful design of the original project.

### A.2 Broader impacts

While our work is largely foundational and concerned with providing the research community with the appropriate tools for the training and evaluation of agents in game-like environments, special caution is always imperative should this research be applied to human-AI collaboration.
Even though our goal is to improve collaboration, safeguards should be applied to make sure that humans are always safe from harm.
Especially so in real-world applications where accidents could potentially result in bodily harms.
Since our work is still far removed from any real-world application, we do not expect that our work in its present form carries the risk of materialising these harms.
Some form of unsupervised environment design in collaborative environments might be part of future systems and we therefore acknowledge these risks.
This work of course also carries potential to improve human-AI collaboration and we make an important contribution to advancing the field with potential impacts in all kinds of human-machine interaction.

### A.3 Infrastructure & tools

We ran our experiments on a server running Ubuntu 22.04, equipped with NVIDIA Tesla V100-SXM2 GPUs with 32GB of memory and Intel Xeon Platinum 8260 CPUs.
All training runs are executed on a single GPU only.
We trained our models using Jax [] and Flax [] with 1, 2 and 3 as random seed for training DCD methods and 1 to 8 as random seeds for the populations. Training the DCD methods usually finishes in under 24 hours, only SoftMoE and PAIRED based methods take longer.
SoftMoE based policies often take an extra 50% wall-clock time to train.
Noticeable is also that our S5 implementation is the fastest, usually needing 30% less time.
Both are compared to the default architectures training time.
In the longest case the combination of a SoftMoE-LSTM policy trained with PAIRED takes about 80 hours to complete training.

### A.4 Hyperparameters

**Table 4: Hyperparamters of the learning process.**
| Description | Value |
| --- | --- |
| Optimizer | Adam [] |
| Adam $\beta_{1}$ | $0.9$ |
| Adam $\beta_{2}$ | $0.999$ |
| Adam $\epsilon$ | $1\cdot 10^{-5}$ |
| Learning Rate $\eta$ | $3\cdot 10^{-4}$ |
| Learning Rate Annealing | - |
| Max Grad Norm | $0.5$ |
| Discount Rate $\gamma$ | $0.999$ |
| GAE $\lambda$ | $0.98$ |
| Entropy Coefficient | $0.01$ |
| Value Loss Coefficient | $0.5$ |
| # PPO Epochs | $8$ |
| # PPO Minibatches | $4$ |
| # PPO Steps | $400$ |
| PPO Value Loss | Clipped |
| PPO Value Loss Clip Value | $0.2$ |
| Reward Shaping | Yes (linearly decreased over training) |

**Table 5: Values used for a grid search over hyperparameters governing the learning process. Finally used values appear in bold.**
| Description | Value |
| --- | --- |
| Learning Rate $\eta$ | [$1\cdot 10^{-4}$, $\mathbf{3\cdot 10^{-4}}$, $5\cdot 10^{-4}$, $1\cdot 10^{-3}$] |
| Entropy Coefficient | [$\mathbf{0.01}$ $0.1$] |
| # PPO Steps | [$256$, $\mathbf{400}$] |
| # Hidden Layers | [$2$, $\mathbf{3}$, $4$] |
| Reward Shaping Annealing Steps | [0 0, $2500000$, $5000000$, until end] |

**Table 6: DR hyperparameters.**
| Description | Value |
| --- | --- |
| $n$ walls to place | Sampled uniformly between 0 0 – $15$ |
| $n$ onion piles to place | Sampled uniformly between $1$ – $2$ |
| $n$ plate piles to place | Sampled uniformly between $1$ – $2$ |
| $n$ pots to place | Sampled uniformly between $1$ – $2$ |
| $n$ goals to place | Sampled uniformly between $1$ – $2$ |

**Table 7: PLR specific hyperparameters.**
| Description | Value |
| --- | --- |
| UED Score | MaxMC [] |
| PLR replay probability $\rho$ | $0.5$ |
| PLR buffer size | $4,000$ |
| PLR staleness coefficient | $0.3$ |
| PLR temperature | $0.1$ |
| PLR score ranks | Yes |
| PLR minimum fill ratio | $0.5$ |
| PLR^⟂ | Yes |
| PLR^∥ | Yes |
| PLR force unique level | Yes |

**Table 8: ACCEL hyperparameters.**
| Description | Value |
| --- | --- |
| UED Score | MaxMC [] |
| PLR replay probability $\rho$ | $0.8$ |
| PLR buffer size | $4,000$ |
| PLR staleness coefficient | $0.3$ |
| PLR temperature | $0.1$ |
| PLR score ranks | Yes |
| PLR minimum fill ratio | $0.5$ |
| PLR^⟂ | Yes |
| PLR^∥ | Yes |
| PLR force unique level | Yes |
| ACCEL Mutation | Overcooked Mutator |
| ACCEL $n$ mutations | 20 |
| ACCEL subsample size | 4 |

**Table 9: PAIRED hyperparameters. All PPO hyperparameters are the same between the student and the teacher. The minimax implementation follows to original one in [] and we stick to it too.**
| Description | Value |
| --- | --- |
| $n$ students | 2 |
| UED Score | Relative regret [] |
| UED first wall sets budget | Yes |
| UED noise dim | $50$ |
| PAIRED Creator | OvercookedUED |

We overview all hyperparameters for training in Table 4 and provide details on the hyperparameter search used in Table 5.
This search was conducted on smaller single layout runs to determine reasonable values as complete runs would have been computationally infeasible.
Furthermore we show the hyperparameters for each DCD method separately: DR hyperparameters in Table 6, PLR hyperparameters in Table 7, ACCEL hyperparameters in Table 8, and PAIRED hyperparameters in Table 9.
DR hyperparameters govern how Overcooked levels are generated randomly and apply to all other processes in which a random level is sampled, for instance, in PLR, in which case the same hyperparameters apply.

### A.5 Neural network architectures

This work employs an actor-critic architecture using a separate actor and critic in which the critic is centralised for training via MAPPO [].
For the actor, the observations are of shape $h\times w\times 26$, while for the centralised critic, we concatenate the observations along the last axis to form a centralised observation, i.e. the centralised observation has shape $h\times w\times 52$ following prior work [].

Figure: Figure 6: Basic architecture featuring a convolutional encoder and an RNN.
Refer to caption: x6.png

All our networks feature a convolutional encoder $f_{c}$.
This encoder always features three 2D convolutions of $32$, $64$ and $32$ channels with kernel size $3\times 3$ each and pads the input with zeros.
Our default activation function is ReLU [, ] which we apply after every convolutional block.
We feed the output of $f_{c}$ to a feed-forward neural network $f_{e}$ with three layers with $64$ neurons, ReLU and LayerNorm [] applied each.
$f_{e}$ takes the flattened representation produced by $f_{c}$ and produces an embedding $e\in\mathbb{R}^{b\times t\times 64}$ that we feed into a recurrent neural network (either LSTM [] or S5 []) to aggregate information along the temporal axis.
We use this resulting embedding $e_{t}\in\mathbb{R}^{b\times 64}$ to produce action logits $l\in\mathbb{R}^{b\times 6}$ to parameterise a categorical distribution in the actor-network or directly produce a value $v\in\mathbb{R}^{b\times 1}$ in the critic network using a final projection layer.
This architecture is inspired by previous work on Overcooked-AI, specifically [], see Figure 6 for an overview.
We also test the use of a S5 layer [] in which case we use $2$ S5 blocks, $2$ S5 layers, use LayerNorm before the SSM block and the activation function described in the original work, i.e. $a(x)=\textrm{GELU}(x)\odot\sigma(W*\textrm{GELU}(x))$.

Figure: Figure 7: Soft MoE architecture featuring a convolutional encoder, the mixture of experts layer and an RNN.
Refer to caption: x7.png

In the case of the SoftMoE architecture, we follow the same approach as in [] and replace the penultimate layer with a SoftMoE layer.
As in their work we use the PerConv tokenisation technique, i.e. given input $x\in\mathbb{N}^{h\times w\times 26}$ we take the output $y\in\mathbb{R}^{h\times w\times 32}$ of $f_{c}$ and construct $h\times w$ tokens with dimension $d=32$ that we then feed into the SoftMoE layer.
We always use $32$ slots and $4$ experts for this layer, see [] for details on this layer.
The resulting embedding is then passed into the two remaining linear layers before being also passed to RNN and used to produce an action or value, equivalent to the description above, compare Figure 7.

**Table 10: Number of trainable parameters in each model.**
|  | CNN-LSTM | SoftMoE-LSTM | CNN-S5 |
| --- | --- | --- | --- |
| Parameter Count | 197,254 | 316,102 | 193,670 |

Lastly, we describe our networks in terms of parameter count in Table 10.

### A.6 Additional analysis

#### A.6.1 Evidence of overfitting in Overcooked agents

**Table 11: Comparing the layout a policy was trained on versus on which it was being evaluated. The policies heavily overfit the training layout. All policies we tested exhibit this property.**
|  | Asymm | Cramped | Counter | Forced | Coord |
| --- | --- | --- | --- | --- | --- |
| Asymm | 343.4 | 0.0 | 0.0 | 0.0 | 0.0 |
| Cramped | 1.6 | 185.6 | 0.0 | 0.0 | 0.0 |
| Counter | 0.0 | 0.0 | 128.0 | 0.0 | 0.0 |
| Forced | 0.0 | 0.2 | 0.0 | 141.2 | 0.0 |
| Coord | 0.0 | 0.0 | 0.0 | 0.0 | 144.6 |

We show that agents heavily overfit their training layout in Overcooked in Table 11.
This is to be expected but verifying is nonetheless important.

#### A.6.2 Performance across levels

**Table 12: Mean episode solved rate for the different methods averaged over the respective testing layouts. The best result is shown in bold. We report aggregate statistics over three random seeds.**
| Method | CNN-LSTM | SoftMoE-LSTM | CNN-S5 |
| --- | --- | --- | --- |
| DR | $0.02\pm 0.0\%$ | $6.31\pm 10.14\%$ | $0.00\pm 0.0\%$ |
| PLR^⟂,∥ | $0.00\pm 0.0\%$ | $0.33\pm 0.3\%$ | $0.00\pm 0.0\%$ |
| Pop. PAIRED | $0.00\pm 0.0\%$ | $\mathbf{14.62\pm 7.6\%}$ | $0.00\pm 0.0\%$ |
| ACCEL^∥ | $0.00\pm 0.0\%$ | $0.08\pm 0.1\%$ | $0.00\pm 0.0\%$ |

To accompany the overall performance meassured by reward in the main paper in Table 3 we also meassure the mean solved rate on display it in Table 12.

#### A.6.3 Performance on individual levels

**Table 13: Performance on all evaluation layouts. We show the mean episode reward R and the mean episode solved rate SR. The overall best result per layout is presented in bold.**
| Layout | Method | CNN-LSTM | SoftMoE-LSTM | CNN-S5 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | R | SR | R | SR | R | SR |
| Cramped | DR | 1.70 | 0.0% | 1.54 | 0.2% | 0.00 | 0.0% |
| PLR^⟂,∥ | 1.12 | 0.0% | 5.02 | 2.1% | 0.14 | 0.0% |  |
| Pop. PAIRED | 1.44 | 0.0% | 37.02 | 57.7 % | 0.50 | 0.0% |  |
| ACCEL^∥ | 0.92 | 0.0% | 0.60 | 0.0% | 0.60 | 0.0% |  |
| Coord | DR | 0.00 | 0.0% | 0.00 | 0.0% | 0.00 | 0.0% |
| PLR^⟂,∥ | 0.00 | 0.0% | 0.00 | 0.0% | 0.00 | 0.0% |  |
| Pop. PAIRED | 0.00 | 0.0% | 16.78 | 14.6% | 0.00 | 0.0% |  |
| ACCEL^∥ | 0.00 | 0.0% | 0.04 | 0.0% | 0.02 | 0.0% |  |
| Forced | DR | 0.00 | 0.0% | 0.02 | 0.0% | 0.00 | 0.0% |
| PLR^⟂,∥ | 0.00 | 0.0% | 0.02 | 0.0% | 0.02 | 0.0% |  |
| Pop. PAIRED | 0.00 | 0.0% | 0.00 | 0.0% | 0.00 | 0.0% |  |
| ACCEL^∥ | 0.00 | 0.0% | 0.00 | 0.0% | 0.00 | 0.0% |  |
| Asymm | DR | 0.58 | 0.1% | 8.64 | 4.4% | 0.00 | 0.0% |
| PLR^⟂,∥ | 0.08 | 0.0% | 0.10 | 0.0% | 0.08 | 0.0% |  |
| Pop. PAIRED | 0.28 | 0.0% | 15.64 | 14.2% | 0.08 | 0.0% |  |
| ACCEL^∥ | 0.14 | 0.0% | 0.04 | 0.0% | 0.02 | 0.0% |  |
| Counter | DR | 0.00 | 0.0% | 0.00 | 0.0% | 0.00 | 0.0% |
| PLR^⟂,∥ | 0.00 | 0.0% | 0.00 | 0.0% | 0.00 | 0.0% |  |
| Pop. PAIRED | 0.00 | 0.0% | 1.38 | 0.0% | 0.00 | 0.0% |  |
| ACCEL^∥ | 0.00 | 0.0% | 0.00 | 0.0% | 0.00 | 0.0% |  |

We list the performance of every individual method on every single layout in Table 13.
Most notable is that some layouts are harder to learn than others.
Our agents especially seem to struggle with layouts requiring more complex forms of interaction, i.e. Coordination Ring, Counter Circuit and Forced Coordination.
Forced Coordination especially seems difficult to solve as no run achieves noticeable performance on it.
This might be due to the specific features of the layout, i.e. that agents have access to several objects and need to hand them over the counter to produce any result.

#### A.6.4 Population training curves

Figure: (a) Coordination Ring (6x9)
Refer to caption: x8.png

To both verify that our implementation is correct and to give an intuition into the performance of the members of the population, we present the training curves over all 8 seeds of training an FCP population in Figure 8.

#### A.6.5 Detailed results with FCP populations

**Table 14: Zero-shot results using SoftMoE-LSTM policies playing with an FCP population of experts trained on the respective layout exclusively. We report the mean episode reward and standard deviation. The best result per layout is put in bold.**
| Method | Asymm | Counter | Cramped | Forced | Coord |
| --- | --- | --- | --- | --- | --- |
| Random | $7.43\pm 12.19$ | $8.89\pm 4.65$ | $66.02\pm 38.28$ | $\mathbf{1.95\pm 1.92}$ | $20.49\pm 7.82$ |
| Stay | $5.32\pm 12.07$ | $0.38\pm 1.11$ | $20.67\pm 33.05$ | $0.00\pm 0.00$ | $0.95\pm 2.73$ |
| DR | $18.18\pm 1.69$ | $6.86\pm 5.27$ | $65.05\pm 5.15$ | $1.09\pm 0.21$ | $17.88\pm 10.27$ |
| PLR^⟂,∥ | $7.64\pm 0.89$ | $5.60\pm 1.29$ | $60.35\pm 6.89$ | $1.76\pm 0.86$ | $21.90\pm 1.26$ |
| Pop. PAIRED | $\mathbf{24.51\pm 3.44}$ | $\mathbf{11.11\pm 1.67}$ | $\mathbf{81.92\pm 6.33}$ | $1.59\pm 0.57$ | $\mathbf{29.72\pm 4.72}$ |
| ACCEL^∥ | $8.60\pm 0.98$ | $10.23\pm 0.85$ | $65.46\pm 4.62$ | $1.81\pm 1.25$ | $19.19\pm 1.93$ |

**Table 15: Zero-shot results using SoftMoE-LSTM policies playing with an FCP population of experts trained on the respective layout exclusively. We report the mean solved rate and standard deviation. The best result per layout is put in bold.**
| Method | Asymm | Counter | Cramped | Forced | Coord |
| --- | --- | --- | --- | --- | --- |
| Random | $8.52\pm 17.52\%$ | $5.00\pm 6.70\%$ | $69.43\pm 38.45\%$ | $0.00\pm 0.00\%$ | $30.89\pm 3.83\%$ |
| Stay | $6.81\pm 18.04\%$ | $0.02\pm 0.14\%$ | $21.75\pm 33.71\%$ | $0.00\pm 0.00\%$ | $0.14\pm 0.74\%$ |
| DR | $24.19\pm 4.60\%$ | $4.56\pm 5.32\%$ | $72.11\pm 6.29\%$ | $0.01\pm 0.01\%$ | $23.76\pm 18.85\%$ |
| PLR^⟂,∥ | $8.84\pm 1.31\%$ | $2.04\pm 0.95\%$ | $68.14\pm 1.21\%$ | $\mathbf{0.11\pm 0.12\%}$ | $30.89\pm 3.83\%$ |
| Pop. PAIRED | $\mathbf{32.48\pm 4.00\%}$ | $\mathbf{7.91\pm 1.38\%}$ | $\mathbf{85.54\pm 6.08\%}$ | $0.09\pm 0.07\%$ | $\mathbf{48.31\pm 11.08\%}$ |
| ACCEL^∥ | $9.58\pm 1.12\%$ | $6.79\pm 0.91\%$ | $69.01\pm 2.03\%$ | $0.06\pm 0.06\%$ | $24.13\pm 6.01\%$ |

We present detailed zero-shot cooperation results per layout in Tables 14 and 15.
As indicated through the averaged performance discussed in main text, we also find that PAIRED performs best on four of the five individual layouts in terms of zero-shot cooperation.

### A.7 Validating the implementation

As an open-source benchmark, we place an emphasis on a correct implementation of the benchmark, including all the baselines.
We do so in two important ways.
Firstly, we base our implementation on the implementation of the minimax benchmark [], making sure that we use publicly available code for all unsupervised environment design algorithms.
Secondly, we test the implementation and adaption of the Overcooked-AI environment by fixing the generated training layouts to a single layout during training.
This allows us to train on the 5 classic Overcooked layouts using our own implementation.
Our implementation is capable of solving these layouts, see Figure
8.
We do this in part to argue for the fact that our benchmark is hard to solve and this is not a function of poorly configured or wrongly implemented algorithms.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: For the error analysis, it is interesting that the models perform well in layouts like asymmetric advantages and cramped room. Both of those layouts are capable of being solved by a single agent to get a high cross-play reward. Can you draw any conclusions about how existing UED objectives might be biasing agents towards solving single agent problems even though the reward scheme technically makes it cooperative?
soundness: 3
strengths: This paper was a compelling first stab at building agents capable of generalizing across partners and problems. The authors did their due diligence with showing the speed of their environment, the performance of difference SOTA algorithms for UED across a variety of network architectures, and justifying the distinction between this work and prior methods for enabling ZSC, such as population based approaches. Overall, the implementation and evaluation details seem well thought through and justified, and the results point towards a clear gap in the multi-agent UED literature.
summary: MARL research has focused on studying how to generalize across novel partners on the same task. However, there is little work on generalizing to novel partners across novel tasks in the Overcooked-AI environment. This work introduces a new challenge for using DCD methods on fully cooperative tasks in Overcooked. They show how SOTA DCD methods cannot build agents that learn to coordinate with novel partners across novel problems, even with recent advances in neural networks, making this an open challenge for the MARL community.
weaknesses: While most of the paper was well written, Section 6 on benchmarking the challenge was a little confusing. Specifically, the difference between generalization performance and zero-shot cooperation performance could've been written with more clarity. It was not obvious if the authors were referring to generalization with partners during training across novel layouts, generalization to partners not seen during training for a single layout within distribution of the generator, and generalization in to novel partners across novel layouts.

Also, Figure 7 is a little unclear. While it is explained in the error analysis section, for someone quickly looking at it it is not obvious what the takeaway should be, or how the number of visits should relate to the return. (as a nitpick, the colors for the number of visits could be in a different pallet to be more obvious, not super necessary but may be helpful). In general, while benchmarking and understanding the limits of the challenges of existing UED approaches is very important and will lead to a high quality paper, the methods for analysis could use more than a few sentences of justification if possible. I.e. if the authors think state-coverage is a useful metric for ability to cooperate and that is why they choose to analyze the frequency of cells visited, then this motivation should be spelled out.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Can the authors please clarify the sampling procedure for each of the UED methods?
2. Can authors clarify what OvercookedUED provides on top of OvercookedJAX and minimax, from an implementation perspective?
soundness: 1
strengths: As the authors have pointed out, the vast majority of the coordination literature studies the coordination challenges that arise when paired with novel partners in the same environment as seen as train time (though with possible variations, such as different deck orderings in Hanabi [1].)  

As a result, there is indeed a significant gap in the coordination literature regarding test-time generalization to scenarios unseen during training, including to new layouts, different number of agents, new dynamics or new reward functions. A benchmark that allows to jointly evaluate the coordination and generalization ability of AI agents is a welcome contribution, and could drive significant algorithmic innovations. However, for the reasons explained below, I do not believe this paper fulfills this role. 

Other strengths include the writing, which I found clear and easy to follow, the commitment to open-sourcing the benchmark, and the thorough reporting of hyperparameters in the appendix. Finally, adopting the Dec-UPOMDP formalism allows to more rigorously define the extent of generalization the model should reach. 

[1] The Hanabi Challenge (https://arxiv.org/abs/1902.00506)
summary: This paper connects the Overcooked-AI implementation from `JaxMARL` [1] with the `minimax` [2] library for Unsupervised Environment Design (UED) to create a new benchmark, the Overcooked Generalization Challenge (OGC). 

OGC is meant to assess the generalization and coordination ability of agents when faced with novel partners in levels unseen during train time, where each level is a specific instantiation of the overarching Dec-UPOMDP.

The authors then evaluate four popular UED methods combined with three different policy architectures by looking at the mean episode return in self-play on held-out layouts (Table 3) and mean episode return when paired with fictitious co-play (FCP) policies in held-out layouts (Figure 5). In both cases, the held-out layouts are the 5 original layouts introduced in Overcooked-AI [3]. 

Finally, authors perform some analysis in an attempt to explain the low performance of DCD methods in the benchmark. 

[1] https://github.com/FLAIROx/JaxMARL/tree/main/jaxmarl/environments/overcooked

[2] https://github.com/facebookresearch/minimax

[3] https://github.com/HumanCompatibleAI/overcooked_ai
weaknesses: **1. Limited Scope**

While authors adopt the Dec-UPOMDP formalism, their proposed challenge only evaluates generalization to new layouts. This means they only assess generalization to new states, but not to new observation functions, reward functions or dynamics. As such, the whole benchmark is in fact simpler than the formalism suggests, and is equivalent to simply augmenting the set of initial states in Overcooked.

This problem is exacerbated by the environment being fully observable and deterministic, which in many layouts means that agents can simply react to what their partner is doing for a good, if not optimal, policy. See Section 7 of [1] for a discussion of partial observability and what that implies for limitations of Overcooked as a coordination challenge.

**2. Unclear Contributions**

The authors claim to introduce a new environment, OvercookedUED, and it is listed as one of their 3 main contributions of the paper on page 2. That said, from the description in section 5.1, OvercookedUED is simply a wrapper of the JaxMARL implementation of Overcooked-AI (i.e. Overcooked-JAX) that connects it to the `minimax` library. Since Overcooked-JAX already comes with a way of easily creating new layouts by providing an ASCII string [2], OvercookedUED appears to be only a minor contribution. 

Relatedly, Table 2 seems to report the steps-per-second achieved by running Overcooked-JAX, and it is unclear what information it carries that is specific to the authors' contribution.

**3. Different design space for different UED methods**

The description of OvercookedUED in Section 5.1 lacks key details on how the layout generation is performed for each UED method. Furthermore, there appear to be significant differences in the design space for each method, which can in turn affect performance. Based on the paper:

- For PAIRED, new layouts are generated by the teacher placing objects "sequentially and in a deterministic order". Since the teacher conditions only on the unfinished layout, does that mean that the number of objects of each type is fixed for all layouts generated by PAIRED?

- PLR can select 1 or 2 piles of onions, bowls, pots and serving locations, so the number of objects of each type is clearly variable. The authors do not specify the sampling procedure for the walls, which can greatly influence the layout.

- ACCEL can remove/add walls and move all other objects in the layout, but *not* remove them. There's also no mention on whether it can move agent starting positions.

Because the design space for each method can greatly influence its performance, the authors should clearly detail how each method operates, and also eliminate any discrepancy. In other words, *all layouts theoretically obtainable by one method should also be obtainable by the other ones*. The descriptions in the paper indicate that this is not the case.

**4. No assessment of policy learning ability**

The authors evaluate three different policy architectures. In Table 11, they show that one unspecified architecture can learn to overfit to a single layout. However, they provide no evidence that any of those architectures can learn a policy over multiple layouts. At a minimum, the authors should train each of the architectures on the 5 held-out layouts and show that they can indeed learn a general policy. They could then evaluate the performance gap between those expert agents and those trained in UED.

Without that experiment, it is impossible to conclude whether the poor performance in Section 6 is due to the UED methods failing to provide a suitable curriculum or to the policy networks being unable to learn a general policy.

**5. No assessment of robustness of FCP policies**

Coordination is evaluated in the ad-hoc teamplay [3] framework (despite the authors never using that term and failing to cite Stone et al. 2010), where the final policy is paired with a pool of test-time partners. In the case of the paper, the pool is obtained by Fictitious Co-play (FCP), but the authors do not evaluate whether those policies are themselves robust to new partners. This poses two problems. 

First, this evaluation pool is limited and likely not very diverse. Authors claim that "*There is currently no algorithm to train a population of diverse agents over a distribution of levels.*", but given that each level simply corresponds to a different initial state, algorithms such as TrajeDi [4] or ComeDi [5] could apply out of the box.

Second, there is no evaluation in the paper showing that FCP policies are robust to unseen partners. As such, poor returns in cross-play could be the fault of the trained policy or of the FCP policies themselves. See [6] for an extreme example of this. 

**6. Other minor issues**

- The authors misuse "Zero-Shot Coordination"; it is a specific setting introduced in [7] and requiring that test time partners be trained with the same algorithm used to train the policy being evaluated.
- The authors misuse "Other-play"; it is a method introduced in [7], not an evaluation setting. Instead, the authors should use the term "cross-play".
- "*Notice that the training levels in which our model performs well in are similar to Asymmetric Advantages and Cramped Room, while the worst levels are similar to the other 3 evaluation levels*". This statement is not substantiated by Figure 7. 
- Most figures use the visuals from Overcooked-AI but figure 7 uses the visuals form OvercookedJAX, without explanation. This can be confusing to authors which are not familiar with both works.

[1] Who Needs to Know?, https://arxiv.org/pdf/2306.09309

[2] Overcooked-JAX, https://github.com/FLAIROx/JaxMARL/tree/main/jaxmarl/environments/overcooked

[3] Stone et al., https://ojs.aaai.org/index.php/AAAI/article/view/7529

[4] TrajeDi, https://proceedings.mlr.press/v139/lupu21a/lupu21a.pdf

[5] ComeDi, https://arxiv.org/pdf/2310.15414

[6] ADVERSITY, https://openreview.net/pdf?id=uLE3WF3-H_5

[7] Other-Play, https://arxiv.org/abs/2003.02979

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Is it possible to **only** test partner generalization under the proposed environment (e.g., something similar to Fig. 5 but under seen layouts)? This would allow user to isolate between two types of generalization, diagnose, and develop better UED+ZSC algorithms.
soundness: 3
strengths: - The paper is well written and easy to understand.
- I really like that the authors aim to connect the UED and ad-hoc teamwork literature through this benchmark. The paper highlights clearly what is the gap of existing UED-compatible environments, i.e., environment and partner generalization, and thus is well motivated and novel.
- The evaluation seems to be thorough with many baselines and evaluation scenarios
- The project will be open-source
summary: This works proposes OvercookedUED, an GPU-based cooperative multi-agent environment that is compatible with UED methods. The proposed environment is more complex than existing UED-compatible environments. It also aims to connect UED with ad-hoc teamwork literature by allowing agents to be tested with unseen partners under unseen game levels. Empirically, existing UED methods struggle to produce strong agents in OvercookedUED. Further, UED-produced agents tend to cooperate better when they are more generally capable.
weaknesses: Though I really like the paper at a high-level, there are quite a few significant improvements that can make the paper more mature:

- The environment assumes a single recipe in the game and thus missing an important aspect of the game. There are variants of Overcooked that use multiple recipes that make the game more interesting and require more complex coordination [1,2,3]. I believe including more possible recipes improves the depth of the environment significantly, arguably adding another important layer of generalization.
- line 288: the observation space is not explained in details, it is especially hard for readers without prior Overcooked knowledge.
- It is not clear if the UED agents were just undertrained or the UED algorithms themselves struggle to generate complex layout. I wish the authors include training curves showing the performance on seen and unseen layouts of UED agents.
- The result of per layout generalization performance is missing from the main paper. I believe Table 13 shows this result but is not included in the main paper. I compare Table 13 and Table 14 (which is the raw results of Fig 5) and see that the self-play performance is actually lower than *test cross-play* performance, hinting that the agents are being *carried* by the FCP test partners. I believe that this observation is important and should be included and discussed in the main text.
- FCP does not explicitly generate diverse agents, which might not be a robust method for generating test partners.


Minor comments:
- Random and Stay baselines (Fig 5) are never introduced
- Fig 4 sentence 2: I find the sentence to not be consistent with the figure. Some of the generated layouts are definitely solvable.
- line 54: empty space on the right
- last line of page 2: the citation should not have parentheses as it is a part of a sentence


[1] Wu, Sarah A., et al. "Too many cooks: Bayesian inference for coordinating multi‐agent collaboration." Topics in Cognitive Science 13.2 (2021): 414-432.

[2] Yu, Chao, et al. "Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased." The Eleventh International Conference on Learning Representations.

[3] Charakorn, Rujikorn, Poramate Manoonpong, and Nat Dilokthanakul. "Generating diverse cooperative agents by learning incompatible policies." The Eleventh International Conference on Learning Representations. 2023.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: How do you plan to address the constraints on layout size in future iterations of the OGC to explore more scalable or natural scene representations? Would methods like graph-based representations or item embeddings be a viable approach?

What are the specific computational requirements for running the OGC, and have you established any benchmarks or guidelines to help researchers estimate the necessary resources for effective experimentation?
soundness: 3
strengths: - Originality: The paper introduces a new benchmark, the Overcooked Generalisation Challenge (OGC), which evaluates RL agents' zero-shot cooperation abilities with new partners in previously unseen layouts, addressing gaps in existing cooperative AI research.

- Quality: The study provides extensive empirical results, and includes detailed error analysis, showcasing both successful and challenging areas for current algorithms.

- Clarity: The paper is well-structured, clearly presenting the motivations, methodology, and findings. It is easy for readers to follow the research and apply its insights.

- Significance: The OGC fills the gap in cooperative AI research by focusing on zero-shot coordination, which is essential for real-world applications of AI in dynamic and unpredictable environments.
summary: This paper introduces the Overcooked Generalisation Challenge (OGC), which is a benchmark for testing RL agents' ability to cooperate with new partners in unseen environments. It focuses on zero-shot coordination to improve human-AI collaboration in complex scenarios. The study shows that current algorithms struggle with this challenge, highlighting the need for more adaptable cooperative agents.
weaknesses: - The challenge restricts layout sizes to ensure observability and compatibility with CNN-based encoders, which may limit the exploration of more natural and scalable representations for complex environments.

- The OGC environment, with its emphasis on running parallel layouts and complex training, may demand substantial computational resources. This could limit accessibility for researchers without high-performance computing capabilities and restrict broader experimentation and adoption of the benchmark.

Meta-review: The paper focuses on an interesting problem and has great intentions, however, it has significant limitations as highlighted by reviewer Kg98. Notably, it is a wrapper around two existing libraries, has only a constrained setting to test generalization and uses a different design space for different UED algorithms. We want to point out one detail that was missing in the discussion, the ACCEL paper intentionally randomized the number of total walls for DR and PLR to make the comparison with ACCEL fair, since otherwise this is a confounder. While it wasn't done perfectly, it is clearly the case that fixing the number biases the result to one of the methods. I would suggest re-running the experiments with this setup and adding more diverse novel tasks to get something a bit more robust.

Decision: Reject