The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Design
Authors: Constantin Ruhdorfer
Matteo Bortoletto
Anna Penzkofer
Andreas Bulling
Venue: Accepted by TMLR ()

Abstract: We introduce the Overcooked Generalisation Challenge (OGC) – a new benchmark for evaluating reinforcement learning (RL) agents on their ability to cooperate with unknown partners in unfamiliar environments. 
Existing work typically evaluated cooperative RL only in their training environment or with their training partners, thus seriously limiting our ability to understand agents’ generalisation capacity – an essential requirement for future collaboration with humans. 
The OGC extends Overcooked-AI to support dual curriculum design (DCD).
It is fully GPU-accelerated, open-source, and integrated into the minimax DCD benchmark suite.
Compared to prior DCD benchmarks, where designers manipulate only minimal elements of the environment, OGC introduces a significantly richer design space: full kitchen layouts with multiple objects that require the designer to account for interaction dynamics between agents.
We evaluate state-of-the-art DCD algorithms alongside scalable neural architectures and find that current methods fail to produce agents that generalise effectively to novel layouts and unfamiliar partners.
Our results indicate that both agents and curriculum designers struggle with the joint challenge of partner and environment generalisation.
These findings establish OGC as a demanding testbed for cooperative generalisation and highlight key directions for future research.
We open-source our code.

Published in Transactions on Machine Learning Research (09/2025)
The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown En- vironments Using Unsupervised Environment Design
Constantin Ruhdorfer
constantin wmstantii..uuldarfe@@iis uunts//ur.......ssssuuuggg....
Collaborative Artificial Intelligence University of Stutigart, Germany
Matteo Bortoletto
matteo. muttoohhhrrrrttttt..........itir
Collaborative Artificial Intelligence Universily of Slullgarl, Germany
Anna Penzkofer
Collaborative Artificial Intelligence University of Stutigart, Germany
Andreas Bulling
andreas. buitruusbblliig@oii....
uni-stutlgar..dd www.:uuwinnnniii..iiituugg.r....
Collaborative Artificial Intelligence
University of Stutigart, Germany
Reviewed on OpenReview: https: // openreview. net/ forum? id= X2=
Abstract
We introduce the Overcooked Generalisation Challenge (OGC) new benchmark for eval- uating reinforcement learning (RL) agents on their ability to cooperate with unknown part- ners in unfamiliar environments. Existing work typically evaluated cooperative RL only in their training environment or with their training partners, thus seriously limiting our ability to understand agents' generalisation capacity an essential requirement for future collabo- ration with humans. The OGC extends Overcooked-Al to support dual curriculum design (DCD). fully GPU-accclerated, open-source, and integrated into the minimax DCD benchmark suite. Compared to prior DCD benchmarks, where designers manipulate only minimal elements of the environment, OGC introduces significantly richer design space: full kitchen layouts with multiple objects that require the designer to account for interaction dynamics between agents. We evaluate state-of-the-ar DCD algorithms alongside scalable neural architectures and find that current methods fail to produce agents that generalise ef- fectively to novel layouts and unfamiliar partners. Our results indicate that both agents and curriculum designers struggle with the joint challenge of partner and environment generali- sation. These findings establish OGC as a demanding testbed for cooperative generalisation and highlight key directions for futurr rescarch. We open-source code
1 Introduction
Developing computational agents capable of collaborating with each other has emerged as key challenge
in artificial intelligence (AI) research (Dafoc et al. 2020). Recent years have seen considerable advances in
developing cooperative reinforcement learning (RI) agents (Stone et al. 2010 Hu et. al. 2020 Choudhury
et al. 2020: Ding et al. 2024) and several benchmarks were proposed to evaluate their generalisation
abilitics (Samvelyan et al. 2019 Bard 2020). However, these benchmarks typically trcat genoralisation
"Qur project web-page accessible http:///gallaaaaiie.rraaaaaatttr

---

Published in Transactions on Machine Learning Research (09/2025)
Training
Cooperate with a novel partner in novel level
Hold-Out Evaluation Partners
Environment Distribution
Hold-out Evaluation Levels
Figure 1: In the Overcooked Generalisation Challenge (OGC), during training, agents can access a generator that outputs new training environments. During evaluation, agents are presented with a novel environment and an unknown partner to cooperate with.
to novel environments (Cobbe el al. 2019) and novel partners (Hu et al. 2020 Carroll et al. 2019 as distinct challenges. Yet, future human-A Al collaboration will require agents to generalise along both ares simultancously. For instance, an autonomous robot assisting in disaster response team must coordinate with ever-changing human partners in unfamiliar, dynamic enviromments.
Overcooked-AA A1 (Carroll et al. 2019) has emerged one of the most popular benchmarks for evaluating zero- shot coordination. Nonetheless, agents are typically trained and evaluated on few fixed layouts (Strouse
2021 Yang el al.) 2022 Zhao et al. 2023 Yu et al. 2023 Wang el. al. 2024). This common practice limits the benchmark's ability to assess generalisation in two key ways: First, agents may overfit to specific spatial configurations, interaction bottlenecks, or object placements seen during training, without acquiring coordination strategies transferable to novel environments. Second, agents may implicitly adapt their training partners' behaviour patterns known layouts, but this does not test their ability to infer intent adapt dynamically to the bchaviour of unknown partners. Crucially, these two challenges coordination strategies oflen depend on the structure of the environment. For example, in layouts with narrow passages or asymmetrical tasks, agents need carefully coordinate their movements or take on com- plementary roles. Testing how well agents gencralise to new partners in just one environment overlooks this connection. Similarly, evaluating agents only in new environments with familiar partners misses the challenge of adapting to different behaviours. Real-world teamwork requires agents generalise jointly over both axes.
et al.
closely linked:
To address these limitations, introduce the Overcooked Generalisation Challenge (OGC) novel coopera- tion benchmark based on Overcooked-Ae AI to evaluate RL agents' ability to collaborate with unknown partners and in novel environments (see Figure 1). In contrast to prior work that augments fixed set of test envi- ronments for training Jha et al. (2025), we introduce an unsupervised environment design (UED) approach DDennis et al. 2020), situated within the broader Dual Curriculum Design (DCD) framework, to procedurally generate a large set of diverse training layouts. DCD refers framework in which generator (designer) and level selector (curator) co-evolve to construct curriculum tailored to the agent's learning progress (Jiang et al. 2021a). This enables
evaluate generalisation in more challenging and realistic way, where agents encounter entirely novel environments without prior exposure. As such, the OGC is the first benchmark to combine UED with multi-agent zero-shot cooperation, thus bridging two previously separate lines of rescarch.
We evaluate trained agents on a suite of human-authored test. layouts across three coordination settings: self-play, cross-play. and ad-hoc teamwork. We show that OGC presents a significant challenge for current UED algorithms and scalable neural architoctures. Among the evaluated methods, only PAIRED (Dennis) et al. 2020) combined with a Soft Mixture-of Exxturreof-Experts (SoftMoE) policy Sbbando-Ceron et al. 2024) achieves partial generalisation. Our results reveal key limitation: current dual curriculum design (DCD) methods
2

---

Published in Transactions on Machine Learning Research (09/2025)
with hand-crafted weak procedural designers fail to generate sufficiently diverse and structured training levels, and thus struggle in complex design spaces like Overcooked. By contrast, methods with learned environment generators, such as PAIRED, show better adaptability. These findings highlight the need for joint-curriculum methods that combine effective partner gencralisation with adaptive curriculum generation. Our contributions are as follows:
We introduce the Overcooked Generalisation Challenge (OGC), the first open benchmark that jointly evaluates agents environment and partner generalisation in cooperative multi-agent settings.
We release OvercookedUUDD a JAX-accelerated, open-source extension of Overcooked-A AT that supports unsuperviscd environment design (UED) via integration with statc-of-the-art dual curriculum design (DCD) algorithms in minimax (Jaang et al. 2023) and JaxMARL (Rutherford et al., 2024b
Through extensive experiments, we demonstrate that current DCD algorithms and scalable neural archi- tectures including recent stato-of-the-art models fail to generalisc effectively across environments and partners, thus establishing OGC a challenging new testbed for multi-agent cooperation.
2 Related Work
2.1 Partner Generalisation
Generalisation to novel partners has been studied under the ad-hoc teamwork (Stone et al. 2010 and zero-shot coordination (Hu et al. 2020) paradigms, both motivated by improving human-AT cooperation. A prominent benchmark in this space is Overcooked-Al (Carroll et al.) 2019), where agents must jointly prepare and serve dishes. Many recent. works use this environment to evaluate ad-hoc coordination
capabilities [Strouse et al. 2021 et al. 2023b Yan et al. 2023a Liu et al.
2024 Tan et al. 2024h.
In this setting, self-play often fails to produce agents that generalise to novel partners (Carroll al.) 2019
Consequently, researchers have turned to population-based methods that. train diverse partner policies
and learn best-response strategies in fixed environments (Zhao al. 2023 Yu et al. 2023 Wang et al. 2024 However, these methods scale poorly, as training cost increases lincarly with population size per enviromment Yan et al. (202bb).
In contrast, our setting trains agents across a large distribution of procedurally generated environments. Cooperation evaluated on human-authored levels not seen during training making population- based ap- proaches infeasible and calling for learning strategies that operate effectively across novel partners and tasks,
2.2 Environment Generalisation
RI. agents fail to generalise to new environments out-of-the-box (Zhang et al. 2018a) and instead require sufficiently diverse training levels to generalise well (Zhang et al.) 2018b Cobbe al. 2019 2020 One es-
tablished approach to generate diverse training data is domain randomisation (DR: Jakobi 1997.. However,
DR may produce uninformative samples (Khirodkar et al. 2018). which hinder learning (Dennis et al. 2020
To improve sample quality, unsupervised environment design (UED) (Dennis et al. 2020 adaptively generates levels that match an agent's current capabilities. Prominent UED algorithms include PAIRED (Dennis et al. 2020). Prioritized Level Replay (PLR) (Jiang et al. 2021b). and ACCEL (Parker-Holder ct al. 2022). These methods fall under the broader Dual Curriculum Design (DCD) framework (Jiang al. 2021a), in which generator and curator co-evolve to construct an adaptive training curriculun. While the development of DCD methods has been steady, they have mostly been explored in simple single-agent environments, c.g. in mazes (Dennis et al. 2020: Jiang et al. 2021a Parker-Holder et al. 2022 Jiang et al.
2023 Li et al.) 2023a Beukman et al. 2024), bipedal walker [Wang et al. 2019 2020
Parker-Holder et al.
2022 or car racing environments (Jiang et al. 2021a).
Multi-agent UED, in contrast, remains largely underexplored. Existing works are either closed source Team al. 2021 Bauer et al.) 22023), do not address a (folll-)cooperative setting (Suarez al. 2021 Samvelyan al.] 2023) or only feature multi-agent, path-finding with no agent interaction (Rutherford)
2023
3

---

Published in Transactions on Machine Learning Research (09/2025)
et al. 2024a).
Moreover, the underlying design spaces are shallow often involving only walls, agents, sparse control points or pregenerated levels without design control (Dennis et al. 2020 Parker-Holder et al. 2022
Samvelyan et al. 2023 Nikulin et al. 2023).
In contrast, OGC introduces a fully cooperative multi-agent UED environment with rich object interactions (e.g. pots, onions, plates) and complex spatial dependencies (see Figure 3). The difficulty of each task critically depends on object placement, making level design substantially more challenging. The OGC thus contributes the first open-source cooperative multi-agent UED environment in which agents exposed to novel partners during evaluation.
2.3 Combining Partner and Environment Generalisation
While many benchmarks focus on either environment or partner generalisation Lowe et al. 2017 Foerster et al.) 2018 Hu et al.j 2020),, few evaluate both simultaneously. A recent Overcooked study explored cross-environment cooperation (Jha et al., 2025), but their approach relied training agents across augmented variations of known test levels, therefore implicitly assuming access to the test distribution and significantly constraining the scope of generalisation.
In contrast, the OGC poses a strictly harder challenge: agents must learn to cooperate in entirely novel environments and with unseen partners, with no prior exposure to evaluation layouts or their structure. Training conducted solely via procedurally generated levels using UED, without handcrafted augmentations or test-level tuning. The sctting of (Jha et al., (0255 can be scen as addressing a reducod version of OGC where one assumes access the lesting layouts.
This decoupled setting reveals that current DCD algorithms struggle in high-complexity cooperative do- mains, and motivates a new class of approaches - UED-ZSC methods that jointly tackle unsupervised environment design and zero-shot coordination. We propose OGC as both a benchmark and a testbed to support this emerging line of research, encouraging future methods that generalise across partners and environments in realistic, open-ended tasks.
3 Preliminaries
We formalise our cooperative multi-agent UED setting as a decentralised under-specified partially ob- servable Markov decision process (Dec-UPOMDP) with shared rewards. Dec-UPOMDP is defined as M = A, 2, S. T. OM, DMRRM, 7) in which W is the set of agents with cardinality n, is a set of observations, and S. is the set of true states the environment. Partial observations of E ft are obtained by agent E A' using the observation function O 8 N + Ω. Following Jiang al. (2021a),, level Me is defined as fully-specified environment given some parameters O E O. In it, agents each pick an action ag E A simultaneously to produce joint, action a (aI...., a..) and observe a shared immediate reward R(s, a). Then, the environment transitions to the next state according to a transition function : x A² x... x A" x θ + A(S) where A(S) refers to the space of distributions over S. 7 E [0, specifies the discount factor. Agents learn a policy x. The joint policy æ together with the discounted return R2 = Σ 7'r441 induce a joint action value function Q* E. 1..
RR [R at). Our formulation 2021 by introducing O as a of
extends the Dec-POMDP framework (Oliehoek & Amato 2016 Wu al.
free environment parameters making the model suitable for unsupervised environment design. This follows
2023 but differs from Samvelyan ct al. (2023) in assuming shared rewards and cooperative, rather than general-sum, structure. Within our Dec- UPOMDP, we perform UED to train a policy over a distribution of fully specified environments that enable optimal learning. This is facilitated by obtaining an environment policy A (Dennis et al.) 2020) that specifies sequence of environment parameters @ for the given policy that is to be trained. How A is obtained depends on the DCD method. In OvercookedUED, θ represents the possible positions of walls, pots, serving spots, agent starting locations, and onion and bowl piles adjusted by A throughout training.
previous work [Dennis et al. 2020 Jiang et al. 2021a Samvelyan et al.
4

---

Published in Transactions on Machine Learning Research (09/2025)
OvercookedUED
Curator
step
Retrieve
Update
obs
Overcooked Mutator
Update
Student
Train
evolve
Designer
Editor
Figure 2: Overview of the OGC and how it is typically used in DCD algorithm. The OGC supports teacher-
based UED methods like PAIRED (Dennis et al. 2020) and baiit methods like ACCEL Parker-Holdee
et al. 2022 via mutator functions of existing layouts.
Design Space
Plate Pile
Onion Pile
Agents
Serving
Pot
Figure 3:
The OGC features large design space in which many different elements have to be placed in
relation to each other, creating challenging environments for both environment designers and agent training.
4 The Overcooked Generalisation Challenge
The Overcookod Generalisation Challenge is new benchmark that allows to evaluate agents on their ability to cooperate with unknown partners in previously unseen environments. Unlike existing benchmarks, the OGC combines unsupervised environment design with multi-agent coordination, introducing the first open-source UED testbed for cooperative RL covering both environment and partner generalisation. Built Overcooked-AI AC Carroll et al. 22119). OGC integrates with DCD algorithms to support procedural training, layout mutation, and zero-shot evaluation across complex coordination tasks.
Overcooked-AT is one of the most popular environments for cooperative RL and ad-hoe teamwork. In the enviromment, two agents are tasked with cooking and delivering soup together. Delivering soup entails collecting and putting onions into pot, letting it cook, collecting the soup with plate and delivering it to serving station, which results in reward of 20.
Figure shows how OGC interfaces with various DCD methods. It supports both teacher- based approaches (c.g., PAIRED (Dennis et al.) 2020) and odit-based mothods (o.g., ACCEL (Parker-Holder et al. 2022) using mutator functions that transform existing layouts. Figure B Eilustrates the complexity of layout design in Overcooked, where task difficulty depends critically on the spatial configuration of multiple interdependent objects and agents.
4.1 Environment and Layout Generation
OGC extends the JaxMARL implementation of Overcooked-AI (Rutherford al.) 2024b), which defines discrete action space left, right, up, down, interact, stay and an observation space consisting of 26 binary masks of size h w, encoding the positions of agents, objects, and obstacles. To support large-scale
5

---

Published in Transactions on Machine Learning Research (09/2025)
Original Evaluation Layouts
Handcrafted and Transformed Evaluation Layouts
Base
Rotated
Mirrored
Scaled
Squeezed
Figure First, we propose to evaluate agents in self-play on the five original layouts and several layouts that are created from several symmetry classes to evaluate their ability to generalise. We combine range of transformations shown in the bottom to generate 28 additional layouts. Second, evaluate coordination with novel partners in the original five.
training, we enable parallel rollouts across multiple layouts, requiring all layouts to be padded to a fixed maximum height and width w. This enables fast training and exccution spoeds across hundreds or thousands environments using JAX.
4.2 Curriculum Learning in OGC
OGC exposes two core interfaces for DCD methods: OvercookedUED and the Overcooked Mutator.
OvercookedUED implements a teacher environment where a generator policy sequentially places objects onlo a layout grid. At each step I, the teacher selects a grid cell and places one object from a fixed sequence (walls, agents, goals, ingredient piles, pots, bowls). If the target cell already occupied, the object is placed randomly in free cell of the same type. Placing two elements of the same type in the samee location results in the second being ignored, enabling variable object counts per type consistent with prior UED cesigns (Dennis et al. 2020). For UED methods that lack teacher component (e.g., PLR), OvercookedlUE. also provides a random environment generator that follows the same structure as the teacher but samples object positions uniformly.
The Overcooked Mutator enables layout evolution for edit-based methods. It supports five operations: (1) toggling walls and free spaces, (2-5) moving goal, pot, plate, and onion pile positions. Agent start positions remain fixed. The number of mutations applied can be configured to control curriculum granularity.
All versions leave layout solvability unchecked, following the convention in prior UED work Dennis et al. Jiang et al. 2023) and place responsibility for level quality on the DCD method.
2020
4.3
Evaluation Protocol
We study three evaluation modes:
Self-play, zero-shot and ad-hoc coordination.
We propose to test agents in self-play to evaluate how well they generalise to novel levels. Figure 4 iillustrates our evaluation suite. We use the five original Overcooked-AI layouts (Carroll et al. 2019) and 32 layouts created via geometric transformations from a simple square base layout and randomly generated layouts to assess generalisation. The five original layouts enable comparisons to earlier work, and the transformed layouts expose whether the agent's behaviour is robust (or idcally invariant) layout transformations that do not require different strategy. We secondly propose evaluate ad-hoc teamwork setting (Stone al. 2010), train populations for 24 agents for the original five layouts via: Fictitious Co-Play (FCP) (Strouse

---

Published in Transactions on Machine Learning Research (09/2025)
Tokenisation
Encoder
Action
Gate/Combination
Individual Expert
LSTM
Dense
Figure 5: The SoftMoE-LSTTY agents architecture used in this work. We employed the PerConv tokenisation
technique introduced Obando-Ceron et al. (0024).
DR
PLR
PAIRED
ACCEL
Middle
End
Figure 6: Sample levels generated by the different methods after 15, 000 (Middle) and 30, 000 (End) epochs. Even after considerable training, none of the methods can guarantee the generation of solvable layouts (Middle row, leftmost and rightmost)..
Each population includes low-, mid-, and high-skill checkpoints (10%, 50%, and 100% achieved return) for diversity. MEP 0.01. This setting evaluates the adaptability of trained agents to diverse, possibly brittle partners. Finally, we also evaluate in zero-shot coordination setting (Hu et. al. 2020) in which we test agent's capability adapt to partners that themselves were trained for the zero-shot coordination setting. Both ad-hoc teamwork and zero-shot coordination evaluated on the original five.
et al. 2021 and Maximum Entropy Population Training (MEP) (Zhao et al. 2023
uses an entropy coefficient
We report two metrics. First, the mean episode reward and second, the solved rate: the proportion of episodes where at least two soups are delivered, distinguishing goal-directed behaviour from random actions. Finally, we also conduct qualitative error analysis to examine failure modes across different levels.
Our primary evaluation metric based on episode return serves direct measure of both task efficiency and partner adaptability. Since rewards in Overcooked within fixed time horizon, the return reflects how elficiently agents coordinate to complete tasks under temporal constraints. Additionally, measuring return in cross-play and ad-hoc teamwork settings isolates the agent's s ability to adapt to novel partners without prior joint training, thus capturing adaptability in zero-shot cooperation.
only obtained by successfully delivering soups
5 Experiments
We conducted scrics of experiments establish performance basclines for partner and environment gener- alisation using state-of-the-att DCD methods and policy architectures. We first compare several policy archi- fectures CNN-LSTM, SofLMoF-LSTM, and S5-based models fixed set of evaluation layouts to identify

---

Published in Transactions on Machine Learning Research (09/2023)
Oracle Performance
Overfitting Baseline
Asymm
220.4
277.8
247.9
478.0
0.0
0.0
0.0
0.0
Cramped
241.3
245.5
189.5
259.6
0.6
Counter
91.2
152.7
84.9
0.0
0.0
117.2
0.0
0.0
Forced
196.8
204.5
133.5
0.0
0.2 0.0
199.6
0.0
Coord
197.8
204.5
119.3
0.0
0.0 0.0
199.6
CNN-LSTM
SoftMoE-LLTM
CNN-S5
Asymm
Cramped
Counter
Forced
Coord
Figure 7: Return of policies in all evaluation layouts. Left: evaluation layouts to be used as an oracle. Mof-MoE-LSTM shows the best performance. Right: Shows SoftMoE-LSTM agents trained only on one layout, which overfit.
Results of policies trained across all five
a strong baseline. We then assess whether agents trained with different DCD methods can generalise to novel environments without prior exposure to the test distribution. This is followed by experiments to establish coordination capabilitics with unscen partners by testing agents ad-hoc team play against diverse policy populations (FCP, MEP), as well as in zero-shol coordination with agents trained under the same method but different random seeds. Finally, analyse failure modes through detailed error analysis, identifying struc- tural layout characteristics that correlate with poor performance, such as object spacing and path complexity.
All agents were trained using MAPPO Yuu et al. 2022). a strong cooperative multi-agent baseline. For DCD, we tested diverse algorithms with distinct generation principles (Dennis et al. 2020 Jiang et al. 2023): domain randomisation (DR), priority-based replay (robust parallel PLR- edit-based curriculum parallel ACCEL" and learned environment design (Pop. PAIRED). We excluded POET (Wang et al, 2019h in this analysis as it outputs specialists rather than generalists, which require (Parker-Holder Additionally, we excluded MAESTRO as it is based on prioritised fictitious self-play (Heinrich 2015 Vinyals et al.) 2019)) that is not readily adaptable to the cooperative setting (Strouse et al.
et al.
2022
ct al.
b. We chose these methods as they offer better theoretical guarantees (PLR* vs PLR), better runtime performance (ACCEL and )LR!!), or bocause we found them to perform better empirically (Pop. PAIRED vs PAIRED). To ensure fair comparison, we standardised the environment design space: each method placed between one and 15 walls (either randomly or through a learned teacher policy), along with one two items per object category (pots, onions, serving points, etc.). Layouts were procodurally generated using either teacher actions (PAIRED) stochastic editing (ACCEL, PLR), with training conducted across five seeds, 32 parallel environments for 30,000 training iterations (~ 400M steps). All architectural and training hyperparameters were selected via grid search and are detailed in Appendix A.4
2021).
5.1 Experiment 1: Policies and Baselines
Before evaluating generalisation on OGC, we identified a strong agent architecture that can serve as a policy backbone across all experiments. Comparing network architectures allows us to: (1) understand how architecture affects generalisation, and (2) establish upper-bound oracle performance on evaluation layouts, which will serve as reference points throughout the paper. We explored the following three architectures (see Appendix A.5 for details): CNN-LSTM is standard convolutional encoder followed by an LSTM that demonstrated strong performance in previous work (Yu et al. 2023) SoftMoE- is an enhanced architecture using a Soft
(SoftMoE) module (Obando-Ceron et al. 2024 and PerConv tokenisation, replacing the final layer of the CNN-LSTM. We explore SoftMoE agents because of their strong parameter scaling properties (Obando-Ceron et al. 2024 CNN-S5 a CNN encoder paired with $5 layers (mmtth et al.] (023) instcad of LSTMs, inspired by structured stato-space models (Gu al. 2022 that showed strong performance in meta-RL (Lu al. 2023 Each agent was trained on the five human- designed evaluation layouts (Cramped Room, Asymmetric Advantages, etc.) to assess whether the architecture could
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Mean episode reward for the different methods averaged over the respective testing layouts. The best result is shown in bold. We report aggregate statisties over five random seeds and the 95% confidence interval. We include oracles trained on the five testing layouts to establish an empirical upper bound.
Method
CNN-LSTM
SofM
CNN-S5
DR
2.84 + 7.37
4.21 + 9.52
0.06 + 0.17
PLR
0.18 + 0.05
0.68 + 0.75
0.13 = 0.14
PAIRED
0.52 + 0.39
13.12 + 11.46
0.02 + 0.44
ACCEL
0.24 + 0.41
0.61 0.55
0.06 = 0.12
Oracle
204.44 + 29.61
216.76 + 34.41
166.63 + 23.62
150
125
Forced Coord.
100
Cramped
Return
Coord. Ring
75
Asymm. Adv.
Counter Circuit
Train
25
5000
10000
15000
20000
25000
30000
Policy Updates
Figure Returns over training for both training and evaluation layouts for our SoftMOE-LSTM-PAIREDD policy. The policy has had some success in generalising, but its generalisation gap remains substantial.
fit the joint task. In all experiments that follow, we refer to these oracles they have access to test environments during training and thus represent an empirical upper bound without generalisation.
As shown in Figure (left), all architectures are capable of fitting the evaluation layouts in self-play. SofLMoE-LSTM achieves the highest returns across the board (Table 1,, with lower variance and better stability. CNN-S5 significantly underperforms, suggesting S5 layers may not suit cooperative RL in this setting. To confirm these models did not simply memorise layout-specifie strategies, we trained SoftMoE-LSTM agent single layout and evaluated it on all five. The steep performance drop suggests overfitting, underscoring the importance of multi-layout training.
Key takeaway: We find that SofLMoE-LSTM generalises best among lested archilectures, and adopt it for all subsequent experiments. This result also suggests that mixture-of-expert routing may support generalisation multi-object, sparse-reward environments like Overcooked.
5.2 Experiment 2: Generalisation to Novel Environments
We then tested whether agents trained with unsupervised environment design can generalise to unseen environments. Unlike recent work that trained on augmented variants of test. levels (Jha et al. 2025 agents in the OGC are faced with entirely new test environments without prior exposure.
As can be seen in [ despite using tuned implementations and scalable architectures, most methods fail to generalise, achieving near-zero returns the evaluation layouts. Only Population PAIRED has limited success, significantly outperforming other methods (p < 0.05). with mean solved rate of 14.6% = 7.7. All other methods barely go above 0% solved levels, suggesting that training on randomly generated or edited layouts is insufficient to prepare agents for the coordination structure of evaluation tasks. We check and find that roughly a quarter of randomly generated training layouts are unsolvable, and many others require long, inefficient action sequences. This further highlights why these methods struggle: not only is the evaluation set challenging, but producing feasibly solvable training levels is itself key difficulty. The

---

Published in Transactions on Machine Learning Research (09/2025)
Table Performance on mirrored, rotated and squeezed levels as illustrated in Figure 4 The transformations are grouped such that for each category, the optimal strategy remains the same modulo mirroring and rotation. Large, medium and small are defined in terms of the available movement space. Squeezed and squeezed small define increasingly narrow spaces. The large confidence interval suggests that agents do not learn general strategies for Overcooked. We analyse Sof-MoEELSTM agents.
Method
Large
Medium
Small
Squeezed
Squeezed Small
Average
DR
18.1 30.1
13.1 + 20.2
10.4 + 15.8
11.3 + 19.9
10.3 17.4
13.1 + 21.1
PLR
0.2 0.4
0.7 1.0
1.0 0.6
0.0 0.0
0.1 + 0.1
0.5 0.5
PAIRED
16.2 17.6
31.5 38.4
25.4 + 28.7
4.7 6.9
8.4 8.9
19.9 - 22.2
ACCEL
1.5 2.6
1.4 0.6
0.8 = 0.6
0.2 0.4
0.1 = 0.3
0.9 0.8
Partner Population = MEP
Partner Population = FCP
150
Mean Episode Reward
125
100
75
25
0
Coord. Ring
Forced Coord.
Counter Circuit
Asymm. Adv.
Cramped
Coord. Ring
Forced Coord.
Counter Circuit
Asymm. Adv.
Cramped
Random
Stay
PAIRED
ORACLE
Figure Ad-hoe team-play performance of SOLIMOF-LSSM-PAAREED and other baselines with both an MEP and an FCP expert population. We measure the returns of multiple seeds across the five original layouts. Thinner lines the bars indicate 95% confidence intervals around the means. While SoftMoE- LSTM-PAIRED outperforms simple baselines, it fails to reach oracle performance.
SoftMoEE-LSTM-PAIRED policy only shows mediocre performance on Asymmetric Advantages and Cromped Room and completely fails to coordinate effectively in more complex layouts, such Counter Circuit Forced Coortination. The training curve in Figure 8 confirms this: While the Soft agent converges on training layouts, its generalisation gap on evaluation levels remains substantial and persistent.
To probe this further, we evaluate agents on systematically transformed versions of base layouts including mirrored, rotated, and squeezed variants that preserve the underlying coordination task but alter spatial structure. Idcally, a general policy should perform consistently across such transformations. However, performance fluctuates widely (see Table 2 revealing that agents fail to learn spatially invariant coordi- nation strategies. This suggests that current methods often overfit to superficial spatial patterns rather than acquiring abstract cooperation skills. However, even PAIRED exhibits high variance across layouts, highlighting its limited robustness.
Key takeaway: Even state-of-the-art DCD methods fail to generalise to unseen, complex multi-agent coordination tasks. This suggests that the OGC introduces a richer and more challenging design space than prior UED environments and reveals limitations in current level generation strategics highlighting the need for stronger curriculum learning and generalisation-awar training algorithms.
10

---

Published in Transactions on Machine Learning Research (09/2025)
Coord Ring
Farced Coord
Asymm AdV
Cram ped Room
Counter Circuit
199.2
11.2 16.3
29.6 37.5
3.3
138.8
2.9
302 163.0
200 00000230000
118.8
2.9
8 183.3
195.22199
22.9
158.0
122.0
210.4147.3
150.0
ORACLE
210
55.0
22.9
21.522
178
40.5
199
2158.
257.8
259.2
87.6
97.8
150.2
58.2
68.2
192.4
43.1
197.6
15.0
571.
2237.5
253.8
223.8.224.7
91.0
78.2
216.0
200.0
259.8
239..0
158.0
3.8
2.2
2.2
7.3
6.0
0.2
0.2
0.0
0.0
0.3
31.8
11.6
7
37.3
33.9
13 8
14.5
21.0
35.6
49.5
0.2
0.0
0.0
3.5
0.1
1-
1.2
3.9
0.8
0.1
011
1.0
1.2
33.4
15.6
9.8
10.2
8.7
5
22.6
0.0
0.0
0.5
0.0
PARLD
0.0
0.4
42.9
2.0
0.7
74.8
41
0.7
2
1
Agent
Figure 10: Zero-shot cooperation returns of the oracle and the SotMMOE-LSTM-PATREE) agents (trained with different seeds). Each square shows the performance of the row and column seeds. Self-play performance is thus displayed the diagonal.
5.3 Experiment 3: Generalisation to Unknown Partners
To evaluate whether agents trained with DCD methods can coordinate with unknown partners in previonsly unseon onvironments, we investigated two scttings: Ad-hoc teamplay and zero-shot coordination. Wo evaluated agents against diverse populations of pre-trained partners using two protocols: FCP (Strouse et al. 2021) and MEP (Zhao al.) (002).. Each population consisted of 24 agents with diverse skill levels and learning historics (sec Appendix A.6.4).
F Figure 9 shows the performance of the SolMoo--SSTMPPAIIE agent compared to three baselines: stationary partner (stay), a randomly acting agent (random), and an oracle trained on the evaluation layouts (see above). As can be scen from the figure, the SoftMoE agent consistently outperforms the basclines but fails to reach oracle performance. Notably, it often performs only slightly better than random coordination a sign of poor robustness to novel partners. We hypothesise that this gap stems from the divergence bet the training layouts (often open and simplified) and the evaluation layouts that have different cooperation demands. As shown in Figure 6] current DCD methods tend to converge toward complxt-compllxitt levels that facilitate early success but fail to expose agents to realistic partner dependencies.
We also assessed ZSC by evaluating whether the agent can coordinate with an independently trained copy of itself with a different random seed. This setting removes population diversity and isolates the agent 's gen- eralisation to novel weights and latent partner strategies. As shown in Figure 10) SoftMoE-LSTM--AIREE performs poorly across the more complex evaluation layouts (Coordination Ring. Forced Coordination, Counter Circuil), and even underperforms its oracle counterpart. Interestingly, in the simpler layouts, the agent occasionally achieves higher returns in cross cross-play than in celf-play suggesting that some diversity across seeds may help mitigate overfitting.
Key takeaway: DCD-trained agents despile some progress in environment generalisation struggle lo generalise to novel partners in unknown environments.
5.4 Experiment 4: Qualitative Failure Analysis
We perform a final experiment to investigate the agents' poor performance. To better understand which structures impede performance, we show cell visit patterns for the best and worst-performing layouts in Fig- ure 11 While on many layouts, our PAIRED SoftMoE-LSTM agent reached good self-play performance (up maximum mean reward of 84.4; top row), it delivers few to no soups in others. Poorly performing layouts often feature narrow corridors or large object distances, suggesting that agents fail in environments requiring
11

---

Published in Transactions on Machine Learning Research (09/2023)
Best
84.4
79.4
71.0
56.8
Worst
9.4
9.0
6.4
0.0
Fewer visits
More visits
Figure 11: Sample levels in which our models perform best and worst. The number of visits to each grid cell is shown as a heatmap overlay, while the mean return is below. The worst layouts tend to feature narrow corridors or large distances between items.
fine-grained coordination and collision avoidance. We previously identified key failure mode: (1) agents fail to learn spatially invariant coordination strategies. This experiment reveals second: (2) performance degrades significantly in more complex spatial layouts, particularly those that demand structured movement and proximity-based interaction. These insights underline the difficulty of OGC and the need for curriculum strategies that better expose agents to high-complexity, high-coordination scenarios during training.
6 Discussion
Our results suggest that generalising to both novel partners and novel environments remains fundamental challenge in cooperative reinforcement. learning. While e.g. Tha et al. (2025) explored cross-environment play, a preliminary analysis showed that their agents also do not generalise to entirely novel layouts hatps/paatrrr..ree To show this, we retrained their agents and tested them on the 32 evaluation layouts discussed Experiment 2. When evaluated on the 32 transformed layouts from Experiment 2, their method only achieved an average return of 0.46 + 2.6, performing worse than the proposed UED-based approaches (e.g., PATRED with SoftMoE-LSTM), These agents appear to learn brittle stratogics tied to the five original Overcooked layouts, and are unable to adapl to structural variations unfamiliar partners.
Our findings also have direct implications for future research on UED and DCD: While previous studies, such as Jiang et al. (2021a), found that PLR outperformed other DCD methods in navigation- tasks, we show that this result does not generalise to more complex, multi-agent. cooperative envirouments. Tn our experiments, PAIRED consistently outperformed PLR and ACCEL. We attribute this to the increased design space complexity offered by the OGC: The environment contains multiple object types, coordination bottlenecks, and sparse rewards, all of which demand deliberate environment construction. In simpler domains like mazes or locomotion tasks, randomly generated or lightly curated curricula may suffice. Tn OvercookedAl, however, this approach fails to expose agents to the kinds of structured coordination tasks they must solve at test time.
This leads three key conclusions: First, DCD methods must scale with environment complenity, Benchmarks that rely on narrow design spaces (e.g., only walls in maze) are insufficient for evaluating the capabilities of currieulum-earning algorithms. The OGC reveals that without principled curriculum generation, agents may never encounter useful learning signals. Second, Current DCD methods do not scale natively to realistic cooperative tasks. Even with tuned architectures and training regimes, we observed poor generalisation across environments and partners. This highlights the importance of new met that integrate environmental and partner generalisation. Third, the OGC provides a critical testbed for advancing this next generation of methods. By supporting both axes of generalisation, the OGC offers foundation for future research into UED-ZSC methods capable of producing robust, general-purpose cooperative agents that perform well in open-ended multi-agent settings.
12

---

Published in Transactions on Machine Learning Research (09/2025)
7 Limitations & Future Work
Despite its advantages, we also identified two limitations of the OGC: First, to support parallel training and JAX-based acceleration, we constrain all layouts to fixed maximum height and width. While we included a partial observation that theoretically be computed independently of size, similar to the vector-based observation used for behaviour cloning agents in (Carroll et al. 2019), batching across layouts in OvercookedUUED stiill requires the layouts to be scaled to the same height and width. Future work could explore layout representations that scale more naturally, such as graph-structured inputs or object-centric embeddings, to remove these spatial limitations.
Second, while OGC evaluates coordination under environmental and partner variation, does not explicitly test agents' ability to reason about their partners' beliefs, intentions, or mental models. Such capabilities -often studied under theory-of-mind or agent modelling frameworks (Rabinowitz et al., 2018 Bard et al,,
2020 Gandhi et al. 2021 Bara et al. 2023 Bortoletto et al. 2024b are likely to be important for achieving robust zero-shot human-AI collaboration. Future work could explore reasoning about other agents in previously unexplored environments.
Finally, while our experiments focus on DCD methods, OGC is compatible with a wide range of generalisation approaches. Future work could for instance explore few-shot adaption partners and environments (Bauer et al. 2023 explore representation learning approaches for better generalisation. We hope that the benchmark provide will inspire and support further work in these areas.
8 Conclusion
We introduced the Overcooked Generalisation Challenge (OGC), the first open-source benchmark for evaluating cooperative multi-agent reinforcement learning (MARL) agents on both environment and partner generalisation. Built on Overcooked-AN and integrated with dual curriculum design (DCD) methods, OGC enables procedural training and rigorous testing in complex, multi-object environments. Compared to prior UED benchmarks, OGC presents significantly larger and more structured design space, exposing key limitations in existing environment generation and coordination strategies. Through extensive experiments, we demonstrated that even state-of-the-ar DCD algorithms struggle to train agents that generalise across layouts and partners. These findings position OGC as a diagnostic tool for probing the frontiers of gen- cralisable cooperation. Beyond DCD research, OGC also provides infrastructure for evaluating human-AI interaction through ad-hoc leamwork and zero-shol coordination. We hope that OGC will catalyse the development of new learning methods what we denote UED-ZSC algorithms that jointly address the challenges of task and partner diversity in open-ended multi-agent scttings.
Broader Impact Statement
This work introduces a benchmark for studying generalisation in cooperative multi-agent learning. While fundamental, may inform future systems that support human-AT collaboration in domains such as assis- tive robotics or simulation-based training. Poor generalisation such settings could lead to coordination breakdowns deployed without safeguards.
Acknowledgments
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting C. Ruhdorfer. A. Penzkofer was funded by the Deutsche Fosschngggenoisccaatt (DFG, German Research Foundation) under Germany's Excellence Strategy EXC 2075 390740016.
References
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/ 1117.66077006 2016.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Cristian-Pail Bara, Ziqiao Ma, Yingzhuo Yu, Julie Shah, and Joyce Chai. Towards collaborative plan acquisition through theory of mind modeling in situated dialogue. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 2958 2966, 2023.
Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, Н. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, lain Dunning, Shibl Mourad, Hugo Larochelle, Mare G. Bellemare, and Michael Bowling. The hanabi challenge: A new frontier for ai re- scarch. Artificial Intelligence, 280:103216, 2020. ISSN 0004-3702. doi: 2019.103216.
Jakob Baner, Kate Baumli, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Thompsn, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nico- las Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroocker, Satinder Singh, Jakub Syg- nowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei M Zhang. an open-ended task space. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd,, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Leorning Research, pp. 1887-1935. PMLR, 23-29 Jul 2023.
adaptation in
Michael Beukmann Samuel Coward, Michael T. Matthews, Mattie Fellows, Minqi Jiang, Michael D. Dennis, and Jakob Nicolaus Focrster. Refining minimax regret for unsupervised environment design. In Forty- first International Conference on Machine Leurning, ICML 2024, Vienna, Austria, July 21-27, 2024. OeenReview.net, 2024. URL htttps://openreviaart wtt.:aaete..toooommdddaaaaaaalaa
Matteo Bortoletto, Constantin Ruhdorfer, Adnen Abdessaied, Lei Shi, and Andreas Bulling. Limits of theory of mind modelling in dialogue-based collaborative plan acquisition. In Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1-16, 2024a.
Matteo Bortoletta, Lei Shi, and Andreas Bulling. Neural reasoning about agents' goals, preferences, and actions. Proceedings of the AAAI Conference on Artificial Intelligence, 38(1):456 464, Mar. 2024b. doi: 10.1609/anai, www.a.........0000011
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman Milne, and Qiao Zhang. JAX: compos- able transformations of
programs, 2018. URL http:://ithh// com/ goom/googee/ja
Micah Carroll, Rohin Shah, Mark Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d' A'Aché- E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Rohan Choudhury, Gokul Swamy, Dylan HadfelldMMnnel, and Anca D. Dragan. On the utility of model learning in hri. In Proceedings of the 14th ACM/IEBS International Conference on Human- Robot Inter- action, HRI '19, pp. 317-325. IEEE Press, 2020. ISBN 9781538685556.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, PMLR, 09-15 Jun 2019.
pp.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learn- vng, ICML '00 JMLR.org, 2020.
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R. McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. Open Problems in Cooperative AI, December 2020.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and wero-shot transfer via unsupervised environment. design. Tn Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Lin Ding, Yong Tang, Tao Wang. Tianle Xie, Peihao Huang, and Bingsan Yang. A Cooperative Decision- Making Approach Based on Soar Cognitive Archilecture for Mulli-Unmanned Vehicles. Drones, 8(4): 155, April 2024 ISSN 2504-446X. doi: 10.3390 cr000000rone
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Coun- terfactual multi-agent policy gradients. In Proceedings of the Sherty-Second AAAT Conference on Artificial Intelligence and Thirtieth Innovatice Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence,
AAAI Press,
2018. ISBN 978-1-57735-800-8.
Kunihiko Fukushima. Cognitron: A self-organizing multilayered neural network. Biological Cybernetics, 20: 121 126, 1975.
Kanishk Gandhi, Gala Stojnic, Brenden M. Lake, and Moira Dillon. Baby intuitions benchmark (BIB): Discorning the goals, preferences, and actions of others. In Thirty- Fifth Conference on Neural Information Processing Systems, 2021.
Albert Gu, Karan Goel, and Christopher Re Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations, 2022.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL http://github. com/ Con/google/flax
Johannos Hcinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 105-813, Lille, France, 07-09 Jul 2015. PMLR.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735 -0055-00--- 12 1997.
Hengyuan HU, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-play" for zero-shot coordination. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. $399-44100 PMLR, 13 18 Jul 2020.
Nick Jakobi. Evolutionary roboties and the radical envelope-of-noise hypothesis.. Adaptive Behavior, 6(2): 225-368, September 1997. ISSN 1741-2633. doi: 10.1177/1559722970000022
Kunal Jha, Wilka Carvalho, Yancheng Liang, Simon S. Du, Max Kleiman-Weiner, and Natasha Jaques. Cross-cnnironment coopcration enables zcro-shot multi-agent coordination, 2025. URL org/abs/2500 2aabbb//554.111 12714
Minqi Jiang, Michael D Dennis, Jack Parker-Holder, Jakob Nicolaus Foerster, Edward Grefenstetle, and Tim Rocktäschel. heplay-guided adversarial environment design. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Marina Meila. and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4940-4950. PMLR, 18-24 Jul 2021b.
Minqi Jiang, Michael Dennis, Edward Grefenstette, and Tim Rocktäschel. minimax: Efficient basclines for autocurricula in JAX. CoRR, abs/2311.122 12716, 2023. doi: ARI./.231...111111
15

---

Published in Transactions on Machine Learning Research (09/2025)
Rawal Khirodkar, Donghyun Yoo, and Kris M. Kitani. Adversarial domain randomization. hbss/181..004911 2018.
CoRR,
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Wenjun Li, Pradeep Varakantham, and Dexun Li. Generalization through diversity: improving unsupervised environment design. In Proceedings of the Thirty-Secona International Joint Conference on Artificial Intelligence, IJCAT '23, 2023a. ISBN 978-1-956792--3-44 doi: 224963/jcaai.2022
Yang Li, Shao Zhang, Jichen Sun, Yali Du, Ying Wen, Xinbing Wang, and Wei Pan. Cooperative open-ended learning framework for zero-shot coordination. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 20470- 20484. PMLR, 23 29 Jul 2023b.
Jijia Liu, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang, Lim alm-powered hierarchical language agent for real-time human-ai coordination. In Mchdi Dastani, Jaime Simão Sichman, Natasha Alechina, and Virginia Dignum (eds.), Proceedings of the 23rd International Conference on Aulonomous Agents and Multiagent Systems, AAMAS 2024, Auckland, New Zealand, May 6-10, 2024, pp. 1219-1228. ACM, 2024. doi: 10.5555 00555/3336637336622222
Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiv environments. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning, In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advunces in Newral Information Processing Systems, volume 36, pp. 47016-44644 47001. Curran Associates, Inc., 2023.
Michael T. Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Thomas Jackson, Samuel Coward, and Jakob Nicolaus Focrster. Craftax: A lightning-fast benchmark for open-ended re- inforcement learning. In Forty-first International Conference Machine Learning, ICML 2024, Vi- enma, Austria, July 21-27, 2024. OpenReview.nee, 2024. URL, hg4wXirqcV
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Pro- ceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, pp. 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077.
Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Viacheslav Sinii, Artem Agarkov, and Sergey Kolesnikov. Xnand-ninig:d:: Scalable meta-reiinorceemnt learning environments JAX. In Intrinsicnlly- Motivated and
Learning Workshop @NeurIPS2023, 2023.
Johan Samir Obando-Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep RL. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview .ne, 2024. URL httpe://openreviee..eeee wwwp:arrrreeeeeeeeoouuuuiiiiee X9VMnfFxwml
Frans A. Olichock and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer International Publishing, 2016. ISBN 9783319289298. doi: 10.1007/966-33310--28222
Jack Parker-Holder Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Focrster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In Kamalika. Chaudhuri,
16

---

Published in Transactions on Machine Learning Research (09/2025)
Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 17473 17498. PMLR, 17 23 Jul 2022.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. Machine theory of mind. In Jennifer Dy and Andreas Krause (ods.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4218 -227..
PMLR, 10- 15 Jul 2018.
Alexander Rutherford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, and Jakob N. Fo- orster. No regrets: Investigating and improving regret approximations for curriculum discovery. Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Urrich Paquet, Jakub M. Tom- ezak, and Cheng Zhang (eds.), Advances Neural Information Processing Systems 38: Annual Con- ference on Newrol Information Processing Systems 2024, NewrIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024a. URL http://paperr html adbed11ccfaasf72c44a0eeeerrr Att.:///aee....aaaaaaaaaaa----r-
wttpc/ppppereeeee/ppppeeeeee f:1ess/paper/2224//m
Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ingvarsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Lange, Shimon Whiteson, Bruno Lacerda, Nick Hawes, Tim Rocktäschel, Chris Lu, and Jakob Foerster. Jaxmarl: Multi-agent environments and algorithms in jax. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS '24, pp. 2444 2446, Richland, SC, 2024b. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9798400704864.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. Torr, Jakob Focrster, and Shimon Whiteson. The star- craft multi-agent challenge. In Proceedings of the 18th International Conference on Aulonomous Agents and MultiAgent Systems, AAMAS '19, pp. 2186 2188, Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450363099.
Mikayel Samvelyan, Akbir Khan, Michael D Dennis, Minqi Jiang, Jack Parker-Holder, Jakob Nicolaus Foer- ster, Roberta Railcanu, and Tim Rocktäschel. MAESTRO: Open-ended environment design for multi-agent reinforcement learning. In The Eleventh International Conference on Learning Represenlations, 2023.
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023.
Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. Ad hoc autonomous agent teams: Col- laboration without pro-coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 1504 1504-1509, 2010.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (cds.), Advances in Neural Information Processing Systems, volume 34, pp. 14502-14515. Curran Associates, Inc., 2021.
Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, and Phillip Isola. The neural mmo platform for massively multiagent research. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 2021.
Joseph Suarez, Kyoung Whan Choc, David Bloomin, Hao Xiang Li, Nikhil Pinnaparaju, Nishaanth Kanna, Daniel Scott, Ryan Sullivan, Rose Shuman, Lucas de Alcantara, Herbie Bradley, Chenghui Yu, Yuhao Jiang, Qimai Li, Jiaxin Chen, Xiaolong Zhu, Louis Castricato, and Phillip Isola. Neural mmo 2.0: A massively multi-task addition to massively multi-agent lcarning. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 50094-500 50104. Curran Associates, Inc., 2023.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In The Tw Twelfth International Conference on Learning Representations, 2024.
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina. Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michaöl Mathicu, Nat McAlcese, Nathalic Bradley- Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Pitt, Valentin Dalibard, and Wojciech Marian Czarnecki. Open-ended learning leads to generally capable agents, 2021.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, L. Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyun Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575:350 354, 2019.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. CoRR, abs/19001 abs/1901.01753, 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeff Clune, and Kenneth O. Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning, 2020.
Xihuai Wang, Shao Zhang, Wenhao Zhang, Wentao Dong, Jingxiao Chen, Ying Wen, and Weinan Zhang. Zsc-eval: An evaluation toolkit and benchmark for multi-agent zero-shot coordination. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Urrich Paquet, Jakub M. Tom- ezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Con- ference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL lnttp://papers.. nips. wwwpsicpppppp.......ee. wcppaperrpppppepeeppeeeeee.... htcp//papr___eeeee_eeeeeeaee files/papers pateee//aprr/222////2/ 2024/hash/ 4ar119654468822200aaaaaaa baaabbeebb-bsstracc-aaaatatass _nd Benchmarks Track.html
Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang, haiyin piao, and Hankz Hankui Zhuo Coordinated proximal policy optimization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Xue Yan, Jiaxian Guo, Xingzhou Lou, Jun Wang, Haifeng Zhang, and Yali Du An efficient end-to-end training approach for zero-shot human-ai coordination. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. Curran Associates, Inc., 2023a.
2636 2658.
Xue Yan, Jiaxian Guo, Xingzhou Lou, Jun Wang, Haiseng Zhang, and Yali Du. An efficient end-to-end train- ing approach for zero-shot human-ai coordination. Advances in Neural Information Processing Systems, 36:2636-2658, 2023b.
Mesut Yang, Micah Carroll, and Anca D. Dragan. Optimal behavior prior: Data-cfficient human models for improved human-ai collaboration. CoRR, //s/22111001600 2022. doi: 10.48550/ 144555/AAXIVV ARXIV.2222 22222.2211000
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and YI WU. The surprising effectiveness of ppo in cooperative multi-agent games. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24611 24624. Curran Associates, Inc., 2022.
Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning zero- shot cooperation with humans, assuming humans are biased. In The Eleventh International Conference on Learning Representations, 2023.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. CoRR, 1bs/1806.07937, 2018a.
Chiyuan Zhang, Oriol Vinyals, Rémi Munos, and Samy Bengio. sludy on overfiuting in deep reinforcement learning. CoRR, abs/ has/1804.06893, 2018b.
Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population-base appse training for zero-shot human-ai coordination. Proceedings of the AAAI Conference on Artificial Intelligence, 37(5):6145-6153. Jun. 2023. doi: 10.1609 aaai x3755.2577..
A Appendix
A.1 Accessibility of the Benchmark
We make our challenge available under the Apache License 2.0 via a code repository: https://git. hcics. simtech. uni-stuttgart. httppaatreddeppllcppojooeet////e
Our environment is built on top of the ex- isting minimax project (accessible under Apache License 2.0 via /ttps://gittub. http:///ggtaab..ccc//aaoooreeee cct./aacoooookeeercccll minimax) and is thus accossible to rescarchers who already familiar with the project. minimax is exten- sively documented, fast, and supports multi-device training. For all details, including full description of the advantages of minimax, we kindly refer the reader to the accompanying publication (Jiang et al. 2023). Our Overcooked adaption is extended from the one in JaxMARL also accessible under Apache License 2.0 via https:///ithuu: www////AAOO/Iaa///...... JaxMARL, Our codc includes extensive documentation and examples of how it may be used. Additionally, our code written a modular fashion and other multi-agent environments can be integrated with the runners.
A.2 Infrastructure & Tools
We ran our experiments on a server running Ubuntu 22.04, equipped with NVIDIA Tesla V100-SXM2 GPUs with 32GB of memory and Intel Xcon Platinum 8260 CPUs. All training runs are exocuted on a single GPU only. We trained our models using Jax (Bradbury et al.) 2018) and Flax (Heek et. al. 2023) with 1, 2, 3, 4 and 5 as random seed for training DCD methods and to as random seeds for the populations. Training the DCD methods usually finishes in under 24 hours, only SoftMoE and PAIRED-based methods take longer. SoftMoE-basse policies often take an extra 50% wall-clock time to train. Noticeable is also that our S5 implementaioon is the fastest, usually needing roughly 30% less time. Both are compared to the default architectures' training time. In the longest case, the combination a SoftMoE-LSTM policy trained with PAIRED takes about 80 hours to complete training. Our benchmark should be runnable on any system that features single CUDA-compatible GPU. Although in our experience our experiments will require 32GB VRAM to run.
A.3 Extended Related Work
We present an overview over how our environment compares to other UED environments in Table 3
A.4 Hyperparameters
We overview all hyperparameters for training Table and provide details on the hyperparameter scarch used in Table 5 This search was conducted on smaller single layout runs to determine reasonable values complete runs would have been computationally infeasible. Furthermore
show the hyperparameters for each DCD method separately: DR hyperparameters in Table 6 PLR hyperparameters in Table 7| ACCEL hyperparameters in [Table 8] and PAIRED hyperparameters in Table 9 DR hyperparameters govern how Overcooked levels are generated randomly and apply to all of her processes in which a random level is sampled, for instance, in PLR, in which case the same hyperparameters apply.
In the experiment displayed in Figure we show how policies behave when trained on all versus on only one layout and then decide the SoftMoR architecture for our agents. Since these considerably easier
19

---

Published in Transactions on Machine Learning Research (09/2025)
Table 3: Overview of benchmarks for unsupervised environment design and procedurally generated envi- ronments. Closed-source benchmarks are marked in gray these cannot be evaluated on by the rescarch community.
Name
Multi- agent
Zero- shot
GPU accel- erated
Open
Partial
Img.
Source
obs.
obs.
coop.
XLand
Towm
po21
Bauer
2023
Laser lag
Bamvelyan
al.
2023
MultiCar Cacing
Sammelyan at sl.
2025
CoinRun Cobbe
ProcGen Cobbe
2D Mazes Cobbe
20.99
Dennie
2020
Car CacRnging
Jinnn ot ml
2021a
Bipedal Walker
Wang et al.
2019
AMaze
Jaang at
2023
XLand-M
MiniGrid
(Nikaltn et
Craflax
JaxNav Rutheeroord
2024n
OvercookedUED (ours)
Table 4: Hyperparamters of the learning process.
Description
Value
Optimizer
Adam Kingma & Ba
2015
Adam $1
0.9
Adamn B2
0.999
Adam
1 - 10
Learning Rate 7)
3 10-4
Learning Rate Anncaling
Max Grad Norm
0.5
Discount Rate Y
0.999
GAE X
0.98
Entropy Coefficient
0.01
Value Loss Coefficient
0.5
#
PPO Epochs
8
# PPO Minibatches
4
# PPO Steps
400
PPO Value Loss
Clipped
PPO Value Loss Clip Value
0.2
Reward Shaping
Yes (linearly decreased over training)
problems train these models on fewer environment steps in total. We train the overfitting baseline for roughly 1/30 (12, 800, 000 steps in the environment) of the experience of all UED methods and only use 1, 000 outer loops. For the Oracle baseline, we use 1/2 of the experience (200, 000, 000 steps in the environment) and only 5, 500 outer loops. Both are trained until convergence, and to speed up training, we deploy 100 environment simulators. Recall that the UED methods use 30, 000 training loops and 400, 000, 000 steps in the environment. Notably, in the case of population PAIRED these steps apply per student and do not inchude any additional steps taken in the teacher environment.
20

---

Published in Transactions on Machine Learning Research (09/2025)
Table 5: Values used for a grid search over hyperparameters governing the learning process. Finally used values appear in bold.
Description
Value
Learning Rate
[1 10 3 10 4, 5. 10
1- 1013
Entropy Coefficient
0.001 0.1]
# PPO Steps
[566, 400
# Hidden Layers
[2. 3, 4
Reward Shaping Annealing Steps
[0, 2500000, 5000000, until end]
Table 6: DR hyperparameters.
Description
Value
n walls to place
Sampled between 0 15
n onion piles to place
Sampled uniformly between 1 2
plate piles to place
Sampled uniformly between 1 2
n pots to place
Sampled uniformly between 1 --2 2
n goals to place
Sampled uniformly between 1 2
A.5 Neural Network Architectures
This work employs an actor-critic architecture using a separate actor and critic in which the critic is cen- tralised for training via MAPPO (Yu et al. 2022).. For the actor, the observations are of shape x w x 26, while for the centralised critic, we concatenate the observations along the last axis form a centralised observation, i.e. the centralised observation has shape h w x 52 following prior work Yuu et al. 2023
All our networks feature a convolutional encoder fer This encoder always features three 2D convolutions of 32, 64 and 32 channels with kernel size 3 x 3 each and pads the imput with zeros. Our default activation function is ReLL (Fukushima 1975 Nair & Hinton 2010)) which we apply after every convolutional block. We feod the output of fc to a feed-forward neural network fe with three layers with 64 neurons, ReLU and LayerNorm (Ba et al.] 2016)) applied each. Se takes the flattened representation produced by La and produces an embedding
oxxrx64 that we feed into a recurrent neural network (either LSTM (Hochreiter & Schmidhuber 1997) $5 (Smith et al.) 2023)) to aggregate information along the temporal axis. We use this resulting embedding e, R6x04 to produce action logits I R to parameterise categorical distribution in the actor-network or directly produce value v E R in the critic network using a final projection layer. This architecture is inspired by previous work on Overcooked-Al, specifically {Yu et al. 2023), see Figure 12 for an overview. We also test the use of a S5 layer (Smith et al. 2023) in which case we use 2 S5 blocks, $5 layers, use LayerNorm before the SSM blook and the activation function described in the original work, ie. a(x) GELU(x) o oW GELU(x)). In the case of the SoftMoE architecture, we follow the same approach as in (Obando- Ceron et al., 2024) and replace the penultimate layer with a SoftMoE layer. As in their work use the PerConv tokenisation technique, i.e. given input e x we take the output E R2x of fa and construct h x tokens with dimension d= =d=30 32 that we then feed into the SoftMoE layer. We always use 32 slots and 4 experts for this layer, see Obaado-Ceeoo et al. 2024 for details on this layer. The resulting embedding is then passed into the two remaining linear layers before being also passed to RNN and used to produce an action or value, equivalent to the description above, also compare Figure
Lastly, we describe our networks in terms of parameter count in Table 10
21

---

Published in Transactions on Machine Learning Research (09/2025)
Table 7: PLR specific hyperparameters in addition to the DR hyperparameters.
Description
Value
UED Score
MaxMC (Jiang al.
2021a
PLR replay probability P
0.5
PLR buffer size
4, 000
PLR staleness coefficient
0.3
PLR temperature
0.1
PLR score ranks
Yes
PLR minimum fill ratio
0.5
PLR
Yes
PLR
Yes
PLR force unique level
Yes
Table 8: ACCEL hyperparameters in addition to the DR hyperparameters.
Description
Value
UED Score
MaxMC (Jiang et al.
2021a
PLR replay probability p
PLR buffer size
4, 000
PLR staleness coefficient
0.3
PLR temperature
0.1
PLR score ranks
Yes
PLR minimum fill ratio
0.5
PLR
Yes
PLR
Yes
PLR force unique level
Yes
ACCEL Mutation
Overcooked Mutator
ACCEL mutations
20
ACCEL subsample size
4
A.6 Additional Analysis
A.6.1
Implementation Details
The OGC is implemented in Jax (Bradbury al. 2018) and integrated into minimax (Jiang et al. 2023) As such, it be tested with all available DCD algorithms present in minimax. To achieve this extend minimax with runners, replay buffers etc. that are compatible with multiple agents. Building on estab- lished library eliminates sources of error and presents users of the challenge with familiar experience, We present the steps-per ser-sossss (SPS) on our setup given varying degrees of parallelism in Table 11 and com- pare it to the GPU-accelerated maze environment minimax includes AMaze. Given sufficiently large numbers of parallel environments, OGC can be run at hundreds of thousands of SPS. While less than AMaze, the OGC is more fully-featured environment in which multiple agents take steps and interact.
A.6.2 Performance Across Levels
To accompany the overall performance measured by reward in the main paper in Table we also measure the mean solved rate in Table 12
A.6.3
Performance on Individual Levels
We list the performance of every individual method on every single layout in Table 13
Most notable is that
some layouts are harder to learn than others. Our agents especially seem to struggle with layouts requiring
22

---

Published in Transactions on Machine Learning Research (09/2025)
Table 9: PAIRED hyperparameters. All PPO hyperparameters are the same between the student and the
teacher. The minimax implementation follows original one in (Dennis al. 2020 and we stick to too.
Description
Value
walls to place
Sampled between 0 15
students
UED Score
Relative regret
Dennis et al. 2020)
UED first wall sets budget
Yes
UED noise dim
50
PATRED Creator
OvercookodUE.
Flatten
Linear + ReLU + LN
Linear + ReLU + LN
Linear + ReLU LN
Action
LSTM/ S5
Encoder
Value
x 64
x 64 64 x 64
64 x 64
64 x or
Figure 12: Default architecture featuring a convolutional encoder and an RNN.
complex forms of interaction, i.c. Coordination Ring, Counter Circuit and Forced Coordination. Forced Coordination especially seems difficull to solve as no run achieves noticeable performance it. This might be due to the specific features of the layout, i.e. agents have access to several objects and need to hand them over the counter to produce any result.
A.6.4 Population Training Details
Both populations were trained over random seeds. As architecture, used simple CNN encoder without RNN as in prior work Zhao et al. (2023); ; Yu et al. (022).. To give an intuition into the performance of the members of the population, we present the training curves over all 8 seeds of training an FCP population in Figure 13] MEP was trained with the same architecture, with the same amount experience per agent and achieved practically identical results. As in prior work (Zhao et al. 2023 we set the population entropy coefficient during training a 0.01.
A.6.5 Detailed Results with Populations
We present detailed zero-shot cooperation results per layout in Table 14 and 15 As indicated through the averaged performance discussed the main text, we also find that PAIRED performs best on four of the five individual layouts in terms of zero-shot cooperation.
A.7 Training Curves and Evaluation
In Figure 15 Figure 16 and Figure 17 we show the returns of our agent during training in seen training levels, as well as the five unseen evaluation levels. The results for the SoftMoF. architecture are displayed Figure 15. the results for the S5 Figure 16 and the results for the CNN-LSTM in Figure 17 Interestingly, while (SoftMoE) PAIRED performs the best in our evaluations it does reach the highest training returns, instead it achieves the highest training return, while keeping the generalisation gap small.
23

---

Published in Transactions on Machine Learning Research (09/2025)
Table 10: Number of trainable parameters in each model.
CNN-LSTM
SoftMoE-LSTM CNN-S5
Parameter Count
197,254
316,102
193,670
Table 11:
Average
for diffcrent numbers of parallel environments mcasured by taking 1,000
steps with randomly sampled actions to show how our adapted Overcooked environment performs compared simpler single-agent UED environment.
# Parallel Eaaus
32
256
1024
4096
16384
AMaze
264
8, 141
67, 282
264, 142
058,306 306
321,, 678 2, 2,017, 526
O OveDcookedUED
151
4, 921
40, 011
156, 696
631, 836
Table 12: Mcan Solve rate averaged over the respective testing layouts. The best result is shown in bold. We report aggregate statistics over five random seeds. As a baseline we include an Oracle version for all architectures, which was trained on the five testing layouts directly.
Method
CNN-LSTM
SoftMoE-LSTM
CNN-S5
DR
4.81 + 13.3%
3.78 9.8%
0.00 0.0%
PLR 1.l
0.03 = 0.0%
3.68 9.9%
0.00
Pop. PAIRED
0.08 + 0.2%
14.72 + 12.2%
0.00 + 0.0%
ACCEL
0.00 + 0.0%
0.04 0.1%
0.00 0.0%
Oracle
97.2 + 7.7%
99.7 0.6%
97.2 + 2.8%
24

---

Published in Transactions on Machine Learning Research (09/202))
Table 13: Performance on all evaluation layouts. We show the mean episode reward R and the mean episode solved rate SR. The overall best result per layout is presented in bold, excluding oracle results.
Layout
Method
CNN-LSTM
SoftMoE-LSTM
CNN-S5
R
SR
SR
R
SR
DR
0.02
0.0%
0.01
0.0%
0.00
0.0%
PLR
0.00
0.0%
0.16
2.1%
0.01
0.0%
Cramped
Pop. PAIRED
0.78
0.0%
8.92
9.0%
0.00
0.0%
ACCEL
1.16
2.7%
1.52
3.4%
0.28
2.1%
Oracle
197.56
100.0%
201.56
100.0%
129.64
94.0%
DR
4.98
0.0%
13.2
0.0%
0.00
0.0%
PLR-II
0.09
0.0%
0.44
6.6%
0.06
0.0%
Coord
Pop. PAIRED
0.4
0.0%
28.76
18.0%
0.1
0.0%
ACCEL
0.00
0.0%
0.28
0.0%
0.00
0.0%
Oracle
278.56
100.0%
293.00
100.0%
252.6
100.0%
DR
1.78
0.0%
3.48
0.0%
0.00
0.0%
PLR
0.00
0.0%
0.50
0.5%
0.00
0.0%
Forced
Pop. PAIRED
0.04
0.0%
2.64
6.0%
0.00
0.0%
ACCEL
0.00
0.0%
0.04
0.0
0.00
0.0 %
Oracle
194.12
100.0%
204.96
100.0%
146.69
96.0%
DR
7.32
0.1%
3.38
4.4%
0.32
0.0%
PLR-II
0.82
0.0%
3.29
11.2%
0.62
0.1%
Asymm
Pop. PAIRED
1.68
0.6%
33.42
40.4%
0.10
0.0%
ACCEL
0.04
0.0%
1.20
0.0
0.00
0.0 %
Oracle
238.68
100.0%
242.40
98.4%
206.36
99.7%
DR
0.11
0.0%
0.96
0.0%
0.00
0.0%
PLR-II
0.00
0.0%
0.00
0.0%
0.00
0.0%
Counter
Pop. PAIRED
0.00
0.0%
1.12
0.0%
0.00
0.0%
ACCEL
0.00
0.0%
0.00
0.0%
0.00
0.0%
Oracle
113.28
86.0%
142.00
100.0%
95.60
96.0%
25

---

Published in Transactions on Machine Learning Research (09/2025)
200
Mean Reward
Mean Reward
100
100
0
2
4
Steps in Environment
1e7
Steps Environment
1e7
(a) Coordination Ring (6x9)
(b) Cramped Room (6x9)
150
Mean Reward
Mean Reward
200
100
100
50
2
3
Steps Environment
1e7
Steps in Environment
1e7
(c) Forced Coordination (6x9)
(d) Asymmetric Advantages (6x9)
Mean Reward
100
2
Steps in Environment
1e7
(e) Counter Circuit (6x9)
Figure 13: Runs used for the FCP evaluation populations with random seeds 8 for the OGC with bands
reporting standard error a/ Layouts were padded to a total size of 6 x 9 to be compatible with the policies trained via DCD.
26

---

Published in Transactions on Machine Learning Research (09/2025)
Mean Episode Reward Mean Episode Reward
Partner Population MEP
100
Random
Stay
DR
PLR
Partner Population FCP
PAIRED
ACCEL
ORACLE
100
Coordination Ring Forced Coordination
Counter Circuit Asymmetric Adv.
Cramped Room
Figure 14: Ad-hoe teamwork results of the SoftMoF-LSTM policy paired with an FCP and MEP population
trained on the respective layout. Error bars show standard error.
Table 14: Zero-shot results using SoftMoE--SSTM policies playing with an FCP and MEP population of experts trained the respective layout exclusively. We report the mean episode reward and standard deviation. The best result per layout is put in bold.
Method
Asymm
Counter
Cramped
Forced
Coord
FCP
Random
7.43 + 12.19
8.89 = 4.65
66.02 + 38.28
1.95 1.92
20.49 7.82
Stay
5.32 + 12.07
0.38 = 1.11
20.67 33.05
0.00 0.00
0.95 2.73
Oracle
126.44 + 27.13
22.63 = 7.82
120.9 10.86
22.08 = 12.89
59.64 22.17
DR
19.47 8.79
9.02 = 4.67
59.62 10.49
1.32 0 0.60
21.63 11.01
PLR-.I
8.75 + 1.99
2.13
60.55 5.03
+.75
20.28 2.41
Pop. PAIRED
45.71 14.65
11.72 + 2.49
72.2 + 13.86
2.46 + 1.05
23.99 - 4.94
ACCELI
11.30 + 6.37
10.03 = 2.43
59.83 + 4.43
2.46 = 1.05
18.29 + 1.69
MEP
Random
8.0 + 9.12
22.46 + 13.34
+
2.55 + 2.76
31.85 + 19.69
Stay
4.86 + 7.21
5.2 10.85
31.55
0.0 = 0.0
1.53 - 3.61
Oracle
135.07 + 30.27
39.33
13.53
138.07 + 10.0
56.1 25.41
67.86 - 10.89
DR
21.18 + 10.27
18.93 = 4.26
56.63 + 10.28
1.86 = 0.74
32.19 + 6.23
PLR-II
8.89 + 2.03
20.68 + 1.61
56.66 3.38
2.38 1.13
29.71 2.24
Pop. PAIRED
39.02 = 22.74
20.85 5.72
64.51
2.04
33.29 = 6.11
ACCEL!
12.00 + 6.40
19.06 = 1.73
51.94 5.28
2.42 2.44
28.91 + 1.51
27

---

Published in Transactions on Machine Learning Research (09/202)
Table 15: Zero-shot results using SoftMoE-LSTM policies playing with FCP and MEP population of experts trained on the respective layout exclusively. We report the mean solved rate and standard deviation. The best result per layout is put in bold.
Method
Asymm
Counter
Cramped
Forced
Coord
FCP
Random
8.52 + 17.52%
5.00 + 6.70%
69.43 + 38.45%
0.00 + 0.00%
30.89 +: 3.83%
Stay
6.81 + 18.04%
0.02 = 0.14%
2
0.00 0.00%
1.14:0.749
Oracle
69.67 = 16.39%
27.39 = 19.02%
31.30 20.97% 20.97%
92.02 1.19%
96.96 = 2.23%
DR
25.99 + 13.37%
6.49 = 4.65%
67.85 8.57%
0.03 = 0.05%
30.26 21.34%
PLR
10.95 + 3.10%
3.82 + 2.18%
68.11 1.83%
0.07 + 0.05%
25.99 + 7.39%
Pop. PAIRED
60.18 + 18.29%
9.08 + 4.51%
77.71 - 83.7%
0.21 + 0.30%
35.16 + 11.97%
ACCEL
13.84 + 10.17%
7.11 + 3.77%
67.2 + 2.48%
0.31 + 0.68%
21.33 = 3.81%
MEP
Random
9.25 + 2.02%
36.04 + 4.38%
67.75 5.48%
0.00 0.00%
54.9 5.55%
Stay
4.91 + 1.46%
5.85 2.71%
29.56 5.92%
0.00 + 0.00%
1.02 0.51%
Oracle
91.02 + 1.12%
52.60 11.37%
96.86 + 2.27%
56.16 + 21.25%
75.23 + 0.91%
DR
29.51 + 17.44%
28.18 7.52%
65.81 8.62%
0.10 0.09%
51.61 6.33%
PLR IIIl
10.26 + 3.43%
31.81 + 4.53%
65.71 3.43%
0.26 = 0.32%
49.97
Pop. PAIRED
50.39 + 29.73%
30.66 + 12.23%
74.21 = 7.05%
0.29 + 0.49%
45.16 - 17.49%
ACCEL
14.16 + 10.53%
27.36 + 4.51%
63.61 + 4.46%
0.83 1.81%
49.71 + 2.32%
28

---

Published in Transactions on Machine Learning Research (09/2025)
Method PAIRED
100
50
0
150
Retum
100
50
0
Method PLR
200
150
100
Method ACCEL
200
150
100
0
5000
10000
15000
20000
25000
30000
Foreed Coordination
Cramped Room
Coordination Ring
Asymmetric Adv
Counter Circuit
Train Return
Figure 15: Returns training and evaluation levels over the duration of training for our SoftMoE archi- tecture.
29

---

Published in Transactions on Machine Learning Research (09/2025)
Method PAIRED
150
125
100
50
0
Method
150
125
100
Retur
50
0
Method PLR
150
125
100
J.M
25
Method ACCEL
150
125
100
75
25
5000
10000
15000
20000
25000
30000
Foreed Coordination
Cramped Room
Coordination Ring
Asymmetric Adv
Counter Circuit
Train Retur
Figure 16: Returns in training and evaluation levels over the duration of training for our S5 architecture.
30

---

Published in Transactions on Machine Learning Research (09/2025)
Method PAIRED
150
125
100
25
0
150
125
Retum
50
25
0
Method PLR
150
125
100
yaur pwam W
75
60
25
Method ACCEL
150
125
100
75
0
0
5000
10000
15000
20000
25000
30000
Forced Coordination
Cramped Room
Coordination Ring
Asymmetric Adv
Counter Circuit
Train Return
Figure 17: Returns in training and evaluation levels over the duration of training for our CNN-LSTM architecture.
31


Reviews:
[{"review_id": "aMp4HN1BZm", "reviewer": "Reviewer_EvB3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths:__\n\n* The motivation for this work, studying out-of-distribution generalization in cooperative settings, is compelling and well-articulated.\n* The GPU-accelerated implementation of the testbed environment using JAX, achieving up to 2M steps per second, makes large-scale experiments feasible. The integration with the minimax framework ensures compatibility with existing DCD methods.\n* The authors provide qualitative failure analysis, providing insights into the limitations of existing approaches.\n\n__Weaknesses:__\n\n* The proposal of the benchmark lacks a theoretical analysis of why environment and partner generalization should be coupled, or what fundamental properties make this problem difficult. \n* The metrics used in evaluating cooperative tasks, such as average return and solvable rates, are not direct and accurate indicators of agent collaboration efficiency and activeness. \n* The five hand-designed layouts for evaluation may not adequately represent the full spectrum of cooperative challenges. \n\n__Questions:__\n\n* This benchmark looks specific for DCD algorithm benchmarking. Could this benchmark be used for other generalization methods in RL, such as meta-learning, domain adaptation, and invariant representation learning?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the Overcooked Generalisation Challenge (OGC), the first benchmark specifically designed to evaluate reinforcement learning agents' ability to perform zero-shot cooperation with novel partners in previously unseen environments. This benchmark requires simultaneous generalization across both environments and partners. The authors also contribute OvercookedUED, a GPU-accelerated, open-source environment built on JAX that integrates with state-of-the-art dual curriculum design (DCD) methods. They provide comprehensive baseline evaluations using current DCD algorithms (PAIRED, PLR, ACCEL, DR) combined with modern neural architectures (CNN-LSTM, SoftMoE-LSTM, CNN-S5). The empirical results reveal that even state-of-the-art methods struggle with both novel environment and partner generalization, highlighting the difficulty of the proposed benchmark and the limitations of current approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors may consider formalizing additional metrics to capture the cooperation efficiency to better examine the generalization with novel partners.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yeamO1MIGO", "reviewer": "Reviewer_vxd9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The benchmark addresses the very important problem of generalization in human-AI collaboration, which is crucial for future real-world approaches. It's technically well executed by building on existing tasks (overcooked) and frameworks (minmax). Also, the framework is built with scalability in mind (highly parallelized JAX implementation) and provides interfaces for different forms of DCD (proposing entirely new levels and modifying a given level). Overall, this results in a highly accessible, useful, and valuable benchmark.\n\nAdditionally, the empirical evaluation shows that the proposed benchmark is indeed beyond the capabilities of the current SOTA and thus relevant. \n\nOne weakness of the evaluation is the number of seeds and how the results are presented. The paper uses 3 seeds and reports the mean and standard deviations. While this will most likely not change the overall finding that SOTA approaches struggle with the challenge, an appropriate number of seeds (>=10) and statistically more robust quantities (e.g., confidence intervals) should be provided. This would also allow future publications to directly “quote” the results from the paper, making the benchmark more accessible and the findings generated with it more significant. \n\nAdditionally, while the work's main focus is establishing the benchmark, not a solution, I would appreciate further insights into why SOTA methods fail. For example, one point I was wondering about is the ratio of solvable vs unsolvable environments generated by the curriculum (in particular, the random one. Intuitively, I would suspect that just placing everything randomly does not give many valid environments.) This could already indicate if the issue is more on the curriculum generation side or actually on the policy learning side?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the “overcooked generalization challenge (OGC)”, a new benchmark for generalizable, cooperative multi-agent RL. Building on overcooked AI the challenge focuses on generalization capabilities to both new environments (i.e., kitchen layouts) and partners. For training, randomly generated environments can be used but the framework provides interfaces for unsupervised environment design and curriculum design. \nThe paper also provides an empirical evaluation of existing SOTA approaches on the new benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major: Run experiments for more seeds and provide statistically robust uncertainty measures\n\nMinor: Provide some additional analysis of why SOTA fails.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JTSf7xdV4r", "reviewer": "Reviewer_AnxK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors present a well-executed and methodologically rigorous contribution that addresses a fundamental gap in cooperative multi-agent evaluation. The benchmark design is particularly strong, as it thoughtfully combines environment and partner generalization into a unified framework that better reflects real-world collaboration scenarios. The experimental methodology is comprehensive and well-justified, featuring careful selection of baseline policies and architectures alongside systematic evaluation across multiple coordination settings. The empirical analysis provides valuable insights that challenge assumptions from simpler domains, demonstrating that results from navigation-based tasks do not transfer to complex cooperative environments. The authors effectively identify specific failure modes and provide clear explanations for why current methods struggle, particularly highlighting agents' inability to learn spatially invariant coordination strategies. Overall, the paper makes a compelling case for the necessity of joint environment-partner generalization and establishes clear directions for future research through the proposed UED-ZSC framework.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents the Overcooked Generalization Challenge (OGC), a benchmark that addresses the critical gap in evaluating cooperative multi-agent reinforcement learning systems regarding joint environment and partner generalization. The work makes three core contributions:\n- Unlike prior work that augments fixed test environments, OGC uses unsupervised environment design (UED) to procedurally generate diverse training layouts and then evaluates agents in entirely novel environments with unknown partners.\n- Development of a JAX-accelerated extension of Overcooked-AI that supports parallelization and integration with state-of-the-art dual curriculum design (DCD) algorithms.\n- A systematic evaluation of multiple DCD algorithms (DR, PLR, PAIRED, ACCEL) and neural architectures (CNN-LSTM, SoftMoE-LSTM, CNN-S5), demonstrating that current state-of-the-art methods largely fail at joint generalization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Suggested changes (non-critical):\n- The paper assumes familiarity with the Overcooked-AI environment but would benefit from a brief description to provide necessary context for readers who are unfamiliar with it and its challenges.\n- The introduction lacks a clear definition of Dual Curriculum Design (DCD). A concise definition should be provided when it is first introduced.\n- Please verify and update citations with proper venue information where applicable, in relation to works cited as CoRR/arXiv preprints.\n\nMinor points (e.g., typos, personal suggestions):\n- Figure 6 caption: \"(middle row, leftmost and rightmost)\" should capitalize \"Middle\" if referring specifically to the \"Middle\" row label.\n- Figure 8: \"Counter Cicuit\" -> \"Counter Circuit\"\n- A.4: \"12,800,00 steps\" -> \"12,800,000 steps\"\n- A.6.2: \"solved rate on display it in Table 12\" is poorly phrased.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)