Partner Modelling Emerges in Recurrent Agents (But Only When It Matters)
Authors: Ruaridh Mon-Williams
Max Taylor-Davies
Elizabeth Mieczkowski
Natalia Vélez
Neil R Bramley
Yanwei Wang
Thomas L. Griffiths
Christopher G. Lucas
Venue: NeurIPS 2025 poster ()
Keywords: multi-agent reinforcement learning
partner modelling
emergent representations
collaboration
Primary area: reinforcement_learning
TL;DR: Recurrent neural networks spontaneously model partners during collaboration — without specialised architectures — but only when partner-specific adaptation improves task performance.

Abstract: Humans are remarkably adept at collaboration, able to infer the strengths and weaknesses of new partners in order to work successfully towards shared goals. To build AI systems with this capability, we must first understand its building blocks: does such flexibility require explicit, dedicated mechanisms for modelling others—or can it emerge spontaneously from the pressures of open-ended cooperative interaction? To investigate this question, we train simple model-free RNN agents to collaborate with a population of diverse partners. Using the 'Overcooked-AI' environment, we collect data from thousands of collaborative teams, and analyse agents' internal hidden states. Despite a lack of additional architectural features, inductive biases, or auxiliary objectives, the agents nevertheless develop structured internal representations of their partners' task abilities, enabling rapid adaptation and generalisation to novel collaborators. We investigated these internal models through probing techniques, and large-scale behavioural analysis. Notably, we find that structured partner modelling emerges when agents can influence partner behaviour by controlling task allocation. Our results show that partner modelling can arise spontaneously in model-free agents—but only under environmental conditions that impose the right kind of social pressure.

Partner Modelling Emerges in Recurrent Agents (But Only When It Matters)
Ruaridh Mon- " Max Taylor-Davies*
Elizabeth Mieczkowski² Natalia Vélez²
Neil R. Bramley'
Yanwei Wang
Thomas L. Griffiths Christopher G. Lucas't
University of Edinburgh
2Princeton University
Massachusetts Institute of Technology
Abstract
Humans are remarkably adept at collaboration, able to infer the strengths and weak- nesses of new partners order to work successfully towards shared goals. To build AI systems with this capability, we must first understand its building blocks: does such flexibility require explicit, dedicated mechanisms for modelling others- or can it emerge spontaneously from the pressures of open-ended cooperative inter- action? To investigats this question, we train simple model-free RNN agents collaborate with a population of diverse partners. Using the 'Overcooked-Ar envi- ronment, we collect data from thousands of collaborative teams, and analyse agents' internal hidden states. Despite a lack of additional architectural features, inductive biases, or auxiliary objectives, the agents nevertheless develop structured internal representations of their partners' task abilities, enabling rapid adaptation and gen- eralisation to novel collaborators. We investigated these internal models through probing techniques, and targe-scale behavioural analysis. Notably, we find that structured partner modelling emerges when agents can influence partner behaviour by controlling task allocation. Our results show that partner modelling arise spontaneously in model-free agents-but but only under environmental conditions that impose the right kind of social pressure.
Introduction
While humans are certainly impressive 'solo' learners and problem-solvers, our capacity for coopera- tion and collaboration is even more remarkable- cenarkablc-eenalinng to achieve goals beyond the reach of any single individual, and leverage the complementary abilities of others while sharing or mitigating the costs of action. In particular, humans display exceptional flexibility in adapting unfamiliar partners and task contexts. Indeed, it could be argued that this capacity for flexible collaboration is one of the primary contributors the success of our species- wethhut it, it is hard ancestors could have developed the culture and civilisation that persist to this day 2 As we develop artificial agents that will operate alongside us, occupying our homes and workplaces, it is crucial that they too can share in this collaborative process 44]
how
One explanation for the powerful flexibility of human collaboration lies in our well-developed "Theory of Mind' (ToM)-our general faculty for inferring and representing the latent mental properties (such goals, beliefs, desires intentions) that drive others' behaviour [5). As with many other social contexts, effective collaboration with diverse partners requires an individual to not only react to the actions of others, but also attempt to predict, and where appropriate to influence them processes which rely on the formation of predictive representations ('mental models') ) of other agents' decision- making 66] g. In collaborative contexts, we
ToM to infer and represent the varying
"Equal contribution. Correspondence to ruaridh. mw@ed. ac. uk or taylor- dasior-ddsesssss ed. ac. uk 1 'Equa senior authorship
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

strengths and weaknesses of different partners 1 10 IT] For example, imagine being assigned to work with unfamiliar classmates on a group project for machine learning class. Effectively dividing up the different tasks will require representations of each contributor's abilities: who will be best suited to implement the code, write the report, and deliver the presentation.
In this paper, we examine whether artificial agents, when trained to collaborate with different partners but without explicit mechanisms for agent modelling, spontaneously develop internal representations of their partners' abilities. We investigate this question in fully cooperative setting, where agents optimise a shared goal (i.e., a single reward function), but have no prior knowledge of each other's attributes or action policies. Crucially, we train reinforcement learning agents with generic recurrent architectures and only task reward supervision -tere are no auxiliary objectives or architectural priors pushing agents to model one another. This stands in contrast to prior work, such as Rabinowitz et al.'s 'Machine Theory of Mind' framework, which relies on specialised components optimised explicitly to infer other agents' internal states [12). We find that despite these minimal inductive biases, agents develop structured, internal representations that (i) encode the different competencies of their partners; (ii) generalise to previously unseen collaborators; and (iii) emerge selectively, depending on agents' ability to control task allocation. Together, our findings suggest that partner modelling can arise within artificial agents solely from the demands of flexible cooperation, without explicit incentives or specialised architectures.
2 Related work
2.1 Ad hoc teamwork
The field of ad hoc teamwork (AHT) deals with the problem of developing agents that learn to collaborate 'on the fly' with previously unseen 'teammates', without any prior coordination [13] AHT shares some basic elements with the field of multi-agent reinforcement learning (MARL); but where MARL typically assumes control of all agents in the environment, in AHT we control only a single agent (often called the 'AHT agent' or 'learner'; we will use 'ego agent' throughout), with teammates' actions governed by either simple heuristics or pre-trained (frozen) RL policies. AHT also considers only settings where all agents share a common cooperative objective-whil individual agents might have additional goals or small differences in reward function, they are never in conflict with one another. The focus of research in AHT has mainly been on producing agents that can adapt to the varying 'play styles' (policies) of different partners or human collaborators [1]. In contrast to self-play training (where agents co-adapt to each other), ad hoc agents must generalise to novel partners under zero shot (no prior interaction) or few shot (minimal adaptation rounds) conditions. AHT can thus be viewed in large part as the problem of rapidly inferring novel teammate's s underlying parameters or characteristics. Accordingly, many approaches have involved explicit inference and representation of these characteristics; traditionally via forms of Bayesian belief-updating over a discrete tcammate space [15] 16 I7 18), or more recently using soarallnetwork-baad encoders to learn latent representations of teammate policies [12] 19 20). In contrast, our work uses the AHT setting to study implicit, emergent partner modelling in simple RNN agents without additional architectural components auxiliary objectives.
2.2 Agent modelling
While many AHT approaches leverage some method for representing different teammates, the problem of modelling other agents in a shared environment is not unqiue to the AHT setting (21]. Recent work on agent modelling has primarily employed deep neural aetwork-based approaches, where a dedicated module is optimised explicitly to produce useful representations of other agents' properties via some auxiliary objective. These representations can then be used to condition the controlled agent's own action policy [21 22). allowing them to adapt their behaviour directly to the properties of the different agents with whom they must interact. For example, He et al. (23 extend the DQN architecture with an additional network that produces representations of the opponent policy. In contrast, Raileanu et al. 24 avoid having to maintain a separate model of other agents by using the learner's s own current policy to infer others' goals via maximum likelihood. Numerous other works have employed some form of encoder-decoder architecture, typically trained via a reconstruction loss
to learn latent embeddings that facilitate behaviour prediction 25 [2 19
26
27 28

---

Beyond these explicit modelling approaches, parallel line of work has examined social influence and coordination in multi-agent RL. For example, Jaques et al. [29] show that giving agents intrinsic motivation to shape others' behaviour improves cooperative outcomes, while work on zero-shot coordination demonstrates that agents trained with diverse partners can adapt to unseen teammates without additional training 300 [3]. These studies highlight the general idea that social prediction
and adaptation are key to successful collaboration.
2.3 Emergent representations in model-free RI
In contrast to the explicit approach common across most agent modelling research, a different line of work has explored the implicit representations that emerge spontaneously in model-free RL agents trained only to achieve particular high-level task. For example, multiple works have investigated the representations that develop within RL. agents trained on simple navigation tasks, finding for example that agents encode target distance, reachability, and progress from their starting location [32], and that structured 'mental maps' the environment emerge in the memories of 'blind' RNN agents 33] Other research has explored the information encoded by model-free puzzle-solving or game-playing agents, isolating goal representations in maze-solving networks [44,, humanlike chess concepts in AlphaZero 35 and planning- like abilities in RNN agents trained to play sokoban [36].
Our work is also closely connected the meta- meta-learnngg literature. Recurrent policies trained across many tasks or partners can act as implicit meta-learners, encoding past experience in hidden states to support rapid adaptation [37] This view frames recurrence as a way for agents to learn about collaborators, not just task features. Our findings extend this perspective by showing that such implicit meta-learning can give rise to eartner-specife internal models under the right collaborative pressures.
2.4 Cognitive and economic perspectives
Work in cognitive science. anthropology, and economics converges on the idea that prediction underpins intelligence and collaboration. Clark (38) argues that perception and action are driven by hierarchical prediction. Byrne and Whiten's 'Machiavellian Intelligence' hypothesis [? J proposes that human intelligence evolved to anticipate others' behaviour. Harsanyi [39 formalised this idea in economics through Bayesian games, where agents reason about hidden partner traits. Grosz and Kraus [40] further emphasise the need for shared predictive structures to coordinate group plans. Our results align with these perspectives, showing that predictive partner models can arise spontaneously simple recurrent agents under collaborative pressure.
2.5 Attention
Recent work uses attention-based architectures to study social reasoning. Long et al. 4 analyse collaboration via attention weights, while Decision Transformer [2] frames RL as sequence mod- elling. In contrast, we use a simple RNN to show that structured partner representations can emerge without strong inductive biases. Future work could apply attention mechanisms to probe whether agents implicitly track partner positions or trajectories.
3 Problem Formulation
As is standard in the AHT literature, we formulate the problem as a two-agent partially observable Markov decision process (POMDP). This is defined by the tuple M (S, O2, O2, A,, P.r, 7) {1,2] (with Ô O x O2, A = A1 x A2). S is the environment state space, P S x A + (SS) denotes the state transition function, r: S x A + IR is the shared reward function, and is the discount factor. At each timestep t the ego agent (agent 1) receives an snapshot of the environment of E O information from which may be retained in future timesteps as part of internal memory state h2 with a recurrent function h4 = f( o!). The ego agent acts according to its learned policy (a, h4) and the partner (agent 2) acts according to its (fixed) pre- pre-trained policy т., governed by latent parameters x' E Z, sampled from distribution p(=") ) at the beginning of each episode. These traits, such as how quickly given partner can perform each task, are not directly observable to the ego agent agent-rather, as in
where O, and A, denote the observation and action spaces of agent
3

---

"work on
+ Counter
Figure I: An illustration of our task setting, based on the 'cramped room' layout of Overcooked-Al The ego agent (green) has learned internal representation of how competent (fast) their partner (blue) is at each the two subtasks; it uses this representation to determine which subtask the partner should work on.
the context of human theory of mind, they must be inferred (explicitly or implicitly) from observed behaviour.
The ego agent is trained only to maximise the expected cumulative reward across episodes, max, E.-~p(=") (xwx,my(=")) Ei where denotes a trajectory of states, observations, and actions sampled from the ego policy and partner policy T2(=") under the transition dynamics P. Each episode involves different partner sampled from a distribution over latent traits =" E Z. Importantly, no explicit architectural mechanisms auxiliary objectives encourage modelling of these latent traits. We are interested in whether, under these minimal conditions, internal partner models nevertheless emerge implicitly within the ego agent's recurrent state f4-
Methods
4.1
Environment
Critical to any test of our hypothesis is that the environment imposes collaborative pressure; i.c. the ego agent's 's optimal policy depends on the latent characteristics of their partner, and so modelling those characteristics conducive achieving high joint reward. We provide this through Overcooked-Ar Al 43). fully cooperative environment where agents work together to prepare soups, tasked with maximising the throughput
Time (for additional results in a second cooperative environment, see Appendix [A). Each agent must navigate a shared kitchen to gather ingredients, cook them, and serve the completed soups making success heavily dependent labour. The environment difficulty can be modified via different recipes, which vary in complexity and number of ingredients, and different kitchen layouts, which introduce specific constraints (c.g. encouraging agents to pass items to perform the task successfully, or forcing agents to navigate around each other in cramped spaces). Figure T iillustrates one such layout.
ASoup
coordination and division of
4.2 Agent Architecture
The ego agent's policy is implemented gated recurrent unit (GRU) 4F recurrent neural network (RNN). At each timestep t, the agent processes an observation o, and updates its hidden state via h4 = GRU (o,, f1.1). The hidden state he acts as a general dynamic memory, evolving as the interaction unfolds. The ego policy is trained using Proximal Policy Optimisation (PPO) [5]. implemented in JAX [46] to facilitate efficient parallel training.

---

4.3 Experimental Design
Achieving high reward in the Overcooked-Al environment requires agents to successfully coordinate two different subtasks: preparing ingredients ('task I I'), and serving soup ("task 2'). We train our ego agent alongside distribution of partners who vary in how competent they are at the two subtasks (operationalised as how frequently they can take actions towards each task). To introduce direct connection between partner properties and optimal ego agent behaviour, we grant the ego agent control over which subtask its partner is working towards at any given time. Our hypothesis is that effective task allocation will require the ego agent to learn how to recognise and represent the abilities of different partners- that is, after training, the RNN hidden state dynamics should be optimised to encode how fast a given partner is at tasks and 2 (as illustrated in Figure D..
Within this framework, we carry out three experiments to probe different dimensions of partner modelling:
4.3.1 Experiment 1: Does the pressure to allocate subtasks drive the emergence of partner modelling?
This experiments tests whether placing the responsibility for subtask allocation to the ego agent encourages it to develop internal representations of its partner's 's capabilities. In particular, we ask whether the ego agent can learn to reliably allocate tasks effectively by representing different partners' ability profiles? To test this, we generate a distribution of partners, each characterised by two-dimensional vector = [e1, 22]] denoting the cooldown interval (i.e, the number of timesteps inverse
between consecutive actions) for each subtask. These cooldowns can be interpreted proxies for subtask skill: a lower 2; reflects higher proficiency in subtask i, allowing the partner to act more frequently. The ego agent is trained alongside population of partners with o; sampled independently from the set {1. 2, 3, 4, 7, 9). During evaluation, the ego agent is paired with partners whose cooldowns are sampled from different set, {0, 2, 3, 8, 10]. While some individual cooldown values overlap (c.g. 2 and 3), the pairs of cooldowns (01, 82) used for training and evaluation are disjoint, ensuring that no test partner configuration was encountered during training. The ego agent has a constant cooldown of 2 for both tasks, allowing for a balance between dominating the tasks (if too fast) and slowing learning (if too slow), and is equipped with additional action that allows it to dictate which subtask the partner contributes to at each timestep.
4.3.2 Experiment 2: Can agents adapt online to new partners within episode?
Successful collaboration in the real world often requires us deal with sudden, unexpected changes. In this experiment, we examine whether our ego agent can adapt dynamically to different partners within single episode. During each training episode, there is 50% probability that the partner is 'switched' at a random timestep between 30 and 70% of the 600-timestep duration (the random dynamics ensure that the agent cannot memorise a fixed timing pattern). To ensure non-triviality, the post-switch partner is always sampled with a mirrored ability profile (i.c. if the initial partner is faster task 1, the new partner is faster at task 2, and vice versa). When evaluating the agent after training. we simplify the analysis by performing the switch in every episode, always at exactly = 300, and only in a single direction (faster task faster at task 2).
4.3.3 Experiment 3: Can blind agents develop partner models from task reward alone?
Inspired by the findings of Wijmans et al. 33 who showed that 'blind' agents trained for PointGoal navigation develop internal map-like representations of their environment despite having access only to proprioceptive feedback, we ask related question- can blind versions of our Overcooked agents, trained purely to maximise cooperative task reward, still develop internal models of their partners' capabilities?
To investigate this, we remove all visual input from our ego agents and restrict them receiving only egocentric signals (information only about their current grid cell, including their location, orientation, and any objects they are holding). Importantly, they are unable to perceive their partner's location, or directly observe their behaviour.
To generate variation in partner competence, first train iighpeerfrmmacc sieea-rre agent in self-play, and then inject controlled levels of noise into its policy. This produces a range of partner
5

---

behaviours from reliably competent to frequently erratic. The ego agent must infer where on this spectrum each partner lies purely from its own direct experiences and task rewards. Full details of the partner generation and evaluation procedures are provided in the Appendix.
4.4 Evaluation Overview
To investigate how and when internal partner models emerge, we analyse agent behaviour across five Overcooked-Ai layouts (see Appendix). Our evaluation is motivated by three key questions: (i) Can the ego agent collaborate effectively with previously unseen partners? (ii) Do hidden states encode partner traits such as speed or competence (iii) Does the agent update its internal model in response mid-episode changes in partner attributes? To address these questions, we analyse overall reward (total number of soups delivered), adaptation curves (how quickly reward accumulates over time), linear probe accuracy (how well partner traits can be 'decoded' from RNN hidden states by optimising single linear layer with input size dim(h) and output size 1) and UMAP projections [47 (capturing the structure of hidden states). We also compare against three baselines, each designed to isolate a different factor affecting partner modelling: a feedforward MLP, which lacks memory; a single-parinet RNN, trained on a fixed partner to assess the role of training diversity; and a non- infiuential RNN, trained across the full distribution of partners but without the ability to influence their behaviour. A more detailed description of each baseline is provided in the Appendix.
5 Results
5.1
Collaborative agents adapt to unseen partners
To establish the importance of modelling different partners to our chosen environment, we compare the performance of our RNN ego agent policies against two simpler baselines: purely ford-forward MLP policy, and an RNN policy trained alongside only a single constant partner. Figure 2 shows the results of this comparison across five different layouts. We find that the RNN policy trained against diverse partners outperforms both the MLP policy and the auglle-artteer-rriine RNN variant; presumably by virtue of learned ability to model partner parameters. The single exception to this is fivebyfive_ II, in which the MLP agent achieved the highest reward-posse fact that fivebyfive_v / is simpler layout, requiring less spatial coordination and allowing simpler behavioural strategies to perform well. Importantly, the higher performance of the multi-partner partner-RNN with respect the single- sagle-partner RNN does not reflect simple memorisation of different partners, since the partners encountered in evaluation were unseen during training. In addition to superior performance, we find that evaluation episodes with the pealtippartner-triinne RNN ego agent yield higher correlation between which task the partner spends most time on, and which task they' 're fastest at (C)uure[C)-sae
due to the
that the ego agent's 's task allocation decisions are informed by a representation of different partners' ability profiles.
5.2
Agent memory encodes partners' abilities (when there is pressure to do so)
key idea behind our experimental design is that having the ability to influence which subtask a partner performs will pressure the ego agent to learn hidden representations that encode the task speeds of different partners. To investigate the hypothesis further, we trained RNN policies under three different conditions: our 'full' setup including both partner diversity and task influence, plus the 'single-pan pareler'prnerrr and 'non-influential" variants described in Section 4.4 For each condition, we then extracted the trained agent's 's hidden states during evaluation episodes alongisde 46 different unique partners in 5 different layouts (with 20 seeds per condition/laoou//aatneee)))
Figure 3 shows, for each condition and layout, 2D UMAP projection of these hidden states (averaged over the final 50 timesteps of each rollout), coloured by the difference in task speeds for each partner (task 1 speed - task 2 speed). We can see that for the multi- partner-trainee trained RNN, there high degree of structure, with less for the non-influential variant. For each condition and layout, we also trained single-layer linear probes at different values of t to predict partner task speeds from hidden states averaged over (0, t). Figure 33 shows the accuracy of these probes on test set of held-out partners: we see that, for the initial hidden states at t = O, our linear probes perform no better than a baseline trained on random data. As the episodes progress and the ego agent is able to interact with and observe each partner, probe
pealll-pattee--rriinee variant, and less still for the

---

MLP
201 engle-perteer
RNW
coord/ing. roggr2
cromped Joom
cramiped_oomm va
Overall
12-
cocnd jin.
Ovalyfive
Trougoo
CramtoOd room
coamoed room, v3
Overall
0.03
0.02
0.0
100
150
230 Teetstep
352
100
120
202 Srrette
250
10
100
150
200
250
Tiaesttg
MLP
RAN
RNN (single-partner)
RR 0.03
R² 0.28
R2 0.21
of Lin.sio tock
p 1.43e 03
p < le
p x le
0.8
0.6-
inpecio of
02
-0.75
-0.50
-0.25
0.05
0.25
0.50
0.75
-0.75
-0.50
-0.25
0.00
0.25 0.50
0.75
-0.75 -0.50
25.25
0.00
0.25 0.50 0.75
Tack spood 239k Steed
199t
5p000 Daok spocs
Task 3 speed 1a5k speed
Figure 2: Comparison between different ego agent policies in the overcooked environment. Each policy evaluated in five different layouts against different combinations of partner speed parameters, over 10 seeds. (A) average episode reward, per-layout and overall (B) throughput (rate of soup production), per-layout and overall, with shaded areas giving bootstrapped 95% confidence intervals (C) correlation between how much faster the partner agent was at task I vs task 2 and the proportion of time the partner spent performing task 1 (over all layouts). Shaded areas show 95% confidence intervals over the slope; higher correlation indicates more efficient task allocation.
accuracy increases across all three conditions- but is highest for the multi-partner-trained RNN, and significantly impaired for the non-influential RNN.
These results offer convineing evidence, first of all, that our ego agent has learned encode meaningful information about different partners in memory, without being explicitly trained to do so. They also demonstrate the importance of environmental pressure the emergence of these representations. In particular, when the ego agent is stripped of its ability to influence partners' behaviour directly, hidden states contain significantly less information about partner task abilities (as measured by linear (rcodeaablityy)-srrkiin ever encountered a single partner during training! For a replication of these results in second environment, see Appendix A
even less than those of the ego agent that only
5.3
Agents can adapt online to new partners
So far, we have established that our RNN ego agents, once trained, can adapt different partners across different episodes. A stronger version of flexible cooperation involves adapting to new partners 'online', without the environment or internal agent state being reset. To this, train an ego agent alongside partners whose task speeds may change to once per episode. Across three layouts, we then evaluate this agent over number of 600-timestep rollouts where they are paired initially with

---

comppdd cramped
cramped
by
Mutt-aafrenttaanee PNW
100L us JO0SRU 106L20
Single-paninn Singin-prttnnntraaee FANN
No. infhurccc PAN
Partner's spead
Mullipatter AAN Single RNN
speed
0.85
0.85
0.80
0.79
0.09
Figure 3: A comparison of the hidden states of RNN ego agents trained under different conditions. (A) UMAP embeddings RNN hidden states averaged over the final 50 timesteps of each episode, coloured by the partner's difference in task speeds (speed 1 speed 2), for five different layouts of the Overcooked environment. (B) Mean test accuracy of linear probes trained to recover partner speeds from sets of RNN hidden states accumulated up to different timesteps (with shaded areas giving bootstraped 95% confidence intervals).
partner that fast at task and slow at task 2, then switched / 300 to partner with the opposite profile.
To measure how well our agent copes with this scenario. we track the average soup throughput over time. From the results in Figure 2B, we expect that the throughput should initially increase to a steady state; we anticipate that will then drop sharply at 300 the partner's task speeds are reversed. After this point, if the ego agent is capable of adapting online to the new partner, the throughput should increase once more to new steady state; if not, should remain low. Figure 4 shows that for all three tested layouts, the throughput does indeed increase again after = 300, demonstrating As further evidence, Figure 43 shows that, on average,
presence online adaptation partner is directed to allocate their time mostly to task for / < 300 and mostly to task 2 for / 300. Finally, we also visualise in Figure how the ego agent's hidden states are affected by the switch. UMAP projections of hidden states averaged over f < 300 align roughly with the distribution of embeddings for "baseline' episodes with (constant) partners fast at task and slow at task over
'We note that the throughput does not fully recover post-switch due to an asymmetry in the subtask complexities: because task requires more steps than task 2 to execute, the overall team efficiency is higher when the partner fast task 1 and slow task than the reverse.

---

oramped
oramppdd
cramppod
0.3
0.4-
0.2-
300
300
partner switched
pre-swits post-ewith
form non wwech solavily
Figure 4: A demonstration of online adaptation. (A) Average throughput (rate of soup delivery) during episodes where the partner is switched halfway through from being faster at task to faster at task 2. (B) From the same episodes, the average proportion of time spent by the partner performing each subtask before and after the switch. (C) UMAP embeddings
the average pre-switch and post-switch RNN hidden states from each episode. Also shown are the distributions (approximated via KDE) for embeddings of hidden states from non-switch baseline episodes with partners matching the pre- and post-switch speeds respectively.
t >> 300, they move match the distribution for episodes with partners slow at task I and fast task 2 (see supplementary videos for visual examples of this adaptation).
5.4 Partner modelling emerges even in blind AI agents
We find that structured partner modelling also emerges in blind agents trained without any architec- tural biases toward modelling their partner. Despite relying purely on egocentric observations and scalar task reward, these agents generalise to new partners and outperform both recurrent agent trained on single partner and memoryless MLP baseline. Evaluated across five Overcooked layouts (with six novel partners per layout and 10 random seeds per permutation), the blind RNN agents trained with collaborators with a diverse range of competencies achieved an average throughput of 9.43 soups per episode. This compares just 5.8 for the single-partner RNN and 1.08 for the MLP. These results show that the interaction structure and memory can enable adaptive behaviour to the capabilities of partner agents corresponding videos and a breakdown of the results, are included in the supplementary materials.
under harsh observational conditions. Further details, including
Discussion
In this paper, have studied the question of whether representations of other agents' relevant at- tributes can emerge simply as a result of envirommental pressure to collaborate effectively with diverse partners. the absence of dedicated architectural features or auxiliary objectives, we found that RNN agents trained to play a version of the cooperative game 'Overcooked' nevertheless developed structured internal representations of their partners' task abilities. In an additional experiment, showed that these representations enable agents to adapt online to new partners within the same episode. Finally, we also demonstrated that the development of structured representations is signifi- cantly weakened when agents are denied the ability to influence partner behaviour. Taken as whole, our results serve illustrate the idea that social intelligence can emerge from specific environmental

---

pressures acting in concert with general mechanisms for learning and memory, rather than necessarily relying on unique architectures. We believe that this is important to bear in mind as we seek to develop artificial agents that bridge the gap towards humanlike social cognition and behaviour.
It is notable that structured partner modelling also emerged in blind agents trained without visual input, relying solely on egocentric signals and task reward. Despite having no explicit access to their partner's actions or state, these "blind" agents developed internal representations that enabled them to effectively collaborate with partners displaying a diverse range of competencies. This emerged despite minimal inductive biases and limited observational input. This indicates that the structure of the interaction itself is sufficient to drive the emergence of partner-aware policies.
While we feel our work represents valuable contribution to the study of emergent partner modelling, we highlight various limitations that might serve as starting points for future research. First and foremost, our experiments used only a single cooperative environment (Overcooked-Al)--a) we are confident that our results will generalise to other environments and task settings, an obvious target for future work is to confirm this empirically. Of particular interest would be open-ended environments that allow us to study how partner representations evolve over time in a more continuous setting; or those with more complex or overlapping subtasks. Relatedly, future work might study whether our findings scale to more than one teammate-we believe that they should, provided that the environment imposes sufficient pressure (i.e. all teammates' behaviour is relevant to task completion). That said, inference will become more demanding with additional agents, and so the ego agent may learn to rely on shortcuts such as modelling the average behaviour of its teammates.
A further limitation is that we have restricted ourselves to studying relatively simple forms of representation. In the real world, people engage in highly complex modelling of their social partners, including through hierarchical representations that deal with how others are perceiving them in turn. It would be interesting to investigate whether these simple, general agent architectures are capable of acquiring such sophisticated capabilities, and under what environmental conditions. Related to this is the fact that our implementation of 'influence" was very strong, essentially taking the form of direct control. Humans typically influence their collaborators in much more nuanced ways future work might reduce this gap via some form of communication system, where the partner learns to follow (or ignore) high-level instructions from the ego agent.
Aside from these limitations, further avenue is explore transformer-basee agents, where attention may offer a natural probe of whether agents implicitly track their partners' positions or strategies. Another direction is test whether internal partner representations can be transferred between agents and tasks via hidden state initialisation. For example, one could evaluate whether seeding an agent's s memory with the final hidden state from a previous rollout improves adaptation when encountering the same partner; probing the portability and generality of the learned representations. Comparing 'transplanted" and 'cold-start' agents would provide insight into the extent to which internal partner models support efficient reuse and generalisation. Another possible avenue would involve direct comparisons between the implicit partner modelling approach we study here and various explicit methods: we expect that the latter would perform better in the specific modelling contexts they were trained for, at the cost of reduced flexibility to changes in task environment or partner attribute space.
Finally, while our primary motivation is the desire for artificial agents capable of collaborating flexibly with humans, we believe that version of our approach might also be used shed light on the evolutionary differences in collaborative and cooperative behaviour observed across different animal species. To this end, future work might explore using large-scale evolutionary simulations to further study the interplay of environmental pressures and agent architectures; an approach which has recently proven fruitful for investigating other social behaviours such altruism (48)
7 Acknowledgements
This work was supported by the EPSRC CDT in RAS (EP/LL01634//) and the National Defense Science and Engineering Graduate (NDSEG) Fellowship Program awarded E.M. We thank J. Shah and many others for their invaluable support and expertise.
10

---

References
[1] Michael Tomasello, Alicia Melis, Claudio Tennie, Emily Wyman, and Esther Herrmann. Two key steps in the evolution of human cooperation: The interdependence hypothesis. Current Anthropology, (5(6))673-692, 2012.
[2] Sarah Blaffer Hrdy. Mothers and Others: The Evolutionary Origins of Mutual Understanding. Harvard University Press, Cambridge, MA, 2009.
[3] Joseph Henrich. The Secret of Our Success: How Culture Driving Human Evolution, Domesticating Our Species, and Making Us Smarter. Princeton University Press, 2018.
[4] Ruaridh Mon-Williams, Gen Li, Ran Long, Wenqian Du, and Christopher G Lucas. Embodied large language models enable robots to complete complex tasks unpredictable environments. Nature Machine Intelligence, pages 1-10, 2025.
[5] Ian A Apperly and Stephen A Butterfill. Do humans have two systems to track beliefs and belief-like states? Psychological review, 116(4))953, 2009.
[6] Chris Baker, Rebecca Saxe, and Joshua Tenenbaum. Bayesian theory of mind: Modeling joint belief-desire attribution. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
[7] Rebecca Saxe and Nancy Kanwisher. People thinking about thinking people: the role of the cemporo-paritat junction in "theory of mind". Social neuroscience, pages 171-182. Psychology Press, 2013.
[8] Mark K Ho. Rebecca Saxe, and Fiery Cushman. Planning with theory of mind. Trends in Cognitive Sciences, 26(11):998 (10(111959-990) 2022.
[9] Yang Xiang, Natalia Vélez, and Samuel Gershman. Collaborative decision making is grounded in representations of other people's competence and effort. Journal of Experimental Psychology: General, 152(6):1565. 2023.
[10] Yang Xiang, Natalia Vélez, and Samuel J Gershman. Optimizing competence in the service of collaboration. Cognitive Psychology, 150:101653, 2024.
[11] Carolyn Baer and Darko Odic. Mini managers: Children strategically divide cognitive labor among collaborators, but with self-serving bias. Child Development, 93(2):437 450, 2022.
[12] Neil Rabinowitz. Frank Perbet, Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. Machine theory mind. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4218 1127.. PMLR, 10-15 Jul 2018.
[13] Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano Albrecht. survey of ad hoc teamwork research. In European conference on multi-agent systems, pages 275-293. Springer, 2022.
[14] Yancheng Liang. Daphne Chen, Abhishek Gupta. Simon S Du, and Natasha Jaques. Learning to cooperate with humans using generative agents. arXiv preprint 1rXiv:2411.1 13934, 2024.
[15] Piotr J. Gmytrasiewiez and Prashant Doshi. framework for sequential planning in multi-agent settings. J. Artif. Int. Res., 24(1):49-79, July 2005.
[16] Samuel Barrett, Peter Stone, and Sarit Kraus. Empirical evaluation of ad hoc teamwork in the pursuit domain. In The 10th International Conference on Autonomous Agents and Multiagent Systems Volume 2, AAMAS '11, page 567-574, International Foundation for Autonomous Agents and Multiagent Systems, 201
[17] Stefano V. Albrecht and Subramanian Ramamoorthy, A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS '13, page 1155-1156, 2013.

---

[18] Stefano V. Albrecht, Jacob W. Crandall, and Subramanian Ramamoorthy. Belief and truth in hypothesised behaviours. Artificial Intelligence, 235:63-94. 2016.
[19] Georgios Papoudakis and Stefano V Albrecht. Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint arXiv:2001. 2020.
[20] Arrasy Rahman, Ignacio Carlucho, Niklas HAl[pnne, and Stefano V. Albrecht. A general learning framework for open ad hoc teamwork using graph-based policy learning. Journal of Machine Learning Research, 24(298):1-744 2023.
[21] Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A compre- hensive survey and open problems. Artificial Intelligence, 258:66-95, 2018.
[22] Ruaridh Mon- Non-WWlliams,, Theodoros Stouraitis, and Sethu Vijayakumar. A behavioural trans- former for effective collahoration between a robot and non-stationary human. 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pages 1150-1157. IEEE, 2023.
[23] He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé, III. Opponent modeling in deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1804-1813. PMLR, 20-22 Jun 2016.
[24] Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4257-4266. PMLR, 10-15 Jul 2018.
[25] Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations multiagent systems. Jennifer Dy and Andreas Krause, editors, Pro- ceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1802-1811. PMLR, 10-15 Jul 2018.
[26] Luisa Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, and Katja Hofmann. Deep interactive bayesian reinforeement learning via meta-learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '21, page 1712-1714, 2021.
[27] Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 575-588. PMLR, 16-18 Nov 2021.
[28) Georgios Papoudakis, Filippos Christianos, and Stefano V. Albrecht. Agent modelling under partial observability for deep reinforcement learning. In Proceedings of the 35th International Conference Neural Information Processing Systems, NIPS '21, 2021.
[29] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Leibo, and Nando De Freitas Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning, pages 3040-3049. PMLR, R, 2019.
[30] Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7187-7194, 2020.
[31] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In International Conference on Machine Learning, pages 4399 4310- PMLR, 2020.
[32] Kshitij Dwivedi, Gemma Roig, Aniruddha Kembhavi, and Roozbeh Mottaghi. What do navigation agents learn about their environment? In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10266- 10276, 2022.
12

---

[33] Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee, Ari S. Morcos, and Dhruv Batra. Emer- gence of maps in the memories of blind navigation agents. In The Eleventh International Conference on Learning Representations, 2023.
[34] Ulisse Mini, Peli Grietzer, Mrinank Sharma, Austin Meek, Monte MacDiarmid, and Alexan- der Matt Turner. Understanding and controlling a maze-solving policy network, 2023.
[35] Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, UIrich Paquet, and Vladimir Kramnik. Acquisition of chess knowl- edge in alphazero. Proceedings of the National Academy of Sciences, 1 119(47):222662551 19, 2022.
[36] Arthur Guez, Mehdi Mirza, Karol Gregor, Rishabh Kabra, Sebastien Racaniere, Theophane Weber, David Raposo, Adam Santoro, Laurent Orseau, Tom Eccles, Greg Wayne, David Silver, and Timothy Lillicrap. An investigation of model-free planning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Muchine Learning Research, pages 2464-2473. PMLR, 09-15 Jun 2019.
[37] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05566 2016.
[38] Andy Clark. Whatever next?
predictive brains, situated agents, and the future of cognitive science. Behavioral and brain sciences, 36(3):181-204, 2013.
[39] John C Harsanyi. Games with incomplete information played by "bayesian" players, i-iii part i. the basic model. Management science, 14(3):159-182, 1967.
[40] Barbara Grosz and Sarit Kraus. Collaborative plans for complex group action. Artificial Intelligence, 86(2):269-355 -357, 1996.
[41] Qian Long, Ruoyan Li, Minglu Zhao, Tao Gao, and Demetri Terzopoulos. Inverse attention agents for multi-agent systems. arXiv preprint arXiv:2410.21794, 2024.
[42] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 14844484-5, 2021.
[43] Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garðar Ingvarsson, Timon Willi, Ravi Hammond, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whiteson, Bruno Lacerda, Nick Hawes, Tim Rocktäschel, Chris Lu, and Jakob Nicolaus Foerster. Jaxmarl: Multi-agent rl environments and algorithms in jax. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[44] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar, October 2014. Association for Computational Linguistics.
[45] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[46] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018.
[47] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
13

---

[48] Max Taylor-Daves, Gautier Hamon, Timothé Boulet, and Clément Moulin-Frier. Emergent kin selection of altruistic feeding via non-episodic neuroevolution. In Pablo García- Sánchez, Emma Hart, and Sarah I.. Thomson, editors, Applications of Evolutionary Computation, pages 496-509, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately represent the paper's empirical contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Our Discussion section includes some limitations of the current work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed information for reproduceability will provided in the (supplemen- tary) technical appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Links to the experiment and analysis code will be provided in the (supplemen- tary) technical appendices.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
CoodeSumeiiie ssi mnyliccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These details will be provided in the (supplementary) technical appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All results figures include confidence intervals where appropriate.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information will be provided in the (supplementary) technical appendices.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:::////// http:///wrrrppp.ccccpccccc/beeea
Answer: [Yes]
Justification: Our work conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We are not aware of any particular societal impacts of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
The paper does not pose any such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the authors of the Overcooked-AN environment, which the only existing asset we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not include the release introduction of any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not use any experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not use any experiments with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We have not used LLMs for any non-standard component of our work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neuriin 2ttpp/////////oooeeeeee///////// for what should or should not be described.
21

---

Appendix
A
CoinGame results
To ensure that our findings generalise beyond the specific environment of Overcooked, we con- ducted some preliminary experiments using modified version of the JaxMARL suite's CoinGame environment
43
A.1 Environment
In our version of CoinGame, agents (the ego agent and one partner) must work together to collect red and blue coins within a small gridworld (see Figure At any given timestep, the partner is in either 'red mode' (tries only collect red coins while ignoring blue) 'blue mode' (vice versa). Different partners are characterised by a two-dimensional 'skill profile" [s), which controls their probability of successfully executing actions when in each mode (the ego agent has 8, 86 1). As in our Overcooked experiments, the ego agent can exert influence by switching their partner's mode. The ego agent is rewarded based on the total number of coins collected (Nua + Tolue over an episode; they are thus incentivised to find the most efficient 'division of labour' between themselves and their partner.
A.2 Experimental procedure
As in our main Experiment
(..1 we trained ego agents for le7 timesteps under four conditions:
1. MLP (ego agent uses simple MLP in place of an RNN)
2. No-influence RNN (ego agent has control which coin type the teammate pursues)
3. Single-partner RNN (ego agent only exposed to a single partner type during training)
4. Multi-partner RNN (ego agent paired with multiple partner types during training and has influence)
During training, the single-agent RNN was always paired with teammate with skill profile [0.2, all other cases partners were sampled uniformly from the set {[x, 1 - x)
0.6, ). During evaluation, partners were always sampled uniformly from the set [[=,1- x] V E {0.1, 0.3, 0.5, 0.7, 0. The ego agent thus never encountered during evaluation partner they had previously seen in training. From these evaluation episodes we recorded the total number of coins collected, the number of timesteps where the teammate was pursuing the 'correct' coin colour (based on their skill profile), and the hidden states of the ego agent RNN (where applicable). As with our experiments in Overcooked, the hidden states were analysed via UMAP projections and linear probe accuracy.
A.3 Results
Looking at Figure 5B, we see that the multi-partner-RNN ego agent outperforms the single-partner- RNN agent, which turn outperforms the no-influence-RNN and the MLP agents-rrpliicting the trend we observed in our Overcooked results. Also corroborating our previous results is Figure 6 which shows that hidden states extracted from the multi-partner- RNN are more structured with respect to partner skill profiles than those from the single-partner no-influence variants. Finally, we computed the correlation to partner skill profile of individual RNN hidden unit activations. As Figure 7 shows, we found multiple units with strong positive or negative correlations (16/32 with [corr| 0.5); as well some units with close-to-zero correlation (likely encoding task/environment information unrelated to partner properties).
22

---

CoinGame environment
Comparison of ego agent per ferfommnce
Ro-ii deighopiiee RAM Poalter
Noviuurso to digg eypaeie RNM
Figure 5: (A) An example CoinGame environment layout episode start. The two coloured circles are coins, the red square a teammate currently in 'collect red coins' mode and the black square represents the ego agent. (B) A comparison of the evaluation performance of different ego agents trained in the CoinGame environment, showing the mean total number of coins collected by ego agent and teammate (left) and the mean proportion of time spent by the teammate pursuing the 'correct' coin colour (i.e. the one matched to their highest skill level). Error bars give bootstrapped 95 % confidence intervals over 5 random training seeds.
UMMP projections of RNN hidder
Liner epieode
Bo-iraarr RhM
Sing
degio sorter nult No ao-intuvec RAM
1200
(M4 componett
Uoich
25 23
15
100
135
200
Figure 6: (A) UMAP embeddings of RNN hidden states averaged over the final 50 timesteps of each episode, coloured by partner skill profile. (B) Mean test accuracy of linear probes trained to recover partner skill from sets of RNN hidden states accumulated up to different timesteps (with shaded areas giving bootstraped 95% confidence intervals).
B Additional details
B.1
Training setup
B.1.1
Ego agent training
The ego policy trained on a single GPU using Proximal Policy Optimisation (PPO), running syn- chronously across 256 parallel Overcooked-Al environment instances. Both agent and environment are implemented in JAX with jax. for accelerated gradient updates and rollout collection. Each rollout lasts 400 timesteps in Experiments 1 and 3, and 600 timesteps in Experiment and agents are rewarded for every successful soup delivery. Training runs for 15 million timesteps against a distribution of partner agents. For the first 5 million timesteps. a decayed reward shaping used to aid policy learning (rewarding the agent for putting onions in the pot and for cooking the soup when it contains the correct ingredients). For experiments I and 2, partner agents could at any given timestep perform one of two subtasks: (1) placing ingredients into a pot, or (2) serving soup (which involved picking up a bowl, ladling the soup, and delivering it). Each subtask is handled by a separate, pre- treitrained neural network that specialises in the specific behaviour. The RNN ego agent has an additional action that allows to set the partner's current subtask. For experiment (with the blind agent), the ego policy is trained against distribution of partners who perform subtask to varying competencies. We randomised the starting state of each episode during both training and testing. The RNN hyperparameters used for the experiments shown in Table
B.1.2 Partner agent training
Each partner policy consists of two ford-forward subtask networks trained independently in self-play using PPO with reward shaping. To obtain subtask- -specfc policies, each network paired with
23

---

Conelaaions activations individaa hidden uniti skill
-2.8
03
00
92
26
Necenart
Figure 7: Top: correlation coefficients between individual hidden unit values and partner skill profile computed over 1000 evaluation episodes. Bottom: normalised average value each hidden unit for each distinct partner.
Table I: RNN training hyperparameters
Parameter
Value
FC FC_DIM_ SIZE
DIISIIIIIE
128
GRU HILDHIDDEN,CC DIM
128
ACTIVATION
relu
LR
5e-4
ANNEAL_LP
True
LR_ WARMUP
0.05
NUM_ENVS
256
NUM_STEPS
256
UPDATE EPOCHS
UPDATE_EPOCHS
4
NUM MINIBATCHES
64
TOTAL TIMESTEPS
le7
CLIP _EP
0.2
ENT
ENT_COEF COEF
0.01
GAMMA
0.99
GAE GAE_LAMBIN LAMBDA
0.95
SCALE_ CLIP_
False
VF_ UF_COEF
1.0
0.25
partner that always performed the other subtask. To ensure robustness we introduced randomness during training: the subtask network had 30% probability of waiting any given step and 30% probability that its partner would take a random action. This prevented over-fitting to specific interaction patterns or timings and ensured that the learned policies could operate independently. We found that this approach allowed us to produce policies for the partner agent faster than relying on manually specified rules heuristics.
After training, these subtask networks were combined to form the full partner agents capable of performing both subtasks. To create a distribution of partner behaviours with varying speeds, we controlled the speed at which each subtask policy could be exccuted when interacting with the ego agent. For example, partner agent might be able to execute the ingredient placement policy every four timesteps and the serving policy every two timesteps. This enables us to simulate a range of partners with different speeds and proficiency across the two tasks, while utilising the same robust underlying subtask policies.
For experiment 3, we created distribution of partners with different competencies by adding certain probability of the partner taking random action any given timestep (e.g, a competent partners might have 5% chance of acting randomly; an incompetent partner might have a 95% chance).
24

---

B.2 Agent architectural design
The architecture includes both input and output fully connected (FC) layers. After the convolutional neural network (CNN) processes the observation, an FC layer maps the resulting embedding to the hidden size of the GRU. The GRU's output is then passed through separate two-layer multilayer perceptrons (MLPs) for the actor and critic heads, which generate action logits and scalar value estimate, respectively. Observations are pre-pre pe-processed to ensure each agent has local, self-centred view of its environment, Furthermore, the CNN output is normalised before being fed into the GRU stabilise training and improve performance.
B.3 Evaluation details
B.3.1
Throughput
ASoup The throughput (rate of soup production, used Figures and 4) given by A To obtain point estimate of the throughput at time t, we fit the slope of the cumulative reward curve over the sliding window [t 25, + 25] by the method of least squares.
B.3.2 UMAP projections
To obtain 2D embeddings of RNN hidden states (as seen in Figures 3 and 4), we use the official Python implementation of the UMAP algorithm hyperparameters min sinnddis=11..11 O and m__neihbbor=-N- where N is the number of hidden state datapoints being projected. These hyperparameters were selected to ensure focus on the 'global', high-level structure of the embedding space wrt the partner parameters, rather than small-scale localised clusterings. For all other hyperparameters we use the library default values.
with
B.3.3 Linear probe analysis
For our linear probe analysis (used in Figure 3), optimise linear layer to perform the classification task 7, + where 7 is the averaged RNN hidden states up to time from a single episode, and is the partner's speed at either task or 2 for that episode. We train probes separately for different values of as well as ego agents trained under different conditions. Each probe is trained for a total of le3 steps using the Adam optimiser with learning rate of lc-2. We use train-test split of 80-20 over rollout seeds (i.c. where we have 20 seeds per partner speed combination, we randomly assign 16 of those seeds to the train set and 4 to the test set). We report the distance-aware classification accuracy over the test set. For comparison, we train a 'baseline' probe under identical conditions using random F (sampled from the standard Normal distribution with the same shape the hidden states) and the true labels.
B.4 Other experiment details
B.4.1
Experiments 1-2
To reduce the chance of poor policy convergence due random seed, ran the entire training process over five random seeds per environment layout, and selected the ego agent policy that achieved the highest average episode return at the end of training. During testing, agent with 24 different partners not encountered during training and collected 20 episode rollouts per partner (over 20 random seeds, which randomised the starting state), for total of 2400 different rollouts. The seeds that achieved the highest return are as follows:
paired the ego
Table 2: Best performing seed (1-5) for each layout and model
MI.P
RNN
RNN Online
RNN Single Partner
Cramped Room
2
3
5
FiveByFive
2
4
5
Coord Ring
2
3
5
Cramped Room V3
2
2
5
Cramped Room V4
2
2
2
25

---

For experiments I and 2, we used an ego agent with a fixed cooldown interval of (across both subtasks). We found that if the speed of the ego agent was higher than this, then it would experience less pressure to model the partner agent, converging to policy that was independent of partner properties. Conversely, at lower speeds, the ego agent struggled converge to effective policy.
B.4.2 Experiment
In Experiment 3, we test a 'blind' ego agent that only receives local observations of its own square (position, orientation, and held object) can implicitly model the competence of its partner. For the partner policy, we use the ingredient-preppae preparation subnetwork (subtask 1). During training, the ego agent is paired with a distribution of partners that vary their probability taking random actions (versus optimal actions); representing different levels of competence. For testing, we introduce six novel partners with fixed randomness levels E {0, 0.05, 0.1, 0.9, 0.95, 13. This allows us to investigate whether the ego agent adapts its behaviour to both highly competent and highly erratic partners. We hypothesise that the ego will collaborate efficiently with skilled partners, while taking on more of the task when paired with less capable ones.
B.5 Environment layouts
For both training and evaluation, we use five environment layouts with different spatial constraints: crumped room, coord ring, fivebyivve cramped_roome and cramped_room, chosen on the basis that they incentivise agents work together rather than independently. and are no larger than 5x5 tiles (to ensure convergence successful policies). The five layouts depicted in Figure
All layouts were
sip
(d) Cramped Room
(e) Cramped Room
(a) Cramped Room
(b) FiveByFive
(c) Coord Ring
V3
V4
Figure 8: Layouts used in the Experiments
B.6 Baselines
To isolate the conditions under which partner modelling emerges, we utilise three baselines that allow us disentangle the effects of training diversity, memory, partner information and architecture on collaborative performance. First, to see if memory plays role, we include a feedforward network (FFN) MIP baseline. This tests whether memory is necessary for adapting diverse partners (poor performance would suggest that memory is important in partner modelling). We next consider single partner specialist, which a GRU recurrent agent trained with one partner. This baseline probes whether exposure to diverse partners is necessary to develop generalisable partner model poor performance in this baseline would outline that training diversity is cruical for emergent modelling, Finally, we used non-influential without the ability to influence them. During rollout with the non-influential RNN, the partner mode switched halfway through training to allow the ego agent retain memory of the partner's ability in both subtasks.
RNN trained over distribution of partners, but
B.7 Illustrative videos of agent-partnce interactions
To illustrate the experiments and interaction dynamics, we include example videos from the RNN policy trained on a distribution partners. Although many policies are trained throughout the experiments, these examples are used to highlight the kind of adaptations (and occasional failures to adapt) that emerge under novel test conditions. Videos are included to show the ego agent interacting with two previously unseen partners (using random seed per layout) in each experiment. In Experiment 1, the agent interacts with a fixed partner throughout the episode - one that competent
26

---

(i.e.. speed 1) at ingredient preparation and one at serving. In Experiment 2, the partner switches partway through the episode (at time step 300), from a skilled ingredient preparation partner skilled serving partner, or vice versa. During training, partner switches occur at variable times or not all, so the agent cannot anticipate when whether change will occur. In Experiment 3, the blind ego agent is paired with both an unskilled partner and competent one (at ingredient preparation)- to visualise the large variations in partner ability. These videos are provided in the supplementary materials and demonstrate how the trained policy generalises to new human collaborators.
B.8 Compute
All experiments were run on A100 GPUs. totalling 462 GPU-hours, Policy training used 225 GPU hours for all three experiments. A total of 37,400 rollouts were simulated, totalling 207 GPU-hours.
B.9
Code availability
The full code for this paper is available at: httt:///gstuu...ccmmmrarddddddnn partner_modellle
27


Reviews:
[{"review_id": "Xu1iPh8Dqu", "reviewer": "Reviewer_xMnq", "summary": "This work investigates the factors behind partner modelling in RL agents. The authors use the cooperative Overcooked-AI environment, where they train different variants of agents together with a partner agent whose policy is pre-trained. The experiments vary in the perception of the agent-in-training (\"blind agent\"), the abilities of the agents (ability to direct the partner agent, partner being slow in one task, slow in both tasks, or exhibiting random behaviour) and in the permanence of the pairing (the fixed-policy partner agent may get swapped during an episode). This variety of experiments shows that the main factor of the emergence of partner modelling is being able to direct the partner to perform different tasks, and that even a \"blind\" agent will still learn to model some aspects of their game partner.", "questions": "- Please elaborate on the agent architecture design.\n  - In the appendix, an FC layer is mentioned. Is there just one input FC layer? Or is there also an output FC layer that converts the GRU output into actions? Are there any other pre-processing or post-processing steps?\n- Figure 2A: Why does the MLP perform better in the five-by-five environment? It seems to also be quite close in the coord_ring environment.\n  - This may indicate that partner modelling may also be environment-dependent.\n- Figure 3A: Was there any further investigation of the example No-influence RNN for the cramped room v4? There seems to be a clear separation, but not on task speed.\n- Linear probing: How fast does the ego agent re-encode the partner’s task speed? I think this would be interesting to see, especially in the context of Figure 4A, to see if this re-encoding delay has an impact on the lower throughput even after 300 epochs.\n- Additionally, why does the throughput never recover after the partner switch?\n  - If the ego agent was trained with agents who can be both slow and fast with different tasks, shouldn't it also be able to switch policy to the correct task speed? This is especially true considering that the UMAP embeddings  in Figure 4C seem to show that this is the case, yet the throughput is not increased.\n- Appendix A.3.2 — You mention \"We hypothesise that the ego will collaborate efficiently with skilled partners, while taking\non more of the task when paired with less capable ones.\" — was this hypothesis tested?", "limitations": "The main limitations I can see, which are not mentioned in the papers, are:\n\n- Only two agents in the environment — It is unknown if the ego agent would build more complex internal representations of multiple agents, or converge to a different strategy without partner modelling.\n- Fixed policy of the partner agent — Would two learner agents be able to converge to a similar strategy?\n- Allowing the ego agent to control the other agent — This to me comes close (but not quite) to the other agent just becoming an extension of the ego agent, with the ego agent just learning to control this different \"module\" of its own architecture.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Originality\n\nThe experimental setup and the ideas in this work are novel. While it may be simplistic, I think this is a good step in partner modelling. The closest work I am aware of is Jaques et al., [1], where similarly to this work, influence on other agents is examined as a factor in agent performance.\n\n### Quality\n\nThe quality of this work is high. All claims are based on performed analyses, and where appropriate, statistical significance testing is also performed.\n\n### Clarity\n\nThe clarity is also high; however, I had a few comments as to what could be improved:\n\n- Figure 1 — The speedometers might be a bit confusing — I initially thought they were showing time taken to perform a task, inferring the opposite relationship (i.e., the arrow being on the left is faster, or it takes less time to perform a task).\n- Line 135: JAX should be cited.\n- Line 221: alongisde $ \\to $ alongside\n- Line 209 (and elsewhere): wrt $ \\to $ w.r.t., or spelled out.\n- Figure 3b — there is a gap in the confidence intervals for the multi-partner RNN line plot.\n- Linear probes — what are their dimensions? Is it just one layer of input size GRU h_dim $ \\to $ output size 1, with the 1 being the speed prediction?\n\n### Significance\n\nThe work has a good amount of significance for the field of MARL. The insights of requiring influence over another agent for a better development of ToM are an interesting datapoint, and would be a good starting point to see if this translates to more than two agents, and to more complex environments.\n\nAdditionally, the authors provide all the code, including Docker files for reproducibility, so building upon this work should be relatively straightforward.\n\n### References\n\n[1] Jaques et al., 2019. Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning. _Proceedings of the 36th International Conference on Machine Learning_, in _Proceedings of Machine Learning Research_ 97:3040-3049, https://proceedings.mlr.press/v97/jaques19a.html.", "was_revised": true, "final_justification": "Based on the discussions and rebuttals I continue to recommend acceptance. Most formatting and clarity issues were/will be resolved in the camera-ready version, and I still think, as I said in my comments and review, that this paper will be a good contribution to NeurIPS.", "rebuttal": "**Thank you for your thoughtful and detailed engagement with the paper.**  We appreciate your recognition of the paper’s originality, clarity, technical quality, and broader significance. We're pleased the experimental design and partner modelling results were well received. Below, we respond point by point and outline revisions we have made or are making to clarify and expand on the points you raised, which we believe further strengthen the paper.\n\n\n### Questions\n\n#### **Q1. Please elaborate on the agent architecture design**\nThank you for the question - we have updated the appendix to clarify this. The architecture includes both input and output FC layers: after the CNN processes the observation, an FC layer maps the embedding to the GRU’s hidden size. The GRU output is passed to separate two-layer MLPs for the actor and critic, producing action logits and a scalar value, respectively. Observations are pre-processed to give each agent a local, self-centred view, and the CNN output is normalised before it enters the GRU. \n\n#### **Q2. Figure 2A: Why does the MLP perform better in the five-by-five environment?**\n\nThis is a good point, and we agree that it indicates that partner modelling is environment dependent. We have updated the manuscript to include an acknowledgement and brief discussion of this idea.\n\nIn the five-by-five layout, spatial coordination is less critical than in cramped_room variants (due to the simpler and more open geography), making feedforward models more competitive. Additionally, these plotted results are from the fully observable setting, where agents can often infer their partner’s behaviour from the current state alone — reducing the benefit of temporal modelling. In contrast, the performance gap is larger in the partial and blind settings, where temporal reasoning becomes more important.\nAnother possible factor is that agents are evaluated with novel partners. RNNs may overfit to training partners, whereas MLPs, lacking memory, may generalise better when spatial context is sufficient.\n\n#### **Q3. Figure 3A: Was there any further investigation of the no-influence RNN in cramped_room_v4?**\n\nYes we did investigate this but we were not able to find any explanation for the apparent structure of the no-influence UMAP plot for cramped_room_v4. We suspect that it is likely an artifact of the UMAP procedure (a phenomenon which is not uncommon) – as evidence for this, we created corresponding plots using PCA and t-SNE in place of UMAP, and did not see a similar grouping appear (while the structure of points in the influence case was preserved to at least some degree).\n\n#### **Q4-5. Linear probing and throughput after partner switch**\nThank you for this helpful point — we are revising the manuscript to clarify why the throughput plateau is lower post-switch. The observed drop in throughput after 300 epochs is actually due to an asymmetry in the speed dynamics between the ego agent and the partner agent when their roles are reversed. In the runs plotted, the partner performs serving soup quickly (1 action per step) and preparing ingredients slowly (1 action every 7 steps). Later, this pattern reverses. The ego agent acts at a constant rate (1 action every 3 steps) for both tasks.\n\nServing soup involves a longer sequence of actions (e.g. retrieving a bowl, ladling soup, serving) than placing onions in a pot. The system performs better when the faster agent handles the more complex task offsetting its length and improving soup throughput.\n\nThe rise in the curve reflects the ego agent adapting to this dynamic—learning to coordinate with a changing partner. Once the throughput plateaus, it indicates the agent has successfully adapted to the partner’s new behaviour.\n\n#### **Q6. Appendix A.3.2 — Was the hypothesis about task allocation tested?**\nYes, Experiment 3 (the blind agent setting) serves as a partial test of this hypothesis. In this setting, the ego agent was paired with partners of varying competence, and the blind agent had to infer its partner’s ability and decide whether to complete the entire task alone or specialise in a subtask.\n\n### Limitations\n\n#### **Two agents only**\n\nThis is an interesting direction for future work, and we have updated the manuscript to discuss it more explicitly. There is no reason to believe that the same kind of implicit representation would not scale to more than one teammate, provided the environment imposes sufficient pressure—i.e. all teammates’ behaviour is relevant to task completion. That said, inference becomes more demanding with additional agents, as more evidence is required to accurately identify their individual properties.  The ego agent may rely on useful shortcuts, such as modelling the average behaviour of its teammates or inferring which teammate is best suited to each subtask. \n\n#### **Two learner agents**\n\nGreat question — we expect that two learner agents could converge to similar strategies through mutual adaptation, though likely via different dynamics. Using a fixed-policy partner lets us focus on the specific question of how the ego agent models and adapts to its teammate. This is also related to the next question about the level of ego agent control/influence over their teammate; if both agents are learning, then it wouldn’t make sense to give one agent control over the other’s subtask, and so we would require a more nuanced mechanism for determining subtask allocation. We have added a brief discussion of this point in the revised manuscript as a direction for future work.\n\n#### **Direct control**\n\nThis is a fair point. We used an extreme form of ‘influence’ to give us the clearest/simplest test of whether partner modelling emerges driven by environmental pressure (i.e. there is a very strong link between ability to represent teammate properties and overall reward). A more realistic setting might involve some form of communication, where the partner learns to follow (or ignore) high-level instructions (and this could even be bidirectional). We have updated the discussion to highlight this as a direction for future work.\n\n### Other Comments\n\nThank you for these helpful suggestions. We have revised Figure 1 to clarify the meaning of the speedometers and avoid potential misinterpretation. We have also cited JAX at first mention, corrected the typo on line 221, and standardised usage of “w.r.t.” throughout. Moreover, we have clarified the linear probe architecture in the text: it consists of a single-layer linear decoder mapping the GRU hidden state to a scalar output (speed prediction).\n\n*Regarding Figure 3B, thank you for flagging this. We're unsure which specific gap in the confidence intervals is being referred to — we’d be grateful for clarification so we can investigate further.*"}, {"review_id": "ZHj5UhwjfL", "reviewer": "Reviewer_FxZn", "summary": "The authors explore the learned representations of RNN-based RL agents using population-based training in the game Overcooked. They explore whether these agents are implicitly learning to model partner behavior in response to different environment pressures. To do so, they rely on several factors, from the time an agent takes to accomplish a subtask, to dimensionality reduction techniques on its hidden states, to variations in reward and throughput between different training techniques. They find that agents are capable of generalizing to meaningfully different partners without hidden state resets, that agents following egocentric observations still are capable of modeling partner behavior, and that the hidden state representations for these PBT agents cluster agent behavior much better than baselines.", "questions": "- Your findings are demonstrated exclusively in Overcooked-AI. Can you provide evidence or discussion regarding how the observed emergence of partner modeling might generalize to other cooperative domains, especially those with more complex or less structured task decompositions (e.g., sequential social dilemmas or zero-sum games implemented in JAX frameworks)? Are there plans or preliminary results for such extensions?\n- The paper focuses on relatively simple partner representations (task speeds for two subtasks). How would your approach handle more complex or overlapping subtasks, or scenarios where partner traits are less easily parameterized? Have you considered or attempted experiments in such settings, and do you expect reward-driven implicit modeling to scale in fidelity compared to explicit partner modeling?\n- The related work section is somewhat limited, missing literature on partner modeling, zero-shot coordination, representation learning, and especially meta-learning. Given that RNNs trained with diverse partners are effectively performing memory-based meta-learning, can you expand the literature review to better situate your contribution within these fields? Are there key works from cognitive science, anthropology, economics, or RL that would help motivate your hypothesis that prediction is central to intelligence?\n- Have you compared your RNN-based implicit partner modeling to models that use explicit partner modeling modules? For example, how do value estimates, adaptation speed, or representation fidelity differ between your approach and architectures with explicit partner modeling? Is the mapping from hidden state to partner speed similar to that from explicit models?\n- Your discussion could be enriched by explicitly addressing the relationship between your findings and large language models (LLMs), particularly those trained via RLHF on human preferences. Do you see parallels between the emergence of partner modeling in your agents and the behaviors of LLMs in collaborative or alignment settings?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- Creative experiments. The pressures introduced such as partners completing tasks at different speeds and agents being blind but still following a cooperative reward seem like very clear cases where it might be optimal to predict a partner’s behaviors, and they are quite distinct from much prior work in Overcooked\n- The paper uses multiple metrics (reward, adaptation curves, linear probe accuracy, UMAP projections) and appropriate baselines (MLP, single-partner RNN, non-influential RNN) to support its claims\n- Results include confidence intervals and are evaluated across multiple seeds and layouts, enhancing reliability\n- The paper promises detailed experimental details, code, and data in the supplementary material, supporting reproducibility\n- The demonstration that \"blind\" agents can still develop partner models based only on egocentric signals and reward is a significant and surprising result, as well as the results on reset-free online partner adaptation, which is a much more natural form of cooperation\n# Weaknesses\n- All experiments are conducted in Overcooked-AI. While this is a rich environment, the generality of the findings to other cooperative domains or real-world settings is not demonstrated, and it would be particularly interesting to apply this evaluation to sequential social dilemmas (https://arxiv.org/html/2503.14576v1) or zero-sum games in Jax\n- The partner modeling representations are fairly simple (task speeds) and presuppose there being two subtasks. However, in more complicated settings with overlapping subtasks, it would be interesting to see whether reward alone leads to predictive representations that have the same fidelity as explicit partner modeling\n- The related work and overall references to prior work feels rather light, there is a large body of work on partner modeling, zero-shot coordination, and representation learning in RL. While some of the key papers are touched on, it might serve the authors better to motivate their hypothesis more in the introduction by drawing from a wide range of literature in cog sci, anthropology, economics, and RL which suggest prediction may be at core of intelligence and that it’s an optimal strategy. Relatedly, the meta-learning literature isn’t referenced, but it could really enhance the literature review, especially since RNNs training to collaborate with many partners are effectively doing memory-based meta-learning.\n- In the discussion of impact, it might be nice to reference the relationship between these findings and LLMs. LLMs trained with RLHF on human preferences are a very similar setup to the agents in this paper, so this work may offer insight into why it works well or what LLMs may be doing under the hood.\n- The specific methodology used is incremental, albeit the application case is novel. Studying representations learned by agents in social settings is a very interesting use case, but there are not particularly novel techniques used to do that, even though the experiments are creative relative to what the Overcooked literature has historically featured.\n- No comparison to models using explicit partner modeling. It would’ve been nice to see how the value estimates differ between RNNs using explicit partner modeling and not using explicit partner modeling, and if the mapping from the hidden state to partner speed is similar as the ones from partner modeling\n- Nitpick: Doing this study with an RNN is fine, but with a transformer the authors could have done more than just predict speed, they also may have been able to track attention to different grid cells as a way of seeing if the memory-based agent is tracking where the other agent is or may be going implicitly \n- Nitpick: Figure 1 is very large, and this may be taking space that would be better used to make the other figures harder to read. Figure 2 for instance has a lot of information and the labels aren’t too legible, and 2a may be easier to follow as a bar graph colored by model type and the x ticks being the layout names. Reducing figure size but increasing text size and/or breaking the figures up should make for an easier read and let the authors include more results.", "was_revised": true, "final_justification": "I will keep a score of 4 and believe this paper should be accepted into the conference. It is a valuable insight which has implications for how we think about the design of other multi-agent systems such as LLMs, which might benefit from additional conversational pressures that require better anticipating others' behaviors. The paper could benefit from additional environments and baselines, but these reasons are not sufficient to justify rejection.", "rebuttal": "**Thank you for the thoughtful and constructive feedback.** We appreciate the recognition of our experimental design, evaluation, and key findings on implicit partner modelling. We have addressed all of the points raised, including conducting preliminary experiments in a second environment, and believe the paper is stronger as a result.\n\n### Weaknesses\n\n#### **1. Generalisation beyond Overcooked-AI**\n\nWe agree that demonstrating generalisation beyond Overcooked-AI is important. Our goal was to show that partner modelling can emerge under minimal assumptions in a rich cooperative domain. We expect the same reward- and influence-driven dynamics to arise in other settings where agents adapt to diverse partners under partial observability.\n\nIn response to this point, we conducted preliminary experiments in a second environment: a modified version of CoinGame (from the JaxMARL suite). In our version of the environment, the ego agent collaborates with a teammate to collect red and blue coins in a small gridworld, and is rewarded based on the total number of coins collected (i.e. fully cooperative reward function). While the ego agent can collect both coin colours, the teammate is only able to collect either red or blue – and, analogously to our Overcooked setup, the ego agent has an additional action allowing them to control which colour coins the teammate attempts to collect. Consistent with our results on Overcooked, we find the following:\n\n- RNN agent with influence + trained alongside both teammate types outperforms RNN without influence, RNN trained only alongside a single partner, and MLP (in terms of total coins collected) (Table 1)\n- In evaluation episodes with the influence RNN agent, the teammate spends close to 80% of their time pursuing the “correct” coin colour (relative to a ~50% baseline) (Table 1)\n- Teammate types are more (linearly) recoverable from the hidden states of RNN agents with influence than those without (Table 2)\n\n**Table 1** (mean/std taken over 5 training seeds and 1000 eval eps):\n\n| Agent              | Mean coins collected (std) | Mean teammate time on best task (std) |\n|--------------------|----------------------------|----------------------------------------|\n| MLP                | 17.2 (0.483)               | 0.529 (0.0139)                         |\n| RNN no-influence   | 15.5 (0.216)               | 0.485 (0.0000)                         |\n| RNN single-partner | 16.3 (0.515)               | 0.485 (0.0036)                         |\n| RNN full           | 25.7 (1.97)                | 0.793 (0.0471)                         |\n\n**Table 2** (mean/std taken over 5 training seeds and 1000 eval eps):\n\n| Agent              | Mean linear probe accuracy (std) |\n|--------------------|----------------------------------|\n| RNN no-influence   | 0.703 (0.0451)                   |\n| RNN single-partner | 0.639 (0.0215)                   |\n| RNN full           | 0.874 (0.0594)                   |\n\nThese results suggest that the phenomenon is not limited to Overcooked, but instead reflects more general properties of cooperative RL under the right environmental pressures. We will expand upon these results in the revised manuscript.\n\n#### **2. Handling more complex or overlapping subtasks**\n\nOur approach can scale to more complex or overlapping subtasks, but this depends on two key factors: (i) whether the environment applies sufficient pressure to represent nuanced or entangled partner traits (rather than allowing simpler heuristics), and (ii) how observable those traits are in the teammate’s behaviour. If partner traits are behaviourally identifiable and reward-relevant, we expect implicit modelling to scale accordingly. We have clarified this in the revised manuscript.\n\n\n#### **3. Related work**\n\nWe appreciate this point and will expand the related work section to better situate our contribution within partner modelling, meta-learning, representation learning, and cognitive science. To support the view that prediction is central to intelligence, we are incorporating several key references, including:\n\n- **Clark (2013)** – *Whatever next? Predictive brains, situated agents, and the future of cognitive science*.  \n  *Behavioral and Brain Sciences, 36(3), 181–204.*  \nArgues that cognition and action are driven by hierarchical prediction.\n\n- **Byrne & Whiten (1988)** – *Machiavellian Intelligence: Social Expertise and the Evolution of Intellect in Monkeys, Apes, and Humans*.  \n  *Oxford University Press.*  \nSuggests human intelligence evolved for social prediction, motivating partner modelling.\n\n- **Harsanyi (1967)** – *Games with Incomplete Information Played by “Bayesian” Players*.  \n  *Management Science.*  \nFormalises agents as reasoning over others’ hidden traits.\n\n- **Wang et al. (2016)** – *Learning to Reinforcement Learn.*\n    *Proceedings of the 38th Annual Conference of the Cognitive Science Society (CogSci), 1315–1320.*  \n    Demonstrates that recurrent policies can learn internal learning algorithms, enabling fast adaptation to novel tasks—supporting our use of recurrence for implicit partner modelling.\n\n- **Grosz & Kraus (1996)** – *Collaborative Plans for Complex Group Action*.  \n  Highlights that effective collaboration requires commitment to shared goals, even under decentralised or partial knowledge.\n\n\nWe are happy to incorporate any additional suggestions for relevant work.\n\n#### **4. Connection to LLMs and RLHF**\n\nWe thank the reviewer for raising this connection and have added a discussion to the manuscript. Our findings suggest a parallel between emergent partner modelling in recurrent agents and the internal representations learned by large language models (LLMs) trained via reinforcement learning from human feedback (RLHF). In both cases, models receive only a scalar reward signal (team reward in our case, human preference in RLHF) and nonetheless develop internal representations that support effective interaction. Notably, this modelling arises without explicit objectives for partner inference or prediction, indicating that social reasoning can emerge as a byproduct of reward optimisation under the right pressures.\n\nA particularly relevant point of contact lies in the role of influence. We find that partner modelling emerges more robustly when agents can influence partner behaviour—for example, by assigning tasks. This mirrors findings in LLMs, where richer user modelling appears in interactive, multi-turn settings, where the model’s outputs shape future feedback (Ouyang et al., 2022;). Together, these results point to a general principle: when a learner’s reward depends on the actions of others and the learner can influence those actions, internal models of others are likely to emerge—even in the absence of explicit modelling objectives or architectural biases.\n\n\n#### **5. Methodological novelty**\n\nWe appreciate this observation — we agree that both the analysis methods and agent architectures are standard, and we chose them intentionally. Our goal was to demonstrate that structured, partner-specific representations can emerge robustly — not as an artefact of architectural choices or analysis techniques, but as a consequence of the collaborative training setting. That said, we agree that exploring richer architectures or novel analysis methods would be a valuable direction for future work. We will add a note on this in the revision.\n\n#### **6. Explicit vs implicit partner modelling**\n\nWe did not compare against explicit architectures, as our aim was to test if implicit modelling emerges in standard agents without scaffolding. We would expect explicit approaches to perform better on the specific traits they target, but the strength of the implicit RNN approach lies in its flexibility: it can represent any partner attribute that supports reward maximisation. A direct comparison — in terms of adaptation speed, value estimates, and generalisation — would be a valuable direction for future work. We will add a note on this in the revision.\n\n\n#### **7. Transformer architecture**\n\nThank you for this point—it's a good suggestion and we agree it opens up a very interesting direction for future work. Transformers could support additional analyses, such as examining attention over grid cells to investigate whether the agent implicitly tracks the other’s position or trajectory.\n\nIn this work, we chose an RNN because it is a standard architecture for memory-based agents in MARL (Foerster et al., 2016), and offers simple, interpretable internal dynamics. Our aim was to explore whether structured representations emerge without strong architectural inductive biases, and the RNN is well suited to that goal.\nWhile transformers could also be applied here, they typically require larger datasets or pretraining to be effective (Chen et al., 2021), which makes it less suitable for our setting. Additionally, to our knowledge, transformers are mainly used in offline RL (Chen et al., 2021). That said, using transformers to probe agent representations (particularly via attention) would be a promising direction for future work, and we note this in the revised paper.\n\n#### **8. Formatting**\n\nThank you for this point - we have updated the formatting of the paper accordingly to make the figures easier to read and to help include more results.\n\n\n### References\n\nChen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., & Abbeel, P. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. Advances in Neural Information Processing Systems, 34, 15084–15097.\n\nOuyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Christiano, P. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.\n\nFoerster, J. N., Assael, Y. M., de Freitas, N., & Whiteson, S. (2016). Learning to Communicate with Deep Multi-Agent Reinforcement Learning. Advances in Neural Information Processing Systems, 29."}, {"review_id": "6vJ0QQWqb5", "reviewer": "Reviewer_we3J", "summary": "This paper investigates whether artificial agents can spontaneously develop internal models of their partners' abilities during cooperative tasks, even without explicit architectural components for modeling others. They find that agents develop structured hidden representations of their partners’ competencies, enabling better task allocation and generalization to unseen partners. Crucially, this modeling emerges only when the agent has some influence over task assignment.", "questions": "- See weakness above.\n\n- Why is an RNN used instead of attention? A recent work [1] analyzes collaboration with unseen agents through attention weights. How does this work compare with [1]?\n\n- How does the ego agent collaborate with humans?\n\n- The experiments are conducted in two-agent settings. How does an ego agent collaborate with more agents? Do the current experimental results still hold true? Could the author provide a scalability analysis?\n\n- If the number of patterns is very large (e.g., 20 or 50 patterns), will the ego agent still be able to encode meaningful representations?\n\n- In the partially observable environment, how does the observation radius affect an ego agent’s ability to model partners?\n\n[1] Inverse Attention Agent for Multi-Agent System", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n- The paper is well-written and easy to follow.\n- The use of UMAP visualizations and linear probing to decode partner traits from hidden states provides convincing evidence for internal partner representations.\n- The paper includes multiple controlled settings to isolate and validate the conditions under which modeling emerges.\n\n### Weaknesses:\n\n#### Limited Experiments\nMany MARL papers conduct experiments on at least two environments and with multiple settings. Do the findings hold true in other environments and under different settings?", "was_revised": true, "final_justification": "The authors have addressed most of my conceptual questions. I still believe that the paper could benefit more from additional experiments on (1) collaboration with humans, (2) scalability with the number of agents, (3) partially observed environments, (4) additional environments, and (5) collaboration with or comparison to attention-based agents.\n\nNonetheless, the current paper is well-written and easy to follow. It clearly analyzes the current experimental results. Thus, I am recommending a borderline accept.", "rebuttal": "**Thank you for your thoughtful feedback on our paper.** We appreciate your recognition of the paper’s clarity, the use of analytical tools like UMAP and linear probing, and the careful design of controlled experiments. We’ve addressed all of your points, including obtaining preliminary results in a second environment to strengthen the empirical validation of our claims. We have incorporated, or are incorporating, your questions and suggestions into the revised manuscript, which we believe is stronger as a result and hopefully addresses your concerns.\n\n### Questions\n\n#### **Q1 Why is an RNN used for attention…**\n\nThank you for bringing this work to our attention. Long et al.’s paper shares a similar motivation (representation and theory of mind in multi-agent settings), but their approach is different. While we focus on implicit partner modelling, they train agents explicitly to predict/infer each other’s attention. We will update the related work section to clarify this distinction.\n\nRegarding the broader point about RNNs versus attention/transformers, in this work, we chose an RNN because it is a standard architecture for memory-based agents in MARL (Foerster et al., 2016), and offers simple, interpretable internal dynamics. Our aim was to explore whether structured representations emerge without strong architectural inductive biases, and the RNN is well suited to that goal.\n\nWhile transformers could also be applied here, they typically require larger datasets or pretraining to be effective (Chen et al., 2021), which makes it less suitable for our setting. Additionally, to our knowledge, transformers are mainly used in offline RL (Chen et al., 2021). That said, using transformers to probe agent representations (particularly via attention) would be a promising direction for future work, and we now discuss this explicitly in the revised manuscript.\n\n#### **Q2. How does the ego agent collaborate with humans?**\n\nWe do not evaluate collaboration with human partners, but this is an important direction for future work. A key challenge is mutual adaptation: unlike fixed-policy partners, humans adjust in response to the agent, changing the learning dynamics. Understanding how internal representations evolve under such bidirectional influence would be especially valuable. We have updated the manuscript to highlight this as a promising area for future work.\n\n#### **Q3. Scaling to more agents**\n\nThis is an interesting direction for future work, and we have updated the manuscript to discuss it more explicitly. We see no reason why the same kind of implicit representation would not scale to more than one teammate, provided the environment imposes sufficient pressure—i.e. all teammates’ behaviour meaningfully affects task success. That said, inference becomes more demanding with additional agents, as more evidence is required to identify their individual properties. The ego agent may adopt useful approximations, such as modelling the average behaviour of its teammates or inferring which teammate is best suited to each subtask.\n\n#### **Q4. If the number of patterns (?) is very large…**\n\nThanks for the question. In our experiments, the ego agent is exposed to over 50 unique partner pairings, with most variation along the axes of speed and randomness. As partner diversity increases across other behavioural dimensions, we would expect the agent’s internal representations to expand accordingly—encoding not just speed, but also traits like reliability, preferences, or coordination style. Exploring this broader space is an interesting next step. We will update the manuscript to highlight this as an interesting future direction.\n\n#### **Q5. In the partially observable environment…**\n\nIn a partially observable environment, the ego agent can't always see its partner and must rely on memory to infer their behaviour. It can still model the partner, but this takes longer, as observable evidence of the partner’s attributes is more sparse. With a radius of 1 (a 3×3 grid centred on the agent), the agent gets some local observations and can adapt fairly quickly. In the blind case, with no visual input, adaptation is slower (but does still occur). The RNN was crucial here, allowing the agent to use past experience to model both the partner and the environment state—for example, it may try to pick up soup, notice no onion arrives, infer the partner is incompetent, and adjust its strategy accordingly.\n\n### Limitations / weaknesses\n\n#### **Do the findings hold true in other environments and under different settings?**\n\nTo address this point and test the robustness of our findings, we have conducted a preliminary experiment in an additional environment; and we will expand upon these results in the revised manuscript. \n\nWe agree that demonstrating generalisation beyond Overcooked-AI is a critical next step. Our goal in this paper was to establish the emergence of partner modelling under minimal assumptions, using a rich cooperative domain that allows for clear analysis of agent memory and adaptation. That said, we expect the underlying mechanism — implicit modelling driven by reward and influence — to be broadly applicable. We believe similar dynamics are likely to emerge in other cooperative settings where agents must adapt to diverse partners, especially under partial observability.\n\nTo validate this, we conducted preliminary experiments in a second environment: a modified version of CoinGame (from the JaxMARL suite). In our version of the environment, the ego agent must collaborate with a teammate to collect red and blue coins in a small gridworld, and is rewarded based on the total number of coins collected (i.e. fully cooperative reward function). While the ego agent can collect both coin colours, the teammate is only able to collect either red or blue – and, analogously to our overcooked setup, the ego agent has an additional action allowing them to control which colour coins the teammate attempts to collect. Consistent with our results on overcooked, we find the following:\n\n- RNN agent with influence + trained alongside both teammate types outperforms RNN without influence, RNN trained only alongside a single partner, and MLP (in terms of total coins collected) (Table 1)\n- In evaluation episodes with the influence RNN agent, the teammate spends close to 80% of their time pursuing the “correct” coin colour (relative to a ~50% baseline) (Table 1)\n- Teammate types are more (linearly) recoverable from the hidden states of RNN agents with influence than those without (Table 2)\n\n**Table 1** (mean/std taken over 5 training seeds and 1000 eval eps):\n\n| Agent              | Mean coins collected (std) | Mean teammate time on best task (std) |\n|--------------------|----------------------------|----------------------------------------|\n| MLP                | 17.2 (0.483)               | 0.529 (0.0139)                         |\n| RNN no-influence   | 15.5 (0.216)               | 0.485 (0.0000)                         |\n| RNN single-partner | 16.3 (0.515)               | 0.485 (0.0036)                         |\n| RNN full           | 25.7 (1.97)                | 0.793 (0.0471)                         |\n\n**Table 2** (mean/std taken over 5 training seeds and 1000 eval eps):\n\n| Agent              | Mean linear probe accuracy (std) |\n|--------------------|----------------------------------|\n| RNN no-influence   | 0.703 (0.0451)                   |\n| RNN single-partner | 0.639 (0.0215)                   |\n| RNN full           | 0.874 (0.0594)                   |\n\nThese results suggest that the phenomenon is not limited to Overcooked, but instead reflects more general properties of cooperative RL under the right environment pressures.\n\n### References\n\nChen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., & Abbeel, P. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. Advances in Neural Information Processing Systems, 34, 15084–15097.\n\nFoerster, J. N., Assael, Y. M., de Freitas, N., & Whiteson, S. (2016). Learning to Communicate with Deep Multi-Agent Reinforcement Learning. Advances in Neural Information Processing Systems, 29."}, {"review_id": "LaOmLd2P7j", "reviewer": "Reviewer_ocq2", "summary": "This paper explores whether and when recurrent agents, trained to perform a collaborative task with another agent, develop representations of their partners \"skills\". The authors find that, when the recurrent agents are able to interact with their teammate (through assigning their teammate one of two tasks), encoding of the partner's skills emerges in the recurrent agents. The authors further demonstrate that their recurrent agents are able to adapt when their teammate switches skills halfway through the trial. This work provides evidence that artificial models can develop representations relevant for collaborative tasks, even in the absence of any added architectural features.", "questions": "Q1 - Why do the authors think the RNN struggled (relative to the baseline models) on the configuration fivebyfive_v1?", "limitations": "The authors do a good job addressing their work's limitations. The only exception to this is that they do not note that linear decoding is just one way in which to measure the representation of the partner's skills in their RNN model. Other, more sophisticated strategies of decoding could yield higher accuracy, even for the baseline models.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "STRENGTHS: \n\n1. This paper was especially well motivated and written. It was a pleasure to read. \n\n2. The experimental design was, for the most part, very well done and allowed for clear conclusions. I especially appreciated the well chosen controls/baselines that the authors compared their recurrent agents against.  \n\n3.  This work tackles a really important question and the results have important implications for how we understand what is needed to design artificial agents that can cooperate effectively. I especially like the way the authors put it - \"social intelligence can emerge from specific environmental pressures acting in concert with general mechanisms for learning and memory, rather than necessarily relying on unique architectures\". I find it likely that this work will help shape a lot of future research. \n\n4. The future directions noted by the authors are interesting and help frame the impact of the work. \n\nWEAKNESSES: \n\n1. While strong, I think the details on the experiments could be improved in a few specific instances: \n    a. The authors mention (in Sec. 5.1) that the recurrent agents were able to generalize to partners that had been unseen in training. However, there is some overlap in the skill range that is sampled during learning ({1, 2, 3, 4, 7, 9}) and the skill range that is sampled during testing ({0, 2, 3, 8, 10}). Do the authors explicitly ensure that the same partner configuration can't emerge in both train and test (e.g., v = (2, 3))? If not, that's fine, but this overlap should be noted. \n    b. The online adaptation to a partner that switches is a cool experiment and the way of testing by switching from the partner being better at task 1 and then task 2 at the halfway point makes the analysis clear. But, I was unclear if the recurrent agent was trained on a similar set-up. That is, if it was trained on partners that switched at the halfway point and went from being good at task 1 to task 2. If so, the recurrent agent could learn to switch its policy without actually paying attention to its partner. I think this is not what the authors are doing, but it would be helpful to get more details. \n\n2. While I agree with the authors that their results demonstrate clearly that recurrent agents trained on a diversity of partners outperforms the other baseline models, the single-partner RNN and the no-influence RNN both appear to have information in the recurrent activations that can be used to determine their partner's skills (Fig. 3B - above chance performance for all models). In addition, the authors use a simple linear decoder - there may be more complex decoding schemes in which the activations of the baseline models can lead to even higher accuracy. Therefore, saying that partner modeling only emerges when the RNN can interact with its partner feels a little misleading. I think re-phrasing the results to say that partner modeling emerges more strongly when the RNN can interact with it's partner would be better. \n\nCOMMENTS: \n\n1. The UMAP analysis of the hidden activations is interesting, but I think other analysis could help complement this and make the conclusions stronger. For instance, computing the correlation between each hidden unit's average activation and the partner's difference in skill (task 1 - task 2) could lead to the identification of whether the partner modeling occurs at the level of individual units or at the level of the population. The authors could also apply Dynamic Similarity Analysis to identify whether different partner's skills lead to different RNN dynamics. \n\n2. I was intrigued by the fact that the recurrent agent did better than the baseline models on every configuration except fivebyfive_v1 (Fig. 2). Could the authors comment on why this might be? \n\n3. There are a few papers that the authors may be interested in (but, to be clear, are not obligated to cite): \n    a. https://elifesciences.org/articles/85694 - This work shows that RL agents can learn to cooperatively hunt (and \"assign\" roles) without any architecture bias, echoing the authors results. \n    b. https://www.nature.com/articles/s41586-018-0102-6 - This work shows that RNN agents trained to perform path integration (i.e., to update where in space the agent is given a sequence of movement directions and speeds) develops computations that are like that in the brain. Leveraging these representations with RL leads to policies that outperform naive RL agents. \n    c. https://proceedings.neurips.cc/paper_files/paper/2024/hash/285b06e0dd856f20591b0a5beb954151-Abstract-Conference.html - This work demonstrates that an RNN model that is trained to path integrate only one agent is not optimal for path integrating two agents, illustrating the need for specific computations in multi-agent settings.", "was_revised": true, "final_justification": "I recommend accepting this paper as it is of high quality and interest to the NeurIPS community", "rebuttal": "**Thank you for the detailed and thoughtful feedback.** We’ve clarified the setup, improved the analysis, and incorporated the suggested related work. We appreciate your input and believe it has strengthened the paper.\n\n\n### Strengths\n\nWe appreciate the positive assessment of the motivation, experimental design, and broader implications of our work. We're encouraged that the framing aligns with future research directions.\n\n### Weaknesses\n\n#### **Weakness 1A: Train/Test Partner Overlap**\n\nThank you for the great point. While some individual speed values overlapped between training and test, the sets were disjoint in terms of speed pairs (e.g. (2,3)), as all speed pairs used during training were excluded from the test set. This ensured that the ego agent was never paired at test time with a teammate policy it had encountered during training. We apologise for the confusion, and will update the manuscript to make this clearer. \n\n#### **Weakness 1B: Online Adaptation and Training Set-up**\n\nThis is also a good point. During evaluation, to simplify the analysis, we always switched the teammate properties in a single direction and at t=300. During training, however, both the timing and the nature of the switch were randomised: the ego agent encountered teammates whose properties switched in both directions (i.e. faster at task 1 → faster at task 2, or vice versa), and t_switch was sampled uniformly between 25% and 75% of the episode length. Additionally, we performed the switch only in 50% of episodes. This was done exactly to mitigate the concern that the model might memorise a fixed timing or pattern, rather than truly adapting to the change online. We will also update the manuscript to clarify this.\n\n\n#### **Weakness 2: Partner Modelling and Decoder Analysis**\n\nThank you for the feedback; we acknowledge that our wording could be clearer and more specific here. While the baseline models do contain some partner-related information, as evidenced by above-chance decoding performance, the influencing condition shows a substantially stronger emergence of partner modelling. We will update the manuscript to clarify this.\n\n\n\n### Comments\n\n#### **Comment 1: Additional Analysis**\n\nThis is a good point. We chose the UMAP projections as a way to produce clear visual evidence of our findings, and the linear probe analysis for its conceptual simplicity. That being said, we agree that additional analyses would help strengthen our conclusions. The idea of looking at per-unit correlations to better understand at what ‘level’ the representations are being learned is appealing, and has a nice mechanistic interpretability-like flavour. We also thank the reviewer for pointing us to DSA, which we were not previously familiar with. We will make sure to incorporate at least one additional analysis approach into the revised manuscript. \n\n#### **Comment 2: Performance Drop on fivebyfive_v1 Configuration**\n\nThis is a good observation - we believe the exception in fivebyfive_v1 highlights that partner modelling is environment dependent. In fivebyfive_v1, spatial coordination is less critical than in cramped_room variants (due to the simpler and more open geography), making feedforward models more competitive. Additionally, these plotted results are from the fully observable setting, where agents can often infer their partner’s behaviour from the current state alone — reducing the benefit of temporal modelling. In contrast, the performance gap to MLPs is larger in the partial and blind settings, where temporal reasoning becomes more important. Another possible factor is that agents are evaluated with novel partners. RNNs may overfit to training partners, whereas MLPs, lacking memory, may generalise better when spatial context is sufficient.\n\n#### **Comment 3: Related Work**\n\nThank you for sharing these relevant and insightful papers. We’ll incorporate all of the papers into the literature review to make it more comprehensive and to better situate our contributions within the broader context. It was nice to read the papers - they are relevant and great suggestions."}]

Author rebuttal:

Meta-review: This paper asks whether agents need explicit partner-modeling modules, or whether such representations can emerge spontaneously under the right social pressures. Using Overcooked-AI, the authors train recurrent RL agents with different partners and show that hidden states encode partner skill levels, which they suggest enables rapid adaptation and generalization. Results support the claim that partner modelling emerges most clearly when agents have influence over task allocation. The authors also extend the analysis with preliminary experiments in CoinGame, suggesting the findings are not limited to one domain.

Reviewers were broadly positive. They praised the clarity of the experimental setup, the strong empirical evidence, and the use of multiple probes (UMAP, linear decoding, adaptation curves) to support the claims. Several highlighted the work’s potential to inform research on social intelligence in artificial agents, noting parallels to human theory of mind and even LLM alignment. 

Concerns were mainly on scope and depth: all main experiments are in Overcooked, the representations are fairly simple (mostly task speed), and the literature review could be broader. Some reviewers also suggested comparing against explicit partner-modelling architectures or trying transformers for richer analysis, but agreed that the simplicity of the current approach strengthens the central claim.

After rebuttal, the authors added clarifications and preliminary results in CoinGame, addressed questions on training/test splits, online adaptation, and expanded their discussion of related work and future directions. Reviewers were satisfied and in some cases raised their scores. The contribution is limited in scope, but solid, and merits acceptance as a poster.
Decision: Accept (poster)