Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models
Authors: Logan Cross
Violet Xiang
Agam Bhatia
Daniel LK Yamins
Nick Haber
Venue: ICLR 2025 Poster (2025)
Keywords: LLM agents
language agents
theory of mind
multi-agent
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce Hypothetical Minds, an LLM-based agent that outperforms MARL and LLM baselines on multi-agent tasks using a novel Theory of Mind module

Abstract: Multi-agent reinforcement learning (MARL) methods struggle with the non-stationarity of multi-agent systems and fail to adaptively learn online when tested with novel agents. Here, we leverage large language models (LLMs) to create an autonomous agent that can handle these challenges. Our agent, Hypothetical Minds, consists of a cognitively-inspired architecture, featuring modular components for perception, memory, and hierarchical planning over two levels of abstraction. We introduce the Theory of Mind module that scaffolds the high-level planning process by generating hypotheses about other agents' strategies in natural language. It then evaluates and iteratively refines these hypotheses by reinforcing hypotheses that make correct predictions about the other agents' behavior. Hypothetical Minds significantly improves performance over previous LLM-agent and RL baselines on a range of competitive, mixed motive, and collaborative domains in the Melting Pot benchmark, including both dyadic and population-based environments. Additionally, comparisons against LLM-agent baselines and ablations reveal the importance of hypothesis evaluation and refinement for succeeding on complex scenarios.

Published as a conference paper at ICLR 2025
HYPOTHETICAL MINDS: SCAFFOLDING THEORY OF MIND FOR MULTI-AGENT TASKS WITH LARGE LAN- GUAGE MODELS
Logan Cross
Violet Xiang"
Agam Bhatia*
Daniel L.K. Yamins
Nick Haber
Stanford University
'Department of Computer Science
DDepartment of Psychology
3 Graduate School Education
ABSTRACT
Multi-agent reinforcement learning methods struggle with the nonstationarity of multi-agent systems and fail to learn online when tested with novel agents. Here, we leverage large language models (LLMs) to create an autonomous agent that can handle these challenges. Our agent, Hypothetical Minds, consists of a cognitively- inspired architecture, featuring modular components for perception, memory, and hierarchical planning over two levels of abstraction. We introduce the Theory of Mind module that scaffolds the high-level planning process by generating hy- potheses about other agents' strategies in natural language. It then evaluates and iteratively refines these hypotheses by reinforcing hypotheses that make correct predictions about the other agents' behavior. Hypothetical Minds significantly improves performance over previous LLM-agent and RL baselines on range of competitive, mixed motive, and collaborative domains in the Melting Pot bench- mark, including both dyadic and population-base environments. Additionally. comparisons against ..-M-tgent baselines and ablations reveal the importance of hypothesis evaluation and refinement for succeeding on complex scenarios.
INTRODUCTION
fundamental challenge artificial intelligence is creating agents that can rapidly adapt to oth- ers in multi-agent environments. Whether in competitive settings like games, collaborative tasks like team projects, or mixed-motine scenarios like negotiation, success requires building predictive models of another agent's behavior by inferring their hidden strategies and latent capabilities. One approach involves training reinforcement learning models that can learn adaptive policies. However, multi-agent reinforcement learning (MARL) methods suffer from various drawbacks, including high sample complexity, poor generalization to agents not seen in training, and limited reasoning capa- bilities (Wong al. 2023).
Another approach deploys foundation models as the backbone to agents, with specialized modules that mediate the decomposition long horizon planning (Wang et 2024 2023a Brohan al., 2023a Park et al. 2023). LLMs are not only powerful reasoners and in-context learners, but they are also particularly suited for social tasks given the utility of language for scaffolding Theory of Mind (ToM) (Astington & Baird] 2005; de Villiers) 2007] 2021 Kosinski] 2023] Gandhi et al.) 2024). Thus, we introduce Hypothetical Minds (HM), a LLM agent that produces adaptive policies in competitive, cooperative, and mixed-motive multi-agent scenarios. At its core, HM addresses the challenge of interacting with agents whose behavior is driven by latent features first inferring these features through natural language approximation of Bayesian inference, then conditioning its policy on these inferences.
"Corresponding author: I edu Code: http:///ttttt...ooooooo///////tt

---

Published as a conference paper at ICLR 2025
A
Deparils mndladt
Memory
Population
Events and rewards ds from past observations
Theory Mind Module'
TiTapeeet witsoes
nosoyettad thair selssrs
Previously seen
selegeellle
Subgoal module*
Observations
Φ(T)
Textual map (partiall--bbsevebble
Action Planner
Environment
Ego View
fez
Preprocess egocentric
*calls
LUN
Theory of Mind Module
infire ald pothoess
bekinder
Figh stuthy
best
aha bataniar doveried
LLM
LLM
fran nenery
canditioned typothents
High -aval Ktan
((T))
Vakkdde typothends prodethess
selggee medale
Figure 1: A. Hypothetical Minds architecture and model workflow. B. ToM module generates hypotheses about agent strategies. Previously generated hypotheses and values are shown for refine- ment. Top k hypotheses predict agent's next behavior Φ(T). considering counterfactual scenarios. Highest-valued valued hypothesís informs high-level planning. Later, hypotheses are evaluated against ob- served behavior é(T), updating values with intrinsic reward. Hypotheses are validated at a threshold.
HM builds the generative agents architecture, integrating modules for perception, memory, and hierarchical planning with novel ToM machinery (Park et al. 2023). The Theory Mind module generates and evaluates hypotheses about other agents' strategies, scoring them based on their ability to predict future actions. Thus, our agent finds useful explanations of other agents' behaviors in- context, affording it the ability adapt to the inferred strategies and achieve high rewards.
We evaluate our model on the Melting Pot MARL benchmark that encompasses diverse challenges and social dynamics (Agapiou et al. 2022]. Collaborative Cooking Asymmetric requires effec- tive coordination and division of labor among agents. Running With Scissors necessitates strategic reasoning about opponents' policies and the ability to exploit predictable patterns in competitive setting, with the eight-player version offering a unique challenge to the model's scalability. More- over, the mixed-motive environment Prisoner's Dilemma involves a tension between individual and collective interests. Diverse evaluation scenarios stress test playing with wide array of agents with fixed or adaptive policies, necessitating contextual adaption. Hypothetical Minds surpasses LLM agent baselines majority of evaluation scenarios in every environment, and performs better than MARL baselines on 3/4 environments despite the fact that those methods are trained on upwards of billion steps. To our knowledge, Hypothetical Minds is the first LLM agent for diverse multi-
2

---

Published as a conference paper at ICLR 2025
agent environments, with strong performance across collaborative, competitive, and mixed-motive domains, unlike previous works limited to collaborative settings. Our contributions are as follows:
We propose the Hypothetical Minds model (HM), an embodied L.MM agent for multi-agent environments that integrates modular components for perception, memory, and hierarchical planning conditioned on ToM inferences.
HM incorporates novel Theory of Mind (ToM) module, which generates, evaluates, and refines hypotheses about other agents' strategies or goals in natural language. Through ablations and comparisons against LLM-agent baselines, we identify the critical role of hypothesis evaluation and refinement within the ToM Module.
We demonstrate the effectiveness of HM Minds across multiple multi-agent environments, including competitive, collaborative, and mixed-motte domains, and 30 distinct evalua- tion scenarios. Our agent significantly outperforms LLM-agent and RL baselines in every environment and in a large majority of evaluation scenarios, showcasing its generalizability.
2 RELATED WORK
2.1 LLM-BASED AGENTS
A burgeoning area of research involves building autonomous agents rooted in large language models Wang al. (2024) : Sumers et al. (202)). This involves deploying LLMs as central controllers
across many different domains by leveraging their extensive background knowledge from training.
Applications span a wide range from equipping LLMs with external tools to interface with databases
and APIs (Schick et al. 2024, Shhnn et al. 2023 Qin et al., (2023) to using them for high-level
planning and control in robotics (Huang et al. [022] Brohan al. 2023b Rana al. 2023 Brohan et al. (2233)). The most relevant branch of this research direction includes works where LLMs are used as planners embodied virtual environments. Voyager autonomously builds complex skills in Minecraft by storing and retrieving behaviors in a skill library of executable code and uses the skill library to solve progressively harder tasks (Wang et al. 2023a 2024). In this work, we use an LLM for long horizon high-level planning and predicting the future states of other agents in multi-agent environments. Previous papers have also incorporated LLM--ased agents into embodied multi-agent environments (Liu et al. 2023a). Park et al. (2023) introduce interactive simulation of rich social environment, where each agent autonomously selects goals and builds relationships with others. We extend the cognitive module framework developed in this work for multi-agent environments of varied dynamics by introducing the novel Theory of Mind module. SAMA uses an LLM to plan out sequences of subgoals for language-basee based goal-conditioned RL policies in environments requiring multi-agent coordination Li et al. (2023). Another study builds cooperative embodied agents, by using an LLM for planning and communication between agents Zhang et al. (2023b). ProAgent develops method for improving zero-shot coordination in Overcooked by using an LLM to infer the intentions of teammates based on the present state Zhang et al. (2023a). As these works have focused solely on collaborative domains, here we present generalizable and scalable method that addresses the challenge of inferring other agents intentions' across a wide spectrum of social dynamics.
2.2 REASONING AND HYPOTHESIS SEARCH WITH LLMS
LLMs have shown impressive reasoning abilities, augmented by Chain-of-Thoughh methods that scaffold the thought process (Wei et al. 2022: Zhang et al. 2023c). Our hypothesis generation ap- proach builds on this work by implementing structured reasoning steps for theory of mind inference Tree-of- oheouuhh (Yao et al.) (024)) extends chain-of-thought by maintaining
and decision-maker and evaluating multiple reasoming paths in parallel, similar to our parallel evaluation of multiple hy- potheses about opponent strategies. ToT evaluates different solution paths of a reasoning trajectory with an external verifier or LLM as judge, whereas we score hypotheses about an agent based on the quality of predictions they make about that agent's behavior. The structured prompting in ToM module draws on broader findings about effective prompt engineering for complex reasoning Zhang et al. 2023c). For example, separation between reasoning and action in prompts draws inspiration from the ReAct framework (Yoo et al. 2023).
Recent work has specifically investigated LLMs' capabilities for hypothesis generation and testing.
Wang et al.] (2023b) explores LLMs' inductive reasoning by generating and evaluating hypotheses on the Abstraction and Reasoning Corpus (ARC), while Qiu al. (2023) refines I.L.M-eeerated
3

---

Published as a conference paper at ICLR 2025
hypotheses using task-specific symbolic interpreters. Similarly, we generate, evaluate, and refine hypotheses based on feedback, though in our case computing values for each hypothesis by predict- ing another agent's goals. STaR (Zelikman et al., 2022) also learns from feedback by finetuning language models on rationales that produced correct answers.
2.3 MULTI-AGENT DECISION-MAKING AND THEORY OF MIND
Decision-making in multi-agent settings has been extensively studied through multi-agent reinforce-
ment learning (MARL,, with benchmarks like Melting Pot providing diverse social interaction sce- narios to evaluate agents (Agapiou et al. 2022). While algorithms like MADDPG (Lowe et al. 2017) and MAPPO (Yu al. 2022a) have advanced MARL through centralized training approaches, spe- cialized models incorporating Theory of Mind principles (Rabinowitz et al. 2018: Wang et al. [ 2022) have shown promise in modeling other agents' intentions. However, even sophisticated approaches like OPRE /Vezhnevets et al. (020)) struggle to generalize in complex environments like Melting Pot's Running
Yu et al. 2022b Huh & Mohapatra 2024; Vezhnevets al. 2020; [Selar al.
With Scissors (Agapiou et al. 2022), highlighting MARL's limitations in compute requirements and training stability. In cognitive science, the Rational Speech Act framework (Goodman & Frank 2016) and Bayesian inverse planning (Baker et al.] (2099 model how humans inter others' goals and beliefs through Bayesian inference, providing inspiration for our ToM module.
3
METHOD
3.1
PARTIALLY-OBSERVAALL MARKOV GAMES
Our method is directly applicable to any multi-agent environment where states are partially observ- able and agent(s)' policies are hidden. We formally define this as a Markov game for N players in partially observable setting. Let the finite set S represent the possible states of the game. Each player receives observations given an observation function x - O, representing their limited point of view. Additionally, each player can take actions from their action space A', and when all players choose actions (a .a........ a N) E A' X ... X A := A, the state transitions according to a probability distribution T: x A D(S). The reward function for each player i is represented as r' : S x A >> IR, mapping the current state and joint actions to a real-valued reward.
3.2 HYPOTHETICAL MINDS MODEL
HM consists of several cognitive modules that altogether form an embodied LLM agent (Figure The egocentric observations are represented by a textual map/state representation, which is added to a memory system after every step. The memory system also logs rewards and other important state information like the inventories from previous interactions in Running With Scissors. Two cognitive modules depend on an LLM, a Theory Mind module and a Subgoal module, which output high-level goals and action plans respectively. An action planner takes action plan (i.e. "move to coordinate (13, 5)") and creates a sequence of actions that achieves that action plan with a pathfinding algorithm. Each cognitive module is explained in more detail in the Appendix. Below we describe the key novel contributions of our method that implement hierarchical planning.
Theory of Mind Module In multi-agent environments, other agents' behavior can be influenced by various latent variables, such as their strategies (e.g.. "always defect"), goals (c.g., "trying to pick up an object"), competence levels (e.g.. "highly skilled" vs. "negligent") and locations in an environment. These latent variables are often not directly observable and must be inferred from partially-observabbe information from your perspective as the ego agent. We represent these latent variables as multidimensional space θ = 0,, 02.02.... Om. where each dimension 0: corresponds to a specific latent variable in an embedding. The ToM module generates hypotheses about these latent variables in natural language. It then maintains set of hypotheses H. = h1, .2..... A, in its working memory, where each hypothesis f;2 represents a belief about the latent variables h, = p()). A hypothesis time is generated by asking an LLM to infer another agent's strategy, conditioned on HM's memory M of important past observations O. Additionally, the LLM is shown the top k valued previously generated hypotheses, such that it can perform hypothesis refinement (see Appendix and code for more details and prompts):
h LLM(M, hcl)
4

---

Published as a conference paper at ICLR 2025
where M is memory buffer storing agent's past actions, observations, and rewards.
Each hypothesis h2 is scored based on how well it predicts the other agent's 's future behavior, noted here formally distribution of trajectories p(7). We formalize this scoring mechanism using likelihood function p(r[h;; representing the probability of an agent exhibiting trajectory T given the hypothesis hę. The best hypothesis h" is selected using the Maximum Posteriori (MAP) estimate:
p(T ((T())))((h))
h" = arg max p(h,/T) = arg max hcEH
(2)
p(T)
KI
where p(h:) is the prior probability of hypothesis hú and p(T) the marginal probability of the observed action a and has no effect on the argmax. The likelihood is approximated by a hypothesis evaluation mechanism described below. The LLM predicts the other agent's future behavior con- ditioned on each hypothesis separately. Hypotheses leading to correct predictions will have higher values reflecting higher likelihoods. The prior p(h:)) corresponds both to the background knowledge embedded in the weights of an L.M from pretraining and to the refinement mechanism that shows the top valued hypotheses to the LLM when the LLM generates a hypothesis. By continuously updating and selecting the best hypothesis based on observed information, the ToM module can effectively infer the latent variables governing the other agents' behavior and adapt its own strate- gies accordingly. When the environment has more than other agent, the ToM module maintains separate hypothesis streams for each other agent, with unique hypotheses and values tracked per agent.
Drawing cognitive modeling approaches (Rescorla 1972] Daw & To- bler 2014), multiple hypotheses are scored with a value system VA, = E[r] where reflects in- trinsic reward based on the accuracy of the predictions the hypothesis generates. We compute self- supervised intrinsic rewards bootstrapped from the LLM's own predictions. Let ((() be particular behavior, feature from observed trajectory, and d(T) be the predicted behavior by the LLM, such as the inventory played by an agent in Melting Pot environments. The intrinsic reward function r, can then be defined as:
Hypothesis Evaluation
if (TT()) (TT) -c if (TT(T) + ((T)
where e a hyperparameter. Vn, is then dynamically updated with a Rescorla Wagner update rule (Rescorla 1972). expressed
8 8= == Vh. VA, + Vn, 8
modulated by learning rate via a prediction error 8. The learning rate dictates how much to weigh recent interactions, a useful property when playing against evaluation scenarios where agents change their strategy within episode. When the value of hypothesis meets a threshold Vine the ToM module marks the hypothesis as validated and uses this hypothesis to condition high-level plans (using the highest-valued one if multiple pass the threshold). This hypothesis will then continue to be used for planning until it no longer makes good predictions and its value subsequently falls below the threshold. If no hypothesis meets the threshold, by default the latest generated hypothesis (with the most updated information) is used for conditioning high-level plans. The highest value hypothesis could also be used in our code. We set the validation threshold Vthr = 0.7 apriori based on analysis of the Rescorla- Wagner dynamics. With reward values of c and learning rate 0.3, a hypothesis requires consistent predictive accuracy to reach and maintain this threshold. The top k hyperparameter mediates cost trade-offs, representing how many of the top_k old hypotheses are continually evaluated in parallel. Each ToM module call uses (3 top_k) LLM calls if no hypothesis is validated yet and only 2 calls after validation, thus scaling linearly at the step in computational cost when hypotheses are not validated and sublinearly when they are frequently validated. (See Table 4 in Appendix for all hyperparameters and Table 5 and Appendix Section for information about computational costs). The default value as reported in the results use top_==5.
Conditioning High-Level Plans The ToM module then conditions its high-level plans on the inferred latent variables represented by the hypotheses. A high-level plan is a natural language
5

---

Published as a conference paper at ICLR 2025
With Repeated
Running
Reward
Reward
Mix Guilile
Rock Paper
ROck
Pager
S05s0rs
Miked Strabeay Rassoons
Best
Fip After 2
Gullele
Rock
Peper
Schssers
After
Evaluation Scenaria
Evaluation on Scenario
Collaborative Cooking Asymmetric
Prisoners Dilemma
Reward 5
Reward
Cospinsor
skilled Unhelpful
ludonhator
Face
Our
Out Faet 201
Evaluation Scenario
Evaluation Scenario
Hypothetical Minds
OPRE
PPO
ReAct
Reflexion
PlanReAct
Figure 2: Results for all models. Average reward per episode (with normalized steps for variable length episodes) for each environment and scenario. seeds are generating for each model, with errorbars reflecting the SEM across those 5 episodes.
description of HM's overall strategy, goal, intention, conditioned on the best hypothesis h" and memory of past events:
LLM(M,
By conditioning the high-level plans the hypotheses, HM can adapt its strategy based its understanding of the other agents' latent states.
Subgoal Module Finally, the Subgoal module selects sequence of subgoals. Let 91. ................................ 9k be a sequence of subgoals, where each subgoal is an action or short sequence actions that the agent needs take to achieve the high-level plan
g LLM
The sequence is generated by conditioning the L.LM on the high-level plan, observations, and mem- ory, and prompt to achieve the high-level plans. The LLM outputs sequence of specified subgoal function calls, which are then parsed and mapped to the corresponding actions in the environment by a hardcoded Action Planner (see Appendix).
4 EXPERIMENTS
Here investigate the following to analyze the generalizability and scalability of our method:
Q1. How does Hypothetical Minds perform compared to LLM agent and RL baselines in embodied competitive zero-sum environments?
Q2. How does Hypothetical Minds perform compared to LLM agent and RL baselines a collabo- rative domain that requires adaptation to a partner's role and competence?
Q3. How does Hypothetical Minds perform compared to LLM agent and RL baselines in a mixed- motive setting?
Q4. Does the Hypothetical Minds agent scale effectively to enviromments with larger populations of agents?
QS. How do the different components of the Hypothetical Minds agent and the Theory of Mind module contribute to its overall performance?
We directly test our LLM-bbsed agent on the evaluation scenarios in four Melting Pot environments FFiuure 2). The key evaluation method is to test agents against different bots with various policies.

---

Published as a conference paper at ICLR 2025
Reward Per Interaction Relative to Hypothesis Validation
2-
M
Reward
95% of when good hypothesis discovered
Hypothesis velidated
-10 Interaction Number Relative When Hypothesis is Validated
10
Figure 3: HM's reward per number of interactions before or after hypothesis meets the validation threshold and used for high-level strategy selection RWS. Vertical green line indicates the average reward at the point where hypothesis is validated, and positive and negative numbers on the x-axis indicate how many interactions before or after this point. Shaded region represents the range where the good hypothesis is typically first generated with a 95% confidence interval.
Across environments, this consists of 30 distinct evaluation scenarios. Crucially, our agent has no knowledge about which strategies they may playing in the prompts given. Strategies have to be ascertained online within an episode via in-context learning.
Baselines
ReAct synergizes reasoning and acting in language models, allowing them to generate both reason- ing traces and task-specific actions an interleaved manner (Yao al. 2023). Reflexion includes three main components: Actor module that generates actions and text, an Evaluator that scores these actions, and a Self-Reflection module that uses the evaluations to provide constructive feed- back stored for subsequent use (Shinn et al. 2024)). PlanReAct To provide hierarchical baseline to test against our hierarchical model, we include the PlanReAct architecture introduced in (Liu et al., 2023b). This structure allows the agent to plan before interacting with the environment. PPO is model-free RL baseline (Schulman et al. 2017) and we train agents in a population of models with the same parameters. OPRE is hierarchical MARL method where agents learn high-level options as strategic responses to other agents' behaviors (Vezhnevets et al. 2020). The high-level controller selects options based on observations of other agents, and the low-level controller executes actions conditioned on these options. We include the OPRE results from the Melting Pot 2.0 paper (Agapiou jet al., 2022) as baseline to compare against a hierarchical MARI. approach that, like our model, aims to adapt dynamically to other agents' strategies.
Prompts and architectures are shared across baselines to provide fair comparison and natural abla- tions of our model. The Subgoal module provides a baseline actor shared between LLM baselines, and the only difference between PlanReAct and Hypothetical Minds is that high-level planning is mediated by the Theory of Mind module, including hypothesis generation, evaluation, and refine- ment.
4.1 How DOES HYPOTHETICAL MINDS PERFORM COMPETITIVE ENVIRONMENTS?
Running With Scissors in the Matrix Repeated (RWS) zero-sum competitive environment with two players moving around map and collecting yellow, purple, or blue resources that corre- spond rock, paper. and scissors respectively. Zapping your opponent causes an interaction, with one agent getting positive reward and the other agent getting opposite negative reward according to the inventories of resources picked up by each player, mirroring the rock, paper, scissors matrix game.
RWS presents nine distinct evaluation scenarios, which range in complexity. These include three straightforward strategies where opponents consistently play rock, paper, or scissors. The remaining scenarios introduce more dynamic and adaptive strategies (see Appendix for details and full list). Therefore in order to succeed on the scenarios, agent needs correctly infer the strategy and exploit it. Against the simple policies, the agent should play the same type of inventory every round rather than playing randomly anticipating change in its opponent's policy. In contrast, success
7

---

Published as a conference paper at ICLR 2025
against the adaptive strategies demands not only the anticipation of the opponent's next move based on personal previous plays but also selecting the most advantageous counter.
Figure 2 and Appendix Table demonstrate how Hypothetical Minds model consistently achieves large magnitude rewards and performs reliably better than the baselines on every single scenario. Hypothetical Minds performs the best the static strategies, scenarios 0, 6, 7, 8 representing fixed policies that play for the same inventory on every interaction (6: rock, 7: paper, 8: scissors, O: ran- dom sample from 6-8). Therefore it is able to exploit the static strategy reliably once it correctly infers it. The agent is also able to consistently return positive rewards against the difficult scenario 1, the adaptive bot that plays the best response to your last round. In contrast, baselines are failing to achieve consistent positive rewards. OPRE, method designed specifically for a earlier version of this task, gets rewards near zero for every scenario. PlanReAct is the only other model to achieve positive rewards on the difficult best response bot, illustrating the usefulness of hierarchical struc- ture for this evaluation in particular. Reflexion performs second best overall, underscoring the value of evaluative reflection for this environment.
Figure 3 showcases HM's dynamics, depicting the agent's reward per interaction before and after hypothesis validation. Upon validation, the agent consistently achieves high positive returns, while rewards are near zero or negative during the information -aataattnnnntheringgg phase. The upward trajectory in reward after generating a good hypothesis and the significant increase just before the validation threshold demonstrate the agent's s ability to exploit accurate hypotheses effectively.
4.2 How DOES HYPOTHETICAL MINDS PERFORM IN COLLABORATIVS ENVIRONMENTS?
In the Collaborative Cooking: Asymmetric environment, two players on distinct sides of a divided kitchen must collaborate to efficiently cook tomato soup. The layout provides distinct advantages to each side-one side is closer to the goal delivery but farther from the tomato dispenser, and vice versa for the other side. To maximize rewards, the two players should specialize based on their proximity to resources: one handles tomato dispensing and delivery, and the other manages dish retrieval and soup delivery. Evaluation scenarios challenge the focal agent to demonstrate dual forms of adaptation: adjusting the specialized role dictated by their side of the kitchen and to the varying competence of their partner, from skilled and specialized to entirely unresponsive.
Again, Hypothetical Minds achieves higher rewards than the baselines on every scenario (Figure 2). Interestingly, HM performs significantly better than the baselines on the scenarios where there is a functional partner (Appendix Table 1). This suggests that if there is value in a partner, HM can take advantage of this and adapt its behavior accordingly, highlighting the model's 's usefulness for com- plex, dynamic environments where cooperative interaction crucial. The LLM baselines perform relatively well with the negligent partner, where success hinges on repeatedly executing an intuitive sequence of actions- - filling and delivering pots without the need to take into account the actions of another mind. For instance, baseline LLM agents struggled when encountering a pot already at capacity while attempting to add tomatoes. PPO showed the opposite relative weakness, performing moderately with the skilled partner but has dramatic lack of generalization to the unhelpful partner scenario, as it is used to role specialization from self play. OPRE performs poorly on every scenario.
4.3 How DOES HYPOTHETICAL MINDS PERFORM IN MIXED-MOTIVE ENVIRONMENTS?
In the Prisoner's Dilemma in the Matrix Repeated (PD) environment, agents navigate similar grid world RWS and collect resources corresponding to cooperation or defection the iterated prisoner's 's dilemma game. The payoff matrix incentivizes mutual defection in single interaction, but the highest total welfare is achieved through mutual cooperation across an episode.
Hypothetical Minds achieves the highest reward among LLM agents and in 5/10 scenarios (Figure 2 Appendix Table 1). This highlights its ability to perceive the background agent's strategy and adapt accordingly. HM outperforms the LLM baselines relatively more with dynamic partners. With tit-for-tat for example (scenarios 5 and 6). Hypothetical Minds achieves the highest score among all models, by engaging in more consistent cooperation while demonstrating some forgiveness to avoid cycles of alternating defection, pattern that plagues LLM baselines. In scenarios and 9, Hypothetical Minds showeases capacity for "corrective punishment." By defecting against these partners that initially defect, it persuades them to switch to conditional cooperation. The agent then shifts to forgiving cooperation strategy maintain mutually beneficial equilibrium.
8

---

Published as a conference paper at ICLR 2025
Comparing Model Versions for Major Scenarios
HE HR
VMP
MMM
MMP HE
HM tours)
60-
Reward
40
20
0-
Best Response Your Last Move (SC 1)
Pure Rock Opponent (5C 6)
Pure Paper Opponent (SC 7)
Pure Scissors Opponent (SC B)
Evaluation Scenario
Figure 4: Comparing different versions of HM. Errorbars reflect SEM across 5 episodes.
However, OPRE achieves the highest rewards overall on Prisoner's Dilemma. Fixed heuristic poli- cies such as always defecting, playing tit-for-tat, etc. can achieve high rewards without needing to correctly infer the policy it playing against. One explanation that advanced RL methods like OPRE collect resources more efficiently than Hypothetical Minds in Prisoner's Dilemma, due to limited spatial reasoning abilities of ungrounded LLMs like GPT4. One limitation HM that the agent frequently travels across the enviromment to pick up resources that were closer to its original location. On the other hand, HM demonstrates good reasoning about the strategies is play- ing against, and good hypotheses between these two methods. LLM agents provide better abstract reasoning about things like other agent's goals right out of the box due to their extensive training data, but lack the low-level control necessary to maximize performance dense reward embodied environment (which RL excels at).
frequently found and validated. This illustrates the tradcofffs
4.4 How DOES HYPOTHETICAL MINDS SCALE TO ENVIRONMENTS WITH LARGER POPULATIONS OF AGENTS?
Running With Scissors in the Matrix Arena (RWS Arena) is eight-player extension of RWS, where the agent controls one player against a population of strategies. This adds additional com- plexity to the decision-maker makinng process and tests the scalability of models, agents must infer the strategies of separate agents and maintain this information in memory. Hypothetical Minds is up for this by maintaining separate hypothesis evaluation streams for every agent. Additionally, in order to maximize reward, agents should only interact with opponents that knows it can beat with its current inventory. Models are therefore tasked with integrating uncertainty and seeking out opponents for which they have high confidence about their strategy.
Hypothetical Minds achieves higher rewards than the baselines in RWS Arena (Figure 2) and is the best model on 6/8 scenarios (Appendix Table 1). HM performs particularly well on homogeneous populations of rock, paper, or scissors (scenarios 5-7). Scenarios 2-4 consist of heterogeneous pop- ulations where 2/3rds of the population are one pure strategy and the remaining 1/3rd represents the pure strategy that would beat the best response the majority strategy (e-g., scenario 2 2/3 rock and 1/3 scissors). HM performs the best on 2 out of 3 of these difficult scenarios, and performance for scenario is dragged down by one highly negative episode in which the agent was exploited by the minority strategy. In contrast, all baseline models struggle to achieve rewards of more than 10 on average in nearly every scenario. This highlights the difficulty of the environment, for which proper coordination between high-level plans, the embodiment, and memory is crucial for success. These results also suggest that Hypothetical Minds scales well to population- baseed environments which you need to either handle distinct agents differently make population-level inferences.
4.5 ABLATIONS
Our ablation studies focused three key aspects: (1) comparing performance across different base LLMs, (2) analyzing the components of the ToM module, and (3) analyzing the sensitivity of the ToM module's hyperparameters.

---

Published as a conference paper at ICLR 2025
First, compared GPT-4, GPT-3.5, and Llama-3-70B 11OB-Instruct across environments (Figure 5)- GPT-4 significantly outperformed the other LLMs. Llama-3 performed moderately on RWS Re- peated and Collaborative Cooking, placing between GPT-4 and GPT-3.5, but fell below GPT-3.5 on Prisoner's Dilemma. Due to context length limits, we could not evaluate Llama-3 on RWS Arena.
Second, conducted detailed ablation analysis of the ToM module using RWS Repeated as test environment. We evaluated five variants: 1. Vanilla Mind Prompting (VMP): A simplified version using single API call for all ToM steps, without hypothesis evaluation refinement. 2. Modular Mind Prompting (MMP): Removes hypothesis evaluation and refinement but queries the LLM separately for each reasoning step (ie. hypothesis generation and planning the next strategy are separate). 3. Modular Mind Prompting Hypothesis Evaluation (MMP HE): This version introduces hypothesis evaluation in comparison to the previous model. Hypothesis Evaluation and Refinement w/o Mind Prompting (HE + HR): Focuses only on hypothesis evaluation and refinement, removing ToM-specific prompting. 5. Full Hypothetical Minds: The complete model combining MMP HE with hypothesis refinement.
Results show MMP consistently outperformed VMP FFiuree Figure Sp. Adding hypothesis eval- uation (MMP+ + НE) and refinement (full HM) improved performance, with both variants achieving positive returns above 10 across all scenarios. The full model showed particular strength against the challenging best response bot (SC 1). This demonstrates that valuation and refinement may be most beneficial for complex strategies that are difficult to generate or reason about from scratch.
While GPT-4 could sometimes identify correct hypotheses through reasoning alone, it did so incon- sistently, highlighting the value of systematic hypothesis evaluation. The HE + HR variant excelled with adaptive opponents but struggled against fixed strategies, suggesting that explicit theory of mind modeling is crucial for recognizing static opponent behaviors (Rakoczy) 2022) Additionally, removing ToM prompting led to verbose, less focused hypotheses (see (see Figure 9).
Third, we examined key hyperparameters in the ToM module. The 'top_k' parameter controlling how many top hypotheses to evaluate (default=5) trades off computational costs. Performance improves most dramatically when maintaining any hypotheses beyond just the last generated one ('top.k= = 0' equivalent to the MMP model), particularly for challenging scenarios like scenario (Figure 11 Figure 12). This reflects categorical difference when the best hypothesis is kept around, giving the agent some memory of past guesses, while poor hypotheses naturally fall out of the top Increasing 'top_k' beyond 3 shows diminishing returns for RWS Repeated and may become detrimental past 5. The relationship between 'top_k" and computational cost is plotted in Figure 13 Figure 14) Once top k >> O, most cost variance stems from other factors, and higher costs (> $10 per episode) actually correlate with decreased performance. This occurs primarily because hypoth- esis validation reduces subsequent LLM calls, leading to both lower costs and better performance. Tests with Llama3 showed performance is relatively robust to learning rate a (which weights re- cent predictions) and the validation threshold (Figure 15 Figure 16). While smaller learning rates performed better, the model achieved above-chance performance across all parameter values tested. This robustness makes sense: a good hypothesis will eventually be validated across a reasonable range of these parameters, leading to high rewards for the remainder of the episode.
CONCLUSION
We have presented Hypothetical Minds, a model that demonstrates strong performance across di- verse multi-agent environments through its novel approach theory of mind modeling and hypothe- sis evaluation. Our work has several limitations that suggest promising directions for future research. The current implementation requires human intervention establish scaffolding and prompting for the agent. Additionally, knowledge of game rules and mechanics must be explicitly encoded in the prompts rather than learned from the euvironment. An important avenue for future research is developing methods for agents to learn these concepts and appropriate types of scaffolding au- tonomously from environmental feedback. This could enable more general-purpose agents that can flexibly adapt to novel multi-agent scenarios without manual configuration.
The success of our approach in structured game environments suggests broader applications to real- world multi-agent systems where understanding and adapting to people is crucial. Future work should investigate how the principles of the ToM module demonstrated here could extend to more complex social interactions such adaptive tutoring personalizing chat bot responses to users with hidden preferences. Such advances would represent important progress toward autonomous agents that can effectively navigate diverse social tasks.
10

---

Published as a conference paper at ICLR 2025
AUTHOR CONTRIBUTIONS
L.C. conceptualized and designed the Hypothetical Minds framework, developed the core code- base, conducted experiments, analyzed results, and wrote the manuscript. V.X. contributed to the framework design, codebase, and implementation. A.B. assisted with experiments. D.Y. and N.H. N.H. supervised the research, provided guidance on the project direction, and secured funding, All au- thors reviewed and provided feedback on the manuscript.
ACKNOWLEDGMENTS
We thank members of the Yamins and Haber labs at Stanford University for their valuable feedback and suggestions throughout the development of this work. This work was supported by the following awards: To Y.L.K.Y: Simons Foundation grant 543061, National Science Foundation CAREER grant 1844724, National Science Foundation Grant NCS-FR 2123963, Office of Naval Research grant S5122, ONR MURI 00010802, ONR MURI S5847, and ONR MURI 1141386 493027.
REFERENCES
John P Agapiou, Alexander Sasha Vezhnevets, Edgar A Duéñiez-Guzmán, Jayd Matyas, Yiran Mao, Peter Sunehag. Raphael Köster, Udari Madhushani, Kavya Kopparapu, Ramona Comanescu, et al. Melting pot 2.0. arXiv preprint arXiv:2211.13746 2022.
Janet Wilde Astington and Jodie A Baird. Why language matters for theory of mind. Oxford
University Press, 2005.
Chris L Baker, Rebecca Saxe, and Joshua Tenenbaum. Action understanding as inverse planning. Cognition, 11(3))329-349) 2009.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-acttion models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023a.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, i say: Grounding language in robotic affordances. In Conference on Robot Learning, pp. 287-318. PMLR, 2023b.
Nathaniel Daw and Philippe N Tobler. Value learning through reinforcement: the basics of dopamine and reinforcement learning. In Neurocconomics, pp. 283-298. Elsevier, 2014.
Jill G de Villiers. The interface of language and theory of mind. Lingua, 117(11): 1878, 2007.
Jill G de Villiers. The role (s) of language in theory of mind. In The neural basis of mentalizing, pp- 423-448. Springer, 2021.
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. Understanding social reasoning in language models with language models. Advances in Neural Information Processing Systems, 36, 2024.
Noah D Goodman and Michael C Frank. Pragmatic language interpretation as probabilistic infer- ence. Trends in cognitive sciences, 20(1 (0(11)8818 1y818-829 2016.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05600 2022.
Dom Huh and Prasant Mohapatra. Representation learning for efficient deep multi-agent reinforce- ment learning. arXiv preprint 2eXiv:2406.0284) 2024.
Michal Kosinski. Evaluating large language models in theory of mind tasks. arXiv e-prints, pp- arXiv-2302, 2023.
Wenhao Li, Dan Qiao, Baoxiang Wang, Xiangfeng Wang. Bo Jin, and Hongyuan Zha. Se- mantically aligned task decomposition in multi-agent reinforcement learning. arXiv preprint arXiv:2305. 10865, 2023.
11

---

Published as a conference paper at ICLR 2025
Ryan Liu, Howard Yen, Raja Marjich, Thomas L Griffiths, and Ranjay Krishna. Improving interpersonal communication by simulating audiences with language models. arXiv preprint arXiv:2311.00687 2023a.
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, et al. Bolaa: Benchmarking and orchestrating Ilm-augmented autonomous agents. arXiv preprint NXiv:2308.05960 2023b.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperatiee-coppettive environments. Advances in neural informa- tion processing systems, 30, 2017.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Anmual ACM Symposium on User Interface Software and Technology, pp. 1-22, 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16788 2023.
Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, al. Phenomenal yet puzzling: Testing in- ductive reasoning capabilities of language models with hypothesis refinement. arXiv preprint arXiv:2310.08559 2023.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International conference on machine learning, pp. 4218- 4227. PMLR, 2018.
Hannes Rakoczy. Foundations of theory of mind and its development in early childhood. Nature Reviews Psychology, (00)223-223,, 2022.
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. arXiv preprint arXiv:2307.06133. 2023.
Robert A Rescorla. A theory of pavlovian conditioning: Variations in the effectiveness of rein- forcement and non-reinforcement. Classical conditioning, Current research and theory, 2:64-69, 1972.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools. Advances in Neural Information Processing Systems, 36, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Melanie Sclar, Graham Neubig, and Yonatan Bisk. Symmetric machine theory of mind. In Interna- tional Conference on Machine Learning, pp. 19466619466. PMLR, 2022.
Yongliang Shen, Kaitao Song. Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang, Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36:38 10888111888800 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive archi- tectures for language agents. arXiv preprint wtt.::200000222.... 2023.
Alexander Vezhnevets, Yuhuai Wu, Maria Eckstein, Rémi Leblond, and Joel Z Leibo. Options as responses: Grounding behavioural hierarchies in multi-agent reinforcement learning. In Interna- fional Conference on Machine Learning, pp. 9733-9742. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2025
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023a.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
Ruocheng Wang. Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D Goodman. Hypothesis search: Inductive reasoning with language models. arXiv preprint (rXiv:2309.05600, 2023b.
Yuanfei Wang, Fangwei Zhong, Jing Xu, and Yizhou Wang. Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind. arXiv preprint arXiv:211 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
Annie Wong, Thomas Bäck, Anna V Kononova, and Aske Plaat. Deep multiagent reinforcement learning: Challenges and directions. Artificial Intelligence Review, 36(6):5023-5056 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances Neural Information Processing Systems, 36, 2024.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:2461 1 24624, 2022a.
Xiaopeng Yu, Jiechuan Jiang, Wanpeng Zhang, Haobin Jiang, and Zongqing Lu. Model-based op- ponent modeling. Advances in Neural Information Processing Systems, 35:28208-28221, 2022b.
Eric Zelikman, Jesse Mu, Noah Goodman, and Yuhuai Tony Wu. Star: Self-taught reasoner bootstrapping reasoning with reasoning. 2022.
Ceyao Zhang, Kaijic Yang. Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song-Chun Zhu, et al. Proagent: Building proactive cooperative ai with large language models. arXiv preprint arXiv:2308. 1339, 2023a.
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tian- min Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv preprint 00X 2023b.
Zhuosheng Zhang, Yao Yao, Aston Zhang, Xiangru Tang, Xinbei Ma, Zhiwei He, Yiming Wang, Mark Gerstein, Rui Wang, Gongshen Liu, et al. Igniting language intelligence: The hitchhiker's guide from chain-of ofor--of--thhhht reasoming to language agents. arXiv preprint 21X.17211.11199, 2023c.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
Sceuario
Agent Type
HM (burs)
Reflexion
ReAct
PlanReAct
PPO
OPRE
Running With
Scissors Repeated
Mix all
50.8
21.0 1
10.2
-10
3.9
5.3 1
2299 1.8
0.0 1 0.0
1: Best Response
23.2
4.7
.3.5
13.+-3 3.5
.2.3
4.2
+..1
0.5+16 1.6
-0.1+.00
DJ1
54.6
29.3
12.5
17.1
7.2
16.6
11.1
0.0.0.00
3: Flip After
40.6 + 11.1
50
12.8-77
13.2
0.8
0.0
4: Flip After
48.5 6.3
19
-115
9.0
-2.4
0.0
Gullible
5:1
82432
6: Rock
50.5
17.
14.3=86
15.6
12
0.2
Paper
8: Scissors
63.8 5.1
31.
29.0 4.3
14.433
4,2 1.4
403 133100 0.0
Running
Scissors Arena
Mix
13.3
1.1 3.6
132
80
0.55 1.5
0.1 0.2
1: Gullible
-2.6 3.4
-18 2.
-5.6
0.
-0.4 1.0
-0.6 0.2
Rock Flip Afler
16.2
-1.7 222
4,4
21132
-1.0 11
0.5
3: Paper Flip After
11.6 4.5
1.6 2.2
4.36
0.913
1.
0,3
4: Scissors + Flip After
-2.6 + 5.7
10.6 2.2
-12
-24
-2.1
02+033
5: Rock
21.8
3.0
1.4
4.3 3.1
7.1
1.7008
0.5 1 0.4
6: Paper
24.9 2.5
11
-10.7
40+3.
-2.4
04
Scissors
1.8
2.22
0.7
Collaborative Cooking
Asymmetric
0: Skilled Partner
628.3 35.1
292.6 + 38.2
225.4 60.0
402.9 + 70.1
244.0 76.1
21.5 + 15.0
Semi-skilled Partner
2: Unhelpful Partner
426.8 42.5
398. 33.6
383.7 21.4
292.6 + 36.7
0,0 0.0
62.0 62.0
Prisoner's 's Dilemma the Matrix
40.6 12.9
44.5
$7.9
37.0
67.2
1: Cooperator
78.7 = 2.4
1.7
71.3 = 100
2:22
40.0 5.4
102.5 : 1.0
Defector
21.9 0.5
2..1
2.
2.0
34.5
3: Grim Reciprocator
31.6 4.7
28.8
29.2 1.3
25.2
13.0 1.1
36.3 1.0
Grim Recipricttor strikes)
30.7
5: Tit-for-tat
52.5 1.1
3.
35.8-07 0.7
42.2+211
15.4 2.1
37.4 0.5
Noisy
41,9
7: Cooperates then defects
20.9 0.5
25.1 1.0
28.4 0.7
2.9
8.8 0.9
38.3 + 0.9
Defoctor that turns tit-for-tat
42.1
31.0
30.1
368
2.0
38.7 28.7-00
9: Defector that turn to noisy tit-for-tat
45.3 1.8
34.7 1.7
36.6
1.6
40.6
15,1 +
46.7 0.9
Table 1: Average reward and SEM for different agents substrates and scenarios. Note that the variance in scenarios that are a union of two scenarios may be related to which scenarios were sampled.
Bunning with Scissors Repested
Running with Scissors Arena
Col aboratvee Cooking Asymmetric
Prisoners Dilerma Repcated
<0
40
Average Reward
Average Reward
Average Reward
Average Reward
30-
RED
120
Macdl Type
Mode
Model Vpe
Macel Npe
Figure 5: Comparing base LLM models. seeds are generated per LLM.
14

---

Published as a conference paper at ICLR 2025
Interaction History
Good hypothesis found: { ('Opponent_ "'pponent_strateee""e strategy's think my opponent likely to play counter strategy to my strong strategies, favoring rock when I play scissors and scissors when I play paper...'', 'value": 0.942}
0,
9
:1
Reward:3.6
GPT4
1. Opponent__eeee serirentt_exx_iieetttrrr"" Based on the previous interactions and the guess about the opponent's policy/strategy, it seens that the opponent is likely to adapt their strategy counter my strong plays. In the last interaction, I played strong scissors strategy, and the opponent played strong paper strategy. Given this pattern and the guess that opponent favors rock when play scissors, opponent's next inventory likely rock/yellow': 5, 'paper/purple": 1, 'scissors/blue' 1).
strong rock strategy,
Lnventory<("roc//yellloo inventory=("
2. pe__ext__iveenten
To counter the opponent's 's likely strong rock strategy, should adopt a strong paper strategy. This neans next inventory should be inventory={" rock/yellow': "paper/purple": 5, 'scissors/blue": 1), which will give an advantage 1f the opponent indeed plays strong rock strategy.
Figure 6: Example of GPT4-based agent finding a good hypothesis and employing good reasoning to select its next target inventory.
Pypentends Raver Ietrentio 1.3 step" suaeed
Hlaest typertedee L'puover uesteyy. dh
Inteacttion
6e
Intrcction
morer contuueely sharie
mary Kaying merrr Fraeure
powtiosi intrecia int.
Intrrctiond
mirerr contiteity stailg sare
daryy dlying mearre riseree
streng aesicanting solee
edlening
Pypentonss tetmentie Lith the Guate vdice
Rlpest wblued typrtesto l'peoen certtey"
Interctioon
thete
1990 Too shary Vaaying heaee Hraeine
covennenty san arittey
silar" ricaooctiin toringg from au))
Intrrctionn 16
waee pranise
Figure 7: Example of a successful evolution of a hypothesis for MMP + HE HR playing a Best Response bot. shows the process how the LLM-based agent generated a good hypothesis about the opponent's strategy, which was eventually validated.
15

---

Published as a conference paper at ICLR 2025
Comparing Model Versions for all Scenarios
(ours)
50
Reward
20-
10-
2 0 1
7
8 Scissors
Mixed Strategy
Best Response
Flip After 2
Flip After 1
Gullible
Rock
Paper
Evaluation Scenario
Figure 8: Average reward per episode all evaluation scenarios for each version of the ToM module using the full version. Errorbars reflect SEM across 5 episodes. Reward is normalized per 1200 timesteps reduce variance with variable length episodes.
Example generated strategy for HE HR playing rock bot
Given the interaction history and the current stace of the it seems that strategy that adapts the opponent's recent choices unpredictabillde could be effective, The epponenz's strategy is not entirely clear Ezom the interaction history, but there are patteras of losses and wias that can be used inform strategy.
maintaining elemens
Hore's high-level strategy dEserve apponent's recent strategy:
player.0:
at to determi na if the opponent 's favoring a partiaular reasurce. Tf pattern is detented, prepare to counter it. interaction history: If win interaction, introduce slight wе lase, counter the
2. Adapt based our naxt choice to avoid being predictable. opponent's a last winning resource, In the case of tie, cither maintain the current rescurce or switch based on the observed pattern of the opponent. collection: Afficient avoiding unnecessary collection strategy
randonness
callecting resources that align with resources.
help
building a strong inventory for the next duel.
4. St stratagic dual initiation: Initiate duals when wа have strong inventory that the opponent's Likely choice. Use opponent's a strategy and position.
and observation to infer the
5. Avoid unnenessary resourcs collection: Dnee we have enough of a particular resource te play strong strategy, avoid collecting more of that resource and instead positioning for dual collecting dferentnt requirea it,
if atrategy
6. Maintein flexibility: Be ready to adapt our strategy based on new information from the opponent's actions and the outcomes of interactionss
Now, letrs summarize this strategy in Python dictienazy Iornal: 11 ""yython
"my.strategy'; 'AdapLive Strelegy with Randomness and Countering', Irue, wttp:arrccenttooonnntteettt
True, 'adapt, wtw.aedooon.nterreeoooooo..oooo Crue, "eficen..ssurreeeeeetttttt Tzue, True, sedduumneesaryrresoo...olleee 'ifwinneext's rontuo slight randomeess to avoid predictability', " 'if.loze_mext': 'counter epponens's is last winsing resource', "if.teeneett; 'mainiain ewitch. based pattern'
Irue,
shaantain.leexbiliie True, inberaattionuuidellneee.
opponent
Example of failure mode for HE HR playing rock bot. Generated strategies make little sense given the interaction history and requested format.
Figure 9:
16

---

Published as a conference paper at ICLR 2025
Example of successful embodied exploration at the beginning of an episode
First subgoala: Subgool Plan: Since I am currently at (21, 10) with an orientation tacing North and high-leve. strategy is to col yellow resourcea to play strategy, Hovever, there Therefore, my firat subgoal 1s to explore the environment to rind yel boxes. should move towards the center of the where an more likely to find resources and encounter the other player for patential duel.
to find yellew
towards
ye low box locations.
locations my inmediate vicinity.
map, should move I will move to (20, 10), then Lo (13, 10), and continue in this direction until I find yellow boxes or reach nore central location on the mag. If I encounter yellow boxes along the
at edge
(West) towards
cencer.
boxes, will will adjust path: collect then. Once I have collected 5 yellow ogponent initiate duel.
Aotion Plan:
"python {
action.plan: move.co 10), (20, 10)1, move.col (20, 101, (13, 101), move.to 101, (18, 101), move.to 10), 117, 10)1, move=so 10), (16, 101)
Figure 10: Example of a subgoal module exploring the environment at the beginning of an episode build up a cognitive map of where resources located.
B HYPERPARAMETERS AND COMPUTATIONAL COSTS
Hyperparameter
Value
Alpha (a)
0.3
Intrinsic reward e
1.0
Threshold for Hypothesis Value (Vir)
0.7
Top A: Hypotheses Considered (k)
5
Table 2: Default Hyperparameters of the Hypothetical Minds model
Performance by Top-k Value
Reward
7
Top-k Parameter
Figure 11: Average reward per episode on Rumning With Scissors Repeated (all scenarios) for four values of top k. Top k reflects the number of top hypotheses to continue evaluating (default=5). Errorbars reflect SEM across 45 episodes. Reward is normalized per 1200 timesteps to reduce variance with variable length episodes.
17

---

Published as a conference paper at ICLR 2025
Performance by Top-k
Reward
Scenarto
Average reward per scenario on Running With Scissors Repeated for four values of top.k. Top_k reflects the number of top hypotheses to continue evaluating (default=5). Errorbars reflect SEM across 5 episodes scenario. Reward is normalized per 1200 timesteps to reduce variance with variable length episodes.
Figure 12:
Performance vs Cost by Top-k Value
Reward
-20
Cost ($)
Figure 13: Scatter plot showing the relationship between computational cost and reward for different top-k parameter values in the hypothesis generation process. Each point represents an episode, with color indicating the top-k value used. Reward and cost is normalized per 1200 timesteps to reduce variance with variable length episodes.
18

---

Published as a conference paper at ICLR 2025
Cost vs Top-k
Input Tokens vs Top-k
Output Tokens vs Top-k
800000
50000
700000
40000
Input Tokens
600000
Output Tokens
Cost ($)
30000
400000
300000
20000
200000
100000
1
7
3
Top-k Parameter
Top-k Parameter
Top-k Parameter
Comparison of computational costs across different top-k values in three metrics. Left: Total monetary cost in dollars. Middle: Number of input tokens consumed. Right: Number of output tokens generated. All metrics are normalized per episode length (1200 steps). Errorbars reflect SEM across 45 episodes.
Figure 14:
Performance by Learning Rate Alpha
30
25
Reward
10
0.1
0.3
0.5 Alpha Parameter
0.7
0.9
Figure 15: Average reward per episode on Running With Scissors Repeated (all scenarios) for cach four values of the learning rate alpha Alpha reflects how much to weigh recent interactions when evaluating the prediction accuracy of the hypotheses, with higher values weighing recent interactions more heavily (default=0.3). Errorbars reflect SEM across 10 episodes in two representative scenar- ios. Reward is normalized per 1200 timesteps to reduce variance with variable length episodes.
19

---

Published as a conference paper at ICLR 2025
Performance by Hypothesis Validation Threshold
Reward
23
0.1
Value Threshold
Figure 16: Average reward per episode on Running With Scissors Repeated (all scenarios) for each five values of the value threshold hypothesis validation. thr reflects how confident we want to be before validating hypothesis as correct such that we do not generate evaluate more hypotheses until that value falls under the threshold (default=0.7). Errorbars reflect SEM across 10 episodes in two representative scenarios. Reward is normalized per 1200 timesteps to reduce variance with variable length episodes.
Hyperparameter
Value
Alpha (a)
0.3
Counterfactual reward c
3.0
Threshold for Hypothesis Value (Vihr)
3.0
Top Hypotheses Considered (k)
5
Table 3: Hyperparameters of the HE + HR model
Hyperparameter
GPT-4
GPT-3.5
Llama3
Model
"gpt-4-1 1 100-pprevww"
"gpt-3.5-turbo-1066
"Meta-Llama-3-177
Max tokens
4000
2000
2000
Temperature
0.1
0.2
0.2
Top
1.0
1.0
1.0
n
10
Table Hyperparameters of Various Models
B. EXPERIMENTS COMPUTE RESOURCES
In Table 5, we show the cost of rumning an episode with 1000 steps. For Llama3 experiments which
required 4 GPUs,
experiments three compute nodes with A40s L40s.
Money ($)
Time (mins)
HM-GPT4
10
45
Reflexion
42
ReAct
14
PlanReAct
21
HM-GPT3.5
24
HM-Llama3
58
Table 5: Experiment costs for a single episode of RWS Repeated. Rounded to nearest integer.
20

---

Published as a conference paper at ICLR 2025
C ENVIRONMENTS
C.I RUNNING WITH SCISSORS REPEATED
Specifically, here we evaluate our model on the Running With Scissors the matrix: Repeated environment (RWS) in the Melting Pot multi-agent decision-making benchmark (Agapiou et al. 2022]. This is zero-sum competitive environment with two players moving around a map and collecting yellow, purple, or blue resources that correspond to rock, paper, and scissors respectively. In addition to movement, the agents have an action to fire an "interaction" beam which initiates a duel with the other player when that player is within range of the beam. An interaction results in one agent getting positive reward and the other agent getting an opposite negative reward according to the inventories of resources picked up by each player. Specifically player will collect inventory, which is only observable by that player:
p (Pyellows Ppurples Pblue).
Reward is determined by matrix multiplication operations mirroring the rock, paper, scissors matrix game:
Vi "oww Arow Vcol+
Frow
Fool -Yrow
where o2
and
2j-1 Pj
0
-10
+10
Arow
+10
0
-10
-0
+10
0
The partially-obseraabee input in Melting Pot consists of a 5x5 window around the agent such that it can three grids in front of itself and one behind it, and two each side.
C.1.1 SCENARIOS
Description of scenarios for each substrate are reproduced directly from (Agapiou et al. 2022):
SCO:
Versus mixed strategy opponent. Here the focal agent must defeat an opponent that was trained to play a pure strategy: either rock, paper, or scissors. However, the specific oppo- nent is sampled at test time it could be any of those. All opponents commit strongly to their choice, aiming to collect at least three resources before interacting. To defeat them, the focal agent should scout out which pure strategy its opponent is playing and then collect the resources to implement its counter strategy. Since this a one-shot interaction, success requires the focal agent to pay close attention which resources are missing since they provide a clue to which strategy their opponent is implementing.
SCI:
Versus opponent who plays the best response to what the focal player did in the last round. Here the focal agent must defeat an opponent who may change their strategy with each interaction. The opponent will always select the best response to what the focal player selected in the previous interaction. For instance. if the focal player plays rock in in- teraction then its opponent will play paper in the next interaction. On the first interaction of each episode it chooses commits strongly to its choice, aiming to collect at least five resources before interacting. To win the focal player can trick its opponent into choosing specific strategy and counter- ing it. This requires changing strategy from interaction to interaction, cycling around the three options.
of the three pure strategies random. The opponent always
SC2: Versus opponent who sometimes plays a pure strategy but sometimes plays the best response to what the focal player did in the last round. Focal player must defeat an opponent sampled from the union of the background populations used in SC 0 and SC 1. The probability of sampling a pure opponent is 3/4 while the probability of sampling a best response opponent is 1/4.
SC3: Versus mixture of opponents who often flip to other strategies after two interactions. Focal player must defeat an opponent that may initially play any pure strategy and, with proba- bility 1/3, may flip after the second interaction the best response to the best response
21

---

Published as a conference paper at ICLR 2025
its initial strategy. For example if it starts out playing rock then the best response to that would be paper, so after the second interaction it would switch to playing the best response to paper i.c. scissors. It only weakly commits to its strategy for the first two interactions. That is, it aims to collect only one resource before interacting. After two interactions, at the point when it changes strategy, it also starts committing more strongly to its choice, aiming to collect five resources before each interaction. With probability 2/3, the opponent instead plays a pure strategy throughout the entire episode. In half of the pure opponent episodes the bot fully commits to its pure strategy, aiming to collect five resources before interacting, while the other half of the time it commits less strongly, aiming to collect only one resource before interacting. Note that opponents may be weakly committed to their strategy for the first two interactions regardless of whether they will ultimately flip strategy or not so it's not possible for the focal agent to observe weak commitment early on as cue to predict whether or not their opponent will later flip strategies.
SC4:
Versus mixture of opponents who either flip to another strategy after one interaction and keep it forever or continue to change, always best responding to what the focal player just did. Two kinds of opponents are possible. Both change their strategy after the first interaction. With probability 3/4 the opponent will be bot that flips to a different strategy after the first interaction and then follows it till the end of the episode. the best response to the best response to its initial strategy (so if it initially plays rock then it will flip to scissors). With probability 1/4 the other kind of opponent is sampled. This opponent is identical to the one in SC 1. Both kinds of opponents always fully commit to their choice, aiming to collect at least five resources before interacting so its not possible to observe the opponent's commitment level to predict which kind they are. To win the focal player must figure out which kind of opponent it is playing against and either best respond by selecting the same choice in all interactions after the first if paired with the first kind of opponent, or apply the cyclic strategy described as the solution to SC 1 if paired with the second kind of opponent.
always flips to
SC5:
Versus gullible opponent. Here the focal agent must defeat an opposing agent that was trained to best respond to agents playing pure strategies. The opponent should attempt to scout out what strategy the focal agent playing so can pick the appropriate counter. To defeat it, the focal agent should feint toward one resource and then collect the counter to its counter. So for example, if the focal agent successfully feinted that it would pick rock, inducing its opponent to pick paper, the focal agent should then collect and play scissors. This opponent fairly weak.
SC6: Versus pure rock opponent. Opponent always plays rock, and commits to it strongly, aiming to collect five resources before interacting. The focal player gets a high score when it picks paper and commits strongly to that choice.
SC7: Versus pure paper opponent. Same SC 6 but opponent plays paper so focal player should play scissors.
SC8: Versus pure scissors opponent. Same as SC 6 but opponent plays scissors so focal player should play rock."
It should be noted that scenario 5 includes gullible opponent, that attempts to scout out what strategy you are playing and pick the appropriate counter. From first principles, to beat this oppo- nent an agent should feint towards one resource to fool the opponent and then select the counter to its counter. Since our ToM module selects strategies on higher-level of abstraction than the embodiment, was not specifically designed to beat this opponent. Integrating high-level strat- egy information with relevant embodied information is the subject of future research. However, in practice the gullible bot does not effectively scout out our agent, and it frequently plays the same strategy, which is exploited by our agent, leading to positive rewards on this scenario on average for all model versions.
C.2 RUNNING WITH SCISSORS ARENA
We also evaluate our model on Running With Scissors in the matrix: Arena environment in the Melt- ing Pot multi-agent decision-makinn benchmark (Agapiou et al. 2022). This environment has the same dynamics as Running With Scissors in the matrix: Repeated with the main exception being that
22

---

Published as a conference paper at ICLR 2025
their are 8 players in this substrate playing larger 25 by 24 matrix with 11 by 11 observability window, skewed towards viewing more in front of the agent. All scenarios under this agent rep- resent one focal resident (agent) playing against 7 others with varying fixed and dynamic policies, attempting to maximize reward decided by the payoff matrix equivalent to the one in Running with Scissors Repeated.
C.2.1
SCENARIOS
SCO: Versus background population containing bots implementing all three pure strategies. Here one focal player joins seven from the background population. The background pop- ulation contains bots who implement all three pure strategies: rock, paper, and scissors. They may either commit to their strategy moderately (aiming to collect three resources be- fore interacting) or more strongly (aiming to collect five). The task for the focal agent is to watch its opponents, see what strategy one of them is implementing, and act accordingly.
SC1: Versus gullible bots. Here one focal player joins seven from the background population. The background population consists entirely of weak bots who were trained to best respond to agents playing pure strategies. They are weak opponents.
SC2: Versus mixture of opponents who play rock and some who flip to scissors after two interac- tions Here one focal player joins seven from the background population. The focal player should pay attention what each prospective partner has collected since 2/3 of them play rock while 1/3 play scissors after the first two interactions. Choosing paper to best respond to rock is bad choice if accidentally paired with an opponent playing scissors.
SC3: Versus mixture of opponents who play paper and some who flip to rock after two interac- tions. Like SC2 but with bots playing paper and bots switching from paper to rock.
SC4: Versus mixture of opponents who play scissors and some who flip to paper after two teractions. Like SC 2 but with bots playing scissors and bots switching from scissors to paper.
SC5: Visiting population of pure paper bots. Here one focal player joins seven from the back- ground population. All seven background bots play paper so the focal player can get a high score by playing scissors.
SC6: Visiting population of pure rock bots Here one focal player joins seven from the back- ground population. All seven background bots play rock so the focal player can get a high score by playing paper.
SC7: Visiting a population of pure scissors bots. Here one focal player joins seven from the background population. All seven background bots play scissors so the focal player can get high score by playing rock.
C.3 PRISONERS DILEMMA REPEATED
The Prisoners Dilemma in the Matrix Repeated environment is a sixxed-motive one where two indi- viduals collect resources that represent 'defect' (red) or 'cooperate' (green) and compare inventories in an encounter, analogous to the Running With Scissors substrates. Consequences of the inventory comparison are congruent with the classic Prisoner's Dilemma matrix game, exposing tension be- tween reward for the group and reward for the individual. Reward is delivered after an interaction weighted by the proportion of items in each agent's inventory, as described above for Running With Scissors. The payoff matrix for the interaction is
3
0
Arnw AT
5
C.3.1 SCENARIOS
SCO: Pariner may play either cooperate defect The optimal strategy is simply uncondi- tionally defect. However, given that the focal doesn't 't know the strategy of the background player, good strategy is more subtle. A reasonable strategy is to be a grim reciprocator cooperator, which would cooperate with the cooperator, and defect to the defector. Alterna- tively the focal player might try to ascertain whether the background player is exploitable.
23

---

Published as a conference paper at ICLR 2025
Doing so, however, carries risk, for if the background player were to be a Grim recipro- cator (like in other scenarios), this would cause them to defect for the rest of the episode.
SCI: Pariner typically plays cooperate, The optimal strategy simply to unconditionally de- fect. The same considerations about uncertainty of the background player's strategy from Scenario 0 apply here.
SC2: Partner rypically plays defecr The optimal strategy is simply to unconditionally defect. However. because the focal player doesn't a priori know the strategy of the background player, they must first try to find out their strategy. This can be done by looking which resources they collect or by paying attention to the results of the first few interactions. Once the focal has identified its background partner is defecting then it may have confidence that it should defect well. The focal player should also consider the possibility that the background bot is corrigible, i.e. that it could be persuade to switch from defection to cooperation. This is not the case here but the background populations used in SC and SC are corrigible.
SC3: Partner is a hair-trigger grim reciprocator, i.e, one who initially cooperates but, if defected once, will retaliate by defecting forever after. The optimal strategy is simply to coop- erate. Grim reciprocator background players are non-exploitable. and there is no way to know how they will react to defection ahead of time. Because of this uncertainty, testing for exploitability can lead to poor performance of the focal player. Conditional cooperators who cooperate first but retaliate if defected should achieve high score.
SC4: Partner is two-strikes grim reciprocator, i.e. who initially cooperates, but if defected on twice, will retaliate by defecting forever after The optimal strategy is simply to coop- erate. Grim reciprocator background players are non-exploitable, and there is no way to know how they will react to a defection ahead of time. Because of this uncertainty, testing for exploitability can lead to poor performance of the focal player. In principle, it would be possible to defect once against the background player leading to higher reward. But since it is not possible and testing it against a hairtrigger grim reciprocator leads to defection, in practice is better simply to cooperate. Conditional cooperators who cooperate first but retaliate if defected on should achieve high score.
know the background player is a two-strikes grim reciprocator,
SCS: Partner is a tit-for-tat conditional cooperator The optimal strategy is simply to cooper- ate. Defecting against tit-for-tat agent, even occasionally, might lead to miscoordinated interactions where one player cooperates and the other defects, in an alternating way. For- giveness is one way to break out of such cycles of recrimination. Conditional cooperators who cooperate first but retaliate if defected on should also be forgiving to ensure they do well in this scenario.
SC6: Partner is a tit-for-tat conditional cooperator who occasionally plays defect instead of cooperate. Like the previous scenario, except the tit-for-tat background player occasionally will defect instead of cooperate. This is known as trembling hand in game theory. A strict tor- focal player would occasionally fall into miscoordinated interactions with the background player resulting in alternating cooperation and defection. As in SC5, focal conditional cooperators must also be forgiving to ensure they do well in this scenario. Forgiveness is even more important here since the background player will defect relatively frequently itself but will still implement tit-for-tat retaliation when defected on itself.
SC7: Partner plays cooperate for a while then switches to defect Similar considerations to the previous scenarios. A good strategy is a grim reciprocator, or tit-for-tai tat for the focal player. Unconditional cooperation would be exploited by the background player.
SCS: Partner tries to take advantage of the focal player by playing defect, but if pumished, part- ner then switches to tit-for-tat conditional cooperation Related to Scenario the optimal strategy is for the focal player to persuade the background player to stop defecting by pun- ishing it through defecting itself. Once persuaded, the background player implements a conditional cooperation (tit-fortat) strategy. So it is safe to start cooperating with them once you have verified that they are themselves consistently cooperating.
SC9: Partner tries to take advantage of the focal player by playing defect, but if punished. partner then switches to noisy tit-for-tat conditional cooperation Like the previous scenario, except
24

---

Published as a conference paper at ICLR 2025
the focal player must implement more generous form of conditional cooperation after persuading the background player to switch from defection.
C.4 COLLABORATIVE COOKING ASYMMETRIC
In the Collaborative Cooking Asymmetric substrate, players need to collaborate to follow recipes. The environment described in (Agapiou et al.] 2022) follows the regular pseudoreward scheme, which is turned off by default. The asymmetric environment version of the Collaborative Cook- ing with an asymmetric advantages map. This is to test whether players can choose high-level strategies that play their strengths.
C.4.1 SCENARIOS
SCO:
Collaborate with skilled chef Here the background player implements a particular policy that can be very effective when its partner does its part. The two players are on two distinct and disconnected sides of the map. On one side the goal delivery location close to cooking pots and the tomato dispenser is far away whereas delivery location is far from the cooking pots but the tomato dispenser is close. The players should collaborate, each specializing in the part of the task that is most efficient for them to do on the side of the map where they spawned. The background player implements this kind of policy, which depends the actions of its partner to complete the task. The background player was trained with the V-MPO algorithm.
the other side the the goal
SCI: Collaborate with a semi-skilled apprentice chef This scenario is similar to SC but the background player is not as well trained. In fact the background population used here is the same as in SCO but from an earlier point in training. The importance of evaluating cooperation with bots of varying skill levels, and different points in training.
SC2: Succeed despite an unhelpful parmer In this scenario the background player never moves or helps in any way. On this map is less efficient implement all steps of the recipe alone versus to work together with a partner. But it is still possible for either player to perform all the steps on their own. The task is to realize that the background player won't do their part of the joint policy so the focal agent had better do everything itself.
D METHODS
D.1 TEXTUAL MAP
Pixel images of the global state are preprocessed into a text-based state representation. The images are divided into 8x8 patches, each corresponding to cell within the Melting Pot grid that entities can occupy. Each patch may represent one of four types: an agent, a resource, a wall, or blank space. The patches are labeled by comparing them manually labeled reference patches of each type of entity, including the numerous possible body orientations and hat colors an agent can em- body in each environment. Automating the process with computer vision is a candidate for future work. The global state can then fully represented text by coordinates in a Height x Width grid and the entity label at each coordinate. Egocentric states are then created from this according to the partially-observable 5x5 box around an agent (dependent on its orientation). We tried several representations of feeding this textual map to GPT, and in practice the best representation consists of printing each entity type with a list of all the coordinates where that entity type is present. For ex- ample "Player Position: ['player '': [(21, 4)]} Observable Yellow Box Locations: [(13, 10), (14, 11]]]. Observable Blue Box Locations: []. Observable Purple Box Locations: [(13, 11), (15, 11)] encodes the player position and orientation (south) along with the observable box locations. The player's 's current inventory is also included in the textual state representation, as it is in the default state representation for Running With Scissors and Prisoner's Dilemma.
D.2 MEMORY
The memory system consists of two parts. The first data structure appends the observed states in the previous step to lists of each entity type in a tuple with the step it was observed. For example: "yellow box': [((13, 3), 'Step: 1087'), ((13, 4), 'Step: 1087'), ((7, 3), 'Step: 1091')]. The LLM is
25

---

Published as a conference paper at ICLR 2025
prompted that its memory can be outdated and therefore should take the step it was last observed into account.
The second data structure in the memory system contains list of the agent's inventories and rewards from the interactions that occurred so far. This specific information, distinct from the previously observed states, is relayed to the ToM module.
D.3 THEORY OF MIND MODULE
The Theory of Mind Module is queried periodically after discrete events, For the in the Matrix substrates, this occurred after an interaction. For collaborative cooking, this occurred after dish was delivered.
The ToM module consisted of multi-step process, as depicted in Figure for the * in the Matrix substrates:
1. Record the observed behavior from the other agent's trajectory Φ(T). Here this refers to whether they played rock, paper, scissors, the argmax of the inventory (or cooper- ate/defect in Prisoner's Dilemma). Since the opponent's inventory is never observed, we have to estimate it given the inventory Hypothetical Minds played and the reward it ceived. Thus, we ask the LLM to estimate the opponent's 's inventory given this information, and note the output as the empirical opponent's 's inventory.
2. Evaluate Hypotheses about opponent's 's strategy. In the previous interaction, the top
k hypotheses are used generate predictions about the opponent's next inventory @(T). These argmax of the inventory predictions are compared to the argmax of the empirical opponent's 's inventory (did they play rock, paper, scissors). As described in the main text, Hypotheses that led to correct predictions get positive intrinsic reward and negative otherwise. a hypothesis is validated, meeting Vinr. then step is skipped and this hypoth- esis is used for step 4 until the hypothesis falls below the threshold (meaning its not longer making good predictions)
3. Generate new and refine old hypotheses. The LLM is tasked with generating hypoth- esis about the other agent's s strategy given the entire interaction history (see prompt below for more details). The prompt also includes the top k hypotheses generated far if the hypotheses have a value above 0 (meaning at least one correct prediction) such that the LLM can refine previously generated hypotheses.
4. Guess opponent's 's next goal. The LLM is prompted to guess the opponent's 's next inventory given hypothesis and the interaction history. If no hypothesis has yet surpassed Vihr. then guesses made for the top hypotheses and the last generated hypothesis. The prediction from the last generated hypothesis would be used to select a counter inventory in the next step. Moreover, all the predictions will be used for step 2 (hypothesis evaluation) after the next interaction. If hypothesis crosses Vinrs then only it will be used in this step.
5. Select goal counter opponent. The LLM is prompted to select counter inventory given the prediction about the opponent's 's next inventory. Since this step involves straight- forward reasoning given the opponent's predicted inventory, it done simultaneously the API/LLM call with the previous step. Therefore, the LLM is specifically tasked with out- putting both the predicted opponent's next inventory ((T) and its own goal/target inventory in a single API/LLM call.
In RWS Arena, step is done separately than step 5. The LLM is first prompted to guess the next inventory for the opponent they just played. Then in subsequent step, it is asked again to select which opponents to seek out, and to guess what inventory they will play (this could a different opponent than the one interacted with in the last round). Therefore, the ToM module can evaluate the hypotheses for each opponent based on the quality of the predictions they make for that particular opponent, and this process is separable from selecting the target inventory in the next interaction (see prompts).
A similar process occurs for Collaborative Cooking Asymmetric. Step is hardcoded and not LLM dependent: the other agent's actions/behavior Φ(T) are labeled by custom code given the observa- tions (abstracting away the problem of action recognition from textual observations). These actions
26

---

Published as a conference paper at ICLR 2025
include "Teammate picked up dish", "Teammate put down dish", "Teammate picked up cooked soup in dish", "Teammate delivered cooked soup". "Teammate picked up a tomato", or nothing. In step 4, the LLM is prompted to guess the teammate's 's next behavior Φ(T) in natural language. An- other LLM instance is used in step
assess whether the prediction was correct or not, prompting the LLM to output True False, and giving each hypothesis the appropriate intrinsic reward. Step is completed
separate API call for Collaborative Cooking, and the LLM is prompted: "what strategy do you want to take next and why? Teammate's 's observed strategy: ". Think step by step about how to adapt to their behavior and maximize all resources and efficiency accordingly."
D.4 SUBGOAL MODULE
The subgoal module is responsible for generating efficient subgoal plans for the agent. Given the high-level strategy and the current state of the game, the module decomposes the strategy into sequence of subgoals in the form of action function calls to efficiently implement the strategy. The subgoal module uses the LLM to generate these plans as a sequence of action function calls (usually 3-6). The prompt includes the current step of the game, the high-level strategy/target inventory pre- viously decided upon, details about the current observations (including player position, orientation, inventory, observable resource locations, other agent locations), valid movement locations, memory, and instructions about the action functions.
Action functions for the * in the Matrix substrates:
move.to(sre.coord, target_coord); Efficiently move agent from source coordinate to target coordinate.
fire at(target coord): Stay around specified coordinate and fire interaction when opponent is spotted to initiate duel.
Action functions for Collaborative Cooking Asymmetric:
wirv__tosrr_ccooee _orordc Efficiently move agent from source coordinate to target coordinate. Only move to valid move_to locations where counters objects are not present.
interact(target coord): Move to and interact with the entity at the target coordinate, such as picking up ingredients or delivering dishes of cooked soup. To place an object on a counter to free your hands, use interact counter coord).
coord): Wait for the pot at target_coord to finish cooking. Check the progress of the pots and only valid locations where pots are present.
wait(target.coord)::
D.5 ACTION PLANNER
The action planner turns the sequence of subgoals specified by the subgoal module into se- quence of atomic actions compatible with the Melting Pot environment. These actions include step forward, backward, left, or right, turn left or right, fire zapping beam, and noop. For the move_to ssource.coordinate
target_coor dinntte function, the A algorithm find the most efficient path given set of obstacles. Walls always considered obstacles, and other resources are conditionally added as obstacles. If a path can be found without picking up another resource, the action planner will return it. However, in many cases the language model picks target coordinate where other resources cannot be avoided. The action planner will use a priority system in this case, privileging paths where only resources of the same type as the target coordinate will be picked up. The f ireett aat function returns a sequence of actions to turn towards the opponent and zap them when the agent within the extent of the zapping beam's range. the opponent is not in view, then the fire at function turns the agent continually clockwise until the other agent is found.
D.6 SELF-REFLECTION COLLABORATIVE COOKING
For Collaborative Cooking Asymmetric, an evaluator and self-reflection mechanism was added as in the Reflexion baseline (Shinn et al. 2024). This was added such that if the agent was making action plans that did not change the state of the world, for example trying to pick up a tomato while holding dish, these action plans were not repeated with the same state information. By first reflecting on
27

---

Published as a conference paper at ICLR 2025
whether the previous action plan was successful, the agent able to make less mistakes over the course of the episode. This additional cognitive module for self-reflection could in principle also be added for the other substrates, but not necessary for good performance the in the matrix games because the state transitions were simpler for LLMs to understand.
E BASELINES
E. REACT
The ReAct (Yao al. 2023) agent combines reasoning traces with task-specific actions an inter- leaved manner. This approach allows the agent to generate both reasoning steps and actions within the same language model framework. In our context, the reasoning consists of chain of thought reasoning preceding subgoal plan in the specified format of utilizing action functions. The agent prompted to think about the other agents' strategies and come up with a subgoal plan accordingly. Thus, ReAct is functionally an ablation of Hypothetical Minds such that there is only subgoal module and not a theory of mind module. Three example responses are shown as few-shot prompts, consistent with the ReAct framework.
E.2 REFLEXION
Reflexion adds evaluation and self-reflection to the ReAct agent backbone serving as the Actor module. After subgoal plan is completed, the LLM queried evaluate the outcomes of that plan. Outcomes are represented as the reward during the plan, and salient state information pre and post plan. This state information includes the position and the inventory of the agent for the s in the matrix games. For Collaborative Cooking Asymmetric, the given state information included position, what the agent is holding, and the state of the two pots.
E.3 PLANREACT
We include the PlanReAct architecture introduced in (Liu et al.) 2023b) This model first generates a high-level plan in language and then feeds this plan to subgoal module that outputs a subgoal plan based the high-level plan. Thus, the only difference between PlanRe- Act and Hypothetical Minds is that high-level planning is mediated by the multiple processing steps of the theory of mind module, including hypothesis generation, evaluation, and refinement.
hierarchical baseline.
E.4 PPO
We train RL agents in a population of PPO agents (Schulman et al. 2017) on each substrate. The weights are randomly initialized for each agent in the population and weights are not shared. There- fore the agents during training than traditional self-play. Models were trained in PyTorch using the Ray Rllib pipeline and this starter code https://github.. cowm/strived//Metiine Peeeting-Po-CContsst-2222. Optimal pa- rameters were searched over and the final models were trained for 1e8 steps.
playing against identical copies of themselves and
greater diversity
E.5 OPRE
We include the Options as REsponses model (OPRE) introduced in the first version the Running With Scissors environment (Vezhnevets et al., 2020). OPRE is hierarchical MARL method where agents Iearn high-level options as strategic responses to other agents' behaviors. The high-level controller selects options based observations of other agents, and the low-level controller executes actions conditioned on these options. As this was included as baseline the Melting Pot 2.0 paper, reuse the results reported in that paper (Agapiou et al. 2022).
F ABLATION DETAILS
The Hypothesis Evaluation + Hypothesis Refinement model had different evaluation procedure than the other models. Rather than computing values based on predicting the opponent's 's inventory.
28

---

Published as a conference paper at ICLR 2025
here extrinsic reward and counterfactual reward. If hypothesis is used online for goal selection, then the rewards received in the next interaction can be directly used for evaluating it. For the other considered hypotheses, target inventory given the hypothesisstrratey and the given situation and 2. after the next interaction ask GPT again to reason about what the reward would have been if it played the inventory from 1. GPT is asked to output, positive, negative, or neutral, which we convert to reward with the parameter.
simulate counterfactual reward by 1. asking GPT to generate
G PROMPTS
All prompts can be seen our code. The most representative prompts also reproduced below.
29

---

Published as a conference paper at ICLR 2025
G.1
RUNNING WITH SCISSORS: ARENA PROMPTS
RWS Arena System Message
You are Agent 0 in the eight player 'runniiggwithh.ccssorrs Melting multiagent reinforcement learning enviranment with resourcea
is 25x24 (x by y) grid collect and walls navigate around. Players move around the map and collect resources of discrete types corresponding rock, paper, and scissers strategles Yellow box rock Purple box paper Blue box seissors. Rock/yellow beats scissors/blue, paper/purple beats rock/yellos
and scissor/bllue bests paper/purple.
In addition to movement, the agents have an action to fire an "interactien" beam which initiates a duel with one player getting positive reward and the other agent getting an opposite negative reverd according to their inventories.
All players carzy an inventory with count of resources picked up since last reapawn and for each respawn start with inventory of resource each. This inventory is visible in the state with the key 'inventory'. To play pure strategy atrongly, pick up at least 5 resources the color and then fire the interaction beam at another player. To commit less strongly to strategy, pick around 3 resources of the color and then fire the interaction beam at another player.
more of
Usually you will only want to pick up one type of resource before an interaction, in order to gain the most information about the ether players' strategies and to not waste time collecting other resources. You also
maximize the number of interactions so after you pick up dual reset your inventory and gain
4-6 resources, you should seek information about the other players' strategies.
Your opponents will also almost always only pick up type of resource before an interaction.
For example, playero.inventory - 17, 1, 1] (Yellow, Purple, Blue) is good inventory that will lead to an informative duel, whereas player0.inventory [2, 2, 2] (Yal lox, Purple, Blue) will not be informative Your reward the result of a matrix mulziplication involving your inventory vector format, and your opponent's inventory vector, and payoff matrix transpose iyouriineentor) arp.array[[1)) -10, 10), [10, 0, -101, [-10, 10, 011)
to rock paper scissors.
A.payoff opponent_inventory where Appayoff
The reward usually ranges from (5, -5) depending players (the min is -10 and max 10, but it is rare to get these magnitudes). Typically +/- 3-5 is a high magnitude, and a reward near 0 suggeats both players played a similar inventory.
the inventories of both
you can observe an 11x11 grid around your agent depending on your position and orientation (you can see more front you behind).
State Description:
This environment is partially-abervaabe
Previcusly seen states will represented memory, but note that these states could potentially be outdated. For example, the other agent could collect resource that you previcusly Given the parciallly-oseerabb. nature of the ervironment, to exploze the environment appropriately and select goals based on the information ve gathered.
will need
Also pay attantion on to your opponents' positions when you see them in order to duel with then and gain information about their strategy.
To find specific player, you can first move towards the last known map.
then meve randomly around Hanging around the center of the map and waiting is not good strategy for this rnvirnnmen..
location. of
player
player come you
Afzer you gather information about your opponents' strategies, seek out opponents whose strategy you know and can exploit and play counter-straten
30

---

Published as a conference paper at ICLR 2025
G.1.1 HYPOTHETICAL MINDS
Subgoal Module Message
Curzent State Description:
Slobal Maр Size: {map_size] grid (Nal1s are located at the boundaries of the map and in other places that are invalid for move.to).
Valid Locations for movelto: (movable.lccationg
Player Pasition: {player.position)
Player Orientation: (player orientation)
Player Inventory (yellow, purple, bluel: (player.irventory)
Egocentric Observaticns Size: 11x11 grid around your agent.
curzently observe the following based your position and orientation: Observable Yellow Rox locations (format: ((x,y), distance from surrent
location)) feellow.ocatioosstttllllttt....t
Observable Blue Box Locations: (blue_locations.withhcmc wisttttthhhttnee
observable Purple Box Locations: (purple locations with distance)
Observable Opponent Locations: (osponent.lccation
Previously seen states from memory (format: (1x,)), step last observed, distance from current location)): (self aaa...mmorr.ssttee..
Execution Outcomes:
(execution.outcomee)
Error for extracting and executing actions from the response: htt..aatttoo.......sspossseeee
Rewards:
[rewards.str]
Strategy Request:
You are at step (step) of the game.
You have decided to execute high-level stratecy/target inventory previous response given what you predicted your opponent will do. strategy, including first achieving beesf.blenex..nnetttrreeeee
Select subgoals target iyvnext.inventory: Once you achieve the target inventory, STOP picking up resources and inmediately seek out duel with opponent close to you that you exploit based your hypothesis about their strategy and your current inventory.
achieve
So once you've ve picked up about 5-7 resources in total, seek out a duel to receive rewards, get more information about strategies, and reset your inventory.
your 'ypotheses about player's strategy:
If you've generated hypothesis about player's strategy, you to inform your strategy about whether to interact. with them not. Each strategy is paired with value how well it explains the data observed so far, starting at 0.
this
A hypothesis is validated when its value is greater than: (self. haa..ooooyypohhesssttar
Your task is to devise efficient ent action plans for player (self.agens id], reason through what the next subgoals should given the state information. response should broken up into two parts:
1. Subgoal Plan based the current state and the high-level strategy
you previously specified above, decompose this strategy into sequance of subgoals and actions to efficiently implement this strategy. Think step step about this. This could be fairly long.
2. Action Plan output this sequence of actions in the following Python dicticnary format, parsable by ast. starting with:
(1 'action_plan': ['move_to((11) ((11, 7), (9, 3))', 'move_to((9, 5), (13, 5)) 11
Example response 1, and 3 formatted similarly, detailing other strategies and actions.
31

---

Published as a conference paper at ICLR 2025
ToM Module User Message
An interaction with another player has occurred step (step), (self.interace interaction historyl selll.le.a.aass played 1-]l-
What my opponent's likely inventory in the last round given the inventory I played and the reward received?
Think step by step about this. First think about what rescurce you had the most of in your inventery, and then think about which resource would beat that you received negative reward of -1 or worse or which resource would lose to yours if you received positive reward of or more.
If you received small magnitude reward near and in between (-1, 1), then your opponent may have played a similar inventery to you. Then depending the magnitude of the reward and the number of resources you played, you can infer the opponent's s inventory and whether they played that strategy strengly (5+ of that rescurce) weakly ( 3 of that resource).
An inventory of l'rock/yellow': 1, 'paper/purple': 1, Sscissors/blue'.
1} is not possible because you need at least 2 resources a type to play a quel.
Here are some example interactiens to help you reason about how the reward function works:
'younoinenttoor' ['rock/yellow'
'paper/purple':
1, 'scissors/blue': 1}, 'rewards'; -2.285,
jpssibllc.ppoment..mnnttrrr. (' ('rock/ycllow': 1, 'papor/purplo':
5, 'scissors/blue' 1}
'your.inventorry: (' ('rock/yellow's
'paper/purple':
'scissors/blue': 1), 'rewards' 3.571, pposibble.ppontttiiveeee.eee.e
(' ('rok/yellow':
'paper/purple': 1, 'scissors/blue': 6)
your inventory': {' (orok/yylllw'' 1, 'paper/purple': 4,
scissorss/buee: 1), 'rewards': 2.0, 'possible.oponettiiiee innippooenttivvetorr""" nventory'
(' rock/yellow': 3, 'paper/purple': 1, 'scissors/blue' 1}
In the 2nd part of your response, output the predicted eppenent's inventory in following Python dicticnary format, parsable by ast. ast.literal.ew () starting with: 'possible opponent.inventorr.... ('rock/yellow': 1, 'paper/purple': 1, /scissors/blue': 5)
Example output:
Given that last played a strong paper strategy with an inventory of ('rock/yellow': received a reward of -3.428, I believe my opponent played a strong scissors strategy.
'paper/purple': 5, 'scissors/blue's 1) and
The reward suggests that paper was beaten by their scissers, which means their inventory likely had a higher count of blue/scissors resources.
possible inventory for them could be ('rock/yeloo'' 1, "paper/pupple': 1, 'scissors/blue" 5) or similar distribution favoring scissors.
32

---

Published as a conference paper at ICLR 2025
ToM Module User Message
Total Rewards: (rewards.str)
Strategy Request:
interaction with another player has occurred step (step),
[ [-1]].
The total interaction history with this opponent is:
self-improvement is focus:
Here are your previous hypotheses about the algorithm this epponent is playing: [self.top hypotheses [self.last played id]). what is your opponent's likely policy given the inventories and the reward function? Think step by step about this given the interaction history.
If your previous hypotheses are useful, you can iterate and refine them to get better explanation of the data observed far. If hypothesis already explains the data very well, then repeat the hypothesis They may be playing the same pure pelicy every time, complex strategy to counter you, or anything between.
this response.
They are not necessarily strategy, you are just playing an algorithm.
smart agent that adapts to your
Are you getting high positive or negative reward when playing the same type inventory? For example, getting high positive reward every time you play many paper rescurces. If so, this oppenent may be playing a pure strategy and you can exploit this by playing the counter strategy.
Once you have output a hypothesis about this opponent's s strategy with step by step reasoning, you can the hypothesis inform your strategy.
the 2nd part of your response, summarize your hypothesis in concise message following Pythen dictionary format, parsable by ast.literalevval) starting with: 'rock/yellow's 1, 'paper/purple': 1, 'scissors/blue':
'I think my opponent is always playing a pure scissors strategy and collecting arcund 5 blue resources.')
(' "Opponent.strategr"
Otherwise:
what is this opponent's likely policy given the inventories and the reward
function? Think step by step about this given the interaction history.
They may be playing the same pure policy every time, complex strategy to counter you, or anything between. They are not necessarily smart agent that adapts to your strategy.
Are you getting high positive negative reward when playing the same type of inventory? For example, getting high positive reward every time you play many paper resources. If so, this opponent may be playing a pure strategy and you can exploit this by playing the counter strategy. Once you have output hypothesis about this opponent's strategy with step by step reasoning, you can use hypothesis to inferm your strategy.
In the 2nd part of your response, summarize your hypothesis in concise message following Python dictionary format, parsable by ast.litcral.cwe () starting with: ['Oppenent.stratoge think my opponent always playing a pure scissors strategy and collecting around 5 blue resources.")
'I
You will be prompted again shortly to select subgoals and action plans to execute this strategy thăa achieves the target inventory, so do not include that in your response yet right now.

---

Published as a conference paper at ICLR 2025
ToM Module User Message
An interaction with self. self. last.played.id has occurred at step (step), The tetal interaction history with (self. played is: You previously made the fol following guess about this player's strategy: Think step by step and predict what this opponent will play the next time you interact with them. Given the above mentioned guess about the opponent's s policy/strateqy, and the last inventory you played (if their strategy is adaptive, it may not be), what is their likely inventory in the next round. In the 2nd part your response, output the predicted opponent's s next inventory in following Pythen dictionary format, parsable by 'ast. bit..iiterllevall () starting with "'python. Example response 1: "opponent.next.iveeete.. Given that my opponent is playing rock policy, believe their next inventory will be (wrentory((rock/yyllloo" 5, "paper/purple': 1, 'scissors/blue': 1}. "'python preedicteooppoentt.xxttmvveeooee
("rock/yellow': 5, paper/purple': 1, 'scissors/blue' } " Example response 2: Opponeet..ext_inveeooyy..s Since my guess is that this player is playing a scissors policy, I precict that their next inventery will be ('rock/yellow':
'paper/purple": 1, 'scissors/blue': 5). 'pyyhhnn { 'predicted opponent next inventory': ("rock/yellow's 1, 'paper/purple':
"scissors/blue": 5) } ... Exanple response 3: ooponent.actt.mventorr"" Since my opponent is following paper strategy, predist their upcoming inventory will be 1, 'paper/purple': 5, 'scissors/blue'; 1}. 'python { 'predicteddoppoment i ('rock/yellow': 1, 'paper/purple': 5, 'scissors/blue' 1)
soventory=''occ//ollll
34

---

Published as a conference paper at ICLR 2025
ToM Module User Message
An interaction with self. dast--paaeed.i has occurred at step (step), The total interaction history with self, played id is: The total interaction history overall is: You previously made the following guesses about all the other players' strategies:
High-level strategy Request: Provide the next high-lovel strategy for your player aelf.agent.id. This response should include step by step reasoning in parts 1-3 about which strategy to select based on the entire interaction history in the following format: 1. 'Opponents to seekout'; Given the hypotheses about your opponent's strategies and their values, chich players should you seek out to duel with next and why? If possible, select opponents you have good hypothesis about you can exploit it and maximize your reward, Try to select multiple players if possible as one player might be hard to find is respawning. Are you noticing any patterns across the population as whole?
possible.oponett.stracc.
Given the above mentioned guess about the opponent's s pelicy/strategy what is their likely inventory in the next round. Given the opponent's s likely inventory in the next round, what In the
pupoonet..exx..nveeeeee
"wy.next.inventoryy.
should your next inventory be to counter this?
4th part of your response, output the opponent to seekcut, the predicted opponent's next inventery, and your next inventory in following Python dicticnary format, parsable by 'ast.literall iettllittral.o starting with "pyyhhnn Example response 1: 1. 'oponent.to.sekoutt: Given that that player.1 and player.5 is playing a rock policy, I believe should seek out either player or player 5 duel with next. 2. rOpponent next inventory': Given that these opponents are playing a rock policy, believe their next inventory will something like aaventor=(""ooc//ylllos
fairly certain
5, 'paper/purple': 1, 'scissors/blue" 1}. 'My.next.inveatorry' Given that these players are playing rock policy, believe next inventory should be paper policy Livertoory=(xroc///ellloo 1, 'paper/purple': 5, 'scissors/blue'; 1}. "'python { 'epponents.to to..kooutt: ['player.1', 'player.5'), 'prealateeopppeen..eeett.vveeeee
('rock/yellow': 5, 1) imy.next,inventorr''
'paper/purple': 1, "scissors/ba
('rock/yellow': 1, 'paper/purple': 5, 'scissors/blue': 1) Example response
Considering all the interactions, player player and player seem to heavily favor the scissors/blue strategy with consistent picks. Engaging either of these players could offer high reward Based on the observed behavior of player.2, player-3, and player-s, it is likely they will continue with a strong scissors/blue strategy, potentially having inventory of ('rock/yellow': 1, "paper/purple': 1, ascissors/blue" 5). 3. iny.eettinvve.oory'' To effectively counter the scissors/blue strategy, my optimal approach would be to adopt a rock/yallow strategy. An inventory geared towards this would be ('rock/yellow': 5, 'paper/purple': 1, " 'scissors/blue's ['player.2', 'player.3', 'player_5' 'predicted opponent next inventory': ("apooo/""""""//"e"""" ('rock/yellow's 1, 'paper/purple': 1, 'scisse bacissors/bue"" 5), 'my next inventory'; ('rock/yellow': 5, 'paper/purple': 1, "scissors/blue's 1)
1. poponents.to.ssccourt
opportunity. 2. "oponen_.nxx.iiveettoyy
1}. "'python { "apponents.to.seekoooo
35

---

Published as a conference paper at ICLR 2025
G.2 RUNNING WITH SCISSORS: REPEATED
The prompts for Running with Scissors Repeated remain very similar as Running with Scissors: Arena with the modification of the prompts being in player setting.
G.3 PRISONER'S DILEMMA: REPEATED
System Message
You are Agent aaanta.geet.00 in the two player (prisoner.dilemt in matrix' Melting Pot multiagent reinforcement learning environment that is an 23x15 ix by y) grid with resources to collect and walla to navigate around. Your geal is to maximize reward, Players can move around the map and collect resources of 2 discrete types corresponding to cooperate and defect strategies Green box cooperate Red box defect. You and another agent will collect resources and compare inventories agents
an encounter in repeated rounds. act to fire "interaction" beam which initiates duel compare inventories and distribute rewards. All players carry inventory with the count of resources picked since last respawn and for each respawn start with inventory of resource each. This inventory is visible in the state with the key 'inventory'. llere is how the payoff matrix in the generic prisoner dilemma works:
addition movement,
If both players cooperate, they both receive a reward of around 3,
Tf player cooperates and the other defects, the defector receives reward of around 5 and the cooperator seceives reward of around 0,
If both players defect, they both receive reward of around 1.
nuance using the payeff rules repeated and the rewards distributed based the inventory resources cal lected by each player.
is that the rewards are classic prisoner dilerma, but the interaction is
both agents cooperate, the with more cooperate resources will receive reward lower than the with less cooperate resources,
one agent cooperates and the other defects, the more defect resources defector has, higher reward
defector, If both agents defect, the one with more defect zesources will receive with
higher zeward
defect resources.
Your goal before each interaction is zo try infer what the other player wi11 play and their strategy over time is affected by your plays. You will only
to pick up type resource before an interaction. For example, the inventories ('cooperate/green''
'defect/red': 1) and 3, sdefect/red'. 3} will both result in the zeward, so don't waste time collecting more than you need. Player will also always only pick up one type of rescurce before an interaction. To play strategy strongly, pick up at least resources or more of only one color and then fire the interaction beam at the other player. To commit less strongly to a strategy, pick up around 2 resources of only one color and then fire the interaction beam at the other player. State Description This environment is pert:aaly-observabl
{' cooperate/grennt
you can observe 5x5 grid around your agenz depending on your position and crientation (you front of you than bahind). Previously memory, but note that these states could potentially be outdated. Per example, the other agent could collect resource that previously saw. Given the nature of the enviranmen environment appropriately and select goals based on the information you've ve gathered. Also pay attention to Player I's position when you see it in order to duel with them and gain information about their strategy. Your geal is to maximize reward attained over an entire episode, so keep in mind the long-term consequences your actions.
in states will be represented in
you will need to explore the
events gestalt manner.
36

---

Published as a conference paper at ICLR 2025
G.4 COLLABORATIVE COOKING PROMPTS
System Message for Collaborative Cooking Asymmetric
You are 7layer [selt.agent.id]] in Coll Collaborative cooking Asymmetria envi ronment, the goal 15 to cook and del deer tonato soup dishea with partner. The envi ronment delivery locations, and dish
diepensers. kitchen with Lomato dispenser, specific parts picking up ingredients, putting soup in dish, and deerinn cooked soup dishes. There is an impessable bartlez in the midcle of the kitcten that separates the agents' aides at x-4, where the pota are located. The goal is to efriciently
consista
perform actiona agent lof
kitchen
the other tonato soup as possible Lo naximise the collective reward. However, communication not posaible, so you must Inter your partner's strategy from their actiors and adapt put 3 tomatoes
together
many dishes
eccordingly coordinate taske. To cook tomate it finished cooking, 3. pus the cooked de delr to the ivery location. Your team receives reward of 20 for each successfully delivered dish. Only interact with objects tonato
pos, dish,
pick up
your side of the kitchen. You cannet pick up tomato from the tomato another item leke diah your hand.
You only hold apensser
up dish before you pick up cooked soup fron pot. The environment is partielly observable, and you can only see a 5x5 grld azound your agent. You will be prompted at different points to provide high-level strategies and lower-level action plans achieve then. these three functions
action plans:
move.to(src.roord, tazget.coord) Efficiently move agent from souice coordinate to counters objects are not present. Use speringly.
coordinate. Only
valid move.to locations where
interact coordd: Mave to and interact with the entity at the cocrdinate, such picking ingredients delivering dishes of cooked soup. To place object down interact ccodnter.corrd.. Mostly use this function.
counter to your hands,
wait (target.coord) Check the progress present. You probably only want to use this when both pot are full to moximize efficiency.
Mait for the got target,coord to finish cooking.
pots
only
lecations where pots
Mest of the tine you will just want to use the inceraes function because it both moves to and interacta with cojects, therefore all the cooking stepa can be completed leted with the interact function. To put down an item to pick something else interact with items
counter
your handa,
floor delivery
location.
G.4.1 HYPOTИETICAL MINDS
High Level Strategy Message (in ToM Module HM) Collaborative Cooking Asymmetrie
Strategy Request:
You are at step (step} of the game. Provide strategy for agent {self.agent.id).
Your response should outline high-level strategy what strategy do you want to take next and why?
Teammate's s observed strategy: (selS.teemate.ssratee)) Think step by step about how to adapt to their behavior and maximize all rescurces and efficiency accordingly.
This response will be shown to you in the future in order for you to select lower-level actions to implement this strategy.
Example response:
High-level strategy: want to focus on cooking tomato scup dishes. You will be prompted again shortly to select subgoals and action plans to execute this strategy, so do not include that in your response yet.
37

---

Published as a conference paper at ICLR 2025
Subgoal Module Message
Strategy Request:
You at step (step) of the game.
Your task is to devise efficient action plans for agent (self.agent_id), reason through what the next subgoals should be given the state information.
Your previously specified high-level strategy is:
Your response should be broken up into two parts:
1. Subgoal Plan Based on the current state and the high-level strategy you previously specified, decompose this strategy into sequence of subgoals and actions to efficiently implement this strategy. For every subgoal, think step by step about the best action function and parameter to use for that function. This could be fairly long.
Action Plan Output this sequence actions in the following Python dictionary format, parsable by ast. eval () starting with:
[f 'actior_pian': [' ['interact ((5, 1))'] H
Example response 1:
Subgoal Plan: Given the current state and my high-level strategy to focus on cooking tomato soup dishes, I should:
Move to the tomato dispenser and pick up tomato.
|I 'action_plan'' I' l'interact ((5, 1))'1 1I
Example response 2:
Subgoal Plan: Given the current state and my high-level strategy to focus on delivering tomato soup dishes, I should:
Move to the dish dispenser and pick up a dish, then plate the cooked soup.
'action_plan'' ('interact ((3, 4 'interact ((4, 2))"]
Example response 3:
subgoal Flan: Next I should move to the delivery locaticn and deliver the cooked soup.
(I aaction_plan'; [' ('interact ((3, 1))'1 }}
38

---

Published as a conference paper at ICLR 2025
Evaluate Action Outcomes/Self- Reflection
User Message Preamble:
If subgoal failed:
You are an action plan evaluator. The last subgoal included an interact action that failed. Your task to look the subgoal the agent took, the state of the envi ronment before and after the subgoal, and
why the subgoal was unsuccessful and provide feedback about what the agent should do next time.
We will next plan an entire new action plan, so suggest specific action plans and action functions to next when applicable.
If subgoal succeeded: You are an action plan evaluator.
Your task to look the action plan the agent took, state of the environment before the plan and the state of the envirenment after the pian,
and evaluate whether the action plan successful, and if not, provide feedback about what failed and what the agent should do next time.
Take into account that your teammate could have influenced the outcome of the subgoal in some circumstances. Suggest specific action plans and action functions to use next when applicable.
Infer Teammate Strategy Message
User Message:
Based on the observed actions of your teammate (player.1), what do you think their strategy is? Are they specializing in any specific activity or subtask?
Teammate's observed actions:
teelftttammate.ccom
Here are your previous hypotheses about the strategy your partner is playing: (selć.top hypotheses).
Think step by step and provide an analysis of their strategy, any specialization you infer from their behavior, and their competence. Then analyze how you adapt your strategy maximize efficiency and coordination with your teammate.
Remember communication is not allowed.
39

---

Published as a conference paper at ICLR 2025
Prediet Teammate Behavior Message
dish has been delivered step (step).
You previously guessed that your teannate's s (player 1) policy is:
(possible teammate strategy)
Based on the proposed hypothesis about your teammate (player.1), what do you think they will do next?
Output concise label about your tcanmate's s next behavicr in the following Python dictionary format, parsable by ast. l ()
starting with:
python {{ "prediccrod.nett_bbevvvoo 'placing tomatoes into pot (4,2)' })
Evaluate Predicted Behavior
A dish has been delivered at step (step).
You previously guessed that your teanmate's s (player.1) would perform
this behavior in this round: (predicted next behavior) Here is the observed behavior of your tearmate (player 1) this round: [1atest.teamate.attin
Did your prediction match the observed behavior? Concisely output True False in the below Python dicticnary format, parsable by lit.literal () starting with:
(I
True }}
40


Reviews:
[{"review_id": "kNSwXuZXu5", "reviewer": "Reviewer_Mz5L", "summary": "The paper introduces \"Hypothetical Minds\" (HM), an advanced autonomous agent framework leveraging LLMs for multi-agent tasks. HM incorporates a cognitively inspired architecture with distinct modules for perception, memory, and multi-level planning. A crucial innovation is the Theory of Mind (ToM) module, which actively constructs, assesses, and refines hypotheses about other agents’ behaviors in natural language. This hierarchical planning, alongside effective strategy adaptation, enables HM to excel in diverse multi-agent scenarios.\n\nThe system is evaluated using the comprehensive Melting Pot benchmark across four different environments, each containing multiple evaluation scenarios that represent competitive, cooperative, and mixed-motive tasks. The results show that Hypothetical Minds consistently outperforms established LLM-agent and RL baselines, demonstrating superior adaptability and generalization.", "questions": "- How does the agent determine the reward threshold (Vthr) for validating hypotheses in the Theory of Mind module, and how sensitive is the model’s performance to this threshold?\n\n- Could you provide more details on the specific algorithms or heuristics used in the preprocessing step to generate textual map representations from pixel images?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "**Originality**: The paper introduces a novel and sophisticated use of Large Language Models in the context of multi-agent systems by incorporating a Theory of Mind module. This innovative approach allows the agent to not only predict but also adapt to other agents' strategies using natural language. The hypothesis generation and evaluation mechanism represent a significant advancement in the field, setting this work apart from traditional RL and existing LLM-based systems.\n\n**Quality**: The authors have thoroughly validated their approach through comprehensive and rigorous experiments. Evaluations across diverse scenarios in the Melting Pot benchmark demonstrate the robustness and adaptability of the Hypothetical Minds model. The detailed ablation studies provide clear evidence for the effectiveness of the modular components, particularly the hypothesis evaluation and refinement mechanisms.\n\n**Clarity**: The paper presents its methods and findings clearly. The inclusion of detailed empirical results, ablation studies, and illustrative examples helps in comprehending the functioning and impact of the Hypothetical Minds model. The explanation of complex mechanisms such as the Theory of Mind module and hypothesis refinement is done in an accessible manner without compromising on technical depth.\n\n**Significance**: This work addresses critical challenges in the development of autonomous agents, specifically in terms of strategy adaptation, nonstationarity, and generalization across diverse multi-agent environments. The significant performance improvements over state-of-the-art baselines highlight the potential of this approach to influence future research and applications in multi-agent systems. The results show that Hypothetical Minds can be a foundational framework for building more sophisticated and adaptable autonomous agents.", "weaknesses": "**High Computational Cost**: Utilizing state-of-the-art LLMs, particularly GPT-4, incurs significant computational expenses and longer inference times, which can be impractical for some applications. Have you considered conducting experiments on smaller language models and test the effectiveness of the framework?\n\n**Manual Preprocessing for State Representation**: The current method relies on extensive manual preprocessing to convert pixel images into textual map representations. This approach is time-consuming and may not generalize well across various environments.\n\n**Presentation Issues**: The paper would benefit from a more comprehensive introduction to the key problem being addressed. It takes a while for the reader to fully grasp the challenge the model aims to solve. Additionally, there are some formatting issues, such as citations referring to preprint versions (e.g., Wang et al., 2023b; Toolformer) which should be updated to the respective peer-reviewed conference versions if available. Appendix G (Line 1346) contains a broken link that needs fixing. Figures and Tables are not links but rather plain text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t0hS98OnSW", "reviewer": "Reviewer_7nHy", "summary": "1. The proposed Hypothetical Minds model (HM) is an embodied LLM agent for multi-agent environments. It integrates modular components for perception, memory, and hierarchical planning conditioned on Theory of Mind (ToM) inferences.\n\n2. The Theory of Mind (ToM) module in HM generates, evaluates, and refines hypotheses about other agents’ strategies or goals in natural language. The critical role of hypothesis evaluation and refinement within the ToM Module is identified through ablations and comparisons against LLM-agent baselines.\n\n3. The effectiveness of Hypothetical Minds across multiple multi-agent environments is demonstrated in the Melting Pot benchmark. The agent significantly outperforms LLM-agent and RL baselines in every environment and in a large majority of evaluation scenarios, showcasing its generalizability.", "questions": "1. Are Chain of Thought, Tree off Thought and Theory of Mind related ? Please share with examples connecting to your proposal in the paper.\n\n2. What is SAMA in L143.\n\n3. L233 – How the prior probabilities are calculated?\n\n4. L267 -268 - .Each ToM module call uses 3 LLM calls + top k times for behaviour prediction …\n\nPlease explain what is done top k times. It is not clear from the above phrasing\n\n5. L342-343 – We directly test our LLM based agent on the evaluation scenarios in four Melting Pot environments (Fig. 3). \n\nI believe the reference should be Figure 4 instead of 3 right? If so, please correct it\n\n6. Figure 4 . …. 3 seeds are generated per LLM. \n\nWhat are the three seeds?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The Theory of Mind Module has been introduced to predict the next behavior of the agent and to make high-level strategy based on best hypothesis.\n\nThe proposed Hypothetical Minds agent significantly improved performance over previous LLM-agent and RL baselines on a range of competitive, mixed motive, and collaborative domains including both two-person and population-based environments. A thorough literature survey with a well planned set of experiments and evaluation in the Melting Pot Benchmark have been carried out.", "weaknesses": "1. The roles played by the various latent variables in the Theory of Minds module and the weightages assigned with these variables have not been discussed.\n\n2. How the values of the latent variables are identified is not clear?\n\n3. Lines 033 to 035 in the paper mention that: However, multi-agent reinforcement learning (MARL) methods suffer from\nvarious draw backs, including high sample complexity, poor generalization to agents not seen in training, and limited reasoning capabilities. \n\n– Citation is necessary for the above claim.\n\n4. Dependence on human subjects to set up scaffolding and prompting for the agent is an important drawback of the proposed system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jgX76INVlO", "reviewer": "Reviewer_3iBb", "summary": "The authors present \"Hypothetical Minds\" a novel approach to scaffolding LLMs by introducing hypothetical reasoning scenarios that help models better understand and respond to complex queries. They demonstrate that by creating imaginary scenarios and personas, LLMs can improve their reasoning capabilities and provide more nuanced responses across various tasks, including ethical reasoning and creative problem-solving. The work introduces a systematic framework for implementing hypothetical reasoning, supported by extensive empirical evaluation across multiple model architectures and benchmarks.", "questions": "- I'm somewhat surprised not to see Rational Speech Act mentioned. Have you considered extending the ToM module to recursive reasoning about others' ToM?\n- How does the performance of the hypothetical scaffolding approach vary with the complexity of the scenario being presented?\n- Have the authors considered the potential limitations of using hypothetical scenarios that might introduce unintended biases?\n- What is the relationship between the number of hypothetical scenarios used and the model's performance improvement?\n- How does this approach compare to other scaffolding techniques in terms of computational efficiency?\n- There are a few relevant papers out there, that somewhat similarly have LLMs simulate other agents or people and hypothetical scenarios, that might be worth discussing. E.g., There was a paper with that kind of flavor from Tom Griffiths's lab called \"Improving interpersonal communication by simulating audiences with language models\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The methodological innovation of using hypothetical scenarios as a scaffolding technique is both novel and well-justified, with clear connections to cognitive science literature on human learning. Love to see studies leveraging tools from cog sci to improve AI systems\n- The experimental design is comprehensive, testing the approach across different model sizes and architectures, which strengthens the validity of the findings\n- The authors provide detailed ablation studies that effectively isolate the impact of different components of their scaffolding approach\n- The qualitative analysis of model outputs offers valuable insights into how hypothetical reasoning affects model behavior", "weaknesses": "- The theoretical foundation could be strengthened by more explicitly connecting to existing work on prompt engineering and chain-of-thought reasoning\n- The evaluation metrics could be expanded to include more diverse tasks beyond the current set, particularly in domains requiring complex logical reasoning\n- I think having more seeds to decrease SEM and confirm statistical significance across all experiments would help strengthen the results\n- The scalability analysis of the approach could be more thorough, especially regarding computational overhead and implementation costs\n- Consider including a more detailed error analysis to understand when and why the hypothetical scaffolding approach fails\n\n*Note: Very happy to re-evaluate my score if authors address my concerns.*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c1RPc55x1q", "reviewer": "Reviewer_PcwQ", "summary": "This paper introduces Hypothetic Minds a new module LLM-based framework for approaching Multi-agent problems across Cooperative, Competitive and Mixed Intention Settings. Their key contribution is in the form a Theory of Mind module which generates hypotheses about strategies followed by other agents and uses these hypotheses to make predictions of next actions. The hypotheses are evaluated  based on the ground truth observations and refined using a Rescorla Wagner update rule. The authors test their framework on 4 games  from the Melting Pot benchmark.", "questions": "Q1. What exactly is Hypothesis Refinement in the ablation studies? Does it mean that in MMP + HE the agent only generates an initial set of hypothesis and does not generate any more hypothesis during the episode? Or is it just that the LLM is not shown the previous hypotheses? \n\nQ2. What is the rationale for the choice of the Rescorla Wagner update and the hyperparameter 'c'. How does this choice and the value of c impact the convergence of the agent during the episode? Furthermore, how does the choice of top-k influence this?  \n\nQ3. \"A hypothesis at time t is generated by asking an LLM to infer another agent’s strategy, conditioned on HM’s memory M of important past observations O\" - how does this work in cases where there are more than one opponents/partners? Does the LLM generate a unique hypothesis for each agent? Is the next action prediction based on all or one of these hypothesis? \n\nQ4. \"Each ToM module call uses 3 LLM calls + top k times\" This statement is a bit unclear - does this mean there are (3 + K) LLM calls or 3*K LLM calls ? \n\nQ5. Why are other Collaborative Cooking environments not included in the experiments?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1. The authors propose a well motivated plug-and-play Theory of Mind module for modular LLM frameworks that uses hypotheses about other agents as natural language representations of latent variables and then utilize and update these hypotheses to select the best actions. \n\nS2. Their method works across all three settings: Competitive, Cooperative and Mixed intention which is important for developing general purpose multi-agent frameworks. \n\nS3. Most of the paper is clear and easy to follow.", "weaknesses": "W1. Lack of generalization: The method is only tested in the limited setup of test scenarios of the Melting Pot benchmark. The Theory of Mind module seems to be a general addition that can be extended to any task. Comparison and experimental results in benchmarks which have been previously evaluated with strong LLM-based methods are necessary to demonstrate the generalizability of the framework and the theory of mind module. For example: \na. Games like Avalon [1], Werewolf [2] for Mixed Intention and Competitive setting\nb. Games like the Hanabi challenge [3], ThreeDWorld Multi-Agent Transport [4] (TDW-MAT) and Communicative Watch-And-Help [4] (C-WAH) for cooperative setting. \n\nW2. Within the Melting Pot benchmark, only a few games have been picked. Does this framework also work effectively on the other setups? Are there any technical limitations restricting the evaluations in the other setups? Even within the collaborative cooking game, only the asymmetric advantages environment tested. However, other maps like Forced Coordination where neither player can complete the task entirely on their own are necessary tests for the ability of this framework. It is insufficient evidence to only evaluate only one game for those three settings. \n\nW3. Missing ablation studies on top-k, v_thresh and Rescorla Wagner update. While these are hyperparameters, it is important to show how they affect the overall performance as they are part of the Theory of Mind modules which is the key contribution in this paper. \n\nW4. Concerns about the Hypothesis Refinement step: In the ablation study in Figures 5: MMP + HE is better in 2/4 scenarios (This trend is also seen in Figure 8). This leads to a question about the necessity of the Hypothesis Refinement step. \n\nW5. While most of the paper is clear, the Ablation studies section with those additional definitions is difficult to follow. Furthermore the conclusion is incomplete and disorganized.\n\n[1] Shi, Zijing, et al. \"Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon Game.\" arXiv preprint arXiv:2312.17515 (2023). \n\n[2] Xu, Yuzhuang, et al. \"Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf.\" arXiv preprint arXiv:2309.04658 (2023). \n\n[3] Bard, Noam, et al. \"The Hanabi Challenge: A New Frontier for AI Research.\" Artificial Intelligence 280 (2020): 103216.\n\n[4] Zhang, Hongxin, et al. \"Building Cooperative Embodied Agents Modularly with Large Language Models.\" arXiv preprint arXiv:2307.02485 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)