Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models
Authors: Ziqiao Ma
Jacob Sansom
Run Peng
Joyce Chai
Venue: EMNLP 2023 Findings (2023)
Keywords: theory of mind
large language models
mental states
TL;DR: We taxonomize machine ToM into 7 mental state categories and delineate existing benchmarks and call for a situated evaluation of ToM.

Abstract: Large Language Models (LLMs) have generated considerable interest and debate regarding their potential emergence of Theory of Mind (ToM). Several recent inquiries reveal a lack of robust ToM in these models and pose a pressing demand to develop new benchmarks, as current ones primarily focus on different aspects of ToM and are prone to shortcuts and data leakage. In this position paper, we seek to answer two road-blocking questions: (1) How can we taxonomize a holistic landscape of machine ToM? (2) What is a more effective evaluation protocol for machine ToM? Following psychological studies, we taxonomize machine ToM into 7 mental state categories and delineate existing benchmarks to identify under-explored aspects of ToM. We argue for a holistic and situated evaluation of ToM to break ToM into individual components and treat LLMs as an agent who is physically situated in environments and socially situated in interactions with humans. Such situated evaluation provides a more comprehensive assessment of mental states and potentially mitigates the risk of shortcuts and data leakage. We further present a pilot study in a grid world setup as a proof of concept. We hope this position paper can facilitate future research to integrate ToM with LLMs and offer an intuitive means for researchers to better position their work in the landscape of ToM.

Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models
Ziqiao Ma
Jacob Sansom
Run Peng
Joyce Chai
Computer Science and Engineering Division, University of Michigan
roihn, c Jumicch
Abstract
forms of reasoning and decision-making (Pereira et al., 2016; Rusch et al., 2020). Inspired by hu- man ToM, AI researchers have made explicit and implicit efforts to develop a machine ToM for so- cial intelligence: Al agents that engage in social interactions with humans (Krämer et al., 2012; Kennington, 2022) and other agents (Albrecht and Stone, 2018). Likewise, machine ToM can hance agents' capacity for interactions (Wang al., 2021). explainable decision decision-making (Akula et al., 2022), dialogue communication (Qiu al., 2022:
Large Language Models (LLMs) have gener- ated considerable interest dehate regarding their potential emergence of Theory of Mind (ToM). Several recent inquiries reveal a lack of robust ToM in these models and pose a pressing demand develop new benchmarks, as current ones primarily focus on different aspects of ToM and are prone shortcuts and data leak- age. In this position paper, we seek to answer two road-blocing questions: (1) How can we laxonomize holistic landscape of machine ToM? (2) What is more effective evaluation protocol for machine ToM? Following psycho- logical studies, taxonomize machine ToM into mental state categories and delineate ex- isting benchmarks to identify under-explored aspects of ToM. We argue for a holistic and situ- ated evaluation of ToM to break ToM into indi- vidual components and treat LLMs as agent who is physically situated in environments and socially situated interactions with humans. Such situated evaluation provides more prehensive assessment of mental states and po- tentially mitigates the risk of shortcuts and data leakage. We further present pilot study in a grid world setup as a proof of concept. We hope this position paper can facilitate future research to integrate ToM with LLMs and of- fer an intuitive means for researchers to better position their work in the landscape of ToM.
Takmaz et al., 2023), and collaborative task plan- ning (Bara et al., 2023).
Machine ToM has received an increasing amount of attention in the research community, especially as the field is reshaped by large language mod- els (LLMs) such as ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023). This highlights an ongo- ing debate and discussion whether machine ToM has emerged in LLMs. While LLMs have demonstrated some capability of inferring commu- nicative intentions, beliefs, and desires (Andreas, 2022; Kosinski, 2023; Bubeck et al., 2023), re- searchers also reported concerns regarding a lack of robust agency in LLMs for complex social and belief reasoning tasks (Sap et al., 2022; Shapira et al., 2023a) and in-context pragmatic communi- cation (Ruis al., 2022). Emerged not emerged, that remains question (or may not even be the central question ask). our view, existing eval- uation protocols do not fully resolve this debate. Most current benchmarks focus only on a (few) aspect(s) of ToM, in the form of written stories, and are prone to data contamination, shortcuts, and spurious correlations (Trott et al., 2022; Aru et al., 2023; Shapira et al., 2023a). Prior to embarking extensive data collection for new ToM benchmarks, is crucial address two key questions: (1) How can we taxonomize a holistic landscape of machine ToM? (2) What is a more effective evaluation pro- tocol for machine ToM?
1 Introduction
The term theory of mind (ToM, sometimes also referred to as mentalization or mindreading) was first introduced by Premack and Woodruff (1978) as agents' ability to impute mental states to them- selves and others. Many aspects of human cog- nition and social reasoning rely ToM model- ing of others' mental states (Gopnik and Wellman, 1992; Baron-Cohen, 1997: Gunning, 2018). This is crucial for understanding and predicting others' actions (Dennett, 1988), planning over others' be- liefs and next actions (Ho et al., 2022), and various

---

To embrace the transformation brought by LLMs and explore their full potential in understanding and modeling ToM, this position paper calls for a holistic investigation that taxonomizes ToM using the Abilities in Theory of Mind Space (ATOMS) framework (Beaudoin et al., 2020). After a review of existing benchmarks under this framework, we put forward situated evaluation of ToM, one that treats L.MMs agents who are physically situated in enviromments and socially situated in interac- tions with humans. We hope this paper will offer an intuitive means to identify research priorities and to help gain a deeper understanding of, as well as to effectively utilize, LLMs in ToM modeling for Al agents in the future.
nAI, 2023), researchers have searched for evidence of an emergent ToM in LLMs. Kosinski (2023) presents 20 case studies each of the unexpected contents task (Perner et al., 1987) and the unex- pected transfer (Sally-Anne) task (Baron-Cohen et al., 1985). With direct comparisons to children's performance, the findings have been cited as poten- tial evidence for spontancous emergence of ToM in LLMs. Bubeck et al. (2023) present similar behavioral study with 10 cases of belief, emotion, and intention understanding, concluding that GPT- 4 has an advanced level of ToM after qualitative comparison with predecessors. Other case studies have also shown aspects of machine ToM (Li et al..
2023; Holterman and van Deemter, 2023).
Limitations of ToM capabilities in LLMs. The above findings contradict the conclusions drawn in Sap et al. (2022)'s carlier study, which shows a clear lack of ToM in GPT-3 (Brown et al., 2020) on SOCIALIQA (Sap al., 2019) and ToMI (Le et al., 2019) benchmarks. As potential account, there has been criticism that the cognitive inquiries are anecdotal and inadequate for evaluating ToM in LLMs (Mareus and Davis, 2023; Mitchell and Krakauer, 2023; Shapira et al., 2023a). Follow- ing the same evaluation protocol, Ullman (2023) demonstrates that simple adversarial alternatives to Kosinski (2023) can fail LLMs. To further under- stand if the most recent variants of LLMs possess robust ToM, Shapira al. (2023a) present com- prehensive evaluation over tasks and probing methods, showing that robust machine ToM is absent even in GPT-4 and that LLMS are prone to shortcuts and spurious correlations. Based on the ongoing debate, it can be concluded that, while LLMs exhibit some level of sensitivity at under- standing others' mental states, this capability is limited and falls short of achieving robust human- level ToM (Trott et al., 2022; Shapira et al.. 2023a).
2
Large Language Models as Theory of Mind Agents
Since the advent of pre-trained language models, the research community has questioned whether they possess intrinsic mental states represent the environment (Li al., 2021; Storks al., 2021; Hase al., 2023) and comprehend the mental states of others (Sap et al., 2019; Zhang et al., 2021) through the textual description (observation) of be- havioral cues. The relatively recent breakthroughs of LLMs have created many discussions and de- bates, primarily concerning the extent to which LLMs possess various capabilities required for machine ToM. In this section, we first survey recent research presenting evidence and counter-evidence for the emergence of ToM in LLMs. We conclude the discussion with the limitations of current evalu- ation protocols
2.1 Do Machine ToM Emerge in LLMs?
Evidence for emergent ToM in LLMs. Prior to the rise of large language models, there has been growing evidence and acknowledgment of a narrow and limited sense of agency in smaller language models. Andreas (2022) argues that language mod- els have the capacity to predict relations between agents' observations, mental states, actions, and ut- terances, as they infer approximate representations of beliefs, desires, and intentions of agents men- tioned in the context. These representations have causal influence on the generated text, similar an intentional agent's state influencing its com- municative actions under Delief-Desire-Intentiie (BDI) agent model (Bratman, 1987). Amidst the excitement surrounding the release of GPT-4 (Ope-
2.2 Roadblocks in ToM Evaluation in LLMs
Given the pressing need for robust machine ToM in LLMs and large-scale ToM benchmarks, Searchers echo several difficulties in the evaluation protocol. Presently, ToM benchmarks suffer from three primary issues summarized follows.
Limited aspects of ToM. The evaluation of ma- chine ToM lacks consistency in the literature due to the ambiguity surrounding the specific mental states being targeted. Existing benchmarks often focus on limited numbers of mental states, such as the intention (Yoshida et al., 2008), belief (Grant

---

et al., 2017), emotion (Sap et al., 2019), and knowl- edge (Bara et al., 2021) of another agent. While all of these are necessary building blocks of machine ToM, we echo Shapira et al. (2023a)'s concern that the ToM capability of LLMs may have been over- claimed based on evaluations from only specific aspect of ToM. To give comprehensive assess- ment of holistic machine ToM, laxonomy is essential to enable researchers to effectively posi- tion their work with different focuses and priorities, which may be orthogonal to each other.
this ambiguity is the latent nature of mental states, the subject has privileged access to them while others can only infer the existence of these men- tal states based on observable behaviors or expres- sions (Dretske, 1979; Blakemore and Decety, 2001; Zaki et al., 2009). Thus, it is impossible to di- rectly access and assess the mental states of a hu- man, and ToM must be tested indirectly through humans' ability to understand the relationship be- tween mental states and behaviors, especially by predicting how agents behave based on their mental states (Swettenham, 1996; Phillips et al., 2002).
Data contamination. Data contamination refers the lack of a verifiable train-test split that is typi- cally established test the ability of machine learn- ing models to generalize (Magar and Schwartz, 2022). LLMs typically learn from internet-scall data, potentially giving them access during training to the data used to test them (Bubeck et al., 2023; Hagendorff, 2023). For ToM evaluation specifi- cally, the training corpora of LLMs may contain research papers detailing these psychological stud- ies. Many past studies used identical or slightly altered language prompts test LLMs, leading to potential contamination issues (Ullman, 2023). To critically evaluate the performance of LLMs on ToM tasks, researchers must have access to the datasets used to train them (Dodge et al., 2021), which are unfortunately not available.
While the exact definition of ToM remains cen- tral debate, the AI community looking at what psychologists have viewed as an initial step. In this paper, we follow Beaudoin et al. (2020)'s ' taxonomy of ToM sub-domains, i.e., the Abilities in Theory of Mind Space (ATOMS). As shown in Figure 1, the space consists of 7 cate- gories of mental states, including beliefs, inten- lions, desires, emotions, knowledge, percepts, and non-literal communication. We selected this taxon- omy because was derived from comprehensive meta- antta-analysis of ToM studies. The meta-analysis focused on young children aged 0-5 years the early stage of cognitive development, such that the setups are simpler and more comparable, avoiding complicated physical and social engagements that cannot be trivially deployed on LLMs.
benefit from
Shortcuts and spurious correlations. The avail- ability of shortcuts and spurious features has trig- gered many concerns that a model may leverage them to perform highly bustly acquiring the desired skill (Sclar et al., 2023; Ullman, 2023; Shapira al., 2023a). Recent find- ings suggest that LLMs tend to learn surface-level statistical correlations in compositional tasks, po- tentially leading to an illusion of systematic learn- ing (Dziri et al., 2023). In all likelihood, LLMs are capable of learning ToM shortcuts in a similar manner.
Beliefs. Beliefs are informational states that peo- ple judge to be true, usually decoupled from moti- vational states (Dennett, 1995; Eccles and Wig- field, 2002). Beliefs, the most studied mental states in the field of ToM, are usually tested in the form of false belief tasks, including the unex- pected contents test (Perner et al., 1987), the un- expected transfer (Sally-Anne) Test (Baron-Cohen et al., 1985), the second-order false belief (Ice- cream Van) Test (Perner and Wimmer, 1985). Re- searchers also studied their connection to actions and emotions (Swettenham, 1996).
benchmark without ro-
3 Towards A Holistic Landscape of Machine Theory of Mind
Intentions are choices with commit- ment, usually associated with concrete actions to- wards a goal (Cohen and Levesque, 1990). As a critical component of ToM, Kennington (2022) has called for a more explicit treatment of intentions. Intentions have been extensively explored in psy- chology tests, e.g., behavioral re-enactment (Melt- zoff, 1995), action prediction (Phillips et al., 2002), intention explanation (Smiley, 2001), and intention attribution to abstract figures (Castelli, 2006).
Intentions.
3.1 Abilities in Theory of Mind Space (ATOMS) Framework
The evaluation of machine ToM lacks clarity and consistency across various literature, primarity due the ambiguity surrounding the specific mental states being targeted. This ambiguity not unique to the field of AI but is rooted in the complicated cognitive underpinnings of ToM. At the core of

---

Involuntary lins Percegr-Actte Link
11
Intention Artriburion
Percepts
Visucl Perspective Taking
19
B Avottabe Bonchmaaks
Exphnation
Auditory Perspective Taking
6-
Prediction of Actions
Intentions
Multiple Desires
6.
Discrepant Intentions
Discrepint Desires
Destres
Completion of Failed Actions
Tasire-Action Contrudictio
Identity Palse Deliefs
Destre Indiuenced, ActionEmotton
Second- Order
Contees Tulon Belisfa
Abilities in Theory of Mind Space (ATOMS)
Percepes Knowledge
Kouwladgg--uu Link
0.
meooing Been
Emetin
Dercept
Bellefs
Knowledge
Locaiinn Taler Belinfs
InformationKnnnleeet Link
Figure? 2: Number of avail- able benehmarks for each mental state in ATOMS.
Comprchenstre Measures
Xnowiedge Pretond Play Link
Reliet-aased ActonnEmatons
Moral Emotions
Tlmmrr
Mixed Emuttion
White Lite
Lidden Emootions
Emotion Regulation
literal Communication
Emotions
Fgooentric Liees
Distrepant Fmottinn
Tepp
Comprshonsive Minasunt
Insencaar
hena
Four Partiocial Gatte
Typical Emotional Reactions
13
cnyveedra 10
Comprehenstre Measures
Acypical Emotional Reactions
Figure I: The ATOMS framework of Beaudoin et al. (2020), which identified 7 categories of mental states through meta- meta-analysii of ToM studies for children.
Figure 3: A comparison of
benchmark settings.
Tested Agent
Situatedness
ATOMS Mental States
Benchmurks and Task Formulations
Imput Modality
Physical
Social
Belief
Intention
Sym.
Text
Noeling.
Per
Int.
Per
Int.
1st
2nd+
Act.
Com.
Des.
Emo.
Know,
EPISTEMIC REASONING (Cilacs,
Infer
ToMi (Ncomoollz atal, 2018)
QA
T
/
Hi-ToM al.
/
MINDGAMES dilco and Lemooid, 2023)
Infer
T
/
ADV-CSFB iStopes
CONVENTAB.CCM (Chang Cho, 2010)
Infer
H
SOCIAL IQA OSperal,
RE:ST (Trnoey sl. 2035)
H
FAUXPAS-FAI (Shapin
H.Al
COKE (Wo atal. 2013)
NLG
Al
TOM-IN-AMC 2023)
G4C ZDou al, 30236)
NLG H.Al
/
VISUXLBELIEFS (yyethoc et al. 2016)
TRIANGLE COPA (Godoo, 2016)
QA
H
Carionn
MSED Jia 2022)
Images
BIB (Gandii 2021)
Infer
2D Grid
AGENT (Shu al..
Sim.
MToM (fahinoiisc 2018)
Infer
2D Grid
SYMMTOM (Sdaretal. 2012)
MARL
Grid
MINDCRAFF (Btas cta., 2021)
Infer
H
31 Sim.
CPA (ftor 2015)
Sim.
Table I: A taxonomized review of existing benchmarks for machine ToM and their settings under ATOMS. We further break beliefs into first-order beliefs (1st) and second-order beliefs or beyond (2nd+); and break intentions into Action intentions and Communicative intentions. Tasks are divided into Inference, Question Answering. Natural Language Generation, and MultiAgent Reinforcement Learning. Input modalities consist of Text (Human, AI, or Template) and Nonlinguistic ones. The latter further breaks into Cartoon, Natural Images, 2D Grid World, and 3D Simulation. The Situatedness divided into None, Passive Perceiver, and Active Interactor. Symmetricity refers to whether the tested agent is co-situated and engaged in mutual interactions with other ToM agents.
Desires. Desires are motivational states that do not necessarily imply commitment, though they are usually emotionally charged and affect ac- tions (Malle and Knobe, 2001; Kavanagh et al., 2005). Typical studies along this line include the Kucmy-Yucky Task (Repacholi and Gopnik, 1997) for discrepant preferences from different individu- als, the multiple desires within one individual (Ben- nett and Galpert, 1993), and the relationship be- tween desires and emotions/actions (Wellman and Woolley, 1990; Colonnesi al., 2008).
Emotions. Emotions are mental states associated with an individual's feelings and affective expe- riences, which could impact beliefs and behav- iors (Frijda et al., 1986; Damasio, 2004). Most ToM studies on emotions focus on typical (Knafo et al., 2009) and atypical (Denham, 1986) emo- tional reactions to situations. Other studies also encompass affective perspective taking (Borke, 1971), understanding hidden emotions (Harris et al., 1986), and morally related emotions (Pons and Har- ris, 2000).

---

Knowledge. Many controversies revolve around the definition of knowledge as justified true be- liefs (Gettier, 2000). In the context of AI, knowl- edge typically consists of information and orga- nized representations of the world, which can be used to simplify understanding and address intri- cate reasoning and planning (Schank and Abelson, 2013). ToM studies usually involve understanding the absence of knowledge (Aronson and Golomb, 1999) as well as the connection between knowl- edge and perception (Ruffman and Olson, 1989) and attention (Moll et al., 2006).
tions with other ToM agents), We summarize our review in Table and discuss our observations and under-explored aspects of ToM evaluation.
Many aspects of ToM are under-explored. As shown in Figure 2, we notice an overwhelming re- search focus on the intention and belief aspects of machine ToM. Several other aspects of ToM have not received enough attention. While the field of NLP has thoroughly explored different facets of emotion and non-literal communication, e-g., in the context of dialogue systems, ToM has rarely been explicitly mentioned as motivation. More connec- tions and integrative efforts are clearly needed.
Percepts. Humans are situated in the physical and social environments. To enable AI agents to operate in the world and communicate with hu- mans, the sensory and social aspects of perception are crucial in a machine ToM. Along this line, psy- chological studies have investigated the perceptual perspective taking (Masangkay et al., 1974) and understanding the influence of limited perception on actions (Hadwin et al., 1997).
Lack of clear targeted mental states. Explic- itly mentioning the Sally-Anne Test (Baron-Cohen et al., 1985) as inspiration, Grant ct al. (2017) de- veloped the predecessor of TOMI (Le et al., 2019). Similarly, Nematzadeh et al. (2018) cited the Ice- cream Van Test (Perner and Wimmer, 1985) motivation and the FAUXPAS-EAI (Shapira et al., 2023b) benchmark followed the study of Baron- Cohen al. (1999). While these benchmarks are cognitively grounded and target one particular as- pect of ToM, the majority often incorporate multi- ple mental states without clear descriptions, which could make it challenging to measure the actual progress (Raji al., 2021).
Non-literal communications. Being able to un- derstand non-literal and figurative communication helps humans to perform pragmatic inference and reason about hidden words behind their written meanings (Giora, 2003). Non-literal communica- tion has been recognized as an advanced ToM ca- pability, spanning a wide spectrum of humor and deceptions (Happé, 1994), sarcasm (Sullivan et al.. 1995), and faux-pas (social gaffe) situations (Baron- Cohen et al., 1999).
Lack of situatedness in physical and social en- vironment. Figure 6 illustrates the configurations of benchmarks. Each bar in the chart represents a distinct benchmark characteristic, and each seg- ment within the bar illustrates the proportion of benchmarks with one specific setting. An immedi- ate observation is a noticeable lack of benchmarks that encompass both physical and social environ- ments, which highlights an existing research dispar- ity in the field. We notice that many existing bench- marks are story-based, which verbalize the agent's perception of the environment and the behaviors of other agents in the form of story episodes, usu- ally with language templates. The semantics of the environment are given by high-level events (e.g., Sally entered the kitchen). Many aspects of physi- cal and social situatedness are overlooked in these benchmarks, c.g., spatial relations, the task and motivation of agents, and their action trajectories.
3.2
A Taxonomized Review of Benchmarks
The ATOMS framework can serve as an intuitive reference for researchers to identify their research priorities and situate their work better in the land- scape of literature. We further take the initiative to provide systematic review of existing benchmarks for machine ToM under the umbrella of ATOMS. Although there are independent research initiatives on certain ToM facets like intention classification, emotion modeling, and aspects of non-literal com- munications, we primarily focus on those that ex- plicitly target ToM or inferences of latent mental states. Besides the ToM dimensions in ATOMS, we further characterize the benchmarks on their task formulation, input modalities, physical and social situatedness, and symmetricity (whether the tested agent is co-situated and engaged in mutual interac-
Lack of engagement in environment. We point out that existing benchmarks primarily adopt pas- sive observer role test language agents. Yet the crucial aspects of interaction and engagement be- tween the agent and other entities involved have
We maintain a repository for relevant literature at https: //gittht thub. com/ Mars- tin/ www.wwwwwwwwweeeeee/oeeeeeeeeeew theory-of oi-mind

---

been overlooked. Among all the benchmarks we reviewed, only three of them treat the tested model as an active agent, one that perceives the physical and social context, reasons about others' mental states, communicates with other agents, and inter- acts with the environment to complete pre-defined tasks (Sclar et al., 2022; Bara et al., 2021, 2023).
Situated evaluation mitigates shortcuts. By em- ploying situated evaluation, the risk of taking short- cuts can be mitigated. Many of the existing ToM benchmarks are either limited in scale or adopt text templates to verbalize a (few) predefined sce- nario(s) and prompt LLMs for answers, giving an- swers away from syntactic structures and positional information (Le et al., 2019; Sclar et al., 2023). In situated setting, on the contrary, rely simu- lated environments manipulate evaluation data at scale, so that the environment, the states, and the action traces in the environment can be randomized to avoid the statistical spurious correlations. While situated evaluation can mitigate shortcuts, it does not eliminate the issue completely. For example, Aru et al. (2023) have reported that shortcuts can emerge in grid world setups if the design is not careful enough and randomness is limited. We em- phasize that careful design and consideration are still required to curate any ToM benchmark.
4 Towards A Situated Theory of Mind
4.1 Why A Situated ToM?
There have been concerns that cognitive inquiries are inadequate for gaining insight into understand- ing ToM for LLMs (Mitchell and Krakauer, 2023; Shapira et al., 2023a). However, believe that the primary problem lies in using story-based probing as proxies for psychological tests, which situate human subjects in specific physical or social en- vironments and record their responses to various cues. We, therefore, call for situated evaluation of ToM, in which the tested LLMs are treated like agents who are physically situated in environments and socially situated in interactions with others.
4.2 A Preliminary Exploration in Grid World In this section, we present a proof-of-coneept study on a situated evaluation of ToM on LLMs. We choose to conduct our pilot study in Mini- Grid Bneevalerr-Boivedd et al., 2018), (0)8, simple and commonly used environment for ToM studies in the machine learning community (Rahinowitz al., 2018; Sclar al., 2022). Through basic grid world representation, we can create tasks to challenge LLMs to reason about many aspects of physical and social situatedness, e.g., spatial relations, par- tial observability, agent's action trajectories, and from there, their beliefs, intent, emotions, etc. This is in stark contrast to existing story-based ToM benchmarks, which only contain high-level event episodes. We demonstrate that diverse range of challenging ToM tests, covering all mental states from ATOMS, can be effectively created in a situ- ated manner using simple 2D grid world.
Situated evaluation covers more aspects of ToM. Although it is possible to frame the situations as nar- ratives and cover all mental states using text-only benchmarks, certain aspects of ToM can only be ef- fectively studied within specific physical or social environment (Carruthers, 2015). This is because humans have the ability to infer the mental states of others through various modalities such visual perception, actions, attention (gazes gestures), and speech (Stack al., 2022). For instance, study- ing perceptual disparitics can be challenging with text-only datasets, as they often reduce complex scenarios to rule-based manipulations over nega- tions in the prompts (Silco and Lernould, 2023). Benchmarks that are not situated also face chal- lenges when it comes to implementing coordination between agents, e.g., aligning intentions towards joint actions (Jain et al., 2019) and pragmatic gen- eration (Zhu et al., 2021a; Bao et al., 2022).
Environment and Task Setups We introduced 9 different ToM evaluation tasks for each mental state under ATOMS, and reality-checking task to test LLMs' understanding of the world. It is im- portant to acknowledge that our experiment serves as a proof of concept and does not aim to cover the entire spectrum of machine ToM, as our case studies are far from being exhaustive or systematic.
Situated evaluation mitigates data contamina- tion. A situated ToM evaluation can mitigate data contamination, as researchers can design scenarios in simulated settings that are unlikely to be part of the LLM's 's training data. Carefully designed bench- marks can also incorporate seen and unseen envi- ronments assess generalization to new tasks and new environments, fundamentally addressing the issue of data contamination (Gandhi et al., 2021).
Given the sequence of actions, predict the closest object at the end of the trajec- tory. The task is designed to test LLMs' under-
Reality Check:

---

Rooas
Rooew
Stage1:
Stage2:
Stage3:
Staged: Agent Dreen: drop roneat
Stage5:
Stage6:
Agent Bed: wander
Agant WAndOT
Red: 9019
sd1e 0oenz-Green: gote roome!
Agant Ried:
rocea) Agent-Greent
Agant
Agent-Breen: 9939
Agent Agent-Breen:
Trajectory
Hall
Where would & Look for O (F cander delief)
Where would think that & nould lock for O
Red
Red
Secced-arder belled)
Figure 4: An overview of the first and second order false belief task illustrated in a grid world setup. We simulate the unexpected transfer scenarios with agents, and verbalize the environment and action traces test if LLMs hold a correct understanding of the agents' false beliefs.
Frightened No strong cmotion
Stage1:
Stage2:
Stage3: AgenttRRe: approoch AponE-ARite
Agent BentttReelhittsSseelle
Agent-Sed:: Agent-Yellew vasel!
How would A feel about the A
Frightened strong ceotion
Seittt-iitthttteWWellll sclle
Agentssed: putth Agent-Velloz 1nto Agest-Whste: idle
Agent=Red:
Agent-Ahite
Figure 5: An overview of the morally related emotional reaction tasks illustrated in grid world setup. We simulate scenarios where agent either directly witnesses is ignorant of morally related event, and verbalize the environment and action traces to test if LLMs hold a correct prediction of the agent's emotional reaction.
standing of relocations in the grid world.
We detail two case studies and leave examples of each task in Appendix A.
Short-term Intention: Given an incomplete tra- jectory and a goal, predict the next action.
Case Study Beliefs. Our belief experi- ments emulate the classic unexpected transfer tasks (Baron-Cohen et al., 1985; Perner and Wim- mer, 1985). As is shown in Figure 4, we simulate this disparity of belief state and world state in Min- iGrid. The first-order belief task features main room with three connected side rooms, two agents named Red and Green, and a ball. Each instance of the belief experiment begins with Green placing the ball in Room#2 while Red watches. Red then enters a separate Room#1 and shuts the door. While Red is inside of this closed room, Green transfers the ball to Room#3. Red presumably holds false belief about the location of the ball, believing it is in Room#3. Similarly,
Long-term Intention: Given an incomplete jectory and a list of subgoals, predict the next subgoal that the agent is planning to achieve.
Desire: Given complete trajectory, predict if the agent demonstrates a preference for objects.
Percepts: Given a complete trajectory, predict if the agent has partial or full observation.
Belief: The classic unexpected transfer task with possible first and second order false belief.
Non-literal Communication: Given trajectory and statement from the agent, judge whether the agent is being deceptive.
Knowledge: Given a trajectory, predict the ob- ject whose location is unknown to the agent.
in Room#2 though implement the second-order belief task to an incorrect belief that one agent holds about the belief of another. After Green has finished trans- ferring the ball, it navigates to the room originally
Emotion: The classic perception-emation link test, where emotions are evoked in response to witnessing an emotionally stimulating situation.

---

Models
Avg. Accuraty
54.1%
GFT3.5 One Shot
50.1%
0.8-
GPT-4 Zero Shot
60.9%
one Shot
65.7%
Accuracy
0.6-
Rnd.
0.4
0.2
0.0-
Reality Check
Short
Long Intent
Desire
Percept
1st bellef
2nd bellef
Non-Literal
Knowledge
Emotion
Intent
Comm.
Figure 6: The LLMs' performance across the 10 tasks is illustrated. Each bar shows how one LLM performed with a specific prompting method. Overall, the tasks are tough for the LLMs tested. While one-shot and CoT prompting help certain cases, their effectiveness is not consistent across the board.
containing the ball and shuts the door. Red then navigates to the room now containing the ball and sees the true location of the ball. Still, Green pre- sumably possesses a false belief about Red's belief. In both tasks, LLMs are queried with two versions of the world: false one with the ball in the original room, and true with the ball in the third room (its actual location). LLMs must correctly respond that the agents hold false belief.
Results and Discussion. We observe that LLMs exhibit some level of sensitivity for some mental states. Especially, GPT-4 scores up to 91% % zero- shot accuracy and 96% one-shot accuracy in the long-term intention task. However, we also high- light the shortcomings of LLMs in some mental states of ATOMS varying degrees, especially, in terms of predicting preferences, perception limita- tions, missing knowledge, and higher-order beliefs. These findings align with previous research (Sap et al., 2022; Trott et al., 2022; Shapira et al., 2023a), further confirming that LLMs are not yet reliable and comprehensive ToM agents. From the reality- checking task, we observe that GPT-3.5 reaches 78% accuracy with CoT prompting and GPT-4 sig- nificantly surpasses its predecessors with 83% zero- shot accuracy and 95% one-shot accuracy. Solving this reality check by no means implies that L.LMs have a general perception ability of the real world, but that as a proof of concept, they demonstrate a certain (but still limited) level of situated aware- ness within the context of basic abstract grid world. This implies that researchers can begin uti- lizing them as powerful building blocks for situated agents in complex ToM tasks. We note that it is always possible come up with more challenging reality-checking questions to expose the limitations of LLMs, or to provide more guided prompts to assist LLMs in successfully completing ToM tasks. Undoubtedly, further research is required along this exciting yet challenging trajectory to advance ToM in LLMs and AI agents built upon LLMs.
Case Study 2: Emotions. While the belief tasks highlight the importance physical situatedness, we further demonstrate that social interactions can be simulated in the grid world. As is shown in Figure 5, We design morally related events that stimulate emotions (e.g., fear, appreciation). In this task, LLMs are queried to predict the emotional response of Agent-Whi te, who either directly wit- nesses or is ignorant of this event. LLMs must correctly respond that the agent holds an emotional reaction only it observes the event.
Experiment Setups. For each task, we create 100 instances following a prompt template that con- sists of [environment description], [agent description], [observability statement], [task statement], [actions sequences], [QA]. We select GPT-4 (gpt-4-0314) and Chat- GPT (gpt-3.5-ture)) 5-turbo-061 for evaluation on the 9 tasks.2 2 Following prior work (Hu al., 2022; Shapira et al., 2023a), we adopt MC-probing for LLMs that don't produce probabilities, which di- rectly instructs LLMs to generate only the letter corresponding to the answer. Besides zero-shot evaluation, we also explored one-shot learning and Chain-of-Thought (CoT) prompting (Wei et al., 2022). More details are available in Appendix B.
Discussions and Action Items
5.1 The Scope of Machine Theory of Mind
WWo use the ChatCompletion. create function from openai package.
Be specific about the mental states studied. Ex- isting benchmarks often lack a clear target mental

---

state, making challenging to interpret the results and measure the actual progress. To mitigate the risk of overestimating LLMs' ToM capabilities, is recommended that future benchmark develop- ers provide specific details regarding the targeted mental state(s) they intend to assess.
Consider a mutual and symmetric ToM. ToM is symmetric and mutual in nature, as origi- nally imputes the mental states of self and others. Prior research is largely limited to passive observer roles (Grant et al., 2017; Nematzadeh et al., 2018; Le et al., 2019; Rabinowitz et al., 2018) or speaker in speaker-listener relationship (Zhu al., 202 Zhou et al., 2023b). We encourage more studies on how humans and agents build and maintain com- mon ground with human ToM and machine ToM through situated communication (Bara et al., 2021; Sclar et al., 2022). Besides, more research is needed to understand LLMs possess early forms of intrinsic mental states given observation cues of the world. While we need to develop ma- chines that impute the mental states of humans, humans should also develop theory of AI's mind (ToAIM) (Chandrasekaran al., 2017) by under- standing the strengths, weaknesses, beliefs, and quirks of these black box language models.
Broaden the Scope of Machine ToM. A breadth of mental states and their sub-domains have al- ready been covered by AT benchmarks (Table 1). We observed an overwhelming emphasis on the benchmarks and modeling of beliefs and intentions, while other aspects have received insufficient at- tention. Still, there are considerably many blank spaces in the landscape of machine ToM, especially for more complicated forms of knowledge, desires, perspective-tasking and emotional experiences be- yond typical social situations.
5.2 Design New Theory of Mind Benchmarks
Avoid shortcuts and spurious correlations. The evaluation of LLMs itself presents sigmificant chal- lenges, not only in the case of ToM. Existing bench- marks suffer from issues such as data leakage and spurious correlations. Especially, shorteut solu- tions have been consistently reported in recent years (Le et al., 2019; Shapira et al., 2023a; Aru et al., 2023). We are in pressing need of new bench- marks with scalable sizes, high-quality human an- notations, and privately held-out sets for evaluation. Avoid unfair evaluations from prompting. Pre- vious work has shown that CoT prompting can im- prove the performance of LLMs in ToM tasks (Li et al.. 2023; Moghaddam and Honey, 2023; Shapira et al., 2023a). Various recent prompting mecha- nisms have also been developed to improve LLM's capability on ToM tasks (Zhou et al., 2023a; Leer al., 2023). In the evaluation of LLMs' ToM capabilities, we recommend the careful documenta- tion of prompts used and the avoidance of implicit human guidance to ensure a fair comparison.
5.3 Neural Language Acquisition and ToM
Both psychological studies (Bloom, 2002; Tomasello, 2005) and computational simula- tions (Liu et al., 2023) have demonstrated the effectiveness of ToM, especially intention, in language acquisition. Instead of concentrating on eliciting ToM in LLMs, we should contemplate whether certain ToM elements should be inherently present in LLMs or perhaps introduced alongside language pretraining. More research is needed understand the connection between neural word acquisition and ToM development in machines.
Conclusion
In this position paper. we survey and summarize the ongoing debate regarding the presence of machine ToM within LLMs, and identify the inade- quate evaluation protocols as the roadblock. Many benchmarks focus only on a few aspects of ToM. and are prone to shortcuts. To mediate this issue, we follow the ATOMS framework to offer a holistic review of existing benchmarks and identify under- explored aspects of ToM. We further call for sit- uated evaluation of ToM, one that is physically situated in environments and socially situated in interactions with humans. We hope this work can facilitate future research towards L.Ms as ToM agents, and offer an intuitive means for researchers to position their work in the landscape of ToM.
Move on to a situated ToM. We call for situ- ated evaluation of ToM, in which the tested LLMs are treated like agents who are physically situated environments and socially situated in interac- tions with others. situated setup covers a wider range of ToM aspects. With carefully designed benchmarks with diverse environments and unseen test sets, a situated setup can help address data contamination issues and assess generalization to new tasks and environments. Furthermore, situ- ated setup allows for more complicated evaluation protocols than simple inference and QA tasks.

---

Ethical Statement
survey and open problems. Artificial Intelligence. 258:66-95. 95.
The dataset created in this study includes instances that are synthetically generated from planners and RL algorithms, as well as ones created by humans. Human subjects research is approved by the Univer- sity of Michigan Health Sciences and Behavioral Sciences Institutional Review Board (IRB-HSBS) under eResearch ID HUM00234647. The text gen- erated by LLMs could potentially contain harm- ful, toxic, offensive content. The authors have ensured that the data does not contain personally identifiable information or offensive content.
Jacob Andreas. 2022. Language models as agent mod- cls. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3769-5779 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
James N Aronson and Claire Golomb. 1999. Preschool- ers' understanding of pretense and presumption of congruity between action and representation. Devel- opmental Psychology, 35(6):1414.
Jaan Aru, Ageel Labash, Oriol Corcoll, and Raul Vi- cente. 2023. Mind the gap: Challenges of deep learn- ing approaches to theory of mind. Artificial Intelli- gence Review, pages 1-16.
Limitations
Our current benchmark only covers 100 instances for each task, adding up to only 1000 instances. Our experiment serves as a proof of concept and does not aim to cover the entire spectrum of ma- chine ToM, as our case studies are far from being exhaustive systematic. In the future, plan to create more systematic benchmark with a larger scale and various forms of evaluation. Additionally, it is worth noting that the ATOMS framework derived from human ToM studies conducted with children under the age of 5. Consequently, this framework primarily focuses on the early devel- opmental stages of ToM, capturing the naive and potentially rudimentary aspects of ToM. For more advanced ToM capability, we point some recent frameworks proposed by Osterhaus and Bosacki (2022) and Stack al. (2022).
Yuwei Bao, Sayan Ghosh, and Joyce Chai. 2022. Leam- ing to mediate disparities towards pragmatic commu- nication. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (ACL).
Cristian-Paut Paaiinn-Paall Bara, Ziqiao Ma, Yingzhuo Yu, Julie Shah, and Joyce Chai 2023. Towards collaborative plan acquisition through theory of mind modeling in situ- ated dialogue. In Proceedings of the Thirty-Secone International Joint Conference ligence, IJCAI-23, pages 2958-2966. International Joint Conferences on Artificial Intelligence Organi- zation. Main Track.
Artificial Intel-
Cristian- Bara, CH-Wang Sky, and Joyce Chai. 2021. Mindcraft: Theory of mind modeling for situ- ated dialogue in collaborative tasks. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 1112-1125.
Simon Baron-Cohen. 1997. Mindblindness: An essay on autism and theory of mind. MIT press.
A Acknowledgements
This work was supported in part by NSF IIS- 1949634, NSF SES-2128623, and by the Automo- tive Research Center at the University of Michigan. Without implying any agreement with the contents presented in this work, the authors extend their appreciation to Susan Gelman for her valuable feed- back. The authors would like thank all anony- mous reviewers for their valuable feedback.
Simon Baron-Cohen. Alan M Leslie, and Uta Frith. 1985. Does the autistic child have "theory of mind"? Cognition, 21(1):37-46.
Simon Baron-Cohen, Michelle O' O'riordan, Valerie Stone, Rosie Jones, and Kate Plaisted. 1999. Recognition of faux pas by normally developing children and children with asperger syndrome or figh-fuuctioning autism. Journal of autism and developmental disor- ders, 29:407-418.
Cindy Beaudoin, Élizabel Leblanc, Charlotte Gagner, and Miriam H Beauchamp. 2020. Systematic review and inventory of theory of mind measures for young children. Frontiers in psychology. 10:2905.
References
Arjun R Akula, Keze Wang. Changsong Liu, Sari Saba- Sadiya, Hongjing Lu, Sinisa Todorovic, Joyce Chai, and Song-Chus Zhu. 2022. Cx-tom: Counterfac- tual explanations with theory-of-mind for enhancing human trust in image recognition models. Iscience, 25( I): 103581.
Mark Bennett and Linda Galpert. 1993. Children's un- derstanding of multiple desires. International Jour- nal of Behavioral Development, 16(1):15-33.
Sarah-Jayne Blakemore and Jean Decety. 2001. From the perception of action understanding of in- tention. Nature reviews neuroxcience, 2(8):561-567.
Stefano V Albrecht and Peter Stone. 2018. Autonomous agents modelling other agents: A comprehensive

---

Paul Bloom. 2002. How children learn the meanings of words. MIT press.
Jesse Dodge, Maarten Sap. Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pages 1286-1305.
Helene Borke. 1971. Interpersonal perception of young children: Egocentrism or empathy? Developmental psychology. 5(2)2663.
Michael Bratman. 1987. Intention, plans, and practical reason. The University of Chicago Press.
Fred I Dretske, 1979. Simple seeing. In Body, mind, and method. pages 1-15. Springer.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine
Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, et al. 2023. Faith and fate: Limits of transformers on com- positionality. arXiv preprini arXiv:2305. 18654.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee. Yuanzhi Li, Scott Lund- berg, al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4, arXiv preprint 127122
Jacquelynne S Eccles and Allan Wigfield. 2002. Moti- vational beliefs, values, and goals. Annual review of psychology, 13111:009133..
Benjamin Eysenbach, Carl Vondrick, and Antonio Tor- ralba. 2016. Who is mistaken? arXiv preprint 1611v:161..01155
Peter Carruthers, 2015. Perceiving mental states. Con- sciousness and cognition, 36:498-500 507.
Nico Frijda et al. 1986. The emotions. Cambridge University Press.
Fulvia Castelli. 2006. The valley task: Understanding intention from goal-directed motion typical devel- opment and autism. British journal of developmental psychology, 24(4):655-668;
Kanishk Gandhi, Gala Stojnic, Brenden Lake, and Moira Dillon. 2021. Baby intuitions benchmark (bib): Discerning the goals, preferences, and actions of others. Advances in Neural Information Process- ing Systems, 34:9963-9976.
Arjun Chandrasckaran, Deshraj Yadav, Prithvijit Chat- topadhyay, Viraj Prabhu, and Devi Parikh. 2017. takes two to tango: Towards theory of ai's mind. arXiv preprint 1rXiv:17040007
Edmund Gettier. 2000. Is justified true belief knowl- edge? Causal Theories of Mind, page 135.
Maxime Boiisve,,,Boisvert, Lucas Willems, and Suman Pal. 2018. Minimalistic gridworld environment for gymnasium.
Rachel Giora. 2003. On our mind: Salience, context, and figurative language. Oxford University Press.
Michael Cohen. 2021. Exploring roberta's theory of mind through textual entailment.
Alison Gopnik and Henry M Wellman. 1992. Why the child's theory of mind really is a theory. Mind & Language,
Philip R Cohen and Hector J Levesque. 1990. Intention choice with commitment. Artificial intelligence, (2(23)):13-2611
Andrew Gordon. 2016. Commonsense interpretation of triangle behavior. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30.
Cristina Colonnesi, Carolien Rieffe, Willem Koops, and Paola Perucchini. 2008. Precursors of theory of mind: A longitudinal study. British Journal of Devel- opmental Psychology, 26(4)756
Erin Grant, Aida Nematzadeh, and Thomas L Griffiths. 2017. How memory-augmentt neural networks pass false-belief task? In CogSei.
Antonio R Damasio. 2004. Emotions and feelings. In Feelings and emotions: The Amsterdam symposium, volume 5, pages 49-57. Cambridge University Press Cambridge.
David Gunning. 2018. Machine common concept paper. arXiv preprint 1rXiv:1810.07526
Julie Hadwin, Simon Baron- Cohon, Patricia Howlin, and Katie Hill. 1997. Does teaching theory of mind have an effect on the ability to develop conversation in children with autism? Journal of autism and developmental disorders, 27:519-537.
Susanne A Denham. 1986. Social cognition, prosocial behavior, and emotion in preschoolers: Contextual validation. Child development, pages 194-201.
Daniel Dennett. 1995. Do animals have beliefs. Com- parative approaches to cognitive science, 111.
Thilo Hagendorff. 2023. Machine psychology: Inves- tigating emergent capabilities and behavior in large language models using psychological methods.
Daniel Dennett. 1988. Précis of the intentional stance. Behavioral and brain sciences, 11 495555555505.

---

Francesca GE Happé. 1994. An advanced test of theory of mind: Understanding of story characters' thoughts and feelings by able autistic, mentally handicapped, and normal children and adults. Journal of autism and Developmental disonlers, 24(2):129-155
Ariel Knafo, Carolyn Zahn-Waxler Maayan Davidow, Carol Van Hulle, JoAnn L Robinson, and Soo Hyun Rhee. 2009. Empathy in early childhood: Genetic, environmental, and affective contributions. Annals of the New York Academy of Sciences, 1167(1):103- 114.
Paul L Harris, Kara Donnelly, Gabrielle R Guz, and
Rosemary Pitt- Watson. 1986. Children's understand- ing of the distinction between real and apparent emo- tion. Child development, pages 895-909,
Michal Kosinski. 2023.
Theory of mind may have spon- taneously emerged in large language models.
Nicole C Krämer, Astrid der Pütten, and Sabrina Eimler. 2012. Human-agent and human-robot inter- action theory: Similarities to and differences from numan-human interaction. In Human-compan in- teraction: The agency perspective, pages 215-240. Springer.
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zor- nitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2023. Methods for measuring, up- dating, and visualizing factual beliefs in language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu- tational Linguistics, pages 2706-2723.
Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLI)) pages 5872-5877, Hong Kong. China. Association for Computational Linguistics.
Yinghui He, Yufan Wu, Yilin Jia, Rada Mihalcea, Yu- long Chen, and Naihao Deng. 2023. Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models. In Find- ings the Association for Computational Linguistics: EMNLP 2023.
Mark K Ho, Rebecca Saxe, and Fiery Cushman. 2022. Planning with theory of mind. Trends Cognitive Sciences.
Courtland Leer, Vincent Trost, and Vineeth Voru- ganti. 2023. Violation of expectation via metacog- nitive prompting reduces theory of mind predic- tion error in large language models. arXiv preprint arXiv:2310.0694
Bart Holterman and Kees van Deemter. 2023. Does chatgpt have theory of mind? arXiv preprint arXiv:2305. 14020.
Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2022. A fine- grained comparison of pragmatic language under- standing in humans and language models. arXiv preprint arx::22122.08011
Belinda Z Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural lan- guage models. In Proceedings of the 59th Annual Meeting of the Association for Compurational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers). pages 1813-1827.
Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Raste- gari, Svetlana Lazebnik, Ali Farhadi, Alexander G Schwing, and Aniruddha Kembhavi. 2019. Two body problem: Collaborative visual task completion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 6689- 6699.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023, Camel: Communicative agents for" mind" explo- ration of large scale language model society. arXiv preprint arXhv 17760.
Ao Jia, Yu Hc, Yazhou Zhang, Sagar Uprety, Dawei Song, and Christina Lioma. 2022. Beyond emotion: A multi-mone modal dataset for human desire understand- ing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- giex, pages 1512-1522.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clin- ton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. 2022. Pre- trained language models for interactive decision- making. Advances in Neural Information Processing Systems, 35:31199-3122
Andy Liu, Hao Zhu, Emmy Liu, Yonatan Bisk, and Graham Neubig. 2023. Computational language ac- quisition with theory of mind. In The Eleventh Inter- national Conference on Learning Representations.
David Kavanagh, Jackie Andrade, and Jon May. 2005. Imaginary relish and exquisite torture: the elabo- rated intrusion theory of desire. Psychological re- view, 112(2):446
Casey Kennington. 2022. Understanding intention for machine theory of mind: a position paper. In 2022 st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pages 450-453. IEEE
Inbal Magar and Roy Schwartz. 2022. Data contamina- tion: From memorization to exploitation. In Proceed- ings of the 60th Anmual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers).
pages 157-165.

---

Bertram Malle and Joshua Knobe. 2001. The distinc- tion between desire and intention: A conk-conceptuall analysis. Intentions and intentionality: Foundations of social cognition, 45:67.
Ann T Phillips, Henry Wellman, and Elizabeth S Spelke. 2002. Infants' ability to connect gaze and emotional expression to intentional action. Cogni- rion, 85(1):53778.
Gary Marcus and Ernest Davis. 2023, How not to test gpt-3.
F. Pons and P. Harris. 2000. Test of Emotion Compre- hension: TEC. University of Oxford.
David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory mind? Behavioral and Brain Sciences, 1(4):515-526.
Zenaida S Masangkay, Kathleen A McCluskey, Cur- tis W McIntyre, Judith Sims-Knight, Brian E Vaughn, and John H Flavell. 1974. The early development of inferences about the visual percepts of others. Child development, pages 357-366.
Liang Qiu, Yizhou Zhao, Yuan Liang, Pan Lu, Weiyan Shi, Zhou Yu, and Song-chun Zhu. 2022. Towards socially intelligent agents with mental state transition and human value. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 146-158.
Andrew N Meltzoff. 1995. Understanding the intentions of others: re-enactment of intended acts by 18-month- old children. Developmental psychology, (5)8888
Melanie Mitchell and David C Krakauer. 2023. The de- bate over understanding in ai's large language models. Proceedings of the National Academy of Sciences, 120(13)::2211900722
Neil Rabinowitz, Frank Perbet, Francis Song. Chiyuan Zhang. SM Ali Eslami, and Matthew Botvinick. 2018. Machine theory mind. In International conference on machine learning, pages 4218-4227. PMLR.
Inioluwa Dehorah Raji, Emily Denton, Emily M Bender, Alex Hanna, and Amandalynne Paullada. 2021. Ai and the everything in the whole wide world bench- mark. In Thirty-fifth Conference on Neural Informa- rion Processing Systems Datasets and Benchmarks Track (Round 2).
Shima Rahimi Moghaddam and Christopher J Honey. 2023. Boosting theory-of-man -fnd performance in large language models via prompting. arXiv preprint 2en:2334.719900
Henrike Moll, Cornelia Koring, Malinda Carpenter, and Michael Tomasello. 2006. Infants determine others' focus of attention by pragmatics and exclusion. Jour- nal of Cognition and Development, 7(3):411 430.
Betty M Repacholi and Alison Gopnik. 1997. Early reasoning about desires: evidence from 14-and 18- mouth-olds. Developmental psychology. 33(1):12.
Aida Nematzadch, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. 2018 Evaluating theory of mind in question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2392-2400, Brussels, Belgium. Association for Computational Linguistics.
Ted K Ruffman and David R Olson. 1989. Children's ascriptions knowledge to others. Developmental Psychology, 25(4))601.
Laura Ruis, Akbir Khan, Stella Biderman. Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2022. Large language models are not zero-shot communi- cators. arXiv preprint arXiv:2210.1499
OPenAI. 2022. Chatgpt: Optimizing language models for dialogue.
Tessa Rusch. Saurabh Steixner Kumar, Prashant Doshi, Michael Spezio, and Jan Gläscher. 2020. Theory of mind and decision science: towards typology of tasks and computational models, Neuropsychologia, 146: 107488.
OpenAl. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
Christopher Osterhaus and Sandra I. Bosacki. 2022. Looking for the lighthouse: A systematic review of advanced of tests beyond preschool. De- velopmental Review, 64: 10102..
Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? on the limits of social intelligence in large LMs. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 3762-3780, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Gonçalo Percira, Rui Prada, and Pedro A Santos. 2016. Integrating social power into the decision-making of cognitive agents. Artificial Intelligence, 241:1 144.
Josef Perner, Susan R Leekam, and Heinz Wimmer. 1987. -hree-year-olds' difficulty with false belief: The case for a conceptual deficit. British journal of developmental psychology, 5(2):125-137.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Bras, and Yejin Choi. 2019. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pages 4463- 4473, Hong Kong, China. Association for Computa- tional Linguistics.
Josef Perner and Heinz Wimmer. 1985. "john thinks that mary thinks that..." attribution of second-order beliefs by 5-to 10-year-old children. Journal of ex- perimental child psychology, 39(3):437-471.

---

Roger Schank and Robert P Abelson. 2013. Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology press.
Ece Takmaz, Nicolo' Brandizzi, Mario Giulianelli, San- dro Pezzelle, and Raquel Fernández. 2023. Speak- ing the language of your listener: Audience-aware adaptation via plug-and-play theory of mind. In Find- ings of the Association for Computational Linguistics: ACI. 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint 1xxiv:707.00644)
Michael Tomasello. 2005. Constructing language: A usage-based theory of language acquisition. Harvard university press.
Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. Minding lan- guage models' (lack of) theory of mind: A plug-and- play multi-character belief tracker.
Jennifer Tracey, Owen Rambow, Claire Cardie, Adam Dalton, Hoa Trang Dang, Mona Diab, Bonnie Dorr, Louise Guthrie. Magdalena Markowska, Smaranda Muresan, et al. 2022. Best: The belief and sentiment corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2460- 2467.
Melanie Sclar, Graham Neubig, and Yonatan Bisk. 2022. Symmetric machine theory of mind. Proceedings of the 39th International Conference on Machine Learning. volume 162, pages
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap. and Vered Shwartz. 2023a. Clever hans neu- ral theory of mind? stress testing social reasoning in large language models.
Sean Trout, Cameron Jones, Tyler Chang, James Michaelov, and Benjamin Bergen. 2022. Do large language models know what humans know? arXiv preprint 25
Natalie Shapira, Guy Zwirn, and Yoav Goldberg. 2023b. How well do large language models perform on faux pas tests. In Findings the Association for Compu- tational Linguistics: ACL 2023.
Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-min tasks.
Qiaosi Wang, Koustuv Saha, Eric Gregori. David Joyner, and Ashok Goel. 2021. Towards mutual theory of mind in human-ai interaction: How language reflects what students perceive about a virtual teaching assis- tant. In P Proceedings of the 2021 CHI Conference on Human Factors Computing Systems, pages 1-14.
Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin Smith, Shari Liu, Dan Gutfreund, Elizabeth Spelke, Joshua Tenenbaum, and Tomer Ullman. 2021. Agent: A benchmark for core psychological reason- ing. In International Conference on Machine Learn- ing, pages 9614-9625. PMLR.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. 2022. Chain-of-hhough prompting elicits rea- soning in large language models. In Advances in Neural Information Processing Systems.
Damien Sileo and Antoine Lernould. 2023. Mindgames: Targeting theory of mind in large language models with dynamic epistemic modal logic. arXiv preprint MiXiv:2305.03353.
Patricia Smiley. 2001. Intention understanding and partner-sensitive behaviors in young children's 's peer interactions. Social Development, 10(3):330-33 3344.
Henry M Wellman and Jacqueline Woolley. 1990. From simple desires to ordinary beliefs: The early development of everyday psychology. Cognition, 35(3):245-273
Caoimhe Harrington Stack, Fffaat Farhana, Xinyu Shen, Simeng Zhao, and Angela Maliakal. 2022. Frame- work for multi-dimensional test of theory of mind for humans and ai systems. In The Tenth Annual Conference on Advances in Cognitive Systems.
Jincenzi Wu, Zhuang Chen, Jiawen Deng, Sahand Sabour, and Minlie Huang. 2023. Coke: A cognitive knowledge graph for machine theory of mind. arXiv preprint arXiv:2305.0533
Shane Storks, Qiaozi Gao, Yichi Zhang, and Joyce Chai 2021. Tiered reasoning for intuitive physics: Toward verifiable commonsense language understanding. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 4902 4918.
Wako Yoshida, Ray Dolan, and Karl J Friston. 2008 Game theory of mind. PLoS computational biology, 4(12)e1000254.
Mo Yu, Yisi Sang, Kangsheng Pu, Zekai Wei, Han Wang, Jing Li, Yue Yu, and Jie Zhou. 2022. Few-shot character understanding in movies as an assessment to meta-learning of theory-of of mind, arXiv preprint arXiv:2211.00 .46684.
Kate Sullivan, Ellen Winner, and Natalie Hopfield. 1995. How children tell a lie from a joke: The role of second-ord -oddr mental state attributions. British jour- nal of developmental psychology, 13(2):191-204.
S Swettenham. 1996. Can children be taught to under- stand false belief using computers? child psychology & psychiatry & allied disciplines, 37 (2), 157-165.
Jamil Zaki, Niall Bolger, and Kevin Ochsner. 2009. Un- packing the informational bases of empathic accuracy. Emotion, 9():478.

---

Chen Zhang and Joyce Y Chai. 2010. Towards con- versation entailment: An empirical investigation. In Proceedings of the 2010 Conference Empirical Methods in Natural Language Processing, pages 756- 766.
Haode Zhang. Yuwei Zhang, Li Ming Zhan, Jiaxin Chen, Guangyuan Shi, Xiao Ming Wu, and Albert YS Lam. 2021. Effectiveneess of pre-training for shot in- tent classification.. In 2021 Findings of the Asso- ciation for Computational Linguistics, Findings of ACL: EMNLP 2021, pages 1114-1120, Association for Computational Linguistics (ACL).
Pei Zhou, Aman Madaan, Srividya Pranavi Potharaju, Aditya Gupta, Kevin R McKee, Ari Holtzman, Jay Pujara, Xiang Ren, Swaroop Mishra, Aida Ne- matzadeh, et al. 2023a. How far are large language models from agents with oheory-of-mind? arXiv preprint 2r30v23310.0355.
Pei Zhou, Andrew Zhu, Jennifer Hu, Jay Pujara, Xiang Ren, Chris CallisonBurch, Yejin Choi, and Prithviraj Ammanabrolu. 2023b. I cast detect thoughts: Learn- ing converse and guide with intents and theory- of-mind in dungeons and dragons. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguisties.
Hao Zhu, Graham Neubig, and Yonatan Bisk. 202 Few-shot language coordination by modeling theory of mind. In International Conference Machine Learning, pages 12901-12911. PMLR.
Hao Zhu, Graham Neubig, and Yonatan Bisk. 202 lb. Few-shot language coordination by modeling theory of mind. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12901-12911. PMLR

---

A Task Settings and Data Collection
by the agent, and the agent's location and face direc- tion for each step. For more details on prompting. please refer to section B.
In this section, we provide an in-depth explana- tion of the ten tasks outlined in section 4.2. Task 0 serves as a "reality check" assess LLMs' grasp of the physical world, particularly relocations within the grid world. Tasks through cach empha- size distinct facets of ToM. All these tasks utilize MiniGrid, a streamlined 2D grid-world environ- ment. CChevalier-Boiseerr et al., 2018).
The data for Task are autonomously generated
using seeds and a shortest-path planner.
Task Short Term Intention
Task setting:
navigate to one of the two objects
Q: Which action will
Task O: Reality Check
the agent take next?
Left
B. Right
Task setting: visiting 8-2 of the objects Which item is the agent closer to at the end of given trajectory?
C. Forward
Figure 8: Illustration for Short Term Intention task
A. Yellow ball
In this scenario, the agent is tasked with visiting either of the two objects in the grid world. The LLM provided with the goal object. Rather, must determine the goal object by examining and understanding the agent's trajectory. In this task, the agent has full observation in the world, and there are exactly two objects placed. The object types and colors are randomly generated. The size of the room range 6 by 6 to 12 by 12.
B. Blue ball
Figure 7: Illustration for Reality Check task
In this scenario, the agent is tasked with visiting 0-2 of the objects in the grid world. The agent has full observation in the world for efficient naviga- tion. At least objects (12 maximum) are placed in the environment. To test LLM's ability to understand the physical actions taken by the agent, we ask it about the distance between the agent and various objects after it accomplishes a number of actions. The action planner either a shortest path planner towards specified objects or a random ac- tion generator.
be randomly sampled from the
To test an LLM's understanding short-term intention, we ask agent given its previous trajectory.
predict the next action of the
The agent's trajectory halts at random step. with the exception of the precise moment when the optimal paths to the two objects diverge. This "cutting point" is set as the mean for the normal distribution from which the stopping point sampled.
After the agent has finished 10 random actions, right after has visited one object with an opti- mal action planner, the task-related question will be generated with the following format:
exception and also serves
By restricting the cutting point in this manner, we guarantee the trajectory included the prompt to be optimal for reaching the potential goal objects. This reduces the ambiguities of our experiments, and thus improves the significance of our results.
After having taken these actions, which item is the agent closer to?
A. cobject1.colors <object1 name>
B. object2.colors <object2. name>
The task-related question has the following for-
mat:
Here object1 and object2 both exist in the environment, and one of them is guaranteed to be the target object if there is such a goal. There are always two options for this task.
Which action will the agent take next?
A. left
B. right
Along with the task-related question, the prompt includes description of the grid world environ- ment, the action space of the agent (only going forward, turn left right in this task), a board-like depiction of the initial state, the list actions taken
C. forward
The LLM should able to choose which action would be next were the agent to continue its optimal trajectory to the goal object. The data for Task are

---

autonomously generated using seeds and shortest- path planner.
Task 2: Long Term Intention
2 Open the door
(Ongoing)
14
@Pickup ball (Pending)
① Pickup key (Finished)
Task setting: complete three subgoals in few steps possible
Task setting: pickup objects the environment, It may may not have its own preference over some objects. object (if any) does the agent prefer?
Q: which subgoal is the agent currently
trying to complete?
A. Locate and pick up a key
B.
Locate and go through a door
A. Blue ball
C. Navigate to the object in the new
В. White ball
C. No preference
room
Figure 10: Illustration for Desire task
Figure 9: Illustration for Long term intention task
In this scenario, the agent needs to complete the following subgoals as few steps as possible: I) Locate and pick up a key; 2) Locate and go through door; 3) Navigate to the object in the new room. There are two rooms in this setting, which are connected by locked door. The key of the door always in the room in which the agent is initially located. The object is always placed in the other room.
preferred object type and in the other 50% the agent lacks preference). We also deduct from the final reward given to the agent when it takes a large number of steps to finish picking up three objects.
This task tests whether an LLM is able to deter- mine the desire of the agent (for one object type or the other) by examining its trajectory. We prepared the following question for this task:
Which object does the agent prefer?
We provide an LLM with a subset of the agent's trajectory and ask it which subgoal the agent is currently trying to complete.
A. white ball
B. blue ball
The task-related question has the following for- mat:
C. preference
The data for Task are autonomously generated using seeds and Reinforcement Learning. We use the PPO algorithm (Schulman et al., 2017) to train the model. In the scenario wherein preference is present, the preferred object type yields 10 times more reward than the non-preferred one. In the scenario wherein the preference absent, both object types yield identical rewards.
Based on the agent's trajectory thus far, which subgoal is the agent currently trying to complete?
A. Locate and pick up key
B. Locate and go through door
C. Navigate to the object in the new room
The data for Task 2 are autonomously generated using seeds and shortest-paie -att planner.
Task 4: Percept
In this scenario, the agent is instructed to navi- gate in two rooms and reach the goal in the other room fast as possible. In contrast with previous task settings, the agent has either very limited visual range (3 3 grid in the front), or an "in- finitely" large visual range (for practical purposes, the visual range actually 101 x 101 grid). Nat- urally, an agent with smaller viewing range will
Task 3: Desire
In this scenario, the agent is required to pick up three objects soon as possible. There 2 types of objects the world (e.g., blue balls and white balls), 3 of each. The agent may or may not have a preference for one object type (we stratify the data such that in 50% the episodes the agent has

---

Human POV
LLM POV
transfer ball from one room another. Presum- ably, therefore, the agent falsely believes the ball be its previous location. The second-order belief task extends the first-order belief task by enabling the agent with the false belief to see the ball its new location. The other agent, however, does not witness the first agent rectifying its false belief. so it presumably holds second-order false belief (about the belief state of the first agent). By vary- ing the observations that each agent makes (e.g.- whether or not each agent sees the transfer of the ball from one room to the another) as well as the order of these events, this task setting allows us to check an LLM's first-order and second-order belief capabilities.
3x3 view range only
Short
Perception Range Long
Different behaviors perception
Directly see the layout
Task setting: navigate green ball given perception range.
Given only environment layout agent's trejectory. what is the most likely perceptual range that the
possesses?
blocks
B, Infinitely many blocks
*The perceptual
refers the number cells that
agent can
Rather than asking the LLM directly where to find the object, provide two board-like belief states options. We then query the LLM about which board-like state the agent is more likely believe.
Figure Illustration for Percept task
make more mistakes (c.g., navigating to dead end) while trying to reach the goal object. Obstacles are randomly placed in each room to block the agent's view.
Data for tasks are collected via rule-based planners with several scenarios.
LLMs are expected only look the trajec- tory of the agent in an environment, and determine whether the agent has a limited view range or a nearly full view range. The question format as follows:
Task 7: Non-Literal Communication
have navigated to two abjects.
have not visited white ball yet.
Based on the agent's actions, what is the most likely perceptual range that the agent possesses? The perceptual range refers to the number of cells that an agent can see in front of it.
Then visit yellow bail
Navigate white ball
Task setting: visiting the objects, report its progress with Q: The agent claims that navigated ted to blue ball. Based the agent' actions, is it telling the truth?
point
A. blocks
B. infinitely many blocks
Yes No
We manually collected 100 trials in total for both situations.
Figure 12: Illustration for Non-literal communication task
Task 6: First and Second Order Belief In this scenario, there are two agents in the environ- ment. Both them are initially the main room (on the left side the whole grid world; see Figure 4). On the right side, there are three small rooms. Agents can freely go in and out of each room. They can see everything inside the current room and can the other rooms through the door if is open. This task reproduces the unexpected transfer (Sally-Anne) test (Baron-Cohen et al., 1985; Perner and Wimmer, 1985), with both first-order and second-order belief checking. the first-order belief task, one agent does not see the second agent
This task focuses specifically on one form non-literal communication: lying. Within this task, the LLM is told explicitly that there is an agent tasked with navigating to all of the objects within the grid world. In each instance of the task, how- ever, the agent only visits subset of the objects in the environment. The LLM is subsequently told that the agent has claimed success in visiting a par- ticular object. This object is randomly selected so that sometimes it an object that has actually been visited and other times is not.

---

The question format is follows:
it know the position of <color> <name> before?
Based on the agent's actions, is it
telling the truth?
A. Yes
A. yes
B. No
B. no
The data for Task are autonomously generated using seeds and a rule-based planner.
To successfully complete this task, the LLM must combine its knowledge about the physical oc- currences taking place within the grid world with its knowledge about lying, vital component in Beaudoin al. (2020)'s category of non-literal communication.
Task 9: Emotions
This task (see Figure 5) requires LLMs to infer the emotions of agents in situated context from their physical behaviors. Specifically, it involves variations on a theme involving small lake of wa- ter. In every variation, one agent pushes another agent into the lake. some of these variations, an observer is privy to the situation. The prompt then asks the LLM about this observer's feelings towards both the victim and the perpetrator. Pre- sumably, the observer should experience sympathy for the victim and anger (or similarly negative emotion) for the perpetrator.
Task 8: Knowledge
Agent starts to look for two balls
20
Then it
explores empty room
Agent directly goes blue ball
32
Finally the white found
How would <observer name> most likely feel about <pusher name>?
A. no strong emotion
B. angry
Task setting: pick both two objects the environment with limited visibility. One their location is known. The other is unknown. LLM will only see the environment layout and agent's trajectory.
In other variations, helper comes along and pulls the victim out of the lake. Presumably, the observer should feel positive emotions (e.g. respect, gratitude) for this helper. The question format is as follows:
Q: Based on the agent's s actions, does it know the position blue ball before? A. Yea
Figure 13: Mustration for Knowledge task
How would <observer. name> most likely feel about <helper name>?
This task requires the agent to pick up two ob- jects that exist in the environment. Both two objects are known be placed separately two of the four rooms.
A. strong emotion
B. respectful
B Prompting and Reproducibility
In this scenario, the agent is informed of the loca- tion of one object, while the location of the second object remains unknown. The agent is instructed to collect the objects in specific sequence. Ideally, if the agent knows object's location, should proceed directly the appropriate room. Other- wise, it will have to search for the object in the -et-to-be-explored rooms.
In this pilot study, our data curation follows a uniform structure across all tasks similar to prior work (Li et al., 2022), deviating only slightly to account for task-specific circumstances."
Environment Description Each prompt begins with description of the two-dimensional world wherein the task will take place. Specifically, prompting code provides LLMs with the dimen- sions of the game board and a method reference specific cells (column-first Cartesian coordinates).
We include the agent's entire trajectory in the prompt. We then ask the LLM to determine the agent knows the position of one object before. The question format is presented as follows:
'The data for this pilot study is available at https:// // huggingface co/ ht/ddaaaas///eeee /stss//seddddieee (aaichh22DAA0 220///0--ATSSSS
Based on the agent's actions, does

---

Each prompt subsequently itemizes the various ob- jects that are situated in the world, along with their coordinates and attributes. Although this prompt- ing structure could be easily adapted to handle many different types of attributes, we focused only on color for the sake of simplicity. Additionally, this section assigns each object "label": single letter that the prompt uses to represent the object in printed grid representation.
portion of the prompt, so their contents vary, how- ever, they are always multiple-choice. Addition- ally, they always contain a set of instructions below heeding the LLM to return only a single letter its response.
Agent Description, Observability, and Task The next section of each prompt is a detailed de- scription of the agent(s) occupying the grid world. In the example below, which was taken from task I, only one agent occupies the grid world. In multi- agent tasks, this section details the various proper- ties of all agents. Whether single or multi-agent, however, the basic properties are the same. The prompt first details the position and direction of the agent. It is located in a specific cell, and always faces up, down, left, or right. The prompt then spec- ifies the various actions that an agent is capable of taking (e.g. "forward", "left", and "open"). The agent's labels are then specified. Within the printed grid representation, the agent is always represented V-like shape depending on the direction that it facing. For instance, the prompting tool uses to represent an agent that is facing left. Finally, the prompt specifies two key attributes of the agent: (I) its level observability (e.g. whether or not it can see into adjacent rooms) and (2) its goal. Some- times these two descriptors are heavily modified, restricted, or removed altogether that they do not interfere with the task. For instance, when testing LLMs for percepts, the prompt does not specify the visual range of the agent.
Action Sequences Following the agent descrip- tion section, the prompt prints out a board represen- tation, multi-line sequence of plain text that ap- pears two-dimen wwo-dimensional when printed out. Here, the various objects and agents depicted in position by their associated labels. Additionally, the config- uration of the walls specified by a perimeter of W's. Next, the prompt specifies sequence of ac- tions that take place over the course of the episode being considered by the LLM. In the episode de- picted below, the agent navigates part of the way to red box. These actions always specify the new position and orientation of the agent.
Questions and Answer Candidates Finally, each prompt contains a question that the LLM must answer. These questions are the most tasc-spec

---

Sample Prompt for Task
The agent has full observability, meaning it can see the entire world
This is a grid-like 2D world The grid world consists of 6 rows and 6 columns, 0-based
The agent has been instructed to navigate to one of the two objects in the environment, although you do not know which
We use (i,j) to represent the i-th column (from left to right) and j-th row (from top to bottom).
The following is list of objects in this world. Each line starts wit ihh the object's position iisitionon and is followed by its attributes
This is the starting state of the board:
4
(2, 3): key, grey; represented by this
0
W
W
W
W
label: G
1
W
0
0
0
O
W
(4, 4): box, red; represented by this
2
W
0
0
W
label: H
3
W
O
G
O
O
W
4
W
O
O
H
W
Walls are depicted using the symbol W
5
W
W
W
W
There is agent (2, 2) facing left
This list contains sequence actions taken by the agent (Step 1) The agent took action left and is now at (2, 2) facing down (Step 2) The agent took action left
The agent can take the following actions:
left: makes the agent face left of where it is currently facing
and is now at (2, 2) facing right (Step 3) The agent took action forward
right: makes the agent face right of where it is currently facing
and is at (3, 2) facing right
(Step 4) The agent took action forward
forward: makes the agent move one step the direction it currently facing
and is now at (4, 2) facing right
(Step 5) The agent took action right
and is now at (4, 2) facing down
open: makes the agent open a door that it is in front of
Which action will the agent take next?
pickup: makes the agent pick up the object that it is in front of drop: makes the agent drop item
A: left
B: right
C: forward
that it is holding
Please ONLY respond using the letter corresponding to your answer Do not generate any text other than the letter
stay: makes the agent stay where it currently is for timestep
The agent is represented by the following labels depending on which direction it is facing:
Facing left:
Facing up:
Facing right:
Facing down:


Reviews:
[{"review_id": "RMQf5YtpYc", "reviewer": "Reviewer_myQj", "summary": "", "questions": "1. For data contamination, why would it be necessary to have access to the datasets used to train LLMs before evaluating the performance of LLMs on ToM tasks? There are tons of benchmarks used for LLM evaluation, why would ToM tasks be different from others?\n2. In line 257-261, why would ATOMS be developed from young children, but \"cannot be easily deployed on state-of-the-art AI agents\"?\n3. Do you also input the graphs in the 10 tasks as illustrated in Figure 4 to GPT-4/ChatGPT in your experiment setup? If not (according to the appendix in my understanding), how are the images used?\n4. Why do you choose to use a zero-shot setup for evaluation instead of few-shot prompting?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions (“structured review for existing formulation, evaluation, and limitations”, “situated evaluation that can be interesting to researchers studying theory of mind in the community”). We appreciate their constructive feedback on our efforts, and would like to make the following responses. We are looking forward to the reviewer’s further comments during the discussion period.\n\n### Responses to Weaknesses\n\n> **W1.1**: Although the community may benefit from the study and advocated holistic landscape by resolving some of the current limitations, it is not convincing from the paper why each part of the taxonomy derived from 0-5 years old children would be necessary for evaluating theory of mind in a large language model. \n\nThe taxonomy is based on developmental psychological studies, as these aspects have been considered as a part of human’s ToM and are demonstrated during human’s early development (e.g., before 5 years old). ToM evaluations are often conducted on pre-school children to detect early signs of developmental problems (e.g., autism).  While possessing these 7 mental states does not imply a complete human-level ToM, a lack of understanding for any of these mental states can imply a limitation in LLM’s ToM capability.\n\n> **W1.2**: More importantly, it is not clear why the proposed 10 tasks would be better at evaluating theory of mind compared to all other baselines (e.g., emotion), especially when there is no detailed results and analysis in either the main paper or the appendix.\n\nWe believe a situated perspective can cover more aspects of ToM and avoid a crucial portion of shortcuts introduced by textual templates and data contamination (section 4.1). The goal of this paper is to offer a new perspective of evaluating ToM in large models and showcase examples of designing such benchmarks. The proposed 10 tasks are used as a proof of concept to demonstrate how to systematically and holistically study situated ToM that covers many aspects of mental states, which go beyond individual types of mental states (e.g., emotion as raised by the reviewer) which are separately studied in the past. \n\nPosition papers often focus on general discussions on the state-of-the-art, key limitations, and future directions. Here we go beyond the traditional form and use an example as a proof of concept to supplement our discussion. We will certainly add more details about this example in the revision.  \n\n> **W2**: Theory of mind evaluation may be impacted by RLHF in large language models, but the situated ToM evaluation (at least from the 10 tasks introduced in the paper), are not related to how the LLMs would respond to different aspects defined.\nWe are not sure what the review meant by “situated ToM evaluation are not related to how the LLMs would respond”. We try our best to clarify with the following aspects, and would appreciate any additional clarifications from the reviewer.\n\n- **ToM evaluation and RLHF**: While previous research shows that RLHF improves LLM’s understanding of user’s instruction intention, which is an aspect of LLM’s ToM **capability**, we are not aware how existing ToM **evaluation** can be affected by RLHF. To the best of our knowledge, we compare the LLMs before and after RLHF on their performance on ToM tasks, which also applies to our proposed situated ToM evaluation methods.\n- **Does human alignment improve LLM’s situated ToM?**: Previous research shows that RLHF improves LLM’s understanding of user’s instruction intention, which is only one aspect of LLM’s ToM **capability**. We use 10 situated tasks to model a variety of ToM which go beyond instruction intention. We conducted further study to evaluate the role of human alignment on these tasks. To do so, we compare the results based on LlaMA-7B (no alignment) and Vicuna-7B (with alignment) on 30 randomly selected test for each task (due to the time constrain of the rebuttal period) as shown below. We use LM-probing to select the choice with the highest probability following Brown et al. 2020 and Shapira et al. 2023a. Surprisingly, we found that these models make similar predictions on the same task, leading to similar performance. We see that human alignment may have a limited impact on situated ToM capabilities for relatively smaller LLMs (i.e., 7B models). \n\n| **LLM** | **Reality Check** | **Short Intent** | **Long Intent** | **Desire** | **Percept** | **1st belief** | **2nd belief** | **NL Com.** | **Knowledge** | **Emotion** |\n| ----------- | ----------- | ------------ | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n|  LLaMA-7B  | 46.67% | 16.67% | 26.67% | 50.00% | 43.33% | 50.00% | 46.47% | 33.33% | 40.00% | 44.44% |\n| Vicuna-7B | 46.67% | 16.67% | 26.67% | 50.00% | 43.33% | 50.00% | 50.00% | 33.33% | 40.00% | 44.44% |\n\nAs we are not sure if we completely understand the reviewer’s comment, we would appreciate additional clarification if our answers do not address the concern.  \n\n### Responses to Questions\n\n> **Q1**: For data contamination, why would it be necessary to have access to the datasets used to train LLMs before evaluating the performance of LLMs on ToM tasks? There are tons of benchmarks used for LLM evaluation, why would ToM tasks be different from others?\n\nData contamination is one of the major concerns for evaluating pre-trained models. There is no exception for LLM as they are trained in internet scale data, and we don’t know if the model has access to the test set during pretraining. Indeed it’s the case that there are tons of benchmark tasks (including existing story-based ToM benchmarks), due to the nature of the closed models, we have to take these evaluations with a grain of salt (Anna Roger’s blog attested to that: https://hackingsemantics.xyz/2023/closed-baselines/). \n\nMore specifically to ToM evaluation, the training corpora of LLMs contain many research articles describing cognitive tests and psychological studies. Many of the previous studies employed the same language prompt for testing children, or only made minimal change of wording. This leads to a data leakage issue in LLM’s ToM evaluation. This concern was shared by many previous works, e.g., Ullman 2023.\n\nOur situated ToM Task, using simple grid-world setup, intends to alleviate the potential problem of data contamination. \n\n\n> **Q2**: In line 257-261, why would ATOMS be developed from young children, but \"cannot be easily deployed on state-of-the-art AI agents\"?\nWe believe that there might be a misunderstanding of our original statement, we mean exactly the opposite: \n> The meta-analysis has focused on young children aged 0-5 years at the early stage of their cognitive development, and the experimental setups could be relatively simple and comparable, **without** complicated social interactions that cannot be easily deployed on state-of-the-art AI agents. \n\nAn extensive body of previous works has shown that very basic skills, e.g., commonsense reasoning, both physical commonsense and social commonsense, which are developed at a very young age, turned out to be a notorious problem for AI agents (e.g, many works from Yejin Choi’s group have demonstrated that). We will provide more discussion to clarify this statement. \n\nAs ATOMS was derived from early children research, human studies are usually in simple forms of visual and verbal prompts, which can be adapted to LLMs. In more advanced ToM studies, complex social interactions and decision making are involved, which cannot be transferred into LLM evaluation without additional computational modeling.\n\n\n> **Q3**: Do you also input the graphs in the 10 tasks as illustrated in Figure 4 to GPT-4/ChatGPT in your experiment setup? If not (according to the appendix in my understanding), how are the images used?\nWe did not use the image of the grid world, but a symbolic representation of the grid world in the form of text. This is a common practice to prompt LLMs for similar 2D grid world from prior work, e.g., [1].\n\n[1] Pre-Trained Language Models for Interactive Decision-Making. In NeurIPS 2022. https://arxiv.org/abs/2202.01771\n\n> **Q4**: Why do you choose to use a zero-shot setup for evaluation instead of few-shot prompting?\n\n The zero-shot setup is the best to consider as a first step [2] for evaluating LLM’s ability. As this is a position paper rather than a benchmark paper, we only include zero shot as a proof of concept. We believe that more work is needed in the future to design more comprehensive sets of ToM benchmarks and test more methods to apply LLMs.\n\nStill, we present the one-shot results upon the reviewer’s request. We found that many ToM tasks benefit from one shot of demonstration, while some suffer from this setting.\n\n| **Case** | **Reality Check** | **Short Intent** | **Long Intent** | **Desire** | **Percept** | **1st belief** | **2nd belief** | **NL Com.** | **Knowledge** | **Emotion** |\n| ----------- | ----------- | ------------ | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n| **GPT3.5 zero shot**  | 57% | 78% | 33% | 50% | 49% | 47% | 24% | 66% | 62% | 75% |\n| **GPT3.5 one shot**  | 51% | 39% | 45% | 42% | 53% | 49% | N/A* | 53% | 47% | 67% |\n| **GPT4 zero shot**  | 83% | 50% | 91% | 4% | 62% | 65% | 45% | 81% | 59% | 72% |\n| **GPT4 one shot**  | 95% | 66% | 96% | 40% | 61% | 63% | 45% | 77% | 53% | 60% |\n\n*N/A as the prompt exceeded the maximal context window.\n \n[2] Best practices for prompt engineering with OpenAI API. By OpenAI, https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. As many researchers are trying to find connections between powerful LLMs and theory of mind, this paper provides a structured review of existing task formulation and evaluation, and pointed out the current limitations from a clear taxonomy perspective. \n2. More importantly, the paper advocates a situated evaluation of ToM by first proposing a prototype environment. This can be interesting to researchers studying theory of mind in the community.", "reasons_to_reject": "1. Although the community may benefit from the study and advocated holistic landscape by resolving some of the current limitations, it is not convincing from the paper why each part of the taxonomy derived from 0-5 years old children, would be necessary for evaluating theory of mind in a large language model. More importantly, it is not clear why the proposed 10 tasks would be better at evaluating theory of mind compared to all other baselines (e.g., emotion), especially when there is no detailed results and analysis in either the main paper or the appendix.\n2.  Theory of mind evaluation may be impacted by RLHF in large language models, but the situated ToM evaluation (at least from the 10 tasks introduced in the paper), are not related to how the LLMs would respond to different aspects defined.", "questions_to_authors": "1. For data contamination, why would it be necessary to have access to the datasets used to train LLMs before evaluating the performance of LLMs on ToM tasks? There are tons of benchmarks used for LLM evaluation, why would ToM tasks be different from others?\n2. In line 257-261, why would ATOMS be developed from young children, but \"cannot be easily deployed on state-of-the-art AI agents\"?\n3. Do you also input the graphs in the 10 tasks as illustrated in Figure 4 to GPT-4/ChatGPT in your experiment setup? If not (according to the appendix in my understanding), how are the images used?\n4. Why do you choose to use a zero-shot setup for evaluation instead of few-shot prompting?\n", "paper_topic_and_main_contributions": "This paper surveys and studies theory of mind in large language models, by first introducing previous evidence in finding and not finding theory of minds in LLMs, and then point out the limitations in previous evaluation. The authors then follow the Abilities in Theory of Minds Space (ATOMS) to validate the current ToM benchmarks, and propose a situated evaluation of ToM in the grid world scenario consisting of 10 different tasks. The tasks include beliefs, intentions, desires, emotions, knowledge, perception, and non-literal communications defined in ATOMS, and showed that similar to previous findings, certain but limited level of world understanding presents in LLMs.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Park et al., 2023. Generative Agents: Interactive Simulacra of Human Behavior"}, "extra_text": {}}, {"review_id": "JEGNxcQzbV", "reviewer": "Reviewer_HssE", "summary": "", "questions": "A: It is unclear to me how the conclusion “LLMs have reached a level of maturity where researchers can begin utilizing them as powerful building blocks for situated agents in complex ToM tasks.” (L524-527) is derived. \nEvaluating ToM in situated physical environments poses a challenge related to the model's ability to perceive the physical environment accurately. If an LLM performs poorly at perceiving the physical environment, it becomes difficult to evaluate its ToM capability. In the preliminary study, the paper proposes the task “Reality Check” for testing LLM’s understanding of the world. This task involves predicting the closest object around the end of the trajectory. However, it is hard to conclude that good performance on this task reflects the model's ability to understand the world. Therefore, I find it hard to derive the stated conclusion about the maturity of LLMs for situated agents in complex ToM tasks.\n\nB: Furthermore, Figure 5 shows that only GPT4 outperforms the random baseline for the \"Reality Check\" task, while ChatGPT also achieves random performance on the \"Reality Check\" task. Therefore it is difficult for me to see why “LLMs have reached a level of maturity where researchers can begin utilizing them as powerful building blocks for situated agents in complex ToM tasks.” Also, to provide a more comprehensive analysis, the paper could investigate the performance of other open-sourced LLMs.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing many aspects of our contributions and consider our work as “comprehensive”. We appreciate the in-depth review, and would like to make the following responses to clarify some misunderstandings. We are looking forward to the reviewer’s further comments during the discussion period.\n\n### Responses to Weaknesses\n\n> **W1**: The paper lacks an in-depth discussion on how a situated evaluation could mitigate shortcuts and spurious correlations. The section of L438-448 (Situated evaluation mitigates shortcuts) appears to focus more on how situated evaluation can help with data contamination (L193) through the configuration of a vast number of diverse environments and complex tasks. However, it does not provide a detailed analysis of how a situated evaluation can help mitigate the issue of shortcuts and spurious correlations.\n\nWe thank the reviewer for raising this concern about the clarity of our discussion! \n\nWe agree that L438-448 is more about data contamination than shortcuts. A situated ToM evaluation can mitigate data contamination as the grid world (and possibly more complex environment) are not likely to be included in the training data of LLM.  A shortcut introduced by training/testing splits in previous works on benchmarks does not apply here, as we do not pre-train any LLMs.\n\nThe shortcuts are taken at the evaluation time. Many of the existing ToM benchmarks adopt text templates to verbalize a (few) predefined scenario(s) and prompt LLMs for response. Researchers have shown that LLMs can get the correct answer by utilizing syntactic structures or positional information, especially in few-shot settings. In our situated ToM setting, we can use simulated environment to manipulate evaluation data, so that the environment, the states, the trajectories in the environment can be randomized at scale, and we can avoid the statistical spurious correlations. \n\nWe emphasize that while our proposed situated ToM evaluation can mitigate shortcuts, it does not eliminate the issue completely. First, we echo Aru et al. (2023) that shortcuts can still emerge in grid world setups if the design is not careful enough and randomness is limited (line 440). We note that careful consideration is required to design any benchmark, which is not specific to ToM. Also, we mentioned that shortcuts can be introduced in the few-shot examples and human prompts introduced at inference time (line 563 and below). \n\nWe will add more detailed discussion in the revised version. \n\n> **W2**: The paper presents an interesting preliminary study as a proof-of-concept study on a situated evaluation of ToM in LLMs. Since, to me, it is questionable if the text-based grid world setting truly resembles the physically situated and socially situated environment, and therefore it is unclear to me what conclusion or insights the paper seeks to draw from this study.\n\nWe thank the reviewer for raising this concern about the motivation for using text-based grid world for proof of concept! There are two reasons we specifically chose text-based grid environment for our investigations. \n\nFirst, text-based grid world provides a good abstraction of the physical world in a simplified environment to study many aspects of embodied agents. It has been used by several recent works, e.g., for grounding [1], planning and decision making [2], interaction and language acquisition [3], etc.\n\nSecond, situated tasks involve perception and reasoning about the surroundings (e.g., through computer vision). In the grid world, the low-level world state (e.g., the grid setup and where the agent is in the grid) can be captured by symbols. This setup allows us to focus on ToM reasoning without being distracted by machine perception. Through basic grid world representation, the agent is tasked to reason many aspects of **physical situatedness**, e.g., spatial relations between agents, their states, their action trajectories, and from there, the goals, intent, emotions, etc. This is in stark contrast from the existing story-based ToM where the semantics of the world is given by natural language stories without requiring the agent to reason about the world (e.g., based on the change of the world and traces of action trajectories).  \n\n[1] A Benchmark for Systematic Generalization in Grounded Language Understanding. In NeurIPS 2020. https://arxiv.org/abs/2003.05161\n\n[2] Pre-Trained Language Models for Interactive Decision-Making. In NeurIPS 2022. https://arxiv.org/abs/2202.01771\n\n[3] BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning. In ICLR 2019. https://arxiv.org/abs/1810.08272\n\n### Responses to Questions\n\n> **Q1**: It is unclear to me how the conclusion “LLMs have reached a level of maturity where researchers can begin utilizing them as powerful building blocks for situated agents in complex ToM tasks.” (L524-527) is derived. Evaluating ToM in situated physical environments poses a challenge related to the model's ability to perceive the physical environment accurately. If an LLM performs poorly at perceiving the physical environment, it becomes difficult to evaluate its ToM capability. In the preliminary study, the paper proposes the task “Reality Check” for testing LLM’s understanding of the world. This task involves predicting the closest object around the end of the trajectory. However, it is hard to conclude that good performance on this task reflects the model's ability to understand the world. Therefore, I find it hard to derive the stated conclusion about the maturity of LLMs for situated agents in complex ToM tasks.\n\nSorry for the confusion caused by this statement. We will rephrase it in the next iteration. We mainly want to say that LLMs have demonstrated incredible performance in a range of NLP tasks. Given an increasing amount of work in applying LLMs in ToM, it’s time to systematically study LLMs for situated ToM. \n\n- **We are not trying to say they already reach maturity for ToM**, but rather they are at the stage where we can put situated ToM evaluation on the table. We totally agree with the reviewer regarding perception which plays an integral role in reasoning, which is also one of the motivations for calling for “situated ToM”.  The frame of “situatedness” is really trying to bring perception into the picture of ToM. Also, as the reviewer pointed out, poor perception will make it difficult to evcaluate ToM. This is exactly one of the reasons we chose the grid-world setup as a proof of concept here. Future work will need to take more complex forms of perception into account.\n\n- **We are not trying to say they already reach maturity for world understanding**. Reality check here is a simplified setting which serves as a reference pointing with respect to other tasks.  Solving this reality check by no means implies the agent’s ability in general perception of the real world. With our current experimental setup, we show that LLMs like GPT-4 can to some extent solve some simplified perception tasks (Reality Check) but can also fail some ToM tasks in similar protocol. We will add the clarifications in the revised version. \n\n> **Q2.1**: Furthermore, Figure 5 shows that only GPT4 outperforms the random baseline for the \"Reality Check\" task, while ChatGPT also achieves random performance on the \"Reality Check\" task. Therefore it is difficult for me to see why “LLMs have reached a level of maturity where researchers can begin utilizing them as powerful building blocks for situated agents in complex ToM tasks.”\n\nWe apologize for the confusion on the performance of ChatGPT in reality checking. Here we provide additional results that ChatGPT can obtain 78% performance on reality check using zero-shot Chain-of-Thoughts (CoT) reasoning, compared to 57% under zero-shot multiple choice (MC) probing. While CoT significantly improves ChatGPT’s ability in reality checking, many ToM tasks don’t benefit (and sometimes suffer) from CoT reasoning due to false rationale.\n\n| **Case** | **Reality Check** | **Short Intent** | **Long Intent** | **Desire** | **Percept** | **1st belief** | **2nd belief** | **NL Com.** | **Knowledge** | **Emotion** |\n| ----------- | ----------- | ------------ | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n| GPT3.5 zero shot **MC** | 57% | 78% | 33% | 50% | 49% | 47% | 24% | 66% | 62% | 75% |\n| GPT3.5 zero shot **CoT** | 78% | 60% | 48% | 49% | 51% | 24% | 16% | 65% | 52% | 62% |\n\n> **Q2.2**: Also, to provide a more comprehensive analysis, the paper could investigate the performance of other open-sourced LLMs.\n\nAs this is a position paper rather than a benchmark/analysis paper, the goal of our preliminary experiment is to present a proof of concept and showcase possibilities to design situated ToM benchmarks using 2D grid world and evaluate LLMs on it. We believe that more work is indeed needed in the future to design more comprehensive ToM benchmarks and test more variants of LLMs.\n\nStill, we provide some more results on public LLMs upon the reviewer’s request. We conducted further study to compare the results based on LlaMA-7B (no human alignment) and Vicuna-7B (with human alignment) on 30 randomly selected test for each task (due to the time constrain of the rebuttal period) as shown below. We use LM-probing to select the choice with the highest probability following Brown et al. 2020 and Shapira et al. 2023a. Surprisingly, we found that these models make similar predictions on the same task, leading to similar performance. We see that human alignment may have a limited impact on situated ToM capabilities for relatively smaller LLMs (i.e., 7B models). \n\n| **LLM** | **Reality Check** | **Short Intent** | **Long Intent** | **Desire** | **Percept** | **1st belief** | **2nd belief** | **NL Com.** | **Knowledge** | **Emotion** |\n| ----------- | ----------- | ------------ | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n|  LLaMA  | 46.67% | 16.67% | 26.67% | 50.00% | 43.33% | 50.00% | 46.47% | 33.33% | 40.00% | 44.44% |\n| Vicuna | 46.67% | 16.67% | 26.67% | 50.00% | 43.33% | 50.00% | 50.00% | 33.33% | 40.00% | 44.44% |\n\n### Presentation Improvements\n\nWe thank the reviewer for the helpful suggestions regarding presentation improvements. We will add detailed descriptions using the additional page and redo Figure 3 to make it clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper performs a comprehensive review of existing works on machine ToM in LLMs, presenting both evidence and counter-evidence for the emergence of ToM.\n- The paper proposes to taxonomize ToM into different mental states and evaluate it based on these mental states in order to ensure clarity and consistency in evaluating ToM across different literature.\n- The paper performs a taxonomized review of existing ToM benchmarks and identifies under-explored aspects of existing ToM benchmarks. \n- The paper motivates the need for a situated evaluation of ToM, one that is physically situated in environments and socially situated in interactions with humans.\n- The paper presents a preliminary study that showcases a practical demonstration of constructing evaluation tasks related to 7 different aspects of ToM under the setting of a text-based grid world. ", "reasons_to_reject": "- The paper lacks an in-depth discussion on how a situated evaluation could mitigate shortcuts and spurious correlations. The section of L438-448 (Situated evaluation mitigates shortcuts) appears to focus more on how situated evaluation can help with data contamination (L193) through the configuration of a vast number of diverse environments and complex tasks. However, it does not provide a detailed analysis of how a situated evaluation can help mitigate the issue of shortcuts and spurious correlations. \n- The paper presents an interesting preliminary study as a proof-of-concept study on a situated evaluation of ToM in LLMs. Since, to me, it is questionable if the text-based grid world setting truly resembles the physically situated and socially situated environment, and therefore it is unclear to me what conclusion or insights the paper seeks to draw from this study.", "questions_to_authors": "A: It is unclear to me how the conclusion “LLMs have reached a level of maturity where researchers can begin utilizing them as powerful building blocks for situated agents in complex ToM tasks.” (L524-527) is derived. \nEvaluating ToM in situated physical environments poses a challenge related to the model's ability to perceive the physical environment accurately. If an LLM performs poorly at perceiving the physical environment, it becomes difficult to evaluate its ToM capability. In the preliminary study, the paper proposes the task “Reality Check” for testing LLM’s understanding of the world. This task involves predicting the closest object around the end of the trajectory. However, it is hard to conclude that good performance on this task reflects the model's ability to understand the world. Therefore, I find it hard to derive the stated conclusion about the maturity of LLMs for situated agents in complex ToM tasks.\n\nB: Furthermore, Figure 5 shows that only GPT4 outperforms the random baseline for the \"Reality Check\" task, while ChatGPT also achieves random performance on the \"Reality Check\" task. Therefore it is difficult for me to see why “LLMs have reached a level of maturity where researchers can begin utilizing them as powerful building blocks for situated agents in complex ToM tasks.” Also, to provide a more comprehensive analysis, the paper could investigate the performance of other open-sourced LLMs.", "paper_topic_and_main_contributions": "With the advent of large language models, there has been an ongoing debate on whether Machine Theory of Mind (ToM) has emerged in LLMs. This position paper calls for a holistic and situated evaluation of ToM in LLMs.\n\nThe paper provides a survey and summary of existing works, presenting both evidence and counter-evidence for the emergence of a machine ToM in LLMs. The paper proposes to taxonomize ToM into different mental states and evaluate it based on these states. Specifically, the paper follows ATOMS to taxonomize machine ToM into 7 mental states and performs a holistic review of existing ToM benchmarks to identify under-explored aspects of ToM. Moreover, the paper calls for a situated evaluation of ToM, one that is physically situated in environments and socially situated in interactions with humans. As part of its contribution, the paper also presents a preliminary study in the text grid world as a proof of concept. \n\nIn conclusion, this paper offers a comprehensive analysis of existing literature on ToM in LLMs and emphasizes the need for a situated evaluation benchmark and a taxonomy of ToM into different mental states for evaluation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n- L380: “Figure 5” -> “Figure 3”\n\nPresentation Improvements:\n- Unclear Corresponding Characteristics: In L381 - 384, the paper mentions that “Each layer within the figure corresponds to a specific characteristic of the benchmark”. However, in the same paragraph, the paper does not provide a clear description of what the corresponding characteristics are. The reader is left to infer from L350-351, which suggests the characteristics could be related to “task formulation, input modalities, physical and social situatedness, and symmetricity”. To improve clarity, the paper should explicitly describe the corresponding characteristics for each layer in Figure 3. Also, in the case where the inner layer's characteristic is \"input modalities,\" it is unclear why \"env-only\" is considered an orthogonal setting to \"text-only\" and \"multimodal.\"\n-  The specific meaning of the colors used in Figure 3 is not clear from the paper. The paper should improve Figure 3 and provide clear descriptions."}}, {"review_id": "vzKoR4rSfz", "reviewer": "Reviewer_92YE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our paper “solid and comprehensive” and “useful for NLP community”, and we thank the reviewer for the positive feedback that:\n> This paper provides a deep understanding in the landscape of ToM and effective observations for future research. \n\nWe will make our data and prompts available for the community to ensure the reproducibility of our pilot study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The content of this paper is solid and comprehensive, which is useful for NLP community. I suggest this paper should be presented at the conference.", "reasons_to_reject": "I didn't see any obvious weakness.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Large Language Models (LLMs) have generated considerable interest and debate regarding the potential emergence of a machine Theory of Mind (ToM). This paper seek to answer two road-blocking questions: (1) How can we taxonomize a holistic landscape of machine ToM? (2) What is a more effective evaluation protocol for machine ToM to avoid spurious correlations? They follow the ATOMS framework to offer a holistic review of existing benchmarks and identify under-explored aspects of ToM. They further call for a situated evaluation of ToM, one that is physically situated in environments and socially situated in interactions with humans. They conduct a preliminary study in the grid world scenario is presented over 10 different task setups as a proof of concept.This paper provides a deep understanding in the landscape of ToM and effective observations for future research.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, this paper addresses the growing discussion of ToM in the context of LMs, working towards a more robust evaluation protocol. 
The reviewers overall slantly positive on the paper, but don't necessarily arrive at the strongest consensus on its strengths and weaknesses.

Collectively, as AC I see the following points as the clearest strengths and weaknesses:
S: Rigor/structured approach to ToM, concrete and situated evaluations
W: Unclear whether the approach to eval (derived from children + 10 tasks initiated based on taxonomy) make sense, concerns about spurious correlations are not fully addressed

In a brief reading of the paper, I agree that the paper makes important progress on a timely topic with substantive soundness. I will say I do share the concern that matter of spurious correlations, which seems to be at the heart of why evaluations for ToM in particular will be challenging to design + trust, has not been adequately resolved. If accepted, I encourage the authors to spend the majority of the additional page strengthening this topic. Separately, I think the community would be collectively more excited/inspired by the work if the results offered more striking analysis: I am left a bit unwhelemd by the results/discussion of results. I understand a well-designed evaluation is itself a great service to the community, but I think making the results (and/or their analysis) richer can really help draw interest to the evaluation as well, which is likely of interest for driving future work in this space.
Decision: Accept-Findings