Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
Authors: Yufan Wu
Yinghui He
Yilin Jia
Rada Mihalcea
Yulong Chen
Naihao Deng
Venue: EMNLP 2023 Findings (2023)
Keywords: Higher Order Theory of Mind
Chain of Thought Prompting
Large Language Models
Deception
TL;DR: We introduce a benchmark called Hi-ToM for evaluating higher-order ToM reasoning in LLMs, and do experimental evaluation on GPT-4, GPT-3.5, Claude and Guanaco.

Abstract: Theory of Mind (ToM) is the ability to reason about one's own and others' mental states. ToM plays a critical role in the development of intelligence, language understanding, and cognitive processes. While previous work has primarily focused on first and second-order ToM, we explore higher-order ToM, which involves recursive reasoning on others' beliefs. %We also incorporate a new deception mechanism in ToM reasoning. We introduce Hi-ToM, a Higher Order Theory of Mind benchmark. Our experimental evaluation using various Large Language Models (LLMs) indicates a decline in performance on higher-order ToM tasks, demonstrating the limitations of current LLMs. We conduct a thorough analysis of different failure cases of LLMs, and share our thoughts on the implications of our findings on the future of NLP.

HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
Yinghui He
Yufan Wu
Yilin Jia
Rada Mihalcea
Yulong Chen Naihao Deng'
University of Michigan Westlake University Chuihui, umwyf, kripf, mihalcea, dnaihao)@uuicch. edu
Abstract
Coueta fRISS
Theory of Mind (ToM) is the ability rea- son about one's own and others' mental states. ToM plays critical role in the development of intelligence, language understanding, and cog- nitive processes. While previous work has pri- marily focused first and second-order ToM, we explore higher-order ToM, which involves recursive reasoning on others' beliefs. in- troduce HI-ToM, Higher Order Theory of Mind benchmark. Our experimental evaluation using various Large Language Models (LLMs) indicates decline in performance higher- order ToM tasks, demonstrating the limitations of current L.MSs. We conduct thorough analy- sis of different failure cases of LLMs, and share our thoughts on the implications of our findings the future of NLP.
THEY DON'T KNOW THAT WE KNOW THEY KNOW WE KNOW.
Figure A scene shot from the TV series Friends that exhibits fourth-order Theory of Mind (ToM).
work has been confined to first-order and second- order ToM, where LLMs are asked to perform in- ference on others' belief of reality in one two passes, e.g., the first and second-order questions in Figure 2 (see Section for more comprehensive discussion of ToM background and the evaluation of ToM LLMs).
1 Introduction
Theory of Mind (ToM) refers to the ability to un- derstand and reason about the mental states of others such as intentions and beliefs, and also to distinguish them from onc's own (Premack and Woodruff, 1978). Such an ability has been con- sidered crucial point the development of intel- ligence functions (Premack and Woodruff, 1978; Bretherton and Beeghly, 1982; Frith and Frith, 2003), and previous research has demonstrated that ToM reasoning is highly related to linguistic and cognitive processes (Perner, 1991; Sperber and Wil- son, 2002). ToM has thus been widely used pro- tocol to evaluate the language understanding and reasoning ability of intelligence agents (Premack and Woodruff, 1978; Takano al., 2006), such as young children (Osterhaus and Koerber, 2021).
Higher-order ToM, referring to third-order rea- soning and beyond, requires recursive reasoning others' beliefs in multiple passes. Figure 1 shows higher-order ToM example from the TV series Friends. In Figure 1, a character says "They don't 't know that we know they know we know" when she and the other character try to recursively identify the situation. Such an example underscores that human beings are capable of higher-order ToM in daily interactions. In addition, evidence shows that higher-order ToM is not only essential to commu- nicate effectively in complicated scenarios, such multi parti conversations (Liddle and Nettle, 2006; De Weerd et al., 2015: Ridinger and McBride, 2017: De Weerd et al., 2022), but also enables better emotional support and empathetic commu- nication (Mitchell and Phillips, 2015). However, because of lack of higher-order ToM datasets in the NLP community, there is significantly less search on higher-order ToM compared to the lower orders.
With the recent advance large language mod- els (LLMss, research has been undertaken to eval- uate the language skills of LLMs using ToM (Sap et al., 2022; Ullman, 2023). Most of the previous
"Contributed equally to this work.
"Coreesponding author of this work.
Previous work has mainly constructed ToM

---

The milk on the table
Sally exited the roam
ing higher-order ToM reasoning and analyzing the abilities of current LLMs on high-order ToM. Fur- thermore, we share our thoughts on the future of NLP and the way forward with LLMs enhancing LLMs from the perspective of human intelligence, understanding humans through the lens of LLMs, and enhancing LLMs' ToM abilities for better NLP applications. We release our dataset and code at https://88 tit..u....... homm/ying-hui-he/Htee dataset.
Saily
Alec
Annw
Boxs
Anne transferred the milk onto the box
Alex exited the room, then Anne exited the room
Alex
Anne
Outside the room, the three interacted with each other
(Alex lied to all) the fridge!
(Sally secretly told Anne)
Ales
Salv
Background and Related Work
the table!
Theory of Mind. Most of prior work focuses on first sccond-order ToM (Nematzadch et al., 2018; Le et al., 2019; Sap et al., 2022), while higher- order ToM (third-order and beyond) remains under- explored. The concept of "orders" refers to the number of mental state attributions that are required to answer a particular question or reason about a particular scenario. For instance, third-order ToM question can be "Where does Anne think that Sally thinks that Isabella searches for the milk?", where Sally's ' reasoning about Isabella is of second-order, and Anna's reasoning Sally's reasoning is of third-ordes
0th
Where is the milk?
1st
Where does Anne think the milk is?
2nd
Where does Sally think Anne thinks the milk is?
3rd
Where does Alex think Sally thinks Anne thinks the milk is?
Figure 2: A sample from Hi-ToM dataset, which con- tains communications among agents, and questions that address (-th (reality) to 3-rd ToM reasoning..
benchmarks using automatic story generation scripts. Although simple and inexpensive, this method cannot be directly extended to generating stories of higher-order ToMs because the generated stories contain insufficient information for raising a higher-order question. In this paper, we build upon previous work and introduce HI-ToM, multiple- choice question benchmark consisting of Sally- Anne-like stories (Figure 2), specifically designed for higher-order ToM evaluation. Unlike previous datasets, HI- ToM contains questions from zeroth- order to fourth-order ToM, and incorporates agent communications in the stories. We manually check the quality of the constructed data, and empirically find that HI-TOM ToM presents greater diversity and challenges compared to previous datasets.
Higher-order ToM is useful in social interac- tion such as maintaining social networks (Lid- dle and Nettle, 2006), winning limited bidding (de Weerd and Verheij. 2011), efficiently coopera- tion (De Weerd et al., 2015; Ridinger and McBride, 2017), and unpredictable negotiations (De Weerd et al., 2022). Researchers from cognitive science investigate second-order and higher-order ToM among young children via complex forms of false- belief tests, such as the Sally-Anne belse- experiment (Baron-Cohen et al., 1985).
Evaluating ToM in LLMs. Sap et al. (2022) find that GPT-3's ToM ability is well below humans the ToM1 dataset (Le et al., 2019), which a ToM evaluation dataset consisting of questions up to the second order. Kosinski (2023); Bubeck et al. (2023) show the promising performance of recent LLMs such as GPT-3.5 and GPT-4 on ToM tasks. How- ever, it is questionable whether LLMs have genuine ToM ability, especially for higher-order ToM. UlI- man (2023) find that for GPT-3.5, small variations that maintain the principles of ToM can cause a flip of the answer. Different from previous work that only evaluates LLMs' ToM ability up to the second order, we take a step forward and evaluate LLMs' ability in higher-order ToM settings. Also,
We experiment with various LLMs, including GPT-4 (OpenAI, 2023), GPT-3.5-turbo (OpenAI, 2022), Claude, and Guanaco (Dettmers et al., 2023), on HI-TOM under a zero-shot setting. Fur- thermore, we test the chain-of-thouge prompt- ing (Wei et al., 2022) and conduct a thorough analysis of LLMs' performances on different story types in HI-ToM and their failure cases. Our work demonstrates that the claim of LLMs having gen- uine ToM abilities (Kosinski, 2023; Bubeck al., 2023) is questionable, especially in the cases of higher-order ToM, where several rounds of recur- sive reasoning are required. To our knowledge, we are the first to introduce a benchmark for evaluat-

---

Component
Num.
Example
HI-ToM One-Chapter Story
Room
30
kitchen 4 bedroom
Emma, Charlotte, Benjamin, Aiden and Isabella
Object
37
lemon peach
tered the workshop.
Container
red enveeelleee blue_bottee bottle
2 The pear is in the red treasure chest.
Agent
Jack 2. Ella Noah
3 Emma moved the pear the blue_ suitcase.
Emma exited workshop.
5 Charlottc exited the workshop.
Table I: Basic components, numbers of choices for each component (Num.), and their examples in Hi-ToM ries.
Benjamin lost his watch.
7 Benjamin exited the workshop.
8 Aiden moved pear to the blue_crate.
9
Aiden exited the workshop. Isabella moved the pear to red treasuree chest.
10
we are the first one pioneering in adding the decep- tive communication protocol in ToM setups, which takes an initial step toward evaluating LLMs' abil- ity in real-world scenarios. Concurrent our work, Ma al. (2023) surveyed the existing ToM bench- marks and conducted preliminary experiments on situated evaluation of ToM for LLMs.
Isabella likes the red hox.
12
Isabella exited the workshop.
13
Aiden publicly claimed that the pear is in the
diueer now.
14
Emma privately told Isabella that the radish is in the red udd_suicese now.
Table 2: An example HI-ToM one-chapter story with agent communications. Random distractors are inserted in lines 6 and I, where the latter introduces "red_ box" as distractive answer choice.
3 The HI-TOM Dataset
To systematically examine how effectively LLMs reason Theory of Mind (ToM) at different orders, each story is coupled with five questions that re- quire the zeroth to fourth level of ToM reasoning, respectively. Following Nematzadeh et al. (2018) and Le et al. (2019), we automatically generate HI-
moves the object (or not) and leaves the room (Scenes 2, 3, and 4).
Agent communication: Outside the room, agents may be involved in two types of com- munications: public, where an agent shares infor- mation with every agent, and private, where an agent only speaks to another agent privately, or they can remain silent (Scene 5).
ToM stories. Additionally, we manually review the generated stories, questions, and answers to ensure that they consistent with each other, and they are logically correct.
For agent communications, information to be deceptive in order emulate the dynamics of the complicated social life. also adds another layer of complexity the ToM reasoning process, which requires the answerers to not only reason about an agent's perceptions of other agents knowledge of the objects location, but also reason about whether an agent would trust another agent. In this way, we evolved the simplistic toy stories in the previous dataset (Le et al., 2019), and solved core problem in the previous evaluations that the answers may be simply found from the objects original or final location.
set the shared
3.1 Dataset Design
Story Design. HI-TOM stories consist of four fundamental elements: rooms, objects, containers, and agents, as shown in Table A story narrates events occurring in one or more rooms, where mul- tiple objects are placed inside their respective con- tainers. Each story features five rational agents. Each story comprises one to three chapters. Each chapter corresponds
single round in the object- finding game. In each chapter, we design multi- ple actions and optional communication protocols among agents:
Moreover, we pose a constraint that the listener would update their world knowledge based on the information given by the speaker if the speaker ex- its the room later than the listener. This is based on the assumption that the listener would be unawarc of any changes after their exit, but the speaker might possess more up-to-date knowledge as they leave later. Additionally, we assume that Alex and Sally, who give out information publicly or pri- vately, will believe that all the listeners trust their
At least agent enters room, where they observe all the objects, other agents, and their actions in that room (e.g., Figure Scene 1).
Entry:
Object Movement: When in room, each agent can choose whether move an object before their exit. Such actions are done in a sequential manner. In other words, the later agent can only perform such an action after the former agent

---

respective information. full assumption list is attached each story, as shown in Table in Ap- pendix B.1.
Datasets
ToMiToM-cnn -asy
TOMI
Hi-TOM
1st
2nd
3rd
Each chapter involves entry and object move- ment, while agent communication is optional. In Hi-ToM, half of the stories have at least one chap- ter with agent communications, while the other half only contains chapters without communications.
Comm.
iine
15.05
8.86
26.47
#Agent 3.22
2.75
5
#Container
7.39
Table 4: Comparison between HI-TOM and other datasets. 1st. 2nd, 3rd, and 4th refer whether dataset contains story-question pairs of specific ToM order. Comm. stands for the existence of agent communica- tions. #Line, #Agent, and #Container represent average number per story.
Question-Answer Design. Following Le et al. (2019), for each story, we provide five questions that are progressively built from lower-order ques- tions to higher ones shown in Table 3.
Order
Question
Oth
Where is O really?
of all agents and derive the correct answer to each question. Further details and pseudocode related to story generation process can be found in Appendix B.2.
Ist
Where does AI think 0 is?
2nd
Where docs A2 think A/ thinks 0 is?
3rd
Where does A3 think A2 thinks A/ thinks O is?
Where does A4 think A3 thinks A2 thinks Al
4th
thinks 0 is?
Additionally, based on Le et al. (2019), we fur- ther incorporate distractor sentences that relate an agent with a random container, such as "Jack likes the red container". This reduces the regularity and predictability of the stories. Table shows an exam- ple one-chapter story with agent communications and random distractors.
Table 3: Questions asked in story involving object and five agents. A/ to A4 are randomly chosen from the five agents. Othh, 1st, 2nd, 3rd, and 4th represent ToM orders of the questions.
Following Sap et al. (2022), we adopt the multiple-choice setting and provide the correct an- swer along with several distractor choices.
3.3 Dataset Characteristics
3.2 Data Generation
Table 4 shows a comparison between HI-TOM and the other ToM datasets. First, unlike previous datasets, HI-TOM is the only benchmark that con- tains third and fourth-order stories, which suggests that Hi-ToM is more challenging and requires higher-order ToM reasoning. Also, we first intro- duce communications among agents, which poses greater challenges to LLMs to reason about human interactions. addition, stories in Hi-ToM are significantly longer, with a larger number of agents and containers per story. This requires the LLMs' capability to comprehend the complete storyline and reason about each agent's 's beliefs.
To generate the aforementioned ToM stories with higher-order questions among agents, adapt the generation scripts from Nematzadeh et al. 22018), which are originally limited to first or second-order ToM stories.
Our script takes a list of story components Rooms, Objects, Conlainers, Agents, as well the number of chapters É as inputs, and outputs a story with i chapters along with five questions from zeroth to fourth order ToM.
For the generation each chapter, we randomly choose the story components and fit them into the chapter template. Specifically, we randomly deter- mine whether or not each agent moves the object to another container. Then, we incorporate agent communications in certain chapters, where we use the phrases "publicly claim" and "privately tell" to encode public and private communications.
Notably, Hi-TOM features a larger pool of poten- tial answers and a balanced distribution of correct answers throughout the story. In ToM/ToM-enu easy and ToMr, all the correct answers appear within the last two containers the only two containers in the corresponding story. In contrast, the pro- portions of correct answers appearing in the first, second, third, and final quarters of the Ht-ToM sto- ries are 28.7%. 27.2% 18.8%, and 25.3%, respec- tively. The even distribution of correct answers
To generate the questions and answers for each story, we integrate the relevant story components into a predefined question template. Subsequently, we utilize an answer generator to track the actions

---

eliminates the position bias of correct answers be- ing concentrated in specific segments of the stories in HI-TOM.
Accuracy (%)
Model & Methods
w/o
w/
Overall
dec.
dec.
VP
33,33
33,33
33.33
4 Experimental Setup
Guanaco 65B
35,00
26.99
30.99
32.17
CoTP
2.34
4.1 Models
VP
49.33
42.00
45.67
Claude -instant
46.00
We evaluate the following four LLMs on HI-ToM:
52.33
40.33
46.33
CoTP
+0.66
1. GPT-3.5-Turbo (OpenAI, 2022) and GPT-
VP
28.67
26.33
27.50
GPT-3.5
31.50
4 (OpenAI, 2023) are closed-sourced models from OpenAI. We use gpt-4-32k and gpt-
-turbo
C6TP
35.67
35.33
35.50
+7.00
+8.00)
3. 7-5-uubo for experiments, which are con-
GPT-4 -32k
VP
60.42
55.81
58. 11
58.99
64.04
55.72
59.88
ducted on June 14th~ 2023.
CoTP
+3.60
-0.09
+1.77
Claude-instant is close-sourced model pub- lished by Anthropic
Table Standard accuracy results of the four tested models on HI-ToM stories. "w/o dec." and "w/ dec." indicate accuracy in stories with and without deception, respectively. The performance increase and decrease from VP to CoTP prompting style highlighted.
3. Guanaco (65B) is an open-sourced model fine- tuned from LLaMA (Touvron et al., 2023).
We adhere to the default parameter configura- tions across all the examined language models.
recursive successor of the lower-order ones for the same story.
4.2 Methods
For each HI-TOM story, we conduct trials using two prompting styles: Vanilla Prompting (VP) and Chain-of-Thought Prompting (CoTP). In VP prompting, the model needs to pick the best answer from given set of options without explanation. CoTP prompting requires the model to offer a step- by-step explanation of its thought process along with the answer. Appendix C.1 provides an exam- ple (Table 8) and more details of our prompting methods.
Experimental Results
Table 5 presents the accuracy scores of the four LLMs. All the LLMs we evaluate exhibit less than 60% accuracy scores, demonstrating that HI- ToM is challenging even for the most sophisticated LLMs.. Figure depicts the joint accuracy scores of GPT-4 and GPT-3.5 under various settings. As the story length decreases or the ToM order increases, LLMs' performance decreases across various set- tings. In addition, L.MS perform worse when there are deceptive agent communications involved in the story. The trend observed in Guanaco and Claude aligns with that of GPT-4 and GPT-3.5, as shown in Appendix C.2.
4.3 Evaluation
We evaluate the model performance using both standard accuracy (hereafter referred to as accu- racy) and joint accuracy. Adapted from (Le et al., 2019). joint accuracy represents a more stringent metric than standard accuracy. It considers an answer as correct only when the related question, along with all preceding, lower-order questions within the same story are answered correctly. For instance, the third-order question in Table is con- sidered correct only if the model correctly answers the zeroth, first, second, and third-order questions above it. Joins accuracy effectively reveals the model's genuine ability in higher-order ToM rea- soning, as the model may only reason the higher- order ToM correctly if it is able to reason the lower- order ToM because the higher-order question is
The experimental results also reveal the follow- ing noteworthy patterns:
CoTР prompting yields insignificant perfor- mance gains. We observe no substantial improve- ment in accuracy when transitioning from VP to CoTP in 5. Furthermore, in the assessments involy- ing stories with deception, the switch in prompting methods even leads to decrease in accuracy. We hypothesize that as there are more steps involved, there are higher chances of deceptive information misleading steps in between. The chain may then amplify the error in that step, leading to a cascade of errors throughout the reasoning process.

---

CoTP
VP
8. CoTP
8 VP
0.7
0.45
0.35
1
0.7
0.5
0.65
0.3
0.2
0.15
0.6
0.2
0.1
GPT-4
0.95
0.79
0.5
0.2
0.1
2
0.95
0.75
0.35
0.1
0.05
2-
0.55
0.2
0.1
0.1
2
0.6
0.35
0.15 0.15
0.95
0.55
0.2
0.16
0.15
0.93
0.58
0.2
0.15
0.15
1
0.45
0.3
0.1
0.05
0.95
0.4
0.21
0:05
0.05
Aoeurooe 2007
-0.6
2
0
2
0
1
3
0.4
0.95
0.4
0.15
0.03
0.7
0.5
0.2
0.1
0.1
0.5
0.15
0.3
0.1
0.03
GPT-3.5
0.2
0.9
0.55
0.2
0.05
0.3
0.05
0.7
0.35
0.1
0.05
0.45
0.15
0.15
0.05
0.05
0.0
0.3
Figure 3: Joint accuracy of GPT-4 and GPT-3,5 on HI- Hi-ToM stories w/ or w/o deceptive agent communications. The r-axis stands for ToM orders, and the y-axis is for story lengths (number of chapters). CoTP and VP respectively represent chain-of- of-thought and multiple-choice-e www.://wwweeeaeeeeee prompting styles. The devil sign (*) signifies accuracy results on stories with deception, while other results pertain non-deceptive stories.
Order:
communications challenge the LLMSs in their ToM reasoning process.
0.7
accurat
0.6
Discussion and Analyses
0.3
6.1 Underlying Patterns in Correct LLM Predictions
0.2
Although the overall accuracy of the models leaves room for improvement, we observe a higher fre- quency of correct model choices under specific conditions. We thus examine the scenarios where models have higher answer accuracy.
Deception times per story
Figure 4: Joint accuracy of GPT-4 on Hi-ToM stories with 0 to 4 sentences of deceptive agent communication. Oth-order (reality) accuracy is not included, since the answer to the real room of the objects is not affected by deceptive communications.
LLMs handle answers that appear at the begin- ning and end better. When dealing with long three-chapter stories, LLMs frequently overlook key information, such as the movement of spe- cific container agent conversations. Yet, they tend pay special attention the beginning and the end of the story.
Increased ToM order leads to decreased perfor- mances. As the ToM order increases from the zeroth to the fourth, the joint accuracy goes from near perfect to near zero. We also observe the dras- tic decline in the conventional accuracy scores in Appendix C.2.
In Figure 5, we highlight GPT-4's higher perfor- mance when the correct answer aligns with the first or last container mentioned in the story, com- pared to other cases, as demonstrated by the higher values on the diagonal. This suggests that LLMs are better at handling answers that appear at the beginning or at the end. In contrast, The accuracy when the correct answers are the middle contain- ers (i.e. neither the first nor the last) is similar to those that are not in those containers, shown in Figure 14 in Appendix C.2. We observe similar patterns of the Claude model focusing on the be- ginning and end of stories, shown in Figure 15 in Appendix C.2. Our findings about position bias in LLMs align with other works on LLMs (Wang et al., 2023).
LLMs' performance decreases there more deception communications involved. The per- formance drops when deception communications are involved. Table 5 and Figure reveals a worse performance on stories with agent communications, To further investigate the models' reasoning abili- ties in handling deceptive agent communications, we plot the resulting accuracy versus the number of deceptive communication sentences ("deception times") per story for GPT-4 in Figure 4. As shown in Figure 4, as deception times increase from to 4, the joint accuracy experiences drops of 32%, 18.1%, 16%, and 11% % respectively for the four ToM orders. This suggests that the deceptive agent

---

Correct answer same as the 1st-order answer
Correct answer different from the 1st-arder answer
Last 0.7
0.3
First
0.62
0.38
0.6
70
60
Accuracy
50
-Last-
0.46
0.54
-First-
0.38
0.62
40
30
Correct
Incorrect
Correct
Incorrect
Figure 5: Frequency of GPT-4 correctly or incorrectly answering question of three-chapter story, based first
2nd
3rd
4th
on whether or not the correct answer is the container mentioned in the story. "Last"/"Trstt and ""Last"""-Fisst" indicate whether or not the correct lies the last/first container.
Figure Standard accuracy GPT-4 on 2nd, 3rd. and 4th-order questions, categorized by whether the correct answer matches the corresponding Ist-order answer.
consists of simple patterns rather than complex and nuanced reasoning scenarios, leading its frequent simplification of the questions. In addition, LLMs may possess limited contextual understanding of the story. They may struggle to retain and integrate information from multiple steps or make connec- tions across different parts of the text, leading to oversimplification of the question.
LLMs perform better the answers across or- ders are the same. We observe that LLMs per- form better on question sets where the higher-order answer coincides with lower-order answer. In Figure 6, we see clear performance disparity be- tween LLMs answering correctly if the answers the same across orders versus the answers being dif- ferent across orders. However, this may result from LLMs' tendency to predict the same answers across orders. We find that 72.4%, 64.6%, and 59.8% of GPT-4's second, third, and fourth-order answers match their first-order responses. In contrast, only 30.9%, 20.9%, and 22.2% of the corresponding correct answers in HI-ToM are the same as their first-order answers. This suggests that GPT-4's hanced performance on certain questions may be due the coincidence of correct answers across different ToM orders.
Commonsense errors. LLMs have demonstrated remarkable performance on standard benchmarks of commonsense reasoning (Bian et al., 2023). However, when it comes to ToM reasoning, even advanced models like GPT-4 prone to mak- ing mistakes in handling commonsense knowledge. One key aspect that contributes these errors is the disparity between the models' knowledge com- monsense facts and their ability to effectively apply that knowledge in the complex reasoning process. While LLMs may possess a vast amount of ex- plicit commonsense knowledge, they can struggle to appropriately utilize this knowledge while avoid- ing overgeneralization. In addition, the frequent commonsense errors HI-TOM might be due that HI-ToM is newly constructed, and therefore LLMs have never seen such data before. In con- trast, LLMs' pre-training corpus might contain the data in the publicly available commonsense bench- marks, leading to the high performances on LLMs on those benchmarks (Magar and Schwartz, 2022). Hallucinations.
6.2 Classifying Reasoning Errors
To provide a comprehensive overview of the fail- ure cases of LLMs in ToM reasoning, we manu- ally evaluate total of 300 step-by-step responses across all ToM orders by ourselves, comprising 150 from each of GPT-4 and GPT-3.5. Table 6 describes the five most prevalent error types with corresponding examples. Figure provides the frequencies of these errors GPT-4's responses across different orders. We also show the results for GPT-3.5 and do a comparison between the two LLMs in Appendix C.2. As the ToM order in- creases, LLMs tend to demonstrate higher fre- quency of errors. Here we provide hypotheses and discussions for each of the eтror types:
Hallucination is a well- known phenomenon in LLMs' generation process (McKenna et al., 2023). In our experiments, LLMs may have relied on superficial cues and statistical associations to answer the questions, rather than gaining solid understanding of the underlying context and meaning. Hence, they might resort to fabricating baseless details to bridge the logic gap between the true story and their erroncous re- sponses.
Insufficient reasoning depth. We notice that LLMs tend to skip steps in their reasoning process and end up with an answer lower-order ques- tion, as we observed carlier in Section 6.1. One reason can be that the pre-training corpus often

---

Error Types
Description
Example
Insufficient Reasoning-Doe Deptt
Oversimplify the question
Where does Jack think Hannah thinks William thinks the carrot is? green_box, as that's where Jack last saw
and skip the required multi- step reasoning.
Commonsense Errors
Generate outputs that violate
Aiden exited pantry after step 8, he can witness the move in the pantry after the exit, so
common sense.
Hallucinations
Fabricate ungrounded de-
(In the story, Benjamin saw a cat, but did not talk about it) @:... But there's another twist. Suppose Ella also learns that Benjamin lied about seeing a cat to distract everyone from his real plan...
tails or facts.
Temporal Ignorance
Confuse or ignore the tempo- ral order of events.
Lily exited the hallway (step 8) after Amelia moved the com the red hasket (step 11),
Spurious Causal Inference
Auribute cause-and-effect relationship between unre- lated events.
.... Carter privately told Emma that the tomato the green drawer. Private communications are heard by others, so Emma no reason doubt Carter's information.
Table 6: Types of reasoning errors commonly made by LLMs, with their description and example erroneous responses
(())
questions (D) from our experiment results on GPT-4.
two-way relation between artificial intelligence and human intelligence.
Frequency
0.5
0.3
Enhance LLMs' ToM ability from the perspec- tive of human intelligence. According Kah- neman (201 1), human decisions are supported and guided by the cooperation of two capabilities or two systems: System for intuitive, imprecise, fast, and often unconscious decisions ("thinking fast"), and System 2 for more complex situations with logical and rational thinking ("thinking slow"), This theory has inspired works in computer vision and natural language processing communities explicitly equip models with the two systems (Hill et al., 2021; Miech al., 2021).
0.0
2nd
3rd
insufficient reasoning-depthh Commonsense errors
Temporal Ignorance Spurious causal inference
Hallucinations
Figure 7: Ratio GPT-4 answers containing the five reasoning errors. The r-axis corresponds to ToM orders.
Lack of temporal information. We observe that LLMs' understanding of the sequence of agent ac- tions is often skewed, as the actions are closely listed in Hi-ToM stories. This confusion in tempo- ral order is also found in Yuan et al. (2023). This error may be attributed biases inherent in the pre-training corpus which leads L.MMs' lack of genuine understanding temporal relations.
Through our examination of LLMs' ToM ability, find failure cases that resemble the character- isties of System thinking; for instance, LLMs may invent causes and intentions ("hallucination",, or substitute an easier question for a difficult one ("insufficient reasoning-depthh) Furthermore, the significant performance drop from zeroth to fourth order ToM in Hr-ToM suggests that LLMs may be more inclined to System 1 thinking rather than System 2 thinking, higher-order ToM requires careful in-depth logical inference.
Spurious Causal Inference. The current learning paradigm for LLMs is designed to capture the sta- tistical correlations among the data (Devlin et al., 2019). Through such paradigm, it is difficult for LLMs to capture the underlying logie behind these correlations (Jin al., 2023). As result, LLMs may make incorrect or misleading causal in- ferences based solely on these superficial patterns.
However, there exists line of research com- bining the symbolic reasoning process (Simon and Newell, 1971; Winograd, 1971). which aligns with System thinking, with connectionist paradigm or neural learning (Rumelhart al., 1986; Le- Cun et al., 2015), which captures the intuitive and pattern recognition aspects of System thinking
7 Implications on the Future of NLP
We believe our work has important implications on the future directions of NLP with respect to the

---

(Shavlik, 1994; Hitzler, 2022). This integration holds the promise of enabling AI systems to per- form complex tasks that require both logical de- duction and statistical generalization. We believe our findings of ToM limitations of LLMs alongside this previous line of research clearly points to direction where neural and symbolic approaches are combined in order to achieve abilities that are more closely aligned to human intelligence.
tasks and applications, enhancing the abilities of LLMs opens up exciting possibilities for improving specific NLP tasks that benefit from these models.
8 Conclusion
In this puper, we introduce HI-TOM, the first ToM benchmark that contains higher-order ToM tasks, We demonstrated that LLMs' performance suffers a significant drop in ToM tasks from lower to higher order. By proposing HI- ToM, we hope to address the challenges of ToM in complicated scenarios and spark further research on enhancing the reasoning ability of LLMs.
Understanding humans through the lens of LLMs. ToM plays crucial role in understand- ing human intelligence, as it is an important as- pect of human cognition that enables us to make inferences about others' thoughts, emotions, and behaviors. Enabling progress in ToM reasoning in LLMs entails progress in emulating the function- ing of human mind, which in turn offers intriguing possibilities for gaining insights into human inter- actions and the emergence of intelligence. While it is important to recognize that the analogy between human and artificial intelligence has its limitations and a subject of debate within the NLP com- munity (Bender et al., 2021), recent research has explored the extrinsic understanding of human in- teractions through multi-agent systems (Park et al., 2023). This approach allows us to observe how LLMs can mimic and simulate aspects of human behavior and communication. By studying LLMs and their intrinsic properties, such as the emergence of intelligence, we can gain valuable insights into the fundamental processes underlying human cog- nition (Wijmans et al., 2023). Researchers have also developed methods to elicit human-like behav- ior from LLMs, providing further opportunities to explore and understand the capabilities and limita- tions of these models (Belrose et al., 2023). While LLMs offer close-up view of human-like lan- guage processing, it is crucial approach the topic with caution and recognize the complexities and nuances of human intelligence and behavior.
Furthermore, we present our insights on the fu- ture of NLP and discuss potential directions for enhancing LLMs. Our aim is to stimulate research that draws inspiration from human intelligence, strives to understand humans better, and ultimately leads the development of NLP applications that better cater the needs of humans.
Acknowledgement
We thank the anonymous reviewers for their valu- able feedback and discussion. This paper's draft version was accepted to the non-archival track of the ToM workshop at ICML 2023. We would also like to extend our appreciation to the reviewers from the ToM workshop for their feedback.
Ethical Considerations
The data used in our study were collected from the API of language models. No sensitive or personal information was included.
Our research aims at examining the high-order ToM reasoning ability of LLMss, and we demon- strate the insufficient higher-order ToM abilities of the current LLMs. There is no direct misuse of our findings. However, we recognize that future LLMs with stronger ToM abilities may become more pow- erful generating misinformation and even manip- ulating people if used by some ill ill-intended parties. Therefore, we advocate for the responsible use of LLMs and the associated technologics.
Enhance LLMs' ToM abilities for better NLP applications. In daily life, our ToM ability plays a vital role in understanding others' intentions, therefore helping us in our communication. In HI-ToM, we enable higher-order ToM reasoning, which in turn can lead to improvements in LLMs performance on tasks such as deception detection, emotional support, empathetic communication, and others. Additionally, since LLMs represent foun- dational models that are used across various NLP
We adhere to the principles of transparency and openness. All methods and findings are reported completely and honestly. Furthermore, make our code public upon acceptance. We invite readers to utilize this resource for a more compre- hensive understanding of our methods and results.
will

---

Limitations
be too big? In Proceedings of the 2021 ACM confer- ence on fairness, accountability, and transparency. pages 610-623.
The limitations of our work can be stated from the following perspectives.
Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. 2023. Chatgpt a knowledgeable but inexperienced solver: An investigation of com- monsense problem in large language models, arXiv preprint arXiv:2303.1642.
1. Due to the constraint of computing resources and budget, only test four LLMs. However, we try our best to select the representative LLMs from close-sourced open-sourced LLMs including GPT-3.5 and GPT-4 from OpenAl, Claude from Authropic, and Gua- naco from the community.
Inge Bretherton and Marjorie Beeghly. 1982. Talk- ing about internal states: The acquisition of an ex- plicit theory of mind. Developmental psychology. 18(6):906.
2. Due to the scope of this paper, we only demon- strate the insufficient ToM abilities of LLMs. Future works may further investigate how dif- ferent training paradigms such as training with or without reinforcement learning with human feedback (RLHF) affect the ToM ability of these LLMs.
Séhastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- genee: Early experiments with gpt-4, arXiv preprint arXiv:2303.1273
Harmen De Weerd, Rineke Verbrugge, and Bart Verheij. 2015. Higher-order theory of mind in the tacit com- munication game. Biologically Inspired Cognitive Architectures, 11:10-21.
3.
We acknowledge that our dataset was con- structed based on specific rules, which means its dialog syntax may differ from genuine con- versations. In the real world, higher-order interactions might occur in a more implicit manner, embedded within more intricate dia- logues and questions. We plan to address this in future research.
Harmen De Weerd, Rineke Verbrugge, and Bart Ver- heij. 2022. Higher-order theory of mind is especially useful in unpredictable negotiations. Autonomous Agents and Multi-Agent Systems, 36(2):30.
Harmen de Weerd and Bart Verheij. 2011. The advan- tage of higher-order theory of mind in the game of limited bidding. In Proceedings Workshop Reason- ing about other Minds, CEUR Workshop Proceed- ings. volume 751, pages 149-166.
4.
We share our thoughts on the future of NLP and research with LLMs, hoping to stimulate research that draws inspiration from human intelligence, understands humans better, and serves humans better. We admit that there exist alternative ways of moving forward NLP research. We welcome feedback and open discussion on how we can collectively advance NLP research.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized Ilms. arXiv preprint arXiv:2305. 143144
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minncapolis, Minnesota. Association for Computational Linguistics.
References
Simon Baron-Cohen, Alan Leslie, and Uta Frith. 1985. Does the autistic child have a theory of mind? Cognition, 21(1):37-46.
Uta Frith and Christopher Frith. 2003. Development and neurophysiology of mentalizing. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 358(1431):453 473.
Nora Belrose, Zach Furman, Logan Smith, Danny Ha- lawi, Igor Ostrovsky, Lev McKinney, Stella Bider- and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprini arXiv:2303.08112
Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. 2021. Grounded language learning fast and slow. International Conference on Learning Representa- tions.
Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models
Pascal Hitzler. 2022. Neuro-symbolic artificial intelli- gence: The state of the art.

---

Zhijing Jin, Jiarui Liu. Zhiheng Lyu, Spencer Poff, Mrin- maya Sachan, Rada Mihalcea, Mona Diab, and Bern- hard Schölkopf. 2023. Can large language models infer causation from correlation?
Christopher Osterhaus and Susanne Koerber. 2021. The development of advanced theory of mind in middle childhood: A longitudinal study from age 5 to 10 years. Child Development, 1875:18872-1888.
Daniel Kahneman. 2011. Thinking, fast and slow. macmillan.
Joon Sung Park, Joseph CO'Brien, Carrie J Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:230..0344..
Michal Kosinski. 2023. Theory of mind may have spon- taneously emerged in large language models.
Matthew Le. Y-Lan Boureau, and Maximilian Nickel. 2019. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)) pages 5872-5877.
Josef Perner. 1991. Understanding the representational mind. The MIT Press.
David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515-526.
Garret Ridinger and Michael McBride. 2017. Theory of mind ability and cooperation. Manuscript, Univ. California, Irvine.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature, 521 (7553)44446-444
Bethany Liddle and Daniel Nettle. 2006. Higher-order theory of mind and social competence in school-age children. Journal of Cultural and Evolutionary Psy- chology, 4(3-4):231-244
David E. Rumelhart, James L. McClelland, and PDP Re- search Group. 1986. Parallel Distributed Process- ing: Explorations in the Microstructure of Cognition: Foundations. The MIT Press.
Ziqiao Ma, Jacob Sansom, Run Peng, and Joyce Chai. 2023. Towards holistic landscape of situated theory of mind in large language models. In Findings of the Association for Computational Linguisties: EMNLP 2023. Association for Computational Linguistics.
Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022. Neural thcory-of-mind" on the limits of social intelligence in large Ims. arXiv preprint 2rxiv:2210.13312.
Jude W Shavlik. 1994. Combining symbolic and neural learning. Machine Learning, 14:321-331.
Inbal Magar and Roy Schwartz. 2022. Data contamina- tion: From memorization to exploitation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 157-165, Dublin, Ireland. Association for Computational Linguistics.
Herbert A Simon and Allen Newell. 1971. Human problem solving: The state of the theory in 1970. American psychologist, 26(2):145.
Dan Sperber and Deirdre Wilson. 2002. Pragmatics, modularity and mind-reading. Mind & language, 17(1-2):3-23.
Nick McKenna, Tianyi Li, Liang Cheng, Moham- mad Javad Hosseini, Mark Johnson, and Mark Steed- man. 2023, Sources of hallucination by large lan- guage models inference tasks. arXiv preprint arXiv:2305.1455 14552.
Masanori Takano, Takaya Arita, et al. 2006. Asymmetry between even and odd levels of recursion in a theory of mind. Proceedings of ALife X, pages 405-411.
Antoine Miech, Jean-Band -aptiste Alayrac, Ivan Laptev, Josef Sivie, and Andrew Zisserman. 2021. Think- ing fast and slow: Efficientt text-to-visual retrieval with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 9826-9836.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Rachel LC Mitchell and Louise H Phillips. 2015. The overlapping relationship between emotion perception and theory of mind. Neuropsychologia, 70:1-10.
Tomer Ullman. 2023. Large language models fail trivial alterations to theory-of-mine tasks. arXiv preprint arXiv:2302.08399
Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Thomas I. Griffiths. 2018 Evaluating theory of mind in question answering. arXiv preprint arXiv:1808.09352
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators.
OpenAl. 2023. Gpt-4 technical report.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.1199h)
TB OpenAI. 2022. Chatgpt:
Optimizing language mod- els for dialogue. OpenAI.

---

Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee. Ari S Morcos, and Dhruv Batra. 2023. Emergence of maps in the memories of blind navigation agents. arXiv preprint arXiv:2301.1326.
A Author Contributions
Yufan Wu and Yinghui He conceived of the idea and planned the experiments. Yufan Wu and Yinghui He took the lead in writing the script for dataset generation. Yilin Jia, Yufan Wu, and Yinghui He carried out the experiments of testing language models. Yulong Chen and Naihao Deng supervised the project. Naihao Deng came up with the high-level idea for the paper, which later refined in team discussions. All authors contributed the writing and editing of this paper. Specifically, Yufan Wu and Yinghui He drafted the paper. Yilin Jia contributed to the writing for experiment setups. Rada Mihalcea, Yulong Chen, and Naihao Deng helped edit the paper.
Terry Winograd. 1971. Procedures as represen- tation for data in a computer program for under- standing natural language. Technical report, MAS- SACHUSETTS INST OF TECH CAMBRIDGE PROJECT MAC.
Chenhan Yuan, Qianqian Xie, and Sophia Ananiadou. 2023. Zero-shot temporal relation extraction with chatgpt. arXiv preprint 4rx44:230..0454.
B HI-ToM Details
B.1 Assumptions
Our simplified deception and belief mechanisms based four assumptions. Table 7 shows the original assumption list attach to each story in the dataset and prompt into LLMs.
Note: You should assume the following.
(1) An agent witnesses everything and every novement before exiting a room.
(2) An agent A can infer another agent B's mental state only if A and have been in the same room, or have private or public interactions.
(3) Note that every agent tend to lie. What character tells others doesn't affect his actual belief. An agent tend to trust a agent that exited the room later than himself. The exit order is known to all agents.
(4) Agents in private communications know that others won't hear them, but they know that anyone can hear public claims.
Table 7: Assumption list attached to each Hi-ToM story and prompt into LLMs.
B.2 Story Generation Details
Algorithm and Algorithm 2 provide the pseu- docode for the generation process of each chapter and the whole story HI-TOM.
In Algorithm 1, the function MOVE is employed to populate the story components into the template, thereby producing sentence that describes the movement. The function COMMUNICATE gener- ates content related the "tell" action. Meanwhile, the RANDOM RANDOM_DDSTTAAAE DISTRACTOR function introduces random distractors into the story.

---

In Algorithm 2, the question generator Q_ GEN randomly picks the agents and the object appearing in the story and populates them into predefined question template. Then, the answer generator A_GEN =_EEN& generates the answer to the correspond- ing question based on dictionary that traces the beliefs of different orders of each agent.
Claude, Joint
CoTP
@ CoTP
Algorithm 1 Hi-ToM Chapter Generation Algorithm
0.9
0.6
0.4
0.3
0.15
1 0.7
0.15
0.05
Input: agents, room, conts, obj
Output: chop
0.85
0.5
0.25
0.05
0.65
0.3
0.1
0.05
1: function CHAP(agents, room, cords, obj)
3
2: for agent agents do
3: set no_movve to random boolean value
4: move 6 MovE(agent, conts, obj, no move)
VP
83, VP
5: add move into chap
0.6
0.3
0.15
0.15
0.15
0.5
0.2
0.05
0.05
0.05
6:
end for
7:
com t COMMUNICATE(ogeent)
0.45
0.35
0
0.6
0.25
0.2
0.15
0.1
8:
rd f RANDOM DISTRACTOR carioo/agggntsccntti
9:
add com and rd into chop
0.6
0.45
0.15
0.05
05
3-
0.5
0.3
0.1
10: return chap
11:
end function
Figure 8: Joint accuracy results of Claude-instant.
Experiment Details
C.1 Prompting inputs
In our experiments, the average number of tokens in single prompt is 453.3, and the total token number of our prompts each model is 543968, including VP and CoTP prompts on stories with or without deception.
Table is a sample CoTP prompt in our experi- ments. We specify the range of the story, question, choices, and assumptions to enhance the models' understanding. We also order each line of the story to indicate the chronological order. The provided answer choices are all the containers appearing in the story.
Guanaco,
CoTP
8, CoTP
C.2 Supplementary Results
0.4.5
0.1
0.1
0.1
2
0.6
0.2
0.05
Figure 9 and Figure 8 shows the detailed joint accu- racy results of Guanaco and Claude. The joint ac- curacy generally decreases as the story length and the ToM orders increase, aligning with the results GPT-4 and GPT-3.5. The overall joint accuracy performance of Claude better than Guanaco.
0.7
0.25
0.05
0
3 0.2 0.15
6
3
2
4
VP
M. VP
1-0.55
1-
2-0.55
0.35
0.15
0.1 0.1
2- 0.45
0.3
0.15 0.15 0.15
3-0.45
0.3
0.2
0.2
0.15
0.4
0.2
0.1
0.05 0.05
Figure 10 to Figure 13 show the standard ac- Claude-
à
3
1
2
curacy results of GPT-4, GPT-3.5-ture instant and Guanaco 65B, as the break-down details of Table 5. Among the four models, GPT-4 has the highest and most stable performance, reaching nearly perfect accuracy on the zeroth order and higher than 20% on the fourth.
Figure 9: Joint accuracy results of Guanaco 65B.
Under CoTP prompting, each model reaches a high performance on zeroth-order questions, espe-

---

Read the following story answer the multiple-choice question. Think step-by-step. Provide the answer first, and then explain it. Story:
William, Jack, Charlotte,
Hannah entered the hall.
Noah saw monkey.
carrot is in the red_basket.
William made no movements and stayed in the hall for minute.
william exited the hall.
Hannah saw a mouse.
Jack made no movements and stayed in the hall for minute.
Jack exited the hall.
Charlotte made no movements and stayed in the hall for 1 minute.
18 Charlotte exited the hall.
11 Charlotte dislikes the tomato.
12 Noah moved the carrot to the green_envelope.
13 Noah exited the hall.
14 Hannah moved the carrot to the red_basket.
15 Hannah exited the hall.
16 william, Jack, Charlotte, Noah and Hannah entered the waiting_roon.
17 Charlotte publicly claimed that carrot is in the green_envelops
18 Hannah privately told Charlotte that the carrot is in the blue_container.
Question: Where does Charlotte think Jack thinks Hannah thinks William thinks the carrot is?
Choices: A. erren_envlopp,, B. red basket, C. cout_ccnnaiier, D. red_crate, E. green_drawer, F. blue_bucket, G. green_cupboard, H. red_bottle, I. green_treasure_chhest J. blue_cupboard,
K. red_pantry, L. red_container, M. blue_bathtub, N. red_envelope, 0. blue_pantry
Note: You should assume the following. (1) An agent witnesses everything and every movements before exiting a room. (2) An agent can infer another agent B's mental state only if A and B have been in the same roon, have private or public interactions. (3) Note that every agent tend to lie. What character tells others doesn't affect his actual belief. agent tend to trust agent that exited the room later than himself. The exit order is known to all agents. (4) Agents in private communications know that others won't 't hear them, but they know that anyone hear any public claims.
Table 8: An example CoTP prompt of a one-chapter HI-TOM story with a fourth-order question.
Algorithm 2 Hi-ToM Story Generation Algorithm
Input:
Number of chapters: & E (1,2,3)
Story components: Rooms, Objects, Containers, Agents
Output:
story, guestion, onswer function STORY(É, Rooms, Objects, Containers, Agents)
1:
GPT-4, Standard
2: for i to do
CoTP
M CoTP
3: randomly choose room,obj, conts, agents
4: clap x CHAP(room, obj, conts, agents)
1.0
0 8.85
0.8
0.65 0.6
0.9
0.65 0.55
0.3
5: add chop into story
6: end for
0.95
0.8
0.6
0.35 0.3
2
1.0
0.55 0.45
0.5
0.5
7: question Q GEgnAge Objects)
0.95
0.55
0.3
0.4
0.45
1.0
0.45 0.45
0.2
8: answer <- A_ Se_GGE((AA (ggents, Objects)
9: return story, question, onswer
3
1
2
à
10: end function
VP
8. VP
1.0
0.7
0.6
0.5
0.4
1
1.0
0.6 0.35 0.35
0.4
0.95
0.8
0.45
0.4
0.4
2
1.0
0.6
0.6
0.6
0.45
0.9
0.55
0.35
0.5
0.5
0.45
0.55
0.55
0.5
6
2
à
Figure 10: Standard accuracy results GPT-4
cially for stories without agent communications, and their performance deteriorates with increased ToM order and story length. Yet, under VP prompt- ing, Claude and Guanaco exhibit a uniform perfor- mance of around 50% across all the orders.

---

GPT-3.5, Standard
the beginning or the end of story.
CoTP
CDTP
0.95
0.45
0.45
0.05
0.1
0.55 0.4
0.3 0.3
0.2
Second
0.42
0.58
Third 0.58
0.42
2
0.9
0.6
0.2
0.15
0.2
0.7 0.5
0.35
0.4
0.25
0.2
0.2
3-0.65 0.55 0.15 0.15
0.05
-Second
0.52
-Third
à
VP
0. VP
Correct
Incairect
Correct
Incorrect
1
0.7
0.7
0.3
0.25
0.1
0.3
0.3
0.3
0.05
Second last
0.49
D.51
0.52
0.3
0.5
0.2
0.05
0.2
0.45
0.3
0.55
0.2
0.2
last
2
3
Second last
0.5
0.5
last
0.49
0.51
Figure
Il: Standard accuracy results of GPT-3.5-turbo.
Correct
Incorrect
Correct
Incorrect
Figure 14: Performance of GPT-4 in the cases when the correct answer does or does not lie in a certain position.
Claude, Standard
CoTP
M. CoTP
1
0.9
0.7
0.6
0.55
0.5
1-
0.7
0.25 0.3 0.3
0.3
Figure 15 shows similar observations for Claude. The plots for the last and first positions of con- tainers show higher frequency in the top-left and bottom-right cells, while the plots for other posi- tions do not imply such pattern.
2
0
0.85
0.6
0.4
0.3
0.05
2-0.65 04 0.35 0.4 0.3
0.5
0.55
0.25
2
à
VP
3 VP
1.
0.6
0.65
0.5
0.7
0.45
1- 0.5
0.35 0.35
0.45
0.4
0.45
0.5
0.25
0.45
0.6
0.6
0.4
0.25
Last 0.52
0.48
First 0.53
0.47
2
-Last
0.42
0.58
-First
0.36
0.64
Figure 12: Standard accuracy results of Claude-instant.
Correct
Incorrect
Incorrect
Guanaco, Standard
Second
0.43
0.57
Third-
0.51
0.49
CoTP
0.4
0.35
0.45
0.45
0.85
0.5
0.3
0.3
SSecon
0.44
0.56
Third
0.42
0.58
2
Correct
Incorrect
Incorrect
3
0.7
0.35 0.4
0.3 0.2
0.2
0.15
0.25
0.15
0.3
2
Second
0.42
0.58
Third last
0.38
0.62
VP
M
1-0.55
Second
0.44
0.56
-hhird
0.45
0.55
last
2
2-0.5
0.45
0.5
0.4
0.55
2-0.45
0.25
0.45
0.45
Correct
Incorrect
Incorrect
0.45
0.45
0.35
0.4
0.4
0.4
0.45
0.6
0.4
0.5
2
2
3
Performance of Claude in the cases when the Figure 15:
correct answer does or does not in a certain position.
Figure 13: Standard accuracy results of Guanaco 65B.
Figure 16 details the appearance frequency of the five reasoning errors in the step-by-step responses of GPT-3.5. Compared to the error ratios of GPT-4 (Figure 7), the frequencies of commonsense errors, hallucinations, and spurious causal inference are significantly higher, implying GPT-3.5's immature perceptions of the world and deficient logical reasoning abilities. The occurrence of insufficient reasoning depth and temporal ignorance escalates
Figure 14 illustrates the performance compari- son of GPT-4 between the case when the correct answer is in certain position in the middle of the story, and the when it is We observe that GPT-4 does not significantly perform better when the correct answer lies in the middle of the story. This serves as a contrast to Figure 5, highlighting the better ability of GPT-4 to capture answers at

---

GPT-3.5
0.9
Frequenty
0th
1st
2nd
3rd
4th
Insufficient reasoning-depthh Commonsense errors
Temperal ignorance Spurious causai inference
Hallucinations
Figure 16: Ratio of the five reasoning errors in GPT- 3.5's responses.
in higher-order responses.
The comparison between Figure and Figure 16 yields that GPT-4 has not resolved the errors of commonsense. insufficient reasoning depth, and temporal ignorance, while hallucinations and spuri- ous causal inference have been largely addressed.


Reviews:
[{"review_id": "5y2CAuLNZ3", "reviewer": "Reviewer_YSgP", "summary": "", "questions": "See weaknesses", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review. We are glad that you found our dataset interesting, and the analysis informative and insightful. Below, we address each of your concerns:\n\n\n* **Response on the zero-shot setting (weakness 1):** \nRegarding evaluations on only zero-shot settings, our primary objective is to assess the innate Theory of Mind (ToM) ability in LLMs. We believe that an LM with genuine ToM ability could master ToM tasks even under a zero-shot setting, as discussed in Shapira et al. (2023).  In the few-shot setting, the provided examples may bias the LLMs, and the models may end up simply imitating instead of genuinely conducting ToM reasoning. Such issues have been mentioned in prior work (Jin et al., 2022). Following previous research (Jin et al., 2022; Shapira et al., 2023), we also use a zero-shot setting to test the innate ability of LLMs. We will clarify this point in the revision.\n\n\n* **Response on the necessity of high-order ToM (weakness 2):**\nWe appreciate the feedback. To strengthen the motivation of our dataset, we will include the following additional arguments in the paper:\n\t* One of the motivations to construct a higher-order ToM is that several recent papers claimed that LLMs acquire the ToM ability (Kosinski, 2023; Bubeck et al., 2023). As Liddle and Daniel (2021) show that human beings are able to conduct higher-order ToM ability, we want to check if LLMs indeed possess the genuine ToM ability like human beings as claimed by those papers. \n\t* From an application perspective, we believe that higher-order ToM ability is crucial for generalist LLMs. It enables better NLP applications for social goods such as emotional support and empathetic communication (Mitchell et al., 2015). For instance, in an emotional support scenario where the user seeks advice on dealing with multiple friends or families, LLMs need to first comprehend the complicated scenarios and the multi-layer thoughts of the people involved. Such applications require higher-order ToM to recursively reason about the feelings or thoughts among the multiple people involved.  In other use cases where LLMs need to consider the beliefs of multiple agents, such as telling a spy story, advising game strategy, and planning business or legal strategy, a generalist LM also utilizes its higher-order ToM ability.\n\n* **Response on the unrealistic introductory example (weakness 2):**\nWe agree that we should add a concrete real-world example to motivate the construction of our dataset. We will modify Figure 1 in the revision as follows: We may use the scene shot from the famous TV series Friends: Phoebe says “They don't know that we know they know we know!” when she and Rachel try to identify whether Monica and Chandler know that Rachel and Phoebe know they are dating each other. Here is the [link](https://www.youtube.com/watch?v=wFmTEKnn64s) to the video clip in Friends, where the ToM increases from zeroth order to fourth order.\n\n* **Response on the paper structure:**\nWe acknowledge the overlap in content and we will reduce some discussions in Section 6 on the importance of ToM ability and the applications of ToM in LLMs.\n\n* **Response on incorporating the recent works:**\nWe're grateful for the reference to the recent work on LLMs' ToM capabilities. We will evaluate the models with this approach in our follow-up revision.\n\n**Reference**\n\nBubeck, Sébastien, et al. \"Sparks of artificial general intelligence: Early experiments with gpt-4.\" arXiv preprint arXiv:2303.12712 (2023).\n\nJin, Zhijing, et al. \"When to make exceptions: Exploring language models as accounts of human moral judgment.\" Advances in neural information processing systems 35 (2022): 28458-28473.\n\nKosinski, Michal. \"Theory of mind may have spontaneously emerged in large language models.\" arXiv preprint arXiv:2302.02083 (2023).\n\nLiddle, Bethany, and Daniel Nettle. \"Higher-order theory of mind and social competence in school-age children.\" Journal of Cultural and Evolutionary Psychology 4.3-4 (2006): 231-244.\n\nLucy, Li, and David Bamman. \"Gender and representation bias in GPT-3 generated stories.\" Proceedings of the Third Workshop on Narrative Understanding. 2021.\n\nMalkin, Nikolay, et al. \"GPT Perdetry Test: Generating new meanings for new words.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n\nMitchell, Rachel LC, and Louise H. Phillips. \"The overlapping relationship between emotion perception and theory of mind.\" Neuropsychologia 70 (2015): 1-10.\n\nShapira, Natalie, et al. \"Clever hans or neural theory of mind? stress testing social reasoning in large language models.\" arXiv preprint arXiv:2305.14763 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. An interesting dataset that is easy for people to create themselves (given the code with be released) and shows where current power LLMs still lag. \n2. The analysis of model behaviors is very interesting and informative. They give insights into both the kinds of examples future datasets should contain and how new approaches can try to address the shortcomings of the current models and improve performance on such tasks.", "reasons_to_reject": "1. The authors only evaluate the LLMs on zero-shot settings, all the models considered in this paper are capable of in-context or few-shot learning, these settings should be considered before claiming that the models perform poorly on this dataset. 3-4 step-by-step reasoning can easily be hand-written for these experiments. \n2. I understand that theory of mind is an important reasoning skill that models should have. But after reading this paper, I am not super convinced that higher than the second-order theory of mind is a necessary reasoning ability in generalist language models (but maybe for those designed for specific tasks like chess or negotiation). Also, the examples in the dataset, at least the ones shown in the paper, are not very realistic. I am not arguing that this ability should not exist in LMs but I think a convincing realistic example should be provided to motivate the construction of this dataset in the introduction even though the examples in the dataset are synthetic.\n3. Not a weakness per se, but I am not sure section 6 is adding much to the overall point of this paper. It seems like an extension of the introduction motivating why ToM is a necessary property to have. In my opinion, it should be absorbed in the introduction itself. On that note, this paper could easily be a short paper on some reorganization. \n4. Finally, again not a criticism, but there has been some recent work on improving LMs' theory of mind capabilities which claim to work on any order of ToM: https://arxiv.org/abs/2306.00924. Not a reason to reject since this work just came out, but in a future version, the authors should evaluate their models with this approach.", "questions_to_authors": "See weaknesses", "paper_topic_and_main_contributions": "This paper contributes a new dataset for measuring higher-order theory of mind capabilities in NLP models. Existing datasets in this domain are synthetically constructed and measure reasoning in stories up to 2nd order ToM. This paper takes their framework and introduces new templates to extend it to up to 4th order while introducing distractor sentences as well as previous work. \n\nThey compare the performances of LLMs like GPT-4, Claude, and Guanaco on this task using zero-shot simple and chain-of-thought prompting and show these models show a decreasing trend of performance as the order of ToM is increased showing the difficulty of this task. The authors show interesting discussions and analyses of model behaviors both when the models are right and when they are wrong.\n\nIn the end, the authors also discuss the implications of these results on NLP.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pjKzfngHup", "reviewer": "Reviewer_Trb7", "summary": "", "questions": "[Q1] I have a major concern, and I'd solicit Authors to elaborate on this point. LM are basically probability distributions over text sequences: as such, LMs are unfit to deal with logical reasoning or matters such as acquiring any sort of propositional knowledge (what is needed to face ToM related tasks).\nIn this sense, one is not surprised for the poor perfomance attained in the evaluation. Authors should then explain how they expect that whatever LM trained for dealing with ToM may generalize to previously unseen stories. How a device such as a LM is expected to be helpful in higher-order tasks, such as deciding whether 'Where does A4 think A3 thinks A2 thinks A1 thinks object O is?'? \nAdditionally, putting at the heart of the system a closed resource such as GPT* may be detrimental to investigate inner mechanisms and implementation details. Are the Authors willing to elaborate on these issues?\nSuch issues should be either addressed in the introductory Section, or discussed after providing the experimental results. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We appreciate the time you took to review our work and offer constructive comments. Below, we address each of your concerns:\n\n-   **Response on LMs’ incompatibility on ToM tasks:**\n    We will add a discussion on this point after the experimental results as follows:\n\t-   We acknowledge the view that LLMs, primarily being probability distributions over text sequences, might not excel in higher-order ToM tasks. However, several recent papers claimed that ToM has spontaneously emerged in LLMs as a byproduct of their development, or LLMs have acquired genuine ToM ability (Kosinski, 2023; Bubeck et al., 2023). Our findings challenge such a claim and expose the deficiency of current SOTA LLMs, especially for higher-order ToM.\n\t-   Still, we want to seek methods to equip LLMs with higher-order ToM ability, as it may help the model better deal with complicated scenarios in downstream tasks like emotional support and strategy advising. A recent paper [Sclar et al., 2023](https://arxiv.org/abs/2306.00924) introduces a new plug-and-play prompting method, SYMBOLICTOM, to endow LMs with a dramatically enhanced ability to reason about higher-order ToM. Thus, we believe that when equipped with proper modules, LLMs have the potential to complete ToM tasks, and this proposed method is one potential way. We will evaluate LLMs on our dataset with this approach in our follow-up revision.\n\n-   **Response on testing closed-source models:**\nWe will add a discussion on this point after the experimental results as follows:\n\t- We concur with concerns about the opacity of closed-source models like GPT*. The detrimental effects of using black-boxed LLMs include unknown pre-training corpus, uncertainty about the reasons behind the model's errors, and difficulties in improving models’ performance (Ye et al., 2022; Ignat et al., 2023; Rodgers, 2023).\n\t- Therefore, in our paper, we also tested a competitive open-sourced LLM – Guanaco 65 B, which was the SOTA for the open-sourced ones when we wrote our paper. We believe that open-source models offer a clearer pathway to enhancing the logical reasoning capabilities of LMs (Rodgers, 2023). We advocate for increased community involvement in benchmarking these models.\n\nWe hope our responses address your concerns and provide clarity on our methodology and findings. We are open to further suggestions to improve our manuscript.\n\n  \n**Reference**\n\nBubeck, Sébastien, et al. \"Sparks of artificial general intelligence: Early experiments with gpt-4.\" arXiv preprint arXiv:2303.12712 (2023).\n\nKosinski, Michal. \"Theory of mind may have spontaneously emerged in large language models.\" arXiv preprint arXiv:2302.02083 (2023).\n\nIgnat, Oana, et al. \"A PhD Student's Perspective on Research in NLP in the Era of Very Large Language Models.\" arXiv preprint arXiv:2305.12544 (2023).\n\nRodgers, Anna. \"Closed AI Models Make Bad Baselines.\" Towards Data Science, 2023. Accessed 23 May 2023.\n\nSclar, Melanie, et al. \"Minding Language Models'(Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker.\" arXiv preprint arXiv:2306.00924 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes an interesting work on a relevant cognitive device, such as Theory of Mind. This is a relevant theme to NLP, since it lies at the base of many cognitive processes, such as those involved in NL understanding.\n", "reasons_to_reject": "- The computational approaches employed in the experiments seem insufficient to deal with the task at hand: or, equivalently, the task might not be appropriate for Language Models.", "questions_to_authors": "[Q1] I have a major concern, and I'd solicit Authors to elaborate on this point. LM are basically probability distributions over text sequences: as such, LMs are unfit to deal with logical reasoning or matters such as acquiring any sort of propositional knowledge (what is needed to face ToM related tasks).\nIn this sense, one is not surprised for the poor perfomance attained in the evaluation. Authors should then explain how they expect that whatever LM trained for dealing with ToM may generalize to previously unseen stories. How a device such as a LM is expected to be helpful in higher-order tasks, such as deciding whether 'Where does A4 think A3 thinks A2 thinks A1 thinks object O is?'? \nAdditionally, putting at the heart of the system a closed resource such as GPT* may be detrimental to investigate inner mechanisms and implementation details. Are the Authors willing to elaborate on these issues?\nSuch issues should be either addressed in the introductory Section, or discussed after providing the experimental results. ", "paper_topic_and_main_contributions": "This paper proposes an annotated resource to investigate to what extent LMs are able to account for theory of mind. Theory of Mind (ToM) is a theory that tries to explain one's own and others' mental states; it is at the heart of many cognitive processes, including language understanding. \nCollected data also contain information on inferential steps as these are performed in the frame higher-order ToM: higher-order ToM involves recursively reasoning on others' beliefs, and is thus a major issue for the sort of knowledge globally grasped by LMs. Experiments were conducted on the newly released resource, employing two prompting approaches: Vanilla Prompting (VP) and Chain-of-Thought Prompting (CoTP); results were computed based on two metrics, standard and joint accuracy. In the latter setting, an answer was scored as correct only if all lower-order related questions within the same story were correctly answered. The GPT-4 and GPT-3.5 LMs were tested, showing a drop in performance on higher-order ToM tasks. The main results based on the analysis of results were as follows:\n- CoTP prompting yields insignificant performance gains;\n- Increased ToM order leads to decreased performances;\n- LLMs' performance decreases as there are more deception communications involved.\n\nI have a major concern, and I'd solicit Authors to elaborate on this point. LM are basically probability distributions over text sequences: as such, LMs are unfit to deal with logical reasoning or matters such as acquiring any sort of propositional knowledge (what is needed to face ToM related tasks).\nIn this sense, one is not surprised for the poor perfomance attained in the evaluation. Authors should then explain how they expect that whatever LM trained for dealing with ToM may generalize to previously unseen stories. How a device such as a LM is expected to be helpful in higher-order tasks, such as deciding whether 'Where does A4 think A3 thinks A2 thinks A1 thinks object O is?'? \nAdditionally, putting at the heart of the system a closed resource such as GPT* may be detrimental to investigate inner mechanisms and implementation details. Are the Authors willing to elaborate on these issues?\nSuch issues should be either addressed in the introductory Section, or discussed after providing the experimental results. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 189: \"we incorporate distractor sentences that **relates** an agent\" -> 'relate'\n"}, "extra_text": {}}, {"review_id": "8LfZ6WBhnm", "reviewer": "Reviewer_ZmFn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review. We are glad you found our work interesting and the discussion thoughtful and insightful. Below, we address each of your concerns:\n\n\n- **Response on whether we’re assessing LMs’ genuine ToM ability (weakness 1):**\n\t-   We concur that current LLMs tend to capture spurious patterns from the context as well as the question. Therefore, we made some efforts on the data construction to mitigate the impact of spurious patterns:\n\t\t-   We made sure that the correct answers to the questions were not only located at the beginning or the end of the stories, but were evenly distributed (as mentioned in Section 2.3, Line 214-221). This prevents LLMs from hitting high accuracy by consistently choosing the original or final location of the object.\n\t\t-   We informed LLMs of the common sense and assumptions used in our story setting at the end of each query (Appendix B.1, Line 800-804). This discourages LLMs from generating unreasonable stories by overlooking obvious facts (Shapira et al., 2023), and avoids obtaining a very low accuracy.  \n\t- While LLMs indeed have a tendency to exploit data patterns, as mentioned in Shapira et al. (2023), our benchmark aims to present a more intricate challenge that is harder to solve through mere pattern recognition. We believe the unpredictable nature of our stories and questions helps reflect a more genuine form of ToM reasoning.\n\n  \n\n-   **Response on lack of samples (weakness 2):**\n\t-   We would like to point out that the size of our dataset (number of stories) is **1.2k**, instead of 146. The numbers in Table 1 reflect the numbers of different story components. For example, “30 rooms” means that there are 30 possible choices of room names (e.g. Kitchen) in our stories. \n\t-   The 1.2k size is comparable to the size of the test set in the previous dataset (ToMi, 2023), so we believe that the dataset is large enough for a valid evaluation. We will highlight these clarifications in our revision.\n    \n\n  \n\n-   **Response on lack of implementation details for the question and answer generator (weakness 2):**\n\t-   We agree that we need to explain more clearly how we implemented the question and answer generators, and we will modify Line 816-820 in the camera-ready version as follows:\n\t\t-   For the generation of a question, we randomly select an object and several names, then populate them into the pre-defined question template.\n\t\t-   To produce the answer, we construct a comprehensive dictionary that documents the position of each object and the multi-layer beliefs of every agent. The answer is then directly retrieved using the names and objects selected into the question.\n\t-   For the detailed code, our implementation of the question and answer generator inherits the publicly released code of the paper “Evaluating Theory of Mind in Question Answering” (2018), as we specified in the Data Generation Section (Line 163-164). Also, we plan to release our code upon acceptance, as mentioned in our paper (Line 555-556).\n    \n\n  \n\n-   **Response on the motivation of deception (weakness 3):**\nWe understand that a motivation for the deceptive communication should be presented, and we will add additional explanations for the motivation in the revised version as follows:\n\t-  Our story setting was created to emulate the dynamics of the complicated social life, where sophisticated people take different actions (agents move the object) and strategically communicate with each other (agents conduct deceptive communications).\n\t- Also, deceptive communications can add another layer of complexity to the ToM reasoning process, which requires LLMs to not only reason about agents’ perceptions of other agents’’ knowledge of the object’s location, but also reason about whether an agent would trust another agent. In this way, we evolved the simplistic toy stories in the previous dataset (ToMi, 2019), and solved a core problem in the previous evaluations that the answers may be simply found from the object’s original or final location.\n\n-   **Response on reproducibility:**\nWe will make our datasets as well as the generation code publicly available upon acceptance.\n\n  \n\nWe hope we clarified your concerns, and we would be happy to engage in further discussions as needed.\n\n**Reference**\n\nShapira, Natalie, et al. \"Clever hans or neural theory of mind? stress testing social reasoning in large language models.\" arXiv preprint arXiv:2305.14763 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors explore a higher-order Theory of Mind (ToM) benchmark by constructing and validating ToM stories along with corresponding questions and answers. This has the potential to become a valued resource for evaluating the cognitive abilities of LLMs.\n2. The authors evaluate cutting-edge LLMs including GPT-4, GPT-3.5-Turbo, Claude, and Guanaco. They demonstrate that the performance of LLMs on the proposed higher-order ToM benchmark is far from perfect.\n3. The authors provide detailed analysis of the experimental results and offer insights into how ToM-related research could influence future developments in the LLM field. Their discussion is thoughtful and insightful.", "reasons_to_reject": "1. A major concern is whether such story-grounded question-answering benchmarks can genuinely assess LLMs' ToM abilities. As discussed in the paper \"Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models\", LLMs tend to answer questions based on linguistic patterns and spurious correlations rather than actual cognitive processes. The authors should address this issue with proper discussion.\n2. Based on Table 1, the proposed Hi-Tom dataset is relatively small, containing only 146 samples. This raises concerns about the validity of evaluation. Additionally, the authors only present pseudocode for the question and answer generators without explaining their actual implementation details.\n3. In designing agent communications, the authors introduce two modes: public and private. They set that the shared information is deceptive. However, the authors do not sufficiently explain the motivation behind incorporating deceptive information. While LLMs' performance decreases with more deceptive communications, this seems obvious and does not justify the necessity of deception.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces Hi-ToM, a new benchmark for evaluating higher-order Theory of Mind (ToM) reasoning in large language models (LLMs). It contains stories with questions up to fourth-order ToM, incorporating agent communications. The stories are longer with more agents and containers compared to prior ToM datasets, requiring comprehending the complete storyline. The dataset is generated using scripts adapted from prior work, with manual verification for quality. Experiments using GPT-3.5, GPT-4, Claude, and Guanaco show performance declines as ToM order increases, especially with deceptive communications, demonstrating limitations in recursive reasoning. Failure analysis reveals insufficient reasoning depth, temporal ignorance, and other deficiencies. The key contribution is the new higher-order ToM benchmark that highlights challenges for LLMs in complex multi-agent reasoning.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers provided scores 4,4,3 for soundness and 3,3,3 for excitement.

The following strengths and weaknesses were prominent in the reviews:

Strengths:
- addresses interesting cognitive device relevant to NLP (R2)
- interesting dataset that should be easy to use (R1, R3)
- analysis of model behavior (R1, R3)
- evaluate a range of LLMs (R3)

Weaknesses:

- only considers zero-shot settings (R1)
- questionable realism of examples (R1)
- it was questioned whether the task might be inappropriate for LLMs (R2, R3)
- dataset is small (R3) and implemention details are underspecified (R3)
Decision: Accept-Findings