FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Authors: Hyunwoo Kim
Melanie Sclar
Xuhui Zhou
Ronan Le Bras
Gunhee Kim
Yejin Choi
Maarten Sap
Venue: EMNLP 2023 Main (2023)
Keywords: theory of mind
benchmark
interaction
conversation
large language model
llm
TL;DR: We introduce FANToM, a benchmark for stress-testing machine theory of mind in information-asymmetric conversations.

Abstract: Theory of mind (ToM) evaluations currently focus on testing models using passive narratives that inherently lack interactivity. We introduce FANToM, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering. Our benchmark draws upon important theoretical requisites from psychology and necessary empirical considerations when evaluating large language models (LLMs). In particular, we formulate multiple types of questions that demand the same underlying reasoning to identify illusory or false sense of ToM capabilities in LLMs. We show that FANToM is challenging for state-of-the-art LLMs, which perform significantly worse than humans even with chain-of-thought reasoning or fine-tuning.

FANTOM:
A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Hyunwoo Kim
Melanie Sclar
Xuhui Zhou o
Ronan Le Bras 9
Gunhee Kim
Yejin Choi
Maarten Sap
< Allen Institute for Artificial Intelligence
University of Washington
& Carnegie Mellon University
Scnul National University
Abstract
Kailey: Hey guys, I'll go grab a coffee.
Theory of mind (ToM) evaluations currently fo- cus on testing models using passive narratives that inherently lack interactivity. We introduce S FANTOM, new benchmark designed to stress-test ToM within information cammattioo-asymmttrii conversational contexts via question answering. Our benchmark draws upon important theoreti- cal requisites from psychology and necessary empirical considerations when evaluating large language models (LLMs). In particular, we formulate multiple types of questions that de- mand the same underlying reasoning to identify illusory or false sense of ToM capabilities in LLMs. We show that FANTOM is challenging for state-of-the-arr LLMs, which perform sig- nificantly worse than humans even with chain- of-thought reasoning fine-tuning.
Sally: See you, Kailey Hay Linda, did you get a clo??
Linds: Yooh, I got goidon recriever. She's so adorable.
David: What's her favorite food!
Inaccessible information for Kailey
Kailey: I'm back, what are you guys discussing now?
Linda was just telling that her dog special
can her move to music!
Accessible information for Kailey
Fact Question What the breed of dog?
/ Full Fact Answer Linda has golden retriever.
Limited Fact Answer There is no information on the breed Linda's dog.
Theory of Mind Questions
1 Introduction
Belief Question
Omniscient-view Belief
What bread would
Kalley belleves
Existing evaluations for language models' theory of mind (ToM) i.e.. the ability to understand the men-
Kailey think Linda's dog is?
Linda has golden retriever.
/ Kailey-centric Belief Kailey does not know the breed.
tal states (c.g., thoughts, beliefs, and intentions) of
Answerability Questions (about Fact Question)
others (Premack and Woodruff, 1978), is primarily focused using situation descriptions (i.e., nar- ratives) the target domain (Nematzadeh al., 2018; Le al., 2019; Sap et al., 2022; Shapira et al., 2023a). However, ToM capabilities play an even more important role in understanding dynamic so- cial interactions, as they form a crucial component of effective communication (Frith, 1994; Schober, 2005). Furthermore, as narratives condense situa- tion information into short texts, reporting biases cause them to include spurious correlations surface cues (Gordon and Van Durme, 2013). These can be exploited by large language models (LLMs) to display illusory ToM i.ee. a false sense of robust social reasoning by models.
the answer to this question?
A: Linda, David, Sally Q: Does David know the correct answer this question? A: Yes
Info Accessibility Questions (about the Full Fact Answer)
Who knows about this information? A: Linda, David, Sally
Q: Does Sally know about this information? A: Yes
Figure 1: An example question set in L FANTOM.
in interactions i.e.. conversations. As conversa- tions present interactions in their raw form, they are much less susceptible to reporting biases, and are more aligned with real-world scenarios requiring ToM reasoning. FANTOM consists of 10K ques- tions covering 256 multiparty conversations around a certain topic while characters enter and leave the discussion, leading to distinct mental states be- tween characters due to information asymmetry.
In this work, we introduce FANTOM, an En- glish benchmark for stress-testing machine ToM
'ht tps: /hyyn/.ywwww//m kim/fantom WWo do not believe that current L. possess an actual ToM. Please see $8 for further discussions.
The goal of FANTOM is to effectively measure how well models can track the belief of multiple

---

characters in conversations where some informa- tion may be inaccessible to some participants. For example, in Figure 1, Kailey briefly steps away from the conversation to get a cup of coffee, while the others continue discussing Linda's 's new dog. The information exchanged during Kailey's ab- sence remains unknown to Kailey, and only the information shared after Kailey's s return becomes accessible. We convert factual question-answer pairs obtain multiple challenging questions about characters' beliefs concerning the inaccessible in- formation. Our aim to design questions at differ- ent levels that evaluate a model's capability for a coherent understanding of others' mental states. In doing so, we are particularly interested in identify- ing instances of illusory ToM, which we define as situations where model may answer some ques- tions correctly but fails answer others that require the same type of ToM reasoning.
Design Considerations for S FANTOM
We go over the important design choices that we made when constructing FANTOM. Our goal is to incorporate (1) social interactions that necessi- tate natural theory of mind (ToM) reasoning ($2.1). (2) essential theoretical prerequisites for validating ToM from psychology ($2.2), and (3) empirical findings that must be taken into account when eval- uating large language models ($2.3).
2.1 Grounding in Social Interactions
To capture the interactive aspect of ToM, we ground our task in natural social interactions i..., conver- sations. By doing so, we gain two key benefits: (1) minimizing reporting bias (Gordon and Van Durme, 2013) and (2) aligning with real-world scenarios. Since narratives are condensed descriptions of in- teractions, the process of deciding what to include or exclude can introduce reporting bias, resulting in artifacts that models exploit. For instance, includ- ing "Carlos did not see this, so he does not know currently where the apple is." in a narrative for ToM evaluation provides a significant clue about the other's mental state. However, such explicit hints are rarely present in real-world interactions. Conversations, on the other hand, present inter- actions in their raw form, without those explicit hints about others' mental states. During conver- sations, we reason through the intermediate steps from scratch, thereby grounding the benchmark in conversations enables a more realistic and unbiased assessment of ToM.
The analysis of evaluation results on FANTOM reveals several interesting findings ($4): (1) First, existing neural models score significantly lower than humans on individual questions and on the full set of questions by more than 70% on aver- age. (2) While chain-of-thouuht reasoning (CoT) does improve performance in most models, it does not substantially bridge the gap with human per- formance. (3) Although our benchmark is not meant for training, we observe that fine-tuning can help models achieve scores higher than human per- formance on individual question types. However, when it comes to metrics that require coherent re- sponses across multiple question types, the fine- tuned model still significantly underperforms com- pared to humans. (4) Additionally, we find that models exhibit different error types depending on the format of questions, despite all questions requir- ing the same underlying reasoning. (5) Moreover, our results indicate that CoT has selective im- pact on performance, showing improvement only in specific scenarios.
2.2 Meeting Theoretic Requirements
We follow the two important criteria outlined by Quesque and Rossetti (2020) that must be met when designing a task to validate ToM: "non- merging" and "mentalizing".
(1) "Non-merging": Evaluation should require the respondent to maintain distinction between the others' mental state and its own. For exam- ple, suppose somcone is asked about the other's belief regarding the location of the TV remote con- troller, and both are believing it to be on the sofa. If the respondent answers that the other believes it is on the sofa, it becomes unclear whether the response is based on the respondent's own belief or the other's (i.e., merging mental states). Such merging scenario is unsuitable for validating ToM. Since machines lack emotions or intentions (Gros et al., 2022), we exploit information asymme-
To the best of our knowledge, FANTOM is the first benchmark to introduce conversation -based ToM evaluation for iiguaae--bseed models. Our benchmark design and experiment results yield important insights into the debate around ToM (Whang, 2023) and the development of artificial general intelligence (Metz, 2023) in LLMss. We release our benchmark to spark further discussions on evaluating the ToM capabilities of LLMs.

---

try when constructing our benchmark to simulate the mon-merging mental state scenarios. We design multiparty conversations where specific informa- tion is inaccessible to certain characters. While machines do not possess their own point of view, they act as omniscient observers during our evalu- ation since we provide the entire conversation as imput. As result, the mental states of the model and the character can be regarded distinct with respect that information.
3
/ FANTOM Overview
Following the success of previous works (Kim et al., 2022; Chen et al., 2023), we automati- cally construct full conversations using the large language model (LLM) InstructGPT davinci-003 (Ouyang et al., 2022). We also generate theory of mind (ToM) question-answer pairs related to the conversation participants' beliefs using a specially designed pipeline. In preliminary explorations, we find off-the-shelff LLMs struggle with directly gen- erating ToM question-answer pairs for given con- versation. Our pipeline consists of three steps: (1) generate conversations with information asymme- try ($3.1), 1), (2) generate fact question-answwr (QA) pairs ($3.2), and (3) construct ToM (e.g., belief) QA pairs from the fact QA pairs ($3.3). We use dif- ferent evaluation methods for each question types ($3.4), and validate the final dataset ($3.5).
(2) " "Mentalizing": Lower-level processes should not be accounted for successful performance of ToM tasks. If a simpler process can explain phe- nomenon, it should always be preferred over more complex one when interpreting the results. For instance, recognizing joy by observing laughter is more of visual discrimination than reasoning mental representations.
If the correct answer for a ToM task has a high degree of word correlation with a salient part of the given input, it becomes difficult to de- termine whether the model is accurately ascrib- ing the other's mental state or simply following a shortcut pattern matching (i.e., the lower-level process). Therefore, such cases should be discour- aged when evaluating ToM in neural language mod- els. In FANTOM, we create false answers that have high word correlation with the input to ver- ify whether the models can overcome the shortcut pattern matching when reasoning mental states.
3.1 Information-Asymmerrii Conversations FANTOM consists of small talk conversations in- volving multiple characters, with each conversation centered around a topic (e-g., pets, risk-taking, per- sonal growth). Each topic has several subtopics, e.g. the topic "pets" may include subtopics "breed" and " "special moves". Initially, the conversation begins with two or three characters. As the conversation progresses, characters join and leave the discussion and the conversation's 's subtopic changes over time. Conversations include explicit indications of leav- ing and joining, such utterances like "Hey guys, I'll go grab a coffee."
2.3 Seeking Comprehensive Evaluation
"Hey, I'm back, what are you guys discussing now?" shown in Figure 1. During the absence of a character, the conversa- tion continues and information is shared among the remaining participants, creating a natural informa- tion asymmetry that reflects real-life interactions. After a series of utterances, the character who was absent (re)joins the conversation, unaware of the information that was previously shared with other participants. More details are Appendix A.I.
Since the performance of LLMs varies significantly based on given prompts (Webson and Pavlick, 2022), we adopt series of reiterative questions at various levels for the same input context, includ- ing free-form response questions, multiple-choice questions, and straightforward yes or no questions. The inclusion of free-form response questions is important as it aligns with the common usage of LLMs in contrast to multiple-choice questions that are prevalent in existing benchmarks (Sakaguchi et al., 2021; Hendrycks et al., 2021). Although their formats are different, all questions in FANTOM fundamentally aim to ascertain the same underlying reasoning: "who is aware of the information?" ?" As a result, FANTOM enables us to identify illusory ToM instances wherein models deliver accurate re- sponses for one format but struggles to do so for another format.
Many existing ToM tasks involve some form of asymmetry between characters (Braüner et al., 2020). For example, in the Sally-Anne task, Sally does not know that Anne relocated the object, while the observer is aware of the action. In the Smarties task, the character in the story does not know the label changed, whereas the observer is fully aware of this situation. This inherent asymmetry ensures two distinct mental states (i.c., the non non-meree mor-megging cri- terion; $2.2) to be present during the experiments.

---

3.2 Factual Question-Answer (QA) Pairs
"OMNISCIENT-VIEW BELIEF A" and "PERSONX- CENTRIC BELIET A", respectively.
The conversations in FANTOM include factual question- answer pairs (FACTQ) about the inacces- sible information-i.e. the information that a spe- cific character is unaware of. An example question would be "What is the breed of Linda's 's dog?" in Figure 1. More details are in Appendix A.2.
-eastion-answerrr
(3) ANSWERABILITY QILISTI: Given the FACTQ, we ask models "List all the characters who know the correct answer to this question". In essence, we are interested in whether the model can identify who among the participants can correctly answer the FACTQ. This is a meta-question that necessitates two-step reasoning: first determining the answer itself, and second, identifying the characters who have access to this knowledge.
There are two distinct types of answers for each FACTQ: (1) FULL FACT A and (2) LIMITED FACT A. The FULL FACT A incorporates the full infor- mation in the preceding conversation where the character PersonX was absent. On the other hand, LIMITED FACT A relies only on the conversation in which PersonX participated. The former answer is based on information that PersonX does not have access to, while the latter answer only takes into ac- count the accessible information for PersonX. For cases where no information was shared regarding the FACTQ, the LIMITED FACT A indicates that no information has been provided. Examples of these answers are in Figure 1. "Linda has a golden retriever." is based on the preceding conversation where Kailey was absent, while "There is no infor- mation on the breed of Linda's 's dog" is based the conversation Kailey participated.
(4) INFOACCESS Q[LIST]: Here, we provide the FULL FACT A with the FACTQ and ask the model "List all the characters who know this information". Essentially, this question aims to identify the in- dividuals who have knowledge or access this information. Since the information is explicitly provided the model, only the second reasoning step of the ANSWERABILITY Q[LIST] is required.
(5) ANSWERABILITY Q(Y/N] and (6) INFOAC- CESS Q[vNJ= We ask models to determine, through simple binary response (yes or no), whether each character is capable of answering the question or knows the information. For exam- ple, we ask models "Does David know the correct answer to this question?" and "Does Sally know about this information?" (Figure 1).
3.3 ToM QAs based on Fact QAs
For each FACTQ, we build six types of ToM QA. Construction details can be found in Appendix A.3.
(1) BELIEFQIDIST.J and (2) BELIEF [ [Chorcej* These questions are created by rephrasing the FACTQ ask beliefs of characters in the conver- sation. We are particularly interested in PersonX's belief about the inaccessible information from the previous conversation, in which PersonX did not participate. For example, the FACTQ of "What is the breed of Linda's 's dog?" in Figure is converted to "What breed would Kailey think Linda's 's dog is? The (ELIEFQIDSTT. requires free-form response, while [HLIIE((CHOICE] provides multiple-choice options for the same question.
3.4 Evaluation
Each question is provided to the model along with the conversation as input. This makes the model an omniscient observer, having access all infor- mation shared in the conversation. On the other hand, PersonX was absent for a while, thereby an information asymmetry naturally arises between the model and PersonX. Responses that include in- accessible information for PersonX indicate a lack of ToM in the model.
Input context types FANTOM comprises two types of input conversations: short and full. In the case of short input, the model is provided with the conversation that only includes the part where the specific speaker left and (re)joined, while excluding the other earlier and later parts of the conversation. On the other hand, a full conversa- tion encompasses the entire discussion on the main topic, including all subtopics. As a result, this is significantly longer than the short input.
The options for BELIEFQ are created by rephras- ing the FULL FACT A and LIMITED FACT A. For example, the "Linda has a golden retriever" in Figure 1 is converted to "Kailey believes Linda has a golden retriever." Since the FULL FACT A re- flects information that is not accessible to PersonX and the LIMITED FACT A incorporates only the information accessible to PersonX, we label the converted FULL FACT A and LIMITED FACT as

---

BELIEF (IDIS.] When given a belief question regarding PersonX, the model should generate a response that incorporates only the information accessible to PersonX. We use cosine similarity to measure the distance between SentenceB (Reimers and Gurevych, 2019) embeddings of cach option and response. A correct response should al- ways be closer to the PERSONX-CENTI A than the OMNISCIENT-VIEW BELIEF A.
refine all 10 conversations. As for safety, no con- versations were voted as being unsafe. We also request workers to verify the answers provided for BELIEFQ(CHOOCEJS. We remove all question sets that were marked as erroneous by the worker (~~8.6%).
Statisties FANTOM is composed of 256 conversations with 1,415 BELIEFQIDIST.S and BELLIEF[CCHNCEJS. 703 FACTQS, ANSWERABIL- ITY QILISTIS, and INFOACCESS QILISTIS, respec- tively. Additionally, there are 2,689 ANSWERABIL- ITY QLY/NIS and INFOACCESS YINNS.. Given that the ANSWERABILITY QLY/NJS and INFOACCESS Q(Y/NJS iterate over all characters present in the conversations, they have the highest count among all the question types.
BELIEF
To accurately assess the performance of the re- sponse, we also calculate the token Fl score for responses that are considered correct based on the distance metric, following the convention of vari- ous QA tasks (Rajpurkar& et al., 2016, 2018). When comparing distances in the embedding space, non- sensical responses (c.g., repetition of character names) can be deceptively closer to PERSONX- CENTRIC BELIEF A, resulting in mislcading accu- racy. Therefore, models must score high on both the distance and Fl metrics for the BELIEFQIDIST.)
The average number of turns in the input context is 13.8 (short conversation), and the average num- ber of words in each turn is 1..9. For reference, the corresponding statistics for ToMi (Le al., 2019) are 4.9 and 4.2, respectively. More statisties can be found Appendix A.5.
BELIEFQ [CHOICE] The model should choose be- tween the OMNISCIENT-VIEW BELIEF and the PERSONX-CENTRIC BELIEF A. The correct an- swer is the PERSONX-CENTRES BELIEF A.
4 Experiments
ANSWERABILITY Q[LIST] and INFOACCESS Q[LIST] correct response must include all char- acters who have access the answer or informa- tion while excluding all characters who do not. No partial marks are assigned.
We test a total of thirteen recent instruction-tuned neural language models: GPT-4 (gpt-4-0613 and gpt-4-0314; OpenAI, 2023), Chat- GPT (gpt-3.5-turro-06613 OpenAl, 2022), Instruct- GPT (davinci-003 and curie-001: Ouyang et al., 2022). Flan-T5-XL and Flan-T5-XXL (Chung et al., 2022), Fl2 (Tay et al., 2023), Falcon Instruct (7B and 40B; Almazrouei et al., 2023), Mistral Instruct 7B (Jiang et al., 2023), Zephyr 7B (HuggingFace, 2023), and Llama-2 Chat 70B (Tou- vron al., 2023). Descriptions for each model are in Appendix B.
Baseline Models
ANSWERABILITY QLY/NJ and INFOACCESS
Q(Y/N] The model should respond with "yes" or "true" for all characters who have access to the an- swer or information, and with "no" or "false" for all characters who do not. More details are in Ap- pendix A.4.
Although our benchmark not meant for train- ing, we also fine-tune Flan-T5-XL (Chung et al., 2022) by randomly splitting FANTOM according to the conversation's 's main topics. We then test the model on unseen conversation topics. More details can be found in Appendix B.
3.5 Dataset Validation & Statistics
Validation To ensure the quality of our bench- mark, we go through a manual validation process for all conversations and question-ansen ing Amazon Mechanical Turk (MTurk). We con- duct validation on the entire conversations in our dataset using 32 annotators who passed qualifi- cation test for assessing conversation coherence. We ask workers to flag conversations that are inco- herent or unsafe (e.g., unethical, biased, harmful, dangerous, or offensive). Each conversation is val- idated by three workers. While 10 conversations received votes for incoherence. none achieved a majority vote indicating they were incoherent. We
pairs us-
Human Performance We also measure human performance by asking graduate students in com- puter science. We ask BELIEF Q(CHOCCE) ANSWER- ABILITY Q[LIST], and INFOACCESS Q given conversation. As is redundant to ask human tes- tees binary questions when they have already been asked ANSWERABILIT Q(LIST) and INFOCCEESS QL we do not ask ANSWERABILITY Q

---

37.5
90.6
90.6
Human
80
0613 [Cenbber
$4.3
(June]
40
20
4.1
12.3
6.1
0.3
0
0.1
3844
27.6
36.337.8
40
25.3
20.3
21.9
43.9
17:1
10.8
27.5
CherGPT
Falcon instrcct 40B
Mistral Instract 7B
All Cuestio Types
[chiceQ[Choice]
Antwenailin(
Info AccessQ [Lis]]
Figure 2: Results of (ELIEFQ(CHOCC)). ANSWERABILITY QuAST and INFOACCESS Quuis)), given the short conversation context. Full results with all models, input types, and metrics
Table 9.
and INFOACCESS QEY/NJ- To ensure a fair compar- ison with the models, we give the same instructions to humans and no other tutorials, examples, or extra instructions were given. Student volunteers solved 32 sets in total.
All Question Types
All AnswerabilityQ: [List Y/N]
Model
InfoAccessss [L + Y/N]
Hurnan
87.5
90.6
90.6
Mistral Instruct CoT
0.1
2.4
9.1
0.0
Llama-2 Chat Col
0.4
ChaGPT 0613 CoT
3.7
20.7
17.1
Metries We report accuracy for BELIEFQDDSTT)) BELIEEQ[CCHOCC)) ANSWERABILITY Q[LIST]= and INFOACCESS QILISTI- The weighted Fl scores are reported for ANSWERABILITY Q(YNN] and IN- FOACCESS QIYINI- We additionally report the "All" score for the ANSWERABILITY Q and IN- FOACCESS Q requiring models to be correct on both list-type and binary-type questions. For BELIEFQ and FACTQ, also report the to- ken Fl scores measure the word overlap between the answer and model's free-form response.
GPT-4 0613 CoT (Jun)
26.6
40.2
57.7
GPT-4 0613 CoT (Oct)
14.8
31.4
41.1
Flan-TS XI. FT
53.7
55.9
54.4
Table 1: Results of models with zero-shot chain-of- thought (CoT) and fine-tuning (FT) for the short conver- sation context. Full results with all models, input types, and metrics are in Table
models' performance sharply drops when evalu- ated for coherent reasoning across multiple ques- tion types with the same underlying theory of mind (ToM) reasoning (i.e., All Question Types). These findings suggest that some instances of success- ful LLM ToM reasoning in FANTOM should be interpreted as illusory.
Moreover, we report the ALL score which re- quires the models to answer all six ToM question types ($3.3) in the set correctly for the same in- formation piece in the conversation. This metric aims measure how well the models show consis- tent understanding across different types of ques- tions. To compare with human performance, we also report the ALL score, which only excludes the B BELIEFQIDIST.) from the ALL* score.
Chain-of- and Fine-tuning Table sum- marizes the results when we apply zero-shot chain- of-thought (CoT) reasoning or fine-tuning to mod- els. For CoT, we follow Kojima et al. (2022) and use the prompt "let's think step by step". We ob- serve an improvement in scores with CoT applied. However, there are still significant score gaps com- pared to human performance.
4.1 Results
All the models exhibit scores that are significantly worse than human performance. Table 9 shows the full results of state-of-the-art large language models (LLMs) on _ FANTOM. We break down the table and highlight each discussion point below.
We also find fine-tuned Flan-T5 XL still falls short of human performance in metrics that demand consistent accuracy across multiple questions-ice. the All scores.3 Although our benchmark is not intended for training purposes, developing models with a coherent ToM reasoning remains challeng- ing, even with explicit training on the data.
Illusory Theory of Mind Figure 2 shows the re- sults of a few selected models. We find models perform significantly better (CHOCE)(CHOICN) compared to ANSWERABILITY Q[LIST] and IN- POACCESS Q[LIST)- Despite the ANSWERABIL- ITY QiLISTI and INFOACCESS QILISTI being pre- requisites for solving BELIEFQICHOICE) they are much more challenging for models. Furthermore,
Comprehending Facts vs. Distinct Beliefs Fig- ure shows the token Fl scores for FACTQ and ac-
*We find fine-tuning achieves scores comparable with hu- man performance on individual question types (see Table 9).

---

Model
AnswerabllitQs [Y/]]
InfaAccessQs [Y/N]
FactQ2 Belieff2 [Dist.
62,9/6.7
Mistral Instruct 7B
61,5
70.4
59.8
60.9
56.6
40B
72.2
50
1.ama-2 Chat 708
61.4
80.4
29.3
26.2
165
24.6
26.2
ChaGGT
InstrucGGPT davinci-003
67.0
78.4
73.2
GPT-4 0314
64.0
76.3
GPT-4 0613 (June)
85,9
90.3
CheciPT
InstruntGH
Llama-2
Falcon Instrct
Matral Instrct
0613 (October)
(613))
0314
0613
devinc--088
Chat
Figure 3: Results of FACTQ and (ELIEFQ/DIST.) for models given the short conversation context. Full results with all models, input types, and metrics are in Table 9.
Table 2: Results of ANSWERABILITY QIYNI and IN- FOACCESS QIYNN when given the short conversation context. Full results with all models, input types, and metrics are in Table 9.
curacy for BELIIEQ(DIST... The token Fl scores for FACTQ can be seen measure of model's basic comprehension capability for interactions. Scor- ing high in FACTQ indicates the model good at identifying the most relevant information piece to answering the question. Despite its small size, Mis- tral Instruct 7B shows the strongest performance among the open-source models.
Reasoning Complexity Table 2 compares mod- cls' performance between ANSWERABILITY QIY/NI and INFOACCESS Q(Y/N)- As ANSWER- ABILITY Qs require an additional step of reasoning compared to INFOACCESS Qs, models consistently perform worse on ANSWERABILITY QLY/N] com- pared to INFOACCESS QIYN]. However, this pat- tern is not consistent across models for ANSWER- ABILITY Q[LIST] and INFOACCESS Q (see Figure 2). This may be because models signifi- cantly struggle with ANSWERABILITY QiLıSTI and INFOACCESS QILISTI, potentially resulting in the absence of meaningful performance patterns.
On the other hand, BELIEI aims to mea- sure model's understanding of individual char- acters' perspective of particular information i.e., belief. To meet the mentalizing criterion (sce $2.2), we deliberately design the incorrect an- swers in BELIEFQ(DIST.) to have greater word over- lap with the context than correct answers. Also, BELIFFQDDIST. are rephrased questions inquiring about PersonX's belief for the facts in FACTQ, thereby the two question types share significant word overlap. However, the same information that was used to answer FACTQ should not be included the response for DELIEFQ/DIST. on PersonX as it is from the conversation that PersonX missed. As a result, certain models with higher token F1 scores for FACTQ have lower scores for BELIEF [IST.] compared to models that perform worse on FACTQ (e.g., InstructGPT davinci-003 vs. Llama-2 Chat and Mistral Instruct). This suggests the models lack the ability to comprehend distinct perspectives of individual characters, leading them to reproduce similar responses to FACTQ for BELIEFQIDIST..)
Short vs. Full Conversations When a model is provided with the full conversation (Table 9, bot- tom), its performance noticeably decreases com- pared when it is given only the relevant parts of the conversation (Table top). The decrease can be attributed to the model's need to identify the relevant information within the full conversa- tion, whereas it does not have to do so for the short conversations. This indicates theory of mind rea- soning becomes even more challenging for models when it needs to be combined with different types of reasoning (e.g., search).
4.2 In-depth Analysis
What types of errors do models make? Figure and 5 summarize the error types of ANSWER- ABILITY Q and INFOACCESS Q for each model with and without chain-of-thought (CoT) reason- ing. For list-type questions, models make more errors by including characters who are unaware of the information in the responses, rather than cluding characters who are aware. Interestingly, when CoT is applied, the error of including un- aware characters decreases, whereas the error of excluding characters who are aware increases for most models.
Free- Response vs. Choice We observe a pat- tern where models score significantly worse in free-response questions than choice questions (BELIEFQ(DIST.) vs. BELIEFQ(CHOCEE) Figure? and 2).4 However, many of them still achieve scores either below or around 50, which is the ran- dom baseline for those binary choice questions.
"This pattern is consistent for ANSWERABILITY Q and INSWERABILIV QIyNN, as well as for INFOACCESS Qiuisn and INFOACCESS QIYNN (see Table 9).

---

Mistral Inatruct
ALL FOR EACH CHARACTER
Mistral instruct Col
Model
Answer Consistency
Faloon
Instruct
Mistral Instruct
27.8
45.1
LJama 2 Chat
Mistral Instruct CoT
26.9
41.9
U Chat CoT
Falcon Instruct 40B
10.7
19.1
InstruciPPT InstrucGPT davinci 003 CoT
Falcon Instruct 40B + CoT
16.9
27.4
Llama-2 Chat 70B
27.1
43.3
CharGPT 0613
Llama Chat 70B CoT
15.2
24.3
ChatGPT 0613 CoT
InstructGPT davinci-003 InstructGPT davinci-003 + CoT
GPT-4 GPT-4 0613 CoT (June)
33.2
58.4
CharGPT 0613
35.0
51.6
GPT.4 0613 [Detolowr)
CharGPT 0613 CoT
(Octolber)
GPT-4 (Junc) GPT-4 0613 + CoT (June)
66.8
100
200
300
400
300
600
700
59.2
73.4
Included Unsware Character
Did Excluded Character
GPT-4 0613 (Octoher)
48.7
62.2
GPT-4 0613 + Coll (October)
51.2
66.9
Figure 4: Analysis of model errors for ANSWERABIL- ITY QILISTI and INFOACCESS QILISTI-
Table 3: accuracy and consistency (%) mod-
els' responses for each character within the given versation context.
Mistral Instruct Mistral GoT
Faloon
Second-Order
Falcon Instruct 40B CoT
Overall
Cyclic
Acyelic
Ulama22 2 Chat 708 U Chat 706 CoT
Mistral Instract CsT
27.0
39.5
40,8
38.3
InstructGPT davindi-033 InstructPPT davind davind 003 CoT
Falcon Instrnet 408
39.3
42.6
Falcon Instruct CoT
67.9
76.3
75.5
77.2
CharGPT 0613
Laana-2 Chat
15.0
20.5
21.0
20.1
ChatGPT 0613 CoT
CoT
GPT-4 GPT-4 0613 CoT (June)
InstrucGPPT davinci-003
15.4
19.4
23.2
15.6
InstruciGPT davinci CoT
23.9
20.5
23.7
17.3
ChuGPT 0613 CharGPT 0613 CoT
21.2
31.1
31.2
30,9
GPT.44 0613 (Detolerr GPT 0613 CoT (October)
47.8
42.6
44.9
40.4
(Juee)
GPT-4 0613 CoTl (Junc)
65.9
67.6
69.1
66.0
False Pasitive
Falsa Nagative
Irrelacent Responss
0613 CaT (Oktoher)
62.6
Figure 5: Analysis of model errors for A ANSWERABII- ITY Q(YINI and INFOACCESS QLY/N]-
Table 4: BELIEFQ results for first and second order ToM beliefs.
In the case of binary questions, false positives and false negatives correspond to including char- acters who unaware and excluding characters who are aware in the response for list-type ques- tions, respectively. If the model fails generate yes or no response, we mark as irrelevant. Mod- els tend to exhibit false negative responses more frequently for binary questions compared to list- type questions. Similarly, CoT primarily helps the model in reducing the false positive error rates, but the reduction in false negative error rates is not consistent across models. This suggests that CoT selectively improves reasoning specifically for determining characters who are unaware of the information, rather than characters who are aware.
to answer all six types of ToM questions correctly regarding the specific character. For consistency, measure the ratio of consistent model responses across ANSWERABILITY Q and InfoAccess for each character. Table 3 shows the accuracy and consisteney of the models' responses for each character within the given conversation context. Overall, we observe a pattern where models that score low in accuracy also show low consistency. While CoT generally improves model performance (see Table 9), find that it does not always lead to improved accuracy and consistency. The decrease in ALL FOR EACH CHARACTER score when CoT applied suggests that CoT has a selective impact on different question types.
How accurate and consistent are models' swers for a given character? For accuracy, we report the ALL FOR EACH CHARACTER score which determined by whether the models are able
Are there differences in performance in terms of the order ToM beliefs? Table 4 presents the results of BELIEFQ with respect to different

---

orders of ToM beliefs. Similar to Le et al. (2019), models perform better on the second-order belief questions than those with first-order beliefs. To fur- ther investigate the performance on second-order belief questions, we analyze the results based on the cyclic and acyclic patterns in them. The cyclic second-order belief questions inquire about Char- acter I's belief regarding Character 2's belief about Character (e.g., What does Linda think about Kai- ley's belief on the breed of Linda's 's dog?): while the acyclic second-order questions focus on Char- acter I's belief about Character 2's belief regarding Character 3 (e.g., What does David think about Kai- ley's belief on the breed of Linda's 's dog?). Models show better performance on the cyclic questions than acyclic ones, which include more characters to track. However, when CoT is applied, the increase in score for acyclic questions is greater than that of cyclic ones, suggesting CoT helps multi-tracking.
upon essential theoretical requisites and empirical considerations required for validating ToM in large language models (LLMs). The conversations in our benchmark involve information asymmetry, with characters joining and leaving the discussion while it continues, to simulate distinct mental states. To identify illusory ToM, we crafted multiple types of challenging belief questions regarding the con- versation participants' mental states by convert- ing factual questions. Our evaluation results show that coherent ToM reasoning is challenging for cur- rent LLMs, performing significantly worse than humans even when using thain-of-thougee of-thought reason- ing or fine-tuning.
Although there has been recent debates around whether current LLMs possess ToM capabilities or not (Whang, 2023), our results indicate that this capacity has not yet emerged in any manner. Previ- ous instances of success on well-known psychology ToM tests may be attributed to exposure during the pretraining phase (Ullman, 2023). Our work high- lights the need for novel interaction-oriente bench- marks that introduce scenarios not encountered dur- ing training, and also aligning more closely with real-world use cases as LLMs are increasingly be- ing deployed in interactive settings.
5 Related Work
Existing Theory of Mind Benchmarks Many theory of mind (ToM) benchmarks, inspired by the false belief test from psychology (Wimmer and Perner, 1983), evaluate models on reasoning beliefs about object locations with narratives (Grant et al., 2017: Nematzadeh et al., 20:8; Le et al., 2019). Other works such as Shapira et al. (2023b) build benchmarks based on the Faux Pas Test (Baron- Cohen et al., 1999). Also, ToM-related benchmarks focus on reasoning emotions and mental states in narratives (Rashkin et al., 2018; Sap et al., 2019).
Our results also shed light on broader issue in neural models the lack of internal consistency (Elazar et al., 2021). We find they often fail to pro- vide consistent answers to questions requiring the same underlying ToM reasoning. To address this concern, future works can explore various direc- tions, such as grounding reasoning in pragmatics (Kim et al., 2020), visual information (Bisk et al., 2020), or belief graphs (Sclar al., 2023).
Theory of Mind in Large Language Models Al- though qualitative assessments might imply a de- gree of ToM in large language models (LLMs; Whang, 2023), more comprehensive quantitative investigations reveal that they have yet to achieve human-level ToM across various benchmarks (Sap et al., 2022; Shapira et al., 2023a). LLMs struggle to reason ToM robustly (Ullman, 2023), though their performance can be improved through few- shot samples and chain-of-thought prompting (Sap et al., 2022; Moghaddam and Honey, 2023) as well as specific inference methods (Sclar et 2023).
Another issue that our work touches upon is the reporting biases inherent in language models. We observed that models often exhibit biases in their responses, showing a tendency to overly rely on the information they are conditioned on, such as prefer- ring answers that have high overlap with the context (Sugawara et al., 2018). However, to achieve suc- cessful ToM reasoning, it is crucial to distinguish between accessible and inaccessible information for a particular agent, rather than blindly using all information available the model. One potential approach to mitigate this is to combine pretraining with interactive learning (Sap et al., 2022).
6 Conclusion & Discussion
We introduced [ FANTOM, a new benchmark for stress-testing theory of mind (ToM) capabil- ities of neural language models in conversations via question answering. Our benchmark is built
In the spirit of encouraging future research in this direction, we make our benchmark publicly available at https:/ / /hhpn//.wwwww///// hwt.:wiiffffff.. fantma.

---

7 Limitations
coherent ToM reasoning; instead, they primarily rely on word correlations.
Although FANTOM is the first benchmark, to the best of our knowledge, to cover theory of mind (ToM) reasoning in conversational interactions, it is currently limited to small talks on specific topics. Additionally, our benchmark only considers only single type of relationship between conversation participants, where they do not have prior knowl- edge of each other. However, social reasoning can become much more dynamic when variables such as relationships (e.g., family, friends, co-workers) are introduced. ToM is essential in all conversa- tional interactions, hence we strongly encourage future works to evaluate ToM in a wider range of diverse conversation scenarios.
Acknowledgement
We thank the participants who contributed to the human performance measurement. We also appre- ciate colleagues on the Beaker Team the Allen Institute for Al for helping with the compute infrastructure. This work was supported in part by DARPA MCS program through NIWC Pacific 1660001-1--240313 Hyunwoo Kim and Gunhee Kim are supported by the Institute of Information & communications Technology Planning & Eval- uation (ITPP) grant funded by the Korea govern- ment (MSIT) (No.2019-0-0106) SW StarLab; and N0.22022-0-01566 Fundamental research on con- tinual meta-learning for quality enhancement of casual videos and their 3D metaverse transforma- tion). Lastly, we also thank OpenAl, as well as Google Cloud Compute.
Our evaluation solely focuses on language-based models. However, it is important to note that ToM extends beyond a single modality (Piaget, 1956; Wu and Keysar, 2007). For instance, the well- known Sally-Anne test (ssttWiini (Wimmer and Perner, 1983; Baron-Coher et al., 1985) is typically conducted face-to-face experiment, where visual cues af- fect the performance of the participants. Therefore, interesting future work will involve examining the capabilities of multi-modal models in relation to ToM reasoning.
References
Ebtesam Almazrouei, Hamza Alobeidii, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-4OB: an open large language model with of performance.
Lastly, as we generate full conversations with large language models, conversations may contain offensive contents (Weidinger et al., 2021). How- ever, we specifically select casual topics for small talks (c.g.. pets, personal growth, traveling) to mini- mize the likelihood of offensive content generation. Also, we manually validate all conversations in our benchmark with crowdworkers from Amazon Mechanical Turk.
Simon Baron-Cohen, Alan M Leslie, and Uta Frith. 1985. Does the autistic child have a "theory of mind"? Cognition, 21(1):37-46.
Simon Baron-Cohen, Michelle O'riordan, Valerie Stone, Rosie Jones, and Kate Plaisted. 1999. Recognition of faux pas by normally developing children and children with asperger syndrome or fignttiuntiining autism. Journal autism and developmental disor- ders, 29(5):407-441
8 Societal and Ethical Considerations
We acknowledge that the term "theory of mind" (ToM) may evoke anthropomorphic connotations regarding Al models. However, we emphasize that the purpose of our work not to promote anthro- pomorphism of AI models. Rather, our focus lies in exploring the limitations of existing language models in social reasoning. While the concept of ToM attempts to capture the ability to attribute mental states to oneself and others (Premack and Woodruff, 1978), it is important to clarify that Al models do not possess subjective consciousness true understanding of intentions, beliefs, or de- sires. Our experiment results also demonstrate that current large language models do not exhibit any
Yonatan Bisk, Ari Holtzman. Jesse Thomason. Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lap- ata, Angeliki Lazaridou, Jonathan May. Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735. Online. Association for Computational Linguistics.
Torben Braüner, Patrick Blackbum,, and Irina Polyan- skaya. 2020. Being deceived: Information asym- metry in second-order false belief tasks. Topics in Cognitive Science, 12(2):004-534
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu, Zhou Yu, and Dilek Hakkani- Tur. 2023. PLACES:

---

Prompting language models for social conversation synthesis. In Findings of the Association for Com- putational Linguistics: EACI. 2023, pages 844-868, Dubrovnik, Croatia. Association for Computational Linguistics.
Hyunwoo Kim, Jack Hessel, Liwei Jiang. Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2022. Soda: Million-scale dialogue dis- tillation with social commonsense contextualization. ArXiv, abs/22 22 10465.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling cnrructionfinettune language models. arXiv preprini arXiv: 11416.
Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. 2020. Will sound like me? improving persona consistency in dialogues through pragmatic self- consciousness. In Proceedings of the 2020 Confer- ence on Empirical Methods Natural Language Processing (EMNLP), pages 904-916. Online. Asso- ciation for Computational Linguistics.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni. Guotong Xie. Zhiyuan Liu, and Maosong Sun. 2023. Ultraeeddacck: Boosting lan- guage models with high-quality feedback.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199 22213.
Ning Ding, Yulin Chen, Bokai Xu, Shengding Hu, Yu- jia Qin, Zhiyuan Liu, Maosong Sun, and Bowen Zhou, 2023. Ultrachat: A large-scale geto-generated multi-round dialogue data. htgp::///ittii com/ thunlp/ultren trachatt..
Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019. Revisiting the evaluation of theory mind through question answering. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 5872-5877, Hong Kong, China. Association for Computational Linguistics.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhi- lasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transac- tions of the Association for Computational Linguis- tics, 9:1012-1031
Cade Metz. 2023. Microsoft says new A.I. shows signs of human reasoning. The New York Times.
Uta Frith. 1994, Autism and theory of mind in everyday life. Social development, 3(2):188-122
Shima Rahimi Moghaddam and Christopher J Honey. 2023. Boosting theory-of-mind performance in large language models via prompting. arXiv preprint arXiv:2304.11490
Jonathan Gordon and Benjamin Van Durme. 2013. Re- porting bias and knowledge acquisition. In Proceed- ings of the 2013 workshop on Automated knowledge base construction, pages 25-30.
Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. 2018. Evaluating theory of mind in question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2392-2400, Brussels, Belgium. Association for Computational Linguistics.
Erin Grant, Aida Nematzadeh, and Thomas L Griffiths. 2017. How can meeorr--augmente neural networks pass a false-belief task? CogSci.
David Gros, Yu Li. and Zhou Yu. 2022. Robots-dont- cry: Understanding falsely anthropomorphic utter- ances in dialog systems. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 3266-3284, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
OpenAL. 2022. Chatgpt: Optimizing language models for dialogue.
OpenAL. 2023. Gpt-4 technical report. ArXiv,
abs/2303.08774.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Míshkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training Language Models to Follow Instructions with Human Feedback. arXiv preprint arXiv:2203.02155
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Con- ference on Learning Representations (ICLR).
HuggingFace. 2023. Zephyr 7b alpha.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023, Mistral 7b. arXiv preprint arXiv:2310.0683
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora

---

with web data, and web data only, arXiv preprint arXiv:2306.01116.
9th International Joint Conference on Natural Lan- guage Processing JEMNLP-JJCNLL pages 4463- 4473, Hong Kong, China. Association for Computa- tional Linguistics.
Jean Piaget. 1956. Child's Conception of Space. Rout- ledge.
Michael F Schober. 2005, Conceptual alignment in conversation. Other minds: How humans bridge the divide between self and others, pages 239-252.
David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 155::515-526.
Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. Minding lan- guage models' (lack of) theory of mind: A plug-and- play multi chulii-haraacter belief tracker. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 13960-13990 13980, Toronto, Canada, Association for Computational Linguistics.
François Quesque and Yves Rossetti, 2020. What do theory-of-ming tasks actually measure? theory and practice. Perspectives on Psychological Science, 15(2)388-336.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't 't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, Association for Computational Linguistics.
Natalic Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldherg, Maarten Sap, and Vered Shwartz. 2023a. Clever hans or neural theory of mind? stress testing social rea- soning large language models. arXiv preprint arXiv:2305. 14763.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 28922,92,, Austin, Texas. Association for Computational Linguistics.
Natalie Shapira, Guy Zwim, and Yoav Goldberg, 2023b. How well do large language models perform faux pas tests. In Findings of the Association for Compu- tational Linguistics: ACL 2023.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading compre- hension questions easier? In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4208-4219 Brussels, Belgium, Association for Computational Linguisties.
Hannah Rushkin, Antoine Bosselut, Maarten Sap, Kevin Knight, and Yejin Choi. 2018. Modeling naive psy- chology of characters in simple commonsense stories. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 2289-2299, Melbourne, Aus- tralia. Association for Computational Linguistics.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gar- cia, Jason Wei, Xuezhi Wang. Hyung Won Chung. Dara Bahri, Tal Schuster, Steven Zheng, et al. 2023. U12: Unifying language learning paradigms. In The Eleventh International Conference on Learning Rep- resentations.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter AL- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint XxXiv:2307.09280
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: An adver- sarial winograd schema challenge at scale. Commu- nications of the ACM, 64(9):99-100) 106.
Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXivv2302.00839)
Maarten Sap. Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022, Neural theory-of-mind? on the limits of social intelligence in large LMs. In Proceedings of the 2022 Conference on Empirical Methods in Nat- urul Language Processing, pages 3762-3780, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguistics.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin. Jonathan Uesato, Po-Sen Huang. Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh,

---

al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04333
[Conversation history) Ançleaving character):" We use predefined list of 64 reasons for leaving the conversation. Ta- ble 7 shows all reasons for leaving. We append the previous conversation history to the input prompt to make the conversation continue from the previous one.
conversation.
Oliver Whang. 2023. Can a machine know that we know what it knows? The New York Times.
Heinz Wimmer and Josef Perner 1983. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children's 's understanding of deception. Cognition, 13(1):103-128.
We the following prompt when a character is selected to join: "Now {joining character) comes back after leaving the conversation because of the reason (leaving reason}.
Shali Wu and Boaz Keysar. 2007. The effect of cul- ture on perspective taking. Psychological science, 18(7):600-606.
They now discuss (subtopic). Remember to indicate that {joining character} is joining the conversation. Do not mention the details in the previous conversations, {Conversation history) \n(joining character):".
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley, 2023. Baize: open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint 21Xhv:230.00196.
A
FANTOM Construction
Full examples of question sets in FANTOM can be found in Table 5 and Table 6.
Extracting the inaccessible information for Per- sonX Whenever a character (re)joins the conver- sation, we extract the inaccessible information by asking GPT-4 (gpt-4-0314) what information was shared in the preceding conversation where the character PersonX did not participate. We provide the previous conversation and the current one as in- put to GPT-4 with the prompt "What information was shared before PersonX joined, but was not mentioned after PersonX joined?" appended to it. To ease the task, the joining of the character is explicitly denoted by inserting a script between the conversations, as follows: "Previous conversation \n[PersonX joined the conversation]\nCurre
A.1 Generating Conversations with Information Asymmetry
informatiion-symmetii conversations To cre- ate the conversations in our benchmark, we use a predefined set of subtopies for each main topic and employ templates to generate scripts. For ex- ample, for the topic "pets" subtopics may include "breed", "special moves", and "favorite food". Fol- lowing Kim et al. (2022), we use specific speaker prefixes with English names sampled from the Top-1K names in the US SSN database for more natural conversations. We append each utterance with speaker prefixes. We randomly shuffle the subtopics for each topic and generate conversations for each subtopic. We generate the first conver- sation with the following prompt: "[Character (Character n}
conversation We observe quality improvements for the output generated by GPT-4 with the inclusion of the hint script. The returned result can be viewed as a versation summary explicitly covering the previous context.
1}, (Character 2), met for the first time at this social event. They are having conversation on their (topic). They now discuss (subtopic). \n{Character 13:" The initial con- versation starts with two or three characters and there can be up to five characters who are partici- pating in the conversation at the same time.
A.2 Generating Factual QA Pairs
We construct factual question-answer (QA) pairs related to the inaccessible information. First, we generate three non-yes-or-no questions and denote these as "FACTQS" and obtain them by prompting GPT-4, given the inaccessible infor- mation text. We obtain "FACTQS" by prompt- ing GPT-4 with the following: "(inaccessible Informattion)/nnnBaet
Then, for each subtopic, we randomly select characters to join or leave the conversation. We use the following prompt when character is se- lected to leave: "Now, (leaving character) leaves the conversation because of the reason '(leaving reason}'. They now discuss (subtopic). Remember to indicate that (leaving character) is leaving the
on this, formulate three non-yes-or-no questions that can be answered by this conversation summary."
Next, we generate two distinct types of answers for each FACTQ with GPT-4. (1) First, we gener-

---

A.3 Constructing Belief QAs with Factual QAs
Sabrina: So, what was the most challenging workout expe- rience you ever had?
Anna: Definitely when decided to try out CrossFit, I'm not going to lie, it kicked my butt!
BELIEFQIDISE and BELIEFQ[CHOOCE]] We first convert FACTQS into first-order or second-order ToM questions asking about beliefs of characters in the conversation. We are particularly inter- ested in PersonX's belief or knowledge about the inaccessible information from the previous con- versation, in which PersonX did not participate. We prompt GPT-4 with the following: "(FACTQ } An'AnCovertt this into a theory of mind ques tion asking (character name)'s 's belief about this."
Sabrina: Wow, that sounds intense. What kind of exercises did you do?
Anna: We did a lot of different things like high intensity interval training and Olympic lifting with barbells and dumb- bells.
Sabrina: That definitely takes dedication! How did you stay motivated during it?
Anna: It was tough but kept reminding myself why I wanted to get fit in the first place and that helped me stay focused on my goals.
Gina: Hey, I'm hack! What were you guys talking ahout?
Sabrina: We were just discussing our most challenging workout experiences. What do you think about when it comes to making a good workout playlist?
Next, we convert the FULL FACT As and LIM- ITED FACT As into answers about beliefs. Since the FULL FACT As reflect information that is not accessible PersonX and the LIMITED FACT A incorporates only the information accessible to PersonX, we label the converted FULL FACT A and LIMITED FACT A as "OMNISCIENT- VIWW BELIEF A" and OPERSONX-CENTRIC BELIEF A", respectively. For the conversion, we prompt GPT-4 with the following format: "Question: FACTQ \n\nAnswer the question using the
Anna: Music is really important when it comes to getting in the zone while working out. For me, I like upbeat and energetic songs that get me going.
Gina: Yeah, something with high tempo can really help push you during those tough workouts! also like adding in some of my favorite classic songs that give me extra motivation keep going.
FACTQ: In what ways did Anna motivate herself when faced with difficult workouts?
What does Gina believe are the ways in which Anna motivated herself when faced with difficult workouts? OMNISCIENT- NEW BELIEF A: Gina believes that Anna motivated herself by reminding herself why she wanted to get fit in the first place and staying focused on her goals. PERSONX FERSONX-CENTRIE CEETTRIC BELIEF A: Gina believes that Anna motivated herself by listening to upheat and energetic songs during her workouts.
BELIEFQ:
following sentence. (FULL FACT A or LIM- ITED FACT A AnAnswer:".
A.4 Evaluation for ANSWERABILITY Q(Y/N] and INFOACCESS QLY/N]
ANSWERABILITY QiLasi]: List the characters who know the correct answer to this question.
We use pattern matching to parse the yes or no an- swers from model responses. We regard "yes". " "knows", "does know", and "true" as responses representing "yes". Similarly, we regard "no", "does not know", "doesn't I know", and "false" as responses representing "no".
INFOACCESS QILIST List all the characters who know this
information.
Answer: Sabrina, Anna
ANSWERABILITY Q(vNj: Does Sabrina know the correct
answer this question?
INFOACCESS Qivnn: Daes Sabrina know this information?
Answer: Yes
A.5 Statistics for FANTOM
Table 5: A sample from FANTOM.
Table 8 compares the basic statistics of FANTOM and ToMi (Le et al., 2019).
ate an answer denoted as "FULL FACT A", which is based on the preceding conversation where Per- sonX was absent. This answer incorporates the full information by providing GPT-4 with the pre- vious conversation i.e., the source of the inac- cessible information for PersonX. (2) Second, we generate another answer referred to as "LIMITeD FACT A", which relies only the conversation where PersonX participated. In this case, conversation
B Experiments
Human performance evaluation A total of student volunteers participated in the evaluation. For each question set, we assign a single testee. They solved a total of 32 sets. To ensure fair comparison, no additional tutorials, examples, or extra instructions were provided beyond what was given to the models.
give GPT-4 the PersonX-participaal along with the FACTQ. We prompt GPT-4 with the following: "[context)\nnnnuestt } "nAnswwer:"
Baseline models The GPT models are propri- etary models from OpenAl based on the decoder- only transformer architecture. Flan-T's T5 and Flan-
(FACTQ

---

UL2 are open-source (i.e., Apache 2.0) mod- cls from Google trained on instruction-phraaed datasets. They are based on the encoder-decoder transformer architecture. Falcon Instruct is another open-source (i.e., Apache 2.0) model trained on RedefinedWeb (Penedo et al., 2023) and Baize (Xu et al., 2023). Llama-2 Chat (Touvron et al., 2023) is a fine-tuned 70B large language model, optimized for following user requests in dialogue format. Mis- tral Instruction (Jiang et al., 2023) is 7B language model fine-tuned to follow instructions, which is reported to surpass the Llama-2 Chat 13B model. Zephyr (HuggingFace, 2023) is a model based on Mistral, further fine-tuned on UltraChat (Ding et al.. 2023) and aligned with UltraFeedback (Cui et al., 2023).
Zachary: Have you guys thought about how much money you 'Il need for retirement?
Hazel: I'm still trying to figure that out. I know it's impor- tant to save, but it can be hard when there are so many other expenses.
Cory: Yeah, that's true. It's also gond to plan for how you want your lifestyle to look when you retire too. Will you travel or stay close to home?
Hazel: That's a tough one! think a combination of both might work best for me travel while we'ne re younger and then spend more time closer to family as we get older.
Zachary: Sounds like a great plan! What kind of invest- ments have cach of you made?
Cory: Well, I've started investing in some mutual funds and my 401(k). How ahout the two of you?
Hazel: I've been contributing regularly to my IRA and also investing in index funds. 7achary?
Zachary: Same here mutual funds and an IRA with occa- sional stock investments as well.
Hazel: Sorry guys, but my pet needs attention. I need to go
check on him.
Results of other models Table 9 shows the re- sults for other large language models not included in Figure 2. Given the random baseline score is 50 ANSWERABILITY QIY/NI-
Cory: Oh, okay. Have good one!
Hazel: See you later.
Cory: Well, now that Hazel is gone, we can talk about the psychology of retirement. What are some of the emotional aspects to consider when planning for retirement?
for BELJEFQQCHHOCEH and INFOACCESS Q(Y/N), most of the models show low performance on our benchmark.
Zachary: It can be hard to think about the future and envi- sion what life will be like retirement without having any prior experience with it. People also tend to underestimate how much money they'l 'Il need for comfortable lifestyle and overestimate their ability to save for it over time.
Fine-tuning details We fine-tune Flan-T5- XL with learning rate=2e-5 and weight decay=0. 01, evaluating per epoch and using early stopping with patience (batch size = 3 for Flan- TS-XL). We observe an increase in validation loss after the first epoch. We also add special tokens be- fore and after the completions to prevent the model from over-generating, which we find in early exper- iments. We also fine-tune text-curie-001 (Ouyang et al.. 2022) for two epochs using standard parame- ters from the OpenAl API.
Alec: Hey, I'm back! What were you guys talking about? Cory: We were discussing retirement planning. Zachary was just mentioning how difficult can be think about the future and envision what life will be like in retirement without having any prior experience with it. Alec: Yeah, that's true. It's important to save as much as possible now so you have enough money for retirement later. Have you guys started investing yet?
Zachary: Yes, we both have investments mutual funds and IRAs, and Cory has a 40)(k). How ahout you Alce? Alec: I've been saving some money into a savings account and I'm looking into investing in index funds soon. What other advice do you have for someone starting out with their retirement planning?
Cory: Do your research on different investment options available to figure out which ones are best for your goals. There are also some tax benefits associated with certain types of investments that could make them more attractive than others depending on your situation.
FACTQ: What kind of funds has Hazel been investing in?
BELIEFQ: What does Alec believe are the types of funds Hazel has been investing in?
OMNISCIENT-VIEW BELIEF A: Alec believes that Hazel
has been investing in index funds and contributing regularly to her IRA.
PERSONX: BIROONXCCENEEE E-CENTRIC BELIEF A: Alec does not know the types of funds Hazel has been investing in.
ANSWERABILITY QIaasr:: List all the characters who know the correct answer this question.
INFOACCESS Qiaasry: List all the charucters who know this information.
Answer: Hazel, Zachary, Cory
A ANSWERABILITY QiviN]: Does Alec know the correct answer to this question?
INFOACCESS QIvNN:: Does Alec know this information? Answer: No
Table 6: Another sample from FANTOM.

---

bathroom break
coffee break
forgot something important
forgot to print some documents
forgot to recieve package
forgot to return package
forgot run errands
forgot submit documents
have meeting starting soon that I need to prepure for have previous engagement that need attend quickly have werk-related emergency that requires my immediate attention
have unexpected visitor my door have errands have to attend to someone who just walked in have check on something have go the have pick up prescription have pick up dry cleaning have to print scan documents have receive delivery
to recharge laptop
to return borrowed item
to take care of family matter
to take care of an unexpected task unexpected visitor
his/her pet needs attention
his/her family is calling
incoming delivery
must respond to phone call
Arg. #Questions per Context
Avg.
Avg.
Avg.
need to check on friend or family member who needs assistance
Total #Questions
Dataset
#Turns (Partial)
#Tumss
Tum
(Full)
Length
need to finish a task that's time-sensitive
need to get a phone call
ToMi
6K 10K
6.0
4.9
need to get some coffce
FANTOM
12.9
13.8
24.5
21.9
need to go to the toilet need to grab a snack or drink
Table 8: Statistics of FANTOM and ToMi (Le et al., 2019).
need to have a quick chat with someone else need to make phone call need to make quick trip to the drug store need to make quick trip to the grocery store need to pick up package need to receive parcel
need to recharge cellphone
need to register for event
need to schedule haircut or salon appointment
need to schedule another appointment
need step away for moment stretch and clear my mind need to step for moment
need to submit some papers
need to take care of some paperwork documents
need to take care of some personal matters
need to take care of something related my health need to take
of something urgent
need to troubleshoot something parking expiring
remembered something needs to be taken care of rememhered receive package
remembered submit some papers
remembered care some paperwork documents remembered remembered
care some personal matters
care something urgent want to go grab drink
want to go grab coffee
want to go take some fresh air want to go to the bathroom
Table 7: Predefined reasons for characters leaving the conversation.

---

FACT QUESTIONS
ALL QUESTION
ALL QUESTION
Model
QUESTIONS
QUESTIONS
QUESTIONS
TYPES
Choice
Dist.
TokenF]
AlI
List
Y/N
All
List
Y/N
TokenFl
Human
87.5
93.8
90.6
90.6
90.6
90.6
Flan T5 XL
0.0
0.1
30.5
40.1
3.2
6.5
17.2
62.7
1.4
11.0
51.0
22.9
Flan-TS-XXI.
0.1
0.3
27.3
42.1
2.2
2.4
15.1
54.9
1.7
10.8
50.4
22.9
5.7
49.9
Mistral Instruct
0.0
27.6
26.2
50.8
2.4
28.3
61.5
9.1
27.5
70.4
56.6
Zophyz 7B
0,0
0.0
58.5
42.0
41.9
0.4
12.6
60.6
2.6
33.4
61.0
55.0
Falcon Instruct 7B
0.0
0.0
43.9
20.2
26.4
0.9
13.2
52.4
2.1
13.7
56.4
33.5
Falcon Instruct 40B
0.0
0.0
54.3
24.6
33.6
13.4
19.1
59.4
5.8
10.8
72.2
50.0
2.4
InstrucGPP curie-001
0.0
21.0
14.7
42.6
0.1
7.3
54.2
0.0
58.2
47.3
InstricGPP davinci-003
0.0
0.4
17.7
16.5
44.5
9.3
56.3
67.0
16.8
33.8
78.4
60.9
SunRt CONVERSTION
ChaiGPT 0613
0.0
53.5
26.2
50.8
3.1
40.0
64.2
13.3
43.9
73.2
59.8
GPT- 0314
77.6
GPT-4 0613 (Junc)
8.2
12.3
73.3
65.3
28.6
85.9
29.0
36.4
90.3
62.9
GPT-4 0613 (October)
2,4
4.1
68.4
56.1
44.6
16.9
36.3
75.7
17.9
21.9
91.5
64.9
Flan-T5 TS-XL, XI. CoT
0.0
0.0
43.0
26.4
15.4
0.9
9.2
57.1
1.6
8.4
65.3
21.5
Flan UL2 + CaT
0.0
0.0
24.7
32.4
7.3
1.1
10.2
57.3
0.3
59.4
15.6
Mistral Instruct
0.0
31.5
6.0
63.6
7.8
67.4
33,9
Zephyr 7B CoT
0.0
69.6
22.4
3.7
24.7
64.0
1.1
58.5
27.4
Falzon Instract 7B CoT
0.0
0.0
42.4
45.3
17.9
0.9
9,5
49.5
2.1
5.9
56.2
19.0
Falcon Instruct 40B CoT
0.0
0.0
$1.7
72.1
18.4
1.6
12.9
58.4
0.9
5.9
65.1
19.5
Llama-2 Chas 70B + CoT
0.0
58.5
31.5
19.3
6.0
26.8
63.6
7.8
28.3
67.00
33,9
curie
InstructGPT davinci- 003 CoT
1.3
34.8
22.2
41.6
9.3
49.4
80.4
16.8
42,9
86.6
49.9
ChatGPT 0613 CoT
2.1
3.7
$8.5
45.2
44.7
20.7
45.5
76.7
17.1
36.1
79.1
53.4
GPT-4 0314 CoT
1.0
2.8
39.0
31.2
36.8
21.9
38.
83.7
10.7
22.5
73.2
56.2
(June)
66.7
54.3
GPT4 0613 (October) CaT
6.8
14.8
74.7
55.0
40.0
31.4
40.4
86.6
41.1
46.4
91.3
52.8
InstrucGPPT curie 001 Flan T5 T5-XL FT
0.0
0.0
56.6
54.7
43.6
3.7
4.4
91.9
5.8
5.9
91.8
35.8
26.5
53.7
93.4
63.5
42.4
55.9
78.7
86.7
54.4
75.0
86.2
49.3
Flan T5 XL
0.0
0.0
3.8
38.2
4.7
0.0
5.8
11.1
0.3
4.4
9.9
8.7
Flan T5 XXL.
0.0
0.0
3.8
36.6
0.0
2.6
10.6
0.0
1.3
8.6
8.6
Flan- UL2
38.9
Mistral Instruct
0.0
25.7
25.0
51.3
1.6
25.8
55.8
5.8
19.4
64.9
53.5
Zephyr 7B
0.0
0.0
61.6
31.5
40.2
D.1
10.4
49.7
2.7
17.0
48.9
41.8
Falcon Instruct 7B
0.0
0.0
16.8
34.8
7.0
0.1
2.8
48.1
0.1
1,4
58.3
10.7
Falcon Instruct 40B
0.0
13.3
56.4
16.2
0.5
16.7
58.3
0.7
18.9
58.3
23,3
Llama-2 Chat 70B
0.0
37.1
1.9
52.8
2.2
69.2
InstructGPT curie-001
0.0
26.7
16.4
40.3
0.0
5.1
51.2
0.0
55.3
46.1
InstructPPT davinci-003
0.0
0.0
14.9
12.6
42.3
6.5
39.3
63.1
11.6
26.0
76.3
59.3
REL CONVERSATION
ChaiGPT 0613
0.0
48.4
30.8
50.4
1.7
30.8
56.7
7.1
39.3
69.7
59.3
GPT-4 (June)
53.5
30.6
GPT4 0613 (October)
0.9
46.0
44.4
8.0
69.0
14.8
23.2
85.1
62.8
Flau-TS mou--T5-XL 15-XL+ CoT
0.0
0.0
4.0
37.0
5.4
0.0
4.8
9.7
0.3
4.4
9.9
8.9
Flau-UL2 + COT
0.0
0.0
3.8
36.6
8.7
0.4
5.3
12.6
0.0
4.0
9.8
10.5
Mistral Instruct 7B CoT
0.0
58.4
31.5
19.3
6.0
26.8
63.6
7.8
28.2
67.0
33,9
Zephyr
46.4
Falcon Instruct 7B CoT
0.0
40.4
45.1
17.0
0.6
9.3
45.0
0.7
48.0
17.2
Falcon Instruct 40B CoT
0.0
0.0
40.5
66.1
18.7
0.9
11.0
49.0
0.4
6.2
55.3
19.0
Ilama-2 Chat 70R + CoT
0.0
0.0
53.6
28.6
20.8
2.3
20.1
56.9
4.0
21.1
63.9
32.2
0.3
53.0
38.1
InstrucUPPT davinci-003 CoT
20.6
6.5
78.9
83.7
47.1
ChaiGPT 0613 CoT
0.6
1.8
52.8
47.2
41.1
11.8
34.3
70.6
15.4
32.5
74.3
51.5
GPT4 0613 (June) CoT
10.1
15.4
70.1
61.2
43.9
29.6
42,0
86.1
45.6
53.3
91.0
$2.2
GPT-4 0613 (Oetober) CoT
0.9
60.9
46.0
44.4
8.0
31.7
69.0
14.8
23.2
85.1
62.8
InstrucGPPT curic-001 Flan-T5-XL++
0.0
0.0
55.1
53.7
43.6
0.7
0.7
88.3
0.1
87.3
25.4
21.3
47.1
62.4
49.3
87.2
87.2
50.2
Table 9: Zero-shot results from humans and large language models on CoT denotes ohain-of-thought reasoning and FT denotes fine-tuning.
1 FANTOM with same instructions.


Reviews:
[{"review_id": "CFnpI1ZFVf", "reviewer": "Reviewer_we1R", "summary": "", "questions": "- I am curious why the authors chose the current task setting of having participants explicitly leave and rejoin during conversations to measure information asymmetry. What difference does it have compared to say, masking some parts of the conversation when inputting into a model?\n- What is the total cost of constructing the dataset?\n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback that our benchmark is well designed and that it “can be a large importance to the LLM research community interested in studying its capabilities in conversation”!\n\n**1. The justification of the emphasis on information asymmetry:**\nThank you for bringing up this question! In psychology, many existing theory of mind tasks can be viewed as having some form of asymmetry between characters [1]. For example, in the Sally-Anne task, Sally does not know that Anne relocated the object, while the observer is aware of the action. In the Smarties task, the character in the story does not know the label changed, whereas the observer is fully aware of this situation. This inherent asymmetry ensures two distinct mental states to be present during the experiments. Therefore, information asymmetry is a key component when designing a theory of mind benchmark. We will clarify this in the updated draft.\n\n**2. The purpose of having participants explicitly leave and rejoin the conversation:**\nThank you for asking! By explicitly surfacing the absence of the character, it becomes clearer for the observer that the specific character does not have access to certain information. Many existing theory of mind tasks in psychology also include some form of explicit indication of absence (e.g., Sally-Anne task). This ensures a clear distinction in the mental states between the character and the observer (i.e.,  fulfilling the non-merging criterion in Section 2), while maintaining a natural and self-contained context. Without such explicit indication, the character's presence or absence becomes ambiguous, especially since our benchmark relies solely on text.\n\nMasking certain areas of the text violates the non-merging criterion which is the basic requirement of our task: if the part where the certain character was absent is masked when given to the model, the model will also lack access to the information, eliminating the information asymmetry.\n\n**3. Total cost of constructing the benchmark:**\nIt was less than 1000 USD.\n\n**4. More information needed in abstract:**\nWe will make sure to update the abstract to be more informative.\n\n[1] Braüner, Torben, Patrick Blackburn, and Irina Polyanskaya. \"Being Deceived: Information Asymmetry in Second‐Order False Belief Tasks.\" Topics in Cognitive Science 12.2 (2020): 504-534.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a benchmark for evaluating the theory-of-mind in conversation settings, which can be of large importance to the LLM research community that will be interested in studying its capabilities in conversations\n- The benchmark is well-designed, both in terms of underlying theory and coverage\n- The evaluation is conducted on a large set of LLMs and are able to provide insights on their capabilities in conversation settings", "reasons_to_reject": "- The emphasis on information-asymmetric conversation settings feels a bit arbitrary and is not justified well enough.", "questions_to_authors": "- I am curious why the authors chose the current task setting of having participants explicitly leave and rejoin during conversations to measure information asymmetry. What difference does it have compared to say, masking some parts of the conversation when inputting into a model?\n- What is the total cost of constructing the dataset?\n", "paper_topic_and_main_contributions": "The authors present FANToM, a benchmark for analyzing theory-of-mind capabilities of language models in conversation settings. This setting primarily focuses on information-asymmetric conversations where some individuals have partial information. They construct a benchmark using GPT-3.5 to create samples based on a set of rules, and perform a number of validations. They then evaluate the benchmark on several LLMs, from which they find that LLMs significantly underperform humans in the tasks, and that chain-of-thought reasoning improves performance but only to a certain extent in certain scenarios.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The abstract could be a bit more informative"}, "extra_text": {}}, {"review_id": "PapBKZBBZU", "reviewer": "Reviewer_o4fC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive endorsement!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Benchmark of language models on ToM.", "reasons_to_reject": "N/A", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a dataset, FanToM, to benchmark model performance on Theory of Mind (ToM).  It provides empirical results of language models on the dataset. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iPhVooxe0o", "reviewer": "Reviewer_iirp", "summary": "", "questions": "InfoAccess and Answerability seem to be very similar? As in, I would expect that a character who knows a given piece of information implies that it would know the correct answer to the question? Not sure what exactly the difference is here.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive acknowledgement of our work as a new interesting benchmark, as well as your recognition of our benchmark design with key ToM principles!\n\n**1. How this dataset, or its principles, can be used to make existing models better:**\nWhile we have discussed that applying chain-of-thought reasoning and fine-tuning can improve the vanilla performance, we will also discuss potential future directions for researchers to tackle this challenging benchmark better. For example, tracking mental states with belief graphs [1], structured prediction between question types to ensure consistency, and procedural prompting in the order of the easiest questions to the most difficult questions (e.g., InfoAccessibility questions to Answerability questions to Belief questions).\n\nAlso, we would like to gently note that the principles of our benchmark (e.g., information asymmetry, minimizing reporting bias, theoretical requirements) can be crucial for clarifying the recent illusion around the ToM capabilities of AI models. These principles also introduce underlying key components for social reasoning, such as understanding information access and asymmetry. As efforts increase in evaluating social reasoning of LLMs, it is important to establish elaborate benchmark designs early on. Thankfully, other reviewers such as 8JV5 and we1R have acknowledged this importance too. We believe our work can stimulate meaningful discussions within the research community.\n\n\n**2. Difference between InfoAccessibility and Answerability:**\nThank you for bringing up this point! You are right in saying that “if a character who knows a given piece of information implies that it would know the correct answer to the question.” This is a very obvious reasoning step for us humans. However, as we show in Table 1, this does not hold true for models, as the performance difference between the InfoAccessibility and Answerability question is significant. We designed these two questions precisely to tease out whether the models can follow this internally consistent reasoning process. We will expand on this important point in the discussion of the revised draft.\n\nTo clarify, the key difference between InfoAccessibility and Answerability questions lies in the depth of reasoning required. The InfoAccessibility question simply requires to reason who knows the given information piece (i.e., one-step reasoning). In contrast, the Answerability question requires (1) determining the answer of the question, and (2) reasoning which character has access to this information piece (i.e., two-step reasoning). \n\n[1] Melanie Sclar, et al. 2023. Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13960–13980", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "An interesting new asymmetric-information motivated theory of mind benchmark in the multiparty conversation setting. The authors source from key ToM principles in the construction and design of their dataset, and existing models are shown to perform badly on this even with significant chain of thought prompting. ", "reasons_to_reject": "I'm curious to see how this dataset, or its principles, can be used to make existing models better. It works as, and *only* as, an evaluation of the capabilities of existing models, but the paper leaves a lot to be desired on how we can build systems that can possess better ToM capabilities. Some experimental design on this front would have significantly improved the impact of the paper. ", "questions_to_authors": "InfoAccess and Answerability seem to be very similar? As in, I would expect that a character who knows a given piece of information implies that it would know the correct answer to the question? Not sure what exactly the difference is here.", "paper_topic_and_main_contributions": "The paper presents a new benchmark for testing theory of mind in multi-party conversations with asymmetric information. Specifically, in conversations, the benchmark tests the belief of characters (what does X believe), the answerability of questions (who knows the right answer to X), and which characters possess information (who has information on X). The dataset is comprised of 256 conversations with ~6k questions total, a mixture of the aforementioned types. Evaluation of existing models shows that they perform significantly worse compared to human performance. ", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VCJVldl2ry", "reviewer": "Reviewer_8JV5", "summary": "", "questions": "- Do you plan to release the benchmark and if so, under which license? What is the timeline for this? \n- Abstract: Please state the language that is investigated / benchmark focuses on in the abstract. \n- Intro + Related Work: Please add peer-reviewed papers of high quality as citations about the „debate around ToM“ instead of citing 2 New York Times articles only (one behind a paywall…)\n- l. 154f: You create „false answers that have high word correlation with the input to verify whether the models can overcome the shortcut pattern matching when reasoning mental states“. I might have missed it in the results/analysis section but as far I saw you only briefly touched upon lexical overlap in 517f in a different context. However, the community might be interested in an analysis of this and potential implications. \n- l. 409 + corresponding App: Could you please state how many question sets each of the annotators evaluated with information on whether some worked on substantially more than others? Why did you only ask 1 annotator per question set? Why use AMT and not a different way of evaluation where trust in annotator reliability is a little higher?\n- Results Table: \n    - Would it be possible to generate a results table where all models’ performances are included (this can go in the appendix, but I find it hard to compare performances across many models with 2 tables).\n    - Would it be possible to provide a type of indication of differences in model performance that are statistically significant?\n- l. 570, 588, etc.: Can you please elaborate on what is meant with CoT having „a selective impact on different questions types“? Do you have any intuition about why this could be the case?\n\n// edit: Thanks to the authors for their detailed responses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the appreciation of our benchmark to be “a novel and useful contribution” and our efforts for “embedding our work into the larger theoretical field of ToM”!\n\n**1. Release of dataset and codes:**\nOur dataset and codes are currently fully prepared to be publicly released upon acceptance, under CC-BY license. We are enthusiastic to share our dataset and code, so that other researchers can reproduce and build on our work!\n\n**2. Evaluating a closed, proprietary model on the benchmark that was generated from it:**\nThank you for that point! In light of the theme of this year’s EMNLP, we evaluate not only various open- and closed-source models but also GPT-4, to show that even the model that was used to generate the dataset is unable to solve it. Also, although it is a closed model, it is often seen as the most capable model and can thus serve as an “upper bound or existential proof that can be achieved with some unspecified combination of current methods and data” [1]. We will make sure to include the discussion around this issue in the limitations section of the updated draft.\n\n**3. Human validation and the quality of generated conversations:**\nWe conducted an additional round of validation on the entire conversations in our dataset using 32 annotators who passed a qualification test for assessing conversation coherency on Amazon Mechanical Turk. We asked workers to flag conversations that were incoherent or unsafe (e.g., unethical, biased, harmful, dangerous, or offensive). Each conversation was validated by three workers. While 10 conversations received votes for incoherency, none achieved a majority vote indicating they were incoherent. This is in line with previous works showing machine-generated conversations to have great quality [2, 3]. As for safety, no conversations were voted as being unsafe.\n\n\n**4. Moving up discussions of anthropomorphism:**\nGood point! We fully agree with you that it is important to clear any potential misunderstanding regarding anthropomorphism! We will move up the discussions about potential anthropomorphism to the introduction section, including the nature of the conversations in our benchmark. We will also indicate that these are \"machine-generated\" conversations, \"simulating\" human interactions, starting from the abstract.\n\n**5. Measuring human performance with workers from Amazon Mechanical Turk (AMT):**\nUnfortunately, AMT was our best option, as our institution has prohibited the use of Prolific due to data concerns. We would like to emphasize that we sought to have an identical setup to the models without introducing any separate qualifications or tutorials. Nevertheless, we recognize that this might have introduced some noise in measuring human performance. In our updated draft, we will set an additional human performance baseline by recruiting offline annotators (e.g., graduate students at our institution). However, we would like to gently remind that even the recent best performing models are still far below the current human baseline. For the present baseline, each worker solved an average of 12 question sets.\n\n**6. The selective impact of chain-of-thought (CoT) reasoning:**\n[L570] Results in Figure 2 & 3 show that CoT reduces false positive errors (i.e., saying yes for unaware characters for binary questions, and including them in the response for list type questions), while it increases false negative errors (e.g., saying no for characters who are aware of the info for binary questions, and excluding them in the response for list type questions).\n\n[L588] CoT decreases the likelihood of correctly answering all questions related to a specific character (i.e., 'All for Each Character' in Table 2) for some models, while improving the overall performance shown in Table 1. We will elaborate this in the updated draft.\n\n**7. Other comments:**\nThank you for the suggestions! We will reflect all your comments: mentioning base language, benchmark focus, additionally citing articles not behind paywalls, updating a full table with all model results in appendix, and adding statistical significance.\n\n[1] Anna Rogers 2023 [Closed AI Models Make Bad Baselines - Hacking semantics](https://towardsdatascience.com/closed-ai-models-make-bad-baselines-4bf6e47c9e6a)\n\n[2] Chen, Maximillian, et al. \"PLACES: Prompting Language Models for Social Conversation Synthesis.\" Findings of the Association for Computational Linguistics: EACL 2023. 2023.\n\n[3] Kim, Hyunwoo, et al. \"Soda: Million-scale dialogue distillation with social commonsense contextualization.\" arXiv preprint arXiv:2212.10465 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I like the idea to extend the evaluation of LLMs regarding ToM to conversation-like text. In my opinion, such a benchmark can benefit the research community and is likely to be leveraged be researchers working on ToM in addition to previous benchmarks. I thus think the benchmark is a novel and useful contribution. \n\nConstructing the benchmark using an LLM  is an interesting research question in itself (which, however, also comes with some caveats, see Reasons to Reject) and a potential use case of the benchmark from a meta perspective. It can further be acknowledged that the authors seek to embed their work into the larger theoretical field of ToM and try to address relevant points when developing main design criteria. \n\nThe authors provide a first evaluation of their benchmarkd on a range of LLMs including different versions of the same proprietary and open-source models and describe the obtained results in detail. They address main research questions including the identification of potentially illusory ToM according to their definition.", "reasons_to_reject": "A main reason to reject is the lack of the benchmark itself: How can this be used as a real benchmark if it is not available to anyone? What does „plan to release“ mean? The authors should please state whether they will release the benchmark or not and under which license. As the authors produced FANToM with GPT-based models, it would be helpful if they could elaborate on potential restrictions. \nIf the the benchmark is not released, the authors should make at least the code for benchmark construction and model evaluation fully available to ensure that other researchers can try to reproduce results (at least on the open-source portion of the LLMs) and potentially build on your work. \n\nA main aspect of the novelty of this work is the evaluation of LLMs regarding ToM in natural conversations vs. other types of conversations / narratives. These conversations are insofar natural as they are not template-based but text obtained by prompting GPT-3.5 text-davinci-003/GPT-4 which are, as far as I understood, in turn also evaluated on the text is generated. This point is a weakness that is in my opinion part of the paradigm that a proprietary, highly under-documented model can be evaluated on a benchmark for the construction of which it was given substantial information about this benchmark. I very much see the point in evaluating on other LLMs, especially open-source models, but unless the authors own that model, I do not think it is very clear what exactly happens with the input. I would have liked to at least see some kind of comment on this.\n\nThe construction process is interesting and sound given that one agrees with the paradigm that natural conversations do not need to be generated by a human but can be generated by model. What I am missing regarding the naturalness of conversations is a more nuanced (report of) human validation. The authors state that they conducted a human validation of the benchmark but do not provide any details (e.g., guidelines and validation protocol, how many annotators, what criteria are applied in the recruting process, how well do they agree on which dimensions, etc.). Hence, I am not completely sure about the quality of the benchmark w.r.t. this dimension. Also, it would be helpful if the authors could talk about potential anthropomorphism not only in the ethics section but much earlier to make clear that these conversations are actually a machine-generated version of human mental states. This does not mean that the benchmark is not interesting or of bad quality, but that this should be made known to the reader right in the introduction.\n\nAnother point concerns the AMT evaluation yielding the relatively low human upper bound: As far as I understand from the appendix, each question set is evaluated by one annotator. Each annotator could do 1+ question sets. The authors do not report how many question sets each annotator evaluates. Quite some work has shown that AMT annotation might not be the best choice in every scenario due to spammers, unreliable annotators, and this is especially true if, in fact, only one annotator evaluated each questions set.  In case of a potential new benchmark which could be assumed as a substantial extension over previous work, it would be nice to at least have either more annotators per question set and/or one other kind of validation / human performance checkpoint where trust in annotators can be rated higher than for AMT. This could be any kind of non-AMT evaluation, e.g., through offline recruiting, a custom online evaluation, or other service platforms such as Prolific, etc.", "questions_to_authors": "- Do you plan to release the benchmark and if so, under which license? What is the timeline for this? \n- Abstract: Please state the language that is investigated / benchmark focuses on in the abstract. \n- Intro + Related Work: Please add peer-reviewed papers of high quality as citations about the „debate around ToM“ instead of citing 2 New York Times articles only (one behind a paywall…)\n- l. 154f: You create „false answers that have high word correlation with the input to verify whether the models can overcome the shortcut pattern matching when reasoning mental states“. I might have missed it in the results/analysis section but as far I saw you only briefly touched upon lexical overlap in 517f in a different context. However, the community might be interested in an analysis of this and potential implications. \n- l. 409 + corresponding App: Could you please state how many question sets each of the annotators evaluated with information on whether some worked on substantially more than others? Why did you only ask 1 annotator per question set? Why use AMT and not a different way of evaluation where trust in annotator reliability is a little higher?\n- Results Table: \n    - Would it be possible to generate a results table where all models’ performances are included (this can go in the appendix, but I find it hard to compare performances across many models with 2 tables).\n    - Would it be possible to provide a type of indication of differences in model performance that are statistically significant?\n- l. 570, 588, etc.: Can you please elaborate on what is meant with CoT having „a selective impact on different questions types“? Do you have any intuition about why this could be the case?\n\n// edit: Thanks to the authors for their detailed responses.", "paper_topic_and_main_contributions": "The authors propose a benchmark to test Theory of Mind in natural conversations which the authors view as raw interactions as opposed to previous benchmarks consisting of template-based narratives potentially explicitly stating implicit cues. The authors state that FANToM is designed to evaluate a given model’s capability for a coherent understanding of other’s mental states at the example of tracking a model beliefs regarding multiple characters in natural conversations with some info inaccessible to some of these characters. An additional goal is to identify so-called illusory ToM which the authors describe as detecting cases where the model answering some questions correctly, while failing others requiring the same type of ToM reasoning.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a benchmark for theory of mind conversations, which is novel and timely wrt to the current pace of LLM development. Authors in their rebuttal also confirm the intention to release this benchmark for general use ASAP, which is the most valuable part. Contingent on this release, the paper can be very helpful for LLM evaluations. I would also urge authors to release a datasheet, detailing aspects like cost of data construction, intended usage, etc. This recurs in reviewers’ questions and underlines how any reader or user will have the same questions. 
Additionally, like reviewers point out, some of the discussions, specifically around anthropomorphism and limitations wrt it should appear much earlier in the paper to establish the context of benchmark usage.
With these updates, the work would be of great use to the community.
Decision: Accept-Main