MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
Authors: Damien Sileo
Antoine Lernould
Venue: EMNLP 2023 Findings (2023)
Keywords: logic
epistemic logic
theory of mind
language models
TL;DR: We generate epistemic logic problems to isolate a well defined component of theory of mind and evaluate language models capabilities.

Abstract: Theory of Mind (ToM) is a critical component of intelligence but its assessment remains the subject of heated debates. Prior research applied human ToM assessments to natural language processing models using either human-created standardized tests or rule-based templates. However, these methods primarily focus on simplistic reasoning and require further validation. Here, we leverage dynamic epistemic logic to isolate a particular component of ToM and to generate controlled problems. We also introduce new verbalization techniques to express these problems in English natural language. Our findings indicate that some language model scaling (from 70M to 6B and 350M to 174B) does not consistently yield results better than random chance. While GPT-4 demonstrates superior epistemic reasoning capabilities, there is still room for improvement. Our code and datasets are publicly available.

MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
Damien Sileo
Antoine Lernould
Lille, Inria, CNRS Centrale Lille, UMR 9189 CRIStAL, F-59000 Lille, France damien. si 1 @ ineii.ira...
Univ.
Univ. Lille
CRIStAL, F-59000 Lille, France
Abstract
Sally-Anne problem (Baron-Cohen et al., 1985): Sally puts a marble in box while Anne is with her. Salty leaves for a moment and Mary puts the marble for the marble? [ANSWER=BoX]
Theory of Mind (ToM) is critical component of intelligence but its assessment remains the subject of heated debates. Prior research ap- plied human ToM assessments natural lan- guage processing models using either human- created standardized tests or rule- based tem- plates. However, these methods primarily fo- cus simplistic reasoning and require further validation. Here, we leverage dynamic epis- temic logic isolate particular component of ToM and to generate controlled problems. We also introduce new verbalization techniques to express these problems in English natural lan- guage. Our findings indicate that some lan- guage model scaling (from 70M 6B and 350M to 174B) does not consistently yield re- sults better than random chance. While GPT-4 demonstrates superior epistemic reasoning ca- pabilities, there is still room for improvement. Our code and datasets are publicly available'
basket. Where will Sally look
Le et al. (2019) deem these problems simplistic and extend them to track second-order beliefs (e.g. the belief of Sally about Anne's beliefs).
In our study, we generate dynamic epistemic logic (DEL) problems and develop verbalizations to transform them into natural language infer- ence problems. DEL is a branch of modal logic that can model an individual's knowledge about particular facts or about other agents' knowl- edge. DEL also enables reasoning about the impact of consecutive public announcements: Alice and Bob have mud their head. Their father says that at least of them is muddy. He asks Alice and Bob they are muddy. Do Alice and Bob know that they are muddy? [AN- SWER=NOJ They answer that they don't know. Do Alice and Bob now know that they are muddy? [ANSWER=YES]
1 Introduction
Theory of Mind (ToM) is the cognitive ability to at- tribute mental states, such as beliefs, desires, and in- tentions, to oneself and others, allowing individuals to understand and predict behavior based on these inferred mental states. It is an important require- ment for general text understanding artificial in- telligence (Navarro et al., 2020), but claims about ToM are prone to bias from human expectations (de Waal, 2016). Kosinski (2023) recently sparked debate by showing that scaling large language mod- els (LLMs) improves performance at standardized tests designed to measure ToM. However, these tests were widely discussed in academic research and might have leaked into the training corpora of LLM. Earlier work generated synthetic exam- ples instead, extending the bAbi (Weston et al., 2016) framework. Nematzadeh et al. (2018) pro- posed a dataset of fixed templates based on the
Bob would have answered YES to the first ques- tion if Alice was not muddy, so after Bob's first answer, Alice can know that she is muddy. DEL can formalize certain ToM problems, making it valuable perspective for ToM assessment. The problems we create can require tracking multiple agents' beliefs and reasoning about higher-order beliefs Our dataset encompasses numerous vari- ations of the Muddy Children and Drinking Logi- cians problems (van Eijck, 2014). This controlled test bench offers new appreciations of language model scaling and presents the first dataset with a complexity that challenge supervised learning models. The dataset and the scripts to generate is publicly available!
'The same holds if we switch 'For example, Anne's belief about Sally's belief about Anne's belief about Mary's belief.
Alice.

---

2 Related Work
in robotics. Van De Pol et al. (2018) explored the plausibility of epistemic logic for ToM by investi- gating its theoretical computational tractability.
Logical Reasoning in Natural Language Process- ing Logic shares profound connections with NLP. Early systems were built around logic, and more recent approaches incorporate logical reasoning into neural networks (Hamilton et al., 2022; Helwe al., 2022). Another line of research closer to ours investigates the logical capabilities of NLP models using textual datasets and labels generated with log- ical reasoning tools. RuleTaker (Clark et al., 2020) explores this area with propositional logic, while LogicNLI addresses first-order logic (Tian al., 2021). Richardson and Sabharwal (2022) examine the satisfiability problem in natural language. Sileo and Moens (2022) targets probabilistic logic. Our study the first to focus on modal logic, specifi- cally epistemic logic, in natural language.
Dynamic Epistemic Logic Problem Generation and Verbalization
3.1 Problem definition
Our objective is to simultancously create dynamic epistemic logic problems and their corresponding natural language representations, with (PREMISE, HYPOTHESIS, LABEI.) format.
An epistemic logic problem can be decomposed into the following components:
Agents:
A set of N individuals, each assigned a different arbitrary name.
A set of Boolean predicates. Here, we use N predicates, one corresponding each agent (e.g.. has mud on her head).
Predicates:
Theory of Mind in NLP To measure ToM capa- bilities of NLP models, Nematzadeh et al. (2018) created examples using Sally-Ann templates, and Le et al. (2019) added complexity to the data by incorporating second-order knowledge. Both stud- ies framed their examples as tasks. Kosinski (2023) employed handcrafted tests evaluate language models' next-word prediction capabilities. Ullman (2023) showed LLM brittle- ness to interventions on these datasets and Ma et al. (2023) consolidated the prior datasets into a prin- cipled evaluation suite. The Social-IQA dataset (Sap et al.. 2019) covers a broad spectrum of social commonsense, encompassing aspects of theory of mind and challenges like comprehending desires and emotions. Cohen (2021) investigated whether natural language inference models captured veridi- cality with epistemic verbs like know and think, using handcrafted patterns. This task was incorpo- rated into the BIG-Bench framework (Srivastava et al., 2022) as the epistemic-reasonie it targets only one shallow aspect of epistemic rea- soning. Bara et al. (2021) used a Minecraft dataset for real-time belief deduction in collaborative tasks, Shapira et al. (2023b) highlighted LLM struggles in faux pas tests. Shapira al. (2023a) conducted stress tests on LLMs' social reasoning capabilities.
Observabilities:
The description of each agent's initial knowledge of the predicate values. We repre- sent observabilities with a boolean matrix O of size N NxN, where Oi,=1 means that agent i initially knows whether predicate j is true.
list of expressions (predi- cates or agent knowledge about predicates) that are shared to all agents. Announcements are made se- quentially, and each new announcement can change what the agents know, even if it is the same an- nouncement is repeated twice.
Announcements:
Hypothesis: An expression that may contain predicates and knowledge of agents about partic- ular expressions after the announcements, given the agents, observabilities, and announcements grouped into a premise.
task, but
3.2 Setups: connecting predicate and observabilities
The concrete choice of predicates dictates the struc- ture of observabilities. For example, the predi- cate "Alice has mud on her head" is observable by agents other than Alice, but "Alice has mud on her hand" could be observable by everyone. We group predicates and observabilities into what we call se- tups to generate textual descriptions. We define the following setups:
Epistemic Logic and ToM Bolander (2018) showed that the Sally-Ann problem could be mod- eled with epistemic logic. Van Ditmarsch and Labuschagne (2007) examined more general con- nections between DEL and ToM, while Dissing and Bolander (2020) demonstrated DEL's applicability
Forehead -ud setup
PREDICATE; <AGENT,>3 's forehead is muddy. O ONES(N)= IDENTITY(N)

---

Forehead-mud-mirrog setup
Thirst setup
Norehead-muc setup
Explidi setup
0.9
5-4hot
0.7
Humann
0.5
160M410M
1.0
70M 160M410M
2,00
160M410M 16 1:40
T0M 160M410M
1:40
Pythia size
Pythia
Pythia size
Pythia size
Figure Accuracy of Pythia language models on MindGames setups.
Forehead-mud mud-mirror setup
Thirst setup
Forehead- mud setup
Expliit setup
D-shot
0.7
S-shol
0.6
0.5
350M
1:30
6.70
17:00
350M
1.30
1740
350M
1.30
6.70
1740
350M
1.30
6.70
1740
GPT3 size
GPT-3 size
GPT-3 size
CPT-3 size
Figure 2: Accuracy of GPT-3 family (ada, cabbage, curie, davinci) language models on MindGames setups.
Forehead-mud-miin setup
e.g. Alice cannot know whether Bob is not muddy. We use can to acknowledge that an agent could theo- retically infer something but fail see it. A Kth order belief is a first-order belief about a 1I order belief. We consider everyone, not everyone, and nobody as possible subjects for the setup predi- cates. Subjects are uniformly sampled among these quantifiers and the list of individual agents. We transform abstract problem representations into nat- ural language and code that can be fed to a model checker determine whether a hypothesis is en- tailed by the premise. We use the SMCDEL model checker (Benthem et al., 2018 8), an announcement logic based on the S5 (Lewis et al., 1959) modal logic. This implementation is the most cited pub- licly available epistemic logic as of April 2023. We discard examples where the premise contains contradiction To generate diverse and gender- balanced random English surnames, we use Cen- susName (Qian al., 2022).
PREDICATE;: <'G forehead is muddy,
O: ONES(N)
OBSERVATION: There is mirror in the room.
Thirst setup
PREDICATEJ: <AGENT,> is thirsty.
O: IDENTITY(N)
Explicit setup
PREDICATE;: <AGENT;> picked red card. O: RANDDOOL N,, (sum(O))=N
OBSERVATION: Each person draws a card, face unrevealed (red or black). <AGENT;> card is revealed to <.GENT,>. for all i,j where O,,==>
3.3 Problem verbalization
We then construct a problem for given setup with the following natural language template:
[Premise] There are <N> persons. Everyone is visible to others. <OBSERVATION> publicly announced that someone <PREDICATE>> <(0 N]
ANNOUNCEMENTS>
4 Experiments
[Hypothesis] <{1 K]th ORDER BELIEF>
4.1 Problem generation parameters
[0 NJ denotes uniform sampling from 0 to N. We restrict announcements first-order be-
We randomly sample (2, 4} agents, we observed that problems were sufficiently challeng-
liefs. first-order belief has the following struc-
"We identify contradictions by examining whether an un- used predicate is entailed or not by the premise. 'https: //pppt/pppp../ oorg/ prwppptt.////pooppoooooopooooo pptoject/cennussaneet censusname/
ture: <AGENT> (can know whether can know that cannot know that cannot know whether)

---

ing with only three agents, and we use K for the same reason. We use knowledge predicate nega- tions 80% of the time to encourage richer infer- ences (as the fact that an agent does not know something conveys information to others) in an- nouncements and 50% of the time otherwise.
Human evaluation We present 50 test samples per setup to two NLP researchers only instructed to perform entailment detection. Inter-annotator agreement is 0.89, and average accuracy is 94%
Pythia language models We select the Pythia (Biderman et al., 2023) language models for our open-source scaling experiments. We use the checkpoints trained on the deduplicated corpus (deduped) with checkpoint sizes of 70M, 160M, 410M, 1B, 1.4B, 2.8B, and 6.9B.
4.2 Controlling for example difficulty
Shortcuts, like hypothesis only bias (Gururangan et al., 2018; Zhang et al., 2023), can lead to the answer without correct reasoning. To control for shortcuts, trained relatively shallow super- vised model (deberta-small (He et al., 2021), 6 layers, 44M backbone parameters) on a training set combining all setups (ensuring that there was no duplicate and no example that was also in the test set). We used 11 1.2k training examples for epochs and a learning rate of 3e-5 and 3.73k test and validation examples. Overall validation accu- racy was 83%. We also experimented with simpler lexical baselines like TF-IDF which did not capture negations well enough. We assumed that examples correctly predicted by deberta-small with high con- fidence contained shorteut cues. We used these deberta-small predictions and confidence as addi- tional metadata. We found that the evaluated lan- guage models already failed on easy examples. So we used a random subset of the validation and test subsets for our experiments, but our dataset can be filtered by difficulty using the provided confidence level and the discrepancy between deberta-small prediction and ground truth.
OpenAl API We evaluate the OpenAl GPT-3 (Brown et al., 2020) models, specifically the ada, babbage, curie, and davinci checkpoints, through the public API. We assume that their model sizes respectively 350M, 1 1.3B, 6.7B, and 174B and we use the default temperature. Figure displays the results for various Pythia model sizes. We observe that scaling improves 5- shot? reasoning, but it has no impact on zero-shot reasoning. In contrast to the emergence results reported by Kosinski (2023), Figure 2 does not show clear scaling trend for GPT-3 models on MindGames data, which suggests that the emer- gent behavior they observed was not due robust epistemic logic capabilities.
4.4 Qualitative analysis with ChatGPT
We also run brief qualitative analyses with GPT- 3.5 and GPT-4 (OpenAI, 2023), as May 2023. On 20 randomly sampled problems, we found that GPT3 was 60% correct and GPT-4 70% correct. We present a brief qualitative analysis of the re- spective models.
We limit the number of agents to 3 and dedu- plicate then undersample the problems to generate 400 test cases with perfect balance of True/False labels per setup. We refer to the resulting dataset as MindGames.
As shown in Appendix A, GPT3.5 tends to an- swer that there is not enough information and perform correct inferences only when it requires very shallow reasoning. GPT-4 can solve this par- ticular example. However, some problems are still challenging, as shown in Figure 3. GPT-4 rarely answers that there is not enough information and its reasoning looks has the surface form of epis- temic reasoning, but occasionally contains glaring mistakes.
4.3 Scaling experiments
We conduct zero-shot experiments and few-shots with a range of language models. We use standard prompting to follow Kosinski (2023) setup. We use the Im-eval-harnese software (Gao et al., 2021) to measure whether language model per- plexity favors the correct reasoning in multiple- choice setting, with natural language inference prompt from Brown al. (2020): <PREMISE> Question: <HyPOTHESIS> True or False ?" with two possible continuation choices, True and False. We evaluate two families of language models:
5 Conclusion
We developed a novel dataset aimed at evaluating epistemic logic reasoning, addressing a particular
'Most errors arose from failing to distinguish between know whether and know that "Increasing number of examples did not improve valida- tion accuracy.

---

User:
Moghaddam and Honey (2023) recently showed that more advanced prompting schemes made sig- nificant differences. addition, we did not explore the full range of Pythia models due to computa- tional limitations.
[Premise] There are two persons. Everyone is visible to others. There is mirror in the room. It is publicly announced that someone's forehead is muddy. It is publicly announced that not everyone's forehead is muddy. is publicly announced that not everyone's forehead is muddy. [Hypothesis] Robert can now know whether or not everyone's s forehead is muddy. Is the hypothesis true or false ?
Task complexity, annotators variation The task proposed is relatively complex, and raises ques- tions about the profiles of annotators that would match the results of a symbolic reasoner. The framework of DEL. itself can also provide insights on theory of mind, as DEL solver perfectly solves this task, even though we could feel uncomfortable attributing ToM to the solver. We might argue that failing on simple DEL examples disproves ToM, but proving failure is difficult, mentioned in the previous paragraph.
GPT-4:
False. Based on the information provided, Robert cannot know for certain whether not everyone's forehead is muddy. The statements provided are: There are two persons. Everyone is visible to others. There is mirror in the room. It is publicly announced that someone's forehead is muddy. It is publicly announced that not everyone's forehead is muddy. Since it is publicly announced that not everyone's forehead is muddy, Robert can deduce that least one person does not have a muddy forehead. However, this information alone does not allow Robert to definitively determine whether not his own forehead is muddy.
Ethical considerations
This work involves human annotations. However, we used procedurally generated data, ensuring no confidential or harmful content. Besides, annota- tions were carried out during the researchers' work- ing hours. For these reasons, Institutional Re- view Board has determined that it was exempted from formal review according to internal guide- lines.
Figure 3: GPT4 conversation
aspect of ToM. Our results reveal that this task con- tinues to pose challenges for contemporary large- scale language models. When future models can solve MindGames for 2-3 agents, the difficulty can be easily scaled up with more agents. Future stud- ies could better explore human performance on our dataset, taking into account factors such as age and educational background. Additionally, further in- vestigation can examine the impact of fine-tuning on other downstream tasks and assess how well Transformer circuits model Kripke structures that represent modal logic problems.
References
Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021. MindCraft: Theory of mind modeling for situ- ated dialogue in collaborative tasks. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 1112-1125, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Simon Baron-Cohen, Alan M Leslie, and Uta Frith. 1985. Does the autistic child have "theory of mind"? Cognition, 21(1):37-46.
6 Limitations
Johan Benthem, Jan Fijck, Malvin Gattinger, and Kaile Su. 2018. Symbolic model checking for dy- namic epistemic logic s5 and beyond*. Journal of Logic and Computation, 28:367-402.
Theory of mind is a complex subject, and our study takes deliberately specific angle, leaving multiple open problems:
Language Our work is centered on English, the method could be adapted to other languages us- ing vubjeet-ver-oojeett structure. Besides, restricted our study to templates that do not cover the full variety of the English language.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle Briee,, Fric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373
Thomas Bolander. 2018. Seeing is believing: For- malising false-belief tasks in dynamic epistemic logic. Jaakko Hintikka on Knowledge and Game- Theoretical Semantics, pages 207-236.
Prompt structure and models scaling We fo- cused on shro and few-shot prompting, which were sufficient to (Kosinski, 2023), however,

---

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are fow-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Empirical Methods in Natural Language Processing: System Demonstrations
Michal Kosinski. 2023. Theory of mind may have spon- tancously emerged in large language models, arXiv preprint arXiv:2302.00083.
Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 5872-5877, Hong Kong, China. Association for Computational Linguistics.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as soft reasoners over language. In Pro- ceedings of the Twenty-Ninte International Joint Con- ference on Artificial Intelligence, IJCAI-20, pages 3882-3890. International Joint Conferences on Arti- ficial Intelligence Organization. Main track.
Clarence Irving Lewis, Cooper Harold Langford, and P Lamprechh. 1959, Symbolic logic, volume 170. Dover publications New York.
Xiaomeng Ma, Lingyu Gao, and Qihui Xu. 2023. Tom- challenges: A principle-guide dataset and diverse evaluation tasks for exploring theory of mind. arXiv preprint arXiv:2305.
Michael Cohen. 2021. Exploring roberta's theory of mind through textual entailment. philarchive.
F. de Waal. 2016. Are We Smart Enough to Know How Smart Animals Are? W. W. Norton.
Shima Rahimi Moghaddam and Christopher Honey. 2023. Boosting theory-of-mind performance in large language models via prompting. arXiv preprint wrXiv:230.111900.
Lasse Dissing and Thomas Bolander. 2020. Implement- ing theory of mind on a robot using dynamic epis- temic logic. In Proceedings of the Twenty-inth International Joint Conference on Artificial Intel- ligence, IJCAI-20, pages 1615-1621. International Joint Conferences on Artificial Intelligence Organi- zation. Main track.
Ester Navarro, Sara Anne Goring, and Andrew R. A. Conway. 2020. The relationship between theory of mind and intelligence: A formative g approach. Jour- nal of Intelligence, 9.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.
Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. 2018. Evaluating theory of mind in question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2392-2400, Brussels, Belgium. Association for Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computa- tional Linguistics.
OpenAL. 2023. Gpt-4 technical report.
Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams. 2022. Per- turbation augmentation for fairer nlp. arXiv preprint arXiv:2205. 12586.
Kyle Richardson and Ashish Sabharwal. 2022. Pushing the limits rule reasoning in transformers through natural language satisfiability. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 36, pages 1 11209-11219.
Kyle Hamilton. Aparn Nayak, Bojan Božić, and Luca Longo. 2022. Is neuro-symbolic ai meeting its promises in natural language processing? a struc- tured review. Semantic Web, pages 1-42.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Bras, and Yejin Choi. 2019. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pages 4463- 4473, Hong Kong, China. Association for Computa- tional Linguistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using clectra-style pre- training with gradient-disentangeee disentangled embedding shar- ing. Xiviv preprint arXiv:21 11. 1100543.
Chadi Helwe, Chloé Clavel, and Fabian Suchanek. 2022. Logitorch: A pytorch- based library for logical reason- ing on natural language. In The 2022 Conference on

---

Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yeji Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2023a. Clever hans or neural theory of mind? stress testing social rea- soning in large language models. arXiv preprint arXiv:2305. 14763.
Natalie Shapira, Guy Zwirm, and Yoav Goldberg. 2023b. How well do large language models perform on faux pas tests? In Findings of the Association for Compu- tational Linguistics: ACL 2023. pages 10438-10451, Toronto, Canada. Association for Computational Lin- guistics.
Damien Sileo and Marie-Francine Moens. 2022. Prob- ing neural language models for understanding of words of estimative probability. arXiv preprint arXiv:2211.03358.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.06115.
Jidong Tian, Yitian Li, Wenging Chen, Liqiang Xiao, Hao He, and Yaohui Jin. 2021. Diagnosing the first- order logical reasoning ability through LogicNLI. In Proceedings of the 202/ Conference on Empiri- cal Methods in Natural Language Processing, pages 3738-3747, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Tomer Ullman. 2023. Large language models fail on trivial alterations theory-of-mine tasks. arXiv prepriat uxXi:2302..8399
Iris Van De Pol, Iris Van Rooij, and Jakub Szymanik. 2018. Parameterized complexity of theory of mind reasoning in dynamic epistemie logic. Journal of Logic, Language and Information, 27:255-294.
Hans Van Ditmarsch and Willem Labuschagne. 2007.
My belief's about your beliefs: case study in theory of mind and epistemic logic. Synthese, 19--99.
Jan van Eijck. 2014. Dynamic epistemic logics. Johan van Benthem on logic and information dynamics, pages 175-202.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. In 4th In- ternational Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Honghua Zhang, Liunian Harold Li, Tao Meng, Kai- Wei Chang, and Guy Van den Brocck. 2023. On the paradox of learning reason from data. In I/CA/ 23. pages 3365-3373. International Joint Conferences on Artificial Intelligence Organization. Main Track.

---

A Qualitative example with GPT3.5
User:
[Premise] There are three persons. Everyone is visible to others. It is publicly an- nounced that someone's forehead muddy. is publicly announced that Her- bert knows whether somcone's s forehead is muddy. [Hypothesis] Herbert can now know that Herbert's forehead is muddy. Is the hypothesis true false ?
GPT-3.5:
It is not possible to determine the truth value of the hypothesis based on the given premise. While it is publicly announced that someone's forehead is muddy, it is not clear who the muddy forehead belongs to. Therefore, is not possible to determine whether Herbert's forehead is muddy or not based on this information alone. However, it is stated that Her- bert knows whether someone's forehead is muddy, so Herbert may know whether their own forehead is muddy or not based on additional information not provided in the premise.
Figure 4: GPT-3.5 conversation


Reviews:
[{"review_id": "zMCFombhgl", "reviewer": "Reviewer_GnXA", "summary": "", "questions": "A. In the case of few-shot learning, was the type of examples systematically controlled? Does forehead-mud setup benefit or hinder the reasoning in foreheaf-mud-mirror setup? These two setups have similar structure, but different reasoning outcomes.\n\nB. How does the use of negation in the first-order belief (can know that v.s. cannot know that) affect the performance. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the thoughtful comments and valuable feedback on our submission.\n\n[Human evaluation] \nWe could disantangle language from reasoning by showing visual scenarios to humans, but human accuracy is already relatively high, and the objective our our study is to test verbal scenarios on LLM. Human evaluation acts as a sanity check, to validate the verbalization of the logic problems and the adequation of the logic solver.\n\n\n[Variety of setups]  The setups require similar types of reasoning (an agent knows something if it is true in all possible situations).\nThe mirror setup is not necessarily more complex, as it mostly acts as a shortcut (an agent can look at itself and knows that other agents can look at themselves).\nThe main difference lies in the commonsense assumptions about what information is initially accessible in specific situations. The explicit setup ablates that aspect by stating explicitly what is initially accessible. Our results suggest that this does not help. \n\n[Statistical association] \nWe chose explicit language over fluency, but ensured that the text is human readable. The reasoning is the center point of our work.\nWe note that there also are statistical associations in the logic itself [1].\nFor instance, specific claims have less chance to be true. (“someone knows that someone is muddy” is more likely that “Mary knows that Paul is muddy”, regardless of the premise). To control for this type of phenomena, we trained a relatively shallow supervised model (deberta-small, 6 layers, 44M backbone parameters) model on a training set combining all setups (11.2k examples, 3 epochs, LR 3e-5). Overall accuracy was 83%. Simpler lexical baselines did not capture negations well enough.\nWe assumed that examples correctly predicted by deberta-small with high confidence contained cues.\nThe dataset we enclosed with the initial submission already contains “deberta-pred” and “deberta-confidence” columns. We found that LLM already failed on easy examples, so we did not eliminate them. Our dataset can be filtered by “difficulty” using deberta confidence level and the discrepancy between deberta-small prediction and ground truth.\n\n\nA: We controlled the type of examples in the few-shot learning, to be charitable to the language models.\n\nB: The performance is low in both cases (when comparing accuracy when the hypothesis contains vs. does not contain negation).\n\n[1] Zhang, Honghua, et al. \"On the paradox of learning to reason from data.\" arXiv preprint arXiv:2205.11502 (2022). https://arxiv.org/abs/2205.11502", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper carefully designed a fairly challenging dataset based on dynamic epistemic logic problem to understand Theory of Mind. The dataset is systematically controlled with different variations, and a template for verbalization. \n\n2. They conducted scaling experiment with the designed dataset, showing that only scaling only improves 5-shot but not 0-shot reasoning. This raises the question whether language models' emergent behavior of reasoning demonstrate their ToM capacity or simply imitating the pattern.", "reasons_to_reject": "1. [Human evaluation] The dataset is only evaluated by two NLP researchers instructed to perform entailment detection. Given this task is fairly challenging for humans when the situation only communicated verbally, I wonder if the verbal set-up has increased unnecessary difficulty for language models. It would be helpful to see baseline human evaluations, to disentangle the challenge to imagine a situation based on languages, and to perform ToM and perspective taking. \n\n2. [Variety of set-ups] The authors have four set-ups (forehead-mud, forehead-mud-mirror, thirst, explicit), but did not provide any detailed by-type analysis on these variations. These details would be important to include to understand the nature of ToM, as these set-ups do not have similar complexity. For example, the forehead-mud-mirror setup requires the understanding the consequence of looking at the mirror in addition to forehead-mud setup.\n\n3. [Statistical association] While the verbalization provides a systematic way to scale up and generate test cases, it hinders the naturalness of language and make it difficult to tease apart reasoning from mimicking the statistical patterns, especially in the case of few-shot reasoning. The dataset would benefit by introducing more varieties of language template other than \"it is publicly announced that someone\", with detailed analysis on how the linguistic properties of the template is affecting the results. For instance, the recency of mentioning the target and the use of discourse connections would likely to change the reasoning accuracy. ", "questions_to_authors": "A. In the case of few-shot learning, was the type of examples systematically controlled? Does forehead-mud setup benefit or hinder the reasoning in foreheaf-mud-mirror setup? These two setups have similar structure, but different reasoning outcomes.\n\nB. How does the use of negation in the first-order belief (can know that v.s. cannot know that) affect the performance. ", "paper_topic_and_main_contributions": "This paper designed controlled experiments to assess PLMs' ability on Theory of Mind (ToM). The authors leveraged dynamic epistemic logic problems and verbalized these questions in English natural language. The results show that language model scaling does not consistently achieve better performance. The dataset contributes to the increasing debate on LMs' cognitive abilities, and to what extent scaling helps with the performance. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "krT1Y9WmQu", "reviewer": "Reviewer_fwC4", "summary": "", "questions": "Question A: Could the authors elaborate on the predicated used in Section 3.2, e.g., ONES, IDENTITY?\n\nQuestion B: Could the authors elaborate on the experiment setups mentioned in Weakness 3?\n\nQuestion C: I understand that the 4-page limit can be challenging for a short paper. If this paper gets accepted, what are the authors’ plans for the extra page?\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the thoughtful comments and valuable feedback on our submission.\n\nWeakness 1: \nThe templates are bound to rules in the previous work we cited, while we used a form of context-free grammar, and a modal logic solver, which can involve diverse reasoning patterns. The complexity stems from the logic and not the templates. We will update our phrasing in the updated version.\nTo control for shortcuts, we trained a relatively shallow supervised model (deberta-small, 6 layers, 44M backbone parameters) model on a training set combining all setups  (11.2k examples, 3 epochs, LR 3e-5). . Overall accuracy was 83%. Simpler lexical baselines did not capture negations well enough.\nWe assumed that examples correctly predicted by deberta-small with high confidence contained shortcut cues.\nThe dataset we enclosed with the initial submission already contains “deberta-pred” and “deberta-confidence” column. We found that LLM already failed on easy examples, so we did not eliminate them. Our dataset can be filtered by “difficulty” using deberta confidence level and the discrepancy between deberta-small prediction and ground truth.\n\nWeakness 2:\nWe will split the examples of line 53 into multiple lines and replace pronouns by name using additional space, and define modal logic, epistemic logic, and dynamic epistemic more explicitly.\n\nWeakness 3:\nFor the text completion evaluation:\nWe used the default parameters of LM evaluation harness v0.3.0. (1.0 for Pythia, 0.0 for GPT-3, which is the default value for the completion API). As mentioned on line 214, we use perplexity to compare the score of true and false answers.\nExamples for few-shot learning are randomly selected from the same setup.\n\nFor the ChatGPT experiments:\nWe used the ChatGPT interface (starting new chats for each example) and the default OpenAI API for qualitative evaluation, using the prompt of the qualitative analysis examples (“Is the hypothesis true or false ?”).\n\n\nA: This notation represents special matrices (Identity(N) is the identity matrix of size N, Ones(N) is the all-ones matrix of size N×N). \n\nB: See answer to weakness 3.\n\nC: In addition to the requested references and clarifications, we would be glad to use the additional space to discuss the problem of shortcuts and detail the additional analysis derived from the automatic difficulty scoring of the examples.\n\n[1] Zhang, Honghua, et al. \"On the paradox of learning to reason from data.\" arXiv preprint arXiv:2205.11502 (2022). https://arxiv.org/abs/2205.11502", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strength 1: The paper introduces the MindGames dataset. The empirical result shows that it is a challenging and useful testbed for ToM assessment in LLMs. There is a clear resource contribution for the commonsense reasoning and language modeling community.\n\nStrength 2: The problem formulation is cognitively-motivated, and the problem definition in Section 3.1 is potentially generalizable to more complex social interactions.\n", "reasons_to_reject": "Weakness 1: The paper needs to situate itself better in related work and be more cautious with the claims it made. I am worried about the authors' claim that this work “presents the first dataset with a complexity that can challenge supervised learning models (lines 65-66)”. First, to my knowledge, epistemic reasoning has been explored in terms of ToM in language models (as authors also cited in Section 2.1), and various other ToM benchmarks are known to be challenging (see missing references). Second, while authors argued that previous work adopts “human created standardized tests or rule-based templates” and “focus on simplistic reasoning and require further validation” (lines 6-9), I am not convinced that the task introduced by the authors is “complex” since it still relies on language templates and there is no analysis on potential shortcuts.\n\nWeakness 2: This paper can benefit from an improvement in writing and presentation. For example, non-experts can get very confused about the exact meaning of “epistemic logic” as there seems to be no clear definition in this work. The setups in lines 61-63 need to be described in more detail and the predicates in Section 3.2 are never introduced. In this case, people without a psychology background can get confused. The example given in line 53 has too many pronouns and is hard to read even for human readers.\n\nWeakness 3: The experiment setups need to be described in more detail. In section 4.3, only 20 sessions are randomly chosen for evaluation on GPT-4. Are the 20 sessions from the same setup as the performance on the 4 different setups clearly different? What is the hyper-parameter of inference using OpenAI API (e.g., temperature)? What are the probing techniques used (e.g., generating Y/N or perplexity)? Did the authors fix the context in 5-shot learning settings across different models? \n", "questions_to_authors": "Question A: Could the authors elaborate on the predicated used in Section 3.2, e.g., ONES, IDENTITY?\n\nQuestion B: Could the authors elaborate on the experiment setups mentioned in Weakness 3?\n\nQuestion C: I understand that the 4-page limit can be challenging for a short paper. If this paper gets accepted, what are the authors’ plans for the extra page?\n", "paper_topic_and_main_contributions": "This paper evaluates large language models (LLMs) by introducing a language inference task of dynamic epistemic logic (DEL) problems. The problem formulation is cognitively motivated, and the focus is well-scoped under the knowledge and perception aspects of the theory of mind (ToM). The paper describes the techniques to create the MindGame datasets, and evaluates the performance of the Pythia and GPT families. The key finding of this work is that scaling does not guarantee better performance on DEL inferences, and that the task is challenging even for the best-performing LLMs like GPT-4.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "See Weakness 2.", "Justification_for_Ethical_Concerns": "Human evaluation is involved in this research. Authors need to confirm that an Institutional Review Board (IRB) review is completed prior to the study."}, "extra_text": {"Missing_References": "[1] MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks. Cristian-Paul Bara et al. 2021\n\n[2] How Well Do Large Language Models Perform on Faux Pas Tests? Natalie Shapira et al. 2023.\n\n[3] Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models. Natalie Shapira et al. 2023"}}, {"review_id": "iP3f8Wo7vA", "reviewer": "Reviewer_6Ahj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the thoughtful comments and valuable feedback on our submission.\nOur qualitative analysis using ChatGPT indicates that while the problem statement is correctly understood, performing the reasoning itself remains difficult. This aligns with findings from the BIGBench results [1], which showed logic tasks like logical_deduction and analytic_entailment to be challenging for GPT-3.\n\n[1] Srivastava, A., Rastogi, A., Rao, A., Shoeb, A.A.M., Abid, A., Fisch, A., Brown, A.R., Santoro, A., Gupta, A., Garriga-Alonso, A. and Kluska, A., 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The systematic set of problems is interesting and well-designed.\nThe results are very clear: LLMs fail at the task, way below what humans do.", "reasons_to_reject": "The failure of the models is not understood: is it due to some predicates which are not understood by the model in the same way as it is implemented in the logic? Is it because reasoning steps are hard to follow (a baseline control of inferences outside of the ToM domain may be feasible)?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper offers a systematic way to construct Theory of Mind tests for LLMs. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a challenging dataset to assess LMs' abilities in Theory of Mind (ToM). The reviewers find the paper well-written and clear, and they consider the approach novel, sound, and interesting. The authors have successfully addressed the concerns raised by the reviewers. I recommend accepting the paper.
Decision: Accept-Findings