How FaR Are Large Language Models From Agents with Theory-of-Mind?
Authors: name: Pei Zhou; name: Aman Madaan; name: Srividya Pranavi Potharaju; name: Aditya Gupta; name: Kevin R. McKee; name: Ari Holtzman; name: Jay Pujara; name: Xiang Ren; name: Swaroop Mishra; name: Aida Nematzadeh; name: Shyam Upadhyay; name: Manaal Faruqui
Published: 2023-10-04 06:47:58

Abstract: "Thinking is for Doing." Humans can infer other people's mental states from
observations--an ability called Theory-of-Mind (ToM)--and subsequently act
pragmatically on those inferences. Existing question answering benchmarks such
as ToMi ask models questions to make inferences about beliefs of characters in
a story, but do not test whether models can then use these inferences to guide
their actions. We propose a new evaluation paradigm for large language models
(LLMs): Thinking for Doing (T4D), which requires models to connect inferences
about others' mental states to actions in social scenarios. Experiments on T4D
demonstrate that LLMs such as GPT-4 and PaLM 2 seemingly excel at tracking
characters' beliefs in stories, but they struggle to translate this capability
into strategic action. Our analysis reveals the core challenge for LLMs lies in
identifying the implicit inferences about mental states without being
explicitly asked about as in ToMi, that lead to choosing the correct action in
T4D. To bridge this gap, we introduce a zero-shot prompting framework, Foresee
and Reflect (FaR), which provides a reasoning structure that encourages LLMs to
anticipate future challenges and reason about potential actions. FaR boosts
GPT-4's performance from 50% to 71% on T4D, outperforming other prompting
methods such as Chain-of-Thought and Self-Ask. Moreover, FaR generalizes to
diverse out-of-distribution story structures and scenarios that also require
ToM inferences to choose an action, consistently outperforming other methods
including few-shot in-context learning.