MMToM-QA: Multimodal Theory of Mind Question Answering (2024)

Abstract: Theory of Mind (ToM), the ability to understand people’s mental states, is an essential ingredient for developing machines with humanlevel social intelligence. Recent machine learning models, particularly large language models, seem to show some aspects of ToM understanding. However, existing ToM benchmarks use unimodal datasets – either video or text. Human ToM, on the other hand, is more than video or text understanding. People can flexibly reason about another person’s mind based on conceptual representations (e.g., goals, beliefs, plans) extracted from any available data. To address this, we introduce a multimodal Theory of Mind question answering (MMToM-QA) benchmark. MMToM-QA comprehensively evaluates machine ToM both on multimodal data and on different kinds of unimodal data about a person’s activity in a household environment. To engineer multimodal ToM capacity, we propose a novel method, BIPALM (Bayesian Inverse Planning Accelerated by Language Models). BIP-ALM extracts unified representations from multimodal data and utilizes language models for scalable Bayesian inverse planning. We conducted a systematic comparison of human performance, BIP-ALM, and state-of-the-art models, including GPT-4. The experiments demonstrate that large language models and large multimodal models still lack robust ToM capacity. BIP-ALM, on the other hand, shows promising results, by leveraging the power of both model-based mental inference and language models.1

heading: 1 Introduction
text: Theory of Mind (ToM) is the cognitive ability to ascribe hidden mental states (e.g. goals, beliefs, and desires) to other individuals based on their observed behavior. A hallmark of human social intelligence, ToM serves as the foundation for a wide
1Code and data are available at the project website: https: //chuanyangjin.com/mmtom-qa.
range of social interactions and a pillar of commonsense reasoning (Lake et al., 2017). Systems designed to safely and productively interact with humans in an open-ended manner, such as assistive robots (e.g., Dautenhahn, 2007; Hadfield-Menell et al., 2016; Patel and Chernova, 2022; Puig et al., 2023), AI teachers (e.g., Wang et al., 2021), and autonomous vehicles (e.g., Chandra et al., 2020), would greatly benefit from incorporating ToM reasoning capabilities. The recent advancements in machine learning, especially in the realm of Large Language Models (LLMs), have spurred increased interest in assessing these models’ aptitude for ToM reasoning (e.g., Rabinowitz et al., 2018; Kosinski, 2023; Sap et al., 2019, 2022; Ullman, 2023; Shapira et al., 2023; Shu et al., 2021; Moghaddam and Honey, 2023; Nematzadeh et al., 2018; Gandhi et al., 2021, 2023; Kim et al., 2023). Many of these assessments use either text-based or video-based benchmarks inspired by classic ToM experiments in the cognitive science literature (Wimmer and Perner, 1983).
While the recent ToM benchmarks provide welldesigned, cognitively informed tools, they share several notable limitations. One such limitation is the dependence on massive training data, which raises the concern that these models work by finding data patterns in a way that deviates from humanlike ToM reasoning (e.g., Ullman, 2023; Sap et al., 2022; Shapira et al., 2023). This paper focuses on a different but related limitation: These benchmarks rely on unimodal data, either in the form of videos (e.g., Gandhi et al., 2021), or textual descriptions of actions and environments (e.g., Kosinski, 2023; Sap et al., 2022; Gandhi et al., 2023). But ToM reasoning goes beyond merely text comprehension or video understanding. It is about forming a causal model of another person’s mind, which connects mental variables to possible actions (Baker et al., 2009; Saxe, 2012; Jara-Ettinger et al., 2016; JaraEttinger, 2019). Such a model can infer mental
16077
states from either words or vision separately, or fuse the separate information to form a single coherent mental scene. By examining multimodal ToM reasoning, we can both gain insight into the computational models that underlie human ToM and offer a stronger test for current ML models, particularly LLMs.
To systematically evaluate the ability of ML models to infer mental states from multimodal data, we developed a novel Multimodal Theory of Mind Question Answering benchmark (MMToM-QA). As shown in Figure 1, the benchmark includes as input both videos and text describing the activity of a person in a household environment. The benchmark also includes questions associated with different points in each of the videos. The questions refer to the mental states (goals and beliefs) of the person described by the video or text. Each question has two possible options, neither surely true nor surely false, but with one option significantly more likely to be true given the observations. Some questions can be adequately answered based on a single modality, but some questions require fusing information from both modalities (e.g. understanding the woman’s goal in Figure 1). We validated our benchmark through human experiments, showing that people are adept at answering the questions in the benchmark and providing a human baseline.
We propose a novel multimodal ToM model,
Bayesian Inverse Planning Accelerated by Language Models (BIP-ALM). As illustrated in Figure 3, BIP-ALM first extracts symbolic representations about the physical scene and the actions of the person from both video and text inputs. Using these symbolic representations, BIP-ALM then extends Bayesian inverse planning (BIP) (Baker et al., 2017), a cognitively grounded ToM method originally designed for visual data, to reason about the multimodal data. To accelerate the inference in real-world scenarios such as household activities in our benchmark, BIP-ALM uses a language model (LM) finetuned on human activity data to evaluate the likelihood of hypotheses about the person’s belief and goal. By doing so, it takes advantage of the robustness of Bayesian inverse planning, as well as the scalability and open-endedness of LMs.
We compared the performance of BIP-ALM and several state-of-the-art models for text QA or multimodal QA, including GPT-4(V). We found that existing models, however impressive in other QA benchmarks, make large and systematic errors in our benchmark, and fail to match human performance. In contrast, BIP-ALM significantly outperforms these models.
In sum, our main contributions include (1) the first benchmark for multimodal ToM, (2) a novel ToM reasoning method, BIP-ALM, that combines Bayesian inverse planning and LMs to conduct
robust yet efficient ToM inference based on multimodal data, and (3) a systematic comparison of different ML models and human ToM.

heading: 2 Related Work
text: Theory of Mind Benchmarks. Existing ToM benchmarks are based on either videos or text. Visual-based benchmarks (e.g., Gandhi et al., 2021; Shu et al., 2021; Netanyahu et al., 2021) typically use animations of goal-directed agents to evaluate different concepts in ToM. Text-based QA benchmarks (e.g., Le et al., 2019; Shapira et al., 2023; Hewitt and Cohen, 2021; Gandhi et al., 2023; Kim et al., 2023; He et al., 2023) adapt or extend a classic false belief test, the Sally-Anne test (Wimmer and Perner, 1983). Questions in these benchmarks ask a model to select the true hypothesis about a person’s knowledge and belief based on a given premise. Triangle COPA (Gordon, 2016) asks questions about the mental states of agents based on text descriptions of abstract shapes acting like social agents. Moreover, there are QA benchmarks (e.g., Zadeh et al., 2019; Sap et al., 2019) that do not specifically test ToM, but evaluate social intelligence in general. Lastly, there have also been multi-agent challenges (e.g., Sclar et al., 2022; Puig et al., 2020, 2023) evaluating ToM as part of the tasks. Unlike existing benchmarks, our MMToMQA evaluates machine ToM on multimodal data to test both goal and belief inference. We also go beyond simple visual and textual stimuli and evaluate ToM using long everyday activities in complex environments. Critically, ToM QAs ask questions about people’s mental states. This is fundamentally different from VQAs (e.g., Antol et al., 2015; Zellers et al., 2019) which do not require mental state inference. Table 2 in Appendix A provides a detailed comparison.
Multimodal Question Answering. There have been several multimodal QA benchmarks developed in recent years (e.g., Talmor et al., 2021; Singh et al., 2021; Sanders et al., 2023; Fu et al., 2023; Li et al., 2023). These benchmarks focus on the ability of models to detect and retrieve relevant information from multimodal inputs (e.g., images, videos, text, tables) to answer factual questions. However, there have not been multimodal QA benchmarks for ToM, an ability fundamentally different from the kind of multimodal information retrieval tested in the existing benchmarks.
Machine Theory of Mind. There have been
two main approaches to engineering machine ToM: end-to-end methods such as Theory of Mind neural networks (e.g., Rabinowitz et al., 2018), and modelbased methods such as Bayesian inverse planning (e.g., Baker et al., 2017; Shu et al., 2021). Both types of approaches focus mostly on unimodal data and simple domains. Recent studies have suggested that machine ToM may also emerge in LLMs such as GPT-4 (Kosinski, 2023; Bubeck et al., 2023). However, more systematic evaluations have shown that apparent ToM capacities in LLMs are not yet as robust as humans (Sap et al., 2022; Shapira et al., 2023; Sclar et al., 2023), and often fail to pass trivial variants of common tests (Ullman, 2023). Our BIP-ALM model builds on the strengths of these different methods. It extends Bayesian inverse planning to fuse multimodal data and conduct inference in complex scenarios with the use of language models.

heading: 3 MMToM-QA Benchmark

heading: 3.1 Overview
text: Our benchmark consists of 134 videos of a person looking for daily objects in household environments, in line with cognitive studies examining mental attributions to agents navigating an environment (e.g. Baker et al., 2017). On average, each video has 1,462 frames, depicting 36 human actions. Based on these videos, we constructed 600 questions about a person’s goals and beliefs. Each question is paired with a clip of the full activity in a video (as RGB-D frames), as well as a text description of the scene and the actions taken by the person in that clip. All questions have two choices. The questions are categorized into seven types (see Figure 2), evaluating belief inference and goal inference in rich and diverse situations. Each belief inference type has 100 questions, totaling 300 belief questions; each goal inference type has 75 questions, totaling 300 goal questions. We provide another set of synthetic human behavior data in household environments for model training. This training set includes 1,000 procedurally synthesized videos with ground-truth annotations of the scene, objects, goals, and beliefs.

heading: 3.2 Question Types
text: Questions fall into two categories – belief inference and goal inference. There are several types within each category (Figure 2), evaluating different aspects of multimodal ToM reasoning. Unlike
existing ToM evaluation that isolates goal and belief inference, questions in our benchmark require a joint inference of goal and belief, asking about one conditioned on another. Type 1.1: True belief, short-term. In the scenarios this type refers to, a person is about to open a container. The question focuses on an object the person has not seen so far and treats it as a hypothetical goal object. The question is then whether the person believes the object is in the container. This type examines true belief ; that is, the inference that the person believes the goal is in the container is more consistent with the current action, and it is also the same as the actual world state. Type 1.2: False belief, short-term. This type is similar to Type 1.1, but differs in a significant way: the hypothetical goal object is not inside the container that the person is about to open, so the person has a false belief. In this case, the correct answer should still be that it is more likely that the person thinks there is such a goal object inside the container, given the current action, even though they would not find what they want there in reality. Type 1.3: Belief tracking, long-term. In the scenarios this type refers to, the person passes by a container but does not check it. After a while, they have still not found a goal object, but are also not going back to check the container they passed by. This suggests that they do not think the goal object is inside that container. This question tests whether a model can use the long-term observation of past actions to make judgments consistent with history, not just the most recent action. Type 2.1: Goal inference given true belief. This question targets a person’s unknown goal. In the scenarios that this type refers to, the person walks towards a container, where there is a hypothetical goal object that the person has not observed so far. The person, on the other hand, has observed the other hypothetical goal object but has not picked it up in the past. The correct inference is that it is more likely that the person wants the goal object that they have not seen so far and thinks it is inside the container (true belief). This type tests whether a model can infer the goal given a true belief. Type 2.2: Goal inference given false belief. This type is similar to type 2.1, but the person has a hypothetical false belief. In these scenarios, a specific object is inside a container that the person is walking towards. However, the question states that this person thinks that there is no such object inside the container (false belief). So, the most likely explanation is that the person’s goal is a different object, which they think might be inside the container. Type 2.3: Goal inference given updated belief. Unlike Type 2.1 and 2.2, in a Type 2.3 question, the
video does not end with the person walking towards a container. Instead, the person opens the container and then closes it without taking an object from it. The correct inference is that the person’s goal is an item not yet seen rather than anything inside the container. To correctly answer this type of question, a model has to infer how a person may update their belief and change their plan (e.g., closing the container without picking anything from it) to pursue the goal accordingly. Type 2.4: Goal inference given future actions. Questions in Type 2.1, 2.2, and 2.3 ask about goals that are consistent with the belief and the latest action. In contrast, in Type 2.4, a model needs to consider possible future actions as well. Specifically, one of the hypothetical goal objects is an unobserved object at a location that is still far away from the person and is not directly related to the latest action (which is walking to a nearby container). But, in these scenarios, the person is on a path to potentially reach the location of that object. For instance, in the Type 2.4 example illustrated in Figure 2, a person is walking towards the right side of the room, and so they might want to search through all possible locations on the right side, including the dishwasher. This gives rise to the correct answer, dish bowl, which is located inside the dishwasher. As such, Type 2.4 requires a model to reason about the spatial relationships (the locations of objects and the person as well as the person’s heading direction) in a visual scene and predict possible future actions for a goal.

heading: 3.3 Procedural Generation
text: We designed a procedural generation method for creating the benchmark. First, we procedurally synthesized a large set of videos in a realistic household embodied simulator, VirtualHomeSocial (Puig et al., 2020). As Puig et al. (2020) have demonstrated, such procedural video generation can create synthetic human activities that are human-like and well-annotated (including groundtruth states, goals, and beliefs). It also alleviates the concerns of cost and privacy that come with realworld human activity video collection. At each step in a video, we sampled a question type and two opposing hypotheses based on the definition of the type. Finally, we generated the text description and the question based on the ground-truth state, actions, and the sampled hypotheses using GPT4 to create the text input for the question. Using the same procedural generation, we synthesize the
videos in the training set. We provide more details in Appendix B.7.

heading: 3.4 Evaluation Protocol
text: We can evaluate a model in three conditions: (1) Multimodal QA in which both the video and text inputs are present, (2) Text QA with only the text input, and (3) Video QA with only the video input. We evaluated all models in a zero-shot manner, which is the standard setting in recent literature on ToM QA evaluation (Shapira et al., 2023). Crucially, we do not provide any example QAs during training. We expect a model to learn how a person updates their mental state and acts accordingly in a physical environment from the human behavior data in the training set, and generalize the learned knowledge to answer the questions at test time.

heading: 4 The BIP-ALM Model
text: To infer the mental state of a person based on video and text inputs, we propose a novel machine Theory of Mind method, Bayesian Inverse Planning Accelerated by Language Models (BIP-ALM), which builds on Bayesian inverse planning (BIP) (Baker et al., 2017). Prior works have shown that BIP can reverse engineer human ToM reasoning in simple domains. BIP-ALM extends BIP by (1) building unified representations about a scene, a person’s actions, and the mental state hypotheses from multimodal inputs, and (2) finetuning a language model to efficiently conduct inverse symbolic planning, based on unified symbolic representations.
Figure 3 provides an overview of the method. We first extract symbolic representations of the states and actions from both the video and the text. We then align and fuse representations extracted from different modalities, to form a unified representation of the event and the physical scene. This unified representation allows us to use a principled method to infer a person’s mental state based on inputs from any given modality. We then use an inverse symbolic planner to compare the two hypotheses extracted from the question and produce the answer. We introduce each module below and provide more details in Appendix C.

heading: 4.1 Unified Symbolic Representations
text: Visual Perception. Our visual perception module processes visual data and transforms it into symbolic representations. For each frame, we adopt the method in Blukis et al. (2022) to create a voxel map and construct a scene graph.

heading: States
text: Text Parsing. We use GPT-4 to parse text, to extract symbolic representations of the initial state as well as subsequent actions. GPT-4 first parses the text into three components – the description of the environment state, the human actions, and the question. Each component is further translated into symbolic representation by GPT-4. For state, we generate predicates such as In(apple, fridge). For action, we generate action commands such as walk towards kitchen. Finally, we translate the question into two hypotheses about the goal and the belief. For each hypothesis, the goal is represented by the goal object (e.g., apple), and the belief is represented by a predicate (e.g., In(apple, fridge)), or its negation (¬In(apple, fridge)), indicating the hypothetical location of the object.
Fusion. The fusion module aligns and integrates information from different input streams. Specifically, we transform the scene graphs from the video input to a set of predicates (similar to those extracted from the text), which describe the spatial relationships between entities and the status of objects. We first form the symbolic representation of the initial state by combining predicates from the video and the text. We then align the actions parsed from the text with the actions detected from the video, and truncate the video frames into several intervals, each corresponding to an action. We term each interval, a time step t. Starting from the initial state, we update the state predicates from the previous step with the new predicates obtained from the video frame corresponding to the current step. By doing so, we can construct a symbolic state sequence and a symbolic action sequence, as illustrated in Figure 3. In addition, we form two
hypotheses based on the hypothetical goals and beliefs parsed from the question.

heading: 4.2 Inverse Symbolic Planner
text: We formulate an agent’s behavior as a forward generative model using a Partially Observable Markov Decision Process (POMDP) (Kaelbling et al., 1998), defined by the tuple ⟨S,A, T , G,R,Ω, O, γ⟩. st ∈ S and at ∈ A are the state and the action at time t. T (st|s, a) are the state transition probabilities. g ∈ G is a goal, which defines the reward of the agent rt = R(st, at, g). ot ∈ Ω is the agent’s observation at t derived following the observation function, ot = O(st). Finally, γ ∈ (0, 1] is a discount factor. The belief of an agent is modeled as a probability distribution over the state b(s). In this work, we factorize the belief of the full state into beliefs about the possible locations of individual objects. Conditioned on both the goal and the belief, a rational agent will take actions based on the optimal policy π(at|g, bt) to maximize its return ∑∞t=0 γtrt.
Given this forward generative model, we can conduct inverse inference about the agent’s goal and belief. By assuming a deterministic state transition, we can jointly infer the goal and belief of an agent given observed states and actions as follows:
P (g, bt|s1:t, a1:t−1) ∝ t∏
τ=1
π(aτ |g, bτ )P (bτ |bτ−1, sτ )
· P (b0)P (g) (1)
Given two hypotheses about goal and belief, H1 = ⟨g1, bt1⟩ and H2 = ⟨g2, bt2⟩, we can evaluate
which one is more likely to be true as
P (g1, b t 1|s1:t, a1:t) P (g2, bt2|s1:t, a1:t) = π(at|g1, bt1)P (bt1|b̂t−1, st) π(at|g2, bt2)P (bt2|b̂t−1, st)
· ∏t−1
τ=1 π(a τ |g1, b̂τ )∏t−1
τ=1 π(a τ |g2, b̂τ )
, (2)
where b̂τ is the estimated belief at a past step τ < t. Since the hypothetical belief in the question is about the belief at the current step, we estimate the belief in the past steps to form a full belief hypothesis. In this work, we assume a uniform distribution for the initial belief. We represent the agent’s observation as the subset of the state predicates that the agent can observe, and update the agent’s belief accordingly. This gives us an estimated agent belief b̂τ = b̂τ (sτ ) at each past step.
Based on Eqn. (2), we need to evaluate (1) the likelihood of the last action at given the hypothetical belief and goal (π(at|g, bt)), (2) the probability of a hypothetical belief at the last step (P (bt|b̂t−1, st)), and (3) the likelihood of all past actions given the hypothetical goal and the estimated belief prior to the current step ( ∏t−1
τ=0 π(a τ |g, b̂τ )). The computational bottleneck here is the policy. Conventional methods rely on planning or reinforcement learning to acquire such a policy. Inspired by the recent use of LLMs for decision-making (Huang et al., 2022; Li et al., 2022), we adopt a language model to amortize the policy. In particular, we symbolically represent the belief at each step as a list of possible locations of the corresponding goal object. We then prompt a language model with the symbolic representations of the state st, goal g, and estimated belief b̂t, and generate the likelihood of the observed action at based on the output logits. Figure 5 in Appendix B.2 illustrates how the likelihood estimation works in a qualitative example. We can finetune the language model on the ground-truth state, belief, goal, and action sequences in the training dataset.

heading: 5 Experiments

heading: 5.1 Baselines
text: Human baseline. We conducted a human experiment to validate our questions, and to evaluate human performance. Participants (N=180) were recruited online via Prolific. We randomly sampled 120 questions (20% of all questions) from all types.
Large language models (LLMs). We evaluate LLMs on the text-only version of MMToM-QA, including GPT-4 (OpenAI, 2023), GPT-3.5, GPT-J (6B) (Wang and Komatsuzaki, 2021), and LLaMA 2 (7B) (Touvron et al., 2023). We also evaluate SimToM (Wilf et al., 2023) and SymbolicToM (Sclar et al., 2023), two recent approaches that improve ToM in LLMs through better prompting. We apply them to GPT-4 and create two baselines: SimToM w/ GPT-4 and SymbolicToM w/ GPT-4. For all LLMs, we prompt them with the text input.
Large multimodal models (LMMs). We evaluate GPT-4V (OpenAI, 2023), InstructBLIP (Dai et al., 2023),Video-LLaMA 2 (Zhang et al., 2023), and LLaVA (Liu et al., 2023) in both multimodal and video-only conditions. For all LMMs, we uniformly sample a few frames from each video following prior works (Dai et al., 2023). We use the largest versions for all LMMs.
We finetuned GPT-J (6B) and LLaMA 2 (7B) and created two BIP-ALM models: BIP-ALM w/ GPT-J and BIP-ALM w/ LLaMA 2.

heading: 5.2 Results
text: We summarize the main results in Figure 4. On the multimodal version, humans achieve 93% accuracy averaging across question types. For each tested question, the majority of the participants chose the correct answer, validating our question designs. When only given unimodal data, human performance drops in accuracy overall, with video-only questions being harder to answer than text-only. While most questions suffer, the performance of text-based reasoning on the true-belief/false-belief
remains the same. This is of note as these questions have been the main target of text ToM QAs.
The baselines show performance that is close to random guessing across all types in all conditions, except GPT-4(V) and SymbolicToM w/ GPT-4 in multimodal and text-only conditions, as shown in Table 1. GPT-4(V) reaches human-level accuracy on Type 1.1 and shows competitive performance on Type 1.3. However, it also makes systematic mistakes in harder questions that involve false beliefs (Type 1.2). This suggests that GPT-4(V) can understand the true world state from the text but confuses belief with the true world state. GPT-4(V) also struggles with goal inference. Its accuracy on Type 2.3 is particularly low. We hypothesize that this is because it mistakenly thinks that the goal has to be one of the objects inside the container the person opens and fails to recognize that the person updates the belief after checking inside the container. SymbolicToM can improve GPT-4’s performance on a few types by removing irrelevant text, but it still struggles with harder types.
Our BIP-ALM models outperform all baselines by a large margin. Even without finetuning, as the ablated study in Appendix B.1 shows, our model with small pretrained LMs can already achieve better results than using much larger pretrained LMs
(e.g., GPT-4) alone. BIP-ALM also can flexibly conduct ToM reasoning with any unimodal or multimodal data thanks to the unified representations.
As reported in Appendix B.1, we examined the effect of few-shot or chain-of-thought prompting (Table 4) and model sizes (Table 5). We did not find any setting that can consistently improve a baseline’s performance in all types. We also finetuned Video-LLaMA 2 (13B) on our training set for video instruction tasks following Zhang et al. (2023). As Table 6 shows, the finetuned model performs moderately better in a few simpler question types (e.g., Type 1.1), but its overall performance is still not better than chance, unlike our method.
Generalization evaluation. We created an additional test set, the human test set, for generalization evaluation. It has 40 videos and 120 questions. To generate the videos in this set, we used 2 new apartments unseen in the training set and the main test set. We recruited 3 participants who had no prior exposure to the system to control the avatar to reach assigned goals via the human interface so that we could collect real human belief updates and human actions. We then used the same method to generate the questions. We report the model performance on this human test set in Table 7 (Appendix B.1). It shows that our method can generalize to both real
human behavior and unseen physical environments.

heading: 6 Discussion & Conclusion
text: We presented MMToM-QA, the first multimodal benchmark for machine ToM. We conducted a systematic evaluation of human performance, stateof-the-art methods, and our BIP-ALM model. We summarize the key findings as follows.
How does each modality contribute to ToM? From a video, a model gets the dynamic state change as well as what objects the agent is walking towards and is passing by at a given step. A model needs this information to determine the agent’s expected action plans given its mental state. Because of the partial observations caused by the limited camera view and occlusion, the text provides additional state information that is sometimes unavailable in the video. A model requires information about the true world state to determine an agent’s observation and whether it has a false belief. This is illustrated in Figure 6 in Appendix B.2.
Do LLMs and LMMs understand ToM? GPT4 and GPT-4V excelled on questions that only require retrieving information about the true world state. However, they still cannot reason about the mental state of a person and track the change in the mental state over time. We found more specifically that they have poor judgment on goals, which was not evaluated in existing text-based benchmarks.
What are the successes and failures of BIPALM? Instead of directly mapping the multimodal inputs to beliefs and goals, BIP-ALM conducts model-based inference by imaging possible actions given a hypothetical mental state and state context via language models. This results in a better performance in the main test set and enables the method to generalize to real human behavior in unseen environments. We also observed several failures. First, BIP-ALM cannot imagine missing state information from videos. Second, it uses symbolic state representations and only conducts symbolic planning. However, in Type 2.4, one needs to imagine the future path of a person given continuous spatial information such as the current path, facing direction, and the locations of potential goal objects. Finally, the action likelihood estimated by the LMs sometimes could be inaccurate due to LMs’ occasional failures in planning.
Limitations and future work. First, MMToMQA only includes videos of people looking for objects in household environments. In the future, we
would like to extend this to more diverse scenarios. Second, we intend to incorporate additional ToM concepts such as desires, emotions, and constraints in a future version of the benchmark. Finally, we intend to enrich the representations in BIP-ALM with broader relations and predicates, extending its reach even further to more complex scenes and human behaviors. This could be potentially achieved by finetuning larger LMs in broader datasets that are collected in simulators or crowdsourced (text descriptions of real-world human behaviors).

heading: Ethics Statement
text: The ability to understand humans’ mental states is a crucial foundation for building human-centered AI systems that can safely and cooperatively interact with humans. Benchmarking state-of-the-art machine learning models’ Theory of Mind capacity can provide us insights into whether current machine learning models can indeed adequately understand humans. We believe that our benchmark is a significant contribution towards this effort. Both our MMToM-QA benchmark and the BIP-ALM model are built on prior studies in cognitive science and thus are grounded in the cognitive theories of human behaviors. Such cognitively grounded benchmarks and models can help develop AI systems that are more aligned with humans’ social cognition. While we do not foresee any potential harm from our work, we recognize the need to ensure the diversity and fairness of our benchmark. We have made our best effort to increase the diversity of the avatars used to generate the videos. We have also validated our benchmark in a human experiment. We welcome feedback and suggestions from the community to further improve our benchmark.

heading: Acknowledgements
text: This work was supported by the DARPA Machine Common Sense program and a grant from Lockheed Martin.

heading: A Comparison of Theory of Mind Benchmarks
text: We provide a comparison of Theory of Mind benchmarks in Table 2, summarizing the evaluated ToM concepts, the size of the test set, available modalities of the inputs, the generation method, and the evaluation for each benchmark. From the table, we can see that our benchmark is the only one that provides multimodal inputs. Additionally, commonly used text-based ToM QA benchmarks do not evaluate goal inference, whereas questions in our benchmark ask about both goals and beliefs.

heading: B Benchmark Details

heading: B.1 More Quantitative Results
text: We conducted an ablated study to show the effect of finetuning language models for BIP-ALM. As Table 3 shows, finetuning GPT-J and LLaMA 2 significantly boosts the performance of BIP-ALM. It is also interesting to see that even before finetuning, BIP-ALM with pretrained GPT-J or LLaMA can already outperform much larger models including GPT-4. This further demonstrates the advantage of conducting inverse symbolic planning for multimodal ToM reasoning.
We evaluated open-sourced models (LLaMA 2, InstructBLIP, and Video-LLaMA 2) with different model sizes (Table 5 in Appendix B.1). All baselines performed no better than chance, regardless of the model sizes. As shown in Table 4 in Appendix B.1, we found no meaningful improvement for almost all baselines after using different few-shot or
chain-of-thought Kojima et al. (2022) prompting. Only GPT-4 in the text-only condition has an improvement in simple types (e.g., Type 1.3) with few-shot prompting. The accuracies for LLaVA (w/ 1-shot) and LLaVA (w/ 2-shot) are notably low. The model often fails to generate options a or b, as it faces difficulties in processing prompts of this length. We finetuned Video-LLaMA 2 (13B) on our training set for video instruction tasks following Zhang et al. (2023). As Table 6 shows, the finetuned model performs moderately better in a few simpler question types (e.g., Type 1.1), but its overall performance is still not better than chance.
We further evaluated the generalization of models on the human test set with real human behaviors in unseen environments (Table 7).

heading: B.2 Qualitative Results
text: Figure 5 demonstrates how the inverse symbolic planner enabled by the language model in our BIPALM model can estimate the likelihood of a given hypothesis. In Figure 5A, given the goal of getting a water glass, our model reasons that Elizabeth is more likely to open the microwave if she believes that there is a water glass inside the microwave, even though there is not any water glass inside the microwave according to the true world state. By imagining reasonable actions conditioned on hypothesized mental states, our model can successfully infer that Elizabeth has a false belief. In contrast, GPT-4 selects (b) as the more likely option, failing to recognize the false belief. Figure 5B depicts how the likelihood of a hypothesis changes after the model observes different actions. Specifically, in this case, the model first thinks that it is more likely that Karen is going to open the oven if the goal is to get a plate because there is a plate inside the oven. However, if the goal is truly to get a plate, at the next step, Karen should not close the oven but pick up the plate instead. If the goal is not to get a plate but a salmon, on the other hand, then Karen should close the oven and continue to look for salmon in other places. Therefore, the fact that Karen closes the oven without picking up the plate suggests that it is unlikely that her goal is to get a plate and that it is still quite possible that she wants to get a salmon instead. The action likelihood ratios estimated by our model at these two steps reflect this reasoning. Consequently, our model answers the question correctly.
Figure 6 illustrates how BIP-ALM may form different state information from different modalities
and how different state information may shape the policy estimated by our model. In particular, from the video, we know that the person is close to the microwave but we do not know where the cupcake is. So the policy conditioned on the state extracted from the video (Svideo-only) tries to guess where the person is going to look for the cupcake (e.g., cabinet). From the text, we know that there is a cupcake inside the microwave, but we do not know that the agent is already close to the microwave. Thus the policy conditioned on the state extracted from the text (Stext-only) predicts that the person is going to walk towards the microwave. After fusing information from both the video and the text, we then know the full state information. Conditioned on this fused state Smultimodal, the policy then predicts that the person is going to open the microwave, which is the ground-truth action of the person at the next step. This demonstrates that BIP-ALM can more accurately estimate the person’s policy when having access to both modalities, which explains why it performs the best in the multimodal
condition.

heading: B.3 Discussion on SimToM and SymbolicToM
text: The two recent approaches evaluated in our benchmark, SimToM (Wilf et al., 2023) and SymbolicToM (Sclar et al., 2023) have previously shown promising results in existing text-based ToM QA benchmarks. However, our experimental results demonstrate that there is still a large gap between their performance and the human performance on our MMToM-QA benchmark. We provide more discussions on these two recent approaches as below.
SimToM utilizes the concept of perspectivetaking to filter context based on what the charac-
ter in question knows before answering a question about their mental state. This approach marginally enhances the performance of GPT-4 on MMToMQA.
SymbolicToM (Sclar et al., 2023) constructs symbolic graphical representations of each character’s belief states, retrieves relevant sentences from the graph, and feeds them into an LLM to answer a given ToM question. As questions in MMToM-QA include only one person and exclude high-order beliefs, the method automatically extracts all sentences containing the relevant objects in the questions. This approach of filtering out useful information enhances the performance of LLMs (e.g., GPT-4).
On the other hand, the performance increase of SymbolicToM on MMToM-QA is not as significant as on the ToMi benchmark. The ToMi benchmark only has simple scenarios with a few objects and locations, and very short action sequences. In con-
stant, MMToM-QA demands the inference of mental states from long human activities in complex environments. Given the observations, the inference in MMToM-QA also has a varying degree of uncertainty. Therefore, in contrast to achieving
100% accuracy on the ToMi benchmark, SymbolicToM with GPT-4 has an overall accuracy of 63% on MMToM-QA. This is lower than the accuracy of BIP-ALM which uses much smaller language models.
In addition, unlike humans and our BIP-ALM model, SimToM and SymbolicToM cannot answer multimodal or video-only questions, since they are purely text-based methods.

heading: B.4 More Details About the Human Experiment
text: Each question in each condition (multimodal, textonly, or video-only) was answered by 5 participants. 180 participants were recruited via Prolific (mean age = 29.7; 65 female). They were paid $12 per hour. The study was approved by an institutional review board. All data collected through Prolific have been anonymized. The instructions and con-
sent form are shown below.
This is a study about how people interpret other people’s actions. We will ask you to answer questions based on typical household activities. The study usually takes 20 minutes. Consent By selecting the “I Agree” button below, you acknowledge that: You must be at least 18 years old to participate. Your participation in this research is voluntary. You may decline to answer any or all of the following questions by closing this window in your browser. You may decline further participation, at any time, without adverse consequences. Your anonymity is assured; the researchers who have requested your participation will not receive any personal information about you.

heading: B.5 Available Data
text: As shown in Figure 7, our benchmark provides RGB-D images, instance segmentation maps, human 3D poses, ground-truth scene graphs, groundtruth actions, and camera data.

heading: B.6 Benchmark Statistics
text: The environment in each question features an apartment that includes a bedroom, kitchen, living room, and bathroom. On average, each apartment contains 10.1 distinct types of containers (e.g., fridge, cabinets) and surfaces (e.g., desks, kitchen tables, coffee tables) types, and 16.4 instances of containers and surfaces. There are on average 12 different object types, summing up to approximately 26.3 object instances in an apartment.
Figure 8 provides an overview of the distribution of text and video lengths across all questions in our benchmark. In comparison to existing ToM benchmarks, MMToM-QA features a more extensive text and video context, with an average of 1595 tokens and 1902 frames, respectively. Such lengths demand advanced information retrieval and fusion capabilities. The longer visual and textural context also increases the difficulty of paying attention to the relevant information to reconstruct the mental state of a person.

heading: B.7 Details of the Procedural Generation
text: Figure 9 provides an overview of the procedural generation of questions in our benchmark. To procedurally generate videos, we first sample different apartments, the goal, and the initial state of an agent, then generate a sequence of actions using a planner. In particular, we formulate the agent in POMDP. The belief of the agent is represented as the probability of finding each object at a location. The agent can observe any objects
within the same room that are not inside closed containers. We adopt the same planner in Puig et al. (2020), which has been verified to be able to synthesize human-like plans in household environments. Given the action sequence, we then render the RGB-D video frames and record ground-truth data including the instance segmentation maps, 3D
human poses, scene graphs, actions, ground-truth agent beliefs, and the true goal.
To generate questions, we first use an “ideal observer” model to keep track of all possible goal and belief hypotheses at any given step. This model has access to the ground-truth observation of the agent at each step and eliminates the belief hypothe-
ses that are inconsistent with the observations. It also has access to the planner of the agent and will evaluate each remaining goal and belief hypothesis pair by simulating actions conditioned on the hypothesis. If the simulated actions are consistent with the actual actions taken by the agent, we then further eliminate that hypothesis. Finally, to generate a question of a certain type at a given step, we sample two hypotheses that fit the design of the question type, with one from the set of possible hypotheses produced by the ideal observer model and the other being a hypothesis ruled out by the model.
At any given moment in a video, we ensure that our benchmark poses at most one question. To achieve this, we randomly select one question from the pool of possible questions at every step. We further sample a subset of the remaining questions to achieve a balanced distribution of question types.

heading: B.8 Utilizing GPT-4 for Enhanced Text Generation
text: We first translate the symbolic representations of state and action descriptions into natural language using simple templates. We then use GPT-4 to enhance the phrasing and diversify the expression. The prompts are provided as follows.
Improving state descriptions: We are describing where things are in an apartment. Please improve the grammer of the description without changing the meaning. Please use only one line break to separate descriptions about each room. Original: There is a kitchen and a livingroom. 4 kitchencabinets are inside the kitchen. There is nothing inside the 1st kitchencabinet from left to right. There is nothing inside the 3rd kitchencabinet from left to right. There is nothing inside the 2nd kitchencabinet from left to right. A waterglass and a wineglass and 2 dishbowls are inside the 4th kitchencabinet from left to right. Improved: There is a kitchen and a living room. The kitchen has four cabinets. The first, second, and third cabinets, from left to right, are empty. The fourth cabinet, from the left, contains a water glass, a wine glass and two dish bowls. Original: {state description in the question}
Improved:
Improving action descriptions: Please improve the following descriptions about a person’s actions without changing the meaning. Original: [name] is in the kitchen, walktowards stove. Improved: [name] is in the kitchen. [name] walks towards the stove. Original: [name] is in the livingroom, walktowards kitchen, walktowards 1st kitchencabinet, open 1st kitchencabinet, close 1st kitchencabinet. Improved: [name] is in the living room. [name] walks to the kitchen, approaches the first cabinet, opens it, and then closes it. Original: {action description in the question} Improved:

heading: C BIP-ALM Implementation Details
text: C.1 Visual Perception
We adopt Blukis et al. (2022) to obtain a voxel map for each frame. Specifically, we first get the instance segmentation map from the RGB image. Combined with the depth map and the camera data, we create a 3D point cloud, with each point representing the pixel on the instance segmentation map. We group the 3D point cloud into a voxel map and then estimate the 3D bounding boxes of the objects. We can also estimate the 3D human pose in each frame. In the current experiments, we use ground-truth instance segmentation maps and ground-truth 3D human poses. In future work, we plan to also evaluate our model on segmentation and pose estimation results acquired from off-theshelf computer vision models.
Using the 3D bounding boxes and the 3D human pose, we construct a scene graph. Each node in the graph is either an object or a person. For nodes representing containers, we indicate whether they are open or closed (which in our scenarios can be detected from the change in the sizes of their bounding boxes). There are two types of edges in a graph – (1) inside edges indicating containment and (2) close edges indicating proximity. Note that our BIP-ALM model is not restricted to these relationships and can be applied to broader scenarios with more types of spatial relationships.

heading: C.2 Text Parsing
text: Utilizing GPT-4, we parse the provided question to extract representations about the state context, action context, question, and the two options. In this subsection, we detail all the prompts used.
To extract representations regarding the state context, we use the following prompt:
State Extraction: Please extract the description of the rooms and where things are in an apartment, found after the phrase “What’s inside the apartment” and before the description of a person’s actions. Keep the line breaks. Input: {question} Extracted:
State Parsing: Please parse the following description about where things are in an apartment. Each sentence should follow the pattern ‘[something] is/are in/on the [somewhere].’ Use a ‘.’ to separate the sentences, and keep the original line breaks.
Original: The living room contains a sofa, a desk, a cabinet, and a coffee table, and the cabinet holds chips, a wine glass, and an apple. Parsed: A sofa, a desk, a cabinet and a coffeetable are in the livingroom. Chips, a wineglass and an apple are in the cabinet.
Original: The kitchen has an oven, a microwave, and four cabinets. The oven contains a salmon, the microwave holds a cupcake, the third cabinet from the left has a wine glass, the fourth cabinet is empty. The first and second kitchen cabinets each holds a plate. Parsed: an oven and a microwave and 4 kitchencabinets are in the kitchen. A salmon is in the oven. A cupcake is in the microwave. A wineglass is in the 3rd kitchencabinet. Nothing is in the 4th kitchencabinet. A plate is in the 1st kitchencabinet. A plate is in the 2nd kitchencabinet.
Original: {extracted states}
Parsed:
To parse the human actions, we employ the following prompt:
Action Extraction: Please extract the exact description of a person’s actions (starting from the initial location), found after the phrase “[someone]’s action” and before the question. Please do not include the question, choices, or the answer. Input: {question} Extracted:
Action Parsing: Please parse the description of a person’s actions. Use a ‘.’ to separate each action, and remove all occurrences of the word ‘and’ in the description.
Original: Jennifer is in the bedroom. She proceeds to the kitchen and strides towards the oven, preparing to open it. Parsed: In the bedroom. walktowards kitchen. walktowards oven. about to open oven.
Original: Mark is in the bathroom. He then walks to the kitchen. He sequentially approaches the oven, the second, and third kitchen cabinets, opening and closing each one in turn. Parsed: In the bedroom. walktowards kitchen. walktowards oven. open oven. close oven. walktowards 2nd kitchencabinet. open 2nd kitchencabinet. close 2nd kitchencabinet. open 3rd kitchencabinet. close 3rd kitchencabinet.
Original: {extracted actions} Parsed:
To parse and analyze the question, we prompt GPT-4 to determine if a question falls under the “Belief Inference” or “Goal Inference” category, and extract all the hypothetical beliefs, hypothetical goals, and conditions in the question:
Question Parsing: Please determine the type of inference for the input question: either “Belief Inference”, which inquires about a person’s belief regarding an object, or “Goal Inference”, which seeks to understand a person’s objective. If a question falls under the “Belief Inference”, please identify the [object] and the [container] that the object may or may not be inside in choices (a) and (b). If a question falls under the “Goal Inference”, please identify the two possible objects that the person is looking for in choices (a) and (b). If the input contains a statement indicating that someone believes there isn’t an [object] inside a [container], please also identify both the [object] and the [container] mentioned. Otherwise, return ‘NaN.’
Input: ... (detailed descriptions about states and actions) ... If Elizabeth has been trying to get a plate, which one of the following statements is more likely to be true? (a) Elizabeth thinks that there is a plate inside the fridge. (b) Elizabeth thinks that there isn’t any plate inside the fridge. Output: Belief Inference. plate, fridge.
Input: ... (detailed descriptions about states and actions) ... If Jennifer has been trying to get a plate, which one of the following statements is more likely to be true? (a) Jennifer thinks that there is a salmon inside the oven. (b) Jennifer thinks that there isn’t any salmon inside the oven. Output: Belief Inference. plate, fridge. salmon, oven.
Input: ... (detailed descriptions about states and actions) ... Which one of the following statements is more likely to be true? (a) Mark has been trying to get a plate. (b) Mark has been trying to get a cupcake. Output: Goal Inference. plate, cupcake. NaN.
Input: ... (detailed descriptions about states and actions) ... If Mary think there
isn’t an apple inside the microwave, which one of the following statements is more likely to be true? (a) Mary has been trying to get an apple. (b) Mary has been trying to get a bottle of wine. Output: Goal Inference. apple, wine. apple, microwave.
Input: {question} Output:

heading: C.3 Representation Fusion
text: Figure 10 illustrates how BIP-ALM fuse complementary information extracted from the video and the text to form unified symbolic representations. In particular, the orange predicates can only be extracted from the video and the blue predicates can only be extracted from the text. After merging predicates extracted from both modalities, we can then construct a full state sequence.
When fusing information from different inputs, we may encounter conflicting predicates. In the case of conflict, we only keep the predicates from the text and remove the contradictory predicates from the video. This is because the predicates from the video are more likely to be erroneous due to noisy visual perception. However, more broadly speaking, such conflict-resolving mechanisms can be calibrated by the reliability of different modalities in a given domain.

heading: C.4 Belief Representation and Update
text: A person’s belief consists of beliefs about the locations of individual objects as illustrated in Figure 11. At each step, we estimate the person’s observation and update the estimated belief accordingly b̂t. Then for each step, we construct a symbolic representation of the belief about each object as a list of possible locations of the object that have non-zero probabilities in the belief.
Note that for representing the negation of a predicate, we can simply remove the location from the list of possible locations in the symbolic belief representation.

heading: C.5 Prompt for Language Models in Inverse Symbolic Planner
text: As introduced in the main paper, we employ a language model (either GPT-J or LLaMA 2) to amortize the policy and estimate the likelihood of the
person’s last action at given the hypothetical belief bt and goal g. We present the language model with the specified prompt below and then record its likelihood of predicting the accurate action.
goal: {hypothetical goal} state: {state} belief (possible locations the person suspects the {hypothetical goal} could be): {hypothetical or predicted belief} action:
For the belief inference questions, to represent a hypothetical belief that the goal object is not at a location L, we exclude that hypothetical location from the list of all possible locations in the prompt for the language model and subsequently estimate the action likelihood (π(at|g, bt)) conditioned on not believing that the object could be at that location. To compare that with the opposing belief, i.e., that the object could be at the location L, we assess the action likelihood conditioned on not believing that the goal object is at an alternative location L′ by removing L′ from the possible location list in-
stead. If the removal of the hypothetical location (L) results in the lowest action likelihood compared to all alternatives (L′), then it is more likely that the person does believe that the goal object could be at this location (L).

heading: C.6 Training Details
text: In the experiments, we finetuned GPT-J (6B) and LLaMA 2 (7B) for our BIP-ALM method. Adhering to our evaluation protocol, we employed the ground-truth state, belief, goal, and action from 1,000 training videos, excluding the use of any example QAs. The models were trained using the state, belief, and goal as inputs at specific timestamps, with the aim of predicting the corresponding action. This approach enhanced the Inverse Symbolic Planner’s ability to estimate the likelihood of human actions at a specific moment conditioned on the hypothesis about the goal and belief. The training data comprised 20,000 samples, consistent with the input and output of the Inverse Symbolic Planner defined in Appendix C.5.
For finetuning, we incorporated the Low-Rank Adapters (LoRA) method (Hu et al., 2021). The
training process leveraged the AdamW optimizer, with a set learning of 5 × 10−5 and a batch size of 4. We trained both models for 5 epochs, which took about 20 GPU hours on a single A100 GPU.
D Implementation Details of Baselines
In all baselines, we use gpt-4-0613 for GPT-4 and text-davinci-003 for GPT-3.5.
We sample a few frames from each video for LMMs, following the standard settings. In particular, we sample 16 frames, 30 frames, 6 frames, and 8 frames from each video for InstructBLIP, VideoLLaMA 2, LLaVA, and GPT-4V respectively. We use Vicuña-13B for InstructBLIP and LLaMA-213B-Chat for Video-LLaMA 2,
Finetuning Video-LLaMA 2. We followed Zhang et al. (2023) to create a video instruction dataset using our training data to finetune VideoLLaMA 2. In particular, based on our training videos and the ground-truth annotations, we generated 7,088 training examples for describing either the environment or the actions of the person in a given video clip. For each video clip, we sampled 8 frames and used the ground-truth scene graph at the last frame or the action sequence during the whole clip to generate the ground-truth descriptions for the training data. We then finetuned the VideoLLaMA 2 (13B) model on our training data on 2 A100 GPUs.

heading: E Full Version of the Example Questions in Figure 2

heading: E.1 Type 1.1 Example
text: The video input: https://youtu.be/ 4zoDwQk91ak.
The text input:
What’s inside the apartment: The apartment consists of a bedroom, kitchen, living room, and bathroom. In the bedroom, there is a coffee table and a desk, with a remote control resting on the coffee table. The kitchen is equipped with four cabinets, a fridge, a kitchen table, a microwave, and an oven. The first and third cabinets, from left to right, are empty, while the second cabinet houses a condiment bottle. The fourth cabinet contains a water glass. Inside the fridge, you’ll find a bottle of wine, a
dish bowl, and two plates. The microwave holds a cupcake, and the oven contains a salmon, two cupcakes, and a plate. The living room features a cabinet, a sofa, a coffee table, and a desk. The cabinet is filled with two plates, a bottle of wine, two wine glasses, a condiment bottle, a water glass, a bag of chips, and an apple. A remote control, a wine glass, and a book are placed on the coffee table. Lastly, the bathroom has a cabinet, which is currently empty.
Actions taken by Elizabeth: Elizabeth is initially in the bathroom. She then proceeds to the kitchen and heads towards the oven. After opening and closing the oven, she moves to the second kitchen cabinet, opens it, and then shuts it. She repeats this action with the third and first kitchen cabinets. Subsequently, she walks towards the fourth kitchen cabinet, opens it, and then closes it. Finally, she moves towards the fridge, preparing to open it.
Question: If Elizabeth has been trying to get a bottle of wine, which one of the following statements is more likely to be true? (a) Elizabeth thinks that there is a bottle of wine inside the fridge. (b) Elizabeth thinks that there isn’t any bottle of wine inside the fridge.
Correct answer: a.

heading: E.2 Type 1.2 Example
text: The video input: https://youtu.be/mgvh1lY_ Z38.
The text input:
What’s inside the apartment: The apartment consists of a bedroom, kitchen, living room, and bathroom. In the bedroom, there is a coffee table and a desk, with three wine glasses and a dish bowl placed on the coffee table. The kitchen is equipped with four cabinets, a fridge, a kitchen table, a microwave, and
an oven. The first, second, and fourth cabinets, from left to right, contain a dish bowl each, while the third cabinet houses a plate. The fridge contains two apples, a dish bowl, and a salmon. The microwave holds two cupcakes, and there is a salmon in the oven. The living room features a cabinet, a sofa, a coffee table, and a desk. The cabinet is filled with a plate, a bag of chips, a water glass, a remote control, a bottle of wine, and a condiment bottle. Lastly, the bathroom has a cabinet, which is currently empty.
Actions taken by Jennifer: Jennifer is situated in the living room. She heads towards the cabinet and is about to open it.
Question: If Jennifer has been trying to get a cupcake, which one of the following statements is more likely to be true? (a) Jennifer thinks that there isn’t any cupcake inside the cabinet. (b) Jennifer thinks that there is a cupcake inside the cabinet.
Correct answer: b.

heading: E.3 Type 1.3 Example
text: The video input: https://youtu.be/ lN810N3KdjM.
The text input:
What’s inside the apartment: The apartment consists of a bedroom, kitchen, living room, and bathroom. In the bedroom, there is a sofa with a book on it and a cabinet containing a remote control, a wine glass, two dish bowls, a bottle of wine, and a condiment bottle. The kitchen is equipped with a fridge, sofa, dishwasher, eight cabinets, an oven, a microwave, and a kitchen table. The fridge contains an apple, three plates, and a bottle of wine, while a bag of chips rests on the sofa. Inside the dishwasher, there is a plate, a water glass, and a wine glass. The first
to the seventh cabinets, from left to right, are all empty. However, the eighth cabinet houses a wine glass. The oven contains a salmon, the microwave is empty, and the kitchen table is adorned with a plate, a wine glass, two apples, two books, and a cupcake. The living room features a sofa with a water glass and a book on it, and a desk. Lastly, the bathroom has a cabinet, which is currently empty.
Actions taken by Charles: Charles is in the kitchen. He walks to the seventh kitchen cabinet, opens and closes it. He repeats the same action with the sixth kitchen cabinet. Subsequently, he moves towards the dishwasher.
Question: If Charles has been trying to get a salmon, which one of the following statements is more likely to be true? (a) Charles thinks that there is a salmon inside the fridge. (b) Charles thinks that there isn’t any salmon inside the fridge.
Correct answer: b.

heading: E.4 Type 2.1 Example
text: The video input: https://youtu.be/ NsOPbJWPn1c.
The text input:
What’s inside the apartment: The apartment consists of a bedroom, a bathroom, a living room, and a kitchen. In the bedroom, there is a coffee table with a plate on it. The bathroom houses a cabinet, which is currently empty. The living room is furnished with a cabinet, a coffee table, a sofa, and a desk. The cabinet is filled with two apples, a condiment bottle, three wine glasses, two water glasses, a cupcake, two bags of chips, a remote control, and a bottle of wine. Both a water glass and a wine glass are placed on the coffee table.
The kitchen is equipped with a fridge, an oven, a kitchen table, and a microwave. Inside the fridge, there are two apples. The oven contains a salmon. Meanwhile, the microwave houses a salmon and two cupcakes.
Actions taken by James: James is in the kitchen. He strides towards the stove, opens it, and then shuts it. He then opens the fridge, closes it, opens the microwave, and closes it as well. Finally, he walks towards the living room and approaches the cabinet.
Question: Which one of the following statements is more likely to be true? (a) James has been trying to get a bottle of wine. (b) James has been trying to get an apple.
Correct answer: a.

heading: E.5 Type 2.2 Example
text: The video input: https://youtu.be/ Fn6s47ZtxMQ.
The text input:
What’s inside the apartment: The apartment consists of a bedroom, bathroom, living room, and kitchen. In the bedroom, there is a sofa, a cabinet, a desk, and a coffee table. A book rests on the sofa. The cabinet contains a remote control, three cupcakes, a wine glass, an apple, and a bag of chips. The coffee table holds two books and a dish bowl. The bathroom houses a single cabinet, which is currently empty. The living room is furnished with a sofa, a desk, and a coffee table. A dish bowl and a book are placed on the sofa, while a plate sits on the coffee table. The kitchen is equipped with a dishwasher, an oven, a kitchen table, eight cabinets, a microwave, and a fridge. Inside the dishwasher, there is a dish bowl. The oven contains a salmon and a plate. The second kitchen cabinet from the left holds an apple, while the fourth and fifth cabinets
contain two dish bowls and another apple respectively. There is a water glass inside the seventh cabinet. The first, third, sixth, and eighth cabinets are empty. The microwave contains a condiment bottle. The fridge stores two cupcakes, a dish bowl, a plate, and a bottle of wine.
Actions taken by Mark: Mark is in the kitchen. He then advances towards the seventh kitchen cabinet.
Question: If Mark thinks there isn’t a water glass inside the 7th kitchen cabinet, which one of the following statements is more likely to be true? (a) Mark has been trying to get a water glass. (b) Mark has been trying to get a cupcake.
Correct answer: b.

heading: E.6 Type 2.3 Example
text: The video input: https://youtu.be/ IUJW6Zv0EWA.
The text input:
What’s inside the apartment: The apartment consists of a bedroom, bathroom, living room, and kitchen. In the bedroom, there is a cabinet and a sofa. The cabinet contains a condiment bottle, an apple, two wine glasses, and a plate. The sofa holds three books. The bathroom features a cabinet, which is currently empty. The living room is furnished with a desk and a sofa, with a book resting on the sofa. The kitchen is equipped with eight cabinets, a sofa, an oven, a fridge, a kitchen table, a microwave, and a dishwasher. The first kitchen cabinet, from left to right, contains a bag of chips. The second and fourth cabinets are empty. The third cabinet houses a wine glass and a dish bowl. The seventh cabinet stores two plates. The fifth, sixth, and eighth cabinets are empty. The oven contains a cupcake. The fridge holds a plate and a dish bowl. The kitchen table
is adorned with an apple, a bottle of wine, a plate, and a water glass. The microwave contains a condiment bottle and a salmon. Lastly, the dishwasher has a water glass inside.
Actions taken by Mary: Mary is situated in the living room. She proceeds towards the kitchen and heads to the second kitchen cabinet. She opens it, then promptly closes it. She then opens the fourth kitchen cabinet and closes it as well. Following this, she opens the dishwasher and closes it. She then moves towards the sixth kitchen cabinet, opens it, and closes it. She repeats this action with the seventh kitchen cabinet. Finally, she walks towards the first kitchen cabinet, opens it, and then closes it.
Question: Which one of the following statements is more likely to be true? (a) Mary has been trying to get a bag of chips. (b) Mary has been trying to get a condiment bottle.
Correct answer: b.

heading: E.7 Type 2.4 Example
text: The video input: https://youtu.be/Y4H_ 9cXR5mw.
The text input:
What’s inside the apartment: The apartment consists of a bedroom, bathroom, living room, and kitchen. In the bedroom, there is a sofa, a cabinet, a desk, and a coffee table. A book rests on the sofa. The cabinet houses an apple, a wine glass, two books, and two cupcakes. The coffee table holds a book, a water glass, a wine glass, and a remote control. The bathroom contains a single cabinet, which is currently empty. The living room is furnished with a sofa, a coffee table, and a desk. A water glass sits on the sofa, and a remote control is on the coffee table. The kitchen is equipped
with eight cabinets, a microwave, a fridge, a dishwasher, a kitchen table, and an oven. The fourth and seventh cabinets from the left, as well as the eighth, are empty. The microwave contains a salmon, a cupcake, and a condiment bottle. The fridge is stocked with two bottles of wine, a cupcake, an apple, and two dish bowls. The dishwasher holds a dish bowl, a wine glass, and a plate. The second cabinet from the left contains a water glass. The first cabinet from the left holds a bag of chips and a wine glass. The fifth cabinet has an apple, and the third cabinet contains a condiment bottle. The sixth cabinet is empty. Lastly, there is a salmon in the oven.
Actions taken by William: William is situated in the kitchen. He advances towards the first kitchen cabinet, opens it, and then shuts it. Finally, he moves towards the fifth kitchen cabinet.
Question: Which one of the following statements is more likely to be true? (a) William has been trying to get a wine glass. (b) William has been trying to get a dish bowl.
Correct answer: b.

heading: Future Work
text: In future work, we plan to also evaluate our model on segmentation and pose estimation results acquired from off-theshelf computer vision models. Using the 3D bounding boxes and the 3D human pose, we construct a scene graph. Each node in the graph is either an object or a person. For nodes representing containers, we indicate whether they are open or closed (which in our scenarios can be detected from the change in the sizes of their bounding boxes). There are two types of edges in a graph – (1) inside edges indicating containment and (2) close edges indicating proximity. Note that our BIP-ALM model is not restricted to these relationships and can be applied to broader scenarios with more types of spatial relationships