ECHo: A Visio-Linguistic Dataset for Event Causality Inference via Human-Centric Reasoning
Authors: Yuxi Xie
Guanzhen Li
Min-Yen Kan
Venue: EMNLP 2023 Findings (2023)
Keywords: visio-linguistic commonsense reasoning
theory of mind
chain of thought
TL;DR: We introduce ECHo (Event Causality Inference via Human-Centric Reasoning), a visual-and-linguistic dataset of Theory-of-Mind inferences grounded in multimodal social scenarios.

Abstract: We introduce ECHo (Event Causality Inference via Human-Centric Reasoning), a diagnostic dataset of event causality inference grounded in visio-linguistic social scenarios. ECHo employs real-world human-centric deductive information building on a television crime drama. ECHo requires the Theory-of-Mind (ToM) ability to understand and reason about social interactions based on multimodal information. Using ECHo, we propose a unified Chain-of-Thought (CoT) framework to assess the reasoning capability of current AI systems. Our ToM-enhanced CoT pipeline accommodates various large foundation models in both zero-shot and few-shot visio-linguistic reasoning. We use this framework to scrutinize recent large foundation models such as InstructGPT and MiniGPT-4 on three diagnostic human-centric tasks. Further analysis demonstrates ECHo as a challenging dataset to expose imperfections and inconsistencies in reasoning. Our data and code are publicly available at [https://github.com/YuxiXie/ECHo](https://github.com/YuxiXie/ECHo).

ECHo: A Visio-Linguistie Dataset for Event Causality Inference via Human-Centric ReasOning
Yuxi Xie Guanzhen Li
Min- Yen Kan
National University of Singapore
{xieyuxi,
guanzhenn@u.nn nus.eeeeuuuuuee.mmm mus.edu
knmnynenus. edu.
Abstract
We introduce C ECHO (Event Causality In- ference via Human-Centric Reasoning), di- agnostic dataset of event causality inference grounded in visio-linguistic social scenarios. ECHO employs real-world human-centric de- ductive information built television crime drama. ECHO requires the Theory-of-Mind (ToM) ability to understand and reason about social interactions based multimodal infor- mation. Using ECHo, we propose a unified Chain-of-Though (CoT) framework to assess the reasoning capability of current AI systems. Our ToM-enhanced CoT pipeline accommo- dates various large foundation models in both zero-shot and few-shot visio-linguistic reason- ing. We use this framework to scrutinize recent large foundation models such as InstructGPT and MiniGPT-4 on three diagnostic human- centric tasks. Further analysis demonstrates ECHO as challenging dataset to expose im- perfections and inconsistencies in reasoning. Our data and code are publicly available at https://ght //gttbbb thub. com/YuxiXie/ECHo
Grissom standss in front of Palge and Gina.
We're ruking out surcide. The evidence leads
Grissom
homicide.
Paige closes her moment, Gina stands behind her, holding
murdered?
Paige
Grissom
believe so.
calnness
may funny
sympathy
sodhess
didn't take his own life
knowing that
Parge
Theory-of-No
Chain-of of Thought
Grissom sympothises with Paige as seed upset she over the desth her loved one
We'll rind him, always
Grissom
Figure I: Scheme of our Theory-of- enhanced Chain-of-Thoure reasoning on human factors. We use key characters visual representations.
gence (Goertzel, 2014; Zhong al., 2023), espe- cially towards better social intelligence.
Recently, large language (Chowdhery et al., 2022; Chung et al., 2022; Touvron et al., 2023; OpenAI, 2023) and multimodal (Radford et al.. 2021; Alayrac et al., 2022; Li et al., 2022; Huang et al., 2023) models have exhibited remarkable rea- soning capabilities. However, current large foun- dation models still fall short of adapting per- sonalized scenarios for specific users (Sap et al., 2022; Bubeck et al., 2023). Hence, there has been an increasing focus on human-centric reasoning as means of enhancing artificial social intelli- gence for its integration in human daily life (Bard et al., 2020; Yuan et al., 2020; Moghaddam and Honey, 2023). To this end, ToM is one of the central challenges accelerating communication and ensuring safety in human-computer interac- tion, requiring complex reasoning on how human beliefs and intents may vary across different sce- narios (Yuan et al., 2020; Sap et al., 2022; Jin et al.,
1 Introduction
Social intelligence refers to the ability to under- stand, navigate, and respond effectively in social scenarios. As widely investigated concept in psy- chology and social sciences, it has become promi- nent facet of artificial intelligence (Walker and Fo- ley, 1973; Kihlstrom and Cantor, 2000; Albrecht, 2006; Zadeh et al., 2019). In the development of social intelligence, humans gain the crucial cogni- tive capacity of understanding and reasoning about the mental states (i.e., beliefs, desires, intentions, emotions, and thoughts) of individuals. This piv- otal form of competence is commonly denoted as Theory-of-Mind (ToM) (Premack and Woodruff, 1978; Apperly and Butterfill, 2009; Apperly, 2010). As fundamental ability of social commonsense reasoning (Davis, 2023), ToM is important for ma- chine reasoning to achieve artificial general intelli-

---

2022; Sileo and Lernould, 2023). Specifically, cur- rent AI systems struggle with handling interleaved multi-modalities and faithful surmising for better consisteney and interpretability of reasoning (Lyu et al., 2023; Bubeck et al., 2023).
in zero-shot and few-shot ToMM-enhnnced CoT rea- soning. Our further analysis demonstrates how to use our diagnostic tasks to assess multimodal understanding of human factors, revealing the defi- ciency of current AT systems in maintaining logical correctness and consistency throughout reasoning.
To enhance reasoning consistency and faith- fulness, recent works on large language models (LLMs) propose break down problem into intermediate inferences (Wei al., 2022; Zhou et al., 2022; Xie et al., 2023). The impressive empirical success of this Chain-of- Thouuhh (CoT)
Related Work
ECHO takes further step towards social in- telligence on human-centric inference in visio- linguistic scenarios, probing the ToM capacity of large foundation models via CoT reasoning.
scheme on both textual and multimodal tasks (Wei et al.. 2022; Zhang et al., 2023; Driess et al., 2023) demonstrates a promising paradigm to integrate intermediate step in human- centric reasoning. In this work, we introduce C ECHO (Event Causality Inference via Human- Centric Reasoning), a visio-linguistic dataset with ToM inferences for reasoning in social scenarios. With a focus on human factors, ECHo seeks to diagnose the social intelligence of current large lan- guage and multimodal models. We focus on event causality reasoning that remains challenging for recent LLMs (Kiciman et al., 2023).
Visio- Disiu-Linguistic Reasoning. Datasets and tasks in visio-linguistic reasoning span widely from de- scriptive information extraction (Antol et al., 2015; You et al., 2016; Gao et al., 2017), physical re- lation inference (Johnson et al., 2017; Hudson and Manning, 2019), to complex and deep rea- soning on the event and human factors (Krishna et al., 7017; Zellers al., 2019; Park et al., 2020). ECHO follows this trend to enhance the reason- ing depth towards human-specific facets. Unlike recent works (Shen et al., 2020; ?: Zhu et al., 2023c) of human-centric reasoning, ECHO is inte- grated with rigorous ToM annotations, supporting the final inferences via CoT reasoning for better consistency. Specifically, there long-standing arguments on the development and assessment of ToM in both human psychology and machine intelligence (Premack and Woodruff, 1978; Ap- perly, 2010; Kosinski, 2023; Ullman, 2023). Mea- surement of ToM is usually based on false belief tasks (Dennett, 1978), assessing the ability to dis- tinguish one's 's own (true) belief and others' (false) belief, given the information and experience asym- metry among different individuals. Constructed on crime drama, ECHO contains an abundance of such cases of false belief to probe ToM ability.
ToM inference
We envision ECHO as challenging diagnos- tic benchmark on human-centric reasoning. Each ECHO instance is grounded in a plot from the crime drama CSI: Crime Scene Investigation, en- abling approximation of the real-worid social in- teractions pertaining to the discordance in human beliefs. As shown in Figure 2, our core annotation process begins with ascertaining the identity of specified character. Next, we discern their mental states via emotion interpretation. Leveraging this human-centti understanding, annotators then in- fer the cause or effect of a plot event for causality reasoning. To foster visio-linguistic social intel- ligence, we enhance ToM by guiding annotators to make causal inferences that take into account the mental states (e.g., intentions, emotions, and thoughts) of characters and pinpoint related frames as visual evidence. As such, ECHO is integrated with a unified framework to assess human-centric reasoning in the social context. As detailed in Sec- tion 4, we propose a series of diagnostic tasks to evaluate capabilities to identify roles, reason about emotions, and infer event causality.
Large Multimodal and Language Models. Pre- vious research in this area mainly complies with the paradigm of pre-training and fine-tuning to construct and train large-scale multimodal mod- els to handle interleaved iisual-and-limg linguistic in- formation (Radford et al., 2021; Jia et al., 2021; Zellers et al., 2021; Alayrac et al., 2022; Li et al., 2022; Huang al., 2023). Recently, there is the emergence of offline methods which leverage the capacities of large foundation models to conduct direct few-shot or zero-shot inference (Wu et al., 2023; Yang et al., 2023; Lu et al., 2023; Zhu et al.,
To conclude, we introduce ECHO, challenging lingo-lin corpus of human-centric reasoning in social scenarios. We propose a unified frame- work to evaluate existing large foundation models

---

Official Videos
Stage 1. Characteristics Identification
Grawled Screenplays
HOLLY is Juntor investigator, affillated to Grissom In her youth (25-551
occupation
oge
Automatic Plot Segmentation
Stage
Keyframe-Grounde Grounded Emotion Interpretation
Plot Instonce to Annotate
Intort D
Angen
Cármac O
Sympethy
Toe
Sugris
check
corresponding frames in the video clip?
Stage Event Causality Inference
? probably carl
event
HOLLY gets
Holly Gribbs:
Pretty good.
Grissom: Okay, look, routine rabbery. prints, check the videotape take lots of fun photos. I'll be back in about an hour to you
HOLLY, as junior. listened Grisso''s advice and look deep breath to prepare herself ready the job store.
Holly Gribbs: Okay.
[HOLLL out of the car. Grissom: you get done early channel your walkie.
Post Inference Validation
(GRISSOM pulls out of the parking lot.) (HOLLY heads inside the liquor store.)
Figure 2: Annotation Pipeline of the dataset construction. We detail the second round annotation, where annotators provide ToM inferences in the first two stages following with the event causality inference.
2023a). Similar with Zhu al. (2023a), we pro- pose a framework to enhance visual understanding via LLM prompting (Ouyang et al., 2022; Ope- nAI, 2023) and facilitate LLM reasoning with aug- mented multimodal information (Li et al., 2023).
tors working over weeks after training sessions
Data Source Crawling and Preprocessing. We acquire the official CSI videos with associated screenplays of 177 episodes from the first 8 sea- sons. We construct ECHO via further annotation and task formulation on subset of CS/ containing 15 episodes, each of which usually features one or two cases that are independent from the preceding plots. We develop heuristic rules to automatically denoise, split, and categorize the scripts into plot events for subsequent task formulation.
3 The ECHo Corpus
ECHO contains 2k inference instances collected via our ToM-enhanced CoT scheme. To facilitate the approximation of authentic social interactions, we ground ECHO in CSI: Crime Scene Investi- gation, an American procedural forensics crime series in English (Wikipedia contributors, 2023). With visual evidence and scenes frames and ut- terances and narrations in sereenplays, CSI repre- sents rich multimodal source, spanning widely factual, relational, and inferential data. As shown in Table 2, drama plots bring abundant instances of belief discrepancy and unexpected content for ToM reasoning. This enables our focus on human- centric information distillation and interpretation in ECHO's construction for ToM-enhanced inference.
Round One: Plot Segmentation. The crawled screenplays are distributed discrete plots. We re- fine this segmentation with data cleaning to collect segments of feasible length and substantive con- tents. Diffcrent from previous works of automatic vision-language alignment (Myers and Rabiner, 1981; Frermann et al., 2018), manually synchro- nize the screenplays with the time-stamped video clips pinpoint the main characters for human- centric reasoning. We obtain 1, 542 plots grounded in different scenes in this round, with an average of 3 identified characters in each segment.
3.1
Construction Pipeline
We pair official CSI clips with their screenplays, crawled from publicly available website hosting TV show transcripts' (Frermann al., 2018). We then launch annotation in 3 rounds with 30 annota-
Round Two: Inference Annotation. Each an- notation instance features one specified event for causality inference. We sequentially operationalize
2 AAppendix A details annotator training.
'https://transrrt foreverdremmiinn org/
Details at https: hts:://fi thub. com/ Yuxi Xie//CHo

---

Frame
# Character
Inference
# Clip
100
# Clip
total
featured
total
labeled
effect
FB (44%)
25
19
17
(36%)
1. 292
13, 306
12, 201
4,319
2,017
369
I, 0,3
C/E
cause (56%)
effect (44%)
Task
Instance
Annotation
Avg-Length
FB
objective (52%)
subjective (48%)
Role
2. 017
4, 226
3.43
UC
physical (69%)
social (31 %))
Emotion
2. 017
2,2
2.44
Event
2. 382
4, 280
17.97
Table 2: ToM attribute distribution on 100-
clip subset. FB and UC represent false-belief and umexpected-comee respectively.
Table I: Summative statistics of BCHo.
Pare
sestto
Jey
cf mnees
collagee
need
Puberce
9,2%
21.4%
surpaie
Hoube
Inderss
(a) common roles
(b) emotions
(c) screenplays
(d) causality inferences
Screenplay Agooth terolegn
female
Sympalny (4.4
disqust (4.9%)
mate
sympatiyy
Clp Durstion (minuta))
Suppect 8%6
(e) screenplay and clip length
(f) role iite-emotion co-oceurrence frequencies
Figure 3: ECHo statistics and varieties of keywords. (a) and (b) show the distributions of the 12 most frequent roles and 13 emotions, respectively. (c) and (d) plot the top 10 most common root verbs (inner circle) and their top 3 direct noun objects (outer circle) in screenplays and event causality inferences, respectively. (f) demonstrates the coverage of role types associated with various emotions of equivalent proportions for fair comparison.
annotation in 3 stages as follows:
not strictly enforce one-to-one matching between frames and emotions, since one frame can feature several emotions, and some emotions may be more accurately captured by considering the reactions of other characters. To ensure the completeness and informativeness of selected keyframes, we imple- ment follow-up validation next in Round Three 3. Event Causality Inference. Following the lisouiing human-centric inferences from the previous stages, we ask annotators further infer the cause or effect of a specified event. We en- courage them to consider the annotated roles and emotions to enhance reasoning consistency. Here we determine the events to annotate through two steps: 1) randomly select utterances or narrations that mention the main character(s) and occur in the middle of the plot facilitate effective causal- ity reasoning, and 2) filter out the automatically selected events that are insufficiently meaningful according to annotators' assessment.
1. Characteristics Identification. Given one key character, annotators identify the character's role in the plot. We encourage them consider social attributes such as age range, occupation, and rela- tions (with others) to describe the role. Character roles are not static and can vary in different plots, depending on the nuances of their appearances, be- haviors, and interactions in the specific context. Grounded Emotion Interpretation. We take further step in human-centric reasoning interpret mental states. Annotators choose from 13 primary emotions* categorized by adapting the 27 emotions from Cowen and Keltner (2017) to the crime drama. We also accept free-form input for emotions when no existing options apply. An- notators then extract associated frames that fea- ture related emotions. We take the frames as vi- sual representations of human factors. Specifically, when multiple emotions are identified, annotators are also instructed to select more frames. We do
2. Keyframe- Geyframe-Grounded
We assign to annotators for each instance for quality control. In this round, we collect a total of
"Including anger; boredom; calmness; disgust; doubt; en- trancement; fear; interest; joy: sadness; shame; surprise; sym- pathy. We provide detailed definitions in Appendix C.
For some instances, we additionally assign more annota- tors when the inter-agreement is low.

---

5, 746 annotated instances. We use the ToM infer- ences formulate diagnostic tasks in Section 4.
Notation. Each ECHO instance consists of sequence of visual frames V [fi, f2. fxl SI.N, a textual screenplay T. The frames are manu- ally selected following the role and emotion identi- fication in annotation. We gather frames featuring different characters together to represent the visual content of each clip. We designate each utterance or narration in the text to be an event E, for the causal- ity inference, as T [E1; F2,, .. : Ear] There is key character C' to focus on each instance.
Round Three: Inference Validation. To qualify the annotated instances, we evaluate each data point via both automatic and manual checking. The real- time automatic checking alerts annotators if their inputs fail meet certain informativeness criteria, such as text length and word-level overlap with the existing context. Our authoring team then carry out the manual validation. We particularly focus on instances with lower inter-rater agreement in emo- tion identification or anomalies in the timestamp distribution of selected frames. We specifically as- sess the plausibility, relevance, and completeness of annotations. For example, we reject instances where the event causality inference is weakly as- sociated with either the plot or the annotated roles and emotions. Out of the 5, 746 annotations col- lected from Round Two, we finally retained 4, 280 annotations, as shown in Table 1.
Task One: Role Identification (cf. Annotation Round Two, Stage 1). The psychoanalysis of a person's role in social interactions indicates their identity, helping to infer their intentions, actions, and relations with others (Miller, 1962; Freese and Burke, 1994). Therefore, we test the ability of role identification to probe the fundamental human- centric understanding in ToM reasoning. Given frame(s) fa of the key character C and the corre- sponding screenplay T. we prompt the model to generate the role of C. The role can be defined by age, occupation, or relations with others, as these attributes can exhibit a strong correlation with the human mental states for ToM reasoning.
3.2 Dataset Exploration
Table I and Figure 3e detail the summative statis- ties of collected data for the three tasks. The com- paratively small scale of ECHo enables efficient assessment in the few-shot paradigm. With 4k ToM annotations on 2k inference instances, ECHO approximates authentic human-centric reasoning across visio-linguistic social scenarios. While our ECHO represents a subset of social interactions, we view it as an initial and specific step to explore ToM understanding of social intelligence. As outlined in Table 2, crime drama is rich in ToM-related cases, including false-belief and unexpected-comte
Task Two: Emotion Interpretation (ef. Annota- tion Round Two, Stage 2). Emotions convey clues of mental states beyond verbal messages (Hari and Kujala, 2009). They bridge fundamental under- standing (e.g., role identification) and further in- ference (e.g., intent prediction) in human-centric reasoning. We thus propose emotion interpretation as our second diagnostic task. We formulate this task multi-choice question answering and test the alignment of model and human predictions on 13 candidate emotions, adapted from the laxonomy of Cowen and Keltner (2017) to the crime data.
We further demonstrate the topic and scenario coverage of ECHO in Figure 3, by visualizing the keyword and verb noun frequencies in both in- put screenplays and human annotations. Alone other lines, Figure 3f shows how the correlation be- tween roles and emotions vary in ECHO, reflecting closer alignment with the real-world social char- acteristics compared against other datasets. We an- alyze the potential bias in our data in Section 6.5.
Task Three: Event Causality Inference (cf. An- notation Round Two, Stage 3). Despite the practi- cal success of large foundation models on a wide range of reasoning tasks, there debate as to whether they genuinely execute causal reasoning or just reproduce memorized patterns (Bender et al., 2021; Marcus, 2022). Furthermore, these models still produce imperfections such as erroneous logic and human- factor understanding (Ghazal et al., 2017; Bubeck al., 2023; Zhong et al., 2023; Kici- man al., 2023). Hence, we formulate a subse- quent task as event causality inference to assess the causality reasoning capacity among socially- grounded events. We also utilize the ToM infer-
4 Diagnose Human-Centric Reasoning in Visio-Linguistic Inference using ECHO
ECHO centers on rigorous human-centric infor- mation that supports ToM-enhanced CoT reasoning in visio-linguistic scenarios. We detail the formu- lation of our three sequential tasks to diagnose the human-centric reasoning ability next.

---

[Causality Question] What caused/enabled event Grissom says: you done early use channel seven your walkie?
man and woman sitting car at night. man black shirt and woman white shirt. man's eyes
M captioning
staring off space.
is stops the car in front of the liquor store in the given video cllp? What is the expression on Gil. Grissom's s face when he stares off?
occupation man
question
generation
Holly Gribbs: Pretty good.
M
Grissam is police officer or law enforcement officer. He into distance with expression his face.
Good. Okay, this routine robbery. Dust for prints, check the videotage take
Grissom:
[Role Identification) The probable identity the person is Gil Grissom, an adult male who is crime-salving detective.
photos. back about an hour to pick up.
L
Holly Gribbs: Okay.
[Emotion Recognition] The prebable emotional traits of the sadness, loneliness, and resignation.
(HOLLY gets out of the car.) Gil Grissom: you get done early use channel seven your walkie.
reasoning
[Causality Inference] ensure Hally's safety by providing instructions on how to contact him if the finishes her task early.
(GRISSOM pulls out of the parking lot.) (HOLLY heads inside the liquor store.)
Figure 4: Examples of model inputs and outputs through the To-M-enhanced reasoning process. The language and multimodal models enhance understanding and reasoning of each other in dialogue-like form.
ences from the preliminary tasks as the CoT in- termediate step to test the reasoning consistency among intermediate ToM inferences and the final predictions. Specifically, with the frames V and associated screenplay T, ask models infer the cause effect of given event E, in the context.
question answering. We simplify the framework of Zhu et al. (2023a) by directly generating one task-specific question instead of augmenting itera- tive questions with accumulated contextual infor- mation. Figure 4 demonstrates example of us- ing the LLM-generated question enhance multi- modal understanding for human-factor extraction.
5 ToM-Enhanced CoT Reasoning
Vision-Augmented LLM Reasoning. Recipro- cally, the multimodal model can facilitate LLM reasoning by augmenting information grounded in the vision. To this end, the visual information should be projected into representations that LLMs can understand, such as discrete text words (Hu et al., 2022: Wang al., 2022: Zeng et al., 2022: Yang et al., 2022) and continuous features adapted into the textual space (Tsimpoukelli et al., 2021; Alayrac et al., 2022; Driess et al., 2023; Huang et al., 2023; Li et al., 2023). In our framework, we follow the former line of work to supplement the multimodal model generated textual descriptions into the LLM for vision augmentation. Specifically, the visual information covers knowledge of various granularities, extracted by general captioning and task-specific dnection-promppee task-specific questions here are generated by the LLM to guide reasoning via ToM inference.
Given the inputs of visual frames V and textual screenplay T, our objective to make human- centric inference We follow Wei et al. (2022) to break down the process into intermediate steps R and thus accommodate the three tasks in unified framework to assess large foundation models. As illustrated in Figure 4, our framework fol- lows the Vision + LLM paradigm (Huang et al., 2023; Zhang et al., 2023; Yang et al., 2023) to facil- itate multimodal understanding using the reasoning ability inherently grown in language models.
LLM-EEnanced Multimodal Understanding. Enlightened by the advanced capability of LLMS in complex reasoning (Brown al., 2020; Kojima et al., 2022: Chowdhery et al., 2022), there is an emergent line of research to leverage LLMs to prompt and guide information extraction in visual understanding (Surís et al., 2023; Wu et al., 2023; Yang et al., 2023; Zhu et al., 2023a). To diagnose the ability of human-centric reasoming of current large foundation models, we follow this paradigm enhance multimodal information extraction with LLM reasoning. Specifically, we incorporate the LLM guidance as information-seeking questions to prompt multimodal understanding via visual
answering. The
6 Experiments
We assess the social intelligence of existing large foundation models using the unified framework in Section 5 on our diagnostic tasks in Section 4. We ablate components in our framework on different backboned models and evaluate their effect on both

---

Role Identification
Emotion Interpretation
Models
VL
FS
CoT
BleU-2
Rouge-L
BERT-FI
Macro-P
Macro-R
Macro-F1
1.80
7.86
45.46
23.42
27.67
23.32
BLIP-2
2.35
7.36.o.50
13.96
24.71
30.94
24.87f
x
0.95
7.13
44.39
24.75
6,66
6,56
x
Ja.7
3.58La.ca
41.21/a.11
23.30
7.31
9.42f2.88
MiniGPT-4
X
2.52
10.43
49.33
26.99
8.44
6.05
/
/
X
0.69
5.6614.77
40.13.o.20
21.36
8.31
9.281
X
X
X
0.95
5.08
44.10
23.38
43.64
29.76
X
X
V
1.57fa.02
6.87f1.79
46.21f2.11
26.09
46.33
32.3712.c
X
X
2.571
10.0714.99
19.21ff.11
33.62
43.94
36.5516.79
InstructGPT
X
V
2.07
8.11fa.03
16.89'2.79
34.52
44.26
37 37.0317.27
X
X
3.16
10.48
49.79
24.29
37.61
28.93
X
3.04.a
10. 10.13|o.a
50.07
25.16
43.58
31.02f
6.4813.32
17.801
53.4313.64
34.87
47.59
38.67fa.74
5.79
16.531
33.37
34.34
48.64
38.951 38.95[10.02
Human (Inter-Annotator Agreement)
9.34
18.93
57.12
85.00
67.71
75.36
Table 3: Result Comparison on Role Identification and Emotion Interpretation. VL indicates whether to input full screenplay or utilize frame-only information. FS and CoT represent few-shot and thain-of-thoughhe respectively.
Models
VL
FS
CoT
ToM
BLeU-2
Rouge-L
BERT-FI
BUP2 a #/2
NinTSPT4 InstrucGPPT
#f?
X
2.99
11.89
51.24
boredam
anger
100
MiniGPT-4
x
x
3.31 Tu.3z
3.30
11.99
11 1.64/o.2
51.621o.5e
51.21.4o.03
disgust
x
/
/
3.55
12.05fo.16
51.987o.74
doubt
entrancement
x
x
X
7.44
18.41
59.33
interest
/
x
X
9.78fa.30
21.3012.88
62.2912.96
jey
InstructGPP
7
7
10.351
22.02fa.61
62.80%..44
sadhess
200
10.631a.19 a.11
22.201a.79
68.187a.s6 3.88
sympathy
11.281.u 13.84
22.9714.06
63.6514.30
Human (Inter-Annolator Agreement)
15.70
23.82
61.87
Figure 5: Class-wise perfor- mance comparison of three mod- els on emotion interpretation.
Table 4: Result Comparison on Event Causality Inference. ToM is the human- centric information. /and rrepresent model and human predictions, respectively.
automatic and human evaluation metrics.
ther validate whether the automatic metrics based on our annotated reference answers align with the actual quality of model predictions, we conduct additional human (on a subset 238 instances 10% of the whole set) and GPT-4 evaluation (on the whole set) on event causality inference results.
6.1 Setup
Backhones and Prompt Construction. We use BLIP-2 (Li al., 2023) and MiniGPT-4 (Zhu et al., 2023b), the recent public and reproducible multi-
modal models for visio-linguistic understanding. We evaluate InstructGPT (Ouyang et al., 2022) as the LLM backend considering the reproducibility of model performance, stronger closed-source LLMs such as ChatGPT and GPT-4 (OpenAI, 2023) will be updated periodically. Details of prompt design can be found in Appendix E.
6.2 Results
We compare different models in zero-shot and few-shot settings. In event causality inference, we compare the impacts of CoT in different formats (indicated by "ToM"), including model-generated general intermediate steps (X), model-generated ToM (V), and human-annotated ToM (V).
Evaluation Metrics. For generation tasks, we employ conventional metrics BLeU-2 (Papineni et al., 2002), L (Lin, 2004), and BERTScore -lateellllgeelll (Zhang et al., 2020). Considering the limitation of the automatic metrics capped the reference quality (Zhu et al., 2023a), we conduct qualitative analysis compare model and human predictions. For emotion interpretation as multilabel classification, we adopt the macro precision, recall, and scores as metrics. To fur-
Role Identification. We see a huge gap between model and human generations in zero-shot prompt- ing, while few-shot brings significant performance gain, especially InstructGPT. This demonstrates the stronger ability of LLMs for in-context learning compared with MiniGPT-4. However, we observe trend of performance drop when enhancing rea- soning via COT. This drop may be caused by the uncertainty due to task difficulty, leading to error

---

Human Evaluation (Win Lose)
Models
CoT
ToM
GPT-4 Score
Plausibility
Relevance
Completeness
Overall
5.83
InstructGPT
t.90
7.1% / 5.7%
43% 31%
38% 26%
46% 28%
(Few-Shot)
6.29fo.46
7.1% / 5.2%
35% / /33
78% 12%
56% 25%
6.361..s3
8.0% / /4.%%
50% / 25%
61% 19%
59% / 20%
Table GPT-4 and human evaluation results on event causality inference.
accumulation in reasoning.
corporating textual information into multimodal models such as MiniGPT-4, we see that these mul- timodal models still struggle to handle long in- put contexts. This demonstrates the importance of LLM incorporation for multimodal understand- ing to enhance and guide information extraction and deduction for further reasoning.
Emotion Interpretation. Likewise, InstructGPT shows stronger in-context learning ability. In- terestingly, MiniGPT-4 exhibits substantially poor performance this task compared with the other models. We further diagnose this via the label- wise scores in Figure 5. The poor recall scores of MiniGPT-4 might be one of the reasons for the failure, as tends to conduct single-label classi- fication, neglecting the instruction in most cases. On the other hand, when sufficient information is provided, i.e., with multimodal inputs and CoT rea- soning, there is stable increase in the Fl score. This disparity in how CoT works for role and emo- tion predictions may be attributed to the different degrees of uncertainty in reasoning for the two tasks. For example, models can resort to an expedi- ent strategy to interpret emotions directly based on human facial expressions.
ToM-enhanced CoT Reasoning. The perfor- mance gain from CoT reasoning in Table 4 varies when incorporating intermediate inferences from different sources, where human-annotated ToM (partially labeled) still outperforms the others. This demonstrates the deficiency of current large foun- dation models in cliciting and utilizing the ToM inferences for better reasoning.
6.4 Qualitative Evaluation
compare the InstructGPT-bbssd model with the CoT and/or ToM mechanisms in reasoning against the vanilla version that only uses few-shot prompt- ing in human evaluation. We choose the criteria including plausibility, relevance, completeness, and overall quality and let the evaluators judge which one is better. Table 5 shows the human evaluated win/lose rate on 10% (238 instances) of the whole dataset on event causality inference. To obtain a more complete understanding, we conduct GPT-4 evaluation on the whole set. Specifically, we adapt the scoring framework from Zheng et al. (2023), with our annotated inferences as the reference an- swers, where scores range from 1 to 10.
Event Causality Inference. We evaluate the ef- fect of ToM-enhanced CoT reasoning on both mul- timodal and language models. For MiniGPT-4, basic CoT reasoning without a specified format or content of ToM cannot guarantee improvement in performance. This is in accordance with our observation on role recognition that higher uncer- tainty may cause performance drop. However, as demonstrated by the ToM-enhanced CoT reasoning, our proposed human-centric tasks can benefit the final inference by incorporating ToM information about roles and emotions. Furthermore, despite the incompleteness of labeled human predictions (as shown in Table I), human ToM still exhibits more significant effect in the reasoning process.
We observe the same trend of model perfor- mance on human judgment and GPT-4 scoring terms of the overall quality of predictions. For ex- ample, the CoT framework with human-annotated ToM achieves the highest GPT-4 score and win rate at 6.36 and 59%, respectively. However, we see different trends in the relevance and completeness ratings, where the model-generated ToM-enhanced CoT achieves the best completeness but worst rele- vance scores. This may come from the LLM abil- ities information extraction and understanding which, however, also brings hallucination.
6.3
Ablation Study
We conduct further analysis to probe the impact of different modalities and ToM-CoT reasoning.
Vision vs. Language Models. In the diagnostic tasks, we observe large differences in the genera- tions between multimodal and language models. As the performance drops remarkably when in-

---

(The door opens, HOLLY GRIBBS walks into the office.)
Holly Gribbs: Hello?
(She laoks around and grimaces at the various itemss on the shelves.) (Behind her, GRISSOM walks up to her.)
Gil Grissom: Hi.
(HOLLY is startled. She gasps and turns around.) Gil Grissom: Sorry, Welcome to Forensics, Gil Grissom. I'm your supervisor on graveyard.
Holly Gribbs: Holly Gribbs.
[Event Causality Inference] What probably has caused/enabled the event that Behind her, GRISSOM walks up to her? employee, enters.
[Human] Grissom is warking the office at Forensics, likely his shift, when Holly, He knows she is going to report today, and is ready to greet her.
[Role Emotions] Grissom supervisor in middle age, who may have the feelings of surprise, calmness, slightly sorry, and appreciation. Holly new employee Graveyard youth, who may the feetings disgust, doubt, fear, and surprise.
[(Hum-TTor ToM) MiniGPT-4] Grissom has been waiting for Hally to walk into the office.
[(Hum: ToM) InstructGP]] Holly entered room and Grissom noticed her presence. Grissom has curiosity and interest meeting her. [CoT (MiniGPT-4) Grissom man in black leather jacket, he was wearing dark blue shirt.
Holly's body language changes when Grissom walks up to her.
[MiniGPT-4] Holly Gribbs in front of Grissom wearing black from behind.... [Role Emotions [InstructGPT] Holly Gribbs
employee in the office, who may have the feelings of surprise, curiosity, and
nervousness in the clip, Grissom is her supervisor, who may have the feolings of professionalism, and confusion in the clip.
[InstructGPT] Grissom introducing himself to Holly Gribbs, form of greeting and to establish professional relationship.
We compare the MeM-eenanced inferences between human annotators and different models. We consider three sources of the incorporated ToM information, including human, MiniGPT-4, and InstructGPT generations, shown in green pink.. and blue, respectively. Imperfections are highlighted in yellow
Figure 6:
6.5 Discussion
and data sides. Specifically, information from still frames can cause ambiguity without clip details, shown by the erroneous "confusion" in Figure 6, On the other hand, we acknowledge that our dataset ECHO represents subset of social interactions, but we view it as an initial and specific step explore ToM understanding of social intelligence.
We discuss main findings in qualitative analy- sis by answering the following questions:
Q1. Can models maintain reasoning robustness when input information varies in format?
As shown in Figure 6, LLMs present significantly higher adaptiveness to elicit different input infor- mation for reasoning. For example, InstructGPP directly synthesizes the character emotional traits such "curiosity" in the human-annotated ToM- enhanced inference, while MiniGPT-4 is still at copy-and-paste level in text generation and tends to focus more on descriptive information in vision.
7 Conclusion
We introduce a visio-linguistic dataset ECHO to probe human-ccntric social intelligence. With our ToMMenhanced CoTT' framework, we diagnose the reasoning ability of large foundation models. Ex- periment results and further analysis demonstrate the deficiency of current Al systems and potential bias in ECHO for efficient, correct, and consistent reasoning. We foresce follow-up work on both model and data facets to develop faithful reasoning across broader range of social scenarios.
Q2. Can models maintain consistency and faith- fulness throughout ToM-CoT reasoning?
MiniGPT-4 shows fact-level consistency in infer- ence via reiterating or rephrasing selected spans. However, it struggles in reasoning about implicit intermediate information. On the other hand, de- spite the advancement of LLMs, they may produce problematic hallucination, ie., imperfect predicted ToM such as "confusion" can lead to wrong final inference that may totally contradict the fact.
Limitations
Dataset Scale and Generalizability. As shown in Table and Section 6.5, there can be potential bias in ECHO since we only label half of the fea- tured characters to reason about their ToM. This imbalance of human belief considerations can lead to bias in final inferences as models may only focus on the thoughts and intentions of some of the char- acters. Furthermore, the reliance on crime scene content may restrict its applicability to a specific genre related to crime instead of daily life scenar- ios. While our ToM annotations (e.g., emotions)
Q3. What potential bias exists in ECHO that can lead to erroneous model predictions? One crucial problem we find in the MiniGPT-4 outputs is that it tends to randomly check one emo- tion option when it is not confident in the selection. This indicates a high uncertainty the multimodal model in the mental state interpretation of humans. Possible reasons can come from both the model

---

can represent human's 's mental states in daily life, future work may further explore whether and how ToM inferences in different distributions can assist human-centric reasoning in more general scenarios.
References
Jean -ean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro. Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Si- monyan. 2022. Flamingo: a visual language model for few-shot learning. In NewrIPS.
Dataset Construction. We may lack a detailed analysis of the visual representations to demon- strate how this information complements the tex- tual inputs. In event causality inference, adopted automatic event determination ease the efforts of human annotation, which cannot priori- tize salience within the plot. This means that cer- tain events, potentially more interesting or pertinent for causal reasoning, may go unselected.
Karl Albrecht. 2006. Social intelligence: The new sci- ence of success. John Wiley & Sons.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question an- swering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425-2433. IEEE Com- puter Society.
In future work, we will further refine ECHO to validate and extend lisou-iin ToM inferences to improve the coverage and balance of event topics, reasoning types, and source of inference evidence.
Ethics Statement
Ian Apperly. 2010. Mindreaders: the cognitive basis of" theory of mind". Psychology Press.
We have received approval from the Institutional Review Board (IRB) for our data annotation. We design the training tutorial and experimental ses- sions as guided and reviewed by the IRB to main- tain minimal risks to participants. The review pro- cess took two months to complete.
lan A Apperly and Stephen Butterfill. 2009. Do humans have two systems to track beliefs and belief- like states? Psychological review, 116(4):953.
Nolan Bard, Jakob N. Foerster, Sarath Chandar, Neil Burch, Mare Lanetot, H. Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Ed- ward Hughes, Iain Dunning, Shibl Mourad, Hugo Larochelle, Mare G. Bellemare, and Michael Bowl- ing. 2020. The hanabi challenge: A new frontier for Al research. Artif. Intell., 280: 103216.
Since ECHO contains criminal data with violent content, it may enable malevolent imitation actors or harm to specific groups of people. To avoid this misuse potential of ECHO, we will impose strict rules for access requirements and frequently track the follow-up works to constrain its usage within research-only goals. In the future, will also make regular updates on ECHO to further extend and balance the ToM attributes to alleviate potential bias and ambiguity in datapoints for better generalizability of our diagnostic tasks.
Emily M. Bender. Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In FAceT 21: 2021 ACM Conference Fairness, Accountabillty, and Transparency, Virtual Event Toronto, Canada, March 3-10, 2021, pages 610-624 (00-623.24 20-6623AAAC ACM.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models are few-shot leameer. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Acknowledgements
The computational work for this paper was partially performed on resources of the National Supercom- puting Centre (NSCC), Singapore?. We would like to thank our annotators for their time and efforts in annotating and validating ECHo instances and for our pilot subjects for their insightful feedback for our annotation user interface refinement.
tpt nus, edu. www.s//areeeechirrr.. /haps///ww.nsec.sg/
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter

---

Lee. Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg. Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4, CoRR, abs/2303.12712.
Lee Freese and Peter J Burke. 1994. Persons, identities. and social interaction. Advances in group processes. 11:1-24.
Lea Frermann, Shay B. Cohen, and Mirella Lapata. 2018. Whodunnit? crime drama as a case for natu- ral language understanding. Trans, Assoc. Comput. Linguistics, 6:1-15.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin. Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Morcira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellsterr Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways. CoRR, abs/2204.02311.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Neva- tia. 2017. TALL:: temporal activity localization via language query. In IEEE International Conference on Computer Vision, ICCV 2017, Venice. Ital,, Octo- ber 22-29, 2017, pages 5277-5285. IEEE Computer Society.
Ahmad Ghazal, Todor Ivanov, Pekka Kostamaa, Alain Crolotte, Ryan Voong, Mohammed Al-Kateb, Waleed Ghazal, and Roberto V. Zicari. 2017. Big- bench V2: the new and improved bigbench. In 33rd IEEE International Conference on Data Engineering, ICDE 2017, San Diego CA, USA, April 19-22, 2017, pages 1225- 1236. IEEE Computer Society.
Ben Goertzel. 2014, Artificial general intelligence: Con- cept, state of the art, and future prospects. J. Artif. Gen. Intell., 5(1):1-48
Riitta Hari and Miiamariia V Kujala. 2009. Brain basis of human social interaction: from concepts to brain imaging, Physiological reviews, 89(2):453-477
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai. Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang. Andrew M. Dai, Hongkun Yu, Slav Petrov, Fdd H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts. Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling cerruction-finetune language models. CoRR, abs/2210.11 11416.....
Yushi Hu, Hang Hua, Zhengyuan Yang. Weijia Shi, Noah A. Smith, and Jiebo Luo. 2022. Promptcap: Prompt-guided task-aware image captioning. CoRR, abs/2211 1.09699.
Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. 2023. Language not all you need: Aligning perception with language models. CORR, 0is/2302.14045.
Alan S Cowen and Dacher Keltner. 2017. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the national academy of sciences, 114(38):E7900-E77)
Drew A. Hudson and Christopher D. Manning, 2019. GQA: A new dataset for real-world visual reason- ing and compositional question answering. In IEEE Conference Computer Vision and Partern Recogni- tion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6700-6709. Computer Vision Founda- tion / IEEE.
Emest Davis. 2023.
Benchmarks for automated commonsense reasoning: A survey. CoRR, abs/2302.04752..
Daniel C Dennett. 1978. Beliefs about beliefs (p&w, sr&b). Behavioral and Brain sciences, ((4):568-570.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi. Corey Lynch, Aakanksha Chowdhery, Brian Ichter. Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu. Wenlong Huang, Yevgen Chebotar, Pierre Ser- manet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. 2023. Palm-e: An embodied multimodal language model. CoRR, abs/2303.033 03378.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hsuan Sung. Zhen Li, and Tom Duerig, 2021. Scaling up vi- sual and vision- -anguue representation learning with noisy text supervision. In Proceedings of the 38th In- ternational Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4904-4916 PMLR

---

Zhijing Jin, Sydney Levine. Fernando Gonzalez Adauto, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Schölkopf. 2022. When to make exceptions: Exploring language models as accounts of human moral judgment. In NeurIPS.
Gary Marcus. 2022. How come gpt can seem so brilliant one minute and so breathtakingly dumb the next? Substack newsletter. The Road to Al We Can Trust.
Daniel R Miller. 1962. The study of social relationships:
Situation, identity, and social interaction.
Shima Rahimi Moghaddam and Christopher J. Honey. 2023. Boosting theory-of-mind performance in large language models via prompting. CoRR, abs/2304. 1490.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitmick, and Ross B. Girshick. 2017. CLEVR: diagnostic dataset for compositional language and elementary visual rea- soning. In 2017 IEFE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pages 1988-1999 IEEE Computer Society.
Cory S Myers and Lawrence R Rabiner. 1981. A com- parative study of several dynamic timp-ng algo- rithms for connected-wond recognition. Bell System Technical Journal, 60(7):1389. 1409..
Emre Kiciman, Robert Ness, Amit Sharma, and Chen- hao Tan. 2023, Causal reasoning and large language models: Opening new frontier for causality. CoRR, abs/2305.00050.
OpenAl. 2023. GPT-4 technical report. CoRR, has/2303.08774..
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NewrIPS.
John Kihlstrom and Nancy Cantor. 2000. Social intel- ligence.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In NeurlPS.
Michal Kosinski. 2023. Theory of mind may have spon- taneously emerged in large language models. CoRR, 0ls/2302.02083.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6-12, 2002, Philadelphia. PA, USA, pages 311-318. ACL.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Nicbles. 2017. Dense-captioning events in videos. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, Oc- tober 22-29, 20/7, pages 706-715. IEEE Computer Society.
Jae Sung Park, Chandra Bhagavatula, Roozbeh Mot- taghi, Ali Farhadi, and Yejin Choi. 2020. Visual- comet: Reasoning about the dynamic context of a still image. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V. volume 12350 of Lecture Notes in Computer Science, pages
Junnan Li, Dongxu Li. Silvio Savarese, and Steven C. H. Hoi. 2023. BLIP-2: bootstrapping language-ing pre-training with frozen image encoders and large language models. CoRR, abs/2301.112997
Springer.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: bootstrapping language-image pre-
David Premack and Guy Woodruff. 1978. Does the chimpanzee have theory of mind? Behavioral and brain sciences, 1(4):115-526.
training for unified vasion-language understanding and generation. International Conference on Ma- chine Learning, ICML 2022, 17-23 July 2022, Balti- more, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 12888-12900. PMLR.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763 Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun 7hu, and Jian- feng Gao. 2023. Chameleon: Plug-and-pe aad---dd-ply compo- sitional reasoning with large language models. CoRR, abs/2304.09842.
Chin-Yew Lin. 2004. Rouge: package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
PMLR.
Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022. Neural thcory-of-mind? on the limits of social intelligence in large Ims. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3762-3788 37800 Association for Computational Linguistics.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Bureh 2023. Faithful chain-of- thought reasoning. CoRR, abs/2301.13379.

---

Guangyao Shen, Xin Wang, Xuguang Duan, Hongzhi Li, and Wenwu Zhu. 2020. Memor: A dataset for multimodal emotion reasoning in videos. In MM '20: The 28th ACM International Conference on Multime- dia, Virtual Event / Seattle, WA, USA, October 12-16. 2020, pages 193-502. ACM.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min- Yen Kan, Junxian He, and Qizhe Xie. 2023, De- composition enhances reasoning via self-evaluation guided decoding. CoRR, 0ts/23355.00633.
Zhengyuan Yang. Zhe Gan, Jianfeng Wang. Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study GPT-3 for few-shot knowledge- based VQA. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Con- ference on Innovative Applications of Artificial In- telligence, IAAI 2022. The Twelveth Symposium on Educational Advances in Artificial Intelligence, FAAI 2022 Virtual Event, February 22- March I, 2022, pages 308 1081-3089, AAAI Press.
Damien Sileo and Antoine Lernould. 2023. Mindgames: Targeting theory of mind in large language mod- els with dynamic epistemic modal logic. CoRR, 0as/2305.03353.
Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python exccution for reasoning. CoRR, abs/2303.08122
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302.1397/
Zhengyuan Yang. Linjie Li, Jianfeng Wang. Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023. MM- REACT: prompting chatgpt for multimodal reasoning and action. CoRR, abs/2303. 11381.
Quanzeng You, Hailin Jin, Zhaowen Wang. Chen Fang. and Jiebo Luo. 2016. Image captioning with seman- tic attention. In 2016 IEEÉ Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Ve- gas, NV, USA, June 27-30, 2016, pages 4651 1659.. IEEE Computer Society.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, M. Ali Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. In Advances in Neural Information Pro- cessing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6- 14, 2021, virtual, pages 200-212.
Luyao Yuan, Zipeng Fu, Jingyue Shen, Lu Xu, Junhong Shen, and Song-Chun Zhu. 2020. Emergence of pragmatics from referential game between theory of mind agents. CoRR, abs/2001.00752
Tomer D. Ullman. 2023. Large language models fail on trivial alterations to theory-of-mine tasks. CoRR, abs/2302.08399.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis Phill-P Morency. 2019. Social- iq: A question answering benchmark for artificial social intelligence. In IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8807- 8817. Computer Vision Foundation / IEBE.
Ronald E Walker and Jeanne M Foley. 1973. Social intelligence: Its history and measurement. Psycho- logical reports, 33(3):839-866 864.
Zhenhailong Wang. Manling Li, Ruochen Xu, Lu- owei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang. Mohit Bansal, and Heng Ji. 2022. Language models with image descriptors are strong few-shot video-language learners. In NewrIPS.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition cognition: Visual commonsense reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6720 6731. Computer Vision Foundation IEEE.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NeurIPS.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. MERLOT: multimodal neural script knowl- edge models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021. December 6- 14, 2021, virtual, pages 23634- 23651.
Wikipedia contributors. 2023. Csi: Crime scene investigation - Wikipedia, the free encyclo- pedia. https://en. www.eewwwiipiiiaa...m org/w/index. tmp7title==SII_Crrmme Serriim__Seeeeeeee Scene_Investigation oldid=1152659991. [Online; accessed 21 21-May- 2023].
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael S. Ryoo, Vikas Sindhwani, Johnny Lee, Vin- cent Vanhoucke, and Pete Florence. 2022. Socratic models: Composing zero-shot multimodal reasoning with language. CoRR, abs/2204.0051
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR, abs/2303.0467.

---

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia, April 26-30, 2020. OpenRe- neew.net.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multi- modal thain-of-thoughtt reasoning in language mod- cls. CoRR, abs/2302.009233
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin. Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judg- ing llm-as-a-judge with mt-bench and chatbot arena. CoRR, abs/2306.005655.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: human-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed H. Chi. 2022. Least-to-mont prompting enables complex reasoning large language models. CoRR, abs/2205. 10625.
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhosciny, 2023a. Chatgpt asks, BLIP-2 answers: Automatic questioning towards enriched visual descriptions. CoRR, 0e./2333.0699.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023b. Minigpt-4: Enhancing vision-language understanding with advanced large language models. CoRR, abs/2304. 10592.
Yaochen Zhu, Xiangqing Shen, and Rui Xia. 2023c. Personality-aware human-centric multimodal reason- ing: A new task. CoRR, abs/2304.02313.

---

A Annotator Recruitment and Training
We recruited prospective annotators considering both quality and diversity. First, all selected annotators possess a minimum of undergraduate education and demonstrate familiarity or expertise with the TV series CSI to ensure the annotation quality. Second, to mitigate potential biases, recruitment aimed for balanced distribution across various factors, including gender, academic disciplines, and nationalities. We piloted our preliminary annotation pipeline for refinement. After subsequent refinement, prospective annotators are first asked to watch prepared instruction video and complete pre-annotation quiz to demonstrate their understanding of tasks. Our team manually checked applicants' responses for quality, admitting qualified subjects as participants. We show one example quiz question as follows.
Clip Start Time 00:3 Clip Durution: susomds
Annutatian:
For character appoars iie, transoripti, the following question.
Chandten Ourtott
OVA 195
Choose age cilegory
Filina Cill fonh 20-25)
N.A.
02 Kenitiy the occupation or role charectic the dit.
03 (D Select the emotionši cleibiite within the duration of video dip.
Orwann
Angee
Sympatyy
Calnnuus
Rocuane thoe stan (loine())
Jump tu
Foar B
Greancement
Dageri
Othor
Current
14,000
Q4 Please different framnss
Emmstamps iMt
timestamo
$5S555
format), cower developeen Sharlacte clp
Transcilpti
Rame
bid
Ieap www.tttteeeeegg.
Flies
Wert
WWomen are brughing. A cac Nivada license #OVE 195, driwes by
Cappy
There are two women inside the cor)
[|Charlotte]]: That
han!
Vist
flom Gromtinn Lecur
[[Ulall] Werre warming
Check framenti which contain trecogiiablee character ottt,, walt few seconds for the frames loaded))
frumess) selected
CHARLOTTE recogntae where
[[Chariotte]]:
right way.
General Qn. Select the important elements focus on for each sample.
a short video clip with the provided start and end times
the whole video clip of the episode
transcripts of the clip
align characters in the clip with their names in the transcripts
the key character to focus on for all the annotation questions
different characters to focus on for different annotation questions
Q 2 Relateed Qn. Select the options which be the input Q2.
occupation (for living), e.g. singer, police, officer
role (role in the event), c.g., driver, customer, suspect of the crime
role (relation with others), e.g., the woman who stares at the others
appearance description, e.g., the girl in white shirt
3 Related Qn. What kind of emotions should be selected, and what "Others" for?
some emotions appearing via the key character's facial expressions/actions
all emotions appearing via the key character's facial expressions/actions
possible emotions of the key characters reflected by the others they interact with
"Others": to add emotions that aren't it included in the options
RelRttde Qnl What kind of & How many frames should be selected?
all the frames indicating the change of (emotional, motional) states of the key characters
the frames should preferrably be evenly distributed the clip
there is lower bound of the the selected number
Phttps: httpss:/llllccoc/e// comp. nus edu,

---

there is upper bound of the the selected number
Q4 Q4-Related Qn2. Requirements for the frames to be checked.
contain the facial expressions / recognizable actions of the key character
contain only external information (others reflection) which may indicate the state of the key character
Q5-Related Qn. Select the requirements for the input text.
Keep a low overlap-rate with the transcripts (< 30%)
Start with the text provided in front of the text box
Focus on the key character
Can consider emotion change of the character as labeled in Q3
Try to elaborate the intrinsic logic among events which aren't directly described in the clip/transcripts
O
B Annotation Interface
We provide publicly available webpage to demonstrate the example annotations on an instance in Round Two. Below is example of the detailed annotation questions.
Annotations:
For each character who appears the scene (i.e,, name the transcript), please answer following questions.
Character: Charlotte
Q1: Please choose the most likely age category of Charlotte. Infant or Child (-12)
Child, Teenager or Youth (10-25) Youth Middle-Age (25-55)
Middle/Old-ae Age
Q2: Please identify the occupation (if applicable) or the role (otherwise) of Charlotte in the clip.
e.g. occupation: doctor, officer; 2) role: driver, the girl who sits besides the driver
Please select all the emotion(s) of charlotte within the duration of the clip.
Anger O
Boredom T
Calmness
Disgust O
Doubt O
Entrancement O
Interest
Sadness
Shame
Surprise
Sympathy
Others O
Q4 [ Please select different frames in the order of timestamps (in format), which cover the character development (le,, the changes of emotional or motional states) of Charlotte within the clip duration.
frame
MM
SSSSSS
from current timestamp
resume from here
copy
timestamp
MM
SS.555
copy from current timestamp
resurne from here
Q4-Check (m] Below is the frame(s) you've selected in Q4, please check all the frame(s) which contain the (recognizable) character charlotte, (You may need to wait for a few seconds for the frames to be loaded.)
Based on above annotations, please answer the reasoning question. Please also make sure that your answer starts with the text span in front of the input bar.
Q5 Ii
effect What probably caused/enabled by the event that LISA turns car and CHARLOTTE doesn't recognize where they are?
Over Lap-Rate
Transcripts
(Please keep 50%)
Charlotte
wwt.eeg.....aaaaeeeeeeeeeeeeeeee La/_ppaass/CS__eaam html

---

C
Emotion Categorization
We adapt the original 27 emotions from Cowen and Keltner (2017) to crime plots, merging them into 13 emotions suited for the domain:
Anger: wrath, outrage, fury, violence, irritability, hostility, resentment;
Boredom: blahs, doldrums, ennui, weariness, listlessness, restlessness;
Calmness: blahs, doldrums, ennui, weariness, listlessness, restlessness;
Disgust: contempt, scorn, disdain, aversion, distaste, revulsion:
Doubt: uncertainty, confusion, distrust;
Entrancement: brooding, reverie, contemplation, daydreaming, cogitation, detachment;
Fear: anxiety, dread, fright, panic, nervousness;
Interest: trust, kindness, affection, devotion, acceptance, love, anticipation, friendliness;
Joy: enjoyment, bliss, happiness, relief, delight, pride, thrill, ecstasy;
Sadness: grief, sorrow, gloom, despair, melancholy, loneliness, depression:
Shame: regret, guilt, embarrassment, remorse;
Surprise: astound, shock, astonishment, wonder; Sympathy: commiseration, compassion, feeling.
D Frequently Asked Questions
D.1 The fact that multiple different causes of an event exist complicates the evaluation of the CoT approach for event causality. How is evaluation impacted when multiple causes of an event may exist?
We dealt with the one-to-many problem on two aspects of data collection. First, we provide multiple reference inferences (annotations) for each event, shown in Table 1. Second, narrow the scope of potential causes by employing Theory-of-Mind (ToM) as the intermediate reasoning step. This serves directional constraint reduce the search space of potential CoT chains.
Considering that our annotated reference answers may not capture the full spectrum of possible causes, we conduct both human and GPT-4 evaluations to provide qualitative analysis on the model predictions. The human evaluation results are consistent with the automatic metries assessed in Table 4. Furthermore, the performance gain brought by ToM--costtaineed CoT shows the importance of ToM in our human-centric reasoning task.
D.2 How many instances are rejected and revised during inference validation? What are the major error types and feedback types encountered during annotation and verification?
Out of the 5, 746 annotations collected from Round Two, we finally retained 4, 280 annotations, indicated in Table 3. Among these, 107 were revised and accepted following re- re-annotation. In validation, we check the annotation quality considering three criteria: plausibility, relevance, and completeness. Specifically, we mainly reject instances if they exhibit weak connections between the annotated inferences and the plot contents or the corresponding character roles and emotions.
Feedback from annotators suggests two primary causes for annotation errors: 1) the imperfect event selection (where we reject the entire instance), and 2) insufficient incorporation of ToM reasoning (where we reject outright or send back for re-annotation).
D.3 Why use instructGPT instead of stronger LLMs like ChatGPT and GPT-4?
We use InstructGPT considering the reproducibility of model performance, as stronger closed-source LLMs like ChatGPT and GPT-4 will be updated periodically. For reference, we conduct an additional experiment using ChatGPT (gpt-3.55 3. 5-turbo couubbo-0133 in Table 6 on the task of event causality inference.
E Implementation Details
We detail our experiment setup and prompt construction this section.

---

Models
VI
FS
CoT
ToM
BERT-FI
GPT-4 Score
/
63.18
6.29
InstructGPT
/
63.65
6.36
ChatGPT
64.09
6.98
Table 6: Result comparison on InstructGPT and ChatGPT.
E.1 Setup
For multimodal models, use BLIP-210 (blip2_t5) (Li et al., 2023) and MiniGPT-4" (prerained_minigott__ (Zhu et al., 2023b), the recent public and reproducible multimodal mod- els for visio-linguistic understanding. Specifically, we use cectrain_flaat5xl for BLIP-2 due the computation limit. For LLM backend, we use InstructGPT (text-davinci-on) (Ouyang et al., 2022). In the automatic evaluation text generation tasks, we measure the similarity between model outputs and human annotations, compared with the inter-agreement among annotators using the same metrics, For the multilabel classification task, the macro precision, recall, and Fl scores measure both the model-human and inter-annotator agreement. The inter-annotator agreement considers the instance-wide similarity between annotator and the others.
E.2 Pipeline Construction
Denote the multimodal model and the language model M and L respectively. The process for ToM- enhanced CoT reasoning for event causality inference as follows (Steps 2 and be merged for simplicity, with further post-processing to extract the final answers):
Step 1: Visual Information Extraction using M. .M generates visual descriptions based on the input
video clips. The prompt to M determines the type and focus of the visual descriptions generated.
input: clip frames
output: visual descriptions
Step Role and Emotion Identification using C. We augment the text prompt for with the visual information obtained previously, eliciting roles and emotions associated with given characters.
input: visual descriptions + textual context (screenplay) output: roles and emotions of the specified characters
Step 3: ToM-Enhanced CoT Reasoning using L. Using the predicted roles and emotions inter- mediate reasoning products, construct the CoT prompt for L, allowing it to perform ToM-enhanced reasoning about event causality.
input: visual descriptions + textual context (screenplay) + roles and emotions output: event causality inference
E.3 Prompt Design
For multimodal prompting, we adopt the form of visual question answering where specific instructions will be provided for human-centric information extraction and inference. For LLM prompting, we accom- modate information comprising screenplays, textual descriptions of visual frames in multi multi-granularities, and specific instructions to stimulate reasoning.
1 LLAVIS: http://hhhhh...oo..lsseere////// MMniGPPT4: https://miniggtt-4. github.

---

ROLE
The image is video frame a person. Specifically, the role of person can their occupation or their relation with others.
In this case, what probably is the role of this person?
Answer:
EMOTION
The image is corresponding video frame of person.
Question: What are the possible emotional traits of this person?
Answer Choices (can select more than one choices):
[#1]
Answer:
Table Prompts for BLIP-2 with vision-only inputs. #1 represents the emotion options to choose from.
ROLE
Read the screenplay of video clip as follows:
[#1]
For reference, the image is corresponding video frame of person. Specifically, the role of person can be their occupation or their relation with others.
In this case, what probably is the role of this person?
Answer:
EMOTION
Read the screenplay of video clip as follows:
[#1]
For reference, image corresponding video frame of person,
Question: What are the possible emotional traits this person?
Answer Choices (can select more one choices):
(#2)
Answer:
Table 8: Prompts for BLIP-2 with vision-and-language inputs. #1 represents the screenplay content.
ROLE
Below are frumes of person happening chronological order:
[#1)
Specifically, the role of person can their occupation or their relation with others the plots.
In this case, what probably is the role this person?
Answer:
EMOTION
Below are frames of person happening in chronological order:
[#1]
Question: What are the possible emotional traits of this person?
Answer Choices (can select more than one choices):
[#2]
Answer:
EVENT CAUSALITY
Below several frames (with characters they contain) happening in chronological order in the video clip:
{#1}
What the possible (#3) of the clip event that (#4)?
Answer:
Table 9: Prompts for MiniGPT-4 with vision-only inputs. #1 represents the visual tokens as a placeholder for frame embeddings. #3 and #4 represent the causality type (cause effect) and the event reason about, respectively.

---

ROLE
Read the screenplay of video clip follows:
[#1]
For reference, below series of chronologically ordered frames of person [#2] from the vidco clip:
[#3]
What probably is the role this person?
Answer:
EMOTION
Read the screenplay of video clip as follows:
{#1]
For reference, below are a series of chronologically ordered frames of the person (#2] from the same video clip:
{#3}
Question: What are the possible emotional traits this person?
Answer Choices (can select more than one choices):
[#4]
Answer:
EVENT CAUSALITY
Read the screenplay of a video clip as follows:
[#1]
For reference, below are a series of chronologically ordered frames from the same video clip, with certain frames
annotated to indicate the character(s) they feature:
[#3]
What the possible (#4) of the clip event that (#5}7
Answer:
Table 10: Prompts for MiniGPT-4 with vision-an -an-uang inputs. #2 represents the name of the specified character.
VQA
Below series of chronologically ordered frames video clip, with certain frames annotated indicate character(s) they feature:
[#1]
Based on provided frames, please answer following question.
Question: {#2)
Table 11: Prompts for MiniGPT-4 for clip-based visual question answering.
ROLE
Below the description of video clip where there a character named [#1):
(#2)
Specifically, the role of character can be their occupation or their relation with others in the plots.
In this case, what is the role of (#1)?
Answer:
EMOTION
Below is the deseription of video clip where there is a character namod (#1):
[#2]
Question: What are the possible emotional traits of (#1) )
Answer Choices (can select more than one choices);
[#3)
Answer:
EVENT CAUSALITY
Below the description of video clip where there is character named (#1):
[#2)
What the possible (#3) of the clip event that (#4)?
Answer:
Table 12: Prompts for InstructGPT with vision-only inputs.

---

ROLE
Read the screenplay of a video clip as follows:
{#1]
Specifically, the role of character can be their occupation or their relation with others in the plots.
In this case, what is the role of {#2]?
Answer:
EMOTION
Read the screenplay of video clip as follows:
[#1]
Question: What the possible emotional traits of (#2]?
Answer Choices (can select more than choices):
[#3]
Answer:
EVENT CAUSALITY
Read the screenplay of video clip as follows:
[#1)
What possible (#2) of the clip event that [#3)?
Answer:
Table 13: Prompts for InstructGPT with language-only inputs.
ROLE
Read the screenplay of a video clip as follows:
[#1]
For reference, below is the description of the same vidco clip:
[#2)
In this context, a character's role can be defined by their occupation or their relationships with others within the
plots.
In this case, what is the role of (#3)?
Answer:
EMOTION
Read the screenplay of video clip follows:
[#1]
For reference, below is the description of the same video clip:
[#2]
Question: What the possible emotional traits [#3]?
Answer Choices (can select more than choices):
{#4]
Answer:
EVENT CAUSALITY
Read the screenplay of video clip as follows:
{#1]
For reference, below is the description the same video clip:
[#2]
What the possible (#3) of the clip event that (#4)?
Answer:
Table 14: Prompts for InstructGPT with ission-and-langusee inputs.

---

ROLE
Read the screenplay of a video clip as follows:
[#1]
For reference, below is the description of the same vidco clip:
{#2]
In this context, a character's role can he defined by their occupation or their relationships with others within the
plots.
As an intermediate step discern the role of (#3), we may need to delve deeper into specific and concrete visual cues from the video clip.
Assuming can access the video clip, please generate question that indirectly seeks information about the role of [#3].
Question:
EMOTION
Read the screenplay of video clip follows:
[#1]
For reference, below is the description of the same video clip: [#2]
intermediate step interpret the emotional trait(s) of (43). may need delve deeper into specific and conerete visual from the video clip. Assuming you
the video clip, please generate question that indircetly seeks information about the emotion(s) of (#3).
Question:
EVENT CAUSALITY
Read the screenplay of a video clip as follows:
{#1]
For reference, below is the description the same video clip:
[#2)
In order understand the possible (#3) the clip event that (#4), we may need to delvv deeper into specific and concrete visual cues from the video clip. Assuming you can access the video clip, please construct a question that indirectly seeks information useful for inferring the causality of the event. Question:
Table 15: Prompts for InstructGPT with vision-and-language inputs for information-seeking question generation.


Reviews:
[{"review_id": "nP0TiSDNXa", "reviewer": "Reviewer_DSz5", "summary": "", "questions": "Is the annotation line of role identification, emotion identification, and event causality identification supposed to be a pipeline? In some cases, event causality is not related to emotions. For example, in Figure 2, for the question what has probably caused / enabled the event that Holly gets out of the car? The answer is Holly, as a junior, listened to Grissom's advice... This is not very related to emotions.\n\nHow is evaluation impacted when multiple causes of an event may exist?\n\nHow do you deal with non-deterministic outputs from the model (even a temperature of 0 cannot guarantee a consistent output) and how often is non-determinism an issue?\n\nWho are the annotators and what are their background (are they all similar)? What is the interannotator agreement?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We recognize your thoughtful feedback!\n\n> The fact that multiple different causes of an event can exist complicates the evaluation of the CoT approach for event causality. How is evaluation impacted when multiple causes of an event may exist?\n\nThanks for highlighting the open-endedness of event causality inference.\nWe dealt with the one-to-many problem on two aspects of data collection. First, we provide multiple reference inferences (annotations) for each event, as shown in Table 1. Second, we narrow the scope of potential causes by employing **Theory-of-Mind** (ToM) as the intermediate reasoning step. This serves as a directional constraint to reduce the search space of potential CoT chains.\n\nConsidering that our annotated reference answers may not capture the full spectrum of possible causes, we conduct human evaluations on $10$\\% ($238$ instances) of the dataset to **rank** ($1$ to $4$) the plausibility of the model's predictions independently. \nSpecifically, the ranking is based on a set of criteria: correctness, relevance, completeness, and reasoning consistency (e.g., the extent to which the inference appropriately employs theory-of-mind). ~~&#10003;~~ represents the half-checkmark in Table 4.\n\n|   Models  |   VL   |   FS   |   CoT  |    ToM     |   Avg. Rank   |\n|    :-:    |   :-:  |   :-:  |   :-:  |     :-:    |      :-:      |\n|InstructGPT|&#10003;|&#10003;|&#10007;|  &#10007;  |    $1.42$     |\n|InstructGPT|&#10003;|&#10003;|&#10003;|  &#10007;  |    $1.40$     |\n|InstructGPT|&#10003;|&#10003;|&#10003;|~~&#10003;~~|    $1.37$     |\n|InstructGPT|&#10003;|&#10003;|&#10003;|  &#10003;  |    $1.34$     |\n\nThe human evaluation ranks the model predictions in the same order as that in Table 4. Furthermore, the performance gain brought by ToM-constrained CoT shows the importance of ToM in our human-centric reasoning task.\nWe will add the human evaluation results on the whole dataset in our revised version.\n\n> The relationship between event causality and the previous step of the task pipeline (emotion identification) is not completely clear. Is the annotation line of role identification, emotion identification, and event causality identification supposed to be a pipeline? In some cases, event causality is not related to emotions. For example, in Figure 2, for the question what has probably caused / enabled the event that Holly gets out of the car? The answer is Holly, as a junior, listened to Grissom's advice... This is not very related to emotions.\n\nThanks for the clarifying question.\nYes, we conduct the annotations of role, emotion, and event causality identification sequentially. Annotators are asked to refer to the annotated role(s) and/or emotion(s) when inferring the event causality. This process encourages Theory-of-Mind reasoning for human-centric plot understanding.\nTake Figure 2 for example, while Holly’s action of *getting out of the car* is not directly caused by her emotion (e.g., *entrancement*, *doubt*), it benefits understanding her decision as a junior (her role) who lacks experience of independent investigation.\nIn the next version, we will further clarify the task relationships in the annotation pipeline.\n\n> In crime scenes, some emotions may be more common than others (e.g. humor is not common in crime scenes), so evaluating on crime scene data may not be the most complete way to accomplish the larger goal of probing social intelligence.\n\nThanks for raising the incredibly important issue of probing social intelligence in a **complete** way.\nWe acknowledge that our dataset ECHo represents a subset of social interactions, but we view it as an initial and specific step to explore ToM understanding of social intelligence. As outlined in Table 2, crime scenes are particularly rich in ToM-related cases, including *false-belief* and *unexpected-content*. \nIn light of your comment, would it enhance the clarity and utility if we further emphasize the specialized focus on crime scenes as a distinguishing aspect of our dataset?\nIn this case, we will elaborate more on how ECHo assesses agents' ToM understanding in a specific genre, like humor detection (Patro et al. 2021).\n\n---\nMultimodal Humor Dataset: Predicting Laughter tracks for Sitcoms. Patro et al. WACV 2021\n\n> How do you deal with non-deterministic outputs from the model (even a temperature of 0 cannot guarantee a consistent output) and how often is non-determinism an issue?\n\nThanks for raising your concern about the non-determinism of model predictions.\nWe conduct additional experiments to probe how model performance varies with different temperatures.\nFor reference, we also provide the GPT-4 scoring results based on our annotated inferences.\nWe adapt the scoring framework from Zheng et al. (2023), where scores range from $1$ to $10$.\n\n|   Models  |Temperature|   VL   |   FS   |   CoT  |    ToM     |  BLeU-2  |  Rouge-L  |  BERT-F1  |  GPT-4 Score  |\n|    :-:    |   :-:     |   :-:  |   :-:  |   :-:  |     :-:    |   :-:    |    :-:    |     :-:   |     :-:       |\n|InstructGPT|   $0.0$   |&#10003;|&#10003;|&#10003;|~~&#10003;~~| $10.63$  |  $22.20$  |  $63.18$  |    $6.29$     |\n|InstructGPT|   $0.0$   |&#10003;|&#10003;|&#10003;|~~&#10003;~~| $10.04$  |  $22.03$  |  $62.96$  |    $6.30$     |\n|InstructGPT|   $0.5$   |&#10003;|&#10003;|&#10003;|~~&#10003;~~|  $9.91$  |  $21.72$  |  $62.65$  |    $6.25$     |\n|InstructGPT|   $0.0$   |&#10003;|&#10003;|&#10003;|  &#10003;  | $11.28$  |  $22.97$  |  $63.65$  |    $6.36$     |\n|InstructGPT|   $0.0$   |&#10003;|&#10003;|&#10003;|  &#10003;  | $11.12$  |  $22.80$  |  $63.43$  |    $6.42$     |\n|InstructGPT|   $0.5$   |&#10003;|&#10003;|&#10003;|  &#10003;  |  $9.65$  |  $22.02$  |  $63.08$  |    $6.37$     |\n\nOur results indicate minimal variation in performance when using different temperatures, in terms of both automatic metrics and GPT-4 scores. For instance, the GPT-4 score fluctuated only within a narrow range of $6.25$ to $6.42$ across different settings. So non-determinism does not significantly impact the model's output quality.\n\n---\nJudging LLM-as-a-judge with MT-Bench and Chatbot Arena. Zheng et al. 2023\n\n> Who are the annotators and what are their background (are they all similar)? What is the interannotator agreement?\n\nThanks for your clarifying questions about our annotator background and the inter-annotator agreement.\n\nWe recruited annotators considering two-fold criteria. First, to ensure the annotation quality, all selected annotators possess a minimum of undergraduate education and demonstrate familiarity or expertise with the TV series *CSI*. Second, to control for potential biases and ensure diversity, our recruitment aims for a balanced distribution across various factors, including but not limited to gender, academic disciplines, and nationalities.\n  \nWe provide the inter-annotator agreement in Tables 3 and 4 in our submission. The automatic scoring is designed to treat annotator inputs as a reference to each other for each instance. For example, the macro F1 of the emotion interpretation task is $75.36$.\nWe will make this clear in the next version of our paper.\n\n> **Presentation Improvement**: In round three: inference validation -- describe more details about how to reject annotations.\n\nThanks for your suggestion.\n\nThe inference validation process consists of two parts: automatic and manual validation.\nThe real-time automatic checking alerts annotators if their inputs fail to meet certain informativeness criteria, such as text length and word-level overlap with the existing context.\nSubsequently, our authoring team carry out the manual validation. We particularly focus on instances with lower inter-rater agreement in emotion identification or anomalies in the timestamp distribution of selected frames. We evaluate annotations based on three dimensions: plausibility, relevance, and completeness. For example, we reject instances where the event causality inference shows weak association with either the plot or the annotated roles and emotions.\nBased on our $5,746$ annotations from Round Two, we finally collect $4,280$ annotations (Table 1) after rejection and revisement.\n\nWe will further clarify this in the Section of construction pipeline (Section 3.1) in our paper.\n\n> **Presentation Improvement**: Explain the reason that the pipeline is split into the parts of role identification, emotion identification, and event causality, or if all of these parts are supposed to be parallel, not dependent on, each other.\n\nThanks for your suggestion. \n\nWe conduct the role, emotion, and event causality identifications sequentially in our annotation pipeline. Specifically, the event causality inference is dependent on the prior annotations concerning roles and emotions. These tasks aim to assess social intelligence through foundational aspects of human-centric reasoning, i.e., ToM understanding. Based on these, the task of event causality inference further examines reasoning capacities within socially-grounded interactions.\n\nWe will further clarify this in the Section of task formulation (Section 4) in our revised version.\n\n> **Typo**: On page 4, the second column has a typo of \"for to\".\n\nWe appreciate the reviewer’s correction of the typo and the above suggestions on presentation. We will revise our paper accordingly in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem statement is very clear and the paper has many clear figures to explain the annotation pipeline, dataset statistics, examples of model inputs and outputs, and a thorough evaluation. The appendix has clear figures to describe the process of collecting annotations and of training annotators. Probing social intelligence over multiple modalities is a very practical and interesting problem and a dataset resource to help with this problem is very helpful. The dataset is on crime scenes which facilitate approximation of social interaction.", "reasons_to_reject": "The fact that multiple different causes of an event can exist complicates the evaluation of the CoT approach for event causality.\n\nThe relationship between event causality and the previous step of the task pipeline (emotion identification) is not completely clear. \n\nIn crime scenes, some emotions may be more common than others (e.g. humor is not common in crime scenes), so evaluating on crime scene data may not be the most complete way to accomplish the larger goal of probing social intelligence.", "questions_to_authors": "Is the annotation line of role identification, emotion identification, and event causality identification supposed to be a pipeline? In some cases, event causality is not related to emotions. For example, in Figure 2, for the question what has probably caused / enabled the event that Holly gets out of the car? The answer is Holly, as a junior, listened to Grissom's advice... This is not very related to emotions.\n\nHow is evaluation impacted when multiple causes of an event may exist?\n\nHow do you deal with non-deterministic outputs from the model (even a temperature of 0 cannot guarantee a consistent output) and how often is non-determinism an issue?\n\nWho are the annotators and what are their background (are they all similar)? What is the interannotator agreement?", "paper_topic_and_main_contributions": "The paper aims to probe human-centric social intelligence and assess the reasoning ability of large foundation models in multiple modalities. The contributions of the paper are (1) introducing a visio-linguistic dataset of event causality inference, (2) proposing a chain of thought framework to assess reasoning of AI systems, using large foundation models, and (3) examining models by performing human-centric tasks. Specifically, the dataset is a visio-linguistic dataset with theory of mind inferences for event causality reasoning in social scenarios, with grounding in a plot from crime drama CSI: Crime Scene Investigation. The annotation process involves (1) identifying a specified character as their role, (2) discerning mental states via emotion interpretation, and (3) inferring cause or effect of plot event for causality reasoning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "On page 4, the second column has a typo of \"for to\". \n\nIn round three: inference validation -- describe more details about how to reject annotations.\n\nExplain the reason that the pipeline is split into the parts of role identification, emotion identification, and event causality, or if all of these parts are supposed to be parallel, not dependent on, each other."}}, {"review_id": "U1Yk5HmCsG", "reviewer": "Reviewer_Jxpn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We acknowledge your insightful comments!\n\n> The annotation of ECHo Corpus is a difficult task, which makes obtaining high quality data hard. I believe it is necessary here to justify the quality of the data. Although the authors present data construction details, they did not give quantitative evaluation for the annotation quality (e.g., inter-rater agreement in the classification tasks).\n\nThanks for highlighting the importance of providing inter-rater agreement.\n\nWe provide the inter-annotator agreement in Tables 3 and 4 in our submission. The automatic scoring is designed to treat annotator inputs as a reference to each other for each instance. For example, the macro F1 of the emotion interpretation (multi-label classification) task is $75.36$. We will make this clear in the next version of our paper.\n\nTo validate whether the automatic metrics based on our annotated reference answers align with the actual quality of model predictions, we conduct additional human (on a subset) and GPT-4 evaluation (on the whole set) of the event causality inference task.\nFor GPT-4 evaluation, we adapt the scoring framework from Zheng et al. (2023), with our annotated inferences as the reference answers, where scores range from $1$ to $10$.\nFor human evaluation, we rank ($1$ to $4$) the model predictions on $10$\\% ($238$ instances) of the dataset for simplicity.\nSpecifically, we did the evaluation based on a set of criteria: correctness, relevance, completeness, and reasoning consistency (e.g., the extent to which the inference appropriately employs theory-of-mind). \n~~&#10003;~~ represents the half-checkmark in Table 4.\n\n|   Models  |   VL   |   FS   |   CoT  |    ToM     |   Avg. Rank   |  GPT-4 Score  |\n|    :-:    |   :-:  |   :-:  |   :-:  |     :-:    |      :-:      |      :-:      |\n|InstructGPT|&#10003;|&#10003;|&#10007;|  &#10007;  |    $1.42$     |    $5.83$     |\n|InstructGPT|&#10003;|&#10003;|&#10003;|  &#10007;  |    $1.40$     |    $5.90$     |\n|InstructGPT|&#10003;|&#10003;|&#10003;|~~&#10003;~~|    $1.37$     |    $6.29$     |\n|InstructGPT|&#10003;|&#10003;|&#10003;|  &#10003;  |    $1.34$     |    $6.36$     |\n\nWe observe the same trend of model performance on human ranking and GPT-4 scoring as the automatic metrics in Table 4.\nWe will add the human and GPT-4 evaluation results in our revised version.\n\n---\nJudging LLM-as-a-judge with MT-Bench and Chatbot Arena. Zheng et al. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow. \n\n2. The proposed dataset presents a way to evaluate artificial agents' understanding of Theory of Mind, which is crucial for implementing general artificial intelligence. I believe the contribution made here is solid.\n\n3. The authors proposes a unified method (combining visual and linguistic reasoning) to conduct chain-of-thought reasoning on the proposed data. ", "reasons_to_reject": "1. The annotation of ECHo Corpus is a difficult task, which makes obtaining high quality data hard. I believe it is necessary here to justify the quality of the data. Although the authors present data construction details, they did not give quantitative evaluation for the annotation quality (e.g., inter-rater agreement in the classification tasks ). If the authors could not justify the quality of the proposed data, the later-mentioned experimental results could be less convincing.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose ECHo, which is a visio-linguistic corpous for social reasoning. They further propose a framework to evaluate large language models in zero- and few-shot theory of mind enhanced reasoning. They further show how to leverage existing LLMs and multimodal models to conduct reasoning on ECHo", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QanIUtia3a", "reviewer": "Reviewer_CzKR", "summary": "", "questions": "A. In Characteristics Identification, are different roles of the same character collected for each plot, or does a character's role remains consistent across most plots?\n\nB. In Keyframe-Grounded Emotion Interpretation, if multiple selections are allowed, how do annotators choose key frames for each emotion selection?\n\nC. In Event Causality Inference, are there any limitations on using utterances or narrations as events? How the selection of annotated \"events\" is determined?\n\nD. Sec 5 and Fig 4 are hard for me to understand. It can be helpful to clarify what intermediate steps are involved, what is the input and output of each step, and which model and prompt are used. There are too many prompts in Appendix D and it's hard for me to map them to each intermediate step. I also don't understand why some prompt requires a character name but some do not. How does the model know which person should be focused on?\n\nE. How to get the visual frames V in Sec 5? Are they provided by the annotated dataset or automatically selected from a video? \n\nF. How many instances are rejected and revised during inference validation? What are the major error types and feedback types encountered during annotation and verification?\n\nG. Why use instructGPT instead of stronger LLMs like ChatGPT and GPT-4?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback!\n\n> The experiments lack systematic human evaluation for model performance. It's not clear if the small difference in automatic evaluation measures can drive meaningful conclusions.\n\nThanks for highlighting the importance of systematic human evaluation.\nTo validate whether the automatic metrics based on our annotated reference answers align with the actual quality of model predictions, we conduct additional human (on a subset) and GPT-4 evaluation (on the whole set) of the event causality inference task.\nFor GPT-4 evaluation, we adapt the scoring framework from Zheng et al. (2023), with our annotated inferences as the reference answers, where scores range from $1$ to $10$.\nFor human evaluation, we rank ($1$ to $4$) the model predictions on $10$\\% ($238$ instances) of the dataset for simplicity.\nSpecifically, we did the evaluation based on a set of criteria: correctness, relevance, completeness, and reasoning consistency (e.g., the extent to which the inference appropriately employs theory-of-mind). \n~~&#10003;~~ represents the half-checkmark in Table 4.\n\n|   Models  |   VL   |   FS   |   CoT  |    ToM     |   Avg. Rank   |  GPT-4 Score  |\n|    :-:    |   :-:  |   :-:  |   :-:  |     :-:    |      :-:      |      :-:      |\n|InstructGPT|&#10003;|&#10003;|&#10007;|  &#10007;  |    $1.42$     |    $5.83$     |\n|InstructGPT|&#10003;|&#10003;|&#10003;|  &#10007;  |    $1.40$     |    $5.90$     |\n|InstructGPT|&#10003;|&#10003;|&#10003;|~~&#10003;~~|    $1.37$     |    $6.29$     |\n|InstructGPT|&#10003;|&#10003;|&#10003;|  &#10003;  |    $1.34$     |    $6.36$     |\n\nWe observe the same trend of model performance on human ranking and GPT-4 scoring as the automatic metrics in Table 4.\nWe will add the human and GPT-4 evaluation results on the whole dataset in our revised version.\n\n---\nJudging LLM-as-a-judge with MT-Bench and Chatbot Arena. Zheng et al. 2023\n\n> **Q.A.** In Characteristics Identification, are different roles of the same character collected for each plot, or does a character's role remains consistent across most plots?\n\nCharacter roles are not static but can vary in different plots, depending on the nuances of their appearances, behaviors, and interactions in each specific context.\nFor example, a character's role could be labeled based on their occupation, such as *forensic investigator*, or based on their specific actions within the plot, like *female passenger sitting next to the car driver*.\n\n> **Q.B.** In Keyframe-Grounded Emotion Interpretation, if multiple selections are allowed, how do annotators choose key frames for each emotion selection?\n\nThe minimum number of frames to select is determined by both the clip duration and the number of selected emotions. When multiple emotions are identified, annotators are also instructed to select more frames.\nHowever, we didn't strictly enforce one-to-one matching between frames and emotions. This is because one frame can feature several emotions, and some emotions may be more accurately captured by considering the reactions of other characters in the scene.\nTo ensure the completeness and informativeness of the selected keyframes, we have implemented a follow-up validation process that takes into account the roles and emotions annotated.\n\n> **Q.C.** In Event Causality Inference, are there any limitations on using utterances or narrations as events? How the selection of annotated \"events\" is determined?\n\nThanks for the clarifying question on event determination.\n\nWe select and determine the events through two stages. Initially, we randomly select utterances or narrations as events based on a set of constraints. These constraints include: 1) the event must mention character(s) in the plot, 2) the event should ideally occur in the middle of the plot to facilitate effective causality reasoning, and 3) the event is long enough to form a complete and understandable unit. Subsequently, in Round Two, the automatically selected events that are insufficiently meaningful are discarded. Annotators can reject such instances, pending confirmation from the authoring team.\n\nThe main limitation of our automatic event determination is that it doesn't prioritize salience within the plot. This means that certain events, potentially more interesting or pertinent for causal reasoning, may go unselected.\n\n> **Q.D.** Sec 5 and Fig 4 are hard for me to understand. It can be helpful to clarify what intermediate steps are involved, what is the input and output of each step, and which model and prompt are used. There are too many prompts in Appendix D and it's hard for me to map them to each intermediate step. I also don't understand why some prompt requires a character name but some do not. How does the model know which person should be focused on?\n\nThanks for the clarifying question.\n\nDenote the multimodal model and the language model as $M$ and $L$, respectively.\nThe process for ToM-enhanced CoT reasoning for event causality inference is as follows (Steps 2 \\& 3 can be merged for simplicity, with further post-processing to extract the final answers):\n1. Visual Information Extraction using $M$: $M$ generates visual descriptions based on the input video clips. The prompt to $M$ determines the type and focus of the visual descriptions generated.\n    - input: clip frames\n    - output: visual descriptions\n2. Role and Emotion Identification using $L$: We augment the text prompt for $L$ with the visual information obtained previously, eliciting roles and emotions associated with given characters.\n    - input: visual descriptions $+$ textual context (screenplay)\n    - output: roles and emotions of the specified characters\n3. ToM-Enhanced CoT Reasoning using $L$: Using the predicted roles and emotions as intermediate reasoning products, we construct the CoT prompt for $L$, allowing it to perform ToM-enhanced reasoning about event causality.\n    - input: visual descriptions $+$ textual context (screenplay) $+$ roles and emotions\n    - output: event causality inference\n\nRegarding your question on prompts: In cases where the input includes screenplay contents, we specify character names to improve coreference resolution. For vision-only inputs, character identification can either be specific, pointing out characters featured in frames, or more general, referring to characters simply as \"a person\" when names are not needed.\n\n> **Q.E.** How to get the visual frames V in Sec 5? Are they provided by the annotated dataset or automatically selected from a video?\n\nThe frames are manually selected following the role and emotion identification during annotation. There is also a follow-up step where annotators label the frames with the featured characters. \nWe gather frames featuring different characters together to represent the visual content of each clip.\n\n> **Q.F.** How many instances are rejected and revised during inference validation? What are the major error types and feedback types encountered during annotation and verification?\n\nOut of the $5,746$ annotations collected from Round Two, we finally retained $4,280$ annotations, as indicated in Table 1.\nAmong these, $107$ were revised and accepted following re-annotation.\nIn validation, we check the annotation quality considering three criteria: plausibility, relevance, and completeness. Specifically, we mainly reject instances if they exhibit weak connections between the annotated inferences and the plot contents or the corresponding character roles and emotions.\n\nFeedback from annotators suggests two primary causes for annotation errors: 1) the imperfect event selection (where we reject the entire instance), and 2) insufficient incorporation of ToM in reasoning (where we reject outright or send it back for re-annotation).\n\n> **Q.G.** Why use instructGPT instead of stronger LLMs like ChatGPT and GPT-4?\n\nWe use InstructGPT considering the reproducibility of model performance, as stronger closed-source LLMs like ChatGPT and GPT-4 will be updated periodically.\nFor reference, we conduct an additional experiment using ChatGPT (`gpt-3.5-turbo-0613`) on the task of event causality inference.\n\n|   Models  |   VL   |   FS   |   CoT  |    ToM     |  BERT-F1  |  GPT-4 Score  |\n|    :-:    |   :-:  |   :-:  |   :-:  |     :-:    |     :-:   |     :-:       |\n|InstructGPT|&#10003;|&#10003;|&#10003;|~~&#10003;~~|  $63.18$  |    $6.29$     |\n|InstructGPT|&#10003;|&#10003;|&#10003;|  &#10003;  |  $63.65$  |    $6.36$     |\n|  ChatGPT  |&#10003;|&#10003;|&#10003;|  &#10003;  |  $64.09$  |    $6.98$     |\n\n> Some aspects of the annotation/experiments design require further clarification.\n\nThanks for your above clarifying questions. Please see our responses above. We will revise our paper accordingly in the next version.\n\n> The size of the dataset is relatively small, restricting its use primarily for evaluation purposes; its reliance on CSI content limits its applicability to a specific genre related to crime instead of daily life scenarios.\n\nThanks for raising your concerns on the size and scope of our dataset ECHo.\nWe acknowledge that ECHo only represents a small subset of social interactions, but we view it as an initial and specific step to explore ToM understanding of social intelligence. As outlined in Table 2, crime scenes are particularly rich in ToM-related cases, including *false-belief* and *unexpected-content*. \nIn light of your comment, would it enhance the clarity and utility if we further emphasize the specialized focus on crime scenes as a distinguishing aspect of our dataset?\nIn this case, we will elaborate more on how ECHo assesses agents' ToM understanding in a specific genre, like humor detection (Patro et al. 2021).\n\n---\nMultimodal Humor Dataset: Predicting Laughter tracks for Sitcoms. Patro et al. WACV 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation for the incorporation of ToM is crucial for narrative understanding and ToM serves as a suitable intermediate step for CoT reasoning. \n\nThe dataset is carefully collected and could be useful for future research in human-centric video and narrative comprehension.\n\nThe findings indicate that ToM-enhanced models yield superior performance based on automatic measures.", "reasons_to_reject": "The size of the dataset is relatively small, restricting its use primarily for evaluation purposes. Also, its reliance on CSI content limits its applicability to a specific genre related to crime instead of daily life scenarios.\n\nSome aspects of the annotation/experiments design require further clarification.\n\nThe experiments lack systematic human evaluation for model performance. It's not clear if the small difference in automatic evaluation measures can drive meaningful conclusions.", "questions_to_authors": "A. In Characteristics Identification, are different roles of the same character collected for each plot, or does a character's role remains consistent across most plots?\n\nB. In Keyframe-Grounded Emotion Interpretation, if multiple selections are allowed, how do annotators choose key frames for each emotion selection?\n\nC. In Event Causality Inference, are there any limitations on using utterances or narrations as events? How the selection of annotated \"events\" is determined?\n\nD. Sec 5 and Fig 4 are hard for me to understand. It can be helpful to clarify what intermediate steps are involved, what is the input and output of each step, and which model and prompt are used. There are too many prompts in Appendix D and it's hard for me to map them to each intermediate step. I also don't understand why some prompt requires a character name but some do not. How does the model know which person should be focused on?\n\nE. How to get the visual frames V in Sec 5? Are they provided by the annotated dataset or automatically selected from a video? \n\nF. How many instances are rejected and revised during inference validation? What are the major error types and feedback types encountered during annotation and verification?\n\nG. Why use instructGPT instead of stronger LLMs like ChatGPT and GPT-4?", "paper_topic_and_main_contributions": "This paper presents ECHo, a diagnostic dataset focused on event causality inference grounded in visio-linguistic social scenarios. The dataset is carefully collected from CSI, a crime drama, incorporating video and screenplay information. It includes three annotation tasks: Characteristics Identification, Keyframe-Grounded Emotion Interpretation, and Event Causality Inference. The study also evaluates the performance of LLM and multimodal models in these tasks, demonstrating that a ToM-enhanced model shows improved reasoning capabilities.\n\n----\nThanks for the detailed response.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a multimodal dataset of theory of mind inferences for event causality reasoning in social scenarios, which are set in a crime setting. The paper shows that theory-of-mind enhanced models achieve better performance on the task that models that do not perform such reasoning. 
The reviewers agree that the presented task is an important and timely task, and that the dataset provides a new unique resource for this task; they also agree that the paper is presented very clearly, that the dataset is described well and that the evaluation is thorough. The concerns of the reviewers pertain to the generalizability of findings  / distributions of social inferences in the particular crime domain that was chosen to construct the dataset, but I think that the authors can address this in the discussion in the final version of the paper. The reviewers also had some questions regarding annotator agreement and a human evaluation, which are addressed well in the rebuttal, in my opinion.
Decision: Accept-Findings