ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos
Authors: Te-Lin Wu
Zi-Yi Dou
Qingyuan Hu
Yu Hou
Nischal Reddy Chandra
Marjorie Freedman
Ralph M. Weischedel
Nanyun Peng
Venue: EMNLP 2023 Main (2023)
Keywords: Counterfactual reasoning
Video Question Answering
Commonsense
Multimodal

Abstract: Multimodal counterfactual reasoning is a vital yet challenging ability for AI systems. It involves predicting the outcomes of hypothetical circumstances based on vision and language inputs, which enables AI models to learn from failures and explore hypothetical scenarios. Despite its importance, there are only a few datasets targeting the counterfactual reasoning abilities of multimodal models. Among them, they only cover reasoning over synthetic environments or specific types of events (e.g. traffic collisions), making them hard to reliably benchmark the model generalization ability in diverse real-world scenarios and reasoning dimensions. To overcome these limitations, we develop a video question answering dataset, ACQUIRED: it consists of 3.9K annotated videos, encompassing a wide range of event types and incorporating both first and third-person viewpoints, which ensures a focus on real-world diversity. In addition, each video is annotated with questions that span three distinct dimensions of reasoning, including physical, social, and temporal, which can comprehensively evaluate the model counterfactual abilities along multiple aspects. We benchmark our dataset against several state-of-the-art language-only and multimodal models and experimental results demonstrate a significant performance gap (>13%) between models and humans. The findings suggest that multimodal counterfactual reasoning remains an open challenge and ACQUIRED  is a comprehensive and reliable benchmark for inspiring future research in this direction.

ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos
Te-Lin Wu Zi-Yi Dou"l, Qingyuan Hu" Yu Hou', Nischal Chandra', Marjorie Freedman', Ralph Weischedel', Nanyun Peng', University of California, Los Angeles, UUniversity of Maryland, College Park, IInfrrmatinn Sciences Institute, University of Southern California {telinwu, zdou, hu528, nrchandra, violetpeng)ecs. ucla edu houyu@umd. edu, [mrf, weisched)iii. edu
Abstract
3rd-Person
Multimodal counterfactual reasoning vi- tal ability for AI systems. It involves pre- dicting the outcomes of hypothetical circum- stances based on vision and language inputs, which enables AI models to learn from failures and explore hypothetical scenarios. Despite its importance, there are only few benchmark datasets targeting on evaluating the counter- factual reasoning abilities multimodal mod- els. Further more, existing datasets either only cover reasoning over synthetic environments, focus only on specific types events (e-g. traffic collisions). making them hard to reli- ably benchmark the model generalization abil- ity diverse real-worke scenarios and reason- ing dimensions. To overcome these limita- tions, we develop a video question answering dataset, ACQuIRED, which consists of 3.7K an- notated videos, encompassing a wide range of event types and including both first and third- person viewpoints, ensuring real-world diver- sity. In addition, each video is annotated with questions that span three distinct dimensions of reasoning, including physical, social, and temporal, which can comprehensively evalu- ate the model counterfactual abilities along multiple aspects. We benchmark our dataset against several statc-of-the-an language-only and multimodal models and experimental re- sults demonstrate significant performance gap 13%) hetween models and humans. The find- ings suggest that multimodal counterfactual reasoning remains an open challenge and Ac- QUIRED is a comprehensive and reliable bench- mark for inspiring future research in this di- rection. Our dataset and code are at: https: //gi thub. cwm/ http//PlssaabLLL.. Pbt.au/uua..e...eeeeee
Temporal
What the tested the weight of the dolly before pushing
he tested weight of the dolly before pushing it wouldn't have dragged him down the ladder.
he tested the weight of the daily before pushing it, he would have pushed it harder down the ladder.
9
C00
0000
Physical
What if wasn't wearing gloves?
could get my hands stained with the red
could accidentally cut myself with the cutter.
The ACQUIRED dataset is video question swering (QA) dataset that specifically focuses factual reasoning on diverse real-world events. Our dataset concerns three types commonsense reasoning dimensions: physical, sncial, and temporal, encompasses videos from both third person (upper) and first-person (lower) viewpoints. Each question is curated Each answer is by itself individually judgeable, hence our dataset can be approached either binary True/False or multiple-choice setting.
Figure 1:
a correct and distractor answer.
from what actually occurred based on vision and language inputs. involves mentally simulating alternative scenarios and evaluating their potential outcomes. This cognitive process plays a crucial role in human intelligence, as it allows us to under- stand causality, make predictions, and learn from past experiences. For Al models, developing the capacity for counterfactual reasoning is signifi- cant area of research and a challenging task. By enabling AI models to engage in counterfactual reasoning, we can enhance their understanding of
1 Introduction
Multimodal counterfactual reasoning refers to the ability to imagine and reason about what might have happened if certain conditions were different
The authors contribute cqually.

---

causal relationships and their ability to assess the impact of interventions or changes in conditions. However, despite the significance of counterfac- tual reasoning, it remains a relatively unexplored area of research. To assess the overall reasoning capabilities of models, several visual question an- swering datasets have been proposed on both im- ages (Antol ct al., 2015; Johnson et al., 2017) and videos (Yi et al., 2020; Xu et al.. 2021). These datasets require reasoning skills such as common- sense reasoning, extracting human/object-to-objee relations, and inferring physical properties.
merous settings, making them ideal sources for curating video question answering datasets. In ad- dition, many videos contain unintentional human actions (e.g., the person accidentally falling down the ladder in Figure 1), which naturally enables people to come up with diverse what-if questions.
Inspired by Singh et al. (2021), adopt simi- lar methodology for gathering counterfactual ques- tions. Each question consists of pair of answers, with one being the correct response and the other serving as a distractor. Importantly, the distractor answer represents minimal contrastive counter- part to the correct answer. As we can see from examples in Figure 1, the design of using comple- mentary pairs requires the model to understand the subtle differences between different options, which ensures that the model exhibits an intuitive grasp of counterfactual reasoning. In addition, having one distractor for each question allows for testing mod- els in either True/False or multiple-choice setting. We extensively evaluate numerous strong lan- guage models such as GPT-4, well as state-of- the-art video-language models such as VALOR our ACQUIRED dataset. The experimental results suggest that models struggle to effectively utilize the video contexts and perform counterfactual rea- soning, with multimodal models achieving only comparable and sometimes inferior performance than language-only models. Moreover, the signif- icant gap between the human and model (>13%) performance highlights the challenging nature of our task and room for improvements visual coun- terfactual reasoning.
One specific dataset in the realm of counter- factual reasoning is CLEVRER (Yi et al., 2020), which generates synthetic videos and associated questions in controlled environment, featuring simulated object motion and rendered video frames. This dataset allows for evaluating models using de- scriptive, explanatory, predictive, and counterfac- tual questions, covering a wide range of reasoning scenarios. However, the data generation process in CLEVRER is overly synthetic, limiting its effec- tiveness to assess models' counterfactual reason- ing abilities in realistic contexts. To address this limitation, TrafficQA (Xu et al., 2021) focuses on real-world traffic event cognition and reasoning in videos, specifically targeting scenarios like traffic accidents. It leverages crowdsourcing gather diverse types of questions, including fundamen- tal comprehension, counterfactual inference, and event forecasting. Nevertheless, because TrafficQA concentrates solely on traffic events, it fails to en- compass other real-life events, resulting in sub- stantial domain gap between TrafficQA and general video datasets such Kinetics (Kay et al., 2017; Smaira et al., 2020) and YouTube (Abu-El-Haija et al., 2016; Zellers et al., 2022).
2 Related Work
We will overview three lines of relevant research to this work: visual question answering, visual under- standing models, and counterfactual reasoning.
In this paper, we construct benchmark that can evaluate the counterfactual reasoning abilities of visual models on various kinds of real-world events. We introduce ACQUIRED! that covers multiple di- mensions of counterfactual reasoning and includes videos of both egocentric and exocentric views. Specifically, based on videos in both Oops (Epstein et al., 2020) and Ego4D (Grauman et al., 2022), we crowddsoouree souree IIK questions over 3.7K videos targeting physical, temporal, and social counterfac- tual reasoning. Both the Oops and Ego4D datasets consist of human activities and interactions in nu-
Visual Question Answering Datasets. In Table 1. we list several representative visual QA datasets as well as their key features. The Visual Question An- swering (VQA) dataset (Antol et al., 2015) is one of the pioneering works in this direction and has been standard benchmark for evaluating the reason- ing ability of image-language models (Goyal et al., 2017). Follow-up datasets such as CLEVR (John- son al., 2017) and GQA (Hudson and Manning, 2019) automatically construct compositional ques- tions over real or synthetic images and perform the evaluation in a systematic way. To further evalu- ate the commonsense reasoning ability of models,
'Ahbreviation of: Answering Counterfactual Questions In Real-Lifee Vidcos

---

Reasoning Domain
Dataset
Visual Source
Question Source
Physical
Temporal
Social
Counterfactual
Image QA datasets
VQA (Antol et al., 2015)
Diverse Real-world Event Synthetic Object Diverse Real-world Event Movic
Human
x
x
X
CLEVR (Johnson
2017)
Automutic
GQA (Hudson and Manning, 2019)
Automatic Human
K
K
VCR (Zellers et 2019)
X
Video QA datasets
CLEVRER al., 2020)
Synthetic Object Collision TV & YouTube
Automatic
V
VLEP (Lei at., 2020)
Human
X
X
X
MovieQA (Tapaswi
2016)
Movie
Human
MSRVTT-QA (Xu sl., 2017)
Diverse Real-world Event
Automatic
X
TGIF-QA (Jang et al., 2017)
Tumblr GIF
Automatic Human
X
MarioQA (Mun al. 2017)
Gameplay Video
Automatic
X
TVQA (Lei et al., 2018)
TV
Huraa
Social-IQ (Zadeh al., 2019)
Human
X
TraflicQA (Xu al., 2021)
Traflic Event
Human
NEXT-QA (Xiao ct al., 2021)
Diverse Real-worid Event
Human
X
Causal-VidQA al., 2022)
Diverse Real-world Event
Human
ACQUIRED
Diverse Real-workd Event
Table 1: Comparisons different visual question answering datasets, ACQUIRED first to feature all the dimensions.
VCR (Zellers et al., 2019) crowd-sourues common- sense ceesstion-aswwer pairs associated with ratio- nales over static images extracted from movies. Video question answering is more challenging than image question answering and is gaining increasing attention from the research community, leading to several video QA datasets being constructed (Lei
different viewpoints.
Visual Understanding Models. The creation of vi- sual QA benchmarks allows the development of visual understanding models. Many of the previous works have tried to solve these tasks using com- positional approaches and scene graphs (Santoro et al., 2017; Hu et al., 2017; Hudson and Manning, 2018; Perez al., 2018; Yi et al., 2018; Shi et al., 2019; Gao et al., 2020; Ding al., 2021). For example, Hu et al. (2017) propose to train mod- ular network in an end-to-end manner achieve both effectiveness and interpretability; Hudson and Manning (2018) utilize scene graphs and perform differentiable neural operations on the graphs to perform visual reasoning. Inspired by the suc- cess in pretraining on Internet-scale data (Devlin et al., 2019), pretraining models on large vision and vision-language tasks and then finetuning them specific downstream tasks has become a standard in tackling visual understanding tasks (Sun et al., 2019; Li et al., 2020; Zhu and Yang, 2020; Lei al., 2021; Zellers et al., 2021; Fu et al.. 2021: Zellers et al., 2022; Wu et al., 2022). Existing works in this direction generally train models on large vision-language datasets with objectives such masked language modeling and video-text match- ing. Despite the great progress in this direction. unclear these models can perform counter- factual reasoning. To address this, we benchmark ACQUIRED against state-of-the-as models and sys- tematically study their performance.
al., 2020; Tapaswi et al., 2016; Xu et al., 2017;
Jang et al., 2017; Mun et al., 2017; Lei et al.. 2018).
Among them, CLEVRER (Yi et al., 2020) improves upon CLEVR and uses programmatically generated videos capturing collisions of synthetic objects to evaluate the model reasoning abilities along multi- ple dimensions. Social-IQ (Zadch et al., 2019) and TrafficQA (Xu et al.. 2021) employ videos depict- ing real-world events, wherein Social-IQ primarily emphasizes human social interactions, while Traf- fieQA focuses on traffic events and accidents. To improve the diversity of the captured events, NExT- QA and Causal-VidQA collect videos from diverse domains and have geting different dimensions of reasoning.
questions tar-
As can be seen Table 1, among all the visual QA datasets, there are only a few that attempt to evaluate the counterfactual reasoning abilities of models. In addition, the existing benchmarks are often limited in terms of the video sources and the question types, making it difficult to evaluate the model performance in diverse real-world setting. ACQUIRED is the first dataset that can comprehen- sively evaluate the model counterfactual reasoning abilities spanning three distinct dimensions (i.e., physical, social, and temporal) and cover videos that include wide range of event types and from
Causal and Counterfactual Reasoning. Humans can infer how an event would have unfolded differ-

---

ently without experiencing this alternative reality and it has been long-standing research topic in cognitive psychology (Van Hoeck et al., 2015). To empower such an important ability to artificial in- telligence, researchers have tried to build learning models that can infer causal relations and perform reasoning in various fields (Qin et al., 2019; Yi et al., 2020; Baradel et al., 2020; Abbasnejad et al., 2020; Yue et al., 2021; Wang al., 2021). Our con- structed benchmark provides a valuable resource for developing and evaluating visual models with counterfactual reasoning abilities.
three dimensions: physical, social, and temporal. The physical dimension concerns the knowledge of objects involved in the events and their proper- ties (e.g., shape, size, functionalities, affordances), as well as the motion and location of the events. The social dimension looks at human social be- haviors. particularly attributes such as personal- ity, emotions, inner interests/intentions. and social activities. The temporal dimension regards the aspects of events/activities in their temporal order- ings, duration, and frequency/speed of motions.
The three main dimensions are the building blocks towards comprehensive commonsense rea- soning, and helps systematically analyze in which aspects the models need to be improved upon more. Although some questions can be answered using more than one commonsense dimension, we ask the annotators label with the main one used.
3
The ACQUIRED Dataset
3.1 Dataset Design & Collection
Problem Definition. As illustrated in Figure I and Table 2, each data point in ACQUIRED con- sists of a video and corresponding annotated ques- tion and answer pairs. We inspired by prior works (Clark al., 2019; Singh et al., 2021) to consider the surprisingly difficult nature of the T/F (yes/no) QA formats that could potentially exhibit less unintended biases/artifacts than curating data in the multiple choice (MCQ) settings. In light of this, for each question, we collect one correct and one distractor answer (which can be slightly perturbed version of the correct onc), where both of which are individually judge-able by themselves respectively. And hence, our dataset can be ap- proached as a binary True/False (T/F) prediction task as well as a multiple-choice (MCQ) (2 choices in this case) question answering task.
Video Resources Sampling, We utilize the Oops! (Epstein et al., 2020) dataset for third-person view videos and Ego4D (Grauman et al.. 2022) for first-person views, where both of which feature text descriptions of the video contents. Oops! con- cerns predicting the failing (oops) moment of intended action in video, and hence is event-rich and good testbed for reasoning what could the outcomes turned out differently. Ego4D collects videos of humans performing daily activities the first-person view, which adds a desirable task- knowledge layer on top of its event-richness.
As we are annotating subsets of videos from the aforementioned sources, have the privilege encourage more balanced key events distribution from the vidcos to be annotated. Specifically, we (I) use NLP tools such as semantic role labeling (SRL) to extract key verbs (events) for each video description', and group the videos accordingly. (2) each time sample an event group with a proba- bility inverse proportional to the current launched key event distribution, (2) sample video from the event group in (3), and repeat until reaching desired number of videos (to be annotated).
It is worth noting that the distractors in our dataset are manually curated with certain twists towards the correct answers (examples in Table 2). forcing the models to truly understand the visual concepts involved in the counterfactual questions in order to answer correctly.
In Section 5.2, we will describe our adoption of pairwise consistency metric that requires the model to answer correctly in both correct and dis- tractor directions to be regarded a success, in order to reduce the models' exploiting surface-level heuristics to predict the answers.
The sampling strategy, combined with our pre- defined reasoning dimensions and video domains, is designed to improve the diversity of question- answer pairs.
Commonsense Dimensions. We adopt the commonsense knowledge catcgorization proposed in (Singh et al., 2021), which is inspired by the Theory of Core Knowledge (Spelke and Kinzler, 2007) to collect QAs that focus the following
Collection Workflow. We collect our dataset via
motions, and the social world. "As most videos from Ego4D show tasks performed solely by the camera wearer without social interactions, we do not require the social dimension to be annotated.
TThe capability of reasoning about physical objects, places,
"We use the originally annotated narrations in Ego4D.

---

Sub-sampled Key Video Frames
Question-Answer Pairs
(Temporal) Q: What if the two persons had swerved to their left before reaching the shore?
Correct: They would not have had a beach landing. Wrong:
They would have had a beach landing.
(Social) Q: What if the skier was stranger to the two people standing still?
Correct: skier does not throw snowball.
Wrong: The skier still throws the snowhall.
(Physical) Q: what if the wheel was in a bike?
Correct: Hc would need to take out the screw before being able to set the wheel on the table Wrong: He would set whole bike along with the wheel on the table.
(Physical) Q: What if I let the cutting board lie the counter?
Correct: The cutting board would be dried slower.
Wrong: The cutting board would be dried quicker as it occupies
a larger urea.
Table 2: Sample data points of the ACQUIRED dataset.
Amazon Mechanical Turk (MTurk). Each MTurk worker asked to carefully watch a given video for creating the QA pairs. As depicted in Figure 2, our dataset collection process comprises four main steps: (1) We design a qualification questionnaire focusing on examining one's understanding of the key concepts in our problem design, i.e., the con- cept of counterfactuality, the requirement of video relevancy, common sense reasoning dimensions. and what types of QA pairs are more desirable. (2) Once the workers pass the qualification test, they are directed to an interface where a pretrained (text-only) QA model is deployed in the loop of the QA creation process. Bonus monetary rewards are given if the deployed model fails to predict correctly the creations. (3) Internal members then conduct quality validation on the created sam- ples and provide customized tips and/or feedback to the workers for potential improvements. (4) Lastly, our deployed model is iteratively finetuned on the validated samples after each batch of annotations, which results in constantly improved model to incentivize more challenging sample creations.
Paquinnmeet Vidao Rehoonss
Quality Validation
Getine Rewards
</>
Ousiftortio Boundiol
Ademrrtall Same
Trannd Textod Model
Figure 2: Data collection workflow.
Quality Validation. order to further ensure the sample quality as well as summarize common mistakes to provide custom human feedback to the annotators, our internal members conduct the second- phase manual sample validation in conjunc- tion with the deployed model results. We cross- validate the annotations among internal mem- bers in the ramping-up up phase ensure quality. We also accumulate detailed guidelines from our man- ual validation process for providing effective feed- back. After scaling up, we continue to validate the annotations via uniform subsampling across each annotator. Our validation criteria are well aligned as can be seen in the high 0.85 Kappa score for commonsense dimension agreements; and 0.91 overlapping ratios for video relevancy.
Integrating the model-in-the-loop protocol into the pipeline not only brings benefits in curating more challenging samples, but also helps diversify the answers as the models will not be easily fooled if there are similar patterns existed in the dataset or the questions can be simply guessed without visual inputs.
'We did not use Kappa score for video relevancy because there un unbalanced "agreed" distribution of "yes" and "no" (22:1) in our validation results for this criteria, which would result in unfair Kappa score.

---

Batches
Annotation Drop-Rate (%)
Number of Videos
Videos From
Avg. Fool Rate (%)
Avg. Fool Accuracy
Batch-1
28
50
Oops! Ego4D 51.43
57.69
42.31 48.57
Batch-2
17.3
100
Butch-3
4.3
200
Butch-4
3.5
200
Table 5: Deployed model fooling rates during collection.
Batch-5
2.6
200
Table 3: Annotation drop rate for the first batches. Each
per-second rate (FPS) of videos from either souree is mostly 30.
vidco gives 3 pairs of question correct/distractor answers.
Type
Counts
Key Annotated Events. We plot the distributions of most frequent key verbs (for main event types) and nouns (for entities involved in events) in Fig- ure 3a and Figure 3b, respectively, to have rough visual inspection the diversity of the created sam- ples. The key verbs/nouns are firstly determined by the SRL parses of the question and answer sen- tences (separately considered), and followed by lemmatization. Both plots are summaries of the two video sources, and more plots broken down by video sources and comparisons with existing works are in Append. Sec. A.2.
Total Unique Videos
2,664
Total Unique QA-Pairs
7,853
Type- Token Ratio
0.0158
Verb-Token Ratio (total verb-types)
0.0341
Tork-Token Ratio (total tokens)
0,0034
Noun-Tokes Tokn-Toknnaaa Ratio (total noun-types)
0.0796
Noun- Token Ratio (total # tokens)
0.0063
Physical / Social Temporal (%)
34/ /3/33// 1/33//3 /3
Type
Mean Std
Max Min
Tokens in a Question
11.5
3.5
Tokens in an Answer
6.6
Video Frames (Count)
296.8
207.3 3283
Vidco Duration (scc)
10.7
7.1
11166
3.2
(a) Videos from Oops!
Deployed Model. Table 5 reports the model fool- ing rates in collected data across the two data sources. We encourage our annotators to develop QA pairs that can successfully fool our model by setting up monetary rewards and unlimited trials.
Type
Counts
Total Unique Videos
1,038
Total Unique QA-Pairs
2,695
Type-Toker Token Ratio
0.0205
Verb-Token Ratio (total # video-types)
0.0586
Verb-Toker Ratio (total tokens)
0.0045
Noun-Toker Token Ratio (total noun-types)
0.1054
Benchmarking Models
Noun-Token Rutio (total # tokens)
0.0081
Physical / Social Temporal (%)
We benchmark our dataset with both state-of- the-art language-only and vision-language mod- els. Specifically, we perform experiments with De- BERTa (He et al., 2021), UnifiedQA (Khashabi et al.. 2020), VIOLET (Fu et al., 2021), VALOR (Chen et al., 2023), and VL-Adapter (Sung et al., 2022) on our dataset.
Type
Mean Std Max Min
Tokens in a Question
11.1
3.3
32
6
Tokens in Answer
9.4
6.0
41
5
Video Frames (Count)
399.
54.8
572
240
Video Duration (sec)
13.3
1.8
19.3
8
(b) Videos from Ego4D
Table 4: General statistics of the two video domains.
Language-Only Models. While ACQUIRED is a multimodal dataset that has both vision and lan- guage inputs, previous works (Thomason et al.. 2019) have pointed out that unimodal models sometimes achieve surprisingly strong performance because of the annotation bias. Therefore, we eval- uate both DeBERTa-v3 (He et al., 2021) and the UnifiedQA model family (Khashabi et al., 2020) (state-of-the-art question answering models based on the T5 architecture (Raffel et al., 2020)) on our dataset, which can reflect the dataset biases and pro- vide important reference point for multimodal models. The language-on models answer the tex- tual questions without looking at the videos.
Validation Analysis. Table 3 reports the data drop- rates (majority voted to drop by all three validators) for the first 5 batches. We hope these rigorous safety checks can ensure a good data quality that also closely follows our guideline, and the vali- dation should by no means introduce unnecessary biases as we indeed saw a decrease in the dropping rates in our later collection batches.
3.2 Dataset Statistics
General Statistics. Table summarizes the essen- tial statistics of the collected dataset, where Table 4a is for videos obtained from the Oops! (Epstein et al., 2020) dataset whereas Table 4b is for videos from Ego4D (Grauman et al., 2022). The frame-
Inspired by the superior performance of the re- cent large language models, i.e., the GPT model from OpenAI, we also evaluate its zero-shot perfor-

---

vlin et al., 2019)), and cross-modal transformer module that performs cross-modal fusion. The video and language encoders extract features from the video and language inputs respectively, and the extracted features are then fed into the cross- modal transformer for cross-modal interactions. VIOLET is pretrained on large-scale video-text data with masked language modeling that predicts the original word tokens given the masked inputs, masked visual-token modeling (MVM) that recov- ers the masked video patches conditioned on the unmasked video and language inputs, visual-text matching that aims to align the paired video-text inputs between video and text modality.
we faadd
come hurt prevent
have
ride
Teel
crash
throw.
oos
turn.
wait
help
pull
pick
drive
put
catch
see
Fall
start
hit
run
happen
make
break
hold
use
avoid
get
wear
jump
Rove take
VALOR (Chen et al., 2023). VALOR is a recently proposed multimodal model that can take video, language, well as audio inputs. Similar VIOLET, VALOR also first encodes vision, audio, and text inputs separately, and the encoded features are then fed into a multimodal decoder for text gen- eration. VALOR demonstrates strong performance across a wide range of tasks, including video re- trieval, video captioning, vidco question answering, audio-visual captioning, text-to-audio retrieval, and audio captioning.
(a) Verhs
way bike
ba balage
jump kid
door
side
child
cat.
meterial
minute.
glass
friend
force
day-
wall
person
VL-A (Sung al., 2022). VL-Adapter uses pretrained vision encoder (e.g. CLIP (Radford et al., 2021)) to extract vision features and feed the vision features as well as text tokens to a pretrained language model (e.g. T5 (Raffel et al., 2020)) so that the model can take both vision and language information. When adapting the model for down- stream tasks, because it can be costly to finetune all the model parameters, VL-AAapter investigates different adapter-based parameter-efficient finetun- ing strategies and demonstrates that training the adapter allows them to only update a rather small portion (e.g. 4%) of total parameters and match the performance of finetuning the entire model. Be- cause VL-AAdaterr supports different combinations of pretrained vision and language encoders, employ different versions of CLIP-ViTI and UnifiedQA-Large as the vision and text encoders.
floor
man
glove
hand
face
woman
people
boy
driver
water
second
girl
head
ball guy ground dog time
(b) Nouns
Figure Top-40 frequent word-types in the dataset.
mance the textual parts of our dataset. Specifi- cally, consider both ChatGPT (OpenAT, 2023a) and GPT-4 (OpenAI, 2023b). In addition, we fur- ther include a version of GPT models that can condition on pre-annotated descriptions describing the general contents the videos, to serve as the pscudo visual (and situated) contexts of the ques- tions. Details on how we prompt the GPT family are in Append. Sec. C.I.
Experiments and Analysis
5.1 Training and Implementation Details
VIOLET (Fu al., 2021). VIOLET is video- language model that has three components, includ- ing a vidco encoder (Swin Transformer- (Liu et al., 2022)), a language encoder (BERT-base (De-
We obtain the pretrained weights of all the bench- marking models from their respective open-sourced releases and finetune them on our official training data split. The hyperparameters are manually tuned

---

Dimension Breakdowns Social
Modality
Model
QA-Format
Viewpoints
Overall Accuracy? (%)
Physical
Temporal
T/F
70.12
70.61
70.32
69. 69.19
DeBERTa-V
MCQ
70.35
72.10
68.62
69.01
T/F
68.93
70.22
69.32
66.33
UnifiedQA-bans
MCQ
67.63
68.53
69.01
65.13
T/F
69.59
71.00
69.88
67.18
UnileeQAAlarge
MCQ
70.38
71.57
71.83
67,38
Text-Only
T/F
70.49
70.58
72.20
68.99
UniliedQA-BB
T/F (Pair.
54.91
55.31
56.21
53.26
MCQ
73.40
73.36
75.80
71.60
Vanilla ChatGPT
T/F
52.80
$1.36
48.06
54.04
T/F
55.20
50.82
52.90
52.48
Desc-ChaiGPT
MCQ
42.40
36.96
43.22
47.83
Vanilla GPT-4
T/F
53.80
53.89
53.16
54.32
T/F
56.20
$5.00
58.23
55,56
Desc-GGP--4
MCQ
60.80
61.41
55.48
65.22
T/F
All
66.15
70.20
64.45
60.24
VIOLET
T/F (Pain.)
All
48.25
54.03
44.60
40.63
MCQ
All
69.33
70.20
70.23
67.19
T/F
All
63.83
66.54
62.50
60.02
VALOR
T/F (Pair.)
AlI
43.00
46.51
42.46
37.26
MCQ
AlI
55.06
58.28
51.76
51.69
All
68.75
71.56
67, 94
64.40
Multimodal
T/F
3rd
66.32
66.01
67.90
65.07
1st
72.63
75.49
62.82
All
51.19
54.27
49.56
47.74
VL-Adapter
T/F (Pair.)
3rd
47.82
47.60
49.50
46.40
1st
60.40
62.23
53.44
All
71.53
72.70
70.39
70.25
MCQ
3rd
69.13
70.35
1st
75.34
76.29
72.05
Human Performance
T/F
All
83.60
81.82
77.27
MCQ
All
92.59
90.91
100
90.91
Table 6: Model benchmarking performance on our ACQUIRED dataset.
for each model, and the checkpoints used for test- ing are selected by their validation performance.
both individual judgments are correct in each pair. Training Details. All the models in this work are trained on multi (at least 2-4) Nvidia A100 GPUs on Ubuntu 20.04.2 operating system.
5.2 Experimental Setup
Data Splits. For our official (to-be-released) dataset, we follow a 5 50 ratio and randomly split the train-dexelopment-tes datasets. The train split is mainly to adapt models to our QA task settings as well as the counterfactual reasoning style. We ensure that there are overlaps between videos of different sets and the Oops! and Ego4D videos equally distributed in each of the splits, Evaluation Metrics. Models are evaluated by simple accuracy metric, for both T/F and MCQ settings. We also further ablate the model perfor- mance along the commonsense dimensions and/or viewpoints, for more detailed performance break- down and analysis. We also include the pairwise accuracy in the T/F setting following Singh et al. (2021), where the model is considered correct
We train our models until performance conver- gence is observed on the training split (determined by the development set performance). All of the hyperparameters are manually tuned and searched, with multiple trials for better performance and train- ing convergences.
5.3 Experimental Results
Table reports benchmark performance. The best- performing multimodal model (VL-Adapter) per- forms slightly better than its text-only counterparts, UnifiedQA-large (i.e., the language encoder of our VL-Adapter). While this shows that visual con- texts and multimodality are effective, the perfor- mance gap is not substantial; therefore, there is

---

room for improvement, and more effective meth- ods of multimodal inputs are yet be explored. While text-only UnifiedQA-3B achieves overall better performance in both T/F and MCQ settings, potentially due to its much larger learnable param- eter space, its mediocre pairwise accuracy suggests that the model is still inept at robust counterfactual reasoning in the two facets of the same question. In general, models perform better in the MCQ settings than the T/F ones. This is intuitive because in the MCQ settings, the model is aware that only one of the two given options is correct and only needs compare them and select the more rea- sonable option. (Such a phenomenon is also stud- ied/discovered in (Clark et al., 2019; Singh al., 2021)) the case of ChatGPT, its MCQ setting accuracy is lower than that of the T/F setting com- pared to others. We suspect that ChatGPT might have weaker reasoning ability compared with GPT4, We observe that often ChatGPT refuses give answer in the MCQ settings because of insufficient conditions while leans towards false when it was asked the same question in a T/F setting.
and/or human social knowledge. Viewpoints. We take the best-performing mul- timodal model (VL-Adapter) and ablate its per- formance along different video viewpoints. We find that, despite being pretrained mostly on third- person viewpoint videos, the generalization ability of the models towards first-person viewpoints is sufficiently good. However, as the videos from Ego4D are not intended to explicitly contain failed actions from the camera wearers, it could be more challenging for our annotators construct diverse and subtle counterfactual questions compared the videos from Oops!. Nevertheless, argue that the counterfactual reasoning ability of the models should be equally crucial regardless of vidco view- point, and our dataset can inspire relevant research serving as a first-of-its-kind counterfactual video QA encapsulating videos from varying viewpoints.
6 Conclusions
In this work, we present novel counterfactual- reasoning-focusee video question answering dataset, named ACQUIRED. The dataset provides questions about counterfactual hypotheses over vi- sual events (videos). We collect a correct and a distractor answer for three commonsense reason- ing dimensions: physical, social, and temporal. We benchmarked various state-of-the-art language models (including LLMs like GPT) and video- language models on the collected dataset, where the results demonstrate algorithm performance well below human performance (>13% accuracy). We hope our studies and the collected ACQUIRED dataset can spur relevant future research, specif- ically testing multimodal models' capabilities in counterfactual reasoning. devising assistive AI for remedial and/or cause estimation of observed failures, and more sophisticated visual event under- standing and reasoning.
Perhaps surprisingly, despite the remarkable ca- pabilities of the GPT series, they do perform as impressively, even when provided with descrip- tions transcribing the major visual events in the videos. This suggests that the annotators our curation task indeed closely examine many visual details in order to create more challenging samples.
Human Performance. We randomly sub-sample 500 videos to estimate human performance: these are reported in the last two rows of Table The human performance highlights a significant gap above all the model results, especially for the MCQ settings. We hope future modeling endeavors can close the gap in visual counterfactual reasoning.
Commonsense Dimensions. The rightmost parts Table 6 report the performance breakdown along commonsense reasoning dimensions. We observe a general trend: most of the models perform bet- ter in physical and social dimensions compared to the temporal dimension; the physical dimension generally exhibits the highest performance. That observation implies that, even after being finetuned on dataset, the models still fall short of captur- ing temporal commonsense as opposed to the other two kinds of knowledge. This can also be hypothet- ically attributed to the fact that the pretraining data for the language models encapsulate more physical

---

(2) The videos used in this work are subsets of readily collected ones from both Oops! (Epstein et al., 2020) and Ego4D (Grauman et al., 2022) mother sets, and hence the event distribution can be bounded by the activities they concern. While we argue that the dataset is, to our best knowledge, first of its kind video QA dataset in terms of di- versity and dedication of counterfactual reasoning, the video resources spanning even more diversified situations can be further extended. We will release the manuscripts and our collection tools to help spur future relevant research in such endeavours. 7 Limitations
Amazon Mechanical Turk (MTurk) and ensure that all the personal information of the workers involved (e.g., usernames, emails, urls, demographic infor- mation, etc.) is discarded in our dataset. Although for a single same video, there is only one single worker annotating the corresponding questions, we ensure that the similar type of events get annotated by as diverse worker pools as possible during the collection phase. We do not foresee much unin- tended biases within the annotations as they should focus on the contents of the videos for physical and event temporal domains, and we make efforts on reducing the potential biases on the social domain by providing periodic email-based training to the workers to diversify the creations.
We hereby discuss the potential limitations of our work:
Our work focuses on the three commonsense dimensions: physical, social, and temporal. While they likely span the most common types of the reasoning technique, there could be more, e-g., nu- merical commonsense is not specifically dealt with in this work, nor is non common activities such as fantasies and fictions involved. For future models benchmarked against our dataset, this should be taken account for, i.e., should the models excel at these commonsense dimensions for counterfactual reasoning. we cannot guarantee it is a complete model on all types of reasoning scheme.
This research has been reviewed by the IRB board and granted the status of an IRB exempt. The detailed annotation process (pay per amount of work, guidelines) included in the appendix; and overall, we ensure our pay per task is above the the annotator's local minimum wage (approximately $15 USD / Hour). We primarily consider English speaking regions for our annotations as the task requires certain level of English proficiency.
Techniques. We benchmark the proposed video counterfactuality reasoning task with the state- of-the-art large-scale pretrained language models (both language-only and multimodal). As the coun- terfactual commonsense reasoning and its under- standing are of our main focus, we do not anticipate production of harmful outputs, especially towards vulnerable populations, after training (and evaluat- ing) models on our proposed task.
(3) Unlike Oops!, there not an obvious failed actions occurred in Ego4D, and hence the anno- tated questions could be confounded by more imag- ined situations. We argue that the required rea- soning technique is essentially the same and the models learn on our dataset should generalize well to situations that actually involve failing actions from egocentric visual contexts. However, we courage future research to extend the first-person viewpoint (egocentric) parts to encompass obvious failing actions to collect just-in-time assistive ques- tions and their corresponding remedial responses.
Acknowledgments
Many thanks to J.R. Bronkar for his help on the development of our data curation user interface, and to I-Hung Hsu for his valuable feedback dur- ing our paper preparation. This material is based on research supported by the Machine Common Sense (MCS) program under Cooperative Agree- ment N66001-19-2-444 under Cooperative Agreement HR001 1 12390060, both with the US Defense Advanced Research Projects Agency (DARPA). The views and con- clusions contained herein are those of the authors and should not be interpreted as necessarily repre- senting DARPA, or the U.S. Government.
8 Ethics and Broader Impacts
We hereby acknowledge that all of the co-authors of this work are aware of the provided ACL Code of Ethics and honor the code of conduct. This work is mainly about collecting a sizable video question an- swering dataset that mainly focuses counterfac- tual reasoning abilities and systematically probing such capability from ofte-of-theeart multimodal and large language models.
and the ECOLE program
Dataset. We collect the human annotation of the question-ane pairs to a prompted video via

---

References
Yash Goyal, Tejas Khot, Douglas Summers-Sun Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognitiin (CVPR).
Ehsan Abbasnejad. Damien Teney, Amin Parvaneh, Javen Shi, and Anton van den Hengel. 2020. Counter- factual vision and language learning. In Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR).
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. 2022. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Sami Abu-El Sou-EI-HE El-Haija. Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vjayanarasimhan. 2016. Youtube- 8M: A large-scale video classification benchmark. arXiv preprint.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question an- swering. In International Conference on Computer Vision (ICCV).
Pengcheng Hc, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced bert with disentangled attention. In ICLR.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR).
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. 2020. Cophy: Counterfac- tual learning of physical dynamics. In International Conference Learning Representations (ICLR).
Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. 2023. omni-perception pretraining model and dataset. arXiv preprint.
Drew A Hudson and Christopher D Manning. 2018. Compositional attention networks for machine rea- soning. In International Conference on Learning Representations (ICLR).
VALOR: Vision-audio-languae
Christopher Clark, Kenton Lec, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In North Amer- ican Chapter of the Association for Computational Linguistics (NAACL-HLT).
Drew A Hudson and Christopher D Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Jacob Devlin, Ming-Wi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In North American Chapter of the Association for Computational Linguistics (NAACI-HILTT), pages 4181-4186.
Yunscok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. TGIF-QA: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR).
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR).
Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Josh Tenenhaum, and Chuang Gan. 2021. Dynamic visual reasoning by learning diffcrentiable physics models from video and language. Neural Information Pro- cessing Systems (NeurlPS).
Dave Epstein, Boyuan Chen, and Carl Vondrick. 2020. Oops! predicting unintentional action in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang. Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The kinetics human action video dataset. arXiv preprint.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. 2021. VIOLET: End-to-end video-language trans- formers with masked visual-token modeling.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sab- harwal, Oyvind Tafjord. Peter Clark, and Hannaneh Hajishirzi. 2020. UnifiedQA: Crossing format bound- aries with a single qa system. In Findings of the Association for Computational Linguistics: EMNLP.
Difei Gao, Ke Li, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2020. Multi-modal graph neural network for joint reasoning vision and scene text. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).

---

Jie Lei, Linjie Li, Luowei Zhou. Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: ClipBERT for video-and-language learning via sparse sampling. In Proceedings of the IEEE Con- ference on Compuler Vision and Pattern Recognition (CVPR).
Wei Li, and Peter Liu. 2020. Exploring the lim- its of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR).
Adam Santoro, David Raposo, David G Barrett, Ma- teusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural net- work module for relational reasoning. In Neural Information Processing Systems (NeurIPS).
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Localized, compositional video ques- tion answering. In Empirical Methods in Natural Language Processing (EMNLP).
2018. TVQA:
Jiaxin Shi, Hanwang Zhang, and Juanzi Li. 2019. Ex- plainable and explicit visual reasoning over scene graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Jie Lei. Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020. What is more likely to happen next? video- and-language future event prediction. In Empirical Methods in Natural Language Processing (EMNLP).
Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormo- labashi, Te-Lin Wu, Xuezhe Ma, and Nanyun Peng, 2021. Com2sense: A commonsense reasoning bench- mark with complementary sentences. In Association for Computational Linguistics (ACL).
Jiangtong Li, Li Niu, and Liqing Zhang. 2022. From representation to reasoning: Towards both evidence and commonsense reasoning for video question- answering. In Proceedings of the IELE Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. 2020. A short note the kinetics-700-2023 human action dataset. arXiv preprint.
Linjie Li, Yen-Chun Chen, Yu Cheng. Zhe Gan, Licheng Yu, and Jingjing Liu. 2020. HERO: Hierarchical encoder for videot language omni-representation pre- training. In Empirical Methods in Natural Language Processing (EMNLP).
Elizabeth S Spelke and Katherine D Kinzler. 2007, Core knowledge. In Developmental science, volume 10, pages 89-96. Wiley Online Library.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. 2022. Video swin trans- former. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. VideoBERT: A joint model for video and language representation learn- ing. In International Conference on Computer Vision (ICCV).
Jonghwan Mun, Paul Hongsuck Seo, Nlchae Jung, and Bohyung Han. 2017. MarioQA: Answering ques- tions by watching gameplay videos. In International Conference Computer Vision (ICCV).
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. VL-adapter: Parameter-efffiient transfer learning for vision-and-language tasks. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR).
OpenAI. 2023a. Chatgpt.
OpenAI. 2023b. Gpt-4 technical report.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018 FiLM: Visual reasoning with general conditioning layer. In Asso- ciation for the Advancement of Artificial Intelligence (AAAI).
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. MovieQA: Understanding stories in movies through question-answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019, Counterfactual story reasoning and generation. In Empirical Methods in Natural Language Processing (EMNLP).
Jesse Thomason, Daniel Gordon, and Yonatan Bisk. 2019. Shifting the baseline: Single modality perfor- mance on visual navigation & qa. In North American Chapter of the Association for Computational Lin- guistics (NAACL-HLT).
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020.
Nicole Van Hoeck, Patrick D Watson, and Aron K Bar- bey. 2015. Cognitive neuroscience of human counter- factual reasoning. Frontiers in human neuroscience.
Wenjie Wang, Fuli Feng, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. 2021. Clicks can be cheating: Counterfactual recommendation for miti- gating clickbait issue.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou,

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: state-of-the-ar natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 48845, Online. Association for Computational Linguistics.
commonsense reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR).
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salchi. Aditya Kusu- pati, Jack Hessel, Ali Farhadi, and Yejin Choi. 2022. Merlot reserve: Neural script knowledge through vi- sion and language and sound. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. In Neural Information Processing Systems (NeurIPS).
Te-Lin Wu, Alex Spangher, Pegah Alipoormolabashi, Marjorie Freedman, Ralph Weischedel, and Nanyun Peng. 2022. Understanding multimodal procedural knowledge by sequencing multimodal instructional manuals. In Association for Computational Linguis- tics (ACL).
Linchao Zhu and Yi Yang. 2020. ActBERT: Learning global-local video-text representations. In Proceed- ings of the IEEE Conference Computer Vision and Pottern Recognition (CVPR).
Junbin Xiao, Xindi Shang. Angela Yao, and Tat- Senn Chua. 2021. NEXT-QA: Next phase of question- answering to explaining temporal actions. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR).
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang. Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined atten- tion over appearance and motion. In ACM Interna- tional Multimedia Conference (ACM MM).
Li Xu, He Huang, and Jun Liu. 2021. SUTD :UTT-TraaiiQQA: question answering benchmark and an efficient network for video reasoning over traffic events. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR).
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jia- jun Wu, Antonio Torralba, and Joshua B Tenenbaum. 2020. CLEVRER: Collision events video repre- sentation and reasoning. In International Conference Learning Representations (ICLR).
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Neural Information Processing Systems (NeurIPS).
Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. 2021. Counterfactual zero-shot and open-set visual recognition. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR).
Amir Zadeh, Michael Chan, Paul Liang, Edmund Tong, and Louis-Philippe Morency. 2019. Social- IQA: question answering benchmark for artificial social intelligence. Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR).
Rowan Zellers, Yonatan Bisk, Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual

---

A Details of The Dataset
Our dataset consists of mixture QA pairs col- lected from two data sources: Ego4d and Oops!. For each dataset split, we create an indexing json file and summarize each QA instance with video id (index), domain (physical/sciia/tmmpraal) type (counterfactual), question, correct answer, distractor, and key to the correct answer and video link URL. Our official data release will encompass all the aforementioned essential fields.
yea
woof
cole
stand
ride
ettempt
turn
feel.
stay
prevent
cresh
do
throw
pull
use
fall
start
happen
could
jump
A.1 Dataset Splits
help
get
We split our data into train/val/test based on the ratio 0.45/0.05/0.5, with each unique video only appearing in one split.
break
make
hold
move
run
land
stop
go
(a) Top-40 frequent verbs in Oops! part of ACQUIRED.
A.2 Word Distributions
Figure 4a and Figure plot the most frequent verbs (mainly for events) and nouns (mainly for entities) distributions of the Oops! proportion of our dataset, while Figure 5a and Figure 5b plot the ones of the Ego4D proportions.
chilkruck side weight boat
way
bike.
tree
balance
table
foot
door
hand.
glass
Figure 6a and Figure 6b are distributions of the CLEVRER dataset. Figure 7a and Figure 7b distributions of the TrafficQA dataset. It can be seen from these charts, alongside Table and Table 8 that the event types in both datasets are quite uni- modally towards their original intended domains (which is reasonable), with all four ratios much lower than those of our dataset.
second
Incident
jump
wall
kid
96
cat
person
friend
woman
day
boy
fall
grr
face
water
ground
dog
peoplieed
ball
guy time
Dataset verb-token ratio verb-token ratio
CLEVRER
0.0001
5.14c-6
(b) Top-40 frequent nouns in Oops! part of ACQUIRED.
TrafficQA 0.0053
0,0004
Figure Top-40 frequent word-types Oops!.
Table 7: Verb-token ratio (total # verb-types / total tokens) of CLEVRER and TrafficQA
Should we discard this question group from our dataset (repetitive not fixable all)?
Dataset noun-token ratio
noun-token ratio
CLEVRER 0.0002
2,57e-5
Does this question group need any editing to reduce ambiguity or to further fool the model?
TrafficQA
0.0036
0.0006
Table 8: Noun-token ratio (total noun-types/ total tokens) of CLEVRER and TrafficQA
Check the T/F of the two sentences.
Select the domain that you think this question group can be categorized into.
B Details of Human Annotations
B.1 Data Validation
select one the type that you think this ques- tion group can be categorized into.
We internal validation interface with question-answering setting accept reject sample. This tool also allows to fix wrong do- main categorization and T/F labels annotated by the workers. Specifically, the validation questions include:
To answer this question, do you need to refer the video?
Does this question group conform to our ques- tion format?

---

fall Gartlliu
pull lift
start
go
damage,
close
see.
pour
grab.
touch
nead
fix
work
finish
break
remove
happen
40%
60%
hold
wash
make
drop
pick
happen
wail
put
hame could opeflean
(a) Top-40 frequent verbs in Ego4d part of ACQUIRED.
(a) Top-40 frequent verbs in CLEVRER.
baint plate doo brush metal box
event
Tollowing
paper
left.
wire
tire.
ground
rubber
cyan
dough.
sink
knife.
cloth-
cabinet
Terence
metal
collide
WPP
screw
hand
container
glove
table
minute
type
material
cube
cylinder
food
water floor
piece
time
secd hour forc
sphere
object
(b) Top-40 frequent nouns in Ego4d part of ACQUIRED.
(b) Top-40 frequent nouns in CLEVRER.
Figure 5: Top-40 frequent word-types in Ego4d.
Figure 6: Top-40 frequent word-types in CLEVERER.
C Modelling Details
B.2 Annotation Process
C.1 More on GPT Baselines
The prompts engineered for both ChatGPT and GPT-4 baselines are shown below:
We build user interface collect QA pair annota- tions. In addition, we collect human performance as benchmarking source.
T/F setting without video description GPT Prompt The answer (Question) (Correct/wrong Answer), True or False?
User Interface. Our annotation interface (Figure 8b) is launched with Mturk tasks (Figure 8a). Upon accepting each Mturk HIT, workers will be directed to the annotation web app and do the rest of the task. Workers will be asked to create QA pairs for different domains and assign their T/F labels. the QA pair successfully fools the model, a green tick will be shown.
T/F setting with video description GPT Prompt The video is about (Video
the answer to { Question) is [Correct/wrong Answer), True or False?
MCQ setting without video description GPT Prompt Which of the following is the correct answer to [Question]? (a
answer (b) answer

---

and our entire code-base implemented in Py- Torch.
corse run
inmain
antempt
walt
D Releases Code
cuv.
prevent
halve.
cause
The comprehensive human-annotated datasets will be released upon acceptance, along with clearly stated documentation for usage. We plan to also release the codes for processing the datasets and leave pointers to all of our benchmarking baseline models. We hope that by sharing the essential re- sources, our work can incentivize more interest research on counterfactual reasoning in real-world visually observed events (via the video format), and their applications to robust multimodal Al and assistive technology in both AR and VR.
pass.
check
hit
maintain
stop
happen
960
triple
violate
pay
avoid
cross
remove
rain
drive
skid
be
change
scour
speed turn move
(a) Top-40 frequent verbs in TrafficQA.
Allaniio
bus
nightiffiing
pediesrian
stfeet
sport
load
pickup.
distance
bicycle
blindspat
trisycle
limit
veller
night
lotorcycle
accident
speed
condition
road
weather
cause space
ruck
car
lane
daeiaario
video
type time
(b) Top-40 frequent nouns in TrafficQA.
Figure Top-40 frequent word-types TrafficAA.
MCQ setting with video description GPT Prompt The video about {Video description). Which the following the correct answer (Question }? (a) answer (b) answer
C.2 Training & Implementation Details
Hyperparameters. The training time for the text- only models about 4-5 hours, and it takes about 3-5 hours for multimodal models to converge. We list all the hyperparameters used in Table 9 for the benchmarking models.
Implementation Details. The implementations of the transformer-based models are extended from the HuggingFace' code base (Wolf et al., 2020),
hhtps://pyoochh.org/

---

Models
Batch Size
Learning Rate
&Truining Epochs
iParams
DcBERTa-V3
lc-5
50
86M
UnifiedQA-bess
lc-5
50
220M
UnifedQQA-large
lc-5
50
770M
UnifiedQA-3B
le-5
50
3B
VIOLET
lc-5
20
163M
le-5
20
479M
VL-Adapter
3e.4
20
904M (32M learnable)
Table 9: Hyperparameters in this work: All the models trained with Adam (Kingma and Ba. 2015). We include the numbers model parameters in the column of #Params.

---

QOPS-OA Annotations
Make Sure You iratructions Sclow Doing the
spealiiully wo yours orote anciuse cortoct uwer intnciabee incoret disiuctio arower causdiue
The paitoons llking with thik enowend ei the culagor plysics hos wet donn vit one trom countrtectall bose
Weddre sain astact sseens cacis nokseed ron tods aostuollee and recentive purtern hove eocume mutiee riees
Piymant
Video
Intended Goal vert: A person takes bowl
verl: N/A
Intended Goal ver2: A person walks towards the cooker
Failure ver2: N/A
The link here will work only you accept this HIT. Remember That Copy Back Confirmation Code
(a) Human Annotation Instruction
Counterfactual Physical
Plaas proiiie osomerfentaal phesical cacation for grvn viso prempt
Prsacorotle cu wheche / Truoifalls
Counterfactual Time
Flass
Waxt wous Degpen the spoond man some the before jampop
the secont men carse the fiet man before he ampped the firs mas would ss have umppdd peiille
sapceta
befowe
would
poods
(b) Sample Annotation Interface
Figure 8: MTurk Annotation User Interface: (a) We ask workers to follow the indicated instruction. All the bluc-colored text bars the top of the page expandable. Workers can click to expand them for detailed instructions of the annotation task. (b) We design an user-fricn faiefrlendyy and interactive annotation tool where annotators and simply input their annotations and get instant feedback from model.


Reviews:
[{"review_id": "fDrb64K9Op", "reviewer": "Reviewer_sY4g", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful remarks! We are encouraged that you find our dataset interesting and useful for relevant research directions. We are particularly delighted to know that you find the resource would benefit building future multimodal models on counterfactual reasoning, and that our initial benchmarking results serve as a good guideline for the future research.\n\nPlease find your suggestions and concerns addressed below:\n\n---\n\n**[W1] Internal validation causing altered data distribution and artifacts/biases**\n\nOur internal validation is meant for assuring the quality of the curated data, and mainly conducted to **filter out significant violations to our desired data criteria while introducing minimal potentially biased expert knowledge into the constructed video-question-answer data points**.\nThe primary criteria are: **(1)** whether the QA is truly counterfactual, **(2)** whether the video source is indeed required to solve the QA, and **(3)** are the labeled answers correct and do the classified dimensions make sense (see Section B.1 for more details)?\nOur criteria are *well aligned with some of our preliminary modeling studies*, where the data points deemed not requiring the videos indeed can be more easily exploited by language-only models.\n\nMoreover, we prohibit too subjective judgements by listing the criteria prior to the validation phase, and all the validators are well-trained to fully understand the main criteria/motivations of our dataset collection process, before they can perform the validation tasks. We have three validators which also exhibit high agreements on their judgements towards the quality of the data, please refer to L315-318 and Section B.1 in the appendix.\n\nFurthermore, to ensure the diversity and mitigate the biases, we deployed a model-in-the-loop approach to incentivize more challenging and diverse QA creations, with an event-balancing sampling algorithm (see L267-278 in Section 3.1) to provide the video data points, and hence the data distribution should reflect these implementations more than our subsampled validation process.\n\nBelow we report the data drop-rates (*majority voted to drop* by all three validators) for the first 5 batches (note that each video gives 3 pairs of question- correct/distractor answers):\n| **Batches** | **Batch-1** | **Batch-2** | **Batch-3** | **Batch-4** | **Batch-5** |\n| :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n| Annotation Drop-Rate (%) | 28 | 17.3 | 4.3 | 3.5 | 2.6 |\n| Number of Videos | 50 | 100 | 200 | 200 | 200 |\n\nWe hope these rigorous safety checks can ensure a good data quality that also closely follows our guideline, and the validation should by no means introduce unnecessary biases as we indeed saw a decrease in the dropping rates in our later collection batches.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is an interesting dataset which can be used as a testbed evaluate models ability to do counterfactual reasoning.\n2. The task is challenging with significant performance gap between recent models and human.\n3. The dataset would motivate research in the direction of building multimodal models with better counterfactual reasoning capability.\n4. Benchmarking is quite extensive and covers very recent models.", "reasons_to_reject": "1. The internal manual validation/correction phase may alter the data distribution but is necessary to filter out noisy responses. \nThe author's quality control process is thorough, and the high inter-validator agreement score shows that the validators in general have high agreement on the validation tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new real-life dataset, ACQUIRED, for multimodal counterfactual reasoning. The proposed dataset enables evaluating multi-modal models' ability to do multimodal counterfactual reasoning. The dataset is a collection of video segments annotated with a counterfactual question, and +ve and -ve answer. The authors propose a video question answering question task on this dataset, and benchmark recent models on the task. There is substantial gap between the human and model performance suggesting that the proposed task on the dataset is a challenging one.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dOhNpcBhom", "reviewer": "Reviewer_5Kzu", "summary": "", "questions": "Refer to Reasons To Reject", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and the constructive suggestions! We appreciate that the reviewer agrees that our dataset is useful for learning multimodal models on the counterfactual reasoning ability, and our extensive benchmarking studies helpful for understanding where the proposed dataset stands.\n\nPlease find your suggestions and concerns addressed below:\n\n---\n\n**[W1,2] Comparisons with the existing works: NExT-QA[2] and Causal-VidQA [1]**\n\nThanks for pointing out the related works!\nWe will cite these works and compare ours with them in detail in the revised version.\nBelow we discuss how these works are different from ours, and our merits compared to the prior literature.\n\n**NExT-QA**, while being a video reasoning dataset, does not focus on counterfactual reasoning, which is our main motivation in this work. The dataset primarily concerns the comprehensive temporal reasoning questions including the cause, the future, and the past of an event of interest shown in a given video. The reasoning ability lies in causality, future prediction, and factual understanding from the video contents.\n\n**Causal-VidQA**, while also having counterfactual video questions included in the dataset similar to previous datasets such as TrafficQA, still differs from our work in multiple aspects. We carefully review the paper and hereby discuss their relations to our work:\n\n- The QAs in ACQUIRED are collected in the three reasoning dimensions (*i.e.*, temporal, physical, social) and two video viewpoints (*i.e.*, both first-person and third-person), beneficial for a more systematic analysis on the models’ abilities in different scenarios and providing the possibility to investigate models in cross-view and cross-domain settings. The aforementioned considerations are uniquely taken into account when collecting our dataset, while Causal-VidQA is based on the Kinetics(-700 2020) dataset where the videos are mostly in the third-person view and its questions are mostly focused on the physical reasoning dimension.\n\n- In addition to the defined dimensions (physical, social, temporal), we further encourage the diversity of question-answer pairs by a designed sampling algorithm (Section 3.1, L267-278). On the other hand, the questions in Causal-VidQA mainly concern the proposed objects by off-the-shelf vision models which are not perfect (*e.g.*, their models can underperform recent works like SAM [3] by a large margin), but also limit the main components the curated questions (and answers/reasons) can be based on. In contrast, we did not impose such a constraint and encourage the annotators to focus on the dynamics/motions of the presented videos and pay attention to the subtle visual details when constructing the counterfactuality. The comparison can be elaborated by the following metrics:\n  - **Type-Token Ratio** is the ratio obtained by dividing the types (unique vocabulary) by the total number of words (higher ratio indicates higher degree of lexical variations). We compute the ratios for both verbs and nouns, against both total verbs/nouns and total words.\n| **Datasets** | **Verb-Token Ratio (Total Tokens)** | **Verb-Token Ratio (Total Verbs)** | **Noun-Token Ratio (Total Tokens)** | **Noun-Token Ratio (Total Nouns)** |\n| :--------: | :--------: | :--------: | :--------: | :--------: | \n| Causal-VidQA | 0.000907 | 0.00727 | 0.001750 | 0.02192 |\n| Causal-VidQA (w/o \"happen\") | 0.0009198 | 0.008164 | --- | --- |\n| ACQUIRED | 0.004651 | 0.06874 | 0.0089 | 0.1133 |\n  - Note that we observed that many of the Causal-VidQA questions have the word (verb) “happen” in them, and hence we additionally compute the ratios excluding those *verbs*.\n  - It can be seen that the ACQUIRED dataset indeed contains more diverse action/interaction scenarios, *i.e.*, verb ratios; and the relationship to the distinct objects, *i.e.*, noun ratios, both exhibit much higher ratio scores.\n\n- We have an iterative model-in-the-loop data collection protocol that can mitigate the annotator biases introduced in the annotation process (L279-302). The advantage of this protocol can be demonstrated by the text-only model performance as reported below (we report the Q → A results on the Causal-VidQA validation set as the test set GTs are not provided).  We can see rather significant performance improvements of text-only models compared with the random baseline on Causal-VidQA while much less improvements on ours, demonstrating that the iterative model-in-the-loop data collection protocol not only brings benefits in curating more challenging samples, but also helps diversify the answers as the models will not be easily fooled if **repetitiveness is too severe** (among the dataset) or the questions can be **simply guessed with commonsense** (no video requirement).\n  - **GPT performance is *zero-shot***\n  - The descriptions (desc.) of Causal-VidQA are directly obtained from the original Kinetics-700 dataset, which are essentially short phrases of the video activity classes (labeled for activity recognition).\n| **Datasets** | **Model** | **QA-Format** | **Accuracy (%)** | **Gain (from Random) (%)** |\n| :--------: | :--------: | :--------: | :--------: | :--------: |\n| Causal-VidQA | Random | MCQ | 20.0 | 0 |\n| Causal-VidQA | Vanilla ChatGPT | MCQ | 42.0 | 110 |\n| Causal-VidQA | Vanilla GPT-4 | MCQ | 48.8 | 144 |\n| Causal-VidQA | Desc.-GPT-4 | MCQ | 52.6 | 163 |\n| Causal-VidQA | DeBERTa-v3 | MCQ | 66.2 | 231 |\n| Causal-VidQA | UnifiedQA-3B | MCQ | 65.9 | 230 |\n| ACQUIRED | Random | T/F | 50.0 | 0 |\n| ACQUIRED | Vanilla ChatGPT | T/F | 52.8 | 5.6 |\n| ACQUIRED | Vanilla GPT-4 | T/F | 53.8 | 7.6 |\n| ACQUIRED | Desc.-GPT-4 | T/F | 56.2 | 12.4 |\n| ACQUIRED | DeBERTa-v3 | T/F  | 70.12 |  40.2 |\n| ACQUIRED | UnifiedQA-3B  | T/F | 70.49 | 41.9 |\n\n- We collect more challenging distractors than Causal-VidQA. The distractors in our dataset are manually curated with certain twists towards the correct answers (examples in Table 2), forcing the models to truly understand the visual concepts involved in the counterfactual questions. Furthermore, we also report the pairwise accuracy to evaluate the **true robustness** of the models. On the other hand, the distractors in Causal-VidQA are retrieved from answers in similar questions.\n\n- In summary, while sharing high-level similarities, our work expands the views of counterfactual reasoning into more comprehensive (in terms of knowledge dimensions) and diverse (in terms of QA semantics and events) regimes. In addition, we spend additional efforts on ensuring the diversity and difficulty of our constructed dataset. In light of this, we do not think the existence of such prior work should exclude/diminish the importance of our work, towards the research community. That said, we thank the reviewer for pointing out the two related works and we will be sure to add them to corresponding sections with the discussion above.\n\n[3] Kirillov, Alexander, et al. \"Segment anything.\" *ICCV* 2023\n\n---\n\n**[W3] The QA format and the complexity of our ACQUIRED dataset**\n\nWe would like to point out that our evaluation protocol is beyond the traditional T/F evaluation paradigm and is rather challenging for neural models that are vulnerable to exploit spurious input-output correlations.\n\nWe are inspired by prior works [4,5] to consider the **surprisingly difficult nature of the T/F (yes/no) QA formats** that could potentially exhibit less unintended biases/artifacts than curating data in the multiple choice (MCQ) settings. \nAs shown in Table 2, we focus on curating high-quality correct answers and a strong associated distractor answer that should be a **twist on the correct** one focusing on similar concepts (of the asked questions).\n\nIn addition to having the manually curated distractors, we provide two formulations and two main metrics to evaluate if the models indeed perform correct reasoning instead of utilizing spurious correlations.\nAs mentioned in L230-237 in Section 3.1, the T/F format requires the model to provide a yes/no response to **each of the** correct answer and the distractor, where both **standard accuracy metric** and the **pairwise consistency metric** (see L448-451 in Section 5.2) are used.\n\nThe pairwise consistency metric thus requires the model to answer correctly **in both correct and distractor directions** to be regarded as a success. Therefore, simply using surface-level heuristics will fail as the models will output the same prediction given two minimally different statements.\n\nFurthermore, as mentioned above, the distractors in our dataset are also *curated* that undergo the same model-in-the-loop gamified setup, where the aforementioned **pairwise accuracy** can more robustly evaluate whether the models **fully understand** the contexts and are able to reason over them.\nJudging by the much deteriorated numbers (from Table 5) of the pairwise metric as compared to the standard ones, there indeed is much room for improvements to make the models more robust on their beliefs (they should be close to each other if the models **truly** understand the counterfactual contexts).\n\nWhile we acknowledge the importance and the interestingness of asking the models to provide the rationale behind their decisions, it is still an open question whether this setting can reveal if models can indeed perform reasoning because this is yet another MCQ setting, while our **pairwise accuracy** goes beyond this paradigm and is designed to evaluate if models can robustly perform correct reasonings. \n\n[4] Clark, Christopher, et al. \"BoolQ: Exploring the surprising difficulty of natural yes/no questions.\" *NAACL-HLT* 2019\n\n[5] Singh, Shikhar, et al. \"COM2SENSE: A commonsense reasoning benchmark with complementary sentences.\" *Findings of ACL* 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work propose a VideoQA dataset for Counterfactual Questions.\n2. This work propose extensive experimental results to show how existing model perform on this dataset.", "reasons_to_reject": "1. The novelty of this work is quite limited. Since similar dataset, like Causal-VidQA[1] and NeXT-QA[2] have already existed.\n2. Missing discussion with similar datasets [1][2]\n3. The answer only includes a wrong and and a right answer, however, choosing the right answer don't indicate that the model can really reasoning in counterfactual situation. A more suitable solution is to make the answer process as answer-reason mode. Please refer to [1] and VCR for more details.\n\n[1] From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering, CVPR2022\n[2] Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR 2022\n\n----\nAfter rebuttal:\nThe author partial solves my concerns, so I will also slightly increase my rating.", "questions_to_authors": "Refer to Reasons To Reject", "paper_topic_and_main_contributions": "This paper contribute a VideoQA dataset for Counterfactual Questions.\nTo construct this dataset, the author propose to provide a first- and a third-person viewpoint of the video, and provide the answer with a wrong one and a correct one.\nExperimental results shows that there are still a long way to go in this field.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5QeSHkAmmE", "reviewer": "Reviewer_LAU8", "summary": "", "questions": "In section 3.1 Dataset Collection Workflow:\n- What pretrained (text-only) QA model do you select and what is your reasoning behind this choice?\n- how do you sample for validation? I am interested in details except for the questions,  such as the sample rate and quantitive results like error rate and etc.\nIn section 3.1 Commonsense Dimensions:\n- Why did you categorize the questions as physical, social, and temporal? And the Ego4D dataset contains no social-type questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback and detailed review!  We are encouraged that you find our dataset novel and exciting for the research community on learning multimodal models that can perform counterfactual reasoning over dynamic visual scenes (videos).\n\nPlease find your suggestions and concerns addressed below:\n\n---\n\n**[W1] Unclear annotation details**\n\nWe address all the questions regarding the annotation details below in **[Q]**(s).\n\n---\n\n**[W2] Comparisons with the existing works: NExT-QA and Causal-VidQA**\n\nWe appreciate the reviewer for pointing out additional related works and will discuss in detail below how these works relate to ours, the distinction between them, and our merits compared to the prior literature.\n\n**NExT-QA**, while being a video reasoning dataset, does not focus on counterfactual reasoning, which is our main motivation in this work. The dataset primarily concerns the comprehensive temporal reasoning questions including the cause, the future, and the past of an event of interest shown in a given video. The reasoning ability lies in causality, future prediction, and factual understanding from the video contents.\n\n**Causal-VidQA**, while also having counterfactual video questions included in the dataset similar to previous datasets such as TrafficQA, still differs from our work in multiple aspects. We carefully review the paper and hereby discuss their relations to our work:\n\n- The QAs in ACQUIRED are collected in the three reasoning dimensions (*i.e.*, temporal, physical, social) and two video viewpoints (*i.e.*, both first-person and third-person), beneficial for a more systematic analysis on the models’ abilities in different scenarios and providing the possibility to investigate models in cross-view and cross-domain settings. The aforementioned considerations are uniquely taken into account when collecting our dataset, while Causal-VidQA is based on the Kinetics(-700 2020) dataset where the videos are mostly in the third-person view and its questions are mostly focused on the physical reasoning dimension.\n\n- In addition to the defined dimensions (physical, social, temporal), we further encourage the diversity of question-answer pairs by a designed sampling algorithm (Section 3.1, L267-278). On the other hand, the questions in Causal-VidQA mainly concern the proposed objects by off-the-shelf vision models which are not perfect (*e.g.*, their models can underperform recent works like SAM [4] by a large margin), but also limit the main components the curated questions (and answers/reasons) can be based on. In contrast, we did not impose such a constraint and encourage the annotators to focus on the dynamics/motions of the presented videos and pay attention to the subtle visual details when constructing the counterfactuality. The comparison can be elaborated by the following metrics:\n  - **Type-Token Ratio** is the ratio obtained by dividing the types (unique vocabulary) by the total number of words (higher ratio indicates higher degree of lexical variations). We compute the ratios for both verbs and nouns, against both total verbs/nouns and total words.\n| **Datasets** | **Verb-Token Ratio (Total Tokens)** | **Verb-Token Ratio (Total Verbs)** | **Noun-Token Ratio (Total Tokens)** | **Noun-Token Ratio (Total Nouns)** |\n| :--------: | :--------: | :--------: | :--------: | :--------: | \n| Causal-VidQA | 0.000907 | 0.00727 | 0.001750 | 0.02192 |\n| Causal-VidQA (w/o \"happen\") | 0.0009198 | 0.008164 | --- | --- |\n| ACQUIRED | 0.004651 | 0.06874 | 0.0089 | 0.1133 |\n  - Note that we observed that many of the Causal-VidQA questions have the word (verb) “happen” in them, and hence we additionally compute the ratios excluding those *verbs*.\n  - It can be seen that the ACQUIRED dataset indeed contains more diverse action/interaction scenarios, *i.e.*, verb ratios; and the relationship to the distinct objects, *i.e.*, noun ratios, both exhibit much higher ratio scores.\n\n- We have an iterative model-in-the-loop data collection protocol that can mitigate the annotator biases introduced in the annotation process (L279-302). The advantage of this protocol can be demonstrated by the text-only model performance as reported below (we report the Q → A results on the Causal-VidQA validation set as the test set GTs are not provided).  We can see rather significant performance improvements of text-only models compared with the random baseline on Causal-VidQA while much less improvements on ours, demonstrating that the iterative model-in-the-loop data collection protocol not only brings benefits in curating more challenging samples, but also helps diversify the answers as the models will not be easily fooled if **repetitiveness is too severe** (among the dataset) or the questions can be **simply guessed with commonsense** (no video requirement).\n  - **GPT performance is *zero-shot***\n  - The descriptions (desc.) of Causal-VidQA are directly obtained from the original Kinetics-700 dataset, which are essentially short phrases of the video activity classes (labeled for activity recognition).\n| **Datasets** | **Model** | **QA-Format** | **Accuracy (%)** | **Gain (from Random) (%)** |\n| :--------: | :--------: | :--------: | :--------: | :--------: |\n| Causal-VidQA | Random | MCQ | 20.0 | 0 |\n| Causal-VidQA | Vanilla ChatGPT | MCQ | 42.0 | 110 |\n| Causal-VidQA | Vanilla GPT-4 | MCQ | 48.8 | 144 |\n| Causal-VidQA | Desc.-GPT-4 | MCQ | 52.6 | 163 |\n| Causal-VidQA | DeBERTa-v3 | MCQ | 66.2 | 231 |\n| Causal-VidQA | UnifiedQA-3B | MCQ | 60.7 | 204 |\n| ACQUIRED | Random | T/F | 50.0 | 0 |\n| ACQUIRED | Vanilla ChatGPT | T/F | 52.8 | 5.6 |\n| ACQUIRED | Vanilla GPT-4 | T/F | 53.8 | 7.6 |\n| ACQUIRED | Desc.-GPT-4 | T/F | 56.2 | 12.4 |\n| ACQUIRED | DeBERTa-v3 | T/F  | 70.12 |  40.2 |\n| ACQUIRED | UnifiedQA-3B  | T/F | 70.49 | 41.9 |\n\n- We collect more challenging distractors than Causal-VidQA. The distractors in our dataset are manually curated with certain twists towards the correct answers (examples in Table 2), forcing the models to truly understand the visual concepts involved in the counterfactual questions. Furthermore, we also report the pairwise accuracy to evaluate the **true robustness** of the models. On the other hand, the distractors in Causal-VidQA are retrieved from answers in similar questions.\n\n- In summary, while sharing high-level similarities, our work expands the views of counterfactual reasoning into more comprehensive (in terms of knowledge dimensions) and diverse (in terms of QA semantics and events) regimes. In addition, we spend additional efforts on ensuring the diversity and difficulty of our constructed dataset. In light of this, we do not think the existence of such prior work should exclude/diminish the importance of our work, towards the research community. That said, we thank the reviewer for pointing out the two related works and we will be sure to add them to corresponding sections with the discussion above.\n\n[4] Kirillov, Alexander, et al. \"Segment anything.\" *ICCV* 2023\n\n---\n\n**[W3] Presentation improvements**\n\nThank you, we will improve on our paper presentation in the final version.\nWe would also appreciate that the reviewer could more specifically point out or suggest which parts need more revisions and we will be sure to revise them accordingly.\n\n---\n\n**[Q1] Pretrained text-only QA model**\n\nAs mentioned in L358-365 in Section 4, beside DeBERTa-v3, the strongest text-only model we adopt is the UnifiedQA model [5ab] (specifically its T5-3B variant), mainly for both its computational efficiency and proven strong performance in many QA tasks (please refer to their original paper, where the model is extensively tested on many existing QA benchmarks).\n\n[5a] Khashabi, Daniel, et al. \"Unifiedqa: Crossing format boundaries with a single qa system.\" *Findings of EMNLP* 2020\n\n[5b] Khashabi, Daniel, Yeganeh Kordi, and Hannaneh Hajishirzi. \"Unifiedqa-v2: Stronger generalization via broader cross-format training.\" *arXiv preprint* 2022\n\n---\n\n**[Q2] Validation phase details**\n\nFor the first few annotation batches, we sample all of the curated data for manual validation for both the data quality control as well as consolidating feedback to further train the annotators for our task.\nFrom the 4th batch where we deem the annotators well-trained, we sample 20% of the data for efficiency and the error rate is no higher than 3-4%, judging by our desired data criteria (*i.e.*, whether the created question is counterfactual, whether the common sense dimension is reasonable, and whether the question indeed requires the video, etc.).\n\nBelow we report the data drop-rates (*majority voted to drop* by all three validators) for the first 5 batches (note that each video gives 3 pairs of question- correct/distractor answers):\n| **Batches** | **Batch-1** | **Batch-2** | **Batch-3** | **Batch-4** | **Batch-5** |\n| :--------: | :--------: | :--------: | :--------: | :--------: | :--------: |\n| Annotation Drop-Rate (%) | 28 | 17.3 | 4.3 | 3.5 | 2.6 |\n| Number of Videos | 50 | 100 | 200 | 200 | 200 |\n\n---\n\n**[Q3] Consideration on the common sense dimensions**\n\nWe are inspired by the commonsense benchmark prior work [6], where they are inspired by the **Theory of Core Knowledge** [7], *i.e.*, the capability of reasoning about physical objects, places, motions, and the social world. The three main dimensions are the building blocks towards a comprehensive commonsense reasoning, and helps systematically analyze in which aspects the models need to be improved upon more.\n\nOur primary motivation is to provide a dataset where a model can learn to reason over counterfactuality from different viewing angles, as the learned agent can be robust in assisting humans in both 1st and 3rd person viewpoints. Our initial preliminary studies found that Ego4D, as an egocentric human activity recorded video set, contains much more generic task completions than social interactions (many of the videos only have activities performed by a single person), and hence our annotators found it more difficult than Oops! to annotate the social dimension. We hence made the decision to mainly collect Physical and Time which are more prevalent in the Ego4D set (in this work).\n\nHowever, we do not expect such design choices would hinder the models from learning from the three main commonsense dimensions, as there is still a decent amount of social portion from the Oops! Videos.\n\n[6] Singh, Shikhar, et al. \"COM2SENSE: A commonsense reasoning benchmark with complementary sentences.\" *Findings of ACL* 2021\n\n[7] Spelke, Elizabeth S., and Katherine D. Kinzler. \"Core knowledge.\" *Developmental science* 10.1 (2007): 89-96.\n\n---\n\n**[Misc.] Missing references and typos**\n\nWe will cite the suggested references appropriately in our final version. Specifically for paper [1] and [3], we will include the discussion mentioned above. Thanks again for the presentation suggestion, we will fix the typos and enrich the related works section aiming at a better clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work present a novel counterfactual reasoning-focused video question answering dataset, named ACQUIRED", "reasons_to_reject": "- Unclear annotation details, see questions for details.\n- You seem to have overlooked some existing VideoQA datasets that involve causality, such as NExTQA [1] and Causal-VidQA [3]. I would appreciate a more in-depth analysis of these.\n- A portion of the writing is difficult to understand and requires enhancement", "questions_to_authors": "In section 3.1 Dataset Collection Workflow:\n- What pretrained (text-only) QA model do you select and what is your reasoning behind this choice?\n- how do you sample for validation? I am interested in details except for the questions,  such as the sample rate and quantitive results like error rate and etc.\nIn section 3.1 Commonsense Dimensions:\n- Why did you categorize the questions as physical, social, and temporal? And the Ego4D dataset contains no social-type questions.", "paper_topic_and_main_contributions": "This paper introduces the ACQUIRED dataset, which focuses on answering counterfactual questions in real-life videos. The dataset incorporates three dimensions of reasoning: physical, social, and temporal. The paper benchmarks several video-language models and SOTA text-only model (GPT4) on the dataset and demonstrates that the models struggle to effectively utilize video contexts and perform counterfactual reasoning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- [1] Xiao, Junbin et al. “NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions.” *2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)* (2021): 9772-9781.\n- [2] Grunde-McLaughlin, Madeleine et al. “AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning.” *2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)* (2021): 11282-11292.\n- [3] Li, Jiangtong et al. “From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering.” *2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)* (2022): 21241-21250.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Sec. 3.1, line 294: \"fails to predict correctly the creations\" should be \"fails to correctly predict the creations\"\n- The unnecessarily large margin of related work contrasts with the vague construction procedure of the dataset, as presented in both the papers and the appendix"}}]

Author rebuttal:

Meta-review: Three reviewers provided feedback for this work and were in consensus. They found the newly proposed dataset to be interesting and very useful to the community. The task is challenging and need for such datasets is quite high. The reviewers expect that this would be an impactful dataset and would motivate work in the direction of multimodal models with counterfactual reasoning capabilities. They also found the benchmarking to be thorough and well done. The one major concern was positioning the dataset in light of recent works at CVPR 2022. The authors have responded well and have agreed to add these extensive details into the paper. In light of these reviews and discussion, I recommend acceptance.
Decision: Accept-Main