SlowGuess/NAACL_2022_OCR huggingface dataset:
CoSIM: Commonsense Reasoning for Counterfactual Scene Imagination
Hyounghun Kim* Abhay Zala* Mohit Bansal
{hyounghk, aszala, mbansal} @cs.unc.edu
As humans, we can modify our assumptions about a scene by imagining alternative objects or concepts in our minds. For example, we can easily anticipate the implications of the sun being overcast by rain clouds (e.g., the street will get wet) and accordingly prepare for that. In this paper, we introduce a new dataset called Commonsense Reasoning for Counterfactual Scene Imagination (COSIM) which is designed to evaluate the ability of AI systems to reason about scene change imagination. To be specific, in this multimodal task/dataset, models are given an image and an initial question-response pair about the image. Next, a counterfactual imagined scene change (in textual form) is applied, and the model has to predict the new response to the initial question based on this scene change. We collect 3.5K high-quality and challenging data instances, with each instance consisting of an image, a commonsense question with a response, a description of a counterfactual change, a new response to the question, and three distractor responses. Our dataset contains various complex scene change types (such as object addition/removal/state change, event description, environment change, etc.) that require models to imagine many different scenarios and reason about the changed scenes. We present a baseline model based on a vision-language Transformer (i.e., LXMERT) and ablation studies. Through human evaluation, we demonstrate a large human-model performance gap, suggesting room for promising future work on this challenging, counterfactual multimodal task. $^{1}$
Anticipating what would happen when there is a condition change in a situation is an important ability as it allows preparation for the implications of
the change. For example, when swimming in the sea on a clear day, you might feel safe. However, if someone told you a storm warning has been issued and dark clouds are coming in soon, you would know that it is no longer safe to swim and return to land. It will be also very useful to have AI systems that could reason about the implications of such scenario changes and provide appropriate guidance/warnings; however, current AI systems will have a hard time performing such counterfactual commonsense reasoning.
Many efforts have been made to teach machines how to reason about images (Antol et al., 2015; Zhu et al., 2016; Johnson et al., 2017; Hudson and Manning, 2019) and videos (Tapaswi et al., 2016; Jang et al., 2017; Zhu et al., 2017; Lei et al., 2018). This area has been built upon further by efforts to teach machines to use commonsense knowledge when analyzing visual scenes (Pirsiavash et al., 2014; Wagner et al., 2018; Zellers et al., 2019; Park et al., 2020). Through these efforts, many AI systems have reached near human-level performance on scene understanding tasks. On the other hand, more complex reasoning on scene 'changes' has been less explored. Sampat et al. (2021) applies simple condition manipulations (e.g., "Paint the small green ball with cyan color") on synthetic images in a visual question answering setup. However, this task is based on simple block objects that might not require complex implicit reasoning. Thus, complicated counterfactual commonsense reasoning on scene changes on real-world images and situations remains widely unexplored.
Therefore, in this paper, we introduce a new dataset called Commonsense Reasoning for Counterfactual Scene Imagination (COSIM) to evaluate the commonsense reasoning ability of agents about counterfactual visual scenes imagined via textual descriptions. To be specific, one data instance in our COSIM dataset consists of an image, an initial question-response pair, an imagined
1Our code and dataset are publicly available at: https://github.com/hyounghk/CoSIM.
Human Language Technologies, pages 911 - 923
Question / Initial Response
Figure 1: Example from our COsIM dataset. An image is associated with an initial commonsense question-response pair, a described counterfactual change to the image, and a new response to the question with three distractors.
Is the railway line safe?
Yes because there are safety lights and crosswalk signs
add a train to the tracks.
Answer Choices
New Response: no, although there are safety lights and crossing gates, they don't appear to be working and there is a train coming.
Distractor #1: yes, there are safety lights and crossing gates, they appear to be working and there is a train coming and it will stop.
Distractor #2: yes, although there are safety lights and crossing gates, they don't appear to be working and there is no train coming.
Distractor #3: no, although there are safety lights and crossing gates, there is a power outage and there is a train coming.
visual scene change, and a new response with three distractors. The question is about commonsense which can be inferred from the image and the initial response includes a reasoning/justification for its answer. The imagined visual scene change is a textual description of what to modify in the scene to alter the conditions. The new response follows the same format as the initial response, but should be influenced by the imagined change (see Figure 1).
A model for this task needs to take this context information as input and try to predict the correct new response among other distractors. The distractors look similar to the correct new response but have subtle differences and are semantically different from the correct new response, thus a good model on this challenging new multimodal task cannot take shortcuts and needs to fully understand what each choice means based on the context. For example, as shown in Figure 1, given an image, the initial question-response pair ("Is the railway line safe?") - "Yes because there are safety lights and crosswalk signs"), and the scene change ("add a train to the tracks."), models should choose the correct new response ("no. although there are safety lights and crossing gates, they don't appear to be working and there is a train coming.") among other distractors ("no. although there are safety lights and crossing gates, there is a power outage and there is a train coming.", etc.). To solve this problem, models need to be able to understand the implications of an incoming train and how safety lights and gates operate at a railroad crossing.
We collect 3.5K high-quality and challenging data instances for this new multimodal reasoning task via a crowd-sourcing annotation platform. To collect each data instance and to help reduce individual crowd-worker load, we break the task up into three separate phases: the question collection
phase, the scene change collection phase, and the distractor collection phase. During the distractor collection phase, to help avoid unexpected biases such as text-only, we implement a modified version of Human-And-Model-in-the-Loop Enabled Training (HAMLET) adversarial data collection (Nie et al., 2020) for the validation and test splits. We deploy the model trained on only the textual data and allow annotators to test their distractors against the model as they write (see Figure 2).
Our COsIM dataset features several diverse types of imagined scene changes (object addition/removal, object state changes, etc.; see Sec. 5.2 for the full change type list and examples) which requires to deeply understand the contexts, making the task very challenging. For example, to understand the scene change of "Add another person to the dock ...", the model should figure out what a dock is, where it is located in the image and be able to add one more person onto it via imagination.
As a baseline model for this new multimodal reasoning task, we employ a vision-language Transformer (based on LXMERT (Tan and Bansal, 2019)) which computes vision and language feature matching scores via multi-head self-attention layers followed by cross-modal attention layers, and we report ablation studies on input modality and scene change types. We also show a large human-model performance gap allowing more effective future work from the community on this new challenging multimodal task on commonsense reasoning for imagined counterfactual scene changes.
Visual Question Answering. There have been many efforts to teach machines how to reason about images (Antol et al., 2015; Zhu et al., 2016; Johnson et al., 2017; Hudson and Manning, 2019) and
videos (Tapaswi et al., 2016; Jang et al., 2017; Zhu et al., 2017; Lei et al., 2018), and in some of these tasks, machine performance is approaching human levels. Although these tasks require a complicated reasoning process, they provide very explicit context to solve the problems and might not be enough to evaluate the ability to reason about implicit aspects (i.e., commonsense).
Visual Commonsense Reasoning. Another actively explored line of study has been on visual commonsense reasoning (Pirsiavash et al., 2014; Wagner et al., 2018; Zellers et al., 2019; Park et al., 2020). In addition to using the provided clues in the context, these tasks require commonsense knowledge to reason about given problems, making these tasks more challenging since machines should be equipped with prior or external information. However, these tasks handle static scene understanding for which contexts and conditions are not changed during the reasoning process. On the other hand, our proposed CoSIM introduces an additional dimension of difficulty by integrating imagined scene changes in the context. Moreover, the changes in our CoSIM dataset are imagined (textually) and counterfactual, so imagination-based commonsense is required for the reasoning.
Textual Scene Change. Recent effort has been made on visual understanding by requiring mental simulation of changes to the scene (Sampat et al., 2021). These tasks require simulating change without any visible result, hence increasing the difficulty of VQA tasks. They, however, have been completed in the simpler context of basic shapes and objects and simple questions (E.g. "How many blue objects will be present in this scene?"). Our CoSIM dataset is based on complex real-world images/situations requiring commonsense reasoning about imagined counterfactual scene changes, allowing for evaluation of the ability to anticipate the implications of complex situation changes, thus, future events.
Given a real-world image, models should predict a new response conditioned on the initial question-response pair and the imagined counterfactual scene change.
Initial Question and Response. The initial question-response pair is created only from a given image. The question and response themselves require quite an amount of commonsense reasoning
Figure 2: HAMLET cycle for distractor collection on validation and test splits.
to understand. For example, to understand the response to the question in Figure 1 ("Is the railway line safe?"), models should know that the 'safety lights' and 'crosswalk signs' are devised for keeping people safe around the railway ("Yes because there are safety lights and crosswalk signs").
Imagined Counterfactual Scene Change. The imagined counterfactual scene change is a textual description that modifies the scene in the image. The change affects the reasoning process of the initial question and response, and provides a new context for the new response ("add a train to the tracks").
Response on the Scene Change. Models should respond to the initial question with a proper reason based on the imagined counterfactual scene change. The task is a multi-choice setup to pick the correct response among other distractors ("no, although there are safety lights and crossing gates, they don't appear to be working and there is a train coming"). To choose the correct response, models should understand what the implications and safety concerns of an incoming train are and that the safety lights should be turning on and the crossing gates should be closing when a train is in proximity.
Our COSIM dataset is composed of  $3.5\mathrm{K}^3$  images paired with a commonsense question-response pair, a description of an imagined counterfactual change to the image, a new response to the question based on the effect of the described change, and then three distractor responses to the question (all text is in English).
2Models should derive scene knowledge from the image or clues embedded in the textual context like the initial response.
<sup>3</sup>Low resource setup of this middle-scale size dataset encourages employing effective external commonsense knowledge.
We employ annotators from the crowd-sourcing platform Amazon Mechanical Turk<sup>4</sup>. Our data collection is broken into three separate phases (question, change, and distractor) in order to reduce the workload for each worker. In the question phase, workers are asked to select an image (from three random images) to use, write a commonsense question and then respond to it. In the change phase, they are asked to describe a counterfactual scene change for the image and then write a new response to the initial question. Lastly, in the distractor phase, they are asked to write three distractor responses for the question.
Commonsense Question Collection. To collect the initial question and response, we present three images to the workers and then ask them to choose the one that they want to use (images are taken from Visual Genome (Krishna et al., 2017)). Then using that image, they should come up with a commonsense question about the image. We define a commonsense question as a question that requires logical thought and understanding of what is happening in the image to be able to answer. Then workers are asked to write a response to their question (the initial response). A response consists of two parts, an "answer" that is a direct answer to the question (e.g. "Yes, ...") and then a "justification" that uses visual clues from the image to prove the answer is correct (e.g. "... because everyone is wearing shorts and short-sleeved shirts and a woman can be seen wearing sunglasses.). See Appendix for the collection interface.
Counterfactual Scene Change Collection. In this phase, workers are given the image chosen from the previous commonsense question collection phase and the corresponding initial commonsense question-response pair. Then workers are asked to describe a counterfactual scene change for the image and write a new response to the question based on that scene change (the new response). To help ensure that workers describe a reasonable counterfactual scene change, we provide two guide templates for them to follow when they write. Workers are asked to select the guide template that they believe makes the most sense for them to use for each data instance (see Appendix for collection interface and guide template details).
Distractor Collection. Workers are given the image, the initial commonsense question-response
pair, as well as the counterfactual scene change and new response. Then they are told to write three distractor responses that are similar to the new response but incorrect. To help ensure the distractors pose a challenge but are still distinct, we pre-fill the worker's textboxes with the new response. Then they are told to edit the text enough so the answers become false and distinct.
HAMLET Data Collection. To avoid having unexpected biases such as context+response bias in our textual data, when collecting distractors for the validation and test splits, we implement a HAMLET style collection (see Figure 2). We deploy the model trained only with textual data and allow workers to test their distractors directly against the model in real-time and check whether they are able to fool it. Workers are also permitted to edit the new response from the previous collection phase if it helps make distractor writing better (they must maintain the original meaning/intent of the new response if they choose to edit).
Data Verification. At each collection phase, we ask workers to verify the previous phase's work. If the previous set of work is not good, workers are given a place to flag and describe the reason for flagging. This reasoning is manually reviewed and if it is fair, then that data is removed and prevented from progressing to the next phase.
Worker Qualifications and Payment. For all 3 phases, workers are required to pass certain qualifications before they could begin. As all of the phases require reading and writing English, they were required to be from native English-speaking countries. Workers were also required to have at least 1,000 approvals from other tasks and a  $95\%$  or higher approval rating. Then for each phase, we require workers to pass a qualification test that tests their understanding of their task at each phase. See Appendix for worker totals and pay (+bonus) rates.
5 Data Analysis
We collect 3.5K task instances (3.5K images, initial questions-response pairs, scene changes, new responses, and 10.5K distractors).
5.1 Statistics
Length. Lengths of each part of the data instances are shown in Table 1. While the lengths of questions are relatively short, the lengths of responses and the changes are long. This means that question
<sup>4</sup>https://www.mturk.com
Object Addition
Object Relocation
Figure 3: Scene change examples from our CoSIM dataset. The relevant portions of the change are in italics. Complex changes contain three or more changes within them (this example contains Object Removal, Object Addition, Object State Change).
"Add snow and ice to the road. Add a bus. Place the person inside the bus."
... move the skateboarder much higher up in the air ... move the skateboard further away from the skateboarder so they can not land with the board."
Object Removal
Complex Change
"Remove all the kites and the unfurled sail. Add some people in the foreground with long hair. Have their hair blowing horizontally ..."
"remove the bowls and towels. Add plates of hot food to the counter. the bowl on the stove has steam rising from it."
Component	max.	min.	avg.	sd.
Question	22	3	7.6	3.03
Initial Response	59	4	18.62	8.06
Scene Change	127	3	16.08	13.01
New Response	109	5	23.38	12.40
Distractor	111	5	23.57	12.40
itself does not contain detailed clues and models should figure out which information is needed to answer the question. On the other hand, the long responses contain reasons to justify their answers and require models to deeply understand the reasoning process to solve the problem. Furthermore, models should also carefully read the long textual scene change to capture all the condition modifications and apply them to images.
Vocabulary. Among all data instances in our CoSIM dataset, there are 9,946 total unique words. Within the commonsense questions, initial responses, scene changes, new responses, and the distractors, there are 3,261 / 4,397 / 4,637 / 5,318 / 6,404 unique words, respectively. The unique word count reflects what is shown by the lengths. Questions are on average the shortest part of each data instance and they have the fewest unique words. The new responses and distractors have long lengths and high unique word counts. The high unique word count for the distractors shows their diversity. Figure 4 shows the most commonly occurring keywords in our dataset. Many of the words are related to people and directional positioning.
5.2 Scene Change Type.
Different imagined scene change types are present in our CoSIM dataset. Imagined scenes changes describe a change (with counterfactual thought) to
Table 1: In our COSIM dataset, each part has a different length according to its role and contained information.
Number of changes present	Frequency
1	34.70%
2	35.82%
3	21.39%
Greater than 3	8.08%
Table 2: Frequency of number of change types present per instance (from the validation split).
the image by applying various properties. Some of these scene change types include object addition/removal, object state changes, environment changes, etc. (see Figure 3 for some scene change types and their examples; see Appendix for a figure with a complete list of all the types with examples). These scene change types, while they are seemingly easy to visualize, require a complex understanding of the effect of the change on other elements in the scene. See Figure 5 for type frequencies.
Human/Object Addition. These two scene change types involve introducing new human(s)/object(s) into the image that was not there prior ("A bunch of old men are standing next to the birds ..." / "There are tears in his eyes ..."). The object addition scene change type is the most commonly appearing one.
Human/Object Removal. These two scene change types involve removing human(s)/object(s) that are visible in the image ("... remove the workers ..."/ "Remove the two people's coats").
Object Replacement. This scene change type involves removing object(s) from the image and replacing them with something else ("... replace the plates of fruit by plates of dog biscuits ...").
Object Relocation. This scene change type involves re-positioning object(s). Rather than changing it directly, this type changes its relation to other objects ("space the zebras out. move them a little
Figure 4: Most commonly occurring keywords in our CoSIM dataset. Many of them are related to people and directional positioning.
further away").
Object State Change. This scene change type involves altering the state of object(s) present in the image ("... change her luggage to all have a Burberry pattern ..."). The alteration of object(s) can occur in various forms such as changing color, size, shape, and orientation (e.g., opening a door).
Event Description. This scene change type involves the creation of an event or a description of motion or interaction between objects in the image. This type includes human actions and changes to human emotions ("A pack of lions are approaching the sheep").
Environment Change. This scene change type involves changes that cause large-scale changes to the entire environment either by drastically altering the current environment, creating a new environment, or causing changes in the weather ("there is very thick dust everywhere").
Complex Changes. We define a complex change as a change that contains three or more different scene change types. For example, "someone is throwing snow ball at her" this change introduces a new human, a new object, and defines an interaction between all these and involves someone already present in the image. These complex changes require much thought to understand their full effect and implications. Complex changes make up about  $30\%$  of our dataset. See Table 2 for change types per instance statistics.
We employ a vision-language Transformer as the base architecture of our baseline model for the CoSIM task. To be specific, we use LXMERT (Tan and Bansal, 2019) to compute the score of each context-response pair given an image feature, and select one with the highest score among them.
We employ Faster R-CNN (Ren et al.,
Figure 5: Frequencies of the change types in our COSIM dataset (from the validation split).
2015) to extract object-level visual features  $O = \{o_1, o_2, \dots, o_{N_O}\}$  and bound boxes  $B = \{b_1, b_2, \dots, b_{N_O}\}$  from an image  $I$ , where  $N_O$  is the number of detected object features. For textual feature encoding we use BERT (Devlin et al., 2019) as it is used in LXMERT. We concatenate all the textual data, i.e., question  $Q = \{q_1, \dots, q_{N_Q}\}$ , initial response  $R_i = \{r_{i1}, \dots, r_{iN_{R_i}}\}$ , scene change  $C = \{c_1, \dots, c_{N_C}\}$ , and new response  $R_n = \{r_{n1}, \dots, r_{nN_{R_n}}\}$  along with [CLS] and [SEP] tokens to create a sequence  $W = \{[\mathrm{CLS}], Q, [\mathrm{SEP}], R_i, [\mathrm{SEP}], C, [\mathrm{SEP}], R_n, [\mathrm{SEP}]\}$  where  $N_Q, N_{R_i}, N_C$ , and  $N_{R_n}$  are the lengths of question, initial response, scene change, and new response, respectively.
$$
O, B = \operatorname {F R C N N} (I) \tag {1}
$$
$$
\hat {O} = \operatorname {L i n e a r} _ {O} ([ \mathrm {V - T o k} _ {O} ]; O ] _ {\dim = \mathrm {t}}) \tag {2}
$$
$$
\hat {B} = \operatorname {L i n e a r} _ {B} ([ \mathrm {V - T o k} _ {B}; B ] _ {\dim = \mathrm {t}}) \tag {3}
$$
$$
\hat {V} = \operatorname {L i n e a r} _ {O B} ([ \hat {O}; \hat {B} ] _ {\dim = f}) \tag {4}
$$