SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
L = \operatorname {E m b} (W), \hat {L} = \operatorname {T F} _ {L} (L) \tag {5}
$$
where  $\mathrm{Linear}_O$ ,  $\mathrm{Linear}_B$ , and  $\mathrm{Linear}_{OB}$  are linear layers.  $[\mathrm{V - Tok}_O]$  and  $[\mathrm{V - Tok}_B]$  are visual token attached to object and bounding box sequences (like [CLS] for a language sequence), respectively, and  $[;]_{\mathrm{dim} = \mathrm{t}}$  is concatenation operation along the token-dimension and  $[;]_{\mathrm{dim} = \mathrm{f}}$  is along feature-dimension.  $\mathrm{TF}_L$  is a language Transformer (Vaswani et al., 2017) which consists of self-attention layers. The  $i$ th attention head in the  $l$ th layer  $a_{i,l}$  is computed this way:
$$
a _ {i, l} = \operatorname {S o f t m a x} \left(\frac {Q K ^ {\top}}{\sqrt {d _ {h}}}\right) V \tag {6}
$$
$$
Q = W _ {l} ^ {q} H _ {l - 1}, K = W _ {l} ^ {k} H _ {l - 1}, V = W _ {l} ^ {v} H _ {l - 1} \tag {7}
$$
$$
H _ {l} = \left[ a _ {0, l}; a _ {1, l}; \dots ; a _ {N _ {A}, l} \right] \tag {8}
$$
where  $W_{l}^{q}, W_{l}^{k}$ , and  $W_{l}^{v}$  are trainable parameters,  $N_{A}$  is the number of attention head, and  $d_{h}$  is the
Figure 6: The full model computes the matching scores between [V-Tok] token feature and each [CLS] token feature of the response candidates (a ground-truth response and three distractors), and selects the highest one as a final prediction.
dimension of each attention head. Then,  $\hat{V}$  and  $\hat{L}$  are fed to the cross-attention layers:  $\bar{V},\bar{L} =$ $\mathrm{TF}_X(\hat{V},\hat{L})$  , where  $\mathrm{TF}_X$  is cross-attention layers of vision and language Transformer which consists of self-attention layers as well as cross-attention layers. Scores are computed between visual feature and each of the 4 language features (1 ground-truth and 3 distractors) pair:  $s_k = \mathrm{Linear}(\bar{V}_0*\bar{L}_{k,0})$  where  $*$  is the element-wise product,  $\hat{V}_0$  is the visual token (i.e., [V-Tok]) that is attached in the input layer, and  $\hat{L}_{k,0}$  is the first token (i.e., [CLS]) of  $k$  -th language feature. The model compares the 4 scores to select the pair with the highest score as the final answer. The loss is computed by crossentropy:  $\mathcal{L} = -\sum_{j}^{N}\log p(s_{j}^{*})$  ,where  $s_j^*$  is a score for the ground-truth pair.
Data Splits. We split the dataset into 1,924/800/800 (train/val/test).
Training Details. We use 768 as the hidden size and use Adam (Kingma and Ba, 2015) as the optimizer, setting the learning rate to  $1 \times 10^{-5}$ . See Appendix for more details.
Human Upper Bound Evaluation Setup. We conduct a human evaluation of our CoSIM task to estimate the upper bound that models can reach. We take 50 samples from the validation split and ask two experts to complete the task and average their scores.
Scene Change Types. We collect the type of the Scene Change for the validation set. Two experts are shown each change and then asked to label it into one or more types. See Figure 5 for the change
Model	Accuracy (%)
Val	Test
1	Response-Only	38.37	-
2	TC-Response	44.62	-
3	Full (Image-TC-Response)	49.25	40.87
4	Human	98	-
Table 3: Model results. Human performance is quite high, showing large room for model improvement (TC: Textual Context). We only include the test performance for the full final model based on standard practice of not reporting ablations on test split.
Scene Change Type	Accuracy (%)
1	Object Addition	46.41
2	Object Removal	37.87
3	Object Replacement	43.59
4	Object Relocation	48.72
5	Object State Change	51.70
6	Human Addition	56.40
7	Human Removal	42.10
8	Environment Change	52.70
9	Event Description	51.56
Table 4: Model performance on different change types. While the model generally shows balanced scores over all scene change types, the performance on removal types seems to be lower than addition types.
Multi-Task / Contrastive Learning. To exploit extra commonsense reasoning information, we explore multi-task learning (MTL) with a large-scaled visual commonsense reasoning dataset, VCR (Zellers et al., 2019) dataset through alternating mini-batch training. In one mini-batch, the model is trained on our CoSIM dataset, and in the next, the model is trained on the VCR dataset, and so on. Also, we try contrastive learning to explore potential improvement. Specifically, we compute matching scores between each visual token and [CLS] token of each ground truth text feature in a mini-batch, and compute contrastive loss.
Modality Ablation. We build models with different input modalities and conduct an ablation study. As shown in Table 3, the Response-Only model (which only takes the new response/distractors as input) does not do well (row 1). The TC-Response model (which takes all text data as input) obtains a better score than the Response-Only model (row 1 and 2), but still performs poorly. The Full model (which takes the full image and text data as input) does best (row 3), meaning models need all the
Number of Scene Change Types	Accuracy (%)
1	52.35
2	46.34
3 or more	49.15
Table 5: Model performance on different numbers of change types, showing instances with single scene change type are relatively easier.
Model	Accuracy (%)
1	MTL with VCR	47.37
2	Contrastive Learning	49.50
Table 6: Model performance on multi-task and contrastive learning approaches.
visual and textual input to perform reasonably. $^{5,6}$
Human Evaluation. We conduct a human evaluation to check the upper performance bound for the COSIM task. As shown in row 4 of Table 3, the score is quite high<sup>7</sup>, indicating a large room for improvement from future work.
Scene Change Types. As shown Table 4, our model shows balanced scores over all scene change types in general, however, comparing the addition and removal types (row 1 and 2 for object, row 6 and 7 for human), the performance on removal types is lower than addition types. That is possibly because removing something from an image might be harder to imagine.
Number of Scene Change Types. As shown Table 5, instances with a single scene change type seem to be relatively easier to address than ones with multiple scene change types. This might imply that multiple scene changes make the reasoning process more complex and challenging.
Multi-Task / Contrastive Learning. As shown in row 1 of Table 6, multi-task training with VCR does not seem to help improve performance on our CoSIM dataset, implying our dataset is challenging to address and requires a more complex reasoning process. The performance of the contrastive learning (row 2) is also very close to the Full model's (row 3 in Table 3), meaning more advanced approaches might be needed to tackle our CoSIM dataset/task.
Q: Is it a good place for a beginner to learn to snowboard?
IR: No, because it is really crowded and there is not much space to practice.
CH: It is not crowded, but it has very steep slopes.
Model choice: no, because it has very precipitous slopes and it is not safe for new learners.
Figure 7: Model output examples (Q: question, IR: initial response, CH: scene change).
Q: Is it safe to swim here?
IR: No, it is a pier with lots of boats in and out, so it wouldn't be safe to swim.
CH: there are not a lot of boats, but there is a shark
Model choice: no, there is a shark in the boat.
Output Examples. As shown in the upper example of Figure 7, our model predicts the correct response by understanding the implication of "steep slopes" in the change. In the bottom example, our model fails to understand that "there is a shark" must mean the shark is in the water (as sharks live in the water), and choose a wrong response. We also split changes into sub-parts and compute scores for each part to see on which part the model focuses to answer questions. As shown in Figure 8, the model looks at "Add labels to the spines of all the books" to choose the answer.
We introduced a challenging counterfactual commonsense reasoning task/dataset called CoSIM which features imagined counterfactual scene changes requiring models to imagine the changed
5The standard deviation of the Full model's scores on validation split is 1.52
<sup>6</sup>The average length difference between the predicted responses and the rest is 0.0075 words, and the ground-truth response indices are randomly assigned, thus there is no bias based on the response length and index.
7Inter-annotator agreement (kappa) is 0.9461, which indicates nearly perfect agreement.
Q: Are the books for sale?
Figure 8: The model focuses on a crucial part in the scene change to properly select the answer.
IR: Yes, as you can see the books are organized by names and authors on the shelves for the customers.
CH: Add a trolley in the foreground with many books replaced on it. Add labels to the spines of all the books.
Answer: It seems from the trolley that this is a library. the books on display are probable for reading and maybe to be borrowed but they are likely not for sale.
situation to answer questions. We collected  $3.5\mathrm{K}$  high-quality instances that consist of an image, an initial question-response pair on the image, an imagined scene change, and a new response (with three distractors). The scene changes have different challenging types (such as object addition/removal/replacement, environment change, etc.). We presented a baseline model as a starting point with useful ablation studies and showed a large human-model performance gap allowing useful future works.
We thank the reviewers for their helpful comments. This work was supported by NSF Award 1840131, ARO Award W911NF2110220, DARPA MCS Grant N66001-19-2-4031, and a Google Focused Award. The views contained in this article are those of the authors and not of the funding agency.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6700-6709.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatiotemporal reasoning in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2758-2766.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2910.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In *ICLR*.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. Tvqa: Localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1369-1379.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901.
Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. 2020. Visualcomet: Reasoning about the dynamic context of a still image. In In Proceedings of the European Conference on Computer Vision (ECCV).
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop.
Hamed Pirsiavash, Carl Vondrick, and Antonio Torralba. 2014. Inferring the why in images. Technical report, MASSACHUSETTS INST OF TECH CAMBRIDGE.
Shaoqing Ren, Kaiming He, Ross B Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS.
Shailaja Keyur Sampat, Akshay Kumar, Yezhou Yang, and Chitta Baral. 2021. Clevr_hyp: A challenge dataset and baselines for visual question answering with hypothetical actions over images. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3692-3709.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4631-4640.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems.
Misha Wagner, Hector Basevi, Rakshith Shetty, Wenbin Li, Mateusz Malinowski, Mario Fritz, and Ales Leonardis. 2018. Answering visual what-if questions: From actions to predicted scene descriptions. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0.
Jianwei Yang, Jiasen Lu, Dhruv Batra, and Devi Parikh. 2017. A faster pytorch implementation of faster r-cnn. https://github.com/jwyang/faster-rcnn.pytorch.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In *The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.
Linchao Zhu, Zhongwen Xu, Yi Yang, and Alexander G Hauptmann. 2017. Uncovering the temporal context for video question answering. International Journal of Computer Vision, 124(3):409-421.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995-5004.
We implement different interfaces for our data collection. The commonsense question collection interface allows for workers to choose which image they would like to use when making the question,
as well as an object to focus on (Figure 9). The counterfactual scene change collection and the distractor collection interfaces (Figure 10 and Figure 11) feature a verification checkbox. Workers can check the box if the quality of the data from the previous phase is poor. If it is flagged, the reason is reviewed. $^{8}$  If the reasoning is valid, the instance is removed from the dataset/no longer progressed through the collection phases.
A.1 Counterfactual Change Collection templates
The first guide template is "Keep A, Flip B" and the second is "Flip A, Keep B" (where 'A' means answer and 'B' means justification). For "Keep A, Flip B", workers are told to describe a change that results in the "answer" part of initial response to be the same, but with a different "justification" part (E.g. "yes because people are wearing jackets and winter clothes."  $\rightarrow$  "yes because you can see some snow ..."). In the change they write, they should negate/remove the "justification" part of initial response and add something that could be used for a new "justification". For "Flip A, Keep B", workers are told to describe a change that results in the opposite "answer". The change should also modify the context so that the initial response "justification" part is true, but is no longer valid in proving the answer and a new "justification" part is needed. (e.g., "no, as you can see the man is not soaking wet."  $\rightarrow$  "yes, the man isn't wet and he is under a structure, however ...").<sup>9</sup>
A.2 Worker Totals and Payment
We had a total of 182, 97, 194 workers pass testing for question collection, change collection, distractor collection, respectively. For the question collection phase and the change collection phase, workers are paid 0.35 USD per instance they complete (each takes about 2 minutes). As the distractor collection phase is faster and easier, workers are paid 0.30 USD per instance (takes around 1.5 minutes). In all three phases, an additional bonus of 0.02 USD is given for each high-quality instance they completed, and then for every subsequent group of 25 high-quality instances completed, the bonus per instance is increased by 0.01 USD (0.02 USD bonus per instance for the first 25, 0.03 USD bonus
Once the flag is checked, workers are provided with a textbox where they can explain their reasoning for flagging it.
The proportions of "Keep A, Flip B" and "Flip A, Keep B" are  $42.93\%$  and  $57.07\%$ , respectively.
for the next 25, 0.04 USD bonus for the next 25, and so on). Since there is no limit on how much a worker can write, they can keep stacking the bonus as much as they want. All the payments are at a reasonable hourly rate of 11-12 USD.
B Scene Change Types
The scene change types, while they are seemingly easy to visualize, require a complex understanding of what effect the change has on other elements in the scene. The Object Addition scene change type (the most commonly occurring one) involves introducing new object(s) into the image that was not there prior. The Object State Change scene change type involves altering the state of object(s) present in the image. The alteration of object(s) can take place in various forms such as changing color, size, shape, and orientation (e.g., opening a door). The Event Description scene change type involves the creation of an event or a description of motion or interaction between objects in the image. Please see Figure 12 for the full list of the scene change types and examples.
C Training Details (Reproducibility)
All the model experiments are conducted on a Ubuntu 16.04 system using the NVIDIA GeForce GTX 1080 Ti GPU and Intel Xeon CPU E5-2630. We employ PyTorch1.4 (Paszke et al., 2017) to build our models. We run models up to 50 epochs (each epoch takes around 8 mins) and choose the best ones based on the validation split evaluation. We use 768 as the hidden size and use Adam (Kingma and Ba, 2015) as the optimizer, setting the learning rate to  $1 \times 10^{-5}$ . We initialize the language layer with the pretrained BERT weights and cross-attention layers with the pretrained LXMERT weights. We use 1234/2345/3456 as the random seed values. The number of trainable parameters of our full model is 173M. We employ accuracy as the evaluation metric. We use manual hyperparameter tuning (e.g., learning-rate=\{1 \times 10 $^{-3}$ , ..., 1 \times 10 $^{-6}\}$ , num-of-cross-layer=\{1, 2, ..., 5\}, batch-size=\{2,4,6,8\}, etc.) based on validation scores. We use the implementation of Yang et al. (2017) for the Faster R-CNN (Ren et al., 2015) model. The evaluation splits of our CoSIM dataset are not overlapped with the training split of the Faster R-CNN.
D Potential Risk
Potential models trained on our dataset may learn misleading information accidentally and create unsafe suggestions; therefore, careful use is required when deploying models in a real-world application.
Here are your images, choose only 1 for your HIT:
Open image in new tab for larger view
Please select which image you will be using for your HIT:
Please select which object from the image you will be using for your HIT:
Please avoid writing inappropriate topics or phrases.
Write your Question to the image here:
Are you sure your question matches all criteria?
Write your Answer here.
Figure 9: Collection interface for the commonsense question collection phase. Workers are given three images, and they select the one they wish to use. Then workers are given space to write their question and response. Workers are told to select an object in the image they choose to help them focus their question around something specific.
Here is your image:
Here is the question: $ {question}
Here is the answer: \$\{answer\}
Flag as a bad original question and answer
Write your Change to the image here:
Write your new Answer here:
Which of the 2 methods did you choose
Any comments you have ("the original question/answer was too hard to change", "the task is unclear, etc) If it was tricky or very hard PLEASE use the checkbox above to indicate it was a bad original question and answer:
Figure 10: Collection interface for the change collection phase. Workers are given the selected image and the written question and response from the commonsense question collection phase and then asked to write a change and new response based off that change.
Here is the information you will be using:
Here is your question: $(question)
Here is your original answer: $(answer)
Here is your change: $(change)
Here is your new answer: $(new_answer)
Please carefully read over the 'change' and 'new answer' we give you above and if they do not match criteria of  $\S$  (method), you can flag it here.
- Flag as a bad change and/or new answer
You may wish to modify the new answer a little so that it is easier to get the green light. However, make sure the original meaning of new answer stays the same, meaning it is still a correct answer to the question. You can modify the new answer here but please make sure that your fake answers look very similar to the new answer even if you change it. It might help to copy whatever you write here to each of the fake answer textboxes before you make them.
$(new_answer)
Write your fake answers here:
{$new_answer}
{$new answer}
Note: it might take a few seconds for the light to show once you click, please do not spam click.
Click here to see if system gets fooled.
Figure 11: Collection interface for the distractor collection phase. Workers are given the image and all the context from the previous phases and then asked to write three distractors that are similar to the new response but are distinct/semantically different. The distractor textboxes are prefilled with the new response and during HAMLET collection, workers are given a section to check their distractors against the model. Note: This interface is quite large and relevant portions are stitched together.
"Add snow and ice to the road. Add a bus. Place the person inside the bus."
Figure 12: Scene change examples from our dataset. The relevant portions of the change are in italics.
Human Addition
"There is now a person in each boat. They each also have a fishing pole that is being used on the water."
Object Removal
"Remove all the kites and the unfurled sail. Add some people in the foreground with long hair. Have their hair blowing horizontally..."
Human Removal
"There is no snow and there are no people, but the windows on the building are covered with ice."
Object Replacement
"Change the cake to a round cake covered in something that has the colour of marzipan. Add 13 round balls on the cake with the same colour."
Object Relocation
"...move the skateboarder much higher up in the air... move the skateboard further away from the skateboarder so they can not land with the board."
Object State Change
"Dress everyone in black clothes and give the scene a somber mood. Change the table cloth to white."
Event Description
"Add rice falling from the sky on to the umbrella. show the rice accumulated on the umbrella and falling off in the places where the umbrella dips."
Environment Change
"Dress the others in clothes suited to a period of the Roman Empire. Add a market behind on the raised area with the people there now shopping for produce..."