SlowGuess/NAACL_2021_OCR huggingface dataset:
Baseline 1- Machine Comprehension using RoBERTa: To evaluate the hypothetical VQA task through the text-only model, we convert images into the templated text using scene graphs. The templated text contains two kinds of sentences; one describing properties of the objects i.e. "There is a  $<\mathrm{Z}> < \mathrm{C}> < \mathrm{M}> < \mathrm{S}>$ ", the other one describing the relative spatial location i.e. "The  $<\mathrm{Z}> < \mathrm{C}> < \mathrm{M}> < \mathrm{S}>$  is  $<\mathrm{R}>$  the  $<\mathrm{Z1}> < \mathrm{C1}> < \mathrm{M1}> < \mathrm{S1}>$ ". For example, "There is a small green metal cube." and "The large yellow rubber sphere is to the left of the small green metal cube". Then we concatenate templated text with the action text to create a reading comprehension passage. We use state-of-the-art machine comprehension baseline RoBERTa (Liu et al., 2019) finetuned on the RACE dataset (Lai et al., 2017). Finally, we pre
dict an answer to the question using this reading comprehension passage.
Baseline 2- Visual Question Answering using LXMERT Proposed by (Tan and Bansal, 2019), LXMERT is one of the best transformer based pretrainable visual-linguistic representations which supports VQA as a downstream task. Typical VQA systems take an image and a language input. Therefore, to evaluate CLEVR_HYP in VQA style, we concatenate action text and question to form a single text input. Since LXMERT is pre-trained on the natural images, we finetune it over CLEVR_HYP dataset<sup>6</sup> and then use it to predict answer.
4.4 Systematically incorporating effects of actions into neural models
Baseline 3- Text-editing Image Baseline: In this method, we break-down the QA task with mental simulation in two parts; first, learn to generate an updated image (such that it has incorporated the effects of actions) and then perform visual question answering with respect to the updated image. We use the idea from Text Image Residual Gating proposed in (Vo et al., 2019) to implement the first part. However there are two important distinctions; Their focus is on the retrieval from the given database. We modify their objective and develop text-adaptive encoder-decoder with residual connections to generate new image. Also, editing instructions in their CSS dataset (Vo et al., 2019) were quite simple. For example, 'add red cube' and 'remove yellow sphere'. In this case, one can add the red cube anywhere in the scene. We modify their architecture to precisely place objects to their
optimizer  $\equiv$  adam with eps  $= 1\mathrm{e} - 06$
$^6$  epochs=4, learning rate=5e-05, batch size=8
5architecture=roberta large, epochs=5, learning rate=1e-05, batch size=2, update frequency=2, dropout=0.1,
Nomenclature I: Image, SG: Scene Graph, TT: Templated Text,  $T_A$ : Action Text,  $Q_H$ : Hypothetical Question, A: Answer, FP: Functional Program,': Updated Modality
Baseline 1:
Baseline 2:
Figure 6: Graphical visualization of baseline models over CLEVR_HYP described above.
Baseline 3:
Baseline 4:
relative spatial references (on left/right/front/behind). Once we get the updated image, we feed it to the LXMERT (Tan and Bansal, 2019) finetuned over the CLEVR (Johnson et al., 2017a) dataset along with the question and predict the answer.
Baseline 4- Scene Graph Update Model: Instead of directly manipulating images, in this method, we leverage image scene graphs to convert image-editing problem into graph-editing problem, conditioned on the action text. This is an emerging research direction to deal with changes in the visual modality over time or with new sources of information, as observed from recent parallel works (chang Chen et al., 2020; He et al., 2020).
We first use Mask R-CNN (He et al., 2017) to get the segmentation mask of the objects and predict attributes (color, material, size, and shape) with an acceptance threshold of 0.9. Segmentation mask of each object along with original image is then passed through ResNet-34 (He et al., 2016) to extract precise 3D coordinates of the object. We get the structured scene graph for the image. Then we use seq2seq with attention model originally proposed in (Johnson et al., 2017b) to generate functional programs (FP) for action text and question. The execution engine executes programs on scene graph, implemented as a neural module network (Andreas et al., 2016) to update the scene representation and answer questions.
We learn to update scene graphs according to functional program for the action text using reinforcement learning<sup>7</sup>. The reward function is as-
sociated with our ground-truth program executor and generates reward if prediction exactly matches with ground-truth execution. Once we get the updated scene representation, we use neural-symbolic model<sup>8</sup> proposed by (Yi et al., 2018) to obtain the final answer. It is notable that (Yi et al., 2018) achieved near-perfect performance on the CLEVR QA task in addition to being fully explainable.
5 Baseline Results
In this section, we benchmark models described above on the CLEVR_HYP. The dataset is formulated as a classification task with exactly one correct answer, so we use standard accuracy as evaluation metric. We then analyze their performance according to question and action types.
Quantitative results from above experiments can be visualized in top part of the Table 3. Among the methods described above, the scene graph update model has the best overall performance  $70.5\%$  on original test data. Text-editing model is best over balanced set, but observed to have the poor generalization capability when two actions or reasoning capabilities have to be performed. CLEVR_HYP requires models to reason about effect of hypothetical actions taken over images. LXMERT is not directly trained for this objective therefore, it struggles to do well on this task. The reason behind the poor performance of text-only baseline is due to its limitation to incorporate detailed spatial locations
stopping, batch size=32
supervised pretraining learning rate  $= 7\mathrm{e} - 04$  num iterations  $= 20\mathrm{k}$  batch size  $= 32$  and then finetuning 1e-05, at most 2M iterations with early stopping, batch size  $= 32$
7finetuning learning rate=1e-05, 1M iterations with early
Overall Baseline Performance for Various Test Sets of CLEVR_HYP
Original Test	Balanced Test	2HopTA Test	2HopQH Test
BL1	BL2	BL3	BL4	BL1	BL2	BL3	BL4	BL1	BL2	BL3	BL4	BL1	BL2	BL3
57.2	63.9	64.7	70.5	55.3	65.2	69.5	68.6	53.3	49.2	55.6	64.4	55.2	52.9	58.7
66.5
Performance break-down by Action Types and Reasoning Types for Baseline 3 and 4
Original Test	2HopATest	Original Test	2HopQHTest
BL3	BL4	BL3	BL4	BL3	BL4	BL3	BL4
Add	58.2	65.9	Add+Remove	53.6	63.2	Count	60.2	74.3	And	59.2	67.1
Remove	89.4	88.6	Add+Change	55.4	64.7	Exist	69.6	72.6	Or	58.8	67.4
Change	88.7	91.2	Add+Move	49.7	57.5	CompInt	56.7	67.3	Not	58.1	65.0
Move(in-plane)	61.5	69.4	Remove+Change	82.1	85.5	CompAttr	68.7	70.5
Move(on)	53.3	66.1	Remove+Move	52.6	66.4	QueryAttr	65.4	68.1
Change+Move	53.8	63.3
Table 3: Baseline performance over CLEVR_HYP (BLx represents one of the four Baselines described above).
into the templates that we use to convert image into a machine comprehension passage.
Two of our models (scene graph update and text-editing image) are transparent to visualize intermediate changes in the scene after performing actions. We analyse their ability to understand actions and make appropriate changes as shown in below part of Table 3. For the scene graph method, we compare the ground-truth functional program with the generated program and measure their exact-match accuracy. For the text-editing image method, we generate scene graphs for both images (original image and image after text-editing) and compare them. For attributes, we do exact-match, whereas for location information we consider matching only on the basis of relative spatial location.
Both scene graph and text-editing models do quite well on 'remove' and 'change' actions whereas struggle when new objects are added or existing objects are moved around. The observation is consistent when multiple actions are combined. Therefore, actions remove+change can be performed with maximum accuracy whereas other combinations of actions accomplish relatively lower performance. It leads to the conclusion that understanding the effect of different actions are of varied complexity. Most models demonstrate better performance over counting, existence and attribute query type of questions than comparison questions. The scene graph update and text-editing methods show a performance drop of  $6.1\%$  and  $9.1\%$  respectively when multiple actions are performed on the scene. However, there is less of a performance gap for models on  $2\mathrm{HopQ}_H$  compared to the test set,
suggesting that models are able to better generalize with respect to multiple reasoning skills than complex actions.
We introduce CLEVR_HYP, a dataset to evaluate the ability of VQA systems after hypothetical actions are performed over the given image. We create this dataset by extending the data generation framework of CLEVR (Johnson et al., 2017a) that uses synthetically rendered images and templates for reasoning questions. Our dataset is challenging because rather than asking to reason about objects already present in the image, it asks about what would happen in an alternative world where changes have occurred. We provide ground-truth representations for images, hypothetical actions and questions to facilitate the development of models that systematically learn to reason about underlying process. We create several baseline models to benchmark CLEVR_HYP and report their results. Our analysis shows that the models are able to perform reasonably well (70.5%) on the limited number of actions and reasoning types, but struggle with complex scenarios. While neural models have achieved almost perfect performance on CLEVR and considering human performance as upperbound (98%), there is a lot of room for improvement on CLEVR_HYP. Our future work would include relaxing constraints by allowing a larger variety of actions, attributes and reasoning types. By extending this approach further for natural images, we aim to contribute in the development of better vision+language models.
We are thankful to the anonymous reviewers for the constructive feedback. This work is partially supported by the grants NSF 1816039, DARPA W911NF2020006 and ONR N00014-20-1-2332.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 3674-3683. IEEE Computer Society.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 39-48. IEEE Computer Society.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425-2433. IEEE Computer Society.
Blender Online Community. 2019. Blender - a 3D modelling and rendering package. Blender Foundation.
Li chang Chen, Guosheng Lin, S. Wang, and Qingyao Wu. 2020. Graph edit distance reward: Learning to edit scene graph. In Proceedings of the European Conference on Computer Vision (ECCV).
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12538-12547.
Hao Dong, Simiao Yu, Chao Wu, and Yike Guo. 2017. Semantic image synthesis via adversarial learning. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5707-5715. IEEE Computer Society.
Zhiyuan Fang, Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020. Video2Commonsense: Generating commonsense descriptions to enrich video captioning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 840-860. Association for Computational Linguistics.
David Gaddy and Dan Klein. 2019. Pre-learning environment representations for data-efficient neural instruction following. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1946-1956. Association for Computational Linguistics.
Lee Gomes. 2014. Machine-learning maestro michael jordan on the delusions of big data and other huge engineering efforts.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 4(5):6.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. 2018. IQA: visual question answering in interactive environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 4089-4098. IEEE Computer Society.
Kaiming He, Georgia Gkioxari, Piotr Dóllar, and Ross B. Girshick. 2017. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2980-2988. IEEE Computer Society.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Xuanli He, Quan Hung Tran, Gholamreza Haffari, Walter Chang, Zhe Lin, Trung Bui, Franck Dernoncourt, and Nhan Dam. 2020. Scene graph modification based on natural language commands. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 972-990. Association for Computational Linguistics.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700-6709.
Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. 2017. First quora dataset release: Question pairs. data.quora.com.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017a. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1988-1997. IEEE Computer Society.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017b. Inferring and executing programs for visual reasoning. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 3008-3017. IEEE Computer Society.
Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. 2018. DVQA: understanding data visualizations via question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 5648-5656. IEEE Computer Society.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300.
John Kanu, Eadom Dessalene, Xiaomin Lin, Cornelia Fermuller, and Yiannis Aloimonos. 2020. Following instructions by imagining and reaching visual goals. arXiv preprint arXiv:2001.09373.
Aniruddha Kembhavi, Min Joon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5376-5384. IEEE Computer Society.
Satwik Kottur, José M. F. Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2019. CLEVR-dialog: A diagnostic dataset for multi-round reasoning in visual dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 582-595. Association for Computational Linguistics.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAd-ing comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794. Association for Computational Linguistics.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020. What does BERT with vision look at? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5265-5275, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. 2018. Text-adaptive generative adversarial networks: Manipulating images with natural language. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada, pages 42-51.
Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. 2019. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 12527-12537. Computer Vision Foundation / IEEE.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics* (Demonstrations), pages 48-53. Association for Computational Linguistics.
J. Park, Chandra Bhagavatula, R. Mottaghi, A. Farhadi, and Yejin Choi. 2020. Visualcomet: Reasoning about the dynamic context of a still image. In Proceedings of the European Conference on Computer Vision (ECCV).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. Generative adversarial text to image synthesis. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1060-1069. JMLR.org.
Mengye Ren, Ryan Kiros, and Richard S. Zemel. 2015. Exploring models and data for image question answering. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2953-2961.
Tianxiao Shen, Myle Ott, Michael Auli, and Marc'Aurelio Ranzato. 2019. Mixture models for diverse machine translation: Tricks of the trade. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5719-5728. PMLR.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10737-10746. IEEE.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100-5111. Association for Computational Linguistics.
Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019. WIQA: A dataset for "what if..." reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6076-6085. Association for Computational Linguistics.
Damien Teney, Ehsan Abbasnedjad, and Anton van den Hengel. 2020. Answering visual what-if questions: From actions to predicted scene descriptions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 580-599.
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. 2019. Composing text and image for image retrieval - an empirical odyssey. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6439-6448. Computer Vision Foundation / IEEE.
Misha Wagner, Hector Basevi, Rakshith Shetty, Wenbin Li, Mateusz Malinowski, Mario Fritz, and Ales Leonardis. 2018. Answering visual what-if questions: From actions to predicted scene descriptions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 0-0.
Terry Winograd. 1971. Procedures as a representation for data in a computer program for understanding natural language. Technical report, Massachusetts inst of tech cambridge project mac.
Guangyu Robert Yang, Igor Ganichev, Xiao-Jing Wang, Jonathon Shlens, and David Sussillo. 2018. A dataset and architecture for visual reasoning with a working memory. In Proceedings of the European Conference on Computer Vision (ECCV), pages 714-731.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural-symbolic VQA: disentangling reasoning from vision and language understanding. In Advances in Neural Information Processing Systems
31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada, pages 1039-1050.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6720-6731. Computer Vision Foundation / IEEE.
A.1 Relation of CLEVR_HYP dataset with real-world situations
Teaching methodologies leverage our ability to mentally simulate scenarios along with the metaphors to aid understanding about new concepts. In other words, to explain unfamiliar concepts, we often reference familiar concepts and provide additional clues to establish mapping between them. This way, a person can create a mental simulation about unfamiliar concept and aid basic understanding about it.
For example, we want to explain a person how a 'zebra' looks like, who has previously seen a 'horse', we can do so using example in Figure 7a. This naturally follows for more complex concepts. Let say, one wants to describe the structure of an atom to someone, he might use the analogy of a planetary system, where the components (planets  $\sim$  electrons) circulate around a central entity (sun  $\sim$  nucleus). One more such example is provided in Figure 7b.
(a) learning the concept 'zebra' from the 'horse'
(b) learning about 'animal cell' by comparison with 'plant cell'
Figure 7: Extension of CLEVR_HYP for more complex real-world scenarios.
For humans, learning new concepts and perform