SlowGuess/NAACL_2021_OCR huggingface dataset:
Shailaja Keyur Sampat,\* Akshay Kumar, Yezhou Yang and Chitta Baral
Arizona State University, USA
{ssampa17,akuma216,yz.yang，chitta}@asu.edu
Most existing research on visual question answering (VQA) is limited to information explicitly present in an image or a video. In this paper, we take visual understanding to a higher level where systems are challenged to answer questions that involve mentally simulating the hypothetical consequences of performing specific actions in a given scenario. Towards that end, we formulate a vision-language question answering task based on the CLEVR (Johnson et al., 2017a) dataset. Wethen modify the best existing VQA methods and propose baseline solvers for this task. Finally, we motivate the development of better vision-language models by providing insights about the capability of diverse architectures to perform joint reasoning over image-text modality<sup>1</sup>.
In 2014, Michael Jordan, in an interview (Gomes, 2014) said that "Deep learning is good at certain problems like image classification and identifying objects in the scene, but it struggles to talk about how those objects relate to each other, or how a person/robot would interact with those objects. For example, humans can deal with inferences about the scene: what if I sit down on that?, what if I put something on top of something? etc. There exists a range of problems that are far beyond the capability of today's machines."
While this interview was six years ago, and since then there has been a lot of progress in deep learning and its applications to visual understanding. Additionally, a large body of visual question answering (VQA) datasets (Antol et al., 2015; Ren et al., 2015; Hudson and Manning, 2019) have been compiled and many models have been developed
Figure 1: Motivation for the proposed CLEVR_HYP dataset: an example demonstrating how humans can do mental simulations and reason over resulting scenario.
over them, but the above mentioned "inferences about the scene" issue stated by Jordan remains largely unaddressed.
In most existing VQA datasets, scene understanding is holistic and questions are centered around information explicitly present in the image (i.e. objects, attributes and actions). As a result, advanced object detection and scene graph techniques have been quite successful in achieving good performance over these datasets. However, provided an image, humans can speculate a wide range of implicit information. For example, the purpose of various objects in a scene, speculation about events that might have happened before, consider numerous imaginary situations and predicting possible future outcomes, intentions of a subject to perform particular actions, and many more.
Among the above, an ability to imagine taking specific actions and simulating probable results without actually acting or experiencing is an important aspect of human cognition (Figure 1 gives an example of this). Thus, we believe that having autonomous systems equipped with a similar capability will further advance AI research. This is particu
$^{1}$ Dataset setup scripts and code for baselines are made available at https://github.com/shailaja183/clevr_hyp. For additional details about the dataset creation process, refer supplementary material.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3692-3709 June 6-11, 2021. ©2021 Association for Computational Linguistics
larly useful for robots performing on-demand tasks in safety-critical situations or navigating through dynamic environments, where they imagine possible outcomes for various situations without executing instructions directly.
Motivated by the above, we propose a challenge that attempts to bridge the gap between state-of-the-art AI and human-level cognition. The main contributions of this paper are as follows;
- We formalize a novel question answering task with respect to a hypothetical state of the world (in a visual form) when some action (described in a textual form) is performed.
- We create a large-scale dataset for this task, and refer it as CLEVR_HYP i.e. VQA with hypothetical actions performed over images in CLEVR (Johnson et al., 2017a) style.
- We first evaluate the direct extensions of top VQA and NLQA (Natural language QA) solvers on this dataset. Then, we propose new baselines to solve CLEVR_HYP and report their results.
- Through analysis and ablations, we provide insights about the capability of diverse architectures to perform joint reasoning over imagedtext modality.
In this section we situate and compare our work with related areas such as implicit text generation/retrieval for a visual, visual question answering (VQA) over synthetic images, question answering (QA) involving hypothetical reasoning, and language-based manipulation in visual domains closest to CLEVR_HYP.
Implicit Text Generation for a Visual:
VisualComet (Park et al., 2020) and Video2Commonsense (Fang et al., 2020) have made initial attempts to derive implicit information about images/videos contrary to traditional factual descriptions which leverage only visual attributes. VisualComet aims to generate commonsense inferences about events that could have happened before, events that can happen after and people's intents at present for each subject in a given image. They use a vision-language
transformer that takes a sequence of inputs (image, event, place, inference) and train a model to predict inference in a language-model style. Video2Commonsense focuses on generating video descriptions that can incorporate commonsense facts related to intentions, effects, and implicit attributes about actions being performed by a subject. They extract top-ranked commonsense texts from the Atomic dataset and modify training objective to incorporate this information.
While both involve a visual-textual component and actions, their key focus is about generating plausible events and commonsense respectively. Whereas, our work is related to performing certain actions and reasoning about its effect on the overall visual scene.
Language-based Manipulation in Visual Domain: Learning a mapping from natural language instructions to a sequence of actions to be performed in a visual environment is a common task in robotics (Kanu et al., 2020; Gaddy and Klein, 2019; Shridhar et al., 2020). Another relevant task is vision-and-language navigation (Anderson et al., 2018; Chen et al., 2019; Nguyen et al., 2019), where an agent navigates in a visual environment to find goal location by following natural language instructions. Both above works include visuals, natural language instructions and a set of actions that can be performed to achieve desired goals. In this way, it is similar to our CLEVR_HYP, but in our case, models require reasoning about the effect of actions performed rather than determining which action to perform. Also, we frame this in a QA style evaluation rather than producing instructions for low-level controls.
Manipulation of natural images with language is an emerging research direction in computer vision. (Teney et al., 2020) proposed a method for generating counterfactual of VQA samples using image in-painting and masking. Also, there are works (Dong et al., 2017; Nam et al., 2018; Reed et al., 2016) which use Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) for language conditioned image generation and manipulation. However, both the above tasks are more focused at object and attribute level manipulation rather than at action level.
VQA over Synthetic Images: While natural images-based VQA datasets reflect challenges one can encounter in real-life situations, the require
2Our work focuses on the capability of neural models to reason about the effects of actions given a visual-linguistic context and not on models that deal with intuitive physics.
Figure 2: Three examples from CLEVR_HYP dataset: given image (I), action text  $(\mathrm{T}_A)$ , question about hypothetical scenario  $(\mathbf{Q}_H)$  and corresponding answer (A). The task is to understand possible perturbations in I with respect to various action(s) performed as described in  $\mathrm{T}_A$ . Questions test various reasoning capabilities of a model with respect to the results of those action(s).
1.  $\mathbf{T}_A$ : Paint the small green ball with cyan color.  
 $\mathbf{Q}_H$ : Are there equal yellow cubes on left of purple object and cyan spheres? (A: yes)
2.  $\mathbf{T}_A$ : Add a brown rubber cube behind the blue sphere that inherits its size from the green object.  $\mathbf{Q}_H$ : How many things are either brown or small? (A: 6)
3.  $\mathbf{T}_A$ : John moves the small red cylinder on the large cube that is to the right of purple cylinder.  $\mathbf{Q}_H$ : What color is the object that is at the bottom of the small red cylinder? (A: yellow)
ment of costlier human annotations and vulnerability to biases are two major drawbacks. Contrary to them, synthetic datasets allow controlled data generation at scale while being flexible to test specific reasoning skills.
For the above reasons, following benchmark VQA datasets have incorporated synthetic images; COG (Yang et al., 2018) and Shapes (Andreas et al., 2016) contain images with rendered 2D shapes; SHRDLU (Winograd, 1971), CLEVR (Johnson et al., 2017a), and CLEVR-dialog (Kottur et al., 2019) have rendered scenes with 3D objects; DVQA (Kafle et al., 2018) and FigureQA (Kahou et al., 2017) have synthetically generated charts (bar chart, pie chart, dot-line etc.); VQAabstract (Antol et al., 2015) and IQA (Gordon et al., 2018) involves question-answering over synthetically rendered clipart-style scenes and interactive environments respectively. Our proposed dataset CLEVR_HYP uses CLEVR (Johnson et al., 2017a) style rendered scenes with 3D objects as a visual component. It is distinct from all other synthetic VQA datasets for two key reasons; first, integration of action domain in synthetic VQA and second, the requirement of mental simulation in order to answer the question.
QA involving Hypothetical Reasoning: In the language domain, WIQA (Tandon et al., 2019) dataset tests the model's ability to do what-if reasoning over procedural text as a 3-way classification (the influence between pair of events as positive, negative or no-effect). In vision-language domains, a portion of TQA (Kembhavi et al., 2017) and VCR (Zellers et al., 2019) are relevant. Questions in TQA and VCR involve hypothetical scenarios about multi-modal science contexts and movie scenes respectively. However, none of the above two datasets' key focus is on the model's capability to imagine changes performed over the image.
As shown in Figure 3, the setting of TIWIQ
Figure 3: Example from TIWIQ (Wagner et al., 2018).
Output Description
Ground Truth "the foam is pushed a little by the screw driver" Human Baseline "the foam is pushed because the screw driver drops on it" Hybrid Model Prediction "the foam is pushed by the screw driver" Data-Driven Model Prediction "the foam is pushed by the mustard container
(a benchmark dataset for "physical intelligence") (Wagner et al., 2018) has some similarity with ours. It has synthetically rendered table-top scenes, four types of actions (push, rotate, remove and drop) being performed on an object and what-if questions.
To our best knowledge, TIWIQ dataset is not publicly available. Based on our understanding from their manuscript, we observe following important distinction with this work. Our questions focus on the impact of actions on the whole image, while in TIWIQ questions are about impact of actions on a specific object in the image. Moreover, we frame CLEVR_HYP as a classification task, contrary to TIWIQ which is a generative task. Our CLEVR_HYP dataset has 175k automatically generated image-action text-question samples which is much larger compared to TIWIQ which has only 1020 samples and manually crafted ground-truths.
3 CLEVR_HYP Task and Dataset
Figure 2 gives a glimpse of CLEVR_HYP task. We opt for synthetic dataset creation as it allows automated and controlled data generation at scale with minimal biases. More details are described below.
3 Inputs: Image(I), Action Text  $(\mathbf{T}_A)$  and Hypothetical Question  $(\mathbf{Q}_H)$
1. Image(I): It is a given visual for our task. Each image in the dataset contains 4-10 randomly selected 3D objects rendered using Blender (Blender Online Community, 2019) in CLEVR
(Johnson et al., 2017a) style. Objects have 4 attributes listed in the Table 1. Additionally, these objects can be referred using 5 relative spatial relations (left, right, in front, behind and on). We provide scene graphs<sup>3</sup> containing all ground-truth information about a scene, that can be considered as a visual oracle for a given image.
Attr.	Possible values in CLEVR_HYP
Color	gray, blue, brown, yellow, red, green, purple, cyan
Shape	cylinder, sphere or cube
Size	small or big
Material	metal (shining) or rubber (matte)
Table 1: Object attributes in CLEVR_HYP scenes.
2. Action Text  $(\mathbf{T}_A)$ : It is a natural language text describing various actions performed over the current scene. The action can be one of four:
(i) Add new object(s) to the scene
(ii) Remove object(s) from the scene
(iii) Change attributes of the object(s)
(iv) Move object(s) within scene (might be in plane i.e. left/right/front/back or out of plane i.e. move one object on top of another object<sup>4</sup>)
To generate action text, we start with manually written templates involving the aforementioned actions. For example, action involving change in the attribute of object(s) to a given value, we have a template of the following kind; 'Change the  $<\mathrm{A}>$  of  $<\mathrm{Z}> < \mathrm{C}> < \mathrm{M}> < \mathrm{S}>$  to  $<\mathrm{V}>'$ . Where  $<\mathrm{A}>, <\mathrm{Z}>, <\mathrm{C}>, <\mathrm{M}>, <\mathrm{S}>, <\mathrm{V}>$  are placeholders for the attribute, size, color, material, shape and a value of attribute respectively. Each action text in the CLEVR_HYP is associated with a functional program which if executed on an image's scene graph, yields the new scene graph that simulates the effects of actions.
Functional programs for action texts<sup>3</sup> are built from the basic functions that correspond to elementary action operations (right part of Figure 4a). For the above mentioned 'change' attribute action template, the equivalent functional program can be written as; 'change_attr(<A>,filter_size(<Z>,filter
_color(<C>, filter_material(<M>filter_shape(<S>, scene()))), <V>)'. It essentially means, first filter out the objects with desired attributes and then update the value of their current attribute A to value V.
3. Question about Hypothetical Situation  $(\mathbf{Q}_H)$ : It is a natural language query that tests various visual reasoning abilities after simulating the effects of actions described in  $\mathrm{T}_A$ . There are 5 possible reasoning types similar to CLEVR;
(i) Counting objects fulfilling the condition
(ii) Verify existence of certain objects
(iii) Query attribute of a particular object
(iv) Compare attributes of two objects
(v) Integer comparison of two object sets (same, larger or smaller)
Similar to action texts, we have templates and corresponding programs for questions. Functional programs for questions<sup>3</sup> are executed on the image's updated scene graph (after incorporating effects of the action text) and yields the ground-truth answer to the question. Functional programs for questions are made of primitive functions shown in left part of the Figure 4a).
Paraphrasing: In order to create a challenging dataset from linguistic point of view and to prevent models from overfitting on templated representations, we leverage noun synonyms, object name paraphrasing and sentence-level paraphrasing. For noun synonyms, we use a pre-defined dictionary (such as cubeblock, sphereball and so on). We programmatically generate all possibilities to refer to an object in the image (i.e. object name paraphrasing) and randomly sample one among them. For sentence level paraphrasing, we use Text-To-Text Transfer Transformer (T5) (Raffel et al., 2020) fine-tuned over positive samples from Quora Question Pairs (QQP) dataset (Iyer et al., 2017) for question paraphrasing. We use Fairseq (Ott et al., 2019) for action text paraphrasing which uses round-trip translation and mixture of experts (Shen et al., 2019).
Note that we keep the action text and question as separate inputs for the purpose of simplicity and keeping our focus on building solvers that can do mental simulation. One can create a simple template like “ $\langle \mathrm{Q}_H\rangle$  if  $\langle$  proper-noun/pronoun  $\rangle$ $\langle \mathrm{T}_A\rangle$ ?” or “If  $\langle$  proper-noun/pronoun  $\rangle$ $\langle \mathrm{T}_A\rangle$
3Scene graphs and Functional Programs (for action text and question) are not provided at the test-time. 4For simplicity, we assume that any object can be put on another object regardless of its size, material or shape.
(a) Function Catalog for CLEVR_HYP, extended from CLEVR (Johnson et al., 2017a)
(b) Dataset creation pipeline
Figure 4: CLEVR_HYP dataset creation process with example and function catalog used for ground-truth answer generation. (for more details, see Appendix A.4)
$< \mathrm{Q}_H > ?"$  if they wish to process action and question as a single text input. For example, "How many things are the same size as the cyan cylinder if I add a large brown rubber cube behind the blue object." or "If I add a large brown rubber cube behind the blue object, how many things are the same size as the cyan cylinder?" However, having them together adds further complexity on the solver side as it first has to figure out what actions are performed and what is the question.
By providing ground-truth object information (as a visual oracle) and machine-readable form of questions & action texts (oracle for linguistic components). This information can be used to develop models which can process semi-structured representations of image/text or for the explainability purposes (to precisely know which component of the model is failing).
Output: Answer (A) to the Question  $(\mathrm{Q}_H)$ , which can be considered as a 27-way classification over attributes (8 colors + 3 shapes + 2 sizes + 2 material), numeric (0-9) and boolean (yes/no).
Dataset Partitions and Statistics: We create CLEVR_HYP dataset containing 175k image-action text-question samples using the process men
tioned in Figure 4b. For each image, we generate 5 kinds of action texts (one for each add, remove, move in-plane and move out-of-plane and change attribute). For each action text type, we generate 5 questions (one for each count, exist, compare integer, query attribute and compare attribute). Hence, we get  $5^{*}5$  unique action text-question pairs for each image, covering all actions and reasoning types in a balanced manner as shown in Figure 5a (referred as Original partition). However, it leads to a skewed distribution of answers as observed from 5b. Therefore, we curate a version of the dataset (referred as Balanced partition) consisting of 67.5k samples where all answer choices are equally-likely as well.
Additionally, we create two small challenge test sets (1500 image-action text-question samples each)- 2HopActionText  $(2\mathrm{HopT}_A)$  and 2HopQuestion  $(2\mathrm{HopQ}_H)$  to test generalization capability of the trained models. In  $2\mathrm{HopT}_A$  , we create action text which requires model to understand two different actions being taken on the scene. For example, Add a small blue metal cylinder to the right of large yellow cube and remove the large cylinder from the scene.' and 'Move the purple object on top of small red cube then change its color to cyan'. In
(a) Distribution based on Action Text types and Question types
(b) Distribution of Answer types
Figure 5: Visualization of distributions for actions, questions and answers in Original_Train partition of CLEVR_HYP.
$2\mathrm{HopQ}_H$ , we create questions which require model to understand logical combinations of questions using 'and', 'or' and 'not'. For example, 'How many objects are either red or cylinder?' and 'Are there any rubber cubes that are not green?'.
In Table 2, we provide size of the various partitions and measure the diversity of the dataset in various aspects. For images, we calculate average number of objects present in the scene from the length of scene graph. For balanced partition, the number of images are much less compared to original, but more average number of objects per image. This is most likely due to the need to accommodate integers 4-9 more frequently as ground-truth answers. For textual components, we show average lengths (number of tokens separated by whitespaces) and count unique utterances as a measure of diversity. The original partition of the resulting dataset has  $80\%$  and  $83\%$  unique action text and questions respectively. For balanced partition, length and unique utterances for action text are nearly same as the original partition but for questions, it decreases. Questions in the original partition have been observed to enforce more strict and specific object references (such as small red metal cubes) compared to balanced partition (small cubes, red metal objects etc.), reducing the average length
and uniqueness. It is intuitive for 2Hop partitions to have higher average length and uniqueness for  $T_{A}$  and  $Q_{H}$  respectively. This shows that despite having created this dataset from templates and rendered images with a limited set of attributes, it is still fairly challenging.
4 Models that we experiment with
Models trying to tackle CLEVR_HYP dataset have to address four key challenges;
(i) understand hypothetical actions and questions in complex natural language,
(ii) correctly disambiguate the objects of interest and obtain the structured representation (i.e. scene graphs or functional programs) of various modalities if required by the solver,
(iii) understand the dynamics of the world based on the various actions performed over it,
(iv) perform various kind of reasoning to answer the question.
4.1 Random
The QA task in CLEVR_HYP dataset can be considered as a 27-class classification problem. Each answer choice is likely to be picked with a probability of  $1/27$ . Therefore, the performance of the random baseline is  $3.7\%$ .
Split	#I	Avg. #Obj	#TA	Unique #TA	Avg. TA Len.	#QH	Unique #QH	Avg. QH Len.
Original_Train	5k	6.4	25k	20.7k	12.8	125k	103.7k	22.6
Original_Val	1k	6.7	5k	3.8k	12.8	25k	20.9k	23.1
Original_Test	1k	6.4	5k	3.6k	12.6	25k	20.7k	22.8
Balanced_Train	5k	7.6	25k	21.1k	12.8	67.5k	58.2k	20.3
Balanced_Val	1k	7.6	5k	3.9k	12.7	13.5k	11.5k	20.7
Balanced_Test	1k	7.5	5k	3.7k	12.6	13.5k	11.4k	20.4
2HopTA_Test	1k	6.4	3k	2.6k	18.6	15k	12.5k	22.8
2HopQH_Test	1k	6.4	3k	2.2k	12.6	15k	13.7k	29.3
Table 2: CLEVR_HYP dataset splits and statistics (# represents number of, k represents thousand).
4.2 Human Performance
We performed human evaluation with respect to 500 samples from the CLEVR_HYP dataset. Accuracy of human evaluations on original test,  $2\mathrm{Hop}A_{T}$  and  $2\mathrm{Hop}Q_{H}$  are  $98.4\%$ $96.2\%$  and  $96.6\%$  respectively.
4.3 Transformer Architectures
Pre-trained transformer-based architectures have been observed (Li et al., 2020) to capture a rich hierarchy of language-structures (text-only models) and effectively map entities/words with corresponding image regions (vision-language models). We experiment with various transformer-based models to understand their capability to understand the effects of actions on a visual domain.