Venue: NAACL 2021

# CLEVR_HYP: A Challenge Dataset and Baselines for

Visual Question Answering with Hypothetical Actions over Images

 Shailaja Keyur Sampat; Akshay Kumar, Yezhou Yang and Chitta Baral

Arizona State Universiy, USA

{ssampa17,akuma216,yz.yang,chitta}@asu.edu

Corresponding author

###### Abstract

Most existing research on visual question answering (VQA) is limited to information explicitly present in an image or a video. In this paper, we take visual understanding to a higher level where systems are challenged to answer questions that involve mentally simulating the hypothetical consequences of performing specific actions in a given scenario. Towards that end, we formulate a vision-language question answering task based on the CLEVR Johnson et al. (2017) dataset. Wethen modify the best existing VQA methods and propose baseline solvers for this task. Finally, we motivate the development of better vision-language models by providing insights about the capability of diverse architectures to perform joint reasoning over image-text modality1.

Footnote 1: Dataset setup scripts and code for baselines are made available at [https://github.com/shailajai83/clevr_hyp](https://github.com/shailajai83/clevr_hyp). For additional details about the dataset creation process, refer supplementary material.

## 1 Introduction

In 2014, Michael Jordan, in an interview Gomes (2014) said that "_Deep learning is good at certain problems like image classification and identifying objects in the scene, but it struggles to talk about how those objects relate to each other, or how a person/robot would interact with those objects. For example, humans can deal with inferences about the scene: what if I sit down on that?, what if I put something on top of something? etc. There exists a range of problems that are far beyond the capability of today's machines._"

While this interview was six years ago, and since then there has been a lot of progress in deep learning and its applications to visual understanding. Additionally, a large body of visual question answering (VQA) datasets Antol et al. (2015); Ren et al. (2015); Hudson and Manning (2019) have been compiled and many models have been developed over them, but the above mentioned "inferences about the scene" issue stated by Jordan remains largely unaddressed.

In most existing VQA datasets, scene understanding is holistic and questions are centered around information explicitly present in the image (i.e. objects, attributes and actions). As a result, advanced object detection and scene graph techniques have been quite successful in achieving good performance over these datasets. However, provided an image, humans can speculate a wide range of implicit information. For example, the purpose of various objects in a scene, speculation about events that might have happened before, consider numerous imaginary situations and predicting possible future outcomes, intentions of a subject to perform particular actions, and many more.

Among the above, an ability to imagine taking specific actions and simulating probable results without actually acting or experiencing is an important aspect of human cognition (Figure 1 gives an example of this). Thus, we believe that having autonomous systems equipped with a similar capability will further advance AI research. This is particu

larly useful for robots performing on-demand tasks in safety-critical situations or navigating through dynamic environments, where they imagine possible outcomes for various situations without executing instructions directly.

Motivated by the above, we propose a challenge that attempts to bridge the gap between state-of-the-art AI and human-level cognition. The main contributions of this paper2 are as follows;

Footnote 2: Our work focuses on the capability of neural models to reason about the effects of actions given a visual-linguistic context and not on models that deal with intuitive physics.

* We formalize a novel question answering task with respect to a hypothetical state of the world (in a visual form) when some action (described in a textual form) is performed.
* We create a large-scale dataset for this task, and refer it as CLEVR_HYP i.e. VQA with hypothetical actions performed over images in CLEVR Johnson et al. (2017) style.
* We first evaluate the direct extensions of top VQA and NLQA (Natural language QA) solvers on this dataset. Then, we propose new baselines to solve CLEVR_HYP and report their results.
* Through analysis and ablations, we provide insights about the capability of diverse architectures to perform joint reasoning over image-text modality.

## 2 Related Work

In this section we situate and compare our work with related areas such as implicit text generation/retrieval for a visual, visual question answering (VQA) over synthetic images, question answering (QA) involving hypothetical reasoning, and language-based manipulation in visual domains closest to CLEVR_HYP.

**Implicit Text Generation for a Visual:**VisualComet Park et al. (2020) and Video2Commonsense Fang et al. (2020) have made initial attempts to derive implicit information about images/videos contrary to traditional factual descriptions which leverage only visual attributes. VisualComet aims to generate commonsense inferences about events that could have happened before, events that can happen after and people's intents at present for each subject in a given image. They use a vision-language transformer that takes a sequence of inputs (image, event, place, inference) and train a model to predict inference in a language-model style. Video2Commonsense focuses on generating video descriptions that can incorporate commonsense facts related to intentions, effects, and implicit attributes about actions being performed by a subject. They extract top-ranked commonsense texts from the Atomic dataset and modify training objective to incorporate this information.

While both involve a visual-textual component and actions, their key focus is about generating plausible events and commonsense respectively. Whereas, our work is related to performing certain actions and reasoning about its effect on the overall visual scene.

**Language-based Manipulation in Visual Domain:** Learning a mapping from natural language instructions to a sequences of actions to be performed in a visual environment is a common task in robotics Kanu et al. (2020); Gaddy and Klein (2019); Shridhar et al. (2020). Another relevant task is vision-and-language navigation Anderson et al. (2018); Chen et al. (2019); Nguyen et al. (2019), where an agent navigates in a visual environment to find goal location by following natural language instructions. Both above works include visuals, natural language instructions and a set of actions that can be performed to achieve desired goals. In this way, it is similar to our CLEVR_HYP, but in our case, models require reasoning about the effect of actions performed rather than determining which action to perform. Also, we frame this in a QA style evaluation rather than producing instructions for low-level controls.

Manipulation of natural images with language is an emerging research direction in computer vision. Teney et al. (2020) proposed a method for generating counterfactual of VQA samples using image in-painting and masking. Also, there are works Dong et al. (2017); Nam et al. (2018); Reed et al. (2016) which use Generative Adversarial Networks (GANs) Goodfellow et al. (2014) for language conditioned image generation and manipulation. However, both the above tasks are more focused at object and attribute level manipulation rather than at action level.

**VQA over Synthetic Images:** While natural images-based VQA datasets reflect challenges one can encounter in real-life situations, the requirement 

ment of costlier human annotations and vulnerability to biases are two major drawbacks. Contrary to them, synthetic datasets allow controlled data generation at scale while being flexible to test specific reasoning skills.

For the above reasons, following benchmark VQA datasets have incorporated synthetic images; COG (Yang et al., 2018) and Shapes Andreas et al. (2016) contain images with rendered 2D shapes; SHRDLU (Winograd, 1971), CLEVR (Johnson et al., 2017a), and CLEVR-dialog Kottur et al. (2019) have rendered scenes with 3D objects; DVQA (Kafle et al., 2018) and FigureQA (Kahou et al., 2017) have synthetically generated charts (bar chart, pie chart, dot-line etc.); VQA-abstract (Antol et al., 2015) and IQA (Gordon et al., 2018) involves question-answering over synthetically rendered clipart-style scenes and interactive environments respectively. Our proposed dataset CLEVR_HYP uses CLEVR (Johnson et al., 2017a) style rendered scenes with 3D objects as a visual component. It is distinct from all other synthetic VQA datasets for two key reasons; first, integration of action domain in synthetic VQA and second, the requirement of mental simulation in order to answer the question.

**QA involving Hypothetical Reasoning:** In the language domain, WIQA (Tandon et al., 2019) dataset tests the model's ability to do what-if reasoning over procedural text as a 3-way classification (the influence between pair of events as positive, negative or no-effect). In vision-language domains, a portion of TQA (Kembhavi et al., 2017) and VCR (Zellers et al., 2019) are relevant. Questions in TQA and VCR involve hypothetical scenarios about multi-modal science contexts and movie scenes respectively. However, none of the above two datasets' key focus is on the model's capability to imagine changes performed over the image.

As shown in Figure 3, the setting of TIWIQ (a benchmark dataset for "physical intelligence") (Wagner et al., 2018) has some similarity with ours. It has synthetically rendered table-top scenes, four types of actions (push, rotate, remove and drop) being performed on an object and what-if questions.

To our best knowledge, TIWIQ dataset is not publicly available. Based on our understanding from their manuscript, we observe following important distinction with this work. Our questions focus on the impact of actions on the whole image, while in TIWIQ questions are about impact of actions on a specific object in the image. Moreover, we frame CLEVR_HYP as a classification task, contrary to TIWIQ which is a generative task. Our CLEVR_HYP dataset has 175k automatically generated image-action text-question samples which is much larger compared to TIWIQ which has only 1020 samples and manually crafted ground-truths.

## 3 Clevr_Hyp Task and Dataset

Figure 2 gives a glimpse of CLEVR_HYP task. We opt for synthetic dataset creation as it allows automated and controlled data generation at scale with minimal biases. More details are described below.

### Inputs: Image(I), Action Text (\(\mathbf{T}_{A}\)) and Hypothetical Question (\(\mathbf{Q}_{H}\))

#### 3.1. Image(I):

It is a given visual for our task. Each image in the dataset contains 4-10 randomly selected 3D objects rendered using Blender (Blender Online Community, 2019) in CLEVR

Figure 3: Example from TIWIQ (Wagner et al., 2018).

Figure 2: Three examples from CLEVR_HYP dataset: given image (I), action text (\(\mathbf{T}_{A}\)), question about hypothetical scenario (\(\mathbf{Q}_{H}\)) and corresponding answer (A). The task is to understand possible perturbations in I with respect to various action(s) performed as described in \(\mathbf{T}_{A}\). Questions test various reasoning capabilities of a model with respect to the results of those action(s).

 

(Johnson et al., 2017) style. Objects have 4 attributes listed in the Table 1. Additionally, these objects can be referred using 5 relative spatial relations (left, right, in front, behind and on). We provide scene graphs3 containing all ground-truth information about a scene, that can be considered as a visual oracle for a given image.

Footnote 3: Scene graphs and Functional Programs (for action text and question) are not provided at the test-time.

**2. Action Text (T\({}_{A}\)):** It is a natural language text describing various actions performed over the current scene. The action can be one of four:

* **Add** new object(s) to the scene
* **Remove** object(s) from the scene
* **Change** attributes of the object(s)
* **Move** object(s) within scene (might be in plane i.e. left/right/front/back or out of plane i.e. move one object on top of another object4) Footnote 4: For simplicity, we assume that any object can be put on another object regardless of its size, material or shape.

To generate action text, we start with manually written templates involving the aforementioned actions. For example, action involving change in the attribute of object(s) to a given value, we have a template of the following kind; 'Change the <A>of <Z><C><M><S>to <V>'. Where <A>, <Z>, <C>,<M>,<S>,<V> are placeholders for the attribute, size, color, material, shape and a value of attribute respectively. Each action text in the CLEVR_HYP is associated with a functional program which if executed on an image's scene graph, yields the new scene graph that simulates the effects of actions.

Functional programs for action texts3 are built from the basic functions that correspond to elementary action operations (right part of Figure 4a). For the above mentioned 'change' attribute action template, the equivalent functional program can be written as; 'change_attr(<A>,filter_size(<Z>,filter_color(<C>, filter_material(<MD>filter_shape(<S>,scene())))),<V>)'. It essentially means, first filter out the objects with desired attributes and then update the value of their current attribute A to value V.

Footnote 3: Scene graphs and Functional Programs (for action text and question) are not provided at the test-time.

**3. Question about Hypothetical Situation (Q\({}_{H}\)):** It is a natural language query that tests various visual reasoning abilities after simulating the effects of actions described in T\({}_{A}\). There are 5 possible reasoning types similar to CLEVR;

* **Counting** objects fulfilling the condition
* **Verify existence** of certain objects
* **Query attribute** of a particular object
* **Compare attributes** of two objects
* **Integer comparison** of two object sets (same, larger or smaller)

Similar to action texts, we have templates and corresponding programs for questions. Functional programs for questions3 are executed on the image's updated scene graph (after incorporating effects of the action text) and yields the ground-truth answer to the question. Functional programs for questions are made of primitive functions shown in left part of the Figure 4a).

Footnote 3: For simplicity, we assume that any object can be put on another object regardless of its size, material or shape.

**Paraphrasing:** In order to create a challenging dataset from linguistic point of view and to prevent models from overfitting on templated representations, we leverage noun synonyms, object name paraphrasing and sentence-level paraphrasing. For noun synonyms, we use a pre-defined dictionary (such as cubeblock, sphereball and so on). We programmatically generate all possibilities to refer to an object in the image (i.e. object name paraphrasing) and randomly sample one among them. For sentence level paraphrasing, we use Text-To-Text Transfer Transformer (T5) (Raffel et al., 2020) fine-tuned over positive samples from Quora Question Pairs (QQP) dataset Iyer et al. (2017) for question paraphrasing. We use Fairseq Ott et al. (2019) for action text paraphrasing which uses round-trip translation and mixture of experts Shen et al. (2019).

Note that we keep the action text and question as separate inputs for the purpose of simplicity and keeping our focus on building solvers that can do mental simulation. One can create a simple template like "<Q\({}_{H}\)> if <proper-noun/pronouns> <T\({}_{A}\)>?" or "If <proper-noun/pronouns> <T\({}_{A}\)>,

\begin{table}
\begin{tabular}{p{86.7pt} p{284.5pt}} \hline
**Attr.** & **Possible values in CLEVR_HYP** \\ \hline
**Color** & gray, blue, brown, yellow, red, green, purple, cyan cylinder, sphere or cube \\
**Shape Size** & small or big \\
**Material** & metal (shining) or rubber (mate) \\ \hline \end{tabular}
\end{table}
Table 1: Object attributes in CLEVR_HYP scenes.

 

\(<\)Q\({}_{H}\)\(>\)?" if they wish to process action and question as a single text input. For example, "How many things are the same size as the cyan cylinder if I add a large brown rubber cube behind the blue object." or "If I add a large brown rubber cube behind the blue object, how many things are the same size as the cyan cylinder?". However, having them together adds further complexity on the solver side as it first has to figure out what actions are performed and what is the question.

By providing ground-truth object information (as a visual oracle) and machine-readable form of questions & action texts (oracle for linguistic components). This information can be used to develop models which can process semi-structured representations of image/text or for the explainability purposes (to precisely know which component of the model is failing).

**Output: Answer (A)** to the Question (Q\({}_{H}\)), which can be considered as a 27-way classification over attributes (8 colors + 3 shapes + 2 sizes + 2 material), numeric (0-9) and boolean (yes/no).

**Dataset Partitions and Statistics:** We create CLEVR_HYP dataset containing 175k image-action text-question samples using the process mentioned in Figure 4b. For each image, we generate 5 kinds of action texts (one for each add, remove, move in-plane and move out-of-plane and change attribute). For each action text type, we generate 5 questions (one for each count, exist, compare integer, query attribute and compare attribute). Hence, we get 5*5 unique action text-question pairs for each image, covering all actions and reasoning types in a balanced manner as shown in Figure 5a (referred as Original partition). However, it leads to a skewed distribution of answers as observed from 5b. Therefore, we curate a version of the dataset (referred as Balanced partition) consisting of 67.5k samples where all answer choices are equally-likely as well.

Additionally, we create two small challenge test sets (1500 image-action text-question samples each)- 2HopActionText (2HopT\({}_{A}\)) and 2HopQuestion (2HopQ\({}_{H}\)) to test generalization capability of the trained models. In 2HopT\({}_{A}\), we create action text which requires model to understand two different actions being taken on the scene. For example, 'Add a small blue metal cylinder to the right of large yellow cube and remove the large cylinder from the scene.' and 'Move the purple object on top of small red cube then change its color to cyan.'. In

Figure 4: CLEVR_HYP dataset creation process with example and function catalog used for ground-truth answer generation. (for more details, see Appendix A.4)



2HopQ\({}_{H}\), we create questions which require model to understand logical combinations of questions using 'and', 'or' and 'not'. For example, 'How many objects are either red or cylinder?' and 'Are there any rubber cubes that are not green?".

In Table 2, we provide size of the various partitions and measure the diversity of the dataset in various aspects. For images, we calculate average number of objects present in the scene from the length of scene graph. For balanced partition, the number of images are much less compared to original, but more average number of objects per image. This is most likely due to the need to accommodate integers 4-9 more frequently as ground-truth answers. For textual components, we show average lengths (number of tokens separated by whitespaces) and count unique utterances as a measure of diversity. The original partition of the resulting dataset has 80% and 83% unique action text and questions respectively. For balanced partition, length and unique utterances for action text are nearly same as the original partition but for questions, it decreases. Questions in the original partition have been observed to enforce more strict and specific object references (such as small red metal cubes) compared to balanced partition (small cubes, red metal objects etc.), reducing the average length and uniqueness. It is intuitive for 2Hop partitions to have higher average length and uniqueness for \(T_{A}\) and \(Q_{H}\) respectively. This shows that despite having created this dataset from templates and rendered images with a limited set of attributes, it is still fairly challenging.

## 4 Models that we experiment with

Models trying to tackle CLEVR_HYP dataset have to address four key challenges;

1. understand hypothetical actions and questions in complex natural language,
2. correctly disambiguate the objects of interest and obtain the structured representation (i.e. scene graphs or functional programs) of various modalities if required by the solver,
3. understand the dynamics of the world based on the various actions performed over it,
4. perform various kind of reasoning to answer the question.

### Random

The QA task in CLEVR_HYP dataset can be considered as a 27-class classification problem. Each answer choice is likely to be picked with a probability of 1/27. Therefore, the performance of the random baseline is 3.7%.

Figure 5: Visualization of distributions for actions, questions and answers in Original_Train partition of CLEVR_HYP.

 



### Human Performance

We performed human evaluation with respect to 500 samples from the CLEVR_HYP dataset. Accuracy of human evaluations on original test, 2Hop\(A_{T}\) and 2Hop\(Q_{H}\) are 98.4%, 96.2% and 96.6% respectively.

### Transformer Architectures

Pre-trained transformer-based architectures have been observed Li et al. (2020) to capture a rich hierarchy of language-structures (text-only models) and effectively map entities/words with corresponding image regions (vision-language models). We experiment with various transformer-based models to understand their capability to understand the effects of actions on a visual domain.

Baseline 1- Machine Comprehension using RoBERTa:To evaluate the hypothetical VQA task through the text-only model, we convert images into the templated text using scene graphs. The templated text contains two kind of sentences; one describing properties of the objects i.e. "There is a <Z> <C> <M> <S>", the other one describing the relative spatial location i.e. "The <Z> <C> <M> <S> is <R> the <Z1> <C1> <M1> <S1>". For example, "There is a small green metal cube." and "The large yellow rubber sphere is to the left of the small green metal cube". Then we concatenate templated text with the action text to create a reading comprehension passage. We use state-of-the-art machine comprehension baseline RoBERTa Liu et al. (2019) finetuned on the RACE dataset Lai et al. (2017)5. Finally, we predict an answer to the question using this reading comprehension passage.

Footnote 5: architecture=roberta large, epochs=5, learning rate=\(1\mathrm{e}{-05}\), batch size=2, update frequency=2, dropout=0.1, optimizer=adam with eps=\(1\mathrm{e}{-06}\).

Baseline 2- Visual Question Answering using LXMertenProposed by Tan and Bansal (2019), LXMERT is one of the best transformer based pretrainable visual-linguistic representations which supports VQA as a downstream task. Typical VQA systems take an image and a language input. Therefore, to evaluate CLEVR_HYP in VQA style, we concatenate action text and question to form a single text input. Since LXMERT is pre-trained on the natural images, we finetune it over CLEVR_HYP dataset6 and then use it to predict answer.

Footnote 6: epochs=4, learning rate=\(5\mathrm{e}{-05}\), batch size=8

### Systematically incorporating effects of actions into neural models

Baseline 3- Text-editing Image Baseline:In this method, we break-down the QA task with mental simulation in two parts; first, learn to generate an updated image (such that it has incorporated the effects of actions) and then perform visual question answering with respect to the updated image. We use the idea from Text Image Residual Gating proposed in Vo et al. (2019) to implement the first part. However there are two important distinctions; Their focus is on the retrieval from the given database. We modify their objective and develop text-adaptive encoder-decoder with residual connections to generate new image. Also, editing instructions in their CSS dataset Vo et al. (2019) were quite simple. For example, 'add red cube' and 'remove yellow sphere'. In this case, one can add the red cube anywhere in the scene. We modify their architecture to precisely place objects to their

\begin{table}
\begin{tabular}{l l c c c c c c} \hline \hline
**Split** & **\#I** & **Avg.****+Obj** & **\#T\({}_{A}\)** & **Unique****+T\({}_{A}\)****+Len.** & **\#Q\({}_{H}\)** & **Unique****+Q\({}_{H}\)****+Len.** \\ \hline \hline
**Original\_Train** & 5k & 6.4 & 25k & 20.7k & 12.8 & 125k & 103.7k & 22.6 \\
**Original\_Val** & \(1\mathrm{k}\) & 6.7 & 5k & 3.8k & 12.8 & 25k & 20.9k & 23.1 \\
**Original\_Test** & \(1\mathrm{k}\) & 6.4 & 5k & 3.6k & 12.6 & 25k & 20.7k & 22.8 \\ \hline
**Balanced\_Train** & 5k & 7.6 & 25k & 21.1k & 12.8 & 67.5k & 58.2k & 20.3 \\
**Balanced\_Val** & \(1\mathrm{k}\) & 7.6 & 5k & 3.9k & 12.7 & 13.5k & 11.5k & 20.7 \\
**Balanced\_Test** & \(1\mathrm{k}\) & 7.5 & 5k & 3.7k & 12.6 & 13.5k & 11.4k & 20.4 \\ \hline
**2Hop\(T_{A}\)****+Test** & \(1\mathrm{k}\) & 6.4 & 3k & 2.6k & 18.6 & 15k & 12.5k & 22.8 \\
**2Hop\(Q_{H}\)****+Test** & \(1\mathrm{k}\) & 6.4 & 3k & 2.2k & 12.6 & 15k & 13.7k & 29.3 \\ \hline \hline \end{tabular}
\end{table}
Table 2: CLEVR_HYP dataset splits and statistics (# represents number of, k represents thousand).



relative spatial references (on left/right/front/ behind). Once we get the updated image, we feed it to the LXMERT Tan and Bansal (2019) finetuned over the CLEVR Johnson et al. (2017) dataset along with the question and predict the answer.

Baseline 4- Scene Graph Update Model:Instead of directly manipulating images, in this method, we leverage image scene graphs to convert image-editing problem into graph-editing problem, conditioned on the action text. This is an emerging research direction to deal with changes in the visual modality over time or with new sources of information, as observed from recent parallel works (chang Chen et al., 2020; He et al., 2020).

We first use Mask R-CNN He et al. (2017) to get the segmentation mask of the objects and predict attributes (color, material, size, and shape) with an acceptance threshold of 0.9. Segmentation mask of each object along with original image is then passed through ResNet-34 He et al. (2016) to extract precise 3D coordinates of the object. We get the structured scene graph for the image. Then we use seq2seq with attention model originally proposed in Johnson et al. (2017) to generate functional programs (FP) for action text and question. The execution engine executes programs on scene graph, implemented as a neural module network Andreas et al. (2016) to update the scene representation and answer questions.

We learn to update scene graphs according to functional program for the action text using reinforcement learning8. The reward function is associated with our ground-truth program executor and generates reward if prediction exactly matches with ground-truth execution. Once we get the updated scene representation, we use neural-symbolic model9 proposed by Yi et al. (2018) to obtain the final answer. It is notable that Yi et al. (2018) achieved near-perfect performance on the CLEVR QA task in addition to being fully explainable.

Footnote 8: supervised pretraining learning rate=\(7e{-}04\), num iterations=20k, batch size=32 and then finetuning \(1e{-}05\), at most 2M iterations with early stopping, batch size=32

## 5 Baseline Results

In this section, we benchmark models described above on the CLEVR_HYP. The dataset is formulated as a classification task with exactly one correct answer, so we use standard accuracy as evaluation metric. We then analyze their performance according to question and action types.

Quantitative results from above experiments can be visualized in top part of the Table 3. Among the methods described above, the scene graph update model has the best overall performance 70.5% on original test data. Text-editing model is best over balanced set, but observed to have the poor generalization capability when two actions or reasoning capabilities have to be performed. CLEVR_HYP requires models to reason about effect of hypothetical actions taken over images. LXMERT is not directly trained for this objective therefore, it struggles to do well on this task. The reason behind the poor performance of text-only baseline is due to its limitation to incorporate detailed spatial locations

Figure 6: Graphical visualization of baseline models over CLEVR_HYP described above.

 

into the templates that we use to convert image into a machine comprehension passage.

Two of our models (scene graph update and text-editing image) are transparent to visualize intermediate changes in the scene after performing actions. We analyse their ability to understand actions and make appropriate changes as shown in below part of Table 3. For the scene graph method, we compare the ground-truth functional program with the generated program and measure their exact-match accuracy. For the text-editing image method, we generate scene graphs for both images (original image and image after text-editing) and compare them. For attributes, we do exact-match, whereas for location information we consider matching only on the basis of relative spatial location.

Both scene graph and text-editing models do quite well on 'remove' and 'change' actions whereas struggle when new objects are added or existing objects are moved around. The observation is consistent when multiple actions are combined. Therefore, actions remove+change can be performed with maximum accuracy whereas other combinations of actions accomplish relatively lower performance. It leads to the conclusion that understanding the effect of different actions are of varied complexity. Most models demonstrate better performance over counting, existence and attribute query type of questions than comparison questions. The scene graph update and text-editing methods show a performance drop of 6.1% and 9.1% respectively when multiple actions are performed on the scene. However, there is less of a performance gap for models on 2HopQ\({}_{H}\) compared to the test set, suggesting that models are able to better generalize with respect to multiple reasoning skills than complex actions.

## 6 Conclusion

We introduce CLEVR_HYP, a dataset to evaluate the ability of VQA systems after hypothetical actions are performed over the given image. We create this dataset by extending the data generation framework of CLEVR Johnson et al. (2017a) that uses synthetically rendered images and templates for reasoning questions. Our dataset is challenging because rather than asking to reason about objects already present in the image, it asks about what would happen in an alternative world where changes have occurred. We provide ground-truth representations for images, hypothetical actions and questions to facilitate the development of models that systematically learn to reason about underlying process. We create several baseline models to benchmark CLEVR_HYP and report their results. Our analysis shows that the models are able to perform reasonably well (70.5%) on the limited number of actions and reasoning types, but struggle with complex scenarios. While neural models have achieved almost perfect performance on CLEVR and considering human performance as upperbound (98%), there is a lot of room for improvement on CLEVR_HYP. Our future work would include relaxing constraints by allowing a larger variety of actions, attributes and reasoning types. By extending this approach further for natural images, we aim to contribute in the development of better vision+language models.

\begin{table}
\begin{tabular}{l c c|c c c|c c c|c c c c} \hline \multicolumn{14}{c|}{**Overall Baseline Performance for Various Test Sets of CLEVR_HYP**} \\ \hline  & \multicolumn{3}{c|}{_Original Test_} & \multicolumn{3}{c|}{_Balanced Test_} & \multicolumn{3}{c|}{_2HopTA Test_} & \multicolumn{3}{c}{_2HopOpt Test_} \\ BL1 & BL2 & BL3 & BL4 & BL1 & BL2 & BL3 & BL4 & BL1 & BL2 & BL3 & BL4 \\
57.2 & 63.9 & 64.7 & **70.5** & 55.3 & 65.2 & **69.5** & 68.6 & 53.3 & 49.2 & 55.6 & **64.4** & 55.2 & 52.9 & 58.7 & **66.5** \\ \hline \end{tabular}
\end{table}
Table 3: Baseline performance over CLEVR_HYP (BLx represents one of the four Baselines described above).

 



## Acknowledgements

We are thankful to the anonymous reviewers for the constructive feedback. This work is partially supported by the grants NSF 1816039, DARPA W911NF2020006 and ONR N00014-20-1-2332.

## References

* Anderson et al. (2018) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_, pages 3674-3683. IEEE Computer Society.
* Andreas et al. (2016) Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_, pages 39-48. IEEE Computer Society.
* Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiaesen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_, pages 2425-2433. IEEE Computer Society.
* a 3D modelling and rendering package_. Blender Foundation.
* Chen et al. (2020) Li chang Chen, Guosheng Lin, S. Wang, and Qingyao Wu. 2020. Graph edit distance reward: Learning to edit scene graph. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
* Chen et al. (2019) Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 12538-12547.
* Dong et al. (2017) Hao Dong, Simiao Yu, Chao Wu, and Yike Guo. 2017. Semantic image synthesis via adversarial learning. In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_, pages 5707-5715. IEEE Computer Society.
* Fang et al. (2020) Zhiyuan Fang, Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020. Video2Commonsense: Generating commonsense descriptions to enrich video captioning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 840-860. Association for Computational Linguistics.
* Gaddy and Klein (2019) David Gaddy and Dan Klein. 2019. Pre-learning environment representations for data-efficient neural instruction following. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1946-1956. Association for Computational Linguistics.
* Gomes (2014) Lee Gomes. 2014. Machine-learning maestro michael jordan on the delusions of big data and other huge engineering efforts.
* Goodfellow et al. (2014) Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks. _arXiv preprint arXiv:1406.2661_, 4(5):6.
* Gordon et al. (2018) Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. 2018. IQA: visual question answering in interactive environments. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_, pages 4089-4098. IEEE Computer Society.
* He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross B. Girshick. 2017. Mask R-CNN. In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_, pages 2980-2988. IEEE Computer Society.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_, pages 770-778. IEEE Computer Society.
* He et al. (2020) Xuanli He, Quan Hung Tran, Gholamreza Haffari, Walter Chang, Zhe Lin, Trung Bui, Franck Demonourt, and Nhan Dam. 2020. Scene graph modification based on natural language commands. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 972-990. Association for Computational Linguistics.
* Hudson and Manning (2019) Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 6700-6709.
* Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017. First quora dataset release: Question pairs. _data. quara. com._
* Johnson et al. (2017a) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017a. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_, pages 1988-1997. IEEE Computer Society.



Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017b. Inferring and executing programs for visual reasoning. In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_, pages 3008-3017. IEEE Computer Society.
* Kafle et al. (2018) Kwashal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. 2018. DVQA: understanding data visualizations via question answering. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_, pages 5648-5656. IEEE Computer Society.
* Kahou et al. (2017) Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. _arXiv preprint arXiv:1710.07300_.
* Kanu et al. (2020) John Kanu, Eadom Dessalene, Xiaomin Lin, Cornelia Fermuller, and Yiannis Aloimonos. 2020. Following instructions by imagining and reaching visual goals. _arXiv preprint arXiv:2001.09373_.
* Kembhavi et al. (2017) Aniruddha Kembhavi, Min Joon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_, pages 5376-5384. IEEE Computer Society.
* Kottur et al. (2019) Satwik Kottur, Jose M. F. Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2019. CLEVR-dialog: A diagnostic dataset for multi-round reasoning in visual dialog. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 582-595. Association for Computational Linguistics.
* Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAdging comprehension dataset from examinations. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 785-794. Association for Computational Linguistics.
* Li et al. (2020) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020. What does BERT with vision look at? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5265-5275, Online. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Namam Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Nam et al. (2018) Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. 2018. Text-adaptive generative adversarial networks: Manipulating images with natural language. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada_, pages 42-51.
* Nguyen et al. (2019) Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. 2019. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_, pages 12527-12537. Computer Vision Foundation / IEEE.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 48-53. Association for Computational Linguistics.
* Park et al. (2020) J. Park, Chandra Bhagavatula, R. Mottaghi, A. Farhadi, and Yejin Choi. 2020. Visualcomet: Reasoning about the dynamic context of a still image. In _Proceedings of the European Conference on Computer Vision (ECCV)_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqj Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Reed et al. (2016) Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. Generative adversarial text to image synthesis. In _Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016_, volume 48 of _JMLR Workshop and Conference Proceedings_, pages 1060-1069. JMLR.org.
* Ren et al. (2015) Mengye Ren, Ryan Kiros, and Richard S. Zemel. 2015. Exploring models and data for image question answering. In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_, pages 2953-2961.
* Shen et al. (2019) Tianxiao Shen, Myle Ott, Michael Auli, and Marc'Aurelio Ronzato. 2019. Mixture models for diverse machine translation: Tricks of the trade. In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_, volume 97 of _Proceedings of Machine Learning Research_, pages 5719-5728. PMLR 

Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_, pages 10737-10746. IEEE.
* Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5100-5111. Association for Computational Linguistics.
* Tandon et al. (2019) Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019. WIQA: A dataset for "what if..." reasoning over procedural text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6076-6085. Association for Computational Linguistics.
* Teney et al. (2020) Damien Teney, Ehsan Abbasnedjad, and Anton van den Hengel. 2020. Answering visual what-if questions: From actions to predicted scene descriptions. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pages 580-599.
* an empirical odyssey. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_, pages 6439-6448. Computer Vision Foundation / IEEE.
* Wagner et al. (2018) Misha Wagner, Hector Basevi, Rakshith Shetty, Wenbin Li, Mateusz Malinowski, Mario Fritz, and Ales Leonardis. 2018. Answering visual what-if questions: From actions to predicted scene descriptions. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pages 0-0.
* Winograd (1971) Terry Winograd. 1971. Procedures as a representation for data in a computer program for understanding natural language. Technical report, Massachusetts inst of tech cambridge project mac.
* Yang et al. (2018) Guangyu Robert Yang, Igor Ganichev, Xiao-Jing Wang, Jonathon Shlens, and David Sussillo. 2018. A dataset and architecture for visual reasoning with a working memory. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pages 714-731.
* Yi et al. (2018) Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural-symbolic VQA: disentangling reasoning from vision and language understanding. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada_, pages 1039-1050.
* Zellers et al. (2019) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_, pages 6720-6731. Computer Vision Foundation / IEEE.

 

Appendix

### Relation of CLEVR_HYP dataset with real-world situations

Teaching methodologies leverage our ability to mentally simulate scenarios along with the metaphors to aid understanding about new concepts. In other words, to explain unfamiliar concepts, we often reference familiar concepts and provide additional clues to establish mapping between them. This way, a person can create a mental simulation about unfamiliar concept and aid basic understanding about it.

For example, we want to explain a person how a 'zebra' looks like, who has previously seen a 'horse', we can do so using example in Figure 6(a). This naturally follows for more complex concepts. Let say, one wants to describe the structure of an atom to someone, he might use the analogy of a planetary system, where the components (planets \(\sim\) electrons) circulate around a central entity (sun \(\sim\) nucleus). One more such example is provided in Figure 6(b).

For humans, learning new concepts and performing mental simulations is omnipresent in day-to-day life. Therefore, CLEVR_HYP dataset is very much grounded in the real world. Models developed on this dataset can serve a broad range of applications, particularly the ones where possible outcomes have to be predicted without actually executing the actions. For example, robots performing on-demand tasks in safety-critical situations or self-driving vehicles. In addition, these models can be an important component for other vision and language tasks such as automatic expansion of existing knowledge bases, zero shot learning and spatio-temporal visual reasoning.

### Rejecting Bad Samples in CLEVR_HYP

Automated methods of question generation sometimes create invalid items, classified as 'ill-posed' or 'degenerate' by CLEVR Johnson et al. (2017) dataset generation framework. They consider question "What color is the cube to the right of the sphere?" as ill-posed if there were many cubes right of the sphere, or degenerate if there is only one cube in the scene and reference to the sphere becomes unnecessary. In addition to this, we take one more step of quality control in order to prevent ordinary VQA models from succeeding over CLEVR_HYP without proper reasoning.

In CLEVR_HYP, one has to perform actions described in T over image I and then answer question Q with respect to the updated scenario. Therefore, to prevent ad-hoc models from exploiting biases in CLEVR_HYP, we pose the requirement that a question must have different ground-truth answers for CLEVR_HYP and image-only model. One such example is shown in Figure 8. For image (I), Q1 leads to different answers for CLEVR and CLEVR_HYP, making sure that one needs to correctly incorporate the effect of T. Q2 is invalid for a given image-action text pair in the CLEVR_HYP as one can answer it correctly without understanding T.

### More Examples from CLEVR_HYP

Beyond Figure 10, all rest of the pages show more examples from our CLEVR_HYP dataset. Each dataset item has 4 main components- image(I), action text (T\({}_{A}\)), question about the hypothetical states (Q\({}_{H}\)) and answer (A). We classify samples based on what actions are taken over the image and the kind of reasoning is required to answer questions.

Figure 7: Extension of CLEVR_HYP for more complex real-world scenarios.

 



### Function Catalog

As described in Section 3 and shown in Figure 4, each action text and question is associated with a functional program. We provide more details about these basic functions in Table 4 that was used to generate ground-truth answers for our dataset. Each function has input and output arguments, which are limited to following data types:

* object: a single object in the scene
* object: a set of zero or more objects in scene
* integer: an integer in [0,10]
* boolean: 'yes' or 'no'
* values: possible attribute values mentioned in Table 1

### Paraphrasing

In order to create a challenging dataset from the linguistic point of view and to prevent models from overfitting on templated representations, we leverage word synonyms and paraphrasing methods. This section provides more details about paraphrasing methods used in our dataset.

Object Name ParaphrasingThere can be many ways an object can be referred in the scene. For example, 'large purple metal sphere' in image below can also be referred to as 'sphere' as there is no other sphere present in the image. In order to make templates more challenging, we use these alternative expressions to refer objects in the action text or question. We wrote a python script that takes scene graph of the image and generates all possible names one can uniquely refer for each object in the scene. When paraphrasing is performed, one of the generated names is randomly chosen and replaced. Figure 9 demonstrates list of all possible name variants for two objects in the given image.

Synonyms for ParaphrasingWe use word synonyms file provided with CLEVR dataset generation code.

Sentence/Question Level ParaphrasingFor action text paraphrasing, we use Fairseq Ott et al. (2019) based paraphrasing tool which uses round-trip translation and mixture of experts Shen et al. (2019). Specifically, we use pre-trained round-trip models (En-Fr and Fr-En) and choose top-5 paraphrases manually for each template. For question paraphrasing, the quality of round-trip translation and mixture of experts was not satisfactory. Therefore, we use Text-To-Text Transfer Transformer T5 Raffel et al. (2020) fine-tuned over positive samples from Quora Question Pairs (QQP) dataset Iyer et al. (2017) and choose top-5 per template.

### Computational Resources

All of our experiments are performed over Tesla V100-PCIE-16GB GPU.

Figure 8: Validity of questions in CLEVR_HYP



\begin{table}
\begin{tabular}{l l p{142.3pt}}
**Function** & **Input Type \(\rightarrow\) Output Type** & **Return Value** \\ \hline scene & \(\phi\rightarrow\) object & Set of all objects in the scene \\ unique & object \(\rightarrow\) object & Object if object is singleton; else raise exception (to verify whether the input is unique or not) \\ relate & object x relation \(\rightarrow\) object & Objects satisfying given spatial relation for input object \\ count & object \(\rightarrow\) integer & Size of the input set \\ exist & object \(\rightarrow\) boolean & ‘Yes’ if the input set is non-empty and ’No’ otherwise \\ filter\_size & object x size \(\rightarrow\) object & Subset of input objects that match the given size \\ filter\_color & object x color \(\rightarrow\) object & Subset of input objects that match the given color \\ filter\_material & object x material \(\rightarrow\) object & Subset of input objects that match the given material \\ filter\_shape & object x shape \(\rightarrow\) object & Subset of input objects that match the given shape \\ query\_size & object \(\rightarrow\) size & Size of the input object \\ query\_color & object \(\rightarrow\) color & Color of the input object \\ query\_material & object \(\rightarrow\) material & Material of the input object \\ query\_shape & object \(\rightarrow\) shape & Shape of the input object \\ same\_size & object \(\rightarrow\) object & Set of objects that have same size as input (excluded) \\ same\_color & object \(\rightarrow\) object & Set of objects that have same color as input (excluded) \\ same\_material & object \(\rightarrow\) object & Set of objects that have same material as input(excluded) \\ same\_shape & object \(\rightarrow\) object & Set of objects that have same shape as input (excluded) \\ equal\_size & size × size × boolean & ‘Yes’ if inputs are equal, ’No’ otherwise \\ equal\_color & color × color → boolean & ‘Yes’ if inputs are equal, ’No’ otherwise \\ equal\_material & material × material × boolean & ‘Yes’ if inputs are equal, ’No’ otherwise \\ equal\_shape & shape × shape × boolean & ‘Yes’ if inputs are equal, ’No’ otherwise \\ equal\_integer & integer × integer × boolean & ‘Yes’ if two integer inputs are equal, ‘No’ otherwise \\ less\_than & integer × integer × boolean & ‘Yes’ if first integer is smaller than second, else ‘No’ \\ greater\_than & integer × integer × boolean & ‘Yes’ if first integer is larger than second, else ‘No’ \\ and & object × objset × object × object & Intersection of the two input sets \\ or & object × object × object × object & Union of the two input sets. \\ \hline \hline not\_size & object \(\rightarrow\) object & Subset of input objects that do not match given size \\ not\_color & object \(\rightarrow\) object & Subset of input objects that do not match given color \\ not\_material & object \(\rightarrow\) object & Subset of input objects that do not match given material \\ not\_shape & object \(\rightarrow\) object & Subset of input objects that do not match given shape \\ add & object × object × object × remove & Input set with input object added to it \\ remove & object × object × object × & Input set with input object removed from it \\ add\_rel & × relation × object × remove & Input set with new object (first input) added at the \\  & x relation × object × remove & Given spatial location relative to second input object \\ remove\_rel & × object × × element × & Input set with object (first input) removed from the \\ change\_loc & × object × remove & Given spatial location relative to second input object \\ change\_size & object × size × object removes & Input set with object (first input) location changed to a \\ change\_model & × relation × object & Given spatial location relative to second input object \\ change\_size & object × size × object & Input set with size updated to the given value \\ change\_color & object × color → object change\_material & Input set with color updated to the given value \\ change\_shape & object × material × object change\_shape × object & Input set with shape updated to the given value \\ \hline \end{tabular}
\end{table}
Table 4: (upper) Original function catalog for CLEVR proposed in Johnson et al. (2017), which we reuse in our data creation process (lower) New functions added to the function catalog for the CLEVR_HYP dataset.



Figure 10: More examples from the CLEVR_HYP dataset





## References

* [1] A. K. Burket, J. M. Campbell and R. C. Myers, _Quantum Chromodynamics: The Theory of Quantum Chromodynamics_, Cambridge University Press, 2003.
* [2] S. G. Cohen and D. E. Gross, "The theory of the chromodynamics in a quantum chromodynamics approach", _Phys. Rev. Lett._**57**, 4689-479; 1986.
* [3] T. W. Grimm, _Quantum Chromodynamics: The Theory of Quantum Chromodynamics_, Cambridge University Press, 2003.
* [4] H. B. Nielsen and I. L. Chuang, _Nucl. Phys._**B28**, 111-124; _ibid._**B29**, 111-120.
* [5] N. Metropolis, _Random Matrices: An Introduction to Statistical Mechanics_, Oxford University Press, 1983.

[MISSING_PAGE_POST]



* [19] T\({}_{A^{i}}\): John hides the big object to the right of the brown sphere.
* [20] T\({}_{H}\): How many yellow or cyan objects are there?
* [21] T\({}_{A^{i}}\): All brown things become matte.
* [22] Q\({}_{H^{t}}\): How many any other things are there which are made of the same material as the small cyan object?
* [23] A: 2
* [24] **Classification**: Change action, Counting + Compare Attribute question Split: 2HopQ\({}_{H}\) test
* [25] T\({}_{A^{i}}\): Make all the brown objects shiny.
* [26] Q\({}_{H^{t}}\): Are there any non metal things to the right of the shiny sphere?
* [27] Q\({}_{H}\): Can change action, Existence question with negation Split: 2HopQ\({}_{H}\) test
* [28] T\({}_{A^{i}}\): The gray object is moved to the right of the yellow thing.
* [29] Q\({}_{H^{t}}\): There is a cyan block; what number of big objects are there to the left of it that has the same material as the blue cube?
* [28] **Classification**: Move (in-plane) action, Counting + Compare Attribute question Split: 2HopQ\({}_{H^{t}}\) test
* [29] T\({}_{A^{i}}\): Remove all the yellow cylinders; Then clone the brown object and put it to the left side of the cyan ball.
* [29] Q\({}_{H^{t}}\): How many objects are made of the rubber?
* [30] Q\({}_{H^{t}}\): Add+Remove actions, Count question Split: 2HopT\({}_{A}\) test
* [31] T\({}_{A^{i}}\): Enlarge the purple object; Then add a large red matte sphere to the right of the large purple cube.
* [32] Q\({}_{H^{t}}\): Is there any small object in the scene?
* [33] A: No
* [34] **Classification**: Add+Change actions, Existence question Split: 2HopT\({}_{A}\) test
* [35] T\({}_{A^{i}}\): Add a small brown rubber sphere to the left of yellow matte object; Then swap its position with the purple shiny sphere.
* [36] Q\({}_{H^{t}}\): There is a ball that is to the left of the blue cube; what is its color?
* [37] Q\({}_{H^{t}}\): If there is a ball that is to the left of the blue cube; what is its color?
* [38] Q\({}_{H^{t}}\): When 2HopT\({}_{A}\) test
* [39] Q\({}_{H^{t}}\): If there is a ball that is to the left of the blue cube; what is its color?
* [39] Q\({}_{H^{t}}\): If there is a ball that is to the left of the blue cube; what is its color?