Venue: NAACL 2021

# Automatic Generation of Contrast Sets from Scene Graphs:

Probing the Compositional Consistency of GQA

Yonatan Bitton\({}^{\diamondsuit}\) Gabriel Stanovsky\({}^{\diamondsuit}\) Roy Schwartz\({}^{\diamondsuit}\) Michael Elhadad\({}^{\spadesuit}\)

\({}^{\diamondsuit}\)School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem, Israel

\({}^{\spadesuit}\) Department of Computer Science, Ben Gurion University, Beer Sheva, Israel

{yonatanbitton,gabis,roys}@cs.huji.ac.il elhadal@cs.bgu.ac.il

###### Abstract

Recent works have shown that supervised models often exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. _Contrast sets_Gardner et al. (2020) quantify this phenomenon by perturbing test samples in a minimal way such that the output label is modified. While most contrast sets were created manually, requiring intensive annotation effort, we present a novel method which leverages rich semantic input representation to _automatically_ generate contrast sets for the visual question answering task. Our method computes the answer of perturbed questions, thus vastly reducing annotation cost and enabling thorough evaluation of models' performance on various semantic aspects (e.g., spatial or relational reasoning). We demonstrate the effectiveness of our approach on the popular GQA dataset Hudson and Manning (2019) and its semantic scene graph image representation. We find that, despite GQA's compositionality and carefully balanced label distribution, two strong models drop 13-17% in accuracy on our automatically-constructed contrast set compared to the original validation set. Finally, we show that our method can be applied to the _training_ set to mitigate the degradation in performance, opening the door to more robust models.1

Footnote 1: Our contrast sets and code are available at [https://github.com/yonatambitton/AutoGenOfContrastSetsFromSceneGraphs](https://github.com/yonatambitton/AutoGenOfContrastSetsFromSceneGraphs).

+
Footnote †: _This work was supported by NSF grant DMS-0854655, DMS-

from "Yes" to "No". Since such perturbations introduce minimal additional semantic complexity, robust models are expected to perform similarly on the test and contrast sets. However, a range of NLP models severely degrade in performance on contrast sets, hinting that they do not generalize well Gardner et al. (2020). Except two recent exceptions for textual datasets Li et al. (2020); Roseman et al. (2020), contrast sets have so far been built manually, requiring extensive human effort and expertise.

In this work, we propose a method for _automatic_ generation of large contrast sets for visual question answering (VQA). We experiment with the GQA dataset Hudson and Manning (2019). GQA includes semantic scene graphs Krishna et al. (2017) representing the spatial relations between objects in the image, as exemplified in Fig. 1. The scene graphs, along with functional programs that represent the questions, are used to _balance_ the dataset, thus aiming to mitigate spurious dataset correlations. We leverage the GQA scene graphs to create contrast sets, by automatically computing the answers to question perturbations, e.g., verifying that there is no wall near the puddle in Fig. 1.

We create automatic contrast sets for 29K samples or \({\approx}\)22% of the validation set. We manually verify the correctness of 1,106 of these samples on Mechanical Turk. Following, we evaluate two leading models, LXMERT Tan and Bansal (2019) and MAC Hudson and Manning (2019) on our contrast sets, and find a 13-17% reduction in performance compared to the original validation set. Finally, we show that our automatic method for contrast set construction can be used to improve performance by employing it during _training_. We augment the GQA training set with automatically constructed _training contrast sets_ (adding 80K samples to the existing 943K in GQA), and observe that when trained with it, both LXMERT and MAC improve by about \(14\%\) on the contrast sets, while maintaining their original validation performance.

Our key contributions are: (1) We present an automatic method for creating contrast sets for VQA datasets with structured input representations; (2) We automatically create contrast sets for GQA, and find that for two strong models, performance on the contrast sets is lower than on the original validation set; and (3) We apply our method to augment the training data, improving both models' performance on the contrast sets.

## 2 Automatic Contrast Set Construction

To construct automatic contrast sets for GQA we first identify a large subset of questions requiring specific reasoning skills (SS2.1). Using the scene graph representation, we perturb each question in a manner which changes its gold answer (SS2.2). Finally, we validate the automatic process via crowdsourcing (SS2.3).

### Identifying Recurring Patterns in GQA

The questions in the GQA dataset present a diverse set of modelling challenges, as exemplified in Table 1, including object identification and grounding, spatial reasoning and color identification. Following the contrast set approach, we create perturbations testing whether models are capable of solving questions which require this skill set, but that diverge from their training distribution.

To achieve this, we identify commonly recurring question templates which specifically require such skills. For example, to answer the question "_Are there any cats near the boat?_" a model needs to identify objects in the image (_cats_, _boat_), link them to the question, and identify their relative position.

We identify six question templates, testing various skills (Table 1). We abstract each question template with a regular expression which identifies the question types as well as the physical objects, their attributes (e.g., colors), and spatial relations. Overall, these regular expressions match 29K questions in the validation set (\({\approx}\)22%), and 80K questions in the training set (\({\approx}\)8%).

### Perturbing Questions with Scene Graphs

We design a perturbation method which guarantees a change in the gold answer for each question template. For example, looking at Fig. 2, for the question template _are there X near the Y_ (e.g., "Is there any fence near the players?"), we replace either X or Y with a probable distractor (e.g.., replace "fence" with "trees").

We use the scene graph to ensure that the answer to the question is indeed changed. In our example, this would entail grounding "players" in the question to the scene graph (either via exact match or several other heuristics such as hard-coded lists of synonyms or co-hyponyms), locating its neighbors, and verifying that none of them are "trees." We then apply heuristics to fix syntax (e.g., changing from singular to plural determiner, see Appendix A.3), and verify that the perturbed sample 

does not already exist in GQA. The specific perturbation is performed per question template. In question templates with two objects (\(X\) and \(Y\)), we replace \(X\) with \(X^{\prime}\), such that \(X^{\prime}\) is correlated with \(Y\) in other GQA scene graphs. In question templates with a single object \(X\), we replace \(X\) with a textually-similar \(X^{\prime}\). For example in the first row in Table 1 we replace _dishwasher_ with _dishes_. Our perturbation code is publicly available.

This process may yield an arbitrarily large number of contrasting samples per question, as there are many candidates for replacing objects participating in questions. We report experiments with up to 1, 3 and 5 contrasting samples per question.

Illustrating the perturbation process.Looking at Fig. 1, we see the scene-graph information: _objects_ have bounding-boxes around them in the image (e.g., _zebra_); _Objects_ have _attributes_ (_wood_ is an attribute of the _fence_ object); and there are _relationships_ between the objects (the puddle is to the _right_ of the zebra, and it is _near_ the fence). The original (question, answer) pair is ("is there a fence near the puddle?", "Yes"). We first identify the question template by regular expressions: "Is there X near the Y", and isolate X=_fence_, Y=_puddle_. The answer is "Yes", so we know that X is indeed near Y. We then use the existing information given in the scene-graph. We search for X' that is not near Y. To achieve this, we sample a random object (_wall_), and verify that it doesn't exist in the set of scene-graph objects. This results in a perturbed example "Is there a _wall_ near the puddle?", and now the ground truth is computed to be "No". Consider a different example: ("Is the puddle to the left of the zebra?", "Yes"). We identify the question template "Is the X _Rel_ the Y", where X=_puddle_, Rel=_to the left_, Y=_zebra_. The answer is "Yes". Now we can easily change Rel'=_to the right_, resulting in the (question, answer) pair ("Is the puddle to the right of the zebra?", "No").

We highlight the following: (1) This process is done entirely automatically (we validate it in Section 2.3); (2) The answer is deterministic given the information in the scene-graph; (3) We do not produce unanswerable questions. If we couldn't find an alternative atom for which the presuppositions hold, we do not create the perturbed (question, answer) pair; (4) Grounding objects from the question to the scene-graph can be tricky. It can involve exact match, number match (_dogs_ in the question, and _dog_ in the scene-graph), hyponyms (_animal_ in the question, and _dog_ in the scene-graph), and synonyms (_motorbike_ in the question, and _motorcycle_ in the scene-graph). The details are in the published code; (5) The only difference between the original and the perturbed instance is a single atom: an object, relationship, or attribute.

### Validating Perturbed Instances

To verify the correctness of our automatic process, we sampled 553 images, each one with an original and perturbed QA pair for a total of 1,106 instances (\(\approx\)4% of the validation contrast pairs). The (image, question) pairs were answered independently by human annotators on Amazon Mechanical Turk (see Fig. 3 in Appendix A.4), oblivious to whether the question originated from GQA or from our automatic contrast set. We found that the workers were able to correctly answer 72.3% of the perturbed questions, slightly lower than their performance on the original questions (76.6%).2 We observed high agreement between annotators (\(\kappa=0.679\)).

Footnote 2: The GQA paper reports higher human accuracy (around 90%) on their original questions. We attribute this difference to the selection of a subset of questions that match our templates, which are potentially more ambiguous than average GQA questions (see Section 3).

Our analysis shows that the human performance difference between the perturbed questions and the original questions can be attributed to the scene

\begin{table}
\begin{tabular}{l l l} \hline
**Question template** & **Tested attributes** & **Example** \\ \hline On which side is the \(\mathbf{X}^{\intercal}\) & Relational (left vs. right) & On which side is the _dishwasher_? \(\rightarrow\) On which side are the _dishes_? \\ \hline What color is the \(\mathbf{X}^{\intercal}\) & Color identification & What color is the _cat_?\(\rightarrow\) What color is the _jacket_? \\ \hline Do you see \(\mathbf{X}\) or \(\mathbf{Y}^{\intercal}\) & Compositions & Do you see _leptops_ or cameras?\(\rightarrow\) Do you see _headphones_ or cameras? \\ \hline Are there \(\mathbf{X}\) near the \(Y\)? & & Are there any _cats_ near the boat? \(\rightarrow\)_Is_ there any _bush_ near the boat? \\ Is the \(\mathbf{X}\)_Rel_ the \(Y\)? & Spatial, relational & Is the boy to the _right_ of the man? \(\rightarrow\) Is the boy to the _left_ of the man? \\ Is the \(\mathbf{X}\)_Rel_ the \(Y\)? & & Is the _boy_ to the right of the man? \(\rightarrow\) Is the _zebra_ to the right of the man? \\ \hline \end{tabular}
\end{table}
Table 1: Question templates with original question examples, and generated perturbations modifying the answer. Italic text indicates variables, bold text indicates the perturbed atoms.

 

graph annotation errors in the GQA dataset: 3.5% of the 4% difference is caused by a discrepancy between image and scene graph (objects appearing in the image and not in the graph, and vice versa). Examples are available in Fig. 5 in Appendix A.5.

## 3 Experiments

We experiment with two top-performing GQA models, MAC Hudson and Manning (2018) and LXMERT Tan and Bansal (2019),3 to test their generalization on our automatic contrast sets, leading to various key observations.

Footnote 3: MAC and LXMERT are the top two models in the GQA leaderboard with a public implementation as of the time of submission: [https://github.com/airsplay/lxmert](https://github.com/airsplay/lxmert) and [https://github.com/stanfordnlp/mac-network/](https://github.com/stanfordnlp/mac-network/).

Models struggle with our contrast set.Table 2 shows that despite GQA's emphasis on dataset balance and compositionality, both MAC and LXMERT degraded on the contrast set: MAC 64.9% \(\rightarrow\) 51.5% and LXMERT 83.9% \(\rightarrow\) 67.2%, compared to only 4% degradation in human performance. Full breakdown of the results by template is shown in Table 3. As expected, question templates that reference two objects (\(X\) and \(Y\)) result in larger performance drop compared to those containing a single object (\(X\)). Questions about colors had the _smallest_ performance drop, potentially because the models performance on such multi-class, subjective questions is relatively low to begin with.

Training on perturbed set leads to more robust models.Previous works tried to mitigate spurious datasets biases by explicitly balancing labels during dataset construction Goyal et al. (2017); Zhu et al. (2016); Zhang et al. (2016) or using adversarial filtering Zellers et al. (2018, 2019). In this work we take an inoculation approach Liu et al. (2019) and augment the original GQA training set with contrast training data, resulting in a total of 1,023,607 training samples. We retrain both models on the augmented training data, and observe in Table 2 that their performance on the _contrast set_ almost matches that of the original validation set, with no loss (MAC) or only minor loss (LXMERT) to original validation accuracy.4 These results indicate that the perturbed _training_ set is a valuable signal, which helps models recognize more patterns.

Footnote 4: To verify that this is not the result of training on more data, we repeated this experiment, removing the same amount of original training instances (so the final dataset size is the same as the original one), and observed very similar results.

Contrast Consistency.Our method can be used to generate many augmented questions by simply sampling more items for replacement (Section 2).

Figure 2: GQA image (left) with example perturbations for different question templates (right). Each perturbation aims to change the label in a predetermined manner, e.g., from “yes” to “no”.

\begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{1}{c}{**Training set**} & \multicolumn{1}{c}{**Original**} & \multicolumn{1}{c}{**Augmented**} \\ \cline{2-4}  & Baseline & 64.9\% & 51.5\% \\  & Augmented & 64.4\% & 68.4\% \\ \hline \multirow{2}{*}{LXMERT} & Baseline & 83.9\% & 67.2\% \\  & Augmented & 82.6\% & 77.2\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Model accuracy on the original validation set and on our generated contrast sets with maximum of 5 augmentations. _Baseline_ refers to the original models, _augmented_ refers to the models trained with our augmented training contrast sets.

 

This allows us to measure the _contrast consistency_Gardner et al. (2020) of our contrast set, defined as the percentage of the contrast sets for which a model's predictions are correct for all examples in the set (including the original example). For example, in Fig. 1 the set size is 4, and only 2/4 predictions are correct. We experiment with 1, 3, and 5 augmentations per question with the LXMERT model trained on the original GQA training set. Our results (Table 4) show that sampling more objects leads to similar accuracy levels for the LXMERT model, indicating that quality of our contrast sets does not depend on the specific selection of replacements. However, we observe that consistency drops fast as the size of the contrast sets per QA instance grows, indicating that model success on a specific instance does not mean it can generalize robustly to perturbations.

## 4 Discussion and Conclusion

Our results suggest that both MAC and LXMERT under-perform when tested out of distribution. A remaining question is whether this is due to model architecture or dataset design. Bogin et al. (2020) claim that both of these models are prone to fail on compositional generalization because they do not decompose the problem into smaller sub-tasks. Our results support this claim. On the other hand, it is possible that a different dataset could prevent these models from finding shortcuts. Is there a dataset that can prevent _all_ shortcuts? Our automatic method for creating contrast sets allows us to ask those questions, while we believe that future work in better training mechanisms, as suggested in Bogin et al. (2020) and Jin et al. (2020), could help in making more robust models.

We proposed an automatic method for creating contrast sets for VQA datasets that use annotated scene graphs. We created contrast sets for the GQA dataset, which is designed to be compositional, balanced, and robust against statistical biases. We observed a large performance drop between the original and augmented sets. As our contrast sets can be generated cheaply, we further augmented the GQA training data with additional perturbed questions, and showed that this improves models' performance on the contrast set. Our proposed method can be extended to other VQA datasets.

## Acknowledgements

We thank the reviewers for the helpful comments and feedback. We thank the authors of GQA for building the dataset, and the authors of LXMERT and MAC for sharing their code and making it usable. This work was supported in part by the Center for Interdisciplinary Data Science Research at the Hebrew University of Jerusalem, and research gifts from the Allen Institute for AI.

## References

* Bogin et al. (2020) Ben Bogin, Sanjay Subramanian, Matt Gardner, and Jonathan Berant. 2020. Latent compositional representations improve systematic generalization in grounded question answering. _arXiv preprint arXiv:2007.00266_.
* Feng et al. (2019) Shi Feng, Eric Wallace, and Jordan Boyd-Graber. 2019. Misleading failures of partial-input baselines. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5533-5538, Florence, Italy. Association for Computational Linguistics.
* Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabil, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1307-1323, Online. Association for Computational Linguistics.
* Geva et al. (2019) Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1161-1166, Hong Kong, China. Association for Computational Linguistics.
* Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in you matter: Elevaring the role of image understanding in visual question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 6904-6913.

\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Augmentations** & **Contrast sets** & **Acc.** & **Consistency** \\ \hline
1 & 11,263 & 66\% & 63.4\% \\
3 & 23,236 & 67\% & 51.1\% \\
5 & 28,968 & 67\% & 46.1\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: Accuracy and consistency results for the LXMERT model on different contrast set sizes.

 

Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
* Hudson and Manning (2019) Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Hudson and Manning (2018) Drew Arad Hudson and Christopher D. Manning. 2018. Compositional attention networks for machine reasoning. In _International Conference on Learning Representations_.
* Jia and Liang (2017) Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
* Jin et al. (2020) Xisen Jin, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. 2020. Visually grounded continual learning of compositional phrases. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2018-2029, Online. Association for Computational Linguistics.
* Kaushik et al. (2019) Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. In _International Conference on Learning Representations_.
* Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. _International journal of computer vision_, 123(1):32-73.
* Li et al. (2020) Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld. 2020. Linguistically-informed transformations (LIT): A method for automatically generating contrast sets. In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_, pages 126-135, Online. Association for Computational Linguistics.
* Liu et al. (2019) Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2171-2179, Minneapolis, Minnesota. Association for Computational Linguistics.
* McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
* Ribeiro et al. (2018) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 856-865, Melbourne, Australia. Association for Computational Linguistics.
* Rosenman et al. (2020) Shachar Rosenman, Alon Jacovi, and Yoav Goldberg. 2020. Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3702-3710, Online. Association for Computational Linguistics.
* Stanovsky et al. (2019) Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1679-1684, Florence, Italy. Association for Computational Linguistics.
* Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5100-5111, Hong Kong, China. Association for Computational Linguistics.
* Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
* Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4791-4800, Florence, Italy. Association for Computational Linguistics.
* Zhang et al. (2016) Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Yin and yang: Balancing and answering binary visual questions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 5014-5022.
* Zhu et al. (2016) Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In _Proceedings of the IEEE conference 

on computer vision and pattern recognition_, pages 4995-5004.

 

Appendix

### Ethical Considerations

We created contrast sets automatically, and verified their correctness via the crowdsourcing annotation of a sample of roughly 1K instances. Section2.3 describes the annotation process on Amazon Mechanical Turk. The images and original questions were sampled from the public GQA dataset Hudson and Manning (2019), in the English language. Fig.3 in AppendixA.4 provides example of the annotation task. Overall, the crowdsourcing task resulted in \({\approx}6\) hours of work, which paid an average of 11USD per hour per annotator.

ReproducibilityThe augmentations were performed with a MacBook Pro laptop. Augmentations for the validation data takes < 1 hour per question template, and for the training data < 3 hours per question template. Overall process, < 24 hours.

The experiments have been performed with the public implementations of MAC Hudson and Manning (2018) and LXMERT Tan and Bansal (2019), models: [https://github.com/airsplay/lxmert](https://github.com/airsplay/lxmert), [https://github.com/stanfordnlp/mac-network/](https://github.com/stanfordnlp/mac-network/). The configurations were modified to not include the validation set in the training process. The experiments were performed with a Linux virtual machine with a NVIDIA's Tesla V100 GPU. The training took \({\sim}1\)-2 days in each model. Validation took \({\sim}~{}30\) minutes.

### Generated Contrast Sets Statistics

Table5 reports the basic statistics of automatic contrast sets generation method when applied on the GQA validation dataset. It shows the overall number of images and QA pairs that matched the 6 question types we identified. Tables6 shows the statistics per question type, indicating how productive each augmentation method is. Tables7 and8 shows the same statistics for the GQA Training dataset.

 



### Linguistic Heuristics for Questions Generation

For each question type, we select an object in the image scene graph, and update the question by substituting the reference to this object by another object. When substituting one object by another, we need to adjust the question to keep it fluent. Table 10 shows the specific linguistic rules we verify when performing this substitution.

### Annotation Task for Verifying Generated Contrast Sets

Fig. 3 shows the annotation task that is shown to Turkers to validate the QA pairs generated by our method.

\begin{table}
\begin{tabular}{l c} \hline \# Images & 72,140 \\ \# QA pairs & 943,000 \\ \hline \# Aug. QA pairs & 89,936 \\ **\# Aug. images** & 43,463 \\ \hline \% Aug. images & 60.2\% \\ **\% Aug. QA pairs** & 9.5\% \\ \hline \end{tabular}
\end{table}
Table 7: Training data augmentation statistics

Figure 3: Example of the annotation task at the Amazon Mechanical Turk website



### Examples

\begin{table}
\begin{tabular}{l r r r r} \hline \hline
**Question template** & **\# Aug. QA pairs** & **\# Aug. images** & **\% Aug. questions** \\ \hline On which side is the _X_? & 17,935 & 16,224 & 2.2\% \\ What color is the _X_? & 32,744 & 27,704 & 4.1\% \\ Are there \(X\) near the _Y_? & 2,682 & 2,323 & 0.3\% \\ Do you see \(X\) or _Y_? & 10,666 & 9,704 & 1.1\% \\ Is the _X Rel_ the _Y_? & 6,302 & 5,479 & 0.6\% \\ Is the _X Rel_ the _Y_? & 9,938 & 8,007 & 1.1\% \\ \hline \hline \end{tabular}
\end{table}
Table 8: Augmentation statistics per question template for the training data

\begin{table}
\begin{tabular}{l c r r r r r r r r r} \hline \hline  & \multicolumn{4}{c}{**Original Dataset**} & \multicolumn{6}{c}{**Aug. dataset**} \\ \cline{3-13}  & \multicolumn{3}{c}{} & \multicolumn{3}{c}{**Max 1**} & \multicolumn{3}{c}{**Max 3**} & \multicolumn{3}{c}{**Max 5**} \\ \cline{3-13}  & **Size** & **MAC** & **LXLERT** & **MAC** & **LXLERT** & **Size** & **MAC** & **LXLERT** & **Size** & **MAC** & **LXLERT** \\ \hline On which side is the _X_? & 2,538 & 68\% & 94\% & 56\% & 79\% & 4,927 & 57\% & 80\% & 5,662 & 57\% & 81\% \\ What color is the _X_? & 4,654 & 49\% & 69\% & 48\% & 62\% & 10,506 & 49\% & 62\% & 12,498 & 49\% & 62\% \\ Are there \(X\) near the _Y_? & 382 & 85\% & 98\% & 72\% & 84\% & 867 & 69\% & 80\% & 1,320 & 66\% & 79\% \\ Do you see \(X\) or _Y_? & 1,506 & 88\% & 95\% & 53\% & 63\% & 4,205 & 53\% & 64\% & 6,679 & 53\% & 65\% \\ Is the _X Rel_ the _Y_? & 766 & 85\% & 96\% & 42\% & 67\% & 1,314 & 44\% & 69\% & 1,392 & 44\% & 69\% \\ Is the _X Rel_ the _Y_? & 1,417 & 71\% & 93\% & 38\% & 55\% & 1,417 & 38\% & 55\% & 1,417 & 38\% & 55\% \\ Overall & 11,263 & 65\% & 84\% & 50\% & 66\% & 23,236 & 51\% & 67\% & 28,968 & 52\% & 67\% \\ \hline \hline \end{tabular}
\end{table}
Table 9: Model accuracy by question template and maximum number of augmentations. Italic text indicates variables, bold text indicates the perturbed atoms.





\begin{table}
\begin{tabular}{|c|c|c|} \hline
**Linguistic rule** & **Explanation** & **Examples** \\ \hline  & If the noun is singular and countable: & “a fence”, “men” \\ Singular vs. plural & add “a” or “an” & “a boy”, “an elephant” \\  & If needed, replace “Are” and “Is" & “is there any fence near the boy” \\ Definite vs. indefinite & Do not change definite articles & suggests that there is a boy in the scene graph, which is not always correct \\  & to indefinite articles, and vice versa & “Cats in the image” \\ General vs. specific & Meaning can be changed & “Animals not in the image” \\  & When replacing to general & “Animals not in the image” \\  & or specific terms & The opposite directions not necessarily holds \\ Countable vs. uncountable & If the noun is uncountable, & “A cat”, “water” \\  & do not add “a” or “an” & “A cat”, “water” \\ \hline \end{tabular}
\end{table}
Table 10: Partial linguistic rules to notice using our method.



Figure 5: Scene graph annotation mistakes