SlowGuess/NAACL_2021_OCR huggingface dataset:
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Drew Arad Hudson and Christopher D. Manning. 2018. Compositional attention networks for machine reasoning. In International Conference on Learning Representations.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Xisen Jin, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. 2020. Visually grounded continual learning of compositional phrases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2018-2029, Online. Association for Computational Linguistics.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73.
Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld. 2020. Linguistically-informed transformations (LIT): A method for automatically generating contrast sets. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 126-135, Online. Association for Computational Linguistics.
Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2171-2179, Minneapolis, Minnesota. Association for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856-865, Melbourne, Australia. Association for Computational Linguistics.
Shachar Rosenman, Alon Jacovi, and Yoav Goldberg. 2020. Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3702-3710, Online. Association for Computational Linguistics.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679-1684, Florence, Italy. Association for Computational Linguistics.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100-5111, Hong Kong, China. Association for Computational Linguistics.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy. Association for Computational Linguistics.
Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5014-5022.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference
on computer vision and pattern recognition, pages 4995-5004.
We created contrast sets automatically, and verified their correctness via the crowdsourcing annotation of a sample of roughly 1K instances. Section 2.3 describes the annotation process on Amazon Mechanical Turk. The images and original questions were sampled from the public GQA dataset (Hudson and Manning, 2019), in the English language. Fig. 3 in Appendix A.4 provides example of the annotation task. Overall, the crowdsourcing task resulted in  $\approx 6$  hours of work, which paid an average of 11USD per hour per annotator.
Reproducibility The augmentations were performed with a MacBook Pro laptop. Augmentations for the validation data takes  $< 1$  hour per question template, and for the training data  $< 3$  hours per question template. Overall process,  $< 24$  hours.
The experiments have been performed with the public implementations of MAC (Hudson and Manning, 2018) and LXMERT (Tan and Bansal, 2019), models: https://github.com/airsplay/lxmert, https://github.com/stanfordlp/mac-network/. The configurations were modified to not include the validation set in the training process. The experiments were performed with a Linux virtual machine with a NVIDIA's Tesla V100 GPU. The training took  $\sim 1-2$  days in each model. Validation took  $\sim 30$  minutes.
A.1 Generated Contrast Sets Statistics
Table 5 reports the basic statistics of automatic contrast sets generation method when applied on the GQA validation dataset. It shows the overall number of images and QA pairs that matched the 6 question types we identified. Tables 6 shows the statistics per question type, indicating how productive each augmentation method is. Tables 7 and 8 shows the same statistics for the GQA Training dataset.
# Aug. QA pairs
Max 1	Max 3	Max 5
# Images	10,696	10,696	10,696
# QA pairs	132,062	132,062	132,062
# Aug. QA pairs	12,962	26,189	32,802
# Aug. images	6,166	6,166	6,166
% Aug. images	57.6%	57.6%	57.6%
% Aug. QA pairs	9.8%	19.8%	24.8%
Table 5: Validation data augmentation statistics
Question template	# Aug. QA pairs
Max 1	Max 3	Max 5
On which side is the X?	2,516	4,889	5,617
What color is the X?	4,608	10,424	12,414
Are there X near the Y?	382	867	1,320
Do you see X or Y?	1,506	4,514	7,516
Is the X Rel the Y?	766	1,314	1,392
Is the X Rel the Y?	1,417	1,416	1,416
Table 6: Augmentation statistics per question template for the validation data
A.2 Models Performance Breakdown by Question Type and Number of Augmentations
Table 3 shows the breakdown of the performance of the MAC and LXMERT models per question type, on both the original GQA validation set and on the augmented contrast sets on validation.
The LXMERT model has two stages of training: pre-training on several datasets (which includes GQA training and validation data) and fine-tuning. To avoid inflating results on the validation data, we re-trained the pre-training stage without the GQA data, and fine-tuned on the training sets. Table 2. We discovered lower performance on the original set  $(\sim 5\%)$  with both models, but the same improvement on the augmented set  $(+\sim 10)$ .
# Images	72,140
# QA pairs	943,000
# Aug. QA pairs	89,936
# Aug. images	43,463
% Aug. images	60.2%
% Aug. QA pairs	9.5%
Table 7: Training data augmentation statistics
A.3 Linguistic Heuristics for Questions Generation
For each question type, we select an object in the image scene graph, and update the question by substituting the reference to this object by another object. When substituting one object by another, we need to adjust the question to keep it fluent. Table 10 shows the specific linguistic rules we verify when performing this substitution.
A.4 Annotation Task for Verifying Generated Contrast Sets
Fig. 3 shows the annotation task that is shown to Turkers to validate the QA pairs generated by our method.
Figure 3: Example of the annotation task at the Amazon Mechanical Turk website
Question template	# Aug. QA pairs	# Aug. images	% Aug. questions
On which side is the X?	17,935	16,224	2.2%
What color is the X?	32,744	27,704	4.1%
Are there X near the Y?	2,682	2,323	0.3%
Do you see X or Y?	10,666	9,704	1.1%
Is the X Rel the Y?	6,302	5,479	0.6%
Is the X Rel the Y?	9,938	8,007	1.1%
Table 8: Augmentation statistics per question template for the training data
Original Dataset	Aug. dataset
Max 1	Max 3	Max 5
Size	MAC	LXMERT	MAC	LXMERT	Size	MAC	LXMERT	Size	MAC	LXMERT
On which side is the X?	2,538	68%	94%	56%	79%	4,927	57%	80%	5,662	57%	81%
What color is the X?	4,654	49%	69%	48%	62%	10,506	49%	62%	12,498	49%	62%
Are there X near the Y?	382	85%	98%	72%	84%	867	69%	80%	1,320	66%	79%
Do you see X or Y?	1,506	88%	95%	53%	63%	4,205	53%	64%	6,679	53%	65%
Is the X Rel the Y?	766	85%	96%	42%	67%	1,314	44%	69%	1,392	44%	69%
Is the X Rel the Y?	1,417	71%	93%	38%	55%	1,417	38%	55%	1,417	38%	55%
Overall	11,263	65%	84%	50%	66%	23,236	51%	67%	28,968	52%	67%
Table 9: Model accuracy by question template and maximum number of augmentations. Italic text indicates variables, bold text indicates the perturbed atoms.
A.5 Examples
Linguistic rule	Explanation	Examples
Singular vs. plural	If the noun is singular and countable:
add “a” or “an”
If needed, replace “Are” and “Is”	“a fence”, “men”
“a boy”, “an elephant”
Definite vs. indefinite	Do not change definite articles
to indefinite articles, and vice versa	“”is there any fence near the boy”
suggests that there is a boy in the scene graph,
which is not always correct
General vs. specific	Meaning can be changed
When replacing to general
or specific terms	“Cats in the image” => “Animals in the image”,
“Animals not in the image” => “cats not in the image”,
The opposite directions not necessarily hold
Countable vs. uncountable	If the noun is uncountable,
do not add “a” or “an”	“A cat”, “water”
Table 10: Partial linguistic rules to notice using our method.
Original QA	Augmented QA
On which side is the blanket? Right	On which side is the ornament? Left
What color is the teddy bear to the right of the pillow? Brown	What color is the christmas lights? Yellow
Figure 4:	Is there a couch near the blanket? Yes	Is there a cat near the blanket? No
Do you see a pillow or couch there? Yes	Do you see a dress or a carpet there? No
If the pillow to the left of a cat? No	Is the pillow to the left of a teddy bear? Yes
Is the pillow to the left of a cat? No	No aug. - No relation between (pillow, cat)
(a) First case example - multiple objects
Augmented question: On which side of the photo are the bananas?
Expected answer: right
"bananas" are annotated in green text color in the right side of the image, but it also appears in additional locations
(b) Second case example - missing annotation
Augmented question: Do you see either a brown chair or couch in this picture?
Expected answer: No
We can see a couch in the left side of the image which is not annotated in the scene graph
(c) Third case example - incorrect annotation
Figure 5: Scene graph annotation mistakes
Augmented question: Do you see either any windows or fences?
Expected answer: Yes
We can see an incorrect annotation of "windows" on the person shirt in azure text color.