SlowGuess/ACL_2020_OCR huggingface dataset:
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, pages 6700-6709.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In Empirical methods in natural language processing (EMNLP), pages 787-798.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L. Yuille. 2019. Clevr-ref+: Diagnosing visual reasoning with referring expressions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pages 4185-4194.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visi-olinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265.
Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In IEEE conference on computer vision and pattern recognition, pages 11-20.
Varun K. Nagaraja, Vlad I. Morariu, and Larry S. Davis. 2016. Modeling context between objects for referring expression understanding. In European Conference on Computer Vision, pages 792-807.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A
6560
cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Association for Computational Linguistics, pages 2556-2565.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018. Mattnet: Modular attention network for referring expression comprehension. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1307-1315.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4995-5004.
In this supplementary material, we begin by providing more details on RefCOCOg dataset to supplement Section 2 of the main paper. We then provide Ref-Adv annotation details, statistics, analysis, and random examples, to supplement Section 3 of the main paper. Finally, we provide details of our models (initialization & training, hyper-parameters) and show additional results to supplement Section 5 of the main paper.
A.1 RefCOCOg vs Other Referring Expressions Datasets
RefCOCO, RefCOCO+ (Kazemzadeh et al., 2014) and RefCOCOg (Google-RefCOCO; Mao et al. 2016) are three commonly studied visual referring expression recognition datasets for real images. All the three data sets are built on top of MSCOCO dataset (Lin et al., 2014) which contains more than 300,000 images, with 80 categories of objects. RefCOCO, RefCOCO+ were collected using online interactive game. RefCOCO dataset is more biased towards person category. RefCOCO+ does not allow the use of location words in the expressions, and therefore contains very few spatial relationships. RefCOCOg was not collected in an interactive setting and therefore contains longer expressions.
For our adversarial analysis, we chose RefCOCOg for the following three important reasons: Firstly, expressions are longer (by 2.5 times on average) in RefCOCOg and therefore contains more spatial relationships compared to other two datasets. Secondly, RefCOCOg contains at least 2 to 4 instances of the same object type within the same image referred by an expression. This makes the dataset more robust, and indirectly puts higher importance on grounding spatial relationships in finding the target object. Finally, as shown in Table 4, RefCOCO and RefCOCO+ are highly skewed towards Person object category ( $\approx 50\%$ ) whereas RefCOCOg is relatively less skewed ( $\approx 36\%$ ), more diverse, and less biased.
A.2 Importance of Linguistic Structure
Cirik et al. (2018b) observed that existing models for RefCOCOg are relying heavily on the biases in the data than on linguistic structure. We perform extensive experiments to get more detailed insights into this observation. Specifically, we distort linguistic structure of referring expressions in the Re
6561
RefCOCO	RefCOCO+	RefCOCOg
Outdoor	0.89%	0.88%	1.65%
Food	10.16%	10.07%	8.10%
Indoor	3.10%	3.09%	2.59%
Appliance	0.67%	0.68%	1.03%
Kitchen	3.95%	3.95%	5.40%
Accessory	2.33%	2.33%	2.85%
Person	49.50%	49.70%	37.02%
Animal	13.26%	13.27%	15.05%
Vehicle	7.23%	7.22%	10.71%
Sports	0.73%	0.74%	1.91%
Electronic	1.94%	1.95%	2.56%
Furniture	6.14%	6.12%	11.09%
$fCOCOg$  test split and evaluate the SOTA models that are trained on original undistorted RefCOCOg training split. Similar to (Cirik et al., 2018b), we distort the test split using two methods: (a) randomly shuffle words in a referring expression, and (b) delete all the words in the expression except for nouns and adjectives. Table 5 shows accuracies for the models with (column 3 and 4) and without (column 2) distorted referring expressions. Except for the ViLBERT model(Lu et al., 2019), the drop in accuracy is not significant indicating that spatial relations are ignored in grounding the referring expression.
Using the relatively robust ViLBERT model, we repeat this analysis on our splits Ref-Easy, Ref-Hard and Ref-Adv. We randomly sampled 1500 expressions from each of these splits and then compare performance of ViLBERT on these three sets. As shown in Table 6, we find a large difference in model's accuracy on Ref-Hard and Ref-Adv. This clearly indicates that grounding expressions in both of these splits require linguistic and spatial reasoning.
A.3 Ref-Adv Annotation
We construct Ref-Adv by using all the 9602 referring expressions from RefCOCOg test data split. As shown in Figure 5, we follow a three stage approach to collect these new samples:
Stage 1: For every referring expression in Ref-COCOg test split, we perturb its linguistic structure by shuffling the word order randomly. We show each of these perturbed expression along with im
Table 4: Distribution of object categories in RefCOCO, RefCOCO+, and RefCOCOg datasets.
Model	Original	Shuf	N+J
CMN (Hu et al., 2017)	69.4	66.4	67.4
GroundNet (Cirik et al., 2018a)	65.8	57.6	62.8
MattNet (Yu et al., 2018)	78.5	75.3	76.1
ViLBERT (Lu et al., 2019)	83.6	71.4	73.6
Table 5: RefCOCOg test accuracies of SOTA models on (a) original undistorted split, (b) after randomly shuffling words (Shuf) in the referring expression, and (c) after deleting all the words except for nouns and adjectives  $(\mathrm{N} + \mathrm{J})$ . ViLBERT is relatively more robust than other baselines.
Test	Original	Shuf	N+J
Ref-Easy	86.40	75.06	76.00
Ref-Hard	72.73	51.13	56.60
Ref-Adv	71.08	50.23	57.40
Table 6: Ref-Easy, Ref-Hard, and Ref-Adv test accuracies of ViLBERT on (a) original undistorted split, (b) after randomly shuffling words (Shuf) in the referring expression, and (c) after deleting all the words except for nouns and adjectives  $(\mathrm{N} + \mathrm{J})$
ages and all object bounding boxes to five qualified Amazon Mechanical Turk (AMT) workers and ask them to identify the ground-truth bounding box for the shuffled referring expression. We hired workers from US and Canada with approval rates higher than  $98\%$  and more than 1000 accepted HITs. At the beginning of the annotation, we ask the turkers to go through a familiarization phase where they become familiar with the task. We consider all the image and expression pairs for which at least 3 out of 5 annotators failed to locate the object correctly (with IoU  $< 0.5$ ) as hard samples (Ref-Hard). We refer to the image-expressions for which at least 3 out of 5 annotators were able to localize the object correctly as easy samples (Ref-Easy). On average, we found that humans failed to localize the objects correctly in  $17\%$  of the expressions.
Stage 2: We take Ref-Hard images and ask turkers to generate adversarial expressions such that the target object is different from the original object. More concretely, for each of the hard samples, we identify the most confused image regions among human annotators as the target objects in stage 1. For each of these target objects, we then ask three
Shuffle words
$\mathrm{O}$  Annotation by 5 turkers per sample
$\mathrm{O}$  Split into Ref-Easy and Ref-Hard splits
Consider most confused object in Stage 1 as target object
$\mathrm{O}$  Ask 3 turkers to write adversarial expression
Filter noisy adversarial expressions
$\mathrm{O}$  Annotation by 3 turkers
Referring Expressions	3704
Unique Images	976
Vocabulary	2319
Avg. Length of Expression	11.4
Table 7: Ref-Adv Statistics
Figure 5: Overview of our three-stage Ref-Adv construction process. Given the image, referring expression, ground-truth bounding boxes for all the samples in RefCOCOg test split, we first filter out the hard samples and then construct adversarial expressions using them. Please refer to section 2 for further detail.
Figure 6: Referring expression length distribution for Ref-Easy, Ref-Hard, Ref-Adv datasets.
turkers to write a referring expression while retaining at least three content words (nouns and adjectives) in the original referring expression. This generates adversarial expressions for the original ground-truth Ref-Hard referring expressions.
Stage 3: We filter out the noisy adversarial expressions generated in stage 2 by following a validation routine used in the generation of RefCOCOg dataset. We ask three additional AMT workers to select a bounding box to identify the target object in the adversarial expression and then remove the noisy samples for which the inter-annotator agreement among workers is low. The samples with at least 2 out of 3 annotators achieving IoU  $>0.5$  will be added to Ref-Adv dataset.
A.4 Dataset Analysis, Comparison, and Visualization
In Table 7 we summarize the size and complexity of our Ref-Adv split. Figure 6 shows expression length distribution of Ref-Easy, Ref-Hard, and Ref-Adv. It should be noted that Ref-Adv expressions are longer on average than Ref-Easy and Ref-Hard.
Distribution of object categories in Ref-Easy, Ref-Hard and Ref-Adv is shown in Table 8. In comparison to Ref-Easy and Ref-Hard, Ref-Adv is more balanced and less biased towards Person category. Figure 7 shows the relative frequency of the most frequent spatial relationships in all the three splits. As we can see, Ref-Adv comprises of rich and diverse spatial relationships. In Table 2, we show random selection of the Ref-Easy, Ref-Hard, and Ref-Adv splits.
A.5 Model and other Experiment Details
A.5.1 Datasets
GQA (Hudson and Manning, 2019) contains 22M questions generated from Visual Genome (Krishna et al., 2017) scene graphs. However, in our multi-task training (MTL), we leverage only 1.42M questions that require reasoning on both linguistic structure and spatial relations. We filter these re
Figure 7: Relative frequency of the most frequent spatial relationships in Ref-Easy, Ref-Hard, and Ref-Adv
Ref-Easy 8034 samples	Ref-Hard 1568 samples	Ref-Adv 3704 samples
Outdoor	1.21%	1.90%	1.97%
Food	7.94%	9.80%	9.63%
Indoor	2.81%	2.83%	2.76%
Appliance	0.80%	1.07%	1.11%
Kitchen	4.52%	5.73%	5.77%
Accessory	3.20%	5.44%	5.29%
Person	37.26%	20.88%	21.01%
Animal	15.95%	13.92%	13.90%
Vehicle	10.91%	10.40%	10.26%
Sports	1.45%	5.04%	5.13%
Electronic	2.62%	3.20%	3.31%
Furniture	11.28%	19.73%	19.83%
Table 8: Distribution of object categories in Ref-Easy, Ref-Hard, and Ref-Adv splits.
lational questions by applying the following constraint on question types: type.Semantic = 'rel'. We also apply this constraint for filtering the development set. We denote this subset as GQA-Rel. We considered GQA-Rel instead of GQA for two reasons: 1) GQA-Rel is a more related task to Ref-COCOg; and 2) MTL training with the full GQA set is computationally expensive. For each question in the dataset, there exists a long answer (free-form text) and a short answer (containing one or two words). We only consider the short answers for the questions and treat the unique set of answers as output categories. While the full GQA dataset has 3129 output categories, GQA-Rel contains only 1842 categories.
We follow Yu et al. (2018) in creating the train (80512 expressions), val (4896 expressions), and test (9602 expressions) splits of RefCOCOg. For all our experiments in this paper, we directly use the ground-truth bounding box proposals.
A.5.2 Training
ViLBERT Pre-training We used pre-trained ViLBERT model that is trained on 3.3 million image-caption pairs from Conceptual Captions dataset (Sharma et al., 2018).<sup>3</sup>
Single-Task Fine-tuning on RefCOCOg In order to fine-tune the baseline ViLBERT (Lu et al., 2019) model on RefCOCOg dataset, we pass the ViLBERT visual representation for each bounding box into a linear layer to predict a matching score (similar to RefCOCO+ training in Lu et al. 2019). We calculate accuracy using IoU metric (prediction is correct if IoU(predicted_region, ground-truth region) > 0.5). We use a binary cross-entropy loss and train the model for a maximum of 25 epochs. We use early-stopping based on the validation performance. We use an initial learning rate of 4e-5 and use a linear decay learning rate schedule with warm up. We train on 8 Tesla V100 GPUs with a total batch size of 512.
Negative Mining We used a batch size of 512 and randomly sample negatives from the minibatch for computational efficiency. We sampled 64 negatives from each batch for both Sum of Hinges and Max of Hinges losses. We fine-tune the margin
$^{3}$ ViLBERT 8-Layer model at the link https://github.com/jiasenlu/vilbert_beta
6564
Split	Before MTL	After MTL
GQA-Rel Dev	53.7%	56.0%
GQA Dev	40.24%	42.1%
GQA Test	36.64%	39.2%
Table 9: Performance on GQA-Rel Dev, GQA-Dev and GQA-Test splits before and after MTL training with RefCOCOg (Note: MTL training for all the three rows is performed using GQA-Rel and RefCOCOg).
ViLBERT	Ref-Dev	Ref-Test	Ref-Adv
Without TL and MTL	83.39	83.63	70.90
TL with VQA	82.26	84.14	72.96
TL with GQA	80.60	82.08	70.41
TL with GQA-Rel	81.05	83.12	70.78
MTL with VQA	81.20	82.10	70.82
MTL with GQA-Rel	83.45	84.30	73.92
Table 10: Comparing ViLBERT's Multi-task Learning (MTL) with Transfer Learning (TL) experiments. Ref-Dev and Ref-Test correspond to: RefCOCOg-Dev and RefCOCOg-Test splits respectively.
parameters based on development split. We train the model for a maximum of 25 epochs. We use early-stopping based on the validation performance. We use an initial learning rate of 4e-5 and use a linear decay learning rate schedule with warm up. We train on 8 Tesla V100 GPUs with a total batch size of 512.
Multi-Task Learning (MTL) with GQA-Rel The multi-task learning architecture is shown in Figure 3 in the main paper. The shared layers constitute transformer blocks (TRM) and coattentional transformer layers (Co-TRM) in ViLBERT (Lu et al., 2019). The task-specific layer for GQA task is a two-layer MLP and we treat it as a multi-class classification task and the task-specific layer for RER is a linear layer that predicts a matching score for each of the image regions given an input referring expression. The weights for the task-specific layers are randomly initialized, whereas the shared layers are initialized with weights pretrained on 3.3 million image-caption pairs from Conceptual Captions dataset (Sharma et al., 2018). We use a binary cross-entropy loss for both tasks. Similar to Luong et al. (2015), during training, we optimize each task alternatively in mini-batches based on a mixing ratio. We use early-stopping based on the validation performance. We use an
initial learning rate of 4e-5 for RefCOCOg and 2e-5 for GQA, and use a linear decay learning rate schedule with warm up. We train on 4 RTX 2080 GPUs with a total batch size of 256.
GQA MTL Results Table 3 in the main paper showed that MTL training with GQA-Rel significantly improved the performance of model on Ref-Hard and Ref-Adv splits. In addition, we also observed a significant improvement in GQA-Rel development, GQA development and test splits as shown in the Table 9.
A.5.3 Additional Experiments
In this subsection, we present results of additional experiments using transfer learning (TL) and multitask learning (MTL) with ViLBERT on VQA, GQA, and GQA-Rel tasks. As shown in Table 10, TL with VQA showed slight improvement. However, TL with GQA, TL with GQA-Rel, and MTL with VQA did not show any improvements<sup>4</sup>.
<sup>4</sup>We could not perform MTL with GQA as it requires large number of computational resources.