SlowGuess/ACL_2020_OCR huggingface dataset:
Arjun R. Akula $^{1*}$ , Spandana Gella $^{2}$ , Yaser Al-Onaizan $^{2}$ , Song-Chun Zhu $^{1}$ , Siva Reddy $^{3}$
$^{1}$ UCLA Center for Vision, Cognition, Learning, and Autonomy,  $^{2}$ Amazon AI
$^{3}$ Facebook CIFAR AI Chair, Mila; McGill University
aakula@ucla.edu, sgella@amazon.com, onaizan@amazon.com,
sczhu@stat.ucla.edu, siva.reddy@mila.quebec
Visual referring expression recognition is a challenging task that requires natural language understanding in the context of an image. We critically examine RefCOCOg, a standard benchmark for this task, using a human study and show that  $83.7\%$  of test instances do not require reasoning on linguistic structure, i.e., words are enough to identify the target object, the word order doesn't matter. To measure the true progress of existing models, we split the test set into two sets, one which requires reasoning on linguistic structure and the other which doesn't. Additionally, we create an out-of-distribution dataset Ref-Adv by asking crowdworkers to perturb in-domain examples such that the target object changes. Using these datasets, we empirically show that existing methods fail to exploit linguistic structure and are  $12\%$  to  $23\%$  lower in performance than the established progress for this task. We also propose two methods, one based on contrastive learning and the other based on multi-task learning, to increase the robustness of ViLBERT, the current state-of-the-art model for this task. Our datasets are publicly available at https://github.com/awsAWS-refcocog-adv.
Visual referring expression recognition is the task of identifying the object in an image referred by a natural language expression (Kazemzadeh et al., 2014; Nagaraja et al., 2016; Mao et al., 2016; Hu et al., 2016). Figure 1 shows an example. This task has drawn much attention due to its ability to test a model's understanding of natural language in the context of visual grounding and its application in downstream tasks such as image retrieval (Young et al., 2014) and question answering (Antol et al., 2015; Zhu et al., 2016). To track
Figure 1: An example of the visual referring expression recognition task. If the word pastry is present in the referring expression, models prefer the bounding box  $r1$  (highlighted in green) irrespective of the change in linguistic structure (word order).
progress on this task, various datasets have been proposed, in which real world images are annotated by crowdsourced workers (Kazemzadeh et al., 2014; Mao et al., 2016). Recently, neural models have achieved tremendous progress on these datasets (Yu et al., 2018; Lu et al., 2019). However, multiple studies have suggested that these models could be exploiting strong biases in these datasets (Cirik et al., 2018b; Liu et al., 2019). For example, models could be just selecting a salient object in an image or a referring expression without recourse to linguistic structure (see Figure 1). This defeats the true purpose of the task casting doubts on the actual progress.
In this work, we examine RefCOCOg dataset (Mao et al., 2016), a popular testbed for evaluating referring expression models, using crowdsourced workers. We show that a large percentage of samples in the RefCOCOg test set indeed do not rely on linguistic structure (word order) of the expressions. Accordingly, we split RefCOCOg test set into two splits, Ref-Easy and Ref-Hard, where linguistic structure is key for recognition in the latter but not the former (§2). In addition, we create a new out-of-distribution<sup>1</sup> dataset called Ref-Adv using Ref-Hard by rewriting a referring expression
*Work done in part while AA was intern at Amazon AI.
This is a contrast set according to Gardner et al. (2020)
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6555-6565 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
such that the target object is different from the original annotation (§3). We evaluate existing models on these splits and show that the true progress is at least  $12 - 23\%$  behind the established progress, indicating there is ample room for improvement (§4). We propose two new models, one which make use of contrastive learning using negative examples, and the other based on multi-task learning, and show that these are slightly more robust than the current state-of-the-art models (§5).
2 Importance of linguistic structure
RefCOCOg is the largest visual referring expression benchmark available for real world images (Mao et al., 2016). Unlike other referring expression datasets such as RefCOCO and RefCOCO+ (Kazemzadeh et al., 2014), a special care has been taken such that expressions are longer and diverse. We therefore choose to examine the importance of linguistic structure in RefCOCOg.
Cirik et al. (2018b) observed that when the words in a referring expression are shuffled in random order, the performance of existing models on  $Re-fCOCo$  drops only a little. This suggests that models are relying heavily on the biases in the data than on linguistic structure, i.e., the actual sequence of words. Ideally, we want to test models on samples where there is correlation between linguistic structure and spatial relations of objects, and any obscurity in the structure should lead to ambiguity. To filter out such set, we use humans.
We randomly shuffle words in a referring expression to distort its linguistic structure, and ask humans to identify the target object of interest via predefined bounding boxes. Each image in Ref-COCOg test set is annotated by five Amazon Mechanical Turk (AMT) workers and when at least three annotators select a bounding box that has high overlap with the ground truth, we treat it as a correct prediction. Following Mao et al. (2016), we set 0.5 IoU (intersection over union) as the threshold for high overlap. Given that there are at least two objects in each image, the optimal performance of a random choice is less than  $50\%$ . However, we observe that human accuracy on distorted examples is  $83.7\%$ , indicating that a large portion of RefCOCOg test set is insensitive to linguistic structure. Based on this observation, we divide the test set into two splits for fine-grained evaluation of models: Ref-Easy contains samples insensitive
Ref-Easy	Ref-Hard	Ref-Adv
data size	8034	1568	3704
(83.7% of RefCOCOg)	(16.3% of RefCOCOg)
avg. length in words	8.0	10.2	11.4
Table 1: Statistics of Ref-Easy, Ref-Hard and Ref-Adv. Ref-Easy and Ref-Hard indicate the proportion of samples in RefCOCOg test set that are insensitive and sensitive to linguistic structure respectively.
to linguistic structure and Ref-Hard contains sensitive samples (statistics of the splits are shown in Table 1).
3 An out-of-distribution dataset
Due to unintended annotation artifacts in Ref-COCOg, it is still possible that models could perform well on Ref-Hard without having to rely on linguistic structure, e.g., by selecting frequent objects seen during training time. Essentially, Ref-Hard is an in-distribution split. To avoid this, we create Ref-Adv, an adversarial test set with samples that may be fall out of training distribution.
We take each sample in Ref-Hard and collect additional referring expressions such that the target object is different from the original object. We chose the target objects which humans are most confused with when the referring expression is shuffled (as described in the previous section). For each target object, we ask three AMT workers to write a referring expression while retaining most content words in the original referring expression. In contrast to the original expression, the modified expression mainly differs in terms of the structure while sharing several words. For example, in Figure 1, the adversarial sample is created by swapping pastry and blue fork and making plate as the head of pastry. We perform an extra validation step to filter out bad referring expressions. In this step, three additional AMT workers select a bounding box to identify the target object, and we only select the samples where at least two workers achieve IoU  $>0.5$  with the target object.
Since the samples in Ref-Adv mainly differ in linguistic structure with respect to Ref-Hard, we hope that a model which does not make use of linguistic structure (and correspondingly spatial relations between objects) performs worse on Ref-Adv even when it performs well on Ref-Hard due to exploiting biases in the training data.
Figure 2 shows several examples from the Ref-Easy, Ref-Hard, and Ref-Adv splits. We note that
2On average, there are 8.2 bounding boxes per image.
Easy: A dinning table with cake and drinks
Hard: A chair with a purse hanging from it
Adv: The purse which is hanging from a chair
Easy: Bus
Hard: Bus in the middle of the crowd
Adv: The crowd that the bus is in the middle of
Easy: The larger of two giraffes
Hard: A giraffe eating leaves off the tree
Adv: The giraffe that is not eating leaves off the tree
Easy: A blue snowboard
Hard: A woman wearing a blue jacket and orange glasses next to a woman with a white hood
Adv: A woman with a white hood, next to a woman wearing orange glasses and a blue jacket.
Easy: Water in a tall, clear glass
Hard: The glass of water next to the saucer with the cup on it
Adv: The cup on the saucer, next to the glass of water
Easy: The short blue bike on the right
Hard: The blue bike behind the red car
Adv: The red car behind the blue bike
Easy: The man with the glasses on
Hard: A man holding a cake that is not wearing a tie
Adv: The man holding a cake that is wearing a tie
Figure 2: Examples from Ref-Easy, Ref-Hard, and Ref-Adv splits. As seen, Ref-Hard and Ref-Adv have several words in common but differ in their linguistic structure and the target object of interest.
Easy: A green cushion couch with a pillow
Hard: A green couch across from a white couch
Adv: A white couch across from a green couch
Ref-Adv expressions are longer on average than Ref-Easy and Ref-Hard (Figure 6 in appendix) and consists of rich and diverse spatial relationships (Figure 7 in appendix).
Concurrent to our work, Gardner et al. (2020) also propose perturbed test splits for several tasks by modifying in-domain examples. In their setup, the original authors of each task create perturbed examples, whereas we use crowdworkers. Closest to our work is from Kaushik et al. (2020) who also use crowdworkers. While we use perturbed examples to evaluate robustness, they also use them to improve robustness (we propose complementary methods to improve robustness §5). Moreover, we are primarily concerned with the robustness of models for visual expression recognition task, while Gardner et al. and Kaushik et al. focus on different tasks (e.g., sentiment, natural language inference).
3.1 Human Performance on Ref-Easy,
Ref-Hard and Ref-Adv
We conducted an additional human study (on AMT) to compare the human performance on Ref-Easy, Ref-Hard and Ref-Adv splits. First, we randomly sampled 100 referring expressions from each of the three splits. Each referring expression is then assigned to three AMT workers and are asked to select a bounding box to identify the target object. We considered a sample to be correctly annotated by humans if at least two out of three workers select
the ground-truth annotation. Through this evaluation, we obtained human performance on each of the three splits Ref-Easy, Ref-Hard, and Ref-Adv as  $98\%$ ,  $95\%$ , and  $96\%$  respectively.
4 Diagnosing Referring Expression Recognition models
We evaluate the following models, most of which are designed to exploit linguistic structure.
CMN (Compositional Modular Networks; Hu et al. 2017; Andreas et al. 2016) grounds expressions using neural modules by decomposing an expression into <subject, relation, object> triples. The subject and object are localized to the objects in the image using a localization module while the relation between them is modeled using a relationship module. The full network learns to jointly decompose the input expression into a triple while also recognizing the target object.
GroundNet (Cirik et al., 2018a) is similar to CMN, however it makes use of rich linguistic structure (and correspondingly rich modules) as defined by an external syntactic parser.
MattNet (Yu et al., 2018) generalizes CMN to flexibly adapt to expressions that cannot be captured by the fixed template of CMN. It introduces new modules and also uses an attention mechanism to weigh modules.
ViLBERT (Lu et al., 2019), the state-of-the-art model for referring expression recognition, uses a
6557
Figure 3: Multi-task learning model for referring expression recognition with GQA
pretrain-then-transfer learning approach to jointly learn visiolinguistic representations from largescale data and utilizes them to ground expressions. This is the only model that does not explicitly model compositional structure of language, but BERT-like models are shown to capture syntactic structure latently (Hewitt and Manning, 2019).
4.1 Results and discussion
We trained on the full training set of RefCOCOg and performed hyperparameter tuning on a development set. We used the development and test splits of Mao et al. (2016). Table 2 shows the model accuracies on these splits and our proposed datasets. The models are trained to select ground truth bounding box from a set of predefined bounding boxes. We treat a prediction as positive if the predicted bounding box has IoU  $>0.5$  with the ground truth.
Although the overall performance on the test set seem high, in reality, models excel only at Ref-Easy while performing poorly on Ref-Hard. The difference in performance between Ref-Easy and Ref-Hard ranges up to  $15\%$ . This indicates that current models do not exploit linguistic structure effectively. When tested on Ref-Adv, the performance goes down even further, increasing the gap between Ref-Easy and Ref-Adv (up to  $26\%$ ). This suggests that models are relying on reasoning shortcuts found in training than actual understanding. Among the models, GroundNet performs worse, perhaps due to its reliance on rigid structure predicted by an external parser and the mismatches between the predicted structure and spatial relations between objects. ViLBERT achieves the highest performance and is relatively more robust than other models. In the next section, we propose methods to further increase the robustness of ViLBERT.
Model	Dev	Test	Easy	Hard	Adv
GroundNet	66.50	65.80	67.11	54.47	42.90
CMN	70.00	69.40	69.55	68.63	49.50
MattNet	79.21	78.51	80.96	65.94	54.64
ViLBERT	83.39	83.63	85.93	72.00	70.90
Table 2: Accuracy of models on RefCOCOg standard splits and our splits Ref-Easy, Ref-Hard and Ref-Adv.
5 Increasing the robustness of ViLBERT
We extend ViLBERT in two ways, one based on contrastive learning using negative samples, and the other based on multi-task learning on GQA (Hudson and Manning, 2019), a task that requires linguistic and spatial reasoning on images.
Contrastive learning using negative samples Instead of learning from one single example, contrastive learning aims to learn from multiple examples by comparing one to the other. In order to increase the sensitivity to linguistic structure, we mine negative examples that are close to the current example and learn to jointly minimize the loss on the current (positive) example and maximize the loss on negative examples. We treat the triplets  $(i,e,b)$  in the training set as positive examples, where  $i$ ,  $e$ ,  $b$  stands for image, expression and ground truth bounding box. For each triplet  $(i,e,b)$ , we sample another training example  $(i',e',b')$ , and use it to create two negative samples, defined by  $(i',e,b')$  and  $(i,e',b)$ , i.e., we pair wrong bounding boxes with wrong expressions. For efficiency, we only consider negative pairs from the mini-batch. We modify the batch loss function as follows:
$$
\begin{array}{l} \mathcal {L} (\mathbf {i}, \mathbf {e}, \mathbf {b}) = \mathbf {F} _ {(\mathbf {e}, \mathbf {e} ^ {\prime})} \left[ \ell (\mathbf {i}, \mathbf {e}, \mathbf {b}) - \ell (\mathbf {i}, \mathbf {e} ^ {\prime}, \mathbf {b}) - \tau \right] _ {+} \\ + \mathbf {F} _ {(\mathbf {i}, \mathbf {i} ^ {\prime})} \left[ \ell (\mathbf {i}, \mathbf {e}, \mathbf {b}) - \ell (\mathbf {i} ^ {\prime}, \mathbf {e}, \mathbf {b} ^ {\prime}) - \tau \right] _ {+} \\ \end{array}
$$
6558
Model	Dev	Test	Easy	Hard	Adv
ViLBERT (VB)	83.39	83.63	85.93	72.00	70.90
VB+Sum-H	81.61	83.00	85.93	70.60	72.30
VB+Max-H	82.93	82.70	86.58	70.46	73.35
VB+MTL (GQA)	83.45	84.30	86.23	73.79	73.92
Table 3: Accuracy of enhanced ViLBERT models.
Here  $\ell(i, e, b)$  is the cross-entropy loss of ViL-BERT,  $[x]_{+}$  is the hinge loss defined by  $\max(0, x)$ , and  $\tau$  is the margin parameter.  $F$  indicates a function over all batch samples. We define  $F$  to be either sum of hinges (Sum-H) or max of hinges (Max-H). While Sum-H takes sum over all negative samples, If batch size is  $n$ , for each  $(i, e, b)$ , there will be  $n - 1$  triplets of  $(i', e, b')$  and  $(i, e', b)$ . For  $(i, e, b)$ , there will be one  $(i', e, b')$  and one  $(i, e', b)$ . Similar proposals are known to increase the robustness of vision and language problems like visual-semantic embeddings and image description ranking (Kiros et al., 2014; Gella et al., 2017; Faghri et al., 2018).
Multi-task Learning (MTL) with GQA In order to increase the sensitivity to linguistic structure, we rely on tasks that require reasoning on linguistic structure and learn to perform them alongside our task. We employ MTL with GQA (Hudson and Manning, 2019), a compositional visual question answering dataset. Specifically, we use the GQA-Rel split which contains questions that require reasoning on both linguistic structure and spatial relations (e.g., Is there a boy wearing a red hat standing next to yellow bus? as opposed to Is there a boy wearing hat?). Figure 3 depicts the neural architecture. We share several layers between the tasks to enable the model to learn representations useful for both tasks. Each shared layer constitutes a co-attention transformer block (Co-TRM; Lu et al. 2019) and a transformer block (TRM; Vaswani et al. 2017). While in a transformer, attention is computed using queries and keys from the same modality, in a co-attention transformer they come from different modalities (see cross arrows in Figure 3). The shared representations are eventually passed as input to task-specific MLPs. We optimize each task using alternative training (Luong et al., 2015).
Results and discussion Table 3 shows the experimental results on the referring expression recognition task. Although contrastive learning improves
Figure 4: Predictions of ViLBERT and MTL model (GT denotes ground-truth).  $e1'$  and  $e2'$  are adversarial expressions of  $e1$  and  $e2$  respectively.
the robustness of ViLBERT on Ref-Adv  $(+1.4\%)$  and  $+2.5\%$  for Sum-H and Max-H respectively), it comes at a cost of slight performance drop on the full test (likely due to sacrificing biases shared between training and test sets). Whereas MTL improves the robustness on all sets showing that multitask learning helps (we observe  $2.3\%$  increase on GQA §A.5.2). Moreover, the performance of MTL on Ref-Hard and Ref-Adv are similar, suggesting that the model generalizes to unseen data distribution. Figure 4 shows qualitative examples comparing MTL predictions on Ref-Hard and Ref-Adv parallel examples. These suggest that the MTL model is sensitive to linguistic structure. However, there is still ample room for improvement indicated by the gap between Ref-Easy and Ref-Hard  $(12.4\%)$ .
Our work shows that current datasets and models for visual referring expressions fail to make effective use of linguistic structure. Although our proposed models are slightly more robust than existing models, there is still significant scope for improvement. We hope that Ref-Hard and Ref-Adv will foster more research in this area.
We would like to thank Volkan Cirik, Licheng Yu, Jiasen Lu for their help with GroundNet, MattNet and ViLBERT respectively, Keze Wang for his help with technical issues, and AWS AI data team for their help with Mechanical Turk. We are grateful to the anonymous reviewers for their useful feedback.
6559
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 39-48.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In IEEE international conference on computer vision, pages 2425-2433.
Volkan Cirik, Taylor Berg-Kirkpatrick, and Louis-Philippe Morency. 2018a. Using syntax to ground referring expressions in natural images. In AAAI Conference on Artificial Intelligence.
Volkan Cirik, Louis-Philippe Morency, and Taylor Berg-Kirkpatrick. 2018b. Visual referring expression recognition: What do systems actually learn? In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, pages 781-787.
Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018. VSE++: improving visual-semantic embeddings with hard negatives. In *British Machine Vision Conference*, page 12.
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating nlp models via contrast sets. arXiv preprint arXiv:2004.02709.
Spandana Gella, Rico Sennrich, Frank Keller, and Mirella Lapata. 2017. Image pivoting for learning multilingual multimodal representations. In Empirical Methods in Natural Language Processing, pages 2839-2845.
John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4129-4138.
Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. 2017. Modeling relationships in referential expressions with compositional modular networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1115-1124.
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natural language object retrieval. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4555-4564.