SlowGuess/NAACL_2022_OCR huggingface dataset:
Mauricio Mazuecos, Franco M Luque, Jorge Sánchez, Hernán Maina, Thomas Vadora, and Luciana Benotti. 2021. Region under discussion for visual dialog. In EMNLP, pages 4745-4759.
Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. 2016. Modeling context between objects for referring expression understanding. In European Conference on Computer Vision, pages 792-807. Springer.
Ivandre Paraboni, Alex Gwo Jen Lan, Matheus Mendes de Sant'Ana, and Flavio Luiz Coutinho. 2017. Effects of Cognitive Effort on the Resolution of Over-specified Descriptions. Computational Linguistics, 43(2):451-459.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Yanyuan Qiao, Chaorui Deng, and Qi Wu. 2020. Referring expression comprehension: A survey of methods and datasets. IEEE TMM.
Radolph Quirk, Sidney Greenbaum, Geoffrey Leech, and Jan Startvik. 1980. A grammar of contemporary english (9th Edition). Longman.
Shafin Rahman, Salman Khan, and Fatih Porikli. 2018. Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts. In ACCV, pages 547-563. Springer.
Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR, pages 658-666.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. VideoBERT: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998-6008.
Jette Vithen and Robert Dale. 2008. The use of spatial relations in referring expression generation. In Proceedings of the Fifth International Natural Language Generation Conference, pages 59-67, Salt Fork, Ohio, USA. Association for Computational Linguistics.
Terry Winograd. 1972. Understanding Natural Language. Academic Press, Inc., USA.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45. Association for Computational Linguistics.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934, 1:12
A Spatial prepositions and keywords
For the spatial prepositions, we rely on the Pattern Dictionary of English Prepositions (PDEP) (Litkowski, 2014), a publicly available lexical resource collected as a part of The Preposition Project (TPP). There are 78 prepositions in the spatial class. We removed prepositions with less than 10 samples, archaic and/or literary (e.g. 'pon, betwixt) and those used in a more technical context (e.g. aslant). The final list of prepositions and spatial keywords is as follows:
- Prepositions: aboard, about, above, across, after, against, ahead of, all over, along, alongside, amid, among, around, as far as, at, atop, before, behind, below, beneath, beside, besides, between, beyond, by, by way of, down, for, from, in, in front of, in line with, in sight of, in the midst of, inside, inside of, into, near, near to, neath, of, off, on, on a level with, on top of, onto, opposite, out of, outboard of, outside, outside of, outwith, over, over against, past, round about, short of, this side of, through, throughout, to, toward, towards, under, underneath, unto, up, up against, up
and down, up before, up to, upon, with, within, within sight of.
Keywords: background, back, bottom, center, corner, close, edge, end, entire, facing, far, farthest, floor, foreground, front, furthest, frontmost, ground, hidden, leftmost, left, middle, nearest, part, rightmost, right, row, side, top, upper.
B More error analysis
Here we present more examples from the analysis to further explain the type of error these models perform and the skills they need to improve.
B.1 Errors that are improved
As mentioned in Sec. 6, the cases where the green model is better than the orange baseline, mostly occur when meronomy skills, fuzzy objects, or both are present. In Fig. 6a, we see how the green model, correctly predicts the set of steps under the opera house although this referent does not have clear borders.
In the Fig. 6b, the baseline orange model shows an error when dealing with ordinal referring expressions.
(a) Steps to the opera house.
Figure 6: Errors that are improved.
(b) 2nd set of jewels.
The barplot in Fig. 9, shows the distribution of skill combinations found during the error analysis described in Section 6. The vertical axis details, in decreasing order, the combinations of skills that had the greatest impact in relation to the improvements observed in the green model over the baseline model. The  $x$ -axis expresses the frequency of occurrences of each of the combinations assigned to each example, in relation to the total of samples annotated.
B.2 Errors that remain a challenge
The images presented in Fig. 7, show two examples that still represent a challenge for the models.
Fig. 7b shows a frequent error found during the analysis. In this case, we observe that the models
(a) The surfboard of the blond surfboarder, the one walking towards the line.
(b) Door closest to right of painting.
Figure 7: Error that remain a challenge.
detect objects further to the right than the actual referent. One possible hypothesis is that the models are not able to understand the implicit proximity that the speaker is trying to communicate. When an expression of the form “X is to the right of Y” is given, a speaker implies that he is speaking of the closest object “to the right” of the landmark.
Interpreting Fig. 7a involves identifying the imaginary line formed by the three surfers, recognizing the blond one, and inferring the direction of his walk, which is referenced by the directional preposition 'towards'.
In a similar way as shown in Fig. 9, the barplot in Fig. 10 describes the distribution of the 15 most frequent skills combinations that made both models fail.
B.3 Errors that are not real errors
Fig. 8 shows how some predictions delivered by the models, despite not coinciding with the ground truth (depicted in blue), can in fact be correct.
In particular, Fig. 8a presents an ambiguous RE. Both the ground truth and the prediction by the green model are possible interpretations. The model identifies the face of the woman that is to the left and outside the group of the three women, which is a valid interpretation given the ambiguity of the input referential expression.
For Fig. 8b, the scenario is totally different. Here
(a) Face of woman, on the left of the group of 3.
(b) Donald duck above girl's head.
Figure 8: Error that are not real errors.
Figure 9: Distribution of the 15 most frequent skills combinations where the extended model improves over the baseline model.
Figure 10: Distribution of the 15 most frequent skill combinations where both the extended and the baseline model fail.
the error is more related to the way in which we consider a prediction to be correct, than to a mistake made by the model. Strictly speaking, it is due to the fact that the predicted area is much smaller than that indicated as ground truth, giving an IoU  $<< 50\%$ ; note that this decision method is mentioned as one of the limitations found in Sec. 8. The model ends up adjusting the Donald duck sticker more tightly than what is annotated as ground truth.
In order for our results to be comparable to previous work we did not modify the ground truth boxes.
C Detailed experimental setup
Images are first normalized by the mean and standard deviation of rgb values pre-computed on the ImageNet training set. We resize the images to  $512 \times 512$  pixels while keeping the original aspect ratio by fixing the longest side to 512 and zero
padding the shortest side accordingly.
For the visual encoder we use the pre-trained ResNet-50 (He et al., 2016) model available at the torchvision package from the PyTorch library (Paszke et al., 2019). This model has been trained for a 1000-way classification task on the ImageNet 2012 dataset (Russakovsky et al., 2015). We replace the output classification layer by a convolutional + normalization layer with  $D = 256$  output channels. In this case, given an input image of  $512 \times 512$  pixels, we obtain an output tensor of size  $16 \times 16 \times 256$ . We freeze the first convolutional layer of the network as well as the batch normalization layers. We apply random affine transformations (rotation, translation and scale) as the only augmentation strategy during training.
For the language encoder we use the bert-base-uncased pre-trained BERT (Devlin et al., 2019) model from the HuggingFace's Transformers library (Wolf et al., 2020). As with the visual encoder, we add a projection and normalization layer to project the embeddings output by the model to  $D = 256$  dimensions. We set a maximum expression length to 32 input tokens.
Our loss function takes the form:
$$
\mathcal {L} _ {\text {S o f t -} L _ {1}} + \gamma \mathcal {L} _ {\text {G I o U}} + \mu \mathcal {L} _ {\text {s e g m}}
$$
where the first and second terms act on the output cast by the box regression head while the third on the mask predicted by the segmentation head. We use  $\gamma = 0.1$  in all our experiments. We set  $\mu = 0$  for the baseline model and  $\mu = 0.1$  for the extended one.
We train our models for a maximum of  $E = 90$  epochs using the AdamW (Loshchilov and Hutter, 2018) optimizer with a multi-step decay schedule by a factor of 0.1 at the  $\lfloor 0.6E\rfloor$  and  $\lfloor 0.9E\rfloor$  epochs. Learning rate is set to  $1\times 10^{-4}$  for the whole model except for the visual and language backbones (ResNet and BERT) for which we use  $1\times 10^{-5}$ . Final models are chosen based on validation accuracy.
Table 4 show an extended view of the results presented in Table 3, including recognition performance on the validation subsets for all the datasets and the sample cardinality for the different expression types considered in the paper.
Model	Type	RefCOCO	RefCOCO+	RefCOCOg	ReferItGame
val	testA	testB	val	testA	testB	val	test	val	test
LBYL (Huang et al., 2021)	All	79.67	82.91	74.15	68.64	73.38	59.49	-	-	-	67.47
VGTR (Du et al., 2021)	All	79.20	82.32	73.78	63.91	70.09	56.61	65.73	67.23	-	-
TransVG (Deng et al., 2021)	All	81.02	82.72	78.35	64.82	70.70	56.94	68.67	67.73	-	70.73
Ref. Tr (Li and Sigal, 2021)	All	82.23	85.59	76.57	71.58	75.96	62.16	69.41	69.40	-	71.42
Baseline (M=0, SH=X)	All	81.07	84.85	74.72	69.93	75.95	59.36	70.14	69.40	69.62	66.76
Intrinsic	80.52	84.11	71.24	76.37	80.33	65.85	74.10	73.92	81.52	81.79
Spatial	81.33	85.22	75.42	62.95	70.55	53.32	69.47	68.44	62.92	58.73
Ordinal	64.49	75.91	48.67	35.29	50.00	40.62	56.52	46.25	34.44	28.47
Relational	70.85	77.03	57.39	60.56	67.11	48.02	68.81	67.87	48.51	44.61
Extended (M=8, SH=✓)	All	82.83 (+2.2%)	86.00 (+1.4%)	77.96 (+4.3%)	70.72 (+1.1%)	77.09 (+1.5%)	61.16 (+3.0%)	71.90 (+2.5%)	71.31 (+2.8%)	74.70 (+7.3%)	70.92 (+6.2%)
Intrinsic	82.01 (+1.9%)	83.94 (-0.2%)	72.32 (+1.5%)	77.34 (+1.3%)	81.18 (+1.1%)	68.25 (+3.6%)	76.53 (+3.3%)	75.25 (+1.8%)	85.15 (+4.5%)	83.94 (+2.6%)
Spatial	83.14 (+2.2%)	86.96 (+2.0%)	79.06 (+4.8%)	63.51 (+0.9%)	72.14 (+2.3%)	54.62 (+2.4%)	71.01 (+2.2%)	70.51 (+3.0%)	68.79 (+9.3%)	64.01 (+9.0%)
Ordinal	64.98 (+0.8%)	85.40 (+12.5%)	56.33 (+15.7%)	44.12 (+25.0%)	50.00 (+0.0%)	34.38 (-15.4%)	60.87 (+7.7%)	48.75 (+5.4%)	42.22 (+22.6%)	35.46 (+24.6%)
Relational	72.87 (+2.9%)	79.56 (+3.3%)	60.22 (+4.9%)	60.95 (+0.6%)	68.50 (+2.1%)	49.80 (+3.7%)	70.46 (+2.4%)	70.09 (+3.3%)	56.18 (+15.8%)	51.49 (+15.4%)
Sample size	All	10834	5657	5095	10758	5726	4889	4896	9602	6333	65193
Intrinsic	2613	1737	831	5657	3182	2378	865	1802	2290	22779
Spatial	8180	3903	4236	5031	2523	2470	4012	7763	4028	42277
Ordinal	414	137	300	34	4	32	46	80	90	1173
Relational	2031	1306	744	3362	1800	1464	3456	6651	1205	13154
Table 4: Comparison with other methods from the literature on the RefCOCO, RefCOCO+, RefCOCOg and ReferItGame datasets. For the baseline and the extended model, we consider performance for different expression types. Relative improvements of the extended vs. the baseline model (in %) are shown between parentheses. Last block shows sample cardinality for each expression type and data subset.