SlowGuess/NAACL_2022_OCR huggingface dataset:
In this section we evaluate the impact on performance of the changes proposed in Sec. 4 for the different types of REs discussed in Sec. 3.2. For these experiments, we set the maximum number of epochs to  $E = 60$ . Table 2 shows comprehension performance on the validation set of the ReferItGame dataset for different model configurations and expression types.  $M$  denotes the number of convolutional blocks and "SH" denotes whether we use the segmentation head or not. In this case,  $\mathbf{M} = 0$  and no segmentation head  $(\mathrm{SH} = \mathbf{x})$  corresponds to the baseline model outlined in Sec. 3.1. First, consider the case  $M = 0$ , i.e. no convolutional blocks after the cross-modal encoder. From the table, we see that adding a segmentation head improves performance overall, specially for the expressions that involve some degree of spatial reasoning. For the intrinsic type, improvement is marginal as the model is already able to solve the region-(class)target alignment problem. For more complex expressions, adding a segmentation head constraints the model to focus on the target location (and scale) and helps disambiguate references to objects/regions from its context. If we now consider  $M > 1$ , we observe the following. First, performance improves for all expression types, specially for ordinal and relational w.r.t. the model with  $M = 0$ . Second, for  $M > 1$ , adding a segmentation head seems to have no impact on performance. This can be attributed to a greater flexibility
of the stack of convolutions in capturing spatial and relational information.
5.3 Comparison with the state-of-the-art
Next, we compare the performance of our models against four different methods proposed recently in the literature, namely: LBYL-Net (Huang et al., 2021), VGTR (Du et al., 2021), TransVG (Deng et al., 2021) and the Referring Transformer (Ref. Tr.) (Li and Sigal, 2021). LBYL-Net is a one-stage grounding model based on modeling spatial relations between the referent and its context via a suitable convolution operator. VGTR is a transformer-based one-stage model following an encoder-decoder design and custom grounding modules. TransVG is similar to our baseline model with an additional transformer after the visual backbone and a specific output embedding that feeds the prediction head. Finally, the referring transformer model follows a similar design as DETR (Carion et al., 2020) while tackling simultaneously the RE comprehension and segmentation problems. We show performance for each model and relative improvement of the extended model with respect to the baseline in Table 3. All these models rely on a ResNet-101 as visual backbone, except LBYL which uses darknet-53.
In this paper, we presented two different models: a baseline described in Sec. 3 and a extended model that incorporates  $M = 8$  convolutional layers and the segmentation head explained in Sec. 4. We use a ResNet-50 as visual backbone and the same set of hyper-parameters and training procedure as before, explicitly avoiding dataset specific finetunings. We disaggregate the performance according to the different types of expressions for both the baseline and the extended model.
First, we see that our baseline model is a strong baseline for REC. If we compare average performance (rows "All" in Table 3), we see that our baseline model performs comparably to the best performant methods in the first group. It shows the second best performance on both RefCOCO testA and testB subsets, second and third best performance in RefCOCO+ testA and testB, respectively; and achieves top performance on the RefCOCOg test set. If we consider the extended model, we observe a consistent overall improvement on all datasets. Although our goal is not to get the best possible performance but to highlight the importance of the different expression types when eval
uating RE models, results in the table show that our design is on pair with the state of the art. This is important since both our models follow a simple design and rely on the same training protocol, which is compared to the more complex backbones and per-dataset tuning of hyperparameters of the methods in the first group.
If we compare disaggregated performance for the baseline and extended models, we observe the following. From Table 3, performance improves on all subsets and expression types, with the only exception of the intrinsic and ordinal types in the testA and testB subsets of RefCOCO and RefCOCO+ datasets, respectively. For RefCOCO, this accounts for a  $-0.2\%$  decrease on performance w.r.t. to the baseline. For RefCOCO+, the difference is greater  $(-15\%)$  w.r.t. to the baseline). Note however that for this dataset, the number of ordinal expressions is 34, 4 and 32 for the val, testA and testB subsets. The observed decrease corresponds to a difference of only 2 examples.
In general, we observe a greater improvement for expressions that involve spatial (spatial and relational) and grouping (ordinal).
A detailed summary of these results, including performance on the validation sets and sample cardinalities for all subsets and expression types can be found in Appendix D.
In this section we analyze the predicted output of the extended and baseline models for the image-expression pairs of the ReferItGame validation set. The figures in this section show the extended model prediction in green and the baseline model prediction in orange. The comparative error analysis goal is to shed light over the kinds of linguistic and visual phenomena that the models are able to handle and those they are not. The analysis was divided into two parts and was carried out on a total of 351 examples. First, we explored referring expressions where the green model correctly predicted the ground truth and improved over the baseline depicted in orange. Two such examples are in Fig. 4. Second, we analysed cases where both models failed, exemplified in Fig. 5. More examples are shown in in Appendix B.
We identified different abilities a model should have in order correctly solve a broader set of expressions. They are listed below. Fig. 4 and Fig. 5 illustrate each of the skills.
Model	Type	RefCOCO	RefCOCO+	RefCOCOg
testA	testB	testA	testB	test	test
LBYL	All	82.91	74.15	73.38	59.49	-
VGTR	All	82.32	73.78	70.09	56.61	67.23
TransVG	All	82.72	78.35	70.70	56.94	67.73
Ref. Tr	All	85.59	76.57	75.96	62.16	69.40
Baseline (M=0, SH=X)	All	84.85	74.72	75.95	59.36	69.40
Intrinsic	84.11	71.24	80.33	65.85	73.92
Spatial	85.22	75.42	70.55	53.32	68.44
Ordinal	75.91	48.67	50.00	40.62	46.25
Relational	77.03	57.39	67.11	48.02	67.87
Extended (M=8, SH=✓)	All	86.00 (+1.4%)	77.96 (+4.3%)	77.09 (+1.5%)	61.16 (+3.0%)	71.31 (+2.8%)
Intrinsic	83.94 (-0.2%)	72.32 (+1.5%)	81.18 (+1.1%)	68.25 (+3.6%)	75.25 (+1.8%)
Spatial	86.96 (+2.0%)	79.06 (+4.8%)	72.14 (+2.3%)	54.62 (+2.4%)	70.51 (+3.0%)
Ordinal	85.40 (+12.5%)	56.33 (+15.7%)	50.00 (+0.0%)	34.38 (-15.4%)	48.75 (+5.4%)
Relational	79.56 (+3.3%)	60.22 (+4.9%)	68.50 (+2.1%)	49.80 (+3.7%)	70.09 (+3.3%)
Table 3: Comparison with other methods from the literature on the RefCOCO, RefCOCO+ and RefCOCOg datasets. For the baseline and the extended model, we consider performance for different expression types. Relative improvement percentage of the extended model with respect to the baseline are shown in parentheses.
(a) Sky near clouds
(b) Water near man's back
Figure 4: Examples where our proposed model (green) improves over the baseline (orange). The ground truth is shown as a blue box. The first example is classified as fuzzy objects. The second as meronymy.
(a) The empty  $ski$  to the right of the statue
(b) Grass directly above and behind little boy on bike
Figure 5: Examples that remain a challenge for both models (ours in green, baseline in orange, ground truth in blue). The first example is classified as fuzzy objects, typo and directional. The second is classified as fuzzy objects, viewpoint, and implicit.
- Meronimy: Reference parts of objects, such as peoples' clothes, parts of the body (e.g. "the man's back") or parts of inanimate objects.
- **Viewpoint:** In order to resolve the reference, the hearer needs to assume the point of view of an object such as "behind the little boy".
- Directional: the expressions contain the direction in which we can find the referent relative to the location of the landmark, e.g. "to the right of the statue".
- Fuzzy objects: Reference to regions of objects without prototypical shapes and borders, like "ground", "water" or "sky".
- Occlusion: Reference to an object that is only partly visible because it is hidden by another object or does not fit completely in the image.
- Tipo: the expression contains a typo that can confuse the understanding of the referring expression (e.g. "ski" instead of "sky").
- Implicit: the RE contains an ellipsis such as "grass directly above" meaning "grass directly above the path".
The first three types of errors require the RE to be relational. Meronimic errors require a relation between an object and a part of it. Viewpoint errors occur in REs that are not only relational, but also
spatial in general. They use a landmark, related to the referent, to change the point of view of the interpreter. Directional errors can occur when the direction of the relation in the RE is misinterpreted. Frequently this relation is spatial but not necessarily, it could correspond to an order established by other property, such as size (e.g. "from the smallest to the biggest"). The last four types of errors may happen for all kinds of REs. Although implicit errors are more frequent in long REs such as relational REs.
6.1 Errors that are improved
During our analysis we identified that most of the cases in which the green model improves over the orange one contain meronimy, fuzzy objects or both. Fig. 4a is an example of fuzzy object because the sky is a kind of object that does not have a prototypical shape and borders (such as a person) and whose parts can be spread in different areas of the image. Our model is not only able to select a region near the clouds but it is also able to constrain its prediction using the natural boundary that is the tree on the left. The model is able to reconstruct the relative spatial position of the different parts of the sky.
Fig. 4b shows an example of meronomy. The "man's back" refers to a part of the man. Correctly predicting the referred region in this expression requires both identifying the back of the man (not the whole man) and the appropriate region of water (another instance of fuzzy objects). As the water and the sky, the man's back does not have a clear border wrt the rest of his body.
6.2 Errors that remain a challenge.
We analyzed 220 examples in which both models failed. The first observation that we found is that over  $66\%$  of the errors require more than one skill. This is only  $38\%$  for the 131 examples we annotated where the green model improves over the orange. Fig. 5 illustrates cases that require multiple skills, Fig. 5a not only has a typo (ski should be sky) but it also includes a directional relation (to the right). We can see how a tiny error in a character of the word sky, confuses the models and both predict the section of fences located below in the image, probably because they look like a set of 'ski's. Fig. 5b not only has fuzzy objects because of the grass but it also requires the interpretation of the viewpoint and implicit language for resolution. In order to identify the referent, the
Interpreter needs to take the viewpoint of the boy in the picture to correctly interpret the relation behind as referring to the blue box. The interpreter also needs to realize that the relation above implicitly means above the bike path.
Our second finding is that some kinds of meronomy remain a challenge and constitute  $12\%$  of the examples we annotated. We find that the most challenging meronomic relations are those that are not frequent in the training data (e.g. "the eyebrows of the person"). Similarly, some directional relations (that amount to  $17\%$  of the errors) are more challenging than others: those relations that are normally in the z-axis (e.g. "behind") lead to more errors that those in the y-axis (e.g. "above"). A complete distribution of the annotations can be found in Appendix B. Summing up, there is a lot of room for improving the grounding skills of REC models. We have identified that the main challenges present in the ReferItGame dataset are those that we defined as rare meronomic relations, viewpoint, directional, occlusion, implicit language and typos.
In this work we studied the kinds of errors that reference resolution models make. In particular, relational expressions caused a lot of errors in REC and that motivated the proposal of a model that improves over the previous SOTA for the task by restoring the referential spaciality. We used a new training objective which is segmentation prediction and added convolutional layers to a transformer.
Looking at accuracy only can obscure and hide common errors these model might have. We performed an error analysis and identified which skills the model would need in order to perform correctly. We found that our proposed model improves in dealing with fuzzy objects and meronomy, but still finds it difficult with other skills. By performing this error analysis we learnt there is yet a lot of work to do in making models consider viewpoint or being able to deal with implicit information (conveyed by the common visual context).
Our findings can help have a finer grained look at the predictions of models. This can be relevant for different areas of NLP like grounding, situated dialog systems and human-computer interaction as referring is a crucial skill in communication.
8 Ethical considerations
REC models predict bounding boxes instead of segmentation masks. Bounding boxes can include a lot of background information for some kinds of objects (imagine a broom at a 45 degrees angle). Such segmentation masks ground truths are expensive to annotate.
As in previous work, we count a prediction as correct if its IoU with the ground truth is above  $50\%$ . This binarization can obscure the quantitative analysis in border cases (49% vs 51% IoU).
Regarding datasets, we did not collect the datasets but used available ones for the task. Crowdsourcing raises ethical concerns including fair wage for crowdworkers, work load and exhaustion. In our qualitative analysis we could find examples of exhaustion in the linguistic production of crowdworkers.
Previous work, coming from the field of collaborative reference resolution, state that one of the "desired" applications was helping with surveillance systems (Li et al., 2017; Das et al., 2017b). We do not agree with this use of the technology. Despite every work in referring expressions inevitably helping towards that goal it would need retraining with specific domain data for that. Our proposed model and formulation is not aimed at surveillance nor the datasets used and should perform poorly in such setting.
We are reporting results over 30 experiments in total. Each experiment was running for 2.5 days on a single 1080ti GPU. We estimate  $6.48kgCO_{2}eq$ . for each experiment according to local emissions factor. Debugging, code refactoring and validation runs took around a couple of hundred additional runs.
We thank all anonymous reviewers and area chair for their insightful comments and suggestions. This work used computational resources from CCAD-UNC, which is part of SNCAD-MinCyT, Argentina.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In ICCV, pages 2425-2433.
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. 2018. Zero-shot object detection. In ECCV, pages 384-400.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. Springer.
F. Chantree, A. Kilgarriff, A. De Roeck, and A. Willis. 2005. Disambiguating coordinations using word distribution information. In Proceedings of Recent Advances in Natural Language Processing - 2005.
Volkan Cirik, Louis-Philippe Morency, and Taylor Berg-Kirk. 2018. Visual referring expression recognition: What do systems actually learn? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 781-787, New Orleans, Louisiana. Association for Computational Linguistics.
Herbert H. Clark. 1996. Using Language. 'Using' Linguistic Books. Cambridge University Press.
Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1 - 39.
Robert Dale and Nicholas Haddock. 1991. Generating referring expressions involving relations. In Fifth Conference of the European Chapter of the Association for Computational Linguistics, Berlin, Germany. Association for Computational Linguistics.
Robert Dale and Ehud Reiter. 1995. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Science, 19(2):233-263.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017a. Visual dialog. In CVPR, pages 326-335.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. 2017b. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 326-335.
Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In CVPR, pages 5503-5512.
Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron C. Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 4466-4475. IEEE Computer Society.
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. 2021. TransVG: End-to-end visual grounding with transformers. arXiv preprint arXiv:2104.08541.
David DeVault and Matthew Stone. 2004. Interpreting vague utterances in context. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 1247-1253, Geneva, Switzerland. COLING.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1).
Ye Du, Zehua Fu, Qingjie Liu, and Yunhong Wang. 2021. Visual grounding with transformers. arXiv preprint arXiv:2105.04281.
Hugo Jair Escalante, Carlos A Hernández, Jesus A González, Aurelio López-López, Manuel Montes, Eduardo F Morales, L Enrique Sucar, Luis Villasenor, and Michael Grubinger. 2010. The segmented and annotated iapr tc-12 benchmark. Computer vision and image understanding, 114(4):419-428.
Jia Guo, Chen Zhu, Yilun Zhao, Heda Wang, Yao Hu, Xiaofei He, and Deng Cai. 2020. Lamp: Label augmented multimodal pretraining. arXiv preprint arXiv:2012.04446.
J. Hawkins. 1978. Definiteness and Indefiniteness: A Study in Reference and Grammaticality Prediction. Routledge.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In ICCV, pages 770-778.
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natural language object retrieval. In CVPR, pages 4555-4564.
Binbin Huang, Dongze Lian, Weixin Luo, and Shenghua Gao. 2021. Look before you leap: Learning landmark features for one-stage visual grounding. In CVPR, pages 16888-16897.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021. MDETR-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. ReferItGame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 787-798, Doha, Qatar. Association for Computational Linguistics.
Imtiaz Hussain Khan, Kees van Deemter, and Graeme Ritchie. 2008. Generation of referring expressions: Managing structural ambiguities. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 433-440, Manchester, UK. Coling 2008 Organizing Committee.
Emiel Krahmer and Kees van Deemter. 2012. Computational generation of referring expressions: A survey. Computational Linguistics, 38(1):173-218.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020a. What does BERT with vision look at? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5265–5275, Online. Association for Computational Linguistics.
Muchen Li and Leonid Sigal. 2021. Referring transformer: A one-step approach to multi-task visual grounding. arXiv preprint arXiv:2106.03089.
Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang. 2017. Person search with natural language description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020b. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pages 121-137. Springer.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Ken Litkowski. 2014. Pattern dictionary of english prepositions. In ACL (1), pages 1274-1283.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020. 12-in-1: Multi-task vision and language representation learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11-20.