SlowGuess/NAACL_2022_OCR huggingface dataset:
Jorge Sánchez<sup>2</sup>, Mauricio Mazuecos<sup>1,2</sup>, Hernán Maina<sup>1,2</sup> and Luciana Benotti<sup>1,2</sup>
<sup>1</sup>Universidad Nacional de Córdoba
2CONICET, Argentina
{mmazuecos, hernan.maina}@mi.unc.edu.ar
{jorge.sanchez,luciana.benotti}@unc.edu.ar
Referring resolution is the task of identifying the referent of a natural language expression, for example "the woman behind the other woman getting a massage". In this paper we investigate which are the kinds of referring expressions on which current transformer based models fail. Motivated by this analysis we identify the weakening of the spatial natural constraints as one of its causes and propose a model that aims to restore it. We evaluate our proposed model on different datasets for the task showing improved performance on the most challenging kinds of referring expressions. Finally we present a thorough analysis of the kinds errors that are improved by the new model and those that are not and remain future challenges for the task.
In the context of vision and language modelling, reference resolution can be understood as the task of identifying a region in an image referred by a natural language expression. This task is also known as referring expression comprehension (REC). It is closely related to the visual grounding problem in the sense that in both cases the goal is to identify the parts of the image that support or "ground" a given linguistic expression into the real world. The difference lies in that the referred region in REC is expected to be unique, while there may be multiple support regions in the case of visual grounding, e.g. the expression "a person" might refer to multiple instances of the person class, all of which can be seen as grounding candidates for the natural language phrase. Also note that, in the case of REC, the referred region may correspond to an actual physical object (or groups of objects) or to an abstract visual element that can be perceptually grouped into a meaningful entity, e.g. "the patch of grass at the bottom". REC aims at identifying a specific object or region unambiguously (Mao et al., 2016; Hu et al., 2016; Qiao et al., 2020).
Although the use of natural language queries to guide localization (Bansal et al., 2018; Rahman et al., 2018) has been explored in the computer vision literature in the past, it has mainly focused on the use of rather simple expressions involving class names and intrinsic visual attributes (e.g. colors). Humans use more complex expressions that include spatial and order relations, relative attributes, meronymy, etc. Compared to other recognition problems, REC goes beyond simple visual-linguistic matching and requires some (primitive, implicit) form of visual-linguistic reasoning, e.g. an expression like "the person to the left of the tree" requires that in order to locate the target object ("the person") one has to look at its relative position ("to the left of") with respect to a different element in the scene ("the tree"). REC and visual grounding are particularly relevant to other visual-linguistic problems like visual question answering (Antol et al., 2015) and visual dialog (Das et al., 2017a; De Vries et al., 2017), where being able to link different linguistic elements (words, phrases and syntactic relations) in a sentence to actual regions in an image helps establishing a "common ground" between the agents that take part in the communication process (Mazuecos et al., 2021).
Our research question is what kinds of errors do reference resolution models make and what can we learn from them?. This paper makes the following contributions<sup>1</sup>.
- We propose a method for classifying referring expressions into linguistically motivated groups that allow for a disaggregated analysis.
- We identify expressions that define an spatial relation between two or more regions of the image as the main source of errors.
- We compare two strategies for accounting for the spatial dimension that improve on the state of the
1Code and models available at https://github. com/jadrs/rec
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1971 - 1986
art for REC on different datasets.
- We perform a systematic analysis of the errors and the strengths of the best proposed model.
The paper is structured as follows. In Section 2 we discuss related work from both psycholinguistics and machine learning, Section 3 digs into the particularities of the REC and visual grounding problems and analyzes the types of errors made by current models for the REC task. In Section 4 we propose a model that takes the observed errors into account and Section 5 evaluates the new model on different datasets. Section 6 presents a meticulous analysis of the errors that are improved by the new model and those that are not and remain future challenges.
The automatic processing of referring expressions (REs) has been studied for a long time (Winograd, 1972). Back then their semantic representation was a logical form and REs were classified into intrinsic (e.g. "the big red car") and relational (e.g. "the car by the pedestrian") corresponding to unary and binary logical predicates, respectively (Dale and Haddock, 1991).
In the psycholinguistics community, reference were studied as a collaborative process and focused on the construction of shared knowledge (Clark and Wilkes-Gibbs, 1986; Clark, 1996) through the use of language. Hawkins (1978) proposed a theory in which an speaker 1) introduces a referent, 2) collaborates with the hearer to locate the referred object in some shared set of objects and 3) refers to the totality of objects that satisfy the RE. Viewed as a collaborative process, reference becomes a fundamental phenomena underlying all kinds of grounded dialogs (Dale and Reiter, 1995; de Vries et al., 2017).
REs are a linguistically rich constructs. Phenomena like overspecification, that is when an RE has more attributes than actually needed, has been shown to help identification when the redundant attribute is easy to recognize (Paraboni et al., 2017). Vagueness and the use of gradable attributes emphasize the context dependence when using REs (Quirk et al., 1980; DeVault and Stone, 2004); something called "big" in one scene may be seen as "small" in another. REs with syntactic ambiguities were also shown to be used in human dialogs (Chantree et al., 2005; Khan et al., 2008).
Viethen and Dale (2008) showed that even with fairly simple scenes human speakers frequently use relational descriptions to identify objects.
In this paper we adapt the classification of referring expressions (REs) proposed in (Krahmer and van Deemter, 2012) to the domain of 2D photos of the world. Krahmer and van Deemter typification includes the types: intrinsic (that they call unary predicates), relational (that they call binary predicates), set (that refer to a group of objects), and gradable (that we described above). Intrinsic and relational REs differ in the number of objects that are involved in the description. Intrinsic REs only involve the referent while relational REs use one or more additional objects to identify the target. Referring expressions that identify sets use properties that are shared by the elements in the set. For example the referents of "the white cats" are all cats and white. In general references to set use plural definite descriptions to identify them. The datasets used in this paper are supposed to refer to a single referent and not to a set of referents so we restrict our classification in this paper to singular REs. Finally, Krahmer and van Deemter last type corresponds to gradable REs. REs referring to objects in 2D photos of the world frequently use spatial properties that are gradable. For example, the attribute to the right of is gradable in that "the empty sky to the right of the statue" might refer to the sky touching the statue on the right or the sky further to the right. In this paper we restrict our analysis of gradable properties to spatial properties.
We agree with Cirik et al. (2018) that careful analysis of datasets and proposed models is crucial to make progress in REC. They performed an analysis of REC models by modifying or completely removing the REs and showed that the models could exploit biases on particular datasets to achieve competitive performance. In our work we do a disaggregated performance analysis and meticulous error analysis and not only rely on automatic performance metrics.
The introduction of the transformer architecture by Vaswani et al. (2017) enabled interesting and novel ways of fusing visual and linguistic information (Tan and Bansal, 2019; Lu et al., 2019, 2020). Besides differences on the way both modalities are merged (single- vs two-branch models, cross-modal attention layer design, etc.), these models also differ on the tasks they are (pre-)trained for, ranging from masked token prediction and text
image matching (Li et al., 2020a; Sun et al., 2019) to more elaborated strategies such as label-region alignments (Li et al., 2020b; Guo et al., 2020) or scene-graph prediction (Yu et al., 2020). Most of these tasks can be seen as local (word-to-region) or global (text-to-image) matching tasks and, although effective for pre-training, they do not contemplate other phenomena such as composition and indirection that can be observed in grounding problems. Although large-scale pre-training has been shown to be effective for grounding (Kamath et al., 2021; Li and Sigal, 2021), such methods require the availability of large collections of image-text pairs with explicit alignments between image regions and phrases in text. Because of the scale of such data and the cost of extensive annotations, these approaches have not gone beyond image-text matching phenomena.
In this work, we take inspiration from a recently proposed family of transformer-based architectures that tackle the REC problem as a regression problem, i.e. given an image and a query expression, directly predict the bounding box coordinates of the object or image region referred by it (Deng et al., 2021; Du et al., 2021).
3 Problem definition and motivation
Given an image and a natural language expression, the goal of REC is to predict the location of the referred object or region, e.g. by predicting the coordinates of the bounding box that encloses it more tightly. The expression may include diverse linguistic constructs such as ellipsis, prepositional phrases, conjunctions, etc.
In what follows, we first introduce a simplified yet performant version of the model proposed by Deng et al. (2021). This model will serve as a strong baseline in our experiments because it allows us to evaluate the differences in performance between the baseline and the proposed extensions. Next, we propose a linguistically motivated classification scheme which allows for a disaggregated analysis that will guide the rest of the paper. Finally, we discuss some motivating results in light of our baseline and different types of expressions.
3.1 A strong baseline model
As in (Deng et al., 2021), the input to our model is an image  $I$  and a RE  $e$ . Its output are the bounding box coordinates of the referred object or region. The model consists on the following blocks: a im
age encoder, a language encoder, a transformer-based cross-modal encoder and a box prediction head. An outline of the architecture is shown in Fig. 1.
Image encoder. We feed the image to a pretrained convolutional backbone. We use a ResNet-50 (He et al., 2016) pre-trained on ImageNet with the classification head removed. The output of this network is a feature map of size  $H \times W$  and  $d$  channels, that we further project to  $D$  dimensions using  $1 \times 1$  convolutions. Different from Deng et al. (2021), we do not add any additional transformer layer on top of the convolutional encoder. We flatten the resulting tensor along the spatial dimension and obtain a sequence of  $HW$  visual embeddings of dimensionality  $D$ .
Language encoder. We first map  $e$  into a sequence of  $T$  tokens and encode it using a pretrained language model. We use a pre-trained BERT (Devlin et al., 2019) as the default encoder. The output sequence of (sub-) word embeddings is projected onto  $D$  dimensions using a fully-connected layer.
Cross-modal encoder. We concatenate both visual and language embeddings into a multimodal sequence and feed it to a cross-modal encoder consisting of a transformer architecture with  $L$  layers. Each layer in the encoder corresponds to a multi-head self-attention layer with skip connections (Vaswani et al., 2017). As in (Deng et al., 2021), we add learnable position embeddings to the input of each transformer encoder layer. The output of the cross-modal encoder is a sequence of embeddings of the same length as the input and whose elements can be seen as a re-encoding of the corresponding input embeddings. This re-encoding mechanism is based on the "similarity" between other elements of the same sequence (visual and/or linguistic) as induced by the multi-head self-attention layers within the cross-modal encoder.
Prediction head. The box prediction head is a single fully connected layer followed by a sigmoid. The output of this head corresponds to the normalized coordinates of the target bounding box. Instead of adding a specialized output token as in (Deng et al., 2021), we take as input the average of the first  $HW$  embeddings after the cross-modal encoder, i.e. those corresponding to the visual block. Given a training set of image-expression-box tu
Figure 1: Our baseline model for referring expression comprehension showing its main processing blocks: image and language encoders (in blue and green respectively), cross-modal fusion (in orange) and box prediction head (in gray). The input is an image and a referring expression and the output is a bounding box.
plies  $\mathcal{D} = \{(I_n, e_n, b_n), i = 1, \dots, N\}$ , we adopt a loss formulation based on a combination of the generalized IoU of Rezatofighi et al. (2019) and the soft  $L_1$  loss as in Deng et al. (2021).
3.2 Types of referring expressions and errors
One of the challenges posed by the REC problem is that natural language expressions exhibit different degrees of grounding complexity. In order to better understand the limitations of current models, and motivated by previous work described in Section 2, we propose to classify them into a non-disjoint set of types as follows.
- Spatial. Expressions that include spatial language e.g. "the doorway on the far right". They include prepositions that signals a spatial relation (e.g. behind) or adjectives (e.g. left) and nouns (e.g. foreground) that give spatial cues.2
- Ordinal. Expressions that include ordinal adjectives that determine the position of an object inside a group, e.g. "2nd set of jewels".
- Relational. Expressions that use another object, which is related to the referent, in the RE; e.g. "the instrument right behind the guy with yellow hat". Relational expressions are spatial when the relation is spatial.<sup>3</sup>
- Intrinsic: Expressions that do not fall into any of the above types. They do not use the position of the object to describe it, instead they use only properties that are intrinsic to the referent no matter its position; e.g. "the tall metal fence".
Type	# test	expr len	acc
All	65193	3.5 (2.6)	66.76
Intrinsic	22779	1.5 (1.0)	81.81
Spatial	42277	4.6 (2.6)	58.73
Ordinal	1173	5.9 (2.8)	28.47
Relational	13154	6.7 (3.0)	44.59
Table 1: Accuracy disaggregated by expression type for the RefItGame dataset. The expr len column shows the average expression length and its standard deviation (between parentheses). The last column of the table shows the accuracy for the baseline model on the ReferItGame test set. An RE may fall in more than one type, in that case it is counted in all the types.
Table 1 shows the frequency and lengths of the different types of REs in the Refer-ItGame (Kazemzadeh et al., 2014) dataset as well as the performance of the baseline model introduced in Sec. 3.1. The performance reported is the accuracy of the predicted bounding box. It is considered that the bounding box is correct if it overlaps with more than  $50\%$  of the ground truth one. The table shows that the accuracy on the spatial class is considerably lower than the accuracy for the intrinsic class although spatial expressions are more frequent. The accuracy is even lower for the relational and ordinal expressions. After performing this disaggregated analysis our intuition is that the position embeddings used in the state of the art models are not enough for capturing the spatial and relational information required to handle this type of expressions.
The Fig. 2 shows the cardinality of the intersections of the types as a matrix. Each row at the bottom corresponds to a type. Each column corresponds to a non empty set, and the bars at the top
2A complete list of spatial words used in this paper is in Appendix A.   
3We say an expression is relational if it contains a preposition with one or more nouns to the left and to the right.
show the size of the respective intersections. The filled dots show which type is part of an intersection. The first four columns represent those REs that fall in only one type. The last three columns show combinations of types.
The intrinsic type is disjoint from all others by definition. Almost all ordinal REs are also spatial or relational. There are very few relational REs that are not spatial (an example of such rare RE is "the girl sleeping with the teddy bear"). Most relational REs use an spatial preposition in the the ReferItGame dataset.
4 Restoring referential spatiality
The types of expressions that have the worst accuracy, relational and ordinal, can be associated to limitations of the model in capturing richer spatial relations that go beyond simple matching (names to image and absolute locations to image). In what follows, we propose two generic strategies to inject stronger visual priors into the model after the cross-modal fusion block. In Fig. 3 we illustrate how we reformulate the prediction head of the original model in Fig. 1. We describe the modified architecture below.
First, due to the one-to-one alignment of the embedding sequence at the input and output of the transformer, we can identify the first block of  $HW$  elements as a re-encoding of the original visual embeddings, modulated by the input expression  $e$ . We can rearrange this block as a tensor of size  $H \times W \times D$  so as to restore the spatial structure lost after the flattening operation. This operation is
Figure 2: The RE types proposed in this paper are not disjoint. Their distribution and intersections on the ReferItGame dataset are shown in the columns. The first four columns correspond to REs that belong to only one type.
Figure 3: Prediction head for the extended model. This new prediction head replaces the one from Fig. 1 with an "unflatten" operation and a block of convolutions followed by the regression and segmentation heads. The regression head outputs a bounding box (in yellow) while the segmentation head outputs a mask (in black and white).
represented as the unflatten block in Fig. 3.
We next add a small convolutional network with  $M$  layers (convolution layer with a  $3 \times 3$  kernel and stride of 1, batch normalization and a ReLU nonlinearity), on top of which we attach two different heads: a target regression head as before and a box segmentation head, as shown in Fig. 3.
The box regression head is the same as the base model but applied to the average pooled features after the convolutional block, while the additional segmentation head consists of a simple convolutional layer whose output is normalized to the interval [0, 1] by a sigmoid activation. The goal of this head is to provide a spatial consistency constraint while avoiding the need of requiring additional data. Supervisory signal for this branch is obtained trivially from the existing annotations, i.e. by creating a binary mask from the bounding box coordinates of the target. Learning is formulated by adding an auxiliary pixel-wise binary classification term to the main loss.
The intuition behind adding  $M$  convolutional blocks after the cross-modal transformer as well as the mask prediction head is to ensure that the spatial consistency of the representation that has been broken is restored. The locality and hierarchical nature of the (stacked) convolutions as well as the complementarity of the segmentation and regression heads proves to be an effective approach to improve the accuracy of some kinds of spatial expressions as we will see in the next two sections.
In the following we describe the datasets and experimental setup we use in our experiments. A
detailed description can be found in Appendix C.
We conduct experiments on four different datasets.
ReferItGame. We use this dataset as a set for validation and error analysis. ReferItGame (Kazemzadeh et al., 2014) was collected based on a two-player game where one of the players has to write a RE based on a given object while the second player has to guess its identity by looking only at the image and the generated expression. If guessed correctly, both players receive a game point and swap their roles for the next image.
Images in this dataset are from the IAPR TC-12 corpus (Escalante et al., 2010). The dataset consists of 130121 expressions referring to 96654 different objects across 19894 images. We use a cleaned version of the dataset provided by Hu et al. (2016) and standard splits, i.e. 54127, 5842 and 60103 expressions for training, validation and testing, respectively.
RefCOCO and RefCOCO+. These datasets (Nagaraja et al., 2016) were collected by following the same procedure as in ReferItGame, but using images from MS COCO (Lin et al., 2014). RefCOCO contains 50000 referred objects in 19994 images. Each object is referred by an average of three expressions, for a total of 142210 REs split across 120624 expressions for training, 10834 for validation and two additional splits (testA and testB) with 5657 and 5095 expressions for testing, respectively. The testA split contains multiple people while the testB split contain multiple instances of all other objects.
RefCOCOg. RefCOCOg (Mao et al., 2016) was collected non-interactively using Amazon Mechanical Turk. This dataset contains 104560 expressions for 54822 objects in 26711 images. Compared with RefCOCO and RefCOCO+, expressions in the RefCOCOg dataset are considerably longer (an average of 8.43 vs. 3.61 and 3.53 words, respectively). We use the split proposed by (Nagaraja et al., 2016) for meaningful comparisons with other methods.
We report REC performance using average accuracy. We consider a region prediction as correct if it has an overlap (as measured by the IoU metric) of at least 0.5 with the ground truth box.
SH	M=0	2	4	8
All	X	71.66	72.64	73.00	73.46
✓	71.89	73.03	73.16	73.61
Intrinsic	X	84.76	84.63	84.93	84.10
✓	83.79	84.93	84.14	84.27
Spatial	X	64.28	65.91	66.53	67.36
✓	65.18	66.27	66.74	67.73
Ordinal	X	34.44	42.22	42.22	46.67
✓	36.67	45.56	41.11	45.56
Relational	X	51.29	51.70	52.70	53.57
✓	51.74	52.99	52.99	54.11
Table 2: REC performance on the validation set of the ReferItGame dataset for different expression types and model configurations.  $M$  denotes the number of convolutional blocks while "SH" denotes whether we use the segmentation head or not.  $\mathrm{M} = 0,\mathrm{{SH}} = x$  corresponds to the baseline model outlined in Sec. 3.1.
5.2 Architecture selection