LUNA: Language as Continuing Anchors for Referring Expression Comprehension
Authors: Yaoyuan Liang
Zhao Yang
Yansong Tang
Jiashuo Fan
Jingang Wang
Wei Wu
Philip Torr
Shao-Lun Huang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Referring expression comprehension aims to localize the description of a natural language expression in an image. Using location priors to remedy inaccuracies in cross-modal alignments is the state of the art for CNN-based methods tackling this problem. Recent Transformer-based models cast aside this idea making the case for steering away from hand-designed components. In this work, we propose LUNA, which uses language as continuing anchors to guide box prediction in a Transformer decoder, and show that language-guided location priors can be effectively exploited in a Transformer-based architecture. Specifically, we first initialize an anchor box from the input expression via a small “proto-decoder”, and then use this anchor as location prior in a modified Transformer decoder for predicting the bounding box. Iterating through each decoder layer, the anchor box is first used as a query for pooling multi-modal context, and then updated based on pooled context. This approach allows the decoder to focus selectively on one part of the scene at a time, which reduces noise in multi-modal context and leads to more accurate box predictions. Our method outperforms existing state-of-the-art methods on the challenging datasets of ReferIt Game, RefCOCO/+/g, and Flickr30K Entities.

Under review as a conference paper at ICLR 2023
LUNA: LANGUAGE AS CONTINUING ANCHORS FOR REFERRING EXPRESSION COMPREHENSION
Anonymous authors Paper under double-blind review
ABSTRACT
Referring expression comprehension aims to localize the description of natural language expression in an image. Using location priors to remedy inaccuracies in cross-modal alignments is the state of the art for CNN-based methods tackling this problem. Recent Transformer-basee models cast aside this idea making the case for steering away from hand deednnd components. In this work, pro- pose LUNA, which uses language as continuing anchors to guide box prediction in Transformer decoder, and show that language-guin -ugddgeggdeed location priors be effectively exploited in a Transformer- based architecture. Specifically, we first initialize an anchor box from the input expression via small "proto-decoder", and then use this anchor as location prior in a modified Transformer decoder for predicting the bounding box. Iterating through each decoder layer, the anchor box is first used as a query for pooling multi-modal context, and then updated based on pooled context. This approach allows the decoder focus selectively on one part of the scene at time, which reduces noise in multi-modal context and leads to more accurate box predictions. Our method outperforms existing state- of- of-the-art art methods on the challenging datasets of ReferIt Game, RefCOCO/+/g. and Flickr30K Entities.
INTRODUCTION
Referring expression comprehension (REC) is the task of localizing natural language in images, where expression in plain text describes single group of objects from image, and the objective is to put a bounding box around the target. It provides fundamental values to real-world applications such robotics Tellex et al. 2020), image editing (Shi et al. (2021), and surveillance Li ltt al. (017). With purposes of reducing the search space and mitigating difficulties cross-modal context modeling, early methods rely on prior knowledge about the "likely" locations of the target.
Specifically, region proposals and anchor boxes are the two most common types of location priors. Two-stage models [uu et al. (2017): Wang al. (2018); Yu al. (2018a b): Yang et al. (2019a) leverage region proposals, consisting of thousands of massively overlapping boxes extracted using standalone proposal method Uijlings al. (0113; Zitnick Dollár (2014): Ren al. (015)).
A cross-modal similarity-based ranking method is used to select one proposal as the prediction. Such models cannot recover from proposal failures and generally suffer a low recall rate Yang et al. (2019)). On the other hand, one-stage models Yang et al.) (201b)) 2020); Luo et al. (2020) leverage dense anchor boxes defined over image locations and directly predict a bounding box from inte- grated
feature maps. This approach allows box regression to be conditioned on the input expression, thus addressing the aforementioned issue of two-stage models. The definitions of anchors however, are heuristic and greatly influence model performance.
The most recent Transformer-basee methods Deng et al. (202T): Li & Sigal (021) discard location priors and leverage the powerful correlation modeling power of the Transformer architecture Devlin et al. (2019); Vaswani et al. (2077) for end-to-end prediction. Typically, a Transformer encoder De- vlin et al. (2019) jointly embeds visual and linguistic inputs, and box prediction is made from context feature vector globally pooled from the encoder outputs based on query-to-feature tofattuee similari- ties. The query (which can be black-box learnable feature vector Deng al. (202I) or a linguistic feature vector summarizing the expression [L & Sigal (202T)) serves as a representation of the target and context pooling is guided by similarity-based attention.

---

Under review as a conference paper at ICLR 2023
Expression:
"a white and blue plane behind a truck on a runway"
Image input.
Initial Anchor
Layer
Oulput
Ground Truth
Example anchor boxes learned by our method. The anchor boxes are generated sequen- tially by stack of modified Transformer decoder layers. The next anchor box predicted based on context aggregated by the current anchor box. The last predicted box is used as the final prediction.
Figure 1:
A potential issue with this approach is that without location priors (pre-designed or learnable), these queries rely on pure "content" -based similarities numerous inaccuracies in practice. Li er al. Li & Sigal (202T) demonstrate that even in scenes of relatively simple compositions, the attention of the context feature vector from this approach often peaks at multiple locations, including those outside the target area, which leads to wrong or inaccurate box predictions.
decide context locations, which often contain
Motivated by the above observations, in this work we propose a Transformer-based decoding method for addressing REC, which we term as LUNA (short for LangLiage as contiNuing Anchors). consists in leveraging the input expression for generating a series of continuously updated anchor boxes (shown in Fig. [T that guide object localization. LUNA generates the first anchor box by attending image regions under the guidance of the input expression. This is achieved via cross- attention-basee proto-decoder, which summarizes an object representation based on word-specific visual context and decodes approximate location of the target. Given the initial object repre- sentation and the anchor box, a stack of modified Transformer decoder layers iteratively refine the object representation and update the anchor box. Progressing through each layer, the current anchor box is projected into high-dimensional space and used for pooling multi-modal context; then new anchor box predicted from pooled context. We refer this stack of decoder layers as a continuous sechor-guided decoder. This decoding approach allows the model to focus selectively on one part of the scene at time, and acquire more accurate context information with focused attention.
To evaluate the efficacy of the proposed method, we conduct extensive experiments on the challeng- ing datasets of Referlt Kazemzadeh al. (2014), RefCOCO Yu et al. (2016), ReiCOCO+ Yu et al. (016)) RefCOCOg Nagaraja et al. (2016), and Flickr30K Entities Plummer et al. (2015). for which we improve the state of the art by large margins.
Our contributions are summarized as follows. (1) We proposed simple but effective Transformer- based decoding method which leverages anchors as position priors for tackling referring expression comprehension. (2) Our decoding strategy consists of proto-decoder to estimate the initial lo- calization from language, and a continuous anchor-guided decoder for predicting bounding boxes progressively. (3) We obtain new state-of-the-art results on five REC benchmarks, demonstrating the effectiveness and generality of the proposed method.
RELATED WORK
Referring expression comprehension (REC). Since the proposal of this task in several parallel studies Yu et al. (016)); Hu et al.) (201)); Nagaraja et al. (2016). the state-of-the for tackling has transitioned from two-stage models to one-stage models to the most recent Transformer-busel models. Two-stage models Hu et al. (2017); Wang et al. (2018): 8); Yu et al. [2018ab); Yang et al. (2019a) rank a set of image regions based on their similarities with the re- ferring language expression, where the regions are pre-extracted using region proposal method. Popular proposal methods include unsupervised ones based on hand-crafted features Uijlings et al. & Dollár (2144) and pre- pre-trained neural nets Ren et al. (201)). This propose-anddrante approach is slow and suffers from limitations of the proposal method (such as location inaccuracies from unsupervised methods or biases of the proposal network) [ et al. (2020); Yang et al. (0019)).
paradigm
Zitnick 20t3):[itnicck
To address these issues, one-stage models opt for directly predicting the bounding box from fused multi-modal features, relying on feature fusion mechanisms and dense anchoring Yang et al. 2019hh
2

---

Under review as a conference paper at ICLR 2023
2020): Luo al. (2020). Typically, cross-modal feature fusion module (such as concatenation- based Yang et al. (2019b 2020) or attention attentionnbasse Luo et al. (2020)) integrates extracted visual and linguistic features to generate multi-modks features. Then set of reference boxes, termed anchor boxes, are densely placed at different locations of the multi-modal feature maps, and boxes predicted with respect to anchors in a sliding window fashion. By directly conditioning box predictions on the referring expression, one-stage models circumvent the problems faced by their two-stage precursors. However, the heuristics used for defining the anchor boxes (concerning their scales, aspect ratios, and assignments) can significantly influence the accuracy of the model. Another line of work (e.g., [iao et al. 20200) leverages language as correlation filters and does not resort to anchors, but has not reached the accuracy of anchor- based methods.
Most recently, the powerful Transformer Devlin et al. (2019); Vaswani al. (2017) architecture is adapted to this problem. The core ingredients of Transformer- based methods Deng et al. (202T);
Sigal (02T)) include a Transformer encoder for jointly embedding visual and linguistic features, and a cleverly designed "query", which aggregates multi-modal context used for box prediction. In a ViT-like Dosovitskiy et al. (2021) manner, Deng et al. Deng et al. (2021) leverage a special [REG] feature vector prepended to the input sequence the query, which pools global context in the encoder's self-attention layers. In a DETR-like Carion et al. (2020) manner, Li et al. Li & Sigal (202I) employ Transformer decoder, and use linguistic feature vector (which summarizes the input expression) as a query in the decoder. Either the [REG] token approach or the language-as- query approach relies purely content- have many inaccuracies due to noises in the image and expression. Motivated by previous one- stage models and recent Transformer-basee
similarities for context aggregation, which tend to
detection models that exploit spatial representations for decoding Zhu et al. (2021): Liu et al. (2022). in this work we leverage learnable anchor boxes as queries for helping guide multi-modal context aggregation. Our model learns more accurate cross- modal alignments compared to models in Deng et al. (2021) and Li & Sigal (202T)
Multi-modal understanding. One major line of research in the area of multi-modal (vision- language) understanding is the large-scale pre-training of multi-modal representations transferable variety of downstream tasks. This problem is typically tackled by leveraging large amounts of paired image-text data (typically sourced from the Internet) and training a Transformer-base
model for solving various surrogate tasks, including masked mooal modeling Lu al. (2019); Tann & Bansal (2019): Zhou et al. (2020), contrastive learning Radford et al. (2021). and modulated detec- tion Kamath et al. (2021). Another line of work [u et al. (2020); Hu & Singh (2021) I) takes a multi- task learning perspective and aims to develop a unified model that can address large number of vision-language tasks at once, In addition, growing efforts have been devoted to a variety of vision- language tasks such as visual question answering Antol et al. (2015), image captioning Vinyals et al.) (2015), and temporal sentence grounding Anne Hendricks et al. (2017); Gao et al.] (2017). In this work, we focus on the task of referring expression comprehension and propose anchor-based decoding mechanism based on the Transformer architecture.
3 METHOD
We are interested in the problem of box prediction from language, where the object of interest is represented in two different modalities. Therefore, our model should perform well in two as- pects: aligning the representation of the object across the two modalities, and determining its pre- cise location on the image. We first exploit the global correlation modeling power of a Transformer encoder Devlin al. (2019) for learning fully connected pairwise correspondences between each image patch and each word (please see See 3.1). To localize the language precisely, we leverage a context aggre-
series of iteratively updated anchor boxes as location priors for guiding gation, which is achieved with proto-decoder and a continuous anchor-guice decoder described Sec. (.22 We name the "proto-decoder" this way to give relevance to functionality to output an initial representation of the object and an initial estimate of the object location. The overall pipeline of our model is schematically illustrated in Fig. 2
3.1 VISUAL-LIIGUIISTC ENCODING
We extract visual features, F, E TR HWCC from the input image by leveraging a vision backbone network (such as ResNet101 He et al. (2016) or Swin Transformer Liu et al. (202T)) followed by
3

---

Under review as a conference paper at ICLR 2023
Proto-decode
Continuous Anchor- guided Decoder
Key,
Sty
Qa
Qa
Qu
Visral K.V
Layer
Layer
Layer 6
Transfomer
Encoder
Crues: Altintiinn
Qol
Splil
Lang. Backhone
Vis. Backbooe
Qa
Inil. Anchor
New
Predl
Andhrr
Box
"woman under purple umbrella" "
Lang
Q
Figure 2: Overall pipeline the proposed method. Our model consists of three main components: (1) visual-linguistic Transformer encoder (Sec. 3.1) which jointly encodes the image and the text inputs, (2) proto-decoder (Sec. (322) which produces an initial representation of the object and an initial estimate of its location (the first anchor box), and (3) continuous anchor- anchor-guided decoder (also Sec. S.2. which sequentially updates the anchor box through stack of modified Transformer decoder layers. At each layer, both the output embedding from the previous layer and the anchor box are used as queries. The last anchor box corresponds to the final prediction.
multi-layer perceptron (MLP) for channel reduction. Here, C' denotes the channel number, and H and W denote the height and width, respectively. And we extract linguistic features, Fi E RT''' from the imput expression by leveraging language modeling network (such LSTM Hochreiter & Schmidhuber (1997) or BERT Devlin et al. (2019)) followed by an MLP for channel reduction. Here, denotes the number of words and C denotes the number of channels. The visual features Fo and linguistic features Fi are concatenated along the sequence dimension to form the input to Transformer encoder, S [Fu; Fi), which is visual-linguistic sequence of feature vectors. In the rest of the paper, we use "feature vector" and "embedding" interchangeably.
The multi-modal Transformer encoder. A multi-modal Transformer encoder takes input, and outputs sequence of multi-modal embeddings which align visual cues and linguistic meanings in a common feature space. We denote the encoder output as E [R where H, W, T, and C follow their previous definitions. Our encoder mostly follows the original implementation of BASE
the BERT BASE encoder Devlin et al. (2019), which consists of a stack of 6 identical layers. Each layer mainly includes multi-head self-attention sub-layer and feed-forward sub-layer. To encode position information and also differentiate between the two types of inputs, we use positional em- beddings, denoted as P [Pu;Pt) where P., IR HW xC are fixed image positional embeddings the same as in Carion et al. (020)) and Pi E RTxC' are learnable expression positional embeddings Li & Sigal (021)) P is supplemented to each attention sub-layer.
3.2 DECODING WITH CONTINUING ANCHORS
As described in Secs. M and 2 previous Transformer-basee methods Deng et al. (2021)
1021):000 Li & Sigal
(2021) tackling this task do not exploit location priors during decoding. As illustrated Fig. [ this often leads to noise in the aggregated context feature vector. We use series of "continuing" anchors (in the sense that each anchor box is updated from the previous one) as queries during decoding for addressing this problem.
Query generation with a proto-decoder. The proto-decoder (illustrated in Fig. (a)) generates the first anchor box and an object representation (what we refer to as the object query) from multi-modal context. Without introducing extra human-designed strategy, our learnable anchor could naturally perform role of location prior.
To start with, word-specific multi-modal context is computed via a cross-attention layer. The query input to this layer is concatenation of & E T and P, (the expression positional embeddings described in Sec. 3.II along the channel dimension. Here, E is segment of the encoder output E from positions corresponding to word embeddings. The key and the value inputs this layer is concatenation of &,, E W XC and P, (the image positional embeddings described in Sec. 3.1] along the channel dimension. Similarly, E, is segment of & from positions corresponding image
4

---

Under review as a conference paper at ICLR 2023
region embeddings. The output from the cross-attention layer encodes word-specific multi-modal context and denoted as E IR RTxx2 Next, average pooling is applied on across the sequence dimension to obtain the pooling output R 1Ix2C
Key, Value: V-L Sequence
Tx2C
Cross Allatioo
Lapyer
Value
Query:
Avg. Pool
Qai
Que
Que
V Viuual
Lang.
Emb.
[
1x2C
Qui
Q
1xC
MLP
MLP
Lang.
Pos.
Fos.
ML.P
Aai
A.s...
HWWx2
TX2C
1
Ao
Aa
Pred. Box
(a) Proto-decoder
(b) Anchor-guidde decoder
Figure 3: Details of the proto-decoder and the continuous anchor-guided decoder. (a) Proto-decoder generates the first learnable anchor query and object query. (b) Anchor-guided decoder refine the anchor query after multi-modal context aggregation of each layer.
In final step, split Q along the channel dimension obtain two feature vectors: Qa1 R and Qa E R Q01 the object query we want. And Qa projected by an MLP with output channels and a terminal sigmoid activation. The output is first anchor box, denoted as Ao E 1x The 4 values along the channel dimension have a physical meaning and can be represented 4-tuple (zo, No, wo, ho), where zn and yo are the coordinates of the center point, and tog and ho are the height and width of the anchor, respectively.
Continuous anchor-guided decoding. To provide better guidance for referring localization, in- spired by the DAB-DETRE Liu et al. (2022) detection model, we propose continuous anchor-guided decoder, which uses the anchor query Ao as an initial bounding box and iteratively updates based on mudti-modal context aggregation, illustrated in Fig. (b).
Assuming there are decoder layers (original transformer decoder layer which contains self- attention and cross-attention sub-layers), we use i E [1,2....n] to index each layer. Given an anchor box A;-1, we project to high-dimensional space to obtain an anchor query follows,
Qai (PPE(E;-1): P)( PB(w;-1): PE(h)-1))),
where '[:]' indicates concatenation along the channel dimension, PE is sinusoidal positional en- coding function as we describe next, and 9 is a fully connected layer with C' output channels. To put it into words, to convert the anchor box into high-dimensional embedding for first apply a positional encoding function which operates element-wise on each of the parameters of the box and embeds each parameter into high-dimensional embedding; then we concatenate the four embeddings along the channel dimension and projects it. Note that is shared across all layers. We adopt the same positional encoding function defined in Devlin et (2019):
query,
PE(2)2j sin T PB(2)2j+1 cos (2)/D) D
where D defines the number of output channels, j is the index along the channel dimension, is the input position variable, and T represents temperature. We discuss the syper-parameters D and Sec. [4.2 Note that our proposed anchor formulation different from DAB-DETR, where we project 4 parameters (including height and width) of anchors while DAB-DETR embeds first 2 parameters (coordinates of the center point). We demonstrate the experimental effectiveness of our anchor query formulation in Supplementary Material due to the page limit.
The query input to the i-th cross-attention sub-layer is the following:
Q: =Qoi: Qai),
(3)
where '[:]' denotes concatenation along the channel dimension, Qa E RXCC is the object query, and Qui E 1xC is the anchor query. With the multi-head attention mechanism in the cross-attention
5

---

Under review as a conference paper at ICLR 2023
sub-laye sub-layer, the concatenation of these two types of queries decouples context aggregation Meng al. (2021) , i.e., the output embedding, Qo(i+1) E RI*C, xC, has the top/bottom half of channels generated based on the attention of the object query/anchor query. correspondingly. Qo(++1) 1) is the object query in the next decoder layer. To generate the next anchor box, we predict a set of offsets:
Aai = a (MLP((Q(():)))
(4)
where a denotes the sigmoid function and A&i is the set of box offsets. Note that the MLP is shared across all layers. We obtain a new anchor box by updating the previous one with the predicted offiets:
A-1 Adi
We iterate through all n layers of the decoder according to Eqs. equation equation anchor box, A,, is used as the final prediction.
The last
The loss function. We use the sum of the Generalized IoU Rezatofighi al. (2019) loss and the LI loss for training our model, which is defined L Xa (B, B) + A1 BlI,, where B the ground truth, and B is our prediction, Xo and x1 are balancing weights.
4 EXPERIMENTS
4.1 DATASETS AND METRICS
RefCOCO, RefCOCO+, and RefCOCOg. The RefCOCO dataset YontasetYueete Yu (2016) contains 19,994 images, 50,000 objects, and 142,209 referring expressions. The RefCOCO+ dataset Yu et al. (2016) contains 41,4664 expressions for 49,856 objects 19,992 images. The RefCOCOg dataset Na- garaja al. (016)) contains 85,474 expressions for 54,822 objects in 26,711 images. Referring pression annotations in RefCOCO and RefCOCO+ are collected in an interactive two-player game. In this game, each player is motivated to provide minimally sufficient description for the other to identify the target. As a result, expressions in these two datasets tend to be succinct, averaging fewer than 4 words per sentence. A special property of RefCOCO+ is that location words (such as "top",, "bottom", "left", etc.) are banned in its annotations, which makes it the harder of the two. On the contrary, expressions in RefCOCOg are annotated on Amazon Mechanical Turk in non-competitive way, and thus tend to be longer (8.43 words per sentence on average) and more expressive. This makes RefCOCOg a particularly challenging dataset compared the other two.
ReferIt Game. The ReferIt Game dataset Kazemzadeh et al. (014) contains 20,000 images col- lected from the SAIAPR-12 dataset Escalante al. (2010) with automatically generated referring expressions. Following standard practice Hu et al. (2016); Yang et al. (2019b): Deng et al. (2021); Yang et al. (2020), we split this dataset into 9,000 images for training. 1,000 images for valida- tion, and 10,000 images for testing, with 54K, 5.8K, and 60K referring expressions for these splits, respectively.
Flickr30K Entities. The Flickr30K Entities dataset [Plummer et al. (2015) is a popular benchmark for the phrase grounding task, in which the model is required to localize multiple entities from an image caption. In our experiments, we follow variant of the above setting which is commonly adopted by many previous REC methods Rohhrbach et al. (2016); Yu et al. (2018b); Yang et al. (2019b 2020); Yu et al. (2018a); Deng et al. (2021). i.e., each phrase in given image caption is treated as an independent referring expression without considering the context the whole sentence. This dataset contains 31,783 images and 427K entities, with 29,783 images for training, 1,000 images for validation, and 1,000 images for testing.
Evaluation metrics. We adopt the standard metric of Acc@0.5, which measures the percentage of test samples for which the intersection-over-unie above 0.5. This metric is also often referred to top-1 accuracy simply accuracy in the literature.
between the prediction and the ground truth is
4.2 IMPLEMENTATION DETAILS
We use ResNet- 101 He et al. (206)) and Swin-B Liu al. (2021) ) our visual backbone with classification weights pre-trained on 222g We adopt the uncased BERT BARTB [Deliin et al. 2019) model from the HuggingFace library Liu et al. (2019a) our language model, and initialize

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison with state-of-the-art methods on RefCOco family datasets in terms of Acc@0.5. We indicate the image resolution for one-stage approaches.
Image
Image Res.
RefCOCO
RefCOCO+ val
RefCOCOg val
Method
Backbone
val test test
test test
test
RvG-Tree Hong et al. (019))
FRCN
75.06
78.61 69.85
63.51
67.45 56.66
66.95 66.51
MAnNet eu 2013
FRCN
76.65
81.14 69.99 65.33
71.62 56.02
66.58
67.27
DGA Yang el al. (019a)
FRCN
78.42
65.53
69.07
$1.99
63.28
NMT Liu of al. 2019a)
FRCN
76.41
81.21
70.09
66.46
72.02 57.52
65.87 66.44
CM-Att- -rase Liu et al.
2019b
FRCN
78.35
83.14
71.32
68.09
73.65
58.03
67.99
68.67
DDPN Yu et al. 801
FRCN
76.8
80.1
72.4
64.8
70.5
54.1
FAOA Yang el al. b
Darknet-53
256
72.54
74.35 68.50
56.81
60.23
49.60
61.33
60.36
ReSC-Largel Yang et al. 2020
Darknet-53
256
77.63
80.45
72.30
63.59
68.36
56.81
67.30
67.20
MCN Luo et al. (2020
Darknet-53
416
80.08
82.29 74.98
67.16
72.86
57.31
66.46
66.01
RCCF Liao et al. 2020
DLA-34
512
81.06 71.85
70.35
56.32
65.73
Trans Denn a. 2021
ResNet-101
640
81.02
82.72 78.35
64.82
70.70
56.94
68.67
67.73
RefTR Li Sigal (2021
ResNet-101
640
82.23
85.59 76.57
71.58
75.96
62.16
69.41
69.40
Soq Zhu ct al.
2022)
Darknet-53
640
81.23
85.00 76.08
68.82
75.37
58.78
71.35 71.58
LUNA (Ours)
ResNet-101
640
84.47
86.64
80.18
72.60
77.90
64.51
74.06
72.75
LUNA (Ours)
Swin-B
640
86.12 88.43 82.63
75.96
80.62
68.36
76.51 76.55
with official weights pre-trained on BooksCorpus Zhu et al. (2015) and English Wikipedia. All other parts of our model are randomly initialized. We adopt Adam Loshchilov & Hutter (2019)
as the optimizer, and set the learning rate to 0.0001 for the visual-linguistic Transformer encoder and 0,00005 for the visual and linguistic backbones. Following Deng et al. (2021) Li & Sigal (202T), images are resized to 640 640 and augmented with random intensity saturation and affine transformations. On RefCOCO, RefCOCO+, RefCOCOg, and ReferIt Game, the model is trained for 90 epochs with the learning rate dropped by a factor of 10 after 60 epochs. On Flickr30K Entities, the model is trained for 60 epochs with the learning rate dropped by a factor of 10 after 40 epochs. When adopting pre-training, we follow Li & Sigal (202T) and train our model on Visual Genome Krishna Genome/Krishna al. (2117) for 6 epochs, and fine-tune it on the evaluation dataset for 50 epochs. The maximum expression length is 20 on all datasets. The visual-linguistic Transformer encoder and our decoder both have layers. C, T, and D in Sec. aree empirically set 256, 20, and 128, respectively. Models adopting Swin-B backbone are trained with mini-batches of size 8, and those adopting ResNet-101 backbone are trained with batch size 40. We apply balancing weights on loss function as Au = 2, A 5. We run several experiments with different random seeds, and the difference between different runs generally lies in +0.5%.
4.3 COMPARISON WITH OTHERS
In Table T compare LUNA state-of-the-art methods on standard benchmarks for the REC task. Among the existing method on REC, our method attains the highest Acc@ across all subsets of all datasets with both ResNet-101 and Swin-B visual backbone. Specifically, on the val/test A/test B subsets of ReiCOCO and RefCOCO+, LUNA with Swin-B backbone outperforms the second best method Li & Sigal (202T by absolute margins of 3. 3.89% 1/66//84%/6606666 and 4.38% 71.%%//..//6...... /00% respectively. As for the 101 backbone, LUNA still achieves superior results by absolute margins up 4.1%. On the most challenging RefCOCOg dataset, LUNA is able to achieve the largest improvements by far: On the validation set and test set, LUNA obtains 7.10% and 7.15% absolute improvements over the RefTR model, 5.16% and 4.97% absolute margin over second-best method SeqTR.
Observing the pattern across the three datasets, we can see that the more difficult the dataset is (please see Sec. 4.1 for discussion), the larger the improvement LUNA is able to bring with spect to the state of the art. This corroborates our case for exploiting learnable location priors to facilitate box decoding. As discussed in Sec. 4.T banning the use of location words makes Ref- COCO+ a significantly harder dataset than RefCOCO is, as cross- modal alignment can only rely on appearance information. Similarly, the longer and more flowery descriptions in the RefCOCOg dataset makes purely content-based similarities less reliable as the basis of grounding. The relatively large improvements of LUNA on these two datasets provide evidence that when content-based map-

---

Under review as a conference paper at ICLR 2023
ping is not easy accurate, our proposed learnable location priors make up for it. Moreover, the Referlt Kazemzadeh et al. (2014) dataset, LUNA also surpasses all previous methods.
In Table 2
evaluate LUNA against the state-of-the-ang methods on the Flickr30K Entities Plummer
Table 2: Results Flickr30K Entities and Referlt.
al. 2015) dataset.
Visual Backbone
Bickr30K/Rcfert
Please refer to Sec. 4.1 for a discussion on the specific task setting. Compared Trans VG and RefTR, two other state-of- the-art models based on Transformers, method obtains 2.04% and 2.48% (abso- lute) improvements in terms of Acc@0.5, respectively.
Method
test
test
CITE Plummer ct al.
2018
VGG-16
61.89
Similariy Nel Wang al.
2018
VGG-16
60.89
DDPN
201
ResNet-101
73.30
63.00
FAOA Yang
Darknet-53
68.69
59.30
ReSC- Large Yang cl
2020
Darknet-53
69.28
64.60
Trans Deng ct al.
ResNet-101
79.10
70.73
RefTR Li & Sigal 2021
ResNet-1U1
78.66
71.42
LUNA (ours)
ResNet- 101
79.44
72.67
While pre-training is not the focus of this work, additional pre-training of LUNA on the Visual Gensme/Krishna al. (2017) re-
LUNA (ours)
Swin-B
81.14
73.69
gion description splits can further boost performance, and our method obtains comparable or better results with respect to other methods that employ pre-training. More result and comparison please see Table 7 in Supplementary Material.
4.4 ABLATION STUDY
In this section, we conduct extensive ablation experiments to study the effects of the main model components and alternative design choices in the proto-decoder and the continuous anchor-guided decoder. Experiments this section are conducted on the validation and test sets the RefCOCOg dataset. ResNet- 101 is adopted as the vision backbone and BERT is adopted as the language back- bone, unless otherwise specified.
Table 3: Model component ablations.
Table Ablations on proto-decoder.
Proto-decoder CA-guided decnder
val
Methods
val
test
74.06
72.75
Self-attention
70.42
70.36
69.58
69.82
Reversed proto-decnder
72.18
71.56
71.26
70.55
Ours decodee
74.06
72.75
Model components. In Table 3 we investigate the contributions of the proto-decoder and the contin- uous anchor-gus guided decoder (CA-guided decoder). We first remove the CA-guided decoder from the full model, and use the initial anchor produced from the proto-decoder the prediction. This model variant leads to 4.48% and 2.93% drop in Acc@0.5 on the validation and test sets of RefCOCOg, respectively. These results validate the effectiveness of our proposed CA-guided decoder. We then remove the proto-decoder from the full model, and randomly initialized learnable embeddings of object query and anchor box which send the CA-guided decoder. This model variant leads to 2.80% and 2.2% drop in Acc@0.5 on the validation and test sets of RefCOCOg, respectively. This demonstrates the effectiveness of the proposed proto-decoder.
Alternative studies of proto-decoder. We carefully designed the structure of proto-decoder to generate relyable initial object query and anchor query, and consider several replaceable cases (in- cluding self-attention, visual query etc.) as shown in Table 4 The result shows that our design of proto-decoder out- out-performs other alternative cases. Due to the page limit, more details please see Table 8 in Supplementary Material.
Object representation in the proto-decoder. As described in Sec. 3.2 after obtaining a sequence of word-specific visual context embeddings, we obtain an object representation by averaging the sequence for predicting the initial object query and anchor box. In Table [ we study three other alternatives for summarizing object information. In the '[CLS] Token' alternative, bedding output from the special '[CLS]' token input as representation of the object. In the 'Max Pooling' alternative, we directly pool the maximum activation along each token dimension across the sequence. As for 'Weighted Avg.' alternative, we apply weighted average pooling on the se- quence length dimension generate the object representation, where weights are learned by 2-layer MIP with softmax activation. As shown in Table 5 normal average pooling out-performs other alternatives and adopt it as the default choice.
the em-
8

---

Under review as a conference paper at ICLR 2023
Table Alternative object representations in the proto-decoder.
Table 6: Effects the number of layers in CA-guided decoder.
Layers
Method
val
test
69.58
69.82
CLS Token
72.36
71.09
70.44
70.43
Max Pooling
71.73
71.69
4
72.32
71.64
Weighted Avg.
72.03
71.66
74.06
72.75
Average Pooling 74.06
72.75
72.44
71.81
The number of layers in CA-guided decoder. In Table 6 we study the effects of the number of decoding layers in our proposed CA-guided decoder, i.e. the number of steps that initial anchor updated to predict the final bounding box. We test model variants which adopt 2, 4, 6, and 8 layers. Accuracy improves as more layers are added until layers have been used. Note that in O-layer case, LUNA only uses proto-decoder to predict the result.
4.5
VISUALIZATION
In Fig. 4 we visualize the attention patterns and predictions of LUNA and RefTR Li & Sigal (2021) some challenging examples. Notice that for both models, the prediction is made where query attention is focused. This is expected as the box is predicted from a context feature vector pooled according query attention. In cluttered scene depicted the left, our model correctly localizes the muffins among many similar distractors such as sandwiches in the middle or on top. In the scene on the right, the wing of the airplane depicted from rare angle and not quite noticeable from its background. Our model also correctly localizes the target in this case. In comparison, the query attention of RefTR peaks at wrong locations and leads wrong box predictions.
"the muffins on the bottom of the photo
"the wing of an airplane over a bus"
RefRR
Ours
Predictions GT
Attention Map
Predictions & GT
Attention Map
Figure 4: Visualized query attention from the last decoder layer and the final predictions of our method and RefTR & Sigal (202T). Cyan boxes are the ground truth and red boxes predictions. Examples are from the validation set of RefCOCOg Nagaraja et al. 2016).
CONCLUSION
In this paper, have presented Transformer-based decoding method for referring expression comprehension, which exploits a series of language-guided anchor boxes as helpful spatial cues for guiding context pooling Transformer decoder. Extensive experiments five benchmarks demonstrate its advantage with respect to the state-of-te tate-off-he--il fe-thh--rt art methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, 2017.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
Jiajun Deng, Zhengyuan Yang. Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to- end visual grounding with transformers. In ICCV, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: pre- pre-training of deep bidirectional transformers for language understanding. In NAACL. 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Hugo Jair Escalante, Carlos A Hernández, Jesus A Gonzalez, Aurelio López-López, Manuel Montes, Eduardo Morales, L Enrique Sucar, Luis Villasenor, and Michael Grubinger. The segmented and annotated iapr 12 benchmark. In CVIU, 2010.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for sssion-and-langunee representation learning, In NeurlPS, 2020.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. ICCV, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. In Neural Computation, 1997.
Richang Hong, Daqing Liu, Xiaoyu Mo, Xiangnan He, and Hanwang Zhang. Learning to compose and reason with language tree structures for visual grounding. In TPAMI, 2019.
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified trans- former. ICCV, 2021.
Ronghang Hu, Huazhe Xu, Mareus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In CVPR, 2016.
Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. Modeling rela- tionships in referential expressions with compositional modular networks. In CVPR, 2017.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr-modulated detection for end-to-end multi-modal understanding. In ICCV, 2021.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Referring to Objects in Photographs of Natural Scenes. In EMNLP, 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsoureed dense image annotations. International of computer vision, 123(1):32-3,, 2017.
Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. In NeurIPS, 2021.
10

---

Under review as a conference paper at ICLR 2023
Shuang Li, Tong Xiao, Hongsheng Li, Wei Yang, and Xiaogang Wang. Identity- awarr matching with latent co-attention. In ICCV, 2017.
aware rextual-visud
Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, and Bo Li. A real-time cross- modality correlation filtering method for referring expression comprehension. In CVPR, 2020.
Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha. Learning to assemble neural module tree networks for visual grounding. In ICCV, 2019a.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang. Xianbiao Qi. Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In ICLR, 2022.
Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. Improving referring ex- pression grounding with cross rross-modal creettiongguiedd crasing. In CVPR, 2019b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin-
guistic representations for rision-and-language tasks. In NeurlPS, 2019.
Jiasen Lu, Vedanuj Goswami, Mareus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In CVPR, 2020.
Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In CVPR, 2020.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li. Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE Interna- tional Conference on Computer Vision (ICCV), 2021.
Varun K. Nagaraja, Vlad Morariu, and Larry S. Davis. Modeling context between objects for referring expression understanding. In ECCV, 2016.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to-phee correspondences for richer image- to-sentence models. In ICCV, 2015.
Bryan A Plummer, Paige Kordas, M Hadi Kiapour, Shuai Zheng, Robinson Piramuthu, and Svetlana Lazebnik. Conditional image-text embedding networks. In ECCV, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. ICML, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR, 2019.
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. ArXiv, abs/ 0bs//5511.033 1.03745, 2016.
Jing Shi, Ning Xu, Yihang Xu, Trung Bui, Franck Dernoncourt, and Chenliang Xu. Learning by planning: sanguage-guides global image editing. In CVPR, 2021.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans-
formers. In EMNLP, 2019.
11

---

Under review as a conference paper at ICLR 2023
Stefanie Tellex, Nakul Gopalan, Hadas Kress Kress-Gazit, and Cynthia Matuszek. Robots that use lan- guage. In Annual Review of Control, Robotics, and Autonomous Systems, 2020.
Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search for object recognition. In IJV,, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In CVPR, 2015.
Liwei Wang. Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image-text matching tasks. In TPAMI, 2018.
Sibei Yang, Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression compre- hension. In ICCV, 2019a.
Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach to visual grounding. In ICCV, 2019b.
Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual ground- ing by recursive sub-query construction. In ECCV, 2020.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Emiee-il: Knowl- edge enhanced vision-language representations through scene graph. In AAAI, 2021.
Licheng Yu, Patrick Poirson, Shan Yang. Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mat- Inet: attention network for referring expression comprehension. In CVPR, 2a.
Zhou Yu, Jun Yu, Chenchao Xiang, Zhou Zhao, Qi Tian, and Dacheng Tao. Rethinking diversified and discriminative proposal generation for visual grounding. IJCAI, 2018b.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In AAAI, 2020.
ChaoYang Zhu, YiYi Zhou, YunHang Shen, Gen Luo, XingJia Pan, MingBao Lin, Chao Chen, LiuJuan Cao, XiaoShuai Sun, and RongRong Ji. Seqtr: A simple yet universal network for visual grounding. arXiv preprint urXiv:2203. 16265, 2022.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In ICLR, 2021.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. ICCV, 2015.
Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In ECCV, 2014.
12


Reviews:
[{"review_id": "aEbsN39FUt4", "reviewer": "Reviewer_q7BF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In order to address the referring expression comprehension task, this paper proposes the LangUage as contiNuing Anchors (LUNA) to employ the language-guided location priors to refine box prediction in a Transformer decoder progressively. Such an approach makes the decoder focus selectively on partial scenes and reduces noise in multi-modal contexts. Experiments show that LUNA achieves good referring expression comprehension performances. ", "summary_of_the_review": "The major concerns of this paper are its novelty and on-par performance, as described in [Weaknesses]. The proposed method integrates several recent models and hence achieves on-par performance. It is better to clarify its key insights and technical novelty to understand the contribution better. For a fair comparison, it is better to indicate the model settings of backbones and image resolutions for performance measurement.  ", "strength_and_weaknesses": "[Strengths]\n+ This paper is well-written and easy to follow.  \n+ The ablation study and visualization are informative.\n\n[Weaknesses]\n- The proposed LUNA model integrates the BERT (Devlin et al.,2019) and ResNet-101 (He et al., 2016)/Swin-B (Liu et al., 2021), cross-attention (Vaswani et al., 2017), and DAB-DETR (Liu et al., 2022). Hence, the technical novelty is somewhat limited. It seems that the proto-decoder is the novel design of LUNA.\n- The referring expression comprehension (RES) is one sort of cross-modal localization task. Therefore, after obtaining the initial localization via proto-decoder, employing the guided decoding mechanism proposed by DAB-DETR to refine the localization interactively is a straightforward solution. Is there any critical insight concerning the RES task to make LUNA different from DAB-DETR in the goal of localization block refinement?  \n- Table 1 misses the comparison with the recent methods (VILLA-large, MDETR, and [A]), which show better results than this work. Since the backbones (language and visual) and image resolutions are important factors of the RES task, it is better to indicate these factors for a fair comparison.\n \nRelated paper:\n\n[A] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework. ICML 2022: 23318-23340\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WguKCfYehSn", "reviewer": "Reviewer_uU4Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the referring expression comprehension (REC) problem, ie, localizing the description of a natural language expression in an image. Specifically, they argue that existing Transformer-based methods don't use any location priors, which may result in inaccuracies in practice. To this end, they propose LUNA (LangUage as contiNuing Anchors) for Transformer-based REC models. Firstly, LUNA generates the first anchor box by attending to image regions under the guidance of the input expression. Then, in each layer, the anchor box is progressively updated into a new anchor. Extensive results on ReferIt, RefCOCO/+/g, and Flickr30K Entities have demonstrated the effectiveness of the proposed LUNA.", "summary_of_the_review": "The paper argues that existing Transformer-based REC methods have overlooked the anchor priors. However, I think the motivation of this claim is still unclear. The whole proposed method is more like a post-processing step to further refine the grounding results. Although the proposed method achieves really good performance on these benchmarks, the idea itself is common and not exciting.", "strength_and_weaknesses": "## Strengths\n+ Extensive results are conducted on five benchmarks to validate the effectiveness of the LUNA model. The performance gains on the RefCOCO-family datasets are quite impressive.\n\n## Weaknesses\n+ The motivation of why need to introduce the anchor prior to the Transformer-based models is still unclear. From another perspective, the proposed continuous anchor-guided decoder (cf. Figure 2) can also be regarded as a prediction refinement network. In another world, the proposed decoder can also be used in other existing Transformer-based REC architectures to further boost their performance. It would be more interesting to see more results on other backbones.\n+ For the results on Flickr30K Entities, it would be better to compare the LUNA to baselines with the same backbone (eg, ResNet-101).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R50_eS9AG9", "reviewer": "Reviewer_pGyA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose to address referring expression grounding  using a single stage transformer based approach and leveraging anchor points (guided by language) as region priors. Their outputs are very well grounded with the attention maps learned during the decoding process and improves over state of the art for referring expression grounding on several benchmark datasets. ", "summary_of_the_review": "Please see the above section for details. The results and  method show improvements with interesting components that can be beneficial in other research works as well. \n", "strength_and_weaknesses": "Strengths - The method is simple and effective and relies on image and location only (without prior region / location information ) to learn grounding for referring expressions. This approach of using an initial proto-decoder to decode an estimate bounding box based on language and learning a continuous offset conditioned on prior knowledge is useful for together bounding box localization of referring expressions. They show several ablation studies and results on benchmark datasets to validate the effectiveness of the approach.\n\nWeaknesses - \nIs it possible to measure the performance of grounding just from the proto-decoder?This would tell how much gains are actually bought by the Anchor guided decoder ? \n\nWhat if another prior / box location is used to guide the decoding process ? such as outputs from a pretrained system. In that case, the current method could also act as a corrective mechanism ? \n\nWhat happens if the model predicts a completely different bounding box location in the proto-decoder stage? \n\nIs there any benefit of splitting Q (Ix2C) to IxC and  IxC in Fig 3. (a) instead of learning projections ?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject