Referring Transformer: A One-step Approach to Multi-task Visual Grounding
Authors: Muchen Li
Leonid Sigal
Venue: NeurIPS 2021 Poster (2021)
Keywords: Computer Vision
Multi-modal Learning
Referring Expression Comprehension
Referring Expression Segmentation
Visual Grounding

Abstract: As an important step towards visual reasoning, visual grounding (e.g., phrase localization, referring expression comprehension / segmentation) has been widely explored. Previous approaches to referring expression comprehension (REC) or segmentation (RES) either suffer from limited performance, due to a two-stage setup, or require the designing of complex task-specific one-stage architectures. 
In this paper, we propose a simple one-stage multi-task framework for visual grounding tasks. Specifically, we leverage a transformer architecture, where two modalities are fused in a visual-lingual encoder. In the decoder, the model learns to generate contextualized lingual queries which are then decoded and used to directly regress the bounding box and produce a segmentation mask for the corresponding referred regions. With this simple but highly contextualized model, we outperform state-of-the-art methods by a large margin on both REC and RES tasks. We also show that a simple pre-training schedule (on an external dataset) further improves the performance. Extensive experiments and ablations illustrate that our model benefits greatly from contextualized information and multi-task training.

Referring Transformer: A One-step Approach to Multi-task Visual Grounding
Muchen Li!
Leonid Sigal 1223,4
muchenl i@cs .bc.ucc..
Isigal Segallcs.. .bc.
'Department of Computer Science, University of British Columbia 2 Vector Institute for AI
CCFAR Al Chair
#NSERC CRC Chair
Abstract
As an important step towards visual reasoning, visual grounding (e.g., phrase localization, referring expression comprehension segmentation) has been widely explored. Previous approaches to referring expression comprehension (REC) or segmentation (RES) either suffer from limited performance, due two-stage setup, or require the designing of complex task-specific one-stage architectures. In this paper, we propose simple one-stage multi-task framework for visual grounding tasks. Specifically, we leverage transformer architecture, where two modalities are fused in visual-lingual encoder. In the decoder, the model learns generate contextualized lingual queries which are then decoded and used to directly regress the bounding box and produce segmentation mask for the corresponding referred regions. With this simple but highly contextualized model, we outperform sate-offtheeart methods by a large margin both REC and RES tasks. We also show that a simple pre-training schedule (on an external dataset) further improves the performance. Extensive experiments and ablations illustrate that our model benefits greatly from contextualized information and multi-task training.
Introduction
Multi-modal grounding tasks (egg, phrase localization 03 29
41 48
referring expression compre-
hension [17 19 24 26
30 37
5
52
55 56) and segmentation 18
20
21
29
38
53
56)
aim to generalize traditional object detection and segmentation to localization of regions (rectangular or a pixel level) in images that correspond to free-form linguistic expressions. These tasks have emerged as core problems in vision and ML due to the breadth of applications that can make use of such techniques, spanning image captioning, visual question answering, visual reasoning and others.
The majority of multi-modal grounding architectures, to date, take the form two-stage approaches, inspired by Faster RCNN [44] and others, which first generate a of image region proposals and then associate aaoociate/ground one, or more, of these regions to phrase by considering how well the content matches the query phrase. Context among the regions and multiple query phrases, which often come parsed from a single sentence, has also been considered in various ways (e.g., using LSTM stacks [9. graph neural networks [ and others). More recent variants leverage pre-trained multi-modal Transformers (e.g., ViLBERT 33 34
to fine-tune to the grounding tasks. Such models have an added benefit of being able learn sophisticated cross-modal feature representations from external
'Grundiig and referring expression comprehension have been used interchangeably in the literature. While the terms are indeed trying characterize the task of associating lingual phrase with image region, there is subtle difference that referring expressions tend to be unique and hence need to be grounded to single region, e.g., "person in a red coat next to bus". The grounding task, as originally defined in [4I] is more general where a lingual phrase may be ambiguous and therefore grounded to multiple regions, e.g., "a person".
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

large-scale data, which further improve the performance. However, a significant limitation of all such two-stage methods is their inability to condition the proposal mechanism on the query phrase itself, which inherently limits the upper bound of performance (see Table 3 in [1
To address these limitations, more recently, number of one-stage approaches have been introduced [22] 36 51 (2). Most of these take inspiration from Yolo [43] and the variants, and rely on more integrated visual-linguistic fusion and dense anchoring mechanism to directly predict the grounding regions. While this alleviates the need for a proposal stage, it instead requires somewhat ad hoc anchor definitions, often obtained by clustering of labeled regions, and also limits ability to contextualize grounding decisions as each query phrase is effectively processed independently. Finally, little attention in the literature has been given to leveraging relationship among the REC and RES tasks.
In this work we propose an end-to-end one-stage architecture, inspired by the recent DETR [2 detection framework, which is capable of simultaneous language grounding at both bounding- box and segmentation level, without requiring dense anchor definitions. This model also enables contextualized reasoning by taking into account the entire image, all referring query phrases of interest and (optionally) lingual context (e.g. a sentence from which referring phrases are parsed). Specifically, we leverage transfomer architecture, with a visual-lingual encoder, to encode image and lingual context, and a two-headed (detection and segmentation) custom contextualized tranformer decoder. The contextualized decoder takes as input learned contextualized phrase queries and decodes them directly to bounding boxes and segmentation masks. Implicit 1-to-1 correspondence between input referring phrases and resulting outputs also enables a more direct formulation of the loss without requiring Hungarian matching. With this simple model we outperform state-of-the methods by a large margin on both REC and RES tasks. We also show that a simple pre-training schedule (on an external dataset) further improves the performance. Extensive experiments and ablations illustrate that our model benefit greatly from the contextualized information and the multi-task training.
Contributions, Our contributions are: (1) We propose simple and general one-stage transformer- based architecture for referring expression comprehension and segmentation. The core of this model is the novel transformer decoder that leverages contextualized phrase queries and is able to directly decode those, subject to contextualized image embeddings. into corresponding image regions and segments; (2) Our approach is unique in enabling simultaneous REC and RES using single trained model (the only other method capable of this is (36)): ); showing that such multi-task learning leads to improvements on both tasks; (3) As with other
moansformr-bbbsee architectures, we show that pre-training can further improve the performance and both vanila and pre-trained models outperform state-of-the-art on both tasks by significant margins (up to 8.5% on RefCOCO dataset for REC and 19.4% for RES). We also thoroughly validate our design in detailed ablations.
2 Related works
Referring Expression Comprehension (REC). REC focuses on producing an image bounding
box tightly encompassing a language query. Previous two-staged works [17 19 29 30 [66 refor-
mulate this as a ranking task with a set of candidate regions predicted from a pre-trained proposal mechanism. Despite achieving great success, the performance of two-staged methods is capped by
the speed and accuracy of region proposals in the first stage. More recently, one- me-stage approaches
26 [1] 52 have been used to alleviate the aforementioned limitations. Yang et al. [51, [52) proposed to fuse query information with visual features and pick the bounding box with maximum activation scores from YOLOv3 43 Yang el al. (50) explore language structure guided propagation in the
context of one stage grounding. Liao el al. (26) utilizes CenterNet [ to perform correlation filtering for region center localization. However, such methods either require manually tuned anchor boxes or suffer from semantic loss due to modality misalignment. In contrast, our model learns to better align modalities using mooss-modal transformer and directly decode bounding boxes for each query.
Referring Expression Segmentation (RES). Similar REC, RES, proposed in [8] aims to predict segmentation masks to better describe the shape of the referred region. A typical solution for referring expression segmentation to fuse multi modal information with a segmentation network 38, 53 (6). More recent approaches focus on designing module to enable better multi-modal interactions, e.g., progressive multi-scale
(e.g., [6 31) and train it to output the segmented masks [18 29
fusion used in (21 and modal attention block used in 20) Since localization information
matters in predicting instance segmentations (as noted in Mask RCNN [6)), very recent work [22]
aims to explicitly localize object before doing segmentation. Despite the relatively high performance
2

---

being achieved in RES, existing approaches still struggle to determine the correct referent region and tend to output noisy segmentation results with an irregular shape, while our model is able to produce segmentations with fine-grained shapes even on challenging scenarios with occlusions or shadows.
Multi-task Learning for REC and RES. Multi-task learning is widely applied in object detection and segmentation [2] 16. often, by leveraging shared backbone and task-specific heads. Building on this idea, Luo et al. 136] proposed a multi-task collaborative network (MCN) to jointly address REC and RES. They introduce consistency energy maximization loss that constrains the feature activation map in REC and RES to be similar. While our model is also set up to learn REC and RES tasks jointly, we argue that an explicit constraint tends to downplay the quality of the final predicted mask since the feature map from the REC branch can blur out fine- aen-ggriied region shape information needed by the RES branch (see Figure 2 Hence, we use an implicit constraint where tasks head of REC and RES are trained to output corresponding bounding box and mask from the same joint multi-modal representation. We illustrate that our model can benefit from multi-task supervision, leading to more accurate results as compared to single-task variants.
Pretrained Multi-modal Transformers. Transformer-based pretrained models 5
2 33 47
54
have recently showed strong potential in multi-modal understanding. LXMERT 47 and ViLBERT 33 use two stream transformers with cross-attention transformer layers on top for multimodal fusion. More recent works,
advocate single-stream design to fuse two modalities earlier. The success of the aforementioned models can largely be attributed to the cross cross-modal representations obtained by multi-task pretraining on large amount of aligned image-text pairs. Despite state-of-the-ar ahe performance of such models on the downstream REC task, these models, fundamentally, are still a form of two-stage pipeline where image features are extracted using pretrained detectors or proposal mechanisms. We focus on a one-stage architecture variant that allows visual and lingual features to be aligned at the early stages, Although the focus of our work is not to design a better pretraining scheme, we show that our model can outperform the existing tht with proper pretraining.
Transformer-basee Detectors. More recently, DETR (2 and its variants 13 [8]. were proposed to enable end-to-end object detection. DETR reformulates detection as a set prediction tasks and uses transformers decode learnable queries to bounding boxes. Despite state-of-art performance, DETR is disadvantaged by its optimization difficulty and, usually, extra-long training time. While adopting a similar pipeline, our model focus on aligning different modalities to generate contextualized expression-spect referring queries. We also design our model to get rid of Hungarian matching loss by leveraging one-to-one correspondences between predicted bounding boxes and referring expressions, which leads to faster convergence for our model.
3 Approach
Given an image I and a set of query phrases 2, {p:) that we assume to come from an (optional) contextual text source Q, our goal predict a set of bounding boxes B (b, (b.))......... and corresponding segmentation masks S = {s: li=......M one for each query phrase that localizes that phrase in the image. Note, M is the number of phrases referring expressions for a given image I and is typically between 1 and 16 for the Flickr30k [1] dataset.
As shown in Figure I our referring transformer is composed of four components. Given image- (con)text pair, < II e > a cross-modal encoder generates joint image-text embeddings for each visual and textual token- feature columns and word embeddings respectively. Query phrases 2, and image-text embeddings are then fed into a query encoder which produces query phrase embeddings. The decoder jointly reasons across all these query phrase embeddings and decodes multi-task feature, which is then sent to the detection and segmentation head to produce a set of boxes B and masks S. The result is one-staged end-to-end model that solves the REC and RES tasks at the same time. We will now introduce constituent architectural components for the four stages briefly described above.
3.1 Feature Extraction
Visual & Text Backbone. Starting from an initial image I E Rax we adopt the widely used ResNet [5] to generate its low-resolution feature map f, E R For the corresponding
Forr example, a sentence from which noun phrasestrefering expressions p. were parsed. Where contextual text source is unavailable and only one phrase exists, we simply let Q = Qp.
3

---

Cross Attention
Visual-I Lingull
Query Decader
Deteet
Head
Feost
Query
Pawst
Decoder
small
Query Encoder
Head
Image backbone
Encoder
boy
small bay the grass blue bat a ball
A small playing the grass with blue bat and a ball
Query Enooder
Plinse Freender
Text feature
Image feature
Cross-modal Feature
(b)
Figure I: Referring Transformer. An overview of the proposed architecture is shown in (a). For an image and (con)text input, a visual-lingual encoder is used to refine image features, extracted from convolutional backbone, and lingual features, extracted by BERT. A query encoder and decoder produce features for REC and RES heads, given multi- modal features and query phrases. The detailed structure of the query encoder and decoder is shown in (b). Colored squares denote emheddings for corresponding query phrases.
expression or sentence, we use the uncased base of BERT R.C.N, while N the length of the input context sentence.
S to obtain the representation fo
Visual-Lingual Encoder. The visual-lingual encoder is designed to fuse information from multi- modal sources. For cross-modality encoding, is composed of 6 transformer encoder layers. Specifically, given both image and text features, multi-layer perceptrons are applied first project different modalities joint embedding space with a hidden dimension of C. To provide transformer encoders with positional information, we follow g to add cosine positional embedding Psmg for image features and learnable positional embedding Prest for text features. We then concatenate the projected features into a single sequence E CC(HH--N) To distinguish between modalities, we also deign a learnable modal label embedding Liabel [Ling Erewt] which added the original sequences. The visual-lingual encoder then takes sequence as input, and {Pings Prezt, Elabet are fed into each encoder layer. The encoder output is multi-modal feature sequence fur E BC (IIW-)))
transformer encoder based model, which
3.2 Query Encoder and Decoder
The query encoder and decoder aims to encode decode query phrases, conditioned visual-lingual features from the encoder, into an output bounding box and segmentation. In this stage, we first generate embeddings corresponding to each query phrase. These query phrase embeddings are then fed into the decoder together with the visual-lingual features from the encoder to generate outputs.
Encoding Query Phrases. To enable the decoder to generate the desired output (bounding box and/or segmentation) the queries must encode several bits of crucial information. Mainly, (1) encoding of the query phrases, (2) image encoding and (3) phrase-specific optional (con)text information. For phrase encoding in (I) we use a BERT model with pooling heads which share weights with (con)text encoder; this results in the phrase feature vector fp. E R for the i-th referring phrase. We note that because we use visual- lingull encoder, (2) and (3) are jointly encoded in multi-modal features fur described in Section 3.1 above. However, f,i is phrase-agnostic encoding of the image and (exn)text, To generate phrase-specific context, given phrase p:. average pooling is used to extract the phrase-specific context information f<(p.) from the visual-lingual feature sequence as follows:
(p.)
Tp. Ip.
where lp. and Tp, denotes the left and right bounds phrase p; in the original (con) sentence. Finally, given phrase encoding fp. and phrase-specific context f.(p.) we construct phrase queries

---

using a multi-later perceptron:
Pp. = MLP ((f((p:); Ep.
(2)
where Ep E IC is learnable embedding which serves bias to the formed query.
Decoding, In the decoder, self attention layers are used to enable information flow in a dense connected graph of phrase queries. This allows phrase queries to contextualize and refine each other; the inspiration for this step is taken from [ After that, cross attention layer decodes visual- lingual information given the updated phrase query and feature sequence from the encoder. The design of our decoder similar the transformer decoder, except attention is non-causal.
3.3 Multi-task training
In this section, we demonstrate how the decoded phrase-specific query features can be naturally used train multiple heads for different referring tasks (regression for REC and segmentation for RES).
Referring Comprehension/Dettttion (REC). For referring detection tasks, the final output is computed by simple two-layer perceptron over the decoded phrase-specific query features. We let the detection head directly output center coordinates b (x, y, h, w) for the referred image. To supervise the training, we use a weighted sum of an L1 loss and a Generalized IOU loss [45):
Liet = Aiou (b, b) + A |b
b/11-
(3)
The Aoo and dL1 control the relative weighting of the losses in the REC objective.
Referring Segmentation (RES). Following previous work [2]. we design an FPN-like architecture to predict a referring segmentation mask for each phrase expression. Attention masks from the decoder and image features from the visual-lingual encoder are concatenated as the FPN input, while features from different stages of image backbones used skip connections to refine the final output. The last linear layer project the upsampled feature to a single channel heatmap and a sigmoid function is used to map the feature to mask scores E TR. Ho/ 1x//11 The loss for training RES task is:
(4) Here foce is the focal loss for classifying pixels used in (8) Laice is the DICE/F-1 loss proposed in (39. Afooat and Adice are prper-parameters controlling the relative importance of the two losses.
Lseg = foouat + Adice (s,
Joint Training. While it is possible train referring segmentation and referring detection tasks separately, we find that joint training is highly beneficial. Therefore the combined training loss which we optimize is L Leey
Ldet-
Pretraining the Transformer. train B3 Although in this paper we do not use a large pretraining model and lot of data. We found that simple pretraining strategy on the region description splits of Trsual-Genome dataset (25) makes our model achieve comparable and even better performance against some of state-of-the-art pretrained models. Interestingly, we found that although there no ground truth segmentation provided in Visual-Genome, the RES task can still benefit greatly from pretrained models, likely due to the fine-tuned multi-task representation.
Transformers are generally data hungry and requires a lot of data to
4 Experiments
4.1 Datasets
ReTCocoRaaCCCCO//RCCCCCCC
(REC&RES). RefCOCo, RefCOCO+ (55) and RefCOCOg 40 are collections of images and referred objects from MSCOCO (27). On RefCOCO and Ref-
COCO+ we follow the split used in (55) and report scores on the validation, testA and testB splits. On RefCOCOg, we use the RefCOCO-und splits proposed in 40).
Flickr30k Entities (REC). Flickr30k Entities 41 contains 31,783 images and 158k caption
sentences with 427k annotated phrase. We use splits from [41] 42). Bounding boxes and phrase
annotations are consistent with the previous one-stage approaches [51 52 for fair comparisons.
ReferIt (REC). The ReferItGame dataset 24 contains 20,000 images. We follow setup in [ for splitting train, validation and test set; resulting in 54k, 6k and 6k referring expressions respectively.
5

---

Table I: Comparison on REC task. Performance on RefcoccoieerCCCCO+eeCCCOOOO datasets 55 is reported. Ours" denotes that pretraining used. RN50 and RN101 refer ResNet50 and
ResNet101 15 respectively; DN53 refers to DarkNet53 43
backbone.
Models
Visual Features
Preirain Images
Mulū- task
ReCCOCO
ReiCOCO+
ReICOCOg
lestA
testB
val
test
val-u
Two-stage:
CMN [19
VGGI6
None
71.03
65.77
54.32
47.76
RyG-Tree 17
RN101
None
75.06
78.61
69.85
63.51
67.45
56.66
66.95
66.51
EMase-Frase
80
RN101
None
78.35
83.14
71.32
68.09
73.65
58.03
67.99
68.67
MAnNee 56
RNI01
None
76.65
81.14
69.99
65.33
71.62
56.02
66.58
67.27
NMTree
29
RNI01
None
76.41
81.21
70.09
66.46
72.02
57.52
65.87
66.44
One-stage
RCCF 26
DLA34
None
X
81.06
71.85
70.35
56.32
65.73
SSG
DNS3
None
76.51
67.50
62.14
49.27
$8.80
FAOA 51
DN53
None
X
72.54
74.35
68.50
56.81
60.23
49.60
61.33
60.36
ReSC-Large [22
DN53
None
77.63
80,45
72,30
63.59
36
56.81
67.30
67.20
MCN [6
DN53
None
80.08
82.29
74.98
67.16
72.86
57.31
66.46
66.01
Ours
RN50
None
81.82
85,33
76.31
71.13
75.58
61.91
69.32
69.10
Ours
RN101
None
82.23
85.59
76.57
71.58
75.96
62.16
69.41
69,40
Preteained:
ViIBERT 13
72.34
62.61
ERNIE: 1, 14
RNI01
4.3M
75.89
82.37
66.91
UNTIER
L(
RNIO1
4.6M
X
81.41
87.04
74.17
75.90
81.45
66.70
74.86
75.77
VILLA_
12
RNI01
87.48
74.84
76.17
81.54
66. 66.84
76.18
76.71
Ours
RN50
100k
85.43
87.48
79.86
76,40
81.35
66.59
78.43
77.86
Ours*
RN101
100k
85.65
88.73
81.16
77.55
82.26
68.99
79.25
80.01
4.2
Implementing Details
We train our model with A 32]. The initial learning rate is set to lc-4 while the learning rate of image backbone and context encoder
1e-5. We initialized weights in the transformer encoder and decoder with Xavier initialization [14] For image backbone, we experiment with the popular ResNet-50 and ResNet-101 networks [15 where weights initialized from corresponding imagenettpretraiine models. For the context encoder and phrase encoder, we use an uncased version of BERT model
(8) with weights initialized from pretrained checkpoints provided by HuggingFace For data augmentation, we scale images such that the longest side is 640 pixels and follow do random intensity saturation and affine transforms. We remove the random horizontal flip augmentation used in previous work [5]] since we notice it causes semantic ambiguity on RefCOCO, likely due relative location (e.g., left of/right of) specific queries in the dataset.
49
On Flickrš0k dataset, we set the maxium length of context sentence to 90 and maximum number of referring phrases 16. On the ReferIt and the RefCOCO dataset, only phrase expressions are provided and the task aims to predict a single bounding box for each the expressions. In those cases, the context sentence is taken as the referring phrase expression itself. We the maximum length of context sentence on these two datasets to 40. To fairly compare with pretrained methods, use region description split in the VisualGenome 25 pretrain our model. The dataset contains approximately 100k images and we remove the images that appear in Flickr30k Entities and RECCoCO/ReCOOOO//eCOOOO
validation and test avoid potential the pretrained methods, we train the model on pretraining dataset for 6 epoches. We find that longer pretraining schedule gives better performance, but since the focus of this paper not on pretraining methods, we stick to shorter pretraining schedules save computational resources. All experiments are conducted using Nvidia 2080TT GPU with batch size 32. For all the results given, we run experiments several times with random seeds and the error bars are within +0.5%.
data leak. For all
4.3 Quantitative Analysis
Evaluation Metrics. For referring expression comprehension (REC), consistent with prior works, we use precision as the evaluation metric. We mark a referring detection as correct when the snersectiio-ovvr-uniiee (IoU) between the predicted bounding box and ground truth is larger than 0.5. For referring expression segmentation (RES), we reported the Mean loU (MloU) between the predicted segmentation mask and ground truth mask.
Our model addresses REC and RES tasks jointly. We compare their respective performances with the state-of-the-art in Table I and
REC and RES on ReCOOCOReeCCCOO-OReCCce

---

Table 2: Comparison on RES tasks. Performance ReiCocoRefCCCCC+//CCCCCC datasets 55 is reported. Ours" denotes that pretraining used. RN50 abd RN101 refer ResNet50 and
ResNet101 [5 respectively; DN53 refers to DarkNet53 43 backbone.
ReiCOCO
RefCOCO+
RefCOCOg
Methods
Backhone
val
testA
testB
val
testA
test
val
test
time(ms)
DMN 3W
RNTO1
49.78
54.83
45.13
38.88
44.22
32.29
MAnNet
56
RN101
56.51
62.37
51.70
46.67
52.39
40.08
47.64
48.61
378
NMTree
29
RN101
56.59
63.02
52.06
47.40
53.01
41.56
46
47.8
Lang2seg
RN101
58.90
61.77
53.81
46.37
46.95
BCAM
RN101
61.35
63.37
59.57
48.57
52.87
42.13
CMPC 21
RNJO1
61.36
64.53
59.64
49.56
53 44
43.23
CGAN 35
DN53
64,86
68.04
62.07
51 01.03
55.51
44.06
51.01
51.69
LTS
DNS3
65.43
67.76
63.08
54,21
58.32
48.02
54.40
54.25
Ours
RNSO
69.94
63.20
53.45
58.37
Ours
RNJ01
70.56
73.49
66.57
61. 01.08
64.69
52.73
58.73
58.51
41
Ours'
RNSO
73.61
75.22
69.80
65,30
69
56.98
65.70
65.41
Ours
RN101
74.34
76.77
70.87
66.75
70.58
$9.40
66.63
67.39
Eebra walking with tell slidknn out
sches wilking with the bail slicking oua
xpirads where face art loss stans
clocet nod betwan yellow and Mack biles
king
bry
men tn shipal
glos with
Man the sank
basis with
wase part glam chips
loop
feacing
Detection
Attention Map
MCN
Our
Ground
(b)
Figure 2: Qualitative Evaluation. In (a) comparison MCN
36 REC is shown; orange, blue
and red bounding boxes correspond to outputs from MCN, our model and the ground truth. In (b) similar comparison on RES is made. The attention map drawn from the last layer of the decoder. We add mosaic to all human face to protect personal information.
In Table the model is first compared with previous one-stage and two-stage approaches for REC.
Without bells and whistles, we observe a consistent performance boost of +2.7% + 4%/ 2.1% on RefCOCO, +6.6% + 4 8.5% on RefCOCO+ and +4.4%/ + 5.1% on RefCOCOg. To compare with pretrained BERT methods, we use the pretraining strategy discussed in Section 3.3 As results show, our model achieves comprehensive advantage and shows distinct improvement on some splits, even compared advance BERT models that use 40x more data in pretraining. Table illustrates results on RES task in terms MIoU score. can be seen that our model achieves the best performance; substantially better than the statc-of-art. We further observe that pretraining on the REC task gives huge performance boost to the RES task, even when no segmentation mask is used in pre-training, Multi-task training enables the model to leverage performance boost in one task to improve the other.
We show the inference time for our model in Table 2 Our model can directly decode all query phrases an image in parallel, allowing it to reach real-time performance. Importantly, note the corresponding scores for our model in Table I task model that predicts referring detection box and segmentation mask simultaneously. The only related work that shares this property is the MCN [36). which has substantially inferior performance.
and Table and are based on the output of a single multi-

---

Table 3: Comparison with Poote-of-The-Are Methods. Table illustrates performance on the test set datasets in terms of top-1 accuracy (%)
of ReferltGame 24 and Flickr30K Entities
ReferliCiame
Plickr
Inference time
Models
Backbone
test
lest
F'lcckSUlk(ms))
Two-stage
MAnNee
RNTOI
29.04
320
Similarity Net 48
RNI01
34.54
60.89
184
CITE
42
RNI01
35.07
61.33
196
DDPN
RNI01
63.00
73.30
One-alage
SSG
DNS3
$4.24
25
7SGNet
RN50
58, 63.63
FAOA 61
DNS3
60.67
68.71
23
DLA34
63.79
25
LesC-L
32
DN53
64.60
69.28
36
RN50
37(14)
Ours
RNI01
71.42
78.66
40(15)
Ours
RN50
75.49
79.46
37(14)
Ours
RNI01
76.18
81.18
40(15)
REC on Flickr30k-Entittes. For Flickr30k dataset, previous one-stage works [5] 52 extract short phrases from sentences and treated them as separate referring expression comprehension tasks. We argue that in this setting. queries are mostly short phrases and therefore cannot well reflect the model's ability to comprehend them in context. contrast, our model, given an image and a caption (con)text sentence, aims to predict bounding boxes for all referred entities in the sentence. Doing gives several advantages: 1. We are able to contextualize referring expressions given all other referring expressions and (con)text provided by the sentence. 2. Locations for all phrases can be inferred in one forward pass of the network, which saves a lot of computation as compared to previous one-stage time. Note that our task formulation is consistent with some two-stage models 19. but is unique for a one stage approach.
approaches [51] 52 that process one phrase
In Table 3 we compare our results with state-of-the-an methods. Without pretraining, we obtain huge performance boost compared to both previous one-stage (+13.54%) and two-stage (+7.31 %) state-of-the art methods. By using pretrained models, observe that model tends to generalize better on the test set and gives even better performance. We also provide comparison of inference time both per image and (per-expression), since model can amortize inference across expressions. Per-expression, our inference time substantially lower than all prior methods.
REC on ReferIt. Since ReferIt is a relative small dataset, we use a slightly smaller model which contains cross attention layers in the query decoder. The results are shown in Table Our model is able to perform better, by large margin, than even the latest one-stage methods. We also observe consistent boost brought by pretraining.
4.4 Qualitative Analysis
In Figure we show our qualitative comparison with previous state-of-the-art multi-task model MCN 36 The first two rows of Figure Sauree2e () show failure cases of MCN that can be better handled by our model. We observe that MCN appears to fail because it neglects some attributes in referring expression (e.g.. "yellow drink" and "blue striped shirt"), while our model is able to better model the query and pay attention to object attributes. In the last row, we shows several failure cases of our model. For the first case, the query requires the model to have the ability to recognize number "44". For the second and third case, there is visual ambiguity to identify the nearest glass to the bowl or to determine which bear(brown or white) has the longest leg.
In Figure (bb). (b)). we show qualitative comparison in terms of referred segmentation mask. Compared to MCN, our model is able to output more detailed object shape and finer outlines. Moreover, our model shows the ability to handle shadows (e.g., the right bottom of the donut) and occlusions (e.g., the man occluded by another man's 's arm) and predict smoother segmentation mask. We also give result on a challenging case in the last row, where the texture boundary of the two giraffe is hard to distinguish. Despite imperfections, our model still able to focus on the giraffe's head in the foreground and performs much better than MCN. Part of our model's ability to generate fine- -raiggr mask can be explained by better localization ability brought by the REC task, which case the RES head can focus on tuning the shape and boundary of the mask.
8

---

Table 4: Ablation studies. Table on the top ablates our multi-task and pretraining schehme on
RefCOCO/+/g validation set. Table on the bottom ablates on core components our model.
REC
RES
Pretrain
REC Acct
RES Miout 66.03 38.397 54.61
IEI
23.52%
81.08/ 70.02 68.15
81.82 71.13 69.32
69.94/ 60.90 57.69
4.73%
77.96
85.43 76.40 788.43
73.61 65.30 65.70
4.48%
Flickr30k
Model component
-w/o Query Decoder
49.38
-wlo Context Encoder
73.68
Query Encoder
-wlo Context Phruse Feature
42.05
-wo Context Feature
76.64
-wo Phrase Feature
77.02
Full model
78.13
Table 5: Results on RefCOCO+ Dataset with Different Input Resolutions. Our methods corre- spond to the RN50 model without pretraining.
Models
Resolution
RBC(prec@0.5)
RES(MIoU) testA
testA
testB
val
testB
FAOA [51
256x 256
56.81
60.23
49.60
ReSC-Large 52
256x 256
63.59
68.36
56.81
CMPC 221
320x 320
49.56
53.44
43.23
LTS [22
416x 416
54.21
58.32
48.02
MCN 36
416x 416
67.16
72.86
57.31
50.62
54.99
44.69
Ours
256x 256
70.05
73.29
61.48
58.26
61.09
52.20
Ours
320x 320
70.03
73.23
61.52
58.42
61.48
52.34
Ours
416x 416
71.50
75.87
61.71
61.00
64.48
52.44
Ours
640x 640
71.58
75.96
62.16
61.08
64.69
52.73
4.5
Ablation Studies
We first consider the importance of the multi-task setup in Table (top). Results indicate that multi-task training consistently boosts both REC and RES performance on Refcoco/+/g datasets by considerable margin. More specifically, we observe that REC loss helps the transformer to better locate the referred object and converge faster early stages of training. At the same time, RES loss aids the model with more fine- fine-grained grained information on the shape of the referred region, which helps to further enhance the accuracy. IE here is Inconsistency Error metric originally used in [36] to measure the prediction conflict between the REC and RES task. We can see that joint training of RES and REC greatly reduce the inconsistency between the two tasks. Note that our model also has much lower multi-task inconsistency compared to MCN (36), with corresponding IE score of 7-54%.. This shows that our model can do better collaborative learning.
Next, we validate the design of our network architecture. We report our scores on the Flickr30k test sets. In Table ( (bottom), we ablate the model's s major components and features used to form the query. Without (w/o) context encoder indicates that we directly use learnable embedding to encode text; w/o Query Decoder means that we directly use the average pooled feature from the encoder to predict single referred output. We can see that the context encoder plays an important role in providing good textual representation for further multi-modal fusion. Query encoders are also quite important without which we also observe big performance drop. For the ablation query features, we observe that both context feature and phrase features are crucial without which the performance will decrease considerably. The table also showed that the network will not work without guidance of both context and phrase features since we cannot establish a correspondence between multiple queries and outputs in such a case.
In RES and REC tasks, the size of input image matter of trade off between performance and speed, which is largely effected by the network architecture. Despite that our model is designed to be able to process 640 x 640 images at real time speed, we also test our model at different input resolution for reference, as showed in Table Note that for resolution 256 and 320, we adjust strides in the final stage ResNet keep the number of visual features sent into visual-lingus encoder roughly the same.

---

Table 6: Comparison with Concurrent Work. Performance on RelCOCoRerCOCCOHRRCCOO
datasets 55
Ours" denotes that pretraining used. All methods use ResNet101 image backbone.
Visual Features
Pretrain
Multi- task
RefCoCo testA
RefCOCO+
RefCOCOg
Models
Images
val
testB
val
testA
testB
val-u
test-u
One-stage:
VGTR 10
Bi-LSTM
None
79.20
82.32
73.78
63.91
70.09
56.51
65.73
67.23
TransVG 7
BERT
None
81.02
82.72
78.35
64.82
70.70
56.94
68.67
67.73
Ours
BERT
None
82.23
85.39
76.37
71.38
75.96
62.16
69.41
69.40
Pretrained:
MDETR [3]
RoBERTa BERT
200k
86,75
89.58
81.41
79.52
84,09
70.62
81.64
80.89
Ours"
100k
85.65
88.73
81.16
77.55
82.26
68.99
79.25
80.01
4.6 Comparison with Contemporaneous Work
Concurrent and independent to us, very recently, there are some closely related works that use We will briefly discuss some of the differences.
transformers for visual referring tasks 7 I0 23
To begin, 100 focus REC, while our approach is formulated in multi-task setting and solves
both REC and RES tasks simultaneously. In addition, our model faster and is capable of grounding multiple contextualized phrases, while 7 [0] follow previous one-stage approaches and are only able to infer a single expression at a time; leading. our case, to more accurate results. MDETR [23] leverages contrastive loss and soft token loss to help better match bounding boxes to phrases. In contrast, our method adopts a simple and straightforward method to pre-match bounding boxes to phrases using one-to-one matching; this leads to simpler learning objective.
We provide quantitative comparison on REC task with these approaches, based on their reported numbers in Table 6 Compared to 7 10 our model performs substantially better in all, with an exception of RefCOCO testB (where 7 is marginally better), datasets and splits. The biggest improvements can be seen on RefCOCO+, where our model is 10.4% better (or 6.76 points better), than the closest concurrent work of 77) on the Val split; similar sizable improvements are illustrated on other splits, e.g., 9.2% on RefCOCO+ testB. In addition, our approach is considerably faster in runtime, since our model is able to handle multiple queries simultancously (unlike 77 Compared to [23), in a pretrained model setting, we see that model performs similarly RefCOCO and marginally worse on RefCOCO+ and RefCOCOg. This difference can perhaps be attributed to two factors: (1) larger pretrain dataset and longer triaining schedule. (As reported in (23) MDETR takes 224 GPU days to pretrain, while the pretraining for our model is roughly 28 GPU days) (2) using more sophisticated language model (RoBERTa for 23 BERT for us). Limited experiments in Supplemental Material show that indeed, the use of RoBERTa leads to certain improvements. In addition, we setup our method in multi-task setting to solve RES and REC task at the same time, so our formulation while perhaps marginally inferior on REC is more general overall.
5 Conclusions and Future Work
In this work, we present Referring Transformer, a one-step approach to referring expression compre- hension (REC) and segmentation (RES). We jointly train our model for RES and REC tasks while enabling contextualized multi-expression references. Our models outperform ttateoof-the-ar by large margin on five three datasets for REC/ / RES respectively, while achieving real-time runtime.
One limitation for our model is that we follow the setup in previous works [51 [52 and assume that each expression refers to only one region. In the future, we plan to explore learning predict multiple regions for each referring entity if necessary. Large-scale multi-task pretraining has been demonstrated to be very effective for ViLBEERT and other similar architectures; this is complementary to our focus in this paper, and we expect such strategies to further improve the performance.
6 Acknowledgments and Disclosure of Funding
This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chair, NSERC CRC, NSERC Discovery and Discovery Accelerator Grants. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government Canada through CIFAR, and companies sponsoring the Vector Institute www. www.weetooinsiiittcccmmmmmm ai ai/#partners Additional hardware support was provided by John R. Evans Leaders Fund CFI grant and Compute Canada under the Resource Allocation Competition award.
10

---

References
[1] M. Bajaj, L. Wang, and L. Sigal. G3raphground: Graph-based language grounding In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 4281-4290, 2019.
[2] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillow, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision (ECCV), pages 213-229, 2020.
[3] K. Chen, R. Kovvuri, and R. Nevatia. Query-guided regression network with context policy for phrase grounding. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 824-832, 2017.
[4] X. Chen, L. Ma. J. Chen, Z. Jie, W. Liu, and J. Luo. Real-time referring expression comprehension by single-stage grounding network. arXiv preprint arXiv:1812.03446. 2018.
[5] Y-C. Chen, L. I. Yu, A. EI Kholy, E. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision (ECCV), pages 104-120, 2020.
[6] Y. W. Chen, Y. H. Tsai, T. Wang, Y. Y. Lin, and M. H. Yang. Referring expression object segmentation with caption-aware consistency. In Proceedings of the British Machine Vision Conference (BMVC), 2019.
[7] J. Deng, Z. Yang. T. Chen, W. Zhou, and H. Li. TransVG: End-to-end visual grounding with transformers. arXiv preprint axXiv:214.00855 08541, 2021.
[8] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx; Human Language Technologies, Volume 1 (Long and Short Pupers), pages 4171-4186 2019.
[9] P. Dogan, L. Sigal, and M. Gross. Neural sequential phrase grounding (segground). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4175-4184, 2019.
[10] Y. Du, Z. Fu, Q Liu, and Y. Wang. Visual grounding with transformers. arXiv preprint arXiv:2105.0422 2021.
[11] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, Centernet: Keypoint triplets for object detection. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6569-6578, 2019.
[12] Z. Gan, Y.-C Chen, L. Li, C. Zhu, Y. Cheng, and J. Liu. Large-scale adversarial training for vision-and- language representation learning. In Proceedingy of the International Conference Neural Information Processing Systems (NeurlPS), 2020.
[13] P. Gao, M. Zheng, X. Wang. J. Dai, and H. Li. Fast convergence of detr with spatially modulated co-attention, arXiv preprint arXiv:2101.07448, 2021.
[14] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249 256, 2010.
[15] K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual lcarning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[16] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pages 2961 2961-2969, 2017.
[17] R. Hong. D. Liu, X. Mo, X. He, and H. Zhang. Learning to compose and reason with language tree structures for visual grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019.
[18] R. Hu, M. Rohrbach, and T. Darrell. Segmentation from natural language expressions. In European Conference on Computer Vision (ECCV), pages 108-124, 2016.
[19] R. Hu, M. Rohrbach, J. Andreas, T. Darrell, and K. Saenko. Modeling relationships in referential expressions with compositional modular networks. In Proceddings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1115-1124, 2017.
[20] Z. Hu, G. Feng, J. Sun, L. Zhang, and H. Lu. Bi-directional relationship inferring network for referring image segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
11

---

[21] S. Huang. T. Hui. S. Liu, G. Li, Y. Wei, J. Han, L. Liu, and B. Li. Referring image segmentation via cross-modal progressive comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
[22] Y. Jing, T. Kong, W. Wang. I.. Wang, I.. Li, and T. Tan. Locate then segment: A strong pipeline for referring image segmentation. arXiv preprint 103. 16284, 2021.
[23] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. ADDER--mdulatee detection for end-to-end multi-modal understanding. arXiv preprint (tl::2044.276.. 2021.
[24] S. Kazemzadch, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNIP), pages 787-798, 2014.
[25] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J, Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV), 123(1):3 73, 2017.
[26] Y. Liao, Liu, G. Li, F. Wang. Y. Chen, C. Qian, and B. Li. A real-time cross-modality correlation filtering method for referring expression comprehension. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[27] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pages 740-755, 2014.
[28] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
[29] D. Liu, H. Zhang, F. Wu, and 7.-J. 7ha. Learning to assemble neural module tree networks for visual grounding. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 4673-4682 2019.
[30] X. Liu, Wang, J. Shao, X. Wang, and Li. Improving referring expression grounding with cross- modal attention attention-uuided guided crasing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2019.
[31] J. Long, F. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431- 3440, 2015.
[32] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representcatinss (ICIR),, 2019.
[33] J. Lu, D. Batra, D. Parikh, and S. Lee. ViLBERT: Pretraining task-agnostic visiolinguistic representations vision-and- lan-aauugg tasks. Proceedings of the International Conference on Neural Information Processing Systems (NeurlPS), pages 13-23, 2019.
[34] Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lec. 12-in-1: Multi-task vision and language repre- sentation learning. In Proceedings of the IEFE Conference on Computer Vision and Pattenn Recognition (CVPR), pages 10437-10446, 2020.
[35] G. Luo, Y. Zhou, R. Ji, X. Sun, J. Su, C.-W. Lin, and Q. Tian. Cascade grouped attention network for referring expression segmentation. In ACM International Conference Multimedia, 2020.
[36] G. Luo, Y. Zhou, X. Sun, L. Cao, C. Wu, C. Deng, and R. Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEFF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[37] Mao. J. Huang, A. Toshev, O. Camburu, A. Yuille, and K. Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 11 11-20,22 2016.
[38] E. Margffoy-Tuaa J. C. Pérez, E. Botero, and P. Arbeláez. Dynamic multimodal instance segmentation guided by natural language queries. In Luropean Conference on Computer Vision (ECCV), 2018.
[39] F. Milletari, N. Navab, and S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In International Conference 3D Vision (3DV), pages 565-571, 2016.
12

---

[40] V. K. Nagaraja, V. I. Morariu, and L. S. Davis. Modeling context between objects for referring expression understanding. In European Conference on Computer Vision (ECCV), pages 792-807, 2016.
[41] B. A. Plummer, L.. Wang. C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebrik, Flickr30k entities: Collecting region-to-phle to-raasse correspondences for richer maage-to-senteree models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2641-2649. 2015.
[42] B, A. Plummer, P. Kordas, M. H. Kiapour, S. Zheng, R. Piramuthu, and S. Lazebnik. Conditional image-text embedding networks. In European Conference on Computer Vision (ECCV), pages 249 264, 2018.
[43] J. Redmon and A. Farhadi. Yolov3: An incremental improvement. arXiv preprint 1rXiv:1804.02767, 2018.
[44] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 39(6): 1137-1149, 2017.
[45] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 658-666, 2019.
[46] A. Sadhu, K. Chen, and Nevatia. Zero-shot grounding of objects from natural language queries. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4694-4703, 2019.
[47] H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from transformers. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
[48] L. Wang, Y. Li, J. Huang, and S. Lazebnik. Learning two-branch neural networks for image-text matching tasks. IEEE Transactions Pattern Analyxis and Machine Intelligence (TPAMI), 41(2):394 2018.
[49] T. Wolf, L.. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma. Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame. Q. Lhoest, and A. M. Rush. Transformers: state-of-the-are art natural language processing. In Proceedings the Conference on Empirical Methads in Natural Language Processing (EMNLP), pages 38 38-45, 2020.
[50] S. Yang, G. Li, and Y. Yu. Propagating over phrase relations for one- stage visual grounding. In European Conference on Computer Vision. pages 589 605. Springer. 2020.
[51] Z. Yang, B. Gong, L. Wang, W. Huang, D. Yu, and J. Luo. A fast and accurate one-stage approach to visual grounding. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2019.
[52] Z. Yang. Т. Chen, L. Wang. and Luo. Improving one-stage visual grounding by recursive sub-query construction. In European Conference on Computer Vision (ECCV), 2020.
[53] L. Ye, M. Rochan, Z. Liu, and Y. Wang. Cross-modal self-attention network for referring image segmen- tation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[54] F. Yu, J. Tang, W. Yin, Y. Sun, H. Tian, H. Wu, and H. Wang. Emie-vil: Knowledge enhanced vision- language representations through scene graph. arXiv preprint arXiv:2006. 16934, 2020.
[55] L. Yu, P. Poirson, S. Yang. A. C. Berg, and T. L. Berg. Modeling context in referring expressions. In European Conference on Computer Vision (ECCV), pages 69 85, 2016.
[56] L. Yu, Z. Lin, X. Shen, J. Yang, X. Lu, M. Bansal, and T. L. Berg. Mattne:: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1307-1315, 1315, 2018.
[57] Z. Yu. J. Yu, C. Xiang, Z. Zhao, Q Tian, and D. Tao, Rethinking diversified and discriminative proposal generation for visual grounding. International Joint Conference on Artificial Intelligence (IJCAI) 2018.
[58] X. Zhu, W. Su, L. Lu, B. L. Wang, and J. Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference Learning Representations (ICLR), 2021.
13


Reviews:
[{"review_id": "uyHdXteln8R", "reviewer": "Reviewer_e6Cm", "summary": "This paper proposes a Transformer-based multi-task framework for visual grounding tasks (referring expression comprehension/segmentation). The core part of this model (comparing with DETR) is the Query Encoder which builds the multimodal query embedding for each referring phrase to decode the grounding results. Extensive experiments and ablations illustrate that the proposed method achieves state-of-the-art performance.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1) REC/RES tasks can be viewed as a general phrase grounding task. Utilizing DETR architecture to formulate phrase grounding task as a set prediction problem is intuitive and straightforward.\n\n2) Extensive experiments show the proposed model achieves state-of-the-art performances on both REC and RES tasks. Training of this model is simpler and faster by removing the Hungarian matching loss.\n\nWeaknesses\n1) For the RES task, authors discuss the limitations of previous methods from the perspective of results, but do not clearly explain why the proposed model can achieve huge performance gains. Considering that the proposed model mainly adds a Query Encoder based on DETR to construct multimodal query embeddings for the referring phrases, could the powerful capacity of this model be attributed to the strong multimodal modeling ability of Transformers? I hope authors can further elaborate their contributions and technical novelty comparing with DETR. In addition, what the effects of focal loss and Dice loss which are not adopted by previous methods?\n\n2) Authors argue that an explicit constraint tends to downplay the quality of the final predicted mask since the feature map from the REC branch can blur out fine-grained region shape information needed by the RES branch. I think there could be some quantitative analysis to show that applying the explicit constraints of MCN on the proposed model will downgrade the RES performance to support the above argument.\n\n3) Line 325-326 “Query encoders are also quite …”. But the model component mentioned in Table 4 (right) is “Query Decoder”.\n\n4) How to initialize the designed learnable modal label embedding E_label mentioned in line 160? Could authors give a more detailed explanation?\n\n\n---- Update ----\n\nAfter reading the comments of other reviewers, I agree with Reviewer 7Fb1 that this paper is an intuitive and straightforward application of DETR for the general phrase grounding task, but lacks some novelty comparing with other variants of DETR. The removal of Hungarian matching loss reduces the complexity of the proposed model, but I don’t recognize it as a contribution since the one-to-one correspondence between phrase and referred object naturally makes the Hungarian matching loss unnecessary. In addition, other losses used in this paper are more like performance tricks rather than reasonable design for the phrase grounding task. Therefore, I lean towards rejection.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. see main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "qs1Jietopv0", "reviewer": "Reviewer_7Fb1", "summary": "This paper proposes a simple one-stage multi-task framework for visual grounding tasks resorting to a transformer architecture. The proposed method achieves state-of-the-art performance on both referring expression comprehension and segmentation tasks.\n ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a simple one-stage multi-task framework for visual grounding tasks resorting to a transformer architecture. The proposed method achieves state-of-the-art performance on both referring expression comprehension and segmentation tasks.\n\nStrengths:\n- The paper does a through literature review in multiple research directions. Built upon the transformer models, the paper proposes an effective framework for REC and RES.\n- The authors did experiments on multiple datasets and studied various backbones, which could provide insights for future work.\n\nWeaknesses:\n- This paper is an application of the Transformer model to a new task, which is ok, but incremental and less exciting. \n- Using multitask learning for REC and RES is not new. There are already some multitask learning methods for REC and RES, for example, Luo et al., \"Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation\", CVPR 2020. \n- The paper claims “As with other transformer-based architectures, we show that pre-training can further improve the performance and both vanilla and pre-trained models outperform state-of-the-art on both tasks by significant margins”. But for all the pretrained methods, the model is pre-trained for only 6 epoches, making the conclusion not reliable. I understand that this methodological concern can be addressed with more computing resources. But if that is the case, then I would suggest limiting the scope of the experiments --- less but longer training on fewer datasets. \n\n--------------\n\nThe authors' response resolved my concern on training so I raised the score. However, method-wise, this work is sort of applying the multitask learning idea in Luo et al. on top of the Transformer models (either is not new on REC and RES tasks). This work is a combination of different tricks that were proposed before though it achieves the SOTA performance (no new conceptual thinking on REC and RES). So I find it less exciting and insightful.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "ii-AdPKl9ux", "reviewer": "Reviewer_u5gv", "summary": "The work explores the tasks of referring expression comprehension (REC) and segmentation (RES) as well as the more general visual grounding. This proposed approach falls into a group of \"one-stage\" methods, in that it does not rely on an initial step of obtaining region proposals. The method has an end-to-end Transformer-based design, and it jointly optimizes both REC and RES objectives in a multi-task learning framework. The authors also argue for the use of contextualized query encoding. The experiments are conducted on 4 standard tasks/benchmarks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=Originality\n- While the overall architecture is not identical to any prior work, the key proposed ideas have been explored previously: multi-task REC/RES training has been proposed in [36], while [a] has explored the role of linguistic context from the full caption.\n[a] Yang, Sibei, Guanbin Li, and Yizhou Yu. \"Propagating Over Phrase Relations for One-Stage Visual Grounding.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\n=Quality\n- Overall, the paper seems compelling, with a clear delivery, broad set of experiments on 4 tasks and performance improvement across the board. \n- At the same time, the ablation study is somewhat limited, especially in terms of specific architectural design choices. E.g. the proposed decoding mechanism (self-attention graph) is not ablated (by e.g. comparing it to a transformer design).\n- Moreover, I would like to see an ablation study performed on REC or RES task in addition to the Flickr30kEntities dataset. \n- I would also like to see more analysis on the source of improvement. E.g., if we consider the task of RES, the vanilla model (w/o joint REC training) already achieves 58.39 (from Tab 4 left), improving over the best prior work (54.21), so what is the source of this improvement? Is it perhaps the effect of a different backbone and/or something else? Generally, what explains the better/more fine-grained quality of the predicted segmentation masks? (Which does not appear to be the focus of this work?)\n- The experimental setting on the Flickr30kEntities dataset is different from majority of prior work, in that the authors incorporate the full caption in addition to the individual phrases. While I do believe that this is a meaningful thing to do, I would like to see a clear comparison between prior works that do *not* use the caption and a version of the method w/o a caption, and prior works that *use* the caption vs. the full proposed model. If I read the Table 4 (right) correctly, w/o Context Encoder is comparable in performance to prior works (in Table 3). Also, the authors mention that some prior work also assumes having access to the full caption [1,9] (L282) but those are not reported in Table 3.\n- Overall, the improvement from the joint training is larger for the RES task than for the REC task (~1%). The benefit of RES to REC is less obvious.\n\n=Clarity\n- The authors mention generating \"queries\" throughout the paper (L9, ), but it is not obvious whether they are referring to actual generated text, predicted textual embeddings or whether it refers to \"queries\" as often used in Transformer terminology (query/key/value). This should be clarified early on.\n- Fig 1 is rather confusing! It is hard to parse and relate to the main text. E.g. the names used in Fig 1 and in the main text (L138-14) are not consistent. I also missed the fact that (a) and (b) were separate figures as they appear to be one figure, while in fact (b) is a \"zoomed in\" version.  I do not follow what the \"Encoder outputs\" refers to. The legend colors are also somewhat ambiguous. Overall, Fig 1 does not help in following along with the method description.\n- I do not entirely understand Sec. \"Encoding Referring Queries\", i.e. the contextualized encoding of input queries. (a) What exactly happens in the tasks of REC/RES where no external caption is available, i.e. would not this be redundant to the query itself? Why should it help then? (b) How does this apply to input image queries (in image-text retrieval)?\n-  I also do not understand how the REC/RES task is framed when there are multiple referring expressions associated with the same image, would those be inferred jointly? Would both of them be used as context for each other? If that is true, how is that comparable to prior works?\n\n=Significance\n- The work follows several recent (concurrent) works in an attempt to perform visual grounding in an end-to-end manner in a Transformer-based design. I believe such efforts are of interest to the community, and unifying multiple tasks (REC/RES etc) seems like an additional benefit.\n\n=Related Work\n- It should be acknowledged that [29] and [56] also perform both REC and RES tasks.\n- The paper misses to cite another one-stage approach, [a]. This work also explores the role of linguistic context from the full caption!\n[a] Yang, Sibei, Guanbin Li, and Yizhou Yu. \"Propagating Over Phrase Relations for One-Stage Visual Grounding.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\n=Other\n- L19: The connection between visual grounding and object detection is not that direct, this misrepresents things a little. I.e., in grounding a text phrase is given as *an input*, while in detection we would need to *output* the label (or text phrase, in a generalized case).\n- L155 perceptions => perceptrons\n- L138 \"four stages\" is confusing, as you are arguing that your approach is one-stage.\n- L156: unclear why transformers are permutation invariant? it is common to include the positional encoding in the transformer models\n- From Table 1 we see the biggest gains on the RefCOCOg dataset, any intuition for why that is?\n\n====== UPDATE\n\nI appreciate the additional ablations/comparisons provided by the authors, as they shed more light on the proposed approach. I also found most of the clarifications helpful. The authors have provided additional details re comparison to [a], and promised to include a comprehensive comparison in the paper.\n\nWhile I do not strongly share the novelty concerns expressed by Reviewers 7Fb1 and e6Cm, I agree that the proposed method builds upon several ideas expressed in prior work. (And who does not?..)\n\nI still see value in the proposed method, as explained in the \"Significance\" section above: this one of the first attempts to establish an end-to-end visual grounding pipeline. While it was not required, the authors have compared to several concurrent works in their supplemental material, and showed superior/competitive results.\n\nThus, I still recommend acceptance and raise my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not present any discussion on potential negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a Transformer-based model for the tasks of referring expression comprehension and segmentation. The reviewers agreed that the techniques used by the paper are not entirely novel and have been individually explored before (especially compared to models like DETR and its variants), even though their combination leads to reasonable performance gains on both tasks. I encourage the authors to add in the ablation studies and analyses from their responses, plus provide better comparison (both conceptually and empirically) with respect to existing methods in their revision.
Decision: Accept (Poster)