Referencing Where to Focus: Improving Visual Grounding with Referential Query
Authors: Yabing Wang
Zhuotao Tian
Qingpei Guo
Zheng Qin
Sanping Zhou
Ming Yang
Le Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual Grounding
Primary area: other
TL;DR: Our proposed RefFormer can iteratively refine the target-related context and generate referential queries, which provide the decoder with prior context.

Abstract: Visual Grounding aims to localize the referring object in an image given a natural language expression. Recent advancements in DETR-based visual grounding methods have attracted considerable attention, as they directly predict the coordinates of the target object without relying on additional efforts, such as pre-generated proposal candidates or pre-defined anchor boxes. However, existing research primarily focuses on designing stronger multi-modal decoder, which typically generates learnable queries by random initialization or by using linguistic embeddings.  This vanilla query generation approach inevitably increases the learning difficulty for the model, as it does not involve any target-related information at the beginning of decoding. Furthermore, they only use the deepest image feature during the query learning process, overlooking the importance of features from other levels. To address these issues, we propose a novel approach, called RefFormer. It consists of the query adaption module that can be seamlessly integrated into CLIP and generate the referential query to provide the prior context for decoder, along with a task-specific decoder. By incorporating the referential query into the decoder, we can effectively mitigate the learning difficulty of the decoder, and accurately concentrate on the target object. Additionally, our proposed query adaption module can also act as an adapter, preserving the rich knowledge within CLIP without the need to tune the parameters of the backbone network. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method, outperforming state-of-the-art approaches on five visual grounding benchmarks.

Referencing Where to Focus: Improving Visual Grounding with Referential Query
Yabing Wang ' Zhuotao Tian 2 Qingpei Guo
Zheng Qin 1 Sanping Zhou Ming Yang " Le Wang 1x National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi an Jiaotong University Harbin Institute of Technology, Shenzhen, China AntGroup (wyb7wyb7 qinzheng) xjtu. edu. cn tianzhuotao@1ing inkk cuhk. edu. hk {spzhou, lewang)@x @uw.x. edu cn fqingpei gqp, m. e.yyang)oantgroup.com
Abstract
Visual Grounding aims to localize the referring object in an image given a natural language expression. Recent advancements in DETR- based visual grounding meth- ods have attracted considerable attention, as they directly predict the coordinates of the target object without relying additional efforts, such pre- pre-generated pro- posal candidates or pre-defined anchor boxes. However, existing research primarily focuses designing stronger multi-modal decoder, which typically generates learnable queries by random initialization or by using linguistic embeddings. This vanilla query generation approach inevitably increases the learning difficulty for the model, as it does not involve any target-related information at the beginning of decoding. Furthermore, they only use the deepest image feature during the query learning process, overlooking the importance of features from other lev- els. To address these issues, we propose a novel approach, called RefFormer. It consists of the query adaption module that can be seamlessly integrated into CLIP and generate the referential query to provide the prior context for decoder, along with a task-specific decoder. By incorporating the referential query into the decoder, we can effectively mitigate the learning difficulty of the decoder, and accurately concentrate on the target object. Additionally, our proposed query adaption module can also act as an adapter, preserving the rich knowledge within CLIP without the need to tune the parameters of the backbone network. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method, outperforming state-of-the-as afrthe-rrt approaches on five visual grounding benchmarks.
Introduction
Visual grounding is a challenging multi-modal task that involves localizing a specific object based on given natural language description. This task requires algorithms to comprehend fine-grained human language expressions and accurately establish correspondences with the target objects. In recent years, has gained significant attention in research due its potential for advancing vision-language understanding, such as cross-modal retrieval 42
and image captioning [4] 29
*Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Quertess
Visual Backbone
Task- cific
Decoder
Amanis shootingletes
Linguistic
Refine Update
Backbone
Rounding Box
(4) Existing Method
Visual
Backbone
Tack spo--speeffii
Decoder
Query Adaption
Qbeeties
$ 6
-...
Aman
Lingulstic Backbone
Bocanding Box pagusagy
shooting liitest
(b) Our Proposed Method
(c) Attention Map
Figure I: Comparison of DETR-like method and our proposed method for visual grounding. (a) The existing method typically adopts the random initialization queries directly into the decoder to predict the target object. (b) We introduce the query adaption module (QA) to learn target-related context progressively, providing valuable prior knowledge for the decoder. (c) The attention map of the last layer in every QA module and decoder (bottom). respectively.
Existing works in this field typically follow the object detection framework and incorporate multi- modal fusion to tackle this task. Earlier studies [52] 12
mainly focus on a two-stage pipeline, which first generates a set of region proposals using object detectors, and then finds the best- matched region by interacting these regions with linguistic expressions. However, the performance of this method is limited by the quality of the generated region candidates. To address this issue, some studies 48 47 adopt the one-stage pipeline, which removes the proposal generation stage. Unfortunately, these methods make dense predictions with a sliding window over pre-defined anchor boxes, resulting in sub-optimal performance due to the failure to capture object relations effectively. Recently, some methods [17 7 10 21 8 36) inspired by the DETR [T] structure, which adopt a standard multi-modal transformer framework to establish the multi-modal correspondence (as shown in Figure [ (a)). These methods predict bounding boxes of target objects directly from learnable queries, eliminating the need for extra efforts to obtain candidates, such as region proposals or predefined anchor boxes.
27
3
58
While these methods have shown promising results, their primary focus remains on designing stronger multi-modal decoders. By contrast, much less work has been done to improve the learnable queries, which have been gained extensive attention in the object detection field. The queries that are inputted to the decoder these methods are typically generated through random initialization or by utilizing linguistic embeddings. We argued that this vanilla approach has two critical issues: i) this target-agnostic query inevitably increases the learning difficulty of the decoder. ii) During the query learning process, these methods tend to focus solely the deepest visual features of the backbone, overlooking the texture information that is crucial for the grounding task and present in low and mid-level features, emphasized by 38).
Drawing from these discussions, this paper seeks to address two critical research questions: i) Can produce the target-related referential queries for the decoder to alleviate the learning difficulty that the decoder faces? and ii) How can we effectively incorporate the multi-level visual context in information into the query learning process? We believe that tackling these issues together would promote the learnable query to more comprehensively and accurately learn the corresponding target object information in the image for the visual grounding task.
Considering CLIP (34] carries rich visual- linnua-angg alignment knowledge, thus we adopt it as the backbone of our approach. Existing methods typically apply CLIP on the visual grounding by fine- tuning its parameters, as CLIP's training objective to match entire images with text descriptions, rather than capturing fine-grained alignment between regions and textual elements. This may risk losing the general knowledge of CLIP and require significant computational resources. To tackle the challenges mentioned above, we propose a novel approach called RefFormer. Our approach incorporates query adaptation (QA) module to generate referential queries, which provide the decoder with target-related context (as illustrated in Figure I (b)). By strategically inserting QA

---

module into different layers of CLIP, the query adaptively learns target-related information from multi- level image feature maps, and iteratively refines the acquired information layer by layer. Furthermore, our proposed Reformer can also act an adapter, enabling CLIPP to keep frozen and preserve the original rich knowledge. adopts the bi-directional interaction scheme, performs the multi-modal fusion by incorporating small number of trainable parameters, and residually injects new task- specific knowledge into CLIP throughout the entire feature extraction process. Extensive experiments conducted on five popular visual grounding benchmarks (i.e., Re/COCO/+/g [53] 31), Flickr30K 33 and Referl(Game [8] demonstrate the superior performance of our proposed method.
Our contributions can be summarized as follows: (1) Unlike the previous methods that focus on designing sophisticated multi-modal decoders, we further improve the learning process of the learnable queries, crucial aspect that has been overlooked in existing work. (2) We propose a query adaption module (QA), which can adaptively capture the target-related context, providing valuable referential knowledge for the decoder. (3) We conduct extensive experiments on five visual grounding benchmarks, demonstrating the effectiveness and potential of our method.
2
Related Work
2.1 Visual Grounding
Visual grounding aims to ground the target objects based on natural language descriptions by understanding the given images and expressions. Early work [52 12 27 3 58 primarily focuses on two-stage methods, which formulates the grounding task as a matching task. These methods employ object detectors to generate proposal candidates and then identify the best-matched candidate based on the matching score computed between each proposal and the referring expression. For example, MAtNee [52) proposes to decompose the language expression into three phrase embeddings, which are used to trigger three separate visual modules. While achieving successful performance, two-stage methods heavily rely on the quality of the generated proposals. Based on this, some studies 48 47
have been dedicated to one-stage methods to remove the proposal generation stage. These methods typically fuse visual features and language features first and then densely regress the bounding box on each position of the feature map grid. For instance, FAOA [48] incorporates linguistic embedding into the YOLOv3 detector to establish one-stage pipeline, balancing between accuracy and speed.
have emerged, which leverages the self-attention mechanism to effectively capture intra- and inter-modality relationships and achieve improved performance. Among these methods, the mainstream approach 17 10 21 & 36) adopts DETR-like structures to decode bounding boxes from learnable queries. For example, Transvg 7 and Transvg++ [8] employ a standard multimodal transformer framework. along with the REG token, to establish multi-modaa correspondence and predict the coordinates of the referring object. Notably, the performance improvement of these methods primarily arises from the design of stronger backbones learnable queries, which have received considerable attention in object detection field.
Recently, transformer-based visual grounding methods 17
21
23
57
40
50
36
10
modtt decoders. In this work, we focus on the design of
2.2 Learnable Queries in DETR and Its Variants
In the object detection field, DETR presents end-to-end object detection model that is built in an
encoder-decoder transformer architecture. However, suffers from slow training convergence. To
address this issue, some follow-up works [55] 19
24 54 solve this issue by optimizing the learnable queries in DETR. For instance, Anchor DETR [45] directly treats 2D reference points
49 45
20
26
as queries, while DETR (24 further investigates the role of queries in DETR and proposes the use of 4D anchor boxes as queries. In contrast to these model-level improvements, DN-DETR 20] introduces query denoising training mitigate the instability of bipartite graph matching, which is further enhanced by DINO 54
Additionally, similar research have been explored in other tasks 22 13 37
For example, EaTR [3
formulates a video as a set of event units and treats video-specific event units as dynamic moment queries in video grounding tasks. MTR++ 37 introduces distinct learnable intention queries generated by the k-means clustering algorithm to handle trajectory prediction across different motion modes in motion prediction tasks.
3

---

CLIP
Image
Decoder
QA
QA
QA
Prediction
A man shooting kites
Hc
Expression
CLIP
Figure 2: Overview of RefFormer. It adopts DETR- like structure. consisting of query adapta- tion (QA) module that seamlessly integrates into various layers of CLIP, along with task-specific decoder. By incorporating the QA module, RefFormer iteratively refine the target-related context and generate referential queries, which provide the decoder with prior context.
Preliminary
Considering the impressive vision-language alignment capability of CLIP, we take as the backbone of method to extract image and text representations, and keep the parameters frozen during training. The feature extraction process can be represented as follows:
Image Encoder. For input image V E #xxx3 it is divided into N non-overlapping patches of size P x P, where N₂- pr These patches are then flattened into a set of vectors, represented
{x} E R Ti=-- Next, these vectors transformed into token embeddings using linear projection layer (-). Furthermore, classification token X.ls R2 is added at the beginning of token embeddings. Subsequently, positional embeddings E, are incorporated, and layer normalization (LN) is applied. This process be expressed follows:
z LN([x (X.)) + E")
(1)
where [:] denotes the concatenate operation. The sequence of tokens Z is then passed through transformer layers. Each transformer layer comprises two submodules: the multi-head self-attention (MHSA) and the multilayer perceptron (MLP), with each submodule preceded by layer normalization.
Z (HS ')) z i= 1,.,,
Z ML.((LN(Z.) + Z
where Z NxD denote the output of i-th transformer layer.
Text Encoder. Given an referring expression T, is first transformed into a sequence of word embeddings using lower-cased byte pair encoding representations X,, The word embeddings are bracketed with the [SOS] and [EOS] tokens, producing a sequence of length N₁. Similar to the image encoder, these tokens summed with positional embeddings E, and passed through the L transformer layers to extract the text representations:
Z: MHSA(LN(Z)))) + 2 1,.,,,
Z: MLP(LN((Z)) + +
(5)
where Z =
[xeosi Xx; Xeos] + E,, representing the word embedding layer in text encoder.
4 Method
The framework shown in Figure] 2 the following, we first describe our query adaptation module in Section 4.1 We then introduce our decoder that decodes with referential query and training
objectives in Section 4.2 and Section 4.3 Furthermore, we extend RefFormer dense grounding
task in Section 4.4 Finally, provide discussion in Section 4.5

---

CAMF
7
2/
Down
CAMF
TR
2
Figure 3: Illustration of our proposed Query Adaption Module, which mainly consists of CAMF and TR modules to generate the referential queries and promote the multi-modal features interaction. "R" represents the feature modulation.
4.1 Query Adaptation Module (QA)
In this section, we propose QA module (as shown in Figure that can generate the referential query to provide the decoder with the target-related context, thereby enhancing the decoder's 's grounding capabilities. Importantly, our approach incorporates multi-level features into the query learning process, enabling the queries capture more comprehensive target object information and can be refined layer by layer. Furthermore, QA can also act as an adapter, eliminating the need to fine-tune the entire parameters of the backbone.
Down-pree nown-projection. Considering the image and language representations Z and Zj obtained from the i-th layer of the backbone, we initially use the MLP layers (ba(+) and (-) project them to lower-dimensional features reduce the computation memory:
F Bnd (Z)). F; (ia(Z))
Condition Aggregation and Multi-modal Fusion (CAMF). We randomly initialize N₂ learnable queries Q E N2 where D; denotes the dimension after projected. These queries specifically designed to capture potential target object context. Next, we concatenate these queries with the image features and input them, along with the language features into the CAMF block. Specifically, the CAMF block mainly consists of a cross-attention layer, which takes the image and query features [Q: F.] and language features F, the query respectively. This approach enables us to not only incorporate the expression condition into the learnable queries Q but also to extract relevant information from other modalities, thereby facilitating the fusion of target-related cross- modal features. Besides, we incorporate two learnable regulation tokens Te, F, E RR to modulate the final output of each QA. This process can be formalized as follows:
F F, F.F Fv. Q F. MHCA((T Q
(7)
Q' = LN(()) + Q F LN(F,) + F F. MHCA((T. F. F. = LNFF;) FF;;
(8)
where Q' represents learnable queries that output from the previous QA, while Q" are randomly initialized. The symbol [:] indicates the concatenate operation, and M HCA(,,) and (N(-) denote the multi-head cross-attention layers and layer normalization, respectively.
Telget-related Context Refinement (TR), Following this, we feed the queries Qe and multi-modal chanced feature maps F and FÚ into the TR block. First, we use the queries Q that have aggregated conditions to interact with the multi-modal enhanced image feature maps F., refining the target- related visual context within them.
Q MHCAA(Q' F.), Q' LL(MMP((L)) Q
(10)
Moreover, for feature maps F' and F; that have aggreaged other modality information, the self-attention to further enhance their target-related contextual semantics:
F, F M MHSA F.J. F P.), G LN(MLP((F)) F
(11)
MHSA((F, F.F.F. G; LN(MLP(())) + F
(12)
Up-projection. Finally, we utilize MI.P to restore the channel dimension of the image and language features back to their original sizes. These features are then passed inputs to the next layer of the
5

---

backbone in residual manner. Prior to this, we utilize the regulation token to modulate the features G, and Gr, which helps prevent the multi-modal signal from overpowering the original signal.
Z = D'r x (T.)) Z Z:- = 22., x (T.)) + Zz
(13)
where (.) and (.) denote the MLP layer, and (-) denotes the sigmoid function.
Finally, by iteratively performing the above process, the queries Q can progressively focus on the target-related context, and generate the referential queries to provide the prior context for the decoder.
4.2 Decoding with Referential Query.
Language-guided Multi-level Fusion. By inserting the QA at different layers of CLIP, the referential queries can be adaptively updated using the multi-level image feature maps. Additionally, to enhance the image features in decoder, aggregate the multi-level visual features under the language guidance to yield language-aware multi-level image features. Specifically, given a multi-level image feature set { (including low, mid, and high levels), where k E IC represents selected layer index, we inject the language features Ziast (the final output of the text encoder) into each level of image features using MHCA:
Hi... = -me (Z)aat) H & (2*) H = MHCA(H H Hr...) + H EC
(14)
(15)
where (m)(.) and mm (-) denote the linear project function used to map features to the same dimension. Besides, H.... represents the [SOS] token in H,, which extracts the global information of the text. Subsequently, the multi-level language-awar image features are produced by simple concatenation, followed by a linear projection function @oml (-) to map to the original dimension:
Hum! Concat ()) E K.
(16)
Humi = domi (Hemt)
(17)
Decoding, Following, we first initialize the queries Q' with the same size the referential query Q, and add them together to utilize the prior context in Q. Note that, to avoid interference from Q' during the initial stage, initialize Q' as all-zero matrix. Then, concatenate the queries with
the image features to interact with the language features to aggregate the condition information and produce the multi-modal feature map H mmm- This can be represented
O H Hm M MMCC (Q) Q'; H H.mm) H,, H.)
(18)
O, L. + O., H LN
Hmm
(19)
where (-) is the MIP layer, which regulates the significance of the query Q. As the importance approaches zero, the query degenerate into a vanilla query. Then, we feed the queries Og and multi-modal feature map H mm into the MHCA layer to extract target embddings O E RNDD It can be formulated as:
Ô = MHCA(O, C1 Hom H
(20)
O = (d) +
(21)
where (+)+) represents the linear projection function.
Grounding Head. We built the two MLPs (oboxr-) and (.)) over the target embeddings O. The final outputs consist of the predicted center coordinates of the target object, denoted as b = (x, y, h, w) E RR, and the predicted confidence score y E IR² that encompass the target object:
b Obow (laz(O)) (0), y (O)
(22)
4.3 Training Objectives
Similar to DETR, we employ bipartite matching to find the best match between the predictions (b, y} and the ground-truth targets (begt, Vtge]. In our case, the class prediction is confidence prediction aims to estimate the confidence of query containing a target object. To supervise the training, we use the box prediction losses (L1 and GloU), and cross- cross-entropy loss after matching.
Last Vion Cion (byt. b) + A bll + Aoe foe (Ugt. v)
(23)

---

where À denotes the corresponding loss weight. Additionally, to encourage the referential queries in every QA module effectively focus on the target-related context, we also introduce the auxiliary loss Lour that is similar to the above objective function to provide supervision for them. The final training objective can defined as:
Lrinat Ldet + Aauxhaus
(24)
where Anaz denotes the weight the auxiliary loss.
4.4 Extend Dense Grounding
In addition to object-level grounding, our method can easily extend to the dense grounding task by incorporating a segmentation head. Specifically, similar the MaskFormer [3], utilize the MLP to transform the target embeddings O into mask embeddings M E IR N2 The binary mask prediction 8; [0, 1] R #xW is then computed by performing dot product between the mask embeddings M and the multi- modal feature map H Hm and followed by a sigmoid activation. During
training, we use the mask prediction losses (Focal and Dice), which can be defined as follows:
Leeg = Aocat Lo (Sgtt( + Adice (8gt,
(25)
where Stt denotes the ground-truth mask.
4.5 Discussion
In this work, we aim to explore how to further optimize the learning process of queries. To reduce the learning difficulties posed by vanilla query, we introduce a simple query adaption module to adaptively capture target-related context and iteratively refine it. As illustrated in Figure [3] the attention maps produced by each query adaption module consistently align with our objective: to progressively focus on the target-related context and provide prior context for the decoder. It is worth noting that while "multi-level", "adapter", and "selffattention" may be extensively applied other research fields, our approach aims to integrate them to address the challenges in visual grounding tasks, instead of designing a specific module to achieve mentioned functions individually.
Experiment
5.1 Datasets and Evaluation Metric
RecCocoORetoooOO-RRCCCCCC RefCOCO
53 comprises 19,994 images featuring 50,000
referred objects, divided into train, val, testA, and testB sets. Similarly, RefCOCO+ (53) contains 19,992 images with 49,856 referred objects and 141,564 referring expressions. contains more attributes than absolute locations compared to RefCOCO, and has the same split. RefCOCOg [31 has 25.799 images with 49,856 referred objects and expressions. Following a common version of split 32 i.c., train, val, and test sets.
Flickr30K. Flickr30k Entities annotated phrases. We follow ? to split the images into 29,783 for training, 1000 for validation, and 1000 for testing, and report the performance on the test set.
contains 31,783 images and 158k caption sentences with 427k
ReferItGame. ReferltGame 18] includes 20,000 images with 120,072 referring expressions for 19,987 referred objects. We follow 7 to split the dataset into train, validation and test sets, and report the performance on the test set.
Evaluation Metric. For referring expression comprehension (REC),
Prec @0.5 evaluation protocol to evaluate the accuracy, which is consistent with prior works. In this evaluation, a predicted region considered correct if its intersection-overuniit
(loU) with the ground-truth bounding box is greater than 0.5. For referring expression segmentation (RES), we report the Mean IoU (MloU) between the predicted segmentation mask and ground truth mask.
5.2 Implementation Details
Following 7 36), the resolution of the input image is resized 640 x 640. We employ the pre-trained CLIP as backbone to extract both image and language features, and we freeze its parameters
7

---

Table 1: Comparisons with the state-of-the-art approaches on three benchmarks, i.e., RefCOCo, RefCOCO+, RefCOCOg. indicates models that additionally data beyond RefCOCO series. f indicates that models simply combine RefCOco, RefCOCO+, and ReiCOCOg.
RefOOC
ReCCOCO+
RefCOCOg
Methods
Visual Backhone
val
IeSIA
tet
testA
testB
val
test
Single Datasel:
Two-stage:
MAnNef CVPRERSS
ResNet101
76.65
81.14
09.99
69.99
71.62
56.02
66.58
07.27
CM-A-E CaneNu
2
ResNet101
78.35
83.14
71.32
68.09
73.65
58.03
67.99
68.67
ReLNNMS
ResNet101
70.55
PAREC saann
5G
ResNet101
82,20
85.26
79.21
68.25
72.63
78.96
73.92
73.18
One-stage:
FADA kova
18
DarkNet53
72.54
74.53
68.50
56.81
60.23
49.60
61.33
60.36
ReSC-Large
DarkNet53
77.63
80.45
72.30
63.59
68.36
56.81
67.30
20
MCN cvriato
23
DarkNet53
80.08
82.
74.98
67.16
72.86
57.31
66.46
66.01
PLV-FPN
24
ResNett01
81.93
84,99
76.25
71.20
77:40
61.08
70.45
71.08
Transformer-hace
Trans iorvee
ResNet101
81.02
82.72
78.35
64.82
70.70
56,94 94
68.67
67.73
RefTR SEPS300
21
ResNet101
82.23
85.59
76.57
71.58
75.96
62.16
69:41
69.40
TR
St
DarkNet53
81.23
85.59
76.08
68.82
75.37
58.78
71.35
71.58
QRNeT cvrRazss
50
Swin-small
84.01
85.85
82.34
72.94
76.17
63.81
71.89
73.03
LADS
8
ResNet50
82.85
866.6
78.57
71.16
77.64
59.82
71.56
71.66
TransVG++
ViT-Base/16
86.28
75.39
80.45
66.28
76.
Dynamic MDETR
40
VIT. Base/16
85.97
88.82
80. 12
74.83
81.70
63.44
74
74.49
VG-LAW cypaeuss
ViT-Base/16
86.06
80.32
66.69
75.31
PVD MA0E
Swin Rase
84.52
86. 19
76.81
73.89
78.41
64,25
73
71.51
VIT-Base/32
83.97
77.45
73.55
81.00
62.24
76.33
75.53
Ours
VIT Base/16
86.52
90.24
81.42
76.58
83.69
67.38
77.80
77.60
Multple/Extre Dataseis:
VILLA J. Sppaa 11
ResNet101
82,39
87.48
74.84
76.17
811.4
66.84
76.
76.71
ResNet101
78.43
MDETR jevvnn hp
ResNet101
86.75
89.58
81.41
79.52
84.09
70.62
81.64
80.89
ShiKru-7B
ViT-Large
Ferret-7B ABxINII
31
ViT-Large
87.49
91.35
82.45
80.78
87.38
73.14
83.93
84.76
APE fevmass 35
ViT-Large
85.50
89.10
81,30
73.40
80.70
64.40
83.00
78.00
Pink evos2a
46
ViT-Large
88,30
91.70
84.00
81.80
88.
73.90
83.90
84.30
Ours
ViT-Base/16
88.82
92.52
84.87
80.91
86.64
73.35
83. 15
Ours ?
VIT Large
90.91
93.69
86.56
83.33
89.00
75.78
84.97
84.88
Table 2: Comparison with state-of-the- art approaches on the Flickr30K Entities and Referl(Game.
Table 3: Comparisons with the state-of-the-an dense grounding approaches on three benchmarks for RES task, i.c., RefCOCO, RefCOCO+, and RefCOCOg.
Methods
Flickr30K
ReferltGame
Methods
Re(COCO
ReICOCO+
ReICOCOp
test
test
val
lestA
testB
val
IeStA
testi
val
test
RefTK
21
78.66
42
MAnNe:
56.51
02.37
51.70
46.57
52.39
40.08
47.64
48.61
TGa
79.10
70.73
MEN 28
62.44
64.20
59.71
50.62
$4.99
44.69
49.22
29.40
QRNet 50
$1.95
74.61
65.43
56,21
58.32
54,40
54.25
VG++ 8
$1.19
74.70
58.73
Dynamic MDETR
SegT
VG-LAW
40
76.60
PVD
24.33
75.05
77.11
69.52
63.38
60
56.92
63.13
63.62
Ours
82.10
$2.18
Our
14.77
77.92
72.30
66.70
72.28
60.43
65.51
66.26
during training. The model is optimized end-to-end using AdamW for 40 epochs, with a batch size of 32. We set the learning rate le-4 and the weight decay to le-2. The experiments are conducted on V100 GPUs. The loss weight Aiou. AL1. Ace. and Aauz- we set to 3.0. 1.0, 1.0, and 0. For dense grounding, we set the parameters Afoclls and Adice to 5.0. and 1.0.
5.3 Comparisons with State-of-the-ars Methods
REC Task. For REC task, we compare the performance with the state-of-the-ar REC methods, includ- ing the two-stage methods, one-stage methods, and cunsformer-basee methods. As reported in Table and Table our proposed method achieves the best performance.
particular, when comparing to the transformer-based method Dynamic MDETR, which adopts the DETR-like structure and uses the same backbone as ours, we can see that our method performs better with +0.53% +1.90%, +1.62% on RefCOCO, +1.11%, +2.44%, 66.11% RefCOCO+, and +4.94%, +4.18% on RefCOCOg. Additionally, under multiple/extra datasets setting, our method also surpasses recent sfate-of-the- methods that incorporate large language models utilize training data.
RES Task. Following RefTR 21 and VG-LAW experiments and report the results in Table in terms of mloU, It can be seen that our model
we also conduct the dense grounding
8

---

Table 5: Ablation study of the QA position on RefCOCOg.
Table 4: Ablation study of the generation method of learnable queries on RefCOCOg.
Fusion layer (K))
val
RefFormer layer
val
test
None
65.50
65.54
[4]
65.42 (-9.31)
65.12 (-9.02)
[8)
73.31 (-1.42)
72.47 (-1.69)
[4,8,12)
74.08 (+8.58)
73,82 (48.28)
14.6.8.10.12)
76.33 (+10.83)
75.33 (49.79)
[12]
74.73 (-1.59)
74.16 (-1.17)
12.4.6.8.10.122
75.84 (+10.34)
75.32 (49.78)
[4.8]
72.71 (-2.02)
73.03 (-1.13)
[4,12]
75.33 (40.60)
74.51 (+0.35)
[8.12]
75.61 (+0.88)
74.82 (+0.66)
[4,8,12]
76.33 (+1.6))
75.33 (41.17)
achieves superior performance without extra deliberate design to dense grounding, demonstrating the generalization of our method.
5.4
Ablation Studies
In this section, we conduct the ablation studies to verify the effectiveness the each part of our proposed method on RefCOCOg. Following previous work 7 8), the visual backbone we apply the ViT-Base/32.
Effect on the position of QA. As presented in Table firstly, we can observe that removing the QA would lead to Sharp decline in performance, highlighting the effectiveness of QA. We then explored the impact of QA's position in the CLIP to determine where QA should be added. We chose three groups for the ablation study: {4, 8, 12), {4, 6. 8, 10, 12). and {2, 4, 6, 8, 10, 12]. The results indicate that performance is best when we use the {4, 6, 8, 10, 12} configuration. Therefore, we default to this position in our experiments.
Effect the layers of multi-level fusion. In Table analyze the impact of the fusion layers in the decoder. We first conduct experiments using single-level image features, and then proceed with multi- level features. The results show that utilizing multi-level features significantly improves performance, demonstrating that low- and mid-leve features complement high-level features. Additionally, using the (4, 8, 12} achieves the best performance, which adopt for our experiments.
Effect on the different backbone. In the first line of Table o we apply our method to single- modal encoders, i.e., Swin Transformer Bert. The results demonstrate that method is not only applicable to multi-modal encoders but is also compatible with single-modal encoders.
Effect on the auxiliary loss. In the second line of Table 6 We experiment with auxiliary loss, and the results demonstrate the effectiveness of auxiliary loss. By employing auxiliary loss, the reference query can capture the target-related visual contexts more effectively.
Effect on learnable queries. In the third line of Table we validate the effectiveness of the learnable queries. Specifically, we replace the prior queries generated by the QA module with randomly initialized queries or linguistic embeddings input to decoder while keeping other modules unchanged. We can observe that introducing the prior queries can bring significant performance improvement. This result demonstrates that prior queries aid the decoder in more accurately locating the target object. Additionally, we investigate the accuracy of our referential queries, which are designed to provide prior information to the decoder. Since the channel dimension the QA module is lower, the reference query may not accurately predict the coordinates of the targets.
Convergence curves. In Figure 4 we illustrate the convergence curve of our proposed method com- pared to other open-source DETR-like visual grounding methods. Notably, our method demonstrates accelerated training convergence, reducing the training time by half compared the TransVG, while also outperforming other existing methods.
5.5 Qualitative Results
As shown Fi the attention maps the QA module illustrate the refined process of how the referential query captures the target-related context. Initially, the attention map appears noisy but gradually focuses on the target-related context, such as the couch (a). By incorporating the referential query, the attention map in the decoder accurately concentrates the target object.

---

100
TransVg
Table Ablation studies of backbone, auxiliary
80
RefTR
Ours
loss, and learnable queries on RefCOCOg.
Preci@0...
Method
val
lest
60
Backbone:
Method
Parmss
Swin+Bert
75.25 (+0.64)
75.61 (+0.29)
Timmsgg
144.00
Auxiliary loss:
RefTR
144.73
Wlo Lanr
74.24 (-1.60)
73.82 (-1.50)
20
Learnable queries:
Referential query
52.92 (-22.92)
51.87 (-23.45)
40
60
80
Linguistic embeddings
71.36 (-4.48)
71.07 (-4.25)
Epoch
Random initialization
73.40 (-2.44)
73.12 (-2.21)
Figure 4:
Convergence curves. Our method
Ours
75.84
75.32
achieves better results with fewer training epochs on RefCOCOg.
Image
QA
Decoder
(a) Expression: the couch sitting in directly in front the window
(b) Expression: table in front of kid
(C) Expression: woman skiing with her child
Figure 5: Qualitative results on RefCOCOg. The bounding boxes in green and red correspond to predictions of our model and the ground truth. Columns 2-6 showcase the attention maps generated by each QA module, while the last column represents the attention map from the decoder.
Besides, it important to note that the referential query may not precisely focus on the target object due to the lower feature dimension in the QA module, but it still captures target-related information.
6 Concluding and Remarks
In this paper, we propose a novel approach, called RefFormer that can be seamlessly integrated into CLIP. The RefFormer can not only generate the referential query provide the target-related context for decoder, but also act as the adaptor to preserve the original knowledge of CLIP and reduce the training cost. Extensive experiments demonstrate the effectiveness of method, and visualization results illustrate the refined process of our proposed RefFormer.
Limitations Although our method is specifically designed for the REC task and surpasses existing SOTA methods in REC, there is still significant room for improvement in the RES task. This is because have not yet optimized our approach specifically for the RES task.
7 Acknowledgments
This work was supported in part by National Science and Technology Major Project under Grant 2023ZD0121 3000000,, National Natural Science Foundation of China under Grants 62088102 12326608 and 12106192, Natural Science Foundation of Shaanxi Province under Grant 2022JC-41, and Funda- mental Research Funds for the Central Universities under Grant XTR042021005.
10

---

References
[1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 219-229. Springer, 2020.
[2] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint 23Xiv:2306.15199 2023.
[3] Long Chen. Wenbo Ma, Jun Xiao, Hanwang Zhang, and Shih-Fu Chang. Ref-nms: Breaking proposal bottlenecks in two-stage referring expression grounding, Proceedings of the AAAl conference on artificial intelligence, volume 35, pages 1036-1044, 2021.
[4] Xinpeng Chen, Lin Ma, Jingyuan Chen, Zequn Jie, Wei Liu, and Jiebo Luo. Real-time referring expression comprehension by single-stage grounding network. arXiv preprint 1812.38:8, 2018.
[5] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34:17 17864- 17875, 2021.
[6] Zesen Cheng, Kehan Li, Peng Jin, Siheng Li, Xiangyang Ji, Li Yuan, Chang Liu, and Jie Chen. Parallel vertex diffusion for unified visual grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38. pages 1326-1334, 2024.
[7] Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1779-1779, 2021.
[8] Jiajun Deng, Zhengyuan Yang. Daqing Liu, Tianlang Chen, Wengang Zhou, Yanyong Zhang, Houqiang Li, and Wanli Ouyang. Transvg++: End-to-end visual grounding with language conditioned vision transformer. IEEE transactions on pattern analysis and machine intelligence, 2023.
[9] Jianfeng Dong, Yabing Wang. Xianke Chen, Xiaoye Qu, Xirong Li, Yuan He, and Xun Wang. Reading-strategy inspired visual representation learning for text-to-video retrieval. IEEE transactions circuits and systems for video technology. 32(8):5680-566 2022.
[10] Ye Du, Zehua Fu, Qingjie Liu, and Yunhong Wang. Visual grounding with transformers. In 2022 IEEE International Conference on Multimedia and Expo (ICME), Jul 2022.
[11] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and and-language representation learning. Advances in Neural Information Processing Systems, 33:66 6-66668, 2020.
[12] Richang Hong, Daqing Liu, Xiaoyu Mo. Xiangnan He, and Hanwang Zhang. Learning to compose and reason with language tree structures for visual grounding. IEEE transactions on pattern analysis and machine intelligence, 44(2):684-696 2019.
[13] Jinhyun Jang, Jungin Park, Jin Kim, Hyeongjun Kwon, and Kwanghoon Sohn. Knowing where to focus: Event-aware transformer for video grounding. In Proceedings of the IEÉE/CVF International Conference Computer Vision, pages 13846-13856, 2023.
[14] Jiay Ji, Yiwei Ma, Xiaoshuai Sun, Yiyi Zhou, Yongjian Wu, and Rongrong Ji. Knowing what to learn: metric-oriented focal mechanism for image captioning. IEEE Transactions on Image Processing, 31:4321-4435, 2022.
[15] Dongsheng Jiang, Yuchen Liu, Songlin Liu, Xiaopeng Zhang, Jin Li, Hongkai Xiong, and Qi Tian. From clip to dino: Visual encoders shout in multi-modal large language models. 2023.
[16] Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, and Tieniu Tan. Locate then segment: A strong pipeline for referring image segmentation. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9858-9861 2021.

---

[17] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulate detection for end-to-end modii understanding, In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[18] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgam:: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[19] Feng Li, Ailing Zeng, Shilong Liu, Hao Zhang, Hongyang Li, Lei Zhang, and Lionel M Ni Lite detr: An interleaved multi-scale encoder for efficient detr. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18558-18560 2023.
[20] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13619-33627, 2022.
[21] Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. Advances in neural information processing systems, 34:1 19542-1666, 2021.
[22] Pandeng Li, Chen- Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random to real. Advances in neural information processing systems, 36, 2024.
[23] Yue Liao, Aixi Zhang, Zhiyuan Chen, Tianrui Hui, and Si Liu. Progressive language-customined visual feature learning for one-stage visual grounding. IEEE Transactions on Image Processing, 31 31:4266 4277, 2022.
[24] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201. 12329, 2022.
[25] Sun-Ao Liu, Yiheng Zhang, Zhaofan Qiu, Hongtao Xie, Yongdong Zhang, and Ting Yao. Caris: Context-aware referring image segmentation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 779-788, 2023.
[26] Wenze Liu, Hao Lu, Yuliang Liu, and Zhiguo Cao. Box-detr: Understanding and boxing conditional spatial queries. arXiv preprint arXiv:2307.08335, 2023.
[27] Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. Improving referring ex- pression grounding with modal attention guided erasing. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1950-1959, 2019.
attention-guided
[28] Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 10034-10043, 2020.
[29] Yiwei Ma, Jiayi Ji. Xiaoshuai Sun, Yiyi Zhou, and Rongrong Ji. Towards local visual modeling for image captioning. Pattern Recognition, 138: 109420, 2023.
[30] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-elip: End-to- end multi-grained contrastive learning for video-text retrieval. In Proceedings of the 30th ACM International Conference on Mullimedia, pages 638-647, 2022.
[31] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11 11-20, 2016.
[32] Varun Nagaraja, Vlad Morariu, and Larry Davis. Modeling context between objects for referring expression understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-1, 2016, Proceedings, Part 14, pages 792-807. Springer, 2016.
12

---

[33] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebmik. Flickr30k entities: Collecting eegion-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-8766 PMLR, 2021.
[35] Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all once for universal visual perception. 2024.
[36] Fengyuan Shi, Ruopeng Gao. Weilin Huang, and Limin Wang. Dynamic mdetr: A dynamic multimodal transformer decoder for visual grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[37] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr+: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[38] Mainak Singha, Ankit Jha, Bhupendra Solanki, Shirsha Bose, and Biplab Banerjee. Applenet: Visual attention parameterized prompt learning for few-shot remote sensing image generalization using clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[39] Wei Su, Peihan Miao, Huanzhang Dou, Yongjian Fu, and Xi Li. Referring expression compre- hension using language adaptive inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2357-2365, 2023.
[40] Wei Su, Peihan Miao, Huanzhang Dou, Gaoang Wang, Liang Qiao, Zheyang Li, and Xi Li. Language adaptive weight generation for multi-task visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10857-10866, 2023.
[41] Yabing Wang. Jianfeng Dong, Tianxiang Liang, Minsong Zhang, Rui Cai, and Xun Wang. Cross-lingual cross-modal retrieval with noise-robust learning. In Proceedings of the 30th ACM International Conference on Mullimedia, pages 422-433, 2022.
[42] Yabing Wang, Fan Wang, Jianfeng Dong, and Hao Luo. C12cm: Improving cross-lingual cross- modal retrieval via rossslinguual knowledge transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5651-5659, 2024.
[43] Yabing Wang, Le Wang, Qiang Zhou, Zhibin Wang, Hao Li, Gang Hua, and Wei Tang. Mul- timodal Ilm enhanced cross-lingus cross-moda retrieval. arXiv preprint AKVV:2409.1966 2024.
[44] Yabing Wang, Shuhui Wang, Hao Luo, Jianfeng Dong, Fan Wang, Meng Han, Xun Wang, and Meng Wang. Dual-view curricular optimal transport for cross lins-lingual cross-modal retrieval. IEEE Transactions on Image Processing, 33:1522-1533, 2024.
[45] Yingming Wang, Xiangyu Zhang, Tong Yang. and Jian Sun. Anchor detr: Query design for transformer- based detector. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2567-2575, 2022.
[46] Shiyu Xuan, Qingpei Guo, Ming Yang, and Shiliang Zhang. Pink: Unveiling the power of referential comprehension for multi-modal 1lms. 2024.
[47] Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual grounding by recursive sub-query construction. In Computer Vision-ECCV 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 387-404. Springer, 2020.
13

---

[48] Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach visual grounding In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4683-4693, 2019.
[49] Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: improving end-to-end object detector with dense prior. arXiv preprint ar..v:2104.0033300 2021.
[50] Jiabo Ye, Junfeng Tian, Ming Yan, Xiaoshan Yang, Xuwu Wang. Ji Zhang, Liang He, and Xin Lin. Shifting more attention to visual backbone: Query-moduuated refinement networks for end-to-end visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15502-15512, 2022.
[51] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07774, 2023.
[52] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara I. Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1307-1315, 2018.
[53] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part Il 14, pages 69-85. Springer, 2016.
[54] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.0366 03660, 2022.
[55] Chuyang Zhao, Yifan Sun, Wenhao Wang, Qiang Chen, Errui Ding, Yi Yang, and Jingdong Wang. Ms-detr: Efficient detr training with mixed supervision. arXiv preprint arXiv:2401.0320 2024.
[56] Peizhi Zhao, Shiyi Zheng. Wenye Zhao, Dongsheng Xu, Pijian Li, Yi Cai, and Qingbao Huang. Rethinking two-stage referring expression comprehension: A novel grounding and segmentation method modulated by point. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7487-7495, 2024.
[57] Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. In European Conference on Computer Vision, pages 598-615. Springer, 2022.
[58] Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and Anton Van Den Hengel. Parallel attention: A unified framework for visual object discovery through dialogs and queries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 0252-4261. 2018.
14

---

A
Appendix
Table 7: Ablation study of the direction of the features flow from the QA module on RefCOCOg.
RefFormer direction
val
test
None
65.50
65.54
Only text
70.00 (+4.50)
69.24 (+3.70)
Only image
72.57 (+7.07)
72.56 (+7.02)
Image & Text
76.33 (+10.83)
75.33 (49.79)
170
76.0
785
75.3
355
25.8-
150
74.5
135
73.5
(a) val
(b) test
Figure 6: The performance of different numbers of learnable queries on RefCOCOg.
A.1 Effeet on the RefFormer's direction.
In RefFormer, the QA module can serve an adapter, injecting specific knowledge into the frozen CLIP model. In Table we investigate the direction of feature flow from QA module. We find that using dual-direction approach achieves the best performance. Through QA module, language features have aggregated relevant visual context information. As pointed to (25), incorporating rich visual context into linguistic features aids in achieving strong vision-language alignment and better indicating target objects.
A.2 Effeet on the number of learnable queries.
We depict the performance terms of @.5 according to the number of learnable queries Ng in Figure 6 When we adopt the N,, 3, the performance is best. However, further increases yield large number of N, increases the difficulty of the model.
only slighT improvements in metries, Therefore, we default set the N, = 3 in our experiments.
A.3 Visualization
Due space limitations, present additional visualization results here. As shown in Figure 7 the referential queries gradually focus on the target object and effectively provide target-related context for the decoder. These results demonstrate the effectiveness of our proposed methods.
15

---

Image
QA
Decoder
(a) Expression: man with an oranget with pizza
(b) Expression standing giraffe in the background
(c) Expression: a chair which a boy sitting
(d) Expression:a sandwiich with colby jack cheese, tomato, and lettuce, on fresh cut bread
(e) Expression: a man wearing nike shoes in bright neon green top is playing tennis
Expression: red taxi standing next to yellow sign
(g) Expression: young kid closest to projector
Figure 7: Qualitative results on RefCOCOg. The bounding boxes in green and red correspond to outputs from model and the ground truth. Columns 2-6 showcase the attention maps generated by each QA module, while the last column represents the attention map from the decoder.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction (Section I accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
16

---

The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations of the work in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
17

---

Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully describe the proposed model and implementation details in Section and Section 5.2 and we submit our main code in the form of zipped file in additional supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the dataset).
the model (e.g., with an open-source dataset or instructions for how to construct
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which case
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We submit our main code in the form of zipped file in additional supplemen- tary materials, and we will release the complete code after review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cd
for more details.
18

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips
guides CodeSubmissiiiii swwpiiiissooooooicco.co for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the datasets, metrics and implementation details in Sec. ?? and supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the eтror bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
19

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com-- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: We only provide the GPU type in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
20

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA].
Justification:
This paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswwthcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
This paper does not release new assets.
21

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "hKg3JAzqJk", "reviewer": "Reviewer_RwV9", "summary": "This paper proposes a novel visual grounding framework, called RefFormer, aims to improve the learning process of learnable queries. Specifically, it introduces a  query adaption module (QA) that can be seamlessly integrated into different layers of CLIP, which can not only provide prior information to the decoder, but also act as an adapter to learn task-specific knowledge. The effectiveness of the proposed method is extensively validated through experiments conducted on the five popular visual grounding benchmarks, namely RefCOCO, RefCOCO+, RefCOCOg, Flickr30K, and ReferItGame. Furthermore, the authors extend this approach to dense grounding tasks, demonstrating its effectiveness and generalization.", "questions": "1. Why is a direct use of the referential query not preferred, and why is an additional learnable query introduced in the decoder?\n2. 6. Provide experimental details regarding the channel dimension of the QA module in the down-projection process.", "limitations": "See the above weaknesses.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The proposed method is novel, which integrates query learning and adapter into one module, effectively leveraging various levels of feature information within CLIP.\n2. By introducing QA module, models can adaptively extract target-related information from the backbone and provide prior information to the decoder instead of relying on randomly initialized queries.\n3. This paper achieves better performance with fewer training parameters and data.\n4. Sufficient experiments have been conducted to show the promising results of the proposed method.\n5. The writing of this paper is good and clear.", "weaknesses": "While the paper is clearly written, some areas can still be improved. Some suggestions are as follows：\n1. Line 72 should correct \"RefFormer\" to \"QA\".\n2. A more detailed description is needed for the text side in the QA module, such as how the interaction with visual features is initiated.\n3. Why is a direct use of the referential query not preferred, and why is an additional learnable query introduced in the decoder?\n4. Specify the dimension along which the concatenation operation in Section 4.1 is performed.\n5. Correct \"[r_t, F^i_t]\" to \"[r_t; F^i_t]\" in Equation 9.\n6. Provide experimental details regarding the channel dimension of the QA module in the down-projection process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Line 72 should correct \"RefFormer\" to \"QA\".\n\n Thank you.  We will correct the typos in our paper.\n\n> Q2: A more detailed description is needed for the text side in the QA module, such as how the interaction with visual features is initiated.\n\n In the CAMF block, we take the language features as the query, and interact them with the image features using cross-attention. By doing so,  we can incorporate rich visual context into language features to better indicate the target object.  Subsequently, in the TR block, we employ self-attention to enhance the context-aware language features produced above to refine the expression condition.\n\nThank you, we will add this description in Sec 4.1.\n\n> Q3: Why is a direct use of the referential query not preferred, and why is an additional learnable query introduced in the decoder?\n\nAs outlined in line 189, we feed the referential query into $\\phi_q(\\cdot)$ to adjust its significance. When the referential query is inaccurate (i.e., its significance approaches zero), the query in the decoder degenerates to the vanilla query.\n\nThank you, we will add this explanation in line 189.\n\n> Q4: Specify the dimension along which the concatenation operation in Section 4.1 is performed.\n\nThank you for your suggestion. We will further improve the description details in our paper. In Eq.7,9,11,12, the concatenation is along with the patch dimension. \n\n> Q5: Correct \"[r_t, F^i_t]\" to \"[r_t; F^i_t]\" in Equation 9.\n\n Thank you.  We will correct the typos in our paper.\n\n> Q6: Provide experimental details regarding the channel dimension of the QA module in the down-projection process.\n\n Thank you for your suggestion. In our experiments, we set the channel dimension of the down-projection to 128. Furthermore, we provide an ablation study on the channel dimension of the down-projection below. Increasing the dimension improves performance, but to strike a balance between performance and computational cost, we set the channel dimension to 128.\n\n| Dimension | val   | test  |\n|-----------|-------|-------|\n| 64        | 74.15 | 73.88 |\n| 128       | 76.33 | 75.33 |\n| 256       | 76.62 | 75.39 |\n|           |       |       |\n\nThank you, we will add this experiment to the ablation studies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WrMniQ5R0g", "reviewer": "Reviewer_cZhf", "summary": "The existing one-stage visual grounding methods suffer from cross-modal learning difficulty and focus simply on the deepest visual features. This paper designs a query adaption (QA) module to provide target-related referential queries for the decoder. The proposed architecture Reformer is based on a CLIP model with multiple QA modules inserted in different layers of CLIP. The proposed method can be applied to both REC and RES tasks. The performance is validated on various benchmarks.", "questions": "- How does the paper select the number of inserted QA modules? How does the paper select the indices of inserted layers? Why do more layers or lower layers hurt the performance? \n- In the condition aggregation and multi-modal fusion (CAMF) module, the query seems to interact more with the visual representations (i.e., the upper part in Figure 3). Why is it designed in such a way? What is the advantage over mainly based on textual representations?\n- Does QA support different model architectures?\n- What is the version of the reformer in Tables 2 and 3?", "limitations": "While the motivation is clear and the idea is intuitive, the technical implementation of essential modules/structures lacks theoretical or empirical justification.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "-\tThe paper is generally well-written and easy to follow.\n-\tThe motivation for addressing two issues present in the existing one-stage grounding model is clear, and the idea of enhancing cross-modal interaction is intuitive.\n-\tThe attention maps show the refining process across different layers as expected.\n-\tIt works for both object-level and dense grounding.", "weaknesses": "- The proposed module QA has to be inserted into specific positions inside the VLM to improve grounding performance. Experiments in Table 5 indicate the importance of layer selection. However,  it is unclear why the 5 layers perform the best and why these indices are selected. As mentioned in the introduction, the low and mid-level features are crucial for grounding (line 48), yet lower layers (e.g., 2nd) containing rich low-level features diminish the performance. In other words, the selection of layers is not explained theoretically, while 3 combinations are not enough to justify the selection empirically. \n- In condition aggregation and multi-modal fusion (CAMF) module, it seems that the query interacts more with the visual representations (i.e., the upper part in Figure 3), while the alternative that mainly integrates query with textual representations seems feasible (i.e., place the query to the lower part in Figure 3). The motivation and advantage of design are missing.\n- The backbone and training data information (especially the version of Reformer) is not provided in Tables 2 and 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: How does the paper select the number of inserted QA modules? How does the paper select the indices of inserted layers? Why do more layers or lower layers hurt the performance?\n\n\nWe provide more experiments on layer selection on RefCOCOg below. We categorize [1-4], [5-8], and [9-12] as low-level, mid-level, and high-level layers, respectively. As shown in the table below, we observe that introducing more layers can improve performance. However, with continued layer addition, the performance gains become less significant. Therefore, **to strike a balance between performance and computational cost, we opt for [4,6,8,10,12]**. Compared to IDs 7, 9, and 10, line 8 leads to a slight performance decline. This could be attributed to the fact that shallower layers focus on the local details and convey less semantic information, which **may introduce noise**. Similarly, in object detection, RPN-based models [a,b] do not use C1 feature maps for the same reason. Additionally, we explore other combinations of layers, but the performance changes show low sensitivity.\n| ID | Layer          | val   | test  |\n|------|----------------|-------|-------|\n| 1    | None           | 65.50 | 65.54 |\n| 2    | 3,7,11         | 73.94 | 72.94 |\n| 3    | 4,8,12         | 74.08 | 73.82 |\n| 4    | 3,5,7,11       | 74.43 | 74.18 |\n| 5    | 4,6,8,12       | 74.82 | 74.20 |\n| 6    | 3,5,7,9,11     | 75.06 | 74.94 |\n| 7    | 4,6,8,10,12    | 76.33 | 75.33 |\n| 8    | 2,4,6,8,10,12  | 75.84 | 75.32 |\n| 9    | 4,6,8,9,10,12  | 76.40 | 75.31 |\n| 10   | 4,6,8,10,11,12 | 76.51 | 75.43 |\n|      |                |       |       |\n\nThank you, we will improve Table 5 and the corresponding experimental analysis.\n\n[a] Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(6): 1137-1149.\n\n[b] He K, Gkioxari G, Dollár P, et al. Mask r-cnn[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2961-2969.\n\n> Q2: In condition aggregation and multi-modal fusion (CAMF) module, it seems that the query interacts more with the visual representations (i.e., the upper part in Figure 3), while the alternative that mainly integrates query with textual representations seems feasible (i.e., place the query to the lower part in Figure 3). The motivation and advantage of design are missing.\n\nWe would like to clarify that **reference queries aim to capture the target-related visual context.** Initially, they interact with expressions to aggregate text conditions (in CAMF). Subsequently,  they interact with visual features based on text conditions to capture and refine visual contexts about the target object (in TR). If we place them below, we will not be able to capture the target-related visual features based on the conditional information. Additionally, we provide the performance comparison on RefCOCOg as below. The results further demonstrate the effectiveness of the method we designed.\n| Method          | val   | test  |\n|-----------------|-------|-------|\n| Query on the text side| 74.21 | 73.52 |\n| Ours            | 76.33 | 75.33 |\n|                 |       |       |\n\nThank you, we will add this experiment to the ablation studies.\n\n> Q3: The backbone and training data information (especially the version of Reformer) is not provided in Tables 2 and 3.\n\nFor a fair comparison, we utilize the ViT-Base 32 and train on a single dataset (Flickr30K Entities and ReferItGame, respectively) to compare with other methods. Consistent with prior approaches, we perform ablation studies (Table 3-6) using ViT-Base 32 and train on a single dataset (RefCOCOg). \n\nThank you, we will further improve the experimental details in our paper.\n\n> Q4: Does QA support different model architectures?\n\nWe apply our method to single-modal encoders, i.e., Swin-base + Bert, and conduct experiments on RefCOCOg. The results, as presented below, demonstrate that our method is also compatible with single-modal encoders.\n\n| Method          | val   | test  |\n|-----------------|-------|-------|\n| Swin-base + BERT| 75.25 | 75.61 |\n| CLIP-base           | 76.33 | 75.33 |\n|                 |       |       |\n\nThank you, we will add this experiment to the experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BWMLg0tKj8", "reviewer": "Reviewer_DDuF", "summary": "This paper addresses the generation of queries for the decoder. The authors propose RefFormer to generate the referential query with the prior context. A query adaption module is proposed to capture extensive target-related context and provide valuable referential knowledge for the decoder. Extensive experiments validate the effectiveness of the proposed method.", "questions": "1. Could you provide the number of trainable parameters and the inference speed?", "limitations": "Yes. Limitations are addressed.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a new method for query initialization, where multimodal information is continuously embedded during the feature extraction stage to provide the query with prior knowledge.\n2. Extensive experiments validate the proposed method.", "weaknesses": "This idea seems straightforward but lacks some innovation. The method mentioned in the paper has been applied in other fields, such as R2-tuning. The proposed module is also simple to implement. Although the model achieves good results, I believe the paper does not meet the standards of NIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: This idea seems straightforward but lacks some innovation. The method mentioned in the paper has been applied in other fields, such as R2-tuning. \n\nThank you for your question. We analyze the differences between our approach and R2-tuning [a] from two aspects: motivation and implementation:\n\n**[Motivation]:** [a] designs an **image-to-video transfer framework** that applies CLIP to video temporal grounding tasks without fine-tuning. In contrast, our work focuses on **improving learnable queries in DETR-like structures** for visual grounding tasks. Specifically, [a] designs a parameter-tuning strategy to model spatial-temporal information from coarse to fine using CLIP, which can be served as an adapter. Instead, our work aims to generate prior information for the decoder to improve the learnable query by integrating the QA module into the CLIP layers. \n\n**[Implementation]:** In our work, we focus on improving learnable queries rather than merely tuning gradient flows. The comparison is shown as follows:\n\n1. *The feature interaction in CLIP layers.*  It primarily encompasses three key differences: **1) the generated features (frame-level spatial-temporal features vs. referential queries), 2) vision-language interaction (unidirectional vs. bidirectional), and 3) gradient flow tunning (visual side vs. visual and language sides)**. Specifically as follows:\n\n    + [a] introduces the $R^2$ blocks to interact the patch-level representations with language representations for adaptively pool spatial features, and then combine the pooled features with the [CLS] token to extract the frame-level spatial-temporal features. In our work, we incorporate the learnable referential queries to interact with the language and visual features for capturing target-related visual contexts. \n\n   + The interaction in [a] is unidirectional, focusing solely on extracting query (language)-modulated spatial features. Conversely, the interaction in our work is bidirectional, not only highlighting language-related context within visual features but also integrating context-aware visual features into language features. \n\n   + The gradient flows tuning in [a] that forward in CLIP are solely within the vision encoder and are specific to the [CLS] token, while in our method, they are adjusted in both the vision and language encoders and apply to patch features.\n\n2. *Decoding.*  **[a] apply the traditional convolution layers** to spatial-temporal features to predict the temporal boundary, while **our work adopts the DETR-like structure** and employs **reference queries** to provide the prior information for the decoder.\n\n3.  *Multi-level features.* We introduce language-guided multi-level fusion to **aggregate the image features from the different layers** rather than applying **1D convolutions to generate the temporal feature pyramid in [a]**  for subsequent decoding.\n\nTo further validate the effectiveness of our approach, we integrate the method from [a] into our framework. The results, presented below, demonstrate that our method achieves superior performance.\n| Method                                           | val   | test  |\n|--------------------------------------------------|-------|-------|\n| Decoding with convolution layers                 | 70.02 | 69.87 |\n| Tunning on the visual side and using [CLS] token | 66.97 | 66.12 |\n| Unidirectional interaction                      | 72.45 | 72.02 |\n| Ours                                             | 76.33 | 75.33 |\n|     \n\n**Additionally, we would like to highlight that [a] is a concurrent work that has been accepted by ECCV.** We will reference it and discuss it in the related work.\n\n[a] Liu Y, He J, Li W, et al. $ R^ 2$-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding[J]. arXiv preprint arXiv:2404.00801, 2024.\n\n\n>Q2: Could you provide the number of trainable parameters and the inference speed?\n\nWe provide the number of trainable parameters and the inference speed below and compare them with other open-source methods. When compared to Transvg and RefTR, which also use the DETR-like structure, our approach demonstrates superior performance in both trainable parameters and inference speed.\n| Method  | Trainable parameters(M) | Inference speed(ms) |\n|---------|----------------------|-----------------|\n| ReSC [39]    | 174.85               | 53              |\n| Transvg [7] | 152.61               | 62              |\n| RefTR [19]   | 144.73               | 44              |\n| Ours    | 29.20                | 32              |\n|         |                      |                 |\n\nThank you, we will add this experiment to the experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ga1dRJy79f", "reviewer": "Reviewer_VGeh", "summary": "This paper focuses on the visual grounding task, which proposes a query adaption module that can be seamlessly integrated into CLIP.  By strategically inserting this module into different layers of CLIP, the learnable query can adaptively learn target-related information from multi-level image feature maps, and iteratively refine the acquired information layer by layer.  Additionally, it can offer prior information to the decoder, effectively mitigating the learning difficulty of the decoder, and accurately concentrating on the target object.  Extensive experiments on five visual grounding benchmarks (RefCOCO/+/g, ReferItgame, Flickr30k) validate the effectiveness of the proposed method.", "questions": "See weakness, please.  If the author can solve my above confusion well, I will consider raising the score.", "limitations": "In the limitations section, the author states that this method still has room for improvement in RES tasks. In addition, by visualizing the attention map, I find that this method accurately locates the area of the target object. Therefore, I believe this method holds great potential in RES tasks, and I hope the author can make further advancements in future work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.This paper introduces a query adaption module that can adaptively learn target-related information from different layers in CLIP, and provides the visualization results of the attention map in experiments. This design attracted me and made me feel interested.\n\n2.The proposed QA module can not only capture target information adaptively, but also act as an adapter to avoid adjusting Backbone's parameters. Compared with the previous fine-tuning Backbone's work, this module has advantages and achieves better performance with smaller training parameters.\n\n3.In this task, unlike most prior DETR-like research that primarily concentrates on decoder design, this paper focuses on learnable query optimization , making it innovative in its approach.\n\n4.The authors conduct experiments on multiple visual grounding benchmarks (RefCOCO/RefCOCO+/RefCOCOg, Flickr30K, and ReferItGame) and also experiment on the RES benchmark, achieving improved performance. This provides robust empirical support for the effectiveness of the method.", "weaknesses": "1.It is better to provide the experiment results without using auxiliary loss, which can further observe the influence of auxiliary loss on the referential query.\n\n2.Is it feasible to directly utilize the referential query without subsequent decoding operations? I am interested in the accuracy of the referential query.\n\n3.Typo: On line 72, \"we propose a query adaption module, RefFormer...\". It appears that there might be an error in this sentence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:It is better to provide the experiment results without using auxiliary loss, which can further observe the influence of auxiliary loss on the referential query.\n\nWe conduct the ablation study with auxiliary loss on RefCOCOg, and the results demonstrate the effectiveness of auxiliary loss. By employing auxiliary loss, the reference query can capture the target-related visual contexts more effectively.\n| Method             | val   | test  |\n|--------------------|-------|-------|\n| w/o auxiliary loss | 74.24 | 73.82 |\n| Ours               | 76.33 | 75.33 |\n|                    |       |       |\n\nThank you, we will add this experiment to the ablation studies.\n\n> Q2: Is it feasible to directly utilize the referential query without subsequent decoding operations? I am interested in the accuracy of the referential query.\n\n The referential query is designed to provide prior information to the decoder. Since the channel dimension in the QA module is lower, the reference query may not accurately predict the coordinates of the targets. We conduct the experiment on RefCOCOg, and the results are illustrated below.\n\n| val   | test  |\n|-------|-------|\n| 52.92 | 51.87 |\n|       |       |\n\nThank you, we will add this explanation to the method.\n\n> Q3: Typo: On line 72, \"we propose a query adaption module, RefFormer...\". It appears that there might be an error in this sentence.\n\nThank you. We will rectify \"RefFormer\" to \"QA\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received mixed feedback, with one reviewer recommending rejection while three others favored acceptance. The authors proposed a novel approach called RefFormer, introducing a query adaptation module designed to learn target-related context more effectively. Although Reviewer DDuF pointed out that the idea appears straightforward and lacks significant innovation, noting similarities to existing methods. During rebuttal, the authors provided detailed comparison between their paper and the mentioned ECCV 24 work, highlighting differences in motivation and implementation. Additionally, they integrated the mentioned method into their framework for experimental analysis. The AC believed the rebuttal is persuasive and clearly demonstrate the novelty and innovation of the proposed method. Therefore, the AC has decided to accept this paper.
Decision: Accept (poster)