Title: Referencing Where to Focus: Improving Visual Grounding with Referential Query
Venue: NeurIPS 2024
Accepted: yes
Keywords: Visual Grounding
Abstract: Visual Grounding aims to localize the referring object in an image given a natural language expression. Recent advancements in DETR-based visual grounding methods have attracted considerable attention, as they directly predict the coordinates of the target object without relying on additional efforts, such as pre-generated proposal candidates or pre-defined anchor boxes. However, existing research primarily focuses on designing stronger multi-modal decoder, which typically generates learnable queries by random initialization or by using linguistic embeddings.  This vanilla query generation approach inevitably increases the learning difficulty for the model, as it does not involve any target-related information at the beginning of decoding. Furthermore, they only use the deepest image feature during the query learning process, overlooking the importance of features from other levels. To address these issues, we propose a novel approach, called RefFormer. It consists of the query adaption module that can be seamlessly integrated into CLIP and generate the referential query to provide the prior context for decoder, along with a task-specific decoder. By incorporating the referential query into the decoder, we can effectively mitigate the learning difficulty of the decoder, and accurately concentrate on the target object. Additionally, our proposed query adaption module can also act as an adapter, preserving the rich knowledge within CLIP without the need to tune the parameters of the backbone network. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method, outperforming state-of-the-art approaches on five visual grounding benchmarks.

Referencing Where to Focus: Improving Visual Grounding with Referential Query

Abstract

Visual Grounding aims to localize the referring object in an image given a natural language expression. Recent advancements in DETR-based visual grounding methods have attracted considerable attention, as they directly predict the coordinates of the target object without relying on additional efforts, such as pre-generated proposal candidates or pre-defined anchor boxes. However, existing research primarily focuses on designing stronger multi-modal decoder, which typically generates learnable queries by random initialization or by using linguistic embeddings. This vanilla query generation approach inevitably increases the learning difficulty for the model, as it does not involve any target-related information at the beginning of decoding. Furthermore, they only use the deepest image feature during the query learning process, overlooking the importance of features from other levels. To address these issues, we propose a novel approach, called RefFormer. It consists of the query adaption module that can be seamlessly integrated into CLIP and generate the referential query to provide the prior context for decoder, along with a task-specific decoder. By incorporating the referential query into the decoder, we can effectively mitigate the learning difficulty of the decoder, and accurately concentrate on the target object. Additionally, our proposed query adaption module can also act as an adapter, preserving the rich knowledge within CLIP without the need to tune the parameters of the backbone network. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method, outperforming state-of-the-art approaches on five visual grounding benchmarks.

1 Introduction

Linguistic

Backbone

Visual

Backbone

Task

-

specific

Decoder

Bounding

Box

(

x

1

,y

1

,x

2

,y

2)

(

a

)

Existing

Method

Linguistic

Backbone

Visual

Backbone

Task

-

specific

Decoder

Learnable

Queries

Query

Adaption

Learnable

Queries

Amanis

shooting

kites

Bounding

Box

(

x

1

,y

1

,x

2

,y

2)

&

Update

(

b

)

Our

Proposed

Method

(

c

)

Attention

Amanis

shooting

kites

Existing works in this field typically follow the object detection framework and incorporate multimodal fusion to tackle this task. Earlier studies [52, 12, 27, 3, 58] mainly focus on a two-stage pipeline, which first generates a set of region proposals using object detectors, and then finds the bestmatched region by interacting these regions with linguistic expressions. However, the performance of this method is limited by the quality of the generated region candidates. To address this issue, some studies [48, 47, 4] adopt the one-stage pipeline, which removes the proposal generation stage. Unfortunately, these methods make dense predictions with a sliding window over pre-defined anchor boxes, resulting in sub-optimal performance due to the failure to capture object relations effectively. Recently, some methods [17, 7, 10, 21, 8, 36] inspired by the DETR [1] structure, which adopt a standard multi-modal transformer framework to establish the multi-modal correspondence (as shown in Figure 1 (a)). These methods predict bounding boxes of target objects directly from learnable queries, eliminating the need for extra efforts to obtain candidates, such as region proposals or predefined anchor boxes.

While these methods have shown promising results, their primary focus remains on designing stronger multi-modal decoders. By contrast, much less work has been done to improve the learnable queries, which have been gained extensive attention in the object detection field. The queries that are inputted to the decoder in these methods are typically generated through random initialization or by utilizing linguistic embeddings. We argued that this vanilla approach has two critical issues: i) this target-agnostic query inevitably increases the learning difficulty of the decoder. ii) During the query learning process, these methods tend to focus solely on the deepest visual features of the backbone, overlooking the texture information that is crucial for the grounding task and present in low and mid-level features, as emphasized by [15, 38].

Drawing from these discussions, this paper seeks to address two critical research questions: i) Can we produce the target-related referential queries for the decoder to alleviate the learning difficulty that the decoder faces? and ii) How can we effectively incorporate the multi-level visual context information into the query learning process? We believe that tackling these issues together would promote the learnable query to more comprehensively and accurately learn the corresponding target object information in the image for the visual grounding task.

Considering CLIP [34] carries rich visual-language alignment knowledge, thus we adopt it as the backbone of our approach. Existing methods typically apply CLIP on the visual grounding by finetuning its parameters, as CLIP's training objective is to match entire images with text descriptions, rather than capturing fine-grained alignment between regions and textual elements. This may risk losing the general knowledge of CLIP and require significant computational resources. To tackle the challenges mentioned above, we propose a novel approach called RefFormer. Our approach incorporates a query adaptation (QA) module to generate referential queries, which provide the decoder with target-related context (as illustrated in Figure 1 (b)). By strategically inserting QA

module into different layers of CLIP, the query adaptively learns target-related information from multilevel image feature maps, and iteratively refines the acquired information layer by layer. Furthermore, our proposed Reformer can also act as an adapter, enabling CLIP to keep frozen and preserve the original rich knowledge. It adopts the bi-directional interaction scheme, performs the multi-modal fusion by incorporating a small number of trainable parameters, and residually injects new taskspecific knowledge into CLIP throughout the entire feature extraction process. Extensive experiments conducted on five popular visual grounding benchmarks (i.e., RefCOCO/+/g [53, 31], Flickr30K [33], and ReferItGame [18]) demonstrate the superior performance of our proposed method.

Our contributions can be summarized as follows: (1) Unlike the previous methods that focus on designing sophisticated multi-modal decoders, we further improve the learning process of the learnable queries, a crucial aspect that has been overlooked in existing work. (2) We propose a query adaption module (QA), which can adaptively capture the target-related context, providing valuable referential knowledge for the decoder. (3) We conduct extensive experiments on five visual grounding benchmarks, demonstrating the effectiveness and potential of our method.

2 Related Work

2.1 Visual Grounding

Visual grounding aims to ground the target objects based on natural language descriptions by understanding the given images and expressions. Early work [52, 12, 27, 3, 58] primarily focuses on two-stage methods, which formulates the grounding task as a matching task. These methods employ object detectors to generate proposal candidates and then identify the best-matched candidate based on the matching score computed between each proposal and the referring expression. For example, MAttNet [52] proposes to decompose the language expression into three phrase embeddings, which are used to trigger three separate visual modules. While achieving successful performance, two-stage methods heavily rely on the quality of the generated proposals. Based on this, some studies [48, 47, 4] have been dedicated to one-stage methods to remove the proposal generation stage. These methods typically fuse visual features and language features first and then densely regress the bounding box on each position of the feature map grid. For instance, FAOA [48] incorporates linguistic embedding into the YOLOv3 detector to establish a one-stage pipeline, balancing between accuracy and speed.

Recently, transformer-based visual grounding methods [17, 7, 21, 23, 57, 40, 50, 8, 36, 10] have emerged, which leverages the self-attention mechanism to effectively capture intra- and inter-modality relationships and achieve improved performance. Among these methods, the mainstream approach [17, 7, 10, 21, 8, 36] adopts DETR-like structures to decode bounding boxes from learnable queries. For example, Transvg [7] and Transvg++ [8] employ a standard multimodal transformer framework, along with the REG token, to establish multi-modal correspondence and predict the coordinates of the referring object. Notably, the performance improvement of these methods primarily arises from the design of stronger backbones or multi-modal decoders. In this work, we focus on the design of learnable queries, which have received considerable attention in object detection field.

2.2 Learnable Queries in DETR and Its Variants

In the object detection field, DETR presents an end-to-end object detection model that is built in an encoder-decoder transformer architecture. However, it suffers from slow training convergence. To address this issue, some follow-up works [55, 19, 49, 45, 20, 26, 24, 54] solve this issue by optimizing the learnable queries in DETR. For instance, Anchor DETR [45] directly treats 2D reference points as queries, while DAB-DETR [24] further investigates the role of queries in DETR and proposes the use of 4D anchor boxes as queries. In contrast to these model-level improvements, DN-DETR [20] introduces query denoising training to mitigate the instability of bipartite graph matching, which is further enhanced by DINO [54].

Additionally, similar research have been explored in other tasks [22, 13, 37]. For example, EaTR [13] formulates a video as a set of event units and treats video-specific event units as dynamic moment queries in video grounding tasks. MTR++ [37] introduces distinct learnable intention queries generated by the k-means clustering algorithm to handle trajectory prediction across different motion modes in motion prediction tasks.

Amanis

shooting

kites

1

2

3

4

5

6

9

10

QA

11

12

QA

QA

Decoder

1

2

3

4

5

6

9

10

11

12

Image

Expression

Prediction

{𝐻

$

'

}

𝐻

%

𝑄

CLIP

CLIP

𝑄

(

3 Preliminary

𝑄

MM

Interaction

Self

Interaction

!"#

MM Interaction Self Interaction 𝐹 $ ! ! 𝐺 $ ! 𝐺 ! Considering the impressive vision-language alignment capability of CLIP, we take it as the backbone of our method to extract image and text representations, and keep the parameters frozen during training. The feature extraction process can be represented as follows:

𝐹 % % Image Encoder. For an input image V ∈ R H × W × 3 , it is divided into N non-overlapping patches of size P × P , where N v = H × W P 2 . These patches are then flattened into a set of vectors, represented as { x i v ∈ R 3 P 2 } N i =1 . Next, these vectors are transformed into token embeddings using a linear projection layer ϕ e ( · ) . Furthermore, a classification token x cls ∈ R D is added at the beginning of the token embeddings. Subsequently, the positional embeddings E v are incorporated, and a layer normalization (LN) is applied. This process can be expressed as follows:

where [;] denotes the concatenate operation. The sequence of tokens Z 0 v is then passed through L transformer layers. Each transformer layer comprises two submodules: the multi-head self-attention (MHSA) and the multilayer perceptron (MLP), with each submodule preceded by layer normalization.

where Z i v ∈ ❘ N × D denote the output of i -th transformer layer.

Text Encoder. Given an referring expression T , it is first transformed into a sequence of word embeddings using lower-cased byte pair encoding representations X t . The word embeddings are bracketed with the [SOS] and [EOS] tokens, producing a sequence of length N t . Similar to the image encoder, these tokens are summed with positional embeddings E t and passed through the L transformer layers to extract the text representations:

where Z 0 t = [ x sos ; X t ; x eos ] + E t , representing the word embedding layer in text encoder.

4 Method

The framework is shown in Figure 2. In the following, we first describe our query adaptation module in Section 4.1. We then introduce our decoder that decodes with referential query and training objectives in Section 4.2 and Section 4.3. Furthermore, we extend RefFormer to dense grounding task in Section 4.4. Finally, we provide a discussion in Section 4.5.

𝑄

!

CAMF

𝑄

!"#

𝐹

$

!

𝐹

%

!

𝐺

$

!

𝐺

%

!

𝑄

!

Z

$

!

Z

%

!

( 𝑍

$

!

( 𝑍

%

!

Down

Up

CAMF

TR

TR

R

R

𝑟

%

r

$

̃ 𝑟

%

̃ 𝑟

$

4.1 Query Adaptation Module (QA)

In this section, we propose a QA module (as shown in Figure 3) that can generate the referential query to provide the decoder with the target-related context, thereby enhancing the decoder's grounding capabilities. Importantly, our approach incorporates multi-level features into the query learning process, enabling the queries to capture more comprehensive target object information and can be refined layer by layer. Furthermore, QA can also act as an adapter , eliminating the need to fine-tune the entire parameters of the backbone.

Down-projection. Considering the image and language representations Z i v and Z i t obtained from the i -th layer of the backbone, we initially use the MLP layers ϕ i vd ( · ) and ϕ i td ( · ) to project them to lower-dimensional features to reduce the computation memory:

Condition Aggregation and Multi-modal Fusion (CAMF). We randomly initialize N q learnable queries Q ∈ R N q × D l , where D l denotes the dimension after projected. These queries are specifically designed to capture potential target object context. Next, we concatenate these queries with the image features and input them, along with the language features into the CAMF block. Specifically, the CAMF block mainly consists of a cross-attention layer, which takes the image and query features [ Q ; F v ] and language features F t as the query respectively. This approach enables us to not only incorporate the expression condition into the learnable queries Q but also to extract relevant information from other modalities, thereby facilitating the fusion of target-related crossmodal features. Besides, we incorporate two learnable regulation tokens r v , r t ∈ R D l to modulate the final output of each QA. This process can be formalized as follows:

where Q i -1 represents learnable queries that output from the previous QA, while Q 0 are randomly initialized. The symbol [; ] indicates the concatenate operation, and MHCA ( , , ) and LN ( · ) denote the multi-head cross-attention layers and layer normalization, respectively.

Target-related Context Refinement (TR). Following this, we feed the queries ˆ Q c and multi-modal ehanced feature maps ˆ F i v and ˆ F i t into the TR block. First, we use the queries ˆ Q c that have aggregated conditions to interact with the multi-modal enhanced image feature maps ˆ F i v , refining the targetrelated visual context within them.

Moreover, for feature maps ˆ F i v and ˆ F i t that have aggreaged other modality information, we use the self-attention to further enhance their target-related contextual semantics:

Up-projection. Finally, we utilize MLP to restore the channel dimension of the image and language features back to their original sizes. These features are then passed as inputs to the next layer of the

backbone in a residual manner. Prior to this, we utilize the regulation token to modulate the features G v and G t , which helps prevent the multi-modal signal from overpowering the original signal.

where ϕ vu ( · ) and ϕ tu ( · ) denote the MLP layer, and σ ( · ) denotes the sigmoid function.

Finally, by iteratively performing the above process, the queries Q can progressively focus on the target-related context, and generate the referential queries to provide the prior context for the decoder.

4.2 Decoding with Referential Query.

Language-guided Multi-level Fusion. By inserting the QA at different layers of CLIP, the referential queries can be adaptively updated using the multi-level image feature maps. Additionally, to enhance the image features in decoder, we aggregate the multi-level visual features under the language guidance to yield language-aware multi-level image features. Specifically, given a multi-level image feature set { ˆ Z k v } (including low, mid, and high levels), where k ∈ K represents selected layer index, we inject the language features Z last t (the final output of the text encoder) into each level of image features using MHCA:

where ϕ mt ( · ) and ϕ mv ( · ) denote the linear project function used to map features to the same dimension. Besides, H t sos represents the [SOS] token in H t , which extracts the global information of the text. Subsequently, the multi-level language-aware image features are produced by simple concatenation, followed by a linear projection function ϕ vml ( · ) to map to the original dimension:

Decoding. Following, we first initialize the queries Q ′ with the same size as the referential query Q , and add them together to utilize the prior context in Q . Note that, to avoid interference from Q ′ during the initial stage, we initialize Q ′ as an all-zero matrix. Then, we concatenate the queries with the image features to interact with the language features to aggregate the condition information and produce the multi-modal feature map H mm . This can be represented as:

where ϕ q ( · ) is the MLP layer, which regulates the significance of the query Q . As the importance approaches zero, the query degenerate into a vanilla query. Then, we feed the queries O c and multi-modal feature map H mm into the MHCA layer to extract target embddings O ∈ ❘ N q × D . It can be formulated as:

where ϕ r ( · ) represents the linear projection function.

Grounding Head. We built the two MLPs ( ϕ box ( · ) and ϕ cls ( · ) ) over the target embeddings O . The final outputs consist of the predicted center coordinates of the target object, denoted as b = ( x, y, h, w ) ∈ R 4 , and the predicted confidence score y ∈ ❘ 2 that encompass the target object:

4.3 Training Objectives

Similar to DETR, we employ bipartite matching to find the best match between the predictions { b, y } and the ground-truth targets { b tgt , y tgt } . In our case, the class prediction is confidence prediction aims to estimate the confidence of a query containing a target object. To supervise the training, we use the box prediction losses (L1 and GIoU), and a cross-entropy loss after matching.

where λ denotes the corresponding loss weight. Additionally, to encourage the referential queries in every QA module to effectively focus on the target-related context, we also introduce the auxiliary loss L aux that is similar to the above objective function to provide supervision for them. The final training objective can be defined as:

where λ aux denotes the weight of the auxiliary loss.

4.4 Extend to Dense Grounding

In addition to object-level grounding, our method can easily extend to the dense grounding task by incorporating a segmentation head. Specifically, similar to the MaskFormer [5], we utilize the MLP to transform the target embeddings O into mask embeddings M ∈ ❘ N q × D . The binary mask prediction s i = [0 , 1] ∈ ❘ H × W is then computed by performing a dot product between the mask embeddings M and the multi-modal feature map H mm and followed by a sigmoid activation. During training, we use the mask prediction losses (Focal and Dice), which can be defined as follows:

where s gt denotes the ground-truth mask.

4.5 Discussion

In this work, we aim to explore how to further optimize the learning process of queries. To reduce the learning difficulties posed by vanilla query, we introduce a simple query adaption module to adaptively capture target-related context and iteratively refine it. As illustrated in Figure 5, the attention maps produced by each query adaption module consistently align with our objective: to progressively focus on the target-related context and provide prior context for the decoder. It is worth noting that while "multi-level", "adapter", and "self-attention" may be extensively applied in other research fields, our approach aims to integrate them to address the challenges in visual grounding tasks, instead of designing a specific module to achieve the mentioned functions individually.

5 Experiment

5.1 Datasets and Evaluation Metric

RefCOCO/RefCOCO+/RefCOCOg. RefCOCO [53] comprises 19,994 images featuring 50,000 referred objects, divided into train, val, testA, and testB sets. Similarly, RefCOCO+ [53] contains 19,992 images with 49,856 referred objects and 141,564 referring expressions. It contains more attributes than absolute locations compared to RefCOCO, and has the same split. RefCOCOg [31] has 25,799 images with 49,856 referred objects and expressions. Following a common version of split [32], i.e., train, val, and test sets.

Flickr30K. Flickr30k Entities [33] contains 31,783 images and 158k caption sentences with 427k annotated phrases. We follow [7] to split the images into 29,783 for training, 1000 for validation, and 1000 for testing, and report the performance on the test set.

ReferItGame. ReferItGame [18] includes 20,000 images with 120,072 referring expressions for 19,987 referred objects. We follow [7] to split the dataset into train, validation and test sets, and report the performance on the test set.

5.2 Implementation Details

Following [7, 36], the resolution of the input image is resized to 640 × 640. We employ the pre-trained CLIP as our backbone to extract both image and language features, and we freeze its parameters

during training. The model is optimized end-to-end using AdamW for 40 epochs, with a batch size of 32. We set the learning rate to 1e-4 and the weight decay to 1e-2. The experiments are conducted on V100 GPUs. The loss weight λ iou , λ L 1 , λ ce , and λ aux , we set to 3.0, 1.0, 1.0, and 0.1. For dense grounding, we set the parameters λ focal , and λ dice to 5.0, and 1.0.

5.3 Comparisons with State-of-the-art Methods

RECTask. For REC task, we compare the performance with the state-of-the-art REC methods, including the two-stage methods, one-stage methods, and transformer-based methods. As reported in Table 1 and Table 2, our proposed method achieves the best performance. In particular, when comparing to the transformer-based method Dynamic MDETR, which adopts the DETR-like structure and uses the same backbone as ours, we can see that our method performs better with +0 . 53% , +1 . 90% , +1 . 62% on RefCOCO, +1 . 11% , +2 . 44% , +6 . 21% on RefCOCO+, and +4 . 94% , +4 . 18% on RefCOCOg. Additionally, under multiple/extra datasets setting, our method also surpasses recent state-of-the-art methods that incorporate large language models or utilize more training data.

RES Task. Following RefTR [21] and VG-LAW [40], we also conduct the dense grounding experiments and report the results in Table 3 in terms of mIoU. It can be seen that our model

achieves superior performance without extra deliberate design to dense grounding, demonstrating the generalization of our method.

5.4 Ablation Studies

In this section, we conduct the ablation studies to verify the effectiveness the each part of our proposed method on RefCOCOg. Following previous work [7, 17, 8], the visual backbone we apply the ViT-Base/32.

Effect on the position of QA. As presented in Table 5, firstly, we can observe that removing the QA would lead to a Sharp decline in performance, highlighting the effectiveness of QA. We then explored the impact of QA's position in the CLIP to determine where QA should be added. We chose three groups for the ablation study: { 4 , 8 , 12 } , { 4 , 6 , 8 , 10 , 12 } , and { 2 , 4 , 6 , 8 , 10 , 12 } . The results indicate that performance is best when we use the { 4 , 6 , 8 , 10 , 12 } configuration. Therefore, we default to this position in our experiments.

Effect on the layers of multi-level fusion. In Table 4, we analyze the impact of the fusion layers in the decoder. We first conduct experiments using single-level image features, and then proceed with multilevel features. The results show that utilizing multi-level features significantly improves performance, demonstrating that low- and mid-level features complement high-level features. Additionally, using the { 4 , 8 , 12 } achieves the best performance, which we adopt for our experiments.

Effect on the different backbone. In the first line of Table 6, we apply our method to singlemodal encoders, i.e., Swin-Transformer + Bert. The results demonstrate that our method is not only applicable to multi-modal encoders but is also compatible with single-modal encoders.

Effect on the auxiliary loss. In the second line of Table 6, We experiment with auxiliary loss, and the results demonstrate the effectiveness of auxiliary loss. By employing auxiliary loss, the reference query can capture the target-related visual contexts more effectively.

Effect on learnable queries. In the third line of Table 6, we validate the effectiveness of the learnable queries. Specifically, we replace the prior queries generated by the QA module with randomly initialized queries or linguistic embeddings input to decoder while keeping other modules unchanged. We can observe that introducing the prior queries can bring significant performance improvement. This result demonstrates that prior queries aid the decoder in more accurately locating the target object. Additionally, we investigate the accuracy of our referential queries, which are designed to provide prior information to the decoder. Since the channel dimension in the QA module is lower, the reference query may not accurately predict the coordinates of the targets.

Convergence curves. In Figure 4 we illustrate the convergence curve of our proposed method compared to other open-source DETR-like visual grounding methods. Notably, our method demonstrates accelerated training convergence, reducing the training time by half compared to the TransVG, while also outperforming other existing methods.

5.5 Qualitative Results

As shown in Figure 5, the attention maps in the QA module illustrate the refined process of how the referential query captures the target-related context. Initially, the attention map appears noisy but gradually focuses on the target-related context, such as the couch in (a). By incorporating the referential query, the attention map in the decoder accurately concentrates on the target object.

100

TransVg

RefTR

80

Ours

5

0

60

Prec

Method

Params(M)

40

Transvg

144.00

RefTR

144.73

Ours

29.20

20

0

20

40

60

80

Epoch

(

a

)

(

b

)

Image

QA

Decoder

Expression

:

thecouchsittingindirectly in front of the window

Expression

:

atableinfrontofkids

1

1

( c ) Expression : awomanskiingwithherchild

Besides, it is important to note that the referential query may not precisely focus on the target object due to the lower feature dimension in the QA module, but it still captures target-related information.

6 Concluding and Remarks

In this paper, we propose a novel approach, called RefFormer that can be seamlessly integrated into CLIP. The RefFormer can not only generate the referential query to provide the target-related context for decoder, but also act as the adaptor to preserve the original knowledge of CLIP and reduce the training cost. Extensive experiments demonstrate the effectiveness of our method, and visualization results illustrate the refined process of our proposed RefFormer.

Limitations : Although our method is specifically designed for the REC task and surpasses existing SOTA methods in REC, there is still significant room for improvement in the RES task. This is because we have not yet optimized our approach specifically for the RES task.

7 Acknowledgments

This work was supported in part by National Science and Technology Major Project under Grant 2023ZD0121300, National Natural Science Foundation of China under Grants 62088102, 12326608 and 62106192, Natural Science Foundation of Shaanxi Province under Grant 2022JC-41, and Fundamental Research Funds for the Central Universities under Grant XTR042021005.

References

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020.

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195 , 2023.

Long Chen, Wenbo Ma, Jun Xiao, Hanwang Zhang, and Shih-Fu Chang. Ref-nms: Breaking proposal bottlenecks in two-stage referring expression grounding. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 1036-1044, 2021.

Xinpeng Chen, Lin Ma, Jingyuan Chen, Zequn Jie, Wei Liu, and Jiebo Luo. Real-time referring expression comprehension by single-stage grounding network. arXiv preprint arXiv:1812.03426 , 2018.

Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems , 34:1786417875, 2021.

Zesen Cheng, Kehan Li, Peng Jin, Siheng Li, Xiangyang Ji, Li Yuan, Chang Liu, and Jie Chen. Parallel vertex diffusion for unified visual grounding. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 1326-1334, 2024.

Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1769-1779, 2021.

Jiajun Deng, Zhengyuan Yang, Daqing Liu, Tianlang Chen, Wengang Zhou, Yanyong Zhang, Houqiang Li, and Wanli Ouyang. Transvg++: End-to-end visual grounding with language conditioned vision transformer. IEEE transactions on pattern analysis and machine intelligence , 2023.

Jianfeng Dong, Yabing Wang, Xianke Chen, Xiaoye Qu, Xirong Li, Yuan He, and Xun Wang. Reading-strategy inspired visual representation learning for text-to-video retrieval. IEEE transactions on circuits and systems for video technology , 32(8):5680-5694, 2022.

Ye Du, Zehua Fu, Qingjie Liu, and Yunhong Wang. Visual grounding with transformers. In 2022 IEEE International Conference on Multimedia and Expo (ICME) , Jul 2022.

Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems , 33:6616-6628, 2020.

Richang Hong, Daqing Liu, Xiaoyu Mo, Xiangnan He, and Hanwang Zhang. Learning to compose and reason with language tree structures for visual grounding. IEEE transactions on pattern analysis and machine intelligence , 44(2):684-696, 2019.

Jinhyun Jang, Jungin Park, Jin Kim, Hyeongjun Kwon, and Kwanghoon Sohn. Knowing where to focus: Event-aware transformer for video grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13846-13856, 2023.

Jiayi Ji, Yiwei Ma, Xiaoshuai Sun, Yiyi Zhou, Yongjian Wu, and Rongrong Ji. Knowing what to learn: a metric-oriented focal mechanism for image captioning. IEEE Transactions on Image Processing , 31:4321-4335, 2022.

Dongsheng Jiang, Yuchen Liu, Songlin Liu, Xiaopeng Zhang, Jin Li, Hongkai Xiong, and Qi Tian. From clip to dino: Visual encoders shout in multi-modal large language models. 2023.

Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, and Tieniu Tan. Locate then segment: A strong pipeline for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9858-9867, 2021.

Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1780-1790, 2021.

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 787-798, 2014.

Feng Li, Ailing Zeng, Shilong Liu, Hao Zhang, Hongyang Li, Lei Zhang, and Lionel M Ni. Lite detr: An interleaved multi-scale encoder for efficient detr. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18558-18567, 2023.

Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13619-13627, 2022.

Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. Advances in neural information processing systems , 34:19652-19664, 2021.

Pandeng Li, Chen-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random to real. Advances in neural information processing systems , 36, 2024.

Yue Liao, Aixi Zhang, Zhiyuan Chen, Tianrui Hui, and Si Liu. Progressive language-customized visual feature learning for one-stage visual grounding. IEEE Transactions on Image Processing , 31:4266-4277, 2022.

Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201.12329 , 2022.

Sun-Ao Liu, Yiheng Zhang, Zhaofan Qiu, Hongtao Xie, Yongdong Zhang, and Ting Yao. Caris: Context-aware referring image segmentation. In Proceedings of the 31st ACM International Conference on Multimedia , pages 779-788, 2023.

Wenze Liu, Hao Lu, Yuliang Liu, and Zhiguo Cao. Box-detr: Understanding and boxing conditional spatial queries. arXiv preprint arXiv:2307.08353 , 2023.

Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li. Improving referring expression grounding with cross-modal attention-guided erasing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1950-1959, 2019.

Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 10034-10043, 2020.

Yiwei Ma, Jiayi Ji, Xiaoshuai Sun, Yiyi Zhou, and Rongrong Ji. Towards local visual modeling for image captioning. Pattern Recognition , 138:109420, 2023.

Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-toend multi-grained contrastive learning for video-text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia , pages 638-647, 2022.

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 11-20, 2016.

Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Modeling context between objects for referring expression understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14 , pages 792-807. Springer, 2016.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision , pages 2641-2649, 2015.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for universal visual perception. 2024.

Fengyuan Shi, Ruopeng Gao, Weilin Huang, and Limin Wang. Dynamic mdetr: A dynamic multimodal transformer decoder for visual grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Mainak Singha, Ankit Jha, Bhupendra Solanki, Shirsha Bose, and Biplab Banerjee. Applenet: Visual attention parameterized prompt learning for few-shot remote sensing image generalization using clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Wei Su, Peihan Miao, Huanzhang Dou, Yongjian Fu, and Xi Li. Referring expression comprehension using language adaptive inference. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 2357-2365, 2023.

Wei Su, Peihan Miao, Huanzhang Dou, Gaoang Wang, Liang Qiao, Zheyang Li, and Xi Li. Language adaptive weight generation for multi-task visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10857-10866, 2023.

Yabing Wang, Jianfeng Dong, Tianxiang Liang, Minsong Zhang, Rui Cai, and Xun Wang. Cross-lingual cross-modal retrieval with noise-robust learning. In Proceedings of the 30th ACM International Conference on Multimedia , pages 422-433, 2022.

Yabing Wang, Fan Wang, Jianfeng Dong, and Hao Luo. Cl2cm: Improving cross-lingual crossmodal retrieval via cross-lingual knowledge transfer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 5651-5659, 2024.

Yabing Wang, Le Wang, Qiang Zhou, Zhibin Wang, Hao Li, Gang Hua, and Wei Tang. Multimodal llm enhanced cross-lingual cross-modal retrieval. arXiv preprint arXiv:2409.19961 , 2024.

Yabing Wang, Shuhui Wang, Hao Luo, Jianfeng Dong, Fan Wang, Meng Han, Xun Wang, and Meng Wang. Dual-view curricular optimal transport for cross-lingual cross-modal retrieval. IEEE Transactions on Image Processing , 33:1522-1533, 2024.

Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 2567-2575, 2022.

Shiyu Xuan, Qingpei Guo, Ming Yang, and Shiliang Zhang. Pink: Unveiling the power of referential comprehension for multi-modal llms. 2024.

Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual grounding by recursive sub-query construction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16 , pages 387-404. Springer, 2020.

Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach to visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4683-4693, 2019.

Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang. Efficient detr: improving end-to-end object detector with dense prior. arXiv preprint arXiv:2104.01318 , 2021.

Jiabo Ye, Junfeng Tian, Ming Yan, Xiaoshan Yang, Xuwu Wang, Ji Zhang, Liang He, and Xin Lin. Shifting more attention to visual backbone: Query-modulated refinement networks for end-to-end visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15502-15512, 2022.

Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704 , 2023.

Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1307-1315, 2018.

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 69-85. Springer, 2016.

Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and HeungYeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 , 2022.

Chuyang Zhao, Yifan Sun, Wenhao Wang, Qiang Chen, Errui Ding, Yi Yang, and Jingdong Wang. Ms-detr: Efficient detr training with mixed supervision. arXiv preprint arXiv:2401.03989 , 2024.

Peizhi Zhao, Shiyi Zheng, Wenye Zhao, Dongsheng Xu, Pijian Li, Yi Cai, and Qingbao Huang. Rethinking two-stage referring expression comprehension: A novel grounding and segmentation method modulated by point. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 7487-7495, 2024.

Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. In European Conference on Computer Vision , pages 598-615. Springer, 2022.

Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and Anton Van Den Hengel. Parallel attention: A unified framework for visual object discovery through dialogs and queries. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4252-4261, 2018.

A Appendix

1

2

3

4

5

6

Number

73.5

74.0

74.5

75.0

75.5

76.0

76.5

77.0

77.5

val

1

2

3

4

5

6

Number

73.5

74.0

74.5

75.0

75.5

76.0

76.5

test

(a) val

(b) test

A.1 Effect on the RefFormer's direction.

In RefFormer, the QA module can serve as an adapter, injecting specific knowledge into the frozen CLIP model. In Table 7, we investigate the direction of feature flow from QA module. We find that using a dual-direction approach achieves the best performance. Through QA module, language features have aggregated relevant visual context information. As pointed to [25], incorporating rich visual context into linguistic features aids in achieving strong vision-language alignment and better indicating target objects.

A.2 Effect on the number of learnable queries.

A.3 Visualization

Due to space limitations, we present additional visualization results here. As shown in Figure 7, the referential queries gradually focus on the target object and effectively provide target-related context for the decoder. These results demonstrate the effectiveness of our proposed methods.

QA

Decoder

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction (Section 1) accurately reflect our paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

470

472

473

477

479

483

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We fully describe the proposed model and implementation details in Section 4 and Section 5.2, and we submit our main code in the form of a zipped file in additional supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We submit our main code in the form of a zipped file in additional supplementary materials, and we will release the complete code after review.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe the datasets, metrics and implementation details in Sec. ?? and supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because it would be too computationally expensive. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We only provide the GPU type in the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the visual grounding task, which proposes a query adaption module that can be seamlessly integrated into CLIP.  By strategically inserting this module into different layers of CLIP, the learnable query can adaptively learn target-related information from multi-level image feature maps, and iteratively refine the acquired information layer by layer.  Additionally, it can offer prior information to the decoder, effectively mitigating the learning difficulty of the decoder, and accurately concentrating on the target object.  Extensive experiments on five visual grounding benchmarks (RefCOCO/+/g, ReferItgame, Flickr30k) validate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1.This paper introduces a query adaption module that can adaptively learn target-related information from different layers in CLIP, and provides the visualization results of the attention map in experiments. This design attracted me and made me feel interested.\n\n2.The proposed QA module can not only capture target information adaptively, but also act as an adapter to avoid adjusting Backbone's parameters. Compared with the previous fine-tuning Backbone's work, this module has advantages and achieves better performance with smaller training parameters.\n\n3.In this task, unlike most prior DETR-like research that primarily concentrates on decoder design, this paper focuses on learnable query optimization , making it innovative in its approach.\n\n4.The authors conduct experiments on multiple visual grounding benchmarks (RefCOCO/RefCOCO+/RefCOCOg, Flickr30K, and ReferItGame) and also experiment on the RES benchmark, achieving improved performance. This provides robust empirical support for the effectiveness of the method.\n\nWEAKNESSES:\n1.It is better to provide the experiment results without using auxiliary loss, which can further observe the influence of auxiliary loss on the referential query.\n\n2.Is it feasible to directly utilize the referential query without subsequent decoding operations? I am interested in the accuracy of the referential query.\n\n3.Typo: On line 72, \"we propose a query adaption module, RefFormer...\". It appears that there might be an error in this sentence.\n\nQUESTIONS:\nSee weakness, please.  If the author can solve my above confusion well, I will consider raising the score.\n\nLIMITATIONS:\nIn the limitations section, the author states that this method still has room for improvement in RES tasks. In addition, by visualizing the attention map, I find that this method accurately locates the area of the target object. Therefore, I believe this method holds great potential in RES tasks, and I hope the author can make further advancements in future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
This paper addresses the generation of queries for the decoder. The authors propose RefFormer to generate the referential query with the prior context. A query adaption module is proposed to capture extensive target-related context and provide valuable referential knowledge for the decoder. Extensive experiments validate the effectiveness of the proposed method.

STRENGTHS:
1. This paper proposes a new method for query initialization, where multimodal information is continuously embedded during the feature extraction stage to provide the query with prior knowledge.
2. Extensive experiments validate the proposed method.

WEAKNESSES:
This idea seems straightforward but lacks some innovation. The method mentioned in the paper has been applied in other fields, such as R2-tuning. The proposed module is also simple to implement. Although the model achieves good results, I believe the paper does not meet the standards of NIPS.

QUESTIONS:
1. Could you provide the number of trainable parameters and the inference speed?

LIMITATIONS:
Yes. Limitations are addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe existing one-stage visual grounding methods suffer from cross-modal learning difficulty and focus simply on the deepest visual features. This paper designs a query adaption (QA) module to provide target-related referential queries for the decoder. The proposed architecture Reformer is based on a CLIP model with multiple QA modules inserted in different layers of CLIP. The proposed method can be applied to both REC and RES tasks. The performance is validated on various benchmarks.\n\nSTRENGTHS:\n-\tThe paper is generally well-written and easy to follow.\n-\tThe motivation for addressing two issues present in the existing one-stage grounding model is clear, and the idea of enhancing cross-modal interaction is intuitive.\n-\tThe attention maps show the refining process across different layers as expected.\n-\tIt works for both object-level and dense grounding.\n\nWEAKNESSES:\n- The proposed module QA has to be inserted into specific positions inside the VLM to improve grounding performance. Experiments in Table 5 indicate the importance of layer selection. However,  it is unclear why the 5 layers perform the best and why these indices are selected. As mentioned in the introduction, the low and mid-level features are crucial for grounding (line 48), yet lower layers (e.g., 2nd) containing rich low-level features diminish the performance. In other words, the selection of layers is not explained theoretically, while 3 combinations are not enough to justify the selection empirically. \n- In condition aggregation and multi-modal fusion (CAMF) module, it seems that the query interacts more with the visual representations (i.e., the upper part in Figure 3), while the alternative that mainly integrates query with textual representations seems feasible (i.e., place the query to the lower part in Figure 3). The motivation and advantage of design are missing.\n- The backbone and training data information (especially the version of Reformer) is not provided in Tables 2 and 3.\n\nQUESTIONS:\n- How does the paper select the number of inserted QA modules? How does the paper select the indices of inserted layers? Why do more layers or lower layers hurt the performance? \n- In the condition aggregation and multi-modal fusion (CAMF) module, the query seems to interact more with the visual representations (i.e., the upper part in Figure 3). Why is it designed in such a way? What is the advantage over mainly based on textual representations?\n- Does QA support different model architectures?\n- What is the version of the reformer in Tables 2 and 3?\n\nLIMITATIONS:\nWhile the motivation is clear and the idea is intuitive, the technical implementation of essential modules/structures lacks theoretical or empirical justification.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel visual grounding framework, called RefFormer, aims to improve the learning process of learnable queries. Specifically, it introduces a  query adaption module (QA) that can be seamlessly integrated into different layers of CLIP, which can not only provide prior information to the decoder, but also act as an adapter to learn task-specific knowledge. The effectiveness of the proposed method is extensively validated through experiments conducted on the five popular visual grounding benchmarks, namely RefCOCO, RefCOCO+, RefCOCOg, Flickr30K, and ReferItGame. Furthermore, the authors extend this approach to dense grounding tasks, demonstrating its effectiveness and generalization.\n\nSTRENGTHS:\n1. The proposed method is novel, which integrates query learning and adapter into one module, effectively leveraging various levels of feature information within CLIP.\n2. By introducing QA module, models can adaptively extract target-related information from the backbone and provide prior information to the decoder instead of relying on randomly initialized queries.\n3. This paper achieves better performance with fewer training parameters and data.\n4. Sufficient experiments have been conducted to show the promising results of the proposed method.\n5. The writing of this paper is good and clear.\n\nWEAKNESSES:\nWhile the paper is clearly written, some areas can still be improved. Some suggestions are as follows：\n1. Line 72 should correct \"RefFormer\" to \"QA\".\n2. A more detailed description is needed for the text side in the QA module, such as how the interaction with visual features is initiated.\n3. Why is a direct use of the referential query not preferred, and why is an additional learnable query introduced in the decoder?\n4. Specify the dimension along which the concatenation operation in Section 4.1 is performed.\n5. Correct \"[r_t, F^i_t]\" to \"[r_t; F^i_t]\" in Equation 9.\n6. Provide experimental details regarding the channel dimension of the QA module in the down-projection process.\n\nQUESTIONS:\n1. Why is a direct use of the referential query not preferred, and why is an additional learnable query introduced in the decoder?\n2. 6. Provide experimental details regarding the channel dimension of the QA module in the down-projection process.\n\nLIMITATIONS:\nSee the above weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}