Referencing Where to Focus: Improving Visual Grounding with Referential Query

Authors: yabing  wang
zhuotao  tian
qingpei  guo
zheng  qin
sanping  zhou
ming  yang
le  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 18:20:56

Abstract: Visual Grounding aims to localize the referring object in an image given a natural language expression. Recent advancements in DETR-based visual grounding methods have attracted considerable attention, as they directly predict the coordinates of the target object without relying on additional efforts, such as pre-generated proposal candidates or pre-defined anchor boxes. However, existing research primarily focuses on designing stronger multi-modal decoder, which typically generates learnable queries by random initialization or by using linguistic embeddings.  This vanilla query generation approach inevitably increases the learning difficulty for the model, as it does not involve any target-related information at the beginning of decoding. Furthermore, they only use the deepest image feature during the query learning process, overlooking the importance of features from other levels. To address these issues, we propose a novel approach, called RefFormer. It consists of the query adaption module that can be seamlessly integrated into CLIP and generate the referential query to provide the prior context for decoder, along with a task-specific decoder. By incorporating the referential query into the decoder, we can effectively mitigate the learning difficulty of the decoder, and accurately concentrate on the target object. Additionally, our proposed query adaption module can also act as an adapter, preserving the rich knowledge within CLIP without the need to tune the parameters of the backbone network. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method, outperforming state-of-the-art approaches on five visual grounding benchmarks.

Main paper: 
 introduction: 
 Visual grounding is a challenging multi-modal task that involves localizing a specific object based on a given natural language description. This task requires algorithms to comprehend fine-grained human language expressions and accurately establish correspondences with the target objects. In recent years, it has gained significant attention in research due to its potential for advancing vision-language understanding, such as cross-modal retrieval [42, 30, 44, 9, 41, 43] and image captioning [14, 29] . Existing works in this field typically follow the object detection framework and incorporate multimodal fusion to tackle this task. Earlier studies [52, 12, 27, 3, 58] mainly focus on a two-stage pipeline, which first generates a set of region proposals using object detectors, and then finds the bestmatched region by interacting these regions with linguistic expressions. However, the performance of this method is limited by the quality of the generated region candidates. To address this issue, some studies [48, 47, 4] adopt the one-stage pipeline, which removes the proposal generation stage. Unfortunately, these methods make dense predictions with a sliding window over pre-defined anchor boxes, resulting in sub-optimal performance due to the failure to capture object relations effectively. Recently, some methods [17, 7, 10, 21, 8, 36] inspired by the DETR [1] structure, which adopt a standard multi-modal transformer framework to establish the multi-modal correspondence (as shown in Figure 1 (a) ). These methods predict bounding boxes of target objects directly from learnable queries, eliminating the need for extra efforts to obtain candidates, such as region proposals or predefined anchor boxes.While these methods have shown promising results, their primary focus remains on designing stronger multi-modal decoders. By contrast, much less work has been done to improve the learnable queries, which have been gained extensive attention in the object detection field. The queries that are inputted to the decoder in these methods are typically generated through random initialization or by utilizing linguistic embeddings. We argued that this vanilla approach has two critical issues: i) this target-agnostic query inevitably increases the learning difficulty of the decoder. ii) During the query learning process, these methods tend to focus solely on the deepest visual features of the backbone, overlooking the texture information that is crucial for the grounding task and present in low and mid-level features, as emphasized by [15, 38] .Drawing from these discussions, this paper seeks to address two critical research questions: i) Can we produce the target-related referential queries for the decoder to alleviate the learning difficulty that the decoder faces? and ii) How can we effectively incorporate the multi-level visual context information into the query learning process? We believe that tackling these issues together would promote the learnable query to more comprehensively and accurately learn the corresponding target object information in the image for the visual grounding task.Considering CLIP [34] carries rich visual-language alignment knowledge, thus we adopt it as the backbone of our approach. Existing methods typically apply CLIP on the visual grounding by finetuning its parameters, as CLIP's training objective is to match entire images with text descriptions, rather than capturing fine-grained alignment between regions and textual elements. This may risk losing the general knowledge of CLIP and require significant computational resources. To tackle the challenges mentioned above, we propose a novel approach called RefFormer. Our approach incorporates a query adaptation (QA) module to generate referential queries, which provide the decoder with target-related context (as illustrated in Figure 1 (b) ). By strategically inserting QA module into different layers of CLIP, the query adaptively learns target-related information from multilevel image feature maps, and iteratively refines the acquired information layer by layer. Furthermore, our proposed Reformer can also act as an adapter, enabling CLIP to keep frozen and preserve the original rich knowledge. It adopts the bi-directional interaction scheme, performs the multi-modal fusion by incorporating a small number of trainable parameters, and residually injects new taskspecific knowledge into CLIP throughout the entire feature extraction process. Extensive experiments conducted on five popular visual grounding benchmarks (i.e., RefCOCO/+/g [53, 31] , Flickr30K [33] , and ReferItGame [18] ) demonstrate the superior performance of our proposed method.Our contributions can be summarized as follows: (1) Unlike the previous methods that focus on designing sophisticated multi-modal decoders, we further improve the learning process of the learnable queries, a crucial aspect that has been overlooked in existing work. (2) We propose a query adaption module (QA), which can adaptively capture the target-related context, providing valuable referential knowledge for the decoder. (3) We conduct extensive experiments on five visual grounding benchmarks, demonstrating the effectiveness and potential of our method.2 Related Work 
visual grounding: 
 Visual grounding aims to ground the target objects based on natural language descriptions by understanding the given images and expressions. Early work [52, 12, 27, 3, 58] primarily focuses on two-stage methods, which formulates the grounding task as a matching task. These methods employ object detectors to generate proposal candidates and then identify the best-matched candidate based on the matching score computed between each proposal and the referring expression. For example, MAttNet [52] proposes to decompose the language expression into three phrase embeddings, which are used to trigger three separate visual modules. While achieving successful performance, two-stage methods heavily rely on the quality of the generated proposals. Based on this, some studies [48, 47, 4] have been dedicated to one-stage methods to remove the proposal generation stage. These methods typically fuse visual features and language features first and then densely regress the bounding box on each position of the feature map grid. For instance, FAOA [48] incorporates linguistic embedding into the YOLOv3 detector to establish a one-stage pipeline, balancing between accuracy and speed.Recently, transformer-based visual grounding methods [17, 7, 21, 23, 57, 40, 50, 8, 36, 10] have emerged, which leverages the self-attention mechanism to effectively capture intra-and inter-modality relationships and achieve improved performance. Among these methods, the mainstream approach [17, 7, 10, 21, 8, 36] adopts DETR-like structures to decode bounding boxes from learnable queries. For example, Transvg [7] and Transvg++ [8] employ a standard multimodal transformer framework, along with the REG token, to establish multi-modal correspondence and predict the coordinates of the referring object. Notably, the performance improvement of these methods primarily arises from the design of stronger backbones or multi-modal decoders. In this work, we focus on the design of learnable queries, which have received considerable attention in object detection field.In the object detection field, DETR presents an end-to-end object detection model that is built in an encoder-decoder transformer architecture. However, it suffers from slow training convergence. To address this issue, some follow-up works [55, 19, 49, 45, 20, 26, 24, 54] solve this issue by optimizing the learnable queries in DETR. For instance, Anchor DETR [45] directly treats 2D reference points as queries, while DAB-DETR [24] further investigates the role of queries in DETR and proposes the use of 4D anchor boxes as queries. In contrast to these model-level improvements, DN-DETR [20] introduces query denoising training to mitigate the instability of bipartite graph matching, which is further enhanced by DINO [54] .Additionally, similar research have been explored in other tasks [22, 13, 37] . For example, EaTR [13] formulates a video as a set of event units and treats video-specific event units as dynamic moment queries in video grounding tasks. MTR++ [37] introduces distinct learnable intention queries generated by the k-means clustering algorithm to handle trajectory prediction across different motion modes in motion prediction tasks.A man is shooting kites 1 2 MM Interaction MM Interaction Self Interaction Self Interaction 𝑄 !"# 𝐹 $ ! 𝐹 % ! 𝐺 $ ! 𝐺 % ! 𝑄 !Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of the work in the paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 Considering the impressive vision-language alignment capability of CLIP, we take it as the backbone of our method to extract image and text representations, and keep the parameters frozen during training. The feature extraction process can be represented as follows:Image Encoder. For an input image V ∈ R H×W ×3 , it is divided into N non-overlapping patches of size P × P , where N v = H×W P 2 . These patches are then flattened into a set of vectors, represented as{x i v ∈ R 3P 2 } N i=1 .Next, these vectors are transformed into token embeddings using a linear projection layer ϕ e (•). Furthermore, a classification token x cls ∈ R D is added at the beginning of the token embeddings. Subsequently, the positional embeddings E v are incorporated, and a layer normalization (LN) is applied. This process can be expressed as follows:EQUATIONwhere [;] denotes the concatenate operation. The sequence of tokens Z 0 v is then passed through L transformer layers. Each transformer layer comprises two submodules: the multi-head self-attention (MHSA) and the multilayer perceptron (MLP), with each submodule preceded by layer normalization.EQUATIONEQUATIONwhere Z i v ∈ R N ×D denote the output of i-th transformer layer.Text Encoder. Given an referring expression T , it is first transformed into a sequence of word embeddings using lower-cased byte pair encoding representations X t . The word embeddings are bracketed with the [SOS] and [EOS] tokens, producing a sequence of length N t . Similar to the image encoder, these tokens are summed with positional embeddings E t and passed through the L transformer layers to extract the text representations:EQUATIONEQUATION)where Z 0 t = [x sos ; X t ; x eos ] + E t , representing the word embedding layer in text encoder.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: Our paper does not include theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
method: 
 The framework is shown in Figure 2 . In the following, we first describe our query adaptation module in Section 4.1. We then introduce our decoder that decodes with referential query and training objectives in Section 4.2 and Section 4.3. Furthermore, we extend RefFormer to dense grounding task in Section 4.4. Finally, we provide a discussion in Section 4.5.CAMF 𝑄 !"# 𝐹 $ ! 𝐹 % ! 𝐺 $ ! 𝐺 % ! 𝑄 ! Z $ ! Z % ! ( 𝑍 $ ! ( 𝑍 % ! Down Up CAMF TR TR R R 𝑟 % r $ r % r$Figure 3 : Illustration of our proposed Query Adaption Module, which mainly consists of CAMF and TR modules to generate the referential queries and promote the multi-modal features interaction. "R" represents the feature modulation.In this section, we propose a QA module (as shown in Figure 3 ) that can generate the referential query to provide the decoder with the target-related context, thereby enhancing the decoder's grounding capabilities. Importantly, our approach incorporates multi-level features into the query learning process, enabling the queries to capture more comprehensive target object information and can be refined layer by layer. Furthermore, QA can also act as an adapter, eliminating the need to fine-tune the entire parameters of the backbone.Considering the image and language representations Z i v and Z i t obtained from the i-th layer of the backbone, we initially use the MLP layers ϕ i vd (•) and ϕ i td (•) to project them to lower-dimensional features to reduce the computation memory:EQUATIONCondition Aggregation and Multi-modal Fusion (CAMF). We randomly initialize N q learnable queries Q ∈ R Nq×D l , where D l denotes the dimension after projected. These queries are specifically designed to capture potential target object context. Next, we concatenate these queries with the image features and input them, along with the language features into the CAMF block. Specifically, the CAMF block mainly consists of a cross-attention layer, which takes the image and query features [Q; F v ] and language features F t as the query respectively. This approach enables us to not only incorporate the expression condition into the learnable queries Q but also to extract relevant information from other modalities, thereby facilitating the fusion of target-related crossmodal features. Besides, we incorporate two learnable regulation tokens r v , r t ∈ R D l to modulate the final output of each QA. This process can be formalized as follows:EQUATIONwhere Q i-1 represents learnable queries that output from the previous QA, while Q 0 are randomly initialized. The symbol [; ] indicates the concatenate operation, and M HCA(, , ) and LN (•) denote the multi-head cross-attention layers and layer normalization, respectively.Target-related Context Refinement (TR). Following this, we feed the queries Qc and multi-modal ehanced feature maps Fi v and Fi t into the TR block. First, we use the queries Qc that have aggregated conditions to interact with the multi-modal enhanced image feature maps Fi v , refining the targetrelated visual context within them.EQUATIONMoreover, for feature maps Fi v and Fi t that have aggreaged other modality information, we use the self-attention to further enhance their target-related contextual semantics:EQUATIONUp-projection. Finally, we utilize MLP to restore the channel dimension of the image and language features back to their original sizes. These features are then passed as inputs to the next layer of the backbone in a residual manner. Prior to this, we utilize the regulation token to modulate the features G v and G t , which helps prevent the multi-modal signal from overpowering the original signal.EQUATION)where ϕ vu (•) and ϕ tu (•) denote the MLP layer, and σ(•) denotes the sigmoid function.Finally, by iteratively performing the above process, the queries Q can progressively focus on the target-related context, and generate the referential queries to provide the prior context for the decoder.4.2 Decoding with Referential Query.Language-guided Multi-level Fusion. By inserting the QA at different layers of CLIP, the referential queries can be adaptively updated using the multi-level image feature maps. Additionally, to enhance the image features in decoder, we aggregate the multi-level visual features under the language guidance to yield language-aware multi-level image features. Specifically, given a multi-level image feature set { Ẑk v } (including low, mid, and high levels), where k ∈ K represents selected layer index, we inject the language features Z last t (the final output of the text encoder) into each level of image features using MHCA:EQUATIONwhere ϕ mt (•) and ϕ mv (•) denote the linear project function used to map features to the same dimension. Besides, H tsos represents the [SOS] token in H t , which extracts the global information of the text. Subsequently, the multi-level language-aware image features are produced by simple concatenation, followed by a linear projection function ϕ vml (•) to map to the original dimension:EQUATIONEQUATIONDecoding. Following, we first initialize the queries Q ′ with the same size as the referential query Q, and add them together to utilize the prior context in Q. Note that, to avoid interference from Q ′ during the initial stage, we initialize Q ′ as an all-zero matrix. Then, we concatenate the queries with the image features to interact with the language features to aggregate the condition information and produce the multi-modal feature map H mm . This can be represented as:EQUATIONEQUATIONwhere ϕ q (•) is the MLP layer, which regulates the significance of the query Q. As the importance approaches zero, the query degenerate into a vanilla query. Then, we feed the queries O c and multi-modal feature map H mm into the MHCA layer to extract target embddings O ∈ R Nq×D . It can be formulated as:EQUATIONEQUATIONwhere ϕ r (•) represents the linear projection function.Grounding Head. We built the two MLPs (ϕ box (•) and ϕ cls (•)) over the target embeddings O.The final outputs consist of the predicted center coordinates of the target object, denoted as b = (x, y, h, w) ∈ R 4 , and the predicted confidence score y ∈ R 2 that encompass the target object:EQUATIONSimilar to DETR, we employ bipartite matching to find the best match between the predictions {b, y} and the ground-truth targets {b tgt , y tgt }. In our case, the class prediction is confidence prediction aims to estimate the confidence of a query containing a target object. To supervise the training, we use the box prediction losses (L1 and GIoU), and a cross-entropy loss after matching.EQUATIONwhere λ denotes the corresponding loss weight. Additionally, to encourage the referential queries in every QA module to effectively focus on the target-related context, we also introduce the auxiliary loss L aux that is similar to the above objective function to provide supervision for them. The final training objective can be defined as:EQUATIONwhere λ aux denotes the weight of the auxiliary loss.In addition to object-level grounding, our method can easily extend to the dense grounding task by incorporating a segmentation head. Specifically, similar to the MaskFormer [5] , we utilize the MLP to transform the target embeddings O into mask embeddings M ∈ R Nq×D . The binary mask prediction s i = [0, 1] ∈ R H×W is then computed by performing a dot product between the mask embeddings M and the multi-modal feature map H mm and followed by a sigmoid activation. During training, we use the mask prediction losses (Focal and Dice), which can be defined as follows:EQUATIONwhere s gt denotes the ground-truth mask.In this work, we aim to explore how to further optimize the learning process of queries. To reduce the learning difficulties posed by vanilla query, we introduce a simple query adaption module to adaptively capture target-related context and iteratively refine it. As illustrated in Figure 5 , the attention maps produced by each query adaption module consistently align with our objective: to progressively focus on the target-related context and provide prior context for the decoder. It is worth noting that while "multi-level", "adapter", and "self-attention" may be extensively applied in other research fields, our approach aims to integrate them to address the challenges in visual grounding tasks, instead of designing a specific module to achieve the mentioned functions individually. Flickr30K. Flickr30k Entities [33] contains 31,783 images and 158k caption sentences with 427k annotated phrases. We follow [7] to split the images into 29,783 for training, 1000 for validation, and 1000 for testing, and report the performance on the test set.ReferItGame. ReferItGame [18] includes 20,000 images with 120,072 referring expressions for 19,987 referred objects. We follow [7] to split the dataset into train, validation and test sets, and report the performance on the test set.Evaluation Metric. For referring expression comprehension (REC), we use Prec@0.5 evaluation protocol to evaluate the accuracy, which is consistent with prior works. In this evaluation, a predicted region is considered correct if its intersection-over-union (IoU) with the ground-truth bounding box is greater than 0.5. For referring expression segmentation (RES), we report the Mean IoU (MIoU) between the predicted segmentation mask and ground truth mask.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We fully describe the proposed model and implementation details in Section 4 and Section 5.2, and we submit our main code in the form of a zipped file in additional supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
implementation details: 
 Following [7, 36] , the resolution of the input image is resized to 640 × 640. We employ the pre-trained CLIP as our backbone to extract both image and language features, and we freeze its parameters during training. The model is optimized end-to-end using AdamW for 40 epochs, with a batch size of 32. We set the learning rate to 1e-4 and the weight decay to 1e-2. The experiments are conducted on V100 GPUs. The loss weight λ iou , λ L1 , λ ce , and λ aux , we set to 3.0, 1.0, 1.0, and 0.1. For dense grounding, we set the parameters λ f ocal , and λ dice to 5.0, and 1.0.REC Task. For REC task, we compare the performance with the state-of-the-art REC methods, including the two-stage methods, one-stage methods, and transformer-based methods. As reported in Table 1 and Table 2 , our proposed method achieves the best performance. In particular, when comparing to the transformer-based method Dynamic MDETR, which adopts the DETR-like structure and uses the same backbone as ours, we can see that our method performs better with +0.53%, +1.90%, +1.62% on RefCOCO, +1.11%, +2.44%, +6.21% on RefCOCO+, and +4.94%, +4.18% on RefCOCOg.Additionally, under multiple/extra datasets setting, our method also surpasses recent state-of-the-art methods that incorporate large language models or utilize more training data.RES Task. Following RefTR [21] and VG-LAW [40] , we also conduct the dense grounding experiments and report the results in Table 3 in terms of mIoU. It can be seen that our model achieves superior performance without extra deliberate design to dense grounding, demonstrating the generalization of our method.In this section, we conduct the ablation studies to verify the effectiveness the each part of our proposed method on RefCOCOg. Following previous work [7, 17, 8] , the visual backbone we apply the ViT-Base/32.Effect on the position of QA. As presented in Table 5 , firstly, we can observe that removing the QA would lead to a Sharp decline in performance, highlighting the effectiveness of QA. We then explored the impact of QA's position in the CLIP to determine where QA should be added. We chose three groups for the ablation study: {4, 8, 12}, {4, 6, 8, 10, 12}, and {2, 4, 6, 8, 10, 12}. The results indicate that performance is best when we use the {4, 6, 8, 10, 12} configuration. Therefore, we default to this position in our experiments.In Table 4 , we analyze the impact of the fusion layers in the decoder. We first conduct experiments using single-level image features, and then proceed with multilevel features. The results show that utilizing multi-level features significantly improves performance, demonstrating that low-and mid-level features complement high-level features. Additionally, using the {4, 8, 12} achieves the best performance, which we adopt for our experiments.Effect on the different backbone. In the first line of Table 6 , we apply our method to singlemodal encoders, i.e., Swin-Transformer + Bert. The results demonstrate that our method is not only applicable to multi-modal encoders but is also compatible with single-modal encoders.Effect on the auxiliary loss. In the second line of Table 6 , We experiment with auxiliary loss, and the results demonstrate the effectiveness of auxiliary loss. By employing auxiliary loss, the reference query can capture the target-related visual contexts more effectively.Effect on learnable queries. In the third line of Table 6 , we validate the effectiveness of the learnable queries. Specifically, we replace the prior queries generated by the QA module with randomly initialized queries or linguistic embeddings input to decoder while keeping other modules unchanged. We can observe that introducing the prior queries can bring significant performance improvement. This result demonstrates that prior queries aid the decoder in more accurately locating the target object. Additionally, we investigate the accuracy of our referential queries, which are designed to provide prior information to the decoder. Since the channel dimension in the QA module is lower, the reference query may not accurately predict the coordinates of the targets.Convergence curves. In Figure 4 we illustrate the convergence curve of our proposed method compared to other open-source DETR-like visual grounding methods. Notably, our method demonstrates accelerated training convergence, reducing the training time by half compared to the TransVG, while also outperforming other existing methods.As shown in Figure 5 , the attention maps in the QA module illustrate the refined process of how the referential query captures the target-related context. Initially, the attention map appears noisy but gradually focuses on the target-related context, such as the couch in (a). By incorporating the referential query, the attention map in the decoder accurately concentrates on the target object. Besides, it is important to note that the referential query may not precisely focus on the target object due to the lower feature dimension in the QA module, but it still captures target-related information.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We submit our main code in the form of a zipped file in additional supplementary materials, and we will release the complete code after review.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
concluding and remarks: 
 In this paper, we propose a novel approach, called RefFormer that can be seamlessly integrated into CLIP. The RefFormer can not only generate the referential query to provide the target-related context for decoder, but also act as the adaptor to preserve the original knowledge of CLIP and reduce the training cost. Extensive experiments demonstrate the effectiveness of our method, and visualization results illustrate the refined process of our proposed RefFormer.Limitations : Although our method is specifically designed for the REC task and surpasses existing SOTA methods in REC, there is still significant room for improvement in the RES task. This is because we have not yet optimized our approach specifically for the RES task. A.1 Effect on the RefFormer's direction.In RefFormer, the QA module can serve as an adapter, injecting specific knowledge into the frozen CLIP model. In Table 7 , we investigate the direction of feature flow from QA module. We find that using a dual-direction approach achieves the best performance. Through QA module, language features have aggregated relevant visual context information. As pointed to [25] , incorporating rich visual context into linguistic features aids in achieving strong vision-language alignment and better indicating target objects.A.2 Effect on the number of learnable queries.We depict the performance in terms of Prec@0.5 according to the number of learnable queries N q in Figure 6 . When we adopt the N q = 3, the performance is best. However, further increases yield only slight improvements in metrics, as a large number of N q increases the difficulty of the model. Therefore, we default set the N q = 3 in our experiments.Due to space limitations, we present additional visualization results here. As shown in Figure 7 , the referential queries gradually focus on the target object and effectively provide target-related context for the decoder. These results demonstrate the effectiveness of our proposed methods.(c) Expression: a chair which a boy is sitting in (d) Expression: a sandwich with colby jack cheese , tomato , and lettuce , on fresh cut bread• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] .Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.(a) Expression: a man with an orange t-shirt with pizza (b) Expression: standing giraffe in the background(e) Expression: a man wearing nike shoes in a bright neon green top is playing tennis(f) Expression: a red taxi standing next to a yellow sign 
 
 Appendix:  


Decision: The paper received mixed feedback, with one reviewer recommending rejection while three others favored acceptance. The authors proposed a novel approach called RefFormer, introducing a query adaptation module designed to learn target-related context more effectively. Although Reviewer DDuF pointed out that the idea appears straightforward and lacks significant innovation, noting similarities to existing methods. During rebuttal, the authors provided detailed comparison between their paper and the mentioned ECCV 24 work, highlighting differences in motivation and implementation. Additionally, they integrated the mentioned method into their framework for experimental analysis. The AC believed the rebuttal is persuasive and clearly demonstrate the novelty and innovation of the proposed method. Therefore, the AC has decided to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel visual grounding framework, called RefFormer, aims to improve the learning process of learnable queries. Specifically, it introduces a  query adaption module (QA) that can be seamlessly integrated into different layers of CLIP, which can not only provide prior information to the decoder, but also act as an adapter to learn task-specific knowledge. The effectiveness of the proposed method is extensively validated through experiments conducted on the five popular visual grounding benchmarks, namely RefCOCO, RefCOCO+, RefCOCOg, Flickr30K, and ReferItGame. Furthermore, the authors extend this approach to dense grounding tasks, demonstrating its effectiveness and generalization.
strength_weakness: strength_weakness: While the paper is clearly written, some areas can still be improved. Some suggestions are as follows：
1. Line 72 should correct "RefFormer" to "QA".
2. A more detailed description is needed for the text side in the QA module, such as how the interaction with visual features is initiated.
3. Why is a direct use of the referential query not preferred, and why is an additional learnable query introduced in the decoder?
4. Specify the dimension along which the concatenation operation in Section 4.1 is performed.
5. Correct "[r_t, F^i_t]" to "[r_t; F^i_t]" in Equation 9.
6. Provide experimental details regarding the channel dimension of the QA module in the down-projection process.
questions: questions: 1. Why is a direct use of the referential query not preferred, and why is an additional learnable query introduced in the decoder?
2. 6. Provide experimental details regarding the channel dimension of the QA module in the down-projection process.
limitations: limitations: See the above weaknesses.
paper_summary: paper_summary: The existing one-stage visual grounding methods suffer from cross-modal learning difficulty and focus simply on the deepest visual features. This paper designs a query adaption (QA) module to provide target-related referential queries for the decoder. The proposed architecture Reformer is based on a CLIP model with multiple QA modules inserted in different layers of CLIP. The proposed method can be applied to both REC and RES tasks. The performance is validated on various benchmarks.
strength_weakness: strength_weakness: - The proposed module QA has to be inserted into specific positions inside the VLM to improve grounding performance. Experiments in Table 5 indicate the importance of layer selection. However,  it is unclear why the 5 layers perform the best and why these indices are selected. As mentioned in the introduction, the low and mid-level features are crucial for grounding (line 48), yet lower layers (e.g., 2nd) containing rich low-level features diminish the performance. In other words, the selection of layers is not explained theoretically, while 3 combinations are not enough to justify the selection empirically. 
- In condition aggregation and multi-modal fusion (CAMF) module, it seems that the query interacts more with the visual representations (i.e., the upper part in Figure 3), while the alternative that mainly integrates query with textual representations seems feasible (i.e., place the query to the lower part in Figure 3). The motivation and advantage of design are missing.
- The backbone and training data information (especially the version of Reformer) is not provided in Tables 2 and 3.
questions: questions: - How does the paper select the number of inserted QA modules? How does the paper select the indices of inserted layers? Why do more layers or lower layers hurt the performance? 
- In the condition aggregation and multi-modal fusion (CAMF) module, the query seems to interact more with the visual representations (i.e., the upper part in Figure 3). Why is it designed in such a way? What is the advantage over mainly based on textual representations?
- Does QA support different model architectures?
- What is the version of the reformer in Tables 2 and 3?
limitations: limitations: While the motivation is clear and the idea is intuitive, the technical implementation of essential modules/structures lacks theoretical or empirical justification.
paper_summary: paper_summary: This paper addresses the generation of queries for the decoder. The authors propose RefFormer to generate the referential query with the prior context. A query adaption module is proposed to capture extensive target-related context and provide valuable referential knowledge for the decoder. Extensive experiments validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: This idea seems straightforward but lacks some innovation. The method mentioned in the paper has been applied in other fields, such as R2-tuning. The proposed module is also simple to implement. Although the model achieves good results, I believe the paper does not meet the standards of NIPS.
questions: questions: 1. Could you provide the number of trainable parameters and the inference speed?
limitations: limitations: Yes. Limitations are addressed.
paper_summary: paper_summary: This paper focuses on the visual grounding task, which proposes a query adaption module that can be seamlessly integrated into CLIP.  By strategically inserting this module into different layers of CLIP, the learnable query can adaptively learn target-related information from multi-level image feature maps, and iteratively refine the acquired information layer by layer.  Additionally, it can offer prior information to the decoder, effectively mitigating the learning difficulty of the decoder, and accurately concentrating on the target object.  Extensive experiments on five visual grounding benchmarks (RefCOCO/+/g, ReferItgame, Flickr30k) validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.It is better to provide the experiment results without using auxiliary loss, which can further observe the influence of auxiliary loss on the referential query.

2.Is it feasible to directly utilize the referential query without subsequent decoding operations? I am interested in the accuracy of the referential query.

3.Typo: On line 72, "we propose a query adaption module, RefFormer...". It appears that there might be an error in this sentence.
questions: questions: See weakness, please.  If the author can solve my above confusion well, I will consider raising the score.
limitations: limitations: In the limitations section, the author states that this method still has room for improvement in RES tasks. In addition, by visualizing the attention map, I find that this method accurately locates the area of the target object. Therefore, I believe this method holds great potential in RES tasks, and I hope the author can make further advancements in future work.

Comments: title: Comment to the author
Response to Reviewer DDuF
Please let us know whether you have any follow-up questions
Please let us know whether you have any follow-up questions
Re-response to the author:
comment: Thanks to the author's response, which has effectively addressed my concerns regarding the details of this paper. Overall, I find this work to be impressive, I am inclined to maintain my recommendation to accept it. I hope the author can address the aforementioned issues, as well as those raised by other reviewers, in the revised version.
Thank you for your feedback.  Our approach fundamentally differs from [a]. As previously mentioned, our study concentrates on improving learnable queries within DETR-like architectures, as opposed to solely concentrating on parameter adjustments as in [a]. Furthermore, experiments have demonstrated significant improvements resulting from our approach.

We would like to emphasize once again our motivation: 
1) We focus on improving the learning process of the learnable queries, different from the previous work that emphasizes the design of sophisticated multi-modal decoders.
2) We propose a query adaption module that not only adaptively captures the target-related context, providing valuable referential knowledge for the decoder, but can also serve as the adapter. 
3) By strategically inserting the QA module into different layers of CLIP, the query adaptively learns target-related information from multi-level image feature maps, and iteratively refines the acquired information layer by layer. 

Additionally, we conduct extensive experiments to validate the effectiveness of our method and provide the visualization results to demonstrate the reasonability of our proposed referential query.

We sincerely hope you will reconsider our paper and appreciate your valuable suggestions for improving this work.  If there are any other questions or areas you'd like to discuss, we welcome further conversation.
I appreciate the author's rebuttal.  I still think the novelty of this paper is limited. So I keep my score.
Dear Reviewer RwV9,

I am glad for the recognition of our work. Please feel free to raise any further questions, and we are more than happy to continue the discussion with you. Thanks again for your great efforts and constructive advice in reviewing this paper! 

Regards, Authors
Dear Reviewer DDuF,

We hope you are doing well. As the discussion period is coming to an end (Aug 13), we wanted to reach out to see if you have any follow-up questions. If so, we would appreciate the opportunity to respond before the discussion period ends. We believe our above messages should have addressed your concerns, and therefore may warrant an increase in score if you find them helpful as well. Would you please let us know if you have any further questions or concerns? We are happy to continue the discussion.
Thank you very much again for your thoughtful review and help in improving the paper. We appreciate your time and consideration.

Regards, Authors
The authors have addressed my concerns, and it would be better if they could include the supplement experiments in their revision. I tend to accept the paper after the rebuttal.
Thank the author’s careful response. After carefully reviewing the authors' responses to my concerns and those of other reviewers, the author included a large number of experiments for further explanation and analysis. This has deepened my appreciation for the significance of this paper.

The authors have excellently addressed my concerns. As a result, I have decided to raise my score.