HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding

Authors: xiaoshan  yang
changsheng  xu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 17:07:35

Abstract: Visual grounding, which aims to ground a visual region via natural language, is a task that heavily relies on cross-modal alignment. Existing works utilized uni-modal pre-trained models to transfer visual or linguistic knowledge separately while ignoring the multimodal corresponding information. Motivated by recent advancements in contrastive language-image pre-training and low-rank adaptation (LoRA) methods, we aim to solve the grounding task based on multimodal pre-training. However, there exists significant task gaps between pre-training and grounding. Therefore, to address these gaps, we propose a concise and efficient hierarchical multimodal fine-grained modulation framework, namely HiVG. Specifically, HiVG consists of a multi-layer adaptive cross-modal bridge and a hierarchical multimodal low-rank adaptation (HiLoRA) paradigm. The cross-modal bridge can address the inconsistency between visual features and those required for grounding, and establish a connection between multi-level visual and text features. HiLoRA prevents the accumulation of perceptual errors by adapting the cross-modal features from shallow to deep layers in a hierarchical manner. Experimental results on five datasets demonstrate the effectiveness of our approach and showcase the significant grounding capabilities as well as promising energy efficiency advantages. The project page: https://github.com/linhuixiao/HiVG.

Main paper: 
 introduction: 
 Visual Grounding (VG), also known as Referring Expression Comprehension (REC) or Phrase Grounding (PG) [9, 19, 29, 41, 47, 62, 63, 75] , is a fundamental and challenging task at the intersection fields of vision-language understanding, which can be potentially used in a wide range of applications [1, 6, 35] , such as visual question answering [1] , human-machine interaction [6] etc.. Unlike object detection [37, 38] , which requires a predefined and fixed set of categories, grounding is not limited to specific categories but instead needs to identify the specific image region according to the language expression semantics. Thus, grounding is a task that strongly relies on the interaction and alignment of multimodal features.Existing state-of-the-art (SOTA) approaches [9, 10, 16, 55, 71, 76, 78] utilize uni-modal pre-trained detection models or language models (e.g., ResNet [15] , Swin Transformer [39] , DETR [4] , ViT-Det [28] , BERT [11] , RoBERTa [36] etc.) to facilitate grounding learning. These methods separately transfer the language or vision knowledge from pre-trained models by using resource-consuming fully parameter fine-tuning, ignoring the multimodal corresponding information. Therefore, it is natural for us to consider using crossmodal pre-trained models as a solution to the grounding problem. By utilizing language supervision from large-scale unlabeled data, Vision-Language Pre-training (VLP) can acquire comprehensive multimodal representations. Recently, the remarkable success of Contrastive Language-Image Pre-training (CLIP) [48] has demonstrated its ability to learn general visual concepts, which assists many multimodal tasks to achieve remarkable improvements [23, 44, 48, 61] . In visual grounding, there are also works, e.g., CLIP-VG [63] and Dynamic-MDETR [53] , which consider using CLIP. However, existing methods mainly utilize the CLIP as a backbone to extract strong vision and language features, without comprehensively investigating on the significant task gap between the pre-trained CLIP and the downstream grounding, which hinders exploiting the full potential of pre-training models. In this work, we scrutinize the task gap from two aspects. (1) Data bias. There inevitably exists a certain bias in data between the large-scale pretraining and grounding. Directly utilizing the frozen vision backbone of the CLIP may extract visual features sensitive to general objects that are not the focus of the query in visual grounding. For example, as shown in Fig. 1-(a) , the middle giraffe receives highlight attention, but it has little relation to the grounding task. (2) Difference in learning objectives. The visual grounding task needs to find the precise image region that has the target object expressed by the query sentence. In contrast, CLIP works as a multimodal pre-trained model, which is only constrained to coarsely align noisy image and text data [51] in a self-supervised way. In addition, the self-supervised constraint is only performed at the final layer. When directly using the pre-trained CLIP in visual grounding, some valuable fine-grained visual information in the bottom vision layers may be discarded, which brings challenges for accurately locating the object box. For example, as shown in Fig. 1-(a) , the left giraffe receives relatively small attention areas, which leads to inaccurate box of the target object.It is not trivial to address the two kinds of task gaps. (1) For the task gap of data bias, extracting the features of the query text to guide the visual feature learning is a potential way to solve it. However, the query text has a feature space that is very different from the visual space and it is difficult to find the appropriate semantic information from the query features to guide the learning of different vision layers. (2) For the task gap of learning objectives, to adapt the pre-trained CLIP to the grounding task, a straightforward way is fine-tuning the pre-trained weights. Whereas, this scheme may lead to catastrophic forgetting, which is harmful to retain the general knowledge learned by the pre-trained models. Another potential solution is to employ Low-Rank Adaptation (LoRA) [18] by fine-tuning only a few parameters. However, simply applying LoRA does not achieve fine-grained adaptation and even lead to performance degradation. Since high-level features depend on low-level features, and they are susceptible to perturbations of the shallow features. If all layers of a large-scale pre-trained model are adapted simultaneously, perceptual errors in bottom layers may accumulate and amplify. Therefore, it is necessary to consider a hierarchical approach for progressively adapt fine-grained visual features from shallow to deep layers.In this paper, we propose a hierarchical multimodal fine-grained modulation framework to more effectively adapt the pre-trained CLIP to grounding, namely HiVG. It is a concise and efficient endto-end framework that can alleviate two kinds of task gaps (i.e., data bias and learning objectives) through a multi-layer adaptive crossmodal bridge and a hierarchical low-rank adaptation paradigm.Firstly, to address the inconsistency between visual features of the pre-trained CLIP and those required for grounding, as well as establish a connection between multi-level visual and text features, we have designed a multi-layer adaptive cross-modal bridge. Specifically, the cross-modal bridge includes a sample-agnostic semantic weighting module and a multi-head cross-attention module. The weighting module incorporates learnable multi-level sampleagnostic adaptive weights, facilitating the selection of appropriate linguistic features through a residual operation. The multi-head cross-attention utilizes the selected multi-level text features for guiding the learning of the visual features required in grounding. The sample-agnostic semantic weighting scheme is inspired by [3, 8] , i.e., specific layers of a pre-trained model may have distinct responses to certain concepts or semantics that are independent of the input and relevant to the network layers.Secondly, to prevent the accumulation of errors layer by layer in the downstream adaptation process of the pre-trained model, we propose Hierarchical Low-rank Adaptation (HiLoRA) paradigm. Existing methods mainly utilize LoRA [18] as a parameter-efficient fine-tuning (PEFT) method to learn a single round along with the entire model. Different from previous methods [18, 54] , we divide the network layers of the pre-trained CLIP into multiple layer groups. The low-rank adaptation is allocated into multiple stages where each stage relates to several layer groups. Then, during the adaptation process, visual features are recursively and hierarchically adapted from shallow to deep layers in a hierarchical manner. Simultaneously, with the assist of the multi-layer cross-modal bridge, HiLoRA can not only achieve fine-grained hierarchical adaptation, but also enable the low-rank matrix perception based on the vision and language cross-modal information.As show in Fig. 1-(b ) and (c), benefiting from the hierarchical multimodal fine-grained modulation structure, HiVG exhibits heightened sensitivity towards visual region information, demonstrates enhanced comprehension of complex text, and significantly bridges the gap between pre-training and grounding tasks. Our method achieves SOTA performance on five widely used datasets, including RefCOCO/+/g [41, 75] , ReferitGame [21] and Flickr30KEntities [46] . HiVG outperforms the CLIP-based SOTA method, Dynamic-MDETR [53] , on RefCOCO/+/g datasets by 3.15%(testB), 2.11%(testA), 4.30%(test), and also outperforms the strong detectorbased SOTA method, TransVG++ [10] , on the three datasets by 2.30%(testB), 3.36%(testA), 2.49%(test), respectively. Meanwhile, our model can obtain SOTA results on 224×224 small-resolution images without relying on high-resolution images (e.g., 640×640) like other works [10, 55, 71] . Additionally, it significantly accelerates inference processes and is 8.2× faster than TransVG++ (Fig. 4 ).The main contributions can be summarized as three-fold:• We proposed a concise hierarchical multimodal modulation framework, which utilizes the hierarchical structure to gradually adapt CLIP to grounding. HiVG achieves fine-grained interaction between multi-level visual representations and language semantics, and significantly alleviates the task gap between CLIP and grounding. • We are the first to propose the hierarchical multimodal lowrank adaptation structure. HiLoRA is a basic and concise hierarchical adaptation paradigm, which is task-agnostic. • We conducted extensive experiments to verify the effectiveness of HiVG approaches. Results show that our method achieves promising results, surpassing the SOTA methods under the same setting by a significant margin. Besides, our model offers significant computing efficiency advantages. 
related work 2.1 visual grounding: 
 Visual grounding has recently received significant research attention, and it can be categorized into several settings. On the one hand, represented by TransVG [9] , this setting involves full-parameter fine-tuning utilizing pre-trained closed-set detectors and language models. It is considered the most conventional and extensively studied setting. Under this setting, numerous complex two-stage [17, 31, 34, 74] and one-stage [68, 70, 77] methods emerged based on traditional detection networks in the early CNN era. After the introduction of ViT [12, 58] , the Transformer-based networks [9, 10, 16, 20, 27, 40, 42, 55, 71, 72] constantly pushes the accuracy to new limits. However, these works only focus on achieving grounding by using independently pre-trained uni-modal detectors and language encoders while ignoring the alignment of cross-modality information within pre-trained model itself. More recent works, such as QRNet [71] , VG-LAW [55] , TransVG++ [10] , etc., only incorporate language-guided knowledge in vision backbone without attempting multi-level fine-grained alignment of multimodal features. Motivated by this setting, several works, such as CLIP-VG [63] and Dynamic-MDETR [53] , have recently sprung up to the setting of fine-tuning with vision and language (VL) self-supervised pre-trained models. Following this setting, our study delves into a deeper perspective of hierarchical multimodal information and achieves fine-grained interaction of cross-modal features. On the other hand, with the evolution of the pre-training paradigm, many new settings have recently emerged that significantly improve the grounding performance, such as fine-tuning with box-level datasetmixed open-set detection pre-trained models (e.g., MDETR [20] , Grounding-DINO [33] , etc.), fine-tuning with box-level / multi-task mixup-supervised pre-trained models (e.g., UniTAB [69] , UNITER [7] , OFA [60] , etc.), and grounding multimodal large language models (GMLLMs, e.g., Shikra [6] , Kosmos-2 [45] , Ferret [73] , LION [5] , etc.). However, these works require a large amount of fine-grained labeled data, resulting in a relatively high training cost.With the promotion of learning general and transferable crossmodal representations [14, [64] [65] [66] [67] , VLP has become the core training paradigm of modern VL research. Benefiting from self-supervised contrastive learning, CLIP has demonstrated impressive generalization and downstream transfer ability in a series of studies [44, 48] .More recently, some works utilized CLIP to realize grounding transfer, such as adapting-CLIP [26] , ReCLIP [56] etc., but these works are limited to using CLIP features as aids in an unsupervised or zero-shot setting [22, 56] and cannot directly perform grounding.Although CLIP-VG [63] , Dynamic-MDETR [53] , JMR [79] etc., realizes grounding transfer, it does not conduct more in-depth research on the task gaps and the hierarchical cross-modal features. Unlike previous work, our study fills the gap by conducting a more comprehensive study of the cross-modal task gaps between CLIP's pre-training and downstream grounding.LoRA [18] freezes the weights of pre-trained model and injects trainable rank decomposition matrices into each layer of the Transformer [59] , thereby significantly reducing the number of trainable parameters for downstream tasks. Vanilla LoRA has been proposed in the field of natural language processing for Large Language Models (LLM) such as LLaMA2 [57] , GPT-2 [49] , GPT-3 [2] with 175B parameters, etc.. Recently, researchers have attempted to apply vanilla LoRA in the fields of cross-modal tasks [54] . However, since cross-modal tasks primarily emphasize the interaction of multimodal information in contrast to unimodal language or visual tasks, the application of LoRA to grounding tasks remains unexplored. Consequently, we propose HiLoRA as an effective solution for addressing the existing gaps in multimodal downstream transfer. 
methodology: 
 In this section, we propose our hierarchical multimodal fine-grained modulation framework for visual grounding, namely HiVG, which mainly consists of the multi-layer adaptive cross-modal bridge and the hierarchical low-rank adaptation (HiLoRA) paradigm. We will introduce each of these methods in the following sections.Our aim is to achieve fine-grained hierarchical cross-modal feature modulation, so as to narrow the task gap between the selfsupervised pre-training and grounding. Therefore, we integrate the multi-level image and text representations from a hierarchical perspective with the facilitation of multi-layer adaptive cross-modal bridge and the hierarchical LoRA paradigm. Specifically, as shown in Fig. 2 , the network architecture of HiVG consists of a CLIP image encoder, a CLIP text encoder, a grounding encoder and a regression head. Firstly, for any given image I ∈ R 3×𝐻 ×𝑊 and text T ∈ R 𝐿 𝑙 pairs, the visual and text encoders encode the image and text tokens to obtain the visual feature 𝒇 𝑣 ∈ R 𝐿 𝑣 ×𝐻 𝑣 and text feature Simultaneously, to reduce the inconsistency between the visual features of the uni-modal image backbone and those required for grounding, we introduce a multi-layer adaptive cross-modal bridge to the visual encoder that bridges image and text modalities. Each layer of the bridge has a learnable sample-agnostic weighting module, thus enabling the uni-modal visual backbone to perceive hierarchical cross-modal text features.𝒇 𝑙 ∈ R 𝐿 𝑙 ×𝐻 𝑙 ,Additionally, to prevent the accumulation and amplification of perceptual errors in the visual encoder, we propose a hierarchical low-rank adaptation (HiLoRA) paradigm to adapting the pre-trained frozen parameters. During HiLoRA training, the entire adaptation process learns from shallow to deep layers. The gradients backward from the grounding encoder are updated hierarchically and adaptively into the low-rank matrix based on both visual features and hierarchical language features. Besides, the intermediate visual features are aggregated and fed to the grounding encoder, which not only benefits the perception of multi-level visual features but also facilitates direct gradient backward updates without going from deep to shallow in the HiLoRA low-stage training.Finally, in the grounding encoder, we concatenate the multi-level visual features along with the hidden dimension, and leverage the weight 𝑊 𝑚𝑣𝑝 ∈ R (𝑚•𝐻 𝑣 ) ×𝐻 𝑔 of a MLP-based visual perceiver to project them into embedding space 𝒈 𝑣 ∈ R 𝐿 𝑣 ×𝐻 𝑔 with dimension 𝐻 𝑔 to perceive multi-level visual representations:EQUATIONTo prevent any perturbation on [𝐸𝑂𝑆] token and ensure the subsequent constraints remain unaffected, we exclusively utilize the linear projection features 𝒈 𝑙 ∈ R 𝐿 𝑙 ×𝐻 𝑔 of the last layer's text features 𝒇 𝑙𝑎𝑠𝑡 𝑙 to feed the grounding encoder. Finally, the input tokens of the grounding encoder are as follows:EQUATIONwhere 𝑐𝑙𝑠 represents the classification token [𝐶𝐿𝑆], 𝑔 𝑟 represents the learnable [𝑅𝐸𝐺] token, which is used to output the regression results [9] . The [𝐸𝑂𝑆] token is the end token of each sequence within 𝒇 𝑙 and 𝒈 𝑙 . The regression head is employed to conduct bounding box regression, which is a three-layer MLPs [9] , each consisting of a linear layer and a ReLU activation layer. It outputs the final coordinate of the predicted grounding box B = ( x, ŷ, ŵ, ĥ).The visual encoder of CLIP independently encodes the image, and the obtained multi-level visual features may be inconsistent with those required for grounding. Additionally, as inspired by [3, 8] , specific layers of a pre-trained model may exhibit distinct responses to certain concepts or semantics that are independent of the inputRegression headImage: EQUATIONwhere {𝒘 𝑖 𝑙 } 𝑛 𝑖=1 ∈ R 𝑛×𝐿 𝑙 ×𝐻 𝑙 represent the learnable multi-level sample-agnostic adaptive weights within different layers, which can promote different visual layers respond distinctly to specific textual concepts or semantics. The weighted features are obtained by dot product between the sample-agnostic adaptive weights and multi-level features. We then add the weighted features to the original features to obtain the calibrated text features { * 𝒇 𝑖 𝑙 } 𝑛 𝑖=1 . Subsequently, we concatenate and project them into visual embedding space * 𝒇 𝑚𝑙 𝑙 ∈ R 𝐿 𝑙 ×𝐻 𝑣 to perceive multi-level language representations with linear projection weight Finally, we perform a multi-head cross-attention on the calibrated multi-level text features * 𝒇 𝑚𝑙 𝑙 (as key and value) and the layernormalized visual features outputted by the self-attention in the ViT block (as query). Then, we add the resulting semantic-aware visual features 𝒇 𝑠𝑎 𝑣 back to the block as residuals after a FFN operation.EQUATIONAlthough the cross-modal bridge enables the visual encoder to incorporate language information, its residual connection manner cannot adapt the frozen parameters of the pre-trained model. As a result, there is still a discrepancy between the visual features and those required for grounding, which may lead to cumulative and amplified perceptual errors layer by layer. LoRA [18] presents a potentially feasible solution. However, as clarified in the Sec. 1, the vanilla LoRA performs one-round learning also cannot address these issues. To avoid cumulative and amplified perceptual errors, we need to design a hierarchical adaptation paradigm. Instead of directly training specific dense layers in a neural network, vanilla LoRA [18] indirectly optimizes the rank-decomposition matrices of the changes occurring in dense layers while keeping the pre-trained weights frozen. As depicted in Fig. 3-(a) , based on the vanilla LoRA definition, we can substitute the weight updates for a pre-trained weight 𝑊 0 ∈ R 𝑑 ×𝑘 with a low-rank decomposition 𝑊 0 +Δ𝑊 = 𝑊 0 +𝐵𝐴, where 𝐵 ∈ R 𝑑 ×𝑟 , 𝐴 ∈ R 𝑟 ×𝑘 , and 𝑟 ≪ min(𝑑, 𝑘), i.e., the low rank 𝑟 is much smaller than the dimension (𝑑, 𝑘) of the original model. Throughout training, 𝑊 0 remains frozen, while 𝐴 and 𝐵 encompass trainable parameters. For hidden state ℎ = 𝑊 0 𝑥, the forward procedure can be formulated as:EQUATIONTo consider the hierarchical scenario, we first define two concepts, i.e., layer group and LoRA stage. Layer group represents the divisions of the pre-trained network layers, while LoRA stage represents the execution of a small LoRA operation. By dividing the network layers of the pre-trained model into multiple layer groups, the learning of LoRA is divided into multiple stages where each stage relates to several layer groups. As depicted in Fig. 3-(b) , from a hierarchical perspective, Hierarchical LoRA (HiLoRA) structure enables downstream task adaptation progressively from the shallow to deep layer within the network with multiple LoRA stages. Specifically, we define the total layers of the pre-trained network as 𝐿, and then divide it into 𝐺 groups, each containing 𝐿/𝐺 layers. Then, we denote 𝑊 𝑙 0 ∈ R 𝑑 ×𝑘 as the pre-trained weights of 𝑙 𝑡ℎ layer block in the network, where 𝑙 ∈ [1, 𝐿]. We utilize LoRA 𝑗 (1 ≤ 𝑗 ≤ 𝐺) to represent the 𝑗 𝑡ℎ adaptation stage. We denote the low-rank matrices of HiLoRA at the 𝑗 𝑡ℎ stage of the 𝑙 𝑡ℎ layer block as 𝐴 𝑙 𝑗 and 𝐵 𝑙 𝑗 , and𝐴 𝑗 contains {𝐴 𝑙 𝑗 } 𝑗 •𝐿/𝐺 𝑙=1 , 𝐵 𝑗 contains {𝐵 𝑙 𝑗 } 𝑗 •𝐿/𝐺𝑙=1 , then each LoRA stage 𝑗 will update the low-rank matrices of 𝐴 𝑗 and 𝐵 𝑗 . We denote ℎ 𝑙 𝑗 as the hidden state ℎ at HiLoRA 𝑗 𝑡ℎ stage of the 𝑙 𝑡ℎ layer block. Then, the forward process of HiLoRA in each hidden state ℎ 𝑙 𝑗 (𝑗 ∈ [1, 𝐺]) can be formulated as:EQUATIONwhere ⌈•⌉ indicates rounding up to an integer, and ⌈𝑙 • 𝐺/𝐿⌉ stands for calculating the index of layer groups in which 𝑙 𝑡ℎ layer is located.With the assistance of the hierarchical mechanism, we can achieve better multimodal low-rank adaptation of multi-level visual features by utilizing textual semantic-aware visual features provided by the adaptive cross-modal bridge. Specifically, the layer groups of HiLoRA are associated with the insertion positions 𝐶 of the bridge. When 𝑙 > 𝑗 • 𝐿/𝐺, the forward process of HiLoRA in each hidden state ℎ 𝑙 𝑗 can be formulated as:EQUATIONWhile in 𝑙 ≤ 𝑗 • 𝐿/𝐺, the process can be formulated as:EQUATION)During the backward process, the updates are gradually performed from 1 𝑠𝑡 to 𝐺 𝑡ℎ stage, and the learning rate can vary at different stages. Additionally, we use a random Gaussian initialization for 𝐴 and 0 for 𝐵, so Δ𝑊 = 𝐵𝐴 is 0 at the beginning of training. We then scale Δ𝑊 𝑥 by 𝛼 𝑟 , where 𝛼 is a constant in 𝑟 . To mitigate inference latency or parameter increase, we incorporate the low-rank matrix into the pre-trained weights after every training stage.HiLoRA provides a new interaction for refining latent representation, preventing direct gradient propagation of vanilla LoRA from deep to shallow layers. Simultaneously, through its hierarchical mechanism, it can avoid the accumulation of perceptual errors in the fine-tuning process, enabling fine-grained cross-modal interaction. Finally, it is worth noting that HiLoRA represents a basic hierarchical adaptation paradigm that is task-agnostic.To ensure the features learned by the cross-modal hierarchical structure meet the fine-grained and regional properties, we design multiple constraints to facilitate the training of HiVG framework. Contrastive Learning Constraint. To enhance training stability, we employ image-text Contrastive Learning (CL) as a constraint for HiLoRA. CL can also be formed between the grounding expression and the images within a shuffled training batch when differences are adequate. We treat the grounding image-text pairs as positive and all other random pairs as negative. We minimize the sum of two losses, one for text-to-image matching:Table 1 : Comparison with latest SOTA methods on RefCOCO/+/g [41, 75] , ReferItGame [21] and Flickr30k Entities [46] for grounding task. * represents utilizing ImageNet [25] pre-training. † indicates that all of the RefCOCO/+/g training data has been used during pre-training. RN101, DN53, Swin-S, and ViT-B are shorthand for the ResNet101, DarkNet53, Swin-Transformer Small, and ViT Base, respectively. The latest CLIP-based SOTA methods are shaded in gray . We highlight the best performance of the base model in the red colors and bold the best results for the large model. EQUATIONand the other for image-to-text matching:EQUATION)where 𝑁 is the batch size, 𝒗 𝑖 and 𝒕 𝑗 are the normalized embeddings of image in 𝑖 𝑡ℎ pair and that of text in 𝑗 𝑡ℎ pair, respectively. 𝜏 is the temperature to scale the logits, and < •, • > denotes cosine similarity operation. Therefore, the constraint can be formulated as:EQUATIONRegion-Text Contrastive Constraint. Inspired by the image-level contrastive learning, we attempt to construct token-wise regiontext contrastive constraint using ground truth bounding box as a mask to simulate text-to-image matching. Specifically, we extract text aggregation features, i.e., the [𝐸𝑂𝑆] token 𝒕 𝑒𝑜𝑠 , from grounding encoder and compute the similarity 𝒔 𝑖 with each visual token 𝒗 𝑖 after applying normalization and an MLP projection:EQUATIONwhere 𝜎 denotes the sigmoid function. Tokens within the bounding box are considered as positive, while those outside are regarded as negative. Subsequently, we employed Focal loss [30] and Dice/F-1 loss [43] to constrain the aggregated similarity 𝒔 = (𝒔 1 , 𝒔 2 , ..., 𝒔 𝐿 𝑣 ) and the nearest downsampling box mask 𝒎 𝑑 ∈ R 1×𝐻 /𝑃 ×𝑊 /𝑃 :EQUATIONwhere 𝜆 𝑓 𝑜𝑐𝑎𝑙 and 𝜆 𝑑𝑖𝑐𝑒 are the coefficients to control the two loss functions, and 𝑃 is the patch size.Training Loss. The box regression loss is formulated by leveraging smooth L1 loss [13] and Giou loss [50] with coefficient 𝜆 𝑙 1 and 𝜆 𝑔𝑖𝑜𝑢 :EQUATIONwhere B donates the ground truth box. Finally, the overall training loss of the model is determined by the sum of the regression loss and the two framework constraints:EQUATION 
experiments 4.1 implementation details: 
 Datasets and Evaluation Metrics. The effectiveness of our method is validated on five widely utilized datasets, namely the three REC datasets (RefCOCO/+/g [41, 75] ), as well as two PG datasets (Refer-ItGame [21] and Flickr30k Entities [46] ). In PG, the query pertains to a specific phrase, while in REC, the query refers to a referring expression. The text of RefCOCO+/g exhibits greater length and complexity in comparison to that of RefCOCO. We follow the previous researches that employs Intersection-over-Union (IoU) as the evaluation metric. Specifically, a prediction is deemed accurate only when its IoU exceeds or equals 0.5. Finally, we compute the prediction accuracy for each dataset as a performance indicator. Experimental Setting. It is worth emphasizing that, as described in Sec. 2.1, our focus is on the transfer learning of self-supervised pre-trained models for grounding tasks. (1) We follow the basic finetuning setting with the same as CLIP-VG [63] and Dynamic-MDETR [53] , etc.. (2) In particular, we also compare with the traditional setting of fine-tuning with pre-trained detection models (e.g., TransVG [9] , TransVG++ [10] , etc.). (3) Additionally, we also follow the previous works that utilized a dataset-mixed pre-training setting (e.g., MDETR [20] , OFA [60] ) and mix the training data (only includes the RefCOCO/+/g, ReferIt, Flickr30k datasets) for intermediate pretraining. This allows us to compare our results with these works in a relatively fair manner. The details are presented in Tab. 1.RefCOCO/RefCOCO+/RefCOCOg/ReferIt/Flickr. As presented in Tab. 1, we compare our results on five widely used datasets with the latest SOTA works, including CLIP-VG [63] , Dynamic-MDETR [53] , TransVG++ [10] , grounding-DINO [33] and OFA [60] Analysis of Referring Expression Length. As shown in Fig. 4-(c ), we conducted a comparison of different expression lengths on the RefCOCOg dataset. It shows that HiVG exhibits superior comprehension for longer and more complex texts, while its performance remains stable as text length increases. Furthermore, compared to CLIP-VG, our method demonstrates significantly better results.Ablation Study of the Main Modules. We conducted the ablation study on RefCOCOg datasets. As presented in Tab. cross-modal bridge (MACB, default using 12 layers of text features). The weights in the table denotes the sample-agnostic weights. The table shows that our designed structure can effectively utilize multilevel text features and achieve hierarchical adaptation.Ablation Study of HiLoRA. As presented in Tab. 5 and Fig. 4-(d) , we conducted an ablation study on HiLoRA with different LoRA stages and various low ranks. It is observed that employing 3-stage HiLoRA with low rank as 32 achieves the best performance.We visually present the results of several relatively challenging examples in Fig. 5 . The attentions show the [REG] token over vision tokens from the last grounding block of each model. HiVG demonstrates exceptional semantic understanding capabilities in the complex sentences.CONCLUSIONIn this paper, we introduce a hierarchical multimodal fine-grained modulation framework, namely HiVG, which effectively implements fine-grained adaptation of the pre-trained model in the complex grounding task. It is a concise and efficient end-to-end framework that can simultaneously alleviate two kinds of task gaps, i.e., data bias and learning objectives, through a multi-layer adaptive cross-modal bridge and a hierarchical low-rank adaptation paradigm. Our exploration in hierarchical cross-modal features offer new insights for the future grounding research, which has been neglected in past works. 
 
 Appendix:  


Decision: This paper proposes a hierarchical multimodal fine-grained modulation framework, namely HiVG, for the challenging task of visual grounding.  HiVG consists of a multi-layer adaptive cross-modal bridge and a hierarchical multimodal low-rank adaptation paradigm. Extensive results on several popular datasets validate the effectiveness of the proposed method.

(+) On the positive side, the reviewers found the method to be interesting and effective and appreciate the good presentation and impressive results. 

(-) On the negative side, the reviewers still have some concerns on the instability, insufficient discussion, missing related work, and clarity of the writing.

Several technical questions and suggestions were raised by the reviewers. The authors have taken these into consideration. 
Overall, all reviewers agreed to accept the paper after the rebuttal. But most reviewers did not clearly and seriously justify their final ratings. Especially, most "Strengths" comments are short and simple. Besides, the "Confidence" of the Reviewer nq6h whose final rating is Accept is just "Somewhat Confident".

Therefore, the AC  leans towards recommending  Accept (Poster).

Reviews: review: paper_summary: paper_summary: This paper proposes a hierarchical fine-grained multimodal module network HiVG to bridge the data gap and training loss gap between multimodal pretraining and grounding tasks based on CLIP. HiVG includes multiple cross-modal bridges and hierarchical multimodal LoRA. The former can resolve the inconsistency between the visual features extracted by the pre-trained model and the visual features required for the grounding task, while establishing associations between multi-level visual and textual features. The latter prevents the accumulation of perceptual errors by adjusting cross-modal features from shallow to deep in a hierarchical manner. The experiments are conducted on five benchmark datasets, demonstrating the sota performance.
strength_weakness: strength_weakness: - The propsosed Hierarchical LoRA is new for progressivingly adapt the pre-trained model.  
- The experimental results have achieved state-of-the-art (SOTA), comparied with the latest baselines. 
- The ablation study is comprehensive.
limitations: limitations: - The placement of the "frozen" diagram in the light green CLIP Image Encoder part of Fig. 2 is somewhat confusing. I suggest that it should point to the dark green ViT Block part on the right. 
- Hi LoRA can be connected as HiLoRa or Hi-LoRa for better representation.  
- Several works are misssing, such as 
[1] M. Lu, R. Li, F. Feng, Z. Ma and X. Wang, "LGR-NET: Language Guided Reasoning Network for Referring Expression Comprehension," in IEEE Transactions on Circuits and Systems for Video Technology, 2024, doi: 10.1109/TCSVT.2024.3374786.
[2] P. Miao, W. Su, G. Wang, X. Li and L. Xi, "Self-Paced Multi-Grained Cross-Modal Interaction Modeling for Referring Expression Comprehension," in IEEE Transactions on Image Processing, vol. 33, pp. 1497-1507, 2024, doi: 10.1109/TIP.2023.3334099.
[3] J. Ke, J. Wang, J. -C. Chen, I. -H. Jhuo, C. -W. Lin and Y. -Y. Lin, "CLIPREC: Graph-Based Domain Adaptive Network for Zero-Shot Referring Expression Comprehension," in IEEE Transactions on Multimedia, vol. 26, pp. 2480-2492, 2024, doi: 10.1109/TMM.2023.3297312. 
- HiLoRa should be evaluted in other transfer tasks. 
- The cross-modal birdge is not novel.
paper_summary: paper_summary: In order to fully exploit the potential of a CLIP model for visual grounding tasks, this paper studies the problems of data bias and difference in learning objectives. To this end, the multi-level image and text representations are integrated from a hierarchical perspective with the facilitation of the developed multi-layer adaptive cross-modal bridges and the proposed hierarchical low-rank adaptation paradigm.
strength_weakness: strength_weakness: 1.  The paper is well written and easy to follow. The diagrams are beautiful and helpful for understanding.
2.  Extensive experiments, including both quantitative and qualitative analysis, indicate the effectiveness of the proposed HiVG method. Additionally, HiVG also shows its superiority on model parameter size, training/inference costs and model efficiency.
3.  This paper provides a strong baseline for Clip-based visual grounding research.
limitations: limitations: It is a good paper, the design of Hi LoRA is interesting, I just have two small questions:

1.	The proposed cross-modal bridge module learns sample-agnostic adaptive weights within different layers, which have the same length of text token. So how do these weights work on the input texts with different lengths? Is it reasonable for using the same weights to reweight the different tokens as in Eq. (3)?

2.	The authors claim the motivation of designing the Hi LoRA module is “Since high-level features depend on low-level features, and they are susceptible to perturbations of the shallow features. If all layers of a large-scale pre-trained model are adapted simultaneously, perceptual errors in bottom layers may accumulate and amplify”. Although Hi LoRA achieves higher performance than vanilla LoRA, I am little confused that the underlying mechanism of Hi LoRA to solve the above issue. Why using such hierarchical structure can deal with this “error accumulation” problem?
Nevertheless, I generally like this paper, so I vote for accepting this paper.
paper_summary: paper_summary: This paper addresses data biases and differences in learning objectives within the visual grounding task, proposing a multi-layer adaptive cross-modal bridge and a hierarchical multimodal low-order adaptation (Hi LoRA) paradigm. This paradigm adjusts cross-modal features from shallow to deep layers, establishing connections between multi-level visual and textual features. Extensive experiments have demonstrated the effectiveness of HiVG.
strength_weakness: strength_weakness: 1.The paper makes non-trivial advances over the current SOTA methods, the proposed Hi lora prevents  from shallow to deep layers. 
2.This paper is well written, easy to follow.
3.The conducted experiments show that the designed HiVG outperforms related visual grounding methods.
limitations: limitations: 1.I wonder whether HiVG is compatible with other multi-modal downstream tasks.
2.Some symbols mentioned in Section 3 do not appear in Figure 2, making it difficult to read.
3.The multi-layer adaptive cross-modal bridge facilitates interaction between textual and visual information. In the ablation studies provided by the author, only choices of related hyper-parameters are discussed, with no visual validations of how text aids in visual calibration.
4.In the experimental section, the author does not validate how Hi LoRA reduces perceptual errors.
paper_summary: paper_summary: This paper studies the multimodal pre-training for visual grounding. To tackle the task gap between pre-training and grounding, this paper proposes a hierarchical multimodal fine-grained modulation framework. Concretely, a multi-layer adaptive cross-modal bridge and a hierarchical multimodal low-rank adaption paradigm are proposed to tackle the inconsistency between visual features and text-relevant regions,  and establish the connection between multi-level visual and text features, respectively. Extensive experiments conducted on the benchmark datasets exhibit the effectiveness of the proposed method.
strength_weakness: strength_weakness: - The proposed hierarchical multimodal low-rank adaption paradigm is innovative.
- This paper is well organized and easy to read.
- The proposed method exhibits superior performance in visual grounding.
limitations: limitations: This paper provides extensive experiments to validate the effectiveness of the proposed model. I am interested in the ablation study results of training loss in Table 4 in the supplementary material. The results indicate some instability in training when the CLC loss is omitted. Could the authors provide a more detailed discussion of the potential causes of this instability?