Enhancing Fine-grained Multi-modal Alignment via Adapters: A Parameter-Efficient Training Framework for Referring Image Segmentation

Authors: zunnan  xu
jiaqi  huang
ting  liu
yong  liu
kehong  yuan
xiu  li
Venue: ICML_cc_2024_Workshop_WANT
Published: 2024-05-25 14:39:15

Abstract: In the domain of computer vision, Parameter-Efficient Training (PET) is increasingly replacing the traditional paradigm of pre-training followed by full fine-tuning. PET is particularly favored for its effectiveness in large scale models, as it streamlines transfer learning costs and optimizes hardware utilization. However, the prevailing PET methods are primarily designed for single-modal optimization without fine-grained feature extraction design. When applied to multi-modal dense prediction tasks, these methods typically do not match the performance of full fine-tuning methods that utilize more resources. In this paper, we do an investigation of efficient training problems on referring image segmentation. We introduce DenseCrossAdapter, a parameter-efficient module designed to enhance low-rank visual feature propagation by establishing dense interconnections between each layer and all preceding layers. This facilitates robust cross-modal feature interaction. We also suggest using text adapters to improve textual features. Our approach greatly surpasses state-of-the-art methods with only 0.9% to 1.8% backbone parameter updates, evaluated on challenging benchmarks.

Main paper: 
 introduction: 
 In the realm of computer vision, Parameter-Efficient Training (PET) is rapidly emerging as a superior alternative to the traditional approach of pre-training followed by full finetuning. PET is particularly advantageous for large-scale models because it significantly reduces the costs associ-Accepted to the Workshop on Advancing Neural Network Training at International Conference on Machine Learning (WANT@ICML 2024). ated with transfer learning and optimizes the utilization of computational resources.Despite their benefits, current PET methods are often limited to single-modal optimization and lack a fine-grained feature extraction design. This limitation becomes apparent when these methods are applied to multi-modal dense prediction tasks. They typically fail to match the performance of full fine-tuning methods that leverage more extensive resources (Liu et al., 2024c) . One such challenging task is Referring Image Segmentation (RIS), which involves predicting the mask of a target object within an image based on a natural language description. Unlike semantic segmentation, which assigns predefined labels to each pixel, RIS requires a more sophisticated understanding of both language and visual content to accurately identify the described object. The significance of RIS lies in its ability to bridge the gap between natural language descriptions and fine-grained visual perception (Liu et al., 2024b; Ji et al., 2024) . This capability is pivotal for the advancement of artificial intelligence, especially in autonomous systems, image-based retrieval, and human-computer interaction. The complexity of RIS stems from the need to interpret variable context lengths and to understand an open-world vocabulary encompassing a diverse range of object names, attributes, and positional references (Li et al., 2024b; Wu* et al., 2024) . The requirement for precise segmentation of referring objects elevates this dense prediction task to one of the most formidable challenges of vision-language understanding. For these challenging vision-language tasks, a prevalent trend is to scale up foundational models (Radford et al., 2021; Li et al., 2022b; Oquab et al., 2023; Fang et al., 2023) . These models leverage large datasets to learn a comprehensive set of visual features. In return, the scale-up process not only enhances their ability to discern subtleties in visual data but also significantly boosts their generalization capabilities, demonstrating a robustness that is essential for real-world applications. However, there often exists a gap between the pre-trained tasks of these models and the specific requirements of downstream applications. Bridging this gap through efficient adaptation presents a formidable challenge. Recent studies (Wang et al., 2022; Ding et al., 2022; Yang et al., 2022; Liu et al., 2023a) have demonstrated the effectiveness of fine-tuning powerful pre-trained models for referring image segmentation. However, a common challenge is that they typically require full fine-tuning to adapt to dense prediction tasks. This process can lead to the loss of valuable pre-training knowledge, as it involves adjusting a large number of parameters that were previously optimized during the pre-training phase (Kim et al., 2022; Liu et al., 2023a; b) . Moreover, these approaches maintain a distinct set of fine-tuned parameters of pre-trained models for each dataset, which can lead to substantial deployment costs. The problem becomes particularly serious when considering the ever-growing size of pre-trained models, which now include parameters ranging from hundreds of millions to trillions (Li et al., 2022a; Zhou et al., 2022; Chen et al., 2022b; Sun et al., 2023) .Considering the above problems, we extend the discussion of ETRIS (Xu et al., 2023) : Is it possible for a model, with its pre-trained backbone parameters fixed, to surpass the performance of existing full fine-tuned methods? Various parameter-efficient training methods have been developed to achieve an optimal equilibrium between operational efficiency and model performance (Gao et al., 2021; Chen et al., 2022a; Zhou et al., 2022; Wang et al., 2023; Li et al., 2023; Liu et al., 2024b) . However, despite these contributions, most existing methods are limited in their scope, predominantly applied to single-modality tasks (Guo et al., 2020; Houlsby et al., 2019; Chen et al., 2022c) or simple classification problems (Gao et al., 2021; Chen et al., 2022a; Zhou et al., 2022) . There remains a notable gap concerning dense prediction tasks and the nuanced interactions between multiple modalities. Pioneering works like ETRIS and BarleRIa (Wang et al., 2023) aimed to parameterefficient fine-tuning CLIP (Radford et al., 2021) on referring image segmentation, but they faced several limitations: (i) These methods primarily relied on the early-stage fusion of multimodal features from the backbone, missing out on the benefits of more comprehensive global features, leading to suboptimal results. (ii) Furthermore, existing parameterefficient modules, such as Bridger (Xu et al., 2023) and GST (Wang et al., 2023) , are constrained by their limited application of multi-scale modeling. Their approach, which focuses on adjusting channel dimensions and using multihead attention mechanisms, is insufficient for capturing the full complexity of visual data across different scales.Our method addresses this question by introducing a novel approach that enhances the effectiveness of adapting pretrained vision-language models. In detail, we propose an adapter named DenseCrossAdapter, which can be seamlessly integrated into the pre-trained model for dense prediction tasks. There are two tailored modules for Dense-CrossAdapter: (i) a densely connected prior module for capturing the local multi-scale semantics feature maps of the intermediate layer and (ii) a cross-modal attention mod-ule that enables information exchange between visual and textual features. Secondly, we propose incorporating text adapters to enhance the text encoder. We further leverage these enhanced features to improve alignment between visual and linguistic features.Our framework is constructed around a dual-encoder architecture. Unlike previous methods, we have selected DINO (Oquab et al., 2023) to act as our visual encoder. The reason we chose DINO as our vision backbone is based on several insights: (i) DINO's self-supervised learning approach provides robust generalization and is more advantageous for dense prediction tasks compared to CLIP (Radford et al., 2021) . (ii) The lack of multimodal pre-training in DINO, notably in visual-text alignment, poses challenges for its direct application on referring image segmentation. This gap highlights the essential role of our proposed module in enhancing the model's capabilities, particularly in improving fine-grained vision language alignment. Our main contributions are as follows:• We introduce the pre-trained model DINO in RIS tasks and provide an effective training strategy for fine-grained alignment that avoids the need for intricate design. • We propose a novel DenseCrossAdapter that can be seamlessly integrated into the pre-trained backbone to enhance and interact with its intermediate features. This integration enhances DINO's alignment with language and improves its performance on dense prediction tasks. • Experiments demonstrate that our method greatly surpasses state-of-the-art full fine-tuned methods in referring image segmentation, with only 0.9% to 1.8% backbone parameter updates. 
related work: 
 Parameter Efficient Training (PET) aims to streamline the process of adapting pre-trained models to new tasks with minimal parameter adjustments, making it a practical solution for deploying large models to individual users, particularly in the face of expanding model sizes. Previous PET methods can be mainly divided into three types: (i) updating newly added parameters to the model or input (Houlsby et al., 2019; Li & Liang, 2021; Zhou et al., 2022; Li et al., 2024a; Liu et al., 2024a) ; (ii) sparsely updating a small number of parameters of the model (Guo et al., 2020; Zaken et al., 2021) ; (iii) low-rank factorization for the weights to be updated (Hu et al., 2021; Karimi Mahabadi et al., 2021; Hao et al., 2023) . However, previous works applying PET in computer vision mainly focus on classification and generation tasks. How to efficiently update and transfer the pre-trained knowledge space to dense prediction tasks remains a great challenge. Pioneering works like ETRIS (Xu et al., 2023) and BarleRIa (Wang et al., 2023) for referring image segmentation. However, their proposed modules like Bridger (Xu et al., 2023) and GST (Wang et al., 2023) are insufficient for capturing the complexity of multi-scale visual features.Referring Image Segmentation (RIS) aims to segment the target objects referred to by natural language descriptions. It necessitates the models to comprehensively associate diverse visual content and linguistic signals. The genesis of this field can be traced to CNN-LSTM-based methods, such as the Referring Relation Network (RRN) (Li et al., 2018) and the Recurrent Multimodal Interaction (RMI) (Liu et al., 2017) . These early methods used CNN and LSTM networks to separately extract visual and linguistic features, which were then combined to form cross-modal representations for segmentation by an FCN. The advent of the Transformer model has catalyzed a paradigm transformation of integrating features across diverse modalities by attention mechanism (Yang et al., 2022; Liu et al., 2023a; Yan et al., 2023; Liu et al., 2023b) . Among them, MDETR (Kamath et al., 2021) and VLT (Ding et al., 2022) have demonstrated remarkable performance across various Vision-Language (VL) tasks by integrating multi-modal attention interaction and query representation. Capitalizing on the robust image-text alignment capabilities of CLIP, CRIS (Wang et al., 2022) , ETRIS (Xu et al., 2023) and UniLSeg (Liu et al., 2023c) zeroes in on sentence-pixel alignment to harness the wealth of multi-modal correspondences. However, existing methods primarily concentrate on the design of visual-linguistic interactions during the decoding phase, while overlooking the potential to fully excavate the pretrained backbone networks. To this end, we propose DCRIS, a framework that aligns features from different modalities with the assistance of parameter-efficient modules boosting multi-scale comprehensive updating of backbone networks. Compared to existing fully fine-tuning methods, the proposed approach achieves competitive performance while greatly reducing the training overhead. 
framework overview: 
 The overall framework of the proposed model is illustrated in Figure 1 . Our approach freezes the parameters of the pretrained backbone, ensuring parameter efficiency. The core of the model's design philosophy is the DenseCrossAdapter, which is intended to facilitate interaction between crossmodal features and inject dense prediction priors into the pre-trained backbone. This is coupled with the incorporation of text adapters in the text encoder to enhance fine-grained image text alignments.Visual encoder. Our work adapts the distilled DINOv2 with registers (Oquab et al., 2023) as the backbone. This model has been well-pretrained on self-supervised training tasks and is based on ViT-B/14. Specifically, for an input image I ∈ R H×W ×3 , we utilize DINOv2 enhanced with densecross adapters to extract image features. We select the outputs of several middle layers and the last layer as hierarchical vision features f i υ , i ∈ [1, 2, 3], which are used for subsequent dense prediction modules.Text encoder. For the input refer expression T , we utilize the pre-trained text transformer of CLIP (Radford et al., 2021) for extracting text features. Text features f t and sentence-level feature f s are extracted using a CLIP text encoder, augmented with text adapters to guide referring image segmentation. Considering the substantial number of parameters the encoder occupies, and to avoid the loss of valuable pre-training knowledge, we freeze all the encoder parameters during our fine-tuning method to apply it efficiently in downstream tasks.As mentioned in Section 1, although DINOv2 has strong generalization capabilities and is more advantageous than CLIP in tasks that rely more on visual abilities, DINOv2 lacks visual-text alignment in the RIS downstream task due to the absence of multimodal pre-training. Meanwhile, the self-supervising tasks of DINOv2 are limited to patch-level, lacking pixel-level information in dense prediction tasks. To address this and enhance the model's multi-scale modeling capability, we designed and utilized densecross adapters to augment the model's vision backbone. The visual backbone network remains fixed, with training solely focused on the DenseCross adapter parameters.DenseCrossAdapter. As shown in Figure 1 , the proposed DenseCross Adapter mainly differs from previous adapter designs by integrating a densely connected prior module. A cross-modal attention module is included between the activation and up-projection layers. This integration enhances the model's capability to extract dense image features and enriches its multimodal perception abilities. In the densely connected module, a multi-branch convolutional structure is proposed to achieve multi-scale modeling of low-rank visual features and acquire multi-scale prior information. After applying linear projection and non-linear activation, 1×1, 3×3, and 5×5 convolutional kernels are used to progressively integrate outputs from the previous layer. To ensure the module's lightweight and efficiency, 1×1 convolutions are also used before the 3×3 and 5×5 convolutions to reduce channel dimensions. The output of the 1×1 convolution serves as the input for the 3×3 convolution, and the combined outputs of the 1×1 and 3×3 convolutions are used as the input for the 5×5 convolution. Unlike the typical approach of directly merging multi-scale features, this stepwise integration method intricately combines fine details with broader contextual information. To maintain the integrity of the original features, the initial features are added to the final concatenated features.Denote σ as a non-linear activation function (ReLU), Linear down as a downsampling operation of linear projection. Specifically, for given input image features f l v at layer l, this process can be formulated as below:EQUATIONwhere F l v1 , F l v2 , and F l v3 are the output features after applying respective convolutional operations, and F l dense is the final dense feature representation obtained by concatenating the features from all branches and adding the initial featureF l v .Here, (, ) represents concatenating the features along the dimensional direction.Considering that textual information contains valuable references, we utilize it as a global reference prior by integrating it into the vision backbone network via a multi-head crossattention mechanism. This not only regularizes the visual features but also aligns them better with the extracted text features (denoted as f t ). This process can be formalized as:EQUATIONwhere F MHCA denotes multi-head cross attention, F l cross denotes the fused visual feature, and Linear up represents an operation to project the visual features back to get f l dc . We add the DenseCrossAdapter in parallel to the MLP layer in the transformer block as illustrated in Figure1.Text adapter. Acknowledging the disparities between the text features extracted by the CLIP text encoder and the DINO features, we incorporate a text adapter to improve the text encoder for fine-grained alignment of linguistic and visual features. In contrast to the DenseCross adapter, the text adapter adopts a more straightforward design, focusing on effectively processing and integrating text features without complex structures. This minimalist approach ensures efficient handling of textual information while maintaining compatibility with the overall model architecture. By utilizing these enhanced features, we can further improve the alignment between visual and language features. Specifically, for given input text features f l t at layer l, the text adapter employs a standard "Down-ReLU-Up" structure to refine and project linguistic features. This process can be formalized as:EQUATIONwhere Linear down represents a downsampling linear projection and Linear up denotes an upsampling operation to adapt text features back to the original dimension.Following CRIS (Wang et al., 2022) and ETRIS (Xu et al., 2023) , we incorporate a learnable referring image segmentation head, which consists of three main components: a crossmodal neck, a vision-language decoder, and an up-sample projector. These components work together to extract the cross-modal feature F c and the textual feature F l .The cross-modal neck takes multiple adapted visual features ( f i υ , i ∈ [1, 2, 3]) from three layers of the visual encoder (e.g., the 1/3, 2/3, and the last layer of the backbone) and the adapted textual embeddings ft . Specifically, we employ a multi-head cross-attention mechanism (F MHCA ) with convolution to fuse these features, obtaining the fusion features F f . Subsequently, we concatenate a 2D spatial coordinate feature F coord with F f and further fuse them using a 3×3 convolution, which can be formalized as:EQUATIONwhereF f = F MHCA ( f i v , ft) and f c denotes the combined cross-modal feature.The vision-language decoder further merges the composite feature f c with the textual embeddings ft . This fusion process culminates in the generation of multimodal features F mm , encapsulating both visual and linguistic information. Specifically, the decoder consists of three layers, each composed of a multi-head self-attention layer (MHSA), a multihead cross-attention layer (MHCA), and a feed-forward network. Within each decoder layer, the combined features f c are fed into the MHSA layer to capture global contextual information. The MHCA layer further facilitates multi-modal interaction by mapping visual features to queries and textual features to keys and values. Following the MHCA layer, an MLP block, along with layer normalization and residual connections, further processes the output features.An up-sampling projector further transforms the multimodal features F mm and the sentence-level feature f s to extract the cross-modal feature F c and the transformed textual feature F l . f s is first transformed into F l through a linear transformation, then split and reshaped into weights and bias, enabling it to function as a Conv2D layer. This Conv2D layer is used to transform the cross-modal representation into the final mask prediction. The overall transformation is achieved using a 4× upsampling followed by convolution and linear projection:F c = Conv(UpSample(F mm )), F l = Linear(f s ).(5)For the training objective of our model, we utilize a textto-visual contrastive loss, denoted as L con , to optimize the alignment between the text-derived features and their corresponding visual pixels. This contrastive loss is designed to both enhance the connection between text features and corresponding visual pixels, and separate these text features from any unrelated visual elements. The text-to-pixel contrastive loss is calculated as below:EQUATIONwhere P and N denote the class of 1 and 0 in the ground truth, and σ denotes the sigmoid function. The loss thus penalizes incorrect alignments between features and encourages the model to correctly match textual descriptions to their associated visual representations. 
datasets: 
 We employ three challenging referring image segmentation benchmarks in our experiments:• RefCOCO (Kazemzadeh et al., 2014) is widely used as a benchmark for referring image segmentation. It comprises 19,994 images annotated with 142,210 referring expressions for 50,000 objects, which have been sourced from the MSCOCO dataset through a two-player game. The dataset is divided into four subsets, consisting of 120,624 training samples, 10,834 validation samples, 5,657 samples for testA, and 5,095 samples for testB, respectively. The average length of the expressions is 3.6 words, and each image contains a minimum of two objects. • RefCOCO+ (Kazemzadeh et al., 2014) The expressions in G-Ref were collected from Amazon Mechanical Turk and had an average length of 8.4 words, which included more words related to locations and appearances. We present results for both the Google and UMD partitioning methods for G-Ref.In our experiments, we use the DINOv2-B/14 as the vision backbone for DCRIS-B, and DINOv2-L/14 as the vision backbone for DCRIS-L. Both models employ the text en- (Radford et al., 2021) as the textual encoder, with input images resized to 448x448 pixels. The DenseCross Adapter with a dimension of 128 is applied at layers [1, 3, 5, 7, 9, 11] in DCRIS-B and at layers [2, 6, 10, 14, 18, 22] in DCRIS-L. The text adapter with a dimension of 64 is applied at layers [1, 3, 5, 7, 9, 11] in both models. We train the entire framework for 50 epochs using the Adam optimizer with an initial learning rate of 0.0001. The learning rate decays by a factor of 0.1 at epochs 35. Specifically, DCRIS-B is trained on 2 A100 GPUs with a batch size of 32, while DCRIS-L is trained on 4 A100 GPUs with a batch size of 64 and an initial learning rate of 0.0002. Following previous methods (Yang et al., 2022; Tang et al., 2023) , we adopt IoU as the metric to evaluate performance, which calculates the intersection regions over the union regions of the predicted segmentation mask and the ground truth.We conducted a comprehensive comparison between our proposed DCRIS models and a series of previous referring image segmentation (RIS) methods. As demonstrated in Table 1, our approach significantly outperforms state-of-the-art RIS methods on three commonly challenging benchmarks. DCRIS-B achieves an average IoU score of 70.8, while DCRIS-L achieves an average IoU score of 72.8, representing improvements of 3.7% and 6.6% compared to the previous state-of-the-art method. Notably, our DCRIS-L model achieves the highest scores across all evaluation tasks, with strong performance on the RefCOCO+ and G-Ref datasets, which present greater challenges compared to RefCOCO.In addition to comparing against full fine-tuning methods, we also evaluated our models in the context of parameterefficient training methods. Table 1 shows that DCRIS-B (Houlsby et al., 2019) 71.2 73.3 68.3 70.9 1.98M Compacter (Karimi Mahabadi et al., 2021) 73.9 75.8 70.8 73.5 1.62M LoRA (Hu et al., 2021) 73.4 75.7 70.2 73.1 1.57M ETRIS (Xu et al., 2023) 74.5 76.5 72.9 74.6 1.38M In contrast, our method does not have this limitation and can effectively enhance model performance. This demonstrates the robustness and efficiency of our approach in enhancing performance with a modest increase in parameters.Effect of Adapter's layers and dimensions.To determine the optimal configuration of the DenseCross Adapter and Text Adapter, we conducted an ablation study varying both the number of layers and the dimensions of the adapters. Firstly, we evaluated the impact of different numbers of adapter layers. In this experiment, the visual DenseCross Adapter and the textual Text Adapter were inserted at the same layers. The results, shown in performance generally improved as the number of layers increased from 1 to 6, with the best results achieved at 6 layers. Beyond 6 layers, performance gains plateaued and even slightly decreased, suggesting that additional layers did not contribute significantly to further improvement and, in some cases, led to overfitting. Effect of DenseCross Adapter and Text Adapter. We assessed the impact of the DenseCross Adapter (DA) and Text Adapter (TA) by performing an ablation study and reporting the results on validation and test datasets. From Table 3 , it is evident that the baseline model without either adapter achieves the lowest average performance, with an average score of 74.7%. Introducing only the DenseCross Adapter yields a slight improvement, increasing the average performance to 75.0%. This indicates that the DenseCross Adapter enhances visual feature extraction. Applying only the Text Adapter results in a more significant improvement, with the average performance rising to 75.6%, demonstrating the Text Adapter's critical role in processing and integrating textual information. The combined use of both adapters produces the highest performance across all datasets, with an average score of 76.0%. This synergistic effect under-scores the importance of integrating both visual and textual adaptations to enhance the model's capability in cross-modal feature learning and segmentation performance. The notable improvement, particularly on the testB dataset, highlights the robustness and generalization ability provided by the combined approach.Effect of varying the dimensions of the adapters. We investigated the effect of varying the dimensions of the adapters. The results of this study are illustrated in Figure 2 . In this experiment, the adapters were fixed at the optimal layers identified in the previous experiment [1, 3, 5, 7, 9, 11] while varying their dimensions. Configurations tested included combinations such as 64,64; 128,64; 128,128; and 256,128 . The results, presented in Table 4 , showed that the combination of 128,64, which corresponds to the chosen parameter setup for DCRIS-B, proved to be the most effective. Although increasing the dimensions to 128,128 or 256,128 introduced more parameters, the performance improvements were marginal and did not justify the additional computational cost. 
conclusion: 
 In this work, we propose a novel parameter-efficient training (PET) method for multi-modal dense prediction tasks, especially in referring image segmentation. Specifically, we adapt the pre-trained DINO model for referring image segmentation (RIS) by introducing a simple yet effective fine-tuning strategy. Our key innovation is the Dense-CrossAdapter, which can be seamlessly integrated into the visual backbone to improve fine-grained visual-text alignment. We also propose using text adapters to enhance the language encoder's capabilities. Our streamlined approach not only surpasses the performance of fully fine-tuned models but also does so more efficiently in terms of scalability and parameter management. 
 
 Appendix:  


Decision: The manuscript introduces a simple yet effective Parameter-Efficient Training method for multi-modal models, namely the multi-modal dense prediction tasks. All reviewers appreciate the impressive results.