BarLeRIa: An Efficient Tuning Framework for Referring Image Segmentation
Authors: Yaoming Wang
Jin Li
XIAOPENG ZHANG
Bowen Shi
Chenglin Li
Wenrui Dai
Hongkai Xiong
Qi Tian
Venue: ICLR 2024 spotlight (2024)
Keywords: referring image segmentation; parameter efficient tuning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Pre-training followed by full fine-tuning has gradually been substituted by Parameter-Efficient Tuning (PET) in the field of computer vision. PET has gained popularity, especially in the context of large-scale models, due to its ability to reduce transfer learning costs and conserve hardware resources. However, existing PET approaches primarily focus on recognition tasks and typically support uni-modal optimization, while neglecting dense prediction tasks and vision language interactions. To address this limitation, we propose a novel PET framework called **B**i-direction**a**l Inte**r**twined Vision **L**anguage Effici**e**nt Tuning for **R**eferring **I**mage Segment**a**tion (**BarLeRIa**), which leverages bi-directional intertwined vision language adapters to fully exploit the frozen pre-trained models' potential in cross-modal dense prediction tasks. In BarLeRIa, two different tuning modules are employed for efficient attention, one for global, and the other for local, along with an intertwined vision language tuning module for efficient modal fusion.
Extensive experiments conducted on RIS benchmarks demonstrate the superiority of BarLeRIa over prior PET methods with a significant margin, i.e., achieving an average improvement of 5.6\%. Remarkably, without requiring additional training datasets, BarLeRIa even surpasses SOTA full fine-tuning approaches. The code is available at https://github.com/NastrondAd/BarLeRIa.

Published as a conference paper at ICLR 2024
BARLERIA: AN EFFICIENT TUNING FRAMEWORK FOR REFERRING IMAGE SEGMENTATION
Yaoming Wangh Jin Li
Xiaopeng Zhang 2 Qi Tian
Bowen Shi! Chenglin Li'
Wenrui Dai l Hongkai Xiong'
Shanghai Jiao Tong University Huawei Cloud
[wang-yaoming deserve.di, 1c11985, daiwenrui, xi sienghcagkai/es edu. exphistory@gmaii tian. fil 1@huawei com
ABSTRACT
Pre-training followed by full fine-tuning has gradually been substituted by Parameter-Efffcient Tuning (PET) the field of computer vision. PET has gained popularity, especially in the context of large-scale models, due to its ability to reduce transfer learning costs and conserve hardware resources. However, exist- ing PET approaches primarily focus on recognition tasks and typically support uni-modal optimization, while neglecting dense prediction tasks and vision lan- guage interactions. To address this limitation, we propose novel PET framework called Bi-directional Intertwined Vision Language Efficient Tuning for Referring Image Segmentation (BarLeRla), which leverages bi-directional intertwined vi- sion language adapters to fully exploit the frozen pre-trained models' potential cross moos- dense prediction tasks. In BarLeRla, two different tuning modules are employed for efficient attention, one for global, and the other for local, along with an intertwined vision language tuning module for efficient modal fusion. Ex- tensive experiments conducted on RIS benchmarks demonstrate the superiority of BarLeRla over prior PET methods with significant margin, i.e., achieving an average improvement of 5.6%. Remarkably, without requiring additional training datasets, BarLeRla even surpasses SOTA full -uniing approaches. The code available at ht /ttpth///g/ttt/tgg www.hhuw..oomm/mee Not/Nast rondad. wtwronnndddd/a/aaaaaaaaaaaaaaaee BarLeRIa.
INTRODUCTION
In recent years, large-scale models have made significant contributions to advancements in NLP and CV, However, the cost associated with full fine-tuning of large models has become prohibitively expensive. To address this challenge. Parameter-Efficice as a prevalent paradigm (Houlsby et al., 2019 Tie & Deng 2022 Jia et al. 2022; Wang et al., 2023).
Tuning (PET) approaches have emerged
By freezing majority of the pre-trained model and fine-tuning only small subset of parameters, PET approaches offer high efficiency while maintaining performance comparable to full tine-tuning, and are increasingly favored for language dialogue (Karimi Mahabadi al. 2021| Sung al. 2021) as well as visual recognition tasks (Chen et al.) 20226 Jia et [ Despite these advancements, limited research has explored the effectiveness of PET pipelines for adapting dense prediction tasks (Ding et al. 2022 Qian et al. 22033 or facilitating cross-modal fusion.
This paper investigates the generalization ability of Parameter-Efficien Tuning (PET) and examines its affordability for challenging model dense prediction task Referring Image Segmenta- tion (RIS). RIS fundamental segmentation task designed to segment target objects from input images based on given text descriptions (Hu et al., 2016). Different from vanilla segmentation tasks, RIS needs to extract not only spatial and semantic information from images, but also key semantics from textual descriptions, and merge them in order to get the correct segmentation re- sults. Previous studies have approached this task by either concatenating textual embeddings with visual features and incorporating vision-language attention mechanisms to facilitate interactions Yuu
Corresponding author: Xiaopeng Zhang; Wenrui Dai. T: Equal contribution. This work was done when Yaoming Wang interned Huawei Inc.

---

Published as a conference paper at ICLR 2024
Langagee: "girl standing in pink and purple shirt wearing braocleis"
Language: "fourth omang on lowwest row"
62.63
JoU: 71.88
JoU: 31.88
IoU: 61.74
Calie
Lnggage:: gray shir""
Language: "fourth lime"
Iol: 46.32
IoU: 73.89
IoU: 0.00
loU: 82.97
Image
ETRIS
(c) Barl
(d) GT
(a) Image
(b) ETRIS
(c) BarLeRRaa
Figure I: Comparison between BarLeRIa and state-of-the-art PET RIS method ETRIS. We perform experiments using two different referring expressions: detailed or abstracted. In the first row, the expression is detailed and two methods can locate the object given sufficient knowledge, though BarLeRIa outperforms ETRIS a lot. In the second row, only brief expressions are provided, ETRIS locates wrong contours while BarLeRla still segments the target objects well. Best viewed in color.
2018 Li al. 2018 Chen et al., 2019), or by pursuing vision- Lanonnl alignment using uni- al.
modal pre-trained models supplemented with additional training (Liu et al. 2023 Yan et al. 2023)
More recently, leveraging the advancements in vision-language pre-training (Radford et al.
2021
Wang et al. (2022) propose to transfers multi-modal knowledge from CLIP through text-to- -ixeeel contrastive learning, leading to remarkable performance gains. However, these approaches rely on computationally extensive full fine-tuning, which raises concerns about scalability and affordability.
Few works explore integrating PET into RIS task, pioneering work Xu et al. (2023), intro- duces a vision-language bridge that combines vision inductive biases and language information, and achieves comparable performance to full fine-tuning However, this approach primarily focuses on aligmment between the vision and language modalities, while overlooking the core aspect of PET, namely, adapting the biased feature from pre-trained models (Tia et al.) 2022] Wang et al. (202)). Be- sides, local modal fusion is adopted in the proposed bridge network as well as the pre-trained vision language models and the segmentation head (Wang et al., 2022). Consequently, all components of the model repetitively fuse local visual features with textual embeddings without incorporating global prior from the text input to regularize the visual features, which lcading to off-target visual information interference and sub-optimal performance.
Considering the above two issues when incorporating PET into RIS task, Firstly, we propose a novel technique to address the feature adaption problem. The highlight is an intertwined vision language efficient tuning framework for better modal fusion along with feature adaption as a basic design. For both visual and textual branches, we fuse the visual and textual input in front of each frozen layer and adapt each layer's 's shortcut feature distribution via normalizing flow (Wang et al.) 2023). In this way, we keep the backbone frozen, employ modal fusion via the original self-attention mechanism, and are able to adapt the biased features for segmentation tasks.
Second, in order address the global regularization issue, we extract global prior from the text input to regulate the vision features. This regularization is achieved with a limited number of pa- rameters an end-to-end manner. Our proposed method consists of bi-directional efficient tuning framework, which comprises a global prior module and global tuning network. The global prior module leverages the cosine similarity between visual features and textual embeddings to enforce regularization. Moreover, to ensure that the global prior regularization does not conflict with the local intertwined vision language tuning, we introduce global shortcut tuning modules that are de- tached from the pre-trained backbone. By doing so, we establish parallel shorteut tuning network alongside the backbone. Similarly, we extend the intertwined vision language tuning to the shortcut tuning network to facilitate better fusion of the models.
Incorporating the proposed intertwined vision language efficient tuning and the bi-directional effi- cient tuning modules, produce novel PET framework, namely Bi-directional Intertwined Vi-
2

---

Published as a conference paper at ICLR 2024
sion Language Efficient Tuning for Referring Image Segmentation (BarLeRIa), to fully exploit the potential of frozen pre-trained vision-language models. BarLeRla exhibits remarkable performance improvement with only 0.4% to 2.5% tuning parameters compared with the backbone when utilizing CLIP VīT-B as the pre-trained model. Compared to the state-of-the-art PET approach ETRIS (Xu et al.] 2023), BarLeRIaa shows significant improvement, e.g., +2.01 IoU RefCOCO, +5.19 loU on RefCOCO+, and +3.74 IoU on G-Ref, respectively. Compared with the fully fine-tuned large visual language model LISA-7B (Lai al., 2023), BarLeRIa achieves comparable performance with only about 2M learnable backbone parameters and significantly outperforms the untuned 7B model. Besides, BarLeRla also outperforms full fine- time-tunigg state-of-ther of-tt--o--e--rtt approaches, egg. Poly- Former (Liu et al., (023) and UNINEXT (Yan et al. 2023), which need pre-training on extra region- level datasets. As comparison, without extra pre-training, BarLeRIa achieves a new SOTA perfor- mance when adopting EVA-CIIIP (Radford et al. (02T) as the pre-trained vision language model.
In nutshell, our contributions can be summarized as follows:
We find that previous PET methods for RIS task focus on modal fusion and ignore feature dis- tribution adaptation and propose novel intertwined vision language efficient tuning algorithm for both feature adaptation and modal fusion with only 0.4M (ViT-B) learnable parameters.
We reveal that repeating fusing the local visual features with the textual embeddings is another problem for previous approaches and propose a bi-directional efficient tuning framework that enables both local feature fusion and global prior regularization.
We design a novel global shortcut tuning module that tunes only 1.8M (ViT-B) parameters and learns the global prior regularization in parallel with the backbone avoid conflicts with our proposed local intertwined vision language efficient tuning.
2
METHODOLOGY
2.1 PRELIMINARIES
Adapting Shortcut with Normalizing Flow (SNF) (Wang et al. (023) adjusts the shortcut to adapt pre-trained models into downstream tasks. For given skip connection inside the transformer, it can be depicted as y = x x+-f fz) where æ is the input feature, f is a certain architecture of the transformer and y is the output. During fine-tuning, SNF only operates on the shortcut while keeping other parts frozen, i.e., y = s (x) + f(x). For given feature x E R. xd the transformation imposed by SNF is given by:
s(x) = x + X. 1((7" x + 3)
(1)
where X, y, 3 E R the Hadamard product and h(.)) is a smooth non-parameteric non-linearity. Note that SNF allows for multiple concatenated transformations, i.e., y s(s(---(x))) a(x))) + f(x)). The number of transformations denoted as the depth of SNF.
2.2 FRAMEWORK OVERVIEW
The framework of BarLeRIa is depicted Fig. The fundamental design of BarLeRIa is the proposed intertwined vision language efficient tuning algorithm, which is used to enhance modal fusion. Along with it, we employ a bi-directional efficient tuning framework that simultaneously adjusts local features and extracts global priors from the text input, thereby regularizing the visual features. This framework consists of two distinct efficient tuning modules. The first module, known as the local intertwined module, utilizes the intertwined vision language efficient tuning approach to enable efficient modal fusion and multi-modal feature adaptation. The second module, referred to as the global shorteut tuning module, incorporates parallel shorteut module and leverages the global prior generated from the global prior module to complement the local vision features. Finally, the complete vision features, alongside the textual embeddings, are inputted into the learnable referring image segmentation head and generate the corresponding segmentation masks.
2.3 INTERTWINED VISTON LANGUAGE EFFICIENT TUNING
For an input tokenized referring expression T E L and an input tokenized image I e R XW along with visual encoder (01, .. .NN composed of N transformer blocks,
3

---

Published as a conference paper at ICLR 2024
Main Pra mework
GST/
GST
GSTO
Globaal Prier
Side Features
Blnck
Segmentation
Global Prior
Vision Features
Mediale
Projector
The dog walking on the left steps, surrounded
Langunge
Embeddings
Prior
Fuslon
Modole
Language
Learnable
Language Embeddinggs
Concat
Vision Features
Euubeddings
Decoder
Query
Cos Similarity
Tuned
Image Tokens
Global Prior
Figure 2: The framework of BarLeRla. GST the abbreviation of global shortcut tuning. For the visual branch, we fuse the textual embeddings with the visual input in the frozen visual block and further adapt the feature distribution with normalizing flow. For language branch, we concatenate the visual class token the textual imput and achieve modal fusion and feature adaption similarly. Besides, a global shortcut tuning module along with a global prior module is proposed in parallel with the backbone for global visual regularization.
begin by projecting the tokenized expression T T' TW proj- TWproj. where proj RDC Next, we concatenate the projected expression T the tokenized image I (we reshape it into R (HH-W)xC before), and the class token cls as [cls, T", I]. These intertwined embeddings are then fed into the frozen visual encoder Φ and the output given as:
[ols, embed] F(ccs, embed)) + db ([els, T/11, embed))
o....... where F= f .... f1 represents a chain of J invertible feature mappings: fa(z) = + A
M(? + 3.), and T represents the output from the (i 1)-th textual block. Note that take the projected expression T' as input for the frist layer. With the Multi-Head Self-Attention (MHSA) module employed in each visual block for information interaction between tokens, we successfully achieve modal fusion through these frozen visual blocks. Additionally, the shortcut normalizing flow F is applied in each visual block for feature adaptation. As for the textual block vi, we project the visual class token cls into cls' to align its feature dimension with the textual embeddings T. Then, we concatenate the textual embedding with the projected class token cls from the previous visual block to form the input. Furthermore, adapt the shortcut textual embeddings and employ the frozen transformer block to fuse the textual and visual features. Consequently, obtain the output of the textual block th follows:
leverage the shortcut normalizing flow to
[T] F((ds(-1,, TJ) (ce((es)-11
2.4 BI-DIRECTIONAL EFFICIENT TUNING
As discussed in Sec. I bi-direction refers to the combination of local efficient tuning and global prior regularization, and the bi-directional efficient tuning framework consists of two modules: the Global Prior Module and the Global Shortcut Tuning Network.
Global Prior Module. As language is more semantic-rich, the produced language embeddings tend to be more robust compared to the visual ones. Therefore, we propose regularizing visual features through the global prior generated by the language embeddings. Specifically, given the visual encoder & and its output vision feature F., as well as the language encoder z and its output language embeddings Fi, we concatenate the vision features with the language embeddings to obtain intertwined features: [Fi, F.]. Next, we calculate the cosine similarity between the intertwined feature [F, Fu) and the language embeddings F,. This cosine similarity serves as an attention mask, which is then multiplied with the intertwined feature to produce the global prior p:
p cos((F, FoJ, Fi) [Ft, F.)
4

---

Published as a conference paper at ICLR 2024
Global Shortcut Tuning Network. To ensure that the global prior regularization does not con- flict with our local intertwined vision language efficient tuning and to achieve an end-to-end PET pipeline, introduce global shortcut tuning network G that operates in parallel with the vision encoder. This network consists of M modules {91,". ........ ,9s),, each following the design of the transformer block with MHSA and MIP, but with smaller feature dimensions (default setting: 144). Firstly, we transform the global prior p using M linear transformations to obtain adapted priors P1,-..
> PM for each global shortcut tuning module. Then, given the tokenized input image / and the language embedding Fi, we concatenate them with learnable query tokens q as [g. Ft, I1, which serves the input for the global shortcut tuning network. For the first global shortcut tuning module Gi. we feed these concatenated tokens along with the global prior pl as follows:
1 = (1 Fi, 1] P1)
(5)
where represents the output of the first module, and Gi (1, utilizes the global prior p regu- larize the MHSA feed-forward in the global shortcut tuning module. In elaboration, we project the input / into query, key, and value tokens using projection matrices Wq, Wi. and W,, respectively. Additionally, we project the global prior p into complementary value tokens using learnable projec- tion matrices Wp. The two sets of value tokens are then added together, resulting in the new value tokens IW, + IWp. Consequently, we redefine the self self-attention mechanism as:
(Wg VC
(HSA(T) Softmax
(IW, + IWP)
(6)
Here, C' denotes the feature dimension, and for simplicity, we omit the multi-head division. By mod- ifying the value tokens in the MHSA block, we successfully incorporate global prior regularization into the global shortcut tuning module through the introduced global attention. For the remaining M modules, we repeat the global prior regularization process as follows:
1; = G.(
...................... M
The global shortcut tuning network comprises very few parameters and is primarily employed for garameter-efffcien tuning. Subsequently, we proceed to fuse the output shortcut features I1.... IMs from each global shortcut tuning module with the corresponding vision features F,. For given i-th output shorteut feature T; and its corresponding vision features Fy, first interpolate 7; match the height and width the original vision features Fý. Subsequently. we add these two sets vision features obtain the output feature Fout as follows:
F aut Interpolate((.) + F²
(8)
To prevent conflicts with the local intertwined module, we detach F during fine-tuning
2.5 FINAL OBJECTIVE
Following Wang al. 2022) and Xu et al. 2023 segmentation head composed of cross-modal neck, vision-language decoder, and an up-sample projector to extract the cross-modal intertwined feature F.s and the transformed textual feature Fş:
incorporate a learnable referring image
Fes, F, Head(F MM. Fu, Fi)
where FM represents the output from the last global shortcut tuning module, while Fj and F, denote the textual embeddings and the vision encoder features adapted by the local intertwined modules. To train our model, we employ text-to-pixel contrastive loss (Wang al. 2022) our training objective, which encourages the alignment of textual embeddings with their corresponding visual pixels, while pushing textual embeddings away from other irrelevant visual pixels. The text-to-pixel contrastive loss is formulated as follows:
(- log (o (Fr, F.)) iEP [ log -1-a (Frs -F))),
{
Cp F Ft)
iEN
(10)
Cup (Fes, F.)
Σ Cip (Fás P.)
PUN
SEPUNN
where denotes the sigmoid function, P and N represent the classes of and 0, respectively.
5

---

Published as a conference paper at ICLR 2024
Table 1: Comparison with SOTA RIS methods and the PET RIS SOTA method without additional
datasets evaluated using the IoU metric on RefCOCO-related datasets.
RefCOCO
RefCOCO+
G-Ref
Method
Avg
val
lestA
testB
val
lestA
testB
val(u) test(u) val(g)
Traditional
Full
Fine
tuning
MAtNee Yu et al. 2018)
56.5
62.4
51.7
46.7
52.4
40.1
47.6
48.6
50.5
RRN (Li
2018
55.3
57.3
54.0
39.8
42.2
36.1
36.5
43.8
CMSA Ye et al. 2019
58.3
60.6
55.1
43.8
47.6
37,9
40.0
47.0
CAC Chen et
2019
58.9
61.8
53.8
46.4
47.0
44.3
BRINet Hu et al. 2020)
61.4
63.4
59.6
48.6
52.9
42.1
48.0
52.5
CMPC+
et al. 2021 a
61.4
64.5
59.6
49.6
53.4
43.2
CGAN (Luo et al. 2020
64.9
68.0
62.1
51.0
55.5
44.1
51.0
51.7
55.5
LTS Jiig et al. T
65.4
67.8
63.1
54.2
58.3
48.0
VLT Ding et al.] 2021
65.7
68.3
62.7
55.5
59.2
49.4
49.8
56.7
PCAN CCen et al.
20022a
69.5
71.6
64.2
58.3
63.7
48.9
60.0
60.8
57.5
61.6
ReSTR Kim et al.
2022
67.2
69.3
64.5
55.8
60.4
48.3
54.5
54.5
58.8
CRIS Wang et al.
2022
70.5
73.2
66.1
62.3
68.1
53.7
59.9
60.4
63.8
LAVT
Yang al.
2021
72.7
75.8
68.8
62.1
68.4
55.1
60.5
64.9
WiCo
Cheng et al.
2023)
73.5
76.9
68.1
63.4
69.2
55.8
60.2
65.3
Parameter Efficient- Tuning
ETRIS (Xu et al. 2023
70.5
73.5
66.6
60.1 1
66.9
50.2
59.8
59.9
57.9
62.8
Ours
72.4
75.9
68.3
65.0
70.8
56.9
63.4
63.8
61.6
66.5
3 EXPERIMENTS
3.1
EXPERIMENTAL. SETUP
Datasets. We employ three challenging referring image segmentation benchmarks our experi- ments: RefCOCO Kazemzadeh et al. (2014). RefCOCO+ (Kazemzadeh et al. 2014)) and G-Ref Yu
al. 2016). Please refer to the appendix A. for details.
Implementation Details. We train the whole network in an end-to-end manner for 50 epochs using the Adam optimizer with a learning rate of 0.0001. A learning rate decay is employed the 35th epoch with decay factor of 0.1. We train the model using 2 Tesla V100 GPUs with batch size of 32. For ViT-L/14, we train the model using 8 Tesla V V100 GPUs with batch size of 64 and an initial learning rate of 0.0002. Following previous works (Ding et al., 2021 Liu et al. 2017 Wang et al.) 2022 Xu et al.) 2023), we adopt IoU as the metric to evaluate the performance. More details can be referred to in the appendix A.2
3.2 MAIN RESULTS
We conducted a comprehensive comparison between our BarLeRla and series of previous RIS ap- proaches. The results, presented in Tab. demonstrate that our approach significantly outperforms state-of-the-art RIS methods on three commonly used datasets, achieving 6 SOTA and sub-sub- SOTA performance across evaluation tasks. In particular, we surpass the performance of WiCo (Cheng et al., (203), which utilizes an additional ResNet-50 to extract top-down segmentation pro- posals as a pte-stage, In contrast, our BarLeRIa model achieves superior results using only 2.2M parameters and is trained in an end-to-end manner.
Furthermore, we compare our proposed approach with the state-of-the-art parameter-efficient tuning RIS method, ETRIS (Xu et al. 2023). To ensure fair comparison, employ the same CLIP pre-trained vision language model of ViT-B/16 as used in ETRIS and freeze the visual and textual encoders. The tuning backbone parameters in BarLeRIa ETRIS. It is worth noting that we can achieve superior performance to ETRIS only using the local intertwined module with much fewer tuning parameters, and more details shown in Sec. 3.5
only 2.2M. which is comparable to
Overall, BarLeRIa achieves a significant improvement of +3.7 IoU on average across the three RefCOCO-related datasets, demonstrating its superiority over existing RIS methods.

---

Published as a conference paper at ICLR 2024
Table Comparison with full fine-tuning SOTA RIS methods and these methods either utilize large language models or are pre-trained with additional datasets. IoU is utilized as the metric. f denotes that the model is tuned using the mixed RefCOCO datasets.
RefCOCO
RefCOCO+
G-Ref
Method
Avg
val
testA
testB
val
testA
testB
val(u)
test(u)
LISA-7B (Lai et al. 2023)
74.1
76.5
71.1
62.4
67.4
56.5
66.4
68.5
67.9
LISA-7B (It) (Lai
2023
74.9
79.1
72.3
65.1
70.8
58.1
67.9
70.6
69.9
PolyFormer-B Liu et al. 2023
74.8
76.6
71.1
67.6
59.3
72.9
67.8
69.1
69.9
UNINEXT-R50 Yan et al. 2023
77.9
79.7
75.8
66.2
71.2
59.0
70.0
70.5
71.3
ETRIS Xu et al. 2023)
72.4
74.6
69.3
64.5
70.4
56.9
62.6
63.1
66.1
BarLeRla
75.0
77.1
71.2
68.6
73.2
61.2
65.9
66.4
69.8
BarLeRla-Mixed
77.6
79.4
75.3
71.7
75.7
66.0
70.9
71.4
73.5
3.3 COMPARISON TO FULL FINE-TUNING METHIOSS
We further conducted comparison between our proposed approach and existing SOTA full fine- tuning methods. These methods either utilize large language models or are pre-trained with addi- tional datasets that contain region-level information. Without using additional datasets, we select a superior CLIP version, EVA-CLIP, which is still pre-trained using general-purpose datasets. For fair comparisons, we also use the EVA-CLIP pre-trained vision language model as the backbone for ETRIS. As shown Tab. 2 BarLeRla outperforms ETRIS by significant margin, achieving an average improvement of +3.7 IoU using the same EVA-CLIP pre-trained backbone.
Compared to LISA-7B (Lai al., 2023), large vision language model with 7 billion parameters, approach demonstrates sigmifican improvement when LISA-7B is not fine-tuned and achieves comparable performance when LISA-7B is fully fine-tuned. Compared with PolyFormer-P (Liu ct al. 2023) that utilizes Swin-B (Liu al. 2021b) the visual encoder and the BERT transformer as the textual encoder, our proposed BarLeRIa achieves comparable performance without additional region-level pre-training and mixed fine-tuning. It is worth noting that PolyFormer introduces sec- ond pre-training phase to incorporate region-level information using additional datasets, including Visual Genome, three RefCOCO-related datasets, and Flickr30keentitie.. Furthermore, BarLeRIa achieves +3.6 IoU improvement over PolyFormer-B when we additionally employ mixed fine- tuning. UNINEXT (Yan al. 2023) leverages pre-training on Objects365 to learn region-level information and also employs mixed fine-tuning, BarLeRla achieves a +2.2 IoU improvement over UNINEXT-R50 when we also employ mixed fine-tuning with much fewer tuning parameters.
We also conduct experiments using the ViT-Large visual encoder to verify the generalization abil- ity of our method across different architectures. As shown Tab. 3 BarLeRla-L outperforms PolyFormer-L without additional region-level pre-training and mixed fine-tuning. Moreover, com- pared the best-performing RIS method, UNINEXT, BarlLeRla---Miee achieves a clear margin of +1.0 IoU averaged improvement across RefCOCO-related datasets, demonstrating its effectiveness.
3.4 VISUALIZATION
As illustrated in Fig. 3 we present visualization results with different settings under easy scenarios and hard scenarios, respectively. In the figure, (d) SNF means we just use normalizing flow to adapt the visual features without the bridge used in ETRIS, and (e) SNF+ETRIS means we combine SNF with ETRIS. We use these two settings to determine whether SNF is the key to PET RIS approaches. We find that both (d) and (e) lag much compared with our BarLeRla and prove that our proposed two PET modules provide great improvement (more details of the ablation are shown in Sec. 3.5
The first two rows of Fig. 3 represent the easy scenario and all methods can segment objects cor- rectly. The difference is only in the detail and the finesse of the contours. BarLeRIa and BarLeRla- L-mixed achieve the best segmentation IoU while ETRIS performs worst. For the hard scenario, i.e., the last two troduce overly large outlines, indicating that they do not fully understand the text description, while our BarL_eRla fully understands the meaning of the text and accurately segments the target objects.
of Fig. 3 ETRIS fails to locate the object correctly, SNF and SNF+ETRIS in-
7

---

Published as a conference paper at ICLR 2024
Table 3: Comparison with full fine-tuning SOTA RIS methods using ViT-Large the visual back- bone, These methods either utilize large language models or are pre-trained with additional datasets. IoU is utilized as the metric. denotes that the model is tuned using the mixed RefCOCO datasets.
RefCOCO
RefCOCO+
G-Ref
Method
Avg
val
testA
testß
val
testA
testB
val(u) test(u)
PolyFormer-LI f UNINEXT-L BarLeRIa-L
Liu et al.
2023
76.0
78.3
73.3
69.3
61.9
74.6
69.2
70.2
71.6
Yan al.
2023
80.3
82.6
77.8
70.0
74.9
62.6
73.4
73.7
74.4
76.8
79.0
74.0
71.5
76.2
65.4
68.7
69.7
72.7
BarLeRIa-L-Mixer
79.0
80.8
77.0
74.2
77.8
68.3
72.7
73.3
75.4
Language: running"
Language: closest us"
Language: "cul bumana buw!"
I Language: "heaa with face most showing"
(a) Image
(b) GT
(c) ETRIS
(d) SNF
(e) SEF+ETRRS
(9 BarLeRla
(g) BarLeRIa-11
Figure 3:
Qualitative results with different settings. (a) the input image. (b) the ground truth. (c)
ETRIS, (d) SNF without local intertwined module, (e) SNF+ETRIS. (f) our proposed BarLeRIa. (g)
BarLeRIa-L using mixed datasets, Best viewed in color.
3.5 ABLATION STUDY
To establish the efficacy of our proposed approach, we perform ablation studies on the components of our proposed BarLeRIa. We only briefly document the averaged performance of different test splits for RefCOCO, RefCOCO+, and G-Ref respectively (please refer to the appendix for detailed results). Illustrated in Tab. 4 SNF means the abbreviation of Local Intertwined Module, GST denotes Global Shorteut Tuning, and No Global means just the Local Intertwined Module without the Global Shorteut Tuning. As see, just employing existing SNF or combining SNF with ETRIS does not improve segmentation performance. Besides, if only the local intertwined module (No Global in the table), can outperform ETRIS with +2.6 averaged IoU improvement with nearly one-tenth the number of tuning parameters. This result demonstrates that BarLeRIa can greatly surpass existing PET state- of-the-art with fewer learnable parameters and showcases its superiority. Finally, with the proposed Global Shorteut Tuning, BarLeRla achieves further enhancements +1.1 averaged IoU.
the normalizing flow to adjust the feature, LIM is
RELATED WORK
Parameter Efficient Tuning (PET) adjust fraction of the parameters and alleviate the compu-
tational challenges associated with fine-tuning the entire model. One prominent research direction focuses on incorporating lightweight architectures into the frozen backbone and updating only these
newly added architectures during fine-tuning (Houlsby al. 2019 Mahabadi et al. 2021 Lester

---

Published as a conference paper at ICLR 2024
Table 4: Ablation study the components of Barl. LIM is the abbreviation of Local Inter- twined Module, GST denotes Global Shortcut Tuning, and No Global means we just use the Local Intertwined Module without the Global Shorteut Tuning.
Method
SNF
LIM
GST Params(M) RefCOCo RefCOCO+
G-Ref
Avg
ReSTR
86.19
67.0
54.8
54.5
58.8
ETRIS X
x
x
1.39
70.2
59.1
59.2
62.8
SNF
0.18
70.6
59.6
59.3
63.2
SNF+ETRIS
X
X
1.57
70.2
59.9
60.1
63.4
No Global
0.39
71.4
63.1
61.6
65.4
BarLeRIa
V
2.21
72.2
64.2
62.9
66.5
et al.
2021:
Li & Liang 2021 Karimi Mahabadi et al.] 2021 Chen al. 2022b Jie & Deng.
Jia et al. 2022). For instance, AdaptFormer (Chen et al. 2022b) and ConyPass (Jie Deng, introduce bottleneck or convolution modules along the skip connections within transformer layers and adapt the residuals for downstream tasks. Recently, [Wang et al. (2023) proposed lever- aging normalizing flows to adjust the shortcuts rather than the residuals within transformer layers, offering an easily implementable and accessible approach for various architectures. Another line of the PET method involves updating only a subset of the parameters the original model (Sung et al. 2021 Zaken et al. 2021). Zaken et al. (Zaken et al. (021). for example, demonstrate that updating only the bias terms can achieve competitive or even superior performance compared to full fine-tuning. Additionally, some researchers have explored matrix decomposition techniques to re- duce the number of learnable parameters by factorizing the weights of pre-trained models (Hu al.] 2021 & Deng. 2023). which also yield satisfactory performance. Unfortunately, PET approaches for referring image segmentation are less investigated. Recently, Xu et al. (2023) introduced PET referring image segmentation by leveraging the bridge module for information fusion between visual and textual modalities. However, their proposed ETRIS lacks feature adaption and global vi- sual regularization, resulting in unsatisfactory performance. Besides these works, some researchers factorize weights of the pre-trained model based on the low-rank assumption, such that parameters that need be tuned can be largely reduced Hu et al. 202TT.
2022
2022
Referring Image Segmentation (RIS) aims to segment a target instance or region referred by the given text query and is initially introduced by Hu al. (2016). Early methods were predominantly based on the CNN+LSTM approach (Liu et al. 2017 Li et al. 2018), where the image and text inputs were encoded separately using their respective backbones. However, in recent years, trans- former architectures have gained popularity due to their flexibility and scalability (Vaswani et al.) 2017 Dosovitskiy et al. 2020), allowing RIS methods to employ a unified architecture across dif- ferent modalities (Kim et al. 2022 Yang et al. 1021 Liu et al. 2023 Yan et al. 2023). Additionally, the advent of multi-modal pre-trat training (Radford et al.) (021)) has provided RIS models with the ad- vantage of leveraging large-scal pre- pre-training data Wang et al. 2022). Besides, recent work (Cheng jet al. 2023) has proven that the global prior can help the referring segmentation. However, these methods require full fine-tuning of additional per-parameterized model and divide the segmen- tation process into two stages without end-to-end training.
CONCLUSION
In this paper, we pay attention to parameter efficient tuning for referring image segemntation. We reveal that previous approaches focus on vision and language modal alignment, but ignores adapting the biased feature from pre-trained models. Besides, previous approaches fuse the local visual fea- tures with the textual embeddings without introducing global prior from text input to regularize the visual feature. To address these issues, we propose a novel PET framework BarLeRla: Bi-directional Intertwined Vision Language Efficient Tuning for Referring Image Segmentation. which leverages intertwined vision language adapters and bi-directional tuning framework to fully exploit the frozen pre- vre-trained models' potential. We conduct extensive experiments on three RefCOCO-relate bench- marks. BarLeRIa consistently outperforms prior parameter efficient tuning methods with clear margin. Morcover, BarLeRla also surpasses full fine-tuning state-of-tte oftth-----h-rrttt approaches without pre- training using additional training datasets.

---

Published as a conference paper at ICLR 2024
Acknowledgment This work was supported in part by the National Natural Science Foundation of China under Grant 62125109, Grant 62250055, Grant 61931023, Grant 61932022, Grant 22371288, Grant 62320106003, Grant 62301299, Grant T2122024, Grant 62120106007.
REFERENCES
Bo Chen, Zhiwei Hu, Zhilong Ji, Jinfeng Bai, and Wangmeng Zuo. Position- contrastive alignment for referring image segmentation. arXiv preprint arXiv:2212. 2022a.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205.13555, 2022b.
Yi-Wen Chen, Yi-Hsuan Tsai, Tiantian Wang, Yen Yu Lin, and Ming- Hsug- Yang. Referring ex- pression object segmentation with caption-aware consistency. arXiv preprint arXiv:1910.0474 2019.
Zesen Cheng, Peng Jin, Hao Li, Kehan Li, Siheng Li, Xiangyang Ji, Chang Liu, and Jie Chen. Wico: Win-win cooperation bottom-up and top-down referring image segmentation. arXiv preprint arXiv:2306. 10750, 2023.
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 16321-16330, 2021.
Shuangrui Ding, Weidi Xie, Yabo Chen, Rui Qian, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Motion-inductive self-supprvised object discovery in videos. arXiv preprint arXiv:2210.00221, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 1rXiv:2010.11922 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning. pp. 2790-2799, PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106.0906009555 2021.
Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expres- sions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Nether- lands, October 11-14, 2016, Proceedings. Part 14, pp. 108-124. Springer. 2016.
Zhiwei Hu, Guang Feng, Jiayu Sun, Lihe Zhang, and Huchuan Lu. Bi-directional relationship inferring network for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4424-4433, 2020.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203. 2022.
Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv preprint ar arXiv:2207.0000 2022.
Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1060-1068, 2023.
Ya Jing, Tao Kong, Wei Wang, Liang Wang. Lei Li, and Tieniu Tan. Locate then segment: A strong pipeline for referring image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9858 9858-9867, 2021.
10

---

Published as a conference paper at ICLR 2024
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances Neural Information Processing Systems, 34:1022- 1035, 2021.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp- 787-798, 2014.
Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Restr: Convolution free referring image segmentation using transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18145-18154, 2022.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. arXiv preprint arXiv:2308. 000922200020..... 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, arXiv preprint arXiv:2 104.08691, 2021.
Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refinement networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5745-5753, 2018.
Xiang Lisa and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 101.001900 2021.
Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan Yuille. Recurrent multimodal interaction for referring image segmentation. In Proceedings of the IEEE international conference on computer vision, pp. 1271-1280 2017.
Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 18653-18663, 2023.
Si Liu, Tianrui Hui, Shaofei Huang, Yunchao Wei, Bo Li, and Guanbin Li. Cross-modal progressive comprehension for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4761-47755 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu. Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Gen Luo, Yiyi Zhou, Rongrong Ji, Xiaoshuai Sun, Jinsong Su, Chia- Wen Lin, and Qi Tian. Cascade grouped attention network for referring expression segmentation. In Proceedings of the 28th ACM International Conference Multimedia, pp. 1274-1282, 2020.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint (tti:2/006044488 2021.
Rui Qian, Shuangrui Ding, Xian Liu, and Dahua Lin. Semantics meets temporal correspondence: Sele-supe object-centric learning in videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16675-16687, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Li-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24200 2021.

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Yaoming Wang, Bowen Shi, Xiaopeng Zhang, Jin Li, Yuchen Liu, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Qi Tian. Adapting shortcut with normalizing flow: An efficient tuning framework for visual recognition. In 2023 IEEE/CVF Conference on Computer Vision and Put- tern Recognition (CVPR), pp. 15974, IEEE, 2023.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11686-11699 2022.
Zunnan Xu, Zhihong Chen, Yong Zhang, Yibing Song, Xiang Wan, and Guanbin Li. Bridging vision and language encoders: Parameter-efficient tuning for referring image segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023.
Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15325-15336, 2023.
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. arXiv preprint arXiv:2112.02220 2021.
Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In Proceedings of the IEEE Conference on Compuler Vision and Pattern Recognition, pp. 10502-10511, 2019.
Licheng Yu, Patrick Poirson, Shan Yang. Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amster- dam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 69-85. Springer, 2016.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mat- tnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1307-1315, 2018
Elad Ben Zaken, Shauli Ravfogel, and Yoay Goldberg. Bitfit: Simple parameter-efficien fine-tuning for transformer-baat based masked monguage-modeh. arXiv preprint arXiv:2106. 10199, 2021.
A EXPERIMENTAL SETUP
A.1 DATASETS
RefCOcO (Kazemzadeh et al.) 2014) is widely used benchmark for referring image segmentation out of large scale. It consists of 19,994 images with annotations for 142,210 referring expressions related to 50,000 objects. These annotations were gathered through two-player game based on the MSCOCO dataset. The dataset divided into four subsets: 120,624 samples for training, 10,834 for validation, 5,657 for test A, and 5,095 for test B. On average, the referring expressions are 3.6 words long, and each image contains at least two objects.
RefCOCO+ (Kazemzadeh et al. 2014) contains 141,564 referring expressions associated with 49,856 objects across 19,992 images.
designed to be more challenging than RefCOCO by excluding certain absolute-location words. Similar to RefCOCO, RefCOCO+ is divided into four subsets: 120,624 samples for training, 10,758 for validation, 5,726 for test A, and 4,889 for test B.
G-Ref Yu et al. 2016) consists of 104,560 referring expressions related to 54,822 objects in 26,711 images. The expressions in G-Ref were collected from Amazon Mechanical Turk and have an aver- age length of 8.4 words. Compared with the previous two datasets, G-Ref contains more descriptions of locations and appearances in the text annotations. Note that we apply both the Google and UMD split in experiments with the denotation G-Ref(g) and G-Ref(u) respectively.
12

---

Published as a conference paper at ICLR 2024
Language: yummy""
Lamguage: table without anything
(al Image
(b) SKT-ETRIS
[c] BateRla
(d) BareRRl
GT)
Figure 4:
Visualization results with different settings. (a) the input image. (b) SNF+ETRIS. (c)
Barl eRla. (d) -ela-L using mixed datasets. (e) the ground truth. Best viewed in color.
Language: "older sheep"
Language: "small sheep
Language:: "largest sheeg"
Language: "closest little ooe"
(e) Image
(b) SKT+ETRIS
BarLeRla
BarLRRla1*
GT
Figure 5: Visualization results with different settings. (a) the input image. (b) SNF+ETRIS. (c)
BarLeRIa. (d) BarLeRIa-L using mixed datasets. (e) the ground truth. Best viewed in color.
A.2 IMPLEMENTATION DETAILS
We choose VIT-B/16 of CLIP as the pretrained vision language model in most experiments. Fol- lowing Wang et al. (2022), we resize the input images to 416 x 416. For the visual backbone and textual backbone, we follow the settings of CRIS (Wang et al. 2022) and ETRIS (Xu et al, 2023). For the local intertwined module, we employ SNF-deep, which has four transformations in MHSA block and MLP block respectively, each visual transformer layer and text transformer layer. Following Wang et al. (2023). the explicit log Jacobian loss is not used when calculating the training objective. For the global shortcut tuning network that consists of 4 global shorteut tuning modules. We set the feature dimension 144 default setting. We employ 128 learnable query tokens following VPT. The output global shortcut features are fused with the output from the 3, 6, 9, 12 visual encoder layers, respectively. The referring image segmentation head consists of
13

---

Published as a conference paper at ICLR 2024
Table 5: The detailed ablation study the components of BarLeRRa..
RefCOCO
RefCOCO+
G-Ref
Method
Pa)ams(M)
Avg
val
testA
testB
val
testA
testB
val(u) test(u)
val(g)
ReSTR
86.19
67.2
69.3
64.5
55.8
60.4
48.3
54.5
54.5
58.8
ETRIS
1.39
70.5
73.5
66.6
60.1
66.9
50.2
59.8
59.9
57.9
62.8
SNF
0.18
70.8
73.8
67.2
60.5
67.1
51.1
60.0
60.0
58.0
63.2
SNF+ETRIS
1.57
70.4
73.3
66.9
60.5
66.1
53.0
60.6
60.8
59.0
63.4
No Global
0.39
71.7
74.6
68.0
64.0
70.2
55.1
62.2
62.3
60.2
65.4
BarLeRRIa
2.21
72.4
75.9
68.3
65.0
70.8
56.9
63.4
63.8
61.6
66.5
transformer decoder and up-sampling projector, and all the settings follow [Wag et al. (2022) We train the whole network in an end-to-end manner for 50 epochs using the Adam optimizer with the 35th epoch with decay factor of 0.1. We train the model using Tesla V100 GPUs with a batch size of 32. For ViT-L/14, we train the model using 8 Tesla V100 GPUs with batch size of 64 and an initial learning rate of 0.0002. Following previous works (Ding et al. 2021 Liu et al., 2017 Wang et al. 2022 Xu et al. 2023). adopt IoU as the metric to evaluate the performance. The loU calculates intersection regions over union regions of the predicted segmentation mask and the ground truth. Given the cross-modal intertwined feature Foi and the transformed textual feature Fr. the segmentation result is obtained by reshaping (Fes F₂) into H W and then sampann it back to the original image size, fol- 4 44
learning rate of 0.0001. A learning rate decay is employed
lowing Wann al. (2022), where denotes the sigmoid function, and W are the origin shape of the input images.
B MORE EXPERIMENTAL RESULTS
B. DETAILED ABLATION
To establish the efficacy of our proposed approach, we perform ablation studies on the components of our proposed BarLeRRa. In this detailed ablation, we document the performance for different splits respectively. Illustrated in Tab. [] SNF means we use the normalizing flow to adjust the feature, LIM is the abbreviation of Local Intertwined Module, GST denotes Global Shortcut Tuning, and No Global means we just use the Local Intertwined Module without the Global Shortcut Tuning. As can see, just employing existing SNF or combining SNF with ETRIS does not improve segmentation performance. Besides, if we only use the local intertwined module (No Global in the table), outperform ETRIS with +2.6 averaged loU improvement with nearly one-tenth the number of tuning parameters. This result demonstrates that BarLeRIa greatly surpass existing PET state-of-the- art with fewer learnable parameters and showcases its superiority. Finally, with the proposed Global Shortcut Tuning, BarLeRla achieves further enhancements to +1.1 averaged IoU.
B.2 MORE VISUALIZATION
As illustrated in Fig. 6 and we present visualization results with different settings. In the figure, (b) SNF+ETRIS means we combine SNF with ETRIS, and (d)) BarL.eRla-L." denotes we use ViT-large as the visual backbone and employ mixed datasets tuning using BarLeRla. All the results demonstrate that our proposed intertwined vision language tuning algorithm and Bi-directional effi- cient tuning framework successfully adapt the features, introduce global regularization, and yield significant segmentation performance improvement.
C THE DETAILS OF THE REFERRING IMAGE SEGMENTATION HEAD
follow Wang al. 2022 and Xu al. (203)). and incorporate learnalle referring image segmentation head composed of a cross- moosl neck, lasoualanngugge de- coder, and up-sample projector extract the cross-modal intertwined feature and the transformed
As elaborated in Sec. 2.55
14

---

Published as a conference paper at ICLR 2024
language: "giraffe half way
Language: "girafe middle beight"
Image
(b) SNP+ETRIS
(d) BarLeRIa
(d) BarLeRla-LL*
(e) CT
Figure 6: Visualization results with different settings. (a) the input image. (b) SNF+ETRIS. (c)
BarLeRla. (d) BarLeRIa-L using mixed datasets. (e) the ground truth. Best viewed in color.
Language: "donst to hole
Language: "no hold pastyy"
Tal Image
SNF
(0) BacteRta
10 BataRRb
(6)
Figure 7: Visualization results with different settings. (a) the input image. (b) SNF+ETRIS. (c)
BarLeRla. (d) BarLeRIa-L using mixed datasets. (e) the ground truth. Best viewed in color.
textual feature. Here we introduce the referring image segmentation head in detail. Cross-modal Neck: Following Xu al. (2023). given multiple adapted visual features Fj, 2, N from different stages and the adapted textual embeddings Ft. we employ MHSA with convolution to obtain the fusion features Fy. Then, we concatenate 2D spatial coordinate feature Fooord with the fused features F' and use 3 convolution to further fuse them as:
F, = =CCoov((F, Fooord]
(11)
LanguagL Decoder: Then, add the feature F., outputted from the cross-modal neck and the output from the last global shortcut tuning module F.M as F.+ and input the combined fea- tures along with textual embeddings F, to a vision-language decoder following Wang et al. (2022); Xu al. (0023) to achieve the multi-modal features Fm Specifically, the decoder consists of three layers, with each layer composed multi-head self-attention layer, a multi-head cross-attention layer, and a feed-forward network. Within each decoder layer, the input combined feature is ini- tially passed through the multi-head self-attention layer to capture global contextual information. Subsequently, the cross-attention layer is employed to facilitate multi-modal interaction, enabling the transmission of detailed semantic information from the textual features to the visual features. This interaction is achieved by mapping visual features queries and textual features to keys and
15

---

Published as a conference paper at ICLR 2024
values. Following the cross-attention layer, MLP block comprising two layers, along with Layer Normalization and residual connections, is utilized to further process the output features. Finally, the output of the MLP block is employed generate the final segmentation mask.
Up-sample Projector: To obtain mask prediction on each pixel according to the corresponding semantic information, formations on multi-modal features Fmm and sentence-level feature Fi to extract the cross-modal intertwined feature Fes and the transformed textual feature F/2 as:
follow Wang al., (2022); ): Xu et al. al.] (2023) use a projector make trans-
Fet Comu(Upp(Fmm))
Linear (Fi)
(12)
where UpSample denotes 4x upsampling, and convolution and linear projection are used to trans- form Fm and F/ into a suitable dimension following [Wang et al. (2022) Xu et (2023).
D LIMITATION AND FUTURE WORK
Despite its ability achieve state-of-the-an art performance on referring image segmentation, Bar- LeRIa has some limitations. The proposed global shortcut tuning network is relatively larger com- pared with our local intertwined vision language tuning module (1.8M v.s. 0.4M). more efficient approach that imposes global prior back to the visual features without influencing the original fea- ture adaption deserves future research. Besides, because it involves the processing of variable-length sequences, approach is not compatible with convolutional networks. How adapt the convo- lutional neural network to make our method more general also deserves further research. Finally, open-vocabulary zero-shot referring image segmentation is worthwhile direction to explore as multi-modal large-scale models continue to evolve.
16


Reviews:
[{"review_id": "AQqC35jLQF", "reviewer": "Reviewer_iAN9", "summary": "The authors have proposed a novel intertwined vision language efficient tuning algorithm based on the large-scale CLIP model. They claim that the previous methods overlook adapting the biased feature from pre-trained models and global prior regularization. The proposed method achieves state-of-the-art performance on RefCOCO, RefCOCO+, and G-Ref.", "questions": "Could you provide any results of the inference time compared with other parameter tuning methods?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method brings a few trainable parameters into CLIP for both feature adaptation and modal fusion, which achieves the best performance on several RIS datasets.\n2. The proposed method utilizes a novel intertwined structure to assist modal feature fusion.", "weaknesses": "1. The global prior is predicted after the vision-language blocks. Then, the global prior is input to the global shortcut tuning module. However, such a design is quite a long path. The inference speed may be very slow.\n2. The comparison with other methods may be unfair. The proposed method utilizes ViT-Large as its backbone.\n3. The dimensions and meanings of different symbols are not introduced in the paper, such as [Fl, Fv].\n4. The decoder is not introduced in this paper. Although the authors have claimed they follow previous works, it is not detailed enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PHqittbfZo", "reviewer": "Reviewer_eHeC", "summary": "This paper considers the problem of using parameter efficient fine-tuning in referring image segmentation (RIS). A sophisticated PEFT paradigm is proposed, consisting of Adapting Shortcut with Normalizing Flow (SNF),  Local Intertwined Module (LIM), and Global Shortcut Tuning (GST). Results are presented on commonly used RIS benchmarks of RefCOCO, RefCOCO+, and G-Ref. A detailed evaluation shows superior results of the proposed method over previous PEFT method in RIS.", "questions": "Does the side network (Global Shortcut Tuning Network) not operate in parallel with the vision/language encoder? And is it required to cache each $F_v^i$ for the side network? If so, what is the extra time and memory cost for this approximately?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written. Adequate background is provided, the literature review covers very recent approaches, and the paper provides necessary background to the required concepts.\n- It seems novel to me to use a small side network that takes text+image features with attention and output side features containing global priors. And the ablation study shows a significant improvement with this module.\n- The proposed approach gets strong results that beats the previous parameter efficient fine-tuning approach in RIS.\n- The paper conducts a thorough ablation study to show the effectiveness of each proposed module.", "weaknesses": "- Writing in Section 3.3 and 3.4 is not clear enough. It should be possible to write in one equation for (2)(3)(4) with a better subscription. Also it's a bit unclear whether $F_v^i$ is the same as (the $i^{th}$) $[cls, embed]$.\n- (typo): it should be intertwine not interwine in Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hz7uW4Mef1", "reviewer": "Reviewer_8Qir", "summary": "This paper proposes a novel Parameter-Efficient Tuning framework for Referring Image Segmentation. By designing a bi-directional intertwined vision-language adapter, this frame exploits the frozen power of pre-trained vision-language models. In addition, this paper introduces a Global Prior Module and a Global Shortcut Tuning Network to extract global prior from text input to regularize the visual feature.\nCombining these contributions, this paper outperforms previous parameter-efficient tuning methods on RIS benchmarks and even surpasses SOTA full fine-tuning approaches on several tasks of RIS benchmarks.", "questions": "* See in Weaknesses.\n* Is any possible to provide the training time comparison between ETRIS and BarLeRIa?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Strong performance on three referring image segmentation benchmarks.\n* The well-designed Local Intertwined Module: according to the Table 4: Local Intertwined Module improves can outperform ETRIS with very small amount parameters.", "weaknesses": "* The limited improvement brought by Global Shortcut Tuning.  According to Table 5, Compared with No Global, BarLeRIa achieves 72.2 on RefCOCO using 2.21M, while without GST, the model achieves 71.4 using 0.39M. It seems that the model without GST owns a better balance between performance and parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)