Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model
Authors: Zihan Zhong
Zhiqiang Tang
Tong He
Haoyang Fang
Chun Yuan
Venue: ICLR 2024 poster (2024)
Keywords: Parameter-efficient fine-tuning
Segment Anything Model
LoRA
Semantic Segmentation
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: The Segment-Anything Model (SAM) stands as a foundational framework for image segmentation. While it exhibits remarkable zero-shot generalization in typical scenarios, its advantage diminishes when applied to specialized domains like medical imagery and remote sensing. To address this limitation, this paper introduces Conv-LoRA, a simple yet effective parameter-efficient fine-tuning approach. By integrating ultra-lightweight convolutional parameters into Low-Rank Adaptation (LoRA), Conv-LoRA can inject image-related inductive biases into the plain ViT encoder, further reinforcing SAM’s local prior assumption. Notably, Conv-LoRA not only preserves SAM’s extensive segmentation knowledge but also revives its capacity of learning high-level image semantics, which is constrained by SAM’s foreground-background segmentation pretraining. Comprehensive experimentation across diverse benchmarks spanning multiple domains underscores Conv-LoRA’s superiority in adapting SAM to real-world semantic segmentation tasks.

Published as a conference paper at ICLR 2024
CONVOLUTION MEETS LoRA: PARAMETER EFFI- CIENT FINETUNING FOR SEGMENT ANYTHING MODEL
Zihan Zhong
Zhiqiang Tang Amazon Web Services
Tsinghua University
zhongzh222mai 1s. ts nnghua en
zqtang@aaazonn.
Tong He
Haoyang Fang
Chun Yuan
Amazon Web Services ht otonge zon com
Amazon Web Services
Tsinghua University
haoyfanggammzon. com
yuances2. t tsinghua. edu, en
ABSTRACT
The Segment Anything Model (SAM) stands as foundational framework for im- age segmentation. While it exhibits remarkable zero-shot generalization in typical scenarios, its advantage diminishes when applied to specialized domains like med- ical imagery and remote sensing. To address this limitation, this paper introduces Conv-LoRA, simple yet effective parameter-efficient fine-tuning approach. By integrating ultra-lightweight convolutional parameters into Low-Rank Adaptation (LoRA), Conv-LoRA can inject image-related inductive biases into the plain ViT encoder, further reinforeing SAM's local prior assumption. Notably, Conv-LoRA not only preserves SAM's s extensive segmentation knowledge but also revives its capacity of learning high-level image semantics, which is constrained by SAM's seagrounnd-ackkrooun
segmentation pretraining. Comprehensive experimenta- tion across diverse benchmarks spanning multiple domains underscores Conv- LoRA's superiority in adapting SAM to real-world semantic segmentation tasks
INTRODUCTION
The Al community have witnessed the explosion development of a series foundation models in recent years, such as CLIP (Radford et al., 2021), GPT-4 (OpenAl (023) and ViT-22B (Dehghani et al. 2023). Recently, Segment Anything (SAM) (Kiillovv et al.) 2023). a promptable model pre- trained over 1 billion masks and 11 million images, emerged foundation model for image seg- mentation. Despite its impressive zero-shot performance on generic object segmentation, it doesn" t perform well many real-world segmentation tasks certain domains (Tang al. 2023 Di al.,
2023 Zhou et al. 2023). such as natural images (Borji et al. 2019 Fan et al. (020a)). agriculture Sriwastwa et al. 2018 remote sensing (Xu et al. 2018 and medical images Fan et 2020b).
Following the pretraining- finetuinnini paradigm (Dosovitskiy et al. 2020 He et al. 2022 Liu et al., 2021a), it is natural finetune SAM on downstream tasks enhance its performance. However, existing works (Zhang & Liu 2023 Chen et al.] 2023 Shaharabany et al.) 2023) have failed to either analyze or address certain limitations inherent SAM. 1) SAM's image encoder a plain VIT, which is known to lack of vision-specific inductive biases (Chen et al. (0222) that are useful for dense predictions. 2) SAM's pretraining essentially binary mask prediction task that, where, given prompt, it separates foreground object from background. The low-level mask prediction pretraining hinders SAM's s ability to capture high-level image semantic information crucial for tasks like multi-class semantic segmentation.
To tackle the above limitations and still retain SAM's valuable segmentation knowledge acquired during pretraining, we finetune small set of (extra) model parameters while freezing most of SAM's pretrained weights, hence parameter efficient finetuning (PEFT). This raises the question: Can PEFT
Work done while interning at Amazon Web Services.
Our code is public available https: / / thub. http:///togg.o////uuuuooooo/////
master/exampless es/
est.llt.aaauut//m/eeeo aut.omm http://tttoo//oooeooeee

---

Published as a conference paper at ICLR 2024
RGB Images
GT
VPT
LoRA
Conv-LoRA
Comparison of VPT, LoRA, and Conv-LoRA (ours) in binary-class road segmentation (top) and multi-class transparent object segmentation (bottom). Conv-LoRA reinforces image- related local priors, allowing SAM to separate roads from adjacent buildings, while LoRA and VPT struggle in this regard. In the second row, VPT produces reasonable mask for the bowls but erroneously assigns them to the jar/kettle class (indicated by object color), revealing SAM's lim- ited high-level semantic understanding. Both LoRA and Conv-LoRA rectify this misclassification through finetuing SAM's image encoder, with Conv-LoRA delivering a cleaner mask with fewer boundary artifacts.
Figure 1:
enhance SAM encoder with image-related local prior and facilitate the acquisition of high-level semantic information?
In this paper, we propose new PEFT method named Conv-LoRA by diving into Low-Rank Adap- tation (LoRA) (Hu et al. 2021). lh. LoRA introduces slim trainable linear projection layers into each transformer layer of SAM's encoder, thereby helping recover its capacity to extract high-level se- mantic information. Our experiments demonstrate that LoRA surpasses the widely-adopte visual prompt tuning (VPT) (Jia et al. 22022), particularly in the multi-class semantic segmentation tasks. On top of LoRA, Cony-LoRA integrates lightweight convolution layers within its bottleneck struc- ture. Convolution
introduce the image-related local prior (i.e. pixel exhibits stronger correla- tion with its neighbors than the distant pixels) (Chen et al. 2022) through the local spatial operations.
Furthermore, it is essential to inject the local prior into the appropriate scale(s) of image features, considering the potential variations in object scales. To this end, Conv-LoRA draws inspiration from the concept of Mixture-of-Fii Expetxxetttsssees (MoE) (Shazeer et al.) 2017) and incorporates multiple parallel convolutional experts, each specializing in a distinct feature scale. Given that ViT processes image features fixed scale, typically downsampling them by factor of 16 from the original resolution, each expert in Conv-LoRA initially recovers image features at specific scale, applies convolutional operations, and then reverts the features to the default scale. Compared 2022) and vision-specific transformers like Swin Transformer (Liu et al., 2021a), Conv-LoRA pro- vides an implicit way to enforce multi-scale local priors, assuming it can leverage image features at the default scale to reconstruct feature information at higher scales. Fortunately, SAM's supervised pretraining, which involves masks various scales, enables the ViT to acquire knowledge image features beyond the default scale.
ViT-adaptor (Chen al.)
In the spirit of PEFT, we also remove the prompt encoder and add lightweight MLPs in the mask decoder for multi-class prediction. This simple modification has transformed SAM into end-to- end model that can be finetuned on both binary and multi-class semantic segmentation applications. Overall, our contribution be summarized follows:
We present an innovative PEFT technique Conv-LoRA. By incorporating supplementary convolution operations, Conv-LoRA reinforces the local prior of SAM from the perspective of handling the limitation of plain VIT.
Conv-LoRA uses MoE to model the process of dynamically selecting the proper feature scale inject the vision-speeific inductive biases.
Our investigations reveal that SAM's pretraining has impeded its ViT encoder's capacity to learn high-level image semantic information. However, LoRA demonstrates the potential to help SAM recover this crucial ability.
2

---

Published as a conference paper at ICLR 2024
We conduct extensive benchmark encompassing diverse domains, including natural im- ages, agriculture, remote sensing, and healthcare. Conv-LoRA consistently exhibits supe- norr performance over other PEFT techniques in various downstream tasks.
2 RELATED WORK
Parameter Efficient Fine-Tuning (PEFT). Parameter Efficient Fine- Tueing (PEFT) minimizes computational and storage requirements by selectively fine-tuning small subset of model param- eters, while keeping the majority fixed. PEFT encompasses methods such as adapter-based tech- niques, selective parameter tuning, prompt-driven fine-tuning, and Low-Rank Adaptation (I (LoRA) emerging from Natural language Processing (NLP). In the adapter paradigm (Houlsby et al. 2019
Hu et al. 2021 Sung et al. (0022), compact modules are inserted within transformer layers, and other approaches (Guo et al. [020 Zaken et al., 202I) involve fine-tuning small fraction of parameters from pre-traine pre-trained backbones. Prompt tuning (Lester et al. 2021 Li Liang, 202T) adds adaptable tokens to input or intermediate sequences, and LoRA (Hu et al. 2021 introduces trainable low-rank
matrices into transformer layers for weight updates.
PEFT techniques have also proven effective in the Computer Vision (CV) domain. Visual Prompt Tuning (VPT) (Jia et al., 2022) applies prompt tuning concepts (Lester et al.] (02TT) to image clas- sification, while Scale and Shift Feature Modulation (SSF) (Lian al. 2022) uses scale and shift parameters for modulating visual features in image classifiers. Convpass (Jic & Deng 2022) in- troduces a convolutional bottleneck to enhance ViT's performance in image classification. In study, we focus on developing PEFT for SAM in semantic segmentation tasks, specifically enforc- ing multi-scale local priors beyond the default scale, distinguishing our approach from Convpass.
Segmentation Models. FCN (Long et al., 2015) is a key deep image segmentation model that directly generates pixel-wise segmentation maps from images. U-Net (Ronneberger al. 2015) employs an encoder-decoder structure with skip connections to preserve fine-grained spatial infor- mation. Deeplab (Chen al. (2017a) integrates atrous (dilated) convolutions for multi-scale context, while PSPNet (Zhao et al. (017)) uses pyramid pooling module. DANet (Fu et al. 2019). SANet (Zhong et al. 2020). and EMA (Li al. 2019) utilize attention mechanisms for contextual depen- dencies. Transformer architectures like PVT(Waa) et al., (021). J) Swin (Liu et al., 2021b), CvT (Wu) al. (02T), CoaT (Xu et al., 202T). LeViT (Graham et al.j 202T), Segtormer (Xie al.) 2021a), and PVT v2 (Wang al. (022) bring various improvements. SAM (Ji al. 2023), a recent break- through segmentation, offers a universal approach for segmenting diverse objects and regions in images. Fine-tuning SAM on downstream tasks is recommended due to a lack of high-level semantic information and potential domain bias in the pre- pre-training dataset.
Fine-tuning SAM. Some prior works (Chen et al. 2023 Zhang & Liu 2023 Wu al., 2023 Chai
et al. 2023 Shaharabany et al. 2023] Hu et al. 2023] Wang et al. 2023) explore tine-tuning SAM for downstream tasks. These methods include tuning SAM's 's mask decoder or integrating parameter- efficient tuning methods with SAM's image encoder. Some of them (e.g.,(Chen et al., 2023 Zhang
& Liu 2023) (Shhaaraaany et al. (023)) provide end-to-end solutions to automate SAM. Our method further addresses the structural limitation of SAM's image encoder for capturing visual-specific in- ductive biases by introducing convolution operations. And we unveil that SAM's pretraining ham- pers its ViT encoder's 's ability to learn high-level semantic information. We also transform SAM into an end-to-end semantic segmentation model with minor architectural adjustments.
Mlxture-of-Experte Mfxture-of-E Expert (MoE) is designed to expand model capacity while in- troducing small computational overhead. An MoE layer leverages multiple experts to enhance model capacity, while using the gating network to regulate sparsity for computational savings. Feed- Forward Networks (FFN) are commonly employed as the default choice for experts (Shazeer al.) 2017 Riguelme et al.) |021 Bao et al. 2022 Du al.) 20222) Zhou et al., 2022 Fedus et al. [2022] Some efforts (Zuo et al. 2021 Zhou et al. 2022) focus on more efficient gating mechanisms.
In our work, we utilize the concept of MoE, not aiming improving it. We compare MoE used in our work with original MoE three aspects: 1) The original goal of MoE to expand model capacity without excessively increasing computational overhead, whereas ours is to dynamically inject the local prior into the feature maps of different scales. 2) The structures of experts in MoE typically the same, whereas are not. Each expert specializes in a specific scaling operation
3

---

Published as a conference paper at ICLR 2024
Frozen
Dec.
Dec.
Trainable
Pretrained Weights
Pretrained
Moll-Com
Weights
Enc.
Enc.
(a) LoRA
(b) Conv-LoRA
Figure 2: LoRA vs. Conv-LoRA. Both LoRA and Conv-LoRA add an extra trainable encoder- decoder structure parallel the frozen pre-trained weights. Inside the bottleneck of LoRA, Conv- LoRA inserts lightweight convolution operations managed by MoE with negligible extra parameters.
our method. 3) While MoE mostly employed during pre-training, we employ MoE as part for parameter-efficient tuning the downstream tasks.
METHOD
3.1 CONV-LORA
LoRA. First, let's briefly recap the design of LoRA (Hu 2021). which uses encoder-decoder structure to impose a low-rank constraint on the weight updates (fig. 2 (a)). freezes the pre- trained model weights and injects small trainable rank decomposition matrices into each layer of the transformer architecture. Specifically, given pre-trained weight matrix Wo E Rixa, LoRA adds pair of linear encoder W. and decoder Wa. i.e., trainable rank decomposition matrices, along its side. W. and Wa satisfy the low rank constraints We E R Wa E Raxrr and r < min(a, b). With LoRA, the forward pass changes from Woz
h Wor + WaWer
Cony-LoRA aims to incorporate convolution operations between the encoder and decoder com- ponents of LoRA (fig. (b)) On one hand, convolution can inject the image-related local prior, addressing fundamental limitation of the vanilla VIT. On the other hand, the low-rank constraint ensures that the convolution layers remain exceedingly lightweight, preserving the PEFT nature of ConY-LORA.
A pivotal consideration in designing Conv-LoRA is determining the scale of feature maps at which to introduce the local prior. While the feature maps in VIT are uniform in scale, object masks typically encompass a wide range of scales. Therefore, it is crucial to apply convolution operations at the right scale. To tackle this challenge, Experts (MoE) (Shazeer et al., 2017). MoE comprises multiple expert networks and gating module that dynamically selects which expert(s) to activate during the forward pass (fig. 3). Adapting this concept to Conv-LoRA, each expert specializes in convolution at specific scale of feature maps, and a compact gating module learns dynamically choose the expert(s) based on the input data. Mathematically, with Conv-LoRA, eq. (I) changes to:
draw inspiration from the concept of Mixture of
h Wor + Wa Σ C(W2))
where Wo E CCoustCin We E grx Wa E Xr x E XHX B batch size, Cout Cin/Cout is the number of input output channels, H and W correspond the height and width. E, is the ith expert of all n experts. G the gating network with only top-k (default 1) values activated. Refer to appendix for more details of gating.
Oin//
Inside each expert, three key operations are arranged in sequence: an interpolation function that reconstructs feature maps specific scale, 3 3 convolutional layer, and subsequent interpo-

---

Published as a conference paper at ICLR 2024
0x5x interpolate
Expert
Expert 2
S
Expart
25x interpolate
3x3 Corv
3x3 Conv
interpolate
interpolate
Gating
Selected
UnSelested
Figure 3: MoE-Conv. consists of n experts and a gating network for dynamic expert selection. Each expert reconstructs feature maps at specific scale, applies convolution, and returns the feature maps to the default scale. Each expert specializes in one unique feature scale.
lation operation to map the feature maps back to the default feature scale of VIT. Assume expert E; in charge of scale 5;, we can formulate it
(x)x) = Interpolate (eattteCooovxxccee (Inerrsssttttptttt(......o s.)),
(3)
For instance, if 8; = 4, expert F; would initially upscale the feature maps by a factor of 4x, apply the Conv3x3 operation, and finally, downscale the feature maps by 4x.
MoE vs. Multi-scale. In contrast to MoE, another method to address diverse scales is employing multi-scale strategy. This approach utilizes multiple branches to concurrently inject local priors at various scales and aggregates the results. Although seemingly more straightforward, this method comes a higher computational cost when compared to MoE. The efficiency of MoE stems from its capacity to selectively activate sparse experts, thereby minimizing computational overhead. Given our priority on efficient finetuning, we favor MoF discerning choice.
3.2 END-TO-END MULTIICLASS SEGMENTATION WITH SAM
SAM comprises three essential components: an image encoder, a prompt encoder, and a mask de- coder. When provided with an image and prompt, which can take the form of point, box, mask, or text, the mask decoder generates mask of the object associated with the given prompt. While this prompt-based approach renders SAM flexible for integration into larger systems, such in- teractive segmentation or combination of detection and subsequent segmentation, it does pose challenges in making SAM end-to-end model in practical applications. To automate SAM, freeze the prompt encoder, thus always constant prompt tokens to mask decoder, when finetuning it downstream tasks. Moreover, the original mask decoder is designed predict binary masks, distinguishing between foreground and background based on the given prompt. To adapt SAM for multi-class semantic segmentation tasks, introduce a straightforward classification branch (de- picted the red dashed box in fig. 4) within the mask decoder. This extra branch is responsible for predicting classification scores. Additionally, we apply full fine-tuning to the mask decoder as lightweight module. For more comprehensive information, refer to appendix B
EXPERIMENTS
Settings. We perform several experiments using SAM on four real- world scenarios, including med- ical images, natural images, agriculture and remote sensing. We use the batch size of 4 and Adam optimizer with learning rate of 10-4 4 as default, with weight decay of 1 10 -4 larger learn- ing rate of 3 x 10-4 is found useful for the datasets we use in agriculture and remote sensing. The random horizontal flip is applied during training as data augmentation. All the methods are trained for 30 epochs with structure loss (i.e., the combination of weighted IoU loss and binary cross en- tropy loss) unless otherwise specified. Additionally, our Conv-LoRA follows Shazeer et al. (2017) to introduce extra loss for balancing the utilization among the experts. The weight of the extra loss set to 1.0 and 2.0 for binary-class and multi-class semantic segmentation respectively. We set the number of experts to be 8 by default, with each expert specializing in scaling ratio within the
5

---

Published as a conference paper at ICLR 2024
Ox R W
N 477
image embedding
image
2x conv.
masks
mip
trans.
N
Nx
token to image attn.
token to image atti..
x (K
oulput tokens
mip
classilication scores
N C
Figure 4: The modified SAM's mask decoder for multi-class semantic segmentation. The classifica- tion module (within the red dashed box) is new added compared to original SAM's mask decoder. N is the number of output mask tokens, K is the number of classes, C is the number of channels, H and W indicate the height and width the feature map (we omit 'batch size' for simplicity).
continuous range from 1 to 8, And we apply Conv-LoRA to the query, key and value matrices in self-attention layers, same as how LoRA does.
Datasets. Our experiments encompass semantic segmentation datasets from various domains, span- ning natural images, medical images, agriculture, and remote sensing. In the natural image domain, we explore two specific tasks: camouflaged object segmentation (Fan et al. [2000a] Skurowski et al.,
2018 Le et al., 2019) and shadow detection (Vicente al. (0066). Within medical segmentation, we investigate polyp segmentation (Jha et al. [0200] [ernal et a [215]] Tajbakhsh et al. 2015 Vázquez al., 2017 Silva al. 2014) and skin lesion segmentation (Codella al. 2018). For agriculture
and remote sensing, we employ the leaf disease segmentation (Rath) 2023) and road segmentation (Mnih 2013) datasets as representative examples, respectively. We also explore multi-class trans- parent object segmentation using Trans10K-vl (Xie al.) (2020) with 3 classes and Trans10K-v2
(Xie et al. (021b) with 12 fine- fine-graing reee-griied classes. Further details about each dataset can be found in appendix
Baselines. We compare our method with the following methods: 1) Fine-tune SAM's mask de- coder only. 2) BitFit (Zaken al. 202T), which only fine-tunes bias terms in the pre-trained model. 3) Adapter (Houlsby et al. 2219), which inserts the trainable bottleneck layers between the trans- former layers. SAM-Adapter (Chen al. 2023). which further tunes the patch embedded features and learns an extra embedding for high-frequency components for low-level semantic segmentation tasks. And it one of the pioneer works that apply PEFT method to SAM. 5) VPT (Jia al. 2022) which inserts learnable tokens to hidden states for each transformer layer. 6) LST (Sung et al., 2022], which inserts a trainable side network parallel to the frozen backbone. To control the number of trainable parameters, the side adapter network employs pre-trained ViT-Tiny model, similar to SAN (Xu et al. 2023). The features from the frozen backbone and the side adapter network are fused at the global attention layers of SAM. 7) SSF (Lian et al. (022), which inserts learnable scale and shift parameters to modulate visual features during training. 8) LoRA (Hu et al. 2021 inserts trainable bottleneck layers parallel to the frozen linear weight.
To adhere to the page limit, we select representative datasets of different domains to report the main experiment results. Refer to appendix D for the full experiment results. All the experiments for PEFT methods are run for three times to ease the randomness. The average values and the standard enror reported in table
4.1 BINARY-CLASS SEMANTIC SEGMENTATION
In table [ all PEFT methods consistently outperform fine-tuning the mask decoder alone, underscor- ing the importance finetuning the image encoder of SAM. Furthermore, Conv-LoRA surpasses other PEFT techniques across diverse datasets from different domains. Compared to other PEFT methods, Conv-LoRA uses lightweight convolution operations to strengthen the sision-specific local prior, which turns out effective in boosting the segmentation performance. The substantial perfor- mance gaps between SAM trained from scratch and Conv-LoRA also underscore the considerable assistance provided by SAM's pretraining knowledge in enhancing downstream task performance.

---

Published as a conference paper at ICLR 2024
Mediol
Naturdl Imags
Agricultuve
Remiit Seasing
AParsmm /Mi/ Ratio (9)
Method
Kyasiz
CVC-612
153C 2017
CAMO
SBU
Leaf
Roal
S,
Fg.
S
Iie"
DNce
S.T.
BER
IdET
Dice
IoU1
Dice
Spregle
1005
909
94.4
95.1
730
inslled from scnwol
73.8
docoder only
0..5%
$6.5
89.5
85.5
80.9
69.7
79.5
78.5
83.1
69.8
14,58
50.8
63.8
48.6
65.1
RirFi
1.96/0009
10.8
9388
840
81.7
S0.
Achppor
01.2
84.0
59.3
92.0
85.0
0.62%
$1.5
94.3
01.0
2.70
23.6
93.8
50.2
LST
11.49/
93.3
70.2
SAM Malipee
0.62/
92.5
50.6
92.4
52.1
50.6
93.9
93.8
Can-LLRAA
4.02/
94.7
76.8
Table I: Results on binary semantic segmentation. '# Params (M) / Ratio (%)" represents the number of trainable parameters and its proportion relative to the total number. Domain Specific a placeholder referring to methods that specifically designed for the tasks. 'Underlined' denotes the better results compared to PEFT methods. See appendix [D for more details. Compared to LoRA, Conv-LoRA incurs negligible parameter overhead, but delivers clear performance boost.
Method
#Params (M)
Easy
Hard
Ratio
Acc f
mloU
MAE
MBER
Ace
m
MAB
MBER
TransLab
42.19/ 100%
95.77
92.23
0.036
3.12
83.04
72.10
0.166
13,30
docader-only
3.51 / 0.55%
94.68
88.54
0.050
4.24
83.53
68.30
0,186
14.37
98.31
95.73
0.017
1.52
90.42
LoRA
4,00 0.62%
98.44
96.26
0.016
1.35
91.94
83.95
0.083
6.35
Conv-LoRA
4.02/ 0.63%
96.45
0.015
1.27
93.05
84.37
0.075
6.25
Method
Acct
Calegory
0g
Jar
Ireeaer
Window
door
eyoglass
cup
bottle
box
Trous ab
42.19/ 100%
92.67
69.00
57.72
79.85
81.61
72.82
77.50
56.43
Trows2eg
56.20/ / 100%
94.14
22.15
95.35
53.43
67.82
64.20
59.64
60.56
88:52
96.62
75.99
73.98
83.43
57.17
decoder only
/
90.66
09.97
93.66
32.75
39.96
35.87
50.70
45.89
$7.38
73.16
69.36
54.23
56.58
33.77
VPT
4.00 4.00/0.6250
94.42
62.81
97.41
29.25 20.76
52.82
62.09
55.54
63.61
81.12
83.40
79.61
65.29
72.92
44.77
LoRA
40.62%
94
57.82
64.08
87.28
85.28
63.67
49.56
Conv-LoRA
/0.6%%
58.44
65.23
52.73
Table Results on multi-class semantic segmentation. The tables are the results for three-class and twelve-class semantic segmentation respectively.
Additionally, while Conv-LoRA outperforms certain domain-spectie methods that are having more trainable parameters, it may still fall short on specific datasets. It's important note that Conv- LoRA aims to be general-purpose PEFT method for adapting SAM to various domains rather than competing with these domain-speont specific models. Tailoring SAM for specific domain applications with more intricate adjustments might yield superior performance compared to specialized model designs.
4.2 MULTI-CLASS SEMANTIC SEGMENTATION
In table 2 while decoder-only fine-tuning approaches achieve comparable segmentation accuracy with Nomain-specifiy methods, they exhibit substantial gap in terms of mloU metrics. While ac- curacy measures level-level segmentation performance, mloU takes mask class information into account. We suspect that SAM's image encoder encounters challenges in extracting high-level se- mantic information that valuable for classification tasks. Moreover, fine-tuning the image encoder using PEFT methods results in significant boost in mIoU, indicating restoration of its capability to learn high-level image semantics.
Additionally, we conduct linear probing experiments on SAM's image encoder. Specifically, we freeze SAM's ViT-B encoder and train only a linear head on ImageNet-IK. Similarly, we perform linear probing for the ViT-B pretrained with MAE (He et al. 2022). The results reveal that SAM's image encoder exhibits significantly lower ImageNet- IK accuracy compared to the MAE encoder (54.2% vs. 67.7%). Given that SAM's image encoder is initialized using MAE encoder,
7

---

Published as a conference paper at ICLR 2024
MAE
SAM
18
Mean Attention Distance
Mean Attention Distance
Head D Head
Head
Head
Layer Depth
Layer Depth
Figure 5: Mean attention distance of each attention head, with each dot indicating the mean distance across images for one of the 16 heads at one layer. In contrast to MAE, SAM retains the ability to incorporate local information even in deeper layers.
hypothesize the pre-training focused on low-level brngrruundbbcckrrowee mask prediction adversely affects the VIT encoder's ability capture high-level semantic information for classification.
To be specific, SAM was trained on a dataset that exclusively comprises segmentation masks with- explicit semantic information. In theory, minimize loss, the fundamental objective for its encoder is to project pixels into metric space where pixels from the same object are in close prox- imity, while those from distinct objects are distantly positioned. This projection requires an implicit understanding of 'objectness'. focusing on proximity within an image rather than preserving con- sistent representations of the same-type object across different images. This introduces potential challenge in aligning representations with semantics across diverse images.
4.3 ABLATION STUDY
SAM's local prior assumption. SAM's local prior is grounded in its extensive segmentation pre- training. Through supervised training on vast dataset encompassing billion high-quality masks and million images, SAM has honed a robust capability to discern and capture local features within images. Notably, SAM's encoder retains the ViT architecture, which inherently lacks ded- icated local prior. However, this deficiency is effectively compensated by the significant local prior acquired through segmentation pretraining.
We analyze using the mean attention distance as a metric. In fig. 5 SAM exhibits numerous heads in the deep blocks with short mean attention distances, suggesting its heightened focus local information during the later stages of the encoder. In contrast, the MAE pretrained VIT. representing SAM's initialization, displays consistently long mean attention distances among attention heads in the later stages, Consequently, SAM's segmentation pretraining induces a transformative shift VIT's attention heads, steering them from a global-oriented to local-oriented configuration. This transformation underscores the efficacy of SAM's approach in imbuing the model with distinetive local prior, enhancing ability to capture fine-grained details within images.
MoE vs. Multi-scale. Conv-LoRA leverages MoE to dynamically inject the local prior into feature maps of different scales (fig. (a)), as the optimal scale remains unclear. Here we explore the impact of multi-scale strategy, which fuses the features from all scales simultancously (fig, 6 (b)).
We compare performance and training costs in table Dynamic MoE outperforms multi-scale direct addition in terms of performance. This could be attributed to the preference for specific scales' feature maps based on different inputs. When injecting the local prior into feature maps of all scales simultaneously, the discrepancy of the importance diminishes as the information from critical feature maps is smoothed out. However, with dynamic selection of the top-1 experts for each forward pass, the information from these crucial scales takes precedence and isn't 't diluted by other
8

---

Published as a conference paper at ICLR 2024
scales. Dynamic MoE also provides a 1.54x speedup and reduces memory usage by 1.7GB during training. summary, the comparison underscores the effectiveness and efficiency of MoE.
Method
ISIC
Method
Training Speed (Iter /s)
Training Memory (GB)
Jac 1
T-Jac
Dice
Acc 1
Multi-scale
0.79
23.4
Multi-scalo
85.4
93.7
MoE
77.9
70.3
85.9
93.9
MoE
1.22
21.7
Table 3: MoE vs. Multi-scale, the latter fuses features from scales simultaneously. The per- formance and the training cost illustrate the effectiveness and efficiency of dynamic selecting the experts, i.c. injecting the local prior into the feature maps of different scales dynamically.
The 'Optimal' Scale of Feature Map for Different Datasets. While determine the optimal scale for introducing the local prior, we could check whether the 'optimal' scale within given range varies indeed across different datasets.
unable definitively
Specifically, we simply modify Conv-LoRA: use only one expert and specific scaling ratio. We set the scaling ratio 2, respectively. The experiments are conducted Leaf Disease Segmenta- tion dataset and ISIC 2017 dataset.
Leaf Dice f
ISIC 2017
Scaling Ratio
IoU f
Acc f
Jac
T-Jac 1
Dice
Acc
73.6
83.4
95.5
76.8
69.1
85.0
93.7
2
73.8
83.7
95.8
77.3
69.7
85.4
93.8
74.0
83.7
95.9
76.9
68.4
85.1
93.7
8
73.2
83.1
96.0
76.9
68.3
85.2
93.6
Table 4: Comparison of the 'optimal' scale of feature map for local prior injection across different datasets. We modify Conv-LoRA to use one expert with a specific scaling ratio.
table 4 the 'optimal' scale indeed varies across the different datasets. The scaling ratio set to optimal for Leaf Discase Segmentation dataset, whereas it is set to 2 for ISIC 2017 dataset. These results further confirms our assumption and the necessity for our dynamic local prior injection based on different inputs.
For more ablation experiments, analyses (e.g., further analyses of local prior and MoE) and visual- ization, refer to appendix BenndixxEEhnne through appendix[[
5 CONCLUSION
Parameter efficient finetuning (PEFT) is a popular way when adapting foundation models to ious downstream tasks. We present Conv-LoRA, a novel PEFT approach for applying SAM to downstream segmentation applications. Conv-LoRA is simple, generic, and obtains promising re- sults over multiple domains including natural images, agriculture, remote sensing, and healthcare. Moreover, shed light on several aspects of SAM: 1) although the large-scale supervised seg- mentation pretraining can provide image-related local prior knowledge from the data perspective, injecting lightweight convolution operations in the VIT encoder can further boost the exploitation of local prior from another perspective of architecture; 2) the grrerooundbbackrronn pretraining prevents the image encoder from learning high-level semantic information, which can be alleviated through finetuning relatively few parameters in the encoder.
segmentation
Our efforts primarily focus on developing a general PEFT method for SAM. showing stronger per- formance than existing PEFT methods in broad spectrum of benchmarks, other than directly com- peting with ort (SOTA) models in specialized domains. Given that SAM fine-tuned with Conv-LoRA may not yet consistently outperform domain-specifi, SOTA models, believe that tailoring the mask decoder and prompt encoder beyond image encoder finetuning, and combining Conv-LoRA with other PEFT methods can be promising directions for domain-specifee applications.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank all the anonymous reviewers for their helpful comments. This work was sup- ported by the National Key R&D Program of China (202YFFB4701440044700 SSTIC Grant 1TZ2D202339231151000022 Shenzhen Key Laboratory (ZDSYYS20220022202200000)) and Bei- jing Key Lab of Networked Multimedia.
REFERENCES
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Sub- hojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-modaile -fdit-------odlll----peeere. Advances in Neural Information Processing Systems, 35:32897- 32912, 2022.
Anil Batra, Suriya Singh, Guan Pang. Saikat Basu, CV Jawahar, and Manohar Paluri. Improved road connectivity by joint learning of orientation and segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 10385-10393, 2019.
Jorge Bernal, Javier Sánchez, Gloria Bernández-Esparrace Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and gruphics, 43:99-111, 2015.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprini arXiv:2210.09461, 2022.
Ali Borji, Ming-Ming Cheng, Qibin Hou, Huaizu Jiang, and Jia Li. Salient object detection: A survey. Computational visual media, 5:117-150, 150, 2019.
Shurong Chai, Rahul Kumar Jain, Shiyu Teng, Jiaqing Liu, Yinhao Li, Tomoko Tateyama, and Yen-wei Chen. Ladder fine-tuning approach for sam integrating complementary network. arXiv preprint arXiv:2306. 127237766.777, 2023.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834 848, 2017a
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 0557777707777..... 2017b.
Tianrun Chen, Lanyun Zhu, Chaotao Ding, Runlong Cao, Shangzhan Zhang, Yan Wang, Zejian Li, Lingyun Sun, Papa Mao, and Ying Zang. Sam fails to segment anything?-sam-aade Adapting sam in underperformed scenes: Camouflage, shadow, and more. arXiv preprint arXiv:2304.091448 2023.
7he Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.0855 2022.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864-17875, 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299, 2022.
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion anal- ysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018). pp. 168-172. IEFF, 2018
10

---

Published as a conference paper at ICLR 2024
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference Machine Learning, pp. 7480-7512. PMLR, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /99099 2020.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference Machine Learning, pp. 5547- 5569. PMLR, 2022.
Deng-Ping Fan, Ge-Peng Ji, Guolei Sun, Ming-Ming Cheng, Jianbing Shen, and Ling Shao. Cam- ouftaged object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2777-2787, 2020a.
Deng-Ping Fan, Ge-Peng Ji. Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, and Ling Shao. Pranet: Parallel reverse attention network for polyp segmentation. In International conference medical image computing and computer-assiiter intervention, pp. 263-273. Springer, 2020b.
Deng-Ping Fan, Ge-Peng Ji, Ming-Ming Cheng, and Ling Shao. Concealed object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10)x602-6042,, 2022. doi: 10.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1): 5232-5270, 2022.
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 3146-3154, 2019.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12259-12269, 2021.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficien transfer learning with diff prun- ing. arXiv preprint arXiv:2012.07463. 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffiient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0000 096.0. 2021.
Xinrong Hu, Xiaowei Xu, and Yiyu Shi. How to efficiently adapt large segmentation model (sam) to medical images. arXiv preprint arXiv:2306. 13731, 2023.
Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Hävard D Johansen. Kvasir-seg: seg: A segmented polyp dataset. In MultiMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5-8, 2020, Proceedings, Part II 26, pp. 451-462. Springer, 2020.
11

---

Published as a conference paper at ICLR 2024
Wei Ji. Jingjing Li, Qi Bi, Wenbo Li, and Li Cheng. Segment anything is not always perfect: An investigation of sam on different real-world applications. arXiv preprint arX11v2304.05750, 2023.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv preprint Stt1:22077.000399 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint (tt.::2344002453.. 2023.
Trung-Nghia Le, Tam V Nguyen, Zhongliang Nie, Minh-Triit Tran, and Akihiro Sugimoto. Anabranch network for camouflaged object segmentation. Computer vision and image under- standing, 184:45-56, 2019.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.088 18490899.. 2021.
Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu. Expectation- maximization attention networks for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9167-9176, 2019.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.001 2021.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35: 109-123, 2022.
Weicong Liang, Yuhui Yuan, Henghui Ding, Xiao Luo, Weihong Lin, Ding Jia, Zheng Zhang, Chao Zhang, and Han Hu. Expediting large-scale vision transformer for dense prediction without fine- tuning. Advances in Neural Information Processing Systems, 35:35462-35477, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 3431-3440, 2015.
Volodymyr Mnih. Machine learning for aerial image labeling. University of Toronto (Canada), 2013.
OpenAl. Gpt-4 technical report, 2023.
Namuk Park and Songkuk Kim. 200.099202.06799 2022.
How do vision transformers work?
arXiv preprint
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR. 2021.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116-123 2021.
12

---

Published as a conference paper at ICLR 2024
Sovit
Ranjan
Rath.
Leaf discase segmentation dataset, 2023.
URL
leaf Toeteaacssssssaatttt---a--aaaaae http://ec..cccttttttetoottaaaaaa
https: htw.p///ww....ttt.e..e..... com.
Carlos
Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts.
Advances in Neural Information Processing Systems, 34:8583-888 8595, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:
Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assistd Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part 18, pp. 234-241. Springer, 2015.
Tal Shaharabany, Aviad Dahan, Raja Giryes, and Lior Wolf. Autosam: Adapting sam to medical images by overloading the prompt encoder. arXiv preprint arXivv2306.06377 2023.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expprts layer. arXiv preprint arXiv: arXiv:1701.06536 2017.
Juan Silva, Aymeric Histace, Olivier Romain, Xavier Dray, and Bertrand Granado. Toward em- bedded detection of polyps in wee images for early diagnosis of colorectal cancer. International journal of computer assisted radiology and surgery, 9:283-293, 2014.
Przemystaw Skurowski, Hassan Abdulameer, J Blaszezyk, Tomasz Depta, Adam Kornacki, and Koziet. Animal camouflage analysis: Chameleon database. Unpublished manuscript, 2(6):7,
2018.
Apurva Sriwastwa, Shikha Prakash, Swati Swarit, Khushboo Kumari, Sitanshu Sekhar Sahu, et al. Detection of pests using color based image segmentation. In 2018 Second International Confer- ence on Inventive Communication and Computational Technologies (ICICCT), pp. 1393-1396. IEEE, 2018
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems, 35: 13991 2022.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Nima Tajbakhsh, Suryakanth R Gurudu, and Jianming Liang. Automated polyp detection in colonoscopy videos using shape and context information. IEEE transactions on medical imaging, 35(())630-664) 644, 2015.
Lv Tang, Haoke Xiao, and Bo Li. Can sam segment anything? when sam meets camouflaged object detection. arXiv preprini arXiv:2304.04709, 2023.
David Vázquez, Jorge Bernal, F Javier Sánchez, Gloria Fermández-Esparacee Antonio M López, Adriana Romero, Michal Drozdzal, Aaron Courville, al. A benchmark for endoluminal scene segmentation of colonoscopy images. Journal of healthcare engineering, 2017, 2017.
Tomás F Yago Vicente, Le Hou, Chen-Ping Yu, Minh Hoai, and Dimitris Samaras. Large-scale training of shadow detectors with moisily-annotated shadow examples. In Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pp. 816-832. Springer, 2016.
An Wang, Mobarakol Islam, Mengya Xu, Yang Zhang, and Hongliang Ren. Sam meets robotic surgery: An empirical study on generalization, robustness and adaptation. arXiv preprint arXiv:2308.00 07100..0756.... 2023.
13

---

Published as a conference paper at ICLR 2024
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference computer vision, pp- 568-578, 2021.
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):115 424, 2022.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 22 22-31, 2021.
Junde Wu, Rao Fu, Huihui Fang, Yuanpei Liu, Zhaowei Wang, Yanwu Xu, Yueming Jin, and Tal Arbel. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304. 12620, 2023.
Enze Xie, Wenjia Wang, Wenhai Wang, Mingyu Ding, Chunhua Shen, and Ping Luo. Segmenting transparent objects in the wild. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp. 696-711. Springer, 2020.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 14:17077 2021a.
Enze Xie, Wenjia Wang, Wenhai Wang, Peize Sun, Hang Xu, Ding Liang, and Ping Luo. Segmenting transparent object the wild with transformer. arXiv preprint arXiv:2101.0842 2021b.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1945-2955 2023.
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentionll image transform- ers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9981 9990, 2021.
Yongyang Xu, Zhong Xie, Yaxing Feng, and Zhanlong Chen. Road extraction from high-resolution remote sensing imagery using deep learning. Remote Sensing, 10(9):1461, 2018.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-modehs arXiv preprint wwX.vv2216.1011 2021.
Kaidong Zhang and Dong Liu. Customized segment anything model for medical image segmenta- tion. arXiv preprint ar arXiv:2304. 13785, 2023.
Yundong Zhang, Huiye Liu, and Qiang Hu. Transfuse: Fusing transformers and cnns for medical image segmentation. In Medical Image Computing and Computer Assisted Intervention-MiC MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Pro- ceedings, Part 24, pp. 14-24. Springer, 2021.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 2881-2890, 2017.
Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Zhifeng Li, Wei-Shi Zheng, Jonathan Li, and Alexander Wong. aaneeerz--nd--tteettin networks for semantic segmen- tation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13065-13074, 2020.
Tao Zhou, Yizhe Zhang, Yi Zhou, Ye Wu, and Chen Gong. Can sam segment polyps? arXiv preprint arXiv:2304.07558 2023.
14

---

Published as a conference paper at ICLR 2024
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114 2022.
Lei Zhu, Ke Xu, Zhanghan Ke, and Rynson WH Lau. Mitigating intensity bias in shadow detec- tion via feature decomposition and reweighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4702-4711, 2021.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts. arXiv preprint arXiv:21 2021.
15

---

Published as a conference paper at ICLR 2024
A GATING NETWORK OF CONV-LORA
In order to calculate the gating scores G(x) & BBx i.e., the scores for B samples in batch assigned & experts, we need to calculate the values (x) R Rax* of experts first. As the size of input x is B X r x H x W (we use I to represent Wax in eq. (2 for simplicity), we apply global average pooling (denoted as 'AvgPool") followed by a reshaping operation (denoted as 'Reshape') ) to obtain TΛ E R fxr, Specifically, the size of z is changed to B r x 1 x after 'AvgPool', and then changed to x after "Reshape". Then, following (Shazeer et al. 2017), , we apply the trainable gating weight Wg E T and the noise term W Woise E 1TTX to calculate (x)):
Th Reshape(AvgPooo(())....11
H(x); (oh W ) StandardNormal)) Softplus((IA) Wnoise)i)
(4)
We keep only the top values (denoted 'KeepTopK') based on Hx);; of each expert E,, setting the rest to -oc, whose corresponding gate values is equal to 0 after a "Softmax' operation (If G(z), need not compute (z)):
G(z) = (eetmaxxKKeTToop(HCCCC
if o2 is in the top & elements of where KeepTopK(1,, k), otherwise.
(5)
B MASK DECODER FOR MULTI-CLASS SEGMENTATION
SAM's mask decoder follows the design in mask-based segmentation, wherein the image is grouped into N (i.e., the number of output tokens in fig. 4 regions represented by binary masks. How- ever, unlike other mask-based methods such as Maskformer (Cheng et al. [021] and Mask2former (Cheng et al. 2022), SAM does not incorporate a classification module; solely identifies the fore- ground, lacking the ability for multi-class segmentation. What need to do is straightforward: incorporate a classification branch to obtain class predictions for the corresponding masks.
During training, need match the set of N mask predictions and the set of ground truth seg- ments. We follow the design in Mask2Former (Cheng et al., (022), which proposes a memory- friendly way for bipartite matching between the predictions and ground truths. It efficiently alleviates the memory demands in semantic segmentation tasks that necessitate high-resolution mask prediction, particularly for SAM involving the processing of high-resolution input images (1024 x 1024). For the masks that are not allocated to any ground truths, an extra 'no object' category introduced produce K + 1 class predictions, assuming there are originally K categories. For semantic infer- ence, we can exclude the 'no object' category and perform straightforward matrix multiplication between the masks and the classification predictions to obtain pixel-wise predictions.
ensure the one-to-one matching. Hence, the classification branch should
Following Former, we use binary cross-entropy loss Lee and dice loss Laice for mask loss, cross entropy loss Lets for classification predictions:
Lmaiti class
+ NotsLess Amaakk Aroo
(6)
where Lmask = Ace + Adsice Edice-
C DATASET DETAILS
Polyp Segmentation (medical images) Polyp Segmentation, the task to identify abnormal growths known polyps within gastrointestinal endoscopic images, plays critical role in early colorectal cancer diagnosis and treatment planning, and it presents a formidable challenge due to the consid- erable diversity in polyp shapes and sizes. We choose polyp segmentation datasets: Kvasir (Jhal
16

---

Published as a conference paper at ICLR 2024
et al. 2020) and CVC-ClnncDB/CVCC-611 (Bernal et al., 2015). Kvasir contains 1000 images and EVC-ClinicDB also called CVC-612, includes 612 open-access images. Fan et al. (202bb) divides the images into 9:1 ratio for training and testing. Additionally, we randomly divide a validation set comprising 20% of the images from the training set, for validation during training. Furthermore, also use images from CVC-ColonDB (Tajbakhsh al.] 2015), EndoScene (Vázquez et al. 2017) and ETIS (Silva et al., (0144) for testing following the setting in (Fan et al.) (2200). We consistently train all methods for 30 epochs, unless otherwise specified for specific datasets.
Skin Lesion Segmentation (medical images) Skin Lesion Segmentation involves the segmentation of various types of skin lesions within medical images, serving a crucial role in the early diagnosis and treatment of skin disorders, notably skin cancer. However, this task is particularly challenging due to ambiguous boundaries and color variations. We choose ISIC 2017 (Codella et al., 2018 for skin lesion segmentation. ISIC 2017 provides 2000 images for training, 150 images for validation and 600 images for testing.
Camouflaged Object Segmentation (natural images) Camouflaged Object Segmentation focuses on identifying objects that are concealed within complex or visually cluttered backgrounds, which is more challenging comparing to traditional object segmentation. We choose three camouflaged object detection datasets: COD10K (Fan et al. (020a)), CHAMELEON (Skurowski et al. 2018). and CAMO (L et al. 2019). CODIOK contains 3040 training and 2026 testing samples. CHAMELEON includes 76 images collected from the Internet for testing. And CAMO provides 1000 images for train on the combined dataset consists of the training images from CODIOK and CAMO for 20 epochs, and test on the three datasets. Addtionally, we randomly split 10% of the images from the training set for validation.
training and 250 for testing. Following (Fan al. 2020a).
Shadow Detection (natural images) Shadow Detection focuses on the recognition of shadow re- gions within a scene and can facilitate the estimation of lighting conditions or the removal of shad- ows. We choose SBU (Vicente et al. 2016). which is the largest annotated shadow dataset. SBU contains 4085 and 638 images for trainnn and testing. We randomly split 10% of the images from the training set for validation and we train the methods for 10 epochs with balanced binary cross entropy loss.
Leaf Segmentation (agriculture) Leaf segmentation involves the identification of individual plant leaves within agricultural images and plays a crucial role in advancing automation for plant diseases control and high quality food production. We choose a Leaf Disease Segmentation dataset (Rath) 2023), which contains 498 images for training and 90 images for testing. We randomly split the training images into 80% for training. 20% for validation.
Road Segmentation (remote sensing) Road segmentation detects road or street regions within im- ages video frames, and is crucial for applications in autonomous driving, traffic analysis, and urban planning. We choose Massachusetts Roads Dataset (Mnih) (013), which contains 1107 im- ages for training, 13 images for validation and 48 images for testing. And train the methods for 20 epochs.
Multiclass Semantic Segmentation (natural images) We choose Trans10K-v1 (Xie al.. 2020) and Trans 10K-v2 (Xie et al. 2021b) dataset for multi-class transparent object segmentation. Trans10K: vI dataset contains 10428 images, with background as one category and two more categories of transparent objects: Transparent Things (e.g., cups, bottles) and Stuff (e.g., windows). Trans 10K- v2 dataset is based on 1rans10K-vl dataset, with more fine-grained categories annotations. The dataset contains background plus two main categories divided into 11 fine tne-grained categories: 1) Transparent Things containing cup, bottle, jar, bowl and eyeglass. 2) Transparent Stuff containing windows, shelf, box, freezer, glass walls and glass doors. In respect to fine-grained categories and high diversity, Trans 10K-22 is more challenging than 10Kv.. All the datasets use 5000, 1000 and 4428 images for training, validation and testing, respectively.
D FULL EXPERIMENT RESULTS
Here are the full experiment results for binary semantic segmentation. Our Conv-LoRA demon- strates superiority across diverse datasets from various domains compared to other PEFT methods. Noted that models indicated in italics in the following tables are specifically designed for the cor- responding tasks. We re-run the experiments all the datasets using the model code provided by
17

---

Published as a conference paper at ICLR 2024
the authors. We choose methods that are either widely established relatively novel within their respective domains. With the exception of the Leaf Segmentation dataset, which is sourced from Kaggle, we utilize the author's s provided sample code to conduct our experiments.
Polyp Segmentation:
Method
4 Parame (MI
RRatie 9)
S, 1
M/p
Kasder
MAB
S,
Mgp
CVC-612
MAB
Profi Sonwtal 2000
100%
2.9
92.6
95.5
83.7
decodenunly
BilFa
3.96 (2,255
00.8
08.5
67.4
5.2.8:
80.0
91.5 mm
61.3 mm
2,3 3.3.B
Adaeee
3.92 11.615
91.2 103-6-00 cm
14.00 =4
10.3 0:
3.1
80.3 00
02.0
82.5 0::41
222.xxxx
VPT
4.00/0.675
91.5 cm
14.3 mm
00.0 cM
2.8
91.0--
01.7
84.8 1:1
21 41
LST
71.777
89.7
03.3 mm
86.9 mm
3.7
89.4 km
02.4 mm
83.7 0:00
244.5
AAM-Auleiiz
97.4
km
Corr
/0.755
85.5
ParsnccMi
CNC-Clomn38
CVC-T
S.1.
Fy
Fg
Proke Pa 2125
32.55/ 100%
820
94.
7009
400
79.3
90.6
58.2
2.3
93.9
97.1
85.2
deondrroely
3:51 10./0055
787
59.5
5.2
57.9
70.4
41.0
7.
M.4
36.4
Bitfi
81.8
868
15.
3.0
14.7 4mm
87
65.4
91.5
96.
813
Adgevr
83.6 4.
86 46
71.7 mm
85.3 6m
86 269--am
18 1...
92.5 mmm
94. mm
8633 6-0
13--am
SPT
4100/0675
819 mm
K73 km
719 8mm
86.1 kmm
00.0
18:
94.
54M-Adgqu
155 0775
km
16311
00.8
63.
233
92.1
81.
12:
SXP
878 1:4
86.9
73. mm
39 km
85.7 km
07.6
86.
17
92.1 km
919
83A km
14:
4101/0477
1m
07-
715 km
08.4
18
93.
Table 5: Quantitative results for Polyp Segmentation.
Skin Lesion Segmentation:
# Params (M) Ratioo %.)
ISIC 2017
Method
Jac f
T-Jac
Dice t
Acc t
Transfuse Zhang et al. 2021
26.30 / 100%
80.1
74.5
87.5
94.7
decoder-only
3.51 0.55%
69.7
56.6
79.5
91.2
BitFit
3.96 / 0.62%
76.4 + u.4
68.2 + am
84.7 = n.s
93.5 + um6
Adapter
3.92 / 0.61%
76.7 + nse
68.0 1m
85.0 + aM
93.6 + n.
VPT
4,00 / 0.62%
76.9 + use
68.4 1.4
85.1 - w/5
93.7
LST
11.49 / 1.77%
76.4 1.09
84.9 0:9
93.5
SAM-Adapter
3,98 0.62%
76.1 +
6m
84.6 +
93.4 +
SSF
4.42 / 0.69%
76.6 +
68.3 4: 0.2
85.0 = 014
93.6 6
LoRA
4,00 / 0.62%
76.6 + u.3
68.6 + 0.2
84.9 = uz
93.6 + u00
Conv-LoRA
4.02 0.63%
77.6 + 0ST
69.6 + mm
85.7 a.
93.9 m
Table 6: Quantitative results for Skin Lesion Segmentation.
Camouflaged Object Segmentation:
Method
Panees (M)/
Ratee (6)
S,
ArT.
CHAMMRLON
P7
MAR
8-1
CAMO
MARL
S-1
COBHOK
MARL
drondreedly
3.50 20.50 00.99%
87.5
90.5
19.3
57
78.5
3511
56.8
8.7
51.8
67A
70.5
1.7
Adens
392 10.11
93.. am
06.8
1.7.2
a.cm
91.3
82.8
91.
95.3 93.32.00
164.2200
VPT
4000-0.027
93.2-2.0 mm
96.1 2.
15.9 am
1.
$1.4 mm
91.4
82.1 cm
5.0 4.
91.1 km
949 4:
88.3 am
18:00
11.49/11
am
km
2::3
18.0
SIF
94.8
92.
1.5
91.4 2c
826
50
913
95.1 2:0
16.1
1m
LaRA
4.00-0.22
93.8 om
06.7 km
1.
5500 2:
01.92211
48 4.
91.5
05.2 2:2
864 864-20
17
Con-LLAA
402/0.04
984
15
2::0
Table 7: Quantitative results for Camouflage Detection.
Shadow Detection:
18

---

Published as a conference paper at ICLR 2024
Params (M)
SBU
Method
Ratio %
BER
FDRNer et al. 2021
10.77 100%
3.56
decoder-only
3.51 0.55%
14.58
BitFit
0.62%
3.16 = 0.12
Adapter
0.61
2.84 0093
VPT
4,00/ 0.62%
2.70 +
LST
11.49 1.77%
3.18 +
SAM-Adapter
3.98 0.62%
3.14 +
SSF
4,42 0.69%
3.19 1 0046
LoRA
4,00 0.62%
2.74 + 0079
Conv-LoRA
4.02 0.63%
2.54 + 009)
Table 8: Quantitative results for Shadow Detection.
Leaf Segmentation:
Method
Params (M) Ratio (%)
Leaf Dice
IoU
Acc
DeepLabu3 Chen al. 2017b
41.99 100%
62.3
74.1
94.0
oecoder-only
3.51 / 0.55%
50.8
63.8
89.6
BitFit
3.96 / 0.62%
71.4 +: L15
81.7 um
95.5
2 033
Adupter
3.92/ 0.61%
72.1 net
82.4 wss
95.3 0.3
VPT
0.62%%
73.6
83.8
LST
11.49 1.77%
70.2 +: wn
81.1 + nae
95.3 0m5
SAM-Adapter
3.98
71.4
82.1
0.5
SSF
4.42 0.69%
71.5 +
81.8
95.5
LoRA
4,00 / 0.62%
73.7 1 020
83.6 0.3
95.7 - 0.77
Cony-LoRA
4.02/ 0.63%.
74.5
034
0.30
Table 9: Quantitative results for Leaf Segmentation.
Road Segmentation:
Params (M)
Road
Ratio (%)
IoU f
Dice
Ace
LinkNer344MT Batra al. 2019
22.00 / 100%
73.0
decoder-only
3.51 0..5%%
48.6
65.1
96.4
BitFit
3,96 / 0.62%
60.6
75.2
97.5 esc
Adapter
3.92 / 0.61%
61.5 1
75.9
am
97.6 em
VPT
4.00 0.62%
60.2
97.4
LST
11.49 1.77%
60.2 am
74.9 az
97.5 am
SAM-Adapter
SSF
4.42 / 0.69%
61.6 am
76.0
97.6
LoRA
4.00 / 0.62%
62.2 -
76.5 aus
97.6 2
Conv-LoRA
4.02 0.63%
62.6
76.8
97.7
Table 10: Quantitative results for Road Segmentation.
E ADDITIONAL ABLATION
The Rank of LoRA. The performance of SAM for twelve-class segmentation could be improved by introducing more extra parameters (table [T]. However, this may disobey the rule of 'parameter efficiency" PEFT. Exploring the design a more efficient way for introducing "classification prior' for SAM is worthwhile endeavor for the future.
19

---

Published as a conference paper at ICLR 2024
Test
LoRA
#Params (M)
Acc
mloU f
3
4.00
94.80
66.01
F' = 6
4.49
95.15
66.24
/' = 12
5,48
95.23
66.69
21
7.44
95.14
67.02
Table 11: The performance trend when increasing the rank r of LoRA for tweee---lass
Glsba
Expert
Expert
Expert 3
Expert
Expert
Expert
Expert 2
Expert
Selected
UnSolected
(a) MoE (Dynamic)
(b) Multi-scale (Simultaneous)
Figure 6: Comparison between (a) MoE, which the gating network to dynamically select the experts, and (b) Multi-scale, which selects all the experts simultaneously.
Combined with Other PEFT Methods. We also evaluate the combination of Conv-LoRA and VPT on Poly Segmentation in table 12 This combination achieves superior or competitive performance, demonstrating the potential of introducing Conv-LoRA in addition to other PEFT methods. As future work, such combination might motivate techniques that further reduce the number of trainable parameters while ensuring the enhanced performance.
Params Ratio(%)
CVC-ColonDB
ETIS
Method
S
Es f
F f
MAE
Sg f
Es
Fy t
MAE 1
VPT
4.00 0.62%
85.0
88.4
75.0
3.6
86.5
87.4
70.1
1.9
Conv-LoRA
4.03 0.63%
85.1
88.8
75.7
3.4
86.8
88.5
70.2
1.7
Conv-LoRA+VPT
4.23 / 0.66 %
86.0
89.0
76.8
3.5
87.7
88.1
70.3
1.7
Table 12: Performance of combining Cony-LoRA and VPT on Polyp Segmentation.
MoE-Cony vs. Blocks with Various Kernel We further compare the design of our MoE-Cone and convolutional blocks with various kernel sizes. Convolutional blocks with various kernel sizes are commonly used in multi-scale feature fusion. Here we follow the Inception module proposed by (Szegedy al. 2015), which consists of multiple kernel sizes with x 1, 3 x 3, 5 x 5. ensure fair relative comparison, opt for four experts in MoE-Conv to align with the branch number of the Inception structure. We conduct experiments on Leaf Dataset and ISIC 2017 Dataset twice for evaluation.
# Params / /Ratio %
Leaf Dice
ISIC 2017
Method
IoU 1
Acc
Jac 1
T-Jac' f
Dice
Acc
Inception
4.01 0.63%
72.1
82.3
95.0 95.6
76.9
68.5
84.9 85.5
93.3
MoF-Conv
4.01 0.63%
74.0
83.8
77.4
69.5
93,9
Table 13: MoB-Conv vs. Blocks with Various Kernel sizes (indicated as 'Inception')
20

---

Published as a conference paper at ICLR 2024
MoE-Com outperforms the design of convolutional blocks with various kernel sizes. We attribute this to convolutions with various kernel sizes only operate on the default feature scale. Maybe larger kernel size can inject some approximate local prior in features of smaller scale, but it probably can't 't bring local prior into features of larger scales than the default. The features in ViT downscaled 16x from the original image, so injecting priors in larger scale features are generally more useful, as evidenced in table where the optimal scale is usually larger than the default scale.
In table 13
Computational Cost. In table 14 we compare the training/ inference speed and per epoch training time with the ISIC 2017 dataset and a single V100 GPU. While Conv-LoRA is slower in speed than others, it achieves robust performance gains across various semantic segmentation tasks (table I We find that the main cost comes from the upscale and downscale operations. A possible future direction is exploring how to inject local prior without explicitly scaling up and down features. We also notice that there are some other orthogonal works, like token merging (Liang et al. 2022 Bolya al. 2022). that may accelerate Conv-LoRA.
Training min/epoch
Inference ited's
Method
iter/s
BitFit
1.82
18.3
4.62
Adapter
1.73
19.3
4.38
VPT
1.41
23.6
4.39
LoRA
1.64
20.3
4.62
Conv-LoRA
1.22
27.3
3.64
Table 14: Computational cost comparison.
F LOCAL PRIOR ANALYSIS
We elaborate on 'SAM's local prior assumption' mentioned in our abstract in section 4.3 Addition ally, as our Conv-LoRA is designed to reinforce existing inductive biases within the features, we investigate the presence of these biases in the features.
We use two metrics: Mean attention distance, which reflects the extent of local or global in- formation that self-attention layer is aggregating (Dosovitskiy et al.) 2020 Raghu et al.) (021] We calculate the attention distance for each attention head, defined as the average distance between the position of the query patch and the locations to which it attends, weighted by the attention weights. Then we calculate the average attention distance for each layer by computing the mean across 500 randomly sampled images from downstream semantic segmentation tasks. 2) Relative log amplitudes (Park& & Kim 2022). which reflects whether the model tends to reduce or amplify high-freque high-frequency signals (e.g., edges, textures) in feature map. We compute the relative log ampli- tudes of the Fourier-transfornet feature map in each layer and average them across layers. Then we calculate the mean of the relative log amplitudes over 100 randomly sampled segmentation images.
Inductive biases of the features. Extensive pre-training on large-scale datasets has been demon- strated to equip ViTs with inductive biases (Dosovitskiy et al. 2020 Raghu et al. 202I). SAM's large-scale segmentation pretraining further amplifies these inductive biases within the features.
To evaluate the inductive biases of the features, we conduct comprehensive comparisons among randomly initialized ViT, MAE pretrained ViT, and SAM ViT. We use mean attention distance and relative log amplitudes for evaluation. fig. 7 as the training progresses from randomly initialized ViT to MAE ViT and ultimately to SAM VIT. we observe consistent trend: the mean attention distance decreases, while the high-frequency signals in the features increase, indicating focus on local information. These findings further confirm the presence of inductive biases in features following large-scale pre-training. Furthermore, the findings affirm the rationality of Conv- LoRA in reinforcing the inherent inductive biases within the features.
G MoE ANALYSIS
What does MoE learn? MoE learns to dynamically select an appropriate scale for injecting local priors based on input features. To illuminate its functionality, conduct analysis by
21

---

Published as a conference paper at ICLR 2024
Models
Models
SAM
MAE (Random Init)
Mean Attention Distance
4 Log Amplitude
SAM
MAE (Random Init.)
Layer Depth
Frequency
Figure 7: Evaluation of inductive biases within features. Left: Mean attention distance average over all attention heads. Right: Relative log amplitudes of Fourier transformed feature maps. Compared to randomly initialized VIT. SAM and MAE could learn the inductive biases from large-scale data pre-training. And SAM could further amplifies these biases within the features.
tracking the frequency of each expert's selection across different datasets during inference. The detailed results presented in fig.
Notably, distinct datasets exhibit preferences for different experts. For instance, in Leaf Segmenta- tion, MoE tends to favor experts with upsampling ratios of 3 and 4, while the ISIC 2017 dataset, it tends to select the expert with an upsampling ratio 2. Connecting these insights with the optimal scale ablation results in table 4 it become evident that MoE selects the expert adaptively for each sample, and consequently, the distributions of MoE selection across datasets reflect their different data distributions. This observation supports that MoE behaves adaptively and effectively with spect to each sample's property. This adaptability reinforces the significance of MoE tailoring its selection the unique characteristics of diverse datasets, enhancing effectiveness in local prior injection.
Leaf
ISIC 2017
350
Selected Times
Selected Times
000
Expert Index
Expert Index
Figure 8: The number of times each expert is selected during inference for on Leaf Dataset and ISIC
2017 Dataset. Distinct datasets exhibit preferences for different experts.
22

---

Published as a conference paper at ICLR 2024
H LOW-DATA REGIME EXPERIMENTS
Convolutional layers could introduce inductive biases, thereby enhancing data efficiency during fine- luning. Therefore, we undertake experiments in low-data regime validate whether Conv-LoRA can indeed improve data efficiency in semantic segmentation.
Specifically, we conduct experiments under the few-shot setting, wherein acquiring data for down- stream tasks is challenging, and only a limited number of training samples per task are available. Our experiments are performed on the Trans 1rans10K-v2 dataset, which used for twelve-class trans- parent object segmentation. We randomly select the training samples, to meet the settings of 1, 2, 4, and 16 shots (i.e., the number of labeled training examples per class), and the experiments are run for 100 epochs.
# Params Ratio %
Method
Shot
mloU 1
LoRA
4.00 / 0.62% 4.02 / 0.63%
1/2/4/8/16 1/2/4/8/16
13.32/18.01/2227002538383
Conv-LoRA
1455/1988123.0575622//.8.
Table 15: Results of few-shot learning on multi-class semantic segmentation. 'Shot" indicates the number of labeled training examples per class.
In table 15 the performance improvement of Conv-LoRA compared to LoRA, is particularly pro- nounced an extremely low-data setting (e.g., I-shot). The result demenstrates that the introduc- tion of inductive biases in Conv-LoRA contribute to improved data efficiency.
I MORE VISUALIZATION RESULTS
Cony-LoRA vs. LoRA Feature. We visualize the feature maps from the fine-tuned SAM's image encoder, which incorporates LoRA and our Conv-LoRA respectively. fig. g the image features from SAM's image encoder equipped with Conv-LoRA could provide more fine-grained informa- tion, e.g., slim edges, which is beneficial to later mask prediction. This further demonstrates the effectiveness of reinforcing the amage-related local priors with network architecture.
Road Segmentation
Multiclass Transparent Object Segmentation
RGB Image & GT
LoRA
Conv-LoRA
RGB Image & GT
LoRA
Conv-LaRA
Figure 9: Feature map visualization for binary-class and multi-class semantic segmentation. Com- pared to LoRA, applying Conv-LoRA to SAM's image encoder could capture more fine-grained details.
Mask prediction. We also provide more visualization results for the mask prediction across var- ious datasets when applying different PEFT methods (VPT, LoRA and Conv-LoRA). This further confirms the superiority our Conv-LoRA.
23

---

Published as a conference paper at ICLR 2024
RGB Image
GT
VPT
LoRA
Conv-LoRA
Figure 10: Visualization Results Skin Lesion Segmentation.
RGB Image
VPT
LoRA
Conv-LoRA
Figure 11: Visualization Results on Camouflaged Object Segmentation.
24

---

Published as a conference paper at ICLR 2024
RGB Image
GT
VPT
LoRA
Cony-LoRA
Figure 12: Visualization Results on Leaf Disease Segmentation.
RGB Image
GT
VPT
LoRA
Conv-LoRA
Figure 13: Visualization Results Remote Sensing.
F
RGB Image
GT
VPT
LoRA
Conv-LoRA
Figure 14: Visualization Results on Multi-class Transparent Object Segmentation.
25


Reviews:
[{"review_id": "uEDcG8Qk8V", "reviewer": "Reviewer_W4sK", "summary": "In \"Convolution Meets LoRA\" the authors introduce a method for parameter-efficient finetuning of the Segment Anything Model (SAM) in specialized domains where it may initially underperform. The approach's effectiveness is demonstrated across a diverse range of datasets and is rigorously compared against a substantial set of baseline methods.\n\nThe method, referred to as Conv-LoRA, involves the incorporation of a modified version of low-rank adaptation (LoRA) into SAM's image encoder. Conv-LoRA improves upon LoRA by introducing a convolutional layer at its bottleneck, and includes a Mixture of Experts module to dynamically select the convolutional layer's scale of operation. Furthermore, the authors extend SAM's functionality to address multi-class segmentation tasks without the need for explicit prompts, allowing it to be deployed in an end-to-end setting.", "questions": "1. As discussed above, I am not fully convinced of the author's explanation for the improved performance they see with Conv-LoRA compared to LoRA. My understanding is that while ViTs learn about locality in images, this is not a prior that is built into the architecture. What do the authors mean by \"SAM's local prior assumption\" mentioned in the abstract? How can \"image-related inductive biases\" be reinjected on top of features that do not have those biases?\n2. I don't follow the author's reasoning for why SAM's foreground-background pretraining is insufficient (see above). Can you elaborate on this?\n3. My understanding of the cited work Zhang&Liu 2023 is that they also adapt SAM to work without an image-dependent prior and they also extract semantics from the segmentation head. Similarly, Shaharabany, Tal, et al. adapt SAM to work fully automatically. Please make sure to not misrepresent these works in the introduction (see above).\n4. I am confused about the author's choices with regards to the scaling for the convolutional layers. What is the interpolation method used? Is it learned? If not, can the larger upsampling factors do much here with the chosen kernel size of 3x3? Adding a row to table 4 with a scaling factor of 8 would provide valuable insights. \n\n## Minor Suggestions for Improvement\n5. Can you clarify where exactly in the image encoder ViT the Conv-LoRA bypass is added?\n6. In the introduction the authors profess that SAM is underperforming on certain domains. The references supporting this claim can be found in the Related Work section (Tang et al., 2023; Ji et al., 2023; Zhou et al., 2023). I suggest that the authors add these references to the first time the claim is made.\n7. The authors identify the adaptation of SAM for end-to-end multi-class segmentation as one of the major contributions of their work. I suggest that the authors add a paragraph to the related work section discussing other efforts to do so if there are any.\n8. Figures 8, 9, 10 should have their own labels.\n9. Make sure all variable names are defined and used consistently (Figure 4, Appendix A). I wasn't able to follow everything in appendix A. (e.g., where does B come from when looking at eq. 2?)\n10. I suggest adding a figure with example images for the medical domain because it is featured prominently in the abstract.\n11. Consider spelling out low rank adaptation in the abstract before abbreviating it. This would make the paper more welcoming to readers that are new to the field.\n12. I wanna encourage the authors to consider releasing their code. This would not only facilitate further research and collaboration in the field but also help realize the impact of their work by making it easier for users to adapt SAM to new domains.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper aims at solving the important problem of adapting a foundation model for computer vision to new, specialized domains. The authors propose Conv-LoRA - a new method for parameter-efficient fine-tuning (PEFT) of the segment anything model (SAM), which allows the adaptation of SAM to new domains where it may initially underperform. The combination of low rank adaptation (LoRA) with convolutions at various scales is a novel concept and a valuable contribution to the field of parameter-efficient finetuning for vision transformers.\n\nThe authors provide a substantial assessment of the quality of their work by showing a small but robust improvement over various baselines in a diverse set of datasets. While the relationship to some prior work needs to be elaborated upon, the paper includes a good overview of current efforts for PEFT. The authors improve the reliability of their results by running most experiments three times, thus reducing the likelihood of spurious effects stemming from random initialization.\n\nThe paper is generally well-structured and easy to follow. The authors effectively convey their approach and findings to the reader with only minor clarifications needed (as noted in the reviewer comments).\n\nWith the introduction of Conv-LoRA this paper not only expands the applicability of SAM to a wider range of datasets but also introduces a new method of PEFT that could be applicable to Vision Transformers more generally.", "weaknesses": "The paper convincingly demonstrates the effectiveness of the PERF method it introduces, but its motivation and explanation for why it works is unintuitive to me and the supporting evidence is insufficient. The authors claim that adding a convolutional layers reintroduces the inductive biases that are helpful in the image domain and hard-coded into convolutional layers. However, in Conv-LoRA, the convolutional layers are not applied to images but on features that do not necessarily adhere to the locality prior by construction. Can the locality prior truly be reintroduced if it might have already been lost, or, should this rather be regarded as a data-efficient method for finetuning that utilizes the learned locality of the early features in the ViT (see e.g. Raghu, Maithra, et al. 2021)? Unless my understanding of this problem is lacking, the explanation for the good performance of Conv-LoRA should be reformulated as a hypothesis.\n\nSimilarly, the authors identify \"SAM's foreground-background segmentation pretraining\" as a weakness, but SAM actually outputs three (prompt-dependent) masks with the idea of allowing the network to identify an object hierarchy (whole, part, sub-part) for ambiguous prompts. To me that seems closely related to multi-class segmentation and requires understanding of the image semantics. I think the paper could be strengthened by elaborating on and providing evidence for this deficiency of SAM.\n\nWhile the paper includes an overview of other efforts for parameter-efficient finetuning (PERF), a clarification on what sets it apart from other work on PERF of SAM would strengthen their work. Specifically, the authors mention the work of Zhang&Liu 2023 (SAMed), which also uses LoRA to adapt SAM to a new domain (medical images) and from my understanding also repurposes SAM to work for multi-class segmentation in an end-to-end fashion. Considering that Zhang&Liu 2023 is a very recent work and has not been published in a peer-reviewed venue, a direct comparison cannot be expected. However, the authors should revisit their claim in the introduction that Zhang&Liu (2023) fail to address SAM's limitation of not capturing \"high-level image semantic information\" (point 2) and functioning as a \"standalone, end-to-end solution\" (point 3). Similarly, the authors cite Shaharabany et al., 2023, who also adapt SAM to work fully automatically (point 3).\n\nIn summary, the authors show through their extensive set of experiments that their method is an effective way of performing PEFT of SAM to difficult domains and is therefore a valuable contribution. The concerns above are only with regards to the representation of prior work and the explanation of why Conv-LoRA is effective, not with the soundness of the method itself and the scientific rigor in showing its effectiveness. If the authors can address these concerns in the discussion or, where applicable, with minor edits to the language in the paper, I recommend accepting this work.\n\nRaghu, Maithra, et al. \"Do vision transformers see like convolutional neural networks?.\" Advances in Neural Information Processing Systems 34 (2021): 12116-12128.\nZhang, Kaidong, and Dong Liu. \"Customized segment anything model for medical image segmentation.\" arXiv preprint arXiv:2304.13785 (2023).\nShaharabany, Tal, et al. \"AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encoder.\" arXiv preprint arXiv:2306.06370 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5AvJDP6NlV", "reviewer": "Reviewer_3rwQ", "summary": "The authors proposed a parameter-efficient fine-tuning approach, i.e., a Conv-LoRA module combining trainable convolutional parameters with MOE scheme. It is developed to overcome the SAM’s performance drop when applied to specialized domains such as medical imagery and remote sensing. The Conv-LoRA module is integrated into the plain ViT encoder, enhancing SAM’s local prior assumption and its ability to learn high-level image semantics. Several previous parameter-efficient fine-tuning approaches are included in the comparison study. The proposed method enables efficient adaptation (with superior results) to real-world semantic segmentation tasks across various benchmarks and domains.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ a novel LoRA-like add-on module for efficient parameter tuning for SAM, a typical large-vision model. \n+ Superior results of the proposed methods are reported in comparison to vanilla LoRA, VPT, and other adaptor-based methods.", "weaknesses": "- The motivation for introducing the combination of MOE with convolutional parameters as a couple is not clear to me. It is not clear how each of them will benefit the performance as an add-on to the vanilla LoRA.\n- The proposed method reminds me of the inception structure from the GoogLeNet. What will be the difference between the design of multiple down-scale+conv+up-scale blocks and convolutional blocks with various kernel sizes? Is MOE really necessary here? Will simple addition(or average) work?\n- It will be helpful to clarify how different the training procedure with the add-on module will be in comparison to the original SAM training process.\n- It will also be helpful to have a comparison in computational cost for those parameter-efficient tuning methods.\n- As shown in Table 4, the scales vary amongst different datasets. Will this require extra tuning efforts for picking a suitable scale (experts)? Again, will the way how different sizes are combined in the inception structure be a better option?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qfpWgapXLJ", "reviewer": "Reviewer_LN6g", "summary": "This paper proposes an improvement to the Segment Anything Model (SAM) by incorporating fittings to alleviate some of the original model's deficiencies, together with algorithmic novelties to improve performance. Improvements to SAM are claimed in specialized domains (e.g medical images) and in improving fine grained semantic predictions. Methods are proposed to build upon LoRA (Low Rank Approximation) from NLP, which is modified to incorporate a Mixture of Experts setup with convolutional processing (what they call conv-LoRA). This way, it is claimed that the model is able to aggregate signals from multiple experts that can learn image priors in a specialized way, all while learning with minimal computational overhead, or what they call parameter efficient fine tuning. A few other modifications are used to boost performance and usability (freezing prompts, allowing for a classification head for multiclass classification) over SAM. \n\nResults show improvements over other parameter efficient fine tuning models (e.g. VPT), with little overhead from the model's conv fittings.", "questions": "Ablations: I am curious about the effect of various model components on performance. \n- MoE looks a bit opaque. What priors is it learning, and how do we grasp what it is doing? \n- Can the authors give more reasoning for freezing the prompt encoder? I quite liked the prompt encoder in the original model.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Solidly written paper, well motivated ideas for SAM improvement\n+ Novelty in the use of MoE\n+ Improvements in producing fine grained classification (e.g. edges), and in specialized domains", "weaknesses": "- Purely empirical work. No reasoning is given as to how the model improves performance. \n- MoE is not well motivated. I looked up the original paper for insight, which clarifies things somewhat, but the paper in question should describe it better. \n- Performance improvement is very marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7eby8KEN9c", "reviewer": "Reviewer_2Lzf", "summary": "This paper is an improved version of LoRA (low-rank adaptation) that aims to adapt pre-trained Segment Anything Models (SAM) across a diverse array of semantic segmentation tasks. The proposed Conv-LoRA adds an extra convolution operation between the encoder and decoder. Freezing the pre-trained SAM, the encoder, decoder, and extra convolution are trainable to adapt SAM to downstream semantic segmentation tasks. The trainable part of this framework is lightweight. Compared with LoRA and other parameter-efficient fine-tuning approaches, the proposed Conv-LoRA shows better performance for semantic segmentation tasks in medical natural imaging, agriculture, and remote sensing.", "questions": "1. The Mixture of Experts in introduced in the paper seems to be multi-scale strategies that are commonly used in image segmentation. It is unclear whether the added complexity of this method is warranted. A more straightforward explanation would likely render the technique more impactful and accecible.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The manuscript is composed with clarity, presenting a concept that is both coherent and well-motivated.\n+ The experimental setting is clearly stated, and the authors have conducted comparisons with not only LoRA but also a wide array of baseline methods.\n+ The scope of experimentation is thorough. The proposed Conv-LoRA notably surpasses LoRA and a range of other methods aiming for efficient fine-tuning. It also appears to outperform full fine-tuning 100% parameters.\n+ It appears that incorporating convolution operations can improve the SAM features in terms of fine-grained information, such as slim edges, and semantic information (this might be due to fine-tuning).", "weaknesses": "- Lack of comparison with training segmentation model from scratch (random initialization). It remains unclear whether the pre-trained SAM model helps transfer to downstream semantic segmentation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)