MTSAM: Multi-Task Fine-Tuning for Segment Anything Model
Authors: Xuehao Wang
Zhan Zhuang
Feiyang Ye
Yu Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-task learning
segment anything model
low-rank adaptation
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose MTSAM, a framework that extends SAM as a foundation model for multi-task learning.

Abstract: The Segment Anything Model (SAM), with its remarkable zero-shot capability, has the potential to be a foundation model for multi-task learning. However, adopting SAM to multi-task learning faces two challenges: (a) SAM has difficulty generating task-specific outputs with different channel numbers, and (b) how to fine-tune SAM to adapt multiple downstream tasks simultaneously remains unexplored. To address these two challenges, in this paper, we propose the Multi-Task SAM (MTSAM) framework, which enables SAM to work as a foundation model for multi-task learning. MTSAM modifies SAM's architecture by removing the prompt encoder and implementing task-specific no-mask embeddings and mask decoders, enabling the generation of task-specific outputs. Furthermore, we introduce Tensorized low-Rank Adaptation (ToRA) to perform multi-task fine-tuning on SAM. Specifically, ToRA injects an update parameter tensor into each layer of the encoder in SAM and leverages a low-rank tensor decomposition method to incorporate both task-shared and task-specific information.
Extensive experiments conducted on benchmark datasets substantiate the efficacy of MTSAM in enhancing the performance of multi-task learning. Our code is available at https://github.com/XuehaoWangFi/MTSAM.

Published as a conference paper at ICLR 2025
MTSAM: MULTI-TASK FINE-TUNING FOR SEGMENT ANYTHING MODEL
Xuehao Wang', Zhan Zhuang": 1,2 Feiyang Ye Yu Zhang!*
Southern University of Science and Technology 2City University of Hong Kong
University of Technology Sydney
(xuehaowangfi. feiyang. ye.....ye......... u yu. zhang. ust 6 ggmaml com 1225006 63@mail. sustech. edu. edu.cn
ABSTRACT
The Segment Anything Model (SAM), with its remarkable zero-shot capability. has the potential be foundation model for multi-task learning. However, adopting SAM to multi-task learning faces two challenges: (a) SAM has difficulty generating task-specific outputs with different channel numbers, and (b) how fine-tune SAM to adapt multiple downstream tasks simultaneously remains unex- plored. To address these two challenges, in this paper, we propose the Multi-Task SAM (MTSAM) framework, which enables SAM to work foundation model for multi-task learning. MTSAM modifies SAM's architecture by removing the prompt encoder and implementing task task-specific no-mask embeddings and mask decoders, enabling the generation of task-specific outputs. Furthermore, we intro- duce Tensorized low-Rank Adaptation (ToRA) perform multi-task fine-tuning on SAM. Specifically, ToRA injects an update parameter tensor into each layer of the encoder SAM and leverages low-rank tensor decomposition method to incorporate both task-shared and task-specific information. Extensive exper- iments conducted on benchmark datasets substantiate the efficacy of MTSAM in enhancing the performance of multi-task learning. Our code is available at https://ithub.com///eeowwwaw Wan
INTRODUCTION
Empowered by large-scale datasets and computational advancements, large foundation models have revolutionized natural language processing and multi-modal learning, exhibiting remarkable zero- shot capabilities (Radford et al. 2018 2019; Kenton & Toutanova 2019 Lewis et al.) 20200] Brown al. 2020 Radford et al. 2021 Wei al. 2023] Jiang al. 2024). Recently, the Segment Anything Model (SAM) (Kirillvv et al., 2023). J. a foundation model in computer vision for image segmentation, achieves exceptional zero-shot performance through training on a large-scale dataset of 11 million samples. Efforts have been dedicated to expanding the zero-shot capability of SAM to various tasks, including high-quality segmentation (Ke al., 2023). 3D reconstruction (Cen al. 2023 object tracking (Yang et al., (002)), medical image processing (Ma et al., 2024;) Huang et al. personalize segmentation (Zhang et al., 2023), and remote sensing (Shankar et al.) 2023).
2024
Though SAM has achieved remarkable performance in diverse tasks in previous studies, those stud- ies only adopt SAM to specific downstream task by single-task learning, while overlooking the potential of employing SAM the foundation model for multi-task learning. In many real-world computer vision applications, there is usually more than one task that can be considered simultane- ously, such as depth estimation and surface normal estimation tasks in dense scene understanding, Previous works on multi-task learning (Misra et al., 2016 Liu et al., 2019 Ye & Xu 2022 2023] Zamir al. 2018 Liu al.] 2022) have shown that these tasks relevant and benefit each other during the training process. This insight motivates us to adopt SAM as foundation model for multi-tasking learning to enhance the performance of different tasks. However, adopting SAM
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Input 3 X W
Mask 1 x x W (a) Outputs Original SAM.
Mask 2 x x
Mask 3 1xH x W
Input x H x W
Depth
Segmentation 13 x H x W
Surface Normal 3 x H x W
1 x x W
(b) Outputs of MTSAM for multiple tasks.
Figure 1: Comparison between the outputs of (a) the original SAM and (b) the MTSAM proposed in this paper. The original SAM generates segmentation results at three distinct levels, all featuring an identical number of channels. contrast, the MTSAM can simultaneously produce outputs for multiple tasks, utilizing varying numbers of channels.
to multi-task learning presents two challenges: (a) how to generate outputs with varying dimensions for each task by SAM; and (b) how to fine-tune SAM to adapt multiple tasks simultaneously.
To tackle the above challenges, we propose the Multi-Task SAM (MTSAM) framework leverage the rich knowledge from SAM for multi-task learning. Our approach includes two key modifica- tions: (a) adapting the architecture of SAM to accommodate varying channel numbers for each task and (b) introducing novel multi-task parameter-effictt fine-tuning method named Tensorized low-Rank Adaption (ToRA) for fine-tuning the SAM encoder.
Specifically, incorporate task embeddings and remove the prompt encoder from the mask de- coder, enabling the generation of the outputs with dimensions tailored to each task. As illustrated in Figure [ this modification enhances the flexibility of SAM, allowing it adapt various tasks within a unified architecture. For multi-task fine-tuning, ToRA injects an update parameter tensor into each layer of the SAM encoder, with each slice the tensor serving as an update parameter matrix for the corresponding task. Inspired by Low-Rank Adaptation (LoRA) (Hu et al. 2021), we assume low-rank structure for each update parameter tensor and apply low-rank tensor decompo- sition to capture both task-shared and task-specife information. Theoretically, we prove that ToRA's expressive power in multi-task learning surpasses that of LoRA. ToRA exhibits superior parameter efficiency, with sublinear growth in learnable parameters with respect to the number of tasks, in contrast to the linear growth the original LoRA method when applied directly to multiple tasks.
The main contributions of this paper are three-fold. (i) We propose MTSAM, novel multi-task learning framework that extends the capabilities of SAM to perform multi-task learning. Specifi- cally, we modify the original architecture of SAM by removing the prompt encoder and adding task embeddings. This modification enhances the flexibility of the original SAM. (ii) We introduce ToRA, novel multi-task PEFT method, that applies low-rank decomposition to the update param- eter tensor, effectively learning both task-shared and task-specific information simultaneously. with theoretical analysis of its strong expressive power. (iii) We conduct comprehensive experiments on benchmark datasets, demonstrating the exceptional performance of the MTSAM framework.
2 RELATED WORKS
Application of Segment Anything Model (SAM). The remarkable zero-shot generalization ability exhibited by SAM showcases its immense potential for both research and industrial applications.
2

---

Published as a conference paper at ICLR 2025
This potential has captured the attention of researchers, leading to numerous attempts to explore and hamness its capabilities for various downstream tasks, including all purpose matching (Liu et al, 2023b), high-quality segmentation (Ke et all, 2023), 3D reconstruction (Cen et al., (023). object tracking (Yang et al. 2023), medical image processing (Ma et al.] 2024 Huang et al. 2024), per- sonalize segmentation (Zhang et al. (023), , and remote sensing (Shankar al. (023). In contrast to those modifications targeted at single downstream task, the proposed MTSAM aims to simultane- ously learn multiple downstream tasks and extract semantic knowledge from SAM to enhance the
performance of multi-task learning.
Parameter-Es Efficient Fine-Tuning (PEFT), To address the parameter and computational efficiency concerns during fine-tuning of large-scale pre-trained foundation models, various PEFT methods have been proposed, including adapter- -based methods (Houlsby et al.
2019
Lin et al.
2020
prompt tuning methods (Li & Liang] 2021 Lester et al. [021] Chen et al.
2024
Jiang et al.
2023
and LoRA-based methods (Zhong et al. 2023 Kopiczko et al. 2023] Wu et al. 2023
Hu
et al.
1 Ding et al. 2023 Valipour et al. [023 Guo et al.) 2025) Specially, LoRA (Hu et al.
2021
introduces trainable low-rank matrices into transformer layers to approximate update parameter ma- trix, Conv-LoRA (Zhong et al.
2023) inserts Mixture of Experts (MoE) (Jacobs et al. [991]) inside
the bottleneck of LORA, VeRA (Kopizkoo et al., (023) fixes the low-rank matrices and only tunes two vectors, MoLEE (Wu al.
2023) directly uses multiple LoRA which combined by gating func-
tion, SoRA (Ding et al. 2023) enables dynamic adjustments of the rank by using a gate unit, and
DyLoRA (Valipour et al. 2023) follows the idea of nested dropout to train LoRA wide range of ranks. Those methods achieve competitive performance and high parameter efficiency in single-task fine-tuning. However, those methods are not suitable for multi-task learning settings, since they do not consider shared information between multiple tasks. In contrast, the proposed ToRA method can leverage shared information to enhance fine-tuning performance across various tasks.
Multi- Task Learning (MTL). As widely used paradigm, MTL aims to improve the average per- formance of a model by simultaneously learning multiple downstream tasks. To enhance the efficacy of learning multiple tasks simultaneously. some studies focus on decoupling task-shared and task-
specific information through manual design (Misra et al., 2016 Liu et al. 2019 Ye & Xu
2022
2023 Gao et al., 2019; Lin et al.] 2024bb
automatic architecture learning (Guo et al.
2020
Huang et al. 2018 Raychaudhuri et al.) 2022 Sun et al.) 2020). Other approaches propose bal lacc
ing the losses gradients of different tasks during training avoid conflicts between them Chen
let 2018 Yu et al. 2020; Liu et al.] [ Ye et al.] [202]] Navon et al.] [202]] Ye et al.
2023
Lin et al., 2023). Additionally, some works employ task grouping techniques to select related tasks
for joint model training (Fifty et all 202T Song et al., 2022, Standley et al. 2020 Zamir et al.
2018). With the impressive generalization capability of large-scale pre-trained foundation models on downstream tasks, various multi-task purameter-efficien fine-tuning methods (Liu et al.) 2022
2023a) have been proposed. For example, Polyhistor (Liu et al. [022) designs a lightweight hyper- networks for hierarchical vision transformer, and HiPro (Liu et al.] 2023a) uses hierarchical prompt tuning to adapt pre-trained vision-language models. Different from the previous works on multi-task learning, we leverage the powerful SAM and propose the MTSAM framework which uses novel method ToRA to fine-tune the encoder.
3 METHODOLOGY
In this section, we introduce the proposed MTSAM framework and ToRA method.
3.1 PRELIMINARIES
SAM. The original SAM consists of three main modules: a heavyweight image encoder, prompt encoder, and a lightweight mask decoder. Given an image I E 323* where H and W denote the height and width of the image I, respectively. SAM first utilizes the image encoder E, to extract image features F1
R Dx H x W as
(1) where D denotes the dimension of the hidden state. Then the prompt encoder, which consists of a dense mask encoder Em and a sparse prompt encoder Ep, encodes dense masks M E R 1x 4 * and different types of sparse prompts P (i.e., points, box, and text) into mask features FM and
Fi = (T),
3

---

Published as a conference paper at ICLR 2025
prompt features Fp
FM = (M) Fp Ep(P),
(2)
where FM, Fp E D After that, the image features are summed with the mask features, and the prompt features are concatenated with some learnable prompt features Fr. Finally, the mask decoder Da will predict the final segmentation mask output E R f w attention-base -assd feature interactions on image features and prompt features as
by performing
0 = (M + FMS [L. Fp)).
(3)
2022) assumes that each update parameter matrix has low in- trinsic rank and fine-tunes them by freezing the pre-trained model. Formally, for given task, LoRA parameterizes update parameter matrix AW Ra*k corresponding to pre-trained parameter matrix Wo E Raxk by the product of two low-rank matrices, i.c., AW = BA, where E Rdxr and A R with < min(d, k)) Thus, for an imput x, the output h can be calculated by
LoRA. The LoRA method (Hu et al.
W = Wor + AWx = Wor + BAz,
(4)
where the W E Rdxk denotes the parameter matrix after the update. LoRA has been proven as an efficient and effective approach in tinettnning large pre-trained models for specific downstream consider LoRA
tasks. Therefore, two different approaches to directly applying LoRA to the multi-task fine-tuning setting for SAM. One approach uses hard parameter sharing strategy, where all tasks use shared LoRA matrix AW, and we call it LoRA-HPS. However, this hard parameter-sharine strategy may lead to imbal- anced performance on all the tasks due the competition among tasks for the shared LoRA (Zhang & Yang, 2022). Another approach is to train task-specific LoRA for each task and hence each task t uses its own A AW, and we call it LoRA-STL. However, this approach cannot harness the inter-task shared information necessary for fine-tuning across multiple tasks.
important baseline method in experiments. There
3.2 ARCHITECTURE
R
4X7*7
Image Encoder
Fr
Mask Decoder,
0
Wa
Fiz
Mask Decoder,
O7 No R
AW
Figure 2: An overview of the proposed MTSAM. The proposed ToRA used to fine-tune the heavyweight image encoder and generate task-specific image embeddings for each task. MTSAM does not utilize the prompt encoder of the original SAM and modifies the mask decoder of SAM to generate outputs with varying numbers of output channels (denoted by N, for task i).
Despite the tremendous potential exhibited by SAM as a fundamental visual model, its reliance on prompt-guided mask generation presents challenges in achieving end-to-end adaptability to down- stream tasks with varying numbers of output channels. Therefore, we propose the MTSAM to enable end-to-end multi-task fine-tuning for SAM.
As shown in Figure 2 MTSAM follows the standard encoder-decoder architecture of SAM, includ- ing heavyweight image encoder and several task-specific lightweight mask decoders. Different from SAM, MTSAM removes the prompt encoder in the original SAM and modifies the architec- ture the mask decoder.
To fully leverage the rich semantic knowledge acquired by SAM during pre-training, we froze the pre-trat pre-trained parameters the heavyweight image encoder and employ multi-task fine-tuning, which will be introduced in the next section, to update parameter tensors in the self-attention module (i.e., the query, key, and the value) the image encoder. Moreover, we perform fine-tur fine-tuning on the seale and bias parameters within the layer normalization layers of the image encoder.

---

Published as a conference paper at ICLR 2025
N₂
D: Broadcast sum
11
W
Matri multplication
E
16
Fix
Upscaling
Oc
w
N; 16 16
Two- Way Transformer
N;
N,
w
H
W
N;
16
16
4
P..
MLP
N₂
N₁ 8
N₂ x D
Figure 3: The mask decoder of MTSAM for task t, which has N, output channels.
To adapt the entire model various tasks, MTSAM introduces separate mask decoders for each task, generating task-specific outputs. Specifically, as detailed in Figure 3 we introduce trainable task embeddings with distinct numbers of output channels and consequently discard the dense mask en- coder and the sparse prompt encoder in SAM. Formally, the task embeddings E, E IR x. for task where N₂ denotes the number of output channels for task t, is expanded to E DX M by copying Eg for H x
W 16 times. Then, we perform the broadcast sum between the image em- 1V task t and the expanded task embeddings Ey, i.c., (FI, + EA) E Another input to the mask decoder is a learnable token Fp. RN Then two types input are fed into a two-way Transformer (Kirillov et al. 2023) and the outputs consist of hidden image feature representation and hidden token feature, which are fed into an upscaling layer and an MLP layer, respectively. In particular, the upscaling layer uses the transposed convolution operator. Thus, the decoder generates the prediction O₂ E R N.xxx 4 x for task t as
bedding FI.e E DR R N.N. x Dx H x W.
O, Dr(FI, ① E,, FP.))
(5)
3.3 TENSORIZED LOW-RANK ADAPTATION FOR MULTI-TASK FINE-TUNING
U₁
G
Pre-trained
Pre-trained Weights
+
Weights
U/2
A1
(a) LoRA for multiple tasks.
(b) ToRA for multiple tasks.
Figure 4: Comparison between (a) LoRA and (b) ToRA. LoRA uses separate low-rank matrices for the update parameter matrix of each task, while ToRA aggregates the update parameter matrices of all the tasks into an update parameter tensor and applies low-rank tensor decomposition.
To efficiently fine-tune the computationally intensive image encoder in MTSAM, we propose Ten- sorized low-Rank Adaptation (ToRA), a novel parameter-effien multi-task fine-tuning method. The comparison between LoRA and ToRA shown in Figure 4
Suppose we are given T' tasks. For simplicity, we consider the case where each task involves fine- tuning only one layer using PEFT approach, although this can be easily extend to multiple layers. For task t, we denote its update parameter matrix as AW; E Rd Consider all the T tasks, it is natural to aggregate the update parameter matrices all the tasks into an update parameter tensor AW = (AW,,... AWT) E [dxx"' Inspired by LoRA, we impose low-rank assumption AW. This assumption is plausible due to both inter-task relatedness and intra-task low rank each AW, observed in LoRA. Specifically, since different tasks in multi-task learning are typically assumed to related, the update parameter matrices (AW,) W.YTT, may be correlated, making A WW likely to be low-rank along the task axis (i.e., the last axis). In this sense, this low-rank assumption

---

Published as a conference paper at ICLR 2025
on A AW could be viewed as generalization of the low-rank assumption the parameter matrix of linear models (Zhag & Yang 2022).
To achieve low-rank A A. parameterize it via tensor decomposition Paapalxakkis et al. 2016), which is a technique to decompose a tensor into several low-rank factors. There are several tensor decomposition methods (e.g.. CP decomposition (Hitchcock) 1927, Carroll & Chang 1970 and Tucker decomposition (Tucker] (966)) and we choose the Tucker decomposition as it has a good representation ability (Papalexakis et al. 2016). Specifically, we decompose the three-mode update parameter tensor A W E R into a core tensor G E Rexaxv and three factor matrices U/ E Rdxp, U2 R*x0 and U3 E Txxx where p. 9. and denote the dimensions of factor matrices, typically p. 9. < min(d, k). Formally, this can be expressed as
AW- = G X1 U1 X2 U2 X3 U3.
where Xn denotes the n-mode product. Accordingly, the (i. f)-th entry in AW can be written as
(iij,t)) (041t)-C> >5>>>>>> (m, n, m) (2)
(7)
m== n=
where i {1.2,... ............. jE (1.2....,Ae .}. and & E {1,2,.... ,T denote the indices of three mode, respectively. For the three-mode update parameter tensor AW, the first mode represents the output feature dimension, the second mode denotes the input feature dimension, and the third mode is for the task dimension. Hence, according to Tucker decomposition (Tucker) (966). Uj and U2 reflect the main subspace variation of task-shared information corresponding to the first two modes in AW, while U3 reflects the task-specific subspace structure corresponding the last mode of A W. Hence, through the Tucker tensor decomposition, the ToRA method could capture both task-shared and task-specific information.
Initialization. For the proposed ToRA method, the core tensor G is initialized as O, while factor matrices U1. U2 and U3 are randomly initialized from the standard Gaussian distribution. Thus, for each task f, the update parameter matrix AW, is O at the beginning of training.
Training and inference. During training. we utilize Eq. (6 to obtain the update parameter tensor AW based on U1. U2. U3. and G and employ h = W Wox + A AW (G,;;()z l) the forward process for task 1. During the back- pack-propagation process, we freeze the pre-trained matrix Wo and only update U1. U2. Uz. and G. During inference, we can store the updated parameter matrix of task t as W = Wo + (W,,,, t). Thus, there is no additional latency introduced during inference.
Parameter complexity. We present comparison of parameter complexity between LoRA and the proposed ToRA under the multi-task learning setting. To fine-tune the pre-trained matrix Wo E Rdxk the LoRA method decomposes each update parameter matrix AW, as B,A, where B, Rdxr and A, E R' Therefore, for T tasks, the parameter complexity of LoRA is ((Trd+Trk) For the proposed ToRA method, we decompose the update parameter tensor AW E RRdxXxT as G xx Un X2 U2 X3 Uz. Therefore, the parameter complexity of the proposed ToRA method is pqv+ + dp+ kTT (d) since T", p, q, < min(d. (d, k). This implies that the parameter complexity of LoRA increases linearly with the number of tasks T', while the proposed ToRA method exhibits sublinear complexity, thereby demonstrating the parameter efficiency of the proposed ToRA method.
3.4 TRAINING OBJECTIVE
Under the multi-task learning setting, the training objective function of MTSAM is defined as
1 1 Σ f(+))) 14
LMTL
Σ w. Σ wrCi, where L,
(8)
In this formulation, L; represents the loss for task i and wy denotes the corresponding loss weight, ae is the j-th training sample for task i, yd is the ground truth label, f() is the MTSAM model, and E; denotes the loss function specific to task i.
According to the High-Order Singular Value Decomposition (HOSVD) (De Lathauwer et al. 2000) used in the analysis shown in the next section, every high-order tensor can be decomposed into core tensor G and orthogonal (or unitary) matrices. Inspired by this, we utilize the orthogonal

---

Published as a conference paper at ICLR 2025
regularization to enforce the orthogonality of U1 U2 and tensor & through the final dimension to reduce the redundancy. Consequently, this can be formulated as a regularization term:
- 103 IIE
Σ 1(..,) 11
(9)
where -| denotes the Frobenius norm for matrices and I denotes the identity matrix with an appropriate size. Therefore, the overall objective function of the MTSAM is formulated
Liotal = LMTL + XR(UI, U2,9)
(10)
where À is the hyper-paramenn parameter that controls the impact of orthogonal regularization.
3.5 ANALYSIS
In this section, we analyze the advantages of ToRA over LoRA in terms of the expressive power. For LoRA, Zeng & Lee (2024) apply the best low-rank approximation (Eckart & Young 1936) of the error matrix under the spectral norm. For the parameter matrix in a single layer, let W denote the target parameter matrix for task and define the enror matrix E, W W. When using LoRA, the update parameter is denoted by AW, and the minimum difference between the adapted and target models is given by
min [[W + AW,) W. -0, i AW,
(11)
where o, (E,) denotes the r-th largest singular value of E. For ToRA, the optimal approximation formulated as
min - U1 X2 U2 X3 Usll?
(12)
s.t. U rank (U+) S Tk.
where the objective is inherently complex, and the best approximation may not always exist (Kolda Bader, 2009). Therefore, measuring the expressive power of ToRA can be challenging. However, as proven in Theorem [ below, we demonstrate that for any multi-task learning problem solvable by multiple LoRAs, ToRA can also solve the problem by using fewer parameters. This implies that ToRA has superior expressive power for multi-task learning when compared with multiple LoRAs.
Theorem 1. (ToRA's Superiority over Multiple LoRAs in Expressive Power) Assume there are T LoRAs, whose update parameter matrix is denoted by AW, designed to solve each task with a rank To Let AW represent the update parameter tensor and AW (1). A W (2) denote the flattened tensor corresponding the vertical and horizontal concatenation of (AW,) Define p and q as the (2), respectively. Then, there exists a ToRA with core tensor G & R and factor matrices U1 E Rdxp, U2 E Tkkxa U3 E RR such that the Tucker decomposition x1 Un X2 U2 X3 U3 reconstructs AW. Furthermore, ToRA utilizes fewer parameters, satisfying (dp + kq) < En (d k)r.
rank of AW (1) and A W
4 EXPERIMENTS
In this section, empirically evaluate the proposed MTSAM three benchmark datasets, in- cluding NYU12 (Silberman et al., 2012), CityScapes (Cordts et al. 2016), and PASCAL-Contex (Everingham et al. 2010).
We compare the proposed MTSAM with several baselines, including CNN- methods (i.e., Single-Task Learning (STL), Hard-Parameter Sharing (HPS), Cross-Stitch (Misra
Baselines.
et al. 2016). Multi-Task Attention Network (MTAN) (Liu et al. 2019). and NDDR-CNN (Gao
al. 2019)). ). Transformer-baut based approaches (i.c., VTAGML (Bhattacharjee et al. 2023) and Swin-
MTL
Taghavi et al., 2024)), and the method using cross-attention (i.e. DenseMTL (Lopes et al.)
2023
To evaluate effectiveness of the proposed ToRA method, we also compare with LoRA-
STL. LoRA-HPS, and MultiLoRA (Wang et al., 2023) that all fine-tune the MTSAM.
Evaluation metric. For the three datasets, we use multiple metrics to evaluate the performance on each task and we put the introduction of them in Appendix B.T Moreover, following the setup

---

Published as a conference paper at ICLR 2025
Table I: Performance on three tasks (i.e., 13-class semantic segmentation, depth estimation, and surface normal prediction) of the NYUv2 dataset. The best results for each task are shown in bold. T(L) means that the higher (lower) the value, the better the performance. The number of trainable parameters (i.e., Params.) is calculated in MB.
Segmentation
Depth
Surface Normal
Method
Angle Distance
Within 1
Parum. (M)
A
mioU?
Pix Acc"
Abs Err
Rel Err.
Mean
Median
11.25
22.5
30
HPS
10.00%
STL
$3.98
75.38
0.3945
0.1631
22.25
15.63
38.12
64.38
74.81
118.91
+0.45%
Stots-Sitch
53.46
75.49
0,3804
0.1555
23.01
16.33
37.01
62.42
73,02
118.99
+0.6665
MTAN
54.74
75.78
0.3796
0.1549
22:97
16.30
36.91
62.63
73.32
92.35
10.777
NDDR-CNN
53.84
75.23
0.3871
0.1560
22.60
16.07
37.67
63.43
73.92
169.10
+0.911
VTAGML
$8.60
78.63
0.3716
0.1525
22.05
15.70
38.14
64.28
74,50
314.15
+1.70%
DenseMLL
56.65
77,68
0.3569
0.1391
22.03
15.87
37.25
64.67
75.47
423.45
+5.8856
82.78
0.1129
18.94
13.34
71.32
-9.555
LoRA-HPS (x=32)
56.77
78.37
0.3470
0.1412
18.97
13.41
44.56
71.20
80.68
58.84
10.67%
LoRA-STI (e-((1))
62.06
81.72
0.3124
0,1233
16.44
11.39
51.04
77.01
85.31
64.83
20.25%
LoRA-STL (e-3)
58.34
78.61
0.3330
0.1335
16.54
11:42
51.08
76.65
84.99
82.84
16.34%
MalnLaRRA
64.85
83.07
0,3113
0.1220
17:26
12.19
48.28
74.05
83.58
65.12
+20.115%
MTSAM
65.98
83.42
0.2898
0.1144
16.34
11.33
51.22
77.20
85,51
59.59
+23.9399
in (Maninis al., 2019). we use the average of the relative improvement of each task over the HPS architecture as another evaluation metric, which is formulated as
T
K K
(-1)%
sis (M', M MBPS
A
T
j=1
where T denotes the number of tasks, K; denotes the number of metrics for task i, M and MSPS denote the performance of the method b and the HPS architecture for the jth metric in task i, respec- tively, and 8i.j is set to a lower value indicates better performance in terms of the jth metric in task i and otherwise 0.
Implementation details. The batch size set to 4 for NYUv2 and 8 for CityScapes and PASCAL- Context. The cross-entropy loss, L1 loss, and cosine similarity loss used as the loss functions of the semantic segmentation, depth estimation, and surface normal prediction tasks, respectively. The Adam optimizer is used to update fine-tuned parameters. In the Adam optimizer, an initial learning rate is set to 10-3, 3, the linear learning rate scheduler with warmup is adopted while the warmup rate is set to 0.05, and the weight decay is set to 10 The dropout rate is set to 0.1. For the proposed ToRA, we set p q 32, v = 8 on the NYUv2 and PASCAL-Contex datasets, and ? = 16, = on the CitySappes dataset. The hyper-parameter is set to The total number of fine- tuned epochs is set to 200, 50, and 30 for the NYUv2, CityScapes, and PASCAL-Conted datasets, respectively. For CiryScapes and PASCAL-Contert datasets, we use equal weights for each task (i.e., wi equals in Eq. (B)). while for the NYUv2 dataset, we follow (Lopes et al. (023) to set the weights of semantic segmentation, depth estimation, and surface normal prediction tasks to be 1, 1, and 4, respectively.
4.1 RESULTS
The results on the NYUv2, CityScapes, and PASCAL-Context datasets are presented in Tables T 2. and 3 respectively. As can be seen, MTSAM achieves the best average performance across all datasets in terms of As compared to all baselines. Furthermore, MTSAM demonstrates better parameter efficiency, offering advantages in storage and enhancing its practical application value. Moreover, LoRA-STL, which employs separate LoRAs for each task, possesses better perfor- mance than LoRA-HPS with a shared LoRA. This demonstrates the importance of utilizing task- specific components. The superior performance of MTSAM with ToRA over both LoRA-STL and LoRA-HPS suggests that ToRA effectively leverages both task-shared and task-spec information, thereby improving overall performance.
4.2 ABLATION STUDY
Sensitivity wat. rank. We conduct sensitivity analysis to evaluate the impact of ranks of ToRA (i.e., P. g, and v) the performance of MTSAM while keeping the remaining hyper-parameters
8

---

Published as a conference paper at ICLR 2025
Table 2: Performance on two tasks (i.e., 7-class semantic segmentation and depth estimation) in the CityScapes dataset. The best results for each task are shown bold. T()) means that the higher (lower) the value, the better the performance. The number of trainable parameters (i.e., Params.) is calculated MB.
Segmentation
Depth
Method
Param. (M))
mloU?
Pix Ace
Abs Err
Rel Err.
HPS
67.48
90.92
0.0142
45.4262
55.76
+0.00%
STL
68.13 13
91.28
0.0133
45.0390
79.27
+2.17%
Cross-Stitch
68.01
91.29
0.0135
44.4246
79.27
+2.11%
MIAN
68.97
91.59
0.0136
43.7508
72.04
+2.74%
NDDR-CNN
68.02
91.25
0.0137
44.8662
101.58
+ +1.1%
VTAGML
73.70
93.23
0.0138
42.8304
288.35
+5.10%
DenseMTL
69.75
91.45
0.0152
52.1401
333.40
-4.4%
SwinMTL
73.33
92.87
0.0132
38,0720
333.31
88.4%%
LORA-HPS (1=16)
86.23
96.30
0.0123
34,2000
37.35
17.98%
LoRA-STL (r=8)
85.86
96.26
0.0110
34,3000
37.35
20.07%
LoRA-STL (r=16)
82.64
95,28
0.0107
33,4312
43.35
19.62%
MultiL
87.23
96.67
0.0127
31,0091
46.81
19.51%
MTSAM
87.45
96.80
0.0113
33.0086
37.44
+20.99%
Table 3: Performance on four tasks (i.e., 21-class semantic segmentation, 7-class human parts seg- mentation, saliency estimation, and surface normal estimation) in the PASCAL-Contex dataset. The best results for each task are shown in bold. 1()) means that the higher (lower) the value, the better the performance. The number of trainable parameters (i.e., Params.) is calculated in MB.
Method
Seg.
H.Parts
Sal.1
Normal.,
Param. (M)
As
HPS
64.77
57.91
64.10
14.21
30.07
+0.00%
STL
65.14
58.58
65.02
15.94
63.60
-2.55%
Cross-Stitch
64.97
58.63
64.46
15.32
79.46
-1.42%
MTAN
64.56
59.08
64.57
14.74
36.61
-0.33%
NDDR-CNY
65.28
59.18
65.09
15.57
69.25
-1.26%
Hyperformer
71.43
60.73
65.54
17.77
287.32
-1.1%
Polyhistor
70.87
59,54
65.47
17.47
34.18
-2.14%
Polyhistor-Lite
70.24
59.12
64.75
17.40
11.29
-2.72%
L-RA-HPS (r=32)
48.19
46.73
69.50
20.38
74.33
-19.97%
LoRA-STL (I=16)
55.25
71.33
75.72
17.05
86.33
+1.65%
LoRA-STL (r=32)
65.07
72.05
76.41
16.82
110.33
+6.42%
MultiLoRA
72.39
67.78
71.66
20.07
92.80
-0.66%
MTSAM
74.13
71.04
76.28
17.10
74.71
+8.95%
Table Ablation studies on the impact of rank on the NYUv2 dataset. The best results for each task shown in bold. 1(1) means that the higher (lower) the value, the better the performance. The number trainable parameters (i.e., Params.) calculated MB.
Segmentation
Depth
Surface Normal
Angle Distance
Within e
(M))
miolif
Pix Ace
Abs Err
Earr
Mean
Median
30
p 7
0.2966
16:40
11.42
76.85
80%
MISAMM 2 32,
83,42
0.2898
0.1140
16.34
11.33
51.22
77.20
85.51
59.59
+23.00%
consistent with previous experiments. According the results shown in Table 4 we can see that the proposed ToRA method consistently outperforms the LoRA-STL. and LoRA-TPS methods across different combinations of ranks, demonstrating the effectiveness of the proposed ToRA method.
Impact of orthogonal regularization. We conduct ablation studies NYUv2 dataset to evalu- ate the impact of the orthogonal regularization on U1. U2 and the core tensor by employing the same hyper-parametee settings previous experiments. We compare MTSAM with that without the orthogonal regularization on G (denoted by MTSAM (w/o G)) and without the orthogonal regular- ization U1. U2 and G (denoted by MTSAM (w/o U, U2. 9)). According the results shown in Table we can see that the orthogonality regularization on effectively improves the performance across various tasks, demonstrating the effectiveness of the orthogonal regularization.

---

Published as a conference paper at ICLR 2025
Table 5: Ablation studies the impact of orthogonal regularization the NYUv2 dataset. Best
results are shown in bold. T()) means that the higher (lower) the value, the better the performance.
Segmentation
Depth
Surface Normal
Method
Angle Distance
Within ("
Ag
mloL?
Pix Acc
Abs Err
Rel Err,
Mean
Median
11.25 7
22.51
30
MTSAM (w/o Us. U2.
58.71
80.01
0.3309
0.1305
16.88
11.66
50.17
76.05
84.48
+17.01%
MTSAM (w/o 9)
65.22
83.11
0.2984
0.1195
16.56
11.53
50.59
76.63
85.03
+22.30%
MTSAM
65.98
83.42
0.2898
0.1140
16.34
11.33
51.22
77.20
83,51
123.93%
Table 6: The performance of MTSAM with varying on the NYUv2 dataset. The best results for each task are shown in bold. 1() means that the higher (lower) the value, the better the performance. The number of trainable parameters (i.e., Params.) is calculated in MB.
Segmentation
Depth
Surface Normal
Method
Angle Distance
Within
Ab
mIoUt
Pix Ace
Abs Err
Rel Err
Mean
Median
11.25 f
22.5 f
f
0.5
66.05
83,82 83.42
0.2898
0.1137
16.70
11.73
49.81
76.39
84,92
+23.410%
1.0
0.2898
0.1140
16.34
11.33
51.22
77.20
85.51
+23.931%
A 1.5
66.11
83.50
0,2872
0.1135
16.47
11.48
50.68
77.05
85.39
+23.933%
Input
Ground
LoRA-HPS
LoRA-STL
MutsilORA
Normals
Senseg
Figure 5: Comparison among predictions of LoRA-HPS, LoRA-STL, MultiLoRA, and ToRA to fine-tune MTSAM the NYUv2 dataset.
Sensitivity w.r.t. hyper-parameter A. We explore the sensitivity of the performance of ToRA with respect to typer-parameter X. According the results shown in Table we can see that MTSAM is not so sensitive to the hyper-parameter X over [0.5, 1.5], making the setting of À not so difficult.
4.3 QUALITATIVE EVALUATION
Figure 5 shows the predictions of the MTSAM fine-tuned with LoRA-STL, LoRA-HPS, Multi- LoRA, and ToRA on the NYUv2 dataset, respectively. More qualitative results are shown in Figures T Appendix D As can seen, the prediction results of ToRA are better than the baselines for different tasks. As shown in the white boxes, the proposed ToRA method generates more accu- rate results than the baseline methods given the ground truth when dealing with vague and slender objects. Therefore, the proposed MTSAM fine-tuned with ToRA achieves the best performance in both qualitative and quantitative evaluations.
CONCLUSION
In this paper, we propose the MTSAM, which modifies the architecture of SAM and leverages a low-rank tensor decomposition method to fine-tune the encoder of MTSAM. MTSAM introduces task embeddings to generate outputs with the corresponding number of channels, enabling the model be adapted to different tasks. The proposed ToRA use both task-shared and task-specific information during the multi-task fine-tuning process. The experimental results demonstrate the ef- fectiveness of MTSAM. In future work, are interested in applying MTSAM to more applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by National Key R&D Program of China 20222D0160300 and NSFC key grant under grant 62136005.
REFERENCES
Deblina Bhattacharjee, Sabine Süsstrunk, and Mathieu Salzmann. Vision transformer adapters for generalizable multitask learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19015-19026, 2023.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011, 2020.
J Douglas Carroll and Jih-Jie Chang. Analysis of individual differences in multidimensional scaling via an n-way generalization of "eckart-young" decomposition. Psychometrika, 35(3):283-319 1970.
Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Chen Yang, Wei Shen, Lingxi Xie, Xiaopeng Zhang, and Qi Tian. Segment anything in 3d with nerfs. In NeurIPS, 2023.
Yang Chen, Shuai Fu, and Yu Zhang. Mopd: Mixture of models, 2024. URL /ttp::///aiivviii arxiv. org/ wtt.:orrg/abb/222..1000000
prompts distillation for vision-language
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pp. 794-803. PMLR, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. multilinear singular value decompo- sition. SIAM journal on Matrix Analysis and Applications, 2154) 2000.
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation pre-trained language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, (3):211-228,, 1936.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identify- ing task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34:27503 7777503--711.. 2021.
Yuan Gao, Jiayi Ma, Mingho Zhao, Wei Liu, and Alan L Yuille. Nddr-enn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3205-3214, 2019.
Pengsheng Guo, Chen-Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In International conference on machine learning, pp. 3854-3863, PMLR, 2020.
Pengxin Guo, Shuang Zeng, Yanran Wang. Huijie Fan, Feifei Wang, and Liangqiong Qu Selective aggregation for low-rank adaptation in federated learning. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL
net/ htt./formmiii.
Fx3dESGds0

---

Published as a conference paper at ICLR 2025
Frank L Hitchcock. The expression of tensor polyadic as sum of products. Journal of Mathematiccs and Physics, 6(1-4):164-18) 189, 1927.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter--fffcient transfer learning for nlp. In International Conference on Machine Learning, pp. 2090- PMLR, 2019.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL /tttps://ppenrevilee net. / nwwp:orrnttofrtt forum?
Siyu Huang, Xi Li, Zhi-Qi Cheng, Zhongfei Zhang, and Alexander Hauptmann. Gnas: greedy neural architecture search method for multi-attribute learning. In Proceedings of the 26th ACM international conference on Multimedia, pp. 2049-2057, 2018.
Yuhao Huang. Xin Yang, Lian Liu, Han Zhou, Ao Chang, Xinrui Zhou, Rusi Chen, Junxuan Yu, Jiongquan Chen, Chaoyu Chen, et al. Segment anything model for medical images? Medical Image Analysis, 92:103061, 2024.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Weisen Jiang, Yu Zhang, and James Kwok. Effective structured prompting by meta-learning and representative verbalizer. In International Conference on Machine Learning, pp. 15186-15199. PMLR, 2023.
Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James Kwok. forward-backward reasoning in large language models for mathematical verification. In Lun- Wei Ku, Andre Martins. and Vivek Srikumar (eds.), Findings of the Association for Compu- tational Linguistics: ACL 2024, pp. 6647-6661. Bangkok, Thailand, August 2024. Associa- tion for Computational Linguistics. doi: 10. 18851//1/2024.fmmlll
URL https:
waw.lantooooy..orgg/ooooo wwtp///.//20...oooo findings ael.s-cl.1979 3977
Lei Ke, Minggiao Ye, Martin Danelljan, Yifan Liu, Yu- Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Segment anything in high quality. In NeurIPS, 2023.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXhv:2304.02643 2023.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3): 455-500, 2009.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Elora: Efficient low-rank adaptation with random matrices. In The Twelfth International Conference Learning Representations, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3045-3059, 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising requence-to-sequeee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7888 2020.
12

---

Published as a conference paper at ICLR 2025
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers),
pp. 4582-4597, 2021.
Baijiong Lin, Weisen Jiang, Feiyang Ye, Yu Zhang, Pengguang Chen, Ying- Song-Cong Chen, Shu Liu, and James T. Kwok. Dual-balancing for multi-task learning, 2023. URL. https://arxiv.
org/abs/2330 12029
Baijiong Lin, Weisen Jiang, Pengguang Chen, Shu Liu, and Ying-Cong Chen. MTMamba++: Enhancing multi-task dense scene understanding via mamba-based decoders. arXiv preprint arXiv:2408. 15101, 2024a.
Baijiong Lin, Weisen Jiang, Pengguang Chen, Yu Zhang, Shu Liu, and Ying-Cong Chen. MT- Mamba: Enhancing multi-task dense scene understanding by mamba-based decoders. In Euro- pean Conference on Computer Vision, 2024b.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficient transfer learning. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pp. 441-459, 2020.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34: 18878-88890, 2021a.
Liyang Liu, Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. ICLR, 2021b.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 1871-1880 2019.
Yajing Liu, Yuning Lu, Hao Liu, Yaozu An, Zhuoran Xu, Zhuokun Yao, Baofeng Zhang, Zhiwei Xiong, and Chenguang Gui. Hierarchical prompt learning for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10888- 2023a.
Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint arXiv:2305. 2023b.
Yen-Cheng Liu, Chih- Yao Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: Parameter- efficient multi-task adaptation for dense vision tasks. Advances Neural Information Processing Systems, 35:36889-36901 2022.
Ivan Lopes, Tuan Tuan-Hung Vu, and Raoul de Charette. Cross-task attention mechanism for dense multi- task learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2329-2338, 2023.
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15665, 2024.
Kevis-Kokitsi Kokitsi Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasked of multi- ple tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recog- nition, pp. 1851-1860, 2019.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings the nition, pp. 1994-4003 2016.
the IEEE conference on computer vision and pattern recog-
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning bargaining game. arXiv preprint arXiv:2202.01017, 2022.
13

---

Published as a conference paper at ICLR 2025
Evangelos E Papalexakis, Christos Faloutsos, and Nicholas D Sidiropoulos. Tensors for data mining and data fusion: Models, applications, and scalable algorithms. ACM Transactions on Intelligent Systems and Technology (TIST), 8(2):1-44, 2016.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763, PMLR, 2021.
Dripta S Raychaudhuri, Yumin Suh, Samuel Schulter, Xiang Yu, Masoud Faraki, Amit K Roy- Chowdhury, and Manmohan Chandraker. Controllable dynamic multi-task architectures. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10955- 10964, 2022.
Siddharth Shankar, Leigh A Stearns, and CJ van der Veen. Semantic segmentation of glaciolog- ical features across multiple remote sensing platforms with the segment anything model (sam). Journal of Glaciology, pp. 1-10, 2023.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
Xiaozhuang Song, Shun Zheng, Wei Cao, James Yu, and Jiang Bian. Efficient and effective multi- task grouping via meta learning on task combinations. Advances in Neural Information Process- ing Systems, 35:37647-37659, 2022.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pp. 9120-9132. PMLR, 2020.
Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what share for efficient deep multi-task learning. Advances in Neural Information Processing Systems, 33, 2020.
Pardis Taghavi, Reza Langari, and Gaurav Pandey. Swinmtl: A shared architecture for simultancous depth estimation and semantic segmentation from monocular camera images. arXiv preprint arXiv:2403. 10662 2024.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3):
279-30 +79, 1966.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter- efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3266-3279, 2023.
Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. Multilora: Democratizing lora for better multi-task learning. arXiv preprint arXiv:2311.11500 2023.
Yanbin Wei, Qiushi Huang, Yu Zhang, and James Kwok. KICGPT: Large language model with knowledge in context for knowledge graph completion. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8667-8683, Singapore, December 2023. Association for Computational Linguisties. doi: 10. 10.18653/v 1/2023. 10.. findi l'ndings--a emn 580/
URL httpp://aaaaaattooooyo.oooo
14

---

Published as a conference paper at ICLR 2025
Xun Wu, Shaohan Huang, and Furu Wei. Mole: Mixture of lora experts. In The Twelfth International Conference on Learning Representations, 2023.
Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. Track anything: Segment anything meets videos, 2023.
Feiyang Ye, Baijiong Lin, Zhixiong Yue, Pengxin Guo, Qiao Xiao, and Yu Zhang. Multi-objective meta learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan
(eds.), Advances in Neural Information Processing Systems, volume 34, pp. 21338 21351. Cur- ran Associates, Inc., 2021. URL U newrsess. cctppppppccppppprrrpppppppp
files/
Feiyang Ye, Xuehao Wang, Yu Zhang, and Ivor W Tsang. Multi-task learning via time-aware neural ode. In Proceedings of the Sheron International Joint Conference on Artificial Intelligence, pp. 4495-4503, 2023.
Hanrong Ye and Dan Xu. Inverted pyramid multi-task transformer for dense scene understanding. In ECCV, 2022.
Hanrong Ye and Dan Xu. Taskprompter: Spatial-channel multi-task prompting for dense scene understanding. In ICLR, 2023.
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE con- ference on computer vision and puttern recognition, pp. 3712-3722, 2018
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. The Twelfth International Conference Learning Representations, 2024.
Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hong- sheng Li. Personalize segment anything model with one shot, arXiv preprint arXiv:2305.0300 2023.
Yu Zhang and Qiang Yang. survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, (4(12):55865600 2022.
Zihan Zhong, Zhiqiang Tang, Tong Hc, Haoyang Fang, and Chun Yuan. Convolution meets lora: Pa- rameter efficient finetuning for segment anything model. In The Twelfth International Conference Learning Representations, 2023.
Zhan Zhuang, Yulong Zhang, Xuehao Wang, Jiangang Lu, Ying Wei, and Yu Zhang. Time-varying lora: Towards effective cross-domain fine-tuning of diffusion models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
15

---

Published as a conference paper at ICLR 2025
A PROOFS
In this section, we present the proof of Thereom demonstrating the superiority of ToRA over multiple LoRAs. We begin by introducing higher-order singular value decomposition (HOSVD) proposed by De Lathauwer et al. (200)) as our Lemma
Lemma 1. (HOSVD) Every tensor A E IRd xd2 xda can be decomposed into a core tensor G E idaxxa and the left singular vectors UR E Rdaxds of k-mode flattened matrix A(k), as follows
A A X1 (U;UT) X2 (U2U2) Xa (UzU3)
(A XI UJ Ug X3 22U)) xi U1 X2 U2 X3 X3
=9 X1 U1 X2 U2 X3 U3.
According to Property 10 of DD Lathauwer al. 2000
achieve low-rank approximation of lensor through compact HOSVD. Let the rank of A A(k) be RR, and define lensor À by discard-
ing the smallest singular values 0it+ +1 11-20. RR for given values of Ii that is, we set the
(k) (k)
(k)
corresponding parts of the core tensor G to zero. Then we can get
R₁
R₂
RN Σ
A Áll <
Σ
(1)
Σ
(2)
(N)2 iN
4
i2
11
iN IN=1x+1
Lemma 2. Given K matrices Ag E Rmxn, let Rank denote the rank of a matrix. follows trivially that Rank([A1 A2 Ax]) s Σx Rank(AA) and Rank([A] A Akl) Σ Rank( A+)).
Theorem 1. (ToRA's Superiority over Multiple LoRAs in Expressive Power) Assume there are T LoRAs, whose update parameter matrix is denoted by AW, designed solve each task with a rank Tr Let AW represent the update parameter tensor and AW (1). (W denote the flattened tensor corresponding to the vertical and horizontal concatenation of (AW)) Define p and the rank of AW (1) and AW (2), respectively. Then, there exists a ToRA with core tensor G E RXx4xI and factor matrices U1 E Rddp, U2 E 1Rxx U3 E TR such that the Tucker decomposition X1 U1 X2 U2 X3 U3 reconstructs A AW. Furthermore, ToRA utilizes fewer parameters, satisfying (dp + kq) S E,(d ( k)r..
Proof. The weight updates A AW of multiple LoRAs across T tasks can be organized into two flat- tened forms: A AW (1) and A (2). Specially, we define these as follows
AW (1) = [& AW2 AWT).
AWT. (2) AWW AW,
AWT].
express A W using core tensor E [2xxxXT with three unitary matrices U1 E RRd U2 E kx, U3 E T'x These unitary matrices represent the left singular vectors corresponding to the three flatten matrices of AW and G = AW U, X2 U X3 Ug.
According to Lemma 1
From Lemma 2 we have the incqualities p S E. Tr and q < Σ. Tt- Consequently, we deduce that dp + lg < E. fe)r,. Given that T.P, q, are significantly smaller than min(d, k), we conclude that ToRA can achieve the same weight updates with fewer parameters. In particular, this theorem constructs specific instance, indicating that the expressive power of ToRA will be greater.
Li
B EXPERIMENTAL DETAILS
B. METRIC FOR EACH TASK
NYUv2 and CityScapes datasets. For the semantic segmentation task, we use the mean Intersec- tion over Union (mloU) and Pixel Accuracy (Pix Acc) to evaluate. For the depth prediction task, we the Absolute Error (Abs Err) and Real Eror (Rel Err) to evaluate. For the surface normal estima- tion task, we use the mean and the median of angular error measured degrees and the percentage of pixels whose angular error is within 11 1.255 22.5, and 30 degrees evaluate.
16

---

Published as a conference paper at ICLR 2025
Table 7: Performance three tasks (i.e., 13-class semantic segmentation, depth estimation, and surface normal prediction) of the NYUv2 dataset. The best results for each task are shown in bold. T(L) means that the higher (lower) the value, the better the performance. The number of trainable parameters (i.e., Params.) is calculated in MB.
Segmentation
Depth
Surface Normal
Method
Angle Distance
Within e"
Param, (M)
A
mloU|
Pix Acct
Abs Err L.
Err]
Mean
Medion
11,25
22.5
30
Full fine-toning
58.76
80.18
0.3063
0.1223
19.61
13.76
43.14
69.44
79.17
1222.47
+14.57%
MuliiLoRA
64.85
83.07
0.3113
0.1220
17.26
12.19
48.28
74,65
83.58
65.12
102111%
80.93
0.3461
0.1343
13.27
+13.70%
HydraLoRA
66.46
83.90
0.3033
0.1202
16.86
11.60
19:47
75.86
84.54
71.30
+22.11%
MISAM
65.98
83.42
0.2898
0.1140
16.34
11.33
51.22
77.20
85.51
$9.59
123.93%
PASCAL-Contex dataset. For the semantic segmentation task, human parts segmentation task, and saliency estimation task, we use the mean Intersection over Union (mloU) to evaluate. For the surface normal estimation task, we use the mean of angular error measured in degrees to evaluate.
C MORE EXPERIMENT
C.I RESULTS
To evaluate the effectiveness of the proposed ToRA method, also compare with coRA-based methods (i.e., MultiLoRA (Wang et al. (023). Terra (Zhuang et al. 2024), and HydraLoRA (Tian al. 2024)) and the full tine-tuning method that fine-tunes the entire MTSAM. The results shown in Table 7 As can be seen, ToRA achieves better performance and parameter efficiency compared to LoRA-based methods and full fine-tuning.
C.2 ABLATION STUDY
Table 8:
The performance of MTSAM with modified MLP layer and task embeddings on NYUx2
dataset.
Segmentation
Depth
Surface Normal
Method
Angle Distance
Param, (M).
4
mloLt
Pix Acc
Err
Rer)
Mean
Median
11.25 1
22,5
30
MLP
63.50
82.45
0.3294
0.1235
17.93
13.02
45.46
73.28
82.86
65.66
+17.35%
embeddings
13.993%
Impact of task embeddings. To demonstrate the effectiveness of the proposed task embedding, we compared with method of modifying the MLP output dimensions for different tasks on NYUv2 dataset. As shown in Table 8 task embedding performs better. This improvement is due to the interaction between task embeddings and image features through the cross-atie attenttonn mechanism, which enables the decoder better learn the task-specific knowledge and achieve superior results.
D MORE QUALITATIVE EVALUATIONS
Figures 6 7 9 and 10 show the predictions of the MTSAM fine-tuned with LoRA-STL, LoRA- HPS, MultiLoRA. and ToRA on the NYUv2 and CityScapes datasets, respectively. Figure II shows the prediction of MTSAM
high-resolutional images. It be observed that the predictions of MTSAM outperform those of other baselines in different tasks and datasets. In the areas highlighted by the white boxes, MTSAM generates more accurate results. Therefore, using MTSAM with ToRA yields better performance.
E QUALITATIVE EVALUATIONS ON ZERO-SHOP ABILITY
To evaluate MTSAM's performance on unseen data, we applied the model fine-tuned on the NYUv2 dataset make depth predictions CityScapes dataset. Qualitative results are shown in Figure 12
17

---

Published as a conference paper at ICLR 2025
Input
Ground Truth
LoRA-HPS
LoRA-STL
MutsiLoRA
ToRA
Normals
Figure 6: (1/3) Comparison among predictions of LORA-HPS, LoRA-STL, MultiLoRA, and ToRA to fine-tune MTSAM on the NYUv2 dataset.
and illustrate that MTSAM capable of handling unseen data distributions to some extent. However, it is important note that the NYUv2 dataset consists of indoor images, whereas the CityScapes dataset comprises of outdoor images, leading to significant differences in depth distribution and object types. Additionally, the datasets differ in terms of resolution and the hardware used for groud-truth depth predictions. Consequently, MTSAM exhibits some inaccuracies, particularly for distant objects.
18

---

Published as a conference paper at ICLR 2025
Input
Ground Truth
LoRA-HPS
LoRA-STL
MultiLoRA
ToRA
Depth
Normals
Semseg
Depth
Normals
Semseg
Depth
Normals
Semseg
Figure 7: (2/3) Comparison among predictions of LORA-HPS, LoRA-STL, MultiLoRA, and ToRA to fine-tune MTSAM on the NYUv2 dataset.
19

---

Published as a conference paper at ICLR 2025
Input
Ground Truth
LoRA-HPS
LoRA-STL
MultiLoRA
ToRA
Depth
Normals
Semseg
Depth
Normals
Semseg
Depth
Normals
Semseg
Figure 8: (3/3) Comparison among predictions of LORA-HPS, LoRA-STL, MultiLoRA, and ToRA to fine-tune MTSAM on the NYUv2 dataset.
Input
Ground Truth
LORA-HIS
LoRA-STL
MultiLoRA
ToRA
Semsag
Depth
Figure 9: (1/2) Comparison among predictions of LORA-HPS, LoRA-STL, MultiLoRA, and ToRA to finc-tune MTSAM on the CityScapes dataset.
20

---

Published as a conference paper at ICLR 2025
Input
LaRA-1iS
MaltitanA
ToRA
Deyth
Servigg
Degeth
Semvag
Degrs
Sermong
Deptt
Semsa
Figure 10: (2/2) Comparison among predictions LoRA-HPS, Lo MultiLoRA, and ToRA fine-tune MTSAM the CityScapes dataset.
Input
Ground Truth
LoRA-HPS
LoRA-STL
MultiLoRA
ToRA
Figure 11: Comparison among predictions of LoRA-HPS, LoRA-STL, MultiLoRA, and ToRA to fine-tune MTSAM on high-quality images.
21

---

Published as a conference paper at ICLR 2025
Input
Ground Truth
Figure 12: Zero-shot depth estimation of MTSAM which is trained on the NYUv2 dataset and evaluated on the CityScapes dataset.
22


Reviews:
[{"review_id": "RGmlY9fuP4", "reviewer": "Reviewer_zxBa", "summary": "The paper proposes MTSAM (Multi-Task SAM), a framework that extends the Segment Anything Model (SAM) for multi-task learning. SAM's original architecture is limited to single-task applications due to its prompt encoder and uniform output channels. MTSAM addresses these limitations by modifying SAM’s architecture to accommodate task-specific outputs and by introducing Tensorized low-Rank Adaptation (ToRA) for multi-task fine-tuning. ToRA injects a tensor parameter into SAM’s encoder, allowing efficient handling of both shared and task-specific information. Extensive experiments on benchmark datasets (NYUv2, CityScapes, PASCAL-Context) show that MTSAM outperforms existing multi-task learning approaches, both qualitatively and quantitatively, in segmentation, depth estimation, and surface normal prediction.", "questions": "I don't have particular questions. The concerns can be found on the weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) MTSAM successfully extends SAM to a multi-task framework, a novel approach that leverages SAM's strong zero-shot capabilities in a multi-task setting.\n\n2) The proposed ToRA method is parameter-efficient, enabling sublinear parameter growth and effective use of shared information across tasks.\n\n3) The paper provides theoretical justification for ToRA’s superiority over existing methods like LoRA, adding credibility to its parameter efficiency claims.\n\n4) MTSAM outperforms other multi-task learning models across three benchmark datasets, demonstrating its efficacy in varied visual tasks.", "weaknesses": "1) The current work lacks an evaluation of MTSAM’s zero-shot generalization ability, particularly on unseen data distributions. \n\n2) The experiments do not include a direct comparison with full fine-tuning methods, leaving it unclear whether MTSAM’s parameter-efficient fine-tuning can achieve competitive performance without compromising accuracy.\n\n3) There is a lack of comparative experiments to demonstrate the effectiveness of task embedding. Further experiments are needed to confirm the specific advantages of task embedding in multi-task setups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SjIHKFO3Mv", "reviewer": "Reviewer_Sjqm", "summary": "This paper propose the MTSAM, a multi-task segmentation model, which is based on the architecture of SAM. The researchers change the original prompt encoder and mask encoder of SAM into separate mask decoders for each downstream task, and introduce task embeddings to generate outputs with the corresponding number of channels, enabling the model can be adapted to various tasks. Otherwise, the researchers apply a low-rank tensor decomposition method to fine-tune the image encoder of MTSAM. different tasks. The proposed ToRA can use both task-shared and task-specific information during the multi-task fine-tuning process. The experimental results demonstrate the effectiveness of MTSAM.", "questions": "1.Could you provide a more detailed analysis of the advantages over advanced methods like SwinSTL? \n2.Can you clarify why different evaluation metrics are used for segmentation tasks or others across various datasets? What criteria were used to select these metrics?\n3.Would it be possible to include more comprehensive examples of image multitasking in the results section?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "There are several strengths of the paper: \nFor originality, based on the original model structure of SAM, this work makes a simple, direct but effective modification. Inspired by the previous work, it also proposes the Tora, a novel multi-task PEFT method on the idea of low-rank tensor decomposition. It has made effective innovations on the basis of existing work.\nFor clarity, the writing of the article is smooth, and the formulas, figures, etc. are clear and unambiguous.\nFor quality, This work was carried out on the NYUv2, CityScapes, and PASCAL-Context, and the experiments are fairly convincing in terms of the results of the indicators presented.\nFor significance, this work migrates the excellent performance of SAM to multiple downstream tasks, which has certain significance for the further promotion of SAM.", "weaknesses": "The weakness of this paper lies in the following aspects:\n1.The comparative advantages primarily focus on CNN-based methods, without analyzing more advanced approaches like SwinSTL. Additionally, the performance metrics compared to SwinSTL are not significantly superior.\n2.The dataset metric configurations differ without adequate explanation. For instance, it is unclear why different evaluation metrics are applied to segmentation tasks across various datasets.\n3.The results section insufficiently demonstrates the effectiveness of image multitasking, and the supplementary appendix provides limited image results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4JFoEUfQS2", "reviewer": "Reviewer_SfrS", "summary": "The paper proposes MTSAM, a multi-task learning framework that adapts the Segment Anything Model (SAM) for simultaneous execution of different computer vision tasks. By modifying SAM’s architecture and introducing a novel Tensorized low-Rank Adaptation (ToRA) method, MTSAM aims to optimize SAM's capabilities for multi-task scenarios, addressing task-specific output generation and efficient fine-tuning.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe MTSAM framework successfully extends SAM’s architecture, enabling it to handle multiple downstream tasks, a significant improvement over SAM’s single-task limitations.\n2.\tToRA offers a parameter-efficient fine-tuning solution that balances task-specific and shared information, enhancing performance without excessive resource requirements.\n3.\tComprehensive experiments across benchmark datasets (NYUv2, CityScapes, and PASCAL-Context) demonstrate MTSAM's superior performance over existing methods in multi-task learning, indicating practical efficacy.\n4.\tThe theoretical analysis of ToRA’s expressive power is well-presented and aligns with empirical findings, adding depth to the framework’s academic contributions.", "weaknesses": "1.\tThe first and second points in the summary of contribution points appear to be similar, the author is requested to provide good reasons for the significant difference, otherwise it is recommended to merge.\n2.\tIt is hard for me to consider MTSAM as a substantial model innovation improvement, (1) I don't get why the prompt encoder affects the downstream adaptation of the model, (2) the author's improvement of the decoder is more like the integration of multiple decoders for different tasks.\n3.\tI am interested in ToRA, but the author's description of the details is unclear. (1) How to constrain U1, U2, and U3 to learn task-relevant and task-irrelevant information, respectively? (explain or experiment) (2) Why is the decomposition into U1+U2+U3+G just enough? What happens if there are more or less U's? What if there were G?\n4.\tThe proposed ToRA lacks comparison with other LoRA-based improvement methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K2kAGiWfFM", "reviewer": "Reviewer_Zgv2", "summary": "This paper presents an interesting idea to make SAM multi-task capable. As mentioned by the authors (L201-204), despite SAM's success, it's end-to-end adaptability is limited by its prompt-guided paradigm. The paper proposes that one can learn multiple mask decoders, one per task, and then have a TORA (Tensorized Low Rank Adaptation) as opposed to say one LoRA per task to capture the task information. In some sense, the main show is actually TORA, for which the authors' motivation that all the tasks share information while having task-specific requirement makes sense. Experiments are sufficient to support the proposal.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The multi-task challenge with SAM is indeed a real problem that many researchers faced. The original SAM would output multiple potential masks when the prompts are ambiguous which sometimes make it hard for practitioner to adapt it to multiple downstream tasks.\n- The main show is TORA which the paper clearly presented why the final formulation is as given. I appreciate this clarity. In fact, I think ToRA can stand as a separate paper exploring its applicability in other problem domains, given that LoRA is a hotly researched topic currently.\n- Moreover, ToRA has a lower complexity computational wise. This could be quite valuable when the number of tasks really scale up. I am curious about scaling up the number of tasks. \n- The regularization term in Eq 9 is interesting.", "weaknesses": "- Scaling-up experiments are lacking. Is there any way to see a larger number of tasks beyond 3-4? I really would like to stress test ToRA.\n- Are there any out of domain experiments, same task but out of domain datasets?\n- Are there any experiments on the speed? ToRA as described has a computational advantage but there does not seem to be any experiments to back that up, unless I missed them.\n- Table 3 is slightly disappointing with LoRA-STL (r-32) beating MTSAM on some tasks. Not a show-stopper since overall performance seems ok.\n- The qualitative examples need to be better. Resolution is very low and hard to tell. I know cityscapes are like that, but perhaps testing on some higher res samples would be more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)