Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learner
Authors: Hanwen Zhong
Jiaxin Chen
Yutong Zhang
Di Huang
Yunhong Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter-efficient Fine-tuning
Multi-Task Learning
Mixtures of Experts
Quality Retaining
Primary area: machine_vision

Abstract: Multi-Task Learning (MTL) for Vision Transformer aims at enhancing the model capability by tackling multiple tasks simultaneously. Most recent works have predominantly focused on designing Mixture-of-Experts (MoE) structures and integrating Low-Rank Adaptation (LoRA) to efficiently perform multi-task learning. However, their rigid combination hampers both the optimization of MoE and the effectiveness of reparameterization of LoRA, leading to sub-optimal performance and low inference speed. In this work, we propose a novel approach dubbed Efficient Multi-Task Learning (EMTAL) by transforming a pre-trained Vision Transformer into an efficient multi-task learner during training, and reparameterizing the learned structure for efficient inference. Specifically, we firstly develop the MoEfied LoRA structure, which decomposes the pre-trained Transformer into a low-rank MoE structure and employ LoRA to fine-tune the parameters. Subsequently, we take into account the intrinsic asynchronous nature of multi-task learning and devise a learning Quality Retaining (QR) optimization mechanism, by leveraging the historical high-quality class logits to prevent a well-trained task from performance degradation. Finally, we design a router fading strategy to integrate the learned parameters into the original Transformer, archiving efficient inference. Extensive experiments on public benchmarks demonstrate the superiority of our method, compared to the state-of-the-art multi-task learning approaches.

Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning
Hanwen Zhong Jiaxin Chen Yutong Zhang Di Huang² Yunhong Wang"
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China School of Computer Science and Engineering, Beihang University, Beijing, China (hanwenzhong, jiaxinchen, harehhhhhnnnthnhggggg dhuang, (aaaang)bbbas edu. cn
Abstract
Multi-Task Learning (MTL) for Vision Transformer aims at enhancing the model capability by tackling multiple tasks simultaneously. Most recent works have predominantly focused on designing Mixture-of-Expete (MoE) structures and in- tegrating Low-Rank Adaptation (LoRA) to efficiently perform multi-task learning. However, their rigid combination hampers both the optimization of MoE and the ef- fectiveness of reparameterization of LoRA, leading to sub-optimal performance and low inference speed. In this work, propose novel approach dubbed Efficient Multi-Task Learning (EMTAL) by transforming a pre-trained Vision Transformer into efficient multi-task learner during training, and reparameterizing the learned structure for efficient inference. Specifically, we firstly develop the MoEfied LoRA A structure, which decomposes the pre-trained Transformer into low-rank MoE structure and employ LoRA to fine-tune the parameters. Subsequently, we take into account the intrinsic asynchronous nature of multi-task learning and devise a learning Quality Retaining (QR) optimization mechanism, by leveraging the historical high-quality class logits to prevent a well-trained task from performance degradation. Finally, we design router fading strategy to integrate the learned parameters into the original Transformer, archiving efficient inference. Extensive experiments on public benchmarks demonstrate the superiority of our method, compared to the state-of-the-art multi-task learning approaches. The project page is available at httpss:/ggihuu.com///eee (ttpwwen186//MMTII 186/EMTAL
Introduction
Multi-task learning (MTL) 2 for Vision Transformer (ViT) aims at simultaneously learning multiple tasks, which has gained popularity in the computer vision community recently. It solves multiple relevant problems through sharing feature representations and forming a unified multi-task learner, thus enhancing the training and inference efficiency, and reducing the storage overhead. Moreover, by virtue of well-designed MTL framework, the performance of each task can be further improved. Due these merits, MTL has been used in wide range of applications, such the scene understanding 4 and the erudite fine-gr ime-grained recognition (6)
Despite both theoretical 7 and practical 8 9 validations of their potential enhancing the model generalizability. conventional MTI approaches 66 compared to training tasks independently, which is primarily due to reasons. Firstly, they adopt suboptimal learning strategies leading to conflicting task gradients and varying loss scales [[T. which increase competitive interference during task optimization. Secondly, their model structures often fail to extract representative features for each task when utilizing a shared backbone network [12 6
10 often suffer performance degradation when
"Correponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Chanitel lidox
21 2 Ca X
Experts with dissimilar weights
Chonnel-wis wise similarity
Split
Merge
C
MoË
High-rank
FFN
Rearrange
Channel Index
Channel Indox
0 Cn d Co M
Experts with similar weights
Compact
Split
Merge
MSA
FFN
MoLE
Low-rank
Channal
Input
Channel-wis similarity of the up-projection layer
Channel-wise similarity of re-composed experts
Figure FFN as Mixture of Low-rank Experts. Given an up-projection weight matrix in FFN, straightforward way of splitting it into MoE is to divide every K channels into separate experts, resulting in highly dissimilar experts and a high-rank MoE, which is inherently unsuitable for integration with LoRA. In contrast, our proposed MoLE approach rearranges the weight matrix into groups similar channels as experts, creating specialized low-rank experts that are better suited for integrating with LoRA.
Several recent works have attempted to deal with the above issues, which concentrate on the following two aspects. 1) Efficient multi-task learners. As shown in Figure instead of designing complex but incompact network structures [12] that incur a large number tunable parameters, recent works
13 such MLoRE and MOELoRA explore the ad advantages of Mixture-off Experts (MoE) in extracting task-specific features by enhancing the diversity of parameters and features [5 16 [7], and Parameter-Efffiient Fine-Tuning (PEFT) in reducing the tunable parameters and storage overhead
18 19 20 Nevertheless, MLoRE 5 still relies on a substantial number of additional parameters, limiting the overall efficiency and feasibility of training. MOELoRA [4]] adopts unitary LoRA structure to tune the experts, which weakens the learning capability of individual experts. Moreover, both methods utilizes task-driven routers, requiring either static network with fixed number of tasks or a dynamic routing network. The former results significant storage overhead, while the latter increases the inference cost. 2) Multi- Task Optimization (MTO) strategies. Existing works on MTO can be broadly categorized into the gradient- based methods and the loss-based methods. The gradient-based methods [22 2 23 3 24 25] seek to balance the gradients across multiple tasks in the last shared layer, by decreasing differences in their magnitude or direction, and aggregating sub- booddrrddeetts into a unified one. The loss-based methods 26
27 28 optimize the MTL process by balancing the multi task losses. Recently, IMTL [I] is proposed to treat all tasks equally without bias, while AMTL own intrinsic optimization pace due varying levels of training difficulty for distinct tasks, forcing synchronization in MTO disrupts these inherent properties, thus leading to suboptimal solutions. For more detailed discussion of related works, we refer to Appendix A
(2 synchronizes learning progress across tasks. However, as each task has its
To overcome the drawbacks of existing works, we propose a novel MTL framework dubbed Efficient Multi-Tas Asynchronous Learning (EMTAL). Basically, EMTAL consists of the MoEfied LoRA structure, the Quality Retaining (QR) optimization mechanism and the router fading strategy, of which MoEfied LoRA decomposes a pre-trained Vision Transformer model into an efficient multi-task learner, QR accomplishes asynchronous learning multi-task knowledge and enable establishing an efficient unified model by combining with the router fading strategy. Specifically, inspired by MoEfication [30 31) proposed MoEfied LoRA firstly decomposes the FFN layer of Vision Transformer into a MoE structure by clustering similar channels into experts, creating specialized low-rank experts demonstrated in Figure Considering the inherent low-rank property of each expert, LoRA is naturally employed to perform efficient training of MoE. Subsequently. in order to

---

achieve asynchronous learning of multi-task knowledge based on the MoEfied LoRA module. QR constrains logits of early converged tasks retain near the optima when continuously optimizing the insufficiently converged tasks, thus avoiding severe interference between tasks and improving multi-task optimization. Finally, the router fading strategy is combined with MoEfied LoRA and QR by gradually diminishing the muter's fole in the last training epochs, seamlessly integrating learned parameters into the model structure without incurring extra inference time cost and storage overhead.
Freso
Tunable
Tckema
Trannng
Output tokens
Output tokens
Output tokens
Opt
Waight
MoE
Moefled
LoRA
Reparametiizaton
Expart
Expert
Expert
Experit
And Experts
Weight Matrix
Construch
Natrices
Weight
Router
Router
Moefication
Weight
Matrix
Transformer Blocks
Transformer Blocks
Transformer Blocks
Faster
Input tokens
Input tokens
Input tokens
(a) MoE 12
The conven- tional MoE leverages multiple ex- pert networks and gating mech- anism dynamically select the most relevant experts each put. facuses on designing com- plex but incompact network struc- tures, incurring large number of tunable parameters.
(b) LoRA Experts [4] LoRA Experts employ unified low-rank adaptation modules to achieve the parameter efficiency, which however limit expert capacity, and requires either static networks with substantial storage overhead dynamic routers with high in- ference cost.
(c) MoEfied LoRA (Ours). Our method groups similar weights into specialized low-rank experts, enabling seamless in- tegration with LoRA create effi- cient multi-task learner. Besides, by combing with router fading strategy, our method cnsures both training and inference efficiency while substantially reduces storage overhead.
Figure 2: Summary of representative architectures of multi-task learning.
The main contributions this paper are summarized as follows. 1) We propose a novel efficient multi-task learning framework dubbed EMTAL. To the best of our knowledge, our work makes the first investigation on decomposing a pre-trained Vision Transformer model for multi-task learning and reparameterizing the leamned multi-task knowledge into unified model. 2) We design MoEfied LoRA structure, a QR multi-task optimization mechanism combined with a router fading strategy to accomplish an efficient asynchronous multi-task learner. 3) We extensively evaluate the proposed method on challenging multi-task fine-grained visual classification datasets and the VTAB benchmark, and the experimental results demonstrate that method significantly improves the performance of single-tast learning and the state-of of the-art art MTL approaches.
2 The Proposed Approach
this section, we mainly introduce the preliminary concepts of Vision Transformer, and describe the technical details of the proposed EMTAL approach.
2.1 Preliminary
Given an input image E 3 a standard Vision Transformer 32 model with layers first divides I into m non-overlapping patches. which are further fed into a patch embedding layer, generating m D-dimensional visual tokens. After concatenating with a class token, the input tokens are finally formed as X 0 [(I+m)xD Each transformer layer contains a Multi-headed Self- Attention (MSA) [33] block, a Feed-Forward Networks (FFN) block and Layer Normalziation (LN). The tokens of the 1-th layer are generated based on those in the 1)-th layer formulated as below:
x" -SS(LL(X'-)) 1 x = =F=FENLN(XX )) +
(I)
3

---

Similar to 30 31, our work mainly focuses on FFN, which usually consists of two linear layers {W up E I D) up E D, (Wdown down E (rrDD)DD bdown E [R and a GELU activation operation, where represents the scaling factor. Accordingly, FNN processes the normalized input x LN(X") as follows:
FEN GELU(X, W + bup) W down +
doown.
This procedure ensures the effective transformation and projection of the input through the encoder layers, enabling the prediction the class probability distribution y for downstream tasks.
2.2
Framework Overview
What are these ?
3-tball wich
Spyker Dorniar-320
routte
No.feers Orginl
FFN
I Router Fodiig
LoRA
Optimized
e
ViT
t
Elp Elaun
e x
Router
MSA
Expert!
Expert
Nomas arto
E
E
AIB
Charter smiler columns into expert
apooed
MSA
Orgiil
W
Sae
FFN
thrrs
(a). Pre-trained ViT
(b). Decomposing
(c). McEfied LoRA
(d). Optimization
(e). Reparameterization
Figure 3: Illustration of the proposed EMTAL framework. Given pre-trained ViT, we firstly decompose it into a MoE-basse multi-task learner by using the balanced k-means. LoRA is then applied to the low-rank experts, creating efficient multi-task learner dubbed MoEfied LoRA. During multi-task optimization, the Quality Retaining is employed to maintain the high-quality knowledge for tasks that have already converged. Finally, with the aid of the router fading strategy, the learned knowledge is reparameterized back into the pre-trained VIT, eliminating the extra inference cost.
As shown in Figure 3 (a), in order to establish a unified model for Ny tasks, we follow 66 by unifying the label spaces for multiple tasks into an overall one with Nciass classes, and merging the training samples as S = U St. where S; denotes the set of training samples for the i-th task. Supposing pre-trained Vision Transformer with an embedding backbone (-:; 0%) : X the parameters to be frozen in the network, it maps an input E X to the feature space F. We decompose into an efficient multi-task learner dubbed MoEfied-LoR. denoted by 10;: 0,]). where 9e indicates the set of newly employed tunable parameters and [:] refers to the concatenation operation. Subsequently, the Quality Retaining multi-task optimization mechanism as well as the router fading strategy are applied to asynchronously learning the tunable parameters 6,. Finally, we reparameterize 0g into the original backbone, achieving an efficient unified model (1; (
where 0g represents
2.3 MoEfied-LoRR
Basically, MoE-based learners benefits MTL in the following two ways. First, they enable dynamic encoding of different samples across tasks via a router and multiple experts, significantly enhancing feature diversity. Second, they reduce the number of parameters and computational cost, remarkably promoting the training and inference efficiency. However, early attempts include delicately designed
MMoE [12]. ViT [4] and Mod-Squad [13] fail establish a unified structure and inevitably introduce additional inference overhead, considering that LoRA increases inference latency by 20-30% without reparameterization [4. Recently, MoEfication methods 3T aimed at "group together the neurons that are often activated simultaneously" have demonstrated promising performance in pre-trained ViT models. Inspired by this, in this work we attempt to group together the neurons with similar weights" to construct the MoE structure. As the corresponding experts naturally meet the low-rank conditions LoRA (34), we can therefore obtain an excellent efficient multi-task learner by combining with LoRA. By employing router fading strategy to preserve the reparameterization property LoRA, A, we can further eliminate additional inference latency.
constructing effective MoE structures from pre-traine

---

Specifically, as shown in Figure 3 (b), we fistly decompose the FFN into a MoLE (Mixture of Low-rank Experts) structure. To convert the FFN in the /-th layer into MoE, we draw on insights from which view the FFN as a memory bank that retains knowledge from pre-trained models. Each column of the up-projection matrix Wup serves as a key to be matched, which each row of the down-projection matrix W down act as the corresponding value. Since similar keys in the Wa, up tend to serve similar functions, they should be grouped within the same expert. Formally, Wup is clustered column-wiss into K clusters by using the balanced k-means (37], with a mapping function C(.) that assigns the idæ-th column to cluster C(idx). Therefore, columns within the same cluster constitute an expert. Additionally, dup W down should match W up channel by channel, and also adhere to the clustering results of Wup up to construct the experts {E') liz1 the MoEfication process and then split to obtain the corresponding experts, formulated as below:
35
36
EE Consequently, they concatenated for
W = [W Daps W d: W E (((2D+)x(+D))
(3)
E' W (idolC i 1,2, K, E' E ((2D-1)x(PP)) Eipp Ejg E down
(4)
Fi.D: E E D+
(5)
Since the column vectors within these experts are relatively similar, exhibiting low-rank characteristics, they naturally satisfy the low-rank conditions in LoRA [34]. Therefore, we apply set of LoRA parameters (A up: } and {A'ownn B Blown] for E and Edown to enable more efficient learning and improved performance, which is referred as MoLÉ and formally described as below:
Exp Eip + Aup Bup
(6)
Eloonn
E Edown
A B
Moreover, to leverage the benefits of dynamic routing for extracting diverse features during training, we initially establish a sample-driven soft router for each MoLE, denote as W,, E RDxK to reweight the experts. For the (-th layer of MoLE, we calculate weights for each experts as wl by adopting the following formulation:
LN(X") W!
w = softmax
K-
(8)
T
During training, the router is employed to fully optimize the MoLE. The output X i of the decomposed FFN is calculated as follows:
K
X de
GELU(WA (X"'E + EE); ..; wwk (x. (X''E'' Σ
'K + EK )I) Edoon' + bitau (9)
2.4 Quality Retaining Optimization
The goal of multi-task learning (MTL) is to ensure that the final model performs well across all tasks. However, due to distinct task difficulties and individual optimization schedules, achieving optimal performance on all tasks simultancously with a single model can be challenging. As displayed in Figure (d), (d), despite introducing strong priors synchronize the optimization schedules across tasks, existing methods (26] 29, 25
can disrupt the inherent schedules tasks with varying difficulties, creating challenges for MTL optimization. Therefore, maintaining the inherent optimization pace of each task is crucial. Specifically, we allow asynchronous convergence of tasks and introduce the Quality Retaining (QR) MTO strategy to preserve high-quality knowledge from already converged tasks during subsequent optimization.
propose different perspective:
Specifically, at iteration iter, we maintain optimal knowledge bank Z E R.NNetaax..... which records the Exponential Moving Average (EMA) logits of each class learned during the optimization process from iteration 0 to iter 1. This knowledge is distilled into the currently optimized model using a distillation loss. Formally, we maintain the knowledge bank Z using EMA. For a sample in the current training batch with label labelg, we update Zlabel, as follows:
Ziter iter
Zlabel,
Dlabel,
- + (1 m) Ze.
(10)
5

---

where indicates the logits of sample and m E (0, 1) is momentum coefficient. This results in real-time updated knowledge repository Z.
To ensure the retention of high-quality knowledge for already optimized tasks, we employ a straight- forward method, i.e., weighting the distillation process by the reciprocal of the loss from each task. This procedure implies that tasks with lower loss (already optimized) should rely more on the learned knowledge, while tasks with higher loss (still being optimized) will depend more on the ground truth. Therefore, the Quality Retaining loss LQR for samples within a mini-batch Shatch is defined as below:
Nr LQR Σ [=1
KL (softmaxx(z) softmaxx Zlabet, )) 1(s E St),
(11)
LCE.
NE Sbatch
where KT.(-) and 1I indicates the Kullback-Leibler divergence
38 and the indicator function, respec-
tively. LCE. is the Cross-Entropy loss for the 1-th task.
Based on Eq. ( the overall training loss is formulated as L = Em LCE, + LQR. This strategy ensures that the model maintains optimal performance for already converged tasks while allowing other tasks to continue their optimization at their inherent pace.
2.5 Router Fading and Insights on the Unified Model Structure
MTL aims to develop a universal model capable of executing multiple tasks simultaneously. To achieve this goal, we transform the unified pre-trained model into an MoEfied LoRA and develop the quality retaining mechanism to preserve multi-task knowledge as discussed in the previous sections. However, the dynamic routing in MoEfied LoRA limits LoRA's capability of reparameterizing the learned parameters into unified, static pre-trained structure. To address this issuc, we design router fading strategy that gradually diminishes the router's s role in the later stages of training. This approach allows the knowledge embedded within the optimized router be implicitly absorbed follows:
w w + (1 a),
(12)
where a E [0, 1] is the off hyper-paramend Finally, as shown in Figure 3 (e), we completely remove the router after training by setting
O. In the mean time, reparametere knowledge
learned by LoRA back using Eq. (6) and Eq. 7. and concatenate them to replace the original parameters W. up: Dugs W down of the 1-th layer with W.,,
wap Bups W dowmn' below:
W up [E]
E
EK 1.
(13)
b Bup
E
E
E
K'
(14)
W down
E Edown'
down
E down
(15)
The above technique avoids extra computational costs and maintains a unified model structure, delivering a new perspective for multi-task learning.
3 Experimental Results and Analysis
3.1 Datasets and Evaluation Metric
By following 6) we mainly evaluate the performance of our proposed EMTAL method on the challenging Multi-task FGVC benchmark. In addition, we conduct experiments on the Specialized VTAB-Ik dataset to validate the effectiveness over previous solutions. Multi-task FGVC is collection of public datasets specifically for multi-task fine-grained visual classification, including CUB-200- 2011 1 (39. Stanford Cars 40). FGVC-Aircraft [41] and Oxford Flowers [42] In order to make fair comparisons, we adopt the standard training/testing split as depicted in 6 Specialized VTAB- Ik
43 consists of specialist images from specialized equipment, where we employ multi-task learning to fully leverage these expensively annotated data. We follow the standard training/validation splits used in [3 for fair comparisons. The top-1 accuracy is utilized as the evaluation metric. To further demonstrate the effectiveness of our method on the tasks of pixel-to-pixel dense prediction, also conduct experiments the NYUv2 datasct [4). Specifically, we integrate method

---

Table 1: Comparison of the top-1 accuracy (%) the Multi-task FGVC benchmark, by using ViT-B/16 supervised pre-trained on ImageNet-2IK. 'FT" denotes 'Full Fine-tuning. The best results highlighted in bold and the second best ones underlined.
Unified
CUB-200
Stanford Cars
FGVC- Aircratt
Oxford Flowers
Tunable Params. (M)
Inference Time (ms)
Method
Reference
Model
-2011
Mean
MIL. baselines
Separate
Baseline Baseline
86.4
87.6
77.2
343.92
14.30
Union FT
83.1
90.7
78.3
97.5
87.39
85.98
7.15
Gradient-based
Nash-MTL 24
ICLR' 22
88.3
90.2
80.6
99.5
89.65
2.82
7.15
Aligned MT1.
25
CVPR' 23
88.9
90.6
81.6
99.7
90.17
2.82
7.15
MTO Loss-based
GLS 26
CVPR 19
88.4
90.1
80.0
99.6
89.55
2.82
7.15
AMIL 29
ICCV 23
88.2
90.7
81.5
99.7
90.04
2.82
7.15
85.0
91.25
2.82
7.15
Efficient multi-tusk learners
Dual-Prongt
ECCV
X
87.8
73.5
53.1
99.4
78.4
1.1
15.48
Erudite o
CVPR 23
79.7
81.4
70.2
98.1
82.35
101.34
2.74
MLoRE S
CVPR' 24
74.8
59.5
49.9
99.2
70.76
188.01
42.02
MOELoRA 14
SIGIR' 24
88.4
88.2
75.0
99.7
88.04
2.82
38.71
MoKfied LoRA
Ours
88.5
91.3
81.5
99.7
90.27
1.20
7.15
EMTAL-1
Ours
90.5
91.9
81.8
99.7
90.96
0.75
7.15
EMTAL-2
Ours
90.0
83.5
91.35
0.90
7.15
EMTAL-4
Ours
89.8
92.3
85.2
99.7
91.73
1.20
7.15
with TrskPrompter 45 by applying MoEfied LoRA and QR to the FFN layers and the semantic segmentation task head, respectively.
In addition, we evaluate Multi-task FGVC for few-shot learning under 1, 2, 4, and 16 shots, by following existing works [46 47).
3.2 Implementation Details
We utilize ViT-B/16
pre-trained on ImageNet-2IK [32] optimizer [48] to fine-tune our models for 100 epochs and adopt the cosine learning rate decay with linear warm-up for 10 epochs in all experiments. We fix the hyper- paper-parameters
the base model. We the AdamW
Eq. (8) 5, since it exhibits stable performance with distinct values. As for data augmentation, we employ random resize cropping to 224 224 pixels and a random horizontal flip during training and resize to 248 248 pixels with center crop to 224 x 224 pixels. All experiments are conducted single Nvidia GeForce RTX 3090 GPU.
3.3 Comparison with the State-of-the-Art Approaches
To comprehensively evaluate the performance of our approach, we compare with the MTL full
fine-tuning baseline and the following categories of state-of-the-arr approaches: 1) MTO Loss- based methods, including GLS (26) and AMTL 29); 2) MTO Gradient-based methods, including Nash-MTL [24] and Aligned-MTL. 255 combined with vanilla LoRA-16: 3) Efficient multi-task learners methods, including Dual-Prompt [49, Erudite (6, MLoRE [5] and MOELoRA [14. As the performance of EMTAL depends on the inherent low-rank properties, we report the results of our method using distinct ranks including 1, and 4, denoted by EMTAL-1, EMTAL-2 and EMTAL-4,
respectively.
As summarized in Table the proposed EMTAL method consistently improves the performance at different ranks, promoting the top-1 accuracy the Separate full-finetuinn baseline by an average of 3.47%, 3. 3.66% and 4.24%, respectively. More importantly, EMTAL tunes only negligible amount of parameters (i.e., 1.20M) compared to the original model (i.e., 343.92M), and incurs no extra inference
VIT B/16 supervised pre-traine trained ImageNet-211

---

Table Comparison results (%) with the sfte-of-thhe-r PEFT and MTL methods on Specialized VTAB-IK 1k by using ViT-B/16 models supervised pre-trained on ImageNet-21K, 'FT' denotes 'Full Fine-tuning'. The best results are highlighted in bold and the second best is underlined.
Method
Reference
Patch Camelyon
EuroSAT
Resiso45
Retinopethy
Mean
Tunable Params, (M)
baselines
Separate FT
Buseline Baseline
79.7
95.7
84.2
73.9
83.38
343.36
Union FT [
84.3
93.9
83.0
75.2
84.08
85.99
Traditional PEFT
Adapter
50
ICML' 19
76.3
88.0
73.1
70.5
76.98
L,08
LoRA
ICLR'
85.5
95.3
86.
75.3
85.50
2.41
VPT-D
18
BCCV" 22
81.8
83.4
68.4
82.42
2.40
SSF 19
NeurIPS 22
87.4
95.9
87.4
75.5
86.56
0.96
SPT-L
ICCV'
85.7
ARC 20
NeurlPS' 23
84.9
95.7
86.7
75.8
85.78
0.52
PEFT for MTL
25
ICCV'
MOELeRA 14
SIGIR' 24
86,9
96.1
88.4
76.7
87.01
2.41
EMTAL-11
Ours
85.5
96.2
88.0
77.5
86.78
0.34
EMTAL-2
Ours
87.3
95.7
88.1
78.7
87.43
0.49
EMTAL-4
Ours
87.4
96.1
89.1
78.9
87.89
0.78
Table 3: More evaluation results on NYUv2 with ViT-B/16. The best results are highlighted bold.
Method
Semseg mloU f
Depth RMSE
Normal mFrr J
Boundary odsF
Mean A (%)
TaskPrompter-Baa 45
50.40
0.5402 0,5284
18,91
77.60
EMTAL (Ours)
52.90
18.95
77.10
1.57
cost, implying that our framework is significantly more effective and efficient than the traditional separate training paradigm.
Moreover, EMTAL consists of reparameterizable and efficient multi-task learner, Quality Retaining MTO mechanism and router fading strategy. Compared to the state-of tfte-off-hhe-art methods, each component of our approach shows significant advantages. In terms the design of MTL structures, MoEfied LoRA seamlessly integrates low-rank experts with LoRA, resulting in improved performance. Furthermore, the router fading strategy and reparameterization effectively reduce inference time, significantly enhancing the overall efficiency. Meanwhile, by considering the intrinsic task-specific optimization pace, the QR mechanism clearly improves previous MTO strategies. Overall, utilizes sample-driven router and multiple experts to extract diverse feature representations while preserving high-quality knowledge during training, making it highly beneficial for multi-task learning. In this way, method achieves the highest accuracy compared the tate-of-thee-rd approaches, surpassing the second best Aligned-MTL. by 1.56% while tuning fewer parameters.
We further evaluate the performance on Specialized VTAB-lk, where MTL is highly beneficial against previous solutions. As displayed in Table some existing works focus on applying parameter- efficient fine-tuning each task individually to avoid over-fitting. However, we find that performance can be significantly enhanced by incorporating multi-task learning. Specifically, applying AMTL and MOELoRA the vanilla LoRA yields improvements of 0.29% and 1.51%, respectively. Furthermore, when directly applying our EMTAL to these tasks, we observe consistent improvements across different ranks, achieving the highest accuracy compared to the state-of-the-a approaches. Notably, enhances the overall performance by 1. 1.43%, while utilizing fewer parameters. In addition, on the NYUv2 dataset, method significantly enhances performance in semantic segmentation and depth estimation tasks, while achieving comparable results surface normal estimation and object boundary detection tasks. Overall, this led to average relative improvement of 1.57%, validating the effectiveness of our approach for pixel-level prediction tasks. We provide more results, including using self-supervised pre-trained model DINOv2-large and the few-shot learning in Appendix B and Appendix C respectively.
3.4 Ablation Study
In this section, we evaluate the effectiveness of the proposed main components, i.e. MoEfied LoRA and Quality Retaining by extensive ablation studies.

---

Table 4: Ablation results (%) of the main components on the Multi-task FGVC by using ViT-B/16
backbone. The best results are highlighted in bold.
CUB-200
Stanford Cars
FGVC- Airerafi
Oxford Flowers
Tunable Params, (M)
MoEfied LoRA
Quality Retaining
Mean
201
x
x
83.1
90.7
78.3
97.5
87.39
86.26
X
88.5
91.3
81.5
99.7
90.27
1,20
X
V
88.5
91.6
84.4
99.6
91.04
86.26
89.8
92.3
85.2
99.7
91.73
1.20
Table 5: Ablation results (%) of the MoEfied LoRA and router fading strategy on Multi-tast tast FGVC
by using ViT-B/16 backbone. The best results are highlighted bold.
CUB-200
Stanford Cars
FGVC- Aircraft
Oxford Flowers
Tunable Params. (M)
Inference Time (ms)
Method
Mcan
-2011
Union FT
83.1
90.7
78.3
97.5
87.39
86.26
7.15
+MoEfied
85.2
91.3
80.8
98.7
89.20
86.40
13.87
+LoRA [34]
88.2
91.0
81.7
99.6
90.14
1.20
13.87
+oouter fading
81.5
99.7
90.27
1.20
7.15
Table Ablation results (%) of the proposed MoEfied LoRA with different numbers of clusters (i.e. k) and distinct ways to construct experts.
Clusters #
Method
Hyper.
16
64
192
Co-activation
Gradient- cluster
Ours
Params, (M)
1.05 88.83
1.64
2.82
1.20 89.34
1.20
Mean Ace
89.12
90.27
89.34
89.02
89.30
90.27
Effect of the Main Components. We evaluate the proposed components across the Multi-task FGVC Benchmark based on ViT-B/16. As Table shows, MoEfied LoRA consistently boost the performance, achieving an average of 2.88% improvement with less tunable parameters. The results indicate that MoEfied LoRA is significantly beneficial to multi-task learning by providing more diverse features. In the mean time, the Quality Retaining MTO mechanism can further remarkably promote the accuracy, with 3.65% improvement on average. A combination of these two components, i.e. Možfied LoRA and QR, further boosts the overall performance across datasets, implying that MoEfied LoRA and Quality Retaining are complementary in multi-task learning.
On MoEfied LoRA and Router Fading, We further validate the effectiveness of MoEfied LoRA and the router fading strategy across the Multi-task FGVC benchmark based on ViT-B/16. Initially, we begin by decomposing the pre-trained model into MOLE structure. A straightforward clustering and splitting of the FFN, combined with sample-drieen router, can achieve 1.81 % improvement by enhancing the the diversity of feature representations. Furthermore, applying LoRA to the low-rank experts yields a 0.94% gain in performance, the small amount of tunable parameters reduces the risk overfitting, and the low-rank property the experts aligns well with LoRA. Additionally, the proposed router fading strategy gradually diminishes the influence of the router over 50 epochs, effectively preserving the reparamcterizable nature of LoRA and reducing the inference time.
Morcover, we conduct more ablation studies on the proposed MoEfied LoRA. The number of clusters significantly influence the performance of MoEfied LoRA, considering that large number of clusters intends to incur simple experts with few channels, and small number of clusters results in high-rank experts, either of which degrades the effectiveness of MoEfied LoRA. We empirically study the effect of & by using 1, 4, 6, 64 and 192 clusters. As Table 6 displays, MoEfied LoRA reaches the highest accuracy when k = 16. Moreover, we compare different ways to construct experts, including the co-activation clustering [28] that groups weights based on activations for each channel and the gradient-cluster [47] that clusters weights according to cumulative gradients. As shown in Table method achieves the best performance, clearly demonstrating its effectiveness.

---

3.5
Visualization on the Low-rank Property of MoLE
MoLE (Ours)
MoLE (Ours)
Vanilla MoE
Vanilla MoE
Ky Fan 2.k-norm
0.08
Ky Fan 2-k-nom
0.06
0.06
0.04
0.04
0.03
0.02
Rank
Leyers /
(a) Low-rank properties of experts across different ranks the 4-th transformer block.
(b) Low-rank properties of experts different transformer block with fixed rank 1.
Figure 4: Comparison of the low-rank properties by using the vanilla MoE and the proposed MoLE, based on the Ky Fan 2-k norm [52] A higher value signifies a stronger low-rank property.
As shown Figure (a), we measure the low-rank properties of experts by using the Ky Fan 2-k norm [52], and the results indicate that the experts generated by our method consistently exhibit statistically more significant low-rank properties across distinct ranks. Additionally, we analyze the low-rank properties of experts across different layers of ViT when the ranks is fixed as 1. As Figure (b) demonstrates, the low-rank properties of experts are more significant lower layers that those in higher layers.
We kindly refer to Appendix E for more detailed discussion about the broader impacts and limitations of our work..
4 Conclusion
In this paper, we focus on decomposing pre-trained Vision Transformer model for multi-task learn- ing, reparameterizing the learned multi-task knowledge back into the original model and establishing a unified model. We propose novel efficient multi-task learning framework dubbed EMTAL, which mainly consists of the MoEfied LoRA module, the Quality Retaining (QR) mechanism and the router fading strategy. Concretely, MoEfied LoRA decomposes a pre-trained ViT into multi-task learners by clustering similar weight of FFN into experts and applies LoRA tune the experts with low-rank properties. Subsequently, we leverage the inherent asynchronous convergence property of tasks and employ QR to preserve the optimized performance for converged tasks. Finally, the router fading strategy introduced to eliminate extra inference cost. Extensive experiments on the public benchmarks demonstrate that method substantially promotes performance by comparing with the state-of-the-an multi-task learning approaches, while also being effective in few-shot learning with limited data. Our work delivers new perspective for efficient multi-task learning by decomposing pre-trained model and reparameterizing back with low rank updating.
Acknowledgments
This work was partly supported by the National Natural Science Foundation China (Nos. 62202034, 62176012, 62022011), the Beijing Natural Science Foundation (No. 4242044), the Beijing Municipal Science and Technology Project (No. 7231 10001033300)) the Aeronautical Science Foundation of China (2023Z07 105 1002), the Research Program of State Key Laboratory of Virtual Reality Technology and Systems, and the Fundamental Research Funds for the Central Universities.
10

---

References
[1] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems, pages 525-536, 2018
[2] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, pages 5824-5836, 2020.
[3] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, pages 18890, 2021.
[4] Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang, al. M' Mixture-of-expee vision transformer for efficient multi-task learning with mocel-accelerator co-design. In Advances in Neural Information Processing Systems, volume 35, pages
2022.
[5] Yuqi Yang, Peng- Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, and Bo Li. Multi-task dense prediction via mixture of low-rank experts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27927-27937, 2024.
[6] Dongliang Chang, Yujun Tong, Ruoyi Du, Timothy Hospedales, Yi-Zhe Song, and Zhanyu Ma. An erudite fine-grained visual classification model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7268-7277, 2023.
[7] Mickaël Binois, Victor Picheny, Patrick Taillandier, and Abderrahmane Habbal. The kalai- smorodinsky solution for many-objective bayesian optimization. Journal of Machine Learning Research, 21(150):1-42, 2020.
[8] Shikun Liu, Andrew Davison, and Edward Johns. Self-supervised generalisation with meta auxiliary learning. In Advances in Neural Information Processing Systems, 2019.
[9] Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. In International Conference Learning Representations, ICLR 2021, 2021.
[10] Enneng Yang, Zhenyi Wang. Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In International Confer- ence Learning Representations.
[11] Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations, 2021.
[12] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1930-1939, 2018
[13] Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik Learned- Miller, and Chuang Gan. Mod-squad: Designing mixtures of experts as modular multi-task learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11837, 2023.
[14] Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets Ulms: Parameter efficient fine-tuning for multi-task medical ap- plications. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1104-11
100111144 14, 2024.
[15] Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991.

---

[16] Robert A Jacobs and Michael Jordan. Learning piecewise control strategies in a modular neural network architecture. IEEE Transactions on Systems, Man, and Cybernetics, 23(2)3337- 345, 1993.
[17] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- mixture-of-experts layer. arXiv preprint arXiv: 7701. 170.00633, 2017.
[18] Menglin Jia, Luming Tang, Bor- Por-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariha- ran, and Ser-Nam Lim. Visual prompt tuning. In Proceedings of the European Conference on Computer Vision, pages 709-727, 2022.
[19] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. In Advances in Neural Information Processing Systems, pages 109-123, 2022.
[20] Wei Dong, Dawei Yan, Zhijun Lin, and Peng Wang. Efficient adaptation of large vision transformer via adapter re-composing. In Advances in Neural Information Processing Systems, 2023.
[21] Bing Li, Jiaxin Chen, Xiuguo Bao, and Di Huang. Compressed video prompt tuning. Advances in Neural Information Processing Systems, 1115111993177 2023.
[22] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pages 794-803, 2018
[23] Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. arXiv preprint arXiv:2010.05874 2020.
[24] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning bargaining game. In International Conference Machine Learning, volume 162, pages 16428-16446, 16446, 2022.
[25] Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Komushin. Independent component alignment for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20083 2023.
[26] Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019.
[27] Baijiong Lin, Feiyang Ye, Yu Zhang, and Ivor W. Tsang. Reasonable effectiveness of random weighting: litmus test for multi-task learning. Transactions on Machine Learning Research, 2022, 2022.
[28) Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1871 1881- 2019.
[29] Hayoung Yun and Hanjoo Cho. Achievement-basel training progress balancing for multi-task learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16935-16944 2023.
[30] Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer feed-forward layers are mixtures of experts. In Proceedings of Annual Meeting of the Association for Computational Linguistics, pages 877-890, 2022.
[31] Mikolaj Piórczyúski, Filip Szatkowski, Klaudia Balazy, and Bartosz Wojeik. Exploiting transformer activation sparsity with dynamic inference. arXiv preprint arXiv:2310.044.
2023.
12

---

[32] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998- 6008, 2017.
[34] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Aluen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[35] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, 2021.
[36] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of Annual Meeting of the Association for Computational Linguistics, pages 8493-8502, 2022.
[37] Mikko I Malinen and Pasi Fränti. Balanced k-means for clustering. In Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshop, pages 32-41, 2014.
[38] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arXiv preprint 1rxxiv:503302533 2015.
[39] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-201 dataset. 2011.
[40] Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine- grained car detection for visual census estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4502-4508, 2017.
[41] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint 13Xiv:1306.51555 2013.
[42] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729, 2008
[43] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucie, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint w:Xiv:1910.048000 2019.
[44] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Proceedings of the European Conference on Computer Vision, 2012.
[45] Hanrong Ye and Dan Xu. Taskprompter: Spatial-channel multi-task prompting for dense scene understanding. In International Conference on Learning Representations, 2023.
[46] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vasion-language models. International Journal of Computer Vision, 130(9))2337-2348, 2022.
[47] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.04673 2022.
[48] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
13

---

[49] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot. Jennifer Dy. et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In Proceedings of the European Conference Computer Vision, pages 631-648, 2022.
[50] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learming, pages 2790-2799, 2019.
[51] Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-awar visual purameter-efficieet tine-tuning In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1825-118333 2023.
[52] Xuan Vinh Doan and Stephen Vavasis. Low-rank matrix recovery with ky fan 2-k-norm. J. Glob. Optim., 82(4):727-751, 2022.
[53] Rich Caruana. Multitask learning. Machine Learning, 28:41-75, 1997.
[54] Michael Crawshaw. Multi-task learning with deep neural networks: A survey, arXiv preprint arXiv:2009.09796. 2020.
[55] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint SoXiv:1706.050900 2017.
[56] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey, IEEE Transactions Pattern Analysis and Machine Intelligence, 44(7):3614-3633 2021.
[57] Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-enn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3205-3214, 2019.
[58] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3994-4003, 2016.
[59] Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi- task architecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4822-4829, 2019.
[60] David Bruggemann, Menelaos Kanakis, Stamatios Georgoulis, and Luc Van Gool. Automated search for resource-efficient branched multi-task networks. arXiv preprint arXiv:2008. 10292, 2020.
[61] Pengsheng Guo, Chen-Yu Lee, and Daniel Ulbricht. Learning branch for multi-task learning. In International Conference on Machine Learning, pages 3854-3863, 2020.
[62] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7482-7491, 2018.
[63] David Brüggemann, Menelaos Kanakis, Anton Obukhov, Stamatios Georgoulis, and Luc Van Gool. Exploring relational context for multi-task dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15869-15878, 2021.
[64] Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Mti-net: Multi-scale task interaction networks for multi-task learning. In Proceedings of the European Conference on Computer Vision, pages 527-543, 2020.
[65] Ling Zhou, Zhen Cui, Chunyan Xu, Zhenyu Zhang, Chaoqun Wang, Tong Zhang, and Jian Yang. Pattern-structure diffusion for multi-task learning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 451 4514-4523, 2020.
14

---

[66] Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. propagation across depth, surface normal and semantic segmentation. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 4106-4115, 2019.
[67] Zhenyu Zhang. Zhen Cui, Chunyan Xu, Zequn Jie, Xiang Li, and Jian Yang. Joint task- recursive learning for semantic segmentation and depth estimation. In Proceedings of the European Conference Computer Vision, pages 235-251, 2018.
[68] Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas J Guibas. Robust learning through cross-task consistency. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11197-11206, 2020.
[69] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1.39-149, 2022.
[70] Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In Advances in Neural Information Processing Systems, volume 33, pages 2039- 2050, 2020.
[71] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision, pages 270-287, 2018.
[72] Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu. A modulation module for multi-task learning with applications in image retrieval. In Proceedings of the European Conference on Computer Vision, pages 401-416, 2018.
[73] Siwei Yang, Hanrong Ye, and Dan Xu. Contrastive multi-task dense prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3190-3197, 2023.
[74] Wei-Hong Li, Xialei Liu, and Hakan Bilen. Learning multiple dense prediction tasks from partially annotated data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18879-18889, 2022.
[75] Weiyi Lu, Sunny Rajagopalan, Priyanka Nigam, Jaspreet Singh, Xiaodi Sun, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Asynchronous convergence in multi-task learning via knowledge distillation from converged tasks. In NAACL, pages 149-159, 2022.
[76] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23120)) 2022.
[77] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668. 2020.
[78] Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts. arXiv preprint arXiv:21 10.00000 04.04200 2021.
[79] Hanrong Ye and Dan Xu. Taskexpert: Dynamically assembling multi-task representations In Proceedings of the IEEE/CVF International Conference
with memorial mixture-of on Computer Vision, pages 21 21828-21837, 2023.
[80] Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1060-1068, 2023.
[81] Yen-Cheng Liu, Chih-Yar Ma, Junjiao Tian, Zijian He, and 7solt Kira. Polyhistor: Parameter- efficient multi-task adaptation for dense vision tasks. volume 35, pages 36889-36901, 2022.
15

---

[82] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. V1-adapter: Parameter-efficient transfer learning for vision-and aade-and--add -an-gag tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5227-5237, 2022.
[83] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint wxXi:22012.3325. 2020.
[84] Yerlan Idelbayev and Miguel A Carreira-Perpinae Learning the rank of each layer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8049 8059, 2020.
Low-rank compression of neural nets:
[85] Chi Su, Fan Yang. Shiliang Zhang, Qi Tian, Larry S Davis, and Wen Gao. Multi-task learning with low rank attribute embedding for person re-identification. In Pruceedings of the IEEE International Conference on Computer Vision, pages 3739-3747, 2015.
[86] Madeleine Udell, Corinne Horn, Reza Zadeh, Stephen Boyd, et al. Generalized low rank models. Foundations and Trends@ in Machine Learning, 9(1):1-118, 2016.
[87] Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-task learning. arXiv preprint arXiv:1606. 04038, 2016.
[88] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient tross-task generalization via dynamic lora composition. arXiv preprint drXiv:2307.13269, 2023.
[89] Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit- Yeung, James T Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision-language instruction tuning. arXiv preprint arXiv:2312.113399 2023.
[90] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhi- heng Xi, Xiao Wang, Xiaoran Fan, et al. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. arXiv preprint 17X19223/..0997, 2023.
[91] Zeren Chen, Ziqin Wang, Zhen Wang. Huayang Liu, Zhenfei Yin, Si Liu, Lu Sheng, Wanli Ouyang, and Jing Shao. Octavius: Mitigating task interference in milms via lora-moe. In International Conference on Learning Representations, 2024.
[92] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024, 2024.
[93] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: fine-tuning for transformer based masked language-models. In Proceedings of Anmual Meeting of the Association for Computational Linguistics, pages 1-9, 2022.
Simple parameter-efficient
[94] Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parameter-efficient fine-tuning. arXiv preprint arXiv:2306.077 2023.
[95] Zeju Qiu. Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In Advances in Neural Information Processing Systems, pages 79320-79362. 2023.
[96] Weiyang Liu, Zeju Qiu, Yao Feng. Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. Parameter-efficient orthogonal finetuning via butterfly factorization. In International Conference on Learning Representations, 2024.
[97] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rota- tion equivariant cnns for digital pathology. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018
[98] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEFE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
16

---

[99] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 2017.
[100] Kaggle and EyePacs. Kaggle diabetic retinopathy detection, July 2015.
[101] Jiaxi Wu, Jiaxin Chen, Mengzhe Hc, Yiru Wang, Bo Li, Bingqi Ma, Weihao Gan, Wei Wu, Yali Wang, and Di Huang. Target-relevant knowledge preservation for multi-source domain adaptive object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5301-5310, 2022.
[102] Wu Ke, Jiaxin Chen, and Miao Wang. Domain adaptive object detection for uav-based images by robust representation learning and multiple pscudo-label aggregation. In Proceedings of the 1st International Workshop on Efficient Multimedia Computing under Limited, pages 59-67, 2024.
[103] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664, 2021.
17

---

A Related Work
A.1 Multi-Task Learning
Multi-Task Learning (MTL.) 5 aims to improve the generalization performance of models across individual tasks by leveraging shared representations exploit the commonalities and interdepen- dencies between tasks. This approach also reduces the number of parameters and accelerates both training and inference [53 54 55). Existing works are roughly divided into two categories: 1) the efficient multi-task learners and the multi-task optimization methods [56].
Efficient multi-task learners. These approaches concentrate on analyzing the impact of parameter sharing within models and can be broadly categorized into encoder-focused and recoder-focusee approaches, depending on where information is exchanged shared between tasks [56]. In encoder- focused models, task parameters are shared exclusively within the encoder to extract general features,
by leveraging mechanisms such as feature fusion (57 58 59] attention [28 6 and dynamic branching [60] [11, while the decoder consists of independent task-specific heads with no cross-
task information sharing [22 62 1 In decoder-focused models, parameters shared across tasks within the decoder. The model initially makes separate predictions for each task and then refines these results by leveraging inter-task correlations through mechanisms such as multi-model distillation 663 64 65 66). sequential task prediction [67], or cross-task consistency [68]. Some continual learning frameworks (69 49] are also applicable to MTL, incorporating task-specific and task-shared parameters to achieve effective isolation and sharing between tasks.
Multi-Task Optimization Methods. The optimization- based methods 22 70
25 24
focus on balancing how tasks are learned, exploring effective solutions from the perspective of model optimization. These approaches enhance the optimization process of MTL through various design of
multi-task losses [26] 68| 62 73 74 29), which assigns appropriate loss weights to minimize conflicts
among tasks. (75) proposes recording the optimal checkpoint for each task and learning from it by
distilling the soft labels of individual samples. However, this approach may record a local optimal solution, leading to suboptimal results. Gradient manipulations [70 22 23 3 techniques address task interference by directly adjusting gradients, with recent methods emphasizing the formulation of a unified gradient vector subject to diverse constraints.
A.2 Mixture-of-Expent
Mixture-of-Experts (MoE) 15 are originally designed to combine the decisions of a series sub- models, i.e., expert networks, the same input, enhancing conditional computational capabilities and enabling the scaling of parameters in neural networks [7 17 78). Therefore, MoE trains multiple specialized expert networks, each of which maintains its unique set of trainable parameters. This design allows the expert networks to develop distinct internal representations tailored to their respective input data. Additionally, MoE employs router that dynamically weights the outputs of each expert network, enabling their contributions to be combined into the final output.
MoE is also applied in Multi-Task Learning (MTL), effectively partitioning the parameter space and leveraging relevant model components for different tasks, making it a promising solution for MTL [6 17]. ViT 44] customizes the MoE layer within a VIT backbone, and activates task-specific experts during training to mitigate gradient conflicts in MTL. Mod-Squad [ introduces a modular multi-task leaner based on MoE, along with novel loss function, address the gradient conflicts among tasks. MMoE [1] designs a multi-gan gate MoE to ensemble expert networks for various census analysis tasks, each with a different router. TaskExpert [79] generates multi-task predictions for all tasks in a single forward pass simultancously. leading to significantly higher multi-task training efficiency. MLoRE (5) explicitly builds global relationships among all tasks within the MoE structure and introduces low-rank experts, improving the efficiency of MoF. compared to the vanilla approach.
A.3 Low-Rank Updating
Low-Rank Adaption (LoRA) 34 is a parameter- fine-tuning method 34 80 81
82
inspired
by the observation from 83 that the difference in weights between the pre-trained model and the adapted model lies in low intrinsic rank. With the success of low-rank structure [84] a parameter-efficient fine-tuning technique, numerous studies have demonstrated impressive results
85
86 87
as
by combining LoRA and MoE for more efficient and effective model tuning. LoRAHub [88 first
18

---

trains multiple LoRA weight modules on upstream tasks. To adapt the model to a downstream task, employs a gradient-fre optimization method to determine the optimal coefficients for linearly combining the pre-trained LoRA modules. MOELoRA [14] utilizes router network conditioned on task identifier to dynamically combine the outputs of multiple LoRA experts. Similarly, MoCLE [89] designs a router network that is conditioned on the clustering information extracted from each input sample. LoRAMoE [90] splits the LoRA experts into two groups and explicitly learns distinct capabilities for each group. While these mixture-of-LoRA methods densely combine multiple LoRA experts, a sparse mixture LoRA experts offers a more economical alternative, achieving comparable performance while maintaining roughly constant training and inference costs. The Octavius 91 method, for instance, selects the top-2 LoRA experts based on a router that conditions on the entire input instance, representing a more coarse roarse-grained routing mechanism. Besides, earlier MTL. methods use the low-rank structure to model task-generic features and generate task-specific features through linear combinations. VI.-Adapterf [2] exploits adapter-based methods efficiently fine-tune generative models in a multi-task setting, while [87] introduces a framework for training multiple neural networks simultaneously, sharing all shareable layers and learning the sharing strategy in data-driven manner.
B More Experimental Results on DINOv2-large
Table 7: Comparison results (%) with the state-of-the-ar PEFT approaches on the Specialized VTAB- 1k benchmark by using the supe-vipe pre- pre-training ViT-L/14 model, i.e., DINOv2 2-aagee [92].
'FT' denotes Fuull Fine-tuning'. The best results are highlighted in bold and the second best ones are underlined.
Unilied Model
Patch Camelyon
Tunable Params. (M)
Method
Reference
EuroSAT
Resise45
Retinopathy
Mean
Separate FT
Baseline
X
88.1
96.1
90.9
77.2
88.00 38.08
1217.6
BirFit
ACL' 22
X
85.2
96.1
90.7
75.7
86.92
1.08
FacTu 80
AAAI 23
X
87.1
88.7
74.0
86.02
0.48
FacTik 80
AAAI 23
X
86.1
89.5
74,2
86.10
LoRA 34
ICLR' 22
X
88.3
26.4
91.4
27.4
88.38
7.08
GLoRA 9
arXiv' 23
X
85.9
96.0
91.0
76.2
87.27
19:48
OFT 3
NeurlPS' 23
X
88.4
96.4
91.5
77.2
88.38
8.40
BOFT 96
ICLR'S 24
88.9
96.6
91.6
77.3
88.60
7.96
EMTAL-4
Ours
89.4
95.9
91.7
80.1
89.27
2.03
In addition to the ViT-B/16 pre-trained model as displayed in Table we evaluate the performance of our method based on the self-supervised pre-training ViT-L/14 model, i.e., DINOv2-large F As summarized in Table 7 our method consistently outperforms the compared approaches. Notably. compared with the separate full-finetuning (Separate FT) baseline which incurs 1217.6M tunable parameters (4 x 304.4M) and the other alternative methods, EMTAL successfully constructs a unified multi-task framework on this benchmark. Our approach eliminates the need to develop multiple specialized models while delivering superior performance, thereby validating its generalizability when applied larger self supervisdd models.
C More Experimental Results on Few-shot learning
We also evaluate the performance of our method on the challenging few-shot learning task. As illustrated in Figure the following observations can be made regarding the average performance (i) EMTAL, LoRA, Adapter, and VPT exhibit similar performance with extreme limited amount of training data, such with only 1 or 2 shots. (ii) As the amount of training data increases, EMTAL demonstrates a significant advantage. For instance, with shots, EMTAL promotes the accuracy of the second best one by approximately 5%. This highlights that multi-task training has a particularly significant advantage over the separate training, especially in scenarios where reliable annotation information is limited.
DINOv2-
19

---

Performeene on CLB-206-2011
Pasformanc Barifoed Cara
Parformence
Flowas
60
100
62-
70-
7D-
50
Aoc (35)
Aoc
30-
72
Oln VP7
Oun V2T
1P7
SPT
UFT
100
1c0
Adglo:
Adapt
Adple
Adppe
Adpler
Shots
Shots
Figure 5: Comparison results using various separate training approaches in the context of few-shot learning on the multi-task FGVC datasets.
D Detailed Descriptions for the Evaluation Datasets
We provide detailed descriptions about the datasets used for evaluation. The train/val/test splits and the number of classes are summarized in Table
Table 8: The statistics of the datasets used for evaluation. The train/val/test splits are the same as depicted [6]. indicates that the corresponding split is not available.
Dataset
Description
#Classes
Training set
Val.
Test set
Multi-task FGVC
(UB-200-201] 39
Bird species recognition
200
5,994
5,794
Stanford Cars 40
Car classification
196
8,144
8,041
AGrC-Ai 41
Aircraft classification
100
6,667
3,333
Oxford Flowers
Flower species recognition
102
2,040
6,149
Specialized VTAB-IK Ik [43
Patch Camelyon 99
32,768
EuroS AT [98
5,400
Specialized
800/1,000
200
Resisc45
45
6,300
Retinopathy 100
5
42,670
employ the Multi-task Fine-Grained Visual Classification datasets to evaluate the performance of our proposed EMTAL, which consists of CUB-200-2011 (3). Stanford Cars [40. FGVC-Aireraft [41] and Oxford Flowers [42). We also employ Specialized VTAB-1K (43) consisting of specialist images from specialized equipment, where multi-task learning is applied to fully leverage these annotated data.
By following Erudite 66.
E Limitations and Broader Impacts
Our work may have the following potential impacts. Firstly, compared to traditional multi-task learning approaches, EMTAL maximizes the use of limited data for downstream multi-task learning. This capability facilitates the rapid transfer of large models pre-trained on vast datasets to downstream tasks, significantly conserving computational resources. Secondly, our method is designed based on reparameterization, allowing the model to be transferred to downstream tasks without altering the deployed backbone architecture. This approach, which involves simply replacing a set of weights, is more convenient than many multi-task learning methods [4 13
Regarding limitations, our model structure currently employs unified rank across all experts. Despite that this design benefits parallel computing during training, our experiments reveal that different tasks have varying difficulties, necessitating different optimal ranks. For example, the optimal rank for the CUB-20 208-200--011 1 dataset is one, whereas for the Stanford Cars dataset, it is four. Thus, dynamically and adaptively selecting the appropriate ranks for different tasks to improve multi-task learning is promising research direction. Additionally, we observed that the low-rank property of shallow experts more pronounced than that of deep experts the network, suggesting that dynamically adjusting the rank of experts across different layers could lead to improved performance. Consequently,
20

---

adaptively assigning the appropriate rank to these experts is potential key focus of future research. Furthermore, while our method seeks establish unified multi-task model for practical applications. it is essential to consider potential out-of-distribution (OOD) issues that may arise during deployment, as they could impact the performance. To address this challenge and enhance the applicability of our work in real-world scenarios, future research could explore the integration of domain adaptation techniques [101 [02]. Investigating datasets such as WILDS [103), which are tailored for OOD challenges, could provide valuable insights and further improve performance across diverse contexts.
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Question:
Answer:[Yes Yes]
Justification:
We describe it in Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We describe it in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
No theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe it in Section 3.2 and will release code upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [No]
Justification; We describe details of public datasets in Section provide details of our method, and will release code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp wwwpppblccgguiecuceccccceooooope /eegguddsse/Ceebbbinninn htt.SSbbbiiiiooooiiiiiooooi. for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe it in Section D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We describe it in Section 3.5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe it in Section 3.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used the paper, are properly credited and the license and terms of use explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "hgqBzOrRLz", "reviewer": "Reviewer_2Lcf", "summary": "The paper introduces a method to decompose ViT MLP matrices into a mixture of experts based on channel similarity, training each expert with LoRA. The authors also use a quality-retaining loss to effectively optimize during training for multi-task learning.", "questions": "Lines 64-69 are unclear and can be worded differently.\n\nA better description of Figure 2 is required. A summary for each of 2a and 2b would assist in clarity.\n\nline 129: “owe” -> “ought”\n\nNotation is a bit confusing: eg: usually “r” is used for lora ranks, but here it’s a scaling factor\n\nline 255: how exactly is inference time being reduced? At the end of training, the LoRA experts are being merged back into the weights of the model, right?\n\nTable 4: What is PoolFT?\n\nEquation 14: clarify what i is here. Notation seems to be overloaded? Is i the iteration, or the column? Previously, i was used to denote experts.\n\nEquation 15: Clarify what L_t is. There also seems to be a L_CE,t. Are these the same?", "limitations": "The authors have adequately described these sections.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**(S1):** The idea of decomposing the weight matrix into experts based on channel similarity and then using LoRA to tune them is an interesting and creative idea. \n\n**(S2):** The quality-retaining loss is a nice addition to the optimization process.\n\nOverall I think the idea is interesting and may be relevant for other researchers to continue exploring the combination of MoE with LoRA/PEFT for ViTs.", "weaknesses": "**(W1):** Ablation study is not extensive enough. I am interested in seeing whether the router is needed at all, and whether you can just set alpha=0 from the very beginning. As a follow up question: I haven’t closely checked, but does setting alpha = 0 make the method equivalent to LoRA?\n\n**(W2):** Line 204: The models used seem to be supervised fine-tuned backbones, which are not really designed to be generalizable to multiple tasks. The authors should consider experiments with self-supervised or unsupervised backbones like MAE [1] or DinoV2 [2], and compare against these backbones’ generalization capabilities. The authors should also clarify what the source of weights are for all the other models used in comparison, and which model sizes are being used.\n\n**(W3):** The distribution of tasks is not necessarily too different. Consider evaluating the method on tasks which represent greater domain shifts (eg: WILDS [3]).\n\n**(W4):** The comparisons against other PEFT techniques in Table 1, 2 is missing. There are numerous PEFT methods with improvements beyond the original LoRA that are not mentioned. Eg: if we look at the VTAB table in BOFT [4], we see that the EMTAL results are not state-of-the-art.\n\n---\nReferences:  \n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).\n\n[3] Koh, Pang Wei, et al. \"Wilds: A benchmark of in-the-wild distribution shifts.\" International conference on machine learning. PMLR, 2021.\n\n[4] Liu, Weiyang, et al. \"Parameter-efficient orthogonal finetuning via butterfly factorization.\" arXiv preprint arXiv:2311.06243 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestions. Our point-to-point responses are summarized as below.\n> W1. On ablation study.\n\nFirstly, the setting alpha=0 from the very beginning is not equivalent to the vanilla LoRA without clustering. We refer to this as Cluster+Finer LoRA, and provide extra ablation results by adding the router and the router fade strategy, summarized as below:\n| | Mean Acc | Params. (M) | Time (ms) |\n| :-- | :- | :- | :- |\n| Vanilla LoRA | 88.83 | 1.05 | 7.15 |\n| Cluster+Finer LoRA | 89.93 | 1.05 | 7.15 |\n| Cluster+Finer LoRA+Router | 90.14 | 1.20 | 13.87 |\n| Cluster+Finer LoRA+Router Fade (Ours) | **90.27** | 1.20 | **7.15** |\n\nThe results clearly display the effectiveness of the router.\n\n> W2. On the backbone.\n\nThe ViT-B/16 backbone [1] used in our experiments is supervised pre-trained on ImageNet-21K, which is widely used in previous works, and is adopted in our paper for comparisons. However, we do agree that exploring other backbones is necessary. Therefore, we conducted additional experiments using the backbone pre-trained by MAE [2] on the Multi-task FGVC benchmark. The results are summarized below:\n| Method | Pre-trained Model | CUB-200 -2011 | Stanford Cars | FGVC Aircraft | Oxford Flowers | Mean Acc | Params. (M) |\n| :- | - | - | - | - | - | :- | - |\n| MOELoRA | MAE ViT-B/16 | 81.5 | 90.7 | 82.7 | 97.7 | 88.13 | 2.82 |\n| AMTL | MAE ViT-B/16 | 83.9 | 91.6 | 84.8 | 97.7 | 89.53 | 2.82 |\n| EMTAL-4 | MAE ViT-B/16 | 85.3 | 92.1 | 85.1 | 97.7 | 90.07 | 1.20 |\n| EMTAL-4 | Supervised ViT-B/16 | 89.8 | 92.3 | 85.2 | 99.7 | 91.73 | 1.20 |\n\nThe experimental results indicate that our method is effective compared with the SOTA MTL methods by using the self-supervised pre-trained backbones, but fails to surpass that of the supervised ones.\n\n>W3. On more complex tasks.\n\nWe acknowledge the importance of evaluating our method on tasks with greater domain shifts. Several reviewers have raised this point, and we agree that it is crucial to validate our approach on a broader range of benchmarks, including NYUv2, WILDS, PASCAL-Context, and Taskonomy benchmark. Since time constraints limit the comprehensive evaluation on these benchmarks, we perform evaluation the NYUv2 benchmarks to further validate our method's performance on tasks with significant domain differences.\nThe NYUv2 dataset includes tasks such as Semantic Segmentation, Monocular Depth Estimation, Surface Normal Estimation, and Object Boundary Detection. These tasks exhibit substantial domain differences, making NYUv2 a suitable benchmark for evaluating our method's robustness to domain shifts.\n| Method | Semseg mIoU ↑ | Depth RMSE ↓ | Normal mErr ↓ | Boundary odsF ↑ | Mean Δ (%) ↑ |\n| :- | - | - | - | - | :- |\n| TaskPrompter-Base | 50.40 | 0.5402 | 18.91 | 77.60 | - |\n| + EMTAL | **52.90** | **0.5284** | 18.95 | 77.10 | **1.57** |\n\nThe results indicate that our method performs well across tasks with significant domain differences, demonstrating its robustness and generalization capabilities.\n\n> W4. On missing comparison of PEFT methods.\n\nTable 2 is designed not only to validate the extensibility of our method across more benchmarks but also to compare our approach with existing single-task efficient fine-tuning paradigms. This highlights the effectiveness of our multi-task PEFT method. Therefore, we additionally included comparisons with public results of existing PEFT methods in Table 2.\n\nBOFT, GPS [Ref.1] and LoSA [Ref.2] are published after or very close to the submission date of NeurIPS, which are missed in our submission. But we do agree it will make the comparison more compherehensive to add them for comparison. However, unlike most existing works, BOFT employs a larger DINOv2-Large backbone instead of the standard supervised ViT-B/16, making it unfair to directly compare. Due to the time limitation, we mainly add GPS and LoSA for fair comparisons as below.\n| Method | Reference | Patch Camelyon | EuroSAT | Resisc45 | Retinopathy | Mean Acc | Params. (M) |\n\n\n| Method | Reference | Patch Camelyon | EuroSAT | Resisc45 | Retinopathy | Mean Acc | Params. (M) |\n| :- | - | - | - | - | - | :- | - |\n| GPS[Ref. 1] | CVPR' 24 | 87.5 | 96.7 | 88.1 | 76.1 | 87.1 | 1.00 |\n| LoSA[Ref. 2] | CVPR' 24 | 86.6 | 97.1 | 87.0 | 76.7 | 86.85 | 0.77 |\n| EMTAL-4 | Ours | 87.4 | 96.1 | 89.1 | 78.9 | **87.89** | 0.78 |\n\n\n[Ref. 1] Zhang, Zhi, et al. \"Gradient-based Parameter Selection for Efficient Fine-Tuning\". In CVPR, 2024.\n\n[Ref. 2] Otniel-Bogdan Mercea, et al. \"Time- memory- and parameter-efficient visual adaptation\". In CVPR, 2024.\n\n\nThe results demonstrate the effectiveness of our method compared to existing single-task efficient fine-tuning methods. Our approach not only achieves the state-of-the-art performance but also avoids selecting from multiple specialized models trained separately, thereby improving the inference speed.\n\n> Q1-Q3. On presentations in L64-69, Figure 2, L129. \n\nThanks for your suggestions on the presentations. We will carefully improve the presentation and description in the final version.\n\n> Q4&Q6&Q7. On the notation r, PoolFT, Eqs. (14) and (15).\n\nThanks for pointing out these typos. PoolFT should be Union FT that fine-tunes on the joint feature space. i is indeed the iteration step and L_t shoud be $ L _ {CE,t} $. We will correct them in the revision.\n\n> Q5. On reduction of inference time.\n\nIndeed, the LoRA experts are merged back into the pre-trained model. As shown in Table 4, the inference time is reduced from 13.87ms to 7.15ms. \n\n\n[1]: ViT-B/16 supervised pre-trained on ImageNet-21K. https://storage.googleapis.com/vit_models/augreg/B_16-i21k-300ep-lr_0.001-aug_medium1-wd_0.1-do_0.0-sd_0.0.npz\n\n[2]: ViT-B/16 Self-supervised MAE. https://dl.fbaipublicfiles.com/mae/pretrain/mae_pretrain_vit_base.pth", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VhRZMMZ8cX", "reviewer": "Reviewer_fvpH", "summary": "Considering recent trends in multi-task learning (MTL) that design Mixture-of-Experts (MoE) structures and integrate Low-Rank Adaptation (LoRA), this paper proposes Efficient Multi-Task Learning (EMTAL) to address the sub-optimal performance resulting from the rigid combination of MoE optimization and LoRA's reparameterization capability. The approach decomposes the pre-trained Transformer into a low-rank MoE and employs LoRA to tune the experts, termed MoEfied LoRA. Additionally, the paper introduces a Quality Retaining (QR) multi-task optimization strategy that uses historical high-quality class logits to prevent performance degradation.", "questions": "Refer to the weakness section.", "limitations": "Refer to the weakness section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Proposed MoEfied approaches are practical applications that can be applied to real-world scenarios. Reparameterizing pre-trained transformer models is a novel approach.\n2. The concept of \"Grouping together neurons with similar weights\" is both intuitive and plausible.\n3. Asynchronously optimizing the learner by using historical class logits to prevent performance degradation is also a plausible approach.", "weaknesses": "1. The proposed idea of grouping similar weights to compose experts seems quite naive. Further justification for this approach is needed, such as theoretical analysis of how different grouping strategies (e.g., weight, activation) lead to better multi-task performance. Additionally, ablation experiments for various grouping strategies would be beneficial.\n\n2. I'm not convinced that the proposed methods are superior to previous works (e.g., MLoRE) since most experiments in the paper are conducted on homogeneous task settings, focusing on visual classification tasks. Can the proposed approach be generalized to heterogeneous task settings involving various types of vision tasks using benchmarks such as Taskonomy or PASCAL-Context? It seems previous works might suffer from overfitting by naively dividing classes into task groups.\n\n3. Quality Retaining (QR) appears to be applicable only to classification tasks, limiting its application in the context of MTL. It should be compared to previous multi-task optimization approaches mentioned in the paper, which can be applied orthogonally to architecture design.\n\n4. In similar context in 3, experimental setting for comparision with previous works is not appropriate. Direct comparison with proposed methods with loss-based and gradient-based optimization is not appropriate as proposed method mainly focuses on architecture design approach.\n\n5. The concept of reparameterizing a pretrained transformer into an MoE-like structure is novel. However, I am unsure if it is competitive in terms of performance. It would be helpful to demonstrate that the proposed methods show competitive performance compared to learning a similar structure from scratch, if possible.\n\n6. Previous work, such as MOELoRA also insists that their framework is a unified network. Please explain clearly that what is the main difference between your work and previous works. why the proposed method is the only unified model.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your insightful comments and suggestions. Our responses are summarized as below:\n> W1. On grouping strategies.\n\n \nA1. To the best of our knowledge, our work is the first one that employs the idea of grouping similar weights to establish the LoRA experts, and there lack grouping strategies that can be directly used for comparisons. In our opinion, this design incurs the following two advantages: \n\n1). The experts built by grouping have more specialized functions as described in [28], as well as low-rank properties, thus being natural for updating through LoRA. \n\n2). LoRA learns low-rank updates [32] for each expert, restraining the rank of the experts' weights from increasing, which in turn promotes the specialization of the experts. \n\nAs aforementioned, there lack grouping strategies that can be directly used for comparison. Inspired by existing works, we implement the following two methods for comparison: 1) Co-activation similar to [28] that clusters weights based on the activations for each channels; 2) Gradient-cluster inspired by SPT [47] that clusters weights based on the cumulative gradients.\n\nAs displayed below, our method reaches the best results, clearly demonstrating its effectiveness.\n \n| Method | CUB-200-2011 | Stanford Cars | FGVC Aircraft | Oxford Flowers | Mean Acc | Params. (M) |\n| :--------------- | ------------------- | ------------------ | ------------------ | ------------------- | :------- | ----------- |\n| Co-activation [28] |88.1|89.8 | 79.5 | 99.6|89.3 | 1.20 |\n| Gradient-cluster [47]|88.4| 89.9 |79.2|99.7| 89.34 | 1.20 |\n| Ours| 88.5|91.3|81.5|99.7|90.27|1.20 |\n\n> W2: Generalizability to the heterogeneous task settings.\n\nThanks for your suggestion. We mainly follow the work [6], and evaluate on FGVC for fair comparisons. However, we do agree that FGVC adopts a homogeneous task setting. The other reviewer also mentioned this issue, and suggests more results on other heterogeneous benchmarks (e.g. NYUv2). Due to the time limitation, we conduct experiments on NYUv2, as it contains four substantially heterogeneous tasks including Semantic Segmentation, Monocular Depth Estimation, Surface Normal Estimation, and Object Boundary Detection. For comparison, as the released code of the suggested MLoRE approach has some unfixed issues and fails to reproduce the reported results after contacting the authors, we finally select another open-source SOTA method TaskPrompter [Ref.1] for comparison. In implementation our method, we follow the framework of the existing open-source SOTA method TaskPrompter [Ref.1]. For the architecture, we cluster and divide the FFN into experts and apply router fade. For the multi-task optimization, we apply QR to the classification branch of the Semantic Segmentation task head, and omit it in other tasks. \n\nThe comparison results are summarized below:\n \n \n| Method | Semseg mIoU ↑ | Depth RMSE ↓ | Normal mErr ↓ | Boundary odsF ↑ | Mean Δ (%) ↑ |\n| :-- | -- | -- | -- | --- | :--- |\n| TaskPrompter [Ref.1] | 50.40 | 0.5402 |18.91|77.60 | - |\n| + EMTAL | **52.90** | **0.5284** |18.95 |77.10 |**1.57**|\n \n\n\nAs displayed, our method promotes TaskPrompter by 1.57% on average, showing the effectiveness of our method for the heterogeneous task settings. \n\n[Ref.1] Ye, Hanrong, et al. \"Taskprompter: Spatial-channel multi-task prompting for dense scene understanding\". In ICLR. 2023.\n\n\n>  W3&W4: Applicability of Quality Retaining (QR), and comparison to previous multi-task optimization (MTO) methods.\n\n\nIntuitively, QR can be integrated to any task (e.g. detection and segmentation) that contains the classification branch, thus being applicable to typical computer vision tasks including the segmentation. Due to the time limitation, we evaluate our overall method on the segmentation task on NYUv2, by comparing to the SOTA approach TaskPrompter with the ViT-Base backbone. As shown in the second column of the table above, the results clearly displays the applicability and effectiveness of our method on the segmentation task beyond classification. \n\nWe have also add comparison results between QR with the representative MTO methods including the loss based approaches GLS [24] and ATML [27], and the gradient based approaches Nash-MTL [22] and Aligned-MTL[23]. For fair comparisons, we use the same architecture design for all compared methods. The results in Table A in \"Rebuttal.pdf\" clearly show the effectiveness of QR.\n\nWe will add the discussion and results in the final version, and further study the extension of QR to more tasks in future.\n\n\n> W5: Comparison to training from scratch.\n \n\nAs shown in the table below, training a ViT from scratch usually yields extremely poor results, due to overfitting on the limited training data in downstream tasks. In contrast, our method reparametrizes a pre-trained transformer into an MoE-like structure, providing a meaningful initialization, thus leading to better performance. \n \n| Method | CUB-200 -2011 | Stanford Cars | FGVC Aircraft | Oxford Flowers | Mean Acc | Params. (M) |\n| :--------------- |--------------| -------------|-------------|-------------- | :-------|-----------|\n| Training from scratch* | 18.5 | 14.2 | 15.3 | 54.8 | 25.7 | 86.70 |\n| Reparameterization (Ours) | 89.8 | 92.3 | 85.2 | 99.7 | 91.73 | 1.20 |\n\n\n*indicates the best results after carefully tuning the hyperparameters. \n \n \n> W6: On unified model.\n \n\nMost existing approaches including MOELoRA and MLoRE reparameterize the LoRA experts based on the task-driven router's weights, by preserving a specialized model or a bypass branch for each task during inference, thus failing to generate a unified model for different tasks. \nIn contrast, by using the reparameterization and the proposed router fade strategy, our method generates a single model without employing additionally task-relevant models or branches during inference, thus being dubbed as a unified model across tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xDglbk9Y0M", "reviewer": "Reviewer_SX1w", "summary": "This paper proposes to integrate MoE and LoRA into pre-trained ViT to construct a multi-task model. Specifically, the experts are formed in the FFN of the transformer blocks based on the channel similarity, and the LoRA is used for each expert to fine-tune the parameters. On top of that, the paper further introduces the multi-task optimization method to avoid knowledge forgetting when some tasks are optimized faster than others. Finally, the fine-tuned model is unified into a single shared model for fast inference.", "questions": "- To make the multi-task model to be a unified model, the authors propose to gradually fade the role of the router. In other words, the model in the end is not a MoE anymore - it becomes an all-shared multi-task model. In this case, I'm curious what will happen if the router is not present from the beginning. In other words, the FFN channels are clustered into experts based on similarity and each cluster will have its own LoRA module for fine-tuning. Is the router truly necessary?\n- I would also expect more ablation studies on the clustering side because using MoE and LoRA for FFN is not a new idea (a relevant paper is [1]). The most important contribution of this paper comes from clustering the channels to form experts. So how many clusters we should have? If we don't cluster them, what will be the final performance on the MTL datasets? (Please point me to the corresponding results if I missed those.)\n- Since the MTO method is orthogonal to the MTL architecture design, I'm curious can the proposed QR loss be used to enhance other MTL models?\n- For Table 1, to form a fair comparison, the MTO methods should be compared with QR independently on the same model design. The same thing applies to the MTL architectures. They should be compared with EMTAL without QR.\n- The experiments are conducted on classification tasks. However, a lot of practical MTL scenarios are pixel-to-pixel prediction tasks, like NYUv2 and Takonomoy datasets. It's better to see results on those datasets.\n- The inference time should be compared as well since one of the contributions is designed for fast inference.\n\nI'm willing to raise my rating if the questions are answered fully.\n\n[1] Li D, Ma Y, Wang N, et al. MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts[J]. arXiv preprint arXiv 2024.", "limitations": "The authors analyzed the limitations and social impact in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is very easy to follow. All the components are clearly stated.\n- The idea of clustering channels into experts is intuitive and novel. It fully leverages the well-trained parameters from the pre-trained model.\n- The MTO method is useful and has the potential to be used in any multi-task model training.\n- The final model is a unified one with theoretical fast inference.", "weaknesses": "Please refer to the questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review, we will try to address any concerns encountered in the paper.\n\n> Q1: Is the router truly necessary?\n\nIt is an insightful question regarding the role of the router in our framework. Initially, we followed to the conventional MoE framework which includes a router. This comment prompted us to undertake a investigation and conduct additional experiments. From our analysis and experimental findings, we draw the following conclusions:\n\n1. When using MoE for multi-task learning, the router plays a crucial role in adaptively weighting different experts for each token. During the training process, for the same input token, calculating different weights for different experts allows for a nuanced and fine-grained optimization of each expert. This results in a well-specialized ensemble of experts, enhancing the model's overall performance and expertise in handling diverse tasks.\n2. Our experiments with a Router Fade strategy demonstrated a consistent improvement, with a mean accuracy increase of 0.34. This enhancement is non-trivial and underscores the utility of the router in achieving better performance across tasks.\n\nBased on these observations, we conclude that the router is indeed necessary for optimizing performance in our multi-task MoE framework.\n\n| Method | Mean Acc | Params. (M) | Time (ms) |\n| :------------------------------------ | :-------- | :---------- | :-------- |\n| Vanilla LoRA | 88.83 | 1.05 | 7.15 |\n| Cluster+Finer LoRA | 89.93 | 1.05 | 7.15 |\n| Cluster+Finer LoRA+Router | 90.14 | 1.20 | 13.87 |\n| Cluster+Finer LoRA+Router Fade (Ours) | **90.27** | 1.20 | **7.15** |\n\n> Q2: So how many clusters we should have? \n\nIn response to your query about the number of clusters and their impact:\n\n1. Regarding the optimal number of clusters, our approach balances between too many and too few clusters. Having too many clusters would result in each expert having too few channels, limiting their individual expressiveness. Conversely, too few clusters make each expert's low-rank characteristics less pronounced, which dilutes the distinct functionalities of each expert. We recommend that the number of experts similar to the number of attention heads.\n2. Our hyperparameter ablation studies, specifically looking at different numbers of clusters, showed optimal performance of 90.27 with cluster size of 16.\n\n| Clusters # | Mean Acc | Params. (M) |\n| :--------- | :-------- | :---------- |\n| 1 | 88.83 | 1.05 |\n| 4 | 89.12 | 1.09 |\n| 16 | **90.27** | 1.20 |\n| 64 | 89.34 | 1.64 |\n| 192 | 89.02 | 2.82 |\n\nThese findings underscore the importance of carefully selecting the number of clusters to maintain the effectiveness of each expert.\n\n> Q2: What if we don't cluster them?\n\nThe MOELoRA method in Table 1 is the scenario where MoE without clustering. In this setup, experts are directly employed in the FFN without forming clusters, resulting in a Mean Accuracy of 88.04. Moreover, we perform additional experiments in the table above without clustering (the clusters is set to 1).\n\n>  Q3: Applying QR loss on other MTL models?\n\nIndeed, the QR loss can be integrated into any MTL model to potentially enhance its performance. To illustrate this, we conducted experiments using the MOELoRA framework, which is detailed in a recent study. Our findings indicate that incorporating QR loss directly into the MOELoRA model results in a performance improvement. This demonstrates the versatility and effectiveness of QR loss in boosting the capabilities of MTL models, making it a valuable addition to existing and future MTL architecture design methods.\n\n| Method | CUB-200 -2011 | Stanford Cars | FGVC Aircraft | Oxford Flowers | Mean Acc | Params. (M) |\n| :--------- | -------------- | ------------- | ------------- | -------------- | :------- | ----------- |\n| QR | 88.5 | 91.6 | 84.4 | 99.6 | 91.04 | 86.26 |\n| MOELoRA | 88.4 | 88.2 | 75.0 | 99.7 | 88.04 | 2.82 |\n| MOELoRA+QR | 90.3 | 91.7 | 82.8 | 99.6 | 91.15 | 2.82 |\n\n> Q4: Fair comparison of the MTO methods and the MTL architectures.\n\nIn response to your valuable suggestion, we incorporate the content from Table 3 and extend experiment on QR loss within the same model designs, into the Table A of 'Rebuttal.pdf'. This allow for direct and convenient comparisons under the respective categories of MTO and MTL architectures.\n\n> Q5: Extension to pixel-to-pixel prediction tasks.\n\nTo demonstrate the extensibility of our method, we conducted experiments on the NYUv2 dataset using ViT-B due to the time constraints. \n\nSpecifically, we made the following attempts, integrating our method with the existing open-source method TaskPrompter. For MTL structure, we clustered and divided the FFN in the backbone and used the Router fade strategy, keeping the rest of the architecture unchanged. For MTO, We directly applied QR to the semantic segmentation task head for experiments. The results are as follows: \n\n| Method | Semseg mIoU ↑ | Depth RMSE ↓ | Normal mErr ↓ | Boundary odsF ↑ | Mean Δ (%) ↑ |\n| :---------------- | ------------- | ------------ | ------------- | --------------- | :----------- |\n| TaskPrompter-Base | 50.40 | 0.5402 | 18.91 | 77.60 | - |\n| + EMTAL | **52.90** | **0.5284** | 18.95 | 77.10 | **1.57** |\n\nSpecifically, our method significantly enhanced the performance on semantic segmentation and depth estimation tasks, while achieving comparable results on surface normal estimation and object boundary detection tasks. Overall, it resulted in an average relative improvement of 1.57%, confirming its effectiveness for pixel-level prediction tasks.\n\n> Q6: The inference time.\n\nIn addition to the ablation study in Table 4, we add a column to the Table A of 'Rebuttal.pdf', to highlight the efficiency of our unified model during inference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
According to the reviewer's suggestions, we have provided a more fair comparison in Table A in 'Rebuttal. pdf' and compared the inference time of different methods

Meta-review: The final ratings are 5, 5, 7. That said, there seems some unaddressed concerns:

- **Comparison based on DinoV2 ViT-L backbone and BOFT**. The authors did try to explain away verbally but could not run the experiments due to the time constraint.
- **Datasets with greater distribution shift such as WiLDS**. Similarly, the authors did try to explain away verbally but AC wonders why they did not just run WiLDS. They did run on NYUv2.

Further, there are some points from the authors that are somewhat confusing:

- **However, we found that BOFT has not released its training code for VTAB**. This is a bit odd to the AC and reviewers, because the results on VTAB are in the BOFT paper, so it doesn't seem necessary to require the code for comparison.
- **significantly less total parameters (85.99 M vs 1217.6 M)**. This is also confusing, since the ViT-L model has 300M params and not 1200M (which is ViT-G). But this may be a minor mixup.
- **Moreover, a preliminary analogy shows that while BOFT improved the mean accuracy by 0.53% and 0.23% compared to full finetuning**. This was also pointed out by the reviewer that it seems a bit odd to compare the improvement using EMTAL over a supervised initialization with BOFT's improvement over a self-supervised initialization. There was enough time over the rebuttal + author-reviewer discussion period to run these experiments, not sure why it wasn't done. AC does highlight that authors are not required to provide any new results during the rebuttal.

The AC discussed with the reviewers and in spite of these concerns, felt that it is borderline more positive than negative. AC also personally like the idea of splitting a pre-trained model into its "constituent" experts. After careful consideration, AC decides to accept the paper on the condition that the authors **must** address these concerns in the final manuscript.
Decision: Accept (poster)