Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learner

Authors: hanwen  zhong
jiaxin  chen
yutong  zhang
di  huang
yunhong  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 19:47:12

Abstract: Multi-Task Learning (MTL) for Vision Transformer aims at enhancing the model capability by tackling multiple tasks simultaneously. Most recent works have predominantly focused on designing Mixture-of-Experts (MoE) structures and integrating Low-Rank Adaptation (LoRA) to efficiently perform multi-task learning. However, their rigid combination hampers both the optimization of MoE and the effectiveness of reparameterization of LoRA, leading to sub-optimal performance and low inference speed. In this work, we propose a novel approach dubbed Efficient Multi-Task Learning (EMTAL) by transforming a pre-trained Vision Transformer into an efficient multi-task learner during training, and reparameterizing the learned structure for efficient inference. Specifically, we firstly develop the MoEfied LoRA structure, which decomposes the pre-trained Transformer into a low-rank MoE structure and employ LoRA to fine-tune the parameters. Subsequently, we take into account the intrinsic asynchronous nature of multi-task learning and devise a learning Quality Retaining (QR) optimization mechanism, by leveraging the historical high-quality class logits to prevent a well-trained task from performance degradation. Finally, we design a router fading strategy to integrate the learned parameters into the original Transformer, archiving efficient inference. Extensive experiments on public benchmarks demonstrate the superiority of our method, compared to the state-of-the-art multi-task learning approaches.

Main paper: 
 introduction: 
 Multi-task learning (MTL) [1, 2, 3] for Vision Transformer (ViT) aims at simultaneously learning multiple tasks, which has gained popularity in the computer vision community recently. It solves multiple relevant problems through sharing feature representations and forming a unified multi-task learner, thus enhancing the training and inference efficiency, and reducing the storage overhead. Moreover, by virtue of a well-designed MTL framework, the performance of each task can be further improved. Due to these merits, MTL has been used in a wide range of applications, such as the scene understanding [4, 5] and the erudite fine-grained recognition [6] .Despite both theoretical [7] and practical [8, 9] validations of their potential on enhancing the model generalizability, conventional MTL approaches [6, 10] often suffer performance degradation when compared to training tasks independently, which is primarily due to two reasons. Firstly, they adopt suboptimal learning strategies leading to conflicting task gradients and varying loss scales [11] , which increase competitive interference during task optimization. Secondly, their model structures often fail to extract representative features for each task when utilizing a shared backbone network [12, 4, 6, 5] .Figure 1 : FFN as Mixture of Low-rank Experts. Given an up-projection weight matrix in FFN, a straightforward way of splitting it into MoE is to divide every K channels into separate experts, resulting in highly dissimilar experts and a high-rank MoE, which is inherently unsuitable for integration with LoRA. In contrast, our proposed MoLE approach rearranges the weight matrix into groups of similar channels as experts, creating specialized low-rank experts that are better suited for integrating with LoRA.Several recent works have attempted to deal with the above issues, which concentrate on the following two aspects. 1) Efficient multi-task learners. As shown in Figure 2 , instead of designing complex but incompact network structures [12, 4] that incur a large number of tunable parameters, recent works [13, 5, 14] such as MLoRE and MOELoRA explore the advantages of Mixture-of-Experts (MoE) in extracting task-specific features by enhancing the diversity of parameters and features [15, 16, 17] , and Parameter-Efficient Fine-Tuning (PEFT) in reducing the tunable parameters and storage overhead [18, 19, 20] . Nevertheless, MLoRE [5] still relies on a substantial number of additional parameters, limiting the overall efficiency and feasibility of training. MOELoRA [14] adopts a unitary LoRA structure to tune the experts, which weakens the learning capability of individual experts. Moreover, both methods utilizes task-driven routers, requiring either a static network with a fixed number of tasks or a dynamic routing network. The former results in significant storage overhead, while the latter increases the inference cost. 2) Multi-Task Optimization (MTO) strategies. Existing works on MTO can be broadly categorized into the gradient-based methods and the loss-based methods. The gradient-based methods [21, 2, 22, 3, 23, 24] seek to balance the gradients across multiple tasks in the last shared layer, by decreasing differences in their magnitude or direction, and aggregating sub-gradients into a unified one. The loss-based methods [25, 26, 27] optimize the MTL process by balancing the multi-task losses. Recently, IMTL [11] is proposed to treat all tasks equally without bias, while AMTL [28] synchronizes learning progress across tasks. However, as each task has its own intrinsic optimization pace due to varying levels of training difficulty for distinct tasks, forcing synchronization in MTO disrupts these inherent properties, thus leading to suboptimal solutions. For more detailed discussion of related works, we refer to appendix A.To overcome the drawbacks of existing works, we propose a novel MTL framework dubbed Efficient Multi-Task Asynchronous Learning (EMTAL). Basically, EMTAL consists of the MoEfied LoRA structure, the Quality Retaining (QR) optimization mechanism and the router fading strategy, of which MoEfied LoRA decomposes a pre-trained Vision Transformer model into an efficient multi-task learner, QR accomplishes asynchronous learning of multi-task knowledge and enable establishing an efficient unified model by combining with the router fading strategy. Specifically, inspired by MoEfication [29, 30] , the proposed MoEfied LoRA firstly decomposes the FFN layer of Vision Transformer into a MoE structure by clustering similar channels into experts, creating specialized low-rank experts as demonstrated in Figure 1 . Considering the inherent low-rank property of each expert, LoRA is naturally employed to perform efficient training of MoE. Subsequently, in order to achieve asynchronous learning of multi-task knowledge based on the MoEfied LoRA module, QR constrains logits of early converged tasks to retain near the optima when continuously optimizing the insufficiently converged tasks, thus avoiding severe interference between tasks and improving multi-task optimization. Finally, the router fading strategy is combined with MoEfied LoRA and QR by gradually diminishing the router's fole in the last training epochs, seamlessly integrating learned parameters into the model structure without incurring extra inference time cost and storage overhead.(a) MoE [12, 4] . The conventional MoE leverages multiple expert networks and a gating mechanism to dynamically select the most relevant experts for each input. It focuses on designing complex but incompact network structures, incurring a large number of tunable parameters.(b) LoRA Experts [5, 14] . LoRA Experts employ unified low-rank adaptation modules to achieve the parameter efficiency, which however limit the expert capacity, and requires either static networks with substantial storage overhead or dynamic routers with high inference cost.(c) MoEfied LoRA (Ours). Our method groups similar weights into specialized low-rank experts, enabling seamless integration with LoRA to create an efficient multi-task learner. Besides, by combing with a router fading strategy, our method ensures both training and inference efficiency while substantially reduces storage overhead. The main contributions of this paper are summarized as follows. 1) We propose a novel efficient multi-task learning framework dubbed EMTAL. To the best of our knowledge, our work makes the first investigation on decomposing a pre-trained Vision Transformer model for multi-task learning and reparameterizing the learned multi-task knowledge into a unified model. 2) We design a MoEfied LoRA structure, a QR multi-task optimization mechanism combined with a router fading strategy to accomplish an efficient asynchronous multi-task learner. 3) We extensively evaluate the proposed method on challenging multi-task fine-grained visual classification datasets and the VTAB benchmark, and the experimental results demonstrate that our method significantly improves the performance of single-task learning and the state-of-the-art MTL approaches. 
the proposed approach: 
 In this section, we mainly introduce the preliminary concepts of Vision Transformer, and describe the technical details of the proposed EMTAL approach.Given an input image I ∈ R 3×H×W , a standard Vision Transformer [31] model with L layers first divides I into m non-overlapping patches, which are further fed into a patch embedding layer, generating m D-dimensional visual tokens. After concatenating with a class token, the input tokens are finally formed as X 0 ∈ R (1+m)×D . Each transformer layer contains a Multi-headed Self-Attention (MSA) [32] block, a Feed-Forward Networks (FFN) block and a Layer Normalziation (LN). The tokens of the l-th layer are generated based on those in the (l -1)-th layer formulated as below:EQUATION)Similar to [29, 30] , our work mainly focuses on FFN, which usually consists of two linear layers{W up ∈ R D×(r•D) , b up ∈ R r•D }, {W down ∈ R (r•D)×D , b down ∈ R D }and a GELU activation operation, where r represents the scaling factor. Accordingly, FNN processes the normalized input X l n ′ = LN(X l ′ ) as follows:EQUATION)This procedure ensures the effective transformation and projection of the input through the encoder layers, enabling the prediction of the class probability distribution y for downstream tasks.Previous MTO : Match convergence of simple tasks to difficult ones. × E 1 up E 1 down E K up ×E K down Expert 1 Expert K W down × W up + Router Fading + LoRA … E 1 E 2 E K router (c). MoEfied LoRA × N … E 1 E 2 E K Router (dE i ‛ = + E i + = A i B i W‛ = [E 1 ‛; E 2 ‛; …; E K ‛]Frozen TunableAnnoying drop occurs during MTL. As shown in Figure 3 (a), in order to establish a unified model for N T tasks, we follow [6] by unifying the label spaces for multiple tasks into an overall one with N class classes, and merging the training samples as S = N T t=1 S t , where S t denotes the set of training samples for the i-th task. Supposing a pre-trained Vision Transformer with an embedding backbone Φ(•; θ ϕ ) : X → F, where θ ϕ represents the parameters to be frozen in the network, it maps an input x ∈ X to the feature space F. We decompose it into an efficient multi-task learner dubbed MoEfied-LoRA denoted by Φ ′ (•; [θ ϕ ; θ t ]), where θ t indicates the set of newly employed tunable parameters and [; ] refers to the concatenation operation. Subsequently, the Quality Retaining multi-task optimization mechanism as well as the router fading strategy are applied to asynchronously learning the tunable parameters θ t . Finally, we reparameterize θ t into the original backbone, achieving an efficient unified model Φ(•; θ ′ ϕ ).Basically, MoE-based learners benefits MTL in the following two ways. First, they enable dynamic encoding of different samples across tasks via a router and multiple experts, significantly enhancing feature diversity. Second, they reduce the number of parameters and computational cost, remarkably promoting the training and inference efficiency. However, early attempts include delicately designed MMoE [12] , M 3 ViT [4] and Mod-Squad [13] fail to establish a unified structure and inevitably introduce additional inference overhead, considering that LoRA increases inference latency by 20-30% without reparameterization [33] . Recently, MoEfication methods [29, 30] aimed at "group together the neurons that are often activated simultaneously" have demonstrated promising performance in constructing effective MoE structures from pre-trained ViT models. Inspired by this, in this work we attempt to "group together the neurons with similar weights" to construct the MoE structure. As the corresponding experts naturally meet the low-rank conditions in LoRA [33] , we can therefore obtain an excellent efficient multi-task learner by combining with LoRA. By employing a router fading strategy to preserve the reparameterization property of LoRA, we can further eliminate additional inference latency.Specifically, as shown in Figure 3 (b), we fistly decompose the FFN into a MoLE (Mixture of Low-rank Experts) structure. To convert the FFN in the l-th layer into MoE, we draw on insights from [34, 35] , which view the FFN as a memory bank that retains knowledge from pre-trained models. Each column of the up-projection matrix W up serves as a key to be matched, which each row of the down-projection matrix W down act as the corresponding value. Since similar keys in the W up tend to serve similar functions, they should be grouped within the same expert. Formally, W up is clustered column-wise into K clusters by using the balanced k-means [36] , with a mapping function C(•) that assigns the idx-th column to cluster C(idx). Therefore, columns within the same cluster constitute an expert. Additionally, b up , W down should match W up channel by channel, and also adhere to the clustering results of W up to construct the experts {E i } K i=1 . Consequently, they are concatenated for the MoEfication process and then split to obtain the corresponding experts, formulated as below:EQUATIONEQUATIONEQUATIONSince the column vectors within these experts are relatively similar, exhibiting low-rank characteristics, they naturally satisfy the low-rank conditions in LoRA [33] . Therefore, we apply a set of LoRA parameters {A i up , B i up } and {A i down , B i down } for E i up and E i down to enable more efficient learning and improved performance, which is referred as MoLE and formally described as below:EQUATIONEQUATIONMoreover, to leverage the benefits of dynamic routing for extracting diverse features during training, we initially establish a sample-driven soft router for each MoLE, denote as W r ∈ R D×K , to reweight the experts. For the l-th layer of MoLE, we calculate weights for each experts as ω l by adopting the following formulation:EQUATIONDuring training, the router is employed to fully optimize the MoLE. The output X l FFN-de of the decomposed FFN is calculated as follows:X l FFN-de = K i=1 GELU([ω l 1 • (X l n ′ E 1 up ′ + E 1 b ); • • • ; ω l K • (X l n ′ E K up ′ + E K b )])E i down ′ + b down . (9)The goal of multi-task learning (MTL) is to ensure that the final model performs well across all tasks. However, due to distinct task difficulties and individual optimization schedules, achieving optimal performance on all tasks simultaneously with a single model can be challenging. As displayed in Figure 3 (d) , despite introducing strong priors to synchronize the optimization schedules across tasks, existing methods [25, 28, 24] can disrupt the inherent schedules of tasks with varying difficulties, creating challenges for MTL optimization. Therefore, we propose a different perspective: maintaining the inherent optimization pace of each task is crucial. Specifically, we allow asynchronous convergence of tasks and introduce the Quality Retaining (QR) MTO strategy to preserve high-quality knowledge from already converged tasks during subsequent optimization.Specifically, at iteration iter, we maintain an optimal knowledge bank Z ∈ R N class ×N class , which records the Exponential Moving Average (EMA) logits of each class learned during the optimization process from iteration 0 to iter -1. This knowledge is distilled into the currently optimized model using a distillation loss. Formally, we maintain the knowledge bank Z using EMA. For a sample s in the current training batch with label label s , we update Z iter labels as follows:EQUATIONwhere z s indicates the logits of sample s and m ∈ (0, 1) is a momentum coefficient. This results in a real-time updated knowledge repository Z.To ensure the retention of high-quality knowledge for already optimized tasks, we employ a straightforward method, i.e., weighting the distillation process by the reciprocal of the loss from each task. This procedure implies that tasks with lower loss (already optimized) should rely more on the learned knowledge, while tasks with higher loss (still being optimized) will depend more on the ground truth. Therefore, the Quality Retaining loss L QR for samples within a mini-batch S batch is defined as below:EQUATIONwhere KL(•) and 1 indicates the Kullback-Leibler divergence [37] and the indicator function, respectively. L CE,t is the Cross-Entropy loss for the t-th task.Based on Eq. ( 11), the overall training loss is formulated asL = N T t=1 L CE,t + L QR .This strategy ensures that the model maintains optimal performance for already converged tasks while allowing other tasks to continue their optimization at their inherent pace.MTL aims to develop a universal model capable of executing multiple tasks simultaneously. To achieve this goal, we transform the unified pre-trained model into an MoEfied LoRA and develop the quality retaining mechanism to preserve multi-task knowledge as discussed in the previous sections. However, the dynamic routing in MoEfied LoRA limits LoRA's capability of reparameterizing the learned parameters into a unified, static pre-trained structure. To address this issue, we design a router fading strategy that gradually diminishes the router's role in the later stages of training. This approach allows the knowledge embedded within the optimized router to be implicitly absorbed as follows:EQUATIONwhere α ∈ [0, 1] is the trade-off hyper-parameter. Finally, as shown in Figure 3 (e), we completely remove the router after training by setting α = 0. In the mean time, we reparametere knowledge learned by LoRA back using Eq. ( 6) and Eq. ( 7), and concatenate them to replace the original parameters W up , b up , W down of the l-th layer with W up ′ , b ′ up , W down ′ as below:EQUATIONEQUATIONEQUATIONThe above technique avoids extra computational costs and maintains a unified model structure, delivering a new perspective for multi-task learning.3 Experimental Results and Analysis 
datasets and evaluation metric: 
 By following [6] , we mainly evaluate the performance of our proposed EMTAL method on the challenging Multi-task FGVC benchmark. In addition, we conduct experiments on the Specialized VTAB-1k dataset to validate the effectiveness over previous solutions. Multi-task FGVC is a collection of public datasets specifically for multi-task fine-grained visual classification, including CUB-200-2011 [38] , Stanford Cars [39] , FGVC-Aircraft [40] and Oxford Flowers [41] . In order to make fair comparisons, we adopt the standard training/testing split as depicted in [6] . Specialized VTAB-1k [42] consists of specialist images from specialized equipment, where we employ multi-task learning to fully leverage these expensively annotated data. We follow the standard training/validation splits used in [42] for fair comparisons. The top-1 accuracy is utilized as the evaluation metric. To further demonstrate the effectiveness of our method on the tasks of pixel-to-pixel dense prediction, we also conduct experiments on the NYUv2 dataset [43] . Specifically, we integrate our method with TaskPrompter [44] by applying MoEfied LoRA and QR to the FFN layers and the semantic segmentation task head, respectively.In addition, we evaluate on Multi-task FGVC for few-shot learning under 1, 2, 4, 8, and 16 shots, by following existing works [45, 46] .We utilize ViT-B/162 pre-trained on ImageNet-21K [31] as the base model. We use the AdamW optimizer [47] to fine-tune our models for 100 epochs and adopt the cosine learning rate decay with a linear warm-up for 10 epochs in all experiments. We fix the hyper-parameters τ in Eq. ( 8) to 5, since it exhibits stable performance with distinct values. As for data augmentation, we employ random resize cropping to 224 × 224 pixels and a random horizontal flip during training and resize to 248 × 248 pixels with a center crop to 224 × 224 pixels. All experiments are conducted on a single Nvidia GeForce RTX 3090 GPU.To comprehensively evaluate the performance of our approach, we compare with the MTL full fine-tuning baseline and the following categories of state-of-the-art approaches: 1) MTO Lossbased methods, including GLS [25] and AMTL [28] ; 2) MTO Gradient-based methods, including Nash-MTL [23] and Aligned-MTL [24] combined with vanilla LoRA-16; 3) Efficient multi-task learners methods, including Dual-Prompt [48] , Erudite [6] , MLoRE [5] and MOELoRA [14] . As the performance of EMTAL depends on the inherent low-rank properties, we report the results of our method using distinct ranks including 1, 2 and 4, denoted by EMTAL-1, EMTAL-2 and EMTAL-4, respectively.As summarized in cost, implying that our framework is significantly more effective and efficient than the traditional separate training paradigm.Moreover, EMTAL consists of a reparameterizable and efficient multi-task learner, a Quality Retaining MTO mechanism and a router fading strategy. Compared to the state-of-the-art methods, each component of our approach shows significant advantages. In terms of the design of MTL structures, MoEfied LoRA seamlessly integrates low-rank experts with LoRA, resulting in improved performance. Furthermore, the router fading strategy and the reparameterization effectively reduce inference time, significantly enhancing the overall efficiency. Meanwhile, by considering the intrinsic task-specific optimization pace, the QR mechanism clearly improves previous MTO strategies. Overall, it utilizes a sample-driven router and multiple experts to extract diverse feature representations while preserving high-quality knowledge during training, making it highly beneficial for multi-task learning. In this way, our method achieves the highest accuracy compared to the state-of-the-art approaches, surpassing the second best Aligned-MTL by 1.56% while tuning fewer parameters.We further evaluate the performance on Specialized VTAB-1k, where MTL is highly beneficial against previous solutions. As displayed in Table 2 , some existing works focus on applying parameterefficient fine-tuning on each task individually to avoid over-fitting. However, we find that performance can be significantly enhanced by incorporating multi-task learning. Specifically, applying AMTL and MOELoRA on the vanilla LoRA yields improvements of 0.29% and 1.51%, respectively. Furthermore, when directly applying our EMTAL to these tasks, we observe consistent improvements across different ranks, achieving the highest accuracy compared to the state-of-the-art approaches. Notably, it enhances the overall performance by 1.43%, while utilizing fewer parameters. In addition, on the NYUv2 dataset, our method significantly enhances performance in semantic segmentation and depth estimation tasks, while achieving comparable results in surface normal estimation and object boundary detection tasks. Overall, this led to an average relative improvement of 1.57%, validating the effectiveness of our approach for pixel-level prediction tasks. We provide more results, including using self-supervised pre-trained model DINOv2-large and the few-shot learning in appendix B and appendix C, respectively.In this section, we evaluate the effectiveness of the proposed main components, i.e. MoEfied LoRA and Quality Retaining by extensive ablation studies. Effect of the Main Components. We evaluate the proposed components across the Multi-task FGVC Benchmark based on ViT-B/16. As Table 4 shows, MoEfied LoRA consistently boost the performance, achieving an average of 2.88% improvement with less tunable parameters. The results indicate that MoEfied LoRA is significantly beneficial to multi-task learning by providing more diverse features. In the mean time, the Quality Retaining MTO mechanism can further remarkably promote the accuracy, with a 3.65% improvement on average. A combination of these two components, i.e. MoEfied LoRA and QR, further boosts the overall performance across datasets, implying that MoEfied LoRA and Quality Retaining are complementary in multi-task learning.On MoEfied LoRA and Router Fading. We further validate the effectiveness of MoEfied LoRA and the router fading strategy across the Multi-task FGVC benchmark based on ViT-B/16. Initially, we begin by decomposing the pre-trained model into a MOLE structure. A straightforward clustering and splitting of the FFN, combined with a sample-driven router, can achieve a 1.81% improvement by enhancing the the diversity of feature representations. Furthermore, applying LoRA to the low-rank experts yields a 0.94% gain in performance, as the small amount of tunable parameters reduces the risk of overfitting, and the the low-rank property of the experts aligns well with LoRA. Additionally, the proposed router fading strategy gradually diminishes the influence of the router over 50 epochs, effectively preserving the reparameterizable nature of LoRA and reducing the inference time.Moreover, we conduct more ablation studies on the proposed MoEfied LoRA. The number of clusters k significantly influence the performance of MoEfied LoRA, considering that a large number of clusters intends to incur simple experts with few channels, and a small number of clusters results in high-rank experts, either of which degrades the effectiveness of MoEfied LoRA. We empirically study the effect of k by using 1, 4, 6, 64 and 192 clusters. As Table 6 displays, MoEfied LoRA reaches the highest accuracy when k = 16. Moreover, we compare different ways to construct experts, including the co-activation clustering [28] that groups weights based on activations for each channel and the gradient-cluster [47] that clusters weights according to cumulative gradients. As shown in Table 6 , our method achieves the best performance, clearly demonstrating its effectiveness. As shown in Figure 4 (a), we measure the low-rank properties of experts by using the Ky Fan 2-k norm [51] , and the results indicate that the experts generated by our method consistently exhibit statistically more significant low-rank properties across distinct ranks. Additionally, we analyze the low-rank properties of experts across different layers of ViT when the ranks is fixed as 1. As Figure 4 (b) demonstrates, the low-rank properties of experts are more significant in lower layers that those in higher layers.We kindly refer to appendix E for more detailed discussion about the broader impacts and limitations of our work . 
conclusion: 
 In this paper, we focus on decomposing a pre-trained Vision Transformer model for multi-task learning, reparameterizing the learned multi-task knowledge back into the original model and establishing a unified model. We propose a novel efficient multi-task learning framework dubbed EMTAL, which mainly consists of the MoEfied LoRA module, the Quality Retaining (QR) mechanism and the router fading strategy. Concretely, MoEfied LoRA decomposes a pre-trained ViT into multi-task learners by clustering similar weight of FFN into experts and applies LoRA to tune the experts with low-rank properties. Subsequently, we leverage the inherent asynchronous convergence property of tasks and employ QR to preserve the optimized performance for converged tasks. Finally, the router fading strategy is introduced to eliminate extra inference cost. Extensive experiments on the public benchmarks demonstrate that our method substantially promotes performance by comparing with the state-of-the-art multi-task learning approaches, while also being effective in few-shot learning with limited data. Our work delivers a new perspective for efficient multi-task learning by decomposing a pre-trained model and reparameterizing back with a low rank updating. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We describe it in Section D.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We describe it in Section 3.5.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss it in Section E.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and the license and terms of use explicitly are mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Multi-Task Learning (MTL) [52] aims to improve the performance and generalization of models across individual tasks by exploiting commonalities and interdependencies between tasks through shared representations, while reduce parameters and speed up training or inference [52, 53, 54] . Two main research branches, i.e., architecture-focused methods and optimization-based, have been developed through the years [55] .Efficient multi-task learners. Structure-focused methods examine the impact of parameter sharing within models and can be broadly categorized into encoder-and decoder-focused approaches, based on where information is exchanged or shared between tasks [55] . In encoder-focused models, task parameters are shared exclusively in the encoder to extract general features, utilizing mechanisms such as feature fusion [56, 57, 58] , attention [27, 6] , dynamic branching [59, 60] , while the decoder consists of disjointed task-heads with no cross-task information [21, 61, 1] . In decoder-focused models, the model parameters are also shared across tasks in the decoder, while it first make initial predictions for each task and then refine the results by extracting inter-task correlation with the mechanisms such as multi-model distillation [62, 63, 64, 65] , sequential task prediction [66] , or cross-task consistency [67] . Some continual learning frameworks [68, 48] are also applicable to MTL, designing parameters that are task-specific and task-shared to ensure effective isolation and sharing between tasks.Multi-Task Optimization Methods. The optimization-focused methods [21, 69, 70, 61, 71, 24, 23] investigate the effect of balancing how the tasks are learned, explore effective solutions through a perspective of model optimization, and improve the optimization process of MTL via different multi-task loss designs [25, 67, 61, 72, 73, 28] , which can produce suitable loss weights to reduce conflicts among multiple tasks. [74] proposes recording the optimal checkpoint for each task and learning from it by distilling the soft labels of each sample. However, this may record a local optimal solution, leading to suboptimal results. Gradient manipulations [69, 21, 22, 2, 3] can counteract task interference by directly adjusting gradients, with recent methods focusing on formulating a unified gradient vector subject to diverse constraints.Mixture-of-Experts (MoE) [16, 15] are originally designed to combine the decisions of a series of sub-models (i.e., expert networks) on the same input, while enhance conditional computational capabilities and scale parameters in neural networks [75, 76, 17, 77] . Therefore, MoE learns multiple specialized expert networks, each of which maintains its own unique set of trainable parameters. This enables the expert networks to craft distinct internal representations tailored to their respective input data. Additionally, MoE employs a router that dynamically weights the outputs of each expert network, allowing their collective contributions to be combined into the final model output.MoE is also used in MTL, which can better separate the parameter space and adopt relevant model part to different tasks, and become a promising approach for MTL [15, 16, 17] . M 3 ViT [4] customizes MoE layer into a ViT backbone, and activate task-specific experts in training to mitigate MTL gradient conflicts. Mod-Squad [13] proposes a modular multi-task leaner based on MoE and a novel loss to address the gradient conflicts among tasks. MMoE [12] designs a multi-gate MoE to ensemble expert networks for different census analysis tasks with different router. TaskExpert [78] generates multi-task predictions for all tasks in one single forward inference simultaneously, leading to significantly higher multi-task training efficiency. MLoRE [5] builds global relationships among all tasks explicitly in the MoE structure and proposes low-rank experts give MoE better efficiency compared to the naive one.Low-Rank Adaption (LoRA) [33] is one of the method in parameter-efficient fine-tuning [33, 79, 80, 81] , which takes inspiration from [82] that the difference in weights between the pre-trained model and the adapted model resides on low intrinsic rank. Since the low-rank structure [83, 84, 85, 86] has become a successful parameter-efficient fine-tuning technique, a surge of works has shown impressive results that combining LoRA and MoE for more efficient and effective model tuning.LoRAHub [87] first trains multiple LoRA weight modules on upstream tasks, then to adapt the model to a downstream task, a gradient-free optimization method is employed to search for the optimal coefficients to linearly combine this pre-trained set of LoRA modules. MOELoRA [14] utilizes a router network conditioned on a task identifier to dynamically combine the outputs of multiple LoRA experts. Similarly, MoCLE [88] designs a router network that is conditioned on the clustering information extracted from each input sample. LoRAMoE [89] splits the LoRA experts into two groups and explicitly learns distinct capabilities for each group. While these mixture-of-LoRA methods densely combine multiple LoRA experts, a sparse mixture of LoRA experts may be a more economical option, and achieves comparable performance while maintaining roughly constant training and inference costs. The Octavius [90] method, for instance, selects the top-2 LoRA experts based on a router that conditions on the entire input instance, representing a more coarse-grained routing mechanism. Besides, the low-rank structure is used to model task-generic features and generate task-specific features by linear combination in earlier MTL methods. VL-Adapter [81] exploits adapter-based methods to efficiently fine-tune generative models in the multi-task setting, while [86] proposes a framework for training multiple neural networks simultaneously, sharing for all shareable layers, and learning the sharing strategy in a data-driven way.B More Experimental Results on DINOv2-large In addition to evaluating our method on ViT-B/16, we also evaluate its performance using DINOv2large 3 , a self-supervised pre-training model. As summarized in Table 7 , our method consistently demonstrates significant performance improvements over prior approaches. Notably, compared with separate full-finetuning which incurs 1217.6M tunable parameters (4 × 304.4M) and the previous SOTA, EMTAL successfully constructs a unified multi-task framework on this benchmark. Our approach eliminates the need for multiple specialized models while achieving superior performance. Thus, this unified framework proves to be effective when scaled to larger self-supervised models, further validating the robustness and generalizability of our approach.The result is shown in Figure 4 . In terms of average performance, the following observations can be made: (i) EMTAL, LoRA, Adapter, and VPT perform similarly under extreme limitations in training data, such as 1 or 2 shots. (ii) When more shots are available, EMTAL shows a significant advantage.For example, at 16 shots, the difference between EMTAL and the second best is about 5%. This suggests that multi-task training has a particularly significant advantage over individual training in situations where reliable annotation information is limited.We show the detailed descriptions of multi-task classification as follows. The train/val/test splits and the classes are shown in Table 8 . Following Erudite [6] , we employ Multi-task Fine-Grained Visual Classification datasets to evaluate the effectiveness of our proposed EMTAL, which consists of CUB-200-2011 [38] , Stanford Cars [39] , FGVC-Aircraft [40] and Oxford Flowers [41] . We also employ Specialized VTAB-1k [42] , consisting of specialist images from specialized equipment, where multi-task learning is applied to fully leverage these expensively annotated data.This work has the following impacts. Compared to traditional multi-task learning, EMTAL maximizes the use of limited data for downstream multi-task learning. This capability facilitates the rapid transfer of large models pre-trained on vast datasets to downstream tasks, conserving computational resources and minimizing carbon emissions. Our design is reparameterizable, allowing the model to be transferred to downstream tasks without altering the deployed backbone architecture. This approach, which involves simply replacing a set of weights, is more convenient than other [4, 13, 5] multi-task learning methods. However, like other methods [14] , EMTAL relies on pre-trained models. If the upstream model is trained on illegal data, this could lead to rule violations.Regarding limitations, our model structure currently employs a unified rank across all experts, which benefits parallel computing during training. However, our experiments reveal that different tasks have varying difficulties, necessitating different optimal ranks. For example, the optimal rank for the CUB-200-2011 dataset is 1, while for the Stanford Cars dataset, it is 4. Thus, dynamically and adaptively selecting appropriate ranks for different tasks to enhance multi-task learning outcomes is a promising research direction. Additionally, we observed that the low-rank property of shallow experts is more pronounced than that of deep experts in the network, suggesting that dynamically adjusting the rank of experts across different layers could yield better results. Therefore, adaptively assigning the required rank to these experts will be a future research focus. Furthermore, while our method aims to establish a unified multi-task model for practical applications, it is crucial to consider potential out-of-distribution (OOD) issues that may arise during deployment, potentially affecting performance.To address this challenge and enhance the applicability of our work in real-world scenarios, future research could explore the integration of domain adaptation methods. Investigating datasets such as WILDS [100] , which are tailored for OOD challenges, could provide valuable insights and further improve performance across diverse contexts.Answer: [No] Justification: We describe details of public datasets in Section D, provide details of our method, and will release code upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.ViT-B/16 supervised pre-trained on ImageNet-21K.DINOv2-large. 


Decision: The final ratings are 5, 5, 7. That said, there seems some unaddressed concerns:

- **Comparison based on DinoV2 ViT-L backbone and BOFT**. The authors did try to explain away verbally but could not run the experiments due to the time constraint.
- **Datasets with greater distribution shift such as WiLDS**. Similarly, the authors did try to explain away verbally but AC wonders why they did not just run WiLDS. They did run on NYUv2.

Further, there are some points from the authors that are somewhat confusing:

- **However, we found that BOFT has not released its training code for VTAB**. This is a bit odd to the AC and reviewers, because the results on VTAB are in the BOFT paper, so it doesn't seem necessary to require the code for comparison.
- **significantly less total parameters (85.99 M vs 1217.6 M)**. This is also confusing, since the ViT-L model has 300M params and not 1200M (which is ViT-G). But this may be a minor mixup.
- **Moreover, a preliminary analogy shows that while BOFT improved the mean accuracy by 0.53% and 0.23% compared to full finetuning**. This was also pointed out by the reviewer that it seems a bit odd to compare the improvement using EMTAL over a supervised initialization with BOFT's improvement over a self-supervised initialization. There was enough time over the rebuttal + author-reviewer discussion period to run these experiments, not sure why it wasn't done. AC does highlight that authors are not required to provide any new results during the rebuttal.

The AC discussed with the reviewers and in spite of these concerns, felt that it is borderline more positive than negative. AC also personally like the idea of splitting a pre-trained model into its "constituent" experts. After careful consideration, AC decides to accept the paper on the condition that the authors **must** address these concerns in the final manuscript.

Reviews: review: paper_summary: paper_summary: The paper introduces a method to decompose ViT MLP matrices into a mixture of experts based on channel similarity, training each expert with LoRA. The authors also use a quality-retaining loss to effectively optimize during training for multi-task learning.
strength_weakness: strength_weakness: **(W1):** Ablation study is not extensive enough. I am interested in seeing whether the router is needed at all, and whether you can just set alpha=0 from the very beginning. As a follow up question: I haven’t closely checked, but does setting alpha = 0 make the method equivalent to LoRA?

**(W2):** Line 204: The models used seem to be supervised fine-tuned backbones, which are not really designed to be generalizable to multiple tasks. The authors should consider experiments with self-supervised or unsupervised backbones like MAE [1] or DinoV2 [2], and compare against these backbones’ generalization capabilities. The authors should also clarify what the source of weights are for all the other models used in comparison, and which model sizes are being used.

**(W3):** The distribution of tasks is not necessarily too different. Consider evaluating the method on tasks which represent greater domain shifts (eg: WILDS [3]).

**(W4):** The comparisons against other PEFT techniques in Table 1, 2 is missing. There are numerous PEFT methods with improvements beyond the original LoRA that are not mentioned. Eg: if we look at the VTAB table in BOFT [4], we see that the EMTAL results are not state-of-the-art.

---
References:  
[1] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[2] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." arXiv preprint arXiv:2304.07193 (2023).

[3] Koh, Pang Wei, et al. "Wilds: A benchmark of in-the-wild distribution shifts." International conference on machine learning. PMLR, 2021.

[4] Liu, Weiyang, et al. "Parameter-efficient orthogonal finetuning via butterfly factorization." arXiv preprint arXiv:2311.06243 (2023).
questions: questions: Lines 64-69 are unclear and can be worded differently.

A better description of Figure 2 is required. A summary for each of 2a and 2b would assist in clarity.

line 129: “owe” -> “ought”

Notation is a bit confusing: eg: usually “r” is used for lora ranks, but here it’s a scaling factor

line 255: how exactly is inference time being reduced? At the end of training, the LoRA experts are being merged back into the weights of the model, right?

Table 4: What is PoolFT?

Equation 14: clarify what i is here. Notation seems to be overloaded? Is i the iteration, or the column? Previously, i was used to denote experts.

Equation 15: Clarify what L_t is. There also seems to be a L_CE,t. Are these the same?
limitations: limitations: The authors have adequately described these sections.
paper_summary: paper_summary: Considering recent trends in multi-task learning (MTL) that design Mixture-of-Experts (MoE) structures and integrate Low-Rank Adaptation (LoRA), this paper proposes Efficient Multi-Task Learning (EMTAL) to address the sub-optimal performance resulting from the rigid combination of MoE optimization and LoRA's reparameterization capability. The approach decomposes the pre-trained Transformer into a low-rank MoE and employs LoRA to tune the experts, termed MoEfied LoRA. Additionally, the paper introduces a Quality Retaining (QR) multi-task optimization strategy that uses historical high-quality class logits to prevent performance degradation.
strength_weakness: strength_weakness: 1. The proposed idea of grouping similar weights to compose experts seems quite naive. Further justification for this approach is needed, such as theoretical analysis of how different grouping strategies (e.g., weight, activation) lead to better multi-task performance. Additionally, ablation experiments for various grouping strategies would be beneficial.

2. I'm not convinced that the proposed methods are superior to previous works (e.g., MLoRE) since most experiments in the paper are conducted on homogeneous task settings, focusing on visual classification tasks. Can the proposed approach be generalized to heterogeneous task settings involving various types of vision tasks using benchmarks such as Taskonomy or PASCAL-Context? It seems previous works might suffer from overfitting by naively dividing classes into task groups.

3. Quality Retaining (QR) appears to be applicable only to classification tasks, limiting its application in the context of MTL. It should be compared to previous multi-task optimization approaches mentioned in the paper, which can be applied orthogonally to architecture design.

4. In similar context in 3, experimental setting for comparision with previous works is not appropriate. Direct comparison with proposed methods with loss-based and gradient-based optimization is not appropriate as proposed method mainly focuses on architecture design approach.

5. The concept of reparameterizing a pretrained transformer into an MoE-like structure is novel. However, I am unsure if it is competitive in terms of performance. It would be helpful to demonstrate that the proposed methods show competitive performance compared to learning a similar structure from scratch, if possible.

6. Previous work, such as MOELoRA also insists that their framework is a unified network. Please explain clearly that what is the main difference between your work and previous works. why the proposed method is the only unified model."
questions: questions: Refer to the weakness section.
limitations: limitations: Refer to the weakness section.
paper_summary: paper_summary: This paper proposes to integrate MoE and LoRA into pre-trained ViT to construct a multi-task model. Specifically, the experts are formed in the FFN of the transformer blocks based on the channel similarity, and the LoRA is used for each expert to fine-tune the parameters. On top of that, the paper further introduces the multi-task optimization method to avoid knowledge forgetting when some tasks are optimized faster than others. Finally, the fine-tuned model is unified into a single shared model for fast inference.
strength_weakness: strength_weakness: Please refer to the questions part.
questions: questions: - To make the multi-task model to be a unified model, the authors propose to gradually fade the role of the router. In other words, the model in the end is not a MoE anymore - it becomes an all-shared multi-task model. In this case, I'm curious what will happen if the router is not present from the beginning. In other words, the FFN channels are clustered into experts based on similarity and each cluster will have its own LoRA module for fine-tuning. Is the router truly necessary?
- I would also expect more ablation studies on the clustering side because using MoE and LoRA for FFN is not a new idea (a relevant paper is [1]). The most important contribution of this paper comes from clustering the channels to form experts. So how many clusters we should have? If we don't cluster them, what will be the final performance on the MTL datasets? (Please point me to the corresponding results if I missed those.)
- Since the MTO method is orthogonal to the MTL architecture design, I'm curious can the proposed QR loss be used to enhance other MTL models?
- For Table 1, to form a fair comparison, the MTO methods should be compared with QR independently on the same model design. The same thing applies to the MTL architectures. They should be compared with EMTAL without QR.
- The experiments are conducted on classification tasks. However, a lot of practical MTL scenarios are pixel-to-pixel prediction tasks, like NYUv2 and Takonomoy datasets. It's better to see results on those datasets.
- The inference time should be compared as well since one of the contributions is designed for fast inference.

I'm willing to raise my rating if the questions are answered fully.

[1] Li D, Ma Y, Wang N, et al. MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts[J]. arXiv preprint arXiv 2024.
limitations: limitations: The authors analyzed the limitations and social impact in the paper.

Comments: title: Response to Author Rebuttal
Please respond, reviewer fvpH
Please respond, reviewer 2Lcf
Response to the authors
comment: Dear Reviewer 2Lcf,

We are grateful for the opportunity to address your concerns and provide further clarification on our work.

Due to time constraints, we were unable to complete additional experiments on the self-supervised backbone DinoV2. However, we hope the following explanations can help alleviate your concerns.

Firstly, regarding the self-supervised vision backbone, our experiments in rebuttal on the self-supervised vision backbone MAE demonstrated the effectiveness of our method in comparison to prior MTO and MTL structure design methods, achieving a performance of 90.07. Consequently, we believe that our method could also yield promising results on the state-of-the-art self-supervised vision backbone DinoV2, we plan to include these additional validations in the final version of our paper similar to the experiments on MAE.

Secondly, we attempted to scale up ViT-B/16 to DinoV2 ViT-L backbone. However, we found that BOFT has not released its training code for VTAB, nor have the authors provided all experiment settings (e.g. the resolution of input images). Due to time constraints, it is challenging for us to obtain experimental results under fair conditions.

Nevertheless, we have attempted a preliminary comparison as follows:

| Method|Backbone|Unified Model|**Patch Camelyon**|**EuroSAT**|**Resisc45**|**Retinopathy**|**Mean Acc ↑**|**Total Params. (M)** **↓**|Tunable Params. (M) **↓**|Inference Time (ms) **↓**|
| -|-|-|-|-|-|-|-|-|-|-|
| Full Finetuning|DinoV2 ViT-L|✗|88.1|96.1|90.9|77.2|88.07|1217.6|1217.6|19.28|
| LoRA|DinoV2 ViT-L|✗|88.3|96.4|91.4|77.4|88.37|1217.6|7.08|19.28|
| BOFT|DinoV2 ViT-L|✗|88.9|96.6|91.6|77.3|88.60 (+0.23)*|1217.6 (+0)*|7.96 (+0.88)*|19.28 (+0)*|
| Full Finetuning|Supervised ViT-B|✗|79.7|95.7|84.2|73.9|83.38|343.3|343.3|13.10|
| LoRA|Supervised ViT-B|✗|85.5|95.3|86.1|75.3|85.50|343.3|2.41|13.10|
| Ours|Supervised ViT-B|$\checkmark$|87.4|96.1|89.1|78.9|87.89 **(+2.39)***|85.99 **(-257.31)***|0.78 **(-1.63)***|6.56 **(-6.54)***|

()* indicates gain compared to LoRA using the same backbone.

In the above table, it can be seen that on these four datasets, our multi-task method has a significantly less total parameters (85.99 M vs 1217.6 M) and tunable parameters (0.78 M vs 7.96 M) compared to the result of specialized model in BOFT. This makes our method more parameter-efficient. Furthermore, as ours is an unfied multi-task model, the inference time is significantly reduced (6.56 ms vs 19.28 ms). Moreover, a preliminary analogy shows that while BOFT improved the mean accuracy by 0.53% and 0.23% compared to full finetuning and LoRA methods respectively, our method improved by 4.51% and 2.39%. Therefore, we believe that scaling up the backbone to the DinoV2 ViT-L backbone could yield promising results.

Finally, regarding the validation of our multi-task learning method on tasks with greater domain shifts, we understand 'tasks with greater domain shifts' from two perspectives: 1. Tasks comprise data from different domains. 2. Tasks themselves exhibit great domain shifts (heterogeneous tasks).

To better demonstrate our validation in this regard, we have summarized the characteristics of each dataset in the following table:

||Tasks comprise data from different domains|Tasks themselves exhibit great domain shifts (heterogeneous tasks)|
| -|-|-|
| Specialized VTAB| $\checkmark$ |✗|
| WILDS| $\checkmark$|✗|
| NYUv2|✗| $\checkmark$|

As shown in this table, we believe that Specialized VTAB is similar to WILDS datasets in terms of domain shift validation, as it also comprise data from different domains, such as retina images, remote sensing images, satellite images, and histopathologic scans of lymph node sections, which exhibit large domain shifts. Therefore, along with the NYUv2 dataset, we validated our method on tasks with great domain shifts from both perspectives.

On the other hand, given that the WILDS dataset is seldom used for multi-task learning benchmarks, aligning the training details and establishing baselines would require a significant amount of time. 

However, we indeed believe that validation on more datasets would be beneficial, we plan to discuss this in the limitations section and consider multi-task learning with greater domain shifts as future work.

We sincerely appreciate your response to our rebuttal and the time you have dedicated to the review process. We will continue to refine our work based on your valuable suggestions.

Sincerely, 
Authors.
Thank you for your response and clarifications. I think it's important to have comparisons with DinoV2, as it is the state-of-the-art self-supervised vision backbone. BOFT was also made available late last year (Nov 2023), which was well before the submission date of NeurIPS. As I mentioned in the review, I think it's important to compare against this PEFT method using the DinoV2 ViT-L backbone since it shows strong performance on the same benchmarks used in this paper. 

Additionally, I still believe a greater distribution shift should be considered for evaluation (eg: WiLDS). I do appreciate the comparison on NYUv2, but I don't think this is sufficient, since the kinds of images in this dataset seem to be mostly similar to natural images. 

I still think the paper leans towards acceptance, but because of the unaddressed concerns above, I will keep my score.
Reviewer fvpH:

Please let us know whether the authors have addressed your concerns.

Thanks.

-AC
Reviewer 2Lcf:

Please let us know whether the authors have addressed your concerns.

Thanks.

-AC
Thank you for taking the time and effort to review our rebuttal and for your thoughtful consideration. We are truly grateful for the raised rating and will carefully incorporate your valuable suggestions to further improve our work.
Thanks for the detailed response. That's clear to me. I'll raise my score to 7.