Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning

## Abstract

Abstract Vision Foundation Models (VFMs) have demonstrated outstanding performance on numerous downstream tasks. However, due to their inherent representation biases originating from different training paradigms, VFMs exhibit advantages and disadvantages across distinct vision tasks.
Although amalgamating the strengths of multiple VFMs for downstream tasks is an intuitive strategy, effectively exploiting these biases remains a significant challenge. In this paper, we propose a novel and versatile “Swiss Army Knife” (SAK) solution, which adaptively distills knowledge from a committee of VFMs to enhance multi-task learning. Unlike existing methods that use a single backbone for knowledge transfer, our approach preserves the unique representation bias of each teacher by collaborating the lightweight Teacher-Specific Adapter Path modules with the Teacher-Agnostic Stem.
Through dynamic selection and combination of representations with Mixture-of-Representations Routers, our SAK is capable of synergizing the complementary strengths of multiple VFMs. Extensive experiments show that our SAK remarkably outperforms prior state of the arts in multi-task learning by 10% on the NYUD-v2 benchmark, while also providing a flexible and robust framework that can readily accommodate more advanced model designs.

## 1 Introduction

Vision Foundation Models (VFMs), such as DINOv2 (Oquab et al., ), CLIP (Radford et al., ), and SAM (Kirillov et al., ), have gained significant attention due to their impressive performance on various downstream tasks.
This underscores the importance of integrating VFMs into Multi-Task Learning (MTL) (Caruana, ; Zhang & Yang, ; Yu et al., ), which aims to jointly learn multiple tasks with a single network, thereby enhancing model efficiency and performance, with broad applications in areas like autonomous driving (Ishihara et al., ).

In computer vision, multi-task models typically use a shared encoder to extract general features for all tasks, as they share a common interpretation of visual input (Ye & Xu, ). A straightforward approach is to directly replace the encoder backbone with a VFM.
However, VFMs are pretrained on diverse datasets, image resolutions, and objectives, introducing representation biases when applied as feature extractors for downstream tasks. Our empirical study in Figure 1 reveals that these inherent biases yield both advantages and disadvantages across different tasks, with *no single model achieving consistently superior performance across all domains*.
These findings highlight the challenge of accomplishing comprehensive improvements in MTL using VFMs, pointing to the demand for collaborative utilization of multiple VFMs to exploit their complementary strengths.

Several existing works attempt an intuitive solution by extracting image features through multiple VFMs and then concatenating or fusing these features for later decoding (Lin et al., ; Kar et al., ; Zong et al., ; Tong et al., ; ; Man et al., ). While this enhances visual encoding, it comes with a major drawback: The inference of all vision encoders drastically increases computational costs, along with the memory and storage requirements due to the large-scale parameters, rendering it less practical for real-world applications.

Therefore, recent works (Ranzinger et al., ; Shang et al., ; Sariyildiz et al., ) propose more efficient frameworks by distilling multiple VFM teachers into a single student model, which can deliver competitive results on downstream benchmarks. Despite the progress, this many-to-one distillation risks eliminating the representation biases of the VFM teachers, potentially limiting the model’s ability to capitalize on their individual strengths for specific tasks. Zong et al. () further point out that biased information from VFMs can lead to performance degradation when naively fused. Moreover, when matching one student to multiple teachers, reconciling diverse biases in shared parameters could induce optimization conflicts. Our pilot study in Table 2 shows that the student trained by many-to-one distillation does not consistently surpass the teachers in their respective proficient tasks.

To overcome these limitations, we propose a novel approach named SAK, with the goal to build a Swiss Army Knife model from a committee of VFMs to synergize their complementary strengths and enhance performance across multiple downstream tasks.
Considering the key challenge of preserving the representation biases while ensuring model efficiency for deployment, we introduce a multi-teacher knowledge distillation framework. This framework incorporates a shared Teacher-Agnostic Stem alongside multiple Teacher-Specific Adapter Path modules, which produce specialized representations aligned with each corresponding VFM teacher. During distillation, the Teacher-Agnostic Stem is optimized simultaneously by gradients from all VFM teachers, thereby capturing universal knowledge. Meanwhile, the Teacher-Specific Adapter Paths accommodate the heterogeneous representation biases of each teacher, explicitly learning their diverse model characteristics.

Building on the reproduction of representation biases, the next step is to amalgamate the committee’s expertise by exploiting the individual biases.
Specifically, we treat each group of representations as a knowledgeable expert and design a Mixture-of-Representations Router. This router dynamically weighs and combines the most relevant representations, bridging the gap between general-purpose knowledge and task-specific characteristics.
The collaboration of these modules allows the student to harness both commonalities and differences of the teachers, facilitating smoother and more comprehensive knowledge transfer.
Furthermore, SAK is a highly flexible framework that can further benefit from more advanced architectural designs (*e.g*., stronger task-specific decoders) and more powerful models (*e.g*., larger teachers), offering a general solution to multi-task visual learning.

Our contributions are summarized as follows:

- •
We systematically analyze the distinct representation biases of Vision Foundation Models, which result in varying advantages and disadvantages across tasks, underscoring the importance of preserving these biases during distillation from multiple VFM teachers.
- •
We propose SAK, an efficient and effective solution that distills knowledge from VFM teachers into a Teacher-Agnostic Stem with Teacher-Specific Adapter Path modules, sharing common knowledge while retaining the biases. We also introduce Mixture-of-Representations Routers to adaptively amalgamate the complementary and specialized strengths for downstream tasks.
- •
We evaluate SAK on two widely-used multi-task benchmarks, PASCAL-Context and NYUD-v2, showing it remarkably outperforms previous multi-teacher VFM distillation methods and state-of-the-art multi-task models in both performance and robustness.
- •
SAK offers high flexibility and scalability, supporting a broad variety of VFM teachers and downstream tasks, and is compatible with various adapter, router, or decoder head architectures.

## 2 Representation Biases in Vision Foundation Models

**Table 1: Comparison of Vision Foundation Models. Although all utilize the same Vision Transformer (ViT) backbone, they greatly differ in their training paradigms, including data, image resolutions, and training objectives, which lead to diverse representation biases.**
| Model | Training Dataset | Dataset Size | Resolution | Objective |
| --- | --- | --- | --- | --- |
| ViT (Dosovitskiy et al., ) | ImageNet-1k/21k | 1.2M/14.2M | 384 | Supervised classification |
| DINOv2 (Oquab et al., ) | LVD-142M | 142M | 518 | Discriminative self-supervised learning |
| CLIP (Radford et al., ) | WebImageText | 400M | 224 | Image-text contrastive learning |
| OpenCLIP (Cherti et al., ) | LAION-2B | 2B | 384 | Image-text contrastive learning |
| SAM (Kirillov et al., ) | SA-1B | 11M+1B | 1024 | Supervised promptable segmentation |

In this section, we investigate the representation biases of Vision Foundation Models on multiple downstream tasks through empirical studies. We select three representative state-of-the-art VFMs: (1) *DINOv2* (Oquab et al., ), which claims to excel in dense prediction tasks such as semantic segmentation and depth estimation; (2) *CLIP* (Radford et al., ) and its reproduction, *OpenCLIP* (Cherti et al., ), which are recognized for capturing language-aligned semantics and employed as vision encoders in vision-language models; and (3) *SAM* (Kirillov et al., ), which achieves outstanding performance in promptable segmentation. For CLIP and SAM, we use only their image encoders for representation learning.

As summarized in Table 1, although all these VFMs utilize Vision Transformers (ViT) (Dosovitskiy et al., ) as backbones, they differ significantly in their training paradigms regarding datasets, dataset sizes, image resolutions, and training objectives. Consequently, *the representations learned by these models embed heterogeneous biases*, causing each model to focus on different aspects of image features and exhibit strengths and weaknesses in specific tasks.

We conduct comprehensive quantitative and qualitative experiments using the three VFMs on five dense prediction tasks from the PASCAL-Context dataset (Mottaghi et al., ). Among these tasks, intuitively, *semantic segmentation* and *human parsing* require high-level semantics of objects and localized features to generate accurate masks. *Saliency detection* demands an overall understanding of the image to identify its main contents, while *surface normal estimation* and *object boundary detection* depend more on fine-grained representations for precise predictions.

We further provide a pilot study to show the inferiority of ignoring representation biases in knowledge distillation from multiple VFM teachers, which validates the significance of addressing this problem and motivates the development of our methodology.

### 2.1 Quantitative Analysis

To quantitatively analyze the representation biases, we evaluate the performance of the three VFMs directly transferred to each downstream task. We first freeze the models to generate image representations based on their pretrained knowledge, and then train a decoder head to produce final predictions for each task. DINOv2 and CLIP operate at a resolution of 512 on the downstream dataset, while SAM uses an input size of 1,024 as required by its pipeline. All feature maps are resized to $1/4$ of the input resolution before being passed to the head.
To quantify the advantages of VFMs over the conventional ImageNet-pretrained ViT backbone, we calculate their relative improvement over ViT for each task.

Figure 1(Left) illustrates how the representation biases in VFMs manifest in varying strengths and weaknesses in different downstream tasks. Specifically, DINOv2 shows significant improvements in two segmentation tasks, particularly excelling in human parsing with a performance gain of over 30%. It also performs well in object boundary detection, benefiting from its strongly localized features learned from the combination of image-level contrastive objective (Caron et al., ) and patch-level reconstructive objective iBOT (Zhou et al., ). While CLIP achieves lower accuracy than DINOv2 in these three tasks, it still exceeds the baseline by a notable margin of over 5%. Despite being pretrained on a segmentation task, SAM surprisingly underperforms ViT in semantic segmentation, showing a 30% drop, because of its limited semantic understanding—SAM considers solely the object masks and ignores their semantic labels in its promptable segmentation task.
However, SAM is the best in surface normal estimation and object boundary detection, exhibiting strength in capturing pixel-level details and object edges.
We also compute the ratio of mean improvement $\mu$ to standard deviation $\sigma$ across tasks, which can measure the consistency of improvements. A higher ratio indicates better outcomes, as it reflects larger average improvements with smaller dispersion. We can observe that while DINOv2 demonstrates stronger average enhancement, CLIP attains more balanced results, whereas SAM is inferior in both perspectives.

### 2.2 Qualitative Analysis

To validate our quantitative findings, we visualize the final predictions for semantic segmentation and boundary detection using an example image in Figure 1(Right). We observe that DINOv2, while being effective at capturing localized features, is less effective than CLIP in semantic perception, as illustrated in the yellow box of Semseg results. In this case, DINOv2 confuses a chair with a sofa, resulting in misclassification as the background (the holes). Although CLIP excels in object-level understanding with its rich semantic knowledge from the language domain, it falls short in generating fine-grained pixel-level masks. This shortcoming arises because CLIP’s training objective prioritizes image-level contents that are represented only by the class token, which possibly accounts for its lower performance than DINOv2 in the quantitative analysis.

On the other hand, SAM produces exceptional details in both tasks due to its high input resolution, as demonstrated in the red and cyan boxes. In the red box, a complex scene shows a foreground flower blending into the background, yet SAM accurately detects and labels the background in the segmentation mask. Notably, the background is not annotated in the ground truth, as such precise masking requires significant time and effort. We regard SAM’s high resolution as its representation bias, as it stems directly from the model’s training paradigm.
However, SAM’s limitation lies in its semantic knowledge, particularly when integrating semantics from multiple objects. This makes it difficult to attain high-quality semantic segmentation results, even with highly precise masks, echoing the quantitative analysis.

**Table 2: Comparison of a student model trained by many-to-one distillation without preserving representation biases and the oracle derived from VFM teachers. The oracle selects the best result from the three teachers for each task. The student’s 2.34% average underperformance demonstrates the critical importance of maintaining these biases during distillation.**
| Model | Semseg | Parsing | Saliency | Normal | Boundary |
| --- | --- | --- | --- | --- | --- |
| mIoU$\uparrow$ | mIoU$\uparrow$ | maxF$\uparrow$ | mErr$\downarrow$ | odsF$\uparrow$ |  |
| Oracle of teachers | 81.18 (DINOv2) | 74.38 (DINOv2) | 81.48 (CLIP) | 16.21 (SAM) | 75.89 (SAM) |
| Student w/o biases | 80.18 ($\downarrow 1.23\%$) | 69.13 ($\downarrow 7.06\%$) | 82.72 ($\uparrow 1.52\%$) | 16.00 ($\uparrow 1.30\%$) | 71.16($\downarrow 6.23\%$) |

### 2.3 Importance of Preserving Representation Biases

In summary, the inherent representation biases in VFMs result in their uneven performance across tasks, *with no single model achieving the best results in all areas*, as reflected in our quantitative analysis. This motivates the idea of combining multiple VFMs to achieve optimal performance in all tasks. Existing methods (Ranzinger et al., ; Shang et al., ; Sariyildiz et al., ) propose a solution by distilling multiple VFMs into a single student model. However, given that the student model is shared by several teachers, an important question naturally arises: Should we respect their individual representation biases when combining diverse VFMs?

To answer this question, we conduct a pilot study where a student model is distilled from three aforementioned VFMs, using only linear aligners to match the student’s features with those of the teachers, following the setup of the state-of-the-art method (Ranzinger et al., ). In this approach, the representation biases are not explicitly preserved during distillation, leading the student to learn a unified representation aimed at simultaneously matching all three teachers.
We then evaluate its performance on downstream tasks with the same settings as in our quantitative analysis in Section 2.1. We compare the student without biases to an oracle derived from the teachers by selecting the best-performing teacher for each task, which represents the optimal performance of teachers.

From Table 2, the answer is clearly YES. While the distilled student surpasses the oracle in Saliency and Normal, somewhat validating the effectiveness of prior methods, it suffers from drastic performance degradation in Parsing and Boundary, with a drop of over 6%. Given that both the teachers and student utilize a ViT-B backbone in this study, the performance gap would likely widen with larger models. This demonstrates the limitation of naively transferring knowledge from multiple teachers into a student and highlights the importance of preserving the individual biases, leading us to the key question: Can we preserve the representation biases of multiple VFMs during distillation to maximize multi-task performance? Our methodology provides a positive answer to this challenge in the following sections.

## 3 Methodology

### 3.1 Overview

Figure: Figure 2: Overview of our proposed SAK framework, which distills foundational knowledge from a committee of frozen VFM teachers into an efficient student model. The student model operates like a Swiss Army Knife, with the Teacher-Agnostic Stem (TAS) serving as the main branch to learn universal knowledge among teachers. Each Teacher-Specific Adapter Path (TSAP) acts as a specialized tool to preserve the inherent representation bias of each teacher. Task-specific Mixture-of-Representations (MoR) Routers are then employed to synergize the complementary strengths of the teachers’ biases, adaptively combining multi-level representations from both TAS and TSAP to generate tailored features for each task.
Refer to caption: x3.png

The overall framework of the proposed SAK is depicted in Figure 2. As a multi-teacher distillation approach, it employs a committee of VFM teachers, including DINOv2, CLIP, and SAM. The student model comprises a Teacher-Agnostic Stem (TAS) and multiple Teacher-Specific Adapter Path (TSAP) modules. TAS produces general representations shared across all branches, while each TSAP adapts the common representations to align with the specialized domain of its corresponding teacher via distillation.
In this approach, the TSAP modules are optimized explicitly to replicate the unique representation biases of the teachers, all in a parameter- and computationally-efficient manner.
The resulting feature sets are then passed through task-specific Mixture-of-Representations (MoR) Routers for adaptive combination and are finally processed by prediction heads to generate outputs for multiple tasks. We utilize multi-level representations for both the distillation and task decoding procedures, an essential aspect for dense prediction tasks (Ye & Xu, ).

### 3.2 Teacher-Agnostic Stem & Teacher-Specific Adapter Path

We adopt an off-the-shelf Vision Transformer (ViT) (Dosovitskiy et al., ) as the Teacher-Agnostic Stem (TAS) and design a lightweight network branch called Teacher-Specific Adapter Path (TSAP) parallel to the main stem. Given a TAS with $L$ blocks, the forward pass for an input image ${\bm{X}}$ is expressed as:

$$ ${\bm{Z}}_{0}=\texttt{PatchEmbed}({\bm{X}});\quad{\bm{Z}}_{l}=b_{l}({\bm{Z}}_{l -1}),\quad l\in\{1,2,\dots,L\},$ (1) $$

where $b_{l}$ represents the $l$-th block, and ${\bm{Z}}_{l}\in\mathbb{R}^{n\times d}$ denotes its intermediate outputs with $n$ tokens of dimension $d$. Each TSAP module consists of $L+1$ adapters $\{a_{l}\},l\in\{0,1,\dots,L\}$, with one adapter parallel to each patch embedding layer or transformer block. These adapters process intermediate features to adapt them to the teacher-specific representations ${\bm{R}}_{l}\in\mathbb{R}^{n\times d}$ in a residual manner:

$$ ${\bm{R}}_{0}=a_{0}({\bm{Z}}_{0});\quad{\bm{R}}_{l}=a_{l}({\bm{R}}_{l-1}+{\bm{Z }}_{l}),\quad l\in\{1,2,\dots,L\}.$ (2) $$

We utilize the standard adapter structure (Houlsby et al., ), which includes a down-projection layer ${\mathbf{W}}_{down}\in\mathbb{R}^{d\times r}$, a GELU non-linearity (Hendrycks & Gimpel, ), and an up-projection layer ${\mathbf{W}}_{up}\in\mathbb{R}^{r\times d}$, where $r\ll d$ is the reduced dimension. As in prior works (Chen et al., ; Mercea et al., ), we integrate a learnable scaling factor $\alpha$ and a residual connection from the input ${\bm{R}}_{in}\in\mathbb{R}^{n\times d}$ to the output ${\bm{R}}_{out}\in\mathbb{R}^{n\times d}$, which can be formulated as:

$$ ${\bm{R}}_{out}=\alpha{\mathbf{W}}_{up}\texttt{GELU}({\mathbf{W}}_{down}{\bm{R} }_{in})+{\bm{R}}_{in}.$ (3) $$

For a committee of $N$ VFM teachers, we assign a TSAP module with adapters $\{a_{l}^{i}\},l\in\{0,1,\dots,L\}$ to the $i$-th teacher. We then select four evenly distributed blocks from its outputs $\{{\bm{R}}_{l}^{i}\}$ to form multi-level representations $\{{\bm{R}}^{i}\}_{s}=\{{\bm{R}}^{i}_{l}\},l\in{\mathbb{L}}_{s}=\{L/4,L/2,3L/4,L\}$.
Similarly, we have shared multi-level representations $\{{\bm{Z}}\}_{s}$ from TAS.
Benefiting from the lightweight adapters, our distilled student model maintains original efficiency, as each TSAP module accounting for less than 5% of the TAS parameters. Consequently, our SAK framework is able to preserve the representation biases from the teachers without significant increases in computational cost, memory usage, or storage.

### 3.3 Mixture-of-Representations Router

As depicted in Figure 2, we treat the representations from TAS $\{{\bm{Z}}\}_{s}$ as a shared expert providing common knowledge, while the representations from each TSAP $\{{\bm{R}}^{i}\}_{s},i\in\{1,2,\dots,N\}$ serve as proxy experts of VFMs with representation biases mirroring the teachers, resulting in a total of $N+1$ experts. To optimize the multi-task performance, we leverage the Mixture-of-Experts (MoE) mechanism (Jacobs et al., ), which adaptively produces task-specific features from this pool of general-purpose and specialized representations.

To facilitate this, task-specific router networks are trained to generate gate scores for each expert representation, which serve as the weights for a linear combination. As shown in Figure 6, the representations from different VFMs exhibit substantial variation in norm magnitudes. Thus, applying different weights for individual patches within an image can be less effective, as it may disturb the inherent patterns. To address this, we design a Mixture-of-Representations (MoR) Router, which differs from prior works by generating a global gating score across all patches.

Specifically, for each selected level $l\in{\mathbb{L}}_{s}$ and downstream task $t\in{\mathbb{T}}$, our MoR Router $r_{l}^{t}$ takes the teacher-agnostic representation ${\bm{Z}}_{l}\in\mathbb{R}^{n\times d}$ as input, projects its channel dimension to $N+1$ through a two-layer MLP, and then averages over $n$ patches to get a feature vector ${\bm{h}}_{l}^{t}\in\mathbb{R}^{N+1}$. To improve stability, we incorporate the noisy gating technique (Shazeer et al., ) by generating a noise vector ${\bm{e}}_{l}^{t}\in\mathbb{R}^{N+1}$ through an additional MLP. Then we compute the gating score ${\bm{g}}_{l}^{t}\in\mathbb{R}^{N+1}$:

$$ ${\bm{g}}_{l}^{t}=\texttt{Softmax}({\bm{h}}_{l}^{t}+\mathcal{N}(0,1)\texttt{ Softplus}({\bm{e}}_{l}^{t})).$ (4) $$

The output gating scores are used to calculate the weighted sum of representations at each output level. The fused features are then passed through task-specific heads for final predictions.

### 3.4 Training Paradigm

Our training paradigm contains two stages, with teacher parameters always frozen. In the first stage, we train the student model on the ImageNet-1k dataset (Deng et al., ; Russakovsky et al., ), focusing on aligning the outputs of the TSAP modules with their respective VFM teachers. ImageNet is chosen due to its diverse and extensive image samples, providing a strong basis for effective knowledge transfer. To maintain fairness—given that conventional ViT backbones are also pretrained on ImageNet—we opt not to use other larger datasets like those utilized in VFMs and RADIO (Ranzinger et al., ). Following previous findings (Ranzinger et al., ; Shang et al., ), we employ a combination of cosine distance and smooth-L1 losses for distillation. Let ${\bm{T}}_{l}^{i}$ be the $i$-th teacher’s representation at a selected level $l\in{\mathbb{L}}_{s}$, the overall distillation loss is defined as:

$$ $\mathcal{L}_{\text{distill}}({\bm{X}})=\sum_{l\in{\mathbb{L}}_{s}}\sum_{i=1}^{ N}\left(\alpha\mathcal{L}_{\text{cos}}({\bm{R}}_{l}^{i},{\bm{T}}_{l}^{i})+ \beta\mathcal{L}_{\text{smooth-L1}}({\bm{R}}_{l}^{i},{\bm{T}}_{l}^{i})\right).$ (5) $$

where $\alpha=0.9$ and $\beta=0.1$ are weighting coefficients.

In the second stage, we continue training on the downstream multi-task datasets. The distillation loss is still included, allowing the VFM teachers to transfer more specialized knowledge related to the downstream data domain. This ensures that the representation biases are further secured in the student model; otherwise the biases could potentially be diminished due to the issue of catastrophic forgetting (French, ) during downstream fine-tuning.
The overall loss is then formulated as:

$$ $\mathcal{L}({\bm{X}})=\gamma\mathcal{L}_{\text{distill}}({\bm{X}})+\sum_{t\in{ \mathbb{T}}}w_{t}\mathcal{L}_{t}({\bm{X}},{\bm{Y}}_{t}),$ (6) $$

where $\mathcal{L}_{t}({\bm{X}},{\bm{Y}}_{t})$ is the task-specific loss for task $t$, computed using the ground truth ${\bm{Y}}_{t}$. The hyperparameter $\gamma$ balances the distillation loss and the task losses, with a default value of $1.0$ for simplicity, while $w_{t}$ adjusts the importance of each task. We set fixed $w_{t}$ values following the standard practice in MTL (Maninis et al., ; Kanakis et al., ).

## 4 Experiments

### 4.1 Experimental Setup

Datasets.
We conduct experiments on two widely-used multi-task datasets: PASCAL-Context (Mottaghi et al., ) with five vision tasks and NYUD-v2 (Silberman et al., ) with four tasks. Details can be found in Appendix B.

Implementation.
We employ a pretrained ViT backbone for TAS and use simple task-specific heads consisting of MLP and convolution layers for decoding. The VFM teachers are DINOv2, CLIP, and SAM with the ViT-L backbones, unless otherwise stated.
More implementation details are provided in Appendix B to ensure reproducibility.

Baselines.
To evaluate the effectiveness of our method, we consider three categories of baselines: (1) *Single-task baseline*, where individual models are trained for each task using the same ViT-initialized architecture, and *multi-task baseline*, where a shared encoder and task-specific heads are trained jointly. (2) *Multi-teacher VFM distillation approaches*, namely RADIO (Ranzinger et al., ) and Theia (Shang et al., ). We use their released models as encoder backbones, coupled with the same task heads as ours. (3) *State-of-the-art MTL models*, which involves complicated encoder or decoder designs. We assess the overall performance of each model with MTL Gain $\Delta_{m}$ by calculating the average relative difference across all tasks compared to the single-task baseline (Maninis et al., ).

### 4.2 Main Results

Figure: Figure 3: Performance comparison on two datasets, based on ViT-B backbones. MTL Gain $\Delta_{m}$ on two datasets are shown in the legend, respectively.
Refer to caption: x4.png

**Table 3: Comparison with state of the arts on PASCAL-Context, based on ViT-L backbones.**
| Model | Backbone | #Param | Semseg | Parsing | Saliency | Normal | Boundary | ${\Delta_{m}\%\uparrow}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU$\uparrow$ | mIoU$\uparrow$ | maxF$\uparrow$ | mErr$\downarrow$ | odsF$\uparrow$ |  |  |  |  |
| Single-task baseline | ViT-L | 1573M | 81.61 | 72.77 | 83.80 | 13.87 | 75.24 | 0.00 |
| Multi-task baseline | ViT-L | 357M | 79.26 | 68.28 | 84.16 | 14.06 | 71.59 | -2.97 |
| \hdashlinePAD-Net (Xu et al., ) | ViT-L | 330M | 78.01 | 67.12 | 79.21 | 14.37 | 72.60 | -4.95 |
| MTI-Net (Vandenhende et al., ) | ViT-L | 851M | 78.31 | 67.40 | 84.75 | 14.67 | 73.00 | -3.81 |
| ATRC (Brüggemann et al., ) | ViT-L | 340M | 77.11 | 66.84 | 81.20 | 14.23 | 72.10 | -4.71 |
| InvPT (Ye & Xu, ) | ViT-L | 423M | 79.03 | 67.61 | 84.81 | 14.15 | 73.00 | -2.81 |
| InvPT++ (Ye & Xu, ) | ViT-L | 421M | 80.22 | 69.12 | 84.74 | 13.73 | 74.20 | -1.19 |
| TaskPrompter (Ye & Xu, ) | ViT-L | 401M | 80.89 | 68.89 | 84.83 | 13.72 | 73.50 | -1.24 |
| TaskExpert (Ye & Xu, ) | ViT-L | 420M | 80.64 | 69.42 | 84.87 | 13.56 | 73.30 | -0.97 |
| BFCI (Zhang et al., ) | ViT-L | 477M | 80.64 | 70.06 | 84.64 | 13.82 | 72.96 | -1.32 |
| 3D-aware (Li et al., ) | ViT-L | 430M | 79.53 | 69.12 | 84.94 | 13.53 | 74.00 | -1.08 |
| TSP (Wang et al., ) | ViT-L | 423M | 81.48 | 70.64 | 84.86 | 13.69 | 74.80 | -0.22 |
| MLoRE (Yang et al., ) | ViT-L | 407M | 81.41 | 70.52 | 84.90 | 13.51 | 75.42 | 0.16 |
| \hdashlineRADIO (Ranzinger et al., ) | ViT-L | 372M | 81.11 | 71.50 | 85.17 | 13.49 | 74.80 | 0.29 |
| SAK (Ours) | ViT-L | 407M | 84.01 | 76.99 | 84.65 | 13.82 | 76.27 | 2.30 |

**Table 4: Comparison with state of the arts on NYUD-v2, based on ViT-L backbones.**
| Model | Backbone | #Param | Semseg | Depth | Normal | Boundary | ${\Delta_{m}\%\uparrow}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU$\uparrow$ | RMSE$\downarrow$ | mErr$\downarrow$ | odsF$\uparrow$ |  |  |  |  |
| Single-task baseline | ViT-L | 1259M | 54.19 | 0.5560 | 19.22 | 78.09 | 0.00 |
| Multi-task baseline | ViT-L | 346M | 52.42 | 0.5413 | 19.29 | 76.50 | -0.76 |
| \hdashlineInvPT (Ye & Xu, ) | ViT-L | 402M | 53.56 | 0.5183 | 19.04 | 78.10 | 1.64 |
| InvPT++ (Ye & Xu, ) | ViT-L | $\sim$402M | 53.85 | 0.5096 | 18.67 | 78.10 | 2.65 |
| TaskPrompter (Ye & Xu, ) | ViT-L | 392M | 55.30 | 0.5152 | 18.47 | 78.20 | 3.36 |
| TaskExpert (Ye & Xu, ) | ViT-L | 400M+ | 55.35 | 0.5157 | 18.54 | 78.40 | 3.33 |
| BFCI (Zhang et al., ) | ViT-L | 400M+ | 55.51 | 0.4930 | 18.47 | 78.22 | 4.46 |
| 3D-aware (Li et al., ) | ViT-L | 409M | 54.87 | 0.5006 | 18.55 | 78.30 | 3.74 |
| TSP (Wang et al., ) | ViT-L | 402M | 55.39 | 0.4961 | 18.44 | 77.50 | 4.07 |
| MLoRE (Yang et al., ) | ViT-L | 552M | 55.96 | 0.5076 | 18.33 | 78.43 | 4.26 |
| \hdashlineRADIO (Ranzinger et al., ) | ViT-L | 362M | 59.32 | 0.4698 | 17.46 | 79.41 | 8.95 |
| SAK (Ours) | ViT-L | 394M | 63.18 | 0.4313 | 16.25 | 79.43 | 14.05 |

Figure 3 presents a comparison between our proposed SAK and representative baseline methods on both PASCAL-Context and NYUD-v2 datasets, with all methods using the ViT-B backbones. On PASCAL-Context, SAK greatly boosts the performance in Semseg and Parsing, achieving an overall improvement of 1.66% over the previous SOTA. On NYUD-v2, our method establishes a new milestone across all four tasks, increasing the MTL Gain metric from the previous best of 6.33% to 11.11%. We provide more comprehensive comparisons on both datasets using the ViT-L backbones in Tables 3 and 4. Our approach consistently outperforms previous methods, achieving the best results on 7 out of 9 tasks and both MTL Gain metrics. Notably, SAK significantly surpasses the SOTAs in MTL (BFCI (Zhang et al., ), MLoRE (Yang et al., )) by nearly 10% on NYUD-v2, all while using fewer parameters.

### 4.3 In-depth Analysis

We conduct extensive experiments to validate the effectiveness and generalization of our proposed SAK framework. All experimental analyses are based on the ViT-B backbones for both teachers and student and the PASCAL-Context dataset unless otherwise specified.

Ablation study. An ablation study is conducted to discern the individual contributions of the main components in SAK, namely TSAP and MoR Router, as outlined in Table 3. We consider two model variants: (1) a model without the TSAP and MoR Router modules (row 1), which corresponds to the student distilled naively regardless of representation biases, as studied in Table 2; (2) a model distilled with TSAP in the first stage but trained without MoR Routers in the second stage (row 2), where the biased representations from multiple VFMs are simply added together. Firstly, our results confirm that our proposed TSAP effectively preserves the representation biases from the teachers as indicated by a higher average similarity between the student and teachers. Additionally, we prove that a simple fusion of diverse biased knowledge does not lead to an overall improvement and may even fall behind compared to the student without biases. With the synergization of TSAP and MoR Router, our proposed SAK not only preserves and reproduces the representation biases after distillation but also optimally capitalizes on these biases to maximize multi-task performance.

Table 3 reports another ablation on our training paradigm, highlighting the contributions of each stage. The results show that Stage 1, which distills knowledge from VFM teachers on ImageNet, is a primary factor of performance enhancement. Meanwhile, incorporating the distillation loss during Stage 2 consistently boosts final outcomes, regardless of whether Stage 1 is applied. This underscores the effectiveness of transferring specialized knowledge in the downstream data domain.

Impact of teacher selection.
To investigate whether knowledge from all teachers can be effectively incorporated into the student model and how each teacher contributes to downstream tasks, we experiment on different combinations of VFM teachers in Table 4. When using a single teacher, SAK effectively learns the teacher’s representation bias, as the student distilled from DINOv2 or CLIP performs well in segmentation tasks, while SAM’s student is better in tasks requiring finer details. Combining DINOv2 and CLIP continues to improve segmentation tasks, potentially due to their complementary strengths in localized feature learning and semantic understanding. Including SAM further benefits all tasks, leading to the best overall results. We also visualize the gating weights learned by our proposed MoR Routers at the lowest and highest levels in Figure 4. At the lowest level, where VFM teachers share more general knowledge about the details, tasks tend to rely on the shared TAS and SAM’s bias. Conversely, the representation biases become more pronounced at higher levels; therefore, the teacher-specific representations are predominantly selected.

Figure: Figure 4: Weights of different experts learned by MoR Routers.
Refer to caption: x5.png

Figure: Figure 5: Performance w.r.t. downstream data percentage. MTL Gain is computed w.r.t. single-task baseline on full dataset. SAK is the most robust in downstream tasks.
Refer to caption: x6.png

Impact of downstream data size.
To assess the robustness of multi-teacher VFM distillation methods, we conduct experiments using varying numbers of samples among {25%, 50%, 75%, 100%} from the downstream dataset. As depicted in Figure 5, while all models show an upward trend as the number of data samples increases, our SAK consistently outperforms the other two distillation baselines across all settings. Particularly, SAK surpasses the second-best method by a clear margin of over 3% in scenarios with substantially fewer samples such as merely 25%.

Scaling with model size.
In Table 5, we explore the impact of scaling the backbone sizes of the VFM teachers and student by forming various combinations. The results indicate that increasing the capacity of the student model, while keeping the teacher models fixed (row 1 *vs*. row 2, row 3 *vs*. row 4), yields remarkable improvements across nearly all tasks. Additionally, scaling up the teacher models without altering the student (row 2 *vs*. 3) also proves beneficial. These results demonstrate the versatility and robustness of our approach in adapting to models of varying sizes.

Compatibility with different decoders.
It is worth noting that our SAK framework is flexible and does not impose constraints on the design of the backbone, the adapters in TSAP, or the decoder heads. As shown in Table 5, we replace the simple head with the more complex MLoRE decoder (Yang et al., ). Even with a simple head, SAK surpasses MLoRE by 0.8%, and integrating the MLoRE decoder further enhances overall performance by 1.72%.

## 5 Related Work

Knowledge Distillation of Vision Foundation Models.
As large-scale generalists, Vision Foundation Models (VFMs) show superior performance in various tasks with minimal tuning, such as CLIP (Radford et al., ) for vision-language tasks, DINOv2 (Oquab et al., ) for fine-grained recognition, and SAM (Kirillov et al., ) for promptable segmentation. To reduce their computational demands while preserving performance, knowledge distillation (Buciluǎ et al., ; Hinton et al., ) has been widely adopted in compressing VFMs (Vemulapalli et al., ; Sun et al., ; Yang et al., ). More recently, multiple VFMs are distilled into a single student to combine their strengths: SAM-CLIP (Wang et al., ) merges CLIP into SAM via continual learning and distillation. RADIO (Ranzinger et al., ) learns from CLIP, DINOv2, and SAM to enhance performance on downstream tasks. Theia (Shang et al., ) further incorporates Depth Anything (Yang et al., ), showing advantages in robot learning. Different from the straightforward distillation in these methods, we adaptively transfer knowledge from multiple teachers while retaining the unique representation biases to maximize their strengths for multiple tasks.

Multi-Task Learning.
Multi-Task Learning (MTL) aims to train a single model capable of handling multiple tasks simultaneously (Caruana, ; Zhang & Yang, ; Yu et al., ). MTL research primarily falls into two categories: multi-task optimization (Kendall et al., ; Chen et al., ; Yu et al., ) and model architecture design (Long et al., ; Wallingford et al., ; Lu et al., ). Considering vision tasks, most works center on designing architectures, which is further divided into encoder-focused and decoder-focused methods (Vandenhende et al., ). Encoder-focused methods develop encoders to extract features for different tasks (Misra et al., ; Ruder et al., ; Gao et al., ), while decoder-focused methods introduce task-interaction modules in decoder to better capture task-specific features (Ye & Xu, ; Xu et al., ; Ye & Xu, ).

Knowledge distillation has also been applied to enhance MTL (Li & Bilen, ; Jacob et al., ; Ghiasi et al., ; Luo et al., ; Ye et al., ). These methods train a multi-task model to mimic multiple single-task teachers, allowing the student to gain richer information. Xu et al. () propose directly distilling a small multi-task student from a large multi-task teacher. To the best of our knowledge, our work is the first exploration of multi-task distillation with general-purpose knowledge from task-agnostic VFM teachers, as opposed to task-related teachers trained on target datasets.

## 6 Conclusion

Building on our analysis of the representation biases in VFMs, we introduce a novel framework SAK, designed to improve multi-task learning by exploiting the complementary biases of multiple VFMs. Through the integration of a Teacher-Agnostic Stem, Teacher-Specific Adapter Paths, and Mixture-of-Representations Routers, SAK effectively preserves the unique representation biases during distillation, thereby enhancing both accuracy and robustness across multiple downstream tasks.
Our work opens possibilities for including more advanced teachers and students, and provides a solid foundation for future advancements in multi-task visual learning with foundation models.

## Appendix A Additional Analysis of Representation Biases

Figure: Figure 6: Qualitative analysis of representation biases in three VFMs based on a pair of manipulated images. We visualize the representations by calculating the L2 norm, their differences in response to content changes, and their semantic segmentation predictions. CLIP struggles to detect small objects, as they resemble background tokens, leading to poor segmentation; in contrast, DINOv2 excels in small object detection but confuses semantic features in complex scenarios, while SAM emphasizes edges, enhancing pixel-level detail but limiting high-level understanding.
Refer to caption: x7.png

Based on our analysis in Section 2, we further delve into the representations generated by VFMs to understand how the representation biases lead to different outcomes across tasks. In Figure 6, we create a pair of manipulated images by selecting several cat images from the dataset and converting them into dogs while preserving the same shape and pose using ControlNet (Zhang et al., ). We place various sizes of cats and dogs in a shared background image to compose complex and challenging scenes. For the segmentation task, this ensures that the predictions should differ only in labels, while the mask shapes remain the same. We visualize the representations by calculating the L2 norm of each patch token.

The red boxes highlight why CLIP struggles to detect small objects, such as the smallest cat and dog. In the representations, small objects exhibit a similar pattern to neighboring background tokens, indicating that CLIP assigns them the same level of importance as the background. Consequently, CLIP fails to segment these small objects and also struggles to generate precise masks for medium-sized objects, as shown in the cyan boxes. Conversely, DINOv2 performs better at detecting small objects by leveraging representations where these objects receive more attention than the background. However, its semantic understanding is limited in complex scenarios. For instance, DINOv2 confuses the heads of the cat and dog in the cyan boxes, even though this is the most distinguishable feature between the two animals. SAM displays different behaviors, with its representations concentrating more on the edges in the images instead of the entire objects, which explains its strength in capturing pixel-level details and weakness in high-level knowledge, as mentioned in our quantitative analysis. When examining the differences between the representations of the two images, CLIP appears more sensitive to changes in the main objects than DINOv2 and SAM, further underscoring its semantic focus. We also observe the phenomenon of register tokens (Darcet et al., ) in CLIP’s representation, where these artifacts correspond to areas the model deems low-informative background.

Additionally, we extend our findings by providing insightful explanations that link the underlying causes (differences in training paradigms) and effects (variations in downstream performance) of the representation biases:

*DINOv2* excels at capturing features across multiple scales, stemming from its combination of image-level contrastive objective (Caron et al., ) and patch-level reconstructive objective iBOT (Zhou et al., ). Although it lacks semantic-based supervision, its general-purposed visual representations exhibit powerful generalization capability for most downstream tasks.

*CLIP* is trained by aligning image and text embeddings of numerous image-text pairs through contrastive learning, which incorporates rich semantic knowledge from language domain that enhances image-level understanding. However, CLIP tends to prioritize prominent objects in an image while overlooking smaller ones, because (1) the alignment only considers the global class token (patch tokens are optimized implicitly), and (2) the image captions mainly focus on primary contents. This possibly accounts for its lower performance compared to DINOv2 in our quantitative experiments.

*SAM*’s representation bias can be explained by two reasons. First, SAM is supervised by the promptable segmentation task, considering solely the object masks and ignoring their semantic labels. Second, SAM operates with an input resolution of 1,024 in both training and inference, producing nearly four times more patches than other models. Consequently, SAM is adept at capturing pixel-level details and detecting object edges but poor at perceiving the semantics of multiple objects.

## Appendix B Implementation Details

### B.1 Models

Architecture.
We use DINOv2 (Oquab et al., ), CLIP (Radford et al., ), and SAM (Kirillov et al., ) as our VFM teachers, with pretrained models from timm (Wightman, ). For CLIP, we utilize the OpenCLIP (Cherti et al., ) reproduction in our experiments. The model names of the VFMs and multi-teacher VFM distillation baselines, RADIO (Ranzinger et al., )(^1^11[https://github.com/NVlabs/RADIO](https://github.com/NVlabs/RADIO)) and Theia (Shang et al., )(^2^22[https://huggingface.co/collections/theaiinstitute/theia-66a7a6ae80a707547c358cce](https://huggingface.co/collections/theaiinstitute/theia-66a7a6ae80a707547c358cce)), are listed in Table 5. For the Teacher-Agnostic Stem in the SAK student model, we employ ViT-S, ViT-B, and ViT-L backbones, initialized with pretrained weights from timm. While we keep the CLS tokens in the model, we only use patch tokens as representations for both the teachers and student models. Other modules in the student are trained from scratch. The adapters in the Teacher-Specific Adapter Path use a channel reduction ratio of 4.

**Table 5: Model zoo of VFMs and multi-teacher VFM distillation methods.**
| Model | Platform | Backbone | Name |
| --- | --- | --- | --- |
| ViT (Dosovitskiy et al., ) | timm | ViT-S/16 | vit_small_patch16_384 |
| ViT-B/16 | vit_base_patch16_384 |  |  |
| ViT-L/16 | vit_large_patch16_384 |  |  |
| \hdashlineDINOv2 (Oquab et al., ) | timm | ViT-B/14 | vit_base_patch14_dinov2 |
| ViT-L/14 | vit_large_patch14_dinov2 |  |  |
| \hdashlineCLIP (Radford et al., ) | timm | ViT-B/16 | vit_base_patch16_clip_384 |
| OpenCLIP (Cherti et al., ) | ViT-L/14 | vit_large_patch14_clip_336 |  |
| \hdashlineSAM (Kirillov et al., ) | timm | ViT-B/16 | samvit_base_patch16 |
| ViT-L/16 | samvit_large_patch16 |  |  |
| \hdashlineRADIO (Ranzinger et al., ) | torch hub | ViT-B/16 | NVlabs/RADIO/radio_v2.5-b |
| ViT-L/16 | NVlabs/RADIO/radio_v2.5-l |  |  |
| \hdashlineTheia (Shang et al., ) | huggingface | ViT-S/16 | theaiinstitute/theia-small-patch16-224-cdiv |
| ViT-B/16 | theaiinstitute/theia-base-patch16-224-cdiv |  |  |

Input resolution.
As detailed in Table 1, the VFMs are pretrained on diverse image resolutions. Additionally, RADIO-B is pretrained on $768\times 768$, RADIO-L is pretrained on $1024\times 1024$, and Theia on $224\times 224$. In our experiments, we use a unified input size for all models, except SAM. In the first stage, we use an input size of $384\times 384$, while in the second stage, the input size is set to $512\times 512$ for PASCAL-Context and $448\times 576$ for NYUD-v2. For SAM, we pad and resize all images to $1024\times 1024$. Feature maps from all models are interpolated to $1/4$ of the input resolution before being decoded by the heads for a fair comparison.

Teacher/student mismatch.
Since we have VFM teachers operating with different input and patch sizes (14 or 16), their intermediate representations have different spatial resolutions. Take an input size of $512\times 512$ as an example, DINOv2-B/14 teacher outputs a $36\times 36$ map, CLIP-B/16 outputs a $32\times 32$ map, SAM-B/16 outputs a $64\times 64$ map, while the student based on ViT-B/16 outputs a $32\times 32$ map. Moreover, the number of channels may differs between teachers and students when using distinct backbones, such as 1024 for ViT-L teacher and 768 for ViT-B student. To align the student’s representations with the teachers’, we introduce an upsampling layer to match the spatial resolution, and a linear layer for channel mapping when necessary. Note that the linear layer is only required during distillation, adding no overhead to the inference.

### B.2 Datasets

In the first stage, we perform knowledge distillation using the ImageNet-1k dataset (Deng et al., ; Russakovsky et al., ), which contains 1.2 million images. Only the images are used, without any category labels.
To evaluate multi-task learning performance, we use two benchmark datasets in the second stage: PASCAL-Context (Mottaghi et al., ) and NYUD-v2 (Silberman et al., ). PASCAL-Context contains 4,998 training samples and 5,105 testing samples, annotated for five tasks of semantic segmentation (‘Semseg’), human parsing (‘Parsing’), saliency detection (‘Saliency’), surface normal estimation (‘Normal’), and object boundary detection (‘Boundary’). Among them, the surface normal and saliency labels are supplemented by previous work (Maninis et al., ). Meanwhile, NYUD-v2 consists of 795 images for training and 654 images for testing, all from indoor scenes. It provides labels for four tasks: semantic segmentation, monocular depth estimation (‘Depth’), surface normal estimation, and object boundary detection.

### B.3 Data Augmentation

In the first stage, we apply standard image augmentation techniques, including random resizing and cropping to $384\times 384$, along with random horizontal flipping.
For the downstream datasets, we follow the established data augmentation protocols from previous studies (Maninis et al., ; Kanakis et al., ; Ye & Xu, ). We use random scaling with factor between 0.5 and 2.0, random cropping to the required input resolution ($512\times 512$ for PASCAL-Context and $448\times 576$ for NYUD-v2), random horizontal flipping, and random color jittering. Image normalization is also applied throughout both the training and evaluation phases.

### B.4 Evaluation Metrics

We adopt evaluation metrics in previous methods (Ye & Xu, ). Specifically, we measure semantic segmentation and human parsing using the mean Intersection over Union (mIoU). Saliency detection is evaluated by the maximum F-measure (maxF). Surface normal estimation is evaluated by the mean error (mErr) of angles. Object boundary detection is evaluated by the optimal-dataset-scale F-measure (odsF) (Martin et al., ), implemented with the SEISM package (Pont-Tuset & Marques, ). The maximum allowed mis-localization for odsF is set to 0.0075 for PASCAL-Context and 0.011 for NYUD-v2. Monocular depth estimation is evaluated by the Root Mean Square Error (RMSE).

To assess overall performance, we calculate the MTL Gain by determining the average relative difference of multi-task method compared to the single-task baseline (Maninis et al., ). The formula is as follows:

$$ $\Delta_{m}=\frac{1}{T}\sum_{t=1}^{T}(-1)^{l_{t}}\frac{M_{t}-M_{\texttt{ST},t}} {M_{\texttt{ST},t}},$ (7) $$

where $T$ is the number of tasks, $M_{t}$ and $M_{\texttt{ST},t}$ represent the performance of task $t$ for the multi-task method and the single-task baseline, respectively. The value of $l_{t}=1$ if a lower value means better performance for task $t$, and $l_{t}=0$ otherwise.

### B.5 Loss Functions and Weights

Following previous practices in the field (Maninis et al., ; Kanakis et al., ; Ye & Xu, ), our method employs task-specific loss functions. For semantic segmentation and human parsing, we use the cross-entropy loss, while for saliency detection, we apply the balanced cross-entropy loss. Surface normal estimation and monocular depth estimation are optimized using the L1 loss. For edge detection, we use the weighted binary cross-entropy loss, with a weight of 0.95 to positive pixels and 0.05 to negative ones. To balance multiple task losses, we compute a weighted sum of the task-specific losses. The loss weights for Semseg, Parsing, Saliency, Normal, Boundary, and Depth tasks are set to 1, 2, 5, 10, 50, and 1, respectively.

### B.6 Training

As outlined in Section 3.4, the first stage trains the student model, which consists of the TAS and TSAP modules, with the distillation loss. Then in the second stage, we add the MoR Routers and prediction heads, training the entire student model with a combination of distillation loss and task-specific losses. The loss balancing factor $\gamma$ in Equation 6 is set as $1.0$ for simplicity.
Note that VFM teachers are always frozen in both stages. The detailed hyperparameter configurations are listed in Table 6, and the second stage follows the setup used in prior work (Ye & Xu, ). We implement our methodology with PyTorch (Paszke et al., ) and run experiments on NVIDIA H100 GPUs with 96G VRAM.

**Table 6: Training hyperparameters setup.**
| Hyperparameters | Stage 1 | Stage 2 |
| --- | --- | --- |
| #GPUs | 8 | 2 |
| Total batch size | 128 | 4 |
| Optimizer | AdamW | AdamW |
| Base learning rate | 1e-3 | 2e-5 |
| LR scheduler | cosine | poly |
| Weight decay | 1e-2 | 1e-6 |
| Steps | 30 epochs | 40,000 iterations |
| Warmup | 2 epochs | 0 |
| Gradient clipping | 0 | 10 |

## Appendix C Additional Experimental Results

Quantitative analysis of representation biases.
In Figure 1, we provide a quantitative analysis of the representation biases of VFMs, here we present the complete results in Table 7. The ‘ST Full’ and ‘MT Full’ results of ViT is used as the single-task and multi-task baseline in other comparisons, respectively.
We also include the multi-teacher VFM distillation methods, RADIO and Theia, along with our proposed SAK for comparison. In the ‘Freeze’ setting of SAK, we freeze the Teacher-Agnostic Stem and Teacher-Specific Adapter Paths, and tune the Mixture-of-Representations Routers and prediction heads during the second stage. The results indicate that SAK outperforms all three VFMs and two baselines, achieving superior overall performance and more balanced improvements across the five tasks.

**Table 7: Quantitative results of VFMs and multi-teacher VFM distillation baselines on PASCAL-Context. We consider three fine-tuning methods: ‘ST Full’ for fine-tuning the entire model in a single-task setting, ‘MT Full’ for fine-tuning the entire model in a multi-task setting, and ‘Freeze’ for freezing the backbone while training only the decoder heads, resulting in the same results for both single-task and multi-task settings.**
| Model | Fine-tune | #Train Param | Semseg | Parsing | Saliency | Normal | Boundary | ${\Delta_{m}\%\uparrow}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU$\uparrow$ | mIoU$\uparrow$ | maxF$\uparrow$ | mErr$\downarrow$ | odsF$\uparrow$ |  |  |  |  |
| ViT (Dosovitskiy et al., ) | ST Full | 462M | 80.25 | 70.54 | 84.54 | 13.57 | 74.22 | 0.00 |
| MT Full | 116M | 76.76 | 65.26 | 84.39 | 13.98 | 70.37 | -4.04 |  |
| Freeze | 30M | 71.36 | 56.08 | 81.06 | 17.17 | 64.03 | -15.19 |  |
| \hdashlineDINOv2 (Oquab et al., ) | ST Full | 462M | 80.77 | 68.62 | 85.30 | 13.24 | 78.55 | 1.42 |
| MT Full | 116M | 77.89 | 70.57 | 84.89 | 13.62 | 74.27 | -0.56 |  |
| Freeze | 30M | 81.18 | 74.38 | 80.48 | 16.85 | 70.83 | -5.39 |  |
| \hdashlineCLIP (Radford et al., ) | ST Full | 462M | 79.83 | 67.16 | 84.65 | 13.39 | 74.93 | -0.58 |
| MT Full | 116M | 76.84 | 65.87 | 84.61 | 13.91 | 70.50 | -3.66 |  |
| Freeze | 30M | 78.33 | 65.31 | 81.48 | 16.84 | 67.43 | -9.33 |  |
| \hdashlineSAM (Kirillov et al., ) | ST Full | 474M | 71.13 | 68.03 | 86.52 | 13.26 | 79.51 | -0.63 |
| MT Full | 118M | 66.39 | 65.65 | 85.38 | 13.74 | 77.20 | -4.09 |  |
| Freeze | 30M | 49.11 | 59.85 | 81.10 | 16.21 | 75.89 | -15.05 |  |
| \hdashlineRADIO (Ranzinger et al., ) | MT Full | 128M | 78.06 | 68.13 | 85.18 | 13.59 | 72.64 | -1.53 |
| Freeze | 30M | 81.42 | 71.71 | 82.43 | 16.21 | 72.91 | -4.12 |  |
| \hdashlineTheia (Shang et al., ) | MT Full | 115M | 76.51 | 67.53 | 84.38 | 14.56 | 70.34 | -4.33 |
| Freeze | 30M | 62.09 | 54.50 | 81.69 | 16.87 | 63.70 | -17.45 |  |
| \hdashlineSAK (Ours) | MT Full | 134M | 81.65 | 72.38 | 84.87 | 14.05 | 73.23 | -0.03 |
| Freeze | 36M | 80.61 | 71.15 | 83.34 | 15.22 | 71.94 | -3.07 |  |

Detailed results corresponding to Figure 3. We also provide the details of Figure 3 in Tables 8 and 9 with additional state of the arts in multi-task learning included. SAK is distilled from VFM teachers based on ViT-L backbones.
On PASCAL-Context, our approach outcomes a positive MTL Gain $\Delta_{m}$, surpassing the single-task baseline—a groundbreaking improvement not achieved by other models, even those with complex task interaction modules designed particularly for MTL. On both datasets, SAK not only outperforms RADIO and Theia but also surpasses the multi-task SOTAs with significant margins, all while utilizing a simple architecture and considerably fewer parameters.

**Table 8: Comparison with state of the arts on PASCAL-Context, based on ViT-B backbones. For models not providing exact number of parameters in the paper or source code, we estimate their parameter counts.**
| Model | Backbone | #Param | Semseg | Parsing | Saliency | Normal | Boundary | ${\Delta_{m}\%\uparrow}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU$\uparrow$ | mIoU$\uparrow$ | maxF$\uparrow$ | mErr$\downarrow$ | odsF$\uparrow$ |  |  |  |  |
| Single-task baseline | ViT-B | 462M | 80.25 | 70.54 | 84.54 | 13.57 | 74.22 | 0.00 |
| Multi-task baseline | ViT-B | 116M | 76.76 | 65.26 | 84.39 | 13.98 | 70.37 | -4.04 |
| \hdashlineInvPT (Ye & Xu, ) | ViT-B | 176M | 77.33 | 66.62 | 85.14 | 13.78 | 73.20 | -2.28 |
| InvPT++ (Ye & Xu, ) | ViT-B | $\sim$176M | 76.95 | 66.89 | 85.12 | 13.54 | 73.30 | -1.92 |
| TaskPrompter (Ye & Xu, ) | ViT-B | 418M | 79.00 | 67.00 | 85.05 | 13.47 | 73.50 | -1.24 |
| TaskExpert (Ye & Xu, ) | ViT-B | 347M | 78.45 | 67.38 | 84.96 | 13.55 | 72.30 | -1.73 |
| BFCI (Zhang et al., ) | ViT-B | 230M | 77.98 | 68.19 | 85.06 | 13.48 | 72.98 | -1.31 |
| MLoRE (Yang et al., ) | ViT-B | 259M | 79.26 | 67.82 | 85.31 | 13.65 | 74.69 | -0.83 |
| \hdashlineRADIO (Ranzinger et al., ) | ViT-B | 128M | 78.06 | 68.13 | 85.18 | 13.59 | 72.64 | -1.53 |
| Theia (Shang et al., ) | ViT-B | 115M | 76.51 | 67.53 | 84.38 | 14.56 | 70.34 | -4.33 |
| SAK (Ours) | ViT-B | 134M | 81.88 | 74.30 | 84.79 | 14.02 | 74.09 | 0.83 |

**Table 9: Comparison with state of the arts on NYUD-v2, based on ViT-B backbones.**
| Model | Backbone | #Param | Semseg | Depth | Normal | Boundary | ${\Delta_{m}\%\uparrow}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU$\uparrow$ | RMSE$\downarrow$ | mErr$\downarrow$ | odsF$\uparrow$ |  |  |  |  |
| Single-task baseline | ViT-B | 369M | 51.15 | 0.5792 | 19.77 | 77.35 | 0.00 |
| Multi-task baseline | ViT-B | 110M | 49.27 | 0.5823 | 19.92 | 75.88 | -1.72 |
| \hdashlineInvPT (Ye & Xu, ) | ViT-B | 161M | 50.30 | 0.5367 | 19.00 | 77.60 | 2.47 |
| InvPT++ (Ye & Xu, ) | ViT-B | $\sim$161M | 49.79 | 0.5318 | 18.90 | 77.10 | 2.40 |
| TaskPrompter (Ye & Xu, ) | ViT-B | 160M+ | 50.40 | 0.5402 | 18.91 | 77.60 | 2.49 |
| ECS (Shoouri et al., ) | ViT-B | 122M | 50.46 | 0.5332 | 18.42 | 77.89 | 3.53 |
| BFCI (Zhang et al., ) | ViT-B | 200M+ | 51.14 | 0.5186 | 18.92 | 77.98 | 3.89 |
| TSP (Wang et al., ) | ViT-B | 160M+ | 51.22 | 0.5301 | 18.78 | 76.90 | 3.26 |
| SEM (Huang et al., ) | ViT-B | - | 51.34 | 0.5222 | 18.95 | 77.60 | 3.67 |
| \hdashlineRADIO (Ranzinger et al., ) | ViT-B | 122M | 55.03 | 0.5186 | 18.49 | 77.97 | 6.33 |
| Theia (Shang et al., ) | ViT-B | 109M | 51.80 | 0.5367 | 19.70 | 76.08 | 1.83 |
| SAK (Ours) | ViT-B | 126M | 59.93 | 0.4942 | 17.60 | 78.60 | 11.11 |

Comparison with state of the arts using additional backbones. As depicted in Tables 10 and 11, we extend our comparison to state of the arts based on the ViT-S backbones. Additionally, we include models using the Swin-S backbones, though they are substantially larger in model size compared to ours.
Here SAK is distilled from VFM teachers based on ViT-B backbones. Our approach consistently delivers the best performance on both datasets, with particularly strong improvements in segmentation tasks. The slight underperformance in boundary detection and depth estimation can be attributed to the architectural differences between ViT and Swin Transformer (Liu et al., ).

**Table 10: Comparison with state of the arts on PASCAL-Context, based on ViT-S and Swin-S backbones. Note that ViT-S backbone has 22M parameters while Swin-S backbone has 50M parameters.**
| Model | Backbone | #Param | Semseg | Parsing | Saliency | Normal | Boundary | ${\Delta_{m}\%\uparrow}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU$\uparrow$ | mIoU$\uparrow$ | maxF$\uparrow$ | mErr$\downarrow$ | odsF$\uparrow$ |  |  |  |  |
| Single-task baseline | ViT-S | 117M | 77.59 | 66.58 | 84.60 | 14.16 | 70.95 | 0.00 |
| Multi-task baseline | ViT-S | 29M | 73.94 | 61.22 | 83.96 | 14.89 | 67.03 | -4.84 |
| \hdashlineTaskPrompter (Ye & Xu, ) | ViT-S | 132M | 76.57 | 63.22 | 84.54 | 13.93 | 70.60 | -1.06 |
| TaskExpert (Ye & Xu, ) | ViT-S | 55M | 75.04 | 62.68 | 84.68 | 14.22 | 68.80 | -2.50 |
| MLoRE (Yang et al., ) | ViT-S | 44M | 75.64 | 62.65 | 84.70 | 14.43 | 69.81 | -2.36 |
| MQTransformer (Xu et al., ) | Swin-S | 57M+ | 71.25 | 60.11 | 84.05 | 14.74 | 71.80 | -4.29 |
| DeMT (Xu et al., ) | Swin-S | 54M | 72.01 | 58.96 | 83.20 | 14.57 | 72.10 | -4.31 |
| DeMTG (Xu et al., ) | Swin-S | 55M+ | 71.54 | 61.49 | 83.70 | 14.90 | 72.20 | -3.99 |
| ATMPNet (Sirejiding et al., ) | Swin-S | 50M+ | 70.58 | 61.17 | 83.96 | 14.41 | 73.15 | -3.32 |
| TFUT (Xin et al., ) | Swin-S | 63M+ | 72.49 | 63.24 | 84.06 | 14.42 | 73.50 | -2.09 |
| \hdashlineTheia (Shang et al., ) | ViT-S | 29M | 70.82 | 62.71 | 83.95 | 14.87 | 67.52 | -5.03 |
| SAK (Ours) | ViT-S | 34M | 78.66 | 68.46 | 84.66 | 14.33 | 70.28 | 0.43 |

**Table 11: Comparison with state of the arts on NYUD-v2, based on ViT-S and Swin-S backbones.**
| Model | Backbone | #Param | Semseg | Depth | Normal | Boundary | ${\Delta_{m}\%\uparrow}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU$\uparrow$ | RMSE$\downarrow$ | mErr$\downarrow$ | odsF$\uparrow$ |  |  |  |  |
| Single-task baseline | ViT-S | 94M | 48.11 | 0.5911 | 20.27 | 75.17 | 0.00 |
| Multi-task baseline | ViT-S | 28M | 45.87 | 0.6626 | 20.55 | 74.44 | -4.78 |
| \hdashlineMQTransformer (Xu et al., ) | Swin-S | 57M | 49.18 | 0.5785 | 20.81 | 77.00 | 1.03 |
| DeMT (Xu et al., ) | Swin-S | 53M | 51.50 | 0.5474 | 20.02 | 78.10 | 4.89 |
| DeMTG (Xu et al., ) | Swin-S | 55M | 52.23 | 0.5599 | 20.05 | 78.40 | 4.81 |
| ATMPNet (Sirejiding et al., ) | Swin-S | 50M+ | 51.82 | 0.5526 | 20.11 | 78.27 | 4.78 |
| TFUT (Xin et al., ) | Swin-S | 63M | 50.04 | 0.5419 | 20.08 | 78.30 | 3.99 |
| \hdashlineTheia (Shang et al., ) | ViT-S | 28M | 46.54 | 0.6047 | 20.95 | 74.59 | -2.42 |
| SAK (Ours) | ViT-S | 32M | 54.30 | 0.5785 | 19.67 | 76.58 | 4.96 |

Detailed results corresponding to Table 3.
In Table 3, we use a higher similarity between the student and teachers to validate the effectiveness of our SAK in preserving representation biases. The detailed breakdown of these similarity calculations for each VFM teacher is listed in Table 12. The substantially higher similarity scores further confirm our claim.

**Table 12: Cosine similarity between the representations of student and corresponding teachers on the ImageNet-1k validation set.**
| TSAP | DINOv2 | CLIP | SAM | Avg |
| --- | --- | --- | --- | --- |
| ✘ | 0.3566 | 0.5159 | 0.1306 | 0.3344 |
| ✔ | 0.8615 | 0.9138 | 0.8369 | 0.8707 |

Detailed results corresponding to Figure 5. In Figure 5, we investigate the impact of downstream data size by experimenting with different percentages of samples from the PASCAL-Context dataset. From the detailed results in Table 13, we can confidently conclude that SAK is a more robust method compared to the multi-task baseline and multi-teacher VFM distillation methods. Besides the knowledge transferred from VFM teachers through the large-scale ImageNet dataset during the first stage, the more specialized knowledge distilled in the second stage further benefits SAK’s robustness and generalization to diverse data sizes.

**Table 13: Performance w.r.t. different percentages of downstream dataset. MTL Gain is computed w.r.t. single-task baseline trained on full dataset.**
| Data Percentage | Model | Semseg | Parsing | Saliency | Normal | Boundary | ${\Delta_{m}\%\uparrow}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU$\uparrow$ | mIoU$\uparrow$ | maxF$\uparrow$ | mErr$\downarrow$ | odsF$\uparrow$ |  |  |  |
| 25% | Multi-task baseline | 72.10 | 59.71 | 81.40 | 15.63 | 68.87 | -10.32 |
| RADIO (Ranzinger et al., ) | 73.47 | 63.18 | 81.94 | 15.13 | 71.49 | -7.43 |  |
| Theia (Shang et al., ) | 71.12 | 63.65 | 81.73 | 15.57 | 69.18 | -9.20 |  |
| SAK (Ours) | 80.09 | 69.77 | 81.47 | 15.28 | 72.32 | -4.02 |  |
| \hdashline50% | Multi-task baseline | 74.95 | 60.65 | 83.04 | 14.72 | 69.59 | -7.42 |
| RADIO (Ranzinger et al., ) | 76.23 | 63.90 | 83.67 | 14.27 | 72.07 | -4.70 |  |
| Theia (Shang et al., ) | 74.60 | 64.01 | 83.19 | 14.89 | 69.84 | -6.70 |  |
| SAK (Ours) | 81.33 | 70.19 | 83.35 | 14.54 | 72.71 | -1.95 |  |
| \hdashline75% | Multi-task baseline | 75.17 | 61.23 | 84.08 | 14.15 | 70.04 | -6.00 |
| RADIO (Ranzinger et al., ) | 76.82 | 64.26 | 84.59 | 13.76 | 72.29 | -3.42 |  |
| Theia (Shang et al., ) | 74.98 | 64.57 | 83.83 | 14.64 | 70.04 | -5.88 |  |
| SAK (Ours) | 81.32 | 70.51 | 84.33 | 14.28 | 73.01 | -1.16 |  |
| \hdashline100% | Multi-task baseline | 76.76 | 65.26 | 84.39 | 13.98 | 70.37 | -4.04 |
| RADIO (Ranzinger et al., ) | 78.06 | 68.13 | 85.18 | 13.59 | 72.64 | -1.53 |  |
| Theia (Shang et al., ) | 76.51 | 67.53 | 84.38 | 14.56 | 70.34 | -4.33 |  |
| SAK (Ours) | 81.65 | 72.38 | 84.87 | 14.05 | 73.23 | -0.03 |  |

Balancing distillation loss and task losses. In the second training stage, we apply a combination of distillation loss and task-specific losses, weighted by the balancing factor $\gamma$ introduced in Equation 6. We explore the impact of this factor on the final prediction results in Table 14, using teachers and student based on ViT-B backbones. As the balancing factor increases, We observe raising accuracy in segmentation and boundary detection tasks, while performance in saliency detection and surface normal estimation diminishes. This suggests that segmentation and boundary detection rely heavily on knowledge distilled from the VFM teachers, whereas the other two tasks depend more on domain-specific details provided by downstream supervision, resulting in a seesaw phenomenon. Though a factor of $2.0$ yields the best overall performance, we use a default value of $1.0$ for simplicity.

**Table 14: Performance w.r.t. different values of balancing factor $\gamma$.**
| $\gamma$ | Semseg | Parsing | Saliency | Normal | Boundary | ${\Delta_{m}\%\uparrow}$ |
| --- | --- | --- | --- | --- | --- | --- |
| mIoU$\uparrow$ | mIoU$\uparrow$ | maxF$\uparrow$ | mErr$\downarrow$ | odsF$\uparrow$ |  |  |
| 0.5 | 81.19 | 71.98 | 84.95 | 14.01 | 73.06 | -0.22 |
| 0.75 | 81.46 | 72.22 | 84.91 | 14.03 | 73.17 | -0.10 |
| 1.0 | 81.65 | 72.38 | 84.87 | 14.05 | 73.23 | -0.03 |
| 1.25 | 81.76 | 72.51 | 84.84 | 14.06 | 73.29 | 0.03 |
| 1.5 | 81.84 | 72.62 | 84.81 | 14.08 | 73.33 | 0.06 |
| 2.0 | 81.92 | 72.78 | 84.74 | 14.11 | 73.37 | 0.07 |
| 2.5 | 81.97 | 72.88 | 84.68 | 14.14 | 73.39 | 0.06 |
| 3.0 | 81.99 | 72.95 | 84.62 | 14.17 | 73.39 | 0.03 |
| 4.0 | 82.01 | 73.01 | 84.53 | 14.22 | 73.38 | -0.05 |

Gating weights learned by MoR Routers.
We present the complete version of Figure 4 in Figure 7, along with the results from NYUD-v2 in Figure 8. A similar observation across both datasets is that all tasks tend to depend on the shared TAS representations at lower levels. Conversely, at higher levels, the gating weights for each task diverge, showing varied dependencies on different experts. This demonstrate that the biases from different VFM teachers are adaptively leveraged by the tasks, allowing the model to optimize performance through selective representation fusion.

Figure: Figure 7: Gating weights of different experts learned by MoR Routers on PASCAL-Context.
Refer to caption: x8.png

Figure: Figure 8: Gating weights of different experts learned by MoR Routers on NYUD-v2.
Refer to caption: x9.png

## Appendix D Additional Related Work

### D.1 Knowledge Distillation

Knowledge Distillation (KD) is a model compression technique where a smaller student model learns from a larger teacher model, typically using the teacher’s output logits as soft targets to guide training (Buciluǎ et al., ; Hinton et al., ; Ba & Caruana, ; Beyer et al., ). To improve knowledge transfer, some studies have proposed using intermediate representations (Romero et al., ; Ahn et al., ; Heo et al., ; Zagoruyko & Komodakis, ).
Multi-teacher knowledge distillation has also been explored, where knowledge is transferred from multiple teacher models simultaneously or progressively to improve the student’s generalization (Sau & Balasubramanian, ; You et al., ; Fukuda et al., ; Wen et al., ; Cao et al., ). Another branch of work, known as knowledge amalgamation, combines knowledge from multiple teachers tackling different tasks into a student that learns the union of all tasks (Shen et al., ; ; Luo et al., ; Ye et al., ; Vongkulbhisal et al., ; Thadajarassiri et al., ; ). Such methods mainly unify classification tasks from multiple domains. In contrast, our approach addresses multiple heterogeneous vision tasks (*e.g*., semantic segmentation, surface normal estimation) simultaneously.

### D.2 Vision Foundation Models

Vision Foundation Models (VFMs) are large-scale, general-purpose vision models trained on massive datasets, demonstrating exceptional performance across a wide range of downstream tasks, particularly when generalizing to unseen tasks and domains. Some VFMs are trained with specific tasks, such as ViT (Dosovitskiy et al., ), DeiT (Touvron et al., ), and Swin Transformer (Liu et al., ) for image classification, Segment Anything Model (SAM) (Kirillov et al., ; Ravi et al., ) for promptable segmentation, and Depth Anything (Yang et al., ; ) for monocular depth estimation. Others are trained with pretext tasks agnostic to downstream tasks. For instance, CLIP (Radford et al., ) and its derivatives (Zhai et al., ) align features of image-text pairs, DINOv2 (Oquab et al., ) employs self-supervised learning, and Pang et al. () even uses Large Language Model (LLM) as a visual encoder. These models have shown strong capabilities in diverse visual tasks, including low-shot classification, open-vocabulary recognition, semantic segmentation, and visual question answering (VQA).

Previous studies have reported similar conclusions as our studied representation biases when evaluating VFMs on different tasks (Ranzinger et al., ; Zong et al., ; Kar et al., ; Tong et al., ; ).
Therefore, several existing works have sought to leverage multiple VFMs to enhance downstream performance. Typically, they extract image features from multiple VFMs and then concatenate or fuse these features (Lin et al., ; Kar et al., ; Zong et al., ; Tong et al., ; ; Man et al., ; Shlapentokh-Rothman et al., ). While this approach can improve visual encoding capability, it comes with a major drawback: running inference across multiple vision encoders drastically increases computational costs, as well as the memory and storage requirements due to large-scale parameters. Additionally, Zong et al. () highlights that biased information from VFMs can lead to performance degradation when using a simple fusion method.

### D.3 Knowledge Distillation of Vision Foundation Models

Due to their large sizes and extensive training data requirements, vision foundation models are typically computationally intensive to train from scratch. Therefore, distilling VFMs into smaller models with knowledge distillation techniques has become a popular topic (Liu et al., ; Vemulapalli et al., ; Sun et al., ; Yang et al., ; Zhang et al., ). For example, several works (Zhang et al., ; Zhou et al., ; Zhang et al., ; Wang et al., ; Songa et al., ) have derived lightweight versions of SAM. More recently, research has explored distilling multiple VFMs into a single student model (Ranzinger et al., ; Shang et al., ; Sariyildiz et al., ), which is closely related to our approach. Besides the differences already discussed in the main paper, we further provide a detailed comparison in Table 15. RADIO uses strong teachers with substantially larger scales and capacities, for instance, DINOv2-g has 1,100M parameters. Theia also uses a ViT-H teacher with 632M parameters, double the size of ViT-L. Moreover, RADIO is trained on a dataset over 1,000 times larger than ImageNet-1k, even with larger input resolutions and a greater number of total samples seen. Despite the relatively lower training cost, our proposed SAK demonstrates superior outcomes on downstream tasks, exceeding both baselines by notable margins.

**Table 15: Comparison with multi-teacher VFM distillation methods w.r.t. training paradigms and the downstream MTL Gain $\Delta_{m}$ on the PASCAL-Context and NYUD-v2 dataset, respectively. All student models use ViT-B backbones and the same decoder heads.**
| Model | RADIO (Ranzinger et al., ) | Theia (Shang et al., ) | SAK (Ours) |
| --- | --- | --- | --- |
| Teachers | DFN CLIP-H, SigLIP-SO400M, | ViT-H, DINOv2-L, CLIP-L | DINOv2-L, OpenCLIP-L, SAM-L |
| DINOv2-g, SAM-H |  |  |  |
| Dataset | DataComp-1B | ImageNet-1k | ImageNet-1k |
| Dataset Size | 1.4B | 1.2M | 1.2M |
| Resolution | 768 | 224 | 384 |
| #Samples | 614M | 60M | 36M |
| $\Delta_{m}\%$ | -1.53, 6.33 | -4.33, 1.83 | 0.83, 11.11 |

### D.4 Multi-Task Learning

Multi-Task Learning (MTL) aims to train a single model to learn multiple tasks simultaneously (Caruana, ; Ruder, ; Crawshaw, ; Zhang & Yang, ; Yu et al., ). MTL research primarily falls into two categories: multi-task optimization (Kendall et al., ; Liu et al., ; Ye et al., ; Chen et al., ; Yu et al., ; Wang et al., ; Liu et al., ; Javaloy & Valera, ; Chen et al., ; Sener & Koltun, ; Guo et al., ; Lu et al., ) and model architecture design (Ruder et al., ; Long et al., ; Huang et al., ; Wallingford et al., ; Mallya et al., ; Huang et al., ; Rosenbaum et al., ; Meyerson & Miikkulainen, ; Yang & Hospedales, ; Bragman et al., ).

In multi-task dense prediction, most works focus on designing architecture, which can be further divided into encoder-focused and decoder-focused methods (Vandenhende et al., ). Encoder-focused methods develop shared encoders to extract features for different tasks, employing techniques like feature fusion (Misra et al., ; Ruder et al., ; Gao et al., ), attention (Liu et al., ), branched networks (Brüggemann et al., ; Guo et al., ; Vandenhende et al., ; Lu et al., ; Raychaudhuri et al., ), and mixture-of-experts (Liang et al., ; Chen et al., ; Ye & Xu, ). Decoder-focused methods, on the other hand, design complicated decoders to extract task-specific features by modeling task interactions (Xu et al., ; Zhang et al., ; Zhou et al., ; Zhang et al., ; Brüggemann et al., ; Vandenhende et al., ; Ye & Xu, ; Xu et al., ; ; Ye & Xu, ; Zhang et al., ; Xu et al., ; Sirejiding et al., ; Xu et al., ; Sirejiding et al., ; Xin et al., ; Sirejiding et al., ; Lin et al., ; Huang et al., ; Zhang et al., ; Wang et al., ). Additional topics include task-conditional models (Maninis et al., ; Kanakis et al., ; Sun et al., ; Lu et al., ; ), efficient adaptation (Liu et al., ; Xin et al., ; Agiza et al., ; Wang et al., ), efficient computation (Neseem et al., ; Shoouri et al., ; Aich et al., ), regularization (Yang et al., ; Li et al., ), and generative modeling (Bao et al., ; Qiu et al., ).

## Appendix E Additional Qualitative Results

To provide an intuitive comparison between the proposed SAK, VFM teachers, and existing methods, we visualize the task predictions of frozen VFM teachers, RADIO, Theia, and our model with examples from PASCAL-Context and NYUD-v2, as shown in Figures 9, 10 (the same image used in Figure 1), and 11. All methods use ViT-B backbones, with RADIO and Theia fully fine-tuned on the downstream dataset to ensure a fair comparison. Our model noticeably generates better details and fewer errors, especially in semantic segmentation, saliency estimation, and depth estimation.

Figure: (a)
Refer to caption: extracted/5937004/figure/vis_bias8/2008_001946.jpg

Figure: (a)
Refer to caption: extracted/5937004/figure/vis_bias3/2009_004263.jpg

Figure: (a)
Refer to caption: extracted/5937004/figure/pred3/0030.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: please see the weaknesses section.
soundness: 2
strengths: 1. This paper contributes a multi-task learning framework that leverages the strengths and specific biases of multiple Vision Foundation Models (VFMs), presenting an alternative to traditional knowledge distillation approaches. By preserving individual model biases, the method offers a good solution to challenges in multi-task learning, where distinct vision tasks often benefit from different aspects of visual representation.

2. Extensive experimental results are provided, showing substantial performance gains across established benchmarks. 

3. This paper is well organized and well written.
summary: This paper introduces a "Swiss Army Knife" (SAK) model that preserves each model's specific strengths through a framework consisting of a shared Teacher-Agnostic Stem and Teacher-Specific Adapter Paths. The SAK dynamically combines representations via Mixture-of-Representations Routers, allowing for tailored outputs for each task. Extensive experiments on multi-task benchmarks show that SAK outperforms several current methods.
weaknesses: 1. The proposed framework, while innovative, introduces a high level of algorithmic complexity by requiring multiple Vision Foundation Models (VFMs) and integrating Teacher-Specific Adapter Paths and Mixture-of-Representations Routers. Given the inherent complexity of multi-task learning, this layered structure may lead to excessive computational overhead without clear evidence of structural necessity. The paper could improve by providing a more rigorous theoretical justification for this architecture, perhaps through ablation studies that explore simpler configurations to assess if comparable results could be achieved with fewer components.

2. The impressive experimental results may be partly due to extensive parameter tuning, but the paper lacks detailed discussions or tests that could isolate the contributions of the model’s design from effects arising purely from tuning. This raises questions about whether the performance gains reflect true architectural benefits or if they could be replicated by tuning existing simpler models. Including experiments with fixed hyperparameters across different tasks or using cross-validation techniques to verify robustness would strengthen confidence in the model’s structural contributions.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Compared with the base model selected in this paper, RADIO also selected SigLIP and Theia selected ViT-H. It is necessary to provide the principles of base model selection and experiments on the sensitivity of the model to teacher selection.

2. Can SAK be easily adjusted to add new Vision Foundation Models teachers or changes that meet the needs of specific downstream tasks?
soundness: 3
strengths: 1. By distilling multiple Vision Foundation Models (VFMs) into a stem and multiple parameter-efficient branches, the computational cost is greatly reduced

2. By retaining the unique representation bias of each teacher model through the teacher-specific adapter path module, SAK can extract knowledge from multiple VFMs in a task-adaptive manner, thereby effectively improving multi-task performance

3. Comparative experiments and sufficient ablation experiments on PASCAL-Context and NYUD-v2 datasets demonstrate the effectiveness of the proposed method
summary: This paper proposes a new method to achieve multi-task learning by coordinating the advantages of multiple Vision Foundation Models (VFMs). In order to solve the problem that the previous distillation method only uses a single student model that cannot preserve the representation bias between different VFMs, a teacher-agnostic stem and a teacher-specific adapter path modules are proposed to parameter-efficiently preserve the representation bias of different VFMs, and then the fusion coefficients of different branches are learned for different tasks through a mixture-of-representations router. Experiments on the PASCAL-Context and NYUD-v2 datasets have demonstrated the effectiveness of the method.
weaknesses: 1. Lack of experimental results on Image level reasoning and Large Vision-Language Model that are consistent with the comparison method RADIO

2. Lack of upper bounds on results before distillation, i.e., results using encoders of three Vision Foundation Models without distillation and routers

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: One thing the reviewer tries to seek clarification is how is mixture-of-representation being done when student and teacher has different architecture. In this case, the feature dimension of student and teacher is different.
soundness: 2
strengths: (1) This paper systematically study the representation bias in different visual foundation models (VFMs), and identify characteristics of VFMs.
(2) The proposed approach achieve state-of-the-art performance on multi-task learning on various vision tasks.
(3) Thorough comparisons with baseline methods are provided to validate the effectiveness of the method. In addition, visualizations on choices of experts are given to provide insights into the method.
summary: This paper introduces a novel method to consolidate different visual foundation models into one model via a mixture-of-expert fashion method. Through their approach, their method Swiss Army Knife (SAK) achieve state-of-the-art performance across various vision tasks.
weaknesses: (1) The proposed mixture-of-representation router involves conducting weighted sum over both student's and teacher's features, which would increase the inference cost during the process. This is because all teacher models will have to conduct forward propagation to obtain the output representations.
(2) More vision tasks such has instance-level segmentation/detection, depth estimation, could be evaluated.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness above. The reviewer is interested in understanding how well can SAK embed knowledge from multiple vision foundation models and comparisons at the tasks that the vision foundation models are good at will be more convincing.
soundness: 2
strengths: 1. Experiments are extensive. The authors have validated their approach on two public datasets across multiple tasks and provide extensive ablation and analysis to demonstrate the effectiveness of the designs.
2. Inference costs is small. It is plausible that the authors try to embed the multiple vision foundation models into a single model which reduces the inference costs of running multiple models significantly.
3. Writing is clear. Overall, the paper is well-organized and the methodology is easy to follow.
summary: This work studies the problem of Multi-TaskLearning(MTL) under the context of VisionFoundationModels(VFMs). Specifically, the authors propose the solution termed as “SwissArmyKnife”(SAK), which adaptively distills knowledge from a committee of VFMs to enhance multi-task learning. Experiments on two public benchmarks demonstrates the ability of SAK to synergies the complementary strengths of multiple VFMs.
weaknesses: 1. Missing comparison with Foundation Models at other tasks. While the authors have compared their method with prior arts and vision foundation models (teachers) on two specific datasets, the reviewer is more interested in how is the performance of the delivered model at the tasks that the vision foundation models are good at? For example, after stage 1 training, how is performance of SAK compared to SAM at semantic segmentation, how is performance of SAK compared to DINOv2 at depth estimation, fine-grained classification, etc. With these study, we could have a more clear picture how well is SAK trained with the help of other vision foundation models.
2. Not always outperforming the best foundation models in Fig. 1. The reviewer noticed from Fig. 1 (left) that SAK cannot outperform the best foundation models at each task (which is sort of expected as they are the teachers), could the authors explain why SAK could outperform the teacher? and why it cannot outperform the teacher at every task?
3. Training costs. Although SAK has reduced the inference costs dramatically, it is expected that it may lead to more training costs as it will iteratively get the prediction from every teacher. The reviewer wonders how is the training and memory costs of SAK compared to baseline methods.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weaknesses section.
soundness: 3
strengths: 1. The paper makes a novel attempt to analyze the failures of previous distillation methods. The authors examine the representation biases of VFMs quantitatively and qualitatively to demonstrate that the biases or diversified visual representations have a unique contribution to each downstream task.

2. The proposed adapter path and MoR routers are lightweight and easy to implement, which makes it easy to follow.

3. The experiment results are significant and solid, where the abundant analytical results further prove the effectiveness of the SAK method.
summary: The paper proposes SAK to effectively merge different VFMs for downstream vision tasks, mitigating the issue of the divergence of different biases among the different vision representations generated by each model. SAK innovatively proposes a new paradigm for transferring knowledge with the proposed lightweight Teacher-Specific Adapter Path modules instead of using a standalone backbone in a many-to-one distillation manner. Extensive experiments on two commonly-used benchmarks show the effectiveness of SAK.
weaknesses: 1. The training difficulty of the router should be discussed since the gating functions of MoE are always hard to optimize for a balanced routing.

2. The authors are encouraged to provide some results on multimodal LLMs (like LLaVA, Sphinx) to further verify the effectiveness of the proposed methods.

Meta-review: This paper proposes a multi-task learning method to distill knowledge from a committee of Vision Foundation Models (VFMs). Unlike Unlike existing methods that use a single backbone for knowledge transfer, this paper preserves the unique representation bias of each teacher by collaborating the lightweight Teacher-Specific Adapter Path modules with the TeacherAgnostic Stem. It can effectively leverage the information from VFMs through a mixture-of-representations router. Experiments on multiple datasets have demonstrated the effectiveness of the method.

All reviewers are unanimously positive on this submission, and like the contributions including 1) the idea is somehow interesting as a multi-task learning framework to learn the knowledge from VFMs while preserving model biases; 2) the method is parameter and inference efficient; 3) the experiments are extensive and support the proposed claims; 4) the paper is well written. The common concerns from the reviewers are 1) architecture/complexity overhead; 2) missing experiments on other CV/VLM tasks; 3) missing a few more ablations/discussion and details. However, most of the concerns were well addressed after the rebuttal. The AC agrees with the reviewers' unanimous decision.

Decision: Accept (Poster)