Simplifying Knowledge Transfer in Pretrained Models
Authors: Siddharth Jain
Shyamgopal Karthik
Vineet Gandhi
Venue: Accepted by TMLR ()

Abstract: Pretrained models are ubiquitous in the current deep learning landscape, offering strong results on a broad range of tasks. Recent works have shown that models differing in various design choices exhibit categorically diverse generalization behavior, resulting in one model grasping distinct data-specific insights unavailable to the other. In this paper, we propose to leverage large publicly available model repositories as an auxiliary source of model improvements. We introduce a data partitioning strategy where pretrained models autonomously adopt either the role of a student, seeking knowledge, or that of a teacher, imparting knowledge. Experiments across various tasks demonstrate the effectiveness of our proposed approach. In image classification, we improved the performance of ViT-B by approximately 1.4\% through bidirectional knowledge transfer with ViT-T. For semantic segmentation, our method boosted all evaluation metrics by enabling knowledge transfer both within and across backbone architectures. In video saliency prediction, our approach achieved a new state-of-the-art. We further extend our approach to knowledge transfer between multiple models, leading to considerable performance improvements for all model participants.

Published in Transactions on Machine Learning Research (09/2025)
Simplifying Knowledge Transfer in Pretrained Models
Siddharth Jain
siddharth..jo@resserc...ii
Center for Visual Information Technology International Institute of Information Technology, Hyderabad
Shyamgopal Karthik
shyamgopal www.thiiiiiiiiiiteeennnnn......
University of Tübingen
Vineet Gandhi
ugandhi@sitt.om
Center for Visual Information Technology
International Institute of Information Technology, Hyderabad
Reviewed OpenReview: https: https://
https://openrevieu openreview. net/ permformm? id= eQ9AVEDaP3
Abstract
Pretrained models are ubiquitous in the current deep learning landscape, offering strong results broad range of tasks. Recent works have shown that models differing in var- ions design choices exhibit categorically diverse generalization behavior, resulting in one model grasping distinet data-specific insights unavailable to the other. In this paper, we propose to leverage large publicly available model repositories as an auxiliary source of model improvements. We introduce data partitioning strategy where pretrained models autonomously adopt either the role of a student, secking knowledge, or that of teacher, imparting knowledge. Experiments across various tasks demonstrate the effectiveness of our proposed approach. In image classification, we improved the performance of ViT-B by approximately 1.4% through bidirectional knowledge transfer with VTT-T. For semantic segmentation, our method boosted all evaluation metrics by enabling knowledge transfer both within and across backbone architectures. In video saliency prediction, our approach achieved new state-of-the-arl. We further extend our approach to kuowledge transfer between multiple models, leading to considerable performance improvements for all model participants. The code is available at: httpps://iihhub.com/ Bltbb/oooo/yydd/BBBBBBBB Bi-KKD
1 Introduction
Knowledge Distillation (KD) (Bucilu et al. 2006 Hinton et al. 2015 Beyer et al. 2022 intends to transfer knowledge from large 'teacher' model to smaller 'student" model. Traditional KD methods utilize the predictions from the pretrained teacher model to supervise the training of the student model, encouraging it to generalize better than were trained from scratch alone. However, vanilla KD is a two-stage process that begins with training a teacher model and then freezing it distill knowledge into the student model, mcaning that the knowledge can only be transferred from the teacher to the student. Online Knowledge Distillation methods (Zhang et al., 2018 Guo et al. (0200) overcome this limitation by adopting a one-stage training process, jointly training set of student models that learn from each other in peer-teaching manner.
Although online KD methods employ a single stage training process, they distill knowledge into untrained student models. Transferring knowledge in this way neglects the existence of complementary knowledge between pretrained models, a factor that, if considered, can enhance generalization (Gontijo-Lopes et al. 2022 Roth al., (2024). Various design choices such as hyperparameters, model architecture, optimization strategies, and pretraining dataset shape the semantic knowledge model acquires (Bouthillier et al. 2021

---

Published in Transactions on Machine Learning Research (09/202))
Frozen +
Trainable t
Forward
Backward
Teacher
Teacher
Model
Student
Student: @
Model
Teacher
(8) KD
KL+DP
(c)
Figure 1: The figure illustrates the training process for given batch of samples. (a) KD (Hinton et al. 2015) utilizes a pretrained Leacher model and trains a student, model to mimic the teachers predictions on each sample. (b) KL+DP [Roth et al. (024) employs a frozen teacher and the original frozen student (called ttachet-t to jointly guide the training of the student model. They partition the datasct into samples where learning from the teacher is desired and ones where knowledge from the original frozen student should be retained. (c) In Bi-KD (ours), both models are trainable and learn from each other every sample, resulting in bidirectional knowledge transfer. Different from KD and KL+DP, we are able to improve both models simultancously.
Wagner al. 2022 Gontijo-Lopes al. (2022) show that model pairs with diverging training methodologies produce increasingly uncorrelated errors. Even low-accuracy models may caplure some data-specifie insights that high-accuracy models might overlook. |Roth et al. (2024) capitalize this finding by transferring complementary knowledge between any pretrained teacher and student model pair. They propose data partitioning strategy that divides the dataset into instances where knowledge transfer from a teacher is beneficial and those where retaining the student's behavior is preferred. However, their approach operates in unidirectional manner, wherein the teacher remains fixed after pretraining and the student is trained minimize the teacher-student gap. This contrasts with the real-world teacher-student dynamic, in which teacher continuously improves their knowledge and teaching skills through ongoing interactions with the student (Cornelius- White 2007 Wright 2011)
Transferring knowledge among group of pretrained models lets each one benefit from the unique strengths and insights that the others have already developed. They can complement each other's weaknesses and build more robust, generalized representations. When models are trained together, they iteratively update their predictions, effectively "teaching" each other. This simultaneous updating can lead to a performance boost that none of the models might achieve if trained independently (Zhang al. 2018 Zhu et al. 2018 Guo et al. 2020).
To this end, we propose simple method for bidirectional knowledge transfer between pretrained models, challenging task due to their prior training, Building on the data partition strategy introduced by Roth cct al. (2024), we dynamically assign the teacher role to the model with the highest prediction confidence for the ground-truth class, allowing it to transfer knowledge to the other model. Unlike their fixed partitioning approach, our confidence-basee data partitioning evolves as the models improve, adapting throughout the training process (as illustrated in Figure D]. Despite the apparent simplicity of our method, it leads to consistent performance improvements for both the models within single training stage.
We validated our approach through extensive experiments on models with diverse architectures, performance levels, sizes, and training objectives. Our method was tested across multiple tasks, including image classi- fication, semantic segmentation, and video saliency prediction, and was further extended to enable parallel knowledge transfer among multiple models. In all cases, we observed consistent performance improvements.
Overall, we make the following contributions:
We demonstrate the ability for bidirectional knowledge transfer in pretrained models. Specifically, show that knowledge can be transferred across both models simultaneously.
2

---

Published in Transactions on Machine Learning Research (09/2025)
We provide experiments across ImageNet classification, semantic segmentation, and video saliency prediction, where we observe consistent improvements for all participating models. In particular, our method sets a new state-of-the-art in video saliency prediction.
We establish that our framework seamlessly exlends to concurrent knowledge transfer across multiple models, thereby progressively enhancing the performance of each individual model as additional models are integrated.
2 Related Work
Knowledge Distillation (KD), pioncered by Buciluă al. (2006), aimed to compress large teacher models into smaller student models by aligning their soft target distributions. Hinton et al. (2015) refined this approach by incorporating temperature scaling to minimize the difference between the softened class proba- bilities of the teacher and student models. [Beyer et al. (2022) further highlighted the importance of consistent image augmentations and extended training schedules for effective KD.
Building on these ideas, recent works have explored transferring knowledge beyond just output probabilities. Romero et al. (0144) proposed aligning the intermediate feature representations between the teacher and the student models, while Zagoruyko & Komodakis (2017) train student to imitate the attention maps of teacher networks. Park et al. (2019) introduced a method that preserves the structural relationship between the outputs using distance-wis and anglo-wisc losses. However, these methods often require careful layer selection and loss balancing (Yun el al., 2019), making them highly dependent on specific network architectures. To address this, Srinivas & Fleuret (2018) proposed matching the Jacobian of network outputs, which has dimension independent of the model's is architecture.
The work most similar to ours, proposed by Roth al. (2024), introduced data partitioning strategy for knowledge transfer among pretrained models. However, their approach uses fixed data partitioning strategy during training, and the teacher model is not optimized specifically to guide the student. In contrast, our work introduces an evolving data partitioning strategy, which enables continuous improvements in the models through repeated interactions.
Online Knowledge Distillation treats every network as a student and trains them simultaneously from scratch. DML (Zhang et al.) 2018) enables peer student models to learn from each other's predictions using combination of cross-entropy and distillation losses. Anil et al. (2018) extend this concept to large-scale distributed neural networks, accelerating training by updating models concurrently.
Other approaches (Song & Chai 2018 Zhu et al.] 2018) involve designing multiple branch classifiers that are trained together. However, these methods are inflexible as they force networks to share lower layers, restricting knowledge transfer to only the upper layers within a single model. Chen et al. (2020) incorporate a self-atlention mechanism to assess the similarity between network groups, enhancing peer diversity to create a more effective leader. Similarly, KDCL (Guo al. 2020) introduces an ensemble of logits, where the optimal weight distribution is determined using Lagrange multiplier to minimize generalization crror.
More recently, Wu & Gong (202]] introduce extra temporal mean network for each peer, assigning it the teacher role. Li & Jin (2022) propose a proxy teacher that updates its weights based on predictions from the original teacher model, enabling bidirectional distillation with the student model. While these mot optimize the teacher model for distillation, they transfer knowledge between untrained models, disregarding the presence of complementary knowledge between pretrained models. Livanos et al. (2024) address this by transferring knowledge between trained models. They dynamically assign the teacher role to model that correctly classifies an instance while others fail. The teacher then generates a counterfactual instance for each correct prediction, adding it to the training set of incorrect models, which are subsequently retrained. However, this approach requires multiple training stages, making it computationally inefficient.
In contrast, our proposed method improves every pretrained model involved in KD within a single training stage, leading to a more efficient and effective learning process.
Multi-teacher Knowledge Distillation. KD can naturally be extended to learning from multiple pre- trained teachers. Fukuda et al. (2077) combine the distillation framework with data augmentation strategy
3

---

Published in Transactions on Machine Learning Research (09/2025)
Model
Forward
Model
Backward
KD
Model n
Generalization of our proposed method for transferring knowledge between multiple pretrained models. For each sample, we select a teacher model that guides the training of all other models. The teacher is selected based on the highost prediction probability, or the lowest loss, corresponding to the ground-truth. This data partitioning stralegy enables every model participating in the knowledge transfer to learn from each others strengths and address their weaknesses, resulting in all models improving within single training stage.
Figure 2:
by creating multiple copies of the data with the corresponding soft output targets from multiple teachers. You et al. (2017) further extend this approach by incorporating multiple teacher networks in the intermediate layers, considering the dissimilarity between intermediate representations of different examples. Luo et al.
(2019) propose a common feature learning scheme, in which the features of all teachers are transformed into a common space, and the student is required to imitate them all to amalgamate the intact knowledge. Instead of treating all teacher models equally, some works (Liu et al. 2020; Yuan al. 2021 dynamically assign weights to teacher models for different training instances and optimize the performance of the student model.
Although these works utilize predictions from multiple teacher models to reduce variance in network outputs, they fail to optimize teacher networks by considering the complementary knowledge between them. Our proposed method allows every model to benefit from each other's sengths and complement their weaknesses. This results in consistent performance improvements for models in a single training stage, which ultimately leads more robust predictions.
3 Bidirectional Knowledge Transfer
In this section, we first explain the traditional KD approach in Section 3.1 then introduce our proposed method for bidirectional knowledge transfer between two pretrained models in Section 3.2 Section 3.3 explains the extension of our method dense tasks, namely semantic segmentation, and vidco saliency prediction, and finally, Section 3.4 highlights approach for multidirectional knowledge transfer among multiple pretrained models.
3.1 Preliminaries
KD aims to improve the performance of the student network by leveraging the predictions of a teacher network as supervision. Hinton et al.] (2015) propose minimizing the Kullback-Leibler (KL) divergence between the soft targets of the teacher and student models. The distillation loss is formulated as:
T N
CKL12
KL KLlolzz../TT
(1)
where T represents the temperature parameter, denotes the batch size, and (z)) and (z2) correspond class probabilities of student and teacher predictions, respectively. We use Equation [along with task-specific loss overall loss function.
4

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 1: Bi-KD
Input: Training set X, label set y, learning rate 7), epochs maxx, iterations Nmax models fi and f2 parameterized by 01 and 02 respectively
for T=1 1 to Tmax do Shuffle training set X;
for N = to Maxx do
4
Fetch mini-batch x from X:
5
Compute output logits 21 = fi (z; 81) and B2 = f2(2; 02):
6
Obtain data mask my = =Z(0(Z1)gr (zz))
// samples where fi acts as a teacher
Obtain data mask m2 = =I(o(Z)) S g2
// samples where /2 acts teacher
Get the distillation loss Ldist m1 LKL2, +m2 KL1,2i
9
Get the cross-cntropy losses Leen = CE(fi (r;0)),)) and Lees = (2)))2(1;))))11)
10
Compute overall loss L Lee, + Loez + Laist 0c
11
Update 8 6-1 01 17 80,
12
Update 0g2 + 02 !I 30g²
0c
Output: Trained models f1() and f2(62)
3.2
Formulation
Given a training data batch D = (X,3) of N samples, where X { and = = (w) represent the inputs and corresponding labels for C classes, we consider two models fi and fe parameterized by 0 and 0g respectively. Their output logits are computed as follows:
21 (x(:;11), Z = f2(:00)
(2)
For each sample, dynamically assign the teacher role to the model with the highest prediction probability for the corresponding ground-truth class, resulting in data masks my and m2 denoting models fi and fz as teachers respectively:
mi II (Z(2)) > (Z2)ga1) my = [((())) I (Z2)ge)
Here, (+) denotes the softmax probability for the ground-truth class, and I represents the indicator func- tion. This approach partitions the data into samples where the best performing model provides supervision others while considering continuously improving models. The distillation loss is given as:
Laist mm LK + LK
(4)
Overall, the model exhibiting higher confidence acts teacher and the other as the student. The distillation loss is propagated only to the student model, and a stop-gradient operation is applied for the teacher. We also employ task-specific loss (e.g., cross-entropy for image classification, Lr for every model) alongside the distillation loss help preserve prior knowledge, and stabilize training. This combination constitutes the overall loss function:
L= LT, + Laist
3.3 Knowledge Transfer for dense tasks
We further extend our approach for semantic segmentation and video saliency prediction. Instead of the aforementioned confidence-based data partition, assign the teacher role to the model having the lowest loss for each sample. Let LT be the lask-specific loss, the dala masks are then formulated as:
mi = [ < Lrz), m2 = [Cr; 2 LrTz)
(6)
these masks are utilized in Equation to obtain the distillation loss Cdists and the overall loss function is the same as in Equation 5
5

---

Published in Transactions on Machine Learning Research (09/2025)
Semantic Segmentation: and dice loss (Milletari et al. 2016) for our mask loss:
For experiments semantic segmentation, use binary cross-entropy loss
Lmask AceLce + AdiceLdice
(7)
where Ace = 5.0 and Adice 5.0. The final task-specific loss for semantic segmentation is a combination of mask loss and classification loss:
Laamssg Lmaasce + Aelshells
(8)
with dets = 2.0 for correct predictions and 0.1 for incorrect ones.
Saliency Prediction: Video saliency prediction utilizes a combination of Equation 1 and Correlation Coefficient (CC), which calculates the Pearson correlation between the ground-truth truth and the predicted saliency maps, the final task-specific loss:
Lvsp CKL Lcc
(9)
(P..))
Lec ((P.P)xo(QQQ))
(10)
here P & Q are the predicted saliency map and ground-truth respectively, and (PP,Q) represents the covariance between P and Q.
3.4 Knowledge Transfer between multiple models
With the basic knowledge transfor between two models set up, extending to parallel knowledge transfer between multiple models the logical next step. As illustrated in Figure 2 the model with the highest prediction probability corresponding the ground-truth class is selected the teacher for a particular sample. Given K models for fo......,, fK- parameterized by 6g. 0,,.0.,............... respectively. The data mask for model & is computed as:
m. = I arg max ((o( (Zx = k
(11)
where mys with E tO.1....,KK 1}, represents whether the model fs acts as a teacher for a particular sample, and (z())))).... (Zx-1))g) denotes the concatenation prediction probabilities corresponding the ground-truth class for K models. The distillation loss is formulated as:
Laist Σ
Σ m
LK
(12)
i=0 j=0,j
with CKIYa considering models f, and fi as student and teacher respectively. The overall loss given as:
K-1
Σ
Cr. + Caist
(13)
0=

---

Published in Transactions on Machine Learning Research (09/2025)
4 Experiments & Results
In this scction, perform series of experiments evaluate our proposed method image classification, semantic segmentation, and video saliency prediction benchmarks. Section 4.1 introduces the datasets used for various experiments, and Section 4.2 explains the followed training choices. Finally, we discuss our results on various tasks in Section 4.3
4.1 Datasets
We verify the effectiveness our approach on multiple tasks using the following datasets:
ImageNet (Deng et al. 2009) consists of 1.2 million images for training and 50,000 images for validation. We report the results of our knowledge transfer between two multiple models on the validation set.
ADE20K (Zhou et al., 2017 provides 150 object and stuff categories, with 20,210 images in the training set and 2,000 images in the validation set. We use the validation set to evaluate our approach for knowledge transfer on semantic segmentation.
DHF1K (Wang et al. 2018) is a benchmark dataset for video saliency prediction, comprising 600 videos in the training set and 100 vidcos in the validation set. We use the validation for our evaluation.
woody Mathhe & Sminchisescu 2014) is the largest dataset for video saliency prediction in terms of the number of videos, containing 1,707 clips sourced from 69 Hollywood movies. Following the standard evaluation protocol, we use the predefined split of 823 videos for training and the remaining 884 videos for testing.
4.2 Implementation details
We implement all the networks and training procedures Pytorch (Paszke et al. 2019 , and conduct all experiments on a single NVIDIA RTX A6000. We use the Adam optimizer for image classification and video saliency prediction, while experiments on semantic segmentation utilize the AdamW optimizer. For all experiments, the lcarning rate and weight decay set to 1e-6 and le-5 respectively, with the temperature parameter set to 1 in Equation [ All models are trained in full precision for 20 epochs with batch size of 128. We do not apply any data augmentations, learning rate schedulers, or layer-wise learning rate decay, The only exceptions are Vills, for which we employ the default data augmentations and cosine scheduler provided by the timm [Wightman et al. 2019 library.
For experiments on ImageNet, we compare our approach with Roth al. (2024). We independently execute the KL+DP framework twice, each time initializing with the original models. The role of student and teacher is swapped in the second run. The original KL+DP paper presents experiments involving cross-entropy loss, which were shown to degrade performance. We observed a similar decline during our replication of their method. Therefore, to ensure a fair comparison we adopt their best performing configuration, excluding cross-entropy losses. Additionally, we compare against the supervised variant of KL+DP, wherein data partitioning guided by the model's confidence with respect to the ground-trute labels.
For knowledge transfer between semantic segmentation models, we utilize Mask2Former (Cheng et al. 2022) and follow its original experimental setup. Finally, we perform knowledge transfer between stale-of-the-art video saliency models proposed by Zhou et al. (2023) and Girmaji et al. (2025). We adopt their respective experimental setups to train the models from scratch, before transferring knowledge between them.
4.3 Results
Image Classification. For evaluating our approach on ImageNet, we utilize pretrained models listed in Table [ The models were chosen to cover a wide range architectures, performance levels, sizes, and training objectives. We utilize the base and small variants of MAE and DINOv2, respectively. In Table we report Atop-, which represents the change in Top-1 accuracy after transferring knowledge between models. Since Roth et al. (2024) have demonstrated that standard KD negatively impact performance when
7

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Selection models used for experiments on the validation set of ImageNet.
Models
Acc.
# Params. (M)
SeNet 154
He ct al.
2019
81.378
115.09
SWSt-ReaNexE
Xie
78.79
MAE (He al.
2022
83.116
86.57
VIT-B
Dosov 15kkiy e6 al
202
79.152
86.57
PfT-B
82,278
73.76
ResMLP-30 Louvion el a.
2022
79.576
41.69
MamnbaVisiot:
35.1
Res PP--Ii.P24-dist [Touvron ef
2022
80.548
30.02
DINOv2 (Oquab al. 2023:
81.332
23.98
VIT-S (Dsovilakiy
2021
78.812
22.05
Coa'T Ite-mmn (3 al
2021
78.858
11.01
PiT-XS [Heo
10.62
VIP- Dosuvitskty at.
2021
75.466
5.72
Table 2: Comparative results of change in Top-1 accuracy after transferring knowledge between models pretrained on ImageNet using different methods. We conduct three runs for our method and report the mean and standard deviations.
Method
Model
Atop-1
Model 2
Aop-1
KL+DP
0.582
0.306
DINOv2
MAE
Oura
1.494 0.042
0.426 + 0.028
KL+DP
PIT-B
SWSL-ResNext00
Ours
0.834 + 0.029
0.228 = 0.012
KL+DP
0.968
-0.242
DINOv2
Mamba Vision-T2
Ours
1.46 + 0.042
0.061 + 0.018
KL+DP
DINOv2
0.818
SWSL-ResNext101
0.538
Ours
1.587 + 0.012
0.431 : 0.036
KL+DP
ConT-lite-miii
0.436
SeNet154
0.48
Ours
+
KL+DP
Con'--lte-mini
0.386
DINOv2
0.692
Ours
0.567 + 0.026
1.381 = 0.051
KL+DP
PIT-XS
ResMLP-36
0.086
Ours
0.477 = 0.01
0.253 + 0.021
KL+DP
ConT-lite-mini
0.17
PTT-X8
0. 222
Ours
0.425 + 0.032
0.495 + 0.033
KL+DP
0.614
0.518
VIT-B
VIT-S
Ours
1.163 0.009
0.678 + 0.007
KL+DP
VIT-B
0.492
VIT-T
0.828
Ours
1.388 + 0.005
0.895 + 0.003
KL+DP
Ours
VIT-S
0.833 + 0.003
VI--T
0.949 + 0.002
learning from weaker or similarly performing teacher models, we compare our proposed method with their approach, referred to as KL+DP.
From the results presented in Table 2 we observe that Bi-KD consistently improves the performance of both participating models. This finding substantiates our hypothesis that pretrained models serve as effec- live sources for transferring complementary knowledge between another, thereby enabling the enhance- ment of each model independently. Furthermore, our method demonstrates superior performance compared to KL+DP, outperforming it in 19 out of 22 cases. The results provide supporting evidence for our hypoth- esis, demonstrating that simultaneous, bidirectional knowledge transfer, enabled through Bi-KD more effective than the unidirectional knowledge transfer employed by KL+DD, where the frozen teacher model is not optimized for teaching.
Our experiments further indicate that the most significant performance improvements occur when models with differing training methodologies are paired together. For example, pairing the self-supervised DI-
8

---

Published in Transactions on Machine Learning Research (09/2025)
CCA: Canonical Condations Across Components
Deginal
0.0
3-40
0.9
0:00
022
6.0
049
040
0.3
Component Indio
Figure 3: Canonical Correlation Analysis (CCA) of the Top-100 canonical components between the feature representations of ViT-S and VIT-T. The plot compares correlations from the original models with those from Bi-KD, showing that Bi-KD achieves stronger feature-space aligument.
Comparison of each model pair's performance before and after knowledge transfer (first training run) using Bi-KD, along with the performance of their direct ensemble.
Table 3:
Top-1
Top-1
Recovered (%)
Model
Model
Ensemble
Before
After
Before
After
Model
Model
DINOv2
81.332
82.722
MAE
83.446
83.842
84.332
SWSL-
PTT-B
82.278
83.1
ResNext101
84.276
84.506
85.206
28.1
24.7
Mamba
DINOv2
81.332
82.804
Vision-T2
82.506
82.542
83.56
3.4
SWSL-
DINOv2
81.332
82.902
ResNext101
84.276
84.75
85.162
CoaT-
lite-mini
78.858
79.34
ScNet154
81.378
81.834
82.302
49.4
CoaTT
lite-mini
78.858
79.442
DINOv2
8 81.332
82.694
82.732
15
97.3
PIT-XS
77.916
78.388
RrsMLPP-36
79.576
79.85
80.242
Con'T-
lite-mini
78.858
79.238
PIT-XS
77.916
78.366
79.866
37.7
23.1
VIT-B
79.152
80.326
Vir-S
78.842
79.526
VIT-B
79.152
80.5/11
VIT-T
75.466
76.361
80.274
124
18.7
VIT-S
78.842
79.68
VIT-T
75.466
76.412
79.69
98.8
22.4
NOv2 (Oquab et al. 2023) with any model trained using a supervised objective consistently results in performance improvements exceeding 1%. Interestingly, SWSL-ResNext101 (Xie et al. 2017), which has a Top-1 accuracy of 84. 276%, benefits more when paired with the relatively weaker DINOv2 than with the stronger PiT-B (Heo et al. 2021). Similarly, T-It--itommni (Xu et al., 2021) shows a greater improvement when paired with DINOv2 than with SeNet154 (He al.) (2019), despite both having comparable perfor- mance. These results align with the observation made by Gontijo-Lopes et al. (2022) that models trained through different methodologies tend to make uncorrelated errors, thereby making even lower performing models valuable contributors in knowledge transfer.
Another notable observation is that knowledge transfer between models with the same architecture also re- sults in considerable performance improvements. All three models: VIT-B, VIT-S, and ViT-T, consistently yield performance improvements when paired with each other. Interestingly, both ViT-B and ViT-S expe- rience greater performance gains when paired with the smaller VIT-T, rather than with each other. This

---

Published in Transactions on Machine Learning Research (09/2025)
further underscores that even smaller models can capture data-specific insights that may be absent in larger counterparts.
Distillation vs Ground-Truth Labels: We also examine how distillation labels deviate from ground- truth labels. Since the teacher model is chosen based on having higher confidence in the ground-truth class, three distinct scenarios arisc:
Case 1 Both the teacher and the student correctly predict the ground-truth class.
Case 2
The teacher predicts the correct class, but the student predicts incorrectly.
Case 3
Both the teacher and the student predict incorrectly, but the teacher assigns higher confidence to the ground-trut class than the student.
In Cases 1 and 2, the student clearly benefits from distillation. Although Case may negatively impact training, it is worth noting that the teacher still demonstrates better alignment with the ground-truth. Additionally, the task-specific loss function contributes to training stability and mitigates the potential adverse effects of Case 3.
Analyzing the predictions of original VIT-S and ViT-T models on the validation set, we find that Case 1 accounts for 70.9%, Case for 12.5% and Case 3 for 16.6% cases. Post Bi-KD, Case increases 73.1%, Case 2 decreases to 10.1%, while Case 3 remains similar levels, suggesting limited knowledge transfer in Case 3. Overall, the above empirical evidence indicates prodominant benefits of using Bi-KD (Casc 1 and Case 2) and minimal adverse elfects when distillation differs from ground truth labels (Case 3).
Feature Evolution: We further inquire into the evolution of feature representations across the participat- ing models, specifically, whether they become more alike following Bi-KD. To this end, we employ Canonical Correlation Analysis (CCA) to quantify and compare feature representations of the two models, before and after Bi-KD. The corresponding plot for ViT-S and ViT-T models is shown in Figure 3 CCA results show consistently higher correlations after Bi-KD, indicating that feature representations converge more closely between models. Considering the Top-100 components, we compute the mean CCA over the CLS embed- dings from the layer preceding the MLP head, and observe improvement from 0.6768 to 0.7487 post Bi-KD. This suggests that Bi-KD not only improves accuracy but also fundamentally aligns and enhances the models' internal representations.
Finally, Table 3 presents a comparison between the performance of Bi-KD and direct ensemble of its two constituent models. For the ensemble, the final classification is obtained by averaging the softmax scores of the individual models. To quantify this comparison, we calculate the percentage of the ensemble's performance retained by each model using the following formula:
Top- after Top-1hefore hefore
Recovered
(14)
Ensemble Top-Lbefore
As shown in Table 3 our method allows individual models to recover more than half of the ensemble's performance in most scenarios. Larger models, such as SeNet154, SWSL-RssNext101, MAE (He et al. 2022) ViT-B, and PiT-B, prove particularly effective, recovering approximately 60% of the ensemble performance on average. Among these, ViT-B stands out by recovering nearly 90% of the ensemble accuracy in one instance and surpassing the ensemble performance in another.
In contrast, smaller models, such as DINOv2, ViT-S, Tolt--itemmini, PiT-XS (Heo et al. 2021), and VIT-T, recover roughly 40% of the ensemble's performance on average. In particular, DINOv2 and VIT-S per- form consistently well, each recovering at least. 40% of the ensemble performance in all cases and closely approaching it some instances.
Overall, these findings underscore the effectiveness of our approach in enabling individual models, both large and small, to recover noticeable portion of the ensemble-level performance.
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 4: Performance comparison Mask2Former models before and after knowledge transfer on ADE20K dataset. We conduct five runs for each experiment and report the mean and standard deviations.
Before
After
Backbones
mIoU
fwloU
mACC
pACC
mloU
fwIoU
mACC
pACC
R50
47.23
70.95
60.11
81.72
47.78 + 0.12
60.47 + 0.13
81.88 + 0.07
Swin-T 47.7
72.37
61.44
82.7
48:32 + 0.07
72.54 0.07
61.8 0.04
82.87 0.07
Swin-S
51.33
73.26
65.11
83.48
51.77
+ 0.15
73.44 + 0.06
65.44 0.14
83.6 0.05
Swin-T
47.7
72.37
61.44
82.7
48.45 + 0.13
72.51 + 0.05
62.08 = 0.2
82.84 0.05
Table 5: We apply Bi-KD between TMFI-Net and ViNet-A, and compare the resulting models against range of individually trained saliency prediction methods. The last two rows represent the Bi-KD variants.
DHFIK IK
Hollywood-2
Models
CC
NSS
CC
NSS
ViNet (Jiin
2021
0.321
2.957
0.693
TSPP Net
Chang
Zinu
2021
0.529
3.009
0.711
3.91
Waagg al.
0.539
3.082
0.705
TMF1-Net
Zhou et al
2023
0.552
3,188
0.737
4.054
THTD-Net
Moradi el
2024
0.353
3.188
0.726
3.965
ViNet-S
Girman
2125
0.529
3,008
0.728
3,941
ViNet-A
Girmay et al.
2025
0.525
3.019
0.756
4.119
TMF1-Net
Zhou et al
2023
(BI-KD)
0.558
3.216
0.76
4.148
VINet-A
Girmaj et al
2025
(BI-KD)
0.536
3.077
0.762
4.198
Semantic Segmentation. Table compares the performance of Fask2Former models with various back- bones before and after applying proposed knowledge transfer method. The models are evaluated using four standard semantic segmentation metrics: mean Intersection-over-uiii (mloU), frequency weighted Intersection-over-niin (fwloU), mean Accuracy (mACC), and pixel Accuracy (pACC).
Our approach leads to consistent improvements in each metric within a single stage of training, whether transferring knowledge between different architectures or similar ones. Notably, Swin-T (Liu et al. 2021) backbone sees its mloU increase from 47.7 to 48.32 when paired with similarly performing R50 (He et al, 2016) backbone, which further increases 48.45 when paired with the stronger Swin-S (Liu et al. 2021) backbone. While the improvements are not dramatic, their consistency demonstrates the utility of our proposed approach in extracting additional performance from already well-trained models.
Video Saliency Prediction. Table presents comparative evaluation of video saliency prodiction task the DHFIK and Hollywood-2 datasets using two standard metrics: CC and Normalized Scanpath Saliency (NSS). We perform knowledge transfer on the TMF1-Net (Zhou et al. 2023) and ViNet-A (Girmaji et al. 2025) models. The performance of their Bi-KD variants is compared against their original versions as well as other stato-of-the-art methods.
On the DHF1K dataset, TMFI-Net (Bi-KD) establishes a new state-of-the-art, improving its CC from 0.552 to 0.558 and its NSS from 3.188 to 3.216. Similarly, ViNet-A (Bi-KD) demonstrates consistent gains, with its CC increasing from 0.525 to 0.536 and NSS from 3.019 to 3.077.
On the Hollywood-2 dataset, the performance gains are even more substantial. TMFI-Net (Bi-KD) improves from a CC of 0.737 to 0.750 and from an NSS of 4.054 to 4.148. ViNet-A (Bi-KD) achieves a new state-of- the-art, raising its CC from 0.756 to 0.762 and its NSS from 4.119 to 4.198.
These results validate the efficacy of our approach, demonstrating that mutual knowlodge transfer consis- tently enhances performance across diverse architectures and datasets, thereby advancing the -hat--of-theartt in video saliency prediction.
11

---

Published in Transactions on Machine Learning Research (09/2025)
Table 6: Parallel multidirectional knowledge transfer across multiple models. For image classification, we report results for Ioo-way, three- way, and four-way knowledge transfer using our proposed approach. For video saliency prediction, we present results for two-way and three-way knowledge transfer.
Models
Atop1
T-Ite-ieni
0.38
Models
DHFIK
PIT-XS
0.45
NSS
T-lte-mmni
0.46
PIT-XS
ResMLL-21-dist
0.476
0.15
TMFI-Net
ViNet-A
0.558
0.536
3.077
3.216
T-Ite-mmll
0.572
TMFI-Net.
VINet-A
0.561
0.54
3.224
3.087
ResMLP-24-ditt
PTT-XS
0.286
0.63
VINet-S
0.533
3.038
DINOv2
1.28
(b) Video saliency prediction
(a) Image classification
Multi-directional Transfer. Finally, we extend our knowledge transfer framework to support parallel, multidirectional transfer among multiple models within a single training stage. Table demonstrates the effectiveness of our parallel multidirectional knowledge transfer strategy across both image classification and video saliency prediction tasks.
In image classification on ImageNet, all participating models consistently benefit as more models are incorpo- rated into the collaborative learning setup. For example, the performance gain for CoaT--itemmini increases from 0.38 0.46, and PiT-XS improves from 0.45 0.476 when ResMLP-24-dist (Touvron al., 2022) is added. These gains are further amplified, reaching 0.572 for CoaT-lite-mintt 0.63 for PIT-XS, and 0.286 for ResMT.P-24-dist, with the inclusion of DINOv2. These results highlight the scalability and effectiveness of our approach with respect to the number of participating models.
Importantly, our approach is also task-agnostic. In video saliency prediction on DHFIK, TMFI-Net benefits from the knowledge transferred from ViNet-A, achieving a CC of 0.558 and an NSS of 3.216. Incorporating ViNct-S (Girmaji et al., 2025) into the collaborative learning cavironment further boosts the performance of both TMFI-Net and Vi with TMFI-Net achieving new state-of-the-art. These findings alfirm the robustness and generality of our knowledge transfer strategy across both architectures and tasks.
5 Conclusion
In this work, we introduce a simple yet ellective approach for simultaneous multidirectional knowledge transfer between pretrained models. Our method employs a dynamic data partitioning scheme that selects the most suitable teacher model for each sample, resulting in consistent performance improvements across all participating models within single training stage. By enabling each model to serve as both learner and teacher, our framework fosters mutual enhancement and contributes to the improvement of participating models. We demonstrate the effectiveness of our approach across a range of model architectures and tasks, including image classification, semantic segmentation, and vidco saliency prediction. Notably, our method sets a new state-of-the-art in video saliency prediction, underscoring the potential of collaborative knowledge transfer in complex visual understanding tasks. Additionally, we extend our framework to support knowledge transfer among multiple models and find that performance continues improve as more models are added to the collaborative environment. Our results provide compelling evidence for the viability of simultaneous multidirectional knowledge transfer between pretrained models.
Our approach has certain limitations. It builds on models pretrained for a specific task and transfers knowledge between them, which may affect the generalization capabilities of the underlying models. This concern is more pronounced for architectures like DINOv2, whose backbones are widely adoptod across diverse tasks. Moreover, the method depends on ground-truth labels, as a task-specific loss employed stabilize training, restricting its use in fully unsupervised settings. Addressing these limitations will be
12

---

Published in Transactions on Machine Learning Research (09/2025)
important direction for future work, which may include exploring model merging to consolidate the strengths of multiple models into a single, higher-performing system.
References
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E. Dahl, and Geoffrey E. Hinton. Large scale distributed neural network training through online distillation. In International Conference on Learning Represenlations (ICLR), 2018. URL
Lucas Beyer, Xiaohua Zhai, Amélic Royer, Larisa Markoeva, Rohan Anil, and Alexander Kolesnikov. Knowl- edge distillation: A good teacher is patient and consistent. In Conference on Computer Vision and Pallern Recognition (CVPR), pp. 10925-10922 2022.
Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3:747-769,
2021.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 535-541, 2006.
Qinyao Chang and Shiping Zhu. Temporal-spatial feature pyramid for video saliency detection. arXiv preprint arXie:2105.00 04213, 2021.
Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. In Association for the Advancement of Artificial Intelligence (AAAI), volume 34, pp. 3430- 3437, 2020.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Conference on Computer Vision and Pattern 2022.
Recognition (CVPR), pp.
Jeffrey Corneeius-hhite. Learner-centeeed teacher-student relationships are effective: A meta-analysis. Re- view of educalional research, 77(1):113-144 143, 2007.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255. Teee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Inter- national Conference on Learning Representations (ICLR), 2021. URL https:// net/forummff id=YicbFaNTTy
Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ramabhadran. Efficient knowledge distillation from an ensemble of teachers. In Interspeech, pp. 3697-3701, 2017.
Rohit Girmaji, Siddharth Jain, Bhav Beri, Sarthak Bansal, and Vineet Gandhi. Minimalistic Video Saliency Prediction via Efficient Decoder & Spatio Temporal Action Cues. In International Conference Acoustics, Speech, and Signal Processing (TCASSP), April 2025.
Raphael Gontijo-oppes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. In International Conference Learning Representations (ICLR), 2022. URL httpp:://openrevierr http:///pppprreeeeee.etffrrrrrrr BQ
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowl- edge distillation via collaborative learning. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11000-10220 2020.
13

---

Published in Transactions on Machine Learning Research (09/2025)
Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba-transformer vision backbone. anXiv preprint or Kr1::2407.08808, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770 778, 2016.
Kaiming Hc, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Conference on Compuler Vision and Pallern Recognation (CVPR), pp- 16000-16009, 2022.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classi- fication with convolutional neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 558-567, 2019.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Rethinking spatial dimensions of vision transformers. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11936-11945, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Advances in Newral Information Processing Systems (NeurIPS), 2015.
Samyak Jain, Pradeep Yarlagadda, Shreyank Jyoti, Shyamgopal Karthik, Ramanathan Subramanian, and Vincet Gandhi. Vinet: Pushing the limits of visual modality for audio-visual saliency prediction. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3520 3527. IEEE, 2021.
Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge transfer. Advances in Newral Information Processing Systems (NeurIPS), 35:635 649, 2022.
Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neuro-
computing, 415:106 113, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference Com- puter Vision (TCCV), pp. 10012-10022, 2021.
Michael Livanos, lan Davidson, and Stephen Wong, Cooperative knowledge distillation: A learner agnostic approach. In Association for the Advancement of Artificial Intelligence (AAAI), volume 38, pp. 14124- 14131, 2024.
Sihui Luo, Xinchao Wang, Gonglan Fang, Yao Hu, Dapeng Tao, and Mingli Song. Knowledge amalgamation from heterogencous networks by common feature learning. International Joint Conference Artificial Intelligence (IJCAI), 2019.
Stefan Mathe and Cristian Sminchisescu. Actions in the eye: Dynamic gaze datasets and learnt saliency models for visual recognition. IEFF Transactions on Pattern Analysis and Machine Intelligence (PAMT), 37(7))1408 1424, 2014.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference 3D vision (3DV), pp. 565-571. Ieee, 2016.
Morteza Moradi, Simone Palazzo, and Concetto Spampinato. Transformer-basee video saliency prediction with high temporal dimension decoding. arXiv preprint Niw:2401.00944. 2024.
Maxime Oquab, Timothóe Darcet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et Dinov2: Learning robust visual features without supervision. Transactions Machine Learning Research (TMLR), 2023.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Wonpyo Park, Dongiu Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Conference Computer Vision and Pattern Recognation (CVPR), pp. 3967-3976, 2019.
Adam Passke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, ZZeming Lin, Natalia Gimelshein, Luca Autiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. International Conference on Learning Representations (ICLR), 2014.
Karsten Roth, Lukas Thede, A. Sophia Koepke, Oriol Vinyals, Olivier J Henalf, and Zeynep Akata. Fantastic gains and where to find them: On the existence and prospect of general knowledge transfer between any pretrained model. In International Conference on Learning Representations (ICLR), 2024. URL atttps://openreviee rettfforummii0mm550KtCttt
Guocong Song and Wei Chai. Collaborative learning for deep neural networks. Advances in Neural Infor- mation Processing Systems (NeurIPS), 31, 2018.
Suraj Srinivas and François Fleuret. Knowledge transfer with jacobian matching. In International Conference on Machine Learning (ICML), pp. 4723 4731. PMLR, 2018
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaacldin El-Nouby, Edouard Grave,
Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions on Pottern Analysis and Machine Intelligence (PAMT), 45(4):5314 5311-3111,,,, 2022.
Diane Wagner, Fabio Ferreira, Danny Stoll, Robin Tibor Schirrmeister, Samuel Müller, and Frank Hutter. On the importance of hyperparameters and data augmentation for self-supervised learning. In First Workshop on Pre-training: Perspeclives, Pilfalls, and Palhs Forward at ICML 2022, 2022. URL https: //openreview.o wttp://wwww.eettttnnnnneaaeeeeee net./oorumiid-
Wenguan Wang, Jianbing Shen, Fang Guo, Ming-Ming Cheng, and Ali Borji. Revisiting video saliency: A large-scale benchmark and new model. In Conference on Computer Vision and Pattern Recognation (CVPR), pp. 4894-4903, 2018.
Ziqiang Wang, Zhi Liu, Gongyang Li, Yang Wang, Tianhong Zhang, Lihua Xu, and Jijun Wang. Spatio- temporal self-attention network for video saliency prediction. IEEE Transactions Mullimedia, 25: 1161-1174, 2021.
Ross Wightman et al. Pytorch image models, 2019.
Gloria Brown Wright. Studentt-centered learning in higher education. International journal of teaching and learning in higher education, 23(1):92 97, 2011.
Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. Association for the Advancement of Artificial Intelligence (AAAT), volume 35, pp. 10302-10310, 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1492- 1500, 2017.
Weijian Xu, Yilan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-altentional image transformers. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 1981-9990, 2021.
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1285-1294. Association for Computing Machinery, 2017. ISBN 9781450348874. doi:
10.1145/3097983... 3005//90999933398155 URL ott.:/ttts////aaooo/////../00000
15

---

Published in Transactions on Machine Learning Research (09/2025)
Fei Yuan, Linjun Shou, Jian Pei, Wutao Lin, Ming Gong, Yan Fu, and Daxin Jiang. Reinforced multi-teacher selection for knowledge distillation. In Association for the Aduancement of Artificial Intelligence (AAAT); volume 35, pp. 14284 14291, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Compuler Vision (ICCV), pp. 6023 6032, 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference Learning Repre- sentations (ICLR), 2017. URL htttps://openrrevlex..u wttp:///w....eetoooo.oooosss....
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4320-4328, 2018.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 633 641, 2017.
Xiaofei Zhou, Songhe Wu, Ran Shi, Bolun Zheng, Shuai Wang, Haibing Yin, Jiyong Zhang, and Chenggang Yan. Transformer-aaser multi-scale feature integration network for video saliency prediction. 1EEE Transactions on Circuits and Systems for Video Technology, 33(12):7696 7707, 2023.
Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. Advances in Newral Information Processing Systems (NewrIPS), 31, 2018
16


Reviews:
[{"review_id": "AMoLjOi50L", "reviewer": "Reviewer_XJcU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The authors should be commended for highlighting a highly ambitious and quite interesting research direction towards automated joint learning for model ensembles. While mutual distillation has been explored, the idea of extending this to automated ensembles is an interesting research direction that warrants further exploration. \n2. The approach is demonstrated to generalise across different tasks, including classification, segmentation, and saliency. Consistent gains in results show promise. In particular, the model provides strong results in video saliency.\n3. The approach requires little modification and additional machinery to provide results, and could potentially be used as a drop-in replacement for existing distillation and fine-tuning pipelines.\n\n### Weaknesses\n\n1.  A main concern of this reviewer is generalisation. The models are evaluated on a singular task, and exclusively with models that are pretrained for the given task. No hold-out sets or different downstream tasks are evaluated, which raises concerns on overfitting.\n2. A second concern is how the core gating mechanism used in bi-distillation depends on the ground-truth label, which is a signal unavailable at test time or on unlabeled data. However, the narrative leans on a *self-improving zoo*. In contrast, an EMA distillation process is cheaper and can arguably be posed as a bi-distillation process that is amenable to SSL and importantly, does not require two hot models in memory simultaneously to perform training. This is known to be a computational bottleneck, particularly in terms of memory. \n3. Hard, argmax-based masking is nondifferentiable and could be unstable around ties, and the training stability/variance isn’t reported. How does the distillation perform when differences are small or tied?  \n4. Uniform KL with $T=1$ between near-one-hot vectors closely mirrors cross entropy, which makes it unclear how much dark knowledge [1] is actually distilled between the models. Again, an analysis of the dynamics in training would be a welcome contribution, and would help readers gain better understanding of the method.\n5. The paper is missing several training and implementation details. Batch sizes, epoch/step counts, augmentation pipelines, exact TIMM configs/checkpoint hashes, gradient clipping, AMP settings, training time / GPU-hours are not mentioned. While not strictly required, an anonymised GitHub repo with config files could potentially substitute for this omission, but as it stands, the reported results are not reproducible. In particular; details on specific model capacities for some models (e.g. DINOv2 and MAE) are missing. What was the model capacity (Base, or Large) for these models?\n\n### Summary\n\nThe paper broaches an interesting topic on collaborative learning, and is ambitious in framing its narrative. However, the work seems to somewhat implicitly overpromise based on the abstract and introduction, limiting the experiments to bi-distillation. The work is lacking an analysis on how learning dynamics affect training. The paper is tersely written and sparse in detail, resulting in a work that is currently non-reproducible. This reviewer's impression is that the manuscript requires serious revision to be considered for acceptance.\n\n[1] Distilling the Knowledge in a Neural Network, Hinton et al. 2015.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an automated collaborative transfer-learning approach between 2 models via sample-wise peer knowledge distillation. In essence, the authors train two pretrained models jointly, and treat the one with the higher probability on the true label for a given sample as the teacher and its logits are distilled (KL, T=1) into the other. Roles (teacher and student) can swap from sample to sample based on the logits as a measure of confidence or certainty. In dense tasks, this is replaced by selecting the model with minimum loss as the teacher on the same per sample basis. *Data partitioning* is implemented as a binary mask that routes the KL loss from the currently more label-confident peer; all models still receive CE on all samples. The ensemble is then said to form an ensemble based \"collaborative learning environment\", where models jointly distill from one another. \n\nThe central contribution of the authors is the aforementioned methodology for mutual knowledge distillation. Implicitly, the narrative suggests that this constitutes a step toward an automated “self-improving zoo” with minimal changes, no extra teachers or temperatures, and applicability to heterogeneous architectures, however, experiments are limited to two models ($K=2$) trained in joint bi-distillation, although the paper outlines a multi-model approach (presumably $K > 2$). The authors report modest gains (+0.5–1.5% top‑1 on ImageNet for ViT-B paired with a smaller ViT-T, as well as improved segmentation on ADE20k video saliency), and claim to *recover* a large share of the ensemble’s boost inside a single model, argued via a *Recovered % of ensemble improvement* metric. However, it is unknown how much training were performed to achieve said gains, as this is unreported.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The paper should include some additional downstream tasks to show how the model generalises beyond the datasets they were trained on. Additionally, some OOD analysis should be performed, particularly with foundational models such as DINOv2, which has been trained for wide adoption to different downstream tasks / baselines. This is critical for acceptance.\n2. The narrative of an automated joint collaborative training is interesting, but somewhat oversold. The paper presents results for bi-distillation, and hence the narrative should reflect this. Reformulating this so the contribution is clarified is necessary for acceptance.\n3. An analysis of training dynamics, and a post-training analysis on the representation spaces would go a long way to help the reader evaluate the merits of the method. An analysis on how the training dynamics adjust to small differences and ties during gating between the two models is a good start. This reviewer asks themselves; \"did the representations become more alike after mutual distillation\"? This could be tested via a canonical-correlation analysis or mutual information estimate, although the former is simpler to perform. While this would improve the paper tremendously, it is not absolutely necessary for acceptance.\n4. Without details on training and implementation, readers have no hope of being able to reproduce the reported results. At the bare minimum, batch size, number of training steps, and a general training setup should be included. This is crucial for acceptance.", "broader_impact_concerns": "The paper is a work on knowledge distillation, and has no applicable broader impact concerns from the viewpoint of this reviewer.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "51l5dnXbsU", "reviewer": "Reviewer_ppWT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The proposed method is straightforward and shows competitive results.\n2. The proposed method extends to more than the classification task, including dense prediction and video saliency prediction, which were not discussed in previous papers.\n\n**Weaknesses:**\n1. The experiment details are not comprehensive. How the baseline results are derived seems ambiguous. See the Questions.\n2. Roth et al. [1] shows that in an unsupervised setting, in which they partition the data based on model confidence and do not use the classification loss, they are able to achieve similar results using a classification-guided loss. The authors do not discuss whether a similar unsupervised setting is effective for their method.\n\n**Questions:**\n1. For the baseline method, it is unclear how the results are derived. Specifically, for the student model (i.e., model 1), it is stated to be distilled from the teacher model (i.e., model 2). However, it remains ambiguous whether the teacher model is distilled from the original student model or from the already distilled student model. Additionally, it is not specified whether the loss function used in the baseline includes the cross-entropy loss alongside the distillation loss. Clarification on these points would help better interpret the reported performance.\n2. Roth et al. [1] partition the training data into two subsets, using one subset for distillation from a frozen copy of the student model to retain previously learned knowledge. In contrast, the proposed method lacks such an explicit mechanism for preserving prior knowledge. Notably, by skipping training on non-complementary data, resulting in fewer update iterations, the method still achieves improved performance. Could the authors clarify whether distilling from a frozen student model is necessary or beneficial in this context?\n3. The proposed method uses a bidirectional distillation loss. When computing gradients with respect to the parameters of the first model, both directions of the loss contribute to the update. On complementary data, the loss encourages the student model to align with the teacher, while on non-complementary data, it promotes divergence from the teacher. Could this dynamic of combining alignment and contrast depending on the data contribute to the improved performance?\n\n[1] Roth, K., Thede, L., Koepke, A.S., Vinyals, O., Hénaff, O. and Akata, Z., 2023. Fantastic gains and where to find them: On the existence and prospect of general knowledge transfer between any pretrained model. arXiv preprint arXiv:2310.17653.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper builds on a previous method [1] for data partitioning to achieve the transfer of complementary knowledge between pretrained models. Unlike the previous method, which has a frozen teacher and trainable student, the paper proposes to dynamically switch the roles of the models, thus transferring the complementary knowledge bidirectionally. The proposed method is also easily transferrable to groups of multiple models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "I have no such concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kDZvAjJyGZ", "reviewer": "Reviewer_3Rzi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1. The paper is clearly written.\n2. The experiments are extensive with various pre-trained models and tasks.\n\n***Weaknesses*\n1. Some crucial technical details are missing. For the distillation loss, there appears to be no stop-gradient operation. This may be problematic as it allows both models trying to match the prababilities of the other. Could the authors clarify this point?\n2. Each model is learning simultaneously from both the ground truth labels and distillation labels, why is this necessary? The current manuscript does not explain this design choice.\n3. In a related question, how much do ground truth labels differ from the distillation labels? What if distillation labels actually predict a wrong ground truth class despiting being the more confident prediction among the two models? In other words, there needs to be some justifications/analysis for teacher selection based on loss metric.\n4. Since the paper is mostly empirical, I would expect more interprretation results explaining why the method works well in practice, instead of just showing the end results that it outperforms the baseline. For instance, does the performance improvements come from the teachers being dynamic instead of being fixed in KL + DP? Does KL + DP also use ground truth labels as part of the loss (in other words is the comparison fair)?\n5. The comparison focuses primarily on the single baseline of KL + DP.  This may be insufficient to justify the efficacy of the proposed approach (alternative baselines such as self-distllation etc).\n6. Some of the improvements are marginal (Table 4) and reported with any confidence level. This makes it hard to assess the method's efficacy. In addition, all performance are worse than the ensemble (Table 3), this may suggest some sub-optimal teacher selection for the distillation targets. Also it raises the question of why not using the ensemble probabilities as distiallation target?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presemnts a simplet yet effective strategy for knowledge transfer via model distillation. The proposed approach leverages multiple pre-trained models, and each one of them can serve as the teacher for training samples, based on some loss metric. The experiments evaluated this approach on various tasks and pre-trained model pairs, showing improved performance over existing methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses above.", "broader_impact_concerns": "N.A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)