Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model
Authors: Karsten Roth
Lukas Thede
A. Sophia Koepke
Oriol Vinyals
Olivier J Henaff
Zeynep Akata
Venue: ICLR 2024 spotlight (2024)
Keywords: transfer learning
pretraining
weak-to-strong transfer
continual learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We first show that transferable knowledge exists between any pairs of models pretrained on the same data. We then study and propose approaches to transfer this knowledge between any model pair - independent of relative performances & architectures.

Abstract: Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variations to result in networks learning unique feature sets from the data. Using public model libraries comprising thousands of models trained on canonical datasets like ImageNet, we observe that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other – independent of overall performance. Given any arbitrary pairing of pretrained models and no external rankings (such as separate test sets, e.g. due to data privacy), we investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation – a task made particularly difficult as additional knowledge can be contained in stronger, equiperformant or weaker models. Yet facilitating robust transfer in scenarios agnostic to pretrained model pairings would unlock auxiliary gains and knowledge fusion from any model repository without restrictions on model and problem specifics - including from weaker, lower-performance models. This work therefore provides an initial, in-depth exploration on the viability of such general-purpose knowledge transfer. Across large-scale experiments, we first reveal the shortcomings of standard knowledge distillation techniques, and then propose a much more general extension through data partitioning for successful transfer between nearly all pretrained models, which we show can also be done unsupervised. Finally, we assess both the scalability and impact of fundamental model properties on successful model-agnostic knowledge transfer.

Published as a conference paper at ICLR 2024
FANTASTIC GAINS AND WHERE TO FIND THEM: ON THE EXISTENCE AND PROSPECT OF GENERAL KNOWL- EDGE TRANSFER BETWEEN ANY PRETRAINED MODEL
Karsten Roth-.", Lukas Thede A. Sophia Koepke'
Oriol Vinyals?, Olivier Hénaff Zeynep Akata
'Tübingen AT Center & University of Tühingen, 'Google DeepMind HHelmholzz Munich, "Technical University Munich "equal contribution
ABSTRACT
Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variations to result networks learning unique feature sets from the data. Using public model libraries comprising thousands of models trained on canonical datasets like ImageNet, we observe that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other independent of overall performance. Given any arbitrary pairing of pretrained models and no external rankings (such separate test sets, e.g. due to data privacy), investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation a task made particularly difficult additional knowledge can be contained in stronger, equiperformant or weaker models. Yet facilitating robust transfer in scenarios agnostic pretrained model pairings would unlock training guidance, auxiliary gains and knowledge fusion from any model repository without restrictions on model & problem specifies including from weaker, lower-performance models. This work provides a first, dn-depth exploration on the viability of such general-purposs knowledge transfer. Across large-scale experiments, we first reveal the shorteomings of standard knowledge distillation techniques, and then propose a general extension via data partitioning for successful transfer between nearly all pretrained models which can also be done unsupervised. Finally, we assess both the scalability and impact of model properties on successful model-agnostic knowledge transfer.
INTRODUCTION
Training neural networks on specific datasets has become machine learning standard to tackle myriad of research and industry challenges, involving a large number of explicit and implicit decisions that range from architecture choices to specific optimization protocols, the particular choice of data augmentation, data sampling and even the data ordering. All these factors can impact the semantic knowledge model can extract from dataset (Bouthillier et al., 2021; Schmidt et al., 2021: Wang et al., 2023; Raghu et al., 2021; Wagner et al., 2022; Roth et al., 2020; Balestriero et al., 2023; Teney al., 2020; Roth al., 2023), and together provide a unique fingerprint of a model's capabilities. In this work, we first highlight the extent of this statement through extensive experiments. We build on large open model libraries (e.g. timm (Wightman, 2019) or huggingface to compare large num- bers of arbitrary pretrained model pairs. Doing so, we discover the consistent existence of significant complementary knowledge- information about the data that one model (the "teacher") holds that not available in the other one (the "student"). Interestingly, we find that complementary knowledge exists regardless of external performance rankings or factors like model families (CNNs (LeCun and Bengio, 1995), Transformer (Dosovitskiy et al., 2021), MLP (Tolstikhin et al., 2021)), and often aggregates in semantic areas of expertise: For stronger, but especially also similar or weaker teachers (by some test metric), significant knowledge about the data not available to the student can be found.
Such general availability of complementary knowledge raises questions about its potential utility. To answer those, we provide first, in-depth exploration. Specifically, given arbitrary pairs of models

---

Published as a conference paper at ICLR 2024
pretrained the same data without access external ranking measures (s.a. test sets, due to e.g- data privacy, separate test servers,... we explore if transfer of complementary knowledge between any teacher and student is possible without performance degradation. Achieving such transfer through any possible model pair unlocks any freely available or self-generated model collection as an auxiliary resource for gains in canonical and problem-specific pretraining. It also avoids the need for model-specific transfer that require expert knowledge, and reduces the reliance on external evaluation measures for model selection. More importantly however, it also enables improvements of larger models by knowledge transfer from weaker, lower-resource models, without the explicit need for additional data & supervision, or sacrifices in e.g. speed, fairness, interpretability or ease-of-use.
We investigate the limits of knowledge distillation (Hinton et al., 2015; Tian et al., 2020) for this task, which in contrast to data-free approaches (e.g. Wortsman et al. (2022a)), operates independently of model choices. However, standard knowledge distillation frameworks assume information to be distilled to an untrained student. contrast, we only wish to transfer knowledge not available in an already trained student model, which may even outperform its teacher. This crucially entails a success- ful trade-off between knowledge gain and retention. Indeed, for knowledge transfer between arbitrary pretrained models, common distillation ($5.1) exhibits strong model/htypepparameerr dependence and performance drops for the majority of student models, particularly for weaker/eqqipeerfommntt teachers. This can be attributed to catastrophic forgetting (Kirkpatrick al., 2016; Zenke et al., 2017) outweighing the benefits of complementary knowledge transfer from the teacher.
For favorable trade-off between forgetting and knowledge gain, treat the transfer process as a continual learning problem, where a model is continuously presented with new context for data already seen. To encourage retention, we first study weight interpolation (Stojanovski al., 2022; Wortsman et al., 2022b). While better than normal distillation, it is often too strong a constraint when the teachers have niche areas of expertise or are overall stronger. We thus propose to constrain distillation at the data level by partitioning it into two sets one with samples where transfer from a teacher desired, and one where we wish retain the student behavior. This introduces significantly fewer constraints on the model weights to learn from arbitrary teacher context, while reducing forgetting by retaining initial performance on samples where the teacher has limited positive (even detrimental) impact. Moreover, our data partitioning can be achieved without any supervision.
Doing so, we see significant increases in the success rate (non-zero gains of the student) for all teacher-student pairings from 32.5% with normal distillation to 92.5% with data partitioning. Our data-level regularization is the only setting which allows for consistently positive transfer from weaker teachers, while retaining the transfer performance of normal distillation for much stronger teachers and even outperforming normal distillation for equiperformant ones. In addition, it allows for the transfer of specialized knowledge (55.1) and requires no mairing-specifte hyperparameters. Unlike ensembling methods (Lakshminarayanan et al., 2017: Gontijo-Lopes et al., 2022; Sinha et al., 2021; Dietterich, 2000), our approach maintains original inference costs and handles high performance differences. Finally, we study architectural properties and their impact on the transfer process 1$5.1) beyond the transfer method, and look into scalability to knowledge transfer from multiple models, where we find that simple sequential transfer can perform favorably when leveraging our transfer method, achieving clear improvements over transfer from just the single best teacher model.
Overall, our contributions can be summarized as: (1) We discover the consistent existence of comple- mentary knowledge between arbitrary models pretrained on the same dataset- - even if model families or performances differ. (2) We conduct extensive, exploratory studies to investigate the possibility of guaranteed model- and performance-indeeendee performance degradation. (3) We propose a successful transfer method motivated through the lens of continual learning, leveraging a confidence-baseee
transfer of the complementary knowledge without
hyperparameter-free data partitioning approach. (4) We provide studies the relation of general model properties to general knowledge transfer, and (5) investigate knowledge transfer between multiple models. Code will be released upon acceptance.
RELATED WORK
Early works in knowledge distillation focus on compressing large teacher models into smaller student models. Bucila et al. (2006) achieve this by matching the soft targets of the teacher. Hinton et al. (2015) propose temperature scaling for lower probabilities. Recent works extend this with structural attention-transfer (Zagoruyko and Komodakis, 2017) encourages similar feature response context:
2

---

Published as a conference paper at ICLR 2024
25
Equal Performance
Complementory Knowledge
20-
Min. Compl. Knowledge
15p
Euttopy
10
6.2-
6.0
Uniform Distribution
5.8
10
Performance Difference
Complementary Knewledge [%6
(a)
(b)
Figure 1: (a) We show the share of complementary knowledge (p perc. of pos. label flips of teacher w.r.t. student) against perform. differences Ance for 466 teacher/student pairs, and find significant complementary context even for much weaker teachers. (b) Looking at the entropy of the compl.
knowledge distribution over classes, we find context to be more specialized for weaker teachers.
patterns, Romero et al. (2015) and Zagoruyko and Komodakis (2017) propose (weighted) regression- guided student feature activations. However, these approaches are limited to matching student and teacher architectures. Tian et al. (2020) propose contrastive distillation, aligning teacher and student feature spaces with flexibility in feature dimensionalities, while highlighting performance overlap among most distillation objectives. These insights transfer to multiple teacher distillation (Luo et al., 2019; de Carvalho et al., 2022; Shen et al., 2019a;b), e.g. via teacher outputs reweighting (Wu al., 2021; Liu al., 2020; Yu al., 2022; Yuan et al., 2020a). Unlike standard distillation, look at knowledge transfer between arbitrary, already trained models a much more difficult task, particularly when no restrictions (in contrast to e.g. (Yuan et al., 2020b)) are imposed on relative performances or architectures, and initial knowledge should be retained. On a conceptual level, this also connects to recent works on weak-to-strong model generalization for superalignment, for which our work can provide an orthogonal perspective and useful practical insights (Burns et al., 2023).
Wortsman et al. (2022a;b) show that when architecture and pretraining are shared, linear interpolation can be surprisingly effective for suitable loss basins (Neyshabur et al., 2020). Still, model selection through validation metrics is key for diverse collections. In contrast, we combine model knowledge without restrictions by leveraging perspectives from Continual Learning. Approaches are categorized into regularization (limit weight changes (Kirkpatrick et al., 2017; Schwarz et al., 2018; Li and Hoiem, 2016; Rebuffi al., 2016; Castro et al., 2018)), replay (Rebuffi et al., 2016; Lopez-Paz and Ranzato, 2017; Chaudhry et al., 2019; Buzzega et al., 2020; Prabhu et al., 2020) through a data memory, and methods that restructure networks (Mallya and Lazebnik, 2018; Mallya al., 2018; Zhang et al., 2020). On top of that, interpolation has also proven useful when continually adapting from pretrained model (Stojanovski al., 2022), which we include in our transfer study.
3 COMPLEMENTARY KNOWLEDGE BETWEEN PRETRAINED MODELS
Over recent years, thousands of models pretrained completion canonical datasets such as ImageNet have been made publicly available, with variations across all possible training choices ($ 1). which potentially impact generalization the extent which we wish to study in this section. In particular, we use simm (Wightman, 2019), comprising hundreds of models trained on ImageNet under varying conditions, and consider the image classification problem with input space X' and label space y' with = | labels. Let f(- 0) X' + IR be classifier with parameters 9 E O, logits (a, 8) and softmax ((2) with or
exp(z,)/ E; exp(2;) associated with samples E X. We use ft. fa to denote the pretrained teacher and student, with parameters O, and 0, respectively. evaluate the complementarity of knowledge between any fi and fs. we follow the methodology of Lopes et al. (2022) and study the performance Specifically, we measure positive prediction.fli between ft and fa. pPus = 1 Di P. where p indicates positive flip. This quantifies the proportion of samples correctly classified by the teacher but incorrectly by the student the complementary knowledge. For the remainder of this work, we will refer to complementary knowledge and the existence of these positive flips interchangeably.
the ImageNet validation set.
Existence of complementary knowledge. Using t timm, we randomly select 466 (f.. f.) model pairs covering 301 unique models of varying architectures, sizes, and performances. In Fig.
3

---

Published as a conference paper at ICLR 2024
1) Weak Teacher (Diff
ii Eigual Accs. (Diff: +:0)
Hii) Strong Teacher (Diff.
Differance to Avg. Sim [%]]
+12%
Coepl Knowletee ledge
per Class [Kg]
30
40
+10%%
40-
20-
20-
10-
+6%
Top-2%
Top-5%
Top-20% Top-50%
ImageNet Classes
ImageNet Classes
ImageNNtt Classes
Classes with Top-X% X% of Compl. Knowledge
(a)
(b)
Figure 2: (a) Sorted histograms of complementary knowledge per class (positive prediction flips) as a share of total class samples for three teacher-student pairs (weak, equal and strong teacher). Complementary context is centralized around few classes. (b) Semantic similarity between top-X% classes sorted by complementary knowledge amount. Shown relative difference to average class similarity. Classes with the most complementary knowledge are likely semantically related.
investigate the share of complementary knowledge against the difference in performance between fe and fa- This is measured as the difference in validation accuracy Aac acc( )) acc(Jss for each pair. We find large share of positive prediction flips when fi outperforms fa- But even when fo notably underperforms the student model (up to 20%), a high share of positive flips can still be found. Converging around 2%. this percentage is more than an order of magnitude higher than random noise for 1000 classes, the probability of a model to correct sample by chance is around 0.1% (e.g. randomized ResNet50 teachers show complementary knowledge of around only 0.03%). As such, our result indicate consistently existing complementary between any pretrained model.
Understanding the complementary knowledge. To figure out if the complementary knowledge is distributed evenly among classes or if it is systematically grouped within particular subsets, we analyze the distribution prediction flips over all classes. For a selected subset in Fig. 2a, where classes are sorted by the amount of complementary knowledge encoded, a disproportionate amount of context, particularly in the case a weaker teacher. For stronger ones, the complementary context the teacher can provide becomes more evenly distributed. This is further supported when looking at the entropy of these distributions for all (fr. f.)-pairs in Fig. 1 (right), where we a clear trend towards more aggregated context for weaker teachers as the entropy goes down. In all cases however, a significant degree of context grouping remains. We denote these groups of classes with significant complementary knowledge as relative areas of expertise. This notion becomes even more evident as we investigate the semantic relation between them in Fig, 2b. For this, we measure the semantic similarity of classes containing the first 2%, 5%, 10%, 20% and 50% of positive flips (based on the per-class ranking by complementary knowledge as in Fig. 2a) using pretrained language model (CLIP, Radford et al. (2021)) Comparing the measured similarity to the average similarity of all classes, we relative increase in semantic similarity by nearly double on average for the classes that encode the most complementary knowledge.
some classes carrying
In summary, we observed that complementary knowledge between any pair of pretrained models exists, and that this knowledge which a pretrained teacher can pass to the student is centered around areas of expertise comprising semantically-reattee classes. The existence of ubiquitous complementary knowledge motivates our study possible general-purpose knowledge transfer tools.
4 GENERAL KNOWLEDGE TRANSFER METHODOLOGY
This section explains possible knowledge transfer objectives, starting from standard knowledge distillation ($4.1 to our proposed extensions in $4.2, which highlights how and why this problem should be treated continual learning one. $4.3 extends this multiple pretrained teachers.
4.1 KNOWLEDGE TRANSFER THROUGH KNOWLEDGE DISTILLATION
Knowledge Distillation (KL distillation) pioneered by Hinton et al. (2015), which suggests minimizing the KL divergence between a teacher and a student model's soft targets (2)) and (z,):
LKL /a Em KI (a (T(aan.iT)) (z),
(1)
4

---

Published as a conference paper at ICLR 2024
Batch Samples
(x,)
my (xt)
Batch Samples
Teacher Model f
(2mmz
(2,)
Target
Knowledge Transfer
Student Model &
Student-
Kl.
o(ae)
.22
Teacher
Model 1
(Rap)
Figure 3: For general knowledge transfer between any pretrained models, propose data-level regularization: Samples are separated based on if they should be taught through the teacher fs or retained via frozen version of the initial student, fa- All models forward the same batch, and outputs (2a) and o (2st) are merged on sample-level via selection masks m and mat derived from model confidences (Eq. 4). Lastly, we compute the KL-div. to the adapting student's (fz) outputs (2z)).
with temperature T. We use Eq. as our base transfer objective (KL-Dist transfer), as it still remains popular and competitive (Beyer et al., 2022; Rajasegaran et al., 2020; Tian et al., 2020) (see Supp. for other objectives). KL distillation is often extended with auxiliary classification losses (e.g. cross- entropy Ex (Hinton et al., 2015; Tian al., 2020; Rajasegaran et al., 2020; Beyer et al., 2022)) to stabilize the distillation process. We denote the X-weighted combination as XE-KI. distillation, and the associated transfer XE-KI--Dist. transfer or XE-KI.:
Las = x LKL + (1 X) LXE-
Most knowledge distillation research considers the distillation from a trained teacher an untrained student, in stark contrast to our goal knowledge transfer between pretrained models while retaining student knowledge already gained a priori. And indeed, when applied to knowledge transfer between pretrained models in $5.1, standard knowledge distillation struggles to transfer knowledge without performance drops for most teacher-student pairs. We measure this via the transfer delta A ace(fs**) ace(fs) which quantifies the change in the student's top-1 accuracy, with Jan being the student model following the knowledge transfer.
4.2 KNOWLEDGE TRANSFER THROUGH CONTINUAL KNOWLEDGE DISTILLATION
For successful knowledge transfer, favorable trade-off between retaining existing student knowledge and incorporating complementary teacher knowledge is required. This bares semblance to Continual Learning (CL) frameworks (Kirkpatrick et al., 2016; Zenke et al., 2017; Aljundi et al., 2019). which aim to train models on incremental data streams without forgetting previously learned context.
Constraining weight updates. Unlike standard CL however, our problem is not that of an incremental data stream, but of continuous new learning signals from the teacher over the same transfer data. This excludes memory- based approaches, but permits regularization (see $2 for details), where Stojanovski et al. (2022) show that for continuous adaptation trained models, momentum weight interpolation (MCL) proves
effective than existing regularizations. We thus extend base XE-KI. objective with MCL (XE-KL-DDss++C) transfer XE-KL+MCLL. Weight interpolation in MCL retains a slow copy of the student weights Oslow addition fast weights 0fant updated during the transfer. At predefined frequency N and iteration i, (ploww is updated via weight-space interpolation:
gslow i+1 = T galow. (1-7) glast.
(3)
with momentum guiding the weight constraint. Both N and can be tuned to balance the plasticity- stability trade-off. However, as weight interpolation constrains weight updates for all samples equally, it struggles to leverage the relative areas of expertise of teachers to their full extent (c.f. $5.1).
Constraining transfer data. Instead of weight-level restrictions, we suggest regularization on benefit from
the data-level by partitioning the transfer data into samples where the student teacher feedback and ones where the prior knowledge should be retained. In particular, for samples essential to the transfer of complementary knowledge, distill from the teacher ft. However, for
5

---

Published as a conference paper at ICLR 2024
the other sample set, we instead distill from the initial, frozen student model (denoted fat for crudent-teacheed with the goal of retaining the initial model behavior. The selection of these data subsets follows a simple and effective greedy heuristic that assigns each sample depending on the highest prediction probability for the corresponding ground-truth class, giving data masks m and m
m [a, (Ze,s) > (z(2,t,))),
m (z. S a (z(zst,))]
for each sample i and j = ys. and with (2) the softmax output for class j. In practice, we find that the use of these masks provides enough stability to the transfer process where auxiliary classification is no longer required. In addition to that, we also find that this supervised heuristic can be replaced with a fully unsupervised one by assigning samples based the maximum prediction probability for a sample, i.c. choosing the model by confidence. While this can suffer from overconfidence, in practice performance matches the supervised setting. This means that knowledge transfer can be performed without access to labels. We provide visualization of our data partitioning (DP) in Figure 3. The final transfer approach which we refer to as KL-Dist +DP transfer, is thus given as:
Las = /n Ei= m (L
+ m KL (zz (T), (T).::T)) (5)
As can be seen, KL-Dist DP transfer (or KL+DP) requires no additional hyperparameters compared to standard knowledge distillation, with strong robustness towards temperature choices (c.f. Supp.).
4.3 MULLTI-EEACHER KNOWLEDGE TRANSFER
With multiple models available in model zoos, studying how to transfer context between multiple experts is natural extension, for which we suggest studying three approaches. Firstly, in line with standard
knowledge distillation, all teachers can be used at once for knowledge transfer (parallet), while still leveraging our proposed transfer regularizer described above to ensure positive transfer. In particular, the greedy data selection is extended to produce data subsets for each respective teacher model. Secondly, the multi-model transfer process can be done sequentially in line with the continual treatment of the single-model transfer process, After each teacher model transfer, the distilled student is treated as the (new) pretrained student for the subsequent transfer step. Finally, our experimental section also investigates the use of Model Soups (Wortsman et al., 2022a) for the problem of sirreecture-idependdee distilled from each teacher model, producing set of distilled model variants ( ) i,....., with K, teacher models. After transfer, simple weight interpolation between all variants is performed (soup).
knowledge transfer. Here, the student model is independently
EXPERIMENTAL STUDY ON EFFECTIVE KNOWLEDGE TRANSFER
We first conduct a large-scale study of knowledge transfer approaches (c.f. 1-.33) in $5.1, high- lighting the advantage of a continual learning approach, and particularly the superiority of our partiiaatiioonng method. For exploration, we use a supervised variant (Eq. 4) but show in Tab. that the unsupervised variant matches the performance. Finally, we investigate the relation of model properties and general transfer success (also $5.1). and study transfer from multiple models in $5.2. Experimental details. We use NVIDIA 2080Ti compute clusters with PyTorce 1.13.1 (Paszke et al., 2017) and ffcv 0.0.3 (Leclere et al., 2022) for fast data-loading. While performance may be slightly impacted, relative changes are retained (Leclerc et al., 2022), allowing large-scale experiments reasonable compute. For our large-scale evaluations of transfer approaches, we use a 10% stratified ImageNet subset (subsampling per class), then validate our main claims on the full ImageNet dataset. We perform transfer over a constrained budget of 20 epochs to study methods for knowledge transfer with practical requirements. Other optimization parameters are determined via grid searches (see Supp.). Code:
LerGGGeeaa--nnwleeeeeeeeeeeeeeee Knowledge Transferr.rr
5.1 EVALUATION OF DIFFERENT APPROACHES FOR KNOWLEDGE TRANSFER
Effectiveness of standard knowledge distillation for knowledge transfer. To study the suitability of standard KL distillation for general-
knowledge transfer, we select 400 teacher-student pairs (Tab. 5 in Supp. for details), all of which exhibit a significant percentage of complementary knowledge (c.f. $3). Across these pairs for each student model, we measure the percentage of teachers for which a positive transfer delta Amanst. is obtained. Results are visualized in Fig. 4a, and reveal that for the majority of students there are less than 40% of teachers with performance increases. An

---

Published as a conference paper at ICLR 2024
Knowledge Transfer Saccess Rate (%g
Knowledge Transfer Delta
6-
Ki.- Transfer
XC-KL Dist, MCL Transfer
80-
KL-Dist. DP Tranfer
60
40-
20-
-2-
0-
KI-Dist Transfer
XE-KL-Dist.
XE-KL-Dist+
KL.-Dist DP
-10
-5
5
10
Transfer
MCL Transfer
Trmsser
Performance Difference of T'Tchher and Student
(a) Percentage of successful transfers.
(b) Correlation teacher properties & transfer success.
Figure 4: (a) Share of teachers resulting in positive knowledge transfer (success rate) for knowledge distillation variants (blue) and continual learning extensions (orange). Each box represents 400 transfer experiments, with a clear increase for continual learning setups. (b) Transfer delta by binned teacher-student performance difference. For more robust reporting, we show the mean transfer delta of the top 25% for each bin/approach with the same 400 teacher-student pairs. The results show KL-Dist+ DP Transfer enabling consistent gains from weaker and stronger teachers.
Table I: Knowledge transfer results on full ImageNet, from four teacher to eight selected student models (see Supp.Tab, 6). The results provide further evidence that a continual learning approach to the transfer problem allows for more consistent knowledge transfer across arbitrary model pairs.
Students
Type
Acc.
Param.
Amnut KL-Dist.
Amnn KL-Dist+DP
XCIT-PI6 (El-Nouby al., 2021)
Trafo
82.89
189.10
0.1..01001
0.65 (110.26)
PÝT-B (Heo et al. 2021)
Trafo
82.44
73.76
0.33 (#0.86)
0.55 (::0.25)
PIT-XS (Heo et al, 2021)
Trafo
78.19
10.62
(10049))
0.40 (1:0.12)
ScNet154 (Hc ct al. 2018)
CNN
81.23
115.09
-0.05 (+0.266)
0.27 (+:017))
ConvNext (Liu et al., 2022)
CNN
84.57
50.22
-0.51 (1:0.5)
0.33 (1:0.14)
ResNetV2 (He al., 2016b)
CNN
82.80
25.55
-0.09 (1:0344)
0.23 (+0.08)
Mixer-Ble (Tolstikhin al., 2021)
MLP
82.30
(1.0.88)
0.15 (1.0.13)
ResMLPP22 (Touvron et al., 2021)
MLP
80.76
30.02
0.15 (+0.366
0.33 (+0.119)
additional classification loss (XE-KL, $4.1) can raise the median success rate slightly above 40%. In both cases, however, overwriting pre-existent knowledge more often than not overshadows the benefits gained from actual knowledge transfer, particularly when transferring from a weaker teacher model as shown in Fig. 4b (KL-Dist. Transfer), where absolute performance changes after transfer are visualized against initial teacher-student performance differences (as measured on the separate validation data). In addition, we find that these limits also hold when deploying simple additional regularizers such as label smoothing (54%, Yuan et al. (2020b)), with consistently negative transfer for weaker teachers, and reduced effectiveness for stronger ones.
Leveraging continual learning. Treating general knowledge transfer as a continual learning task through weight regularization MCCI) raises median success rates significantly (80%, Fig. 4a). However, we find a lack of efficacy when knowledge is specialized to areas of expertise, and when teachers are stronger, which we address with data-level regularization (KL+DP), raising success rates to 92.5%. As shown in Fig, 4b, these gains can be attributed to positive transfer deltas even for much weaker teachers (see performance differences much lower than zero), and, unlike strict weight-level regularization in e.g. MCL, barely limit gains from much stronger teachers. Indeed, we find that for a number of stronger teachers, particularly where performance differences are not as striking, data-level regularization can even offer an edge over normal distillation.
Full ImageNet experiments. We extend our experiments from above to full ImageNet in Tab. and verify that insights transfer, with KL++P performing significantly more reliably than normal distillation. We also provide a more detailed overview Table 2, highlighting successful transfer from both stronger, but particularly weaker teachers well even if student models are already strong ( +0.3% for PiT-B (Heo et al., 2021), 82.44% ImageNet top-1 and ConvNeXt (Liu et al., 2022), 84.57%, with nearly 5% performance differential). For additional results ImageNet and model information, we refer to tables and 7 in the supplementary. In addition, the supplementary also reveals similar transfer success using KL+DP for other datasets such as CUB200 (Wah et al.,
7

---

Published as a conference paper at ICLR 2024
80-
Trafo
Transfer [%)
80
Transfer Rate (%)
60-
CNN
MLP
60-
40-
20-
20
Top-2%
Top-5%
Top-20%
Top-50% Top-100%
10
200
Classes with Top-X% of the Compl. Knowledge
Number of Student Parameters [M]
(a) Transfer rates for the top-X% pos. flip classes.
(b) Transfer rate by student size and model family,
Figure 5: (a) Transfer rates for the sets of classes containing the top 2% 100% complementary knowledge per model family. (b) Transfer rates versus student size separated by model family.
2011), StanfordCars (Krause et al., 2013), and Caltech256 (Griffin et al., 2007). Furthermore, we leverage full ImageNet runs in Tab. to demonstrate that our unsupervised variant of KL+DP transfer (64.2) performs comparably and in some cases even better than its supervised counterpart (e.g. A wansf. 0.55 vs Arnnst. 0.95 for PiT-B students). This showcases that general knowledge transfer can be done even without supervision by leveraging data-level continual learning regularization.
We do find increased variance due the unsu- pervised data selection. To better understand it, we study the number of positive (teacher cor- rect, student incorrect) and negative flip samples (vice versa) that are assigned to the teacher. Ide- ally. this only includes positive flip samples. For model pairs presented Table using KL-Dist.
Table 2: Examples for the transfer deltas of two student models each distilled with stronger and weaker teacher model on full ImageNet using our KL-Dist+DP transfer approach.
Teacher +
Vulo-D2
ResMLP-36
Student 1 Axe
Agust.
Auc
Amast
+ DP transfer, we find that 72% of positive and 9% of negative flip samples are assigned to the teacher. This means that while simple confidence-bart partitioning does not perfectly assign samples, it still strongly aligns with re- spective areas of expertise.
PiT-B
+2.75
0.86
-2.77
0.31
ConyNext
+0.62
0.44
-4800
Table 3: Comparison between supervised and un- supervised KL-Dist+DD transfer on ImageNet for eight selected students and four teachers, respec- tively. Results show that fully unsupervised knowl- edge transfer between experts is not only possible but can even outperform supervised transfer.
Overall, we find very promising transfer across teacher-students pairs even without supervi- sion. While gains fall short of the total comple- mentary knowledge $3 attributable to the trade- off between retention and transfer- we believe our results to offer strong proof-of-concey for future research and the potential of truly general knowledge transfer.
Students
Areut KL+DP
Supervised
Areett KL-DP Unsupervised
XCIT Nouby et al., 2021)
0.65 (1-0.66
0.96 (4:0.42)
PVT-H (Heo al, 2021)
(+0.25)
0,95 (+00.52)
PIT-XS (Hoo et al., 2021)
0.40 (200.2)
0.35 (2:0.18)
SeNetl54 (He et al., 2018)
0,27 (100.7))
0.25 (1.0.15)
ConsNext
al., 2022)
(110.1))
0.28 (1:0.14)
ResNetV2 (He et al., 20166)
0.23 (10.08)
0.23 (1:0.09)
Mixer-B16 (Tolstikhin et all, 2021)
0.15 (+0.11)
0.17 (+0.15)
Complementary knowledge drives transfer gains. Our experiments above show clear gains when conducting KL+ DP transfer across all kinds of model pairs. In this part, we show that gains indeed stem from the transfer of the complementary knowledge between these pairs (c.f. $3). For that, we analyze the student's ' improvement per class relative to the available complementary knowledge per class, denoted transfer rale, for all 400 transfer runs studied in Fig.4a. Results in Fig. 5a plot the transfer rate against classes containing the top-X% X% of complementary knowledge. After sorting classes by the share of complementary knowledge (c.f. in Fig. 2a, we look at how the first X% of complementary knowledge are transferred). Following our notation from $5.1. 1, smaller percentages mostly contain flips from a teacher's relative area of expertise, which gets softened for larger values. Indeed, when utilizing our proposed KL+DP transfer, we find a clear indication where complementary knowledge associated with stronger areas of expertise in the teacher model have a near- guar-guuretteed chance to be transferred to the student model. This drops when moving towards flips associated with less represented classes. Note that without preference towards relative areas of expertise, one would expect horizontal line at the average transfer rate. This shows that our CL-based treatment of knowledge transfer allows any teacher model to impart specific knowledge and that one can explicitly guide the context learned by student model by choosing leacher with suitable area of expertise.
ResMLP-24 (Touvron 2021)
(::0.19)
0.29 (2:024))
8

---

Published as a conference paper at ICLR 2024
Table 4: Transfer from multiple teachers in sequential, parallel and soup-based fashion (c.f. $4.3).
We find sequential transfer to perform favorably.
Students
Type
Acc.
# Param.
Auant. Mean
Single
Min
Teucher Max
Sequ.
Amant
Multiple Teachers Parallel
Soup
XCYT-PI6 (El-Nouby et al., 2021)
Trafo
82.89
189.1
0.73
0.36
0.93
0.97
0.91
0.63
Twins (Chu et al., 2021)
Trafo 83.68
99.27
0.22
0.20
0.26
0.65
0.30
0.21
PiT-B (Hoo al., 2021)
Trafo
82.44
73.76
0.64
0.31
0.86
1.04
0.77
0.69
Properties of a good student model. We conclude our single model transfer studies by investigating if and how specific student model properties can facilitate the reception of complementary knowledge. Across our experiments performed in Sec. 5.1, we find in Fig. 5b that the transfer rate of complemen- tary knowledge has significant relationship with the model capacity (parameter count). We observe this general trend across architectural families irrespective of the visual inductive biases encoded. However, we highlight that while larger models are generally more receptive towards additional context, CNN-style architecture, or generally models with stronger visual inductive biases, are more prone to have their existing knowledge overwritten, resulting in a lower distillation delta. See Supp. for additional details and experiments.
5.2 KNOWLEDGE TRANSFER FROM MULTIPLE PRETRAINED MODELS
Finally, we study transfer from multiple teachers on full ImageNet, testing sequential and parallel KL+ transfer, and model soups variant by interpolating between transferred student variants (c.f. $4.3). Our experiments include three students and teachers (Supp. B.6), where we focus on Transformers, which in our experiments have shown the highest aptitude towards knowledge reception. Results are shown in Tab. 4, where we compare to the performance of single-teacher transfer.
We find that students can consistently gain knowledge from each teacher when transferring se- quentially. However, as the student improves, returns diminish until the transfer deltas plateau, as forgetting becomes more prevalent as we move further from the initial student pretraining. Never- theless, the sequential transfer of three teachers achieves an average gain of 59% compared to the best single teacher transfer delta (e.g. A 0.26 -> Aranst 0.65 for Twins (Chu et al., 2021) or A 0.86 + Iranst 1.04 for PÝT-B (Heo et al., 2021)). At the opposite end, we find vanilla KL-Dist. transfer to be unsuccesful in the multi-teacher setting, underlining the benefits of KL+DP transfer (see also Supp.). Furthermore, while we found consistent knowledge gain irrespective of the teacher order, our experiments indicate that descending teacher sequence (i.e. strongest first) can actually incur tisproportionately higher forgetting, the model moves away from its base knowledge more quickly. Finally, unlike sequential transfer, parallel transfer of multiple teachers does not improve over the best single teacher transfer performance. This is due reduced amount of retention happening as the respective subset regularization (c.f. $4.3) does not retain enough samples for active knowledge retention. Finally, we find weight averaging of student variants distilled with each respective teacher to perform worst (underperforming the best single-teacher transfer), which we attribute to lack of interpolatability and a subsequent reduction in knowledge retention.
CONCLUSION
In this work, show that any pairing of models trained on the same dataset with different training protocols (e.g. changes in architecture or optimization) exhibits significant complementary knowledge data context encoded in one model and not the other. Based on the presence of complementary knowledge, we offer a first exploration into a general mechanism to transfer it between any pair of models without performance degradation and reliance on external ranking measures. This unlocks any model repository as a resource for model gains, and the option to improve large models with context from weaker, lower resource ones. Our large-scale experiments reveal the limits of simple knowledge distillation continual learning and confidence-
general transfer mechanism, and suggest extensions through the lens of
data partitioning. This raises the transfer success rate from under 40% to over 92%, with positive transfer from both stronger and weaker teachers. We also provide insights into general model properties that facilitate transfer, finding model capacity and reduced visual inductive biases to be beneficial. Finally, showcase transfer from multiple models with our transfer mechanism. Overall, we provide the experimental motivation and first steps towards general- pprepss complementary knowledge transfer tools between arbitrary model architectures.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
Karsten Roth thanks the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program and the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support. This work was supported by DFG project number 276693517, by BMBF FKZ: 01831839A, by the ERC (853489- DEXIM), by EXC number 2064/1 project number 390727645.
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances Neural Information Processing Systems (NeurIPS). 2019.
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gor- don Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar. Hamed Pirsiavash. Yann LeCun, and Micah Goldblum. A cookbook of self-supervised learning. arXiv preprint arXiv:2304. 12210, 2023.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: good teacher is patient and consistent. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Paseal Vincent. Accounting for variance in machine learning benchmarks. In Conference on Machine Learning and Systems (MLSys), 2021.
Cristian Bucila, Rich Caruana, and Alexandru ficulescu-Mizii Model compression. In International Conference on Knowledge Discovery and Data Mining (KDD), 2006.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to- strong generalization: Eliciting strong capabilities with weak supervision, 2023.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
Francisco M. Castro, Manuel J. Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In ECCV, 2018
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations (ICLR), 2019.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In Advances in Neural Information Processing Systems (NeurIPS). 2021.
Marcus Vinícius de Carvalho, Mahardhika Pratama, Jie Zhang, and Yajuan San. Class-incremental learning via knowledge amalgamation. In European Conference Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD) 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A laage- scale hierarchical image database. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Thomas G. Dietterich. Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems (MCS), 2000.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICI.R, 2021.
Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jegou. Xeit: Cross-covariance image transformers. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
Raphael Gontijo-Lppes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. In International Conference on Learning Representations (ICLR), 2022.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. Technical report, California Institute of Technology, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), 2016b.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In European Conference on Computer Vision (ECCV), 2018.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In International Conference on Learning Representations (ICLR), 2021.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. arXiv preprint 1003:0503300531, 2015.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- Demis
drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 2016.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A.
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In IEEE International Conference on Computer Vision (ICCV) Workshops, 2013,
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. FFCV; Accelerating training by removing data bottlenecks. https:/ /ggggubtthh//gggg//////// com/ libffcv ffte/ 2022. commit 2544abd.
Yann LeCun and Yoshua Bengio. Convolutional Networks for Images, Speech, and Time-Series, page 255-258. MIT Press, 1995.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2016.
Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. Pay attention to mlps. In Advances in Neural Information Processing Systems (NeurIPS), 2021a.

---

Published as a conference paper at ICLR 2024
Iou-Jen Liu, Jian Peng, and Alexander G. Schwing. Knowledge flow: Improve upon your teachers. In International Conference on Learning Representations (ICLR), 2019.
Yuang Liu, W. Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neurocomputing, 2020.
7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. IEFE International Conference on Computer Vision (ICCV), 2021b.
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Raphael Gontijo Lopes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. In International Conference on Learning Representations (ICLR), 2022.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning, In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Sihui Luo, Xinehao Wang. Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song. Knowledge amalgamation from heterogeneous networks by common feature learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning mask weights. In European Conference on Computer Vision (ECCV), 2018.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Jinhyuk Park and Albert No. Prune your model before distill it. In European Conference on Computer Vision (ECCV), 2021.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
Ameya Prabhu, Philip H. S. Torr, and Puneet Kumar Dokania, Gdumb: A simple approach that questions our progress in continual learning. In European Conference on Computer Vision (ECCV), 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Maithra Raghu, Thomas Unterthiner, Simon Korblithh, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Jathushan Rajasegaran, Salman Hameed Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. Self-supervised knowledge distillation for few-shot learning. In British Machine Vision Conference (BMVC), 2020.
12

---

Published as a conference paper at ICLR 2024
Sylvestre-Alvis Rebuffi, Alexander Kolesnikov, G. Sperl, and Christoph H. Lampert. icarl: Incre- mental classifier and representation learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations (ICLR), 2015.
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjorn Ommer, and Joseph Paul Co- hen. Revisiting training strategies and generalization performance in deep metric learning. In International Conference on Machine Learning (ICML), 2020.
Karsten Roth, Timo Milbich, Bjorn Ommer, Joseph Paul Cohen, and Marzyeh Ghassemi. Simulta- neous similarity- based self-distillation for deep metric learning. In International Conference on Machine Learning (ICML), 2021.
Karsten Roth, Oriol Vinyals, and Zeynep Akata. Integrating language guidance into vision-based deep metric learning. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentan- glement of correlated factors via hausdorff factorized support. In International Conference Learning Representations (ICIR), 2023.
Robin M Schmidt, Frank Schneider, and Philipp Hennig. Descending through a crowded valley benchmarking deep learning optimizers. In International Conference on Machine Learning (ICML), 2021.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka
Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress and compress: A scalable framework for continual learning. In International Conference on Machine Learning (ICML), 2018.
Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. Amalgamating knowledge towards comprehensive classification. In AAAI Conference on Artificial Intelligence (AAAI), 2019a.
Chengchao Shen, Mengqi Xue, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation. IEEE International Conference on Computer Vision (ICCV), 2019b.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Samarth Sinha, Homanga Bharadhwaj, Anirudh Goyal, Hugo Larochelle, Animesh Garg, and Florian Shkurti. Dibs: Diversity inducing information bottleneck in model ensembles. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
Zafir Stojanovski, Karsten Roth, and Zeynep Akata. Momentum- weight interpolation of strong zero-shot models for continual learning. In Workshop on Interpolation Regularizers and Beyond, held at NeurIPS, 2022.
Mingxing Tan and Quoc Chen. Mixconv: Mixed depthwise convolutional kernels. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Damien Teney, Ehsan Abhasnejad, and Anton van den Hengel. Unshuffling data for improved generalizationin visual question answering. In IEEE International Conference on Computer Vision (ICCV), 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Interna- tional Conference on Learning Representations (ICLR), 2020.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucie, and Alexey Dosovitskiy. MLP-mixer: An all-MLP architecture for vision. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
13

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
Diane Wagner, Fabio Ferreira, Danny Stoll, Robin Tibor Schirrmeister, Samuel Müller, and Frank Hutter. On the importance of hyperparameters and data augmentation for self-supervised learning. In Workshop of Pre-training: Perspectives, Pitfalls, and Paths Forward, held at ICML, 2022.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-uesd birds 200. Technical report, California Institute of Technology, 2011.
Zeyu Wang. Yutong Bai, Yuyin Zhou, and Cihang Xie. Can cnns be more robust than transformers? In International Conference on Learning Representations (ICLR), 2023.
Ross Wightman. Pytorch image models, 2019.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Loees. Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML), 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022b.
Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. One teacher is enough? pre-trained language model distillation from multiple teachers. In Findings of the Association for Computational Linguisties (ACL-IJCNLP), 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
ZeLun Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Self-supervised learning with swin transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Weijian Xu, Yifan Xu, Tyler Chang. and Zhuowen Tu. Co-scale aonv-attentional image transformers. In IEEE International Conference on Computer Vision (ICC)), 2021.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Longhui Yu, Zhenyu Weng, Yuqing Wang, and Yuesheng Zhu. Multi-teacher knowledge distillation for incremental implicitlyreffneed classification. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Fei Yuan, Linjun Shou, Jian Pei, Wutao Lin, Ming Gong, Yan Fu, and Daxin Jiang. Reinforced multi multi-teacher selection for knowledge distillation. In AAAI Conference on Artificial Intelligence (AAAI), 2020a.
Li Yuan, Francis F. H. Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR, 2020b.
Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference Learning Representations (ICLR), 2017.
14

---

Published as a conference paper at ICLR 2024
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning (ICML), 2017.
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry P. Heck, Heming Zhang, and C.-C. Jay Kuo. Class-incremental learning via deep model consolidation. In WACV, 2020.
15

---

Published as a conference paper at ICLR 2024
APPENDIX
FANTASTIC GAINS AND WHERE TO FIND THEM: ON THE Ex- ISTENCE AND PROSPECT OF GENERAL KNOWLEDGE TRANSFER BETWEEN ANY PRETRAINED MODEL
A IMPLEMENTATION DETAILS AND EXPERIMENTAL INSIGHTS
In this section, we descrihe the implementation details of our experiments to evaluate the effectiveness of different approaches and techniques for transferring complementary knowledge between pretrained expert models trained on the same dataset.
For our initial and exploratory experiments use a 10% stratified subset of ImageNet (Deng et al., 2009) to reduce runtimes, in order to conduct wider range of experiments across large number of model pairs. detail, we drew 130 samples per class using the standard ImageNet validation set for evaluation. All of our experiments utilize an SGD optimizer with momentum 0.9 and weight decay 1e-3. Further hyperparameters were individually tuned for each investigated transfer approach.
A.1 IMPLEMENTATION OF DISTILLATION BASED KNOWLEDGE TRANSFER VARIANTS
To set the learning rate for our default knowledge distillation based transfer approach using KL divergence (as in Section 4.1), we conducted parameter search over a set of 33 teacher- student pairs randomly selected from the timm Wightman (2019) library, with learning rates Ir E {le-2, 1e-3, 1e-4, le-5), for which we found a learning rate of 1e-4 to generally work best, albeit regardless of chosen values, the average transfer delta was consistently negative.
Following Section 4.1, we also extend KL distillation with a cross-entropy classification loss. In this case, hyperparameters were determined over a grid comprising learning rates Ir E {le-2, 1e-3, le-4, le-5), softmax temperatures T' {0.1, 1, 4, 10} and weightings À E (0.05, 0.1, 0.25, 0.5). Again, we found that learning rate of 1e-4 was the most effective on average, but found particular variance in the weighting \, where we observed that larger À value placing higher emphasis on distillation is better suited for transferring knowledge from a stronger teacher weaker student, while smaller À seems to be preferable when transferring knowledge from weaker teacher to a stronger student. This further highlights the trade-off between knowledge gain and retention, where for weaker teacher, retention plays much more crucial part ensure overall high performance, as student knowledge is overwritten.
For the softmax temperature, we found that a small temperature of 0. limits the decrease in the student's 's performance when transfering from weaker teacher model, but also limiting the knowledge transfer in general. This results in only small increases the student's performance even when transferring from a stronger teacher model. Hinton et al. (2015) propose to use larger temperature of 4 to match soft targets to better represent smaller probabilities in the output of single sample. However, we do not find larger temperatures to benefit the transfer performance.
In general, we find that particularly the temperature and weighting parameter guide the aggressiveness of the distillation- bassed transfer approach, which is highly dependent on the observed teacher and student dynamic of the provided pair of pretrained expert models. The high variance across such arbitrary model pairs makes normal knowledge distillation, even paired with additional classification loss for stability, not well suited as a general knowledge transfer tool.
A.2 IMPLEMENTATION OF CONTRASTIVE DISTILLATION KNOWLEDGE TRANSFER
While knowledge distillation approaches matching the soft targets of the teacher and student model remain popular, various recent approaches argue that more structural knowledge can be transferred by encouraging the student model to also match intermediate representations of the teacher model (Liu al., 2019; 2020; Wu et al., 2021: 1: Park and No, 2021). Thus, in this section, highlight results of our exploration on the feasibility using intermediate representations and their relations to transfer knowledge between pretrained experts.
16

---

Published as a conference paper at ICLR 2024
We particularly follow Tian et al. (2020), who propose to extend the basic knowledge distillation approach of Hinton et al. (2015) by aligning the feature representations of the teacher and the student models. Here, the student is encouraged to provide feature representations close to the ones of the teacher for similar images while repelling the feature representation of dissimilar images. Unlike other existing distillation approaches operating feature representations, such a contrastive approach puts less restrictions on the architectures of the teacher and the student model, particularly because the feature representations of both models can be cheaply projected into a common feature space using a learned projection layer for both models. This enables the distillation between models of different architectures, and allows us to explore an alternative to our utilized base KL Distillation objective for general knowledge transfer (Sec. 4.1).
To assess the feasibility of popresentationmmathiinn for knowledge transfer between expert models, we implement two contrastive learning approaches. First, we utilize a simple approach that encourages the distances between the feature representations of a pair of images to be similar for both the teacher and the student model. Hence, if two images result in similar feature representations in the teacher's 's embedding space, the student is encouraged to also provide feature representations with close proximity in their respective embedding space. Such relative similarity baased matching has seen success in standard supervised contrastive learning, such as in (Roth et al., 2021; 2022). Using t and denote teacher and student respectively, this gives
LCD KL (o((.))
(6)
where S is similarity matrix containing the cosine similarities of the normalized feature represen- tations of the current batch (S.y = cos sim (norm(s;) sommnorm
(s;), norm(s))), Vi,j E O,..., We denote this
approach CD Distillation.
Secondly, we implement the contrastive representation distillation approach (CRD distillation) of Tian al. (2020). As noted, CRD distillation directly aligns representations by encouraging the student to be close to the teacher for positive pairs (different augmentations of the same image) while pushing apart feature representations of negative pairs (images of different classes). The respective objective is thus given as:
LCRD = arg max max [e((.s/c=1)]oog llog htt, s)] + kiBggt../(c=)))logg - h(t, s)]],
h
where we utilize f, s as shorthand for respective teacher and student representations. In addition, we use h: 8 - [0, 1] represent discriminator estimating whether the feature representation t and are drawn from the same joint distribution or from the respective marginal product. In this setup, denotes the number of negative pairs drawn from the product of marginals.
Both contrastive distillation approaches compute the overall distillation loss Laist as a weighted combination of the respective contrastive loss LCD or LCRD and cross-entropy classification loss LxE as also used in standard KL Divergence distillation objectives Beyer et al. (2022); Rajasegaran al. (2020).
For CD distillation based knowledge transfer, we tested different weightings between the contrastive loss and the classification loss as well different learning rates on small set of teacher-student combinations. On a similar hyperparameter grid as noted in the previous section, we found an equal weighting of both losses in combination with a learning rate of 1e-4 to be most suitable on average, thought with a similar trade-off as depicted in Section 1. For the CRD distillation transfer, we found the hyperparameters as provided in Tian al. (2020) to work well.
A.3 IMPLEMENTATION OF CONTINUAL LEARNING BASED TRANSFER APPROACHES
Finally, we describe hyperparameters and the corresponding hyperparameter studies utilized for our continual learning extension to distillation-basee knowledge transfer (see Section 4.2), in particular the setup for XE- ME-KL-Dis++MLL transfer and KL-Dist+ DP transfer.
For XE-KL+MCL transfer, we conducted a parameter search on learning rate grid with the same resolution as before. However, as there are several other parameters to validate, we only test Ir E (1e-2, 1e-3). In addition to that, we follow Stojanovski et al. (2022) and test the momentum for values in 7 E {0.99, 0.999, 0.99993) and the interpolation frequency N e (2, 10, 50, 100)). For the weighting against the classification objective, X, test 0.5 and 0.7. We conducted the
17

---

Published as a conference paper at ICLR 2024
Table 5: Selection of student and teacher models used for the experiments on the 10% ImageNet subset. Each set of models was selected to contain multiple architecture types and cover a wide range of model sizes and performance levels.
Type
Teacher Models
Type
Acc.
# Param.
CGTT-Lange22-DD1 El-Nouby et al. (2021)
Trafo
82,89
189.10
ConoNect Liu et al. (2022)
CNN 86.64
197.77
ViT--ase-PPt P16 Desovissliy iy et al. (2021)
Tralo
84.53
86.57
VOLO-14 Yuan (2021)
Trafo
PTB Hoo et al (2021)
Itafo
82.44
73.76
RogNety-332 320 Radosavovic ct (2020)
CNN
80.50
145.05
ViT-Relooo Medium- P16
Trafo
82:16
38.75
VGG13 Simonyan and Zisserman (201.)
CNN
71.60
133.05
PTT-X
RegNetx- 320 Radosavovic
(2020)
CNN
80.24
107.81
PYT-XF-dist Hen all (2021)
79.31
11.00
TWINS Chu (2021)
Trafo
VE-Re-Nect101-32x/1 Xie at (2017)
CNN
84.17
194.03
ResNext0 Xie et (2017)
CNN 84.29
88.79
Sehonn-SNNer14411 He et (2018)
CNN
81.23
115.09
SWIN-S3 Liu ct (2021b)
Trafo
83.93
71.13
Wide-ResNer50-255 Be (2016a)
CNN
81.46
TWINS peput Chu (2021)
Trafo
83.14
60.99
ResNec101 (2016a)
VOLO-D2
(2021)
ResNetV2-500 Ho cl. (2016b)
CNN
80.10
25.55
ResML P-36 Tousron et (2021)
MLP
79.77
44.69
ResNe134-v1b ot (20116a
CNN
74.59
21,80
DLA102 Yo al. (2018)
CNN
78.03
33.27
ResNevv2-00-dii He 6301
CNN
82,80
25.55
SWSI RReNextt0
(2021)
Mixrr-L16 Tobsikkin (2021)
VIT-P16 Dasovisskl- al. (2021)
Trafo 81.40
22.05
Mixer-B 16-miil Tolsikhin
(2021)
82.30
59.88
gMLP-S16 Liu et (2021a)
MLP
79.64
19:42
Mixer-B16 Talstkhin et al. (2021)
ML.P
76.61
59.88
COAT-lite Xu al. (2021)
Trařo
79.09
11.01
ResMLP-3. P- Tounron al. (2021)
MLP
79.77
44.69
MixNee and Chen (2019)
78.98
ResMLP-24
Tounron
(2021)
MLP
79.39
30.02
RegNety-000 006 Radosavovic et al. (2020)
CNN
75.25
6.06
RcsMLP-12
Touxron (2021)
MixNet Tan and Chen (2019)
MLP
76.00
4,13
ResMLP-22-dist Touvron et al, (2021)
ML.P
80.76
30.02
XCIT-nano-124 El-Nonby (2021)
Trafo
73.92
3:05
parameter search as random search over the parameter grid. Ultimately, we found a parameter setting using a high momentum of 0.9999 in combination with high interpolation frequency (every other iteration) and a learning rate of 0.01 with weight score 0.7 to work best on average. Unlike simple KL Distillation based transfer, a fixed hyperparameter combination now results in both positive transfer delta on average, and significantly increased number of teachers from which each student can learn from (c.f. Fig. 4a)
For our final proposed KL+DP transfer approach, we again conducted similar parameter search. However, unlike XE-KL+MCL transfer, the KL+DP approach does not introduce additional hyperparameters compared to the standard KL distillation based setup. Consequently, grid of E {1e-3, 1e-4], {0.5, 0.75, 0.9, 1} and T E {0.1, 1, 10). Note that while we ablated the use of an external cross-entropy classification loss, we found the best performance consistently come for x = 1 - by turning of the auxiliary classification objective. This provides strong evidence that external measures for training stability are longer required. Finally, across all remaining experiments, we utilize a learning rate 1e-4 and a temperature of 1. While more in-depth parameter searches could likely provide parameter combination that would improve the average success rate, we believe that results achieved in its current setting to offer sufficient proof-of-concept..
utilize
A.4 MODEL LISTS: LARGE-SCALE STUDIES ON STRATIFIED IMAGENET SUBSETS
Table 5 presents a comprehensive summary of the pretrained teacher and student models employed in our evaluation of various transfer techniques on the 10% subset of the ImageNet dataset ($5.1) I). These models were carefully chosen encompass diverse architecture families, demonstrate varying performance levels, and exhibit a range of model sizes. This selection allows us to thoroughly examine the efficacy of knowledge transfer methods in different scenarios and settings. Note that for the exploration of complementary context ($3) we leveraged an even broader set of 466 teacher-student pairs comprising of 301 individual pretrained models randomly drawn from the t timm Wightman (2019) library.
A.5 MODELS EVALUATED FULL IMAGENET
Table showcases the detailed specifications of the student and teacher models employed in our full-scale ImageNet experiments (refer Section 5.1). In the context of knowledge transfer from multiple teacher models ($4.3), we utilized the same set of teacher models in combination with a subset of student models.
18

---

Published as a conference paper at ICLR 2024
Table 6: Selection of student an teacher models used for the experiments on full ImageNet. The student models were selected to contain multiple architecture types and cover wide range of model sizes and performance levels.
Type
XCTT-brge-2f-pl El-Nouby et (2021)
Tralo
PTT-8 Hou et al. (2021)
Tralo
82.44
73.76
Type
PTF-XS Heo (2021)
Trafo
78.19
11.00
SWSL- N Xic al. (2017)
CNN
84.29
88.79
ScNet154
VOI
(2021)
Com-Next
(2022)
CNN
84.57
50.22
ResMLP-36 Touwron ot al (2021)
MLP
79.77
44.69
BesNaaV2-50-dii He (20166)
CNN
82.80
25.55
ConT-litemini Xu (2021)
Trfo
79.09
11.01
Mixer B16 16-miil Tolstikhin al. (2021)
MLP
82.30
59.88
ResMLP:22
(2021)
B EXTENDED EXPERIMENTAL RESULTS
In this section, present additional experimental results of our experiments conducted in Section 5.
B. ADDITIONAL EXPERIMENTS ON VARIANTS OF DISTILLATION-BASE KNOWLEDGE TRANSFER
In the following subsection, present supplementary experiments conducted enhance the perfor- mance of knowledge transfer variants for knowledge transfer among pretrained models.
Using cross-entropy plus distillation transfer objective. As alternative to the KL divergence used in Equation (1) we additionally investigated the potential of using cross-entropy loss between the soft targets of the teacher and the student model, similar to Hinton et al. (2015). However, our results showed no advantage in using cross-entropy loss over KL divergence. In fact, we observed an average transfer delta that was 1.2 percentage points lower when using cross-entropy loss compared to KL divergence where only the student model's linear layers are trained using KL divergence loss, but found no improvement in transfer performance.
set of 60 teacher-student pairs. We also explored the use of warmup epoch
Restricting the set of classes for computing the distillation-barl transfer loss. our sup- plementary experiments,
investigate the impact of limiting the distillation loss focus only on the top-10 or top-100 most probable classes. This approach aimed to address the challenge posed by the large number of classes the ImageNet dataset, specifically the potential bias to- wards matching the long tails of the soft target distributions. To evaluate this hypothesis, we compared the KI divergence between full soft targets and subsets of soft targets. By selecting the top-10 10 and top-100 most probable classes based on the teacher's predictions, we observed that some teacher-studen pairs exhibited higher divergence over all classes compared the selected subsets. This indicated the inffuence of classes with low prediction probabilities on the KL divergence. Motivated by these findings, we further exam- ined the impact of considering only the top- 10 or top-100 classes on the transfer perfor- mance. Across six teacher-student pairs, using the top-10 10 divergence resulted an average in- crease of 0.20 percentage points transfer delta. Moreover, we observed that the magnitude of improvements aligned with the differences be- tween the top- 10 and total KL divergence. Our findings suggest that limiting the divergence to selected classes can be advantageous when deal- ing with large number of classes, although the magnitude of improvements remains limited.
Knowlodge Transfer Success Rate [%)]
100
80
20
0-
XE-KL Distillation
CD Distillation
CRD Distillation
KL+DP Distillation
Figure 6: Share of teacher increasing student per- formance (success rate) for contrastive distillation (green) vs classificttonnguuiee distillation (bluc) and continual learning based KL+DP (orange).
Contrastive distillation for knowledge trans- fer between arbitrary models To understand how well contrastive distillation techniques are suited for knowledge transfer between arbitrary
19

---

Published as a conference paper at ICLR 2024
Trafo Student Models
CNN Student Models
MLP Student Models
400
10-
200
KL KL-Dist Transfer Knowladge
150
Number Student Parameters
2.0-
100
MLP
KL KL-Dist DP Trunsfer
Knowledge willgge Gain
200
15-
1.-
Knowledge Loss
Knowledge Loss
Knowledge Loss
Figure 7: Share of transferred knowledge (knowledge gain) visualized against the share of knowledge lost for vanilla KL distillation and our proposed KL+DP distillation approach. Student models are grouped by their respective architecture type. Each marker represents one teacher-student pair. The color of the markers represents the size of the student, while marker shapes determine the teacher architecture. The marker size visualizes the teacher's performance. Results showcase a clear benefit of KL+DP, moving most points to areas of positive knowledge transfer (above red diagonal).
pretrained models. we measure the average transfer success rate for both CD and CRD distillation transfer ($A.2), with results shown in Fig. 6. We leverage the same experimental setup on 10% ImageNet as for the other transfer approaches (see $5.1). The experimental results clearly show the contrastive distillation approaches to be unable to improve the student model for most teacher models. On closer examination of the results we can see that the contrastive distillation approaches result in similar levels of knowledge transfer from the teacher to the student, but appear to also incur much stronger overall overwriting, causing the student to lose large portions its previous knowledge. While very suitable for distillation to untrained students, this behaviour is unfortunately not well suited for knowledge transfer between already trained expert models.
B.2 EXTENDED RESULTS ON KNOWLEDGE TRANSFER BETWEEN PRETRAINED MODELS
For our knowledge transfer success rate experiments conducted in Section 5.1, we provide an extended and more detailed version for Figure 4a in Figure 7. Using scatterplot, relate the share of knowledge transferred to the student model (knowledge gain) versus the share of the students pretrained knowledge that is overwritten during the transfer process (knowledge loss). Each student model is denoted by a respective color choice associated with its parameter count. Symbol sizes and colors denote both family and performance the respective teacher models. The red line denotes an equal trade-off between knowledge gain and loss, with upper-diagonal entries indicating a positive knowledge transfer. Comparing the results of vanilla KL-Dist. transfer and the continual learning based KI+DP transfer, we see that a vast majority of points are pushed up the diagonal, allowing for transfer even from weaker models (small symbols, heavily scattered towards the lower diagonal area in the normal knowledge distillation approach). This behaviour also highlights that normal knowledge distillation approaches generally overwrite knowledge instead of augmenting, and is reflected in our correlation studies in Figure 4a.
Overall, these results simply extend the insights provided in the main part this work from a more detailed point of view, highlighting that continual learning treatment of the knowledge transfer problem can significantly raise the transfer success rate. However, we note that this more finegrained perspective provides better support the detrimental aspect of stronger visual inductive biases for general knowledge transfer, as we found CNN students to generally perform worst, even when leveraging KI.+ DP transfer.
20

---

Published as a conference paper at ICLR 2024
Table Knowledge Transfer results on full ImageNet, from four teacher to eight selected student models. The tables include the individual transfer deltas of all teacher-student pairs.
Arseet -L-Sist
Avort KI-Die.-DP (mnspp)
Teachers Students
SWSL. ResNexti01
Volo-12
RedMI.P%
CoaT lite-mii
SWSL. ResNest
Volo-122
RecMi.P16
CoaT mini
SWSL. ResNexti01
Velo-12
ReMMi P36
Coaff
lite mini
P16
0.95
1.40
0.55
0.58
0.93
0.90
0.36
0.42
1.29
1.85
0.57
0.52
PTB
1.16
1.42
0.24
-0.54
0.74
0.31
0.31
1:35
1:59
0.61
0.65
PTEXS
0.54
0.43
0.14
-0.71
0.37
0.23
0.51
0.51
0.29
0.08
S4Net1
0:38
-0.07
-0:20
0.22
0.13
0.38
ComNest
0.23
0.1
1.10
1.57
0.49
0.44
0.26
0.12
0.44
0.38
0.22
0.09
RecNetV2
0.34
0.11
0.23
0.56
0.32
0. 17
0.28
0.13
0.34
0.18
0.29
0.11
Miner B16
0.32
0.22
0.54
-1.00
0.31
0.22
0.11
0005
0.35
0.25
0.07
4002
RcsMLP:24
0.58
0.43
40.36
-0.26
0.57
0.45
0.10
0.20
0.57
0.35
0.10
0.13
Table 8: The table below shows the results of knowledge transfer with our proposed KL-Dist. + DP transfer approach on the full ImageNet. It includes two metrics that describe the changes in the positive and negative prediction flips, and extends the information provided in Table For each student, report the mean and standard deviation over all teacher models, which can be found in Table 6.
Students
Type
Acc.
Parum.
Agasst
Agene
XCTT-P16 (HI-Nouhy al., 2021)
Trafo
82.89
189.10
0.65 (+0.26)
-0.74 (+0.25)
-0.08 (++0.04)
P/T-B (Heo et al., 2021)
Trafo
82.44
73.76
0.55 (=0.25)
-0.64 (+0.28)
-0.08 (1:0.05)
PIT-XS (Heo ul., 2021)
Trafo
78.19
10.62
0.40 (20.12)
-0.45 (200.13)
(#:0.04)
ScNet154 (He
2018)
CNN
81.23
115.09
0.27 (=0.17)
-0.35 (10.114)
-0.07 (+0.04)
ConvNext (Liu et al., 2022)
CNN
84.57
50.22
0.33 (=0.14)
-0.37 (1:0.9))
-0.04 (4:0.06)
ResNerV2 (He et al., 201
CNN
82.80
25.55
0,23 (+0.08)
-0.35 (+0.09)
-0.13 (+00003)
Mix (Tolstikhin
2021)
MLP 82.30
59.88
0.15 (=0.13)
-0.16 (+0.08)
-0.01 (4:0.7)
ResMLP-24 (Touvron et al., 2021)
MLP 80.76
30.02
0.33 (#0.19)
-0.30 (=0.16)
+0.03 (10.04)
The following table shows the individual transfer deltas of the teacher-student pairs from Table and Table 3.
To further support our analysis in Section 5. we have provide additional results regarding the change in the share of positive and negative prediction flips during knowledge transfer. Positive prediction flips poss refer to cases where the teacher was correct, but the student was incorrect. In contrast, negative prediction flips pue refer cases where the teacher was incorrect, but the student was correct. To measure this change, we defined two new metrics, pos-flips delta Apros and neg-flips delta Arres similar the transfer delta. We present the mean and standard deviation for both metries for all student models using our KL+DP transfer approach in Table 8, extending the results from Table
Our goal with knowledge transfer is to transfer complementary knowledge, i.c.. the positive prediction flips. This means that the number samples where the teacher is correct but the student is incorrect should decrease as much as possible. However, we must simultaneously preserve the student's previous knowledge. As a result, the number of samples where the student correct and the teacher is incorrect (negative prediction flips) should not decrease.
The experimental results conclusively demonstrate the effectiveness of our approach reducing the share of positive prediction flips for all student models. This underlines the capability of our approach to transfer complementary knowledge between models. Morcover, the minor changes in the negative prediction flips provide compelling evidence of the approach's 's ability preserve the student's s previous knowledge.
B.3 EXTENDED RESULTS ON THE IMPACT OF DIFFERENT STUDENT MODEL PROPERTIES ON KNOWLEDGE TRANSFER
In this section, we provide a closer assessment of the impact of the student model properties on the knowledge transfer behaviour, as measured through the transfer delta. In particular, we look at performance, size (measured by the number of parameters) and model family. For this assessment, we selected for each model property pairs triplets of students with similar values for two of the three properties to isolate each single variable as well as possible. While an exact intervention can not be made by simply leveraging pretrained models, this setup does provide more controlled insights, which we visualize in Figure 8 for experiments conducted on 10% ImageNet using the KL+DP transfer approach.
21

---

Published as a conference paper at ICLR 2024
Knowledge Transfer Delta
a) Student Performance
b) Student Size
c) Student Architecture
1.3
0.5
1.0-
0.0
Tralo
0.0
0.5
CNN
MLP
-2
75.0
71.5
80.0
82.5
85.0
10
20
40
80 120 200
Small
Medium
Large
XLarge
Student Accuracy
Student Parameters
Student Model Size
Figure 8: Evaluation of the impact of the student model properties a) performance, b) size (measured by the number of parameters) and c) architecture type on the knowledge transfer delta. Each marker represents a selected student model distilled with 20 different teacher models. We group students into pairs or triplets based the remaining model properties by connecting the respective markers.
Note that each marker represents one evaluated student model with all 20 teacher models. We connect the pairs or triples of students that can be compared, with the color of the lines and markers representing the model family of the student.
Our results replicate insights noted in the main part of this work, particularly Figure 5 (right). We find that even when controlling for other factors such as initial accuracy, the overall student capacity appears strongly correlated with the ability receive new knowledge without overwriting previous. This is particularly pronounced behavior in models with strong visual inductive bias such as CNNs. The rightmost subfigure showcases that when looking at the average behavior of a model family (divided into different model sizes), that scale can offer emergent transfer capabilities in CNNs while not available before for any type specific architecture increased sizes can allow for notably improved transferability.
B.4 EXTENDED RESULTS ON ADDITIONAL DATASETS
To substantiate results ImageNet additionally conduct experiments the CUB200 Wah et al. (2011), Caltech256 Griffin et al. (2007), and Stanford- Krause et al. (2013) datasets.
For each datasets combine the nine student and four teacher models shown in Table 6 resulting in a total of 36 teacher-student combination. We fine-tune the classification layer of the student and teacher models using dataset-specific data before initiating knowledge transfer. We employ the dataset's 's training data as the transfer set.
2.5
2.5
2.5-
0.6
0.7
-2.5
Krowledge
2.5-
KL-thl Trankr
KL-tlei Trader
Ki-chll Trankr
33.0
Me-LLbbw MLL Trenkr
Xi-kLibit 5K3. Teade
3.0
ths MCL Trewkr
KI DD Trands
KL Ki. Trande
-7.5
-10
0
10
-7.5
-00
6 Porforminc Differoce Teachor Staten
-15
-10
-75
20 Performanc Diffoomcce Toccerr and Strdeet
-20
Performance (ifffrence Tooheer and Student
(a) CUB200
(b) Caltech256
(c) Stanford Cars
Figure 9: Knowledge transfer delta based teacher-student performance difference for three additional datasets: a) CUB200, b) Caltech256, and c) Stanford Cars. We compare simple KL-Dist. transfer with TE-KL-Dist.++e +MCL transfer and KL-Dist. +DP Transfer. The teacher-student pairs categorized into bins determined by equipartitions of their respective performance differences. To mitigate the influence of outliers, report the mean transfer delta of the top 25% within each bin and approach.
Across all datasets, we consistently observe the KL-Dist. + +DP transfer approach to not only enable the transfer of knowledge from less proficient teachers without compromising student performance but to also demonstrate the capacity to transfer substantial knowledge portions in cases where the teacher outperforms the student significantly, aligning with the effectiveness of the straightforward KL-Dist. transfer. These results are in line with our observations on ImageNet (c.f. Figure 4b) and underline the strengths of KL+DP transfer.
22

---

Published as a conference paper at ICLR 2024
B.5 EXTENDED RESULTS ON KNOWLEDGE TRANSFER UNDER DOMAIN SHIFTS
We further explore knowledge transfer in the setting of domain shift between the teacher and student model. For this purpose we fine tune the teacher model on the domainnet infograph dataset Peng et al.
(2019) before conducting knowledge transfer. The transfer process is executed on the 10% subset of ImageNet. Our comprehensive assessment encompasses a cohort of 9 distinct student models and teacher models (see Table 6).
Knowledge Transfer Delta
0.0
-2.5
-5.0
KL-Dist Transfer
XE-KI-Dist MCL. Transfer
10.0
KisDist DP Transfer
30
-20
-10
Performance Difference of Teacher and Student
Figure 10: Inter-domain knowledge transfer delta analysis for KL-Dist. and KL-Dist. + +DP transfer. We investigate the transfer delta resulting from knowledge transfer from teacher model trained on DomainNet Infograph to an ImageNet-pretrained student model.
Notably, our findings underscore the efficacy of the KL-Dist, +P transfer approach, which facilitates the transfer of knowledge from the Infograph-trainee teacher to the student model on the ImageNet domain, thereby improving the student's performance. In stark contrast, the conventional KL-Dist. transfer demonstrates a substantial decrease in student accuracy, particularly when using a less proficient teacher.
B.6 EXTENDED RESULTS ON THE TRANSFER FROM MULTIPLE TEACHERS
Finally, we present additional insights on the sequential knowledge transfer from multiple teacher models to a single pretrained stu- dent model. For all multi-teache knowledge transfer experiments we select three student models (XCYT-P16, Twins, PiT-B) and three teacher models (SWSL-ResNex1011 VOLO- D2, ResMLP-36) from Tab. 6. Appendix B.6 visualizes the knowledge transfer (knowledge gain), the share the student's 's pretrain knowl- edge heing lost (knowledge loss) and the overall transfer delta over the transfer epochs for the PiT-B Heo al. (2021) student model presented in $5.2. As noted there, we distill the student with three different teacher models (see Table 6). For this particular visualization, we order teach- ers by ascending performance, but find positive continual transfer also to be achievable from other sequences. For each teacher, we allocate fixed transfer budget of 20 epochs. As noted already in Table 2, the figure visually highlights that positive transfer deltas can be gained going from one teacher to the subsequent one (stronger transfer delta compared to the strongest single student, Auia 1.04), but with returns diminishing. We attribute this the increased rate of forgetting while knowledge gain steadily rising, continuously moving the student from its initial pretraining weights induces increasingly stronger knowledge loss, even when leveraging KI+DP transfer.
Knowledge Gain
Knowlofge Gaim/ Loss [%]]
3.0
Knowledge
Knowlodge Transfer Delta
2.5.
2.0
1.5-
1.0.
ResMLP-36
ReSS-R
VOLO-D2
0.5
20
30
40
50
60
Transfer Epoch
Figure 11: Knowledge transfer (knowledge gain) and loss of the student previous knowledge (knowl- edge loss) during the sequential training of PiT-B Heo al. (2021) with three different teacher mod- els sorted by ascending performance.
For further insights, we compare the results of our multi-teacher experiments using KL-Dist. +DP transfer to vanilla Kl.-Dist. transfer (Tab. 9). The results clearly show that sequential KI.-Dist.
23

---

Published as a conference paper at ICLR 2024
Table Knowledge transfer from multiple teachers into a pretrained student using sequential, and soup-based vanilla KL-Dist. transfer (c.f. $4.3). We compare with transfer deltas obtained from the single teacher knowledge transfer.
Ammet
Single
Teacher
Aguet. Sequ.
Multiple Teachers
Students
Type
Acc.
Param,
Mcan
Min
Max
Soup
XCTT-P16 El-Nouby al. (2021)
Transf.
82.89
189.1
0.47
-0.85
1.31
0.48
0.89
Twins Chu et al. (2021)
Transf. 83.68
99.27
-0.04
-1.04
0.63
0.01
0.43
PiT-B Heo (2021)
Transf.
82.44
73.76
0.69
-0.24
1.39
0.80
1.19
transfer cannot achieve larger gains as the best teacher alone but results in performance gains in the range of the average transfer delta across the three teachers. This again shows that rather than transferring only the complementary knowledge vanilla KL-Dist. transfer overwrites the student's previous knowledge with the knowledge of the teacher model. Thus when sequentially transferring knowledge from multiple teachers improvements from the previous transfer are lost during transfer from the subsequent teacher. Note that the vanilla KL-Dist. fransfer approach cannot be directly applied to transfer knowledge from multiple teacher models parallel, hence we omit this baseline.
24


Reviews:
[{"review_id": "k9d69AXIWO", "reviewer": "Reviewer_LBqG", "summary": "This paper investigates if it is possible to transfer complementary knowledge from one model to another without performance degradation. To this end, authors propose different heuristics to design how to switch the knowledge transfer between models. Experiments with various pairs of models demonstrate the effectiveness of the model-agnostic knowledge transfer.", "questions": "See the weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The writing is clear and easy to follow.\n\n2. There are consistent performance improvements compared to different types of baselines.", "weaknesses": "1. As addressed by the authors, different models are trained with different data augmentations, architectures and optimization techniques. The performance improvements are relatively marginal (e.g., Table 1), especially considering some models are not fully trained.\n\n2. In Table 4, do the authors train all the variations with the same number of training steps? The sequential knowledge transfer may benefit from more training steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1zrM6BHhkF", "reviewer": "Reviewer_epvt", "summary": "This paper investigates the phenomenon that different models have complementary information, reflected in their different predictions on each sample. Such complementary information could be due to model architectures, training settings, etc. Then the authors study how to transfer the complementary knowledge from a teacher model to a student model. The authors formulate this as a continual learning problem, which effectively improves upon the knowledge distillation baseline  and achieves better knowledge distillation on diverse models and ImageNet.", "questions": "1. Explain more about the terms. The readers are likely to have an ambiguous understanding of them, including: transfer delta $\\Delta_{transf}$, \"available complementary knowledge per class,\" and transfer rate.\n\n2. I am wondering if the methods have to reason from the \"continual learning\" perspective. In my opinion, the regularization techniques proposed by the authors seem like generally applicable tricks for knowledge distillation. If any procedure involving multiple steps has to be treated as continual learning, maybe training a neural network with SGD is also a continual learning process? I hope the authors can clarify this motivation better in the paper.\n\n3. See the third weakness above.\n\n4. I suggest the author add an oracle study to strengthen the argument. In the examples (e.g. Table 1), the final improvement seems small in scale. To argue that this is actually challenging, the authors can run several ensembles of the teacher-student model and compare it to the improvement from knowledge transfer. Of course, I welcome other variants of similar analytical studies from the authors.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper identifies the \"complementary knowledge\" in different neural networks with grounded evidence. I think the claims are reasonable and well-supported.\n\n2. The authors proposed improvement to the general knowledge distillation approaches from the continual learning perspective, including constraining the weight updates and transfer data. Both of the approaches are reasonable and straightforward to apply.\n\n3. The authors conduct experiments on a wide range of models on ImageNet and show improvement with their improved continual knowledge distillation approach.", "weaknesses": "1. Present the key terms more clearly. For example, I haven't found the definition of transfer delta $\\Delta_{transf}$, which is an important evaluation metric.\n\n2. I think the proposed regularization, including the constraining of weight updates and transfer data, are some tricks for knowledge distillation. Although I don't work directly in knowledge distillation and I will wait for other expert reviewers to justify the novelty, I think the authors need to clarify more about their motivation or form a more explicit connection with continual learning. I also raised a question in the next section, elaborating my concerns.\n\n3. I suggest the authors add some more experiments to make the arguments more thorough. Specifically, a more detailed breakdown of the accuracy numbers would help. For instance, (a) the percentage of [teacher correct, student wrong] samples are changed to correct answers after the distillation, (b) the percentage of [teacher incorrect, student correct] samples are changed to incorrect labels, to understand if the transfer is indeed complementary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H4fQ2LLjaV", "reviewer": "Reviewer_QCZm", "summary": "This paper studies general knowledge distillation (KD), where, given any two models, the student can infer missing information from the teacher model and potentially improve the performance. The authors begin with a comprehensive analysis showing that such complementary knowledge generally exists in any paired models regardless of model capacity or architecture, and existing KD methods cannot leverage the information that student models already carry, i.e., trained students. To this end, the authors propose a continual learning-based extension to the existing KD methods and a data partitioning scheme that, according to the highest prediction probability., simultaneously maintains the useful knowledge of students while learning from the teacher. The extensive experiments conducted on more than 400 models sufficiently verify the effectiveness and provide many insightful analyses.", "questions": "1. What would happen if one applies the proposed method to weaker models, e.g., randomly initialized? I guess it will be degenerated to conventional KD methods. A specific section for weaker/smaller models would be interesting to the community of edge device users/researchers.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The problem studied in this paper, i.e., general knowledge distillation, is interesting and practical. It bridges the gap in the existing literature that a trained student might degrade during the knowledge distillation process.\n\n2. The additional analysis also indicates almost every model could benefit from the other teacher models, even if the teachers are weaker than the students. The result and the evaluation methodology may motivate the community for further research.\n\n3. The proposed method is sound yet easy to implement in practice. The authors also consider different scenarios of distillation, including a single teacher and multiple teachers distilled in different ways/orders.\n\n4. The large-scale analysis (over 400 models) validates the claim and provides various insights, such as the properties of student models.", "weaknesses": "1. The student models considered in this paper are relatively strong. As the authors claim **general** knowledge distillation, the readers will also be interested in the weaker models or even from scratch. However, the authors only consider powerful architectures, such as transformers or ResNet, in the paper.\n\n2. The proposed method is slightly confusing to me. My understanding is that the proposed data partition is built upon the continual learning method since Figure 4(a) clusters KL-Dist + DP Transfer into continual learning. If so, the contribution of each component is not clear enough to me. Though some experiments, e.g., Figure 4(b), present the partition improves the MCL transfer, the contribution of the partition itself is not investigated in the experiments.\n\n3 (Minor) Consistency of italic type. Some \"KL+DP\" are italics in the text, while some are not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "piHByz2CZJ", "reviewer": "Reviewer_yxq3", "summary": "The authors provide an empirical study of the ability to transfer complementary knowledge between different pretrained models without performance degradation. This paper analyzes existing approaches in knowledge distillation and find it insufficient, especially in the case of distilling specific information from weaker teacher models. They go on to propose a data partitioning-based method (into regions of desired teacher behavior and desired student behavior retention) to achieve complementary knowledge transfer between the pretrained models considered in this paper.", "questions": "1. Do you have any intuitions as to why student models that are CNNs exhibit better transfer at scale (while other architectures do not)? (Figure 8 in Supplement)\n\n2. In Table 3, unsupervised DP outperforms supervised DP on several tasks. This seems a bit surprising; in the cases where these methods would be different, your DP approach would be distilling information from the teacher model on instances where the model is both quite confident and incorrect. Do you have an ideas about how this would be beneficial, and does this match your intuitions as to why this method works in general?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper empirically shows that complementary knowledge exists between a large suite of models defined by different architectures and sizes (and even in weaker models that are less well performant than other stronger models). This complementary knowledge is localized to particular classes (what the authors deem as relative areas of expertise)\n\n2. The authors propose a new data-partitioning approach to transfer complementary knowledge, where data is partitioned by which model has a higher probability of the ground truth class (or chosen simply by maximum probability in an unsupervised case).\n\n3. Extensive experimental results that demonstrate that the proposed distillation approach transfers at a higher rate and transfers complementary knowledge from weaker teacher models.\n\n4. The paper also studies different properties of student models that better allow for knowledge transfer.", "weaknesses": "Overall, I think the paper is quite comprehensive. A few points that may be lacking:\n\n1. The results in studying properties of student models is a bit surprising to me. This isn’t a huge weakness, but more exploration of why CNN student models improve with scale and why transformer student models seem to worsen with would strengthen these results.\n\n2. The data partitioning heuristic is reasonable, but some ablations on this approach would be more enlightening. Perhaps in some instances, the student model may be overconfident about particular data points (that either have incorrect labels or are inherently difficult examples to classify), and this data partitioning approach would maintain this overconfidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)