Representation Consolidation from Multiple Expert Teachers
Authors: Zhizhong Li
Avinash Ravichandran
Charless Fowlkes
Marzia Polito
Rahul Bhotika
Stefano Soatto
Venue: ICLR 2022 Submitted (2022)
Keywords: transfer learning
distillation
pretraining
model merging

Abstract: A library of diverse expert models transfers better to a novel task than a single generalist model. However, growing such a library indefinitely is impractical. Hence, we explore the problem of learning a consolidated image feature representation from a collection of related task-specific teachers that transfer well on novel recognition tasks. This differs from traditional knowledge distillation in which a student model is trained to emulate the input/output functionality of a teacher.  Indeed, we observe experimentally that standard distillation of task-specific teachers, or using these teacher representations directly,  **reduces** downstream transferability compared to a task-agnostic generalist model. We show that a simple multi-head, multi-task distillation method using an unlabeled proxy dataset and adding a generalist teacher is sufficient to consolidate representations from task-specific teacher(s). We improve downstream performance, outperforming the teacher (or best of all teachers) as well as the strong baseline of ImageNet pre-trained features. Our method almost reaches the performance of a multi-task joint training oracle, reaping the benefit of the teachers without replaying their training data.

Under review as a conference paper at ICLR 2022
REPRESENTATION CONSOLIDATION FROM MULTIPLE EXPERT TEACHERS
Anonymous authors Paper under double-blind review
ABSTRACT
A library of diverse expert models transfers better to novel task than single generalist model. However, growing such a library indefinitely is impractical. Hence, we explore the problem of learning consolidated image feature represen- tation from a collection of related task-specific teachers that transfer well on novel recognition tasks. This differs from traditional knowledge distillation in which a student model is trained to emulate the input/output functionality of a teacher. Indeed, we observe experimentally that standard distillation of task spee-iie teach- ers, or using these teacher representations directly, reduces downstream transfer- ability compared
task-agnostic generalist model. We show that simple multi- head, multi-task distillation method using an unlabeled proxy dataset and adding generalist teacher is sufficient to consolidate representations from task-specific teacher(s). We improve downstream performance, outperforming the teacher (or best of all teachers) well as the strong baseline of ImageNet pre-trained fea- tures. Our method almost reaches the performance of a multi-task joint training oracle, reaping the benefit of the teachers without replaying their training data.
INTRODUCTION
A promising approach to scale transfer learning to diverse downstream vision tasks is to maintain library of diverse experts pre-trained different tasks. (Deshpande et al. 2021 When presented with a novel downstream task, one can select an appropriate expert and fine-tune the representation with small amounts of task-specific data. This strategy has many practical benefits: fine-tuning task-relevant pre-trained representation is fast, there is no need to store or revisit expert pre-training data. How is such a library of expert representations populated and maintained? Previous work (e.g., Puigcerrvr et al. (2020): AAchilee et al.] (2019): Deshpande et al. (2021)) has assumed a static datasets domain-spectioo
collection of experts created by training on general dataset. Instead, we would like to automatically enrich the diversity of the expert library by accumulating knowledge from transferred downstream tasks, that the overall system performance continually increases over time (life-long meta-learning)
subsets of
One approach to growing the library would be add fine-tuned downstream task-specific models back into the library as a candidate expert for transfer on future tasks. Such a naive approach clearly does not scale and, at minimum, requires developing techniques for selecting experts that sub-linear in the size of the library (Puigcerver et al., 2020 Achille et al.) [219]] Deshpande et al.] 2021). More importantly, as experiments show in Sec. task-specific models fine-tuned on small amounts of data do not provide transferable representations. Task-specific models tend to overspecialize and degrade under-utilized features in their representations, and thus under-perform on new tasks compared to generic pre-trained models.
To address this, introduce representation consolidation where the goal is consolidate knowl- edge from multiple task-specific teacher models into a single expert student representation which transfers to downstream tasks better than any of the individual teacher representations. There three factors that make the problem of representation consolidation unique relative to the existing literature: (1) we assume a representation is to be consolidated from collection of multiple task- specific models, (2) we avoid the need to revisit task-specific training data these models, and (3) focus on improving downstream transfer instead of simply replicating performance the up- stream teacher's tas)(s). To the best of our knowledge, no prior work has analyzed the downstream transfer aspect of distilled domain-expert representations.

---

Under review as a conference paper at ICLR 2022
Carril
enstab deplity
Gaaull
and lald
vcd Eabe
lcadic
Fnicher
ITAkiE
(prity) datseci
(pecty) dataset
Traitional (prsy)) distilaaionn
Traditioal (proxy) distillation multi-lask
endlane & Aeptes
coaloao Degley
adi
saaterr
Tiue Tine
tcachh:
fiserzza
sayp!
(peosy) dataset
tureet danaset
(pecae) denset
Seget daraset
Traditional (proxy) multi-task disiil usod for transfer
(d) Reprosentariin consolidaaiio (ouns)
more Knowledge distillation (a.b) seeks to copy the end-to-end functionality of teachers but often fails to transfer (c). We propose simple yet effective method to learn a consol- idated representation from N teachers that transfers well to downstream tasks (d). Given large unlabeled proxy dataset, we train a single student model using multi-task distillation with a separate classifier head for each of the teacher tasks. To limit student forgetting and representation collapse, we always include an additional generalist teacher (ImageNet model). We show that the resulting consolidated representation transfers better to downstream tasks than any of the individual teachers (including the generalist).
Figure 1:
To earry out representation consolidation, utilize multi-teacher multi-task model distillation (see Fig. T]. We use a simple yet effective method to jointly distill one or several task-specific teacher with a generalist one. Each teacher may operate a different set of classes, and multi-head student is trained to emulate all teachers. Previous work on knowledge distillation has focused on the student model's performance on the teacher's task. Instead, evaluate how well the student representation generalizes to new downstream tasks (whether related or unrelated to the teachers' tasks). In this setting we demonstrate several surprising results:
Task-specific model representations often transfer poorly even to related downstream task.
Consolidating task-specific teacher with generalist teacher (ImageNet) is sufficient to rescue the student. The resulting representation matches or improves the downstream performance over both task-specific and generalist representations, getting the best of both worlds even though no taec-specific data is replayed.
Consolidating multiple related task-specific teacher models can yield better student rep- resentation that exceeds the performance of any one teacher on downstream tasks.
Consolidation performs similarly to multi-task joint training oracle, therefore gaining almost all of the transfer benefit from the teacher's dataset without needing to replay it.
Symbol
Definition
Symbol
Definition
The number tasks task specific tenchers
The downstream tack's dataset
D.
The i-th task-specific teacher's dataset
The consolidated student backbone fine tuned on dataset D)
i-th task-specific teacher's backbone
The downstream model's st's head (chasilier layer linear
(i O: the generalist's backbone)
22 Loss weight for the i-th teacher
task specific teacher's
(classifier layer)
A 1-split d-split
O: the generalist's head)
First 50% rundom split of classes dataset, used Doj
D
The large unlabeled proxy datasee used for
Second 50% random split of classes of dataseet used 12/
disii deeilltionkoomoolddeee
The fistiliefoonsolidate student's backhone
distilledcomselidatedd stadent's (classifier)
Table 1: Glossary of symbols.
2

---

Under review as a conference paper at ICLR 2022
2 REPRESENTATION CONSOLIDATION
Problem statement. We start with a collection of one or more task-specific image classification models {M; fi=1 teachers, trained on corresponding datasets { belonging some domain (satellite images, images of flowers, etc.). We assume models consist of a feature extractor or back- bone (f()) composed with classifier head (+(-) so that Mi = (-)) We consolidate the knowl- edge of these task-specific teachers into a single student representation (.(-) using a proxy dataset D Drroxy (e.g., ImageNet), and then evaluate & by training a SVM on top of it (or fine-tuning it) on a given downstream D/ chosen from some set {D)}- Our goal is that the resulting downstream model (-)) achieves good performance, where o' denotes the student 6, as-is or potentially after tuning on Dj- Fig. highlights how this differs from standard distillation in which the student model h4 (-))--) is simply evaluated on the same task its teachers were once trained to perform.
Forgetting and representation collapse during distillation. Though our problem statement is novel, can existing approaches solve our problem statement? We observe (Sec. 4 that neither the task-specific teachers nor students distilled with standard or state-of-the-arr methods provide suffi ciently transferable representations. They under-perform general pre-training representations (e.g. with ImageNet) when evaluated on downstream tasks from the teachers' domain, and drastically under-perform on tasks outside the teachers domain. Surprisingly, this holds true even though we use Image -eretrrraid weights to initialize the teacher and student networks. Thus, we argue that training task- specifcc teachers or distilling from them suffers from catastrophic forgetting of general knowledge that is crucial for transfer learning.
Intuitively, transfer performance depends on how distinguishable different classes are represented in the penultimate layer feature space. In standard distillation, the student only learns from task- specific teachers trained on smaller datasets and is only required to discriminate the classes in those datasets. This well suited for the traditional distillation problem that evaluates the student on these same tasks. But for representation consolidation, distinguishing unknown downstream classes requires preserving general features that may not relevant to the teachers' specific task. Our strategy is thus to ensure that the student maintains general features as it learns task-specific ones.
Method formulation. A key challenge to our approach is avoiding replay of the images used to train the task-specific teachers. To overcome this, propose using a proxy dataset and generalist model that transfers well (e.g. one pre-trained on ImageNet). This model is referred to as generalist as it is not task specific and contains wide variety of classes, and use it to help avoid forgetting. More specifically, we use multi-head multi-task distillation. We put N + 1 heads, hg. ........................ top of the student backbone & (-). In addition to the task-specific teachers (f(( used in traditional distillation, we also include the generalist teacher trained on ImageNet (denoted as (xf The rationale behind this construction is that learn task knowledge from the task-specific model on the proxy dataset, but also force the student to retain knowledge from the generalist teacher to prevent over-specialization. We learn the student by optimizing the loss:
........"" A, on
{1, (1.....N)
C= Σ RED
No Cassiu ( (1)(0) (2(2)
Ag Caisun (h)(ot())), (2()
(1)
poay
generalist teacher term
traditional distillation
with distillation loss Ldistil- Our key difference from traditional distillation the generalist teacher term. Setting λo = 0 while keeping all other conditions (proxy dataset, network initialization, etc.) yields a standard multi-teacher distillation baseline. Note that this is also distinct from ensemble knowledge distillation, where one averages the homogeneous output of every model and distills the mean onto single student head, since our teacher outputs are heterogeneous (different tasks).
Our method is agnostic to the underlying distillation method. We can use the standard knowledge distillation (KD) loss (Hinton et al. 2015)) which is cross-entropy with temperature
T = 2, i.e. Laisuu (pr, Ps) - -E Pt (c) log(p! (0) where indexes the C classes, and pt =
softmax (N((t(()))/T) Ps softmax (o,(o, (x: (T). Alternatively, we can use a state-of-the-
art distillation method, e.g., CRD (Tian et al. 2020). CRD uses the KD loss and adds a contrastive embedding loss between heRD, (z)( and MERD, (x)) with a pair of trainable linear embedding layers M.RD. and heDD, that are used only in training and discarded afterwards. The teacher dt's
3

---

Under review as a conference paper at ICLR 2022
trained on completely different tasks, so for CRD different embedding layers for each paacher-student-beed pair.
We initialize the student backbone o, and its O-th head hg using the generalist model's weights o.. n4. whereas other heads are randomly initialized. Since it is important to maintain pre-trained model's representational power, we simply set the loss weights Àu
and X; v for < i I ... This forces the learned representation O to balance between learning general and task- specific features, which benefits future downstream transfer. After training the student we evaluate the resulting representation (-(-) on multiple downstream tasks {Dj}. J. For each task j we can either fine-tune the whole model (g(ddd (-) keep the student representation fixed and only learn the classifier head h2 (dx())) which is often referred to as a linear probe.
3 EXPERIMENTAL SETUP
Datasets and downstream tasks. We utilize datasets from a variety of domains to generate teach-
and downstream tasks: Cars196 (Krause et al. 2013). Resisc45 (Cheng al. 2017) (remote
sensing images), iFood (Kaur et al. 2019) and Food 101 (Bossard et 2014)) iFashion (Guo et al.) 2019), DTD (Cimpoi al. (214)) (describable textures), Naturallst (Horn et al. 2018) (species clas- sification, 2019 challenge version). CUB Birds (Wah et al. 2011.. Flowers (Nilsback & Zisserman 2008), Caltech256 (Griffin et al. 2007), and Airerafts (Maji et al. 2013). Among these, iFood and Food101 are the same domain, and Birds and Flowers are subdomains of iNaturalist. We checked for near-duplicates between these datasets using perceptual hash (Buchner) 2021), and found negli- gible duplication: 1 out of 1 134k iNaturalist (50% classes we used) is duplicate of CUB, and 8 out of 130k/100k images are duplicates between iFood and Food101.
To evaluate a consolidated student representation has learned features relevant to a specific do- main, require downstream tasks that are related (in the same domain) to that of each task specifcc teacher. Except for Food101 and iFashion, we split each dataset at random into two disjoint sets which each contain only 50% of the classes. We take the first half of the dataset, named "t-split", and use as Di, to train a task-specific teacher. We use the second half of each dataset, named "d- split", as one of the downstream tasks D/- We primarily evaluate on few-shot downstream transfer, where we randomly sample 5 training images from each available class in Dj, but always use the entire test set for evaluation. We use all samples for those classes with < 5 images. For iFashion, which is multi-task multi-label dataset, use all images but with subset of labels (e.g., those related to clothing category) as Dj for teacher training. We then use all images with a disjoint set of labels (e.g., those related to sleeve style) as a downstream task Dj for evaluating transfer. For iFashion's 's few-shot scenario, we randomly subsample 1000 images for downstream training, and evaluate on all test data. We do not train any teacher on Food101 so the complete set of classes are used as downstream task.
During distillation and consolidation, we use ImageNet (Russakovsky et al. 2015) (ILSVRCI2) as the proxy dataset D prxx for most experiments. However, we also show results with Places365- standard (Zhou et al. 2018) or iNaturalist (Horn et al. 2018) as the proxy.
Compared methods and Criteria. As a reminder, our goal is not to duplicate upstream teacher predictions, but rather to improve transfer performance of distilled representations. Therefore, we evaluate each student's 's o, using its performance when transferred to various downstream datasets (D)) and compare multiple methods for initializing the downstream representation dg:
(1) Imreennet-reeraiieed of, a strong baseline for transfer learning. (2) ImageNet-pretrained of fine-tuned on the soft-labels produced by the pretrinet-rettraieeed model with batchnorms in test mode. This baseline aims to isolate the effect of soft-labels / self-distillation on model performance.
(3) The task-speeific teacher of (or one of the teachers when N > 1) without further distillation. (4) Distillation with N teachers using either standard knowledge distillation (KD) or contrastive representation distillation (CRD). This distills only task-specific teacher(s) on D Pproxy. without the ImageNet teacher hn (d)())) (5) Our consolidated representation di which includes (df (-)) teacher, using either KD or CRD as the underlying distillation loss. (6) A multi-task learning oracle (MTL) that jointly trains ImageNet and teacher data D, rather than unlabeled proxy data.
4

---

Under review as a conference paper at ICLR 2022
Datasct
IFood (t-split)
iFood (d-split)
Food101
Resise45
Dataset
Resisc4S (tsplit)
Resise45 (d-split)
iFood
Food101
(t-split)
(d-spla))
5-shot
5-shot
5-shat
ovaluated head
by b
SVM
SVM hu
SVM hu
evaluated head
R or
SVM
SVM hl
SVM hg
ImageNet pre-train
28.9
70.7
Image pre-train
70.7
37.0
Teacher (t-split)
74.4
34.8
42.8
53.3
Teacher (t-split)
98.2
67.9
14.8
17.9
(trad. distill)
35.3
KD (rna. distill)
97.9
Ours + KD
68.6
38.8
47.2
69.5
Ours + KD
97.0
72.6
28.8
36.4
MTL (oraclo)
71.6
38.9
69.4
(oracle)
73.1
36.2
(a) iFood as task-specific teacher datasct
(b) Resisc45 as task-specific teacher dataset
Table 2: Exp. 0: Accuracies of baselines and representation consolidation, judged with tra- ditional criteria (first column, original network head's old task performance) and transfer learning criteria (last 3 columns, performance of linear SVM head trained on related (bold) & unrelated downstream tasks). Baselines work well for the original task, but underperform in transfer learning. Ours matches or outperforms the best of all baselines in all transfer scenarios and matches the oracle.
For comparison fairness, (2-6) are all ImageNet pre-trained (i.e. initialized with (1)) before further training. Both (4) and (5) are using the same proxy dataset (i.e. ImageNet for most experiments).
We use the transfer accuracy downstream tasks DJ measure each representation's power. We primarily use the few-shot linear probe (train a 5-shot linear SVM as h2 over fixed () on D training set (single training run for full dataset, few-shot performance averaged over 50 random subsampling trials). We also verify our results hold when fine-tuning the student representation (+)())) on sho-shot or full dataset splits.
Implementation details. We will release code to reproduce this paper upon acceptance. For experimental fairness, we use the same network (ResNet50 (He et al., 2016)) hyperparameters / ImageNet initialization for the baselines and ours. For more details, please appendix A
4 RESULTS
Exp. 0: Motivational analysis traditional distill vs. representation consolidation To high- light the difference between representation consolidation and traditional distillation, we use either iFood or Resisc45 (t-split) as 'DI to train teachers, and run distillationcconsoliaationn with 1. We test on iFood (d-split), Food101 (full), and Resisc45 (d-split) as downstream tasks (t-split and d-split are disjoint 50% classes of each dataset; see Section 3
On downstream tasks Dj we follow representation learning's evaluation protocol (few-shot linear probe): train linear SVM ha on Φ(z), x E Dj. evaluate on DJ's test set. We also evaluate each network following the traditional distillation evaluation protocol, i.e. directly evaluate (o) (-)) or h (o.(-)) on the upstream task D 's test set. The results shown in Table 2
If we only focus on the upstream task D then traditional (proxy) distillation almost matches the teacher's performance, and representation consolidation performs worse than both. However, if instead focus the downstream transfer performance DJ, we the opposite trend. In Table 2a with iFood representations, for the teacher-related downstream tasks Food101 and iFood d-split (the disjoint classes split from the same dataset DH), we see benefit of consolidation over both task- specific teacher and generic pre-trained features. On the unrelated downstream task (Resisc45), where the generalist teacher excels, notice that method almost matches the generalist perfor- mance even though we improve over for the food tasks. We see a similar trend in Table 2b) the consolidated student outperforms the teacher and the generalist the teacher-related task (Resise45 d-split) but still retains the generalist teacher's performance on unrelated food tasks. This clearly demonstrates the significant difference between upstream and downstream transfer.
We perform nearly the same as the MTL oracle which trains also on the teacher datasets D,, indicat- ing that while avoid replaying D., simple consolidation method with Dprosy can extract nearly all of its benefit. Note that although we have carefully optimized the MTL oracle for downstream performance, adding state-of- ofte----tthrrrr MTL techniques could outperform MTL. implementation.
5

---

Under review as a conference paper at ICLR 2022
Accuracy (rel. to maage/ett-retrainne model)
Haad
Foodtee
reluned unselated
rove glu
+ XD KD (mad)
KD ImsgeNet
10
0
0
KD (mad) Us ImsgeNet
0
10
ouTs + CRD v. ImageSer of CRD URD urad. CRD uead) d ve. Imagrier
0
10
0
10
0
0
2 0
0
10
Frn
(b) Tally of comparisons (representation conooldation (ours) vx. traditional distill vs. ImageNet pre-trained model) among all 10 teacher dataset D) scenarios.
(a) Excerpt of 10 scenarios). Full figure in appendix Fig.
Figure 2: Exp. with = 1, 5-shot linear SVM downstream transfer. Left: Comparing different representations ten downstream tasks. Teacher-related downstream tasks in bold. Performance relative to ImageNet representation baseline. Excerpt (2 of 10 teacher domain scenarios). Right: Tally of comparisons among all ten DJ domains. On teacher-related downstream tasks, we outper- form or match ImageNet-pretraneet underperforms ours (7/10 related, 10/10 unrelated) and ImageNet (3-4/10 related, 10/10 unrelated). See appendix for full results.
and on other tasks we match ImageNet. Traditional distill often
We only demonstrate that it is not trivial outperform our method using original teacher training data.
Exp. Improving student representation when N = 1. We show that this advantage of con- solidation over baselines holds for a wide range of upstream and downstream datasets. We also compare to CRD to show state-of-the-art distillation often underperforms even when specifically targeting representation learning. Fig. [ summarizes few-shot SVM accuracy using different repre- sentations relative to SangeNet-preerainee (See appendix for the full Fig. and raw numbers).
The conclusions are similar regardless of using KD or CRD consolidation outperforms (Cars 196, Resisc45, iFood, CUB, Airerafts, iNaturalist teachers) or matches (iFashion, DTD, Flowers, Cal- tech256 teachers) ImageNet pre-trained model performance on related downstream tasks, and matches its performance on unrelated ones. contrast, both traditional distillation methods (1) underperform consolidation on related downstream tasks for all teachers except Cars196, iFashion, and Aircraft, and (2) drastically underperform both consolidated and ImageNet features on unrelated downstream tasks- although CRD is better than KD, it still degrades performance. Notably, tradi- tional distillation methods underperform ImageNet even on related downstream tasks for Resisc45, DTD, Caltech256, and (for KD only) Flowers teachers. We match the MTL oracle performance across the board, reaping the teacher dataset's benefit without replaying
Exp. 2: Consolidating representations with N > 1. We can merge multiple task-specific teacher representations that are related to get a better one. In addition, our method is not constrained to merging models with the same architecture of's feature space dimensions like Geyer et al. (2019). To illustrate this, we split the "t-split" of 19rs196 and iFood into five random splits containing 10% of the original classes. We train a ResNet18 teacher on each of the five splits. Then, either traditional distillation or representation consolidation to merge these models with a ResNet50 generalist teacher. We compare the resulting representations with the teacher model's and ImageNet. Fig. 3a shows this result. For downstream tasks, we obtain better performance than using only one of the five teachers on both related and unrelated tasks, especially for iFood and Food101 where the teachers themselves underperform on related downstream task. This shows that our method can benefit from even teachers whose representation is weaker, as long as they have domain knowledge.
Full comparison with traditional distillation and representation consolidation from only one of the five teachers are in the appendix we gain performance on similar D by using five task-specific teachers instead of one, and outperform both traditional distill methods related downstream

---

Under review as a conference paper at ICLR 2022
Accuracy (rel. to ImageNet-prettaineed model)
Accuracy (rel. to ImageNee-pretrinned model)
Techee Contte-Reeaucsene
Teddenti toad
Doditl
Poettt
GR Lamo
Aaka wa
Scrvital
Incahhoppens and madds
Bolllrgg seeei CED
Clulll cadder st
(b) Merge multi-domain Res50 teach- ers
(a) Merging same-domain ResNet18 teachers
Figure 3: Exp. with N > I multi-model merging, 5-shot linear SVM downstream transfer. Left: In the same domain, we are able to consolidate from models with different architectures and improve transfer performance over every single teacher. Right: We can consolidate different domain models and improve over the ImageNet representation. See appendix for full comparisons.
Accumey (rel. ImageNett-retrainee model)
Teadeer sodl)
Texdan Daaaaala coadi
Rodlond
FFod
deomn
calcom
Tgt
Figure 4: Exp. Excerpt of N = 1, fine-tuning results. Showing full-shot (transfer to all classes for Flowers and Birds). Our conclusions are the same between fine-tuning and fixed representation (Fig. See appendix for the full results (few-shot, full-shot, more datasets).
tasks for iFood/Food101. We also show results when using all ResNet50 teachers for the five splits in the appendix. The gaps are closer but the results are similar and the conclusions are identical.
Finally, we explore merging models from different domains to form multi-domain consolidated student. See Fig. We observe that we can outperform ImageNet on all related downstream tasks, but the performance gain is smaller than representation consolidation in just one domain. We show in the appendix that ou outperforms KD on most downstream tasks except Cars196.
Exp. 3: Fine-tuning downstream. We also verify that our conclusions generalize to fine-tuning as well, especially without few-shot sampling. Fig. Fig.4 shows an excerpt of our results that include full dataset transfer. See appendix for fine-tuning using few-shot and full d-splits, whose conclusions are the same as this section's. Fine-tuning allows the representation to change into one that better suits the downstream task, so the benefit of teachers' domain knowledge shrinks compared to fixed oe with linear SVM hig Despite this, the conclusions are the same the fixed da scenario.
Exp. 4: Influence of the Loss weights. Fig. shows the effect of the choice of loss weight on method - when we use Ao = 1. X; = 2N (i 0), we gain performance on the related downstream task but lose performance on unrelated ones, whereas using An = 3. d zy (i > 0) gives us the opposite. We potentially use this trade-off to suppress unreliable teachers' influence.
7

---

Under review as a conference paper at ICLR 2022
Accuracy (rel. ImiggeNd--mmttrriind model)
Toccer Caul06
Aocuney (ee Kongehe greaated mndcc
20-
15-
10-
Cass196 (4-spii)) fewosbot
Recieks (d-ppii) low-sbut
5-
iFood (d-spiit)
0-
Foodl01
iPaahion bloove oak) fio-abee
13
okdnew 1:l
oidacw 3:1
(a) Exp. 4 Tune loss weights do (old) and )1 (new). DJ =Cars196
(b) Exp. D proxy- More in appendix.
Figure 5: Additional studies. 5-shot linear SVM downstream transfer. Left: Tuning loss weights
result in different balance between related and unrelated downstream task performance. Right: Places365 as proxy has similar results as ImageNet proxy. However, training on Places365's super- vised labels using less general proxy (iNaturalist) hinders performance.
Exp. 5:
Proxy data choice. Fig. 5b shows results replacing ImageNet with Places365 iNat- uralist as the Drraxy dataset. Places365 yields similar performances as using ImageNet, while narrower-scope iNaturalist weaker on unrelated downstream tasks. Jointly training on Places365 labels is not helpful, meaning that the improvement comes from the generalist teacher, not the data.
5 RELATED WORK
Our goal is to maximize the transferability of a representation consolidated from multiple teachers. This downstream transfer aspect has received little attention in the literature (Liu et al. 2019 Geyer et al., 2019; Tian et al.] 2020), but this problem formulation closely related to prior work on mudti-model merging and distillation with proxy data.
Multi-model merging. It is useful to combine separate models that perform different tasks into a single model for efficiency and performance benefits. Knowledge Concentration (Gao et al. 2017) combine teachers trained on subsets of 100k classes in the EFT dataset (Gao et al. (017) into one improved single model, using handcrafted sparse connections for the final student layers. Chou et al. (2018) merge CNNs by combining the kernel weights of different models by clustering and lookup, followed by fine-tuning. ZZhao et al. (2020) merge object detection models when some classes may be the background to other models. Vongkulbhisal et al. (2019) combine models with overlapping classes using the combined train set, by deriving soft labels from intra-model class correspondence. Ye et al. 2020) progressively train a GAN to regenerate proxy data for all teachers, and combine teachers layer by layer. [Chkkraborty et al. (2018) and Park & Kwak (2020) aggregate ensembles of members trained on the same task. In one-shot federated learning (Guha et al. 2019), multiple clients with their private data train model each, and finally merge the models using an ensemble or distillation a way that protects their data privacy.
Unlike approach, these methods merge models in order to perform exactly the same task as the teacher models, are not concerned with the performance of the student when transferred to a downstream task, and often (Gao et al., 2017 Chou al. 2018: Zhao al.) 2020 Vongkulbhisal eet al. 2019 Chakraborty et al. 2018 Park & Kwak] 2020) require revisiting the original training
images. We have shown that these teachers themselves have poor transferability compared simple pre-trained baseline, and learning only from them (even using state-of-the-art CRD (Tian
al. 2020 yields suboptimal transfer learning performance.
Multi-model merging for transfer. Knowledge Flow (Liu et al. 2019) connects the student to
multiple teachers' intermediate layers to kick-start target task training, and gradually penalizes its reliance on teacher models over time. [Geyer et (201)) uses IMM (Lee et al. 2017) to merge mul- tiple models using their diagonal approximated Fisher information matrix (FIM) to balance different
8

---

Under review as a conference paper at ICLR 2022
teacher's weight importance, merge, and fine-tune. Computing the FIM requires reprocessing the original teacher training data (unlike our approach that only needs generic proxy data). Furthermore, this method requires homogeneous architectures for students and teachers, while ours works any architecture combinations. These methods directly optimize performance on the downstream task and thus require re- re-running for each different target dataset. Our approach is more efficient, as it only requires consolidating teachers once to improve the pre-trained representation independently of the downstream task. Finally, we note that the representations learned were not compared to a strong baseline (i.e., pre-training on ImageNet), which we argue is a prerequisite for being useful in real- world applications. One of our main contributions is observing the need for including generalist teacher, an insight which is orthogonal to, and could be combined with these previous approaches.
Distillation with representation losses ("representation distillation") tries capture additional structure of feature representations by aligning student and teacher feature activations during distil- lation. Koratana et al. (2019) compress models by adding L2 losses between intermediate represen- tations of the teacher and students. [Aguilar et al.] (2020) use KL divergence and cosine similarity to make the attention and representation of intermediate features of the student and teacher similar.
Tian et al. (2020) adds contrastive representation learning loss to the penultimate layer to preserve feature structures, by maximizing each image's 's student and teacher features' mutual information. Despite their focus on yielding better representations, these methods fully rely on the teacher's transferability. We show that this yields downstream performance worse than generalist baselines.
Proxy data ("data-free") distillation transfers the input-output function of a teacher network to student network without using the teacher's training data. (Yalniz al. 2019. Orekondy et al., 2019) opt to use a large general proxy dataset to query the teacher, and their teacher outputs to on this data train the student. Other methods (Nayak al. 2019 Chen al. 2019 Haroush al. 2020 Chawla et al.) 202T) generate proxy data directly from the trained models, and use this data to train the students. Further. Micaelli & Storkey| (2019): Yin et al. (2020) also encourage generating samples the student and teacher disagree on. Lastly, other methods require the original dataset to compute meta-data information such as feature cluster mean. (Lopes et al., 2017 Bhardwaj et al. 2019) Some train a GAN from the teachers to maximize chosen class predictions (Fang et al. 2019 Yoo et al./ 2019 Ye al., 2020), sometimes also batchnorm statistics (Luo et | Xu al.
2020b), and sometimes on proxy data instead. (Addepalli et al. 20200] Besnier et al. (0200 Some
work combines self- superviedd learning with distillation (Tian et al.) 2020] Xu et al. 2020a Fang
et al.
202T) to improve the representation performance on the original task.
These works try to emulate the original data for standard distillation, and do not concern the student's
transferability or merging multiple teachers. Our simple method already performs comparably to a multi-task oracle that uses the original teacher dataset, indicating that these more complex methods that improve upstream task distillation may be unnecessary when the goal is downstream transfer.
Incremental learning is also related to our overall goal of growing a library of expert representa- tions. These methods continually learn tasks or classes but often with limited access to past training data (Li & Hoiem 2018; Rebufñi et al. [2017) Kirkpatrick et al. [017]] Zenke et al. 2017; Hu al.) 2019 Yin et al. 2020 Prabhu et al. (020). Our approach addresses many of the same challenges by consoldating knowledge in the form of feature representations without revisiting old data used to train teachers, but our "increments" are whole tasks given in the form of trained models instead of data, whose labels may overlap. Multi-task learning (Caruana) 1998) assumes all datasets are available and trains jointly on them whereas we assume task-specific datasets absent. We perform comparably to this oracle.
SUMMARY
In this paper, show that traditional distillation can result in representation suboptimal for down- stream task transfer learning, because it only focuses on preserving the end-to-end input-output mapping of the old task. We propose representation consolidation with the generalist model as an additional teacher. Our method preserves the wide-range transferability of the strong ImageNet baseline and improve the performance for both related and unrelated downstream tasks over tradi- tionally distilled networks. We show that we can merge multiple models in the same domain to get better representation than any single model.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Char- less C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6430-6439, 2019.
Sravanti Addepalli, Gaurav Kumar Nayak, Anirban Chakraborty, and Venkatesh Babu Radhakrish- nan. Degan: Data-enriching gan for retrieving representative samples from trained classifier. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3130-3137, 2020.
Gustavo Aguilar, Yuan Ling, Y. Zhang, Benjamin Yao, Xing Fan, and Edward Guo. Knowledge distillation from internal representations. In AAAI, 2020.
Victor Besnier, Himalaya Jain, Andrei Bursuc, M. Cord, and P. P'erez. This dataset does not exist: Training models from generated images. ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2020.
Kartikeya Bhardwaj. Naveen Suda, and R. Marculescu. Dream distillation: A dtta-indeppndent model compression framework. In Joint Workshop On-Device Machine Learning & Compact Deep Neural Network Representations, NeurIPS Workshop, 2019.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative com- ponents with random forests. European Conference on Computer Vision, 2014.
Johannes Buchner. Imagehash library.
imagehash| 2021. Accessed: 2021-05-27.
R. Caruana. Multitask learning. In Encyclopedia of Machine Learning and Data Mining, 1998.
Rudrasis Chakraborty, Chun-Hao Yang, and B. Vemuri. mixture model for aggregation of mul- tiple pre-trained weak classifiers. 2018 IEFE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 454 4547, 2018.
Akshay Chawla, Hongxu Yin, Pavlo Molchanov, and J. Álvarez. Data-free knowledge distillation for object detection. In WACV, 2021.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Daft: Data-free learning of student networks. In ICCV, 2019.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 2256. doi: 10.1 10.11000000000017..2711 201 7.
1883 Oct 2017. ISSN 1558- URL http: /xx doi wrg. 1109///RRO./
2675998
Yi-Min Chou, Yi-Ming Chan, Jia-Hong Lee, Chih-Y Yi Chiu, and Chu-Song Chen. Unifying and merging well-trained deep neural networks for inference stage. In Proceedings of the 27th Inter- national Joint Conference Artificial Inielligence, pp. 2049-20566 AAAI Press, 2018
M. Cimpoi, Subhransu Maji, Kokkinos, Mohamed, and Vedaldi. Describing textures in the wild. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-36066000 3613, 2014.
Aditya Deshpande, Alessandro Achille, Avinash Ravichandran, Hao Li, Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona. linearized framework and new benchmark for model selection for fine-tuning arXiv preprint arXiv:2102.00 2021.
Gongfan Fang, J. Song, Chengchao Shen, X. Wang, D. Chen, and Mingli Song. Data-free adversarial distillation. ArXiv, abs/1912.. 2019.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, L. Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. ICLR, 2021.
J. Gao, Zijian Guo, Z. Li, and R. Nevatia. Knowledge concentration: :nnvtedge.onneenrraion::: Learning 100k object classifiers in single enn. ArXiv, abs/1711.07607, 1.06677, 2017.
10

---

Under review as a conference paper at ICLR 2022
Robin Geyer, Luca Corinzia, and Viktor Wegmayr. Transfer learning by adaptive merging of multi- ple models. In MIDL, 2019.
G. Griffin, Alex Holub, and Perona. Caltech-256 object category dataset. Technical report, California Institute of Technology, 2007.
Neel Guha, Ameet S. Talwalkar, and Virginia Smith. One-shot federated learning, Machine Learning on Devices Workshop at NeurIPS, 2019.
S. Guo, Weilin Huang, X. Zhang, Prasanna Srikhanta, Yin Cui, Y. Li, M. Scott, Hartwig Adam, and Serge J. Belongie. The imaterialist fashion attribute dataset. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3113-3116, 2019.
Matan Haroush, Itay Hubara, E. Hoffer, and Daniel Soudry. The knowledge within: Methods for data-free model compression. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8491-8499, 2020.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
Geoffrey E. Hinton, Oriol Vinyals, and J. Dean. Distilling the knowledge in neural network. ArXiv, abs/1503.02531, 2015.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, C. Sun, Alexander Shepard, Hartwig Adam, P. Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8769- 8778, 2018.
Wenpeng Hu, Z. Lin, B. Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In ICLR, 2019.
Parneet Kaur, Karan Sikka, Weijun Wang, serge Belongie, and Ajay Divakaran. Foodx-251: A dataset for fine- tineggrineed food classification. arXiv preprint arXiv: 2019.
J. Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, J. Veness, G. Desjardins, Andrei A. Rusu, K. Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barbkkkk,, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 11::5521 3526, 2017.
Animesh Koratana, Daniel Kang, Peter Bailis, and M. Zaharia. Lit: Learned intermediate represen- tation training for model compression. In ICMI., 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-graing graigad categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13) 13), Sydney, Australia, 2013.
Sang- Woo Lee Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung- Tak Zhang. Overcom- ing Catastrophic Forgetting by Incremental Moment Matching (IMM). In Advances In Neural Information Processing Systems 30, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40:2935 2947, 2018.
Tou-Jen Liu, J. Peng, and A. Schwing. Knowledge flow: Improve upon your teachers. 7th Interna- tional Conference on Learning Representations, ICLR 2019, 2019.
Raphael Gontijo Lopes, Stefano Fenu, and T. Starner. Data-free knowledge distillation for deep neural networks. ArXiv, wts///b1../7355,,.1 2017.
Liangchen Luo, M. Sandler, Zi Lin, A. Zhmoginov, and Andrew G. Howard. Large-scale generative data-free distillation. ArXiv, abs/2012.05578, 2020.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. ArXiv, abs/1306.5151 2013.

---

Under review as a conference paper at ICLR 2022
P. Micaelli and A. Storkey. Zero-shot knowledge transfer via adversarial belief matching. In NeurIPS, 2019.
G. K. Nayak, K. R. Mopuri, V. Shaj. R. V. Babu, and A. Chakraborty. Zero-shot knowledge dis- tillation in deep networks. In International Conference on Machine Learning, pp. 4743-475, 2019.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp.
722-729, 2008.
Tribhuvanesh Orekondy, B. Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black- box models. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4949-4958, 2019.
Seonguk Park and Nojun Kwak. Feature-level ensemble knowledge distillation for aggregating knowledge from multiple networks. In ECAI, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, gigh-peforrance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Ameya Prabhu, P. Torr, and P. Dokania. Gdumb: A simple approach that questions our progress in continual learning. In ECCV, 2020.
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint Xrxiv:2009. 13239, 2020.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, G. Sperl, and Christoph H. Lampert. icarl: Incre- mental classifier and representation learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533-5542, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV),, 11(3))211--22, 2015. doi: 10. 1007/s 1263-015-0816-y.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations, 2020.
J. Vongkulbhisal, Phongtharin Vinayavekhin, and M. V. Scarzanella. Unifying heterogeneous clas- siferr with distillation. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11799 2019.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie, The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 201
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In ECCV, 2020a.
Shoukai Xu, Haokun Li, Bohan Zhuang, Jing Liu, Jiezhang Cao, Chuangrun Liang, and Mingkui Tan. Generative low-hitwidth data free quantization. In ECCV, 2020b.
I. Z. Yalniz, H. Jégou, K. Chen, Manohar Paluri, and D. Mahajan. Billion-scale semi-supervised learning for image classification. ArXiv, abs/1905.005s 905.00546, 2019.
12

---

Under review as a conference paper at ICLR 2022
Jingwen Ye, Yixin Ji, X. Wang, Xin Gao, and Mingli Song. Data-free knowledge amalgamation via group-stack dual-gan. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12513- 2020.
Hongxu Yin, Pavlo Molchanov, Zhizhong Li, Jose M. Alvarez, Arun Mallya, Derek Hoiem, N. Jha. and J. Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7712-8721, 2020.
Jaemin Yoo, Minyong Cho, T. Kim, and U. Kang. Knowledge extraction with no observable data. In NeurlPS, 2019.
Friedemann Zenke, Ben Poole, and S. Ganguli. Continual learning through synaptic intelligence. Proceedings of machine learning research, (0:3987-3995, 2017.
X. Zhao, Schulter, G. Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with unified label space from multiple datasets. In ECCV, 2020.
B. Zhou, A. Lapedriza, A. Khosla, Oliva, and Torralba. Places: 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40: 1452-1464, 2018
A IMPLEMENTATION DETAILS
We will release our code to reproduce this paper upon acceptance. We use PyTorch, (Paszke et al., 2019) For all network training, we use SGD with momentum 0.9, weight decay of 10-3, a batch size of 32. Unless noted, we use learning rate decay 0.1 at 50% and 80% of total training. We ini- tialize sask-specific teachers' training and all students' distillation with ResNet50 (He et al. 2016) pre-trained on ImageNet, and we freeze batchnorm statistics during distil/consoliiaation/MM" learn each of on the task-specific Dj by fine-tuning 120 epochs (learning rate decay 70th and 100th epoch) while doing log-scale grid search on the learning rate. For distilling de. our method less sensitive to the choice of learning rate. We use fixed learning rate of 0.01 for each hi and 0.001 for dh, and schedule of 40 epochs. Note that ImageNet pre-training in PyTorch uses 0.001 as the final epoch learning rate. This takes us roughly 4 days on an AWS instance with one NVIDIA V100 for each distilleddconssoliateed representations. When the downstream transfer uses fixed oi we extract (x) on the center image crop, and search ht's SVM hyperparameters using a 5-fold cross-vvliddatio scikit-learn (Pedregosa et al.) (01T). When the downstream transfer uses fine-tuning, we run log-scale grid search of learning rate with 50 epoch schedule.
We
B COMPLETE FIGURES FOR SECTION EXPERIMENTS
We now provide the full graphs and table for our experiments. Note that we have summarized these results and all conclusions in the main paper.
Exp. 1: Improving student representation when N = 1. Fig. 6 and Table 3 show full results
for Fig. Please the main paper for analysis and conclusions.
Exp. 2: Consolidating representations with N > 1. Figs.
shows full results for Fig.
For Figs. and same-domain model merging, in addition to the main paper results merging ResNet18 of and ResNet50 of, we show similar results for all teachers being ResNet50 in Fig. We also show comparison to traditional distill with 5 teachers, representation consolidation with only teacher, and the teacher itself (best out of 5 according to related DJ performance). The conclusions are the same, and merging five teachers using representation consolidation outperforms all baselines on both related and unrelated downstream tasks (except Cars with related DJ against traditional distill).
For multi--dominn model merging in Fig. 9 we compare to traditional distill with multi-task learning. We outperform or match it on downstream DJ except for Cars 196. We also explore using a concate- nation of multiple large unlabeled datasets as Dproxy- With more diverse proxy, the performance
13

---

Under review as a conference paper at ICLR 2022
Accuracy (rel. to Irareeet-ppetrrinnd model)
d:spit)
D-spitt
PPod
ihodd
Phad Nowwahtt
Woughtn
Rochayy
Euaste Myothat
feevatiion
San
(oupl::
IC45
Teachen: Fashion (catogory takkd model
Teacher: DTD (t-ppit) model
Teachan Flwers (t-piit) model
Iebenve pohas
Oiove Tytk
Myahhe
Cirerres
Nuweiee Fiod
featts
Teacher:
Teacher (:4000
iTecd
Sun mmd.
few
The OFC
Foum
47
20
Contte
Fantoy
13 1p
co
Ten te--tht
New shot
Teachen INsturalist (t-spiit mode
Floderrss
grrd
wwa.eeeeorrtttiiee... sufvlabel
single task-specific teacher, 5-shot linear SVM downstream transfer.. All 10 DJ cases. This extensive set of experiments have the same conclusions as the main paper: we match or outperform ImageNet representation while traditional distill often underperforms. See also Table for comparison tally.
Figure 6: Exp. 1 Full results for Fig. 2a (N
14

---

Under review as a conference paper at ICLR 2022
Accuracy (rel. to ImageNeli pretruined model)
gived
Toulldl diengo
Poddii kinget
Posduct
dkoc alo
fe-plo
2ath www.ii
Imaghhin periiieed lake
(Pes KD
3. cpp o4siid CRU
<5 22 KD
(Rooi scadect (fo
Figure 7: Exp. 2 Merging N same-domain ResNet18 teachers, 5-shot linear SVM downstream transfer. Part 1/2 of full results for Fig. 3a architectures (ResNet50 of and ResNet18 ( and improve transfer performance over every single teacher.
We are able to consolidate from models with different
Accuracy (rel. to ImagaNet-prettrinee model)
Teuchets) clascs
Teachoy'y iPand
chesee modd
vle
Fadca
oleve ogo
DE
TITE
wwutht
Figure 8: Exp. 2 Merging N same-domain ResNet50 teachers, 5-shot linear SVM downstream transfer. Part 2/2 of full results for Fig. 3a teachers.
Our conclusions generalize to using all ResNet50
Accuracy (rel. SangeNee-gnetrained model)
Tosher Quail5--Raa - (1uge
Figure 9: Exp. 2 Merging N > different domain ResNet50 teachers, 5-shot linear SVM down- stream transfer. Full results for Fig. 3b > multiple model merging, multiple domains). We can consolidate different domain models and improve over the ImageNet representation and (for most related/unrelated downstream datasets) multi-task traditional distillation.
15

---

Under review as a conference paper at ICLR 2022
Accuracy (rel. to ImageNed--retruineed model)
Teachen 1an 196 modd
Teacher: Rockoks (Hppl)) model
Teacher iPood (oepli)) modki
Rostees
Ronoot
Stepla Dooakei
Migti oo
Food foesh
iead
Sylbse Anvibol
Ualees lowast
Fastial
olucuao fro-dxx
Tencher Washon (category trakk model
Teacher Cans (o5-sBoosoio----- Foodd Dooplid model
staphr)
10
Meriio
10
olopti
BEKT tall
(de wil)
clareng
poaphha-pettinddraadd roaki
Sggtt conalal
Figure 10: Exp. 3 N =
- 1 (first 4 graphs) and N 1 (last graph) teacher(s), 5-shot fine-tuning
downstream transfer. Part 1/2 of full results for Fig. The same conclusions as the fixed represen-
tation scenario in Figs. 6 hold.
drops little for related DJ and stays similar for unrelated Dj suggesting we are somewhat insen- sitive to choice of datasets, but a more diverse proxy data may not provide better model merging performance.
reluted ZYj
unrelated Dy
iFshioon.
ouns d. ImageNet of
others)
DTD, Flowers,
[all
Caltech256.
ouns d KD distill) 0
others)
Cars
iFshioon, Airerani.
10)
Resise45.
KD (trad. distill) de vs. ImageNet of
others)
iNaturalisa,
DTD, Flowers, Caltech256.
(all 10)
ouss + CRD de v.. ImageNet of
(7 others)
iFashion, DTD.
Caltech256.
(all 10)
1ars196, iFashion, Aireratt.
ours + CRD 6 vs. CRD (trad. distill) 6,
(7 others)
(al) 10)
Flowers,
Resise45, Caltech256..
CRD (trad. distill) 6, vx. ImageNet of
(5 others)
iNaturalist.
(all 10)
Table 3: Exp. Detailed tally of Fig. 6 (N
1 single task-specific teacher, 5-shot linear SVM downstream transfer) explaining Fig. 2b On teacher-related downstream tasks, outperform or match ImageNet, and on other tasks we match ImageNet performance. Traditional distill often underperforms ours (7/10 related, 10/10 unrelated) and ImageNet (3-4/10 related, 10/10 unrelated).
16

---

Under review as a conference paper at ICLR 2022
Accuracy (rel. ImageNNet-retrriine model)
Teacher: CarsiNg (1 spliti model
Tracher Raiioo5 (-sphi) model
Teacher. IF Food (t-splin model Jboeet
ihoe Bonga
iPaal (Daa
Teadto clamo
Twech cbuli
Fowdti cbolp))
frnsh
(1 Sgti
Freas (al
spiitt
(t-plit)
S-iooo (1-sptet)
(2-epir))
Hood
16
Carstee
Poadio
Posti
1sl clomno
Alrezaia
Arroh
tall classo
KD
Figure 11: Exp. N (first graphs) and N (last graph) teacher(s), full-shot fine- tuning downstream transfer. Part 2/2 of full results for Fig. The same conclusions as the fixed representation scenario in Figs. hlold.
17

---

Under review as a conference paper at ICLR 2022
Accuracy (rel. Imageeed--retraineed model)
Teachen Can 196 mood
Teacher: Reekoks (Hepi)) model
Teacher iPood (oepli)) modki Ronoos
Rostees
Cant
Steply Dhhakei
20
Food
iead Sendue
Sylbse Anvioll
lonasd
Ticeral mee-bir
Fastial
fru-dex
TrappNeyerrsraaddo rode
couolld IDD (JoNit
FpC. cowsld KD (Fiaas (Fbose,300s) sooy)
Hgpe SONAEEL KD (Nasssi pany)
Figure 12: Exp. 5 N 5-shot linear SVM downstream transfer, different proxies. Full results for Fig.5b) Conclusions similar - as the proxy, Places365 is similar to ImageNet, but narrower-scoped iNaturalist underperforms.
Exp. 3: Fine-tuning downstream. Figs. T0 show full results for Fig. These extensive results show that in both few-shot and full-shot scenarios, and both single-domain and multi-domain scenarios, we have the same conclusions as the fixed representation experiments.
Exp. 5: Proxy data choice. Fig. 12 shows full results for Fig. 5b This shows similar results with different teacher D, i.e. Places365 performs similarly to ImageNet as the proxy, but the narrower- scope iNaturalist underperforms.
C RAw NUMBER ACCURACY TABLES FOR ALL EXPERIMENTS
Table 4
fixed representation few-shot results for Figs. (Figs.
6 7 in the appendix).
Table
fixed representation few-shot results for Fig. S
Table 6
fine-tuning few-shot results for Fig. 10 in the appendix.
Table
fine-tuning full-shot results for Fig. (Fggs I in the appendix.)
D LIMITATIONS
We assume we have perfect knowledge which tasks form the same domain and which tasks do not belong to a domain. Our performance drops when teachers from different domains are consolidated. In future work, we plan to automatically determine how to cluster large amount of teachers into domains. In addition, one of our contributions assume the existence of strong representation baseline such as the ImageNet pre-trained model, which is true for fields like images and language, but not for others (e.g., 3D reconstruction). Our method also takes the teachers as is and learn from them, and any mistakes made by the teachers can be propagated during the consolidation. Possible mitigations include using better teacher that makes less such mistakes, or using regularization on both teacher and student training, such as making similar inputs map to similar outputs.
18

---

Under review as a conference paper at ICLR 2022
Foad101
iFaehion (sleeve)
Cars196
Resise45
(all)
LX11
Flowers
Caltech256
Birds
Aircratts
ImageNe: pretrained model pereeeee-eetiinddd. soft-lahel
31.94
70.66
28.94
36.95
$9.00
60.98
83.84
50.25
54.73
30.71
31.47
69:99
29.21
37.29
59.18
611.4
82.86
81.35
57.19
29.77
Cars196 R-split)
59.39
51.56
9:75
54.27
36,07
59.91
35.51
16.90
Cars196 (-splii) rape comoolid. KD
50.21
70.00
80:00
83.01
50.33
54.02
20.85
Car1196 (--plin) CRD
62.42
$5.96
66 76
19:13
Cans196 (t-plitt rape comsolid. CRD
60.97
71.02
29.14
36.15
59. 59.10
59.47
B1.33
80.23
$4.09
32.04
Cars196 (-splii) MiL oracle
60.49
69:23
25.44
35.15
89:20
611.46
N2.79
79.79
55.37
30.89
Resiso45 (t-spin) KD
8.97
61,62
9.85
12.31
$4.97
37.50
56.06
34.14
11.23
11:57
Resisok5 (t-split) repe cunsolid. KD
30.80
72.64
28:77
36.37
59.03
60.08
82:
50.77
55.47
29. 29.17
Resisok5 (1-spli) CRD)
16.81
68.62
16.95
19.54
56.42
45.00
72.
49.00 00.02
20.89
18.31
Resiset5 (t split) repc. cunsolid. CRD
32.66
74.00
29.61
36.99
59.10
60.46
$4.22
90.73
55.19
31.05
Resise45 (t-splin MTL ocacle
52,51
73.06
29,07
36.21
89.11
59.82
83.74
30.04
57.11
31.19
iFod (i-ppin) KD
12.85
53.57
35,26
44.05
85.39
36,62
70.99
39.33
17.95
44.00
iFvod (i-plit) consolid KD
29.46
59.49
38.85
47.19
59,00 04
57.16
82:66
79.17
51,07
27.31
iFood (i-plit) CRD
14,80
56.57
36.30
64.67
56.03
38.95
74.03
42.56
19.91
16.18
iPood (1-pli consolid CRD
30.54
57.48
78.67
(-splii) MTT.
82.62
79.52
iFashion (ctlegory
KD
37.97
23.45
44.01
iFoshion (calegury repe consolid
70.02
29.23
37.45
59.15
60.76
83.28
81.29
57.10
|Fashion (catogary task CRD
16.11
57.91
14.74
18.35
50.93
41.77
72.11
50:08
20.87
17.52
IFsshion (catogary tusk) rege consolid. CRD
32.36
71.75
37.60
89:72
611.96
54.05
30.91
55.51
31.39
iFashion (category task) MTT. oeacle
31.91
69.35
28.78
36.27
58.75
60.50
83.40
79.93
$7.01
31.28
DTD (t-plitt KD
11.29
53.12
14.11
19.00
56.
46.30
62.48
46.3 38
17.14
14.75
DTD (--plii) repe, consolid, KD
31.35
69.78
29.17
37.38
39.08
61.43
N2.90
81.28
56.09
29.59
DTD (1-plii) CRD
23.39
64.77
24.59
31.08
87.82
55.90
79
66. 36
33.82
24.15
DTD (1-plii) repe, consolid, CRD
33.10
71.58
29.55
38.27
89. 19.10
61.47
84.06
81.01
56.02
$1.35
DTD (-split) MTL, onall
32.76
69.55 56
29.24
36.95
$9.3
61.10
83.22
80.57
$7.66
31.05
Flowers (t (--plii KD
16.72
$8.08
16.68
20.67
$7.05
44.70
77.35
$4.69
31.32
20.26
Hlwess (t-split rggr, cansolid, KD
31.81
69.94
29.46
37.01
59.15
611.05
S4.77
81.18
57,077
30.41
Flower (-splin) CRD
26.84
64 86.86
22.78
27.54
58.
50.05
62.66
66.27
42.05
27
Flowas (-splii) consolid. CRD
53.79
71 71.38
50.05
37.68
80.25
60.42
86.
81. 10
56,56
32.52
Flowers (t-plii MTL oracle
32.15
68.89
27.85
35.19
$9.11
56.78
82.73
79.72
55,97
30.79
Caltech256 (t KD
21.65
59.67
19.84
25.10
$7.35
51.59
75.52
98
36.09
23.32
Caltesh256 (1-plit) repe.
KD
69.36
36.97
N3.14
Caltech256 (--plit)
55.35
Callech256 (--plit) rope,
CRD
Caltech256 (pllp MTL ocacle
30.61
34.92
$9.09
58.92
93.04
78.83
29.21
Binds (c-plii KD
17:15
57.95
17.06
20.72
87.03
46.37
72.16
53.26
19:00
Binds (-splii) rege consdid KD)
31.50
29.16
59.20
60.33
83, $3.90
80:89
66.37
Binds (-split) CRD
23.22
62.63
21.31
25.92
58. $8.10
50,80
77.85
61.92
64.00
23.10
Birds (t-spiiO rege. consolid CRD
32.70
71.65
29:72
37.52
39.25
61.33
64.63
80.56
66.35
31.54
Birds It- t-splii) MiL oraele
31.89
09.85
29.12
36.54
39.30
60.11
$3.00
80.25
64.79
30.26
Aircrafts (t-splin) XD
10.56
44.30
7.81
9,59
34.17
29.16
4N.32
29.18
10.87
$4.51
Aircrafls (l-plii)) repr. consolid. KT)
30.71
69 58
28.25
35.54
50.06
58.67
81. 16
50.5 54
$3.95
55.26
Aircralts (1 splint CRD
14.06
50.09
10.56
11.56
$5.17
33.13
57.85
35.16
14.11
60:44
Aireratt (1-splif) repr. consolid. CRD
32.54
70.54
25.91
36.15
59.17
59.31
83.01
30.27
53.29
$7.25
Aireratt (1-splin) MTL onacle
31,55
69.52
26.91
36.30
39.28
611.40
83.04
79.65
56,31
54.82
(Notaralist (1-ipii) KD
13.51
57.25
14.66
18.68
$4.46
39..09
79.74
41.59
$7,49
16.98
iNotunliit (1-ppii) consllid, KD
30,08
69.67
27.35
34.52
58.70
57.28
84.77
78.01
67.58
27.94
iNaturalist (t-split)
14.92
58.76
15.30
19.41
34.84
40142
80.98
42.71
58.48
18.00
(Notariistt (1-ppii) repr. consolid. CRD
50.87
33.97
56.53
86.06
77.08
(Nstarolist (1-ppii) orade
79.05
Cars196 10%s classes 5 (Rec18) rope. consolid. CRD
44.04
71.99
30.45
$7.30
59.20
60.55
65.64)
$1.01
58.09
$1.86
Cans196 10% classss x5 (Res repe. consolid, KD
42.03
69 19.95
29 29.44
36.97
$9.05
60.15
83.23
$1:48
58.18
31.10
Cars196 10% classes x5 (Res18) CRD
48.45
66.00
23.35
27.10
$9.42
52.48
80.29
06.12
47.90
29.24
Car119 10% ciases 5 (Res]8) KD
45.37
60.74
17.55
20.50
87.1
46.16
73.66
59.55
39.02
23. 23.85
Cars196 10% chases (Res)8)
comolid, CRD
39.56
72.50
29.86
37.12
59,
60.73
85.07
80.00
57.67
33.79
Cars196 10% chases (Res seacher (best 5)
comolid. KD
36,97
69.64
25.62
35.95
58. 92
59.85 09.88
K2.79
80.37
56,80
30.67
Cars196 10% classes
16.9%
20.50
700.5
23.40
iFood
x55 repr. couselid.
32.36
H4.710
79.57
(Food 10%
(Res18) rept. consolid. KD
31.14
83.51
iFood classes (Res18)
22.26
$7.67
46.94
79.60
58.12
classes (8es18)
17.60
51.55
(Hood 10% classes (Res18) rege. consolid. CRD
32.75
71.36
32.22
38.97
39:24
59.18
$4.45
796988
54,017
32.17
iFood 10% clases (8es18) rype consolid KD
30.45
51.25
3R.47
58.
50
91.97
79.33 37
$4.33
30.25
iFood 10% classes (Res15] teacher (best of
14.27
$2.35
23.11
25.40
$6.45
38.34
65.88
44.94
22.55
16.11
Cars196 10% classes 5 rept. consolid. CRD
46.69
71.65
30.10
37.72
$9.00
60.48
85,50
$0.75
57.01
33,00
Cars196 10% chases 5 consolid, KD
45.99
70.10
29.25
37.00
$9.16
60.40
83.82
81.21
57.60
31.42
Cars196 10% classes CRD
53.22
66.77
29.87
Cars196 10% classe
50.33
22.30
56.77
75.
Cans196 classes repe. consolad CRD
41.67
71.77
59.32
60.52
85.37
30.05
Cars196 ciasses roge. consolad.
30.25
36.01
59.54
N2.76
30.33
31.20
Cars196 10% classes toscher (best 5)
42.92
25.95
87.58
51.79
75.66
64.35
43.20
25.01
[Food 10% classes 5 repr. couselid. CRD
32.12
72.04
89:099
79.26
iFood 10% classes x repr. conselid, KD
30.74
20.64
37.34
44.77
$8.91
59.17
84.5 44.58
80.20
$2.47
29 29.83
iBood 10% classes x55 CRD
21.95
65.63
37.04
43.06
$7.12
49.35
81. $1.66
56.65
30.24
24.05
iFood 10% clwses x5 KD
17.70
61.33
35.88
42.71
56.15
46.63
77.50
50.95
25.36
20.49
iFood 10% clasees repr. consolid. CRD
32.64
70.85
32.81
39.57
80.
53.40
84. 36
79.19
53.73
31,
iFood 10% classs repr. consolid. KD
30.05
68.62
32,06
39.37
$8.85
58.67
82.09
79.43
53.85
29.68
iFood 10% classes teacher (best 5)
13.85
54.31
25.16
24.22
36.45
34.39
69.26
44,21
20.54
16.08
Cars:96+8essss555
(1-)) repr. consolid, Cari1964Rssius5..ff. (1-gia) KD
47.85
72.39
39.09
58.63
N3.64
79.90
51.79
29.18
53.27
66.27
85. 32
43.74
75.84
23,16
19:99
Cars/ 96 + Resised + Fooo (p-ppit) repe. conslld.
(Pacc--iNaellgolle Nolelooeee-- proty 10 epoch)
45.71
73.00
32,87
39.50
89.00
59.09
83.39
79.83
52.40
29.46
Cars196 Resiset5 aPood it splil tndd dutill
(ettees Nat procy 10 epochs)
$3.26
66.34
32
39.10
85.43
43.81
75.92
47.75
23.15
20.15
Table 4: Fixed representation linear SVM, 5-shot results (Part 1/2) raw numbers for Figs. the main paper (Figs. [
2 3 in
g 9 in the appendix)
19

---

Under review as a conference paper at ICLR 2022
Food101
iFashion
Cars196
Resisc45
iFood
(all)
(sleeve)
Cars 196 (t-split) repr. consolid. old:new 19rs196 (t-split) repr. consolid. KD oldmnew Cars 196 (t-split) repr. consolid. KD old:new 3:1
61.21
67.83
26.70
33.35
88.92
59.21
70.00
28.42
35.28
89.00
52.80
70.82
29.18
36.53
89.13
Cars 196 (t-split) repr. consolid. KD (ImNet proxy)
59.21
70.00
28.42
35.28
89.00
Cars196 (t-split) repr. consolid. KD (Places proxy)
61.18
69.59
26.69
33.54
88.68
Cars (t-split) repr. consolid. KD (Places proxy label)
53.16
69.32
23.17
28.54
88.23
Cars196 (t-split) KD (Places proxy)
59.98
51.11
9,54
11.10
84.38
19rs196 (t-split) repr. consolid. KD (iNaturalist proxy)
47.76
67.90
25.08
31.07
87.80
Resise45 (t-split) repr. consolid. KD (ImNet proxy)
30.80
72.64
28.77
36.37
89.03
Resisc45 (l-split) repr. consolid. KD (Places proxy)
30.08
72.56
27.59
34.68
88.99
Resise45 (I-split) repr. consolid. KD (iNaturalist proxy)
26.17
72.11
25.71
32.38
88.40
iFond (t-split) repr. consolid. KD (ImNet proxy)
29.44
69.49
38.85
47.19
89.04
iFood (t-split) repr. consolid. (Places proxy)
68.94
36.30
43.97
iFood (t-split) repr. consolid. KD (iNaturalist proxy)
22.87
67.04
33.47
40.18
87.79
Table 5: Fixed representation + linear SVM, 5-shot results (Part 2/2) raw numbers for Fig. in the main paper.
Food101 (all)
iFashion (sleeve)
Cars196
Resise45
iFood
Imateeett-reetrainedd model ImageNet-pretranner soft-label
37.18
68.91
28.25
34.72
89.68
39.84
69.88
28.72
35.31
89.92
Cars196 (1-split) repr. consolid. Cars196 (t-split) trad. distill
55.80
69.91
28.41
34.00
89.91
60.19
61.30
18.98
19.79
87.90
Resise45 (t-split) repr. consolid. Resise45 (t-split) trad. distill iFood (I-split) repr. consolid.
38.30
71.71
28.16
34.33
89.83
21.12
60.93
17.57
19.46
88,05
35.94
68.39
35.27
42.14
89.54
iFood (t-split) trad. distill
27.89
61.58
31.23
39.33
87.91
iFashion category task repr. consolid.
38.57
68.45
28.80
89.29
iFashion category task trad, distill
16.71
54.53
11.49
14.93
89,80
Cars + Resise45 iFood (t-split) repr. consolid.
45.31 45.29
70.84
30.90 30.14
37.62
89.87
Cars196 + Resise45 iFood (t-split) trad. distill
67.05
35.44
87.97
Table 6: Fine-tuning, 5-shot results raw numbers for Fig. 10 in the appendix
Food101 (all)
Flowers (all)
Birds
Cars196
Resise45
iFood
Airerafts
(all)
ImageNet-prette pretrrirtiind model mrageNet-pretrained, soft-label
90.90
96.93
78.55
88.15
87.57
92.32
80.82
97.20
78.92
88.13
87.87
92.54
81.76
Cars196 (I-split) repr. consolid,
91.84
97.08
78.12
87.91
87.81
92.45
81,46
Cass196 (1-split) trad. distill
91.35
96.43
75.76
86.10
86.91
85.35
75.39
Resise45 (t-split) repr. consolid.
91.03
97.14
78.57
88.02
87.75
92.08
81.55
Resise45 (1-split) trad, distill
87.41
96.71
75.90
86.09
85,42
85.
71.57
iFood (I-split) repr. consolid.
90.75
96.86
78.34
88.00
87.75
92.26
80.20
iFood (t-split) trad. distill
89,44
96.34
77.24
87.33
85.65
88.42
76.30
iFood (t-split) teacher
89.09
96.21
77.76
87.92
84.46
87.97
75.92
Aircrafts (t-split) repr. consolid.
90.93
97.08
78.20
88.01
88.94
91.97
81.15
Aircrafts (t-split) trad. distill
87.31
95.87
75.06
85.29
89,54
81.79
72.30
(Naturalist (t (t-spiit) repr. consolid.
91.08
96.74
78.16
87.92
87.81
93.62
82.36
iNaturalist (t-split) trad. distill
88.62
96.02
76.64
86.60
86.61
90.89
79.89
iNaturalist (l-spiit) teacher
89.37
96.37
76.89
87.09
85,59
91.22
79.62
Cars196 + Resise45 ioodd (1-split) repr. consolid.
91.05 91.05
96.74 96.71
78.42
88.15
16
92.76
81 81.43
Cass196 Resise45 iFood (t-split) trad. distill
77.47
87.38
87.27
89.71
77.29
Table 7: Fine-tuning, full-shot results numbers for Fig. appendix.
the main paper (Figs.
11 in the
20


Reviews:
[{"review_id": "xPMozI_UQfo", "reviewer": "Reviewer_vU3z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of the paper is the  _representation consolidation_ method to consolidate knowledge from multiple task-specific teacher models into a single expert student, given a generalist teacher is available. This boils down to adding a generalist teacher to the training. \n\nThe main findings are that consolidating a task-specific teacher with a generalist teacher is sufficient to rescue the student and consolidation performs only slightly worse/similar to a multi-task joint training oracle, at least on image classification tasks. Which brings about one of the main issues of this paper - despite proposing a general framework, only the image classification problem has been addressed. I would have liked to see more complex tasks such as dense predictions can benefit from the proposed method (e.g.., optical flow, depth estimation). The main insight here is - if we can get a generalist that works well on most scenarios, could we squeeze additional performance from the proposed method or not? Otherwise, the main claim of this paper would be - keeping a generalist teacher improves _any_ transfer learning. Which is kind of a bold statement for ImageNet / iFood testing only, IMHO.\n\nFurthermore, most of the times few-shot learning is illustrated, the few examples of full fine tuning are less favorable to the proposed method. \n\nOtherwise, the paper is well written and illustrated, and the experiments support the claims, in the scenario proposed by the authors.\n\n\nStrenghts:\n- general method for a better distilled student\n- easy to implement, provided a generalist is available for the proposed task\n- no data replaying for the generalist network during fine-tuning\n\nWeaknessses:\n- simple training scenario, mostly classification, few-shot learning\n- where fine-tuning is employed (Fig 4/ 10) the results are less convincing", "summary_of_the_paper": "The paper proposes a task-agnostic Knowledge Distillation method that includes a generalist model as an additional teacher, to limit student forgetting and representation collapse. The authors call this method representation consolidation and claim it can benefit both related and unrelated downstream tasks. The authors experiment with ImageNet and iFood image classification tasks, the accuracy of the proposed method is generally higher than using Knowledge Distillation only.", "summary_of_the_review": "I lean towards weakly rejecting the paper, based on the limited applicability/experiments mentioned above. If there is a general claim to be made, I would like to see more evidence and other domains tested, beyond image classification. As it stands, I do not find the technical contribution sufficient for acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "j0cNZWfHTIu", "reviewer": "Reviewer_6iCZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- In this paper, the authors have stated their key difference from the traditional distillation loss item is the generalist teacher item. Thus, I feel the contribution  is not that strong enough, since the generalist teacher is the pre-trained network.\n\n- In addition, the generalist teacher item is just the model pretrained on imagenet dataset. What if in the other situation that the task is in NLP or speech recognition? Does it mean that the method can’t generalize well?\n\n- In order to avoid catastrophic forgetting, do the authors try to add regularization item like the works of incremental learning do?\n\n- The work here aims at transfer learning, but the experiments are set with the classes in the same domain.\n\n- Some advice in the experiments, the chart figures are not very suitable here.\n", "summary_of_the_paper": "- The authors propose a method of learning a consolidated image feature representation from a collection of related task-specific teachers that transfer well on novel recognition tasks.\n\n- To achieve it, the authors utilize multi-teacher multi-task model distillation framework that jointly distills one or several task-specific teachers with a generalist one (trained with Imagenet dataset). \n\n- Each teacher is set to operate on a different set of classes, and a multi-head student is trained to emulate all teachers.\n\n- Experimental results show that the proposed method doesn’t need to revisit original training data of each teacher, but gains better performance.\n", "summary_of_the_review": "- Besides the generalist teacher, what is the other contribution?\n\n- Please discuss the generalization ability of the proposed method.\n\n- Discuss more about the experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Nx596B8xqYD", "reviewer": "Reviewer_HSTC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This paper is well written and easy to follow. I think the study multi-teacher consolidation problem is interesting and important.  In some practical domains (like healthcare), it is difficult to construct a large dataset to train a model,  how to conduct transfer learning from multiple datasets (models) is critical.\n\n2. The authors conduct extensive analysis on multiple datasets.\n\nWeakness:\n1. The technical contribution is limited, as the method is somehow ad-hoc. Different from previous multi-teacher-distillation works, the authors add a generalized model (ImageNet model) during knowledge distillation to prevent representation collapse. However, there is no deep analysis of why this can achieve such a goal. Moreover, this strategy simply treats the generalized model and other teacher models equally, while there should be different roles during the distillation between the generalized model and teacher models.\n\n2. As the technical contribution is limited, I would appreciate the merits of experiments.  Although the authors conducted extensive analysis, most of them are in a few-shot learning setting (i.e., using only several images to train the downstream networks) not a fine-tuning setting. For the few-shot learning setting, the performance of different experimental runs has large perturbations, it is truly difficult to determine the effectiveness of the proposed method. \n\n3. The authors also conduct the experiments under full fine-tuning in Exp. 3. However, the improvement of the proposed method compared with \"imagenet-pretrained + fine-tuning\" is quite incremental (see Figure 4). Therefore, it is difficult to sufficiently support the advantage of the proposed multi-teacher consideration method with the existing experiments.\n\nOther comments:\n\n1. In the first paragraph, the authors mention the overall goal is to \"enrich the diversity of the expert library by accumulating knowledge from transferred downstream tasks\" and in the second paragraph, the authors argue that adding fine-tuned downstream models back into the library is naive and does not scale. However, to achieve the overall goal, the proposed multi-model consideration method still seems to depend on this naive strategy. Maybe the authors can reorganize the motivation part in the introduction to avoid confusion.\n\n2.  It is better if the authors can briefly summarize the size/class distribution of each dataset in the experimental setup section. \n\n3.  As there are performance perturbations for different experimental runs, especially under the few-shot learning setting. Are the results in the tables the average values of different runs or single runs? If they are the results of different runs, it is better to show the std of the results.", "summary_of_the_paper": "This paper studies the problem of multi-model consolidation, which aims to merge (consolidate) the knowledge from multiple models into one model so that we can better transfer this merged model for downstream tasks. Different from the traditional knowledge distillation or multi-teacher distillation, the proposed method aims to transferability of merged models for the downstream tasks, whilst achieving good performance on source tasks. The key technique (technical contribution) is to add a generalized model during the knowledge distillation, which can be regarded as s \"special\" teacher model.  \n\nThe authors conduct extensive experiments on multiple datasets and most of the experiments are under a few-shot linear probe transfer learning setting.", "summary_of_the_review": "The studied problem is interesting and important and the whole paper is easy to follow, while the technical contribution is limited and the experiments are not very strong to support the advantage of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Cx60PgTSfrg", "reviewer": "Reviewer_Swfw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The problem of how to properly use multiple pre-trained models for downstream transfer learning is interesting and practical in real life.\n\n2. The introduced method achieves better performance than distillation from a single ImageNet-pretrained teacher or task-specfic teacher.\n\nWeaknesses:\n\n1. This paper tells a big story of ``life-long meta-learning'' in the beginning, but fails to verify it in the method and experiment part. Specifically, the authors argue that they would like to automatically enrich the diversity of the pre-trained model zoo by accumulating knowledge from downstream tasks. However, I cannot find any evidence of enriching the diversity in this paper.\n\n2. The authors claim that the generalist teacher is important to the transferability of distilled representations. It should be important to carefully discuss the choice of the generalist teacher in this framework, e.g., how does it perform when adopting different or multiple generalist teachers. I did not find anything about it.\n\n3. Similarly, how to properly select task-specific teachers for corresponding downstream tasks? How much does it affect when choosing different numbers of task-specific teachers? In the experimental part, the authors only show the comparisons when adopting ad-hoc teachers, which is less convincing.\n\n4. As shown in Fig. 1, I find the comparison between *distillation directly to the downstream task* and *distillation first and then finetuning to the downstream task* interesting. It is necessary to conduct ablation studies on this comparison while keeping the teachers (a generalist teacher + multiple task-specific teachers) the same. \n\n\n====================\n\nPost-rebuttal:\n\nThanks for the authors' efforts. But unfortunately, I choose to maintain my original score considering the other reviewers' concerns and the overclaiming problem (the method and experiments fail to support the motivation) as I mentioned in the initial review.", "summary_of_the_paper": "The paper introduces a representation consolidation method to properly aggregate the pre-trained knowledge from multiple teachers for transfer learning. It claims that a generalist teacher is necessary for preserving the transferability of distilled representations, and task-specific teachers contribute to better performance in the same-domain downstream tasks. The introduced method performs on par with multi-task training while neglecting the need for teacher datasets.", "summary_of_the_review": "The motivation of this paper (life-long meta-learning) is attractive, but the method and experiments fail to well support it. I tend to reject this paper, and recommend the authors refine it and submit it to the next conference. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors present an approach to consolidate multiple teachers into a single student model that can be adapted to new tasks. The method involves using a proxy dataset to facilitate distillation to prevent having to replay images from the teacher datasets. A multi-task multi-head objective is utilized, agnostic to the loss function, in which two are studied. Downstream task performance is used as the performance measure.

Pros:
- The problem of how to best leverage multiple teachers for a downstream task is important and interesting.
- Presents a method to generate distilled students that can be finetuned to tasks that demonstrates performance gains over baselines (imagenet alone or task specific teacher). 
- Easy to follow and implement.
- Analysis across multiple datasets.

Cons:
- Multiple reviewers expressed concerns about current level of novelty / contribution. In some sense, it is natural to expect that combinations of task-related and generalist distillation would improve performance.
- Main results demonstrate improvements in performance when teacher and tasks are related to one another. But authors do not address how to select task-specific teachers for distillation. Related tasks and their matching to the target task are assumed to be known. Authors cited related prior works that attempt to do this matching, but do not apply it to their study for a full solution.
- Authors do not study variations of generalist teachers. How does changing the generalist teacher impact performance?
- Some reviewers expressed concern presentation is not clear. In particular, the style of figures may not be appropriate to best convey results and analyses of this type of work. Comparing different approaches is difficult looking at thin lines. Tables are perhaps better suited to convey these results.
- Multiple reviewers expressed concerns full-finetuning results are not convincing (Fig 4), though few-shot results look more convincing

Authors and reviewers had interaction, but reviewers maintained their recommendation of weak reject. All reviews unanimous in their decisions. Authors are encouraged to take into consideration all the comments and submit to another venue.
Decision: Reject