Learning Efficient Models From Few Labels By Distillation From Multiple Tasks
Authors: Kenneth Borup
Cheng Perng Phoo
Bharath Hariharan
Venue: Submitted to ICLR 2023 (2023)
Keywords: transfer learning
semi-supervised learning
multi-source distillation

Abstract: We address the challenge of getting efficient yet accurate recognition systems that can be trained with limited labels. Many specialized applications of computer vision (e.g. analyzing X-rays or satellite images) have severe resource constraints both during training and inference. While transfer learning is an effective solution for training on small labeled datasets it still often requires a large base model for fine-tuning. In this paper we present a weighted multi-source distillation method; we distill multiple (diverse) source models trained on different domains, weighted by their relevance for the target task, into a single efficient model using limited labeled data. When the goal is accurate recognition under computational constraints, our approach outperforms both transfer learning from strong ImageNet initializations as well as state-of-the-art semi-supervised techniques such as FixMatch. When averaged over 8 diverse target tasks our method outperform the baselines by 5.6%-points and 4.5%-points, respectively.

Under review as a conference paper at ICLR 2023
LEARNING EFFICIENT MODELS FROM FEW LABELS BY DISTILLATION FROM MULTIPLE TASKS
Anonymous authors Paper under double-bling review
ABSTRACT
We address the challenge of getting efficient yet accurate recognition systems that can be trained with limited labels. Many specialized applications of computer vi- sion (e.g. analyzing X-rays satellite images) have severe resource constraints both during training and inference. While transfer learning is an effective solution for training on small labeled datasets it still often requires large base model for fine-tuning. In this paper we present weighted multi-source distillation method; we distill multiple (diverse) source models trained on different domains, weighted by their relevance for the target task, into single efficient model using limited labeled data. When the goal is accurate recognition under computational con- straints, our approach outperforms both transfer learning from strong ImageNet initializations as well as state-of-the-are semi-supervised techniques such as Fix- Match. When averaged over 8 diverse target tasks our method outperform the baselines by 5.6%--pinnt and 4.inss, respectively.
INTRODUCTION
With recent advances in recognition, there is an increasing interest in deploying deep networks in a variety of downstream applications, be it an- alyzing X-rays, skin conditions, or satellite im- ages. However, in contrast to the increasingly massive training datasets and large networks that power advances in deep learning, many of these downstream applications have severe resouree constraints. In particular, labeled training data expensive. addition, we often require fast and cheap inference privacy and availability concerns, as well as practical limitations often re- quire deployment be local devices. These constraints immediately rule out the standard ap- proach of training large neural network on mas- sive amounts of training data. A key research question for these applications is thus: how do we get efficient but accurate recognition systems that we can train with limited labels?
Source Models
Evaluate
Task Similartty Metrics
Sec. 42
Stallarity
Compute Task Similarity for Source Model Weighting
P
Target Probe Set
Sec. 3.2
t6
90 02 Thock Acmunicy
Multi-Souree Distilation
Sec.
Labeled Target Set
Target Set
Figure I: Overview of our paper and method.
A common approach for dealing with small labeled datasets is transfer learning. Here, one pre-trains a base model on a different problem where large labeled datasets are available, and then fine-tunes this model on the application of interest. This transferred model will often be large, but can then be distilled into a smaller model for deployment (Ba Caruana 2014 Hinton et al.) 2015). While in principle this approach can be effective, in practice this relies heavily on having good source model that is relevant to the target task. There are many ways to compare and choose the best base model (Achille et al.; 2019; Kornblith al. 2019b Recht et al. 2019: Bolya et al. 2021). But this assumes that a single optimal base model exists. What no single source model matches the target task, is likely going to be the case for new problem domains?

---

Under review as a conference paper at ICLR 2023
Source Model Weighting
Multi-Source Distillation
Labels
Laboleed Ropect Tats (Pt)
AlR.. Y)
Av
glabsied
Opp
Labels
ulks. X)
Unlsbeled Tarpet (P!)
Labeled Target Data (D.)
Figure 2: While common multi-source distillation usually weighs a set of S source models, M, h1 o Φs. equally for every target task, we propose to weight the source models by using task similarity metries to estimate the alignment of each source model with the particular target task using a small subset of labeled data, Dr. Since the task similarity metrics are independent feature dimension, can utilize source models of any architecture and from any source task. We show that choosing the weighting, 01,..., a.g, this way we are able to improve performance over transfer from ImageNet and training with FixMatch (see e.g. Table and Figure 3).
This challenge motivates our approach. We propose to produce efficient yet accurate models for new tasks with few labels by task- wimilaarityweiigteee multi- sourc- distillation (see Figure (2) That is, we distill multiple (diverse) source models trained on different domains, weighted by their relevance for the target task. All this is done without access to any other data than that of the target task. Specifically. we propose to first rank diverse set (both in architecture and task) of source models for a particular target task using a task similarity metric. This ranking is used to select and weight the most relevant source models to distill into target model of some suitable arrget-architecture in a semi-supervised learning setting using multi-source distillation (see Figure TT.
Contributions. We summarize our contributions as follows: 1) By analyzing over 200 distilled models we extensively verify that, for single-source cross-domain distillation, the choice of source model important for the predictive performance the target model. 2) We show that task similar- ity metrics can be used to select and weight source models for single- and multi-suurce distillation without access to any source data. 3) We show that our approach yields the best accuracy on multiple target tasks under compute and data constraints. We compare our task similarity- weighted multi- source distillation to two baselines; classical transfer learning and FixMatch, as well as the special case of multi-souree distillation with equal weighting. Averaged over diverse datasets, our method outperforms the baselines with at least 4.5% 4.5%-points and in particular 17.5%-points on CUB200.
similarity-weighted
2 RELATED WORK
We review several research areas relevant to our problem setup and approach.
Knowledge Distillation One key aspect of our problem to figure out how compress multiple models into an efficient target model. A common approach is knowledge distillation (Ba & Caruana 2014 Hinton al. 2015) where an efficient student model is trained to mimic the output of larger teacher model. However, most single-teacher (Adriana et al. (015)] Mirzadeh et al.) 2019 Park et al.)
2019 Cho & Hariharan 2019 Borup & Andersen, [022]] or multi-teacher know knowledge distillation
You et al. 20177; Fukuda al. 2017: et al.) 2019 Liu et al., 20200 research focuses on the closed set setup, where the teacher(s) and the student both attempts to tackle the same task. To the best of our knowledge, compressing multiple models specializing in various tasks different from the target task has rarely been explored in the literature. Our paper explores this setup and illustrate that
carefully distilling multiple source models can bring forth efficient yet accurate models.
2

---

Under review as a conference paper at ICLR 2023
Semi-Supervised Learning and Semi-Supervised Transfer Given target task is specified in a suni-isuereesed setting, is customary to review semi cemi-supereised learning (SSL). The key to SSL
approaches is how to effectively propagate label information from small labeled dataset to large
unlabeled dataset. Along this vein, methods such as pseudo-labelingssef--rr training ([ee et al.
2013
Xie et al.; 2020) or consistency regularization (Tarvainen & Valpola 2017 Berthelot et al.
2019
Sohn et al. 2020) have shown remarkable results in reducing deep networks dependencies on large labeled datasets via unlabeled data. However, most SSL approaches focus on training models from scratch without considering the availability of pre-trained models. Given the increasing availability of large pre-trained models (Paszke al.] 2019 WWof et al., 2019). recent work has started exploring the intersection between transfer learning and SSL (Phoo & Hariharan, 2021 Islam et al. 2021 Abuduweili et al., 202T). However, most of these works focus on how to transfer from a single pre-trained model to the target task. Our paper, however, explores an even more practical setup: how to transfer from multiple pre-trained models to a downstream task where in-domain unlabeled could combine
data are available. In principle, SSL to (potentially) gain even larger improvements, but to keep our method simple we leave such exploration to future work and focus on how to better utilize the available set of pre-trained models.
approach with a lot of previous work on
Multi-Source Domain Adaptation Our setup also bears a resemblance with multi-source domain adaptation (MSDA) (Peng et al.] 2019) in which the goal is to create a target model by leveraging multiple source models. However, MSDA methods often assume that the source and target models share the same label space to perform domain alignment. We do not make such an assumption and in fact, focus on the case where the label space for source and target tasks has minimal to no overlap. Besides, a lot of the MSDA approaches (Zhao et al., 2018 Xu et al., 2018 Peng et al., 2019 Zhao et al. 2020) rely the availability of source data the fact that the source and target tasks share the same model architecture to build domain invariant features. Given the discrepancy in assumptions between multi-source domain adaptation and our setup, we do not consider any methods from this line of work as baselines.
Transfer Learning From Multiple Sources Transfer learning from multiple different tre-trained models has been explored in different setups. [Bolya et al.] (202T) focuses on how to select a single good pre-trained model to use as a model initialization whereas we explore how to construct an efficient model from the pre-trained models (i.e. our target architecture could be different from those of the source models). AAgostinelli et al. (2022) focuses on how to select a subset of pre- trained models to construct an (fine-tune) ensemble, whereas we focus creating single model that could potentially have different architecture complying with computational constraints. [L et al.) 2 focuses creating generalist representation by distilling multiple pre-trained models using proxy/source data (which often requires high capacity models) whereas our goal is to construct an efficient specialist model using the target data. All these works have indicated the importance of exploring how to best leverage a large collection of pre-trained models but due to differences in setup and assumptions, we do not (and could not) compare to them.
Task Similarity / Transferability Metrics The key insight of our approach is to leverage the similarity between the target and source tasks to weigh different pre-trained source models during distillation. Characterizing tasks (or similarities between tasks) is an open research question that has had various successes, A common approach is to embed tasks into a common vector space and characterize similarities between tasks by distances in said space. Representative research along this line of work include Achille et al. (2019): |eng et al. (2020); Wallace et al. (202T). Another related line of work investigates transferability metrics (Tran et al. 2019 Bao et al. 2019 Nguyen et al. 2020 Dwivedi et al. 2020 Dwivedi & Roig 2019 Bolya et al., 2021). After all, one of the biggest use cases of task similarities to predict how well model transfers to new tasks (models trained on similar tasks likely are good candidates for transfer learning). Since it is not our intention to establish which task similaritytrransffrablllt metric is the best for distillation, we use established metrics that capture the similarity between source representations and one-hot labels to weigh the source models. Under this purview, metrics that characterize similarities between features such as CKA (Cortes al. 2012, Kornblith et al. 2019a) and transferability metries based on features (Dwivedi & Roig 2019 Bolya et al. 2021) fits the bill. We evaluate the correlation between these metrics and distillation performance in Section 4.2.1
3

---

Under review as a conference paper at ICLR 2023
Accuroo
100
70
01000 1000
40
Stanfaed Dups
(0.04)
(0.03)
(0.03)
(0.05)
(0.84)
(0.38)
(0:34)
[0.02]
Figure 3: Distillation results for multiple target tasks with each dot representing the test accuracy after distillation from a source model. The coloring represents the task similarity associated with the particular source model (from small to large;
We include the performance when fine-tuning
ImageNet (-), distilling the source model highest ranked by task similarity (>), ). and performing
multi-source distillation with equal weights (=), and weights proportional task similarity power
(-), and 12 ( (=), respectively. The numbers in parentheses are Spearman correlations between the task similarity and test accuracy for the single-source distilled models of each target task.
3 PROBLEM SETUP AND METHOD
We want train accurate model for target task. where we have limited labeled data and compu- tational constraints (e.g. limited compute resources/storrge) during inference. As discussed above, want leverage various pre-trained models (i.e. different architectures trained on different pre- training tasks). Formally, we assume that our target task is specified via small labeled training set D To mitigate the data problem, we propose leverage: (a) set S {M&)s of S source models M, trained on various source tasks different from the target task and (b) a set of unlabeled data D. associated to the target task.
To reflect computational constraints, we assume that the architecture of the target model Mr must be chosen to fit these constraints. This might mean that the architecture we need may be available among the source models in S. We also assume no access to any source data which could be practical due to storage, privacy, and computational constraints.
3.1 SOURIE DISTILLATION
Since the target model might not share the same architecture with any of the source models, we propose train target model by distilling from multiple source models, with each source model weighted based on its relevance for the target task (See Figure 2).. For simplicity, we restrict our models (regardless of source or target) classification models that could be parameterized as M o & (the feature extractor & embeds input x into a feature representation and the classifier head, maps the feature d(x) into predicted class probabilities, P x)).
To train the target model M, = hy o Φy, we utilize a weighted sum of two loss functions. The first loss function is the standard supervised objective,
C labeled det
Σ (X);))ED;;
fCE (x.(),
N
where (. is the classical cross-entropy loss. The second loss function consists distillation objective for each source model M, = h2 &.. Specifically, for each source model (indexed by s) the distillation objective
distill
Σ X
le
for 1,,...,
N
where he is additional classifier head that maps the features from the target task feature extractor of to the label space of the source task (to account for the difference label space). Note, we discard all these classifier heads h after training and merely keep the target classifier head h,-
4

---

Under review as a conference paper at ICLR 2023
Combined, the overall objective function is as follows,
L del A.Cabelled + (1- A)
Σ
a2 C distill
where X, 01,,........................1... .1....,03.. .g E [0, 1]. Here a, is the relative weight assigned each source model such that E Σx= 0, 1. In principle, we could add additional cuni-supervised losses, such as the FixMatch loss (Sohn et al. 2020) to propagate label information from the labeled set D', the unlabeled set Dr for better performance, but we avoid such additional complexities (e.g hyperparameters) and confounding the effect distilling from multiple models. We leave such exploration to future work.
3.2 TASK RELEVANCE WEIGHTING FOR MULTI-SOURCE DISTILLATION
Simply assigning equal weight (i.e. 01 optimal (e.g. weighing source models trained on ImageNet and Chest X-ray equally might not be optimal for recognizing birds). As such, we propose compute the source weights {aa hetss as
as = 1/S) to all source models is usually sub-
where
Eg - 1 Eg for j = 1....,,
where is similarity between the s-th source tasks and the target task, and is hyperparameter used to rescale the distribution of the weights. Larger p would assign more weight to the most relevant source models, while p 0 corresponds to equal weight and p assigns all weight to the single most relevant source model.
Quantifying similarities between tasks is open research question with various successes (Achille et al. 2019 Nguyen et al. 2020). For simplicity, we pick our similarity based on one simple intu- ition: target examples with the same one-hot label should have similar source representations and vice versa. Along this vein, the recently introduced metric, PARC, fits the bill (Bolya et al. (021).
For convenience, we briefly review PARC. Given small labeled probe DJ ((x,, y:) v))) 2= and a source representation of interest o.. PARC first constructs two distance matrices Da,, Dy based on the Pearson correlations between every pair of examples in the probe set;
Dæ,
(x.:))
Dy
(eyrson((y:))) (-m).
PARC is computed as the Spearman correlation between the lower triangles of the distance matrices;
e = PARC( Y) = spearman (fDo, [t. (ill,, [iy(i.J]]ss))
Intuitively, PARC quantifies the similarity of representations via comparing the (dis)similarity struc- tures of examples within different feature spaces: if two representations are similar, then (dis)similar examples in one feature space should stay (dis)similar in another feature space. In Figure we show how ranking source models by PARC correlates well with distillation accuracy, and that selecting an appropriate source model can yield significant improvements.
4
EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL SETUP
Benchmark. For our setting, we modify an existing transfer learning benchmark: Scalable Diverse Model Selection (SDMS) Bolya et al. (202T). In particular, we used the publicly available models to construct the set of source models for each target task. This full set consists 32 modely"] architectures (AlexNet, GoogLeNet, ResNet-18, and ResNet-50 (Krizhevsky al. 2012 He al.) 2016)) trained on 8 different tasks. For the target tasks, we consider 8 different tasks covering various image domains (Natural images: CIFAR-10, CUB200, NABird, Oxford Pets, Stanford Dogs; X- ray: ChestX; Skin Lesion Images: ISIC; Satellite Images: EuroSAT). For the target architecture, use MobileNetV3 (Howard et al. 2019) due to its lower computational requirements compared to any of the source models. We refer the reader to the supplementary material for further details on
We carefully leave any source models associated with the target task if such exists.
5

---

Under review as a conference paper at ICLR 2023
Test: Accuracy
100
Method
80
17.5x
78
6,4x
76
74
IN+Transfor
72
20
Multi-Soures
70
GFLOPS
Equal
Single
Similarity
Souroe-Weighting
Figure 4: Mean test accuracy over the 8 target tasks compared to compute requirements for a single forward pass inference. Using multi- source distillation allows for training of signif- icantly smaller models with small to loss in predictive performance.
Figure 5: Average test accuracy over five tar- get tasks for different source-weighting meth- ods: INVERSE is TASK SIMILARITY weights applied to a reversed sorted list of source mod- els, while RANDOM is sampled weight- on the uniform 4-Simplex. When applicable p= 12.
implementation. We also experimented with VTAB (Zhai et al. 2019) where observe similar results on the natural and specialized tasks. Results on VTAB can be found in Appendix A
Baselines. To show the importance of selecting the right source model distill, consider two baselines based on ImageNet initializations: IN+TRANSFER fine-tunes ImageNet representations using the labeled data, and IN+FIXMATCH fine-tunes the ImageNet representation using laheled and unlaheled data (via FixMatch). We also consider distilling all source models with equal weights show the importance of using task similarity-eeihhiin Additionally, we report the expected performance if we were to pick single source model at random (SINGLE SOURCE (AvG.)).
4.1.1 RESULTS
We present our main results in Figure 3 with exact (and additional) values reported in Table [
Distillation is better than fine-tuning from ImageNet. We observe that within the same target architecture (MobileNetv3)) simply fine-tuning ImageNet representations (IN+TRANSFER) less optimal than distilling from the most similar SINGLE-SOUREL model (selected with our task simi- larity metric). In fact, for fine-graine datasets such as CUB200, NABird, Oxford Pets, and Stanford Dogs, we observe that distilling from appropriate source model SINGLE-SOURR could yield much better performance than fine-tuning from generalist ImageNet representation. More surpris- ingly, even with the aid of unlabeled data, models fine-tuned from ImageNet representations using a label propagation style approach (IN+FiXMATCH) still underperform distillation-ban based methods by at least 3.9% average (Table T..
Distilling all sources equally is not always optimal. Compared to distilling from the most similar source -SINGLE-SOURCE or from a of sources weighted by task similarity (MULTT-SOURCE p 12), distilling all source models equally is far less optimal (see also Figure 5. This vindicates the importance distillation using weighted source models and accentuates that distilling additional irrelevant source models can decrease performance.
Distilling from multiple source models can outperform distilling single best source model. On Oxford Pets (classification of different breeds of cats and dogs), we observe that distilling from multiple weighted sources (MULTI-SOURCE gle most similar source MANGLE-SOURCES which a ResNet-18 trained on Caltech101 (that can recognize visual concepts such Dalmatian dog, spotted cats, cougars, etc.). Although the most similar source model contains relevant information for recognizing different breeds dogs and cats, it might not contain all relevant knowledge from the of source models that could be conducive
12) is much better than distilling from the sin-

---

Under review as a conference paper at ICLR 2023
Table 1: Multi-source distillation compared baselines. Despite requiring fewer computations, MobileNetV3 models trained with our method are highly competitive with baseline methods for more demanding model architectures. We highlight the top methods, which comply with compute requirements (i.e. MobileNerV3) for each target task by bold, underline, and italic, respectively.
Stanford Dogs
CIFAR-10
Oxford Pets
CUB200
Chsst
EmoSAT
NABied
15IC
Menn
AlexNet
IN+Transfer
85.0
18.4
46.2
91.9
67.8
13.0
50.9
29.1
50.3
GILOPN
Fine-tune Selected Source
88.0
30.4
42.9
89.8
74.5
17.9
66.8
41.3
56.5
GoogLeNer
IN+Transfer
91.8
42.8
41.4
96.8
80.5
36.5
84.8
65.9
67.6
(1.1 GFLLPR)
Fine-tune Selected Source
91.6
61.2
48.6
96.9
78.3
33.0
87.8
71.8
71.2
ResNer-18
IN+Transfer
92.2
37.8
45.2
96.6
80.2
34.0
80.2
58.2
65.6
GILOPS)
Fine-tune Selected Source
91.3
58.2
46.4
97.0
75.8
35.4
80.7
69.3
69.3
ResNer-50 (4.4 GFLORN)
IN+Transfer
92.9
42.0
43.4
96.8
79.9
39.9
83.3
65.9
68.0
Fine Selected
93.0
70.8
97.2
79.3
74.7
IN+Transfer
92.4
42.8
47.3
97.4
81.6
37.3
75.9
62.6
67.2
IN+FisMatch
93.5
41.9
38.5
98.1
82.6
42.8
83.4
63.8
68.3
MobileNerV3 GFLOP))
Single Source (Avg.)
89.6
46.5
46.6
97.4
81.8
39.0
79.4
61.9
67.8
(Ours) Multi Source (Equall
90.8
53.5
45.7
97.5
81.5
41.4
82.1
62.1
69.3
(Ours) Single-Source (p + oc)
92.0
59.6
46.8
97.4
81.0
47.4
81.9
71.3
72.2
(Ours) Multi-Source (p = 12)
92.0
60.0
47.7
97.6
82.2
48.3
84.4
69.9
72.8
to recognizing all visual concepts in Oxford Pets. In fact, we observe that the second most similar model is a GoogLeNet model trained on Stanford Dogs that could recognize more different breeds of dogs than the most similar source model (but incapable of recognizing cats). By distilling from mul- tiple weighted sources, we are able to effectively combine knowledge from different source models for accurate target model compared to target model distilled from single source.
Distilling efficient architecture could be better than fine-tuning large models. Figure 4 (and also Table [D. we include the performance when fine-tuning larger architectures trained on ImageNet and the source model (of same architecture) most similar to each target task. few ob- servations is immediate: (a) our choice of task similarity metric is effective for transfer; across all 4 architectures, we observe at least 4% improvement over simple fine-tuning from ImageNet, which validates the results by Bolya et al. (2021), and (b) with the aid of unlabeled data and distillation, the computationally efficient architecture MobileNetV3 can outperform larger architectures fine-tuned on labeled data from the target task (i.e. AlexNet, GoogLeNet, ResNet-18). Although underper- forming fine-tuning a ResNet-50 initialized with the most similar (ResNet-50) source model by a mere average of 2%-points (FINE-TUNT SELECTED SOURCE), using ResNet-50 would require 17.5x more computations during inference to achieve such improvements.
4.2 ABLATIONS
4.2.1
TASK SIMILARITY FOR DISTILLATION
There are many existing metrics for quantifying task similarities but their effectiveness for distilla- tion remains unclear. Given the myriads of metrics, we restrict our focus metrics that can capture similarities between a source representation of target example and one-hot label representation. Along this vein, two questions arise: which metric to for comparing representations and which representations from source model should be used to represent a target example?
For the first question, we look into multiple metrics in the literature that compares various represen- tations: CKA (Cortes et al. 2012), RSA (Dwivedi & Roig 2019), and PARC (Bolya et al. 2021); for the second question. look into the common representations from source model: the feature o and the probabilistic output o d. For simplicity, we focus on single-source distillation (i.c. how to pick the best source model for distillation).

---

Under review as a conference paper at ICLR 2023
Table 2: Spearman correlation between test ac- curacy after all possible single-source distilla- tions and task similarities associated with the source models. We do not use any heuristic e.g. Bolya et al. (202T). Generally feature representations correlate better with distillation performance compared to pseudo-label repre- sentations.
Relative accuracy of top-3 single- source distilled target models selected by task similarity over the average of the best models found in hindsight. We compute the average test accuracy of the top-3 highest ranked target mod- els (ranked by some task similarity) and divide this average with the average of the test accu- racy of the 3 best performing target models.
Table 3:
Stanford Dogs
Scanford Dogs
Oxford Pets
CIFAR-IR
CUB200
ChexX
EunSAT
NABird
CIFAR-11 10
CUB200
ChestX
EuoSAT
NABird
Oxford
ISIC
Msan
ISIC
Mean
Pseutoo
CKA
0.72
0.62
0.23
0.39
-0.04
0.31
0.69
0.11
0.38
Pscudo
CKA
99.1
95.6
97.4
99.6
98.8 89.4
100.0 97.6
97.2
PARC
0.79
0.02
0.17
0.06
0.72
0.45
100.0
95.5
99.6
99.7
99.7
RSA
0.82
0.31
00.11
0.30
0.10
-0.33
0.65
0.38
0.30
RSA
100.0
77.7
96.5
99.7
98.5
87.2
98.6
97.6
94.5
Feature
CKA
0.82
0.39
0.36
0.21
-0.04
0.47
0.69
0.55
0.43
Festure
CKA
100.0 95.6
97.0
99.8
99.0
93.3
100.0
96.4
97.6
PARC
0.84
0.84
0.18
0.42
-0.14
0.81
0.81
0.84
0.58
PARC
100.0
100.0
97.8
99.7
98.3
100.0
97.1
98.5
98.9
RSA
0.03
RSA
100.9 96.7
98.9
94,9
98.8
98.5
Tast Accuracy
100
Absolute Improvement
90
12.5
CUB200
10.0
Uxford Pets
7.5
NABtrd
Stanford Dogs
5.0
CIFAR-10
2.5
ChestX
ISIC
0.0
EuroSAT
-2.
12
0.00 0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
(Equal)
(Smglee)
Task Similurity
Metric Power (p)
Figure 6: Test accuracy of single-source distil- lation and (unnormalized) task similarity score calculated using PARC on the feature represen- tations. The scores are on different scales for different tasks, but all tasks (except ISIC) have positive Spearman and Pearson correlation be- tween test accuracy and task similarity.
Figure 7: Improvement over transfer baseline. Here * is the mean improvement across all tar- get tasks and each represents the performance on a target dataset with given p. Note, = corresponds equal weight for all source mod- els, and p = oc the single highest ranked model.
To establish the effectiveness of our choice of similarity metric, we report the Spearman correlation between the task similarities and the test accuracy of the distilled models in Table 2 We see that features from the source models can better capture the correlation between the source models and the test accuracy of the distilled models, than the probabilistic pseudo-labels labels. In addition, we also see much higher correlation among natural tasks (compared to specialized tasks such as ChestX, EuroSAT, and ISIC) which suggests that our choice of task similarity effective at selecting similar tasks. Besides, also observe much higher correlation when using PARC compared to the other metric, thus validating our choice of using PARC as the default metric.
To further establish that our choice of metric can be used to rank various source models, we look at the relative test accuracy between the top-3 models most similar to the target task and the top-3 best models after performing distillation (see Table 3). Again, observe that all of the three metrics capable of ranking affinity between source models, but ranking the models with PARC outperforms the other two metrics.
8

---

Under review as a conference paper at ICLR 2023
4.2.2 SHOULD YOU WEIGH BASED ON TASK SIMILARITY?
We have established that our choice of similarity metric can capture the correlation between the source model representations and the test accuracy of the distilled models. However, it is not prior clear that weighing different sources based on the ranking of their affinity to the target task would yield better performance for multi-source distillation. As such, we investigate different choices of weighing schemes for a subset of target tasks across different domains (CUB200, EuroSAT, ISIC, Oxford Pets, Stanford Dogs): INVERSE (source model weights are inversely proportional to task similarity). RANDOM (weights are samples from a 4-simplex, EQUAL (setting equal weights for all source models), TASK SIMILARITY (default), and SINGLE (distilling from the most similar source).
Through Figure 5 we find that distilling from single or set of source models ranked using the similarity metric (SINGLE, TASK SIMILARITY) is much more effective than distilling from source models that are weighted randomly or equally (RANDOM or EQUAL). In addition, the fact that INVERSE underperforms IN+TRANSFEL on average suggests that it is crucial to follow the ranking induced by the similarity metrics when distilling the sources and that the metric ranks not only the most similar source models appropriately, but also the least similar source models.
4.2.3 WHAT IS THE EFFECT OF P?
Our task similarity metric gives good ranking of which source models to select for distillation but it is unclear whether the similarity score could be used directly without any post-proe post-processing. To investigate, start by visualizing the relationship between the test accuracy of the models distilled from a single source and our task similarity. From Figure 6 it is clear that the distribution of task
similarities is dependent on the target task. This observation motivates our normalization scheme.
In addition, it is not apriori clear that the appropriate weights should scale linearly with respect to the similarity scores. Thus, we investigate the effect of the rescaling factor, p. for constructing the weights for distillation. In Figure 7 we see that although no rescaling (p 1) outperforms equal weighting (p- 0), is far less optimal than rescaling with p = 12 (our default). This suggests that the relationship between our task similarity and good weights is monotonic but non-linear.
4.3 ADDITIONAL ABLATIONS AND RESULTS
We include additional ablations and results in Appendix A but summarize these findings as follows.
MULTI-SOURCE distillation with ResNet-50 target model; averaged over 8 tasks, task muillarit--weighte multi-source distillation outperforms both TNTRRANFER and equally weighted multi-source distillation. Furthermore, when initializing our target model with the most similar ResNet-50 from the set of source models, performing multi-source distillation improves over multi- nolti-source distillation from an ImageNet initialization.
Multi-source distillation on VTAB; both MULTI-SOURCE and S-GGEE outper- form IN+TRANSFER averaged over the Natural and Specialized tasks.
Multi-source distillation with even fewer labeled samples; MULTI-SOURCE and SINGLE- SOURCE both improve in predictive performance over IN+TRANSFER, indicating our method works with even fewer labeled samples.
Additional investigations of task similarity metrics; we consider both Spearman, Pearson, and Kendall Tau correlation well additional measures of relative accuracy of the top-k selected models all supporting the usefulness of task similarity to weigh source models.
5 CONCLUSION
We investigate the use of diverse source models to obtain efficient and accurate models for visual recognition with limited labeled data. In particular, propose to distill multiple diverse source models from different domains weighted by their relevance to the target task without access to any source data. We show that under computational constraints and averaged over diverse set of target tasks, our method outperforms both transfer learning from ImageNet initializations and ofate-of-the- art semi- supervise techniques.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abulikemu Abuduweili, Xingjian Li, Humphrey Shi, Cheng-Zhong Xu, and Dejing Dou. Adaptive consistency regularization for semi-supervised transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9923-6932, 2021.
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowikes, Stefano Soatto, and Pietro Perona. Vec: Task embedding for meta-learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6429-6438, 2019. ISBN 9781 48038. doi: 10.1 10.1109/CCC220190000663
Romero Adriana, Ballas Nicolas, K Samira Ebrahimi, Chassang Antoine, Gatta Carlo, and B Yoshua. Fitnets: Hints for thin deep nets. Proc. ICLR, 2, 2015.
Andrea Agostinelli, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. Transferability Met- rics for Selecting Source Model Ensembles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7936-7946, 2022. URL httttt//aaxx...co
abs/2111. 13011
Lei Jimmy Ba and Rich Caruana. Do Deep Nets Really Need to be Deep? Advances in Neural Information Processing Systems, (UJauary):2654-266) 2014. ISSN 10495258.
Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An information-theoree approach to transferability in task transfer learning. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 2309-2313. IEEE, 2019.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural informa- fion processing systems, 32, 2019.
Daniel Bolya, Rohit Mittapalli, and Judy Hoffman. Scalable Diverse Model Selection for Accessible Transfer Learning, 2021. ISSN 10495258.
Kenneth Borup and Lars N Andersen. Even your teacher needs guidance: Ground-truth targets dampen regularization imposed by self-distillation. Advances in Neural Information Processing Systems, 14:5316-5555 22226,,,,,2225 2021.
Jang Hyun Cho and Bharath Hariharan. On the Efficacy of Knowledge Distillation. ICCV, 2019.
ISSN 00262714. doi: 10.1016/0266-711444999633
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. Journal of Machine Learming Research, 13:795-8222 2012. ISSN 15324435.
Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy & transfer learning. Proceedings the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June:1 13399-12388, 2019. ISSN 10636919. doi: 10.1 109/CVPR. 2019.01267.
2019-June:12339912288
Kshitij Dwivedi, Jiahui Huang. Radoslaw Martin Cichy, and Gemma Roig. Duality diagram sim- ilarity: generic framework for initialization selection in task transfer learning. In European Conference on Computer Vision, pp. 497-513. Springer, 2020.
Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ram- abhadran. Efficient knowledge distillation from an ensemble of teachers. In Interspeech, pp- 1697-3701,2011
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016 1e11--ecemm770-777.. 2016. ISSN 10636919, doi: 10.1109/CVP.22016.000
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network, arXiv preprint /503.00503.02553 pp. 1-9, 2015. URL http:://aaxii..rrr//b//////
10

---

Under review as a conference paper at ICLR 2023
Andrew Howard, Mark Sandler, Grace Chu, Liang- Slang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, and Le Hartwig Adaml. Searching for MobileNetV3. Proceedings of the IEEE International Conference Computer Vision, 2019- Octob: actoh:1314-10 1314-1324, 2019. ISSN 15505499. doi: 10. 109/1CCV.2019.0000
Ashraful Islam, Chun-Fu (Richard) Chen, Rameswar Panda, Lconid Karlinsky, Rogerio Feris, and Richard J. Radke. Dynamic Distillation Network for Cross-Domaie Few-Shot Recognition with Unlabeled Data. In M Ranzato, A Beygelzimer, Y Dauphin, P S Liang, and J Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. inwpppooceeeiinnnneeeie.pppssppp neurips.ee 3584-3595, Curran Associates, Inc., 2021. URL papppa
//axxiv.org/ab//200000
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual Prompt Tuning. arXiv, 2022. URL httt::tttttiiiiiee wrg/abs. wwa.bb/// abs//203. 2203.
12119
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General Visual Representation Learning. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lec- ture Notes in Bioinformatics), 12350 LNCS:491-507, 2020. ISSN 161 13349. doi: 10.1007/
978-3-030-58558-77 -2.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pp. 3519- 3529. PMLR, 2019a.
Simon Kornblith, Jonathon Beettte_ImageNe_MMMM Trranee..eedes.....ee........ Transfer.BBtteerr.. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2661- 2671,20199
Shlens, and Quoc
Le.
Korn- Proceedings of the
CCMPR_ paper.
Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton, Sutskever Ilya, and Geoffrey E Hinton. Ima- geNet Classification with Deep Convolutional Neural Networks. In F Pereira, CJ Burges, L Bot- tou, and K Q Weinberger (eds.). Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
Dong-Hyur Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop challenges in representation learning, ICML, pp. 896, 2013.
Zhizhong Li, Avinash Ravichandran, Charless Fowlkes, Marzia Polito, Rahul Bhotika, and Ste- fano Soatto. Representation Consolidation for Training Expert Students, 2021. URL http:
//arxiv.orr//abs//10. 08039
Yuang Liu, Wei Zhang. and Jun Wang. Adaptive multi-teacher multi-leve knowledge distillation. Neurocomputing, 415:106-11:: -13, 2020.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Hassan Ghasemzadeh, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher. arXiv preprint arXiv:1902.03399, 2019. URL
http:://axxt wtgg
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pp. 7294-7305. PMLR, 2020.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational Knowledge Distillation. In Pro- ceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019. ISBN 9781728132938. doi: 10.1 10.100//VPP.2009.0000

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL httpppppss/ppppceddng..... neurippg...erursss. wwwpeeepppppppppeeeeeeeeee.eee hcc/papper/211//Eiii bdbca288fee 200.03111111222111117727270-0000
pdf
Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang. Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Xingchao Peng, Yichen Li, and Kate Saenko. Domain2Vee: Domain Embedding for Unsupervised Domain Adaptation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.). Computer Vision - ECCV 2020, pp. 756-774, Cham, 2020. Springer International Publish- ing. ISBN 977-3-030-58539-6
Cheng Perng Phoo and Bharath Hariharan. Self-training For Few-shot Transfer Across Ex- treme Task Differences. In International Conference on Learning Representations, pp. 1- 19, 2021. (tLphtt:://axxi http:///a..e/0.0...00033oa////// Forum?1d=03y56aqpccc
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? 36th International Conference on Machine Learning, ICML 2019, 2019-
June:9413 9424,44444444 2019.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, Chun-Liang Liang Li, Zizhao Zhang. Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Han Zhang, and Colin A Raffel. FixMatch: Simplify- ing Semi-Supervised Learning with Consistency and Confidence. In H Larochelle, M Ranzato, R Hadsell, M F Balcan, and H Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 596-59666888 0666668.60 Curran Associates, Inc., 2020.
Xu Tan, Yi Ren, Di Hc, Tao Qin, Zhou Zhao, and Tic- Yan Liu. Multilingual neural machine trans- lation with knowledge distillation. arXiv preprint arXiv: /90.v 2019.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Anh Tran, Cuong V Nguyen, and Tal Hassner. Transfcrability and hardness of supervised classifi- cation tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 1395-1405 2019.
Bram Wallace, Ziyang WU, and Bharath Hariharan. Can we characterize tasks without labels or fea- tures? In Proceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1245-1254, June 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 10687-10698. 2020.
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3964-3973, 2018.
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1285-1294. 2017.
12

---

Under review as a conference paper at ICLR 2023
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski. Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark. arXiv, 2019. /R..ntt:://arxi..or//////a 04986
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gor- don. Adversarial multiple source domain adaptation. Advances in neural information processing systems, 31, 2018.
Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12975-12983, 2020.
13

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL RESULTS
A.1 RESULTS ON VTAB
We report the results of our VTAB experiment in Table On VTAB, We find that both MULTI- SOURCE and SINGLE-SOURCE distillation outperforms NNTRANNFER on each of the Natural tasks. Particularly, MULTI-SOURCE outperform IN+TRANSFER with 13.9%-points on CIFAR- 10 and 10.6%-points on Sun397 and averaged across Natural MULTI-SOURCE outperforms IN+TRANSEER with 5.1%-Tp 1%-ppinnss..... Average over Specialized both MULTI-SOURCE and SINGLE- SOURCE outperforms IN+NSSNEER although with small margin. Finally, averaged over Structured IN+TRANSEER outperform our methods, but due to the nature these tasks, we do not expect source models to transfer well to these tasks Yet. we still obtain the best accuracy DMLab, dSpr-Loc, and sNORB-Azimuth
Table 4: Top-1 accuracy by dataset in VTAB. The accuracy for each task is in grey, and the average accuracy for each category of tasks is in black. Note, the Mean is the average across all tasks, not categories. The largest value in each column is marked bold. Here MOLTI-SOURCE is with p 9.
Catochi
CIFAR-
DED
Flwerr
SVHN
Natural
Camelyon
EaroAAT
Retsee4
Retinopalyy
Specialized
Clevn Cocnt
Clevn Dist
DMLab
KiTT--Dist
dS on
dSpr Ori
SNORB- Aziio
SNORn Eley
Smootured
Monn
IN-Trnsfer
88.1
47.0
57.4
85.8
82.8
75.3
27.8
66.3
81.0
95.0
80.0
72.7
82.2
73.1
55.9
43.6
75.7
18.7
58.6
21.2
46.0
49.1
62.4
Multi-Sounce
88.6
62.4
67.7
27.4
44.9
23.9
62.2
Single-Souree
88.9
84.5
60.5
45.4
45.2
679
208
40.6
24.2
36.5
42.6
61.6
A.2 RELATIVE ACCURACY OF SINGLE-SOURCE DISTILLATION
Similarly to Table 3] we extend on our evaluation of how well the task similarity selects the best source models for single-source distillation. We report the ratio between the average test accuracy of the top-k target models ranked using the task similarity and the average test accuracy for the actual top-k target models found after the fact in Table Table 3 and Table 7 for k = 1, k 3, and 5, respectively.
We find that generally, using task similarity feature representations rather than the corresponding pseudo-labels yields better rankings, but also that PARC show very little difference between features and pseudo-labels for all considered & (1,3,5).
Relative accuracy over all k. The relative accuracy measure reported above is sensitive to k and the actual accuracy values of the models. T.e. if metric flips the order of the best and second best model when there is notable performance gap between the two models, the relative accuracy for k 1 will be low, and we might be mistaken believe the metric not working well. However, the metric might rank every model for k > 2 perfectly correct, and since we typically utilize the full set of source models, the initial mistake should not be detrimental the selection of task similarity metric. Thus, in Figure g we plot the relative accuracy for each task similarity metric and all k E {I...., S}. We find that while PARC on feature representations outperformed by both PARC and CKA pseudo-labels for k < 3. PARC feature representations outperforms all the other metrics for & 2 3. In particular, from Table g we have that on average over all k < S, PARC, performs the best.
A.3 ABLATION OF p FOR TASK SIMILARITY- WEIGHTED MULTI-SOURCH DISTILLATION
We report the values associated with Figure 7 for each target task and all considered choices of in Table 9
2 'The Structured tasks are mainly (ordinal) regression tasks transformed into classification tasks, and thus it seems reasonable to expect very general features (such as those from ImageNet pre-trained model) to generalize better such constructed tasks than specialized source models.
14

---

Under review as a conference paper at ICLR 2023
Table (Identical Table 3 Relative accu- racy of top-3 single-source distilled target mod- els selected by task similarity over the average of the 3 best models found in hindsight. We compute the average test aceuracy of the top- 3 highest ranked target models and divide this average with the average test accuracy of the best performing target models.
Table 5: Relative accuracy of top-1 single- source distilled target model selected by task similarity over the best model found in hind- sight. We compute the test accuracy of the high- est ranked target model (ranked by some task similarity) and divide this with the test accuracy of the performing target model.
Stanford Dogs
Scanford Dogs
Oxford Pets
Oxford Pets
CIFAR-IR
CUB200
ChexX
EunSAT
NABird
CIFAR-18 10
CUB200
ChestX
EuoSAT
NABird
ISIC
Msan
ISIC
Mean
Pseutoo
CKA
99.6
100.0
96.1
99.5 98.1 100.0
100.0 100.0
99.2
Pscudo
CKA
99.1
95.6
97.4
99.6
98.8
89.4
100.0 97.6
97.2
PARC
99.3
100.0
93.6
99.5
98.3
100.0
98.4
98.6
99.5
100.0
95.5
99.6
99.7
98.8
99.7
RSA
99.3
74.8
94.8
99.5
98.3
86.6
97.8
95.6
93.4
RSA
100.0
77.7
96.5
99.7
98.5
87.2
98.6
97.6
94.5
Feature
CKA
99.6
81.0
92.6
99.8
98.3
100.0
100.0
100.0
96.4
Festure
CKA
100.0
95.6 97.0
99.8
99.0
93.3
100.0
96.4
97.6
PARC
99.6
100.0
94.6
99.5
97.7
100.0
95.1
100.0
98.3
PARC
100.0
100.0
97.8
99.7
98.3
100.0
97.1
98.5
98.9
RSA
100.0
100.0
100.0
96.3
RSA
100.9 96.7 99.8
98.9
94,9
98.9
98.8 98..5
Table 7: Relative accuracy of top-5 single- source distilled target models selected by task similarity over the average of the 5 best mod- els found in hindsight. We compute the results analogously to Table with k = 5.
Table 8: The mean relative accuracy, across all k, for each metric in Figure g The average is bounded in (0, 1]. and corresponds to perfect ordering by task similarity. We find that using feature representations consistently outperforms pseudo-labed and that for both feature represen- tations and pseudo-labels PARC performs the best.
Shanford Dogs
Oxford Pets
CIFAR-10
CUB200
CheatX
EuroAAT
NABird
ISIC
Mean
CKA
PARC
Pseudoo
CKA
99.3
98.7
98.3
99.7
99.0
92.9
99.2
98.4
98.2
Pseudo
0.985
0.990
PARC
99.7
100.0
96.7
99.7
98.9
94.5
99.4
98.4
Feature
0.986
0.993
0.991
RSA
99.7
83.2
97.6
99.8
99.0
84.9
99.2
92.8
94.5
Fealure
CKA
99.7
97.4
97.7
99.8
98.9
96.5
99.2
97.8
98.4
PARC
99.7
100.0
97.9
99.8
99.1
99.7
97.5
99.7
99.2
RSA
99.7
99.7
97.9
99.8
99.2
97.9
98.9
99.7
99.1
A.4 MULTI--OURCE DISTILLATION WITH RESNET-50 AS TARGET ARCHITECTURE
In the main part of the article, we consider the computationally constrained setting, where some compute budget restricts the possible size of our target model. Thus, we use MobileNetV3 models as target models throughout the main paper. However, in Table TO remove the computational budget and allow the target model to be of any architecture, and particularly we use ResNet-50 as the target model.
We compare multi-source distillation (with p O and = 12) initialized with either ImageNet pre-trained weights or the weights of the highest ranked ResNet-50 source model to TRANSFER and FINE-TUNE SELECTED SOURCE. We find that multi-source distillation initialized from ImageNet outperform TRANSFER on average for both equal weighting and p 12, but underperform FINE- TUNE SELECTED SOURCE for both p. However, since FINE-TUNE SELECTED SOURCE is initial- ized from well-selected source model weights, the comparison is not entirely fair. Thus, we also consider the case where we initialize the target model for molti-suurce distillation with the weights of the highest ranked Re source model, and find that for p = 12 solti-source distillation performs par with FINE-TUNE SELECTED SOURCE.
15

---

Under review as a conference paper at ICLR 2023
1.00
Relative
0.98
CKA on Pseudo-label
0.96
PARC Pseudo-label
RSA on Pseudo-label
0.95
CKA on Feature
0.94
Feature
RSA on Feature
10
15
20
25
Top-k Source-Models
Figure 8: Relative accuracy of top-k single-source distilled target models selected by task similarity over the average of the top-k actual best target models found in hindsight. If the ordering by task similarity perfectly correct, the relative accuracy would be 1 for k. See Table for the average of each metric across all k.
Table 9: Test accuracy of multi-source distillation with various choices of p. compared to the base- line methods of IN+T TRANSFER and IN+FIXMATCH. We highlight the largest value for each target task in bold, and the results also visualised in Figure
Stanford Dogs
Oxford Pess
CIPAR-10
CLB200
ChastX
EamSAT
NADidd
ISIC
Mean
IN+FixMatch IN+Trassfe
41.9
38.5
98.1
42.8
Multi-Soucee (Liqua))
90.8
53.5
45.7
97.5
81.5
41.4
82.1
62.1
69.3
Multi-Source (p = 1)
91.1
55.6
46.5
97.9
81.5
42.5
83.3
64.4
70.3
Multi Source (p 3)
91.6
57.7
46.5
97.7
82.3
44.5
84.6
67.4
71.6
Multi-Source (p =
91.8
59.0
46.7
97.5
82.5
46.7
69.1
72.3
Multi-Source (p
59.6
97.6
47.6
Multi (p 12)
Multi-Source (p
81.7
72.6
Single-Source
92.0
59.6
46.8
97.4
81.0
47.4
81.9
71.3
72.2
A.5 NORMALIZATION TASK SIMILARITY FOR SOURCE MODEL WEIGHTING
We propose to choose the weights a = (a1, (ai,...,as) as. as
where
e = ( Cj.
for 1, S
and e, is the task similarity for source model MG, evaluated on the target task, normalized to sat- isfy 6, [0. 1] with min-max normalization over all es, Here, the hyperparameter, p can be used to increase/decrease the relative weight on the highest ranked source models, with the extremes p 0 and p corresponding equal weight and single-source distillation, respectively. An alternative way to obtain our normalization is to use the softmax function on the task similarities,
cxp (%)
05 > exp +
This does not require clipping the task similarity O. and with the temperature, T, adjust the relative weight on particular source models. Here, large T flattens the weights, and T + oo corresponds equal weighting of all source models, while small T increases the weight the highest ranked source models. Quantitatively, the two normalization methods can yield similar transformations with appropriate choices of p and T see Figure 9
16

---

Under review as a conference paper at ICLR 2023
Table 10: Multi-Source Distillation with ResNet-50 target model architecture. We compare fine- tuning of the highest ranked source model (Bolya et al. 2021) with multi- distillation to both ImageNet-initializee target models and target models initialized from the highest ranked ResNet-50 source model. For 12, multi-source distillation performs par with fine-tuning the selected source model. Largest value each target tasks bold.
Stafford Degs
Oxford Pess
CIFAR-10
CUB200
EuroSAT
NABird
Target Model
BSIC
Mean
Initialization
Transfer
ImageNet
92.9
42,0
43.4
96.8
79.9
39.9
83,3
65.9
68.0
Fine- tune Selected Source
Source Model
93.9
70.8
43.9
97.2
81.3
47.4
84.8
79.3
74.7
Multi-Source (Equal)
ImageNet
87.8
57.3
46.1
97.0
78.9
42.4
84.1
64.5
69.8
Multi-Son Source (p 12)
ImageNet
91.5
64.5
45,4
97.0
78.9
49.8
87.1
74.2
73.6
Multi-Source (Equal)
Source Model
68.8
97.4
81.2
43.2
81.9
65.1
Multi-Soure (p 12)
Model
97.0
Metric Power
Softmax
Weight
T-
p=1
0.25
T- 0.125
p-12
T- 0.075
4/8
p=15
2/5
1/5
0/5
Normalized Metrie
Normalized Metric
Figure 9: Transformation of weights for various choices of power (left) or softmax temperature (right). Here S is the number of source models, and we consider equidistantly distributed normalized metrics.
Multi-Soure Distillation variation of SDMS with only 5% labeled samples per task. Table 11:
Again, we compare to the bascline of IN+TRANSFER. Largest value for each target tasks bold.
Stanford Dogs
CIFAR-10
CUB2200
Chesx
FumSAT
NABrr
ISIC
Mean
IN+Transfer
88.0
16.8
43.5
94.8
73,9
14.4
55.0
38.9
53.2
Multi (p
Multi-Source (p
32.3
42.6
95.9
76.7
24.8
60.0
Ingle-Source
87.2
31.4
39.7
95.1
75.4
24.0
58.9
49.7
57.7
A.6 SMALLER AMOUNT OF LABELED DATA
We now repeat the experiment of the main paper across the 8 target datasets with reduced amount of labeled samples. Here, we reduce the number of labeled samples to 5% (rather than 20%) of the training set and report the accuracy in Table I1 We find a similar pattern observed in the main experiment, where MULTI-SOURGE distillation on average outperforms TRANSFER irrespective of the choice of p. For p 9 MULTI-SOURCE outperforms TRANSFER by 6.8%-point average and in particular 15.5%-points on CUB200, whereas the only loss in performance on ChestX with drop of 0.9%-point.
17

---

Under review as a conference paper at ICLR 2023
Pearson correlation between test ac- curacy after all possible single-source distilla- tions and task similarity associated with the source models. Similar to Table
Table 12:
Table 13: Kendall Tau correlation between test accuracy after all possible single-source distil- lations and task similarity associated with the source models. Similar to Table 2
Stanford Dogs
Stanford Dogs
Oxford Pets
CIFAR-10
CUB200
ChatX
EuroAA
NABird
CIFAR-10
CUB200
Chaax
EmroSAT
NABird
Oxford
ISIC
Mean
1SIC
Mean
Pseudo
CKA
0.62
0.85
0.07
0.30
-0.06
0.33
0.67
0:21
0.37
Psenda
CKA
0.51
0.46
0.16
0.28
-0.05
0.24
0.49
0.07
0.27
PARC
0.75
0.74
40.03
0.27
-0.00
0.36
0.63
0.51
0.40
PARC
0.61
0.64
0.01
0.12
0.02
0.36
0.54
0.39
0.34
RSA
0.75
0.13
-0.07
0.38
0.04
-0.09
0.66
0.40
0.27
RSA
0.62
0.17
-0.07
0.22
0.08
-0.01
0.29
0.22
Feature
0.84
0.60
0.39
0.29
0.00
0.71
0.46
Fealure
CKA
0.67
0.34
0.14
-0.05
0.40
0.50
0.38
PARC
0.86
0.73
0.17
0.46
-0.06
0.58
0.77
0.78
0.54
PARC
0.69
0.67
0.14
0.31
-0.10
0.65
0.62
0.67
0.46
RSA
0.90
0.85
0.07
0.45
0.04
0.27
0.87
0.83
0.54
RSA
0.72
0.65
0.02
0.28
0.02
0.19
0.72
0.67
A.7 DIFFERENT MEASURES OF CORRELATION
In order to evaluate the quality of task similarity metric estimate the performance of target model after distillation, we consider the correlation between the computed metric and the actual ob- served performance after distillation. However, since we have no reason to believe that the relation- ship is linear, we consider the Spearman correlation in the main paper. However, for completeness of exposition we report Pearson correlation and Kendall's s Tau in Table|IZand Table respectively. For both these correlation measures, the overall conclusions the same: Using feature represen- tations is preferable to pseudo-labels, and PARC generally outperforms both CKA and RSA, albeit not by much over CKA.
A.8 CHOICE TASK SIMILARITY METRICS
Recently, multiple measures intended to estimate the transferability of a source model have been proposed. However, despite the very recently published Multi-Source Leep (MS-LEEP) and En- semble Leep (F-Leep) no task similarity metric consider the estimation over multiple models at once (Agostinelli et al. 2022). Thus, we consider each source model separately and compute the metrics independent of other source models. This has the added benefit of reducing the number of metric computations required as we do not need to compute the task similarity for all possible combinations of n models from S possible (i.e. (")), which grows fast with S.
Assume X Nxxx and Y R.xxyy and that Kij A(x;, x3) for and Lij = 1(y;, where k. and / are two (similarity) kernels as well as x,, X2 and y Y, are rows of X and Y. respectively. Then we have that CKA defined as
PCKA (X, Y) del
HSIC(K, L) HSTC(K, K K)HSIC(L,L)
where K, L E NNN and HSIC is the Hilbert-Schmidt Independence Criterion,
HSIC(K, L) def 1
112 (KHyLHy), with HN def IN 11T.
In particular, if both & and are linear kernels, then we have that
Y 13
PCKA (X, Y)
where || is the Frobenius norm. We use the linear kernel throughout this paper and refer to Cortes al. 2012) for additional details on CKA.
For RSA, we consider the dissimilarity matrices given by
K def
corrcoef(x,xx) and Liy del
18

---

Under review as a conference paper at ICLR 2023
where X and Y are assumed normalized to have mean 0 and variance 1. We then compute RSA as the Spearman correlation between the lower triangles of K and L,
PRSA (X,
Y) def spearman ([Ki) i < j). [La 5)))
For additional details on RSA, we refer the reader Dwivedi & Roig (011). While Bolya et al. (2021) introduces PARC alongside a heuristic and feature reduction, the PARC metric is almost identical to RSA. However, RSA was introduced compute similarities between two sets of rep- resentations, and PARC was aimed at computing similarities between set of representations and a set of labels associated with the dataset. Thus, our use of PARC, merely differs from RSA in the lack of normalization of Y, which is assumed to be one-hot encoded vectors of class labels from the probe dataset.
B EXPERIMENTAL DETAILS
In the following, we provide some experimental details.
B.1 MAIN EXPERIMENTS
Unless otherwise mentioned, batch size of 128, and loss weighting of = 0.8. We initialize our target models with the ImageNet pre-trained weights available in torchvision stable/mode s), and consider the 31 fine-tuned models from Bolya et al. (021)) publicly avail- our set of source models. The source models consist of
SGI) with learning rate of 0.01, weight decay of 0.0001,
org/
able at gigthu... com/ aoooly/paazee each of the architectures (AlexNet, GoogLeNet, ResNet-18, 18, and ResNet-50) trained on CIFAR-10, Caltech101, CUB200, NABird, Oxford Pets, Stanford Dogs, VOC2007, and ImageNet Note, we always exclude any source model trained on the particular target task, thus effectively reducing the number of source models for some target tasks. For FixMatch we use a batch size of 128 (with a 1:1 ratio of labeled to unlabeled samples for each batch) and fix the confidence threshold 0.95 and the temperature at 1, We keep the loss weighting between the supervised loss and the unlabeled FixMatch loss at À = 0.8.
B.2 VTAB EXPERIMENTS
For each VTAB experiment we consider the full training set (as introduced in Zhai et al. (2019)) as the unlabeled set, D" and the VTAB-IK subset as the labeled set, D We use the Pytorch implementation from Jia et al. (022) available at at|github. com/ KMnP KMnP/vpt
We use SGD with learning rate of 0.005, weight decay of 0.0001, batch size of 128 equally split in 64 labeled and unlabeled samples, and loss weighting of À = 0.9. We train our models for 100 epochs, where we define one epoch as the number of steps required to traverse the set of unlabeled target data, D when using semi-supervised methods, or merely as the number of steps to traverse the labeled set, D', for supervised transfer methods. We initialize our target models with the BiT- M ResNet-50xl model fine-tuned ILSVRC-2012 from BiT (Kolesnikov et al. 2020 publicly available at gigggt.t.bb.... cwm/ wewpgleeereeerrc/bbigggge transffr
We consider the 19 BiT-M ResNet-50x1 models fine-tuned the VTAB-1K target tasks from Kolesnikov et al. (2020) as the set of source models. We always exclude the source model asso- ciated with the target task from the set of source models, and thus effectively have 18 source models available for each target task in VTAB. We use the PARC metric on the source model features to compute the source weighting, but also only use the top-5 highest ranked source models reduce the computational costs of training. Furthermore, we use p = 9 for multi-source distillation.
Note, there is no ResNet-50 trained on ImageNet in the set of source models, thus the total number of models 31 and not 32 as otherwise expected.
19


Reviews:
[{"review_id": "m5LsJ5UACuV", "reviewer": "Reviewer_QfTL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a method to train efficient models with limited labeled data. By using task similarity metrics, they conduct weighted knowledge distillation with pretrained models from different sources. It achieves better results than transferring from ImageNet pretrained model or FixMatch.", "summary_of_the_review": "In conclusion, this work shows task similarity is useful metrics when it comes to distillation with teachers from multiple sources. I think more evidences are needed to prove multi-source distillation is indeed better than single-source distillation.", "strength_and_weaknesses": "Strength:\n- Learning with limited labeled data is meaningful to be solved.\n- The experiments and ablation studies are well-designed.\n\nWeakness:\n- In the main table it's good to show how multi-source distillation works for all architectures.\n- The empirical results are not convincing enough comparing to the single-source distillation with task similarity. Seems it only provides marginal improvement with multi-source but adds way more computation overhead.\n- Extra hyperparameters are required to be tuned (especially p).\n- I don't quite get why this method especially targets efficient models and doesn't work well for large models like resnet50. More analyzes will be very helpful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jGk7eAf_F2P", "reviewer": "Reviewer_Tp8o", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper address the challenge of knowledge distillation for image recognition tasks with few training samples. This paper proved that simply distilling knowledge from a single task will not get good performance. The authors present a weighted multi-source distillation method to distill multiple source models trained on different domains by weighting these models with a task relevance score. Experiments demonstrate that more datasets will bring performance improvement.", "summary_of_the_review": "I think the method and experiments are not strong enough to support the motivation of this paper. Importance issues are forgotten to be discussed. ", "strength_and_weaknesses": "*Strength\n1. The idea of distiallting knowledge from multiple datasets with weights is interesting and reasonable. The authors conduct a lot of experiments to validate the corresponding claims.\n2. The paper is overall clear and well-organized in illustrating the methods, motivation, and experimental results.\n\n*Weakness\n1. The task relevance metric is very important for knowledge distillation, but the authors don't discuss the influence of the different choices of the task relevance metric.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AGhglDIu_zR", "reviewer": "Reviewer_fFzJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper assumes limited label availability and propose a weighted multi-source distillation method, in practice that means distill multiple (diverse) source models trained on different domains, weighing them by their relevance for the target task, assuming such target task is available", "summary_of_the_review": "The paper is interesting overall but there are better backbones available, including transformed-based ones. Also, single source is quite similar to the proposed method. ", "strength_and_weaknesses": "Strengths:\n- The paper shows there are sources that may represent better some given target tasks than others, including the baseline of ImageNet pretrained networks\n- The methods show superior results in particular for fine-grained data such as CUB200\n\nWeaknesses:\n- FixMatch is a semi-supervised technique where the unlabeled data is assumed to be from the same domain\n- The methods assumes the target task is available during distillation time\n- A stronger baseline would be ViT/DINO\n- It seems that, overall, it would be better to select a few tasks (more relevant) than to weigh them all. Maybe a preselection would reach similar results without computational burden (single is almost as good as task similarity)\n- It is interesting that MobileNet performs better than ResNet (among others), which contradicts previous work, and no justification is given.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject