Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance
Authors: Shibal Ibrahim
Natalia Ponomareva
Rahul Mazumder
Venue: ICLR 2022 Submitted (2022)
Keywords: transferability metrics
fine-tuning
transfer learning
discrepancy measures
domain adaptation

Abstract: Fine-tuning of large pre-trained image and language models on small customized datasets has become increasingly popular for improved prediction and efficient use of limited resources. Fine-tuning requires identification of best models to transfer-learn from and quantifying transferability prevents expensive re-training on all of the candidate models/tasks pairs. In this paper, we show that the statistical problems with covariance estimation drive the poor performance of H-score (Bao et al., 2019) — a common baseline for newer metrics — and propose shrinkage-based estimator. This results in up to 80% absolute gain in H-score correlation performance, making it competitive with the state-of-the-art LogME measure by You et al. (2021). Our shrinkage-based H-score is 3-55 times faster to compute compared to LogME. Additionally, we look into a less common setting of target (as opposed to source) task selection. We demonstrate previously overlooked problems in such settings with different number of labels, class-imbalance ratios etc. for some recent metrics e.g., NCE (Tran et al., 2019), LEEP (Nguyen et al., 2020) that resulted in them being misrepresented as leading measures. We propose a correction and recommend measuring correlation performance against relative accuracy in such settings. We also outline the difficulties of comparing feature-dependent metrics, both supervised (e.g. H-score) and unsupervised measures (e.g., Maximum Mean (Long et al., 2015) and Central Moment Discrepancy (Zellinger et al., 2019)), across source models/layers with widely varying feature embedding dimension. We show that dimensionality reduction methods allow for meaningful comparison across models, cheaper computation (6x) and improved correlation performance of some of these measures. We investigate performance of 14 different supervised and unsupervised metrics and demonstrate that even unsupervised metrics can identify the leading models for domain adaptation. We support our findings with ~65,000 (fine-tuning trials) experiments.

Under review as a conference paper at ICLR 2022
NEWER IS NOT ALWAYS BETTER: RETHINKING TRANSFERABILITY METRICS, THEIR PE- CULIARITIES, STABILITY AND PERFORMANCE
Anonymous authors Paper under double-blind review
ABSTRACT
Fine-tuning of large pre-trained image and language models small customized datasets has become increasingly popular for improved prediction and efficient use of limited resources. Fine-tuning requires identification of best models transfer-learn from and quantifying transferability prevents expensive re-training all of the candidate models/tasks pairs. In this paper, we show that the statistical problems with covariance estimation drive the poor performance of H-score (Bao) common baseline for newer metries and propose shrinkage- based estimator. This results in up to 80% absolute gain in H-score correlation performance, making competitive with the ofate-of-the-art LogME measure by You al. (202I). Our shrinkage-based H-score is 3 55 times faster to com- pute compared to LogME. Additionally, we look into less common setting target (as opposed to source) task selection. We demonstrate previously over- looked problems such settings with different number of labels, ratios etc. for some recent metrics e.g., NCE (Tran et al. 2019), LEEP (Nguyen et al. 2020) that resulted in them being misrepresented as leading measures. We propose correction and recommend measuring correlation performance against relative accuracy in such settings. We also outline the difficulties of comparing feature-dependent metries, both supervised (e.g. H-score) and unsupervised mea- sures (e.g., Maximum Mean (Long et al. 2013) and Central Moment Discrepancy (Zellinger et al., 2019)), ) across source models/layers with widely varying feature embedding dimension. We show that dimensionality reduction methods allow for meaningful comparison across models, cheaper computation (6x) and improved correlation performance of some of these measures. We investigate performance 14 different supervised and unsupervised metrics and demonstrate that even unsupervised metrics identify the leading models for domain adaptation. We support our findings with ~ 65,000 (fine-tuning trials) experiments.
et al. 2019)
INTRODUCTION
Transfer learning (TL) is set of techniques of using abundant somewhat related source data (s)(s), (()) to ensure that model can generalize well to the target domain, defined as either little amount of labelled data p(x(1), (()) (supervised), and/or a lot of unlabelled data p( (0) (unsuper- vised TL). TL most commonly achieved either via fine-tuning or co-training. Fine-tuning (FT) is process of adapting model trained on source data by using target data to do several optimization steps (for example, SGD) that update the model parameters. Co-training on source and target data usually involves reweighting the instances in some way enforcing domain irrelevance on feature representation layer, such that the model trained on such combined data works well on target data. Fine-tuning becoming increasing popular because large models like ImageNet (Krizhevsky al.) 2012), Bert (Devlin et al. 2018) etc. are released by companies and are casily modifiable. Training such large models from seratch often prohibitively expensive for the end user.
In this paper, we are primarily interested in effectively measuring transferability before training of the final model begins. Given a souree data/model, transferability measure (TM) quantifies how much knowledge of source domain/model is transferable to the target model. Transferability mea- sures (TMs) are important for various reasons: they allow understanding of relationships between

---

Under review as a conference paper at ICLR 2022
different learning tasks, selection of highly transferable tasks for joint training on source and target domains, selection of optimal pre-trained source models for the relevant target task, prevention of trial procedures attempting to transfer from each source domain and optimal policy learning in rein- forcement learning scenarios (e.g. optimal selection of next task to learn by a robot). a measure is capable of efficiently and accurately measuring transferability across arbitrary tasks, the problem of task transfer learning is greatly simplified by using the measure to search over candidate transfer sources and targets.
Contributions We study both supervised and unsupervised TMs in the context of FT. For supervised TMs, our contributions are three-fold:
1. We show that H-score, commonly used as a baseline for newer supervised TMs, suffers from instability due poor estimation covariance matrices. We propose shrinkage-based estimation of H-score with regularized covariance estimation techniques from statistical literature. We show 80% absolute increase over the original H-score and show superior performance 9/15 cases against all newer TMs across various FT scenarios.
We present a fast implementation our estimator that is 3 - 55 times faster than state-of-the-art LogME measure. Unlike LogME, our optimized implementation for our estimator is tractable even for really high- sigh-dimensional feature embeddings 10
3. We identify problems with 3 other supervised TMs (NCE, LEEP and MLLEP)) in target task selection when either the number of target classes or the class imbalance varies across candidate target tasks. We propose measuring correlation against relative target accuracy (instead of vanilla accuracy) in such scenarios.
For unsupervised TMs, outline computational challenges and propose dimensionality reduction methods for better estimation and effective comparison of such measures when the feature dimen- sions are large and/or different across various source models. We show that with our proposed mod- ifications, even unsupervised TMs can be effective in identifying the best source ImageNet model. Our large set of 65,000 FT experiments with multiple ImageNet models and different regimes gen- erated from CIFAR-100 100 and CIFAR-10 image datasets shows usefulness of our proposals.
This paper organized as follows. Section describes general FT regimes and transfer learning tasks. Section 3 discusses supervised TM and addresses shortcomings of the pioneer TM (H-Score) that arise due to limited target data (subsection3.1) (.1].. subsection 3.2 we demonstrate problems with recent NCE, LEEP and ALEEP metrics and propose way to address them. Section 4 highlights shortcomings of different commonly used unsupervised measures for source selection and proposes alternatives that offer improvements. Finally, Section 5 presents meta study of all metries.
2 TRANSFERABILITY SETUP
We consider the following FT scenarios based on existing literature.
(i) Source Model Selection (SMS): For a particular target data/task, this regime aims to select the "optimal" source model (or data) to transfer-learn from, from collection of candidate models/data. (ii) Target Task Selection (TTS); For a particular (source) model, this regime aims to find the most related target data/task.
In addition, we primarily consider two different FT strategies:
(i Linear FT/head only FT: All layers except for the penultimate layer are frozen. Only the weights of the head classifier are re-trained while fine-tuning.
(ii) Nonlinear FT:: Any arbitrary layer can be designated feature extractor, up which all the layers are frozen; the subsequent layers include nonlinear transformations and are re-trained along with the head on target data.
SUPERVISED TRANSFERABILITY MEASURES
Related Work Recent literature in transfer learning has proposed computationally efficient TMs. We categorize measures that require target labels as supervised TMs. Inspired by principles in
information theory, Negative Conditional Entropy (NCE) Tran et al. (019 uses pre-trained source model and evaluates conditional entropy between target pseudo labels (source models' assigned labels) and real target labels. Log Expected Empirical Predictor (LEEPP (Nguyen et al. 2020
2

---

Under review as a conference paper at ICLR 2022
modifies NCE by using soft predictions from the source model. Both NCE and LEEP do not directly use feature information, hence they are not applicable for layer selection.
Cui et al. (2018) propose representing each output class by the mean of all images from that class
and computing Earth Mover's ' distance between the centroids of the source classes and target classes.
Bao et al. 2 Li al. (2021): y: Huang et al. (202T); You al. (202T): Deshpande et al. (2021)
proposed metrics that capture information from both the (learnt) feature representations and the real target labels. These metries are more appealing as these can be broadly applicable for models that are pre-trained in either supervised or unsupervised fashion. Li et al. (2021) proposed A ALEEP that fits a Gaussian mixture model on the target feature embeddings and computes the LEEP score between the probabalistic assignment of target features to different clusters and the target labels. |Huang jet al. (202T) proposed TransRate - a computationally-frienae
surrogate of mutual information
(using coding rate) between the target feature emheddings and the target labels. |Bao et al. (2019) introduced H-score that takes into account inter-class feature variance and feature redundancy. You et al. 2021 proposed LogME that considers an optimization problem rooted in Bayesian statistics to maximize the marginal likelihood under linear classifier head. Deshpande et al. (202T) introduced LFC to measure in in-class similarity of target feature embeddings across samples.
Finally, Tan et al. (202I) used Optimal Transport to evaluate domain distance, and combined it, via a linear combination, with NCE. To learn such measure, portion of target tasks were set aside, the models were transferred onto these tasks and the results were used to learn the coefficients for the combined Optimal Transport based Conditional Entropy (OTCE) metric. While the resulting metric appears to be superior over other non-composite metrics like H-score and LEEP, it is expensive to compute since it requires finding the appropriate coefficients for the combination. Additionally, our results indicate that both components of the measure seem to be individually sub-optimal in measuring transferability against corresponding supervised and unsupervised measures.
3.1 IMPROVED ESTIMATION OF H-SCORE FOR LIMITED TARGET DATA
H-score (Bao et al. 2019) is one of the pioneer measures that is often used as a baseline for newer su- pervised TMs, which often demonstrate the improved performance. It characterizes discriminatory strength of feature embedding for classification:
H(f) = ir(x()))) ()()))
(I)
where, d is the embedding dimension, f: - h((x!")) E Rd is the target feature embeddings when the feature extractor (h R - Rd)) from the source model is applied to the target sample x!!" RR,
F Ruoxdd denotes the corresponding target feature matrix, Y (()) E ) {1, (1...... ,C} are the target data labels, Σ() Raddd denotes the sample feature covariance matrix of f. z E [flY] Rd and Z R denotes the corresponding target conditioned feature matrix, {*) Rdxd denotes the sample covariance matrix of 2. Intuitively, H(f) captures the notion that higher inter- class variance and small feature redundancy results in better transferability.
We hypothesize that the sub-optimal performance of H-Score (compared to that of more recent metrics) for measuring transferability in many of the evaluation cases, e.g., in (Nguyen al. (220). is due to lack of robust estimation of H-Score - see Fig. for a synthetic example showing the non-reliability of empirical H-score over various sample sizes when compared with its population version. Given that many of the deep learning models in the context of TL have high-dimensional feature embedding space - typically larger than the number of target samples- - the estimation of the two covariance matrices in H-score becomes challenging: the sample covariance matrix of the feature embedding has a large condition number in small data regimes. In many cases, it cannot even be inverted. |ao et al. (2019) used a pseuud-iiverss of the covariance matrix Σ(), However, this method of estimating precision matrix can be sub-optimal as inversion can amplify estimation error (Ledoit Wolf (004). We propose to use well- rell-conditioned shrinkage estimators motivated by the rich literature in statistics on the estimation of high- -ighesninnio covariance (and precision) matrices (Pourahmadi] 2013). We show that the use of such shrinkage estimators can offer significant experiments
gain in the performance of H-score in predicting transferability. In many cases, show, the gain is so significant that H-score becomes a leading TM, surpassing the performance of ofte--oftthe-art measures.
Condition number of positive semidefinite matrix A, is the ratio of its largest and smallest eigenvalues.
3

---

Under review as a conference paper at ICLR 2022
Proposed Transferability Measure We propose the fol- lowing shrinkage based H-score:
80
Htf--oopllttion
hiff-mmpirical
60
(a) - tr DD -1 (11-))
(2)
Scores
Hu(f)
40
Estimating Ex While there several possibilities to obtain a regularized covariance matrix (Pourahmadi 2013), we present an approach that considers a linear op- eration on the eigenvalues of the sample version of the feature embedding covariance matrix. Similar ideas of using well-conditioned plug-in covariance matrices are used in the context of discriminant analysis (Hastie et al. 2001}. In particular, we improve the conditioning of the covariance matrix by considering its weighted convex combination with scalar multiple of the identity matrix:
20-
0
10a
10a
10
Number of samples
Figure 1: Stability of original H(f) and the shrinkage-bar baaaddeeebaedddeeee H, (f) with respect to number of samples. The original H- Score is ~ 75 times larger than the population version of the H-Score (es- timated with sample size of 10%). In contrast, the shrinkage-basee based H-Score is significantly more reliable.
ΣΦ = (1-a)) a) + aaId
(3)
where a E [] is the shrinkage parameter and a is the average variance computed as tr( ((())// The linear op- eration the eigenvalues ensures the covariance estimator is positive definite. Note that the inverse
of ΣX can be computed for every a, by using the degempeccopooition of Σ(O. The shrink- age parameter controls the bias and variance trade-off; the optimal needs to be selected. This distribution-free estimator is well-suited for our application as the explicit convex linear combina- tion is easy to compute and makes the covariance estimates well-conditioned and more accurate
Ledoit & Wolf 2004 Chen et al. (2100; Schäfer & Strimmer 2005).
Understanding (1 (2)) The scaling factor (1 aa) can be understood in terms of regularized covariance matrix estimation under ridge penalty:
1/(1 + A) (x) argmin || > + >
where > 2 0 is the ridge penalty. Choosing À a/ a),, it becomes clear that (1 0z) (+) is the regularized covariance matrix.
Choice of a Ledoit & Wolf (2004) proposed a covariance matrix estimator that minimizes mean squared error loss between the shrinkage based covariance estimator and the true covariance matrix. The optimization with respect to a considers the following objective:
min E E|
s.t. Σ = (1 a)Σ() + ovt, -
(5)
where || = tr //d. This optimization problem permits a closed-form solution for the optimal shrinkage parameter, which given by:
a² E1| 2 / (trEE)/d) Lall]]
(6)
~ min((1/n()) ////nn//e..
(E( " |SST - E - (I(E))) 12ll.113
(7)
where 77) defines valid estimator (not dependent on true covariance matrix) for practical use. For proof, we refer the readers to Section 2.1 and 3.3 in Ledoit & Wolf (2004). We provide some addi- tional discussion on why same a used for the two regularized covariance matrices shrinkage- based H-Score in Supplement Section S
We provide validation of shrinkage-based estimation of H-Score on synthetic classification data. We generated million 1000-dimensional features with 10 classes using Sklearn multi-class dataset generation function (Pedregosa et al. (01IT. Number of informative features is set to 500 with rest filled with random noise. We visualize the original and the population version of the H-score and the shrinkage-basee based H-Score for different sample sizes in Fig. T We observe that the original H- H-Score becomes highly unreliable the number of samples decreases. In contrast, the shrunken estimation of H-Score is highly stable and has a small error when compared with the population H-Score.
Efficient Computation for small target data For small target data (C < n4 < d), the naive imple- mentation Ha( can be very slow. We propose an optimized implementation for our shrinkage- based H-Score that exploits diagonal plus low-rank structure of x(/) for efficient matrix inversion
4

---

Under review as a conference paper at ICLR 2022
and the low-rank structure of [(e) for faster matrix- matrix multiplications. We assume F (and correspondingly Z) are centered. The optimized computation of Ha(f) (f is given by:
Ha(f) = (11))=(1-0)(mmoo))) (IRI - (1-a) a) vee (G)² vee (W (W- (G))
(8)
where R = [V . E RdxC G FR E R W =n +FF
X The derivation is provided in the Supplement Section S2.3 We make a timing comparison
of our optimized implementation of (f) against the computational times of the sfate-of-the-an
MogME measure and demonstrate 3 55 times faster computation (see Table 6 Section 5.3
3.2 A CLOSER LOOK AT NCE, LEEP AND ALEEP MEASURES
Next, we pursue a deeper investigation of some of the newer metrics that reported to be superior to H-Score and bring to light what appears be some overlooked issues with these metrics in Target Task Selection (TTS) scenario. TTS has received less attention than Source Model Selection (SMS). To our knowledge, we are the first bring light some problematic aspects with NCE, LEEP and NLEEP, which can potentially lead to the mis- use of these metrics in measuring transferabil- ity. These measures are sensitive to the num- ber of target classes (C) and tend to be smaller when is larger (see Fig. Left)). Therefore, of these measures for target tasks with dif- ferens C' will most likely result in selecting the task with smaller C. However, in practice, it is not always the case that transferring class classification into a set of binary tasks can create more difficult to learn boundaries (Friedman) al. (000). Furthermore, the measures are also problematic if two candidate target tasks have dif- ferent degree of imbalance in their classes even if C' is the same. The measures would predict higher transferability for imbalanced data regimes over balanced settings (see Fig. Right)). However, imbalanced datasets are typically harder to learn. If these measures are correlated against vanilla accuracy, which tends to be higher as the imbalance increases e.g. for binary classification, the mea- sures would falsely suggest they are good indicators of performance. Earlier work did not consider both these aspects and erroncously showed good correlation of these metrics against vanilla accu- racy to show dominance of these metries in TTS with different C (Nguyen et al. 2020: Tan al. 2021 and imbalance (Tan et al. 2021).
-0.40
-0.45
LEE
ALEE
Number Casses
Class Intalance
Relation of NCE, LEEP & WLLEP to [Left] number of classes (log-scale) and [Right] class imbalance, mar(n1, n2) /minnminnnnn.. n2). for VGG19 on CIFARIO0. For [Left], we randomly select 2-100 classes. For [Right], we randomly select 2 classes and vary the class imbalances.
Figure 2:
task with a smaller C is easier; for example, reframing a multi-
Here, we propose method to ameliorate the shortcomings of to prevent misuse of these measures, so that they lead to more reliable conclusions. We propose to standardize the metrics by the entropy of the target label priors, leading to the definitions in This standardization considers both the class imbalance as well as number of classes through the entropy of the target label priors.
D-NCE daf NCE/H(YY, n-LEEP def
LEEP (I((), n-WLEEP def
(IL)) (9)
Our proposed normalizations in ensures the normalized NCE is bounded between [o. 1]. For
proof, see Supplement Section S2.4 n-NCE in fact equivalent to normalized mutual information and has been extensively used to measure correlation between two different labelings/clustering of samples (Vinh et al.) 2010). Given the similar behavior of LEEP and NCE to different and class imbalance as shown in Fig. 2 suggest the same normalization as given in (9) However, this normalization does not ensure boundedness of n-LEEP score (and by extension n-WLEEP) in the range [0, 1] as in the case of n-NCE.
For scenarios where candidate target tasks have different C, we propose an alternative evaluation criteria (relative accuracy) instead of vanilla accuracy see Section[]oon more details. We provide empirical validation of the proposed normalization to these measures in Table 2 Section 5.1 We also show that our proposed shrinkage-based H-Score is the leading metric even in these scenarios.
5

---

Under review as a conference paper at ICLR 2022
4 UNSUPERVISED TRANSFERABILITY MEASURES
Since most of the existing TMs essentially estimate the difference between source and target distri- butions (be it just the labels as in LEEP and NCE or between distributions of feature embeddings LFCC,, one consider unsupervised discrepancy measures that don't 't
and labels as in H-score use target labels, but rely only on source and target data feature embeddingg. They have been used for regularization in the context of domain adaptation (Li et al. 2020; Zellinger et al. 2019), how- ever they have not been studied for characterizing transferability in the context of FT. It remains to be seen how informative these metrics are as TMs for FT. These metrics are important because they can be used with unlabeled target data when supervised metrics cannot be used.
Related Work early work, [Ben-David et al. (2006) proposed A-distance to estimate domain divergence from unlabelled source and target data. This metric can be empirically estimated with an additional model trained to predict the domain of the data (source or target) and uses the do- main classification accuracy proxy. Training such a model is expensive, requires choosing an architecture and/or tuning hyperparameters and may end up being more expensive than actual TL.
We consider more standard discrepancy measures unsupervised TMs. Maximum mean discrep- ancy (MMD) (Long et al., 2015) measures mean differences between distributions in some rich kernel space and has been previously used to detect covariate shift between input distributions (Ra- banser et al. 2019). Central moment discrepancy (CMD) (Zellinger et al., 2019) measures dif- ferences in mean and higher order moments. Similarly, correlation alignment (CORAL) (Sun & Sacnko 2016) measures the difference in covariance matrices of source and target feature distri- butions. Kullback-Leibler Divergence (KLD) (Kullback & Leibler 1951) is a standard divergence metric for two distributions, which can be calculated either by making an assumption on the distri- butions or using non-pan ooraammmttticc estimators. Optimal transport (OT) (Bonneel al., 2011 1 considers the optimal energy required to shift distributions from source features to target features. We also consider Wasserstein distance (WD) (Kantorovich, 1939), under normality assumptions for source and target features. Supplement Section S3.2 contains formulas for all the unsupervised TMs.
Challenges of comparing feature distribution discrepancy across task pairs Next, we discuss challenges in using unsupervised discrepancy metrics on the feature embeddings of source and target data across model/task pairs in both SMS and TTS scenarios. Given that none of the discrepancy measures (mentioned above) are normalized, direct comparison of these measures across model/task pairs leads to the following challenges:
1. Scale of the features across different source models can be arbitrary, for example ImageNet mod- els with or without Batch Normalization layers.
2. The feature dimension (d) across different source models even for the penultimate layer can vary significantly e.g. from 1024 in MobileNet to 4096 in VGG19. Normalization is not straight- forward for many metries such as MMD, KLD etc. Such differences makes source model/layer selection for FT highly problematic.
may be huge. Measuring discrepancy in high-dimensionas spaces is both challenging and not well-established due to curse of dimensionality (Rabanser et 2019).
Our proposals to address challenges: We address the first challenge (outlined above) by stan- dardizing both source and target feature embeddings via feature-wise standardization using source features' first and second order moments (z-score). See Supplement Section [S.1 for more details. This standardization is in contrast to independent z-score normalization of source and target em- beddings, ensuring that measures that consider differences in moments e.g., CMD can effectively capture such information even after standardization.
To address the second and third challenge (outlined above), we propose dimensionality reduction of feature embeddings before computing unsupervised discrepancy measures. We project feature embeddings to a lower qdimeensional space, where q is taken to be the same across the competing K models/layers and satisfies: g < min min) ft.)....... ((k) [ where |.] operator denotes the cardinality of the feature spaces. The dimensionality reduction allows for more meaningful comparison of
measures across source/target pairs; this is relevant for source/layer selection. More generally, also allows for faster and more robust estimate for limited target samples case (n, < d) for linear and nonlinear FT. In the case of nonlinear FT, the intermediate layers of visual and language models have really large d ~ 103, see Table S2 in Supplement Section S5 for examples.

---

Under review as a conference paper at ICLR 2022
We consider Principal Component Analysis (PCA) and Gaussian Random Projection (RP). Both use linear transformation matrix V to derive the transformed features F FV; the former derives an optimal orthogonal transformation to capture as much of the variance in the data, while the latter samples components from (0, ;) to preserve pairwise distances between any two samples of the dataset. Untrained auto-encoders (AE) are other alternatives that have been used to detect covariate shifts in input distributions by Rabanser et al. (2019). It is not known how sensitive these untrained AE are to the underlying Pechitectur-ustine ity measurement for FT as those maybe more time-consuming than the actual FT. We demonstrate improved correlation performance of unsupervised discrepancy measures with dimensionality re- duction in Table in Section 5. for TTS and Table in Section 5,2 for SMS.
trained AE is less appropriate for use in transferabil-
5 EXPERIMENTS
We evaluate existing TMs and our proposed modifications in various FT regimes and data settings. We draw inspiration from Nguyen et al. (2020) who consider TTS and SMS. The experimental setup highlights important aspects of TMs, e.g., dataset size for computing empirical distributions, covariance matrices and discrepancy measures, number of target classes, and feature dimension etc. Some of these aspects have been overlooked when evaluating TMs, leading to improper conclusions.
Recent work usually considers either supervised or unsupervised domain adaptation. The proposed measures to quantify transferability have mostly relied on the availability of target labels, hence, they have been evaluated in supervised domain adaptation with FT. It has not been previously shown how these measures compare against metrics that only rely on feature distributions e.g. MMD, CMD etc. Therefore, we also provide an empirical evaluation of these measures and demonstrate that even these measures can be effective at identifying the best model to transfer from-this allows for much broader applicability of the use of these discrepancy measures.
Fine-tuning with hyperparameter optimization The optimal choice of hyperparameters for FT is not only target data dependent but also sensitive to the domain similarity between the source and target datasets (Li et al. 2020). We thus tune the hyperparameters for FT: use Adam optimizer and tune batch size, learning rate, number of epochs and weight decay (1.2 regularization on the classifier head). For tuning, we aside portion of the training data (20%) and try 100 random draws from hyperparameters' multi-dimensional grid. With this additional tuning complexity, we performed 650 x 100 FT experiments. See additional information and motivation in Supplement (S4
Evaluation criteria TMs are often evaluated by how well they correlate with the test accuracy after FT the model on target data. Following Tran et al. (219)); Nguuen et al. (2020): [Hunn al. [202T) we used Pearson correlation. We include additional results with respect to rank correlations (c.g.. Spearman) in Supplement Section S7 We argue that considering correlation with the target test accuracy is flawed in some scenarios. In particular, for TTS, it is wrong to compare target tasks based on accuracy when C is different e.g vs 10 classes. In such a case, task with 10 classes will have a high chance of arriving at lesser test accuracy compared to that for task with 5 classes. In this case, more appropriate to consider the gain accuracy achieved by the model over it's random baseline. Hence we use relative accuracy (for balanced classes): Accuracy-1/C This measure is more effective in capturing the performance gain achieved by the same model in transferring to two domains with different C. This also highlights the limitation of NCE, LEEP and ALEEP which are sensitive to C and tend to have smaller values with higher C; these measures do not provide useful information about how hard these different tasks when evaluated on the original accuracy scale.
I/C
Correlations marked with asterisks (*) in Tables 2 > 0.05). Hyphen (-) indicates the computation ran out of memory or was really slow.
are not statistically significant (p-value
5.1 TARGET TASK SELECTION
We consider Small balanced (S-B) target data, small imbalanced (S-IB) and Large Balance (L-B).
See additional details in Supplement Section S6
Validation of Ha (f) against Supervised TMs We empirically compare the bhrsekage-based H- score against the original measure by (Bao et al. 2019) with pseudo-inverse matrix of the sample feature covariance. Table demonstrates 80% absolute gains in correlation performance of (f)
7

---

Under review as a conference paper at ICLR 2022
over H(f), making it leading metric in many cases in small target data regimes. Table 2 demon- strates how various supervised TMs perform on TIS when the number of target classes varies. (f) dominates the performance in both cases, surpassing all supervised TMs.
Table 1:
Correlation comparison of supervised TMs against FT target accuracy. Larger correla- tions indicate better identifiability as quantified by TM. We compared our proposed Hf( against original H(f) and of art measures.
Fine-tuning
Tonget Data
Medel
Regime
H(
Ho.(f)
NCE
LEEP
.VLEEP
TamaRate
LogME
CIFAR-100
ResNer50
VOG19
S-15
S-8
S-8
0.030
0.138
0,03*
0.007
0.771
0.365
0.063
0.513
0.050
0.84
0.617
0.625
0.701
0.805
0.462
0.564
0,735
0:773
0.759
0.433
0.718
0.338
Tinesr
0.437
CIFAR-10
ResNar50
VEG11
S-1B
5-8
S-8
0.091*
0004
0291
0.808
0.711
0.733
0.23
0.746
0.127
0.96
0.812
0.411
0.611
0.62
0,800
0.09
0.287
0.155
0.565
0.320
0.457
0
0,708
0.735
Noulinear
CIAAR-100
VOG119
3-18
S-R
0.16
0487
0.729
0487
0.535
0.580
0342
0.674
0L51
.0009
0.480
0.73
0.700
Table 2: Correlation performance of TMs against relative accuracy for Large Balanced CIFAR-100
data with different number of classes across target sets.
Model
H())
(f)
NCE
D-NCE
LEEP
n-LEEP
WLEEP -0.932
n-VLEEP 0.945
TransRatc 0
LogME
VGG19
0.876
0.971
00949
0,655
-0.477
0.661
0.968
ResNeL50
0.950
0.736
0,950
Validating Dimensionality Reduction for Unsupervised TMs Table shows that unsupervised
discrepancy measures greatly benefit from dimensionality reduction (DR). Our findings are summarized below:
Table 3: Correlation performance of unsupervised TMs (see Section 4 in FT with/without DR in target task se- lection for ImageNet Models.
MMD and KLD greatly benefit from dimensionality reduction as they indicate significant improve-
Thrger
Rogime
CORAL
MMD
CMD
WD
None
2036
0.15
0.60
0.28
0.28
0.51
ment in all cases where the corre- lations hold significance.
S-B
4.16
0.56
0.55
0.22
0.37
0.54
4.45
0.36
VOG19
203
0.02
0.50
2002
0.17
For both CMD) and WD, dimension- ality reduction improve the correla- tion in 2/3 cases.
$ LB
RP
0006
0.40
0.16
0.31
0.20
0.32
PCA
4.2
0.44
0.20
0.07
0,14
0.27
CIFAR-IOP
0.30
S-B
-035
0.05
0.54
0.28
4009
0.45
OT does not gain from DR in esti- mating transferability.
ResNes50
PCA
-043
4.08
0.20
4..39
0.37
0.24
0.25*
0.35
40.00
40.11
0.16
0.5*
PCA
4007
2.159
0.5
0.1
4.2
0.8
5.2 SOURCE MODEL SELECTION
We select 9 small to large ImageNet models. We evaluate SMS for FT under small sample set- ting. We sample 50 images per class from all classes available the original train split of CIFAR- 100/CIFAR-10. We designate 10 samples per class for hyperparameter tuning.
Validation of H. (f) against supervised TMs We highlight that proposed dimensionality reduction techniques in Section 4 are also pertinent for supervised metrics like H-score for SMS with different feature dimensions because the metric is measured in the feature space (despite using target labels). Given that the feature dimensions vary significantly across different models. we apply RP to project to 128 22s-dimensional space (q = 128). This allows for more meaningful comparison H-score across source models and provides the gains of proposed (f) () in SMS as well for small samples given in Table making it again a leading metric SMS.
Table 4: Correlation of proposed (J) without/with Random Projection (RP) for FT in SMS of ImageNet Models in small data regime constructed from CIFAR-100.
Regime
Target
H(S)
Ha OOINo RPI
Ha. (OURP)
NCE
LEEP
ALLEP
TransRate
LogMls 0.705
CIFAR-100
-0.100
0.024
0.859
0.825
0.839
0.822
-0.204
Linear
CIFAR-10
0,276*
0
0.939
0.938
0.936
0.938
0.311
0.923
Nonlincar
CIFAR-100
-0.108
0.125
0.879
0.967
0.976
0.977
8

---

Under review as a conference paper at ICLR 2022
Comparison of Unsupervised TMs with Supervised TMs in identifying top model Table demonstrates which of the metrics can be used to predict the best source model (Top-3 indicates that the best model is in top predictions sorted by this metric). We show that even unsupervised TMs, which can be used both for supervised and unsupervised domain adaptation, exhibit excellent performance in identifying top performing source. Note this performance of unsupervised TMs is with dimensionality reduction (as outlined in Section 4.. For linear FT, all of the supervised mea- sures (except for original H-score and TransRate) and unsupervised metries (except for OT) include the best model in their top 3 predictions. For tuning additional layers, all unsupervised metrics are able to do the same with random projection.
Table 5: Model identification (as measured by Top-3) by supervissed/onsupervisee measures for FT in SMS of ImageNet Models in small data regime.
Supervised
Unsupervised
Regime
Targel
HOT (7(7)
NCE LEEP XLEEP TramRate LopMIE
CORAL
MMD
CMD
KLD
WD
None
RF
Nose
RP
None
RF
Note
RP
None
RF None
RF
CITAR-
Lincar
CIFAR-10
8
1
/
x
Nonlinear
CIFAR- 100
8
7
5.3 TIMING COMPARISON BETWEEN LOGME AND H.
We empirically investigate the computational times of (f) when computed via our optimized implementation in (8). For
Table 6: Timing comparison of LogME and our shrinkage- based H-score. All times are in m/8.
this exercise, we generate synthetic multi-class classification data using Sklearn (Pedregosa et al.] (011) multi-class dataset generation function that is adapted from Guyon (2003). We investigate different values for number of samples (Rt), feature dimension (d) and number of classes (C). For data generation,
d
| =
LegME
H(f)
H, (f)
500
200
1000
20
300
200
66
set number of informative features to be 100 with the rest of the features filled with random noise. Table demonstrates significant computational advantage of H (f) over LogME. We observe 3- - 55 times faster computational times. LogME seems intractable both with respect memory and time for
500
3000
30
39100
9630
1400
300
1000
10
252
130
63
1000
202
1000
50
305
218
1000
1000
50
333
173
116
d ~ 10a as exposed by the nonlinear settings in Table | and
CONCLUSION
We study both supervised and unsupervised TMs in the context of FT. For supervised TMs. our contributions are three-fold. First, we show that H-score measure, commonly used as a baseline for newer supervised TMss suffers from instability due to poor estimation of covariance matrices. We propose shrinkage-bass estimation of H-score with regularized covariance estimation techniques from statistical literature. We show 80% absolute increase over the original H-score and show supe- rior performance in 9/15 cases against all newer TMs across various FT scenarios and data settings. Second, present fast implementation of our estimator that provides a --55 times computational advantage over state-of-the-art LogME measure. Unlike LogME, our optimized implementation for our estimator is tractable even for really high-dimensional feature embeddings identify problems with 3 other supervised TMs (NCE, LEEP and ALEEPP in TTS (an understud- ied FT scenario than SMS) when either the number of target classes or the class imbalance varies across candidate target tasks. We propose an alternative evaluation scheme that measures correla- tion against relative target accuracy (instead of vanilla accuracy) in such scenarios. For unsupervised TMs, we identify challenges with computation and propose dimensionality reduction methods for better estimation and effective comparison of such measures when the feature dimensions are large and/or different across various source models. We show that even unsupervised TMs can be effec- tive in identifying the best source ImageNet model with our proposals. Our large set of 65,000 FT experiments with multiple ImageNet models and different regimes generated from CIFAR-100 and CIFAR- 10 image datasets demonstrates usefulness of our proposals. We leave for future work to explore how predictive various TMs are for co-training regimes (as opposed to FT).
10 Third, we

---

Under review as a conference paper at ICLR 2022
REFERENCES
Y. Bao, Y. Li, S. Huang. L. Zhang, L. Zheng, A. Zamir, and L. Guibas. An information-theores approach transferability in task transfer learning. In 2019 IEEE International Conference on Image Processing (ICIP), 2309-2313, 2019.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In B. Schölkopf, J. Platt, and T. Hoff- man (eds.). Advances in Neural Information Processing Systems, volume 19. MIT
Press, 2006. URL
htw.:ooeeeeinng..eeiii..e.essese http://ss..cpppeer////////ee pdf
Nicolas Bonneel, Michiel van de Panne, Sylvain Paris, and Wolfgang Heidrich. Displacement in- terpolation using lagrangian mass transport. ACM Trans. Graph. 30(6):1-12, December 2011, ISSN 0730-0301. doi: 01144//07778.2022111 2070781. 2024192
URL Inttps://@ec..org//00/
Y. Chen, A. Wiesel, Y. C. Eldar, and A. O. Hero. Shrinkage algorithms for mmse covariance esti- mation. IEEE Transactions on Signal Processing, 58(10):5016-5029, 2010.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge J. Belongie. Large scale fine the-ggaaned categorization and Noooman--speciff transfer learning. CoRR, http/8806.06190 2018. BRLhttp:
www.iiiwwgggabaa/oooeea /106. 06193
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. In C. J. C. Burges, L. Bottou, M. Welling. Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Asso- ciates, Inc., 2013. URL https: /ttpss//pprooeeeasss..eee http:///wsss...pppe//////////// ptte/pppprr213//ieee af21d0c9 2tl.:00077d2222211357222222
Aditya Deshpande, Alessandro Achille, Avinash Ravichandran, Hao Li, Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona. A linearized framework and new benchmark for model selection for fine-tuning, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1 Abs/1810.00405, 2018. URL
04805
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Additive logistic regression: statistical view of boosting (With discussion and a rejoinder by the authors). The Annals of Statistics, 28(2): 337 - 407, 2000. doi:
101 T001228223]
Isabelle Guyon. Design of experiments for the nips 2003 variable selection benchmark. 2003.
Trevor Hastie, Robert Tibshirani, and Jerome Fricdman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/1512.00335. 2015. URL BtL/htpp: http::://axiv.... org/
03385
Long-Kai Huang, Ying Wei, Yu Rong, Qiang Yang, and Junzhou Huang. Frustratingly easy trans- ferability estimation. ArXiv, 0es/2106609362. 2021.
L.V. Kantorovich. Mathematical methods for manufacturing, organizing and planning. 1939. Reprint of the 1939 edition, St. Petersburg University Publishing House, St. Petersburg 2012.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2656- 2666, 2019. doi: 00.1109/CVPR.2219.0000
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep
volutional neural networks. Neural Information Processing Systems, 25, 2012. doi: 10.1145/3065386.
10

---

Under review as a conference paper at ICLR 2022
S. Kullback and R. A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22:79-86, 1951.
Olivier Ledoit and Michael Wolf. A well-conditioned estimator for large- linee-dimensioall covari- ance matrices. Journal of Multivariate Analysis, 88(2):365-411, February 2004. doi: 1016//00047225500000000000999990 00096-4
URL https: /tte::/ddoi.oe .rg./..g///
s004 25 9x (03)
Hao Li, Pratik Chaudhari, Hao Yang. Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. CoRR, abs/2002.11770, 2020.
URL hwtph:t//axxxiyt. org/ abs/ 2002 11770
Li, E. Chen, Z. Ding, L. Zhu, K. Lu, and H. T. Shen. Maximum density divergence for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2020.
Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2663-2673, June 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference Interna- lional Conference on Machine Learning Volume 37, ICML' pp. 97-105. JMLR.org, 2015.
Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. CoRR, 0bss/1805.0093 2018. URL http://arxiv org/ abs 1100/115.0000 00932
A Woodbury Max. Inverting modified matrices. In Memorandum Rept. 42, Statistical Research Group, pp. 4. Princeton Univ., 1950.
Cuong V. Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations, 2020.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, et al. Scikit-learn: Machine learning in python. Mach. Learn. Res., 12(mull): 283-0 November 2011. ISSN 1532-4435.
Mohsen Pourahmadi. High-dimensional covariance estimation: with high-dimensional data, vol- ume 882. John Wiley & Sons, 2013.
Stephan Rabanser, Stephan Günnemann, and Zachary Chase Lipton. Failing loudly: empirical study of methods for detecting dataset shift. In NeurlPS, 2019.
Y. Rubner, Tomasi, and L.J. Guibas. A metric for distributions with applications to image databases. In Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271), pp. 66, 1998. doi: 10.1 109/1C1..1199999 1991.99078771111.
Juliane Schäfer and Korbinian Strimmer. shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical Applications in Genetics and Molecular Biology, 4, 2005.
K. Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recog- nition. CoRR, abs/ 11t./11009.555. 2015.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Gang Hua and Hervé Jégou (eds.), Computer Vixion - ECCV 2016 Workshops, pp. 443-450, Cham, 2016. Springer International Publishing. ISBN 978-3-319-49409-3
Yang Tan, Yang Li, and Shao-Lun Huang. OTCE: A transferability metric for cross-domain cross-task representations. CoRR, abs/2103.13843, 2021. URL httpts///axxx..ooo /bs/ 2103.13843
A. Tran, C. Nguyen, and T. Hassner. Transferability and hardness of supervised classification tasks. In 2019 IEEF/CVF International Conference on Computer Vision (ICCV) pp. -395-1405 2019.

---

Under review as a conference paper at ICLR 2022
Nguyen Xuan Vinh, Julien Epps, and James Bailey. Information theoretic measures for clus- terings comparison: Variants, properties, normalization and correction for chance. Journal of Machine Learning Research, 11(95):2837-285555 2010. URL htttp::/jml..org////
vl1/vinh10a html
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In ICML, 2021.
Werner Zellinger, Bernhard A. Moser, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz, Robust unsupervised domain adaptation for neural networks via moment alignment. Information Sciences, 183:174--91,, May 2019. doi: 00.1016fj.nss.211.000 025. Utthttps: 1// i.nss//..nnnnnnnnnn.....nno. 2019.
12

---

Under review as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
SI ACRONYMS
Table $1: List of acronyms used in the paper.
Terms
Acronyms
Transferability Measure
TM
Target Task Selection
TIS
Source Model Selection
SMS
Fine- tunentuning
Dimensionality Reduction
DR
Principal Component Analysis
PCA
Random Projection
RP
Amall-Balanced
S-B
Small Imbalanced
S-IB
Large- BalaacBalccdd
L-B
S2 SUPERVISED TRANSFERABILITY METRICS
$2.1 ADDITIONAL DISCUSSION ON SAME SHRINKAGE FOR THE TWO COVARIANCES IN BAEIINKAGE-BSEDD H-SCORE
The covariance [ (*) can not be shrunk independently Σ() in the estimation of the two
covariances coupled by the law of total covariance:
9 = [ - M+*)
(S1)
where fy denotes the feature embedding of target samples that belong class Y y and [(v)
Cov(flY) denotes the class-conditioned covariances. We can write
(1 (1-a)ı(h -1 (1-a)E/s(/+) (1-a) ie, Σ = (1 (1 a) + tr(E())
tr(I()) La + (1 (1)))
Ia = (1 - (1-a)E(E(/)) +a
d
d
(S2)
Comparing S2 with SI we see that the same shrinkage parameter a should used when using
shrinkage estimators, to preserve law of total covariance. The first two terms on the right side in (52) can understood as shrinkage of class-conditioned covariances to the average (global) variance.
The third term in (S2 (e.g. (1 (() can then understood as ridge shrinkage as in
S2.2 CHOICE OF IN TERMS OF IMPACT ON SHRINKAGE-BASED H-SCORE
Following the synthetic example showing the unreliability of the original H-Score in Section 3.1 further visualize the effect of using non-optimal values of alpha on the shrinkage-baat based H-Score. We that the shrinkage-bass based H-Score with optimal shrinkage a is much closer to the population version of the original H-Score, especially for smaller sample cases. This validates the use of a as computed in equation 7
13

---

Under review as a conference paper at ICLR 2022
80
H(f)-Population
H(f)-Empirical
60
Ha= a'==aa.
Ha- 0.01
Scores
40
o.1
0.5
Ha-0.9(f)
20
0
102
10a
10
Number of samples
Figure S1: Effect of on shrinkage-based H-Score.
$2.3 DERIVATION OF OPTIMIZED IMPLEMENTATION FOR Ha(f)
We derive an optimized computation for our proposed shrinkage-basee based H-score Ha(f) for small target data (C The < d) as follows:
(1 (1-a).
((acts- (1 (1-a) FTFF)
(f) E!! (1 -))(())
z²Z
nhe
nt
(1 (1-a) a)
la+ (1-WFFT FT F
RRT
tr
1400
71,00
(1 a))
(I -a) F" (niao,, + +FF") F) RR"),
tr
(S3)
1400
(1-a) a)
(tr (R² R) - -(1-a) tr (G' W GG))
(S4)
1400
(1 a1)))
(1- (l- vec (G)² W (W-1G))
(S5)
n4a
where R
VTifr=1." fc Rdxc, G FR Ra W +
Rmexng
S3 follows by Woodbury matrix identity
Max
1950 and
and (S5) follow by trace properties.
$2.4 NORMALIZATION OF NCE
NCE Tran al. 2019) evaluates conditional entropy between target pseudo labels g (source model's 's assigned labels) and actual target labels, as given by:
1 nhe
NCE(Y())
log Py 200 (v:lz))
(S6)
The
The conditional entropy of the target labels conditioned on the dummy labels (source model' labels on target data) is:
HY
Σ PY. (y, log Py (z(2))
(S7)
DEY,EZ
holds that 0 < Z) HYY) (see appendix S2.4.1
Negative Conditional Entropy (NCE)
is given by NCE -H(Y)Z) and it holds that 0 - NCE S H(Y). We normalize the NCE as follows:
NCE
2
(S8)
H(Y)
NCE
0 S 1
H(Y) <
(S9)
14

---

Under review as a conference paper at ICLR 2022
where in
S8
the fact that entropy is greater than 0 for any practical classification task.
$2.4.
PROOF OF BOUNDS ON CONDITIONAL ENTROPY
H(Y 2) =
E (U,
logrrizu11
(S10)
VEViRE2
Σ Pz(2)
Σ Priz(ele)
(S11)
NEY
Σ (Y(
(S12)
2E2
2 0
(S13)
where (S13) holds because H(Y Z = 2) 2 0 and holds with equality if and only if is a determin- istic function of Z.
H(V =
Σ
(S14)
VEY,2E2
(y, 2)
Σ
(Y, log
(S15)
pz(=)
VE
(Y.
pz (z(2)
E py
log
(S16)
pr(y)
Pr,
VEY.BEZ
(V,((,=))
=) pz(2)
(v) log Σ
(S17)
Pr(
Pr, (U,
VEY
SE2
Σ (1) bY
(S18)
VEY
H(Y)
(S19)
where S17 holds by Jensen inequality.
S3
UNSUPERVISED TRANSFERABILITY MEASURES
We denote the target feature embeddings by f: = h(x(()) (3) E Rd, E [m.] and the source feature
embeddings by 9j h(()))) IRd j E [n.]. where h denotes the feature extractor from the source model.
S3.1 SOURCE AND TARGET FEATURE EMBEDDINGS STANDARDIZATION
We standardize both source and target feature embeddings via feature-wise standardization using source features' first and second order moments (z-score) as follows:
9jk: (gik mean(g.
(S20)
fa = (fik mean(g. std st)
(S21)
S3.2 COMPUTATION OF UNSUPERVISED TMS
We list formulae for 6 different distribution discrepancy measures that don't use target labels and instead rely only on source and target feature embeddings.
Correlation Alignment (CORAL) [Sun & Sacnko (2016) measures discrepancy between the second-order statistics (covariances) of the source and target features given by:
1
CORAL
(S22)
d²
where (f) denotes the sample covariance of target features f and (g) denotes the sample covariance of source features g.
15

---

Under review as a conference paper at ICLR 2022
Maximum Mean Discrepancy (MMD) Long et al. (2013) measures the distance of distributions by projecting them into a rich kernel space. In practice, an estimate of the MMD compares the L2 distance between the empirical kernel mean embeddings of the learnt source and target features. Given samples from both source and target feature embedding distributions, we calculate an unbiased estimate:
T2e
n2
R2
MMD
gg)
K. gi f3)
f3), (S23)
Tlg
where x denotes the radial basis kernel given by x(a, b) = -aala-bi To reduce sensitivity of MMD particular kernel, we use combination of radial basis kernels functions with
{1. 5, 10] as per Long et al. (2015).
Central Moment Discrepancy (CMD) Zellinger et al. (2019) compares means and higher order
moments of feature embedding distributions (without projection into kernel space). Given sam- ples from the source and target feature embedding distributions, we only consider the marginal central moments (up to order of 5). The unbiased estimator of the CMD is given by
CMD
(g) 13
+
Σ k= b al
Ica (g) ct (f)llz,
(S24)
where (f) = (
| (g(
B E denote the k-th order moment of
f and g respectively. a, b denotes the joint distribution support of the source and target feature embeddings.
Optimal Transport (OT): This measure considers the optimal energy required to shift distributions from source features to target features. The measure has been well-studied to compare distribu- tions, see Rubner et al. (998) amongst others. In practice, we solve an optimal transport problem given by
OT min
Σ kEn,
s.t. 71
-1, 7"1 72
1, n2
(S25)
YER
where Cy,k || full, denotes the cost of transporting sample feature from source distribution to target distribution. There are entropy regularized variants that include entropy of coupling matrix and efficiently solve the optimization problem with Sinkhorn algorithm (Cuturi 2013).
Kullbach Leibler Divergence (KLD) and Wasserstein Distance (WD) under Multivariate Normal Assumption: We consider two discrepancy measures under multivariate normal distribution sumptions for source and target features (gg Σ(e)) and E(0). The closed-form solutions for these two measures are given below:
1
d + ( -200) (f (g- -) Σリリ (f (f-9)
KLD
log
WD
9 2
+
tr [(0) + EN (/
S4 TUNING HYPERPARAMETERS
Current practices for FT typically involve a selection of values for hyperparameters when retraining the model on target data. Given that the target datasets are typically small in the transfer learn- ing scenarios, the typical strategy is to adopt the default hyperparameters for training large models while using smaller initial learning rate and fewer epochs for FT. It has been believed that adhering to the original hyperparameters for FT with small learning rate prevents catastrophic forgetting of the originally learned knowledge or features. Many studies have used fixed hyperparameters (e.g. learning rate, momentum and weight decay, number of epochs) for FT. However, the choice of hy- perparameters is not necessarily optimal for FT on target tasks. Earlier work has reported that the performance is sensitive the default hyperparameter selection, in particular learning rate, momen- tum (for stochastic gradient descent), weight decay and number of epochs (Mahajan et al. 2018
Korbblith et al. 2019 Li et al. 2020 The optimal choice of these parameters is not only target
16

---

Under review as a conference paper at ICLR 2022
data dependent but also sensitive the domain similarity between the source and target datasets Li jet al. 2020). Therefore, in order to ensure the target task accuracy (against which the correlation of repeat the FT exercise for 100 trials of hyperpa- rameter settings. We employ Adam for FT experiments and optimize over batch size, learning rate, number of epochs and weight decay (1.2 regularization on the classifier head). We select the space of these hyperparameters based on existing literature on FT, c.g. the learning rate is varied in the range [le- I, le- 5], the number of epochs between {25, 50, 75, 100, 125, 150, 175, 2003. the batch size between {32, 64, } and the weight decay in the range [1c - 6, 1 - 2].
transterability metrics is measured) is optimal,
S5 FEATURE EMBEDDING LAYERS FOR LINEAR AND NONLINEAR FINETUNING
Table S2: Feature extraction layer in ImageNet models for nonlinear FT. The names are from pre-
trained ImageNet models in Tensorflow Keras
Linear Fine- Tuning
Nonlimeur
Fine- Tuning
Embedding Layer
dimensions
Feature Embedeing Layer
Dimensions
VGG19
penolimate
4096
boock3 poul
28 x 28 x 256 = 200, 704
ReaNeR50
pemolimne
XF45
camo Jhlck
28 x 28 250
200. 704
NcSNETTO1
penalimate
blackt,o
704
pemolimms
1024
pool2 poo)
28 x 28
128
100, 352
Models
NeT200
pemalimate
1930
poo12-pol
26 x 28
x 128
100, 352
Xception
penulimase
20R8
x
14
728
142, 088
IncepiionN3
pemlimmae
x
Mabil
pemilimut
cam, camr.pw.S.jeh
x 512
100, 352
ElliceenNNetB
penulimmte
1280
26 x
26
144
= 112, 896
S6 TARGET TASK SELECTION EXPERIMENTS SETUP
This evaluation regime is motivated by task transfer policy learning in robotics/reinforcemenn learn- ing. Under this regime, transferability measures can be used to greedily optimize a task transfer policy given a collection of tasks. For instance, robot has automatically select which new object to pick up. Given that the robot has learned to pick up a few objects before, it would be beneficial for the robot to optimally select the most transferable source/task object pair and improve it's ma- neuvering ability throughout the process in highly efficient manner. TMs can also shed light on the relatedness of different tasks in reinforcement learning setups for better understanding.
We currently evaluate target task selection regime on visual classification tasks with both VGG19 (Simonyan & Zisserman (205)) and ResNet50 (He al. 2015) models on subsets of CIFAR-
100/CIFAR-10 data under three different dataset regimes following Nguyen et al. (0220). In all 3 cases outlined below, 20% of the samples from the randomly generated subsets is designated as validation set for hyperparameter tuning to find the model with with optimal validation accuracy. We use all examples in the original test set for evaluating target data. Both training and validation samples in the subsets used for computation of transfer- ability metrics and we report the correlation of these measures against the (relative) test accuracies for the randomly generated subsets.
sample accuracy performance on the
We make a random selection of 5 classes from CIFAR-IOO/CIFAR- 10 and sample 50 samples per class from the original train split, out of which we designate 10 samples per class for validation. We repeat this exercise 50 times (with a different selection of classes), fine-tune the model for each selection (100 hyperparameter tuning trials per selection to find finetuned model with optimal validation accuracy) and evaluate performance of those optimal models in terms of test accuracy. We then evaluate rank correlations of TMs across the 50 experiments with random selection of 5 sub-classes.
Target Data:
Small-Imbalancee Target Data: We make 50 random selections of classes from CIFAR- 100/CIFAR-10 10, sample between 30- 60 samples from the first class and sample 5x the number of samples from the second class. This makes for a binary imbalanced classification task. We again measure performance of transferability measures against optimal target test accuracy.
Target Data with different number of classes: We randomly select 2-100 classes from CIFAR- 100 and include all samples from the chosen classes (500 samples per class). This
17

---

Under review as a conference paper at ICLR 2022
constructs range of large balanced dataset target task selection cases. We evaluate correlation of TMs with relative target test accuracy across the variable number of target classes.
S7 SPEARMAN RANK CORRELATION PERFORMANCE OF SUPERVISED TRANSFERABILITY MEASURES
We present rank correlation performance of all supervised TMs across various FT scenarios (tar- get task selection and source model selection), FT strategies (linear and nonlinear) various data regimes. Table S3 combines setups in Tables [ and 4 and presents Spearman correlation perfor- mance of H., (J) against supervised TMMs Correlations marked with asterisks (*) are not statistically significant (p-value 0.05). Hyphen (+) indicates the computation RAM and/or was really slow.
of memory 128GB
Table S3: Spearman correlation comparison of supervised TMs. Larger correlations indicate better identifiability as quantified by TM. We compared our proposed Ha (f) against original H(f) and state-of-the art measures. For L-B regimes in the table we correlate against relative accuracy. For other rows, we use vanilla accuracy.
FT vcenacio
FT stralegy
Target Data
Model
Regime
HGG)
H.((F)
e-NCE
n-LEEP
n-VLEEP
TranRRate
LegME
VGG19
S-B
-0.19
4107
0.1
0.72
0.67
0.67
0.8
0.56
0.86
003
Thsk Selection
Linear
ResNet50
5418
5-8
-0
0.3
0.0
0.76
033
068
0:38
0.55
0.78
009
0.19
841
0.81
0.82
CIFAR- 10
ResNe150
VGG19
318
5-6
0.21
-0.51
0.72
0.60
0.75
0.28
0.29
0385
023
031
0.03
0.32
0.39
1136
Nonlinear
CIFAR- 100
VOG19
5:10
0005
0.39
0.76
0.62
0.52
0.68
Linear
CIFAR
Small
0.30
0.88
030
Selection
CIFAR-TO
Nonlinear
CIPAR-100
Small 00027
0.96
0.9
0.95
0.9
With respect to Spearman correlations in the table above, our shrinkage-based H-score Ha (f) leads in 7/15 cases and LogME leads 8/15 cases. In terms of Pearson correlations, Ha(f)) leads 9/15 cases (Tables 2 and LogME leads in 4/15 cases. Additionally, LogME seems to be intractable with respect to memory and computational speed for nonlinear settings where feature dimension large (d ~ 105). Our efficient implementation for HJ"() provides 3 55 times computational advantage over LogME.
S8 EXPERIMENTAL CODE AND TYPE OF RESOURCES
We use Tensorflow Keras for our implementation. Imagenet checkpoints (Resnet and VGG) come from Keras https://keras httss//Arrrasiiaaapplaallitttooo For experiments, we use P100 GPUs per model, 15GB RAM per GPU
18


Reviews:
[{"review_id": "uzqNraAXHlU", "reviewer": "Reviewer_NC1U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strenghs**\n\n+ The approach proposed in the paper is interesting. In particular, the idea of studying deeply an existing measure rather than proposing yet another new one is a good idea. In particular, the paper builds on robust knowledge in statistics and shrinkage operators to estimate high dimensional covariance.\n+ The presentation of the proposed measure as well as its efficient computation is well detailed.\n+ Both supervised and unsupervised setting is taken into account.\n+ Experimental results are promising.\n\n**Weaknesses and main concerns**\n\n+ The paper could be improved on its structure. In particular, section 3.1 is very dense and some choices could be better motivated, such as instance the choice of using a linear operation on the eigenvalues( equation 3), choice of alpha (equation 5) in terms of their impact on the transferability criteria.\n+ Section 3.2 is interesting but what about these measures on SCM ? On different datasets than CIFAR? On different transferability tasks ? For instance on text transfer learning tasks?", "summary_of_the_paper": "This paper focuses on transferability measures both in a supervised and unsupervised context. In particular, the authors propose a shrinkage-based estimation of H-score in order to correct its instability and discuss the limitations of the other approaches on two different scenarios: source model selection or target task selection. Experiments are done on these tasks on CIFAR. \n\n\n\n", "summary_of_the_review": "Interesting paper on robust metrics for measuring transferability having some interesting results and good experimental validation.  Some revisions could improve the paper :\n + Improving the motivations of the different choices in terms of their impact and meaning of the transferability measure.\n + Discuss more the different transferability measures in terms of their meaning in relation with the difficulty and hard settings of transferability tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PJciEimlrpA", "reviewer": "Reviewer_7FSx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper has a clear transferability setup that makes intuitive senses.\n2. The paper improves the state of the arts H-score and some newer metrics by studying both supervised and unsupervised TMs. The paper recognizes the issues with some metrics lie with their sensitivity to the number of classes and imbalanced datasets.\n\nWeakness:\n1. The paper only considers the image classification and test on this problem while largely ignoring other tasks. Can the authors try other tasks too?", "summary_of_the_paper": "The paper studies transferability measure like H-score and shows the instability of H-score due to poor estimation of covariance matrices. The authors propose a new shrinkage-based estimation of H-score and provides a fast implementation for even high dimensional feature embeddings.", "summary_of_the_review": "The paper is a good paper in my opinion, it tries to address an important problem question in transfer learning: a good transferability metric. It addresses some issues of existent metrics and proposes a fast implementation of the proposed metric. The main concern is that does this metric only work for image classification tasks?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Li71XPrL8j3", "reviewer": "Reviewer_M6p2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Inside each sections 3 and 4, ideas, intuitions and contributions are well explained and are easy to follow\n- While not proposing fundamentally novel ideas, this paper highlights problems of existing methods\n\n\nWeaknesses\n- Hypotheses are not always justified or empirically verified. For instance, the hypothesis that the sub-optimal performance of the H-score for measuring transferability in many of the evaluation cases is due to a lack of robustness in estimating the H-score (page 3) is not empirically demonstrated. What could be a toy example to demonstrate that hypothesis? \n- As a whole, the paper could use some restructuring and editing. It’s hard to follow the connections between the supervised and the unsupervised case (both in terms of problems and suggested solutions), how the experiments back up the effectiveness of each solution, whether there is a specific call to action or recommendation for future research on task transferability. For instance, I couldn’t understand where the equations of (11) are used in the experiments.\n\nQuestions for Authors\n- Table 6 compares the speed of LogMe against shrinkage-based H-score. How does shrinkage-based H score compare to the original H-score in terms of speed?\n- Could you elaborate on why Table 5 uses a different metric from Table 1, 2, 3 and 4?\n- While I get the use cases for source model selection, could you elaborate on the practical scenarios for target task selection?\n\nTypos Grammar, Presentation Improvements\n- It would be worth it to restructure sections 3 and 4 to highlight the problems in both setups and then list the proposed solutions. Something that is confusing is that the experiment section doesn’t reflect the dichotomy of supervised vs unsupervised you introduced in sections 3 and 4.\n- Why is Section 2 so small? Could it be integrated into another section instead?\n\n", "summary_of_the_paper": "This paper is interested in task transferability measures (both in the supervised and the unsupervised case). Task transferability measures are useful in quantifying how much knowledge of the source domain or model is transferable to the target model (or target domain in the unsupervised setup). Ultimately, one aims at having effective transferability measures to select the best setup for the task of transfer learning.\nThe 4 main contributions of this paper are:\n- Highlighting instabilities in the computation of the H-score which leads to poor estimate, and proposing to correct these instabilities with regularized covariance estimations, demonstrating an 80% improvement over the original H-score\n- Fast implementation that is 3-55 times faster than another state of the art baseline\n- Identify problems with other existing transferability measures (NCE, LEEP and N LEEP) and propose to measure correlation against relative target accuracy (instead of vanilla accuracy).\n- In the unsupervised setup, propose to use dimension reduction to improve the effectiveness of existing and proposed transferability measures.\n", "summary_of_the_review": "While I like that the paper makes a case that existing measures of task transferability have limitations that have been overlooked and that the authors propose simple solutions to fix these problems, I am still confused at the core hypotheses that are not always empirically verified or the actual call to action.\n\nI am willing to raise my score if the authors address the main weaknesses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers transferability measures both in the supervised and unsupervised domain. It identifies instabilities in the way that H-score is computed and proposes to correct the issue with a shrinkage-based covariance estimations. The proposed fix results in 80% absolute gain over the original H-score and makes it competitive with state-of-the-art LogME metric. The new shrinkage-based H-score is much faster to compute.

Reviewers agree that the paper makes interesting and important contributions. In particular, the reviewers appreciate that the paper takes a deeper look at existing metrics and propose valuable fixes instead of proposing yet another new metric. The paper demonstrates depth of statistic knowledge and proposes shrinkage operators to estimate high dimensional covariance. 

There are a few shortcomings of the paper, however, that suggests that the paper can benefit of another round of improvement. In particular, the paper is very dense with little motivation. Some of the choices in the paper can be motivated better. For instance, the hypothesis of lack of robustness in estimating H-score is not demonstrated empirically. The reviewers also felt that the paper should extend experiments to other domains beyond image.
Decision: Reject