Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance

1 INTRODUCTION . Transfer learning ( TL ) is a set of techniques of using abundant somewhat related source data p ( X ( s ) , Y ( s ) ) to ensure that a model can generalize well to the target domain , defined as either little amount of labelled data p ( X ( t ) , Y ( t ) ) ( supervised ) , and/or a lot of unlabelled data p ( X ( t ) ) ( unsupervised TL ) . TL is most commonly achieved either via fine-tuning or co-training . Fine-tuning ( FT ) is a process of adapting a model trained on source data by using target data to do several optimization steps ( for example , SGD ) that update the model parameters . Co-training on source and target data usually involves reweighting the instances in some way or enforcing domain irrelevance on feature representation layer , such that the model trained on such combined data works well on target data . Fine-tuning is becoming increasing popular because large models like ImageNet ( Krizhevsky et al. , 2012 ) , Bert ( Devlin et al. , 2018 ) etc . are released by companies and are easily modifiable . Training such large models from scratch is often prohibitively expensive for the end user . In this paper , we are primarily interested in effectively measuring transferability before training of the final model begins . Given a source data/model , a transferability measure ( TM ) quantifies how much knowledge of source domain/model is transferable to the target model . Transferability measures ( TMs ) are important for various reasons : they allow understanding of relationships between different learning tasks , selection of highly transferable tasks for joint training on source and target domains , selection of optimal pre-trained source models for the relevant target task , prevention of trial procedures attempting to transfer from each source domain and optimal policy learning in reinforcement learning scenarios ( e.g . optimal selection of next task to learn by a robot ) . If a measure is capable of efficiently and accurately measuring transferability across arbitrary tasks , the problem of task transfer learning is greatly simplified by using the measure to search over candidate transfer sources and targets . Contributions We study both supervised and unsupervised TMs in the context of FT. For supervised TMs , our contributions are three-fold : 1 . We show that H-score , commonly used as a baseline for newer supervised TMs , suffers from instability due to poor estimation of covariance matrices . We propose shrinkage-based estimation of H-score with regularized covariance estimation techniques from statistical literature . We show 80 % absolute increase over the original H-score and show superior performance in 9/15 cases against all newer TMs across various FT scenarios . 2 . We present a fast implementation of our estimator that is 3− 55 times faster than state-of-the-art LogME measure . Unlike LogME , our optimized implementation for our estimator is tractable even for really high-dimensional feature embeddings ∼ 105 . 3 . We identify problems with 3 other supervised TMs ( NCE , LEEP and NLEEP ) in target task selection when either the number of target classes or the class imbalance varies across candidate target tasks . We propose measuring correlation against relative target accuracy ( instead of vanilla accuracy ) in such scenarios . For unsupervised TMs , we outline computational challenges and propose dimensionality reduction methods for better estimation and effective comparison of such measures when the feature dimensions are large and/or different across various source models . We show that with our proposed modifications , even unsupervised TMs can be effective in identifying the best source ImageNet model . Our large set of 65,000 FT experiments with multiple ImageNet models and different regimes generated from CIFAR-100 and CIFAR-10 image datasets shows usefulness of our proposals . This paper is organized as follows . Section 2 describes general FT regimes and transfer learning tasks . Section 3 discusses supervised TM and addresses shortcomings of the pioneer TM ( H-Score ) that arise due to limited target data ( subsection 3.1 ) . In subsection 3.2 we demonstrate problems with recent NCE , LEEP and NLEEP metrics and propose a way to address them . Section 4 highlights shortcomings of different commonly used unsupervised measures for source selection and proposes alternatives that offer improvements . Finally , Section 5 presents a meta study of all metrics . 2 TRANSFERABILITY SETUP . We consider the following FT scenarios based on existing literature . ( i ) Source Model Selection ( SMS ) : For a particular target data/task , this regime aims to select the “ optimal ” source model ( or data ) to transfer-learn from , from a collection of candidate models/data . ( ii ) Target Task Selection ( TTS ) : For a particular ( source ) model , this regime aims to find the most related target data/task . In addition , we primarily consider two different FT strategies : ( i ) Linear FT/head only FT : All layers except for the penultimate layer are frozen . Only the weights of the head classifier are re-trained while fine-tuning . ( ii ) Nonlinear FT : Any arbitrary layer can be designated as a feature extractor , up to which all the layers are frozen ; the subsequent layers include nonlinear transformations and are re-trained along with the head on target data . 3 SUPERVISED TRANSFERABILITY MEASURES . Related Work Recent literature in transfer learning has proposed computationally efficient TMs . We categorize measures that require target labels as supervised TMs . Inspired by principles in information theory , Negative Conditional Entropy ( NCE ) Tran et al . ( 2019 ) uses pre-trained source model and evaluates conditional entropy between target pseudo labels ( source models ’ assigned labels ) and real target labels . Log Expected Empirical Predictor ( LEEP ) ( Nguyen et al. , 2020 ) modifies NCE by using soft predictions from the source model . Both NCE and LEEP do not directly use feature information , hence they are not applicable for layer selection . Cui et al . ( 2018 ) propose representing each output class by the mean of all images from that class and computing Earth Mover ’ s distance between the centroids of the source classes and target classes . Bao et al . ( 2019 ) ; Li et al . ( 2021 ) ; Huang et al . ( 2021 ) ; You et al . ( 2021 ) ; Deshpande et al . ( 2021 ) proposed metrics that capture information from both the ( learnt ) feature representations and the real target labels . These metrics are more appealing as these can be broadly applicable for models that are pre-trained in either supervised or unsupervised fashion . Li et al . ( 2021 ) proposed NLEEP that fits a Gaussian mixture model on the target feature embeddings and computes the LEEP score between the probabalistic assignment of target features to different clusters and the target labels . Huang et al . ( 2021 ) proposed TransRate — a computationally-friendly surrogate of mutual information ( using coding rate ) between the target feature embeddings and the target labels . Bao et al . ( 2019 ) introduced H-score that takes into account inter-class feature variance and feature redundancy . You et al . ( 2021 ) proposed LogME that considers an optimization problem rooted in Bayesian statistics to maximize the marginal likelihood under a linear classifier head . Deshpande et al . ( 2021 ) introduced LFC to measure in-class similarity of target feature embeddings across samples . Finally , Tan et al . ( 2021 ) used Optimal Transport to evaluate domain distance , and combined it , via a linear combination , with NCE . To learn such a measure , a portion of target tasks were set aside , the models were transferred onto these tasks and the results were used to learn the coefficients for the combined Optimal Transport based Conditional Entropy ( OTCE ) metric . While the resulting metric appears to be superior over other non-composite metrics like H-score and LEEP , it is expensive to compute since it requires finding the appropriate coefficients for the combination . Additionally , our results indicate that both components of the measure seem to be individually sub-optimal in measuring transferability against corresponding supervised and unsupervised measures . 3.1 IMPROVED ESTIMATION OF H-SCORE FOR LIMITED TARGET DATA . H-score ( Bao et al. , 2019 ) is one of the pioneer measures that is often used as a baseline for newer supervised TMs , which often demonstrate the improved performance . It characterizes discriminatory strength of feature embedding for classification : H ( f ) = tr ( Σ ( f ) −1 Σ ( z ) ) ( 1 ) where , d is the embedding dimension , fi = h ( x ( t ) i ) ∈ Rd is the target feature embeddings when the feature extractor ( h : Rp → Rd ) from the source model is applied to the target sample x ( t ) i ∈ Rp , F ∈ Rnt×d denotes the corresponding target feature matrix , Y = Y ( t ) ∈ Y = { 1 , · · · , C } are the target data labels , Σ ( f ) ∈ Rd×d denotes the sample feature covariance matrix of f , z = E [ f |Y ] ∈ Rd and Z ∈ Rnt×d denotes the corresponding target conditioned feature matrix , Σ ( z ) ∈ Rd×d denotes the sample covariance matrix of z . Intuitively , H ( f ) captures the notion that higher interclass variance and small feature redundancy results in better transferability . We hypothesize that the sub-optimal performance of H-Score ( compared to that of more recent metrics ) for measuring transferability in many of the evaluation cases , e.g. , in ( Nguyen et al. , 2020 ) , is due to lack of robust estimation of H-Score — see Fig . 1 for a synthetic example showing the non-reliability of empirical H-score over various sample sizes when compared with its population version . Given that many of the deep learning models in the context of TL have high-dimensional feature embedding space — typically larger than the number of target samples — the estimation of the two covariance matrices in H-score becomes challenging : the sample covariance matrix of the feature embedding has a large condition number1 in small data regimes . In many cases , it can not even be inverted . Bao et al . ( 2019 ) used a pseudo-inverse of the covariance matrix Σ ( f ) . However , this method of estimating a precision matrix can be sub-optimal as inversion can amplify estimation error ( Ledoit & Wolf , 2004 ) . We propose to use well-conditioned shrinkage estimators motivated by the rich literature in statistics on the estimation of high-dimensional covariance ( and precision ) matrices ( Pourahmadi , 2013 ) . We show that the use of such shrinkage estimators can offer significant gain in the performance of H-score in predicting transferability . In many cases , as our experiments show , the gain is so significant that H-score becomes a leading TM , surpassing the performance of state-of-the-art measures . 1Condition number of a positive semidefinite matrix A , is the ratio of its largest and smallest eigenvalues . Proposed Transferability Measure We propose the following shrinkage based H-score : Hα ( f ) = tr ( Σ ( f ) −1 α · ( 1− α ) Σ ( z ) ) , ( 2 ) Estimating Σ ( f ) α While there are several possibilities to obtain a regularized covariance matrix ( Pourahmadi , 2013 ) , we present an approach that considers a linear operation on the eigenvalues of the sample version of the feature embedding covariance matrix . Similar ideas of using well-conditioned plug-in covariance matrices are used in the context of discriminant analysis ( Hastie et al. , 2001 ) . In particular , we improve the conditioning of the covariance matrix by considering its weighted convex combination with a scalar multiple of the identity matrix : Σ ( f ) α = ( 1− α ) Σ ( f ) + ασId ( 3 ) where α ∈ [ 0 , 1 ] is the shrinkage parameter and σ is the average variance computed as tr ( Σ ( f ) ) /d . The linear op- eration on the eigenvalues ensures the covariance estimator is positive definite . Note that the inverse of Σ ( f ) α can be computed for every α , by using the eigen-decomposition of Σ ( f ) . The shrinkage parameter controls the bias and variance trade-off ; the optimal α needs to be selected . This distribution-free estimator is well-suited for our application as the explicit convex linear combination is easy to compute and makes the covariance estimates well-conditioned and more accurate ( Ledoit & Wolf , 2004 ; Chen et al. , 2010 ; Schäfer & Strimmer , 2005 ) . Understanding ( 1−α ) Σ ( z ) The scaling factor ( 1− α ) can be understood in terms of regularized covariance matrix estimation under a ridge penalty : 1/ ( 1 + λ ) ·Σ ( z ) = argminΣ̂ ||Σ̂−Σ ( z ) ||22 + λ||Σ̂||22 ( 4 ) where λ ≥ 0 is the ridge penalty . Choosing λ = α/ ( 1−α ) , it becomes clear that ( 1−α ) Σ ( z ) is the regularized covariance matrix . Choice of α Ledoit & Wolf ( 2004 ) proposed a covariance matrix estimator that minimizes mean squared error loss between the shrinkage based covariance estimator and the true covariance matrix . The optimization with respect to α considers the following objective : minα , v E [ ||Σ∗ −Σ||2 ] s.t . Σ∗ = ( 1− α ) Σ ( f ) + αvI , E [ Σ ( f ) ] = Σ . ( 5 ) where ‖A‖2 = tr ( AAT ) /d . This optimization problem permits a closed-form solution for the optimal shrinkage parameter , which is given by : α∗ = E [ ||Σ ( f ) −Σ||2 ] / E [ ||Σ ( f ) − ( tr ( Σ ) /d ) · Id||2 ] ( 6 ) ' min { ( 1/n2t ) ∑ i∈ [ nt ] ||fifTi −Σ ( f ) ||2 / ||Σ ( f ) − ( tr ( Σ ( f ) ) /d ) · Id||2 , 1 } . ( 7 ) where ( 7 ) defines a valid estimator ( not dependent on true covariance matrix ) for practical use . For proof , we refer the readers to Section 2.1 and 3.3 in Ledoit & Wolf ( 2004 ) . We provide some additional discussion on why same α is used for the two regularized covariance matrices in shrinkagebased H-Score in Supplement Section S2.1 . We provide validation of shrinkage-based estimation of H-Score on synthetic classification data . We generated 1 million 1000-dimensional features with 10 classes using Sklearn multi-class dataset generation function ( Pedregosa et al. , 2011 ) . Number of informative features is set to 500 with rest filled with random noise . We visualize the original and the population version of the H-score and the shrinkage-based H-Score for different sample sizes in Fig . 1 . We observe that the original H-Score becomes highly unreliable as the number of samples decreases . In contrast , the shrunken estimation of H-Score is highly stable and has a small error when compared with the population H-Score . Efficient Computation for small target data For small target data ( C ≤ nt < d ) , the naive implementation of Hα ( f ) can be very slow . We propose an optimized implementation for our shrinkagebased H-Score that exploits diagonal plus low-rank structure of Σ ( f ) α for efficient matrix inversion and the low-rank structure of Σ ( z ) for faster matrix-matrix multiplications . We assume F ( and correspondingly Z ) are centered . The optimized computation of Hα ( f ) is given by : Hα ( f ) = ( 1− α ) / ( ntασ ) · ( ‖R‖2F − ( 1− α ) · vec ( G ) T vec ( W−1G ) ) , ( 8 ) whereR = [ √ n1f̄Y=1 , · · · , √ nC f̄Y=C ] ∈ Rd×C , G = FR ∈ Rnt×C , W = ntασIn +FF T ∈ Rnt×nt . The derivation is provided in the Supplement Section S2.3 . We make a timing comparison of our optimized implementation of Hα ( f ) against the computational times of the state-of-the-art LogME measure and demonstrate 3− 55 times faster computation ( see Table 6 in Section 5.3 ) . 3.2 A CLOSER LOOK AT NCE , LEEP AND NLEEP MEASURES Next , we pursue a deeper investigation of some of the newer metrics that are reported to be superior to H-Score and bring to light what appears to be some overlooked issues with these metrics in Target Task Selection ( TTS ) scenario . TTS has received less attention than Source Model Selection ( SMS ) . To our knowledge , we are the first to bring to light some problematic aspects with NCE , LEEP and NLEEP , which can potentially lead to the misuse of these metrics in measuring transferability . These measures are sensitive to the number of target classes ( C ) and tend to be smaller when C is larger ( see Fig . 2 [ Left ] ) . Therefore , use of these measures for target tasks with dif- ferent C will most likely result in selecting the task with a smaller C. However , in practice , it is not always the case that transferring to a task with a smaller C is easier ; for example , reframing a multiclass classification into a set of binary tasks can create more difficult to learn boundaries ( Friedman et al. , 2000 ) . Furthermore , the measures are also problematic if two candidate target tasks have different degree of imbalance in their classes even if C is the same . The measures would predict higher transferability for imbalanced data regimes over balanced settings ( see Fig . 2 [ Right ] ) . However , imbalanced datasets are typically harder to learn . If these measures are correlated against vanilla accuracy , which tends to be higher as the imbalance increases e.g . for binary classification , the measures would falsely suggest they are good indicators of performance . Earlier work did not consider both these aspects and erroneously showed good correlation of these metrics against vanilla accuracy to show dominance of these metrics in TTS with different C ( Nguyen et al. , 2020 ; Tan et al. , 2021 ) and imbalance ( Tan et al. , 2021 ) . Here , we propose a method to ameliorate the shortcomings of to prevent misuse of these measures , so that they lead to more reliable conclusions . We propose to standardize the metrics by the entropy of the target label priors , leading to the definitions in ( 9 ) . This standardization considers both the class imbalance as well as number of classes through the entropy of the target label priors . n-NCE def = 1 + NCE/H ( Y ) , n-LEEP def = 1 + LEEP/H ( Y ) , n-NLEEP def= 1 +NLEEP/H ( Y ) . ( 9 ) Our proposed normalizations in ( 9 ) ensures the normalized NCE is bounded between [ 0 , 1 ] . For proof , see Supplement Section S2.4 . n-NCE is in fact equivalent to normalized mutual information and has been extensively used to measure correlation between two different labelings/clustering of samples ( Vinh et al. , 2010 ) . Given the similar behavior of LEEP and NCE to different C and class imbalance as shown in Fig . 2 , we suggest the same normalization as given in ( 9 ) . However , this normalization does not ensure boundedness of n-LEEP score ( and by extension n-NLEEP ) in the range [ 0 , 1 ] as in the case of n-NCE . For scenarios where candidate target tasks have different C , we propose an alternative evaluation criteria ( relative accuracy ) instead of vanilla accuracy — see Section 5 for more details . We provide empirical validation of the proposed normalization to these measures in Table 2 in Section 5.1 . We also show that our proposed shrinkage-based H-Score is the leading metric even in these scenarios .

Summary: This paper focuses on transferability measures both in a supervised and unsupervised context. In particular, the authors propose a shrinkage-based estimation of H-score in order to correct its instability and discuss the limitations of the other approaches on two different scenarios: source model selection or target task selection. Experiments are done on these tasks on CIFAR.