Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance

1 INTRODUCTION . Transfer learning ( TL ) is a set of techniques of using abundant somewhat related source data p ( X ( s ) , Y ( s ) ) to ensure that a model can generalize well to the target domain , defined as either little amount of labelled data p ( X ( t ) , Y ( t ) ) ( supervised ) , and/or a lot of unlabelled data p ( X ( t ) ) ( unsupervised TL ) . TL is most commonly achieved either via fine-tuning or co-training . Fine-tuning ( FT ) is a process of adapting a model trained on source data by using target data to do several optimization steps ( for example , SGD ) that update the model parameters . Co-training on source and target data usually involves reweighting the instances in some way or enforcing domain irrelevance on feature representation layer , such that the model trained on such combined data works well on target data . Fine-tuning is becoming increasing popular because large models like ImageNet ( Krizhevsky et al. , 2012 ) , Bert ( Devlin et al. , 2018 ) etc . are released by companies and are easily modifiable . Training such large models from scratch is often prohibitively expensive for the end user . In this paper , we are primarily interested in effectively measuring transferability before training of the final model begins . Given a source data/model , a transferability measure ( TM ) quantifies how much knowledge of source domain/model is transferable to the target model . Transferability measures ( TMs ) are important for various reasons : they allow understanding of relationships between different learning tasks , selection of highly transferable tasks for joint training on source and target domains , selection of optimal pre-trained source models for the relevant target task , prevention of trial procedures attempting to transfer from each source domain and optimal policy learning in reinforcement learning scenarios ( e.g . optimal selection of next task to learn by a robot ) . If a measure is capable of efficiently and accurately measuring transferability across arbitrary tasks , the problem of task transfer learning is greatly simplified by using the measure to search over candidate transfer sources and targets . Contributions We study both supervised and unsupervised TMs in the context of FT. For supervised TMs , our contributions are three-fold : 1 . We show that H-score , commonly used as a baseline for newer supervised TMs , suffers from instability due to poor estimation of covariance matrices . We propose shrinkage-based estimation of H-score with regularized covariance estimation techniques from statistical literature . We show 80 % absolute increase over the original H-score and show superior performance in 9/15 cases against all newer TMs across various FT scenarios . 2 . We present a fast implementation of our estimator that is 3− 55 times faster than state-of-the-art LogME measure . Unlike LogME , our optimized implementation for our estimator is tractable even for really high-dimensional feature embeddings ∼ 105 . 3 . We identify problems with 3 other supervised TMs ( NCE , LEEP and NLEEP ) in target task selection when either the number of target classes or the class imbalance varies across candidate target tasks . We propose measuring correlation against relative target accuracy ( instead of vanilla accuracy ) in such scenarios . For unsupervised TMs , we outline computational challenges and propose dimensionality reduction methods for better estimation and effective comparison of such measures when the feature dimensions are large and/or different across various source models . We show that with our proposed modifications , even unsupervised TMs can be effective in identifying the best source ImageNet model . Our large set of 65,000 FT experiments with multiple ImageNet models and different regimes generated from CIFAR-100 and CIFAR-10 image datasets shows usefulness of our proposals . This paper is organized as follows . Section 2 describes general FT regimes and transfer learning tasks . Section 3 discusses supervised TM and addresses shortcomings of the pioneer TM ( H-Score ) that arise due to limited target data ( subsection 3.1 ) . In subsection 3.2 we demonstrate problems with recent NCE , LEEP and NLEEP metrics and propose a way to address them . Section 4 highlights shortcomings of different commonly used unsupervised measures for source selection and proposes alternatives that offer improvements . Finally , Section 5 presents a meta study of all metrics . 2 TRANSFERABILITY SETUP . We consider the following FT scenarios based on existing literature . ( i ) Source Model Selection ( SMS ) : For a particular target data/task , this regime aims to select the “ optimal ” source model ( or data ) to transfer-learn from , from a collection of candidate models/data . ( ii ) Target Task Selection ( TTS ) : For a particular ( source ) model , this regime aims to find the most related target data/task . In addition , we primarily consider two different FT strategies : ( i ) Linear FT/head only FT : All layers except for the penultimate layer are frozen . Only the weights of the head classifier are re-trained while fine-tuning . ( ii ) Nonlinear FT : Any arbitrary layer can be designated as a feature extractor , up to which all the layers are frozen ; the subsequent layers include nonlinear transformations and are re-trained along with the head on target data . 3 SUPERVISED TRANSFERABILITY MEASURES . Related Work Recent literature in transfer learning has proposed computationally efficient TMs . We categorize measures that require target labels as supervised TMs . Inspired by principles in information theory , Negative Conditional Entropy ( NCE ) Tran et al . ( 2019 ) uses pre-trained source model and evaluates conditional entropy between target pseudo labels ( source models ’ assigned labels ) and real target labels . Log Expected Empirical Predictor ( LEEP ) ( Nguyen et al. , 2020 ) modifies NCE by using soft predictions from the source model . Both NCE and LEEP do not directly use feature information , hence they are not applicable for layer selection . Cui et al . ( 2018 ) propose representing each output class by the mean of all images from that class and computing Earth Mover ’ s distance between the centroids of the source classes and target classes . Bao et al . ( 2019 ) ; Li et al . ( 2021 ) ; Huang et al . ( 2021 ) ; You et al . ( 2021 ) ; Deshpande et al . ( 2021 ) proposed metrics that capture information from both the ( learnt ) feature representations and the real target labels . These metrics are more appealing as these can be broadly applicable for models that are pre-trained in either supervised or unsupervised fashion . Li et al . ( 2021 ) proposed NLEEP that fits a Gaussian mixture model on the target feature embeddings and computes the LEEP score between the probabalistic assignment of target features to different clusters and the target labels . Huang et al . ( 2021 ) proposed TransRate — a computationally-friendly surrogate of mutual information ( using coding rate ) between the target feature embeddings and the target labels . Bao et al . ( 2019 ) introduced H-score that takes into account inter-class feature variance and feature redundancy . You et al . ( 2021 ) proposed LogME that considers an optimization problem rooted in Bayesian statistics to maximize the marginal likelihood under a linear classifier head . Deshpande et al . ( 2021 ) introduced LFC to measure in-class similarity of target feature embeddings across samples . Finally , Tan et al . ( 2021 ) used Optimal Transport to evaluate domain distance , and combined it , via a linear combination , with NCE . To learn such a measure , a portion of target tasks were set aside , the models were transferred onto these tasks and the results were used to learn the coefficients for the combined Optimal Transport based Conditional Entropy ( OTCE ) metric . While the resulting metric appears to be superior over other non-composite metrics like H-score and LEEP , it is expensive to compute since it requires finding the appropriate coefficients for the combination . Additionally , our results indicate that both components of the measure seem to be individually sub-optimal in measuring transferability against corresponding supervised and unsupervised measures . 3.1 IMPROVED ESTIMATION OF H-SCORE FOR LIMITED TARGET DATA . H-score ( Bao et al. , 2019 ) is one of the pioneer measures that is often used as a baseline for newer supervised TMs , which often demonstrate the improved performance . It characterizes discriminatory strength of feature embedding for classification : H ( f ) = tr ( Σ ( f ) −1 Σ ( z ) ) ( 1 ) where , d is the embedding dimension , fi = h ( x ( t ) i ) ∈ Rd is the target feature embeddings when the feature extractor ( h : Rp → Rd ) from the source model is applied to the target sample x ( t ) i ∈ Rp , F ∈ Rnt×d denotes the corresponding target feature matrix , Y = Y ( t ) ∈ Y = { 1 , · · · , C } are the target data labels , Σ ( f ) ∈ Rd×d denotes the sample feature covariance matrix of f , z = E [ f |Y ] ∈ Rd and Z ∈ Rnt×d denotes the corresponding target conditioned feature matrix , Σ ( z ) ∈ Rd×d denotes the sample covariance matrix of z . Intuitively , H ( f ) captures the notion that higher interclass variance and small feature redundancy results in better transferability . We hypothesize that the sub-optimal performance of H-Score ( compared to that of more recent metrics ) for measuring transferability in many of the evaluation cases , e.g. , in ( Nguyen et al. , 2020 ) , is due to lack of robust estimation of H-Score — see Fig . 1 for a synthetic example showing the non-reliability of empirical H-score over various sample sizes when compared with its population version . Given that many of the deep learning models in the context of TL have high-dimensional feature embedding space — typically larger than the number of target samples — the estimation of the two covariance matrices in H-score becomes challenging : the sample covariance matrix of the feature embedding has a large condition number1 in small data regimes . In many cases , it can not even be inverted . Bao et al . ( 2019 ) used a pseudo-inverse of the covariance matrix Σ ( f ) . However , this method of estimating a precision matrix can be sub-optimal as inversion can amplify estimation error ( Ledoit & Wolf , 2004 ) . We propose to use well-conditioned shrinkage estimators motivated by the rich literature in statistics on the estimation of high-dimensional covariance ( and precision ) matrices ( Pourahmadi , 2013 ) . We show that the use of such shrinkage estimators can offer significant gain in the performance of H-score in predicting transferability . In many cases , as our experiments show , the gain is so significant that H-score becomes a leading TM , surpassing the performance of state-of-the-art measures . 1Condition number of a positive semidefinite matrix A , is the ratio of its largest and smallest eigenvalues . Proposed Transferability Measure We propose the following shrinkage based H-score : Hα ( f ) = tr ( Σ ( f ) −1 α · ( 1− α ) Σ ( z ) ) , ( 2 ) Estimating Σ ( f ) α While there are several possibilities to obtain a regularized covariance matrix ( Pourahmadi , 2013 ) , we present an approach that considers a linear operation on the eigenvalues of the sample version of the feature embedding covariance matrix . Similar ideas of using well-conditioned plug-in covariance matrices are used in the context of discriminant analysis ( Hastie et al. , 2001 ) . In particular , we improve the conditioning of the covariance matrix by considering its weighted convex combination with a scalar multiple of the identity matrix : Σ ( f ) α = ( 1− α ) Σ ( f ) + ασId ( 3 ) where α ∈ [ 0 , 1 ] is the shrinkage parameter and σ is the average variance computed as tr ( Σ ( f ) ) /d . The linear op- eration on the eigenvalues ensures the covariance estimator is positive definite . Note that the inverse of Σ ( f ) α can be computed for every α , by using the eigen-decomposition of Σ ( f ) . The shrinkage parameter controls the bias and variance trade-off ; the optimal α needs to be selected . This distribution-free estimator is well-suited for our application as the explicit convex linear combination is easy to compute and makes the covariance estimates well-conditioned and more accurate ( Ledoit & Wolf , 2004 ; Chen et al. , 2010 ; Schäfer & Strimmer , 2005 ) . Understanding ( 1−α ) Σ ( z ) The scaling factor ( 1− α ) can be understood in terms of regularized covariance matrix estimation under a ridge penalty : 1/ ( 1 + λ ) ·Σ ( z ) = argminΣ̂ ||Σ̂−Σ ( z ) ||22 + λ||Σ̂||22 ( 4 ) where λ ≥ 0 is the ridge penalty . Choosing λ = α/ ( 1−α ) , it becomes clear that ( 1−α ) Σ ( z ) is the regularized covariance matrix . Choice of α Ledoit & Wolf ( 2004 ) proposed a covariance matrix estimator that minimizes mean squared error loss between the shrinkage based covariance estimator and the true covariance matrix . The optimization with respect to α considers the following objective : minα , v E [ ||Σ∗ −Σ||2 ] s.t . Σ∗ = ( 1− α ) Σ ( f ) + αvI , E [ Σ ( f ) ] = Σ . ( 5 ) where ‖A‖2 = tr ( AAT ) /d . This optimization problem permits a closed-form solution for the optimal shrinkage parameter , which is given by : α∗ = E [ ||Σ ( f ) −Σ||2 ] / E [ ||Σ ( f ) − ( tr ( Σ ) /d ) · Id||2 ] ( 6 ) ' min { ( 1/n2t ) ∑ i∈ [ nt ] ||fifTi −Σ ( f ) ||2 / ||Σ ( f ) − ( tr ( Σ ( f ) ) /d ) · Id||2 , 1 } . ( 7 ) where ( 7 ) defines a valid estimator ( not dependent on true covariance matrix ) for practical use . For proof , we refer the readers to Section 2.1 and 3.3 in Ledoit & Wolf ( 2004 ) . We provide some additional discussion on why same α is used for the two regularized covariance matrices in shrinkagebased H-Score in Supplement Section S2.1 . We provide validation of shrinkage-based estimation of H-Score on synthetic classification data . We generated 1 million 1000-dimensional features with 10 classes using Sklearn multi-class dataset generation function ( Pedregosa et al. , 2011 ) . Number of informative features is set to 500 with rest filled with random noise . We visualize the original and the population version of the H-score and the shrinkage-based H-Score for different sample sizes in Fig . 1 . We observe that the original H-Score becomes highly unreliable as the number of samples decreases . In contrast , the shrunken estimation of H-Score is highly stable and has a small error when compared with the population H-Score . Efficient Computation for small target data For small target data ( C ≤ nt < d ) , the naive implementation of Hα ( f ) can be very slow . We propose an optimized implementation for our shrinkagebased H-Score that exploits diagonal plus low-rank structure of Σ ( f ) α for efficient matrix inversion and the low-rank structure of Σ ( z ) for faster matrix-matrix multiplications . We assume F ( and correspondingly Z ) are centered . The optimized computation of Hα ( f ) is given by : Hα ( f ) = ( 1− α ) / ( ntασ ) · ( ‖R‖2F − ( 1− α ) · vec ( G ) T vec ( W−1G ) ) , ( 8 ) whereR = [ √ n1f̄Y=1 , · · · , √ nC f̄Y=C ] ∈ Rd×C , G = FR ∈ Rnt×C , W = ntασIn +FF T ∈ Rnt×nt . The derivation is provided in the Supplement Section S2.3 . We make a timing comparison of our optimized implementation of Hα ( f ) against the computational times of the state-of-the-art LogME measure and demonstrate 3− 55 times faster computation ( see Table 6 in Section 5.3 ) . 3.2 A CLOSER LOOK AT NCE , LEEP AND NLEEP MEASURES Next , we pursue a deeper investigation of some of the newer metrics that are reported to be superior to H-Score and bring to light what appears to be some overlooked issues with these metrics in Target Task Selection ( TTS ) scenario . TTS has received less attention than Source Model Selection ( SMS ) . To our knowledge , we are the first to bring to light some problematic aspects with NCE , LEEP and NLEEP , which can potentially lead to the misuse of these metrics in measuring transferability . These measures are sensitive to the number of target classes ( C ) and tend to be smaller when C is larger ( see Fig . 2 [ Left ] ) . Therefore , use of these measures for target tasks with dif- ferent C will most likely result in selecting the task with a smaller C. However , in practice , it is not always the case that transferring to a task with a smaller C is easier ; for example , reframing a multiclass classification into a set of binary tasks can create more difficult to learn boundaries ( Friedman et al. , 2000 ) . Furthermore , the measures are also problematic if two candidate target tasks have different degree of imbalance in their classes even if C is the same . The measures would predict higher transferability for imbalanced data regimes over balanced settings ( see Fig . 2 [ Right ] ) . However , imbalanced datasets are typically harder to learn . If these measures are correlated against vanilla accuracy , which tends to be higher as the imbalance increases e.g . for binary classification , the measures would falsely suggest they are good indicators of performance . Earlier work did not consider both these aspects and erroneously showed good correlation of these metrics against vanilla accuracy to show dominance of these metrics in TTS with different C ( Nguyen et al. , 2020 ; Tan et al. , 2021 ) and imbalance ( Tan et al. , 2021 ) . Here , we propose a method to ameliorate the shortcomings of to prevent misuse of these measures , so that they lead to more reliable conclusions . We propose to standardize the metrics by the entropy of the target label priors , leading to the definitions in ( 9 ) . This standardization considers both the class imbalance as well as number of classes through the entropy of the target label priors . n-NCE def = 1 + NCE/H ( Y ) , n-LEEP def = 1 + LEEP/H ( Y ) , n-NLEEP def= 1 +NLEEP/H ( Y ) . ( 9 ) Our proposed normalizations in ( 9 ) ensures the normalized NCE is bounded between [ 0 , 1 ] . For proof , see Supplement Section S2.4 . n-NCE is in fact equivalent to normalized mutual information and has been extensively used to measure correlation between two different labelings/clustering of samples ( Vinh et al. , 2010 ) . Given the similar behavior of LEEP and NCE to different C and class imbalance as shown in Fig . 2 , we suggest the same normalization as given in ( 9 ) . However , this normalization does not ensure boundedness of n-LEEP score ( and by extension n-NLEEP ) in the range [ 0 , 1 ] as in the case of n-NCE . For scenarios where candidate target tasks have different C , we propose an alternative evaluation criteria ( relative accuracy ) instead of vanilla accuracy — see Section 5 for more details . We provide empirical validation of the proposed normalization to these measures in Table 2 in Section 5.1 . We also show that our proposed shrinkage-based H-Score is the leading metric even in these scenarios .

Summary: This paper is interested in task transferability measures (both in the supervised and the unsupervised case). Task transferability measures are useful in quantifying how much knowledge of the source domain or model is transferable to the target model (or target domain in the unsupervised setup). Ultimately, one aims at having effective transferability measures to select the best setup for the task of transfer learning. The 4 main contributions of this paper are: - Highlighting instabilities in the computation of the H-score which leads to poor estimate, and proposing to correct these instabilities with regularized covariance estimations, demonstrating an 80% improvement over the original H-score - Fast implementation that is 3-55 times faster than another state of the art baseline - Identify problems with other existing transferability measures (NCE, LEEP and N LEEP) and propose to measure correlation against relative target accuracy (instead of vanilla accuracy). - In the unsupervised setup, propose to use dimension reduction to improve the effectiveness of existing and proposed transferability measures.