An Information-Theoretic Metric of Transferability for Task Transfer Learning

An important question in task transfer learning is to determine task transferability , i.e . given a common input domain , estimating to what extent representations learned from a source task can help in learning a target task . Typically , transferability is either measured experimentally or inferred through task relatedness , which is often defined without a clear operational meaning . In this paper , we present a novel metric , H-score , an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems . Inspired by a principled information theoretic approach , H-score has a direct connection to the asymptotic error probability of the decision function based on the transferred feature . This formulation of transferability can further be used to select a suitable set of source tasks in task transfer learning problems or to devise efficient transfer learning policies . Experiments using both synthetic and real image data show that not only our formulation of transferability is meaningful in practice , but also it can generalize to inference problems beyond classification , such as recognition tasks for 3D indoor-scene understanding . 1 INTRODUCTION . Transfer learning is a learning paradigm that exploits relatedness between different learning tasks in order to gain certain benefits , e.g . reducing the demand for supervision ( Pratt ( 1993 ) ) . In task transfer learning , we assume that the input domain of the different tasks are the same . Then for a target task TT , instead of learning a model from scratch , we can initialize the parameters from a previously trained model for some related source task TS . For example , deep convolutional neural networks trained for the ImageNet classification task have been used as the source network in transfer learning for target tasks with fewer labeled data ( Donahue et al . ( 2014 ) ) , such as medical image analysis ( Shie et al . ( 2015 ) ) and structural damage recognition in buildings ( Gao & Mosalam ) . An imperative question in task transfer learning is transferability , i.e . when a transfer may work and to what extent . Given a metric capable of efficiently and accurately measuring transferability across arbitrary tasks , the problem of task transfer learning , to a large extent , is simplified to search procedures over potential transfer sources and targets as quantified by the metric . Traditionally , transferability is measured purely empirically using model loss or accuracy on the validation set ( Yosinski et al . ( 2014 ) ; Zamir et al . ( 2018 ) ; Conneau et al . ( 2017 ) ) . There have been theoretical studies that focus on task relatedness ( Baxter ( 2000 ) ; Maurer ( 2009 ) ; Pentina & Lampert ( 2014 ) ; Ben-David et al . ( 2003 ) ) . However , they either can not be computed explicitly from data or do not directly explain task transfer performance . In this study , we aim to estimate transferability analytically , directly from the training data . We quantify the transferability of feature representations across tasks via an approach grounded in statistics and information theory . The key idea of our method is to show that the error probability of using a feature of the input data to solve a learning task can be characterized by a linear projection of this feature between the input and output domains . Hence we adopt the projection length as a metric of the feature ’ s effectiveness for the given task , and refer to it as the H-score of the feature . More generally , H-score can be applied to evaluate the performance of features in different tasks , and is particularly useful to quantify feature transferability among tasks . Using this idea , we define task transferability as the normalized H-score of the optimal source feature with respect to the target task . As we demonstrate in this paper , the advantage of our transferability metric is threefold . ( i ) it has a strong operational meaning rooted in statistics and information theory ; ( ii ) it can be computed directly and efficiently from the input data , with fewer samples than those needed for empirical learning ; ( iii ) it can be shown to be strongly consistent with empirical transferability measurements . In this paper , we will first present the theoretical results of the proposed transferability metric in Section 2-4 . Section 5 presents several experiments on real image data , including image classificaton tasks using the Cifar 100 dataset and 3D indoor scene understanding tasks using the Taskonomy dataset created by Zamir et al . ( 2018 ) . A brief review of the related works is included in Section 6 . 2 BACKGROUND . In this section , we will introduce the notations used throughout this paper , as well as some related concepts in Euclidean information theory and statistics . X , x , X and PX represent a random variable , a value , the alphabet and the probability distribution respectively . √ PX denotes the vector with entries √ PX ( x ) and [ √ PX ] the diagonal matrix of√ PX ( X ) . For joint distribution PY X , PY X represents the |Y| × |X | probability matrix . Depending on the context , f ( X ) is either a |X |-dimensional vector whose entries are f ( x ) , or a |X | × k feature matrix . Further , we define a task to be a tuple T = ( X , Y , PXY ) , where X is the training features and Y is the training label , and PXY the joint probability ( possibly unknown ) . Subscripts S and T are used to distinguish the source task from the target task . 2.1 LOCAL INFORMATION GEOMETRY AND HYPOTHESIS TESTING . Our definiton of transferability uses concepts in local information geometry developed by Makur et al . ( 2015 ) , which characterizes probability distributions as vectors in the information space . Consider the following binary hypothesis testing problem : test whether i.i.d . samples xm = { x ( i ) } mi=1 are drawn from distribution P1 or distribution P2 , where P1 , P2 belong to an -neighborhood N ( P0 ) , { P | ∑ x∈X ( P ( x ) −P0 ( x ) ) 2 P0 ( x ) ≤ 2 } centered at a reference distributon P0 . And denote φi = Pi ( x ) −P0 ( x ) √ P0 ( x ) as the information vector corresponding to Pi for i = 1 , 2 . Given k normalized feature functions f ( x ) = [ f1 ( x ) , . . . , fk ( x ) ] , let Ξ ( x ) = [ ξ1 ( x ) , . . . , ξk ( x ) ] be the matrix of feature vectors ξi ( x ) = √ P0 ( x ) fi ( x ) . Given observations xm , feature f is associated with a k-dimensional statistics l ( xm ) = 1m ∑m i=1 ( f ( x ( i ) ) ) for the binary hypothesis testing problem . Let Ef be the error exponent of decision region { xm | l ( xm ) > T } for T ≥ 0 , which characterizes the asymptotic error probability Pe of l ( i.e . limm→∞− 1m log ( Pe ) = E k f ) . Ef can be written as the squared length of a projection : Ekf = k∑ i=1 Efi = k∑ i=l 2 8 〈ξi , φ1 − φ2〉2 + o ( 2 ) ( 1 ) When f ( x ) = log P1 ( x ) P2 ( x ) is the log likelihood ratio , l is the minimum sufficient statistics that achieves the largest error exponent Ekf = ∑k i=1 2 8 ||φ1 − φ2|| 2 + o ( 2 ) by the Chernoff theorem . ( See Appendix A for details . ) In the rest of this paper , we assume is small . 2.2 DIVERGENCE TRANSITION MATRIX . Definition 1 . Matrix B̃ is the Divergence Transition Matrix ( DTM ) of a joint probability PY X if B̃ = [ √ PY ] −1 PY X [ √ PX ] −1 −√PY√PXT . The singular values of B̃ satisfy that 1 ≥ σ1 ≥ · · · ≥ σK = 0 , K , min { |X | , |Y| } . Let Ψ = [ ψ1 , . . . , ψK ] and Φ = [ φ1 , . . . , φK ] be the left and right singular vectors of B̃ . Define functions f∗i ( x ) = φi ( x ) √ PX ( x ) and g∗i ( y ) = ψi ( y ) √ PY ( y ) for each i = 1 , . . . , K − 1 . Makur et al . ( 2015 ) further proved that f∗i and g ∗ i are solutions to the maximal HGR correlation problem studied by Hirschfeld ( 1935 ) ; Gebelein ( 1941 ) ; Rényi ( 1959 ) , defined as follows : ρ ( X ; Y ) = sup f : X → Rk , g : Y → Rk E [ f ( X ) ] = E [ g ( Y ) ] = 0 E [ f ( X ) f ( X ) T ] = I E [ f ( X ) T g ( Y ) ] ( 2 ) The maximal HGR problem finds the K strongest , independent modes in PXY from data . It can be solved efficiently using the Alternating Conditional Expectation ( ACE ) algorithm with provable error bound ( see Appendix B ) . Huang et al . ( 2017 ) further showed that f∗ and g∗ are the universal minimum error probability features in the sense that they can achieve the smallest error probability over all possible inference tasks . 3 MEASURING THE EFFECTIVENESS OF FEATURES IN CLASSIFICATION TASKS . In this section , we present a performance metric of a given feature representation for a learning task . 3.1 H-SCORE . For a classification task involving input variable X and label Y , most learning algorithms work by finding a k-dimensional functional representation f ( x ) of X that is most discriminative for the classification . To measure how effective f ( x ) is in predicting Y , rather than train the model via gradient descent and evaluate its accuracy , we present an analytical approach based on the definition below : Definition 2 . Given data matrix X ∈ Rm×d and label Y ∈ { 1 , . . . , |Y| } . Let f ( x ) be a kdimensional , zero-mean feature function . The H-Score of f ( x ) with respect to the learning task represented by PY X is : H ( f ) = tr ( cov ( f ( X ) ) −1cov ( EPX|Y [ f ( X ) |Y ] ) ) This definition is intuitive from a nearest neighbor search perspective . i.e . a high H-score implies the inter-class variance cov ( EPX|Y [ f ( X ) |Y ] ) of f is large , while feature redundancy tr ( cov ( f ( X ) ) ) is small . Such a feature is discriminative and efficient for learning label Y . More importantly , H ( f ) has a deeper operational meaning related to the asymptotic error probability for a decision rule based on f in the hypothesis testing context , discussed in the next section .

Summary: The paper proposes the H score H(f), a quantity that measure the goodness of feature f(x) for predicting some target y. This heavily builds on Makur et al. (2015) who introduce information vectors, the error exponent, and the DTM matrix. The paper connects H(f) with these quantities to justify the proposal (e.g., H(f) is proportional to the error exponent (Theorem 1)). The actual transferability is measured by the ratio between H(f) and H(f_opt) where the latter can be computed using the approach of Makur et al.