An Information-Theoretic Metric of Transferability for Task Transfer Learning

An important question in task transfer learning is to determine task transferability , i.e . given a common input domain , estimating to what extent representations learned from a source task can help in learning a target task . Typically , transferability is either measured experimentally or inferred through task relatedness , which is often defined without a clear operational meaning . In this paper , we present a novel metric , H-score , an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems . Inspired by a principled information theoretic approach , H-score has a direct connection to the asymptotic error probability of the decision function based on the transferred feature . This formulation of transferability can further be used to select a suitable set of source tasks in task transfer learning problems or to devise efficient transfer learning policies . Experiments using both synthetic and real image data show that not only our formulation of transferability is meaningful in practice , but also it can generalize to inference problems beyond classification , such as recognition tasks for 3D indoor-scene understanding . 1 INTRODUCTION . Transfer learning is a learning paradigm that exploits relatedness between different learning tasks in order to gain certain benefits , e.g . reducing the demand for supervision ( Pratt ( 1993 ) ) . In task transfer learning , we assume that the input domain of the different tasks are the same . Then for a target task TT , instead of learning a model from scratch , we can initialize the parameters from a previously trained model for some related source task TS . For example , deep convolutional neural networks trained for the ImageNet classification task have been used as the source network in transfer learning for target tasks with fewer labeled data ( Donahue et al . ( 2014 ) ) , such as medical image analysis ( Shie et al . ( 2015 ) ) and structural damage recognition in buildings ( Gao & Mosalam ) . An imperative question in task transfer learning is transferability , i.e . when a transfer may work and to what extent . Given a metric capable of efficiently and accurately measuring transferability across arbitrary tasks , the problem of task transfer learning , to a large extent , is simplified to search procedures over potential transfer sources and targets as quantified by the metric . Traditionally , transferability is measured purely empirically using model loss or accuracy on the validation set ( Yosinski et al . ( 2014 ) ; Zamir et al . ( 2018 ) ; Conneau et al . ( 2017 ) ) . There have been theoretical studies that focus on task relatedness ( Baxter ( 2000 ) ; Maurer ( 2009 ) ; Pentina & Lampert ( 2014 ) ; Ben-David et al . ( 2003 ) ) . However , they either can not be computed explicitly from data or do not directly explain task transfer performance . In this study , we aim to estimate transferability analytically , directly from the training data . We quantify the transferability of feature representations across tasks via an approach grounded in statistics and information theory . The key idea of our method is to show that the error probability of using a feature of the input data to solve a learning task can be characterized by a linear projection of this feature between the input and output domains . Hence we adopt the projection length as a metric of the feature ’ s effectiveness for the given task , and refer to it as the H-score of the feature . More generally , H-score can be applied to evaluate the performance of features in different tasks , and is particularly useful to quantify feature transferability among tasks . Using this idea , we define task transferability as the normalized H-score of the optimal source feature with respect to the target task . As we demonstrate in this paper , the advantage of our transferability metric is threefold . ( i ) it has a strong operational meaning rooted in statistics and information theory ; ( ii ) it can be computed directly and efficiently from the input data , with fewer samples than those needed for empirical learning ; ( iii ) it can be shown to be strongly consistent with empirical transferability measurements . In this paper , we will first present the theoretical results of the proposed transferability metric in Section 2-4 . Section 5 presents several experiments on real image data , including image classificaton tasks using the Cifar 100 dataset and 3D indoor scene understanding tasks using the Taskonomy dataset created by Zamir et al . ( 2018 ) . A brief review of the related works is included in Section 6 . 2 BACKGROUND . In this section , we will introduce the notations used throughout this paper , as well as some related concepts in Euclidean information theory and statistics . X , x , X and PX represent a random variable , a value , the alphabet and the probability distribution respectively . √ PX denotes the vector with entries √ PX ( x ) and [ √ PX ] the diagonal matrix of√ PX ( X ) . For joint distribution PY X , PY X represents the |Y| × |X | probability matrix . Depending on the context , f ( X ) is either a |X |-dimensional vector whose entries are f ( x ) , or a |X | × k feature matrix . Further , we define a task to be a tuple T = ( X , Y , PXY ) , where X is the training features and Y is the training label , and PXY the joint probability ( possibly unknown ) . Subscripts S and T are used to distinguish the source task from the target task . 2.1 LOCAL INFORMATION GEOMETRY AND HYPOTHESIS TESTING . Our definiton of transferability uses concepts in local information geometry developed by Makur et al . ( 2015 ) , which characterizes probability distributions as vectors in the information space . Consider the following binary hypothesis testing problem : test whether i.i.d . samples xm = { x ( i ) } mi=1 are drawn from distribution P1 or distribution P2 , where P1 , P2 belong to an -neighborhood N ( P0 ) , { P | ∑ x∈X ( P ( x ) −P0 ( x ) ) 2 P0 ( x ) ≤ 2 } centered at a reference distributon P0 . And denote φi = Pi ( x ) −P0 ( x ) √ P0 ( x ) as the information vector corresponding to Pi for i = 1 , 2 . Given k normalized feature functions f ( x ) = [ f1 ( x ) , . . . , fk ( x ) ] , let Ξ ( x ) = [ ξ1 ( x ) , . . . , ξk ( x ) ] be the matrix of feature vectors ξi ( x ) = √ P0 ( x ) fi ( x ) . Given observations xm , feature f is associated with a k-dimensional statistics l ( xm ) = 1m ∑m i=1 ( f ( x ( i ) ) ) for the binary hypothesis testing problem . Let Ef be the error exponent of decision region { xm | l ( xm ) > T } for T ≥ 0 , which characterizes the asymptotic error probability Pe of l ( i.e . limm→∞− 1m log ( Pe ) = E k f ) . Ef can be written as the squared length of a projection : Ekf = k∑ i=1 Efi = k∑ i=l 2 8 〈ξi , φ1 − φ2〉2 + o ( 2 ) ( 1 ) When f ( x ) = log P1 ( x ) P2 ( x ) is the log likelihood ratio , l is the minimum sufficient statistics that achieves the largest error exponent Ekf = ∑k i=1 2 8 ||φ1 − φ2|| 2 + o ( 2 ) by the Chernoff theorem . ( See Appendix A for details . ) In the rest of this paper , we assume is small . 2.2 DIVERGENCE TRANSITION MATRIX . Definition 1 . Matrix B̃ is the Divergence Transition Matrix ( DTM ) of a joint probability PY X if B̃ = [ √ PY ] −1 PY X [ √ PX ] −1 −√PY√PXT . The singular values of B̃ satisfy that 1 ≥ σ1 ≥ · · · ≥ σK = 0 , K , min { |X | , |Y| } . Let Ψ = [ ψ1 , . . . , ψK ] and Φ = [ φ1 , . . . , φK ] be the left and right singular vectors of B̃ . Define functions f∗i ( x ) = φi ( x ) √ PX ( x ) and g∗i ( y ) = ψi ( y ) √ PY ( y ) for each i = 1 , . . . , K − 1 . Makur et al . ( 2015 ) further proved that f∗i and g ∗ i are solutions to the maximal HGR correlation problem studied by Hirschfeld ( 1935 ) ; Gebelein ( 1941 ) ; Rényi ( 1959 ) , defined as follows : ρ ( X ; Y ) = sup f : X → Rk , g : Y → Rk E [ f ( X ) ] = E [ g ( Y ) ] = 0 E [ f ( X ) f ( X ) T ] = I E [ f ( X ) T g ( Y ) ] ( 2 ) The maximal HGR problem finds the K strongest , independent modes in PXY from data . It can be solved efficiently using the Alternating Conditional Expectation ( ACE ) algorithm with provable error bound ( see Appendix B ) . Huang et al . ( 2017 ) further showed that f∗ and g∗ are the universal minimum error probability features in the sense that they can achieve the smallest error probability over all possible inference tasks . 3 MEASURING THE EFFECTIVENESS OF FEATURES IN CLASSIFICATION TASKS . In this section , we present a performance metric of a given feature representation for a learning task . 3.1 H-SCORE . For a classification task involving input variable X and label Y , most learning algorithms work by finding a k-dimensional functional representation f ( x ) of X that is most discriminative for the classification . To measure how effective f ( x ) is in predicting Y , rather than train the model via gradient descent and evaluate its accuracy , we present an analytical approach based on the definition below : Definition 2 . Given data matrix X ∈ Rm×d and label Y ∈ { 1 , . . . , |Y| } . Let f ( x ) be a kdimensional , zero-mean feature function . The H-Score of f ( x ) with respect to the learning task represented by PY X is : H ( f ) = tr ( cov ( f ( X ) ) −1cov ( EPX|Y [ f ( X ) |Y ] ) ) This definition is intuitive from a nearest neighbor search perspective . i.e . a high H-score implies the inter-class variance cov ( EPX|Y [ f ( X ) |Y ] ) of f is large , while feature redundancy tr ( cov ( f ( X ) ) ) is small . Such a feature is discriminative and efficient for learning label Y . More importantly , H ( f ) has a deeper operational meaning related to the asymptotic error probability for a decision rule based on f in the hypothesis testing context , discussed in the next section .

Summary: In this paper, the authors considered a source domain selection problem in transfer learning. Given a feature representation function f, the H-score is defined as the normalized correlation between the output f(X) and the label Y. The transferability is then measured by the ratio of H-score on the target domain and the optimal one. The authors introduced the information-theoretic and statistical meaning of the H-score. Validation of H-score was confirmed by numerical experimenters with image data.