Deciphering and Optimizing Multi-Task Learning: a Random Matrix Approach

This article provides theoretical insights into the inner workings of multi-task and transfer learning methods , by studying the tractable least-square support vector machine multi-task learning ( LS-SVM MTL ) method , in the limit of large ( p ) and numerous ( n ) data . By a random matrix analysis applied to a Gaussian mixture data model , the performance of MTL LS-SVM is shown to converge , as n , p→∞ , to a deterministic limit involving simple ( small-dimensional ) statistics of the data . We prove ( i ) that the standard MTL LS-SVM algorithm is in general strongly biased and may dramatically fail ( to the point that individual single-task LS-SVMs may outperform the MTL approach , even for quite resembling tasks ) : our analysis provides a simple method to correct these biases , and that we reveal ( ii ) the sufficient statistics at play in the method , which can be efficiently estimated , even for quite small datasets . The latter result is exploited to automatically optimize the hyperparameters without resorting to any cross-validation procedure . Experiments on popular datasets demonstrate that our improved MTL LS-SVM method is computationally-efficient and outperforms sometimes much more elaborate state-of-the-art multi-task and transfer learning techniques . 1 INTRODUCTION . The advent of elaborate learning machines capable to surpass human performances on dedicated tasks has reopened past challenges in machine learning . Transfer learning , and multitask learning ( MTL ) in general , by which known tasks are used to help a machine learn other related tasks , is one of them . The particularly interesting aspects of multi-task learning lie in the possibility ( i ) to exploit the resemblance between the datasets associated to each task so the tasks “ help each other ” and ( ii ) to train a machine on a specific target dataset comprised of few labelled data by exploiting much larger labelled datasets , however composed of different data . Practical applications are numerous , ranging from the prediction of student test results for a collection of schools ( Aitkin & Longford , 1986 ) , to survival of patients in different clinics , to the value of many possibly related financial indicators ( Allenby & Rossi , 1998 ) , to the preference modelling of individuals in a marketing context , etc . Since MTL seeks to improve the performance of a task with the help of related tasks , a central issue to ( i ) understand the functioning of MTL , ( ii ) adequately adapt its hyperparameters and eventually ( iii ) improve its performances consists in characterizing how MTL relates tasks to one another and in identifying which features are “ transferred ” . The article aims to decipher these fundamental aspects for sufficiently general data models . Several data models may be accounted for to enforce relatedness between tasks . A common assumption is that the data lie close to each other in a geometrical sense ( Evgeniou & Pontil , 2004 ) , live in a low dimensional manifold ( Agarwal et al. , 2010 ) , or share a common prior ( Daumé III , 2009 ) . We follow here the latter assumption in assuming that , for each task , the data arise from a 2-class Gaussian mixture.1 Methodologically , in its simplest approach , MTL algorithms can be obtained from a mere extension of support vector machines ( SVM ) , accounting for more than one task . That is , instead of finding the hyperplane ( through its normal vector ω ) best separating the two classes of a unique dataset , ( Evgeniou & Pontil , 2004 ) proposes to produce best separating hyperplanes ( or normal vectors ) ω1 , . . . , ωk for each pair of data classes of k tasks , with the additional constraint that the normal vectors take the form ωi = ω0 + vi for some common vector w0 and dedicated vectors vi . The amplitude of the vectors vi is controlled ( through an additional hyperparameter ) to enforce or relax task relatedness . We study this approach here . Yet , to obtain explicit and thus more insightful results , we specifically resort to a least-square SVM ( as proposed e.g. , in ( Xu et al. , 2013 ) ) rather than a margin-based SVM . This only marginally alters the overall behavior of the MTL algorithm and has no impact on the main insights drawn in the article . Moreover , by a now well-established universality argument in large dimensional statistics , ( Mai et al. , 2019 ) show that quadratic ( leastsquare ) cost functions are asymptotically optimal and uniformly outperform alternative costs ( such as margin-based methods or logistic approaches ) , even in a classification setting . This argument further motivates the choice of considering first and foremost the LS-SVM version of MTL-SVM . Technically , the article exploits the powerful random matrix theory to study the performance of the MTL least-square SVM algorithm ( MTL LS-SVM ) for data arising from a Gaussian mixture model , assuming the total number n and dimension p of the data are both large , i.e. , as n , p → ∞ with p/n→ c ∈ ( 0 , ∞ ) . As such , our work follows after the recent wave of interest into the asymptotics of machine learning algorithms , such as studied lately in e.g. , ( Liao & Couillet , 2019 ; Deng et al. , 2019 ; Mai & Couillet , 2018 ; El Karoui et al. , 2010 ) . Our analysis reveals the following major conclusions : • we exhibit the sufficient statistics , which concretely enable task comparison in the MTL LS-SVM algorithm ; we show that , even when data are of large dimensions ( p 1 ) , these statistics remain small dimensional ( they only scale with the number k of tasks ) ; • while it is conventional to manually set labels associated to each dataset within { −1 , 1 } , we prove that this choice is largely suboptimal and may even cause MTL to severely fail ( causing “ negative transfer ” ) ; we instead provide the optimal values for the labels of each dataset , which depend on the sought-for objective : these optimal values are furthermore easily estimated from very few training data ( i.e. , no cross-validation is needed ) ; • for unknown new data x , the MTL LS-SVM algorithm allocates a class based on the comparison of a score g ( x ) to a threshold ζ , usually set to zero . We prove that , depending on the statistics and number of elements of the training dataset , a bias is naturally induced that makes ζ = 0 a largely suboptimal choice in general . We provide a correction for this bias , which again can be estimated from the training data alone ; • we demonstrate on popular real datasets that our proposed optimized MTL LS-SVM is both resilient to real data and also manages , despite its not being a best-in-class MTL algorithm , to rival and sometimes largely outperform competing state-of-the-art algorithms . These conclusions thus allow for an optimal use of MTL LS-SVM with performance-maximizing hyperparameters and strong theoretical guarantees . As such , the present article offers through MTL LS-SVM a viable fully-controlled ( even better performing ) alternative to state-of-the-art MTL . Reproducibility . Matlab and Julia codes for reproducing the results of the article are available in the supplementary materials . Notation . e [ n ] m ∈ Rn is the canonical vector of Rn with [ e [ n ] m ] i = δmi . Moreover , e [ 2k ] ij = e [ 2k ] 2 ( i−1 ) +j . Similarly , E [ n ] ij ∈ Rn×n is the matrix with [ E [ n ] ij ] ab = δiaδjb . The notations A ⊗ B and A B 1In the supplementary material , we extend this setting to a much broader and more realistic scope , and justify in passing the relevance of a Gaussian mixture modelling to address multi-task learning with real data . for matrices or vectors A , B are respectively the Kronecker and Hadamard products . Dx is the diagonal matrix containing on its diagonal the elements of the vector x and Ai· is the i-th row of A . The notation Å is used when a centering operation is performed on the matrix or vector A. Uppercase calligraphic letters ( A , K , Γ , M , V , ... ) are used for deterministic small dimensional matrices . Finally , 1m and Im are respectively the vector of all one ’ s of dimension m and the identity matrix of dimension m×m . The index pair i , j generally refers to Class j in Task i . 2 RELATED WORKS . Let us first point out the difference between MTL and transfer learning : while MTL makes no distinction between tasks and aims to improve the performance of all tasks , transfer learning aims to maximize the performance of a target task with the help of all source tasks . Yet , both methods mostly sharing the same learning process , in this section , we mainly focus on the MTL literature , which is divided into parameter-based versus feature-based MTL . In the parameter-based MTL approach , the tasks are assumed to share some parameters ( e.g. , the hyperplanes best separating each class ) or their hyperparameters have a common prior distribution . Existing learning methods ( SVM , logistic regression , etc . ) can then be appropriately modified to incorporate these relatedness assumptions . In this context , ( Evgeniou & Pontil , 2004 ; Xu et al. , 2013 ; Parameswaran & Weinberger , 2010 ) respectively adapt the SVM , LS-SVM , and Large Margin Nearest Neighbor ( LMNN ) algorithms to the MTL paradigm . The present article borrows ideas from Evgeniou & Pontil ( 2004 ) ; Xu et al . ( 2013 ) . In the feature-based MTL approach , the tasks data are instead assumed to share a low-dimensional common representation . In this context , most of the works aim to determine a mapping of the ambient data space into a low-dimensional subspace ( through sparse coding , deep neural networks , principal component analysis , etc . ) in which the tasks have high similarity ( Argyriou et al. , 2007 ; Maurer et al. , 2013 ; Zhang et al. , 2016 ; Pan et al. , 2010 ) ; other works simply use a feature selection method by merely extracting a subset of the original feature space ( Obozinski et al. , 2006 ; Wang & Ye , 2015 ; Gong et al. , 2012 ) . We must insist that , in the present work , our ultimate objective is to study and improve “ data-generic ” MTL mechanisms under no structural assumption on the data ; this approach is quite unlike recent works exploiting convolutive techniques in deep neural nets or low dimensional feature-based methods to perform transfer or multi-task learning mostly for computer vision . From a theoretical standpoint though , few works have provided a proper understanding of the various MTL algorithms . To our knowledge , the only such results arise from elementary learning theory ( Rademacher complexity , VC dimension , covering number , stability ) and only provide loose performance bounds ( Baxter , 2000 ; Ben-David & Schuller , 2003 ; Baxter , 1997 ) . As such , the present work fills a long-standing gap in the MTL research .

Summary: This paper provides a theoretical analysis of the inner workings of multi-task learning methods, based on a random matrix analysis applied to Gaussian mixture data model. The analysis is based on MTL LS-SVM with data from a Gaussian mixture model, where the bias of MTL LS-SVM is shown and a simple method is proposed to correct it. Experiments are conducted on a synthetic dataset and image classification task, where superior performance is shown in addition to the theoretical guarantees.