Deciphering and Optimizing Multi-Task Learning: a Random Matrix Approach

Abstract: This article provides theoretical insights into the inner workings of multi-task and transfer learning methods, by studying the tractable least-square support vector machine multi-task learning (LS-SVM MTL) method, in the limit of large (p) and numerous (n) data. By a random matrix analysis applied to a Gaussian mixture data model, the performance of MTL LS-SVM is shown to converge, as n,p→∞, to a deterministic limit involving simple (small-dimensional) statistics of the data.
      
      We prove (i) that the standard MTL LS-SVM algorithm is in general strongly biased and may dramatically fail (to the point that individual single-task LS-SVMs may outperform the MTL approach, even for quite resembling tasks): our analysis provides a simple method to correct these biases, and that we reveal (ii) the sufficient statistics at play in the method, which can be efficiently estimated, even for quite small datasets. The latter result is exploited to automatically optimize the hyperparameters without resorting to any cross-validation procedure. 
      
      Experiments on popular datasets demonstrate that our improved MTL LS-SVM method is computationally-efficient and outperforms sometimes much more elaborate state-of-the-art multi-task and transfer learning techniques.

Reviews and author responses:

This paper provides a theoretical analysis of the inner workings of multi-task learning methods, based on a random matrix analysis applied to Gaussian mixture data model. The analysis is based on MTL LS-SVM with data from a Gaussian mixture model, where the bias of MTL LS-SVM is shown and a simple method is proposed to correct it. Experiments are conducted on a synthetic dataset and image classification task, where superior performance is shown in addition to the theoretical guarantees.

The main contribution of the paper is to introduce the random matrix theory to study the performance of MTL LS-SVM theoretically, which is novel and facilitates the understanding of MTL. The theoretical work seems valid. 

Concerns:
1)	The theoretical analysis is based on MTL LS-SVM with data from a Gaussian mixture model, which limits the generality of the work. Specifically, as the authors state in the paper, the quadratic optimization problem with linear constraints produces explicit solutions, which makes the analysis easier. It would be helpful if the authors provide some insights of generalizing the analysis to other settings. 

2)	In the experiments, it would be necessary to explain the choice of the baselines to justify the results of the comparison. Specifically, why didn’t the authors compare to some multi-task learning representatives? Also, only one dataset is employed for multi-class experiments, which is less than sufficient. 

3)	Is the “low computational cost” claimed in the conclusion more due to the quadratic optimization problem itself?

Minor comments:
Algorithm 1 is suggested to be included in the main content of the paper. 

After rebuttal:
The authors' response addressed some of my concerns and I'd like to adjust my rating to marginally above.


Summary:
The paper provides interesting theoretical insights in multi-task learning using common and specific parameters modeling framework and based on least-squares SVM. Especially, it is theoretically established that the standard MTL LS-SVM is biased. Thereon a method derived from the analysis is proposed to correct the bias and allows to achieve enhanced performances. Empirical evaluations highlight the effectiveness of the method.

Reasons for score: 
Overall, I vote for accepting. The theoretical analysis highlight the intrinsic  relation between task statistics/relatedness and the classification performances. The analysis helps to design an adequate MT models with improved performances. My major concern is about the clarity of the paper notations. Hopefully the authors can address my concern in the rebuttal period. 
 
Pros:
- The paper provides a asymptotical analysis of the decision function $g_i$ related to each task $i$ (learned using a linear MTL LS-SVM) by leveraging on random matrix theory and by assuming large scale $n$ and high dimension $p$ with limiting growth rate. The main result highlights the influence of the task data statistics and the MTL hyper-parameters on the decision function. Essentially the paper shows that the score provided by a task decision function $g_i(\mathbf{x})$ has a Gaussian distribution in the limit case, hence one can estimate its classification error. For me, the proposed derivation is of great interest in real applications. 
-  The derived statistical modeling of $g_i(\mathbf{x})$ allows to control the intercept of $g_i$ in order to minimize the classification error. The key to this error control is to appropriately assign the labels of each task samples according to the tasks relatedness and their data statistics which can be easily computed based on available training data. This leads to a practical and comprehensive MTL algorithm (that should be moved in the main paper). 
- Experimental evaluations on synthetic and classical MTL datasets illustrate that the proposed method systematically ranks in the top two methods out of 5 compared algorithms. This makes the provided analysis convincing.  

Cons: 
- The mathematical notations are dense and render the overall mathematical derivation hard to read.  It might be valuable to expose the main concepts of the paper starting from a two-tasks MTL problem and then generalize to an arbitrary number of tasks. 
-  It might be useful to report the standard deviation along with the average empirical accuracies (Table 1 for instance)
- The analyzed framework relies on a binary MT classification problems. How the presented results transfer to the multi-class classification setting?
- Does the analysis change if instead of the LS-SVM one uses a logistic regression as a model? Also how the proposed approach lifts to non-linear models? 
 

Other comments:
- Table 1 overpasses the page format. 

After rebuttal
- I read the response of the authors. The response addresses most of the concerns raised in the reviews.

** On the motivation of LS SVM and extension to other model **

We covered this indeed key aspect of our work in the main comments to all reviewers, to which you can refer.


** On the data assumptions **

Similarly, this aspect on the data assumptions (Gaussian in the main core then concentrated vectors in the supplementary material) is covered at length in the answer to all reviewers and has been appropriately updated in the new version of the article in a paragraph just after the assumption 1 in the beginning of page 5.


** Datasets used **

We agree with the reviewer that more simulations on real world data would improve the support of our theoretical findings. As an answer, we have performed additional experiments, provided in the supplementary material (Sections 4.3 and 4.4), on (i) the MNIST dataset, on (ii) a multi domain sentiment classification dataset in natural language. These experiments corroborate the claimed advantages brought by our proposed algorithm. Specifically, performance figures are reported in Section 4.3 for natural language data and in Section 4.4 on images (MNIST), while Section 4.5 additionally supports our important result on the absence of "negative transfer" brought by our method, here on both synthetic and MNIST data.
Moreover, since the article aims to propose an improved classification independent of the feature representation, it is fair to compare it to methods that use the same data features. As such, the algorithms compared in the table all systematically either use the same feature data (SURF features, VGG features, tf*idf,...). It would be unfair to compare these against "end to end" MTL learning methods including a (explicit or implicit) step of feature learning. This further justify the use of MMDT, CDLS, ILS and MTL LSVM as baselines and not works exploiting convolutive techniques in deep neural nets.


** On computational complexity aspects **

The low computational cost claimed in the conclusion relates to the fact that the LS-SVM method has a fully explicit (thus computationally cheap) solution, which can in passing be implemented in a sequential manner using a recursive least squares approach. The only additional cost incurred by our method when compared to the baseline LSSVM MTL is linked to the evaluation of anticipated score statistics, but those were shown to *scale  with the number of tasks k* and not the (large) number n and dimension p of the data (so the cost is more than negligible): since these statistics are used to fine-tune the method without resorting to any cross-validation, this is all the more cheaper than a full-fledged "optimized" baseline LS-SVM.

We thank the reviewers for showing keen interest in our ideas, and for their thorough and valuable comments. Common comments are addressed here. Specific ones are in a comment to each review. We also upload a new version of the paper and supp mat.

   Why base our MTL approach on LS-SVM (i.e., linear regression) rather than SVM or logistic regression? 
It is important to insist on the following key messages of the article:
- one of our main findings is the possibility to optimize, *in closed form*, the input labels 'y'. This dramatically improves the performances when compared to binary labels (in {+1,-1}). The closed-form solution though is only accessible with LS-SVM: with implicit optimization schemes such as SVM or logistic regression, the optimal 'y' is not explicit and may even not be solved via convex optimization. The expression of the optimal 'y' as presented in our article also has the advantage to bring key intuitions on the best functioning of MTL in this setting. Finally, as shown in our simulations (compared to other state-of-the-art methods), the gains incurred by optimizing 'y' largely outtake the possible losses incurred by the 'naive' choice of an LS-SVM approach; and tuning our algorithm is much easier and intuitive.
- in the same line, the asymptotic performances and *optimal thresholds* to be set for improved classification are also closed-form and have simple expressions under the LS-SVM framework. This ensures a full control on the performances and a better understanding of the proposed MTL approach.
- moreover, by a now well-established universality argument in large dimensional statistics, it has been shown in closely related works [Mai'19] that quadratic cost functions are asymptotically optimal (as the data dimension and number increase) and uniformly outperform alternative costs (such as SVM or logistic approaches), even in a classification setting (the proof was in fact obtained in the precise large dimensional setting which we consider here). This argument further motivates the choice of considering first and foremost the LS-SVM version of MTL.
An explanation has been appended in the updated version of the article to clarify this important aspect.

Why a restriction to Gaussian data modelling?

As stated in the article, Gaussian data modelling is assumed but is then generalized to *concentrated vector modelling* in the supplementary material. The key reasons for this choice are:
- simplicity of Gaussian models: the main results and notations being potentially heavy (already under the LS-SVM assumption), the Gaussian data model in the paper core lightens the concepts and allows for a better focus on the main messages of the article;  
- universality: this said, Gaussian modelling is  *sufficient* in the sense that the performance of many ML algorithms in large data have recently (and repeatedly) been shown to only depend on first and second order statistics of the (large) data, see e.g., [Seddik'20]. To cover this aspect, the supplementary material generalizes the results to the family of *concentrated random vectors* (which encompass Gaussian vectors as a special case): concentrated random vectors are an extremely appealing data model as they contain, as a particular case, all data generated by GANs (and thus very realistic image data).

Does the analysis change if instead of  LS-SVM one uses a logistic regression as a model? Also how the proposed approach lifts to non-linear models?

Using a random matrix approach, [Mai'19] investigates the problem of high dimensional classification within the general framework of empirical risk minimization including logistic regression, adaboost, SVM,... for a single task.
     The analysis in [Mai'19] is generalizable to the MTL setting, however at the cost of the readability, interpretability and improvements of the theoretical results. As already mentioned in the first comment, one of the main conclusions of [Mai'19] is that (least-square) cost functions are asymptotically optimal; this argument further motivates the choice of considering the LS-SVM.
     As for non-linear models, the work of [Louart'18] derives the asymptotic performance of a neural network with one hidden unit using Random Matrix Theory. The techniques used in this article can be generalized to MTL learning. This is left as future research. 
     
Some of the aspects mentioned in these comments are discussed now in the updated version of the article.
 
*** References ***
[Mai'19] High dimensional classification via empirical risk minimization: Improvements and optimality, Xiaoyi Mai, Zhenyu Liao, 2019- arXiv preprint arXiv:1905.13742, 2019
[Seddik'20] Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures, Mohamed El Amine Seddik, Cosme Louart, Mohamed Tamaazousti, Romain Couillet, ICML 2020
[Louart'18] A Random Matrix Approach to Neural Networks, C. Louart, Z. Liao, R. Couillet, The Annals of Applied Probability.

** On the motivation of LS SVM and extension to other model **

This is indeed a point of importance, raised by all reviewers and which we commented in our answer to all reviewers.


** On the choice of the hyperparameters lambda and gamma **

The article provides a high-level interpretation for the impact of the vector parameter gamma and the scalar parameter lambda, the effects of which are respectively to regularize LSSVM learning and to set the throttle between individual versus collective learning. These hyperparameters intervene deeply inside our theoretical formulas (in Theorem 1) but, as opposed to the label vector 'y', are not amenable to explicit optimization. Yet, as confirmed by supplementary experiments provided in the updated version of the article (see in particular Figure 3 in the supplementary material), the optimization of the input scores 'y' largely compensates for suboptimal choices in gamma,lambda. As such, an 'informed guess', based on our previous discussion on the effects of these parameters, is in general sufficient to reach highly performing MTL-LSSVM. This is quite unlike the conventional (non optimized) MTL-LSSVM with labels 'y' in {-1,+1}, which instead greatly suffers from suboptimal hyperparameters. While we believe it to be mostly unnecessary in practice, a further gradient descent operation (or local grid search) on the theoretical performance approximation, initialized at the informed guess values, could further improve the overall learning performance. This is now mentioned in the updated version of the article in a footnote at page 8.


** On the unbalanced case **

The asymptotic regime indeed requires $n_{ij}/n$ to be non-trivial. This encompasses the case where the $n_{ij}$'s differ, even possibly significantly. In fact, one important consequence of our results lies in their revealing a severe bias in the decision threshold influenced by data unbalancing; our results allow to properly set the threshold (usually taken to be 0) as a consequence. The case of *severely* unbalanced classes (where, say, one of the ratios $n_{ij}/n$ is very small) actually also enters our regime so long that '$n_{ij}$' and 'n' are both quite large; in practice though, it is difficult to decide what is considered as "quite large" or "quite small", and one would usually resort to simulation campaigns which, as far as our experiments tell, support the validity of our results even beyond our theoretical setting (see as example the threshold in the unbalanced setting of Figure 1 of the main article).

** On the clarity of the notations **

We agree with the reviewer that the notations may seem quite dense and that we failed to explain our "implicit" convention in choices of characters. We provided in the updated version (section "Notation") an additional paragraph precisely detailing our notational conventions for ease of read.

** On the extension to multi-class classification **

The literature [Bishop'06] describes broad groups of approaches for  dealing with classification of $m>2$ classes (one-versus-one, one-versus-all or one hot encoding). These naturally extend to multiple tasks. We chose to focus in this article on the most common method, namely one-versus-all, although our main results naturally adapt to other families of methods (we did study these with no striking difference in the results, which we did not consider worth presenting). With respect to multi-class classification, our main contributions may be summarized as follows:
- The "classical" (not improved) one-versus-all approach suffers a severe data unbalancing effect when using binary labels in 'y' (this is because the set of '-1' labels in each binary classification is on average m-1 times larger than the set of positive labels). The method also suffers a centering-scale problem when ultimately comparing the  outputs of the decision functions, whose average locations and ranges may differ significantly; these issues lead to undesirable effects, already reported in part in [Bishop'06, section 7.1.3]. Our multi-class MTL-LSSVM approach simultaneously addresses all these limitations (and this is theoretically proved): specifically, having access to the theoretical statistics of the classification scores allows us to appropriately center and scale these scores. These results were previously discussed in a remark of the supplementary material, which we decided to move to the core of the article (Remark 2) in the updated version.

- In our proposed approach, *each one of the m classifiers of the one-versus-all approach is optimized*, as the labels 'y' used for each binary classification are independently adapted and optimized. This is explained in detail in the article for the two-class case, and naturally extends to the multi-class setting.

Algorithm 1, which we have now moved to the core of the article, summarizes all these steps and contributions. Matlab and Python codes are now linked in the supplementary material and are ready to use for the readers.

** On the motivation of LS-SVM and the extension to other models (SVM, log. regression) **

See comments shared to all reviewers on this important aspect.

** Minor comments **

As requested by the reviewer, Algorithm 1 has been moved to the main article. 
In Table 1, we provide the mean accuracy obtained over 20 trials. The variance of the proposed method and the MTL LSSVM non optimized is provably of order O(1/p) while the accuracy is of order O(1).
They would disrupt the reading of the table without giving more insights. We thus chose to display only the mean accuracy for readability in the main article.

** References **

Pattern recognition and machine learning, Christopher M Bishop,  springer, 2006

** On the Gaussian Distribution **

This important aspect, raised by many reviewers, has been covered in the general answer to all. In a nutshell, our results are valid quite beyond the Gaussian case and support extremely realistic data models, as thoroughly exposed in the supplementary material.


** On the motivation of using Least Square SVM Model **

Similarly, most (if not all) reviewers pointed out this seemingly weak aspect of our work. This also is covered in detail in the answer to all reviewers. There too, the apparent 'weakness' is in fact a strength of our proposed method: being simple *but thoroughly optimized*, it is more than competitive over state-of-the-art alternatives while at the same time being much more intuitive and theoretically fully understood (down to the existence of accurate performance estimates available even before running the algorithm).


** Computational complexity of the proposed algorithm **

Being based on LS-SVM, the solution of which is explicit and easily obtained (if needed) by recursive least squares, the algorithm complexity is extremely low. Besides, as opposed to the standard LSSVM approach (with no optimized labels 'y', no optimized decision threshold) for which hyperparameters need be set by cross-validation (see our new Figure 3 on the importance of properly setting these hyperparameters), our proposed approach is capable of anticipating its own performances and its best parametrization by merely estimating k-dimensional statistics (k being the typical number of tasks) at a cost which does not scale with the data size 'p' and number 'n'. As a result, the additional computational cost incurred by these estimates not only is negligible, but also discards altogether the need of any cross-validation procedure; and this, with an optimized (often drastically improved) performance. Benefiting from the additionally allowed page, we included further discussions on these central aspects of the article in the updated version.

** Minor comments **

Algorithm 1 has been moved into the main article in the revised manuscript.
In the second paragraph, the sentence was meant to be :
A central issue [to i)...] consists in characterizing...

The paper considers the multitask least-square SVM problem. Such a problem consists of k SVM tasks, each being a binary classification problem. The normal vector of the separating hyperplane in each task is “close” to each other, reflecting the commonality of the tasks. For an input data point, the problem asks to predict the classification of the input data point for a given task. This problem has a standard optimization formulation.

The main contribution of the paper is a theoretical analysis for the setting where the training data and the test data are all Gaussian random vectors. It shows that under such setting, the classification score in each task converges, when the number of training data goes to infinity, in distribution to Gaussian variables whose mean and variance can be computed from some statistics of the data and the parameters of the optimization formulation. Therefore, one can use this limiting distribution to set the threshold for assigning the class, given the classification score, by minimizing the misclassification probability.

Strengths:
- Sophistical analysis, very theoretical
- Good experiment result

Weaknesses:
- Analysis may only be possible for special distributions (perhaps Gaussians is a relatively easy case) and it is not fully convincing that it can be widely used. SVM itself is a relatively simple problem, too.
- Experiment does not completely beat the best existing method. Is there an advantage in some other aspect, such as runtime?
- ICLR seems to mainly about representation learning while the problem this paper does not rightly concern processing the input data.

The main body of the paper is very well-written, although I think Algorithm 1 should be presented in the main body instead of being left in the supplementary material. 

Minor comment:
- Second paragraph of Section 1, add “is” to the end of the first line


Overall, I vote for accepting this paper. I list the strong and weak points in the following.
### Strong points
- The theoretical result is sound and significant. It not only matches the simulation well, but also provides a way to optimally choose the decision threshold and training labels. The proposed method is simple but beats some of the more complicated algorithms as demonstrated in the experiments. This is quite a success of applying random matrix theory to study machine learning. It is a stepping stone to utilizing random matrix theory to study more sophisticated problems and design more efficient algorithms.
- The paper is well written and organized. It clearly states the contributions and limitations and put itself in the literature appropriately.

### Weak points
- The studied problem is restricted. It is a binary classification problem under a Gaussian mixture model. The LS-SVM algorithm is also not as common as margin-based SVM. I would like at least some discussions on how to generalize the framework and techniques in this paper to other problems. 
- The asymptotic regime also require the number of samples in each task and in each class to be proportional to each other. What can be said about the unbalanced case?
- The paper does not talk about how to choose the hyperparameters $\lambda$ and $\gamma$. Can the theory provide a way of choosing the hyperparameters optimally?





Meta-review: The reviewers and I all agree that this analysis of multi-task and transfer learning from the random matrix perspective is novel and theoretically sound. While some reviewers expressed concern about the restriction to Gaussian mixtures, the strength of the explicit results undoubtedly justifies this assumption, and the generalization to concentrated random vectors significantly mitigates any concerns. I recommend acceptance.

Decision: spotlight-presentations