Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper provides interesting theoretical insights in multi-task learning using common and specific parameters modeling framework and based on least-squares SVM. Especially, it is theoretically established that the standard MTL LS-SVM is biased. Thereon a method derived from the analysis is proposed to correct the bias and allows to achieve enhanced performances. Empirical evaluations highlight the effectiveness of the method.

Reasons for score: 
Overall, I vote for accepting. The theoretical analysis highlight the intrinsic  relation between task statistics/relatedness and the classification performances. The analysis helps to design an adequate MT models with improved performances. My major concern is about the clarity of the paper notations. Hopefully the authors can address my concern in the rebuttal period. 
 
Pros:
- The paper provides a asymptotical analysis of the decision function $g_i$ related to each task $i$ (learned using a linear MTL LS-SVM) by leveraging on random matrix theory and by assuming large scale $n$ and high dimension $p$ with limiting growth rate. The main result highlights the influence of the task data statistics and the MTL hyper-parameters on the decision function. Essentially the paper shows that the score provided by a task decision function $g_i(\mathbf{x})$ has a Gaussian distribution in the limit case, hence one can estimate its classification error. For me, the proposed derivation is of great interest in real applications. 
-  The derived statistical modeling of $g_i(\mathbf{x})$ allows to control the intercept of $g_i$ in order to minimize the classification error. The key to this error control is to appropriately assign the labels of each task samples according to the tasks relatedness and their data statistics which can be easily computed based on available training data. This leads to a practical and comprehensive MTL algorithm (that should be moved in the main paper). 
- Experimental evaluations on synthetic and classical MTL datasets illustrate that the proposed method systematically ranks in the top two methods out of 5 compared algorithms. This makes the provided analysis convincing.  

Cons: 
- The mathematical notations are dense and render the overall mathematical derivation hard to read.  It might be valuable to expose the main concepts of the paper starting from a two-tasks MTL problem and then generalize to an arbitrary number of tasks. 
-  It might be useful to report the standard deviation along with the average empirical accuracies (Table 1 for instance)
- The analyzed framework relies on a binary MT classification problems. How the presented results transfer to the multi-class classification setting?
- Does the analysis change if instead of the LS-SVM one uses a logistic regression as a model? Also how the proposed approach lifts to non-linear models? 
 

Other comments:
- Table 1 overpasses the page format. 

After rebuttal
- I read the response of the authors. The response addresses most of the concerns raised in the reviews.
review_title: The paper provides interesting theoretical insights in multi-task learning using common and specific parameters modeling framework and based on least-squares SVM. Especially, it is theoretically established that the standard MTL LS-SVM is biased. Thereon a method derived from the analysis is proposed to correct the bias and allows to achieve enhanced performances. Empirical evaluations highlight the effectiveness of the method.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper provides a theoretical analysis of the inner workings of multi-task learning methods, based on a random matrix analysis applied to Gaussian mixture data model. The analysis is based on MTL LS-SVM with data from a Gaussian mixture model, where the bias of MTL LS-SVM is shown and a simple method is proposed to correct it. Experiments are conducted on a synthetic dataset and image classification task, where superior performance is shown in addition to the theoretical guarantees.

The main contribution of the paper is to introduce the random matrix theory to study the performance of MTL LS-SVM theoretically, which is novel and facilitates the understanding of MTL. The theoretical work seems valid. 

Concerns:
1)	The theoretical analysis is based on MTL LS-SVM with data from a Gaussian mixture model, which limits the generality of the work. Specifically, as the authors state in the paper, the quadratic optimization problem with linear constraints produces explicit solutions, which makes the analysis easier. It would be helpful if the authors provide some insights of generalizing the analysis to other settings. 

2)	In the experiments, it would be necessary to explain the choice of the baselines to justify the results of the comparison. Specifically, why didn’t the authors compare to some multi-task learning representatives? Also, only one dataset is employed for multi-class experiments, which is less than sufficient. 

3)	Is the “low computational cost” claimed in the conclusion more due to the quadratic optimization problem itself?

Minor comments:
Algorithm 1 is suggested to be included in the main content of the paper. 

After rebuttal:
The authors' response addressed some of my concerns and I'd like to adjust my rating to marginally above.

review_title: The paper introduces the random matrix theory to study the performance of MTL LS-SVM theoretically, which is novel. Meanwhile, The theoretical analysis is based on MTL LS-SVM with data from a Gaussian mixture model, which limits the generality of the work. 

final_score: rating: 7: Good paper, accept
review_content: Overall, I vote for accepting this paper. I list the strong and weak points in the following.
### Strong points
- The theoretical result is sound and significant. It not only matches the simulation well, but also provides a way to optimally choose the decision threshold and training labels. The proposed method is simple but beats some of the more complicated algorithms as demonstrated in the experiments. This is quite a success of applying random matrix theory to study machine learning. It is a stepping stone to utilizing random matrix theory to study more sophisticated problems and design more efficient algorithms.
- The paper is well written and organized. It clearly states the contributions and limitations and put itself in the literature appropriately.

### Weak points
- The studied problem is restricted. It is a binary classification problem under a Gaussian mixture model. The LS-SVM algorithm is also not as common as margin-based SVM. I would like at least some discussions on how to generalize the framework and techniques in this paper to other problems. 
- The asymptotic regime also require the number of samples in each task and in each class to be proportional to each other. What can be said about the unbalanced case?
- The paper does not talk about how to choose the hyperparameters $\lambda$ and $\gamma$. Can the theory provide a way of choosing the hyperparameters optimally?




review_title: This paper studies the LS-SVM MTL method in multi-task learning in the large-p large-n setting. By utilizing mathematical tools in random matrix theory, it characterizes the asymptotic behavior of the classification score and thus proposes to choose the decision threshold and training labels in an optimal way. The theory matches the simulation results well, and the competency of the proposed method is demonstrated via synthetic and real data analysis.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper considers the multitask least-square SVM problem. Such a problem consists of k SVM tasks, each being a binary classification problem. The normal vector of the separating hyperplane in each task is “close” to each other, reflecting the commonality of the tasks. For an input data point, the problem asks to predict the classification of the input data point for a given task. This problem has a standard optimization formulation.

The main contribution of the paper is a theoretical analysis for the setting where the training data and the test data are all Gaussian random vectors. It shows that under such setting, the classification score in each task converges, when the number of training data goes to infinity, in distribution to Gaussian variables whose mean and variance can be computed from some statistics of the data and the parameters of the optimization formulation. Therefore, one can use this limiting distribution to set the threshold for assigning the class, given the classification score, by minimizing the misclassification probability.

Strengths:
- Sophistical analysis, very theoretical
- Good experiment result

Weaknesses:
- Analysis may only be possible for special distributions (perhaps Gaussians is a relatively easy case) and it is not fully convincing that it can be widely used. SVM itself is a relatively simple problem, too.
- Experiment does not completely beat the best existing method. Is there an advantage in some other aspect, such as runtime?
- ICLR seems to mainly about representation learning while the problem this paper does not rightly concern processing the input data.

The main body of the paper is very well-written, although I think Algorithm 1 should be presented in the main body instead of being left in the supplementary material. 

Minor comment:
- Second paragraph of Section 1, add “is” to the end of the first line

review_title: Very theoretical results

Meta-review: The reviewers and I all agree that this analysis of multi-task and transfer learning from the random matrix perspective is novel and theoretically sound. While some reviewers expressed concern about the restriction to Gaussian mixtures, the strength of the explicit results undoubtedly justifies this assumption, and the generalization to concentrated random vectors significantly mitigates any concerns. I recommend acceptance.
Decision: Accept (Spotlight)