Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper analyzed the principles for a successful transfer in the hard-parameter sharing multitask learning model. They analyzed three key factors of multi-task learning on linear model and relu linear model: model capacity (output dimension after common transformation), task covariance (similarity between tasks) and optimization strategy (influence of re-weighting algorithm), with theoretical guarantees. Finally they evaluated their assumptions on the state-of-the-art multi-task framework (e.g GLUE,CheXNet), showing the benefits of the proposed algorithm.

Main comments:

This paper is highly interesting and strong. The author systematically analyzed the factors to ensure a good multi-task learning. The discovering is coherent with with previous works, and it also brings new theoretical insights (e.g. sufficient conditions to induce a positive transfer in Theorem 2). The proof is non-trivial and seems technically sound.

Moreover, they validated their theoretical assumptions on the large scale and diverse datasets (e.g NLP tasks, medical tasks) with state-of-the-art baselines, which verified the correctness of the theory and indicated strong practical implications.

Minor comments:
The main message of the paper is clear but some parts still confuse me:
1. I suggest the author to merge the Figure 3 and Data generation (Page 4) part for a better presentation. e,g which “diff.covariance” is task 3 or 4 ?  And why we use different rotation matrix Q_i ? 

2. In algorithm 1 (Page 5) , I suggest the author use a formal equation (like algorithm 2) instead of descriptive words.
     -- Step 2, I have trouble in understading this step.
     -- Step 3, how to jointly minimize R_1,\dots, R_k, A_1, \dots, A_k ? we use loss (3)  or other losses ?
     -- I suggest that the author release the code for a better understanding.

3. For theorem 2, can we find some “optimal” c to optimize the right part ? Since 6c + \frac{1}{1-3c}\frac{\epsilon}{\X_2\theta_2} might be further optimized 

4. In section 3.3. (Figure 6)  of the real neural network, the model capacity is the dimension of Z or simply the dimension before last fc-layer ?

5. Some parts in the appendix can be better illustrated:
    (a) I am not clear how proposition 4 can derive proposition 1.
    (b) Page 15, proving fact 8: last line \frac{1}{k^4}sin(a^{prime},b^{prime}) should be \frac{1}{k^4}sin^{2}(a^{prime},b^{prime}). 


Overall I think it is a good work with interesting discoverings for the multi-task learning. I think it will potentially inspire the community to have more thoughts about the transfer learning. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The submission investigates multitask learning (MTL) and develops new theories around MTL with linear models and linear+ReLU. In the experimental section, the authors improve performance in sentiment analysis on subtasks of the GLUE benchmark (building on BERT - highly non-linear neural network) and show a SVD-based task loss reweighting scheme on an multi-label image classification dataset. 

The submission is overall well written though some paragraphs (2.1-2.3, in particular the example section) would benefit from additional effort towards clearer sentences. One issue with the submission is that there is a significant gap between the theory and experimental sections as theory only covers linear models and the experiments don’t include linear models and purely focus on deep networks. The benefits of a bottleneck in multitask learning are well known (based empirical results). However, it is helpful that the additional theoretical results (given strong assumptions) provide some grounding. 
While the model with non-linear activation is mentioned at places, nearly all theorems rely on the linear model instead such that it might make sense to either work towards generalising the theorems or emphasising that most only apply to linear models.

Additional assumptions (1D labels, same input dimensionality across all tasks) should be emphasised to clarify limitations of all derivations. Where previous work addressed model similarity it often looks at models in the context of existing datasets (i.e. taking the data into account to describe boundaries etc) such that the emphasised novelty at looking at data similarity is to be taken with a grain of salt.

Overall, the paper contributes to the conversation around multitask learning but would benefit from comparing again external work on multitask learning (e.g. see under minor) and from bridging between theory and experiments (e.g. experiments with the models described in the theory section - linear/ReLU).

Minor:
- y is used as label and as data terminology at different parts of the text. 
- the model in the first set of experiments has lower capacity than most models individually, suggesting that the capacity should be smaller even for individual tasks to prevent overfitting.
- An ablation over model capacities is mentioned but missing for 3.3
- comparison against existing multitask loss weighting techniques should be performed [1]


[1] Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
Alex Kendall, Yarin Gal, Roberto Cipolla 2017

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper studies how to improve the multi-task learning from both theoretical and experimental viewpoints. More specifically, they study an architecture where there is a shared model for all of the tasks and a separate module specific to each task. They show that data similarity of the tasks, measured by task covariance is an important element for the tasks to be constructive or destructive. They theoretically find a sufficient condition that guarantee one task can transfer positively to the other; i.e. a lower bound of the number of data points that one task has to have. Consequently, they propose an algorithm which is basically applying a covariance alignment method to the input. 
The paper is well-written, and easy to follow. 
Pros:
A new theoretical analysis for multi-task learning, which can give insight of how to improve it through data selection.
They empirically show that their algorithm improves the multi-task learning on average by 2.35%. 

Cons:
There is not much of novelty in the algorithm and architecture. Their method is very similar to domain adaptation but for multi-learning setting.
In the Theorem 2, they have assumed parameter c <= 1/3. They have not provided any insight of how much restrictive this assumption is.  


review_title: Official Blind Review #3

Meta-review: Many existing approaches in multi-task learning rely on intuitions about how to transfer information. This paper, instead, tries to answer what does "information transfer" even mean in this context. Such ideas have already been presented in the past, but the approach taken here is novel, rigorous and well-explained.

The reviewers agreed that this is a good paper, although they wished to see the analysis conducted using more practical models. 

For the camera ready version it would help to make the paper look less dense.

Decision: Accept (Poster)