Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a simple but effective matrix-interleaving method (mint) for multi-task learning, which aims to represent both joint training and independent training.
The model achieves good performance on several supervised and reinforced learning datasets. Though the model resembles FiLM(Perez et al., 2018), it outperforms FiLM by a larger margin in three dataset.

It would be better for authors to give more detailed comparisons with models that work on combining both joint training and independent training.

I would like to see the paper to be accepted for its simplicity and effectiveness.

Typos: chnages -> changes?
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes a matrix-interleaving (Mint) based on neural networks for multi-task learning. The Mint contains a share parameter matrix and a task-specific parameter matrix. 

Authors claim that the proposed Mint have the ability to represent both extremes: joint training and independent training. However, if the relations among tasks make the model between the both extreme cases, how is the performance of the proposed model?

What does "when the shared weight matrices are not learned" mean? Are the shared weight matrices randomly initialized and then fixed without updating?

Theorem 1 requires that each W^(l) is invertible, which implies that W^(l) is a square matrix. This requirement may not be satisfied in many neural networks. In this case, does Theorem 1 still hold? If not, Theorem 1 is not so useful.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper propose a single-network approach to multi-task learning by adding a task-specific linear transformation layer after each fully connected layer. The authors prove that the addition of such a layer keeps the expressive power of the network for each task. They also discuss how the linear transformation (parameterized by a transformation matrix and a bias vector) can be represented in a discrete manner in usual multi-task supervised learning and in a continuous manner (by two other neural networks) in goal-conditioned reinforcement learning. Experiments demonstrate the superiority of the proposed single-network approach.

The proposed approach is single and elegant. It is recommended to weak-reject the paper because of the following key reasons.

(1) Problem formulation is far from clear, perhaps because of the lack of clarity in writing. In particular, the super short Section 2 did not clearly illustrate what's being trained and what's being tested, and whether we care about the generalization performance of each task, or the generalization performance to new tasks (generated from P(T)). Some notations are confusing there---for instance, i seems to be indicating tasks, but then there is a z_k as task indicator. Even for the main proposed approach in Section 3.1, the notations are loosely used in nature. For instance, it is hard to understand what the authors mean by "train separate neural networks to output the Mint matrices and biases"---there is no information about the "training data" for learning those neural networks.

(2) Theoretical justification is at best shallow, or at least in the context that the authors have put it. While having an universal expressive power is good, it is easily achieved by adding an indicator variable (z_k) per layer (similar to task-specific-all-fc in the experiments). So the guarantee does not seem to be closely related to explaining the proposed approach (though the guarantee is nice to have). The authors contrast the guarantee with what FiLM (a competitor approach) can do, but in the experiments FiLM is not taken as a competitor in multi-task supervised learning, leaving a big gap between theory and practice. In the flow presented by the authors, it is strongly suggested to introduce FiLM in more detail and compare it with the proposed approach more clearly in design, theory and experiments.

(3) It is hard to understand whether the experiments are reasonably designed. In particular, the two settings take different sets of competitors, and there is little information on why those competitors are selected, whether they represent state-of-the-art, etc.. The authors highlight that the proposed approach uses much fewer parameters but other than that it is hard to infer why the proposed approach is better. Is it better because there is more overfitting for the competitor's approaches given more parameters? Is it better because it is easier to tune? The task-specific-all-fc (which is of similar # parameters to the proposed approach) result particularly looks suspicious to me but there is no other information to double-check on why the proposed approach is better. In particular, I believe the authors have *not* answered their first proposed question "does our method enable effective multi-task learning both in settings where there is substantial overlap in tasks and where there is little overlap" properly---their best evidence may have been MT10 and MT50 experiments, but even in those experiments, I am not sure whether the authors want to take the results as suggesting there are "substantial overlap" or "little overlap." 

Some other suggestions:

(4) It is suggested to analyze the matrices learned by the proposed approach. Do the matrices contain reasonable task correlations (i.e. for two similar tasks, are the matrices somewhat similar) to understand more about the proposed approach.

(5) It looks a bit strange to me that there is no discussion on regularizing the linear transformation matrices, as it seems possible to embed the task relations through the regularization. Have the authors considered the possibility?

(6) The authors are overly-emphasizing what they want to do (interpolating between independent networks and shared network). This occupies multiple redundant paragraphs in the early sections. It is better to remove some of those and use the space for more solid results, such as clarifying the notations.

(7) One baseline that could have been considered is to just train a fully-shared network (without z_k), and a fully-independent one. Then, use validation to select the better network and compare with the proposed approach.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper introduces a factorization method for learning to share effectively in deep multitask learning (DMTL). The approach has some very satisfying properties: it forces all sharable structure to be used by all tasks; it theoretically captures the extremes of total sharing and total task independence; it is easy to implement, so would be a very useful baseline for future methods; and it is able to effectively exploit task similarities in experiments, and outperforms some alternative DMTL methods.

I have two main concerns with the work: (1) It is most closely related to MTL factorization methods, but does not discuss this literature, or provide these experimental comparisons; (2) the interpretation of why Mint works is not clear: it is not clear that the universality is what makes it work, and there are no experimental analyses of what Mint learns. 

W.r.t. (1), there are several DMTL approaches that factorize layers across shared and task-specific components, e.g., [1], [2]. Such approaches are extensions of factorization approaches in the linear setting, e.g., [3], [4]. Compared to previous DMTL approaches, Mint is more closely related to these linear methods, as it takes the idea of factorizing each model matrix into two components and applies it to every applicable layer. In particular, the formal definition (i.e., without nonlinear activation between M and W) of Mint appears to be a special case of the more general factorizations in [1]; an experimental  comparison [1] would make the conclusions more convincing, e.g., that universality is important.

However, in the Mint experiments, a non-linear activation is added between the two components of each layer. This could void the universality property. Is there some reason why this is not an issue in practice? 

More generally, it is not clear that universality is the important advantage of Mint. Some existing DMTL methods already have this property, including Cross-stitch, which is compared to in the paper. The intriguing difference with Mint is that shared and unshared structure are applied sequentially instead of in parallel. Could there be an advantage in in this difference? E.g., is Mint a stronger regularizer because it forces all tasks to use all shared layers (learning the identity function for shared layers is hard), while something like cross-stitch could more easily degenerate to only use task-specific layers even when tasks are related?

Beyond performance, analysis on what Mint actually learns would be clarifying. Can the sharing behavior be analyzed by looking at the trained Mint layers? Is Mint actually able to learn both of the extreme settings in practice? The non-synthetic experiments in the paper are only performed on tasks that are closely related. 

As a final note, adding layers to non-Mint models to make the topologically more similar to Mint models may not help these other models. It may make them more difficult to train or overfit more easily, since they are deeper, but do not have Mint method to assist in training. Comparisons without these extra layers would make the experiments more complete. Do cross-stitch and WPL share the conv layers across all tasks like Mint in Table 1? They should to make it a clear comparison.

Other questions:
-	What exactly are the “two simple neural networks” that produce the goal-specific parameters for goal-conditioned RL? Do these maintain the universality property?
-	Can Mint be readily extended to layer types beyond FC layers? This may be necessary when applying to more complex models.


[1] Yang, Y. & Hospedales, T. M. “Deep Multi-task Representation Learning: A Tensor Factorisation Approach,” ICLR 2017.
[2] Long, M., Cao, Z., Wang, J., & Philip, S. Y. “Learning multiple tasks with multilinear relationship networks”, NIPS 2017.
[3] Argyriou, A., Evgeniou, T., & Pontil, M. “Multi-task feature learning,” NIPS, 2007.
[4] Kang, Z., Grauman, K., & Sha, F. “Learning with Whom to Share in Multi-task Feature Learning,” ICML 2011.

review_title: Official Blind Review #3

Meta-review: Reviewers put this paper in the lower half and question the theoretical motivation and the experimental design. On the other hand, this seems like an alternative general framework for solving large-scale multi-task learning problems. In the future, I would encourage the authors to evaluate on multi-task benchmarks such as SuperGLUE, decaNLP and C4. Note: It seems there's more similarities with Ruder et al. (2019) [0] than the paper suggests. 

[0] https://arxiv.org/abs/1705.08142
Decision: Reject