Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper provides a mechanism of building multi-task shared layer model, by computing Representation similarity between different layers which in turn computes the task affinity. Tasks that have a higher similarity have more shared layers, while tasks with lesser similarity are branched earlier.

Pros:
1. The question in consideration is very important
2. Yes, it is true that there are no enough studies that has studied this problem in a principled way.
3. The results are shown in some very recent datasets including Taskonomy

Cons:
1. There is not much novelty in this work. The important/ key aspect of this paper is the RSA (Representation Similarity Analysis) or the RDM (Representation Dissimilarity Matrices). This is already proposed. The rest of the paper is mere brute force. They compute the RSA between two tasks, at multiple predetermined layers of a model and whichever layer shows a lesser RSA, they start branching there. This is mostly brute force and part heuristics

2. One of the key challenge in the literature was that NAS/. Zamir et al.(2018) was computationally expensive. Very surprising that the authors did not spend a good analysis over the computation time of the proposed approach. As far I can understand, the computation of the proposed approach is going to be really expensive.
a. Train a task individual DL model on each of the task
b. For every pair of tasks, both the pretrained individual models, at multiple pre-determined layers, we have to compute the RSA
c. Once the big RSA matrix is computed, then we need to compute the correlation. 
d. Thus, compute, at which point to share between these two tasks
e. Repeat this for every pair of tasks.

This is really computationally expensive, and brute force - depending on the sampling rate of the layers

3. I really dont like the comparison made in this paper with NAS - NAS really is a search optimization problem, finding new architectures. However, the technique proposed in this paper is more of a brute force comparison at pre-determined layers to compute which is the better of determined subset of n layers. This is not like a NAS problem. 

4. There is no generalization in this paper- we cannot say that for ResNet always share at this layer. For every pair of tasks, and for every given model, we need to train the individual model and find the correlation for every pair of layers and perform the whole computation again. There is no takeaway for me a researcher from this paper.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents a method to infer multi-task networks architecture, more specifically to determine which part of the network should be shared among different tasks. The main idea is to first train a specific, standalone encoder/decoder network for each task. Subsequently, a task affinity factor is computed by looking at the similarity (or, more likely the dissimilarity) of an holdout set of images feature representations. Knowing these dissimilarities (based on RDM), one can cluster the tasks and make similar tasks share more of their layers. Computational budget can also be taken into consideration in the form of the number of parameters. Results on Cityscapes, Taskonomy, and CelebA datasets show, to some extent, improvements against the state of the art.

The paper is well written and addresses a common problem in multi-task learning. The experiments provided are extensive and cover most of the current multi-task learning methods and interesting problems. I especially like the idea of formalizing the dissimilarity between tasks using RDM. There are, however, a few key points that would need improvement.

First, except for CelebA, the experiments provided use ResNet50 with only 4 different "anchor point" in the network. In other words, the task at hand is limited to selecting the best sharing point between 4 choices. This is not wrong per se, but in my opinion, it does not tackle the main problem: what to do when brute force / exhaustive exploration cannot be fulfilled? CelebA provides a more complex case, but it also requires to change the method from an exhaustive search to a beam search (end of Sec. 3.2). Doing so get us back to a kind of greedy approach, precisely what was advocated against the paragraph before (in the discussion about Lu et al. 2017).

Second, the fact that task affinities are computed a priori leads to the following conclusion: "this allows us to determine the task clustering offline". While I agree that this could be useful, one has to keep in mind that compared to other methods, this one has to first train a network for _each_ task independently, which can take a long time.

Out of curiosity, did you consider other correlation coefficients than Spearman? Why use a rank-based method?

Overall, this is a nice paper, with adequate methodology. It is not groundbreaking, and most of the good results arise when we consider both performance and number of parameters, but it is interesting for the community nonetheless. I am not sure the impact would be very high, since it can probably be replaced by an architecture exhaustive search if the number of tasks and branching points in the network are low, but the formalism of the approach is welcomed.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a novel soft parameter sharing Multi-task Learning framework based on a tree-like structure. The idea is interesting. 

However, the technique details, the experimental results and the analysis are not as attractive as the idea. The proposed method is a simple combination of existing works without any creative improvement. Furthermore, comparing with the MTL baseline, the experimental performance of proposed method does not get obvious improvement while the computation cost increasing significantly.  Besides, there is not enough analysis about the idea this paper proposed. The intuition that more similar tasks share more parameters probably cannot always ensure the improvement of MTL.

review_title: Official Blind Review #3

Meta-review: The authors present an approach to multi-task learning. Reviews are mixed. The main worries seem to be computational feasibility and lack of comparison with existing work. Clearly, one advantage to Cross-stitch networks over the proposed approach is that their approach learns sharing parameters in an end-to-end fashion and scales more efficiently to more tasks. Note: The authors mention SluiceNets in their discussion, but I think it would be appropriate to directly compare against this architecture - or DARTS [https://arxiv.org/abs/1806.09055], maybe - since the offline RSA computations only seem worth it if better than *anything* you can do end-to-end. I would encourage the authors to map out this space and situate their proposed method properly in the landscape of existing work. I also think it would be interesting to think of their approach as an ensemble learning approach and look at work in this space on using correlations between representations to learn what and how to combine. Finally, some work has suggested that benefits from MTL are a result of easier optimization, e.g., [3]; if that is true, will you not potentially miss out on good task combinations with your approach?

Other related work: 
[0] https://www.aclweb.org/anthology/C18-1175/
[1] https://www.aclweb.org/anthology/P19-1299/
[2] https://www.aclweb.org/anthology/N19-1355.pdf - a somewhat similar two-stage approach
[3] https://www.aclweb.org/anthology/E17-2026/
Decision: Reject