Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper works on the problem if training a set of networks to solve a set of tasks. The authors try to discover an optimal task split into the networks so that the test performances are maximized given a fixed testing resource budget. By default, this requires searching over the entire task combination space and is too slow. The authors propose two strategies for fast approximating the enumerative search. Experiments show their searched combinations give better performance in the fixed-budget testing setting than several alternatives.

+ This paper works on a new and interesting problem. Training more than one networks for a few tasks is definitely a valid idea in real applications and related to broad research fields.
+ The baseline setup is comprehensive. The difference between optimal, random, and worst clearly shows this problem worths effort for research.
- I believe this problem setup requires a larger task set. In the paper the authors manually picked 5 tasks. It seems straightforward for a human to manually group them together: segmentation and edge/ surface + depth/ keypoint for 3 networks. It is unclear how better a network can do than a human in one minute or should we expect learning the task split is better than manual design.
- Both technical contributions in Section 3.3 look straightforward. Given the good performance in Figure3, it is fine.
- I am confused by the comparison to Sener and Koltun. How do you change the inference budget for them? If it is changing the number of channels for a single network, I believe it can be improved more.

Overall I believe this is a good paper to open interesting research direction with solid baselines. I am happy to accept this paper and see more exciting future works in this direction.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper focuses on how to partition a bunch of tasks in several groups and then it use multi-task learning to improve the performance.  The paper makes an observation that multi-task relationships are not entirely correlated to transfer relationships and proposes a computational framework to optimize the assignment of tasks to network under a given computational budget constraint. It experiments on different combinations of the tasks and uses two heuristics to reduce the training overheads, early stopping approximation and higher order approximation. 

Please see the detained comments as follows:
1. The experiments are based on the assumptions that the network structures (how parameters are shared across tasks) are fixed. From my perspective, understanding how to optimize the parameters sharing across two tasks should be the first step to study how to optimally combine the training of tasks. Otherwise, different parameter sharing structures across tasks may lead to different conclusions.

2. It requires optimization of the article structure. E.g., algorithm 1 is important and should be in the main context.

3. It is also related to neural architecture search and it requires some discussions.

4. The paper is over-length. 

5. A lot of typos. 

Nits:
Page 1: vide versa -> vice versa

Page 3: two networsk -> two networks

Page 6: budge ?? 1.5

Page 7: overlap between lines (under figure 3)

Page 8: half segmented sentence in section 6.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This submission studies how to group tasks to train together to find a better runtime-accuracy trade-off with a single but large multi-task neural model. The authors perform an extensive empirical study in the Taskonomy dataset. I like Section 6.1 in particular, which shows the difference between multi-task learning and transfer learning. 

My main concern is that the competition between different tasks may stem from the limited capacity of the model during training. It might be possible that with enough parameters, the competing tasks become compatible. If I were the authors, I would first train a bigger model on multiple tasks and then distill it into a smaller one, which does not increase inference time. 

Furthermore, the paper has more than 8 pages. 
review_title: Official Blind Review #2

Meta-review: An approach to make multi-task learning is presented, based on the idea of assigning tasks through the concepts of cooperation and competition. 

The main idea is well-motivated and explained well. The experiments demonstrate that the method is promising. However, there are a few  concerns regarding fundamental aspects, such as: how are the decisions affected by the number of parameters? Could ad-hoc algorithms with human in the loop provide the same benefit, when the task-set is small? More importantly, identifying task groups for multi-task learning is an idea presented in prior work, e.g. [1,2,3]. This important body of prior work is not discussed at all in this paper.

[1] Han and Zhang. "Learning multi-level task groups in multi-task learning"
[2] Bonilla et al. "Multi-task Gaussian process prediction"
[3] Zhang and Yang. "A Survey on Multi-Task Learning"

Decision: Reject