Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a framework for learning multi-task convolutional neural networks. For each layer of the network, the proposed algorithm assigns a subset of the layer's channels to each of the tasks. This is in contrast to existing methods that assign whole layers to tasks. There are two key ideas here: (1) instead of searching in the space of binary assignments of layers to tasks, search in the continuous space of fractions of channels assigned to each layer, subject to some consistency constraints; this allows for using finite differences for gradient estimation which can be fed into a black-box optimization procedure; (2) the use of distillation to estimate the performance of a given assignment, rather than retraining many models. Experimentally, the proposed framework performs relatively well on the Visual Decathlon benchmark.

Overall, I do like the paper's key ideas. However, I am not convinced by the presentation of the paper and by the lack of multi-task learning baselines in the experiments. I will consider changing my score if the following comments are addressed.

- Testing your main premise: "we instead partition out individual feature channels within a layer. This offers a greater degree of control over both the computation required by each task and the sharing that takes place between tasks." 
Actually, based on the paper, it is not clear that assigning channels rather than layers brings about additional gains. To prove that hypothesis, you should carry out experiments comparing your method to existing multi-task learning methods that you surveyed in the paper.

- "share all" performance: this baseline seems to do quite well in Table 2, beating es in 5/10 w.r.t. the average accuracy. Why does this happen?

- Too many hacks hidden in the appendix and it is not clear what works why. Have these values been found by cross-validation? The reader should be able to understand how your method works *exactly*. Examples of these magical values:
-- Number of samples for the ES
-- Number of parameter directions
-- "Distillation training is done for a brief 3000 iterations with a batch size of 4 and a learning rate of 1 which is dropped by a factor of 10 at iteration 2000."

Clarification questions:
- I don't understand what "strategy" means here: "Fixed vs Learned Strategies: Is a uniform strategy applied across tasks or is a task-specific solution learned?"

- Section 4.2 needs to be rewritten to give a more structured exposition of the distillation process. Please add pseudocode if needed. The Figure is not very informative.

- MIP: Generally, this is a very important component of your method, yet it is lacking in detail and left to the appendix. Some questions:

-- how long does it take to solve? Which commercial solver do you use? Please provide more details.
-- constraint (8) is for which k? Is it for all k? Please fix. Also for (10), it should be for all i and j as well as k.
-- what is the "non-linear constraint" that (8-9) are linearizing? Please provide the original MIP formulation (without slack or linearization), and explain the final formulation accordingly.
-- MIP: are you adding slack variables to allow for slightly infeasible solutions in case no M can be derived from \tilde{P}?

Minor:
- "cut down average feature dramatically": unclear, rephrase as needed.
- Figure 6 is low-resolution.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a feature partitioning scheme for efficient multi-task neural architecture search.  The proposed scheme automatically determines the network capacity that should be shared across tasks and kept exclusive for each task.

Overall, this is a well written paper. 

The problem of neural architecture search is important and beneficial to deep learning community to be able to extract the best out of these methods. Multi-task learning is no exception. However, I am not sure why the problem of NAS is so different for multiple outs than a single output. Given multiple outputs, we need to either look at a weighted combination where weights could be provided by user to reflect the priority over the tasks, or individual outputs in a multi-objective approach such as Pareto optimality manner. I find the approach of this paper a heuristic.

Usually, due to negative transfer learning, too much sharing of the weights is detrimental for the performance of an individual task. I could not see this anywhere in the results. Especially, I missed any baselines where such problems were present, which would then be improved due to using this method. How would the results be if all tasks are given equal weight and NAS is performed with respect to their average output (of course normalised appropriately)?

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper studies multi-task learning (MTL) from the deep learning perspective where a number of layers are shared between tasks followed by specific heads for each task. One of the main challenges in this problem is to decide the best configuration among a large number of possible ones (e.g., the number of layers , number of neurons, when to stop the shared part of the network). In this paper, the authors fix the network architecture, and learn which filters (among the already learned ones) should be dedicated to (and hence fine-tuned for) a specific, and which ones should be shared between multiple tasks. 

Instead of deciding on other hyper-parameters such as the number of layers, the authors chose to study how to efficiently share the capacity of the network: to decide which filters should be used for which tasks, and which filters should be shared between tasks. 
Specifically, this is controlled by task specific binary vectors which get multiplied with feature activations for each task, hence blocking or allowing the signal to pass for a specific filter. In addition, they define a different set of binary vectors for the foreground and background passes. This allows simpler tasks to benefit from features learnt from more complicated tasks such as ImageNet classification while avoiding ‘catastrophic forgetting’ at the same time.

Moreover, the authors develop a simple yet elegant strategy to reduce their parameter search space (by using the matrix P which controls the percentage of filters used per task + the percentage of filters shared between each pair of tasks) and quickly evaluate the performance of each configuration (using distillation). The advantages of these approaches are well discussed and validated quantitatively.

The paper is well written and the approach itself appears to be sound and it led to improvement over independent task estimator.  However, I am mostly concerned about the experimental setting: there are no comparisons with any other MTL algorithm. 

The authors perform a search over the matrix P, which is similar to neural architecture search over the entire possible ways of sharing the capacity of a network. This could potentially lead to improvement beyond multi-task learning. Experimental comparison on this could be provided.
I think the paper will make a strong case if it is compared with existing deep MTL algorithms including [Misra et al: Cross-stitch networks for multi-task learning]. In addition, the network seems to share a similar spirit with [Mallya et al: PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning], in that they also share the capacity of the network between tasks, and hence a comparison here seems reasonable. 

Overall, I think this paper makes a borderline case.

Other comments: 
In the supplementary material, providing a detailed description of the algorithm (e.g., pseudo code and an accompanying discussion) that calculates the matrices M from P could help reproduce and build upon the experiments reported in the paper. I wonder if M is uniquely defined from M.
review_title: Official Blind Review #2

Meta-review: This paper considers how to create efficient architectures for multi-task neural networks. R1 recommends Weak Reject, identifying concerns about the clarity of writing, unsupported claims, and missing or unclear technical details. R2 recommends Weak Accept but calls this a "borderline" case, and has concerns about experiments and comparisons to baselines. R3 also has concerns about experiments and baselines, and feels the approach is somewhat ad hoc. The authors submitted a response that addressed some of these issues, but the authors chose to maintain their decisions. The AC feels the paper has merit but given these slightly negative to borderline reviews, we cannot recommend acceptance at this time. We hope the reviewer comments help the authors to prepare a revision for another venue.
Decision: Reject