Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a framework for automatically designing and training multi-task models. Given a set of tasks and a particular backbone network, the framework learns when to share layers for the best performance. For each task, the option is given to either use a shared version of a backbone operation, use a task-specific version, or skip the operation altogether. The network is trained with Gumbel-Softmax to learn in a differentiable way which of the three choices to make for every operation. At the end, discrete decisions are assigned and the final network design is trained from scratch. This approach performs well against other multi-task methods on the benchmarks CityScapes, NYUv2, and Taskonomy.

strengths_and_weaknesses: Strengths:

- The authors propose a straightforward and well-motivated search space for multi-task architectures, and show how to produce assignments in this space for an architecture that performs well against other multi-task methods while not requiring much additional parameter overhead.
- The approach can easily be adopted with new models, and the authors seem to have put a lot of emphasis on coding up a framework that’s easy to use

Weaknesses:

There are a couple other ablations that would have been nice to see to round out the experiments and get a clearer impression of the benefits of the proposed approach:

- I’m curious about the performance of a baseline like [i] trained with Gumbel-Softmax instead of RL. In contrast to this paper with 1 shared op + independent task-specific ops, the routing can instead be learned through some number of available shared ops.
- Is the three-stage training process necessary? How important is the initialization with the pre-trained model before beginning to learn the policy? Does the final model need to be trained from scratch or can the discrete subset of layers be sampled from the “policy-train” stage and finetuned from there?
- The regularization to encourage parameter sharing makes sense and seems to help from the ablations, I wonder about the weighting to encourage sharing particularly in the initial layers, how much does that help?

I don’t necessarily expect that all of those experiments would have been done or need to be done, just that the paper would be stronger with a few extra insights into the final design.

[i] Rosenbaum, Clemens, Tim Klinger, and Matthew Riemer. "Routing networks: Adaptive selection of non-linear functions for multi-task learning." ICLR 2018

Overall:

The proposed method is a nice contribution in the space of multi-task architecture search, the approach itself is not some huge, novel step away from existing methods, but it is well thought out and well executed with a clear paper and solid results.
questions: - Not sure a definition of "operator" is ever provided? Is each parameterized operation (conv layer, normalization function) treated separately?
- Might be nice to see absolute numbers for # params in addition to relative percent drop in Tables 1-3
limitations: limitation + social impact discussion is included in the paper
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The AutoMTL system integrates the Multi-Task Supermodel Compiler (MTS-Compiler), which transforms a user-provided backbone CNN into a multi-task supermodel that encodes the architecture search space. The authors propose a Stretchable Architecture Search Space that offers flexibility in deriving multi-task models with a wide range of model capacity based on task difficulties and interference, and further propose a novel data structure called Virtual Computation Node to encode the search space and enable compiler-based multi-task supermodel transformation. They propose a policy regularization term on the sharing policy to promote parameter sharing for high memory efficiency.
strengths_and_weaknesses: Strengths:
1. The paper proposes a  resource-efficient architecture design, which determines what parameters of a backbone model to share across tasks to optimize for both resource efficiency and task accuracy.
2. AutoMTL takes as inputs an arbitrary backbone CNN and a set of tasks to learn, and then produces a multi-task model that achieves high task accuracy and small memory footprint.
3. This method considers parameter sharing in MTL at the operator level. The operator-level sharing granularity not only enables the automatic support of arbitrary CNN backbone architectures, but also leads to a stretchable architecture search space that contains multi-task models with a wide range of model capacity.

Weaknesses:
1. Even though AutoMTL achieves better results, the number of model parameters is larger than Multi-Task baseline.
2. The inference time of AutoMTL in CityScapes is longer than Multi-Task and AdaShare.
questions: How long does it take to train the model? It seems that the training pipeline is complicated. It would be better to show the comparison of training time among baselines.
limitations: The method is interesting and the model can be adapted to other backbone without implementation cost. However, it seems that the training time and inference time is longer than the baseline. The method cannot achieve best on all evaluation metrics.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper focuses automatically learning to produce efficient neural network architectures for jointly performing multiple tasks given an arbitrary backbone and a set of tasks. The proposed method can efficiently generate efficient network for multi-task learning and achieve good performance on three multi-task learning benchmarks. 
strengths_and_weaknesses: Strengths:
1. The paper develops an efficient programming framework that can automatically learn to produce efficient neural network architectures for jointly performing multiple tasks given an arbitrary backbone and a set of tasks. 

2. It is shown that the method performs better than related multi-task architecture learning approaches in most metrics in three datasets. 

3. And the authors also show that the proposed framework is easier than related literatures e.g. Adashare, for producing multi-task learning network

Weaknesses:

1. The main weaknesses of the paper is its limited novelty as the idea is quite similar to Adashare and its high computational cost for searching the multi-task learning architectures. (Detailed in Questions section.)

2. Another limitation is the effect of the optimization in multi-task learning is not considered while the optimization in multi-task learning can greatly influence the performance of multi-task learning (See Question section for more detailed).

3. The performance gains are not significant while the method requires more parameters than Adashare. (See Questions section for more detailed.)
questions: 1. The idea of learning multi-task architecture is very similar to Adashare while the proposed method has one more operation in each layer (task-specific) such that in a layer each task can keep the layer task-specific (Adashare only have skip or not which make the layer shared across tasks or shared by a subset of tasks or skipped by all tasks). It enhance the power of learning more powerful representations however, it has a larger action space to learn and it can be computationally costly. In this point of view, the novelty of this work is limited.

2. One of the challenge in multi-task learning is the optimization of the multi-task learning network, which is not discussed in this paper and need to be discussed. The optimization of multi-task learning strategies, such as loss weighting, knowledge distillation strategies and gradient-based methods can greatly affects the performance of multi-task learning. Is the optimization of multi-task learning also affects the the network architecture learning and the training of the final network?

3. As mentioned above, the optimization can greatly affects the performance of multi-task learning and it may differs for different architectures. Do the authors tune the hyper-parameters lambda (losses weight) for the proposed method and apply them to other methods? It can be more clear that if all network learned with uniform loss weights or also tune the loss weights for different architectures.

4. The proposed method has more parameters (around +20%) than Adashre while the gain is not significant. The performance gains on NYUv2 dataset and Taskonomy are not significant (Table 6 and Table 7 in appendix). It also looks strange that MTAN uses much more parameters than the vanilla multi-task learning baseline in all datasets and single-task learning in NYUv2. The authors are suggested to check again this.

5. The when applying the proposed method using MobileNetV2 and MNasNet, only Vanilla Multi-task learning baseline and single-task learning baseline are compared, how well are other methods?
limitations: The limitation of methods and potential negative societal impact are discussed in the paper.
review_title: null

Meta-review: metareview: All reviewers are positive about the paper. The AC concurs. 
confidence: Certain
Decision: Accept