Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: 1. This paper proposes a tree-structured multi-task model recommender. It takes as input an arbitrary CNN model and some predefined tasks, and outputs the top-k tree-structured multi-task architectures that achieve good performance within some computation budget limitation.

2. The whole process is in a white-box manner.

3. Extensive experiments validate the effectiveness of the proposed approach
overall_review: Please see the above section: **Technical Quality And Correctness**
review_summary: This paper proposes a tree-structured multi-task model recommender. It takes as input an arbitrary CNN model and some predefined tasks, and outputs the top-k tree-structured multi-task architectures that achieve good performance within some computation budget limitation in a white-box manner.

The proposed approach consists of three key components:
- Branching point detector: automatically detects branching points (computation blocks, such as Residual Block in ResNet50)
- Design space enumerator: lists all the multitask architecture
- Task accuracy estimator: predicts task performance of architecture w/o performing actual training.

Please check the above section **Technical Quality And Correctness** for negative points.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The authors work with Multi-task learning, which aims to solve multiple tasks simultaneously. The authors propose an approach that takes as inputs a backbone model and a set of tasks in interest and then predicts the top-k tree-structured multi-task architectures that achieve high task accuracy while meeting a user-specified computation budget. The authors propose to achieve it using three major components: a task accuracy estimator, design space enumerator, and branching point predictor.

Unlike previous approaches, the authors claim that their approach does branching automatically, without the need for domain expertise, and efficient computation, and can work under a given constraint.
The authors furthermore maintain their assertions from the experiments on various datasets.
overall_review: ## Strengths

* The paper proposes a novel approach for MTL using a tree-structured architecture candidate generation.
* The authors show the robustness of this approach across multiple datasets and settings.

## Weakness

* The authors propose a two-task accuracy estimator. However, it would be nice to share how the task interactions and interference would be if we did not consider this scenario. Why not more than 2? Is it for complexity reasons? Or does this setting achieve best performance.
* The models have only been run on one seed, and it would be very useful to see it run on multiple seeds, reporting both mean and var of performance, to show that their approach is not an artifact of luck, and is robust.
review_summary: Overall, I find the paper exciting and fun to read. The authors introduce concepts that help in the field of MTL and are very useful to the research community in general. Their end-to-end approach is efficient and does not require additional training, unlike many similar approaches. Furthermore, it returns a candidate set of architectures and makes the approach very useful for analyzing the neural network better.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: N/A for reproducibility reviewers
overall_review: N/A for reproducibility reviewers
review_summary: N/A for reproducibility reviewers
review_title: null

Meta-review: metareview: This paper proposes a recommender that, given a convolutional model and a set of tasks, proposes (top-k) tree-structured multi-task architectures that can achieve high accuracy (by figuring out where to branch so that the appropriate parts of the network are shared across tasks) while adhering to a computational budget. The reviewers found the proposed approach, experiments and theory sound and novel, the presentation clear, the empirical performance strong, and the contribution overall important for the research community. This is why I recommend acceptance. 

The reviewers though do point out some flaws that should be addressed as well as possible in the revised version. These relate to motivating components of the approach, clarity with regards to measuring computational budget, and the need to run experiments with additional random seeds.

An important thing to flag is that while the authors claim they will publish their code on Github as soon as possible, as it stands currently their code is not available.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept