Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper brings into question the efficacy of multi-task optimization methods, finding that when adequately tuning the baselines, MTO methods yield solutions similar to optimizing a weighted average of each tasks loss. They show that sensitivity to training hyper parameters lead to an often unfair comparison across methods and drastically different results on the same benchmarks, and provide solutions to standardize the evaluation of these methods.
strengths_and_weaknesses: The issue of insufficient hyperparameter tuning leading to the illusion of performance gains is incredibly important and often overlooked, and this paper does an excellent job of highlighting this fact. The claims of the paper were clearly laid out and the evaluation of sensitivity to hyperparameters was thorough.  

However, I have two main concerns. The first is the organization of Section 4. I don't fully understand what the experimental setup is, the methods evaluated, and the results laid out specifically for the neural machine translation were not easy to follow. I realize that there is limited space so many experimental details can't be put in the main paper, but more summarization of the task setup and evaluation metrics would clear things up a lot. Also more of a suggestion than critique, but I wonder if employing a hyperparameter stability evaluation similar to [5] would be useful in illustrating these methods sensitivity to hyperparameters. 

Second, the issue of hyper parameters not being swept correctly when evaluating methods has been heavily explored in the past few years [1-4], and the issue of reproducibility of results is also a widely accepted issue in the field. Ultimately I believe this paper provides novel contributions in its evaluation and specific focus on MTO methods, but these previous works should be mentioned in the related works. 

With all of this in mind I believe that the ideas in this paper are of interest to the NeurlIPS community, although the presentation of the results needs clarification and the related works mentioned should be added. Overall a very interesting read :D

[1] Su et al. "A Comparison of Strategies for Source-Free Domain Adaptation"

[2] Musgrave et al. "Unsupervised Domain Adaptation: A Reality Check"

[3] Gulrajani et al. "In Search of Lost Domain Generalization"

[4] Greff et al. "LSTM: A Search Space Odyssey"

[5] Xiao et al. "Early Convolutions Help Transformers See Better"
questions: In 4.1 the authors mention that for the English to {French, Chinese} task they "anticipate a large degree of interference among the tasks"; what exactly does this mean? 

In 4.1, I did not understand this baseline: "The blue dashed line corresponds to the Pareto front achieved via proportional sampling with English→French sampling rate ranging from 10% to 90%."Why do you only sample English to French if the task is English to French and Chinese? Am I misunderstanding the evaluation?

Is RLW an MTO algorithm or the baseline of taking the weighted average of the losses?

What dataset is used in 4.1?

Is test cross-entropy the usual evaluation metric used for NMT? What about things like BLEU score?
limitations: n/a
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper performs large-scale experiments to show the recent multi-task optimization methods do not improve the performance over the simple static scalarization of different tasks. The paper also shows that the performance of multi-task learning is sensitive to the chosen hyperparameters. And suboptimal hyperparameters sometimes make some MTO method stand out from others while this trend does not hold after using better hyperparameters They also suggest the potential solution as using the common task framework to make a fair comparison for multi-task learning in the future work.
strengths_and_weaknesses: Strengths:

1. This paper summarizes the previous SOTA MTO methods and make a fair comparison between them and simple scalarization method on NLP translation task.

2. The comparison of SoTA MTO methods and simple scalarization is valuable since it shows with optimal scalarization, the static weighting of different tasks give competitive (or even better) performance than complex MTO methods.

3. Also the analysis of variation of MTL performance with different hyperparameters is very interesting and insightful. 

Weaknesses:

1. Even though the optimal scalarization gives as good performance as other MTO methods, authors seem does not count in the computation cost to find the optimal weighting for different tasks, especially when there are more than 2 tasks.

2. Since the hyperparameter is very important to the MTL performance, it is unclear whether the same group of hyperparameters is optimal for all different loss weights among tasks. 
questions: 1. What is the computation cost of search a good scalarization among tasks? Is there any method to speed up the searching for the optimal loss weights when there are more than 2 tasks.

2. For the scalarization method, does the same group of hyperparameters works optimally for different scalarization of tasks? 
limitations: Yes, they have addressed
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Here, the authors present an extensive empirical evaluation of multi-task optimization (MTO) methods across Multilingual Machine Translation, Multi-MNIST and CelebA datasets. The results showed that the MTO methods tested did not offer performance improvements over scalarization as a method of optimizing MTL models. The authors also provided a comprehensive evaluation of the impact of hyperparameters. They pointed out that this and the different evaluation strategies employed by previous papers might lead to optimistic results in favor of MTO methods.
strengths_and_weaknesses: 1) This work provides a comprehensive benchmarking of MTO methods using language and vision datasets. In light of the recent trend of optimization-based approaches to multi-task learning, this evaluation is of importance in the field of multi-task learning. 

2) The authors presented a discussion and analysis of why previous MTO methods present optimistic results, likely due to poor fine-tuning and evaluation procedures.

3)  MTO methods might present a substantial computational overhead. The authors provided a careful analysis suggesting that time/investment in simpler and well-tuned baselines might be beneficial in the end.

questions: Could the authors make a bit clear the limitations of this work in section five?

limitations: As per the checklist, the authors did mention limitations in Section 5. I believe the limitation is stated in the second paragraph of this section. Would it be possible for the author to clarify this? There is an abrupt transition from one paragraph to another here, and I read the paragraph twice to note the limitation.
review_title: null

Meta-review: metareview: This paper reveals several important facts about multi-task optimization methods by providing a comprehensive benchmarking of MOT methods. They find that with carefully tuning the hyper-parameter,  MTO methods give similar results compared to simply averaging the weight of each task.  Then they go deeper into why this has not been noticed in previous work by studying the evaluation step and provide solutions to standardize the evaluation of these methods. 
In particular, the reviewers have appreciated the comprehensive evaluation and tenable findings based on their experiments. Hence, I recommend that the paper be accepted. 
confidence: Certain
Decision: Accept