Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work is interested in re-examining the efficacy of multi-task optimization methods, termed STMOs in the paper. The authors believe that prior STMO work does not treat the "unitary scalarization" baseline, i.e. the simple sum of objectives, fairly and that, when fairly treated, unitary scalarization can achieve comparable results to SMTOs.

The core contribution of this work is a large set of experiments on 4 multi-task datasets (3 vision and 1 reinforcement learning) in which the authors compare the distribution (mean & std.) of performance using 4 common SMTOs, 2 random weighting schemes, and unitary scalarization. Their results show that no individual method significantly outperforms unitary scalarization across all tasks. Although some methods may outperform unitary scalarization on certain tasks, unitary scalarization is always well within their standard deviation, and moreover the benefits pf SMTOs are never consistent across datasets. In addition, SMTOs have a significant effect on training time, increasing it drastically for little-to-no gain in performance. 

The authors postulate that this is because most  SMTOs can actually be seen as regularizers. The second contribution of this work is a theoretical analysis of SMTOs in terms of their convergence sets - MGDA, PCGrad, and IMTL increase the convergence set of optimization (i.e. the number of solutions a method may stop at), and suggest that this increase in convergence sets can behave similar to early stopping. Overall, their work suggests that current SMTOs, whose focus is on mitigating task conflict, instead benefit multi-task learning through implicit regularization effects - an effect which can be mirrored much more simply with unitary scalarization.
strengths_and_weaknesses: Strengths:
- A lot of multi-task learning research has focused on conflict-mitigating methods to improve negative transfer. However, this exhaustive  set of experiments suggests that this focus has not been as beneficial to multi-task learning as previously believed. This is an important result which suggests that future MTL work should rethink how task conflict is approached.
- The number of experiments run is helpful, and is a meaningful contribution alone - it provides a reasonable reference for future multi-task work to build off of. Moreover, the authors have promised to release the code for all SMTOs.
- The analysis of SMTOs acting as regularizers is interesting - To the best of my knowledge this is the first joint theoretical treatment of the effects of SMTOs on the optimization.

Weaknesses:
- There seems to be a disconnect between the analysis of SMTOs and the empirical results of section 5.1:
	- Much of the analysis is centered around showing that there exist solutions which an SMTO may converge to, yet a unitary scalarization approach will not. To me this suggests that SMTOs will, in some senses, early-stop themselves. However, the results in 5.1 on CelebA suggest that SMTOs converge slower than unitary scalarization, and that the early stopping regularization arises because the max number of training epochs has been reached.
- At face value, the contributions of the paper seem somewhat limited - the core contribution of this paper is a large set of experiments using prior methods. I do not think this is a major weakness, because the results are novel and important. I think the paper would benefit from much more focus on section 5 - the biggest contribution of the paper, in my mind, is the connection between regularization and SMTOs which help to explain the results in section 4. However, section 5 is comparatively small and empirical analysis of this connection is very limited (5.1).
questions: The CelebA experiments in 5.1 suggest that SMTOs also simply converge slower - if training continued, would they continue to overfit similar to unitary scalarization? In other words, is the early stopping exhibited in practice simply an artifact of the number of epochs the models are trained for (a fixed number of epochs for all methods) or because the multi-task methods have truly "stopped"? 
limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors perform an empirical evaluation of various optimizers designed specifically for multi-task learning, particularly those that utilize per-task gradients. The experiments, which are conducted on several image classification tasks and the metaworld MT10 and MT50 multi-task RL benchmarks, generally show that unitary scalarization (simply summing loss for the tasks) is a very strong baseline for multi-task learning when its regularization (l2 + dropout) is tuned, and the benefits of more sophisticated optimizers is dubious, given their computational cost. The authors interpret their findings to mean that more sophisticated multi-task learning objectives essentially perform a form of early stopping, providing some initial theoretical arguments for why this may be the case.
strengths_and_weaknesses: Strengths:
- Thorough evaluation of multi-task learning (MTL) methods, with experimental details well-catalogued
- Surprising empirical result
- Likely to be a useful set of reference experiments within the MTL world
- Most of the paper is very well written

Weaknesses:
- Theoretical section seems hurried, and I'm not totally convinced by the line of argument (see below)
- As the authors point out, more comprehensive hyperparameter tuning would be nice, given the paper's primary contribution is providing a large comparison
- No methodological novelty (this is not a requirement for a strong paper, but without it, the key results need to be very strong)
questions: Overall, my biggest concern with the paper is Section 5. While I can appreciate the general intuition that a larger number of convergence points may lead to early stopping-like behavior, the authors don't convincingly argue that this is indeed the case in practice. For example, the statement that PCGrad has an "enlarged convergence set" based on the points where gradients are exactly opposed seems like a stretch- this is a vanishingly small set of points. Nonetheless, PCGrad performs reasonably favorably to the other baseline algorithms, and the alternative argument that the stochasticity in the objective leads to better performance is not developed in much depth.

One way to address these concerns would be to empirically show that, as their theoretical argument suggests, SMTOs do indeed converge to solutions that are far from the fixed points of unitary scalarization. i.e., at SMTO convergence, is the unitary scalarization update magnitude still large? Alternatively, can the authors compare the flatness of the minima for regularization unitary scalarization and SMTOs? The authors invoke the flatness intuition, but it is not validated.

Regarding methodological novelty, I wonder if the authors can also provide more detailed discussion of their results; currently, while the results are interesting, they are not directly actionable for the community. What do the authors recommend the MTL community do with these results? How can these insights translate into better or more efficient models? I wonder if some of the text allocated to the theoretical section might be better used for such discussion.

As a minor point, Figure 5 is quite hard to interpret, especially for colorblind readers. I would suggest using multiple line styles to make it easier to compare between unit scale and SMTO lines.
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers the multitask learning setup (MTL) in which a model is trained to perform several tasks. Several existing MTL approaches argue against the simple baseline of using the sum per-task losses (unitary scalarization) as the training objective. Instead, SOTA MTL approaches usually manipulate the per-task gradients to obtain a joint update direction. Thus, the more complex per-task methods increase the training time and implementation complexity. The authors show that by adding proper regularization and carefully tuning the unitary scalarization baseline, it can perform on-par with existing MTL approaches. The paper raises some valid concerns regarding the evaluation and experimental methodology in the MTL literature. However, some questions about the experimental setup and missing comparisons/benchmarks impair the authors' claim. 

strengths_and_weaknesses: **Strengths:**
- The paper is well organized and well written.
- This paper offers a novel view of SOTA MTL methods as regularization methods.
- The authors perform an important evaluation of recent MTL methods and raise valid concerns regarding the evaluation and experimental methodology in the MTL literature.
- Novel analysis and view of well-known MTL methods through the lens of regularization.

**Weaknesses:**
- Most MTL approach uses the feature gradients as a surgent to the full task gradient (or uses other approaches to scale the computation), hence resulting in only a small increase of runtime.
- The arguments for presenting MTL methods as regularizers appear quite weak. While interesting and novel, it appears to provide only a partial explanation.
- The experimental setup lacks important comparisons and evaluation on more relevant benchmarks:
  - First, the authors fail to (cite [1], and) compare with recent strong MTL approaches [1,2,3] that show large performance gains w.r.t unitary scalarization. 
  - The results presented in [1,2] show better performance on the Cityscapes benchmark for all methods (for example in terms of mIoU). This might suggest that all methods are under-optimized.
  - A better metric for MTL evaluation is the relative task improvement (compared to STL), used in previous MTL literature [1,2,3].
  - For Cityspcaes, it’s not clear how the early stopping was performed. How do you reduce the different metrics into a single metric as the early stopping criterion? Metrics have different scales and meanings. A good approach is using the relative task improvement as the stopping criterion.
  - The supervised learning benchmarks are insufficient: (i) Multi-MNIST is a toy dataset with only two tasks; (ii) The vast majority of methods quickly overfit the Celeb-A benchmark. Hence, this benchmark is not the best choice for arguing something concerning MTL optimization: the effective optimization process ends early, so it’s hard to learn something meaningful about the optimization dynamics. (iii) Finally, the Cityspcaes dataset consists of only two tasks that were shown to benefit each other in previous works. Hence, we cannot learn from it about optimization with a large number of possibly conflicting tasks.
  - I suggest the authors add more relevant and challenging MTL benchmarks like the QM9 with 11 tasks (used in [1]) and the NYUv2 dataset with 3 tasks (commonly used through MTL lit.).

References:
- [1] Multi-Task Learning as a Bargaining Game. ICML 2022.
- [2] Conflict-Averse Gradient Descent for Multi-task learning. NeurIPS 2021.
- [3] RotoGrad: Gradient Homogenization in Multitask Learning. ICLR 2022.



questions: See above.
limitations: N/A
review_title: null

Meta-review: metareview: This paper criticizes the SMTO methods by building a single experimental pipeline and finding that none of the SMTOs consistently outperform unitary scalarization method, which is the simplest and cheapest method for multi-task learning. To explain the findings, the authors postulate that SMTOs act as regularizers and present an analysis.

Generally, I like such a kind of paper as it would provide a criticized view and corresponding evidence as well as an in-depth analysis, which let us stop and think about the real progress we have made so far, and indeed helps the long-term development of the studied area.

The reviewers generally provided supportive comments and positive overall ratings. I also make an acceptance recommendation for this paper.

confidence: Certain
Decision: Accept