Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an approach for better multi task learning in the presence of objectives with a large difference in gradient magnitudes. They study the problem and define a "The rate of dominance" metric that is based on an eigenvalue ratio of the gradient tensors. they then use this dominance metric as a coefficient to better "balance" gradients across tasks and propose an elegant solution based on the  an approximate version that is only computed on the shared parameters. They report results on common multi-task settings, and either perform on par or outperform other recent methods.

==== After rebuttal ====

I want to thank the authors for a complete and detailed response. The answers make sense to me, and I am still in favour of acceptance - I raise my score to Accept.


main_review: ### Strengths 

* The authors deal with a very interesting problem, that of multi task learning in the presence of objectives with a large difference in gradient magnitudes
* They propose a way of quantifying the imbalance "The rate of dominance" and use it to propose an elegant solution based on the SVD of the gradients of each loss, and an approximate version that is only computed on the shared parameters. 
* the authors compare against recent works and show some empirical gains on sommon multi-task scenarios

### Weaknesses, questions and notes

1) In the illustration of figure 1, we see that there is a common solution for the two tasks in parameter space (towards the top left corner). Is it always possible to "guarantee consistent improvement over all tasks"? what if two tasks are very different (eg learning jointly for rotation invariance and for 6-9 digit recognition). 

2) In section 3 the authors say: " task-specific parameters are independent of each other and shared parameters, and therefore have a single gradient, and thus can be omitted without loss of generality." - Isn't this assumption very strong? I understand (and it seems) that that such an approximation indeed works, yet, the fact is that this might not be true in practice

3) Notation in Fig 2 is defined way after the figure is referenced in the intro of sec 4 and this makes the fig hard to understand. Adding some explanation for $u_i$ and $\sigma_i$ would make it better.

4) it is unclear to me what "vector w denotes a pre-selected task preference" means. Is this a hyperparameter weight per task? how is it set?

5) Beyond differences in the applications and tasks, the following is a missing related work:
> Wang, Weiyao, Du Tran, and Matt Feiszli. "What makes training multi-modal classification networks hard?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
Authors should discuss the relation of the proposed dominance to the overfitting-to-generalization-ratio (OGR) from Wang et al.
Ideally, a comparisson to GradientBlending would be great.

6) In algorithm 1 line 13: why are only the $\theta_{sh}$ parameters updated? how about the update of $\theta_i$

7) the task specific parameters seem to come as "heads" on top of a common shared backbone g() - Would this approach extend to works that use adaptors thoughout the net as in
> Housby et al., Parameter-Efficient Transfer Learning for NLP
> Pfeiffer et al., AdapterFusion: Non-Destructive Task Composition for Transfer Learning

8) Shouldnt the subscript  G_theta in definitin 3 be G_Z ?

9) I did not fully validate the correctness of the proofs of the theorems in the appendix. I think that a 1-2 sentence discussion is however needed on the theorems in the text, eg for Theorem 1, the second condition contains a lot of ndefined notation and it is really hard to understand. This relates with point 4 above and could clarify the role of vector w

10) Why is  Z-aligned, an efficiency-motivated approximation (In conclusions the authors claim that this was created "to ensure practicality"), better than $\theta$-aligned in many (most) cases? 

11) Another work that can be cited and discussed is
> Lu, Jiasen, et al. "12-in-1: Multi-task vision and language representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

summary_of_the_review: Overall an interesting and elegant approach, that shows some gains. There are weaknesses, discussed above; looking forward to the rebuttal from the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors aim to perform gradient task balancing on MTL setting by aligning independent components of the training objective. They claim to be able to handle large differences in gradient magnitude. Not choosing the faster learning rate dominating direction prevent task overfitting. The main contribution is based on the analysis of the individual components of the gradients instead of the individual gradients using the dominance rate. Tasks should have small dominance coefficients. Through a series of experiments the authors show the proposed technique is competitive with other multi-objective optimization approaches.    
main_review: The main strength of the paper is the introduction of the dominance rate, which allows the approach to avoid task overfitting. Another strength of the paper is the  technical presentation that is quite detailed. The z-alignment is also a strong point of the paper due to the solution to the limitation of the linear scaling of the theta-alignment counterpart. 

Regarding the weaknesses of the paper I would point out the first claim that the method provides consistent improvements over all tasks. This claim is not backed up by the experiments. First at Table 2 the authors claim a noticeable improvement of 0.06% for Task-L and 0.16% for Task-R. I believe the results difference to the SOTA method are marginal and, consequently, hard to draw any conclusions. Besides that I suggest the authors to include the work from Lee et al. [1] which proposes a single gradient step update for task balancing that shows better results than pareto-MTL and GradNorm with much higher differences than 0.06 and 0.16%, respectively.  

[1] Sungjae Lee and Youngdoo Son. Multitask learning with single gradient step update for task balancing. arXiv preprint arXiv:2005.09910, 2020. 

The Camera relocalization results shows a slight improvement over the compared methods for orientation. Nervertheless, I believe the 15% orientation improvement over GradNorm is not enough due to the 25% performance degradation when compared to GradNorm. Besides that I missed the reference to Radwan et al. [2] which shows much more accurate results for the 7-SCENES dataset. The compared and presented results are too far from been competitive with the cited paper.

[2] Noha Radwan, Abhinav Valada, Wolfram Burgard. VLocNet++: Deep Multitask Learning For Semantic Visual Localization And Odometry
IEEE Robotics And Automation Letters (RA-L), 3(4):4407-4414, 2018. 

With the inclusion of the referred papers will be possible to better verify the claims related to performance and the comparison to non MOO methods make the method a more generalist approach.

Another suggestion is related to benchmark the proposed method in a high cardinality dataset (hundreds of tasks). Datasets like "PubChem BioAssay Dataset Study" count with 128 tasks that is ideal to check the scale limitations of theta-alignment and the z-alignment capabilities. 

summary_of_the_review: The paper technical contribution is interesting, however the experimental section show the performance gains can be marginal and claims regarding results should be revised. Additional references and experiments are needed to draw conclusions regarding the actual empirical contributions of the method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a gradient-based multi-task learning method to balance multi-task training by aligning the independent components of the training objective. Experiments conducted on several MTL problems are given. The method is claimed to be scalable, robust to overfitting, and able to seamlessly handle multi-task objectives with a large difference in gradient magnitudes. 
main_review: Strengths:
-	The idea of balancing multi-task training by aligning the independent components of the gradient of the training objective makes sense to me. 
-	The proposed method achieves favorable performance on multiple multi-task learning benchmarks.
-	The paper is well written and easy to follow.

Weaknesses:
- The performance on some benchmarks are not that good, such as the results on CELEBA, 


summary_of_the_review: The idea of this paper is interesting and makes sense.
Although the performance improvements against existing methods are not that significant, the stable improvements on multiple benchmarks demonstrate the effectiveness of the proposed method.
review_title: null

Meta-review: This paper presents work on multi-task learning.  The reviewers appreciated the method based on SVD of loss gradients.  However, concerns were raised regarding empirical effectiveness and overall impact.  The reviewers considered the authors' response in their subsequent discussions.  While the methods are interesting, the concerns over their effectiveness would need to be more thoroughly addressed in order to improve the impact of the paper.  As such, it is encouraged that the authors take these suggestions into account in preparing a new version of the paper for a future submission.
Decision: Reject