Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a method to reduce the optimization difficulty for multi-task learning. The key idea is to homogenize task gradients in terms of both magnitudes and directions. Specifically, they rescale task gradients to maintain similar convergence rates over tasks. At the same time, they introduce new network parameter R_k for each task k and train it so that the angle between task-specific and average gradients are minimized when task-head parameters are rotated using R_k. The method improves baselines and outperforms existing methods on DIGITS, CIFAR10, NYUv2, and CelebA datasets.
main_review: I think the key contribution of this paper is to introduce rotation matrices that reduces gradient conflicts by rotating the feature-space z in a way that matches their directions. In particular, the idea to use rotation matrices is reasonable and good in that they will preserve the relational information (e.g. distance) between features.

They claim that the novelty lies on the joint homogenization of gradients in terms of both magnitude and direction. However, IMTL [18] also jointly addressed loss magnitudes and gradient direction. Rather, I think novelty lies on the way they formulate the problem as a Stackelberg game. In other words, they optimize rotation matrices only to match gradient directions while optimizing the remaining network parameters to minimize the task loss. GradNorm used a similar strategy but they focused on adjusting gradient magnitudes. 

In the experiments, their method improves baselines and performs better than other methods on four datasets and for two different backbone architectures. Additional analysis on training stability, effect of sub-dimension size for rotation, and training time comparison provides more comprehensive views.
However, it seems one needs to carefully adjust the training parameters for the rotation matrix, depending on dataset. when I ran the submitted code for mnist, the median cosine similarity for the task 'density' oscillated largely.


I think L130-131 is a bit misleading, because rotating the feature z is different from rotating its gradient g. Rotating z could result in the change of magnitudes in gradients. For example, consider a function f(x,y) = [relu(x),0]. A vector [x,y]=[1,0] and its rotation [x,y] = [-1,0] have different gradient magnitude.

It would be good to add ablation study to analyze the effect of magnitude and direction separately, i.e., magnitude-only and direction-only. Since each of them was explored in existing work, comparing each to corresponding other approaches will be also informative, e.g., magnitude-only to GradNorm and direction-only to PCGrad and GradDrop.

One way to demonstrate the training behavior will be to visualize the actual reduced gradient conflicts, possibly using angles, compared to the baseline. Also, comparing the method with baselines in the task loss curve will show that the method results in lower task loss.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel method for the multitask learning environment, which deals with the conflicting gradients problem (AKA negative transfer). The method comprises two parts for dealing with gradient magnitudes and directions. The latter part is quite novel, well-motivated (with a cartoon example from Fig. 1), and theoretically sound. The method is tested against a number of MTL benchmarks and is shown to produce better-performing models. Overall, the paper presents interest to the MTL community.
main_review: L27: why is gradient cancellation seen as a problem? If a model cannot explain training data, conflicting gradients can be seen as reducing the variance of the solution, yielding a model which is "equally" bad wrt multiple objectives (as shown in Fig.1). In such a case, the proposed solution can be seen as an efficient way to increase model capacity in the right places (where the heads branch out), thus leading to a model with higher capacity (which is not bad per se but requires acknowledgment).

L35: can prioritization of the "lagging" tasks be harmful, e.g., in cases where the task loss is noisy (e.g., due to extreme label noise)?

The usage of a task-specific rotation matrix is interesting; however, could it be replaced with a simpler learned construct, such as a task-conditional batch norm, keeping the alternating training scheme from Alg.1 intact?

The usage of matrix exponential for parameterizing the rotation matrix is interesting; however, in the light of the argument in L151, do the gradients of the unconstrained parameters of R require any treatment (gradient magnitude/LR adjustment)?

One unclear aspect associated with matrices R is whether the preceding activations are the output of a linear layer or a nonlinearity. If activations are nonlinear and given that such a matrix R constitutes an additional linear layer, it would be interesting to see extended single-task baselines with matrix R added in the right place (where it would otherwise branch out from the backbone in MTL).

An ablation study of the magnitude and direction counterparts, particularly where another more traditional method can be applied in place of one of the proposed magnitude components, is missing. The closest I could find in Table 1 reporting IMTL-G+$R_k$

================= Post-rebuttal comment

After reading all reviews and responses, and discussing the common concerns with the other reviewers, one weak point of this work remains the increased capacity of the network (regardless of how the added parameters are optimized). For this reason, I chose to keep my original rating of WA.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes RotoGrad, a multi-task optimisation strategy that normalises the gradient magnitude and (conflicting) direction in a unified manner, to avoid the negative transfer. The proposed method is intuitive, theoretically motivated by the two player-game, and hyper-parameter free (except for learning rate required for updating rotation parameters). 

The paper extensively evaluates the proposed method comparing with other baselines, both in single domain MTL setting NYUv2, and in multi-domain MTL setting CIFAR-10, and CelebA, and have showed improved performances in all settings.
main_review: The paper is clearly written, and well-motivated from the existing Stackelberg leadership model. I also appreciate the authors discuss the memory and time complexity in Section 3.4, and being honest about the current limitation on scaling issues in computer vision tasks, which typically with large representation dimensions, requiring extensive computing time for an exact solution without any approximation.

Generally, I like the idea and particularly for the part that introducing the gradient rotation, which I believe is novel and overlooked in previous multi-task optimization papers I know. Here are some other comments, which I hope the authors could clarify and hopefully could further improve the paper.

-- Subspace rotation. As mentioned in the previous comment, for computing a direct solution requires a $d^2$ rotation matrix for each task with representation dimension $d$. The introduced rotation parameters can be neglectable if $d$ is small. However, if $d$ is large as typically in most computer vision tasks, it introduces more parameters compared to baseline methods, which leads to unfair comparison.  We thus would not know whether the improved performance is based on the design of this method, or simply because of the increased number of learnable parameters. In Table 1, applying the rotation only in the 64 subspace is having a close and worse performance compared to the baseline performance, this verifies my concern here. To be clear on the parameter size and/or FLOPs in each method would be essential for a fair comparison. Maybe adding a linear embedding to map to a smaller embedding size, or applying a block-wise rotation would alleviate this issue?

-- Limitation on multi-task network design. From my understanding, the proposed RotoGrad requires a clear top-down multi-task network design with a **single** shared representation output $z$, which the task-specific rotation function needs to be applied on. It would be tricky, or impossible? to apply this strategy to any design of the network, say MTAN as discussed in the related work, or other popular multi-task network design: such as AdaShare, Cross-Stitch network, from which the shared network parameters are defined implicitly. In this way, the RotoGrad would be very hard to optimise, maybe requiring network-specific modification on choosing the appropriate $z$. This is not a serious issue though, if the improved performance solely on hard parameter sharing is significant enough that surpass the improvements brought by the design of the network itself. A paragraph discussing this issue would be helpful.



limitations_and_societal_impact: This paper studies the theoretical insights and optimisation strategy for multi-task learning, and thus having no societal impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The work proposes RotoGrad, a multitask layer that rotates a shared feature space in a deep neural network to align gradients and mitigate negative transfer effects. The proposed method works within models with multiple prediction heads and a shared backbone, and deals both with magnitude and directional imbalance between task gradients. Authors provide multiple experiments in the computer vision space to show efficacy of the proposed method against other multitask baselines. 

main_review: **Overall:** The paper presents an interesting new method to treat multitask learning systems. If robust, this method can have general use to the community and be a powerful addition to the toolbox. However, there are many unanswered questions and I remained overall unconvinced that the method works without major additional experimentation, including detailed ablation studies, and justifications (too many to address within the short rebuttal period in my opinion). I do think the method shows promise though, so regardless of the decision after this review process I hope the authors continue to work on their method.  

**Pros:**

(P1) Method is clear and intuitive, and tackles a crucial problem within optimal multitask training. 

(P2) Results seem promising and beat other baselines on benchmarks within the main paper. 

(P3) Formulation of the method as a Stackelberg game is interesting and can perhaps provide general insight into these problems, as many modern MTL methods involve optimizing auxiliary losses.

(P4) Results are given for multiple model structures and multiple datasets performing the same tasks. 

(P5) Code is provided for reproducibility. 

**Needs Comment/Revision (major):**

(RM1) Why must the proposed method interact with the forward pass? There are some other methods (see for example Sinha et al. at https://arxiv.org/abs/1806.08028) that propose methods which only interact with the backwards pass and seem to also work well. Either theoretical justification or additional experiments are needed to justify why doing the aggressive thing of altering the forward pass feature space is necessary. 

(RM2) The toy example provided is unconvincing, as the only comparison is to vanilla SGD. It is rather puzzling that even the Rotograd layer within the toy example is optimized using RAdam, which I feel would give Rotograd a severe advantage. 

(RM3) When the proposed method outperforms other methods that’s great, but when all the other methods perform very poorly it raises questions of whether the other methods are implemented correctly. For instance, why is MGDA performing so poorly on the CIFAR/MGDA tasks when the original MGDA paper used that as a benchmark and actually performed quite well? These discrepancies with the prior results need to be addressed for the current results to be fully believed.

(RM4) One strange detail is that GradNorm and GradDrop seem to have different compute overheads within the current implementations, at least within the CelebA experiments. However, as far as I can tell, GradNorm and GradDrop have essentially the same compute overhead (with the limiting factor in the compute overhead the same computation of dL_i/dw for each task loss L_i), so this discrepancy makes me concerned that the present implementations of baselines is not correct. Similarly, RotoGrad seems to be even slower than PCGrad, and PCGrad has to calculate dL_i/dw for all trainable weights while RotoGrad only needs to backpropagate to the feature layer, so this overhead is somewhat concerning as well.

(RM5) Ablation studies are needed to see whether both the magnitude and directional components of Rotograd are both needed. 

(RM6) The magnitude part of the implementation seems to include the initial gradient magnitude as part of the normalized quantity. However, this is incredibly dependent on the network initialization. Did authors account for this in some way? 

(RM7) The digit classification task performances seem to be very dependent on the size of the RotoGrad layer. Looking at the appendix, the network structure for these tasks is very shallow, so how can the authors rule out the effect of the added network capacity within the RotoGrad layer?

**Needs Comment/Revision (less major):**

(RLM1) Proposition 3.1 is correct but is rather obvious - would authors consider rephrasing into nonmathematical language to reduce potential confusion? If two gradients have positive pairwise similarity, then following their sum (for small enough learning rate) will clearly reduce both losses because loss reduction is step_size*(grad_k dot grad_sum). This proposition has also been heavily treated in prior work as well like MGDA and PCGrad, so does not necessarily need independent treatment here and can just be stated.

(RLM2) Authors claim that homogenizing gradients at one layer is equivalent to homogenizing gradients at the weight layers. This is a false statement, as chained gradient computations will cause even initially homogenized gradients at a feature layer to increasingly stray from each other. Authors must justify why they can ignore this effect.

(RLM3) Figures need to be placed closer to the text where they’re referenced. 

(RLM4) Sometimes bolded numbers in tables are not the best results for that column. 

*Post rebuttal*: I thank the authors for their time and effort in crafting their response. A lot of my more technical comments were addressed. I am still very concerned about the extra capacity and don't feel these concerns are properly addressed by pointing out that one other baseline (IMTL-G) also has added capacity, and some of the new results actually may run counter to the proposed narrative - for example RotoGradNorm beating out RotoGrad. The method itself is well defined but I am still not sold on the scientific validity of the analysis. I will keep my rating a 4. 

limitations_and_societal_impact: More discussion could be provided regarding limitations and societal impact. In terms of the former, for example, there are plenty of benchmarks on which RotoGrad seems to not perform the best, so discussion of why needs to be more in detail.
review_title: null

Meta-review: The work introduces an approach to minimize interference in multitask learning RotoGrad. 

After carefully looking over the reviews and rebuttal, I do find the proposed solution interesting and the experimentation section comprehensive (especially with the additional results provided by the authors during the discussion period). 
I do not feel that the concerns of extra parameters problematic in this scenario. 

In general if there is a weakness of this work that I would highlight is that I feel more analysis could have been done to understand the behavior of the rotation matrices and the scaling of the updates done. For example, would the rotation matrices converge before the multitask problem converge to some fix task-specific rotation ? I would have been interested in any form of visualization that could have helped to understand what these matrices do, how they interfere with the baseline model, how they change the learned representation etc. The same for the scaling of the updates. I feel the focus is mostly on looking at the improved performance (which to be fair is the standard way of approaching this) but not as much on understanding how the proposed method works and what impact it has. 

That said, the results are reasonable, and I do think this method will be of interest to the community. 
After a long deliberation, I feel the paper in its current form is not exactly ready for NeurIPS (though is really a very borderline case). I think the additional results the authors provided greatly increase the paper. I would have liked a bit more analysis (or ablation) in understanding the interplay between the added capacity. What the rotation matrices really do. Anything that gives additional understanding of the dynamics of the method would have probably pushed the paper over. I strongly encourage the authors to improve the manuscript and resubmit/
Decision: Reject