Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The submission argues that when training the multiple objectives in a multi task learning framework, orthogonalizing gradients is beneficial toward reducing task competitions and efficient allocation of the learning capacity in the parameters.  This is demonstrated experimentally and subsequently a second order method is used for incorporating this in training. 

Positives: 
1. The concept is sensible, intuitive, and simple. There has a been quite few works (Sener and Koltun 2018) that similarly argue analyzing/regularizing the gradient direction when training multiple task objectives is beneficial. This submission reinforces those. 

2. The implementation of the concept using a second order method is sensible and simple.  

Weaknesses: 
1. Missing comparisons: As reiterated above, several existing works have augmented training of neural networks with terms that are based on directions of gradients of multiple objectives. Some of them, e.g. Sener and Koltun 2018 or Du et al 2018, are very related. Though the submission cites them, no experimental comparison or convincing verbal critique of the differences are provided. The experiments could have included baselines that correspond to the concepts proposed in those prior papers to support the novelty of this submission. If the authors believe the concept of those papers are basically the same as theirs, then this submission should change to an analysis paper rather than appearing to pitch a new regularization term. I would have found an analysis paper as valuable as one with a novel method, but the stance should be clear. 

2. I found the experiments too toy to be convincing. The MultiDigitMNIST is artificial and with limited benchmarking value as its construct doesn't really reflect the construct of multi task learning in the real world (e.g. as in the multi task vision datasets). NYUv2 dataset is more realistic, but the reported results are not clear to show significant and meaningful differences (see table 2). Particularly since the NYUv2 dataset has certain biases with imperfect ground truth from the kinect sensor which questions if differences in 0.001 range are meaningful. The authors can consider more recent and comparatively more reliable multi label datasets like Taskonomy for benchmarking. 

3. Inline with the above comment, qualitative results or any other method for convincing that the achieved results (on datasets other than MultiDigitMNIST) is meaningful seems crucial. 

Further comments: 
4. The single task baselines in table 2 often perform inferior to the multi task baselines. This suggests the NYUv2 dataset could be too small to learn individual networks, whereas many other multi task papers often find single task baselines are hard to beat if they're not starved of parameters (e.g. see Standley 2018 "Which Tasks Should Be Learned Together in Multi-task Learning?"). Please clarify and/or consider using large enough datasets that allow you to benchmark under both high data and low data regimes. 

5. why there is no single task baseline in table 1? 

6. why CosReg is not included in figure 3 plot? 

7. A closer look at the recent works on analyzing task competitions in multi task learning could be useful, and probably supportive of the concept of this submission. For instance standley 2018 Which Tasks Should Be Learned Together in Multi-task Learning seem to suggest that tasks that are related under transfer learning setting did not help each other in multi task setting. Their observation seems related to the pitch of this paper that orthogonal gradients (ie tasks with dissimilar updates) can be optimized better. 

8. In page 4 authors state that computing the regularization term for all layers is complex, hence they do that only for the last layer. This seems okay to me, though I would have found an experiment demonstrating the consequences of this simplification useful (e.g. am experiment showing which layer to pick and a one-time expensive experiment demonstrating that picking one layer vs more layers is not too damaging). 

-------
Comments after rebuttal: 
I read the rebuttal and appreciate authors responses and the attempted improvement. It helped. I think the submission has the potential to be ultimately a useful read for the community, but at this stage more work/revision that wouldn't fit rebuttal time constraints would be needed to achieve that. I still think an experimental comparison with the related methods would be more convincing than verbal. 

The added experiment on SUN RGB-D dataset is appreciated, but all results appear too close (table 2) to suggest one should adopt the proposed method. Also SUN RGB-D limits the tasks that could be learned together in a multi-task experiment given the limited number of labels per datapoint. I would have found using more recent  datasets with higher number of labels (rather than forcing to use depth+semantics which doesn't have to be a good mix, see Standley 2018) and more significant quantitate improvements more convincing. 

Overall I lean toward not accepting the current submission but acknowledge the potential value in resubmission after further work. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper embraces the idea that better multi-task/lifelong learning can be achieved if tasks produce gradients that are orthogonal to the gradients produced by other tasks. The authors propose an approach to regularizing learning in order to incentivize this to happen. However, as they mention themselves, the regularized loss is computationally intractable in general and they only apply it to a subset of their network as a result. Given the computational scalability concerns, it is natural to wonder why researchers in the community would adopt this approach rather than other approaches that also aim to make gradients orthogonal. 

The idea of producing orthogonal gradients across tasks or examples is not new in the context of lifelong/multi-task learning. In fact, just to name a few, [1] demonstrated that noise alone can lead to orthogonal gradients, [2] demonstrated that modular neural network architectures can lead to orthogonal gradients and less interference. Additionally, sparsity naturally leads to orthogonal gradients as does the recent approach in [3].  These approaches achieve orthogonal gradients without adding a significant computational burden to learning. This paper can be greatly improved by discussing past approaches to producing orthogonal gradients and why they are theoretically / empirically worse than CosReg. 

[1] "A theory for how sensorimotor skills are learned and retained in noisy and nonstationary neural circuits". Robert Ajemian, Alessandro D’Ausilio,  Helene Moorman, and  Emilio Bizzi. PNAS'13. 

[2] "Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning". Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. ICLR'18. 

[3] "Meta-Learning Representations for Continual Learning". Khurram Javed and Martha White. 2019. 

Additionally, despite much past work, I tend to think that the entire quest for orthogonal gradients is not particularly well motivated as it is missing half of the story. Orthogonal gradients only address the problem of interference during learning, but don't help maximize transfer during learning. In fact, intuitively Figure 2 showcases that CosReg diminishes transfer during learning in comparison to baselines. Some recent work, such as [4] and [5], argues that what we really want is to maximize the dot product of gradients i.e. their alignment. This perspective achieves the best of both worlds as it incentivizes orthogonality to address interference while also incentivizing positive transfer. I wonder how the authors would position their work relative to the body of work that optimizes for the gradient dot product. Why would we like gradients to be orthogonal if there would otherwise be transfer? Why focus on the cosine rather than the dot product, which naturally comes out of the first order Taylor expansion derivation for each task? 

[4] "On First-Order Meta-Learning Algorithms". Alex Nichol, Joshua Achiam, John Schulman. 2018. 

[5] "Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference". Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. ICLR'19. 

Given my major concerns about the theoretical motivation and comparisons to past work, I do not find the experiments comprehensive enough to prove the value of the proposed approach to the community. At the very least, I would be interested in comparison with additional very relevant baselines and in experiments with more tasks.  

Update After Author Feedback: 

While I really appreciate the authors providing some context about the references, I still feel like the paper would benefit from increased empirical comparison with these past approaches. Unfortunately, I don’t really follow the point they are making about why it is better to produce orthogonality vs. high dot products.  I agree that catastrophic forgetting is more of a problem related to continual learning, but I am not sure why we wouldn't want to maximize transfer even if there was no forgetting or interference.   Given my continued concerns, I am inclined to keep my score the same. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary: In this paper, the author analyzed gradient regularization in deep multitask learning. They empirically discovered a sharper concentration (low variance) in angles between the task gradient distributions could potentially improve the performance in multi-task learning. Then they proposed a new gradient regularization to enforce the gradient for each task be orthogonal. Empirical results (on Multi-digits MNIST and NYUv2 data-sets) indicate a marginal improvement, comparing with baselines.

Main Comments:

The discovering in the paper sounds interesting while the work looks like preliminary and unpolished. Particularly I have the following technical and conceptual concerns:
1.  In high dimensional geometrical space, ** the random high dimensional vectors will be orthogonal with high probability**. The authors can find this conclusion in many places, for example: 

https://courses.cs.washington.edu/courses/cse521/16sp/521-lecture-6.pdf
(Theorem 6.3)
https://www.cs.princeton.edu/courses/archive/fall14/cos521/lecnotes/lec11.pdf
(Corollary 2)

I noticed the author claimed in the paper “Based on empirical observations presented later on we argue that multi-task networks not only benefit when the cosine is non-negative but more so when task gradients are close to orthogonal.” (Page 3).  However, this empirical claim is not convincing for me.  Since gradient of the over-parameterized neural network is in very high dimension, the task gradients closing to orthogonal may happen because of 
(a) the internal high-dimensional geometry property
(b) or the phenomene caused by the deep multi-task learning 

It will be much better and more clear if the author can provide more analytical, theoretical (e.g provable bound even on linear model) or empirical (e.g ablation study) support to understand the behaviors of gradient in high dimensional problem. 

2. In the proposed approach, I am not clear how the author choose the **task weights** (e.g in Eq.1) “w_{t_i}” ? It seems that the author set them as hyper-parameters. Since understanding the task relations and automatically estimating their relationships is the key factor for avoiding negative transfer in multitask learning (e.g Sener [2018]), I think it is not a proper way to simply adjust them as hyper-parameters.

3. In the experimental part, the proposed approach showed almost no or very marginal improved performance. The author should provide more evidences to show the utility or potentials, in order to convince the community to adopt the proposed approach. 

Minor comments:
1. From equation (4) to (5) is not obvious, it will be better to provide the derivation details.
2. The analyzed problem sounds more like the “multi-output” or “multi-label” problem (Section 3 in the paper). In the multi-task learning generally the inputs for each task X are different.

Overall, I feel it is an interesting and promising direction to consider gradient regularization based approach in multi-task learning. However, the current manuscript is not mature for the acceptance.


Reference:
Multi-Task Learning as Multi-Objective Optimization.  Ozan Sener, Vladlen Koltun, NeurIPS, 2018
----------------------------------------------------------------------------------------------------

After rebuttal

I thank the author for detailed rebuttal and efforts for making the paper better.

I accept points  (1), (2) and (5) , then I update my score to weak reject -- 3.

The main reason that I still keep the decision toward rejection is the experimental part. 

Since it is an empirical paper, I suggest the author either systematically show more comparisons and more datasets. 
Or the author can develop some theoretical insights/bounds (e.g point (1)) for enhancing the contribution of the paper.




review_title: Official Blind Review #2

Meta-review: This paper proposes a training approach that orthogonalizes gradients to enable better learning across multiple tasks. The idea is simple and intuitive.

Given that there is past work following the same kind of ideas, it would be need to further: 
(a) expand the experimental evaluation section with comparisons to prior work and, ideally, demonstrate stronger results.
 (b) study in more depth the assumptions behind gradient orthogonality for transfer. This would increase impact on top of past literature by explaining, besides intuitions, why gradient orthogonality helps for transfer in the first place.

Decision: Reject