Regularizing Deep Multi-Task Networks using Orthogonal Gradients
Authors: Mihai Suteu
Yi-ke Guo
Venue:  (2020)
Keywords: multi-task learning
gradient regularization
orthogonal gradients

Abstract: Deep neural networks are a promising approach towards multi-task learning because of their capability to leverage knowledge across domains and learn general purpose representations. Nevertheless, they can fail to live up to these promises as tasks often compete for a model's limited resources, potentially leading to lower overall performance. In this work we tackle the issue of interfering tasks through a comprehensive analysis of their training, derived from looking at the interaction between gradients within their shared parameters. Our empirical results show that well-performing models have low variance in the angles between task gradients and that popular regularization methods implicitly reduce this measure. Based on this observation, we propose a novel gradient regularization term that minimizes task interference by enforcing near orthogonal gradients. Updating the shared parameters using this property encourages task specific decoders to optimize different parts of the feature extractor, thus reducing competition. We evaluate our method with classification and regression tasks on the multiDigitMNIST and NYUv2 dataset where we obtain competitive results. This work is a first step towards non-interfering multi-task optimization.

Under review as a conference paper at ICLR 2020
REGULARIZING DEEP MULTI-TASK NETWORKS USING ORTHOGONAL GRADIENTS
Anonymous authors Paper under double-bling review
ABSTRACT
Deep neural networks are a promising approach towards multi-task learning be- cause of their capability to leverage knowledge across domains and learn general purpose representations. Nevertheless, they can fail to live up to these promises as tasks often compete for a model's limited resources, potentially leading to lower overall performance. In this work we tackle the issue of interfering tasks through a comprchensive analysis of their training, derived from looking at the interaction between gradients within their shared parameters. Our empirical results show that well-performing models have low variance in the angles between task gradients and that popular regularization methods implicitly reduce this measure. Based on this observation, we propose a novel gradient regularization term that minimizes task interference by enforcing near orthogonal gradients. Updating the shared parameters using this property encourages task specific decoders to optimize dif- ferent parts of the feature extractor, thus reducing competition. We evaluate our method with classification and regression tasks on the multiDigiMMNIST, NYUv2 and SUN RGB-D datasets where we obtain competitive results.
INTRODUCTION
Deep neural networks have proven to be very successful at solving isolated tasks in a variety of fields ranging from computer vision to NLP. In contrast to this single task setup, multi-task learning aims to train one model on several problems simultaneously. This approach would incentivize to transfer knowledge between tasks and obtain multi-purpose representations that are less likely to overfit to an individual problem. Apart from potentially achieving better overall performance (Caruana 1997), using multi-task approach offers the additional benefit of being more efficient in memory usage and inference speed than training several single-task models (Teichmann et al. 2018).
popular design for deep multi-task networks involves hard parameter sharing (Ruder] 2017). where model contains common encoder, which is shared across all tasks and several problem specific decoders. Given a single input each of the decoders then trained for a distinct task using a different objective function and evaluation metric. This approach allows the network to learn multi-purposs representations through the shared encoder which every decoder will then use differently according the requirements of its task. Although this architecture has been successfully applied multi-task learning (Kendall et al.) 2018 Chen et al. 2017) it also faces some challenges. From an architec- tural point of view is unclear how choose the task specific network capacity (Vandenhende al.) 2019 Misra et al., 2016) as well as the complexity of representations to share between tasks. Ad- ditionally, optimizing multiple objectives simultaneously introduces difficulties based on the nature of those tasks and the way their gradients interact with each other (Sener & Koltun 2018). The dis- similarity between tasks could cause negative transfer of knowledge (Long et al. 2017; /Zhao al.) 2018 Zamir et al. 2018) or having task losses of different magnitudes might bias the network in favor of a subset of tasks (Chen et al., 2017 Kendall et al., 22018). . It becomes clear that the overall success of multi-task learning reliant managing the interaction between tasks, and implicitly their gradients with respect to the shared parameters of the model.
This work focuses the second category of challenges facing networks that employ hard param- eter sharing, namely the interaction between tasks when being jointly optimized. We concentrate on reducing task interference by regularizing the angle between gradients rather than their magni-

---

Under review as a conference paper at ICLR 2020
tudes. Based our empirical findings unregularized multi-task networks have high variation in the angles between task gradients, meaning gradients frequently point in similar or opposite directions. Additionally, well-performing models share the property that their distribution of cosines between task gradients is zero-centered and low in variance. Nearly orthogonal gradients will reduce task competition as individual task decoders learn to different features of the encoder, thus not in- terfering with each other. Furthermore, we discover that popular regularization methods such as Dropout (Srivastava et al., 2014) and Batchnorm (Toffe & Szegedy, 2015) implicitly orthogonalize the task gradients. We propose a new gradient regularization term to the multi-task objective that explicitly minimizes the squared cosine between task gradients and show that our method obtains competitive results on the NYUv2 (Nathan Silberman & Fergus 2012) and SUN RGB-D (Song) et al. 2013) datasets.
RELATED WORK
Multi-task learning is a sub-field of transfer learning (Pan & Yang 2009) and encompasses vari- cty of methods (Caruana 1997). 'he recent focus on deep multi-task learning can be attributed to the neural network's 's unparalleled success in computer vision (Krizhevsky et al., 2012: Simonyan & Zisserman 2014; He et al. 2016) and its capability to create hierarchical, multi purppos represen- tations (Bengio et al. [203]] Yosinski et al. 2014p. Deep multi- learning is commonly divided into hard or soft parameter sharing methods (Caruana 1997, Ruder 2017). Soft parameter sharing maintains separate models for each task but enforces constraints on the joint parameter set (Yang & Hospedales 2016). In this work we focus solely on hard parameter sharing methods, which maintain a common encoder for all tasks but also contain task-specific decoders that use the learned generic representations.
We further split deep multi- approaches into architecture and loss focused methods. Architecture based methods aim at finding network structure that allows optimal knowledge sharing between tasks by balancing the capacities of the shared encoder and the task specific decoders. Most multi- task related work chooses the architecture on ad hoc basis (Teichmann al. 2018 Neven al.] 2017). but recent research looks to answer the question of how much and where to optimally share knowledge. Cross-stitch networks maintain separate models for all tasks but allow communication between arbitrary layers through specialized cross-stitch units (Misra et al.) 2016). Branched multi- task networks allow for the decoders to also be shared by computing a task affinity matrix that indicates the usefulness of features at arbitrary depths and for different problems (Vandenhendel jet al., 2019). [iu et al. (2019b) introduces attention modules allowing task specific networks to learn which features from the shared feature network to use at distinct layers.
Loss focused methods try to balance the impact of individual tasks on the training of the network by adaptively weighting the task specific losses and gradients. Certain tasks might have dispro- portionate impact on the joint objective function forcing the shared encoder to be optimized entirely for subset of problems, effectively starving other tasks of resources. Kendall et al. (2018) devise weighting method dependent on the homoscedastic uncertainty inherently linked to each task while Chen et al. (2017) reduce the task imbalances by weighting task losses such that their gradients similar in magnitude. [Sener & Koltun (2018) cast multi-task learning as a multi- nujti-obbecctiv optimiza- tion problem and aim find a Pareto optimal solution. They also analyze gradients but with the goal to then scale these such that their convex combination will satisfy the necessary conditions to reach the desired solution. In contrast these approaches our method does not seek to scale gradients, neither directly nor via task weights, but conditions the optimization trajectory towards solutions that have orthogonal task gradients.
The problem of conflicting gradients or task interference has been previously explored in multi-task learning as well as continual learning. ZZao et al. (2018) introduce a modulation module that reduces destructive gradient interference between tasks that are unrelated. Du et al. (2018) choose to ignore the gradients of auxiliary tasks if they are not sharing a similar direction with the main task. Riemer et al. (2018) maximize the dot product between task gradients in order to overcome catastrophic forgetting. These methods have in common the interpretation that two tasks are in conflict the cosine between their gradients is negative, while alignment should be encouraged. Our work differs from this perspective by additionally penalizing task gradients that have a similar direction, arguing that by decorrelating updates the shared encoder is able maximize its representational capacity. A
2

---

Under review as a conference paper at ICLR 2020
similar observation regarding orthogonal parameters is made by Rodriguez et (2016) who propose a weight regularization term for single task learning that decorrelates filters in convolutional neural networks. Another way to minimize interference to encourage sparsity (French [991] Javed & White 2019), although doing so directly might interfere with the network's ability to learn shared representations (Ruder 2017).
Finally our work is in line with recent research (Liu et al. 2019a Santurkar et al. 2018) that em- phasizes the benefit of analyzing gradients to understand neural networks and devise potential im- provements to their training. We share elements with Drucker & Le Cun (199T) and more recently Varga et al. (2017) in that we propose explicit regularization methods for gradients.
ORTHOGONAL TASK GRADIENTS
In this work we present novel gradient based regularization term that orthogonalizes the interaction between multiple tasks. We define multi-task neural network shared encoder fa,, and set of task-specific decoders fe., for each of the I tasks T = {fi,..., tr The encoder creates a mapping between the input space X' and latent feature space IR that is used by each of the decoders to predict the task specific labels y Each the inputs in is associated to a set of labels for the tasks in T, forming the dataset D (z,, vi ... y1""
( JEN of N observations.
For task t E T we define the empirical loss L, A 1 EiEN Lo.fo, (fou (2:)), U:). The multi- task objective can be then constructed as a convex combination individual task losses using the weights W/ R:
CT
Σ wrli TET
Using gradient descent to minimize the multi-task loss in Equation rule for the parameters 0sn:
we obtain the following update
OC, 00,M
82h
8.h
7
We
TET
becomes clear that the overall success of a multi-task network is dependent on the individual task gradients and their relationship to each other. Task gradients might cancel each other out certain task might dominate the direction the encoder's parameters. We further examine the interaction between two tasks t; and tý by looking the cosine of their gradients with respect to the encoder:
OL., 0L.,
cos(t;, ty) = cos(
(3)
a0,, 00
Previous work argues that negative transfer, task interference or competition (Du et al. 2018 Sener & Koltun| 2018 Zhao al. 2018) happens when this cosine is negative, leading tasks with smaller gradient magnitudes in fact increasing their error during training. The interference between tasks lies in the competition for resources in the shared encoder fea- Based on empirical observa- tions we argue that multi-task networks not only benefit when the cosine is non-negative but more so when task gradients are close orthogonal. In continual learning setting makes sense for task gradients to be as aligned possible in order to avoid catastrophic forgetting (Riemer et al.) 2018). multi-task setting is however unclear whether maximizing the transfer between tasks also leads to superior solution for all objectives, especially since there is risk of forgetting. In our experiments we observe a higher performance when orthogonalizing correlated tasks on the SUN RGB-D dataset. [tannley et al. (2099) make a similar finding that learning related tasks does not necessarily improve the musti-task optimization.
In approach minimize the squared cosine during training which diminishes competition as each task will be able to optimize different parameters of the encoder. By also orthogonalizing positive transfer the encoder will produce richer feature space and multi-purpose representations.
3

---

Under review as a conference paper at ICLR 2020
To minimize the cosine between two task gradients simply add the squared cosine the multi- task objective function from Equation [ penalty weight:
with an additional hyper peeem--rrmtter a E R to adjust the
Cu,,, u Lt. + Wi, Cr2 + cos (t,
(4)
We can generalize Equation[4jn 7 tasks by taking the squared Frobenius norm of the cosine distance matrix between gradients. We define Va, the column vector of unit normalized partial derivatives of the task losses with respect to 0sh- The distance matrix for I" tasks (cos"(4; Lj ( can be then efficiently computed by taking the outer product of Ve,, with itself. Subsequently we subtract the constant distance between identical tasks and normalize by accounting for the matrix symmetry as well as the number of task pairs in order to ensure the same bounds for the penalty term.
ahi, dh,,
ahr 30sh
Vo.- =
80th
08m
CT wrCt + T(T -(T-1)
V Bx
Von
11
(5)
ET
The above equation generalizes the gradient regularization term for I" tasks and maintains its range within [0, 1]. Even though W, are and will therefore treat them constants. In all the experiments we add our penalty term to the naive approach of having all tasks equally weighted. Computing the regularization term for each layer in the shared encoder is computationally prohibitive, so in practice we restrict ourselves to computing the loss with respect to only the last layer of the encoder.
our focus is solely on the cosine regularization
Finally, we will refer to (t.,t,) as the distribution of cosines between the gradients of I, and Lj
throughout training, having mean (t),,t,) and standard deviation (t,,t)) )- Our gradient regularization method minimizes (t,,tt)) which will be empirically shown later on.
4 EMPIRICAL ANALYSIS
To illustrate our findings will use the MultiDigitMNIST dataset (Sun 2019) with minor alter- ation to make it more suitable for multi-task learning. MultiDigiMNNIST is dataset constructed by positioning
MNIST digits side by side at an arbitrary location within its half, varying in style and orientation as in the original MNIST dataset. The resulting tasks are classifying the left and right digits in the image, denoted tie and fright respectively. We modify the setup by choosing subset of possible digits for each task, cre- ating two disjoint sets of labels. This allows the tasks to related, as both classify digits, but at the same time avoiding redundancy since each task is optimized on different labels. By making this modification we encourage each decoder to learn task specific features, while still taking advantage of the shared filters trained for generic digit classification. For our experiments we have assigned even digits to tie and odd digits to frigles as shown in Figure The resulting dataset contains 16000 images for training, 4000 for validation and 5000 for testing. It is worth noting that even though the combination of left and right digits are random, it is ensured that individual pairs of digit classes are only present in dataset split. This guarantees that the network evaluated new digit pairings rather than pairings seen during training.
image of 64 by 64 pixels. Each digit is located
9
6
5
2
7
3
8
Figure 1: Sample images from the modified multiDigitMNIST dataset (Sun 2019 Only even digits
are assigned to the while tyighr contains odd numbers. The same combination of digits in an image does not appear in multiple dataset splits.
4

---

Under review as a conference paper at ICLR 2020
Regulaiizaiion
Regularization Batchrorm
None
0.005
Ratchoomm
Drapout
Dropout
0,000
Probabiitt Density
0.873
Accuracy
0.800
0.825
0,800
0,733
0.75
1,00
013
0.2
0.4
0.5
0.6
0.7
0.8
09
cas (Tigs Fright)
0 (tggt/aggt
Figure 2: Cosine distribution between task gradients (tier, tight) during training. Reg- ularization methods implicitly orthogonalize task gradients.
Figure 3: Evaluation of models with different syyee-parametee and random seeds. The fi- nal validation accuracy is plotted against the standard deviation of gradient cosines from the first training epoch.
We perform our experiments using convolutional neural network architecture. The shared encoder consists of two convolutional layers, while the decoders have convolutional and two fully nected layers. The decoders contain convolutions in order to encourage the learning of task -peciic filters. For simplicity our convolutional layers lack bias terms and use stride to replace maxpooling layers. We have tested the configuration with bias terms and max maxpooling layers and do not encounter noticeable difference. Training is performed using the cross-entropy loss and the Adam (Kingmal & Ba 2014) optimizer. To evaluate the overall performance of a model we measure the harmonic mean of the accuracy obtains for both tasks on the validation set.
To derive an accurate picture of the interaction between task gradients we perform our analysis on variety of instances using a range of syper-parameters and random seeds. We vary the network capacity by having different number of filters in the convolutional layers, thus allowing different ratios of resources between encoder and decoders. The training being varied by iterating over dif- ferent batch sizes and learning rates. Each unique configuration is being evaluated on five different initializations.
We compare our method with both regularized and unregularized baselines and show the cosine distributions (e.,.) during sample training in Figure It can observed that the unregularized model displays distribution with high variance, where gradients frequently form sharp and obtuse angles. Intuitively having gradients in opposite directions will hurt performance, while having them in the same direction raises questions about the usefulness of optimizing multiple objectives. On the other hand Dropout and Batchnorm seem to implicitly reduce the variance of these angles, favoring orthogonality between task gradients. This confirms the findings of [Santurkar et al. (2018) that Batchnorm is having a smoothing effect on the loss surface and extends it to multi-task scenarios. Unsurprisingly, the regularized models outperform the unregularized baseline as seen in Table [] This leads to the question whether model explicitly regularized for gradient orthogonality can help the training multi-task networks. Similar the findings of Liu al. (2019a) and Santurkar et al. (011)) we observe in our analysis that high gradient variance is more prominent in the beginning of training and when using smaller batch sizes. We find that unregularized models also reduce their cosine variance in later stages of training as they reach convergence and further explore this in Appendix A Figure B shows the validation accuracy and standard deviation (t.,4,) during the first training epoch of several models. We notice that this initial standard deviation is a good indicator of the final generalization performance of a model. Additionally we observe a clear separation in terms of (tatt))) between regularization methods. We believe that by having non interfering gradients in the beginning of training, the model being guided into different regions of the search space that ultimately prove yield better local minima.
5

---

Under review as a conference paper at ICLR 2020
Based on these observations we evaluate our gradient regularization method and observe that (ta.tt)) is being successfully reduced as seen in Figures 2 and [ The evaluated model contains 20 filters on each convolutional layer, and has been optimized using learning rate of 10-3 with batches containing 64 images. The final test scores over five runs are shown in Table [T Although our method does not on seem to have major impact by itself, it significantly boosts the performance of the model when used in conjunction with Batchnorm. We believe the added complexity of the loss landscape is benefiting from the smoothing effect of Batchnorm (Santurkar et al. 2018), which is why the methods work well together. Although Batchnorm has already a gradient orthogonalizing effect on training, further regularizing them proves be beneficial. It is worth noting that in these experiments none of the multi-task approaches outperform their single task counterparts. We believe this is due to the abundance of data compared with the relative simplicity of the tasks. This stands in contrast to the results on the following benchmark datasets which have much less data compared with the amount variability displayed.
Table 1: Harmonic mean of task accuracies with standard deviation, as well as the standard devia- tion of the cosine distribution on the modified MultiDigiMMNIST dataset. Regularization methods implicitly reduce (tc.t)))
Model
Acc. (%)
(ban (aats)
Single Task
93.7 (0.00)
No reg
90.7 (0.01)
0.27
Dropout
91.0 (0.00)
0.11
Batchnorm
91.3 (0.02)
0.11
CosReg(a 10)
90.7 (0.00)
0.03
CosReg(a 0.1) Batchnorm
92.5 (0.01)
0.09
BENCHMARKS
For our experiments we use the multi-task friendly SegNet (Badrinarayanan et al. 2017) archi- tecture. The model consists of symmetric VGG16 (Simonyan & Zisserman 2014) encoder and decoders. The decoders perform upsampling using the indices obtained from the maxpooling lay- in the encoder. Due to limited number of data points the network is being initialized with the weights from a VGG16 network pre-trained on ImageNet. We use independent decorders for each task in order to evaluate the gradients on the last layer of the VGG encoder. All experiments have been implemented in Py PyTorch and run on a TITAN X PASCAL GPU machine. We compare the performance of our approach with GradNorm (Chen al. 2017) and Kendall al. (2018) as both methods are focused on the interaction between tasks rather than architecture design. It is worth reiterating that our method differentiates itself from these approaches by not scaling task gradients, hence in all experiments the task weights for CosReg are set to 1.
5.1 NYUv2
We evaluate our regularization method on the NYUv2 dataset (Nathan Silberman & Fergus 2012) of indoor scenes. The small dataset of 795 training and 654 test images includes image segmentation, depth and surface normal labels which makes it a suitable benchmark having both classification and regression tasks. The dataset is challenging as it contains indoor images from multiple view- points and under different lighting conditions displaying high variation relative to the number of data points it offers. We do not augment the dataset with auxiliary observations that have only a subset of labels, such as additional vidco frames. The original input images of 640x480 pixels are resized to 320x320, while the target images are downsampled to 80x80. This allows the model to have less memory requirements while still handling images with semantic significance.
Image segmentation. Each pixel in the target image for the segmentation task is labeled as one of 14 classes (bed, chair, window etc.) including the background. We train the task to minimize the pixel-wise cross-entropy loss, while using the mean intersection over union (IoU) metric to evaluate it.

---

Under review as a conference paper at ICLR 2020
Table 2: Task errors for the NYUv2 dataset. Lower values are preferred and the best performance for each task is displayed in bold.
Model
Segmentation mIoU
Depth Estimation RMSE
Surface Normal cos
Single Task
0.663
0.775
0.051
Equal Task Weights
0.670
0.765
0,053
Gradnorm (Chen et al. 2017
0,651
0.747
0,052
Kendall et al. (2018)
0.659
0.745
0,052
CosReg
0.646
0.747
0.053
Depth estimation. The indoor images were captured with a Microsoft Kinect which can collect depth information. Each pixel of the target image for this task is annotated with the distance in meters. We use mean squared error (MSE) loss to optimize the task decoder and evaluate using the root MSE metric.
Surface normals. The surface normals for each image were generated algorithmically and are encoded over three channels representing each axis. Predictions are normalized unit length and trained to minimize the MSE loss. A model's performance evaluated by computing the cosine between the target and predicted surface normals each pixel.
All models are trained using Adam for 25 epochs, with static learning rate of 10-1 Due to memory constraints resulting from having three decoders the variation in loss amplitude is large do to the optimization of three objective functions. In order better adapt these changes we also try scheduling a
batch size of 2. During training
for our gradient regularization method to be multiplied by the average loss of the tasks. The weight has the sole purpose of scaling the regularization loss term and is not being backpropagated through. For these experiments we use an of 10.
The results are shown in Table 2 We also evaluate the single task approach, where one model is being trained to optimize only one objective at time, not receiving any signal from the other tasks. can be seen that adopting a naive multi-task approach of assigning equal weights to each objective produces mixed results. While depth estimation benefits from the multi-task setting the performance for semantic segmentation and surface normal prediction is reduced. Similar to Kendall] et al.) (2018) and Chen et al., (2017) our method also improves the performance of the naive multi-task approach. is worth emphasizing that both CCen et al. (2017) and Kendall et al. (2018) explicitly weigh individual losses to balance the training between tasks. In contrast, method operates solely on regularizing the direction of gradients and not their magnitude, while achieving similar results.
5.2 SUN RGB-D
We further test our proposed method on the SUN RGB-D dataset (Song et al. 2015). Similar to the NYUv2 dataset it contains images describing indoor scenes but with 5285 train and 5050 test images it is significantly larger. The tasks the models trained to solve also semantic segmentation and depth estimation, but for the former we have both fine and coarse labels at our disposal. This allows us construct two similar tasks with different difficulties the coarse segmentation offers 13 classes while for the fine segmentation we are provided with 37. Having such a strong similarity between tasks will bias the mode of the cosine distribution since optimizing one task will inevitably benefit the other. The fact that the tasks have predominantly sharp angles between gradients can be seen Figure 4 We can observe that even in this situation our method successfully produces orthogonal gradients. This setup is interesting as it allows us to test our hypothesis of the benefits of gradient orthogonality even when tasks are not competition with each other.
We train the models using Adam for 25 epochs using batch size of 3 images and exponentially decay the learning rate from a base of 10-4 by 0.5 every epochs. Similar the previous setup we resize the input images to 320 and the outputs to 80x 80, but additionally augment this with horizontal flips. For these experiments we also found the model work best with an a of 10.
7

---

Under review as a conference paper at ICLR 2020
Table Task errors for the SUN RGB-D dataset. The segmentation tasks are correlated as they are performed at coarse (13 classes) and fine (37 classes) granularities.
Model
Segmentation (13)
Segmentation (37)
Depth Estimation RMSE
1- mloU
1- mloU
Single Task
0.649
0.719
0.586
Equal Task Weights
0.646
0.717
0.586
Gradnorm (Chen et al. 2017
0.648
0.723
0,606
Kendall et al. 2018)
0.646
0,716
0.582
CosReg
0.644
0.714
0.584
The results of the benchmark are displayed in Table 3 Similar NYUv2 the single task baselines are outperformed by most multi-task approaches. Al- though the dataset larger we believe each task still does not have sufficient observations to be indiffer- ent to knowledge transfer. As can be observed our method outperforms all other baselines and be- updates
Kendall
Gradnorm
Equual
Probability Density
lieve this is due to the orthogonalization from the correlated segmentation tasks. This sug- gests that tasks that are related under transfer learn- ing do not necessarily help each other during multi- task. A similar observation has been made by [Stan-
dley et al. (2019) who obtain superior multi-task re- sults by combining uncorrelated tasks.
-1.00
-0.7
-0.00
-0.25
0.00
0.25
0.50
0.5
1000
cos(Lsep13,
Eseg37)
CONCLUSION
Figure 4: Cosine distribution between corre- lated segmentation tasks. Even though the gradients are in a highly dimensional space they point in similar directions throughout training. CosReg successfully regularizes even in this circumstance.
In this work we explore the interaction between task gradients during training. Through an empiri- cal analysis on the multiDigitMNIST dataset we ob- serve that unregularized models have high variance in the angles between task gradients, while models with lower variance tend to perform better. Addi- tionally find that common regularization meth- ods such as Dropout and Batchnorm implicitly or- thogonalize gradients throughout training, thus min-
imizing task interference. Based on this finding we propose a novel gradient regularization term that explicitly orthogonalizes task gradients and obtain competitive results on the NYUv2 and SUN RGB-D datasets. Different to recent approaches, this method balances tasks by regularizing the direction of gradients rather than their magnitude.
In future work we would like to further explore the impact of gradient regularization at the begin- ning of training and measure its effects on later stages of the optimization. In our experiments on the multiDigitMNIST dataset we found correlation between the initial cosine variance and final validation score. This topic should be further analyzed evaluate the predictive power this mea- sure and if it can alleviate the need for validation set. Morcover, our gradient regularizer by investigating methods to dynamically scale the loss term. Even though provides simplicity, under the current formulation the cosine loss has upper bound independent of tasks, relying on the manual adjustment of for each domain.
would like to further improve
In line with recent research believe there is lot be gained by analyzing and influencing gra- dients throughout training. This proves to especially true in multi-tane learning where managing the interaction between tasks plays crucial role the success of model.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, (9(12)224812295) 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. JEEE transactions on pattern analysis and machine intelligence, 1788)11798 2013.
Rich Caruana. Multitask learning, Machine learning, 28(1):41-75, 1997.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. arXiv preprint arXiv:1711.02257, 2017.
Harris Drucker and Yann Le Cun. Double backpropagation increasing generalization performance. In JJCNN-91-Seatie International Joint Conference Neural Networks, volume 2, pp. 145-150. IEEE, 1991.
Yunshu Du, Wojciech M Czarnecki, Siddhant Jayakumar, Razvan Pascanu, and Balaji Lakshmi- narayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv: arXiv:1812.02224, 2018.
Robert M French. Using semi-distributed representations to overcome catastrophic forgetting in connectionist networks. In Proceedings of the 13th annual cognitive science society conference, pp. 173-178, 1991.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502,0310 2015.
Khurram Javed and Martha White. Meta-learning representations for continual learning. arXiv preprint wwX.v:1905.125600 2019.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908 1008:31908.03265, 2019a.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1871- 1830, 2019b.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and S Yu Philip. Learning multiple tasks with multilinear relationship networks. Advances in neural information processing systems, pp- 1594-1603, 2017.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994-4003. 2016.

---

Under review as a conference paper at ICLR 2020
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Fast scene understanding for autonomous driving. arXiv preprint arXiv:1708.0233 2017.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, (2(11):1345-111
2009.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint arXiv:1810.11910 2018.
Pau Rodríguez, Jordi Gonzalez, Guillem Cucurull, Josep M Gonfaus, and Xavier Roca. Regularizing cnns with locally constrained decorrelations. arXiv preprint 16110196 2016.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint
10X0::1706.05098, 2017.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyss, and Aleksander Madry. How does batch nor- malization help optimization? In Advances in Neural Information Processing Systems, pp. 2483- 2493, 2018
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In Advances in Neural Information Processing Systems, pp. 527-538, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576, 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955 2014.
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese Which tasks should be learned together in multi-task learning? arXiv preprint 1rXiv:1905.07553 2019.
Shao-Hua Sun. Multi-digit mnist for few-shot learning, 2019. URL https://githhii com/ shaohua011 6/Mu t. Di g tMNTS" ST
Marvin Teichmann, Michael Weber, Marius Zoellner, Roberto Cipolla, and Raquel Urtasun. Multi- net: Real-time joint semantic reasoning for autonomous driving. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 2013-1020.. IEEE, 2018
Simon Vandenhende, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: De- ciding what layers to share. arXiv preprint arXiv:/904.02992 2019.
Dániel Varga, Adrián Csiszárik, and Zsolt Zombori. Gradient regularization improves accuracy of discriminative models. arXiv preprint 1rXiv:1712.099966 2017.
Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-task learning. arXiv preprint arXiv: 1606vv1606.000
2016.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320-3328, 2014.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018
Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu. A modulation module for multi-task learning with applications in image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 401-416, 2018
10

---

Under review as a conference paper at ICLR 2020
A COSINE STD DURING TRAINING
The decrease of (t, during training on MuliiDigiMNNIST can be seen in Figure 5 As opposed unregularized models, networks using Dropout and Batchnorm start training with reduced cosine variance and maintain the values stable after the first epochs. The vanilla network also decreases the standard deviation of its cosine distribution during training but at far slower rate.
Regularization
Dropout
Batchnorm
Mad M wym M M Mamm
Figure 5: Moving standard deviation of cos (then fright) throughout training MultiDigitMNIST. The standard deviation is computed over rolling windows of 50 iterations. can observed that for the vanilla model (tan decreases training progresses, but remains at relatively high values compared to the regularized networks.


Reviews:
[{"review_id": "rkgCFiIAKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper embraces the idea that better multi-task/lifelong learning can be achieved if tasks produce gradients that are orthogonal to the gradients produced by other tasks. The authors propose an approach to regularizing learning in order to incentivize this to happen. However, as they mention themselves, the regularized loss is computationally intractable in general and they only apply it to a subset of their network as a result. Given the computational scalability concerns, it is natural to wonder why researchers in the community would adopt this approach rather than other approaches that also aim to make gradients orthogonal. \n\nThe idea of producing orthogonal gradients across tasks or examples is not new in the context of lifelong/multi-task learning. In fact, just to name a few, [1] demonstrated that noise alone can lead to orthogonal gradients, [2] demonstrated that modular neural network architectures can lead to orthogonal gradients and less interference. Additionally, sparsity naturally leads to orthogonal gradients as does the recent approach in [3].  These approaches achieve orthogonal gradients without adding a significant computational burden to learning. This paper can be greatly improved by discussing past approaches to producing orthogonal gradients and why they are theoretically / empirically worse than CosReg. \n\n[1] \"A theory for how sensorimotor skills are learned and retained in noisy and nonstationary neural circuits\". Robert Ajemian, Alessandro D’Ausilio,  Helene Moorman, and  Emilio Bizzi. PNAS'13. \n\n[2] \"Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning\". Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. ICLR'18. \n\n[3] \"Meta-Learning Representations for Continual Learning\". Khurram Javed and Martha White. 2019. \n\nAdditionally, despite much past work, I tend to think that the entire quest for orthogonal gradients is not particularly well motivated as it is missing half of the story. Orthogonal gradients only address the problem of interference during learning, but don't help maximize transfer during learning. In fact, intuitively Figure 2 showcases that CosReg diminishes transfer during learning in comparison to baselines. Some recent work, such as [4] and [5], argues that what we really want is to maximize the dot product of gradients i.e. their alignment. This perspective achieves the best of both worlds as it incentivizes orthogonality to address interference while also incentivizing positive transfer. I wonder how the authors would position their work relative to the body of work that optimizes for the gradient dot product. Why would we like gradients to be orthogonal if there would otherwise be transfer? Why focus on the cosine rather than the dot product, which naturally comes out of the first order Taylor expansion derivation for each task? \n\n[4] \"On First-Order Meta-Learning Algorithms\". Alex Nichol, Joshua Achiam, John Schulman. 2018. \n\n[5] \"Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference\". Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. ICLR'19. \n\nGiven my major concerns about the theoretical motivation and comparisons to past work, I do not find the experiments comprehensive enough to prove the value of the proposed approach to the community. At the very least, I would be interested in comparison with additional very relevant baselines and in experiments with more tasks.  \n\nUpdate After Author Feedback: \n\nWhile I really appreciate the authors providing some context about the references, I still feel like the paper would benefit from increased empirical comparison with these past approaches. Unfortunately, I don’t really follow the point they are making about why it is better to produce orthogonality vs. high dot products.  I agree that catastrophic forgetting is more of a problem related to continual learning, but I am not sure why we wouldn't want to maximize transfer even if there was no forgetting or interference.   Given my continued concerns, I am inclined to keep my score the same. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJgBsEQgqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission argues that when training the multiple objectives in a multi task learning framework, orthogonalizing gradients is beneficial toward reducing task competitions and efficient allocation of the learning capacity in the parameters.  This is demonstrated experimentally and subsequently a second order method is used for incorporating this in training. \n\nPositives: \n1. The concept is sensible, intuitive, and simple. There has a been quite few works (Sener and Koltun 2018) that similarly argue analyzing/regularizing the gradient direction when training multiple task objectives is beneficial. This submission reinforces those. \n\n2. The implementation of the concept using a second order method is sensible and simple.  \n\nWeaknesses: \n1. Missing comparisons: As reiterated above, several existing works have augmented training of neural networks with terms that are based on directions of gradients of multiple objectives. Some of them, e.g. Sener and Koltun 2018 or Du et al 2018, are very related. Though the submission cites them, no experimental comparison or convincing verbal critique of the differences are provided. The experiments could have included baselines that correspond to the concepts proposed in those prior papers to support the novelty of this submission. If the authors believe the concept of those papers are basically the same as theirs, then this submission should change to an analysis paper rather than appearing to pitch a new regularization term. I would have found an analysis paper as valuable as one with a novel method, but the stance should be clear. \n\n2. I found the experiments too toy to be convincing. The MultiDigitMNIST is artificial and with limited benchmarking value as its construct doesn't really reflect the construct of multi task learning in the real world (e.g. as in the multi task vision datasets). NYUv2 dataset is more realistic, but the reported results are not clear to show significant and meaningful differences (see table 2). Particularly since the NYUv2 dataset has certain biases with imperfect ground truth from the kinect sensor which questions if differences in 0.001 range are meaningful. The authors can consider more recent and comparatively more reliable multi label datasets like Taskonomy for benchmarking. \n\n3. Inline with the above comment, qualitative results or any other method for convincing that the achieved results (on datasets other than MultiDigitMNIST) is meaningful seems crucial. \n\nFurther comments: \n4. The single task baselines in table 2 often perform inferior to the multi task baselines. This suggests the NYUv2 dataset could be too small to learn individual networks, whereas many other multi task papers often find single task baselines are hard to beat if they're not starved of parameters (e.g. see Standley 2018 \"Which Tasks Should Be Learned Together in Multi-task Learning?\"). Please clarify and/or consider using large enough datasets that allow you to benchmark under both high data and low data regimes. \n\n5. why there is no single task baseline in table 1? \n\n6. why CosReg is not included in figure 3 plot? \n\n7. A closer look at the recent works on analyzing task competitions in multi task learning could be useful, and probably supportive of the concept of this submission. For instance standley 2018 Which Tasks Should Be Learned Together in Multi-task Learning seem to suggest that tasks that are related under transfer learning setting did not help each other in multi task setting. Their observation seems related to the pitch of this paper that orthogonal gradients (ie tasks with dissimilar updates) can be optimized better. \n\n8. In page 4 authors state that computing the regularization term for all layers is complex, hence they do that only for the last layer. This seems okay to me, though I would have found an experiment demonstrating the consequences of this simplification useful (e.g. am experiment showing which layer to pick and a one-time expensive experiment demonstrating that picking one layer vs more layers is not too damaging). \n\n-------\nComments after rebuttal: \nI read the rebuttal and appreciate authors responses and the attempted improvement. It helped. I think the submission has the potential to be ultimately a useful read for the community, but at this stage more work/revision that wouldn't fit rebuttal time constraints would be needed to achieve that. I still think an experimental comparison with the related methods would be more convincing than verbal. \n\nThe added experiment on SUN RGB-D dataset is appreciated, but all results appear too close (table 2) to suggest one should adopt the proposed method. Also SUN RGB-D limits the tasks that could be learned together in a multi-task experiment given the limited number of labels per datapoint. I would have found using more recent  datasets with higher number of labels (rather than forcing to use depth+semantics which doesn't have to be a good mix, see Standley 2018) and more significant quantitate improvements more convincing. \n\nOverall I lean toward not accepting the current submission but acknowledge the potential value in resubmission after further work. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkg0F-a2Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: In this paper, the author analyzed gradient regularization in deep multitask learning. They empirically discovered a sharper concentration (low variance) in angles between the task gradient distributions could potentially improve the performance in multi-task learning. Then they proposed a new gradient regularization to enforce the gradient for each task be orthogonal. Empirical results (on Multi-digits MNIST and NYUv2 data-sets) indicate a marginal improvement, comparing with baselines.\n\nMain Comments:\n\nThe discovering in the paper sounds interesting while the work looks like preliminary and unpolished. Particularly I have the following technical and conceptual concerns:\n1.  In high dimensional geometrical space, ** the random high dimensional vectors will be orthogonal with high probability**. The authors can find this conclusion in many places, for example: \n\nhttps://courses.cs.washington.edu/courses/cse521/16sp/521-lecture-6.pdf\n(Theorem 6.3)\nhttps://www.cs.princeton.edu/courses/archive/fall14/cos521/lecnotes/lec11.pdf\n(Corollary 2)\n\nI noticed the author claimed in the paper “Based on empirical observations presented later on we argue that multi-task networks not only benefit when the cosine is non-negative but more so when task gradients are close to orthogonal.” (Page 3).  However, this empirical claim is not convincing for me.  Since gradient of the over-parameterized neural network is in very high dimension, the task gradients closing to orthogonal may happen because of \n(a) the internal high-dimensional geometry property\n(b) or the phenomene caused by the deep multi-task learning \n\nIt will be much better and more clear if the author can provide more analytical, theoretical (e.g provable bound even on linear model) or empirical (e.g ablation study) support to understand the behaviors of gradient in high dimensional problem. \n\n2. In the proposed approach, I am not clear how the author choose the **task weights** (e.g in Eq.1) “w_{t_i}” ? It seems that the author set them as hyper-parameters. Since understanding the task relations and automatically estimating their relationships is the key factor for avoiding negative transfer in multitask learning (e.g Sener [2018]), I think it is not a proper way to simply adjust them as hyper-parameters.\n\n3. In the experimental part, the proposed approach showed almost no or very marginal improved performance. The author should provide more evidences to show the utility or potentials, in order to convince the community to adopt the proposed approach. \n\nMinor comments:\n1. From equation (4) to (5) is not obvious, it will be better to provide the derivation details.\n2. The analyzed problem sounds more like the “multi-output” or “multi-label” problem (Section 3 in the paper). In the multi-task learning generally the inputs for each task X are different.\n\nOverall, I feel it is an interesting and promising direction to consider gradient regularization based approach in multi-task learning. However, the current manuscript is not mature for the acceptance.\n\n\nReference:\nMulti-Task Learning as Multi-Objective Optimization.  Ozan Sener, Vladlen Koltun, NeurIPS, 2018\n----------------------------------------------------------------------------------------------------\n\nAfter rebuttal\n\nI thank the author for detailed rebuttal and efforts for making the paper better.\n\nI accept points  (1), (2) and (5) , then I update my score to weak reject -- 3.\n\nThe main reason that I still keep the decision toward rejection is the experimental part. \n\nSince it is an empirical paper, I suggest the author either systematically show more comparisons and more datasets. \nOr the author can develop some theoretical insights/bounds (e.g point (1)) for enhancing the contribution of the paper.\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a training approach that orthogonalizes gradients to enable better learning across multiple tasks. The idea is simple and intuitive.

Given that there is past work following the same kind of ideas, it would be need to further: 
(a) expand the experimental evaluation section with comparisons to prior work and, ideally, demonstrate stronger results.
 (b) study in more depth the assumptions behind gradient orthogonality for transfer. This would increase impact on top of past literature by explaining, besides intuitions, why gradient orthogonality helps for transfer in the first place.

Decision: Reject