Independent Component Alignment for Multi-task Learning
Authors: Dmitry Senushkin
Iaroslav Melekhov
Mikhail Romanov
Anton Konushin
Juho Kannala
Arno Solin
Venue: ICLR 2022 Submitted (2022)
Keywords: multi-task learning
optimization

Abstract: We present a novel gradient-based multi-task learning (MTL) approach that balances training in multi-task systems by aligning the independent components of the training objective. In contrast to state-of-the-art MTL approaches, our method is stable and preserves the ratio of highly correlated tasks gradients. The method is scalable, reduces overfitting, and can seamlessly handle multi-task objectives with a large difference in gradient magnitudes. We demonstrate the effectiveness of the proposed approach on a variety of MTL problems including digit classification, multi-label image classification, camera relocalization, and scene understanding. Our approach performs favourably compared to other gradient-based adaptive balancing methods, and its performance is backed up by theoretical analysis.

Under review as a conference paper at ICLR 2022
INDEPENDENT COMPONENT ALIGNMENT FOR MULTI-TASK LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
We present a novel cruddient-based multi-task learning (MTL) approach that bal- ances training in multi-task systems by aligning the independent components of the training objective. In contrast to state-of of-te-of-thh-art MTL approaches. our method is stable and preserves the ratio of highly correlated tasks gradients. The method is scalable, reduces overfitting, and can seamlessly handle multi-task objectives with a large difference in gradient magnitudes. We demonstrate the effectiveness of the proposed approach multi-label image classification, camera relocalization, and scene understanding. Our approach performs favourably compared to other gradient-basee based adaptive balancing methods, and its performance is backed up by theoretical analysis.
variety of MTL problems including digit classification,
INTRODUCTION
In multi-task learning (MTL, e.g., Caruana, 1993; Doersch & Zisserman, 2017), several tasks are optimized simultancously using single model and by leveraging shared information across tasks to improve generalization and boost performance for all tasks. MTL is a key component in real-world applications, such natural language processing (Liu et al., 2019c), computer vision (Kendall et al., 2015; Kokkinos, 2017), and reinforcement learning (Parisotto et al., 2016; Teh et al., 2017). However, these systems are difficult to train since different tasks need be properly balanced, which is challenging when the multi-task gradient is dominated by the gradient of one of the tasks.
MTL approaches have demonstrated rapid progress, and in general. they can be divided into two categories. The first group of methods aims at desigming multi-task neural network architectures which should learn both task-shared and task-specifie features by using hard and soft parameter sharing (Liu et al., 2019b; Lu et al.; Houlsby et al., 2019; Pfeiffer et al., 2021). The methods in the second group focus on multi-task optimization strategies. To learn all tasks with equal importance, some methods use gradient replacement (Yu et al., 2020; Lopez-Paz & Ranzato, 2017) or balance the individual loss functions with weights sampled from uniform distribution, learned by using task-dependant uncertainty (Kendall et al., 2018), determined by utilising the learning speed for the task (Liu et al., 2a;,;;;; Lee & Son, 2020), or obtained by minimizing an auxiliary loss (Chen al., 2018). However, combining individual loss functions into one using weighted average can be hard and prone to errors if some task gradients are in conflict with each other. To alleviate this limitation, variety of multi-objective optimization methods have been proposed (Désidéri, 2012: Sener & Koltun, 2018). They aim to find Pareto-stationary solutions for which the performance of any of the objectives can only be improved by deteriorating performance of some other objective. Although these methods demonstrate good performance, their main limitation is local convergence. Specifically, if the objective function is non-convex, which typical in deep learning. the methods converge to poor Pareto-stationary solution. Moreover, there is gap between optimization and generalization performance as proven by Lin et al. (2019).
This work hinges on the realization that improving upon the overall performance in multi-task learning can be formulated by conservatism in the individual learning steps uncorrelated losses. We formulate an approach that aims to guarantee consistent improvement over dynamically constructed linearly independent tasks at every step. This results multi-objective optimization algorithm that automatically balances training in multi-task systems by aligning orthogonal components of the objective. We address the problem of a large difference the task gradient magnitudes through the condition number of the gradient matrix. This number defines the dominance rate in the multi-

---

Under review as a conference paper at ICLR 2022
H-alianedd 847.35
POGmd r(+) --7142
Actems - -05.2)
Strting point
(a) Task 1 objective
(b) Task objective
(c) Trajectories of gradient updates
Figure I: Synthetic multi-task problem. Following Yu et al. (2020), we evaluate our approach and recent baseline MTL methods multi-task optimization problem, where the multi-task objective gradient is dominated by one task. For each MTL method, we visualize the trajectory of gradient updates computed for an equal number of iterations. The proposed O-aligned method can handle the imbalance and converges faster compared to prior state-of-the-ar MTL approaches.
objective task and is obtained the ratio of maximum and minimum singular values. Fig. illustrates the proposed approach and shows its performance compared to other gradient-basee MTL methods when the difference in the task gradient magnitudes large.
The contributions of this paper are summarized as follows. (i) We present novel gradient- MTL approach that dynamically constructs consistent losses and seamlessly handles the problem of a large difference in task gradients. (ii) We provide computationally efficient approximation for the approach with constant time complexity and which is directly applicable to large-scall scale problems. (iii) We formulate a new metric, the dominance rate, an interpretable way of establishing the maximum possible dominance in the gradient system. Additionally, we perform an extensive evaluation of the proposed approach various MTI. benchmarks and demonstrate that our method outperforms prior state-of cate-of-thee the-art MTL methods.
RELATED WORK
A multi-task formulation is well motivated and ubiquitous to many application domains such as computer vision (Bilen & Vedaldi, 2016; Kokkinos, 2017: Zamir al., 2018; Nekrasov al., 2019;
Kendall et al., 2018), natural language processing (Collobert & Weston, 2008; Luong et al., 2015;
Dong et al., 2015), speech processing (Seltzer & Droppo, 2013), and robotics (Wulfmeier et al., 2020;
Levine et al., 2016). In this work we focus gradient-based malti-task learning strategies and refer the interested reader to comprehensive surveys by Caruana (1993), Ruder (2017), and Crawshaw multi-task optimization is to balance the
(2020) for more detailed overview. One way individual loss functions for different tasks. Prior methods to simultaneously learning multiple tasks parameterize the total objective weighted sum of task-specific loss functions whose weighting coefficients are manually tuned (Kendall et al., 2015; Laskar et al., 2017). However, using grid search for optimal weighting is computationally inefficient especially when the number of tasks is large. To address this limitation, Kendall et al. (2018) utilize homoscedastic uncertainty of each task to derive a weighted multi-task loss. Chen et al. (2018) propose a heuristic to tune loss weights based on task gradient magnitudes.
One of the main challenges in MTL is the presence of conflicting gradients when tasks have gradients pointing in opposing directions, i.e. negative transfer (Lee et al., 2018). Among other approaches to mitigate the problem of negative transfer, the methods based on explicit gradient modulation have shown great potential (Lopez-Paz & Ranzato, 2017; Yu et al., 2020; Chaudhry al., 2019). The core idea to replace task gradient which conflicts with some other task by a modified, non-conflicting, gradient. Yu al. (2020) propose 'gradient surgery' technique which makes the new gradient non-conflicting with the average of task gradients at the previous step. However, it requires checking multiple gradients for conflicts at each update step leading
large computational overhead.
On the other hand, multi object-o optimization (MOO) is the process of optimizing a set of possibly contrasting objectives simultaneously (Fliege & Svaiter, 2000; Schäffler et al., 2002; Désidéri, 2012:
2

---

Under review as a conference paper at ICLR 2022
Peitz & Dellnitz, 2018: Poirion et al., 2017). These methods find direction that decreases all KouuuhhhKuh-TTukee (KKT) conditions (Kuhn & Tucker,
objectives relying on multi-objective 1951). The work by Sener & Koltun (2018) extends the classical MGDA (Désidéri, 2012) method to a form that scales well to high-dimensional problems. However, it converges to a non-optimal Pareto-stationary solution if the objective function is non-convex. We empirically compare our approach to a number of MOO methods and demonstrate its superior performance.
3 PROBLEM FORMULATION AND NOTATION
Multi-task learning is concerned with finding the set of parameters for a parametric model that correspond to high average performance across all tasks. More formally, we consider MTL problem defined over the input space X' and a series of targets {)',...., ".' where T is the number of tasks. We consider parametric model f(x: 6sh 0! ..... 0")) with shared, 6sh and task-specific parameters, 0'. Each task has a corresponding differentiable loss function C' C'(g', y²): ) x ) + R. Moreover, we can include prior knowledge user preferences p())) in the model. The MTL problem can be formulated as an empirical risk minimisation (ERM) problem on a data set {(r,, y/ ..........
of i.i.d. data points:
T Σ w L' 0¹) min gah
min Bys [E (0sh, 0¹)] = min g
Σ w/ ( 0 (1) 1=1
0¹ o!.....TT
0¹
0² "!.....7""
The vector of preferences w defines the importance of each task: auxiliary task learning. We consider shared parameters, unless otherwise specified. Since the gradients of O' and 0eh are linearly independent pairwise, task-specific parameters have a single gradient, and thus can be omitted without loss of generality.
1 for MTL; w * for
4 INDEPENDENT COMPONENT ALIGNMENT
One of the key challenges in MTL is overfitting to one of the tasks caused by the dominance of the gradient of some linear component in the loss function. In practice, the overfitting might be defined by dominating direction which has the fastest learning rate compared to others. This dominance can be caused by, for example, gradient interference between the individual loss functions or significant difference in their magnitudes. Such situation is illustrated in Fig. 2a, where the gradient for task 2 dominates task Recent works on the interactions of individual tasks (Zamir et al., 201 8; 2020) demonstrate that tasks may have a common component. This means that the uniqueness of particular task is determined by another, unique, component independent to the common one. In this regard, the interference of gradients leads to faster training rate of the joint subtask reducing the efficiency of learning unique components for each task. Therefore, we need to analyze the independent components of the gradient system, rather than the individual gradients themselves. We define dominance in the system of gradients as follows:
Definition (Dominance rate). The rate of dominance of the system of loss functions is the condition number of the matrix composed of the gradients of these functions over the shared parameters:
D(C C (0) =
max
(Ge)
where Go = (V). VoL ( (2)
Tmin (Ga)
This definition based on the singular value decomposition (SVD) and has geometrie interpretation. The dominance coefficient defines the maximum possible dominance in the gradient system and is obtained as the ratio of the maximum and minimum learning rates of the orthonormal components of the gradient system (as in Fig. 2a). Recently, Wang et al. (2020) propose metric, OGR to quantitatively estimate the significance of overfitting of multi-modal networks for classification. Although related, the metric is obtained by using assumption that target distribution is well approximated by a validation data set. In contrast, our measure does not require such a strong prior and solely based on gradients with respect to the training set.
4.1 MODEL-AGNOSTIC ALIGNMENT
For effective training over all tasks, the system of gradients of loss functions should be consistent and have small dominance rate. We take this design principle, and aim for avoiding domination and
3

---

Under review as a conference paper at ICLR 2022
u2
1212
721
Ug
V.2
(11
V
V
V
(a) Unaltered task gradients
(b) Decomposed directions
(c) Aligned task gradients
Figure 2: Geometric interpretation. (a) Decomposition the original problem to orthogonal components through SVD. (b) Re- weighting the gradient directions by balancing the conditioning. (c) The aligned task gradients defining the step directions. Here, Ui o₁ are left singular vector and singular value of gradient matrix Gg. respectively.
inconsistency altogether. We propose constructing new (consistent) gradient system that best (under conditions that follow) describes the given directions, but whose dominance coefficient is equal to minimum dominance over the task gradients, i.e. unit dominance. More formally, define new aligned gradient as follows.
Definition 2 (llaligned). We define 0-aligned gradient as V C(0) 7 (Go))ow where vector w denotes a pre-selected task preference and Go the solution to the Procrustes problem:
Go arg min || Ge -Q S.t. QQ==1.
3
In order to calculate the f-aligned gradient, the solution to the general Procrustes problem is needed. Schönemann (1966) shows that the problem has analytical solution, namely:
Go -V where U. Go = UEV VVVVenn by SVD).
(4)
The analytical solution allows us to avoid the costly procedure for finding the minimum, which reduces the computational cost of our approach. Additionally, this leads the dominance rate of the resulting matrix being equal unit dominance: D(amin (Go) climinated the possibility of uneven learning of different tasks. Fig. 2c shows sketch of the geometric intuition the aligned gradients after the alignment step.
Smin 1, i.e. have duan
At this stage aim verify that standard gradient descent with f-aligned gradients corresponds to a sensible optimization procedure. We provide the following analysis, which states the convergence for general non-convex loss functions with standard assumptions on regularity. For proof and more theoretical results, see App. B.
Theorem 1. Assume L (O))).... (())......') LT (0) are lower bounded continuously differentiable functions with Lipschitz continuous gradients with A > Assume that the total loss function L(O) is weighted sum of these functions with a pre-selected vector of preferences w. Gradient descent with step size A and 0 -laiineed gradient converges the initial rate of dominance finite, D(Gg) C, and task preferences vector w is consistent with the gradient system, W 2 E.
4.2 EFFICIENT ALIGNMENT FOR ANCODER-DECODER NETWORKS
The main limitation of the f-aligned approach is the requirement of multiple backward passes through the shared part of the model to calculate the gradient matrix. The backward passes are computationally demanding, which results in linear scaling of the training time with respect the number of tasks. Therefore, it challenging to apply the method to large-scale learning problems when the number of tasks is high. To address this limitation, we propose an efficient approximation to the -laiggeed approach, obtained under stronger assumptions on the model architecture.
Suppose that the class of hypotheses consists of factorizable functions, i.e.
f(x; 0th 01..... "!....,0") = (f(g(z, 0=h)), (f)),..,((((,,, 01h))
(5)
4

---

Under review as a conference paper at ICLR 2022
where g(-, (hh) is representation function common to all tasks and f(., 0*) is a function that implements model functions for specific task. This assumption corre- sponds to encoder-decoder type neural network architec- tures which widely used in computer vision applica- tions (e.g., Kendall et al., 2018; Sener & Koltun, 2018; Zamir al., 2018).
8
We consider the original Procrustes problem (Def. 2) with new assumptions. We denote the encoded representation as Z = g ) = g 0). Suppose the Jacobian J BZ affects equally all task gradients, i.e. J = XI He
125
150
175
This assumption is strong, but it is supported by empirical evidence (as also sketched out in Fig. 3) and holds with some accuracy for wide class networks.
25
50
75
100
125
150
175
Following these assumptions, the original problem is for- mulated as follows:
|Goo - Qli 1|J Go Go - JQQI =
Figure 3: Sketch of JJJ = XI. The
Jacobian matrix J was computed for first 50 channels from the output of layer 4 of ResNet-18.
s.t. P-P-I. (6)
In practice, computing the matrix Gz requires only gradi- ents over the latent representation, which can be computed
in one backward pass. Therefore, define efficient method of gradient alignment for encoder- decoder networks as follows.
Definition 3 (Zaaligned). We define Z-aligned gradient L() = min (Gz)
az
Gz 00
where w denotes a pre-selected task preference and Gz is the solution to the Procrustes problem:
Gz arg min Gz P
s.t. P
The value of the rate of dominance in this case will strongly depend on the condition for the Jacobian. The value of the growth rate does not affect the value of the dominance coefficient:
Ad (Gz) 1.
D(a D(amin (Gz)JGz)
Ao Gz
Algorithm 1 Gradient descent with alignment both for the O-aligned and Z-aligned variants.
1: for = 1 to T do
2: 0 = 0j Vo: (irrh 0")
Gradient descent on task-specific parameters
3: end for
4: if f-alignment then
5: Compute Gg
6:
Go PROCRUSTSSSOLVERRCG
Solve problem in Def. D. Compute O-aligned gradient
7:
VaL(0) = dGw
8:
else if Z Z-alignment then
9:
Compute Gz = (8). (ZL
10:
Gz. PROCRUSTESSOOLVERGG
Solve problem in Def. Compute Z-lligned gradient
11:
VoL(O) GZw
12: end if
0ch
hrh v)V)(()
> Gradient descent on the shared parameters
procedure PROCRUSTESSOOVER(()
15: Factorize G, ---VVV
b Compute SVD decomposition > Analytical solution of Procrustes problem > Find a minimal learning rate
16:
--UV"
17:
Compute o min; Σn
18:
return G, a
19:
end procedure
5

---

Under review as a conference paper at ICLR 2022
attributes
Z aligned (ours)
Hard attributes
MGDA-UB
GradNorm
Unnertaintt
24
25.0
22.
22.0
32.
125
o'clock shadow
Black hair
16
Goetes
No beard
32 Straight
Attractive
cydtrows
Gmy makeup
Pale skin
Wearing
carrings
Baps under eyas
11
Brown hair
15
High cheokbones
27
Painty nose
35 Wearing hal
Bald
12 Busny eyebrows
20
Male
28
Receding hairline
36
Waarrng lipstick
Bangs
Chuoby
21
Mouth slighty open
Rosy cheeks
37
Wearing neckiace
Tes
38
Eyeglasses
Narrow eyes
Smiling
Yourg
Figure 4: Performance of MTL methods on CELEBA. The percentage error per attribute on the
CELEBA data set presented as radar chart, where tighter is better. Following Sener & Koltun (2018), we divide attributes into two categories: easy (left) and hard (right). See Sec. 5.2 for more details. We present here the methods with comparable time complexity for clear comparison.
Even though our observations confirm that the Jacobian often satisfies our assumptions (see also Sec. 5), we prove (proof of the theorem below in App. B) that gradient descent with Z-aligned gradients will converge increases the stability of the proposed approach because in worst case it behaves similarly to gradient descent without alignment.
milder setup even if our requirements are not met. Our formulation
Theorem Assume C ()),...... CT (0) are lower bounded continuously differentiable functions with Lipschitz continuous gradients with A > Assume that the total loss function L(0) is weighted sum of these functions with pre-selected vector of preferences w. The gradient descent with step size < x and Zaaligned gradient converges if the Jacobian J OZ is full rank, the initial oB rate of dominance is finite, i.e. D(Gz) < Co and D(J) < Cx,, and task preferences vector w is consistent with the gradient system, [ w|l 2
Alg. summarizes the approaches for the O-aligned and the Z-aligned multi-task learning methods.
EXPERIMENTS
empirically demonstrate the effectiveness of the proposed approach on a number of different multi-task learning benchmarks. Specifically, we tackle the problems of scene understanding, multi- digit, and multi-label classification by evaluating our method on the CITYSCAPES (Cordts et al., 2016), MULTIMNIST (Sabour et al., 2017). and CELEBA (Liu et al., 2015) data sets, respectively. Furthermore, we experiment with image-based localization by jointly predicting camera orientation and translation the 7SCENES data set (Shotton et al., 2013).
Baselines. In this work, we consider the following MTL baseline methods: (1) uniform scal- ing: optimizing a uniformly weighted sum of individual task objectives, i.e. +. (2) Un- certainty (Kendall et al., 2018): using uncertainty to find weighting coefficients for each task; (3) MGDA (Désidéri. 2012): using the gradients of each task to solve an optimization problem for update; (4) MGDA-UB (Sener & Koltun, 2018): optimizing upper bound for the MGDA optimization objective; (5) GradNorm (Chen et al., 2018): normalizing the gradients to balance the learning of multiple tasks, and (6) PCGrad (Yu et al., 2020): performing gradient projection avoid

---

Under review as a conference paper at ICLR 2022
Table MULTIMNIST multi-task results. For each task, highlight the best performing method in bold. We also report the coefficient of dominance for each MTL approach.
Dominance rate ' (Def. 1)
Accuracy
Method
Mcan
Std
Max
Min
Task-L [%]
Task-R [%]]
MGDA
1.63
1.45
38.03
1.004
96.83
95.06
MGDA-UB
1,41
0.36
4.00
1.004
96.76
94.96
PCGrad
1.45
0.37
3.76
1.005
97.04
95,29
GradNorm
3.12
0.83
8,91
1.704
96.51
95.31
Uncertainty
1.47
0.44
5.17
1.007
97.03
95.06
0-aligned (ours)
1.00
0.0
1.00
1.000
97.10
95.45
Z Z-aligned (ours)
1.36
0.29
3.17
1.005
96.98
95.45
the negative interactions between tasks gradients. The proposed approach and the baseline methods implemented using the Py PyTcc framework (Paszke et al., 2019).
Analysis of computational complexity. The main com- putational cost of all algorithms depends on the number of backward passes over the shared parameters for each task and for each iteration of the gradient descent. We denote the number of tasks T and the number of iter- ations the Tran-Wolfee solver (Jaggi, 2013; Sener & Koltun, 2018) K. By Def. 2, the 9-aligned gradient requires only the precomputed matrix Gg and therefore, performs T backward passes on one step. In contrast, the Z-aligned gradient can be computed in a single pass, which is the best possible case. The computational cost for each MTL method is provided in Table 1. We consider the time complexity of the SVD decomposition constant.
Table I: Computational cost per step for
T tasks and K steps.
Method
Cost
MGDA
O(T + K) O(K)
MGDA-UB
PCGrad
O(T)
GradNorm
O(T)
Uncertainty
0(1)
O-aligned (ours)
O
Z-aligned (ours)
O(1)
5.1 SYNTHETIC MULTI-TASK PROBLEM
To illustrate the proposed method on simple problem, we consider multi-task optimization objective consisting of two tasks presented in Fig. which was originally proposed by Yu et al. (2020). Specifically, the multi-task gradient of the objective is dominated by the gradient of one of the tasks leading to suboptimal training dynamics. As result, the standard optimizers (e.g., Adam) improve only the dominating task (the magenta trajectory in Fig. 1c) and struggle to optimize the whole objective. In contrast, the proposed approach is more stable and outperforms MGDA (Désidéri, 2012) and PCGrad (Yu et al., 2020). Morcover, it demonstrates favourable convergence rate compared to the baselines: 5k and 500k iterations are required for our method and PCGrad, respectively.
5.2 MULTI-TASK CLASSTFICATION
We tackle the problem of multi-task classification and evaluate the proposed method on MUL- TIMNIST digit classification and celebrity face image data CELEBA.
Following the same procedure proposed by Sener & Koltun (2018), we convert digit classification into a multi-digit classification problem. Specifically, for each image of the original MNIST data set, we randomly sample a different image from uniform distribution. The two images are then overlaid in such a way that one of the images is put at the top-left and the other one bottom-right. Similarly to Sener Koltun (2018); Yu al. (2020), we generate 60k such images and directly evaluate the proposed approach and the baselines. The resulting task in the multi-task classification problem to classify the digit on the top left (task-L) and bottom right R), respectively. The results are summarized in Table 2. The proposed MTL approach O-aligned surpasses all baselines by a noticeable margin: up to +0.06 % for Task-L and +0.16% for Task-R R compared to the best performing contender. To emphasize the significance of the gradient dominance problem and its influence on the results, we report the coefficient of dominance (Def. 1) for each method, which shows large variability for the baselines.
MULTIMNIST.
7

---

Under review as a conference paper at ICLR 2022
Table Scene understanding and multi-label classification performance. The best score is in bold and the second best score is in italic. For CELEBA, we report the mean-tstd over 10 random seeds.
Scene understanding (CAPES)APES)
CELEBA
Segmentation mloU [%]] f
Instance error
Disparity error
L1 [px] +
MSE (up scale) +
mAce [%]
MGDA
66.72
17.02
0.3294
90.49 + 0,05
MGDA-UB
66.37
18.63
0.3212
90,91 = 0.02
GradNorm
57.24
10.29
0,3536
91.18 = 0.04
PCGrad
54.06
9.91
0,3837
91.53 = 0.05
Uncertainty
60.12
9.87
0.3314
91.33 + 0.03
D-alinned (ours)
66.98
10.81
0,3266
91.32 = 0.04
Z d-aligned (ours)
66.07
10.54
0.3213
91.36 + 0.05
CELEBA. We try our method on a multi-label classification problem by using the CELEBA data (Liu al., 2015), which consists of 200k face images annotated with 40 attributes. Each attribute can be obtained by solving a binary classification problem and thus we convert it to 40- way MTL following (Sener & Koltun, 2018; Yu al., 2020). For all of the methods we use the same CNN architecture based on ResNet-18 18 as proposed by Sener & Koltun (2018). The binary classification eтrоr averaged over all 40 attributes used for evaluating performance and is reported in Table 3 (mean + std over 10 random seeds). Our approach achieves 91.36% classification aceuracy, which outperforms other strong methods such as MGDA-UR (Sener & Koltun, 2018) and Uncertainty (Kendall et al., 2018), indicating the effectiveness of the proposed method when the number tasks is large. The percentage of error per attribute illustrated in Fig. 4.
5.3 SCENE UNDERSTANDING
We also apply the proposed MTL approach to understanding. Specifically, we aim to jointly solve the problem of semantic segmentation, instance segmentation, and depth estimation. We follow Sener & Koltun (2018) and the CITYSCAPES data sets (Cordts et al., 2016) consisting of video frames captured in the streets of urban cities and labelled with instance, semantic segmentations, and depth maps. Similarly to Kendall al. (2018), we use a CNN model with UNet architecture comprising shared encoder based on ResNet-50 and three independent task-specific decoders. In our experiments, we resize all training and validation images to x522 and use standard pixel-wise loss functions for each task: negative log likelihood loss for semantic segmentation and L1 loss for instance segmentation and depth estimation. Table 3, we benchmark our approach in all three tasks and show that the proposed method performs favourably compared to other state-of-the-art MTL methods.
5.4 CAMERA RELOCALIZATION
Finally, we evaluate the performance of the proposed MTL approach the task of image-based localization. Image-based localization, or camera relocalization refers to the problem of estimating the oofgree-of-freedon (DoF) camera pose from visual data with respect been shown that camera relocalization can be solved by casting it as a regression problem (Kendall al., 2018; Laskar et al., 2017; Melekhov al., 2017: Walch al., 2017: Noha et al., 2018), where a 7-dimensional camera pose vector p [t, r] is directly estimated by CNN. The camera pose vector p consists of a translation component t = [4, 12, la] and an orientation component [41, 92, 43. 44), represented by a quaternion. Following Kendall et al. (2018), we utilize the Microsoft 7SCENES (Shotton et al., 2013) data set which a common benchmark for indoor pose regression methods. The data set comprises RGB-D image sequences of seven different indoor scenes from a handheld Kinect camera, and each scene provides training and testing image sequences consisting of 1000-7000 frames at 640x 480 resolution. Similar to Walch et al. (2017); Laskar et al. (2017), we use ResNet-34 as a backbone network in all our experiments to make the comparison between the MTL methods fair. For each method, we compute the median error of camera translation. f, and orientation, r, per scene. We also report the growth rate, R, as a measure of improvement of multi-task learning methods over the uniform scaling approach. It is defined as R = 1- & where P; the mean translation (orientation) error of MTL method; P is the mean translation (orientation)
known environment. has
8

---

Under review as a conference paper at ICLR 2022
Table Camera relocalization performance of the proposed method and existing MTL approaches for the 7SCENES data set. We report median translation t, orientation errors and the growth rate, R for each scene. For each scene, we highlight the best performing method in terms of translation and orientation error in italic and bold, respectively.
Scenes
CHESS
FIRE
HEADS
OFFICE
PUMPKIN
KITCHEN
STAIRS
Mean
R. %
f, m
0.48
1.78
0.46
0.70
0.72
0.90
0.47
0.79
uniform
5.83
11.57
13.04
8.43
6.79
8.88
11.22
9.39
t, m 0.29
0.73
0.81
0,33
0.47
0.33
0.46
0.49
+-3798
MGDA
5.90
11.84
12.59
8.67
6.37
9.08
11.46
9.42
-0.32
t, m 0.30
0.73
0.66
0,59
0.46
0.57
0.55
0.55
+30.38
MGDA-UB
5.92
11.74
12.42
9.09
6.30
8.69
11.76
9,42
-0.32
t, 0.17
0.33
0.25
0.24
0.25
0.26
0.37
0.27
+-6..82
GradNorm
7.08
13.54
14.70
9.65
8.35
9.08
14.07
10.92
11..99
t, 0.17
0.30
0.23
0.23
0.26
0.25
0.37
0.26
+-67.99
PCGrad
r, 9.10
13.17
16.19
10.57
9.00
10.35
13.71
11.73
24.92
f, 0.19
0.33
0.30
0.25
0.25
0.28
0.48
0.30
+62.03
Uncertainty
r, 9.32
16.15
17.86
10.83
11.22
10:20
14.60
12.88
33.17
t, 0.18
0.46
0.69
0.27
0.34
0.29
0.61
0.41
+48.10
O-aligned
r, 5.82
12.07
18.15
9.38
8.99
10.27
12.84
11.07
-17.89
t, m 0.18
0.48
0.43
0.28
0.27
0.32
0.42
0.34
+56.96
Z-aligned
r, 5.40
11.18
12.14
9.57
6.37
8.91
11.54
9.30
+0096
error of the uniform scaling method. The results are provided in Table 4. In terms of camera orientation error, the proposed Z-aligned approach is superior to other MTL methods for the majority of scenes. Regarding camera translation accuracy, Z-aligned can outperform MGDA-UB (Sener Koltun, 2018) and improve the averaged translation error by 0.21 m showing comparable performance to PCGrad (Yu et al., 2020). See App. D.2 for details on the evaluation pipeline and additional results.
6 DISCUSSIONS AND CONCLUSIONS
We have presented a new approach to gradient- based multi-task learning (MTL) that balances training in multi-task systems by aligning the independent components of the gradient of the training objective. In Sec. 4.1, we derived the 'O-aligned' approach and the conditions for it to converge (details in App. B). To ensure practicality, we extended our approach to incorporate heuristics motivated by common encode-decodes networks, which was formulated into the 'Zaaligned' approach in Sec. 4.2. Additionally, we formulate new metric, the dominance rate (Def. 1) that provides an interpretable way of establishing the maximum dominance in gradient system and gives insights to pre-existing MTL approaches.
We also performed an extensive set of experiments for analyzing the practical benefits of our approach. To gain insights, we provided results for a toy benchmark problem and compared our approach to other recent gradient-based MTL approaches on MULTIMNIST to study the effect of the rate of dominance. As for more pure benchmarking, we provided results for MTL problems in image classification (CELEBA), scene understanding (CITYSCAPES)) and camera relocalization (7SCENES), each being large-scale and well-known benchmarks in their respective domains. We show state-of-the-ar performance MULTIMNIST, CITYSCAPES, and 7SCENES, while the differences between methods remains small for CELEBA. The results are particularly interesting given the smaller computational requirements the Z -liigeedd method (see Table
A reference implementation of the methods presented in this paper is currently available as supple- mentary material this submission.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
This paper proposes a new method for multi-task learning, and the contributions of the paper are technical of nature. To back up the statements in the main paper, have provided proofs for the given theorems in the appendix (included after the references). Additionally, we have also provided further theorems and proof's that help the reader assess the applicability of the proposes methodology to other optimization setups.
We also include code package supplementary zip file this double-blind submission. This reference implementation helps understand technical details, and the included scripts can be used for reproducing the numerical experiments in the paper. Upon acceptance, the codes will be available as GitHub repository under the MIT license.
REFERENCES
Hakan Bilen and Andrea Vedaldi. Integrated perception with recurrent multi-task neural networks. In Advances in Neural Information Processing Systems (NIPS), volume 29, pp. 235-243. Curran Associates, Inc., 2016.7 2
Richard Caruana. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning (ICML), pp. 41-48. Morgan Kaufmann, 1993. 1, 2
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference on Learning Representations (ICLR), 2019. 2
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pp. 794-803. PMLR, 2018. 1.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning (ICML), pp. 160-167, ACM, 2008. 2
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3213-3223, 2016. 6,8
Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint
M.Xiv:2009.09796. 2020. 2
Jean-Antoine Désidéri. Multiple-gradien descent algorithm for multiobjective optimization. In European Congress on Computational Methods in Applied Sciences and Engineering (ECCOMAS), 2012. 1, 2, 6,7
Carl Doersch and Andrew Zisserman. Multi-task self superviedd visual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2051-2060, 2017.
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computa- tional Linguistics and the 7th International Joint Conference on Natural Language Processing, pp.
1723-1732. Association for Computational Linguistics, 2015. 2
Jörg Fliege and Benar Fux Svaiter. Steepest descent methods for multicriteria optimization. Mathe-
matical Methods of Operations Research, 51:479-494 2000. 2
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzębski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799. PMLR, 2019. 1
10

---

Under review as a conference paper at ICLR 2022
Martin Jaggi. Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In Proceedings of the 30th International Conference on Machine Learning (ICML), volume 28 of Proceedings of Machine Learning Research, pp. 427-435, PMLR, 2013. 7
Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real- time 6-dof camera relocalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2938-2946, 2015. 2,
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1482-7491, 2018. 1, 5, 6,
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6129 61389,618,,-- 2017. 1, 2
Harold W. Kuhn and Albert W. Tucker. Nonlinear programming. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability. University of California Press, 1951.
Zakaria Laskar, laroslav Melekhov, Surya Kalia, and Juho Kannala. Camera relocalization by computing pairwise relative poses using convolutional neural network. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 920-929, 2017. 2, 8
Hae Beom Lee, Eunho Yang, and Sung Ju Hwang. Deep asymmetric multi-task feature learning. In Proceedings of the 35th International Conference Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pp. 2956-2964. PMLR, 2018. 2
Sungjae Lee and Youngdoo Son. Multitask learning with single gradient step update for task balancing. arXiv preprint arXiv:2005.09911 2020.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 1:1334-1372 1373, 2016. 2
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. In Advances Neural Information Processing Systems (NeurIPS), volume 32, pp. 12060-12070. Curran Associates, Inc., 2019.
Shengchao Liu, Liang Yingyu, and Anthony Gitter. Loss-balanced task weighting reduce negative transfer in muulii-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9977-9978, 2019a.
Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi task learning with attention. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1871-1880, 2019b.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, volume 57. Association for Computational Linguisties. 2019c.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3730-3738, 2015. 6, 8
David Lopez-Paz. and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NIPS), volume 30, pp. 6467-6476. Curran Associates, Inc., 2017. 1,2
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lec. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR).
Minh-Thang Luong, Quoc Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. International Conference on Learning Representations (ICLR), 2015. 2
11

---

Under review as a conference paper at ICLR 2022
laroslav Melekhov, Juha Ylioinas, Juho Kannala, and Esa Rahtu. Image-based based localization using hourglass networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 370-877, 2017. 8
Vladimir Nekrasov, Thanuja Dharmasiri, Andrew Spek, Tom Drummond, Chunhua Shen, and lan Reid. Real-time joint semantic segmentation and depth estimation using asymmetric annotations. In International Conference on Robotics and Automation (ICRA), pp. 7101-7107. IEEE, 2019.
Radwan Noha, Valada Abhinav, and Burgard Wolfram. VLocNet++: Deep multitask learning for semantic visual localization and odometry. 3(4):4407-4414, 2018. 8
Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning, In International Conference on Learning Representations (ICLR), 2016. 1
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, figh-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. 7. 18
Sebastian Peitz and Michael Dellnitz. Gradient-Based Multiohjective Optimization with Uncertainties, pp. 159-182. Springer International Publishing, 2018. 3
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- Fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computatioona Linguistics, pp. 487-503. Association for Computational Linguistics, 2021.
Fabrice Poirion, Quentin Mercier, and Jean-Antoine Désidéri. Descent algorithm for nonsmooth stochastic multiobjective optimization. Computational Optimization and Applications, (2):117-311,
2017. 3
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint
arXiv: 050988 2017. 2
Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. In Advances in Neural Information Processing Systems (NIPS), volume 30, pp. 3856-3866. Curran Associates, Inc., 2017.
Stefan Schäffler, Richard R. Schultz, and Konstanze Weinzierl. Stochastic Method for the Solution of Unconstrained Vector Optimization Problems. Journal of Optimization Theory and Applications, 114:209-2221 222, 2002. 2
Peter Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31 (1):1-10, 1966.
Michael Seltzer and Jasha Droppo. Multi-task learning in deep neural networks for improved phoneme recognition. In IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 1965-6969 IEEE, 2013.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objectivve optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, pp. 527-538. Curran Associates, Inc., 2018. 1, 3, 5, 6, 7, 8, 9, 18,
Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization RGB-D images. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2930-2937, 2013. 6, 8, 18
Yuckai Sun. Notes on first-order methods for minimizing smooth functions, 2015. URL https: Whnote//note--fiictoodeeem-mm
17, 18
12

---

Under review as a conference paper at ICLR 2022
Yee Teh, Victor Bapst, Wojeiech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems (NIPS), volume 30, pp. 4499 1499-4509. Curran Associates, Inc., 2017.
Florian Walch, Caner Hazirbas, Laura Leal-Taixé, Torsten Sattler, Sebastian Hilsenbeck, and Daniel Cremers. Image-based localization using LSTMs for structured feature correlation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 627-637, 2017. 8
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12692- 127022, 2020.
Markus Wulfmeeerr Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Noah Siegel, Tim Hertweck, Thomas Lampe, Nicolas Heess, and Martin Riedmiller. Compositional transfer hierarchical reinforcement learning. In Proceedings of Robotics: Science and Systems, 2020. 2
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 5824-5836. Curran Associates, Inc., 2020. 1, 2. 7,
Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling Task Transfer Learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3712-3722, 2018. 2, 3,5
Amir R. Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjic Cao, Jitendra Malik, and Leonidas J. Guibas. Robust learning through cross-task consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11194-11205, 2020. 3
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2230-6239 2017. 20
13

---

Under review as a conference paper at ICLR 2022
APPENDIX
A NOTATION
())
th loss function
set of a shared parameters shared representation
Z
VoL
V CC
the gradient of loss function over the shared parameters the gradient of loss function over the shared representation matrix of gradients over the shared parameters matrix of gradients over the shared representation
V
Go (V) (0),
VoL
GE (V (0),
VzLT
V ((
0-aligned gradient
(())
Z-aligned gradient
V (
common notation for both aligned gradients
J
02
Jacobian of shared representation over shared parameters a manual weight vector for cumulative loss function
w
(...)
dot product of vectors
My = span(V)
linear hull spanned on the column vectors of V
||
Frobenius norm
B PROOFS OF THEOREMS
Synopsis. In this theorem we prove that the worst case performance of the Z-aligned gradient is no worse than in standard gradient descent. The constraints mentioned in the theorem always hold in practice.
Lemma 1. Assume L(0) to be continuously differentiable and VL(0) to be Lipschitz continuous
with A > 0. Then, the following restriction is valid for any descent with the step or:
a²A |||
[(O,) [(0 o (VV
Proof. First, we substitute -or. Then, according to the fundamental theorem of calculus we receive the equality:
C(0r+1) C(0.) -(0.+++)) -) L(.)) /. (VC(O, + =6), ds.
(10)
Adding and substracting the value (V(())) = So' (V.))), 8) ds leads to:
-(0++1)- = (VL())) 8) + J. (V + 56) VL(()), ds.
(11)
Since the gradient satisfies the Lipschitz condition: [ + =5) VL(0,)| < A| s5 - 0,1 and due inequality ( (x, < ||*|||||||| we can transform the integral, make the reverse substitution and get the final result:
( sds.
((0 - C(0.) -L(V.()) Q2A][r|1?
(12)
Theorem 1. Assume L' (')...., CT (0) are lower bounded continuously differentiable functions with
Lipschitz continuous gradients with A > Assume that the total loss function L(0) is a weighted sum of these functions with pre-selected vector of preferences w. Gradient descent with step size of
S K and flaaigned gradient converges if:
1. Go UEVT (given by SVD) and E diagggr .., RR) where on <C
14

---

Under review as a conference paper at ICLR 2022
2. projuy w| where Hy sVan
Proof. Following the assumptions we find that the cumulative loss function is continuously differen- tiable and has a Lipschitz continuous gradient (denote this constant A) and therefore it satisfies Lemma 1:
((O,) -(0++1) 2 a VeC(O.), - ((0.))
a²A 2
v
c(o.)|
(13)
By definition of 0-aligned gradient we get:
n V L()))))) = Σ
n
o (1, (u,, = OR
o
(w ww up)
(14)
rr'=1
r=l
n
n
(V [(0:), -(0,)))) ) = Σ
oR
(u.. ( R
Σ
w0 T
0,)2
(15)
mr==1
Since S 1/ and Vr oR
1, we can rewrite Eq. (13):
2
C(0.) - ((0+1) 2 0R
a
Ur w
ao
R
oj.
(16)
2
of
r=1
>1
=|| (w,,)
Following the assumption ds > C. Moreover 01 maxxar Then:
|G therefore oi |=
2 VeC(0.))1
((..) C(0++1)
(17)
2 w
The sequence of L(0.) is monotonically decreasing and bounded (under assumption) and therefore converges. Then ((0,) - ((0+11) - if + Dc. Therefore, we have a local convergence of gradient descent:
VoL(0)))1 < a [(0,) - 2(0.11) + as t oo.
2||w||
(18)
The same estimate appears in case of gradient descent. Therefore, the convergence of the proposed method is similar to that of gradient descent.
Theorem Assume (O).... LT (0) are lower bounded continuously differentiable functions with Lipschitz continuous gradients with A > O. Assume that the total loss function (()) is a weighted sum of these functions with pre-selected vector of preferences w. The gradient descent with step size a S 1 and Z-aligned gradient converges if:
A
1. Gz = UEV (given by SVD) and Σ diag(o
where or Co.
2. projny w| where Hy span(V).
3. J az is full rank.
08
4. J PSQ (given by a SVD) and S (Aiag(A,,.. AK). where
Cx- Ax
Proof. Following the assumptions we find that the cumulative loss function is continuously differ-
entiable and has Lipschitz continuous gradient (with constant A > 0) and therefore it satisfies Lemma 1:
a² A
((,,) - L(0 2 a VoL(O), V (()))
1 2())
(19)
2
Since < 3. we can rewrite this inequality:
((O.) - [(0.+)) 2 a
(2VoL())) v L(0)), 7 (())))
(20)
15

---

Under review as a conference paper at ICLR 2022
Consider the left vector of a dot product in the right part of Eq. (20). By definition of Z-aligned gradient we get:
R
((,,(0,) Σ OR(W yr)Jur:
(21)
Vvo((0,) v²'1L(0.) [(),)
(2o, OR)(W ( Jr)Jur.
(22)
Following the assumption J is positive definite. Any positive definite matrix congruent to diagonal with positive and ordered eigenvalues on the main diagonal. Thus replacing all eigenvalues Af with the smallest one X does not increase the right side of Eq. (19) resulting the inequality:
R
(2Vo((0)) V za 2(0.). Zo ((0))) 2 A
-1)
T w
(23)
RR
|| (w,))
Therefore:
aE OF
((0.) L(0++1)
RA
(24)
20727
Following the assumption Co and dx > Cx. Moreover 01 max and \ ||] Therefore, we get the final bound:
GGzØl |||||
CZ 2|/2|
as C2
((0,) L(0++1)
G2
(VL(0)))
(25)
2
The sequence of [(0.) is monotonically decreasing and bounded (under assumption) and therefore converges. Then L(,,) [(0,41) - if - co. Therefore, we have local convergence of gradient descent:
aC?C2Z2 ((0,) c(0...)) + 0 as t + oc.
2 w
(26)
C METHODS WITH MOMENTUM
Definition 4. A function L(x): X + IR is justrongly convex if for any x, E X
L(y) 2 C(x) + (V y) + 1 2 || - y|??.
(27)
Definition 5. A continuously differentiable L(x) X - R. is A-smooth if for any y E X:
[V - V S A| x ae--wll wll-
(28)
Lemma 2. 0-aligned and Z Z-aligned gradient can be expressed as:
T
T
R Σ
)
V L(0.) =
Wkk
s uy
(29)
Proof. We consider only the Zaliigned case. The 0-aligned case may be proved using analogous logic. According to SVD theorem, U V-V in matrix form that in vector form turns into Ur - Gzv,, Then, by definition of Z-aligned gradient:
R T = Gz.
R
ORGZ, = =0R Σ
u.
Σ
RR
U,
(30)
r=
16

---

Under review as a conference paper at ICLR 2022
If we rewrite this dot product, then get:
ORGG.
R
Up
Zz (O.).....
Σ
Σ
R Uy dr
V
(0))
(31)
Tr
Ovr
Since a JGz (.... LogJV (O.).......) (...,
Σash
ask Vol. then the final
statement follows from the definition of the Z-aligned gradient:
((O,) JGz,
wgac VoL'
Σ
Wk (o.)
(32)
Similarly, this equation be derived for O-aligned case:
Vg- ((0,) - GRGo w ΣΣ wK wrdxs (o)
Σ
Σ
wp wpdek VOL.
(33)
Theorem 3. Assume ()),)....; L (0) are strongly convex with constant μ4i 0 and smooth with
constant A₂ > O. Assume fū: - Σt ( Ek=1 wydsk) μig and &. = -i-1, (E wadxx) The heavy ball method with at (VA.- VP.) and Be = max{[1 - vapue, v Vn and the O f-aligned or Z-uligned gradient will converge linearly an optimal value L()*) if:
1. (0-aligned) Go UEV (given by SVD) and E -diagla,, ... nn). where an Co. as
2. (Z-aligned) Gz = UEVT (given by a SVD) and Σ = RR), where as Co.
3. (Z-aligned) full rank.
4. (both) projoy w|l 2 where Hv span(V).
Proof. The conditions of the theorem ensure the existence of finite and non-zero O and Z-aligned gradients. Due to existence Lemma is valid. We will not separate O-alignment and Z-alignment in this theorem due to similarity of the proving procedure. Instead we denote V L(0.) as general aligned gradient.
Consider the convergence of a heavy ball method using matrix notation following Sun (2015). The next point is computed by definition, i.e. 0.+1 = 0, az Vg ((O,) + (,, 0.-1). Then we can estimate the squared discrepancy between the current solution and the optimal point using the explicit expression for aligned gradient from Lemma 2:
0 0" 0 0"
0, - (.,
0
V L(I))
0,
0
at
0,
(. 0.-1)
0-
T Em w/ VoL (0.)
0
0"
at
(34)
Note that for all gradients for some point O on the line segment between 0, and 0* the following expression is true: C (0,) = (O)) 0*), where v² is a Hessian of a function
Lt. Denote H. Et V (O))) Hence, H, is Hessian of total loss function. Then
17

---

Under review as a conference paper at ICLR 2022
get:
0.+1 0" 0: 0
0. (0, 0.-1)
0"
(0 0*)
0, 0"
0
(1 3.)1 0r H,
3
0 0+
I
0
0 0-
(1 + 3r)1 - H,H, -
0, 0"
(35)
0 0"
By strong convexity and smoothness of (0) the eigenvalues of (O)) are in the interval [UR, Au]. Hence cigenvalues of He are between iú and Åg. Therefore, following Lemma 3.1 introduced in Sun (2015), the L2 norm matrix in the right side of Eq. (35) is bounded:
(1 B.) ar H, -Bel
$ max {|1 Vassel ----aall Varul).
(36)
0
Now, we can plug in step size at =
and Eq. (36) into Eq. (35):
(V VA.++
0.+1 0t
0, 0"
0: 0"
0 0"
<
0,11 0"
0-1
0"
(37)
2
where "/- = As Ae"
Hence the heavy ball method with this gradient will converge linearly to an optimal
value L(0*).
D IMPLEMENTATION DETAILS
In this section, we provide additional experimental results and more details on the training pipelines. The proposed approach and all bascline methods are implemented in PyTorch (Paszke et al., 2019).
D. MULTI-LABEL CLASSIFICATION
In addition to the radar chart presented in the main text, we present more detailed results on the CELEBA data set here. Specifically, we report the binary classification accuracy of each attribute for the proposed methods and the baselines on the validation split. The results are summarized in Table 5. Following Sener& Koltun (2018) we use ResNet- 18 with 40 separate fully-connected (FC) layers as task-specific functions. Table 7 shows the architecture used in the experiments in more detail. The images the data set are resized to 64x64x3 as proposed Sener & Koltun (2018). We empirically found that learning rate 10-3 gives better results. The learning rate is gradually decreased during training. All models are optimized using Adam solver and trained for 20 epochs with batch size 256.
D.2 CAMERA RELOCALIZATION
For image-based localization, we follow Kendall al. (2015) and adapt ResNet-34 architecture by removing the last FC layer but keep the convolutional and pooling layers intact. As decoder, we use two separate FC layers with 3 and outputs to regress camera orientation and translation, respectively. The network architecture is presented in Table 7. The 7SCENES (Shotton et al., 2013) data set is used for training and evaluation. The data set consists of 640 x 480 RGB-D images captured by Kinect device and cover 7 different indoor scenes. As preprocessing step. the images are re-scaled in such a way that the smaller image side is 256 pixels. All models were trained on random crops of 224x 2244 pixels and evaluated central crops. We use batch size 128 and train for 120 epochs. The learning rate is initialized to 10-3 and decreased by 10 every 40 epochs. The Adam solver used for optimization. Each method is trained 10 times with different random seeds for every scene and average performance is provided in Table 6.
18

---

Under review as a conference paper at ICLR 2022
Table 5: CELEBA performance. Following Sener & Koltun (2018), we report accuracy the validation split. The best score is in bold and the second best score underlined.
MTL methods
0-aligned (ours)
Z-aligned (ours)
MGDA
MGDA-UR
GraNNorm
PCGrad
Uncertainty
Easy altributes
Attr.
92.39
93.05
93.09
94.02
93.96
93.51
93.69
Altr.
98.79
98.74
98.69
98.80
98.77
98.77
98.84
Attr.
94.51
95.47
30
95.72
95.42
95.52
95.36
Attr.
94,93
95.22
95.45
95.16
95.39
95.38
Attr. 10
96.12
96.12
96.20
96.40
96.07
95.77
96.04
Attr. 14
96.03
96.22
96.09
96.43
96.24
96.42
96.23
Alir. 16
95.88
96.18
96.18
96. 46
96.39
96.13
96.41
Attr. 17
97.27
97.67
97.19
97.91
97.76
97.70
97.78
Attr. 20
97.25
98.01
98. 13
98.53
98.52
98.02
98.41
93.72
93.57
93.54
Altr. 22
96.13
95.91
95.60
96.20
95 99
96.11
96.05
Attr. 26
96.54
96.67
96.44
96.79
96.71
96.74
96.71
Alir. 28
94.18
94.42
94.40
94.60
94.47
94.50
94.43
Attr. 29
94,60
94.73
94.63
95.14
95.11
95.17
95.02
Attr. 30
96.00
96.56
95.97
96.79
96.86
97.02
96.94
Altr. 38
96.19
96.13
95.71
96.51
96.33
96.00
96.34
Hard attributes
Attr.
82.09
82.67
82.08
84.91
84.27
84,91
Attr_
79.94
80.14
81 26
81.36
80.50
80.61
81.30
Attr.
Att.
85,73
85.66
86,09
83.43
85,92
82.22
84,06
Alt..
81.76
81.84
82.68
X2.70
82.48
82.84
82.56
Altr.
89.48
90.67
91.35
91.42
90.45
89.45
91,28
Attr. 11
81.95
84.32
84.89
85.58
85.1
83.73
85.31
12
92.54
92.32
92.19
Alt.. 18
90.64
91.02
91.77
92.14
90.82
91.21
91.67
Alt.. 23
93.39
92.87
92.29
92.75
92.89
93,41
92.36
Allr. 25
75.60
75.14
75.57
76.14
75.87
76.04
75.84
Attr. 27
75.82
75.72
77.26
77.62
76.43
77.15
77.24
Attr, 31
91.17
92.15
92.65
93.04
92.99
92.85
92.68
83.11
82,97
Altr. 33
$1.42
83.27
85.95
85.53
K5.66
84.56
85.63
90.31
Altr. 36
91.30
91.56
92.71
92.61
92.69
92.14
92,86
Altr. 37
83.35
88.15
88.25
88.90
87.68
88.62
88.38
Attr. 39
85.67
86.03
86.93
88.13
86.94
87.07
87.58
19

---

Under review as a conference paper at ICLR 2022
Table Camera relocalization performance of the proposed method and existing MTL approaches for the 7SCENES data set. We report translation and orientation accuracy in terms of meters and degrees, respectively. The reported results are mean-istd over 10 random seeds.
(a) Translation performance (meters).
Scenes OFFICE
CHESS
FIRE
HEADS
PUMPXIN
KITCHEN
STAIRS
Mean
uniform
0.49::0.01
0.91::0.21
1.03:10.44
0.68.1 02
0.70:10.01
0.90:0.02
0.55:10.06
0.73:10.07
0.65.10.07
0.47.10.03
MGDA-UB
0.33+0001
0.73+:0.1
0.7240.16
0.58:40.03
0.48+00.3
0.64=0.03
0.52+0..2
0.57=0.03
GradNorm
0.18+0001
0.33+
0.25#:0.01
0.25#
0.23:+0.00
0.27=0.01
0.43+0.02
0,28-00.0 00
PCGra
0.17::0.0
0.33+00.01
0.241:0.01
0.24+:000
0.23=40.01
0.26:0001
0.40:20.3
0.27=00.01
Uncertainty
0.19:0000
0.37+0002
0.29:00.02
0.26:
0.26:#0001
0.29=0.01
0. 0.4620.02
0.30 0.00
0aaligned
0.26::0.02
0.53+:0.03
0.633:0.03
0.26:10.00
0.33
0.29=10.00
0.59-10.03
0.42:10.01
Z-aligned
0.20:#0.03
0.48+0.04
0.50++0.06
0.29#0.01
0.31=00.22
0.32+0.00
0.68+0.07
0.38 0202
(b) Orientation performance (degrees).
Scenes
PUMPKIN
uniform
6.26:10.22 20
11.73:10.24
13.08:10.15
8.2610.16
6.68:10.15
8.7210.21
11.81:0..24
9.51=0.07
MGDA
6.02:10.10
11.59.10.29
12.99.0.117
0.551
6.5910.09
8.8410.11
11.6110.18
9.46110.06
MGDA-UB
5.95:40.09 09
11.76:20.29
12.96:20.18
8.44
6.42:10.10
8.7320.18
11.58:0.31
9. 0.40:20.07
GradNorm
7.11=0.14
12.77=0.28
14.35+0.55
9.76=0.2
8.75+00311
9.71+0.21
13.91
10.96=0.10
POGrad
8.29:10.21
13.56:20.41
14.96:1.35 59
10.81:0.188
9.86::0.36
10.43:0.19
14.4210.34
11.76::0.15
Uncortainty
9.01+0.35
15.44+0.77
18.6941.08
10.60=0.32
10.19440.48
10.35+0.13
15.09+0.97
12.80=0.29
0aliigedd
7.47:0.49
14.40:11.36
18.9541.30
0.14:2.25
7.72110.67
9.5841.21
14. 14.0840.557
11.62:0.31
Z-aligned
0.81
12.18+0.68
13.5840.68
9.03+0.17
6.31#0.16
9.3840.15
12.20#0..2
9.79+0.17
Table 7: The network architectures used in our experiments for (1) Multi-label classification (CELEBA), (2) camera relocalization (7SCENES), (3) Scene understanding (CITYSCAPES), and (4) Mulati-task classification (MULTIMNIST), Notation: PPM stands for Pyramid Pooling Mod- ule (Zhao et al., 2017); DE depth estimation; IS and SS are instance and semantic segmentation respectively.
CELEBA ResNet-18
7SCENES ResNet-34
CITYSCAPES ResNet-50
MULTIMNIST LeNet
Conv, 7x7, 64, s=2
Conv, 3x3, 64, s=1
MaxPool, 3x3, s=2
1x1, 64
3x3, 64 3x3, 64
3x3, 64
x2
x3
3x3, 64
x3
3x3, 64
1x 256
Conv, 5x5, 10
1x1, 128
encoder
3x3, 128 3x3, 128
x2
3x3, 128
x4
3x3, 128
x4
MaxPool, 2x2
3x3, 128
1x1,
Cony, 5x5, 20
MaxPool, 2x2 FC,
1x1, 256 256
3x3, 256 3x3, 256
3x3, 256
x2
x6
3x3, 256
x6
3x3, 256
1x1, 1024
1x1,
3x3, 512 3x3, 512
3 xx3, 512 3x3, 512
x2
x3
3x3, 512
x3
1x1, 2048
PPM(1) (DE) PPM(2) (IS)
FC, 3 (Translation) FC, 4 (Orientation)
FC, 10 (Left)
decoder
(FC, 2)x40
FC, 10 (Right)
PPM(19) (SS)
20


Reviews:
[{"review_id": "ypCGZW5AmHv", "reviewer": "Reviewer_nRWJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of the paper is the introduction of the dominance rate, which allows the approach to avoid task overfitting. Another strength of the paper is the  technical presentation that is quite detailed. The z-alignment is also a strong point of the paper due to the solution to the limitation of the linear scaling of the theta-alignment counterpart. \n\nRegarding the weaknesses of the paper I would point out the first claim that the method provides consistent improvements over all tasks. This claim is not backed up by the experiments. First at Table 2 the authors claim a noticeable improvement of 0.06% for Task-L and 0.16% for Task-R. I believe the results difference to the SOTA method are marginal and, consequently, hard to draw any conclusions. Besides that I suggest the authors to include the work from Lee et al. [1] which proposes a single gradient step update for task balancing that shows better results than pareto-MTL and GradNorm with much higher differences than 0.06 and 0.16%, respectively.  \n\n[1] Sungjae Lee and Youngdoo Son. Multitask learning with single gradient step update for task balancing. arXiv preprint arXiv:2005.09910, 2020. \n\nThe Camera relocalization results shows a slight improvement over the compared methods for orientation. Nervertheless, I believe the 15% orientation improvement over GradNorm is not enough due to the 25% performance degradation when compared to GradNorm. Besides that I missed the reference to Radwan et al. [2] which shows much more accurate results for the 7-SCENES dataset. The compared and presented results are too far from been competitive with the cited paper.\n\n[2] Noha Radwan, Abhinav Valada, Wolfram Burgard. VLocNet++: Deep Multitask Learning For Semantic Visual Localization And Odometry\nIEEE Robotics And Automation Letters (RA-L), 3(4):4407-4414, 2018. \n\nWith the inclusion of the referred papers will be possible to better verify the claims related to performance and the comparison to non MOO methods make the method a more generalist approach.\n\nAnother suggestion is related to benchmark the proposed method in a high cardinality dataset (hundreds of tasks). Datasets like \"PubChem BioAssay Dataset Study\" count with 128 tasks that is ideal to check the scale limitations of theta-alignment and the z-alignment capabilities. \n", "summary_of_the_paper": "The authors aim to perform gradient task balancing on MTL setting by aligning independent components of the training objective. They claim to be able to handle large differences in gradient magnitude. Not choosing the faster learning rate dominating direction prevent task overfitting. The main contribution is based on the analysis of the individual components of the gradients instead of the individual gradients using the dominance rate. Tasks should have small dominance coefficients. Through a series of experiments the authors show the proposed technique is competitive with other multi-objective optimization approaches.    ", "summary_of_the_review": "The paper technical contribution is interesting, however the experimental section show the performance gains can be marginal and claims regarding results should be revised. Additional references and experiments are needed to draw conclusions regarding the actual empirical contributions of the method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kubQUffVgW9", "reviewer": "Reviewer_PcEb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths \n\n* The authors deal with a very interesting problem, that of multi task learning in the presence of objectives with a large difference in gradient magnitudes\n* They propose a way of quantifying the imbalance \"The rate of dominance\" and use it to propose an elegant solution based on the SVD of the gradients of each loss, and an approximate version that is only computed on the shared parameters. \n* the authors compare against recent works and show some empirical gains on sommon multi-task scenarios\n\n### Weaknesses, questions and notes\n\n1) In the illustration of figure 1, we see that there is a common solution for the two tasks in parameter space (towards the top left corner). Is it always possible to \"guarantee consistent improvement over all tasks\"? what if two tasks are very different (eg learning jointly for rotation invariance and for 6-9 digit recognition). \n\n2) In section 3 the authors say: \" task-specific parameters are independent of each other and shared parameters, and therefore have a single gradient, and thus can be omitted without loss of generality.\" - Isn't this assumption very strong? I understand (and it seems) that that such an approximation indeed works, yet, the fact is that this might not be true in practice\n\n3) Notation in Fig 2 is defined way after the figure is referenced in the intro of sec 4 and this makes the fig hard to understand. Adding some explanation for $u_i$ and $\\sigma_i$ would make it better.\n\n4) it is unclear to me what \"vector w denotes a pre-selected task preference\" means. Is this a hyperparameter weight per task? how is it set?\n\n5) Beyond differences in the applications and tasks, the following is a missing related work:\n> Wang, Weiyao, Du Tran, and Matt Feiszli. \"What makes training multi-modal classification networks hard?.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\nAuthors should discuss the relation of the proposed dominance to the overfitting-to-generalization-ratio (OGR) from Wang et al.\nIdeally, a comparisson to GradientBlending would be great.\n\n6) In algorithm 1 line 13: why are only the $\\theta_{sh}$ parameters updated? how about the update of $\\theta_i$\n\n7) the task specific parameters seem to come as \"heads\" on top of a common shared backbone g() - Would this approach extend to works that use adaptors thoughout the net as in\n> Housby et al., Parameter-Efficient Transfer Learning for NLP\n> Pfeiffer et al., AdapterFusion: Non-Destructive Task Composition for Transfer Learning\n\n8) Shouldnt the subscript  G_theta in definitin 3 be G_Z ?\n\n9) I did not fully validate the correctness of the proofs of the theorems in the appendix. I think that a 1-2 sentence discussion is however needed on the theorems in the text, eg for Theorem 1, the second condition contains a lot of ndefined notation and it is really hard to understand. This relates with point 4 above and could clarify the role of vector w\n\n10) Why is  Z-aligned, an efficiency-motivated approximation (In conclusions the authors claim that this was created \"to ensure practicality\"), better than $\\theta$-aligned in many (most) cases? \n\n11) Another work that can be cited and discussed is\n> Lu, Jiasen, et al. \"12-in-1: Multi-task vision and language representation learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n", "summary_of_the_paper": "The paper presents an approach for better multi task learning in the presence of objectives with a large difference in gradient magnitudes. They study the problem and define a \"The rate of dominance\" metric that is based on an eigenvalue ratio of the gradient tensors. they then use this dominance metric as a coefficient to better \"balance\" gradients across tasks and propose an elegant solution based on the  an approximate version that is only computed on the shared parameters. They report results on common multi-task settings, and either perform on par or outperform other recent methods.\n\n==== After rebuttal ====\n\nI want to thank the authors for a complete and detailed response. The answers make sense to me, and I am still in favour of acceptance - I raise my score to Accept.\n\n", "summary_of_the_review": "Overall an interesting and elegant approach, that shows some gains. There are weaknesses, discussed above; looking forward to the rebuttal from the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CwCX2f3W5Mj", "reviewer": "Reviewer_RbSw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-\tThe idea of balancing multi-task training by aligning the independent components of the gradient of the training objective makes sense to me. \n-\tThe proposed method achieves favorable performance on multiple multi-task learning benchmarks.\n-\tThe paper is well written and easy to follow.\n\nWeaknesses:\n- The performance on some benchmarks are not that good, such as the results on CELEBA, \n\n", "summary_of_the_paper": "This paper proposes a gradient-based multi-task learning method to balance multi-task training by aligning the independent components of the training objective. Experiments conducted on several MTL problems are given. The method is claimed to be scalable, robust to overfitting, and able to seamlessly handle multi-task objectives with a large difference in gradient magnitudes. ", "summary_of_the_review": "The idea of this paper is interesting and makes sense.\nAlthough the performance improvements against existing methods are not that significant, the stable improvements on multiple benchmarks demonstrate the effectiveness of the proposed method. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on multi-task learning.  The reviewers appreciated the method based on SVD of loss gradients.  However, concerns were raised regarding empirical effectiveness and overall impact.  The reviewers considered the authors' response in their subsequent discussions.  While the methods are interesting, the concerns over their effectiveness would need to be more thoroughly addressed in order to improve the impact of the paper.  As such, it is encouraged that the authors take these suggestions into account in preparing a new version of the paper for a future submission.
Decision: Reject