Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning
Authors: Baijiong Lin
Feiyang Ye
Yu Zhang
Ivor Tsang
Venue: Accepted by TMLR ()

Abstract: Multi-Task Learning (MTL) has achieved success in various fields. However, training with equal weights for all tasks may cause unsatisfactory performance for part of tasks. To address this problem, there are many works to carefully design dynamical loss/gradient weighting strategies but the basic random experiments are ignored to examine their effectiveness. In this paper, we propose the Random Weighting (RW) methods, including Random Loss Weighting (RLW) and Random Gradient Weighting (RGW), where an MTL model is trained with random loss/gradient weights sampled from a distribution. To show the effectiveness and necessity of RW methods, theoretically, we analyze the convergence of RW and reveal that RW has a higher probability to escape local minima, resulting in better generalization ability. Empirically, we extensively evaluate the proposed RW methods to compare with twelve state-of-the-art methods on five image datasets and two multilingual problems from the XTREME benchmark to show that RW methods can achieve comparable performance with state-of-the-art baselines. Therefore, we think the RW methods are important baselines for MTL and should attract more attention. 

Published in Transactions on Machine Learning Research (10/2022)
Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi- Task Learning
Baijiong Lin"
www.iiimmallggmmmiccommmm com
Feiyang 2e3..3
120000001@maaissstteeh edu.
Yu Zhang!:4.
yo. yo. wu..hanng.amm wilggnillssmmmmmmcoo
Ivor W. Tsang32
ioor, shorttsang@gmatie com
Department of Computer Science and Engineering, Southern University of Science and Technology Australian Artificial Intelligence Institute, Universily of Technology Sydney
Centre for Fronlier AI Research, A *TAR
Peng Cheng Laboratory
Reviewed on OpenReview:
https: openreview. net/ forun? id= jj
Abstract
Multi-Task Learning (MTL) has achieved success in various fields. However, training with equal weights for all tasks may cause unsatisfactory performance for part of tasks. To address this problem, there are many works to carefully design dynamical loss/ pessgradient weighting strategies but the basic random experiments are ignored to examine their effectiveness. In this paper, we propose the Random Weighting (RW) methods, including Random Loss Weighting (RLW) and Random Gradient Weighting (RGW), where an MTL model trained with random loss /gradient weights sampled from distribution. To show the effectiveness and necessity of RW methods, theoretically, analyze the convergence of RW and reveal that RW has a higher probability to escape local minima, resulting in better generalization ability. Empirically, we extensively evaluate the proposed RW methods to compare with twelve stato-of-the-art methods on five image datasets and two multilingual problems from the XTREME benchmark to show that RW methods can achieve comparable performance with state-of-the thet-e- baselines. Therefore, we think the RW methods are important baselines for MTL and should attract more attention.
1 Introduction
Zhang & Yang 2021 Vandenhende et al. 2021 aims to jointly train several related tasks to improve their generalization performance by leveraging common knowledge among them. Since MTL could not only significantly reduce the model size as well as speed up the inference but also improve the performance, it has been successfully applied to various fields (Zhang & Yang 2021 However, when all the tasks are not highly related, which may be reflected via conflicting gradients or dominating gradients (Yu et al. 2202)), is more difficult to train an MTL model than training them separately because some tasks dominantly influence model parameters, leading to unsatisfactory performance for other tasks. This phenomenon is referred to the task balancing problem (Vandenhende et al. 2021 in MTL.
Multi-Task Learning (MTL) (Caruana, 1997
Recently, several works focus on tackling this issue from an optimization perspective via dynamically weight- ing task losses or balancing task gradients in the training process, called loss balancing and gradient balancing
"Corresponding author.

---

Since there are multiple losses in MTL, they usually are Published in Transactions on Machine Learning Research (10/2022)
methods, respectively. However, all of the existing works take Equal Weighting (EW) which uses the fixed and equal weights in the whole training process basic baseline to test the effectiveness of their methods. We think that this baseline is not sufficient and it is quite necessary to conduct a random experiment where weights are sampled randomly as a bascline to show the effectiveness of those well-designed dynamically weighting strategies, but is missing in the existing literalure.
Therefore, in this paper, we propose the Random Weighting (RW) methods including Random Loss Weight- ing (RLW) and Random Gradient Weighting (RGW) as more reasonable baselines to test loss and gra- dient balancing methods, respectively. Specifically, in each training iteration, first sample loss/ /gsr/iade weights from distribution with some possible normalization and then minimize the aggregated loss passgnradentt weighted by the random loss pass/gratte gradient weights. Although the RW methods seem unreasonable, they can not only converge but also achieve comparable performance with existing methods that use carefully tuned weights. Thus, we think the RW methods are important basolines for MTI, and deserve more attention.
To better understand the effectiveness and necessity of RW methods, we provide both theoretical anal- yses and empirical evaluations. Theoretically, show RW methods are the stochastic variants of EW. From this perspective, we give convergence analysis for RW methods. Besides, we can show that RW methods have a higher probability to escape local minima than EW, resulting in better generalization per- formance. Empirically, we invest igate lots of State-Of-The-Ar (SOTA) task balancing approaches including four loss balancing methods and eight gradient balancing methods mostly under the multi-head architecture as those previous works did. On five Computer Vision (CV) datasets and two multilingual problems from the XTREME benchmark (Hu et al.) (020)), we show that RW methods can consistently outperform EW and have competitive performance with existing SOTA methods.
In summarr, the main contributions of this paper are folde-fold.
We propose the simple RW methods novel baselines and litmus tests for MTL.
We provide the convergence guarantce and effectiveness analysis for RW methods.
Extensive experiments show that RW can outperform EW and achieve comparable performance with the SOTA mothods.
2 Preliminary
Suppose there are T tasks and task has its corresponding dataset D,. An MTL model usually contains two parts of parameters: task-sharing parameters 0 and task-specific parameters {wh Y For example, in CV, 0 usually denotes parameters in the feature extractor shared by all the tasks and th: represents parameters in the task-specific output module for task Let (D 0, v/x) denotes the average loss on D, for task t. (X)T= are task-specific loss weights with a constraint that X 2 0 for all t's. Similarly, 1 denote task-specific gradient weights
Notations.
Conventional Baseline with Fixed Weights. aggregated as a single one via loss weights as
T L(0, (W Σ Xe(DD.0.))
(1)
Apparently, the most simple method for loss weighting is to assign the same weight to all the tasks in the whole training process, i.e., without loss of generality, X = 1 for all i's in each iteration. This approach is a common baseline in MTL and it is called EW in this paper.
Loss Balancing Methods. To achieve task balancing and improve the performance of MTL model, loss balancing methods aim to study how to generate appropriate loss weights (X/TT= Eq. (1] in each itera- tion and some representative methods include Uncertainty Weights (UW) (Kendall et al. 2018). Dynamic Weight Average (DWA) (Liu et al. 2019), IMTL-L LLu et al. 2021b) and Multi-Objective Meta Learning
2

---

Published in Transactions on Machine Learning Research (10/2022)
(MOML) Ye et al. 2021 These four methods focus on using higher loss weights for more difficult tasks measured by the uncertainty, learning speed, relative loss value, and validation performance, respectively. A more detailed introduction of those loss balancing methods is put in Section 6 When minimizing Eq. { the learning rate optimizing each task-specific parameter 1/ will be affected by the corresponding loss weight N3, which the major difference between loss balancing and gradient balancing methods.
Gradient Balancing Methods. This type of method thinks that the task balancing problem is caused by conflicting task gradients and the inappropriate gradient to update task-sharing parameters, thus they solve it via generating appropriate gradient weights {Ai to balance the task gradients and make better update of f in each iteration as
T
0= - 1 Vo
(2)
Noticeably, in such type methods, the gradient weights {1P)T= only affect the task-sharing parameter @ but not task-specific parameters [the], each of which is updated by the 1-th task gradient V (D,;, 0, she).
Some representative works include MGDA-UB (Sener & Koltun 2018), Gradient Normalization (GradNorm) (Chen et al. 2018b), Projecting Conflicting Gradient (PCGrad) (Yu et al.
2020). Gra-
dient sign Dropout (GoadDrop) (Chen et al. 2020). Impartial Multi-Task Learning (IMTL-G) (Liu al.
2021b), Gradient Vaccine (GradVac) (Wang ct al. 2021). Conflict-Avesse Gradient (CAGrad) (Liu et al.
Those eight methods focus on finding an aggregated gradient by linearly combining all the task gradients under different constraints such equal gradient mag- nitude in GradNorm and equal gradient projection in IMTL-G to climinate the gradient conflict. A more detailed introduction of those gradient balancing methods is put in Section
2021a), and RotoGrad (Javaloy & Valera 2022).
Compared with the EW method, those two types of methods use a dynamic weighting process where loss/gn boss/gradicnt weights vary over training itcrations or epochs. Thus, it is natural think how about training an MTL model with random weights. Inspired by this, we propose the RW methods by randomly sampling loss/ boss/grade /raddient weights in each iteration the random experiments for loss/gradient balancing methods, re- spectively. Besides, we think RW methods are more reasonable baselines than EW as the litanus tests for MTL methods.
3
The Random Weighting Methods
In this section, we introduce the RW mothods, including the RLW and RGW methods.
We focus on the update of the task-sharing parameter as is the key problem in MTL. In the following, we mainly introduce the RLW method the RGW method acts similarly to the RLW method. For notation simplicity, do not distinguish between X and X and denote them by X Besides, denote L(O) ((()))) where the datasets {D.) and the task-specific parameters (W)) are omitted for brevity.
RT as random variables and samples them from a random distribution each iteration. To guarantee loss weights in to be non-negative, we can first sample À (A,, Ar) from any distribution p()) and then normalize X into À via a mapping f, where ʃ RT A is normalization function such as the softmax function and A T denotes simplex in RRT ie., À E T- means Et d 1 and X 0 for all t. Note that (()) is different from p()) unless f is an identity function. Finally, RIW updates the 0 by computing the aggregated gradient VoA E(O).
Different from those loss balancing methods, RLW considers the loss weights A=
In this way, the RLW method uses dynamical loss weights in the training process, which is similar to existing loss balancing methods, but RLW uses random weights instead of carefully designed ones in the existing works. Therefore, RLW is a basic random experiment for those loss balancing methods to examine their effectiveness, which indicates RLW is more reasonable baseline than the conventional E.W.
3

---

Published in Transactions on Machine Learning Research (10/2022)
Algorithm 1 A training iteration in RW methods. The random sampling process is the only difference between RW methods and the existing works. The red line and blue line are the only difference between RLW and RGW methods.
1: Input: numbers of tasks T, learning rate datasct (D.)-m: weight distribution p()), normalization function f
2: Output: task-sharing parameter 0', task-specific parameters (v))=1
3: for t 1 to do
Compute loss (,(D;; vt2): Compute gradient 9t Vele
end for Sample weights À from p()) and normalize it into X via f
b Random Sampling
Augei for t = to T do
0' = = 01
10: vi = the v dh Ad( or vi v. v/ (,2,0,
11: end for
Noticeably, the loss weights À are random variables and vary over training iterations, thus it is apparent that the gradient VoA e(O) of RLW is an unbiased estimation of the gradient E[A]TVaL(O)) where E[|] is the expectation of À over the whole training process. This means that the RLW method stochastic variant of the loss balancing method with fixed weights E[A]. In particular, if E is proportional to (,, +44, RLW is a stochastic variant of the conventional EW baseline. In Section 4 we theoretically show that RLW has better generalization performance than EW because of the extra randomness from loss weight sampling, which indicates the RIW method is a more effective baseline than FW.
Similar RLW, in each iteration, RGW first randomly samples gradient weights À from p()), then nor- malizes it obtain À via f. and finally updates the task-sharing parameter 9 by computing the aggregated gradient VgA (()). Following previous works (Sener Koltun 2018 Chen et al. 2020 Liu et al. 2021bb Javaloy & Valera 2022). can also compute the gradient with respect to the final hidden feature representa- tion output from the shared parameter instead of the lask-sharing parameter 0 to reduce the compulational cost. Thus, RGW is a random experiment for gradient balancing methods.
In this paper, we use the standard normal distribution for p(K) and the softmax function for f in both the RLW and RGW methods since it is easy to implement, has a more stable performance compared with other sampling distributions (as shown in experimental results in Section 5.4. and is as efficient as the EW strategy (as shown in experimental results in Section 5.5). Besides, [[[]] is proportional to (+....
proved in Appendix B thus it is fair to compare with the EW strategy.
The training algorithms of both RW methods are summarized in Algorithm 1 The only difference between the RW methods and the existing works is the generation of loss/ gradient weights (i.e., Line 7 in Algorit I Apparently, the sampling operation in the RW methods is very easy to implement and only brings negligible additional computational costs when compared with the existing works. Note that random weights are involved in the update of task-specific parameters in the RLW method but not the RGW method (i.e., Line 10 in Algorithm
4 Analysis
In this section, analyze how the extra randomness from the loss/gradient weight sampling affects the convergence and effectiveness of the RW methods compared with the F.W strategy.
We focus on the update of task-sharing parameter and take RLW as an example for analysis, which can easily be extended to the RGW method. For notation simplicity, we simply use (,( instead (D,;; 8, vh)) to denote the loss function of task t, and define of = Ep, (V.. (D; and μ = Ex[A] in this section and Appendix For case of analysis, we make the following assumptions.
4

---

Published in Transactions on Machine Learning Research (10/2022)
Assumption 1 (First Derivative Lipschitz Contimuity). For each task t, the gradient Vl.(0) is M,-Lipschitz continuous if for any two points Oj and 02, we have
[V(((0)) (2,(02) M, 0211-
Assumption 2 (Polyal-Toojasieviez Condition). For each task i, the loss function &(0) satisfies the Polyak- Lojasiewicz (PL) inequality with constant & for any 0, we have
[e(0))1 2 (c((()) E7).
where 0f is a global minimum of (e) and f = (0)) denotes the optimal function value of (,()).
Assumption 3 (One Point Strongly Convex). For each task t, the loss function (0) is L,-nne point strongly convex with respect to a minimum O, after convolved with noise & if for any we have
(VE4l1 - ()00llpp)),,, - -0a) 2 Lello 0x112.
Note that these three assumptions are widely used in the analysis of non-convex optimization problems (Vaswani et al.] 2019; Safran et al. 2021), thus they could hold for deep MTL. models where deep neural networks used.
4.1 Convergence Analysis
In the following theorem, we analyze the convergence property of Algorithm I for the RLW method.
Theorem 1 (Convergence). Consider the RLW method with objective function L(0) 1 T e(O). Suppose that Assumptions) 7 and & holds. Define the global månimum of RLW as 0. arg ming L((); and the solution in the k-th iteration in RLW method as Og. If n, the step size equivalently the learning rate, satisfies 7 S 1/2e, where = minngegr(cs), we have
E[[(OL+1) C(0.)) 21pc) [C(0k) - -((.) + MIjK
(3)
2
where k = T 1=1 of and M max (M, Then for any positive E, B[[((()) L(0.)) S can be achieved after 2ec log (5e) iterations with v = Me, where Eo = E[C((n) L(0.)].
25c
Theorem shows that the RLW method with a fixed step size has a linear convergence up to a radius around the optimal solution, which is similar the EW strategy according the property of the standard Stochastic Gradient Descent (SGD) method (Moulines & Bach 2011: Needell et al. 2016). Although the RLW method has larger k than the EW strategy, iee,, NEW Σi= M Ei= of S K. which may possibly require more iterations for the RLW method reach the same accuracy as the EW strategy, experimental results in Scction 5.5 show that empirically this does not cause much difference.
4.2 Effectiveness Analysis
Note that converging to flat local minima and escaping sharp local minima are important in neural network training because flat local minima may lead to better generalization (Keskar al.) 2017 Chaudhari et. al. 2019). One popular way to escape sharp local minima is to inject noise into the gradient since sharp local minima has a smaller diameter than the flat one. The most representative work is SGD which can converge to a better solution than Gradient Descent (GD) Lechniques under various settings with the help of noisy gradients HHrdt et al., 2016 Kleinberg et al., 2018). Inspired by this, we analyze the effectiveness of the RLW method from the perspective of stochastic optimization as follows.
It observed that for both EW and RLW methods, the stochastic gradient can be cast as a noise injected for the full gradient, Le., the update step
M 02 - E(Ox) + Ex),
(4)
5

---

Published in Transactions on Machine Learning Research (10/2022)
where Çk is noise with E[[x] 0 and ||&,1] S r, and denotes the intensity of the noise. For EW method, the noise Ea is caused by the randomness of data sampling. While for RLW method, the randomness is not only from data sampling but also from weights sampling, thus the RLW method can have a larger noise & with a larger r than the EW strategy (refer to Appendix A.3
In the following Theorem we show that under the MTL setting, both EW and RIW methods have high probabilities to converge to local minimum 0. with a radius due to the injected noise &n in the update step (i.e., Eq. (
Theorem 2. Suppose on intermediale sequence upk = 0x V e(0k) ((00) and the update slep of Og is as Eq. & 1 log iterations with of S Ma. with probability at p
4.. Then under Assumptions 1 and after K
eeast 1 - 6, we have
233
1lpx 1. v
pô
where 5o = Elll:0 - -o,113, L = minigesT(Lih M maxistsT ?
2nL m2, and B = 7
Based on the definition of you and the update step of 0g in Eq. 14 we have
Ee, [px+1] = =Pk VIVES (1 l(vx - n(Ex)]
=Pk VIVE (M" e(Os- 7 VV e(0x) + Ex))
LPk VIVIE Iu e(0x+1))
Thus the sequence {1Fx} can be regarded as an approximation of using GD to minimize the function Ee, [e {(O)], which indicates that if the sequence {fxx converges local minimum 0. according to Theo- rem 2 Egg [M e(0]] can reach to the same minimum. Therefore, the parameters that lie in the neighborhoods of 6. with a neighborhood size have similar loss function values. Since the RLW method has a larger r than the EW method, thus RLW can converge to a flatter local minimum and achieve better generalization performance than FW.
5 Experiments
In this section, we empirically show the proposed RW methods can converge to a flatter minimum than the EW method in a toy example and then evaluate the proposed RW methods on five computer vision datasets (i.e., NYUv2, CityScapes, CelebA, Office-11, and Office-Home) and two multilingual problems from the XTREME benchmark (Hu et al., 2020). All the experiments are conducted on one single NVIDIA GeForce RTX 3090 GPU. The experimental results on the CityScapes, CclebA, Offico-31, and Office-Home datascts are put in Appendix C
Compared Methods. The bascline methods in comparison include several SOTA task balancing methods as introduced in Section including four loss balancing methods (i.e., UW, DWA, IMTL-L, and MOML) and eight gradient balancing methods (i.e., MGDA-UB, GradNorm, PCGrad, GradDrop, IMTL-G, GradVac, CAGrad, and RotoGrad). For all the bascline methods, we directly use the optimal hyperparameterss used in their original papers. The implementations of the RW methods and the baseline methods are based on the open-source LibMTI. library (Lin & Zhang 2022
Network Architecture. The network architecture we used adopts the Hard-Parameter Sharing (HPS) pattern (Caruana 1993), which shares the bottom layers of the network for all the tasks and uses separate top layers for each task. Other MTL architectures are studied in Section 5.7
Evaluation Metric. For homogeneous MTL problems (e.g., the XTREME benchmark and Office-31 datasot) which contain tasks of the same type such as the classification task, we directly use the average performance among tasks as the performance metric. For heterogencons MTL problems (e.g., the NYUv2 dataset) that contain tasks of different types and may have multiple evaluation metrics for each task, by

---

Published in Transactions on Machine Learning Research (10/2022)
following (Maninis al. 2019: Vandenhende et al, 2021), the average of the relative improvement over the E. method on each metric of each task the performance measure, which is formulated as
(-1) (Mz.
A 100%
1,
FW
where N2 denotes the number of metrics in task t, Mr, denotes the performance task balancing method for the nth metric in task t, M,W is defined similarly for the EW method, and Pt, is set to if higher value indicates better performance for the nth metric in lask and otherwise 0.
5.1 Results on Multi-Fashion-MNISS Datasets
Here, we take the RLW method as an example to empirically confirm whether the RW methods can converge flatter minimum than the EW method.
Multi-MNIST
Multi- FashionMNiST
Multi- Multi-(Fashion MNNIST
9.
85
85
CW RUW
Test Accutety of DR0%)
Test Acounccy Thcck BRUS)
85
80-
75
80-
X
x
70
75
X
60
65
55
60
80
70
95
Test Accuracy Task TL(%)
Test Accuracy of Task TL1%)
Test Accuracy of Tesk TL(%)
Multi- MNIST
Multi-FashioMNNIT
Men--rrashionnMMIITT
May w / aar Auseun
Naximum Eigenvalue
Maximum Eigenvclue
Yw Anwnn
/ yry wn
1.0
Maximum
My Mm
/
0.0
0.9
RLW
0.6
0.6
0.4
20
90
100
40
80
100
100
Fpochs
Number Epochs
Epochs.
Multi-MNIST
Multi-FashionMNIST
Multi-(Fashin+MMNSST
50
Gradient Variance
40
Crndlent Vartance
Mw My
Grudient Variance
40-
40
30-
RUY
30
30
20
40
60
80
100
0
20
40
60
80
100
20
40
60
80
100
Number of Epochs
Number of Epochs
Number of Epochs
Figure 1: Results of the EW and RLW methods the Multi-MNIST,, the Multt-FFsshonMNNIST and the Multi- (PassionITs) datasets. Three figures in each column show the test accuracy of every task at each epoch, the variation of the maximum eigenvalue of Hessian within the training epoch, and the variation of the gradient variance within the training epoch on each datasct, respectively.
Datasets. We consider three image classifications datasets: the MNIST (Sabour et al., 2017 the Multi-PashionMNIST, and the M datascts (Lin et al. 2019). The MNIST dataset is generated by randomly sampling two images with different classes from the MNIST dataset (Leunn) et al. 1998) and then overlaying top of another to obtain image with size 36 36. This

---

Published in Transactions on Machine Learning Research (10/2022)
dataset includes two tasks: classifying the digits on the top-left (Task TL) and bottom-right (Task BR), respectively. With the same approach, the Mulli-PashionMNISST dataset is constructed with the overlap of the PashionMNIST images (Xiao et al. 2017), and the the overlap of the FNSS and MNIST images in (Lin et al.) 2019 For both three datascts, we use 120K and 20K images for training and lesting, respectively.
dataset is constructed with
Implementation Details. These three datasets adopt the same experimental settings. Specifically, by following in (Lin et al.) 2019), a LeNet-based network (LeCun et al. 1998) except the final fully connected layer is used as the shared encoder and a fully connected layer is used as the task-specific output layer. The SGD optimizer with the learning rate as 10 and the momentum as 0.9 is used for training, the batch size is set 256, and the training epoch is set to 100. The cross-entropy loss is used for each task.
Flatness Measures. We compute two widely-used metrics to measure the flatness of a local minimum and the generalization for a deep neural network, i.e., the maximum eigenvalue of Hessian C²)( computed on the whole training dataset (Foret et al.) 2020: and the variance of gradient VZ(() over all batches on the training dataset (Jiang et al., (2220). For both two metrics, a lower value means that the local minimum is flatter and the model has better generalization. Due to the high dimensionality of model parameter A, we calculate the approximated Hessian using the Lanczos algorithm (Ghorbani al., 2019) based the open-source implementation")
Results. The results are shown in Figure [I The top row shows that the RLW method can significantly outperform the EW method on every task in these three datascts. The middle and last rows show that the RLW method has smaller maximum eigenvalue and gradient variance than the EW method. Those results indicate that the RLW method can converge to a flatter local minimum and hence result in better generalization performance, which confirms the theoretical insight in Section 1.2
5.2 Results on the NYUv2 Dataset
Dataset. The NYUv2 dataset (Silberman et al. 2012: is an indoor scene understanding dataset, which consists of video sequences recorded by the RGB and Depth cameras in the Microsoft Kinect. It contains 795 and 654 images for training and testing, respectively. This dataset includes three tasks: 13-class semantic segmentation, depth estimation, and surface normal prediction.
Implementation Details. For the NYUv2 dataset, the DeepLabV3+ architecture (Chen et al 2018a] is used. Specifically, ResNet-50 network pre-trained on the ImageNet dataset with dilated convolutions YYu eet al. 2017) is used as a shared encoder among tasks and the Atrous Spatial Pyramid Pooling (ASPP) (Chen et al. 20) module is used as the task-specific head for each task. Input images are resized to 288 x 384. The Adam optimizer (Kingma & Ba 2015) with the learning rate as 10-4 and the weight decay as 10-5 is used for training and the batch size is set to 8. We use the cross-cntropy loss, L1 loss, and cosine loss the loss function of the semantic segmentation, depth estimation, and surface normal prediction tasks, respectively.
Results. The results of different methods on the NYUv2 dataset are shown in Table ! The top row shows the performance of the widely used EW strategy and we use it as a baseline to measure the relative improvement of different methods as shown in the definition of Ap. Rows 2-5 and 7-14 show the results of loss balancing and gradient balancing methods, respectively.
According to the results, we see that both the RLW and RGW methods gain performance improvement over the EW strategy, which implies that training with extra randomness can have a better generalization ability. Besides, RLW has an improvement. of 1.04% over the EW strategy and it is higher than all loss balancing methods. As for gradient balancing methods, half of those methods have negligible or even negative improvement over the F.W strategy and RGW can outperform five of them. Compared with all

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1: Performance on the NYUv2 dataset with three tasks: 13-class semantic segmentation, depth estimation, and surface normal prediction. The best results for each task on cach mcasure over loss/gradient balancing methods are marked with superscript +/t. The best results for each task on each measure over all methods are highlighted in bold. t (4) indicates that the higher (lower) the result, the better the performance.
Segmentation
Depth
Surface Normal
Methods
mIoU
Pix Acct
Abs Err!
Rel Err,
Angle Distance
Within 1
Ap
Mean
Median
11.25
22.5
301
FW
53.77
75.45
0.3845
0.1605
23.5737
17.0438
35.04
60.98
72.07
+0.00%
UW
54.14
76.92
0.3833
0.1597
23. 23.9989
16.8691
35.33
61.37
72.48
+0.64%
Los Bul.
DWA
53.81
75.56
0.3792
0.1565
23.6111
17.0609
34.89
60.89
71.97
+0.63%
IMTL-L
53.50
75.18
0.3824
0.1596
23.3805
16.8088
35.44
61.43
72.43
+0.35%
MOMI.
54.98
75.98
0.3877
0.1618
23.2401
16.7388
35.90*
61.81*
72.76"
70.76%
RLW (ours)
54.11
75.77
0.3809
0.1575
23.3777
16.73 7385
35.71
61.52
72.45
1.04%
MGDA-UB
50.42
73.46
0.3834
0.1555
22.7827!
16.1432
36,90*
62.88
73.61
10.38%
GradNorm
53.58
75.06
0.3931
0.1663
23.4360
16.9844
35.11
61.11
72.24
-0.99%
Cradient Bal.
PCGrad
53.70
75.41
0.3903
0.1607
23.4281
16.9699
35.16
61.19
72.28
-0.66%
GradDrop
53.58
75.56
0.3855
1.1592
23.5518
17.0137
35.08
60.97
72.02
+0.0%%
IMTL-G
53.54
75.45
0.3880
0.1589
23.0530
16.4328
36.21
62.31
73.15
+0.80%
CradVac
54.891
75.98
0.3828
0,1635
23.6865
17.1301
34.82
60.71
71.81
1..0%%
CACrad
53.12
75.19
0.3871
0,1599
22.5257
15.8821
37.42
63.50
74.17!
+1.36%
RotoCrad
53.90
75.46
0.3812
0,1596
23. 0190
16.3714
36.37
62.28
73.05
+1.19%
RGW (ours)
53.85
75.87
0.3772
0.1562
23.6725
17.2439
34.62
60.49
71.75
+0.62%
baselines, RLW even higher than all of them except the CAGrad and RotoGrad methods, which indicates
the random weights can easily beat the carefully designed ones.
According the above analysis, there are two important conclusions. Firstly, the conventional EW strategy is weaker baseline than RLW and RGW for MTL. Secondly, RW methods competitive with SOTA methods and even perform better than some of them.
5.3 Results on the XTREME benchmark
Dataset. The XTREME benchmark (Hu et al. 2020) is large-scale multilingual multi-task benchmark for cross-lingual generalization evaluation, which covers fifty languages and contains nine tasks. We conduct experiments on two tasks containing Paraphrase Identification (PI) and Part-Of-Speech (POS) tagging in this benchmark. The datasets used in the PI and POS tasks are the PAWS-X dataset YYang et al. 2019 and Universal Dependency v2.5 treebanks (Nivre et. al.) 2020), respectively. On each task, we construct multilingual problem by choosing the four languages with largest numbers of data, i.c., English (en). Mandarin (zh), German (de) and Spanish (es), for the PI task and English, Mandarin, Telugu (te) and
Vietnamese (vi) for the POS task. The statistics for each language are summarized in Table 2 Different
from the NYUv2 dataset where different tasks share the same input data, in those multilingual problems, each language/tant ttask has its own input data.
Table 2: The numbers of training, validation, and test data for each language in PI and POS problems from the XTREME benchmark.
PI
POS
en 49.4
6.9K+188K33.21
zh 40.4K+2.0KK+2000
4.0K+ -2.9K
de 49.4
es
49.4KK2.0KK2.0KK
te
1.0K+0. 1K-0.
vi
14KK-0.8K-00888
9

---

Published in Transactions on Machine Learning Research (10/2022)
Implementation Details. For each multilingual problem in the XTREME benchmark, pre-trained multilingual BERT (mBERT) model (Devlin et al. 2019) implemented via the open-source transformers library (Wolf et al, 2020) is used as the shared encoder among languages and a fully connected layer is used as the langunge-spoeific output layer for each language. The Adam optimizer with the learning rate as x 10-5 and the weight decay as 10-8 is used for training and the batch size set to 32. The cross-entropy loss is used for the two multilingual problems.
Results. According to experimental results shown in Table 3 we can find some empirical observations, which are similar to those on the NYUv2 dataset. Firstly, both the RLW and RGW strategics outperform the EW method. Secondly, compared with the existing works, RLW and RGW can achieve comparable performance with existing loss/ /esr/diieett balancing methods, respectively. Even, RLW RGW methods could outperform all baseline methods. For example, RLW achieves the best performance (i.e., 90.25% average accuracy) on the PI problem and RGW achieves the best average Fl score of 91.16% on the POS problem. It is interesting to find that the performance of RLW and RGW are inconsistent different datasets. There is because the random loss weights in RLW will affect the update of task-specific parameters while in RGW, which has different influence on the performance of different datasots.
Table 3: Performance on two multilingual problems, i.c., PI and POS from the XTREME benchmark. The best results for each language over loss (oss/gradient balancing methods are marked with superscript */t. The best results for each language over all methods are highlighted in bold.
PI (Accuracy)
POS (F1 Score)
Methods
en
zh
de
88
Avg
en
zh
te
vi
Avg
EW
94.29
84.99
89.79
90.94
90.00
95.06
89.01
91.41
86.65
90.53
UW
93.74
85.44*
90.24
91.29
90.18
94.89
88.77
90.96
87.12
90.44
Loss Bal.
DWA
94.69*
84.99
89.49
91.44"
90.15
95.02
89.03
91.87
87.27
90.80
IMTL-L
93.94
84.54
89.39
91.44"
89.82
95.57*
89.93*
91.77
86.11
90.84
MOML
93.89
83.74
89.94
90.99
89.64
95.15
89.11
92.41
87.24
90.98*
RLW (ours)
94.29
85.39
89.94
91.39
90.25
95.01
88.87
92.86
86.85
90.90
MGDA-UB
94.09
84.14
89.14
90.59
89.49
94.89
88.43
91.01
86.04
90.01
GradNorm
94.19
83.59
88.89
91.24
89.47
94.88
88.80
91.78
86.96
90.61
Gradient Bal.
PCGrad
94.19
85.49|
89.09
91.24
90.00
94.85
88.42
90.72
86.71
90.18
GradDrop
94.29
84.44
89.69
90.94
89.84
95.08
89.06
90.65
87.17
90.49
IMTL-G
94.69
84.54
89.39
90.69
89.82
94.93
88.70
91.66
87.00
90.57
GradVac
94.29
84.94
89.19
90.89
89.83
94.87
88.41
90.62
86.47
90.09
CAGrad
94.34
84.59
90.09!
90.64
89.91
94.83
88.65
91.71
86.76
90.48
RotoGrad
93.99
83.89
89.29
90.94
89.52
95.44
89.79
91.42
86.33
90.74
RGW (ours)
94.55
84.99
89.29
91.40
90.06f
95.52
90.13
91.82
87.18f
91.16f
5.4 Robustness on Sampling Distribution
In this section, we evaluate the robustness of the proposed RW methods on the sampling distribution. Taking RLW as an example, we show its robustness by evaluating with five different sampling distributions (i.e., p()) for loss weights, The five distributions are uniform distribution between and 1 (denoted by Uniform), standard normal distribution (denoted by Normal), Dirichlet distribution with a 1 (denoted by Dirichlet), Bernoulli distribution with probability 1/2 (denoted by Bernoulli), Bernoulli distribution with probability 1/2 and a constraint Σt, J, = (denoted by c-Bernoulli). We set as function of f (()) = X/(EI, )e) for the Bernoulli distribution and the c-Bernoulli distribution, a softmax function for the Normal distribution and Uniform distribution, and an identity function for the Dirichlet distribution. We can prove that all the E[A]'s under these five distributions equal (.....,,) +) (refer to Appendix B thus it is fair to compare among them.
Figure 2 shows the results of the RLW method with five sampling distributions on the NYUv2 dataset in terms of Ap, where the experiment each sampling distribution is repeated for 8 times. The results show
10

---

Published in Transactions on Machine Learning Research (10/2022)
1.6
1.4
1.0
0.8
0.4
Uniform
Normal
Dirichlet
Berouuli
c-Bernoulli
Figure 2: Results of the RLW method with different sampling distributions in terms of Ap.
that the RLW method with different distributions can always outperform the EW method, which shows the robustness of the RLW method with respect the sampling distribution. In addition, compared with the uniform, Dirichlet, and Bernoulli distributions, RLW with the standard normal distribution achieves better and more stable performance. Although RLW with the c-Bernoulli distribution performs slightly better than the standard normal distribution, it is more unstable and may need longer training time as shown in Section 5.5 Thus, in this paper, we use the standard normal distribution to sample loss weights.
5.5 Convergence Speed
Here take RIW example to show the efficiency of RW methods. Figure B plots the performance curve on both NYUv2 and CelebA validation datasets to empirically compare the convergence speed of the EW and RLW methods.
On the NYUv2 dataset with tasks, the performance curves of the RIW method with two sampling distribu- tions are similar to that of the EW method, which indicates that the RLW mothod has a similar convergence property to the EW method on this dataset. As the number of tasks increases, i.e., on the CelebA dalasel with 40 tasks, we find that the RLW method with the standard normal distribution still converges fast the EW method, while the RLW method with the c-Bernoulli distribution converges slower. One reason for this phenomenon is that only one task is used to update model parameters in each training iteration when using the c- Bernoulli distribution. Thus, in this paper, use the standard normal distribution, which is efficient as the EW method.
90
-10
Average (%6)
88
-20
4,7%)
86
10
84
82
-60
RLW (Normal)
(Normal) (c-Bernoulii)
RLW (c-Bemoulli)
80
-70
20
40
80 Epochs
100
120
140
160
20
40
100
120
140
Epochs
Figure 3: Comparison the convergence speed of the EW and RLW methods on the 3-task NYUv2 validation dataset (Left) and the 40-task CelebA validation dataset (Right). We lean to use the standard normal distribution the RLW method since is as efficient as the EW method.
11

---

Published in Transactions on Machine Learning Research (10/2022)
5.6 Combination of Loss and Gradient Balancing Methods
The loss balancing methods are complementary to the gradient balancing methods. Following (Liu et al. 2021b), we train an MTL model with different combinations of loss balancing and gradient balancing methods on the NYUv2 dataset to further improve the performance. We use the vanilla EW as the baseline to measure the relative improvement the other different combinations as shown in the definition of Ap.
According the results shown in Table 4 we can see that combined with the UW, DWA, and IMTL-L methods, some gradient balancing methods performs better but others become worse. For example, A, of the GradDrop method drops from 0.08% to -0.42% when combined with DWA. Noticeably, by combining with the proposed RLW method, all the gradient balancing methods can achieve better performance. Besides, on each gradient balancing method, the improvement induced by the RLW method is significantly larger than the other three loss balancing methods as well as the EW method. Moreover, RGW can also improve the performance of loss balancing methods except for DWA. Thus, this experiment further demonstrates the effectiveness of the proposed RW methods.
Table 4: Results of different combinations of loss balancing and gradient balancing methods on the NYUv2
dataset in terms of Ap. The best results in each are highlighted in bold.
EW
UW
DWA
IMTL-L
RIW
Vanilla
+0.00%
0.64%
+0.63%
+0.35%
+1.0%
MGDA-UB
+0.38%
-0115%
+0.47%
-0.59%
+2.01%
GradNorm
-0.99%
+0.87%
-0.95%
+0.54%
+0.89%
PCGrad
-0.16%
-0.72%
+0.19%
+0.38%
+0.97%
GradDrop
+0.08%
-0.25%
-0.42%
+0.36%
+0.93%
IMTL-G
+0.80%
-0.45%
+1.20%
+0.18%
+1.50%
GradVac
+0.07%
-0.03%
+0.89%
+0.69%
+0.97%
CAGrad
+1.36%
-1.7%%
+1.41%
+2.18%
+2.20%
RotoGrad +1.19%
-1.33%
+0.75%
+1.40%
+1.45%
RGW
+0.62%
-0.22%
+0.41%
+0.78%
+1.46%
5.7 Effects of Different Architectures
The proposed RW methods can be seamlessly incorporated into all the MTL architectures. To see this, we take RLW an example and combine it with three different MTT architectures, i.e., cross-stitch network (Misra al. 2016), Multi-Task Attention Network (MTAN) (Liu al. 2019), and NDDR-CNN (Gao) et al. 2019) We use the combination of EW and HPS the baseline to measure the relative improvement of the other different, combinations as shown in the definition of Ap
Table 5: Results of different combinations of task balancing methods and MTL architectures the NYUv2 dataset in terms of Ap. The best results for each architecture are highlighted in bold.
HPS
Cross-stitch
MTAN
NDDR-CNN
EW
+0.00%
+1.43%
+2.56%
+1.90%
CAGrad +1.36%
+2.42%
+2.26%
+2.83%
RLW +1.04%
+2.23%
+2.66%
+2.91%
RLW+CAGrad
+2.20%
+2.76%
+2.92%
+3.53%
According to the results on the NYUv2 dataset. as shown in Table 5 we can see that the proposed RLW strategy outperforms the EW method under all the three architectures. When using the MTAN and NNDR- CNN architectures, RLW achieves better performance than the CAGrad method that performs best in the HPS architecture, which shows the potential of the proposed RLW method when choosing suitable MTL architectures. Moreover, combined with the RIW method, CAGrad can be further improved under the four
12

---

Published in Transactions on Machine Learning Research (10/2022)
architectures. For example, the combinations of RLW and CAGrad can achieve the best Ap of 3.53% under the NDDR-CNN architecture.
6 Related Work
Loss Balancing Methods. This type of method aims to balance different tasks by carefully designing the dynamic loss weights via. measuring the learning speed, uncertainty, relative loss value, and validation performance. For example, DWA (Liu et al. 2019) sets the loss weight of each task to be the ratio of two adjacent losses. UW [Knddll et al. 2018) applics homoscedastic uncertainties as loss weights for cach task and those weights are dynamically updated by backpropagation. IMTL-L (Liu et al. 2021b) learns the loss weights to make the scaled loss value similar for each task. MOML (Ye al. [202]] learns loss weights meta-learning- based hyperparameter optimization manner, where the loss weights are cast the function of model parameters and updated by minimizing the loss on the validation dataset.
Gradient Balancing Methods. This type method aims to find an aggregated gradient to balance different tasks. For example, MGDA-UB (Sener Koltun| 2018) casts MTL as a multi-objective opti- mization problem and directly solve the optimal weights in every iteration by applying Multiple Gradient Descent Algorithm (MGDA) (Désidéri 2012) which aims to find common descending direction among all the gradients via solving quadratic programming problem. GradNorm (Chen et al. 2018b) aims to explicitly learn loss weights by constraining the gradient magnitude of each task to be similar. GradDrop (Chen et al. 2020) thinks that the conflict can be reflected by the inconsistency in signs between the gradient values of different tasks and it eliminates such conflict by masking out the gradient values with opposite signs. PCGrad (Yu et al., 2220)) projects each task's gradient onto the normal plane of the other task's gradient they are conflict reflected in terms of negative cosine similarities between task gradients. Based PCGrad, GradVac (Wang et al. 2021) projects task gradients in more adaptive way where the cosine similarities between task gradients is not always positive. IMTL-G (Liu et al. 2021b) aims to find an aggregated gradient that has projections with an equal length on each lask's 8 gradient. CAGrad (Liu et al. 2021a) aims to find gradient that can decrease all the task losses by solving an optimization problem at every itcration. RotoGrad (Javaloy & Valera, 2022) climinates conflict by jointly homogenizing gradient magniludes and directions. Specilically, it uses learnable rotation matrix to change the gradient direction for each task and then computes weights that constrain all tasks have the same gradient magnitudes.
Task Similarity-based Methods. This type of MTL method aims to improve the performance of MTL by modeling and utilizing the task similaritics between different tasks. For example, Shui et al. (2019) explicitly learn a similarity matrix to model the similarities between different tasks in an adversarial manner and this adversarial loss term has a regularization effect to improve the performance. Similarly, Bai & Zhao (2022) propose regularizer learn the task similarities and improve the performance. Different from these two methods, Fifty et al. (2021) aim to automatically determine task grouping using the learned task similarities.
7 Conclusions
In this paper, propose the RW methods, an important yet ignored baseline for MTL, by training an MTL model with random loss/gradient weights. We analyze the convergence and effectiveness properties of the proposed RW method. Morcover, we provide a consistent and comparative comparison to show that the RW methods can achieve comparable performance with state-of-the-art methods that use carefully designed weights. Though RW methods can not always achieve the best performance empirically, those experimental results still indicate that random experiments could be used to examine the effectiveness of newly proposod MTL methods and RW methods cou attract wide attention as the litmus tests. In our future work, we will apply the RW methods to more MTL applications.
13

---

Published in Transactions on Machine Learning Research (10/2022)
Acknowledgements
This work is supported by NSFC key grant 62136005, NSFC general grant 62076118, and Shenzhen funda- mental research program JCYJ20210332105000000,
References
Guangji Bai and Liang Zhao. raliency-regulariized deep multi-task learning, In Proceedings of the 28th ACM SIGKDD Conference on Knowleddge Discovery and Data Mining, pp. 15 25, 2022.
Rich Caruana. Multitask learning: A knowledge-based based source of inductive bias. In Proceedings of the 10th International Conference on Machine Learning, pp. 41 41-48, 1993.
Rich Caruana, Multitask learning. Machine learning, 28(1):41-77 1997.
Pratik Chaudhari, Anna Choromanska, Stefano Soatlo, Yann LeCun, Carlo Baldassi, Christian Borgs, Jen- nifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valloys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124018, 2019.
Liang-Chiin Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for somantic image segmentation. In Proceedings of the 14th European Conference on Computer Vision, volume 11211, pp. 833 851, 2018a.
Zhao Chen, Vijay Badrinarayanan, Chen-Y Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normaliza- tion for adaptive loss balancing in deep multitask networks. In Proceedings of the International Conference on Machine Learning, pp. 794 803. PMLR, 2018b.
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In Proceedings of the 33rd Advances in Neural Information Processing Systems, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban seene understand- ing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2016.
Jean-Antoine Désidéri. Multiple-rradient descent algorithm (MGDA) for multiobjective optimization. Comptes Rendus Mathematique, 350(5):313-318, 2012.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapler of the Associalion for Compulational Linguislics: Human Longuage Technologies, pp. 4171 4186, 2019.
Chris Fifty, Fhsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. In Advances in Neural Information Processing Systems, volume 34, pp. 27503-27516, 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In Proceedings of the 8th International Conference Learning Rep- resentations, 2020.
Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-cmi enn: Layerwise feature fusing in multi- task cnns by neural discriminative dimensionality reduction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3205-3214, 2019.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. In Proceedings of 36th International Conference on Machine Learning, pp. 2232 2232-2241. 2241. PMLR, 2019.
14

---

Published in Transactions on Machine Learning Research (10/2022)
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Proceedings of the 33rd International Conference on Machine Learning, pp. 1225-2251122 1234. PMT.R, 2016.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 4411 4421. PMLR, 2020.
Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic general- ization measures and where to find them. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-lask learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-batch training for deep learning: Generalization gap and sharp minima. In Proceedings of the 5th International Conference on Learning Representations, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd Internaiional Conference on Learning Represenlations, 2015.
Bobby Kleinberg, Yuanzhi Li, and Yang Yuan. An alternative view: When does sgd escape local minima? In Proceedings of the 35th International Conference on Machine Learning, pp. 2698-2707, PMLR, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278-25) 2324, 1998.
Baijiong Lin and Yu Zhang. Libmtl: A python library for multi-task learning. arXiv preprint arXiv:2203. 14398, 2022.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. In Advances in Newral Information Processing Systems, volume 32, 2019.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Proceedings of the 35th Advances in Neural Information Processing Systems, 2021a.
Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hos Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In Proceedings of the 9th International Conference on Learning Representations, 2021b.
Shikum Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. In Proceedings of TEEE Conference Computer Vision and Pattern Recognátion, pp. 1871-1880, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision, 2015.
Kevis-Kokitsi Maninis, Tlija Radosavovic, and Tasonas Kokkinos. Attentive single-tasking of multiple tasks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851 1860, 2019.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi- task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994 4003, 2016.
Eric Moulines and Francis Bach. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. In Proceedings of the Advances in Neural Information Processing Systems, volume 24, pp. 451- 459, 2011.
15

---

Published in Transactions on Machine Learning Research (10/2022)
Deanna Needell, Nathan Srebro, and Rachel Ward. Stochastic gradient descent, weighted sampling, and the randomized kaczmarz algorithm. Mathematical Programming, 155(1-2):549-55) 573, 2016.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis M. Tyers, and Daniel Zeman. Universal dependencies v2: An ever- growing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 1034-4043, 2020.
Sara Sabour, Nicholas Frosst, and Geolfrey E Hinton. Dynamic routing between capsules. In Aduances in Newral Information Processing Systems, volume 30, 2017.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Proceedings of the 6th European Conference on Computer Vision, pp. 213-226. Springer, 2010.
Itay M Safran, Gilad Ychudai, and Ohad Shamir. The effects of mild over parammmtmettiiiioo the opti- mization landscape of shallow relu neural networks. In Conference on Learning Theory, pp- PMLR, 2021.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Proceedings of the 31st Advances Neural Information Processing Systems, pp. 525-536, 2018.
Changjian Shui, Mahdich Abbasi, Louis-Emile Robitaille, Boyu Wang, and Christian Gagné. A principled approach for learning task similarity in multitask learning. In Proceedings of the 28th International Joinl Conference on Artificial Intelligence, pp. 3446-3452, 2019.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Proceedings of the 8th European Conference on Computer Vision, pp- 746-760, 2012.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Sharan Vaswani, Francis Bach, and Mark Schmidt. Fast and faster convergence of sgd for werrpaaameeteried models and an accelerated perceptron. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, pp. 1194-1204. PMLR, 2019.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition, pp. 5018 5027, 2017.
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In Proceedings of the 9th International Confer- ence on Learning Representations, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: Stato-of-tho-art natural language processing. In Proceed- ings of the 2020 Conference on Empiricul Methods in Natural Language Processing, pp. 38 45, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-muist: novel image dataset for benchmarking
machine learning algorithms. arXiv preprint arXiv:1708.077477 2017.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ith International Joint Conference on Natural Language Processing, pp. 3685-3690 2019.
16

---

Published in Transactions on Machine Learning Research (10/2022)
Feiyang Ye, Baijiong Lin, Zhixiong Yue, Pengxin Guo, Qiao Xiao, and Yu Zhang. Multi-objective meta learning. In Proceedings of the 35th Advonces in Newral Information. Processing Systems, 2021.
Fisher Yu, Vladlen Koltun, and Thomas A. Funkhouser. Dilated residual networks. In Proceedings of the
TEEE Conference Computer Vision and Pattern Recogration, pp- 636 644, 2017.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Proceedings of the 33rd Advances in Neural Information Processing Systems, 2020.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
17

---

Published in Transactions on Machine Learning Research (10/2022)
A Proof of Section 4
Corollary 1. Consider the objective function C(0) defoned in Theorem 1 Suppose that Assumption is satisfied, then the function V is M-Lipschitz continuous, where M maxisisT {M1).
Proof. Note that 5 lr and E Me 1, have Et= M,M S M, where M maxisigT(Mi. Since (0) is Me-Lipschitz continuous, for any two points 01 and 02, we have
T
[V V2(02)I - V (l(((:) V v
((2)
4 (() - ver(02)1
Σ Mr 110 02ll
< M 1 0211-
Thus VC(0) is M-Lipschitz continuous.
Corollary Suppose that Assumption 2 is satisfied, then the function [(0) defined in Theorem satisfies the PL inequality with constant c, where mingger((a).
Proof. Suppose c= minigees sesT(()) and 0. is a global minimum of L(0). Since (0) satisfies the PL, inequality with constant Ct, for any 0, we have
VCO1)
2 V
(0)
Mel (o))
IV
0
(o(((()- - le())))
where the last inequality is due to the PL conditions in Assumption 2 and the definition of c. Since the optimal value of the weighted objective L(0) is no less than the weighting of the respective optimal values of all objectives (l)). We have L(0.) > Σ (O)) So we obtain
1 | 2 c(C(0) L(O-)).
2
Thus ((0) satisfics the PL inequality with the constant c.
A.1 Proof of Theorem i
Based on Corollary and the update stochastic gradient descent, i.e., 0x41 = O2 - E(D;0s) we have
[(0x+1) - L(0x) < (VL(())) 0k+1 - -Ox) + .M1
2 10 - 0x1
-nn(V VA vATe(D:O:)) + Mr²
| (ATe(DD0))''
2
18

---

Published in Transactions on Machine Learning Research (10/2022)
Note that Ex ((D: = Vu E(D; 0k) and
Ex E E(D; 00)111
- EA
E
|
V((D;
0,)1
T
T
< E
Σ
o/
1=1
1=1
where the first inequality is due to the Cauchy- Schwarz inequality and the third inequality is due to 0 S S
and Σ, d 1. Then, by defining x Σ of we obtain
Ex [Ep(((O+))-
n (L((O)) BA P (w))
E(D:
+
Mn²
Ex EDDIIAA
E(D;0a))
M1RK
S -n (VC(Os), ((D +
2
=-nlVp T ((0x))12 + MлRK
2
2m)(((((a)) - [(0.)) + 2
where the last inequality is due to Corollary 2
Then, we have
M1² k.
E[C(O++1) (C(()+))-(((0))1 (1 (((c C(0.)) +
(5)
2
If 1 2nc 0, we recursively apply the inequality 5 over the first k iterations and we can obtain
M7p K
k-1
(IC(0++1)-C11) s 2nc)** (C(Oo) [(0.)) +
Σ (1
2n70)
2
1=
(1 2nc) (C(Gp) [(0.))
4c
Thus the inequality
(3) holds if i) 1 1
2e
According to inequality
the minimal value of a quadratic function (e (1 2nec) 2
Mg²K
is achieved
at 1/=- Mx
2ec By setting ((ax)= - [(O.) Eo. we have
E[((0++1) - C(0.)) S -c(0L)-c(0.) (1..)
2c² (c(()(0k) - C(0.))) [C((K) C(0-)]
MN
2Ec²
1
[C(0k) C().))
MR
28c²
MR
Then if B[C(0++1) [(0.)) e, we have
(1
2ec²
''zo- Therefore, k < Mx
Mr 2
log Ea
A.2 Proof of Theorem 2
Since ipk = 0g V (((Ox) and Ox+1 = -0-- - "n(u e((x) + Ek), we have
Pk+1 4Ph 1/Ex V T llypx n/Ex)-
19

---

Published in Transactions on Machine Learning Research (10/2022)
Since the loss function (0) of task t is Ln-one point strongly convex w.r.t. given point 0, after convolved with noise &, we have
T
VVEgluee(y - -n]]), -0.)
Σ Me (VEel(y- -EE) 40 0.)
Me Lello 11
2 L 0. 112
where L = minggigt Le), the first inequality is due to Ly-one point strongly convex in Assumption 3 and the last inequality is due to the definition of L and >i= M4 1. Then we can get
B|[lsx+1 0-11²)
E1 T)Eu= V e(px= n/xx) 0.11²]
- 111l 0.11 + || + 1V T e(px - m1x))11 - 2 (pxx - 0,. nEx)
- - 0., Vu T flyck r(xx) +2 E(px= - ()(x),
$ |1 0.1 + 1"² Ell T E(px - -161)112] -2 5Px - 0.11
+ 2 (V E(px - n/xx)= - Vu [((x), T}Ex))
< (1 - 117 - 00 + (O. - -(Pp - -1(x))11 2
< (1 - 27/ - 0-l1² +12,, + 1/² 2/ 1112111x-0000 -0.11 + Bl(sp -O..1Ex7
+ 7, [17]]
2 + n/) 0.112 + $ (1 2, + x M2) 00
where the second incquality is due to the convexity assumption and E[gA] = O, the third and forth incqualities are due to the Lipschitz continuity in Corollary we have 17 S n/. then we get
We set p = 2n - xf M and B = (1++)) p 2 0,
Billp++1 0.1 < (1 p)lppx 0.112+ B
(1 D) 1100 0.113 + Σ (1 - --p
j=0
3
$ (1 - 1)0 - 0.112
So if K < 1 log
AE0 we have [[]ppH+- - 0.13] < 29 Then by the Markov inequality, with probability at
least 1 - 8, we have
23 pō
11 - -0,12
A.3 Noise Upper Bound
Suppose the noise produced by the EW method is -V (D:0)) Vn T();)))) and || S R. The noise produced by the RLW method is & = / ((D;0) - Vp ((D:))). We have
((D:)))(V++++ | = VIV ((D;0) - V4 T V (((:)) - Vμ E(D;0)1
= II T =T))E((D 011 + (X -n - (_T)e(D:))) + 11551
Bocause the noise & can be any dircction, there exists constant s such that || = R and { s(X -p--pp - ()) Then, we have ||E|| < (1+2s) ()) RR. R. Thus, the norm of the noise provided by the R.W method has larger supremum than EWW
20

---

Published in Transactions on Machine Learning Research (10/2022)
B Proof of the Mean Value E(x)
Suppose that (t=1=1 1,... 7) are independent and identically distributed (i.id.) random variables sampled from the Uniform standard Normal distributions and is the softmax function. Then have X exp((e) and
Σ m- exp(Am)
B(A,) = E[ex E exp(Am)
+ Cov exp((i),
Em exp(Xm))
where Cov(.-)) denotes the covariance between two random variables, ii.d random variables, we have E[exp(A:)] = E[exp(A;)] and Cov(cxp(A:), 1/ 1/Em=r exp(Am)) 1/ exp(Am)). Therefore, we obtain
Since {delT=1 are
B(A:) = VI
Morcover, we have
I Σ E(A) Σ
k Σ
E X
=1
K
Thus have B(A) ( #). Similarly, we can prove the same result for the Bernoulli and e-Bernoulli
distributions with the normalization function as f(A) = X/(E Xe).
Additional Experimental Results
C.1 Results on the CityScapes Dataset
Table 6: Performance on the CityScapes dataset with two tasks: 7-class semantic segmentation and depth estimation. The best results for each lask on each measure over loss/ /es/rradient balancing methods are marked with superscript #/t. The best results for each task on each measure are highlighted in bold. 1 (L) means the higher (lower) the result, the better the performance.
Segmentation
Depth
Methods
Apt
mIoUt
Pix Acct
Abs Err!
Rel Err,
EW
68.71
91.50
0.0132
45.58
+0.00%
UW
68.84
91.53
0.0132
46.18
-0.09%
Loss Bal.
DWA
68.56
91.48
0.0135
44.49
+0.05%
IMTL-L
69.71*
91.77*
0.0128*
45.08
+1.58%
MOMI.
69.34
91.65
0.0129
46.33
+0.59%
RLW (ours)
68.78
91.45
0.0134
43.68*
+0.69%
MGDA-UB
68.41
91.13
0.0124
46.85
+0.64%
GradNorm
68.60
91.48
0.0133
45.32
+0.01%
Gradient Bal.
PCGrad
68.54
91.47
0.0135
44.82
-0.10%
GradDrop
68.62
91.45
0.0136
45.05
-0.42%
IMTL-G
68.62
91.48
0.0130
44.29
+1.09%
GradVac
68.60
91.47
0.0134
44.92
-0.06%
CAGrad
68.89
91.50
0.0128
44.72
+1.38%
RotoGrad
68.96
91.47
0.0127
43.851
+2.13%
RGW (ours)
69.68t
91.85t
0.0127
43.91
+2.36%*
21

---

Published in Transactions on Machine Learning Research (10/2022)
Dataset. The CityScapes dataset (Cordts et al. 2016 is large-scale urban street scene understanding dataset and it is comprised of diverse set of stereo video sequences recorded from 50 different cities in fine weather during the daytime. It contains 2,975 and 500 annotated images for training and test, respectively. This dataset inchudes two tasks: 7-class semantic segmentation and depth cstimation.
Implementation Details. For the CityScapes datasct, the network architecture and optimizer are the same as those in the NYUv2 dataset. We resize all the images to 128 x 256 and set the batch size to for training. We use the cross-entropy loss and L loss for the semantic segmentation and depth estimation tasks, respectively.
Results. The results on the CityScapes dataset are shown in Table The empirical observations are similar to those the NYUv2 dataset in Table I Firstly, both the RLW and RGW strategics significantly outperform the EW method. Secondly, the RLW method can oulperform most of the loss balancing baselines except the IMTL.-L method. Moreover, the RGW method achieves 2.36% performance improvement and outperforms all of the baselines.
C.2 Results on the CelebA Dataset
Dataset. The CelebA dataset (Liu al.) 2015 is large-scale face attributes dataset with 202,599 face images, each of which has 40 attribute annotations. It is split into three parts: 162,770, 19, 19,867, and 19,962 images for training, validation, and testing, respectively. Hence, this dataset contains 40 tasks and each task is a binary classification problem for one attribute.
Implementation Details. We use the ResNet-18 network as a shared feature extractor and a fully con- nected layer with two output units as a task-specific head for each task. All the images are resized to 64 x 64. The Adam optimizer with the learning rate as 10 3 is used for training and the batch size set to 512. The cross-cntropy loss used for the 40 tasks.
Table 7: Average classification accuracy (%) of different methods on the CelobA dataset with forty tasks. The best results over loss/ /gsaddide balancing methods are marked with superscript */t. The best results are highlighted in bold.
Methods
Avg Acc
EW
90.70
UW
90.84
Loss Bal.
DWA
90.77
IMTL-L
90.46
MOML
90.94
RLW (ours)
90.73
MGDA-UB
90.40
GradNorm
90.77
Gradient Bal.
PCGrad
90.85
GradDrop
90.71
IMTL-G
90.80
GradVac
90.75
CAGrad
90.72
RotoGrad
90.45
RGW (ours)
90.00
Results. Since the number tasks in the CelebA dataset is large, we only report the average classification
accuracy the forty tasks in Table 7
According to the results, the proposed RLW strategy slightly
22

---

Published in Transactions on Machine Learning Research (10/2022)
outperforms the EW method and performs comparably with loss balancing baseline methods. However, we can find that the RGW method and most of the gradient balancing methods are worse or achieve very limited improvement over the EW method, which indicates the gradient weighting is not suitable for the CelebA datasct.
C.3 Results on the Office-31 31 and Office-Home Home Datasets
Datasets. The Office-31 dataset (Saenko et al., 2010) consists of three domains: Amazon (A), DSLR (D), and Webcam (W), where each domain contains 31 object categories, and it contains 4,110 labeled images. We randomly split the whole dataset with 60% for training, 20% for validation, and the rest 20% for testing. The Offico-Home dataset (Venkateswara al., 2017) has four domains: artistic images (Ar), clip art (CI), product images (Pr), and real-world images (Rw). has 15,500 labeled images total and each domain contains 65 classes. We make the same split the Office-31 dataset. For both datasets, we consider the multi-class classification problem each domain as task. Similar to multilingual problems from the XTREME benchmark, each task in both Office-31 and Office-Homm datasets has its own input data.
Implementation Details. We use the same configuration for the Office-31 and Office-Home datasets. Specifically, the ResNet-18 network pre-trained on the ImageNet dataset is used as a shared backbone among tasks and a fully connected layer applied as a task-specific output layer for each task. All the input images are resized 224 x 224. We use the Adam optimizer with the learning rate as 10-4 and the weight decay as 10-5 and set the batch size to 128 for training. The cross-entropy loss is used for all tasks in both datasets.
Results. According to the results shown in Table we can see both the RLW and RGW strategies outper- form the EW method on both two datasets in terms of the average classification accuracy over tasks, which implics the effectiveness of the RW methods. Morcover, the RGW method achioves the best performance (92.55% and 78.07% term of the average accuracy) over all baselines on the Office-31 and Office-Home datasets, respectively.
Table 8: Classification accuracy (%) different methods on the Office-31 and Office-Home datasets. The best results for cach domain over loss/gradien gradicnn balancing methods are marked with superscript */t. The best results for each task highlighted in bold.
Office-31
Ofice-Home
Methods
A
D
W
Avg
Ar
Cl
Pr
Rw
Avg
EW
82.73
96.72
96.11
91.85
62.99
76.48
88.45
77.72
76.41
UW
82.73
96.72*
95.55
91.66
63.94
75.62
88.55
78.05
76.54
Loss Bal.
DWA
82.22
96.72*
96.11
91.68
63.37
76.05
89.08
77.62
76.53
IMTL-L
83.76
96.72*
95.55
92.01
65.46'
79.08*
88.45
78.81
77.95*
MOMT.
84.78%
95.08
96.67*
92.17
64.70
77.03
88.24
80.00
77.49
RIW (ours)
83.76
96.72*
96.67*
92.38*
62.80
76.48
90.57*
80.21*
77.52
MGDA-UB
81.02
95.90
97.771
91.56
64.32
75.29
89.72
79.35
77.17
GradNorm
83.93
97.54
94.44
91.97
65.46
75.29
88.66
78.91
77.08
Gradient Bal.
PCGrad
82.22
96.72
95.55
91.49
63.94
76.05
88.87
78.27
76.78
GradDrop
84.27
95.08
96.11
91.82
64.70
77.03
88.02
79.13
77.22
IMTL-G
82.22
95.90
96.11
91.41
63.37
76.05
89.19
79.24
76.96
GradVac
82.73
97.54
95.55
91.94
63.18
76.48
88.66
77.83
76.53
CAGrad
82.22
96.72
96.67
91.87
63.75
75.94
89.08
78.27
76.75
RotoGrad
82.90
96.72
96.11
91.91
61.85
77.03
90.36t
78.59
76.95
RGW (ours)
84.27
96.72
96.67
92.55'
65.08
78.651
88.66
79.891
78.07!
23


Reviews:
[{"review_id": "KLYFGqAO0Q", "reviewer": "Reviewer_yYkZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Proposal to utilize random weightings as additional baselines for future MTL algorithms is very low cost to implement and should be easily achievable.\n\n* Combination of analyses and experiments provides compelling evidence to support claim.\n\n* The paper is clear and organized well.\n\n* Experiments are performed wrt/ a variety of performance dimensions: application, robustness, speed, experimental (combining loss and gradient weightings)\n\nWeaknesses:\n\n* The theory and experiments are a little too complementary in that they are almost orthogonal. In other words, it would have been nice if some synthetic experiment(s) was provided showing the analytical claims held. However, I consider this absence very minor in the overall amount of evidence supplied.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the specification of task weights in multi-task learning. Specifically, two types of weighting are considered: loss weights which affect shared and task parameters and gradient weights that affect only the shared parameter. For these types, the paper proposes random weightings as competitors which should be used in substantiating performance of multi-task learning algorithms that focus on weighting schemes.\n\nAs evidence that these random weighting schemes should be used, the paper supplies analyses proving convergence of the shared task parameter under random loss weights in expectation and with high probability (the latter with additional assumptions). The paper also supplies experiments showing random weightings are at least competitive with the standard baseline of equal weighting if not often-times superior in performance in a computer vision dataset and NLP dataset and are robust to choice of randomness and NN architecture.\n\nI think this is a very good paper with a nice mix of theory and experiments. The proposal to use random weightings as a standard competitor in future MTL algorithm performance claims is quite achievable and convincingly backed-up. (Note, I did not check the proofs in great detail beyond general plausibility.)", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I have no requested changes.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sDrFc2jwVf", "reviewer": "Reviewer_KBdf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. This paper is easy to read and it is concise while being mostly self-contained. For the most part, the reader not familiar with current multi-task learning state-of-the-art techniques for loss/gradient weighting should be able to get a basic understanding of the methods from reading the first sections of this paper. \n2. The theoretical analysis is a nice backup for the experimental proposal.\n3. Experimental study presents results across a wide range of benchmarks with different tasks and of different modalities. \n\nWeaknesses: \n1. The assumptions used to prove Theorems 1 and 2 should be better clarified. Ideally, it would be nice to separate the assumptions from the statements of these two theorems into a bullet list and explain each assumption individually.\n2. I had a feeling that this paper was written in a hurry as it has a fair amount of typos and grammatical mistakes. Major proofreading would be needed to fix those. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to explore the use of random weights in multi-task learning (MTL). The main underlying idea is to study the performance of using random weights drawn from some probability distribution either when reweighing the individual tasks loss terms optimized by MTL algorithm or when defining the weights of individual tasks’ gradients. The authors first study theoretically their proposed reweighing strategy and show that it 1) converges to an optimal solution within a certain radius and 2) may avoid bad local minima. Then, they present an experimental study revealing that such a simple method can represent a surprisingly strong baseline that remains on par or better than other common weighing strategies and is always better than equal weighing.  \n\nI believe that this paper presents a strong contribution to the multi-task learning community by studying a very intuitive, yet overlooked baseline. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Major:\n\nMy major requested change is regarding the theoretical part where the authors do not clearly state to which extent the assumptions made are restrictive. Below, I explain this in more detail:\n1. It seems that assuming strong convexity of the loss with respect to $\\theta$ in Theorem 1 means that the authors do not consider neural networks which optimize a function that is non-convex in its parameters. It is not stated explicitly but that would limit the scope of this result to some extent as most common MTL methods are based on deep neural networks. \n\n2. In Theorem 2, the authors introduce the notion of local minimum convoluted with noise even though it was never mentioned before. Theorem 2 in general lacks a proper introduction. Ideally, the authors would spend more time to first introducing the intuition and the motivation to have the quantities that they have in the statement. Currently, they are introduced directly in the statement which is cumbersome and makes the whole results less clear. \n\n3. It is a good practice also to explain the pitfall scenarios for random weight generation. Currently, one has an impression that this baseline works well in all cases. Did the authors identify the scenarios where random weights are worse than equal ones? \n\nMinor changes:\n\nSome typos:\n1. “Task balancing” in the abstract is not introduced, consider rephrasing without this specific term\n2. “thus it is apparently” -> “it is apparent”\n3. “RLW has a better generalization performance that EW” -> than\n4. “bring negligibly additional computational costs” -> negligible \n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8sWzw0XyTU", "reviewer": "Reviewer_9Zr7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I really like the proposed idea because it is conceptually simple and appealing. Furthermore, the authors attempted to provide various theoretical and practical evidence to support the proposed approach. I also think there exist several weak points. My comments, which are based on TMLR guidelines, are as follows:\n\n### 1. Would some individuals in TMLR's audience be interested in the findings of this paper?\n\nDefinity **YES**. \n\n1. This paper enhances an intriguing viewpoint: even if the weight of each task is generated at random, empirical performance may be comparable to or better than other modern baselines. Because the random number generation is simple, it would be quite promising in practice.\n2. Extensive empirical validations are conducted to justify the claims.\n3. A convergence analysis of optimization is proposed. (However, this reviewer is concerned about the results.)\n\n### 2. Are the claims made in the submission supported by accurate, convincing and clear evidence?\n\n**Partially**. \n\nThe authors attempted to provide evidence from either theoretical or practical perspectives. Several arguments, however, are neither accurate nor clear. The detailed comments are as follows:\n\n\n1. This paper concentrated primarily on the multi-head based deep multi-task learning problem. In other words, there is a shared parameter $\\theta$ and a task-specific parameter $\\phi_i$. To better fit the paper scope, I think this paper should clearly illustrate this in the introduction section.\n2. [Related work] The introduction of loss balance and gradient balance approaches (in Section 2) could be improved. The related introductions/key works (such as uncertain weights) appear ambiguous, whereas I mean the related works could be listed as a table to demonstrate different techniques. Besides, it would be highly appreciated to discuss other sorts of multi-task learning perspectives such as task-similarity based MTL (e.g., paper [1])\n3. In Section 3, why is further normalisation required if the task weights are generated by a distribution $p(\\lambda)$?\n4. Algo 1 and Algo 2 had very similar structures, and I think these two tables should be combined into a single algorithm.\n5. (Major improvement) I couldn't figure out why it's **theoretically** beneficial in the theoretical analysis. (1) Theorem 1 proposed the optimization convergence behaviour on the proposed algorithm, despite the fact that the theory appears to be quite standard in the optimization. (2) Theorem 2 yielded a number of results in noisy gradient descent, where noise is injected for the gradient (this surely guarantees to escape a local minimum). This does not, in my opinion, demonstrate that noisy $\\lambda$ guarantees a better generalisation property. When we look closely at the proposed framework, we can see that it is essentially a randomised version of task weights. In general, the randomised algorithm may produce more robust results with improved generalisation properties. As a result, I think Theorems 1 and 2 cannot explain the success. According to the TMLR guidelines, I would recommend revising this section to better reflect the contribution.\n6. The proposed algorithm is an alternating optimization in which the task weights and model parameters are alternately updated. In fact, as hyper-parameters, $\\lambda$ is essentially a function of the model parameter. As a result, I would propose a discussion about this section.\n7. It is not surprising that the empirical results are nearly comparable or slightly better than the baselines. I would recommend a clear statement on the results; it is perfectly fine that the empirical performances are not always SOTA, and a clear discussion would be great.\n\n### 3. Minor points\n\nI would suggest that the authors carefully revise the paper because there are several typos and low-level writing issues. E.g,\n\n- Intro “This phenomenon is related to …” should be ‘is referred to as’\n- Intro ‘We think that baseline is not sufficient … to conduct a random experiment, which is missing, as a baseline to test them ’ What is a random experiment?\n- The title in Sec 2 could be expressed as preliminary or problem set-up. \n- Sec 2 “in every iteration” should be “in each iteration”\n- Sec 3 (page 4) “,has a more stable performance.. In Sec 5.3’ I think this part is quite confusing. Sec 5.3 did not show a stable performance?\n- Figure 2 is difficult to understand. I couldn't understand its main point.\n\nRef [1] A Principled Approach for Learning Task Similarity in Multitask Learning. IJCAI 2019.\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a novel way in understanding deep multitask learning. Specifically, rather than learning from data, the weights of learning each task are generated by a sort of **random distribution**. The authors claimed that it would help them escape the local minimum and improve prediction performance. This paper was empirically validated on a variety of real-world datasets, with the empirical performance consistently outperforming baselines.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See the comments and questions in \n-  Are the claims made in the submission supported by accurate, convincing and clear evidence?\n-  Minor points\n\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper examines a simple yet effective method for multi-task learning. The method weights each task randomly through its loss or gradient. The authors demonstrate that random weighting improves generalization performance, allowing the method to perform competitively to other state-of-the-art methods. Some theoretical analyses on the convergence property of the method are provided to strengthen the paper.

The reviewers appreciate the simplicity of the proposed method that comes with competitive performance to serve as a solid future baseline for multi-task learning. The breadth of the experiments is sufficient to demonstrate the strength of the proposed method. Before the revision, most concerns surrounded how the theory connected to the experiments. The authors attempted to resolve the concerns by providing more toy experiments, clarifying and loosening the theory's assumptions, and rephrasing several hand-waving claims. The efforts convinced the reviewers. Ultimately, all reviewers unanimously agreed that the work could be accepted.

Some reviewers are still aware of typos and grammatical errors in the latest version. The authors should do careful proofreading when preparing for the camera-ready version.

Decision: Accept (poster)