Year: 2021

Abstract:
Multi-Task Learning (MTL) has achieved great success in various fields, however, how to balance different tasks to avoid negative effects is still a key problem. To achieve the task balancing, there exist many works to balance task losses or gradients. In this paper, we unify eight representative task balancing methods from the perspective of loss weighting and provide a consistent experimental comparison. Moreover, we surprisingly find that training a MTL model with random weights sampled from a distribution can achieve comparable performance over state-of-the-art baselines. Based on this finding, we propose a simple yet effective weighting strategy called Random Loss Weighting (RLW), which can be implemented in only one additional line of code over existing works. Theoretically, we analyze the convergence of RLW and reveal that RLW has a higher probability to escape local minima than existing models with fixed task weights, resulting in a better generalization ability. Empirically, we extensively evaluate the proposed RLW method on six image datasets and four multilingual tasks from the XTREME benchmark to show the effectiveness of the proposed RLW strategy when compared with state-of-the-art strategies.

heading: 1 INTRODUCTION
text: Multi-Task Learning (MTL) (Zhang & Yang, 2021; Vandenhende et al., 2021) aims to jointly train several related tasks to improve their generalization performance by leveraging common knowledge among them. Since this learning paradigm can not only significantly reduce the model size and increase the inference speed but also improve the performance, it has been successfully applied in various fields of deep learning, such as Computer Vision (CV) (Vandenhende et al., 2021), Natural Language Processing (NLP) (Chen et al., 2021), reinforcement learning (Zhang & Yang, 2021) and so on. However, when all the tasks are not related enough, which may be reflected via conflicting gradients or dominating gradients (Yu et al., 2020), it is more difficult to train a multi-task model than training them separately because some tasks dominantly influence model parameters, leading to unsatisfactory performance for other tasks. This phenomenon is related to the task balancing problem (Vandenhende et al., 2021) in MTL. Recently, several works focus on tackling this issue from an optimization perspective via balancing task losses or gradients.
In this paper, we investigate eight State-Of-The-Art (SOTA) task balancing approaches and unify them as loss weighting strategies. According to the way of generating loss weights, those methods can be divided into three types, including the solving approach such as directly solving a quadratic optimization problem in a multi-objective formulation as weights (Sener & Koltun, 2018), the calculating approach such as projecting conflict gradients (Yu et al., 2020), and the learning approach such as learning weights in a gradient descent manner (Chen et al., 2018b). On the other hand, since there are some discrepancies of the implementation details such as using different backbone networks for training or different metrics for the evaluation among those SOTA weighting methods, leading to inconsistent comparisons, we provide a unified testbed on six CV datasets and four multilingual problems from the XTREME benchmark (Hu et al., 2020) for those SOTA weighting strategies to show a fair comparison.
In addition, inspired by dynamic weighting processes in those SOTA strategies where loss weights vary over training iterations or epochs, we have a sudden whim: what will happen if a MTL model is trained with random loss weights? Specifically, in each training iteration, we first sample the
loss weights from a distribution with some normalization and then minimize the aggregated loss weighted by the normalized random weights. Surprisingly, this seemingly unreliable method can not only converge but also achieve comparable performance with the SOTA weighting strategies. Based on this observations, we proposed a simple yet effective weighting strategy for MTL, called Random Loss Weighting (RLW). It is very easy to implement RLW by adding only one line of code and this strategy does not incur any additionally computational cost. An implementation example of RLW in PyTorch (Paszke et al., 2019) is shown below.
1 outputs = model(inputs) 2 loss = criterion(outputs, labels) # [1, task_num] vector 3 weight = F.softmax(torch.randn(task_num), dim=-1) # RLW is only this! 4 loss = torch.sum(loss*weight) 5 optimizer.zero_grad() 6 loss.backward() 7 optimizer.step()
To show the effectiveness of RLW, we provide both theoretical analyses and empirical evaluations. Firstly, the objective function of RLW can be considered as a doubly stochastic optimization problem when optimizing by stochastic gradient descent or its variants, where the randomness is from both the mini-batch sampling of the data for each task and the random sampling of loss weights. From this perspective, we give a convergence analysis for RLW. Besides, we can show that RLW has a higher probability to escape local minima when compared with fixing loss weights, resulting in a better generalization performance. Empirically, as described before, we compare RLW with SOTA weighting approaches on six CV datasets and four multilingual problems to show its competitive performance.
In summary, the main contributions of this paper are four-fold.
• We provide a unified testbed on six multi-task computer vision datasets and four multilingual problems from the XTREME benchmark for a fair comparison among eight SOTA weighting methods and the proposed RLW method.
• We propose a simple yet effective RLW strategy, which we think is an ignored baseline in MTL.
• We provide the convergence guarantee and effectiveness analysis for RLW. • Experiments show that RLW can achieve comparable performance with SOTA weighting
methods without bringing any additionally computational cost.

heading: 2 PRELIMINARY
text: Suppose there are T tasks and task t has its corresponding dataset Dt. An MTL model usually contains two parts of parameters: task-sharing parameters θ and task-specific parameters {ψt}Tt=1. For example, in CV, θ usually denotes parameters in the feature extractor shared by all tasks and ψt represents the task-specific output module for task t. Let `t(·; θ, ψt) denotes a task-specific loss function for task t. Then the objective function of a MTL model can be formulated as
LMTL = T∑ t=1 λt`t(Dt; θ, ψt), (1)
where `t(Dt; θ, ψt) denotes the average loss on Dt for task t and {λt}Tt=1 are task-specific loss weights with a constraint that λt ≥ 0 for all t. When minimizing Eq. (1) by Stochastic Gradient Descent (SGD) or its variants, the task-specific parameters {ψt}Tt=1 are simply updated based on the corresponding task gradient ∇ψt`t(Dt; θ, ψt), while the task-sharing parameters θ should be updated by all the task losses jointly as
θ = θ − η T∑ t=1 λt∇θ`t(Dt; θ, ψt), (2)
where η is a learning rate. Obviously, for the update of task-sharing parameters θ, the loss weighting (i.e., {λt}Tt=1 in Eq. (1)) influences θ via the aggregated gradient essentially and the gradient
weighting in Eq. (2) during the backward process has the same effect as the loss weighting when they are using the same weights. Therefore, we can ignore the level on which the weights act and focus on the generation of weights. For simplicity, these two types of weighting are all referred to as loss weighting in the following sections.
Apparently, the most simple method for loss weighting is to set a same weight for every tasks, i.e., without loss of generality, λt = 1T for all t. This approach is a common baseline in MTL and it is called Equally Weighting (EW) in this paper. To tackle the task balancing problem and improve the performance of MTL model, there are several works to study how to generate appropriate weights. In this paper, we investigate eight SOTA weighting strategies, i.e. Gradient Normalization (GradNorm) (Chen et al., 2018b), Uncertainty Weights (UW) (Kendall et al., 2018), MGDA (Sener & Koltun, 2018), Dynamic Weight Average (DWA) (Liu et al., 2019a), Projecting Conflicting Gradient (PCGrad) (Yu et al., 2020), Gradient sign Dropout (GradDrop) (Chen et al., 2020), Impartial Multi-Task Learning (IMTL) (Liu et al., 2021), and Gradient Vaccine (GradVac) (Wang et al., 2021).
According to different ways of generating loss weights, we categorize those loss weighting strategies into three types: the learning approach, the solving approach, and the calculating approach. Both GradNorm and UW consider the loss weights {λt}Tt=1 in Eq. (1) as learnable parameters and explicitly optimize them by gradient descent. MGDA casts MTL as a multi-objective optimization problem and directly solves the loss weights {λt}Tt=1 in Eq. (1) by solving a quadratic programming problem. DWA, PCGrad, GradDrop and GradVac directly compute the weights {λt}Tt=1 by combining gradients and/or losses of all the tasks. IMTL is a hybrid strategy, which combines the learning and the calculating approaches. We summarize those strategies from the perspective of loss weighting in Table 5 in Appendix A.
We now unify those eight SOTA methods as loss weighting strategies, i.e., generating loss weights {λt}Tt=1 in Eq. (1). Noticeably, almost all the existing strategies except EW need to incur intensive computation to generate loss weights in every iteration, such as solving a quadratic optimization problem in MGDA, and operating on high-dimensional gradients in PCGrad, GradDrop, IMTL, and GradVac. Different from those strategies, the proposed RLW strategy generates loss weights in a sampling way, thus it is as efficient as EW without bringing additionally computational costs.

heading: 3 THE RLW METHOD
text: In this section, we introduce the proposed RLW method. The RLW method is a simple loss weighting strategy and it considers the loss weights λ = (λ1, · · · , λT ) ∈ RT as random variables. Formally, the objective function of the RLW method is formulated as
LRLW(θ) = Eλ [ λ>`(D; θ) ] = Eλ[λ]>`(D; θ), (3)
where E[·] denotes the expectation and `(D; θ) = (`1(D1; θ), · · · , `T (DT ; θ)) where we omit the task-specific parameters {ψt}Tt=1 in Eq. (3) for brevity. To guarantee loss weights in λ to be nonnegative, we can first sample λ̃ = (λ̃1, · · · , λ̃T ) from any distribution p(λ̃) and then normalize λ̃ into λ via a mapping f , where f : RT → ∆T is a normalization function for example softmax function and ∆T denotes a convex hull in RT , i.e. λ ∈ ∆T means ∑T t=1 λt = 1 and λt ≥ 0 for all t. Note that in most cases p(λ) is different from p(λ̃).
In Eq. (3), p(λ) is usually too complex to compute its expectation Eλ[λ], thus a stochastic approximation scheme is adopted to minimize Eq. (3). When the mini-batch SGD (Bottou, 1991) or its variants is used to minimize Eq. (3) as most deep learning models did, Eq. (3) can be viewed as a doubly stochastic optimization problem, where the randomness is from both the mini-batch data sampling for each task and the randomly sampling of the loss weights. In the following, we show that the approximated gradient ∇θλ>`(D̃; θ) is an unbiased estimation of the true gradient of LRLW(θ), where D̃ denotes a mini-batch data sampled from all the tasks. Specifically, as D̃t is a mini-batch data sampled from Dt to calculate the stochastic gradient ∇θ`t(D̃t; θ) to approximate the full gradient ∇θ`t(Dt; θ) for task t, we have ED̃[∇θ`(D̃; θ)] = ∇θ`(D; θ). Therefore, when we further randomly sample a weight vector λ, we have
Eλ [ ED̃[∇θλ >`(D̃; θ)] ] = Eλ[λ]>∇θ`(D; θ) = ∇θLRLW(θ),
which verifies that∇θλ>`(D̃; θ) is an unbiased estimation. In practice, it is very easy to implement the RLW method without modifying network architecture or bringing additionally computational costs. Specifically, in each iteration, we first sample λ̃ from p(λ̃) and normalize it to obtain λ via appropriate normalization function f , and then minimize the aggregated loss weighted by λ. The entire algorithm of RLW (i.e., minimizing Eq. (3)) via SGD is shown in Algorithm 1. Apparently, the only difference between the proposed RLW strategy and the widely used EW strategy is Line 7 in Algorithm 1 and it is very easy to implement with only one line of code.
Algorithm 1 Optimization Algorithm for RLW by SGD Input: numbers of iterations K, numbers of tasks
T , learning rate η, dataset {Dt}Tt=1, weight distribution p(λ̃) 1: Randomly initialized θ0; 2: for k = 1 to K do 3: for t = 1 to T do 4: Sample a mini-batch data D̃t from Dt; 5: Compute loss `t(D̃t; θk); 6: end for 7: Sample weights λ̃ from p(λ̃) and normalize
it into λ via f ; . RLW is only this 8: θk+1 = θk − η∇θ ∑T t=1 λt`t(D̃t; θk); 9: end for
In this paper, we use six different distributions for p(λ̃) in the proposed RLW method, including uniform distribution between 0 and 1 (denoted by Uniform), standard normal distribution (denoted by Normal), Dirichlet distribution with α = 1 (denoted by Dirichlet), Bernoulli distribution with probability 1/2 (denoted by Bernoulli), Bernoulli distribution with probability 1/2 and a constraint ∑T t=1 λ̃t = 1 (denoted by constrained Bernoulli), and normal distribution with a random mean and a random variance both sampling from a uniform distribution U(0, 1) for each task (denoted by random Normal). We set f as a function of f(λ̃) = λ̃/( ∑T t=1 λ̃t) if p(λ̃) is the Bernoulli distribution or the constrained Bernoulli distribution and a softmax function for the other types of distribution. When sampling from the first five types of distribution, E[λ] is simply proportional to ( 1T , · · · , 1 T ), thus it is fair to compare with the EW strategy. When p(λ̃) is a random Normal distribution, it means each λ̃t is sampled from a normal distribution with random mean and variance, thus it is intractable to compute the expectation for p(λ) and combining with such distribution can further show the effectiveness of RLW.
When sampling from a Bernoulli distribution, the weights for all tasks are either 0 or 1, i.e., λt ∈ {0, 1} for all t. In this way, just a subset of tasks contributes to updating the task-sharing parameters θ. This manner can be viewed as the mini-batch sampling on the task level. If considering an additional constraint that ∑T t=1 λ̃t = 1, it implies only one task is involved in the update of the tasksharing parameters in each iteration. Although there are some works (Dong et al., 2015; Liu et al., 2015a; Søgaard & Goldberg, 2016; Subramanian et al., 2018; Sanh et al., 2019; Liu et al., 2019b) adopting this strategy to train a MTL model, it is a special case in the proposed RLW strategy and beyond existing works, we also provide theoretical analyses to show the effectiveness of the proposed RLW method.

heading: 4 ANALYSIS
text: As the optimization procedure of the RLW method can be viewed as the doubly stochastic optimization, this strategy increases the randomness compared with the fixed loss weights methods optimizing via SGD (denoted by FW), where EW is a special case. In this section, we focus on analyzing how the extra randomness from the loss weights sampling affects the convergence and effectiveness of RLW compared with FW.
In the case of without misunderstanding, we simply use `t(θ) instead of `t(Dt; θ) to denote the loss function of task t for brevity in this section and Appendix B. For the ease of the analysis, we need to make the following assumption.
Assumption 1. The loss function `t(θ) of task t is Lt-Lipschitz continuous w.r.t θ, and that EDt [‖`t(Dt; θ)‖2] = σ2t . Loss weights in λ satisfy Eλ[λ] = µ.
In the following theorem, we analyze the convergence property of Algorithm 1 for RLW.
Theorem 1. Suppose the loss function `t(θ) of task t is ct-strongly convex. We define θ∗ = arg minθ LRLW(θ) and denote by θk the solution in the k-th iteration. When η, the step size or equivalently the learning rate in SGD, satisfies η ≤ 1/2c, where c = min1≤t≤T {ct}, under Assumption 1 we have
E[‖θk − θ∗‖2] ≤ (1− 2ηc)k‖θ0 − θ∗‖2 + ηκ
2c , (4)
where κ = ∑T t=1 σ 2 t . Then for any positive ε, E[‖θk − θ∗‖2] ≤ ε can be achieved after k =
κ 2εc2 log ( ε0 ε ) iterations with η = εcκ , where ε0 = E[‖θ0 − θ∗‖ 2].
Theorem 1 shows that the RLW method with the fixed step size has a linear convergence up to a radius around the optimal solution, which is similar to FW according to the property of the standard SGD method (Moulines & Bach, 2011; Needell et al., 2016). Although RLW has a larger κ than FW, i.e., κFW = ∑T t=1 µ 2 t · ∑T t=1 σ 2 t ≤ κ, possibly requiring more iterations for RLW method to reach the same accuracy with FW, our empirical experiments in Appendix C.1 show that this does not cause much impact in practice.
We next analyze the effectiveness of the RLW method from the perspective of stochastic optimization. It is observed that the SGD method can escape sharp local minima and converge to a better solution than Gradient Descent (GD) techniques under various settings with the help of noisy gradients (Hardt et al., 2016; Kleinberg et al., 2018). Inspired by those works, we first provide the following Theorem 2 and then leverage this theorem to show that the extra randomness in the RLW method can help RLW to better escape sharp local minima and achieve a better generalization performance than FW.
For the ease of presentation, we introduction some notations. Here we consider the update step of these stochastic methods as θk+1 = θk−η(∇µ>`(θk)+ξk), where ξk is a noise with E[ξk] = 0 and ‖ξk‖2 ≤ r. Here r denotes the intensity of the noise. For the analysis, we construct an intermediate sequence ϕk = θk − η∇µ>`(θk). Then we get Eξk [ϕk+1] = ϕk − η∇Eξk [µ>`(ϕk − ηξk)]. Therefore, the sequence {ϕk} can be regarded as an approximation of using GD to minimize the function Eξk [µ>`(ϕ− ηξk)].
Theorem 2. Suppose ∇`t(θ) is Mt-Lipschitz continuous and ‖ξk‖2 ≤ r. If the loss function `t(θ) of task t is ct-one point strongly convex w.r.t a local minimum θ∗ after convolved with noise ξ, i.e., 〈∇Eξ`t(ϕ− ηξ), ϕ− θ∗〉 ≥ ct‖ϕ−θ∗‖2, then under Assumption 1 we have ‖ϕK−θ∗‖2 ≤ 2βρδ with
probability at least 1−δ afterK = 1ρ log ( ρε0 β ) iterations with η ≤ cM2 , where ε0 = E[‖ϕ0−θ∗‖ 2], c = min1≤t≤T {ct}, M = max1≤t≤T {Mt}, ρ = 2ηc− η2M2 and β = η2r2(1 + ηM)2.
Firstly, Theorem 2 only requires that `t(θ) is ct-one point strongly convex w.r.t θ∗ after convolved with noise ξ, which is much weak than the convexity assumption and can hold for deep neural networks. Moreover, this theorem implies that for both RLW and FW methods, their solutions have a high probability to get close to a local minimum θ∗ depending on the noise ξ. Note that by adding extra noise, the sharp local minimum will disappear and only the flat local minimum with a large diameter will still exist (Kleinberg et al., 2018). On the other hand, those flat local minima may satisfy one point strongly convexity assumption in Theorem 2, thus the diameter of the converged flat local minimum is affected by the noise intensity. Due to the extra randomness from loss weights sampling, the RLW method can provide a stronger noise (i.e. a larger r) than FW (referred to Appendix B.3). Hence RLW can better escape sharp local minima and converge to a flatter local minimum than FW, resulting in better generalization performance.

heading: 5 EXPERIMENTS
text: In this section, we empirically evaluate the proposed RLW method by conducting experiments on six computer vision datasets (i.e., NYUv2, CityScapes, CelebA, PASCAL-Context, Office-31, and Office-Home) and four multilingual problems from the XTREME benchmark (Hu et al., 2020). Due to page limit, experimental results of the CityScapes, CelebA, Office-31, Office-Home datasets and two multilingual problems are put in Appendix C.

heading: 5.1 DATASETS
text: The NYUv2 dataset (Silberman et al., 2012) is an indoor scene understanding dataset, which consists of video sequences recorded by the RGB and Depth cameras in the Microsoft Kinect. It contains 795 and 654 images with ground-truths for training and validation, respectively. This dataset includes three tasks: 13-class semantic segmentation, depth estimation, and surface normal prediction.
The PASCAL-Context dataset (Mottaghi et al., 2014) is an annotation extension of the PASCAL VOC 2010 challenge. It contains 10,103 images, which are divided into two parts: 4,998 for training and 5,105 for validation. We consider two tasks with annotations in this dataset: 21-class semantic segmentation and 7-class human part segmentation. By following (Maninis et al., 2019), we generate two additional tasks, including the saliency estimation and surface normal estimation tasks where their ground-truth labels are computed by the label distillation using pretrained state-of-theart models (Bansal et al., 2017; Chen et al., 2018a).
The XTREME benchmark (Hu et al., 2020) is a large-scale multilingual multi-task benchmark for cross-lingual generalization evaluation, which covers fifty languages and contains nine tasks. We conduct experiments on four tasks containing Named Entity Recognition (NER), Part-Of-Speech (POS) tagging, Natural Language Inference (NIL), and Paraphrase Identification (PI) from this benchmark. On each task, we construct a multilingual problem by choosing the four languages with the largest number of data. The more details are provided in Appendix C.6.

heading: 5.2 IMPLEMENTATION DETAILS
text: The network architecture used adopt the hard-parameter sharing pattern (Caruana, 1993), which shares bottom layers of the network for all tasks and uses separate top layers for each task. Other architectures with more parameter sharing manners are provided in Section 5.7. The implementation details on each dataset are introduced in the following.
For the NYUv2 dataset, the DeepLabV3+ architecture (Chen et al., 2018a) is used. Specifically, a ResNet-50 network pretrained on the ImageNet dataset with dilated convolutions (Yu et al., 2017) is used as a shared encoder among tasks and the Atrous Spatial Pyramid Pooling (ASPP) (Chen et al., 2018a) module is used as task-specific head for each task. Input images are resized to 288 × 384. The Adam optimizer (Kingma & Ba, 2015) with the learning rate as 10−4 and the weight decay as 10−5 is used for training and the batch size is set to 8. We use the cross-entropy loss, L1 loss and cosine loss as the loss function of the semantic segmentation, depth estimation and surface normal prediction tasks, respectively. For the PASCAL-Context dataset, the network architecture is similar to that on NYUv2 dataset with a shallower ResNet-18 network used as the shared encoder due to constraints of computing resources. All input images are resized to 512×512. The Adam optimizer with both the learning rate and weight decay as 10−4 is applied for training and the batch size is set to 12. The cross-entropy loss is used for two segmentation tasks and saliency estimation task, while the normal estimation task uses the L1 loss. For each multilingual problem in the XTREME benchmark, a pretrained multilingual BERT (mBERT) model (Devlin et al., 2019) implemented via the open source transformers library (Wolf et al., 2020) is used as the shared encoder among languages and a fully connected layer is used as the language-specific output layer for each language. The Adam optimizer with the learning rate as 2 × 10−5 and the weight decay as 10−8 is used for training and the batch size is set to 32. The cross-entropy loss is used for the four multilingual problems.

heading: 5.3 EVALUATION METRIC
text: To measure the performance of MTL models in a scalar metric, for homogeneous MTL problems (e.g., the Office-31 dataset) which contain tasks of the same type such as the classification task, we directly average the performance metrics among tasks. For heterogeneous MTL problems (e.g., the NYUv2 dataset) that contain tasks of different types, by following (Maninis et al., 2019; Vandenhende et al., 2021), we compute the average of the relative improvement over the EW method on each metric of each task as
∆p = 100%× 1
T T∑ t=1 1 Nt Nt∑ n=1 (−1)pt,n(Mt,n −MEWt,n ) MEWt,n ,
where Nt denotes the number of metrics in task t, Mt,n denotes the performance of a task balancing strategy for the nth metric in task t, MEWt,n is defined similarly for the EW strategy, and pt,n is set to 1 if a higher value indicates better performance for the nth metric in task t and otherwise 0.

heading: 5.4 RESULTS ON THE NYUV2 DATASET
text: The results on the NYUv2 validation dataset are shown in Table 1. It is noticeable that the proposed RLW strategy can achieve comparable performance with SOTA baseline methods. Firstly, RLW with six weight distributions can always outperform EW, which implies that training in a doubly stochastic manner can have a better generalization ability. Secondly, RLW can achieve a balanced improvement on all tasks. That is, RLW has comparable or better performance on each metric in each task when compared with EW, resulting in a large ∆p. Different from RLW, many baseline methods achieve unbalanced performance on all the tasks. For example, IMTL significantly outperforms other methods on the normal prediction task but has unsatisfactory performance on the other two tasks. Hence, this can be one advantage of the proposed RLW strategy since MTL aims to improve the generalization performance of each task as much as possible. Thirdly, RLW with some distributions (i.e., “constrained Bernoulli” and “Normal”) can improve the generalization performance by more than 1%, which is significantly better than baseline methods. Even, RLW with the constrained Bernoulli distribution can entirely dominate not only the EW method but also some baseline methods such as GradNorm, UW, DWA, and GradDrop, on each task, which demonstrates the effectiveness of the RLW method.
Moreover, we compare the average time of training one epoch for each loss weighting strategy with the same batch size (i.e., 8) on a single NVIDIA GeForce RTX 3090 GPU. The relative training speed of each method over the EW method is reported as ∆t in Table 1. Noticeably, the proposed RLW strategy is as computationally efficient as EW, while some baseline methods are computeintensive. For example, PCGrad and GradVac take about twice the time of EW for each epoch because of computing the gradients of parameters. MGDA spends a lot of time to solve a complex quadratic programming problem. Furthermore, ∆t of those baseline methods will increase as the network becomes deeper, while the proposed RLW strategy is architecture-agnostic and always as efficient as EW.
By combining the above analysis, we think that the proposed RLW method is an effective and efficient loss weighting strategy for MTL.

heading: Segmentation Depth Surface Normal

heading: 5.5 RESULTS ON THE PASCAL-CONTEXT DATASET
text: The results on the PASCAL-Context validation dataset are shown in Table 2. The empirical observations are similar to those on the NYUv2 dataset. Specifically, RLW with different distributions can outperform EW, which means RLW has a better generalization ability. Most baseline methods have
unsatisfactory performance on this dataset and the best baseline, i.e., MGDA, achieves the largest ∆p of 0.18%. Thus, RLW outperforms many baseline methods. Moreover, RLW with the constrained Bernoulli distribution achieves the highest improvement of 0.46%. Furthermore, RLW does achieve more balanced improvement than some baseline methods. For example, GradNorm performs not so good on the saliency estimation task, leading to the lowest ∆p. Although IMTL significantly improves the performance of the surface normal estimation task, it performs unsatisfactorily on the other tasks especially the semantic segmentation task. Moreover, the relative training speed of each method over the EW method is similar to the NYUv2 dataset and hence we omit it in Table 2.

heading: 5.6 RESULTS ON THE XTREME BENCHMARK
text: We study four multilingual problems from the XTREME benchmark (Hu et al., 2020) and show experimental results of POS and PI in Table 3. Due to the page limit, the results of NLI and NER are placed in Table 12 in the Appendix. Different from heterogeneous MTL problems on the NYUv2 and PASCAL-Context datasets, in these multilingual problems, each language has its own input data, which is usually called homogeneous MTL problems (Zhang & Yang, 2021). According to the results in Tables 3, RLW with diverse distributions still outperforms EW in all the two multilingual problems, which further shows the effectiveness of RLW on different type of MTL problem.
Besides, RLW achieves comparable and even better performance than those baseline methods. For example, on the POS multilingual problem, RLW has the highest average F1 score and it achieves the best average accuracy on the PI problem.

heading: 5.7 RLW WITH DIFFERENT ARCHITECTURES
text: The proposed RLW strategy can be seamlessly combined with other MTL network architectures without increasing additional computation cost. To see this, we combine the RLW strategy with three SOTA MTL architectures, i.e., cross-stitch network (Misra et al., 2016), Multi-Task Attention Network (MTAN) (Liu et al., 2019a), and CNN with Neural Discriminative Dimensionality Reduction layer (NDDR-CNN) (Gao et al., 2019) and evaluate all the methods on the NYUv2 dataset. Experimental results when using the MTAN architecture are shown in Table 4. Due to page limit, the results for the cross-stitch and NDDR-CNN networks are put in Tables 6 and 7 in the Appendix, respectively.
According to the results, we have some observations. Firstly, compared with the performance when using the DMTL architecture (i.e., the results on Table 1), the performance of each loss weighting strategy with the deeper MTAN architecture is improved, especially on the surface normal estimation task, which is due to the larger capacity of the MTAN. Secondly, the RLW strategy with different distributions can outperform EW, which indicates the effectiveness of RLW with more advanced and deeper architectures. Thirdly, compared with baseline methods, the proposed RLW method can achieve competitive performance. For example, RLW with the random normal distribution can improve over EW by 0.76% and is among the top-3 methods.

heading: 6 CONCLUSIONS
text: In this paper, we have unified eight state-of-the-art task balancing methods from the loss weighting perspective. Based on randomly sampling task weights from distributions, we propose a simple RLW strategy that can achieve comparable performance with state-of-the-art baselines. We analyze the convergence property of the proposed RLW method and the double stochasticity that can help escape sharp local minima. Finally, we provide a consistent and comparative comparison to show the effectiveness of the proposed RLW method on six computer vision datasets and four multilingual tasks from the XTREME benchmark. In our future studies, we will apply the proposed RLW method to more fields such as reinforcement learning.

heading: ETHICS STATEMENT
text: In this paper, we propose the RLW method, a simple yet effective loss weighting strategy for multitask learning. This method is a general machine learning algorithm and we do not apply it to any specific applications. Thus there is no any negative ethics impacts in our paper.

heading: REPRODUCIBILITY STATEMENT
text: For the experiments in this paper, all the datasets we used are publicly available and implementation details including data splits and hyperparameters (i.e., the learning rate, the weight decay, and the batch size) are provided in Section 5.2 and Appendix C. Besides, the source code is put in the supplementary material. For the theoretical results in this paper, we have stated the full set of assumptions in Section 4 and the complete proofs are provided in Appendix B.

heading: APPENDIX

heading: A SUMMARY OF LOSS WEIGHTING STRATEGIES
text: In this section, we summarize the eight SOTA methods introduced in Section 2 from a perspective of loss weighting. We define G = [g1, · · · , gT ] and ut = gt/‖gt‖, where gt = ∇θ`t(Dt; θ) denotes the gradient of `t(Dt; θ) with respect to θ. Let RT+ denote the non-negative subspace in the T -dimensional space RT , 1 denote (1, · · · , 1), I denote an identity matrix, diag(a) denote a diagonal matrix with its principal diagonal a, [·]+ be the ReLU operation, sgn(·) represent the sign function, and I(·) denote the indicator function. The summary of different methods is in Table 5.

heading: B PROOF OF SECTION 4

heading: B.1 PROOF OF THEOREM 1
text: Since `t is ct-strongly convex w.r.t θ, for LRLW(θ) = λ>`(θ), for any two points θ1 and θ2 in Rd, we have 〈
∇λ>`(θ1)−∇λ>`(θ2), θ1 − θ2 〉 = T∑ t=1 λt 〈∇`t(θ1)−∇`t(θ2), θ1 − θ2〉
≥ T∑ t=1 ctλt‖θ1 − θ2‖2. (5)
Since 0 ≤ λt ≤ 1, we have ∑T t=1 ctλt ≥ c, where c = min1≤t≤T {ct}. Then for any λ, LRLW(θ) is c-strongly convex.
With notations in Theorem 1, we have
‖θk+1 − θ∗‖2 = ‖θk − θ∗ − η∇λ>`(D̃; θk)‖2 = ‖θk − θ∗‖2 − 2η 〈 θk − θ∗,∇λ>`(D̃; θk) 〉 + η2‖∇λ>`(D̃; θk)‖2.
Note that Eλ [ ED̃[∇λ >`(D̃; θk)] ] = ∇µ>`(D; θk) and
Eλ [ ED̃[‖∇λ >`(D̃; θk)‖2] ] ≤ Eλ [ ED̃[‖λ >‖2‖∇`(D̃; θk)‖2] ]
≤ Eλ [ T∑ t=1 λ2t ] · T∑ t=1 σ2t
≤ T∑ t=1 σ2t ,
where the first inequality is due to the Cauchy-Schwarz inequality and the third inequality is due to 0 ≤ λt ≤ 1. Then, by defining κ = ∑T t=1 σ 2 t , we obtain
Eλ [ ED̃[‖θk+1 − θ∗‖ 2] ] ≤ ‖θk − θ∗‖2 − 2η 〈 θk − θ∗,∇µ>`(θk) 〉 + η2κ
≤ (1− 2ηc)‖θk − θ∗‖2 + η2κ. (6) If 1− 2ηc > 0, we recursively apply the inequality (6) over the first k iterations and we can obtain
E[‖θk+1 − θ∗‖2] ≤ (1− 2ηc)k‖θ0 − θ∗‖2 + k−1∑ j=0 (1− 2ηc)jη2κ
≤ (1− 2ηc)k‖θ0 − θ∗‖2 + ηκ
2c .
Thus the inequality (4) holds if µ ≤ 12c .
According to inequality (6), the minimal value of a quadratic function gε(η) = (1− 2ηc)ε+ η2κ is achieved at η∗ = εcκ . By setting ‖θ0 − θ∗‖ 2 = ε0, we have
E[‖θk+1 − θ∗‖2] ≤ g‖θk−θ∗‖2(η∗)
= (1− 2‖θk − θ∗‖ 2c2
κ )‖θk − θ∗‖2
≤ (1− 2εc 2
κ )‖θk − θ∗‖2
≤ (1− 2εc 2
κ )kε0.
Then if E[‖θk+1 − θ∗‖2] ≥ ε, we have ε ≤ (1− 2εc 2 κ ) kε0. Therefore, k ≤ κ2εc2 log ( ε0 ε ) .

heading: B.2 PROOF OF THEOREM 2
text: Since ϕk = θk − η∇µ>`(θk) and θk+1 = θk − η(∇µ>`(θk) + ξk), we have
ϕk+1 = ϕk − ηξk −∇µ>`(ϕk − ηξk). Since the loss function `t(θ) of task t is ct-one point strongly convex w.r.t a given point θ∗ after convolved with noise ξ, similar to inequality (5), we have〈
∇Eξ[µ>`(ϕ− ηξ)], ϕ− θ∗ 〉 ≥ c‖ϕ− θ∗‖2,
where c = min1≤t≤T {ct}. Since ∇`t(θ) is Mt-Lipschitz continuous, for any two points θ1 and θ2 in Rd, we have
‖∇µ>`(θ1)−∇µ>`(θ2)‖ = T∑ t=1 µt‖∇`t(θ1)−∇`t(θ2)‖ ≤ T∑ t=1 Mtµt‖θ1 − θ2‖. (7)
Note that ∑T t=1Mtµt ≤ M , where M = max1≤t≤T {Mt}. Therefore, ∇µ>`(θ) is M -Lipschitz continuous. Then we can get
E[‖ϕk+1 − θ∗‖2] = E[‖ϕk − ηξk −∇µ>`(ϕk − ηξk)− θ∗‖2] ≤ E[‖ϕk − θ∗‖2 + ‖ηξk‖2 + ‖∇µ>`(ϕk − ηξk)‖2 − 2 〈ϕk − θ∗, ηξk〉 − 2 〈 ϕk − θ∗,∇µ>`(ϕk − ηξk) 〉 + 2 〈 ∇µ>`(ϕk − ηξk), ηξk 〉 ]
≤ ‖ϕk − θ∗‖2 + η2r2 + E[‖∇µ>`(ϕk − ηξk)‖2]− 2ηc‖ϕk − θ∗‖2 + 2E[ 〈 ∇µ>`(ϕk − ηξk)−∇µ>`(ϕk), ηξk 〉 ]
≤ (1− 2ηc)‖ϕk − θ∗‖2 + η2r2 + η2E[‖M(θ∗ − (ϕk − ηξk))‖2] + 2η3r2M ≤ (1− 2ηc)‖ϕk − θ∗‖2 + η2r2 + η2M2‖ϕk − θ∗‖2 + E[〈ϕk − θ∗, ηξk〉]
+ η2M2E[‖ηξk‖2] + 2η3r2M ≤ (1− 2ηc+ η2M2)‖ϕk − θ∗‖2 + η2r2(1 + ηM)2,
where the second inequality is due to the convexity assumption and E[ξk] = 0, the third and forth inequalities are due to the Lipschitz continuity. We set ρ = 2ηc− η2M2 and β = η2r2(1 + ηM)2. If ρ ≥ 0, we have η ≤ cM2 , then we get
E[‖ϕk+1 − θ∗‖2] ≤ (1− ρ)‖ϕk − θ∗‖2 + β
≤ (1− ρ)k‖ϕ0 − θ∗‖2 + k−1∑ j=0 (1− ρ)jβ
≤ (1− ρ)k‖ϕ0 − θ∗‖2 + β
ρ . So if k ≤ 1ρ log ( ρε0 β ) , we have E[‖ϕk+1 − θ∗‖2] ≤ 2βρ . Then by the Markov inequality, with probability at least 1− δ, we have ‖ϕK − θ∗‖2 ≤ 2β
ρδ .

heading: B.3 NOISE UPPER BOUND
text: Suppose the noise produced by the FW method is ξ̄ = ‖∇µ>`(D̃; θ)−∇µ>`(D; θ)‖ and ‖ξ̄‖2 ≤ R. The noise produced by the RLW method is ξ = ‖∇λ>`(D̃; θ)−∇µ>`(D; θ)‖. We have
‖ξ‖2 = ‖∇λ>`(D̃; θ)−∇µ>`(D̃; θ) +∇µ>`(D̃; θ)−∇µ>`(D; θ)‖2 = ‖(λ> − µ>)∇`(D̃; θ)‖2 + 2 〈 (λ> − µ>)`(D̃; θ), ξ̄ 〉 + ‖ξ̄‖2.
Because the noise ξ̄ can be any direction, there exists a constant s > 0 such that ‖ξ̄‖2 = R and ξ̄ = s(λ> −µ>)∇`(D̃; θ). Then, we have ‖ξ‖2 ≤ (1 + 2s)‖λ−µ‖2‖∇`(D̃; θ)‖2 +R. Thus, the norm of the noise provided by the RLW method has a larger least upper bound than FW.

heading: B.4 CONVERGENCE WITH DECREASING LEARNING RATE
text: In the following theorem, we show the convergence analysis and generalization bound of the RLW method under the strong convexity case with decreasing step sizes.
Theorem 3. Suppose the loss function `t(θ) of task t is ct-strongly convex and Assumption 1 holds. Then if we choose ηk = α/k and α > 1/2c, we have
E[‖θk − θ∗‖2] ≤ max{2α2κ(2αc− 1)−1, ‖θ0 − θ∗‖2}
k ,
where κ = ∑T t=1 σ 2 t and c = min1≤t≤T {ct}. Moreover, by defining the optimal loss function as LRLW(θ∗), we can obtain the following generalization bound as
E[‖LRLW(θk)− LRLW(θ∗)‖] ≤ max{α2Lκ(2αc− 1)−1, L‖θ0 − θ∗‖2}
k ,
where L = max1≤t≤T {Lt}.
Proof. According to inequality (6), by setting η = α/k, we have
E[‖θk+1 − θ∗‖2] ≤ (1− 2αc
k )‖θk − θ∗‖2 +
α2κ
k2 .
Then by induction we can get
E[‖θk+1 − θ∗‖2] ≤ Q
k ,
where Q = max{2α2κ(2αc− 1)−1, ‖θ0 − θ∗‖2}. Since `t(θ) is Lt-Lipschitz continuous, similar to inequality (7), for any two points θ1 and θ2 in Rd, we have
‖∇LRLW(θ1)−∇LRLW(θ2)‖ ≤ T∑ t=1 Ltµt‖θ1 − θ2‖.
Therefore, LRLW(θ) is L-Lipschitz continuous, where L = max1≤t≤T {Lt}. Then we can get
E[‖LRLW(θk)− LRLW(θ∗)‖] ≤ L
2 E[‖θk − θ∗‖2] ≤
LQ
2k ,
where we reach the conclusion.

heading: C ADDITIONAL EXPERIMENTAL RESULTS

heading: C.1 EXPERIMENTS ON CONVERGENCE
text: In Figure 1, we empirically compare the convergence speed of the EW and RLW methods in terms of the performance curve on both the NYUv2 and CelebA validation datasets.
On the NYUv2 dataset with three tasks, the performance curves of the RLW method with three different distributions are both consistent with the EW method, which indicates the RLW method has a similar convergence property to the EW method. As the number of tasks increases, i.e., on the CelebA dataset with forty tasks, we find that the RLW method with the Normal and Bernoulli distributions still converges as fast as the EW method, while the RLW method with the constrained Bernoulli distribution converges slower. One reason for this phenomenon is that only one task is used for updating the parameters in each training iteration.
In summary, the proposed RLW strategy combined with all the distributions except the constrained Bernoulli distribution has similar training efficiency to the EW method.

heading: C.2 ADDITIONAL RESULTS OF RLW ON DIFFERENT ARCHITECTURES
text: The results of different loss weighting strategies with the cross-stitch and NDDR-CNN networks are put in Tables 6 and 7, respectively. The empirical observations are similar to those with the MTAN architecture (i.e., Table 4). Firstly, the performance of each loss weighting strategy with the deeper NDDR-CNN architecture is improved on the three tasks, while only the surface normal estimation task can be improved with the cross-stitch network. Secondly, the RLW strategy significantly outperform EW on both architectures. Thirdly, compared with the baseline methods, RLW can achieve comparable or even better performance than all the baseline methods except IMTL on both two architectures. For example, RLW with the constrained Bernoulli distribution has an improvement of 1.79% on the cross-stitch architecture, which is slightly lower than the best performed method, IMTL. Besides, RLW with the Normal distribution is among the top-2 methods based on the NDDR-CNN network.

heading: C.3 RESULTS ON THE CITYSCAPES DATASET
text: Dataset The CityScapes dataset (Cordts et al., 2016) is a large-scale urban street scenes understanding dataset and it is comprised of a diverse set of stereo video sequences recorded from 50 different cities in fine weather during the daytime. It contains 2,975 and 500 annotated images for training and validation, respectively. This dataset includes two tasks: 7-class semantic segmentation and depth estimation.
Implementation Details The network architecture and optimizer on the CityScapes dataset are the same as those of the NYUv2 dataset. We resize all the images to 128 × 256 and set the batch size as 64 for training. The Adam optimizer (Kingma & Ba, 2015) with the learning rate as 10−4 and the weight decay as 10−5 is used. We use the cross-entropy loss and L1 loss for the semantic segmentation and depth estimation tasks, respectively.
Results According to the results shown in Table 8, we can see that all the loss weighting strategies except the GradDrop method outperform EW. Especially, the PCGrad and MGDA methods can achieve a significant improvement of 2.79% and 2.60% in terms of ∆p, respectively. Compared with those baseline methods, the RLW strategy not only outperforms EW by using the six different distributions but also achieves comparable performance with those baseline methods. For example, RLW with the Dirichlet distribution can improve over EW by 1.95% and is among the top-3 methods.

heading: C.4 RESULTS ON THE CELEBA DATASET
text: Dataset The CelebA dataset (Liu et al., 2015b) is a large-scale face attributes dataset with 202,599 face images, each of which has 40 attribute annotations. It is split into three parts: 162,770, 19,867 and 19,962 images for training, validation and test, respectively. This dataset contains 40 tasks and each task is a binary classification problem for one attribute.
Implementation Details We use the ResNet-18 network as a sharing feature extractor and a fully connected layer with two output units as a task-specific head for each task. All the input images are resized to 64 × 64. The Adam optimizer with the learning rate as 10−3 is applied for training and the batch size is set to 512. The cross-entropy loss is used for the 40 tasks.
Results Since the number of tasks is large, we only report the average classification accuracy on the forty tasks in the CelebA dataset and the results are shown in Table 9. The proposed RLW strategy slightly outperforms EW and performs comparable with all the baseline methods.

heading: C.5 RESULTS ON THE OFFICE-31 AND OFFICE-HOME DATASETS
text: Datasets The Office-31 dataset (Saenko et al., 2010) consists of three domains: Amazon (A), DSLR (D), and Webcam (W), where each domain contains 31 object categories. The Office-31 dataset contains 4,110 labeled images and we randomly split these samples with 60% for training, 20% for validation, and the rest 20% for test. The Office-Home dataset (Venkateswara et al., 2017) has four domains: artistic images (Ar), clip art (Cl), product images (Pr), and real-world images (Rw). It has 15,500 labeled images in total and each domain contains 65 classes. We divide the entire data in the same proportion as Office-31. For both two datasets, we consider the multi-class classification problem on each domain as a task. Similar to multilingual tasks from the XTREME benchmark, each task in both Office-31 and Office-Home datasets has its own input images, thus MTL problems in these two datasets are homogeneous MTL problems (Zhang & Yang, 2021).
Implementation Details We use the same configuration for both Office-31 and Office-Home datasets. Specifically, the ResNet-18 network pretrained on the ImageNet dataset is used as a sharing backbone among tasks and a fully connected layer is applied as a task-specific output layer for each task. All the input images are resized to 224 × 224. We use the Adam optimizer with the learning rate as 10−3 and set the batch size to 64 for training. The cross-entropy loss is applied for all tasks of both two datasets.
Results According to the results on Office-31 and Office-Home datasets shown in Table 10, it is noticeable that RLW with six different types of distributions outperforms EW on both datasets. In addition, the RLW method performs better than most baseline methods. Moreover, the RLW method with the constrained Bernoulli distribution achieves the best average classification accuracy on both Office-31 and Office-Home datasets, and this strategy achieve 100% accuracy on the DSLR domain in the Office-31 dataset.

heading: C.6 ADDITIONAL RESULTS ON XTREME BENCHMARK
text: Datasets The datasets used in the NER, POS, and PI tasks are the Wikiann dataset (Pan et al., 2017), Universal Dependency v2.5 treebanks (Nivre et al., 2020), and PAWS-X dataset (Yang et al., 2019), respectively. For the NIL task, the MultiNLI dataset (Williams et al., 2018) and the XNLI dataset (Conneau et al., 2018) are used for training and evaluation, respectively. For the NER, NIL, and PI tasks, the same four languages including English (en), Mandarin (zh), German (de), and Spanish (es), are used. For the POS task, Telugu (te) and Vietnamese (vi) instead of German and Spanish are used. The numbers of data for all the tasks are summarized in Table 11.
Results The results on the NLI and NER multilingual tasks are shown in Table 12. The empirical observations are similar to those on the POS and PI tasks shown in Table 3. Specifically, the RLW method with all the distributions outperforms EW in the two multilingual problems. In addition, RLW achieves comparable and even better performance than those baseline methods. For example, on the NLI multilingual problem, RLW achieve the highest average accuracy.

heading: C.7 RESULTS OF DIFFERENT BACKBONES ON NYUV2 DATASET
text: Those loss weighting strategies are agnostic to both architectures (or sharing pattern) and backbone networks. Tables 13 and 14 show consistent results when applying the SOTA methods and the proposed RLW strategy on different backbone networks, i.e., ResNet-18 and ResNet-101, respectively.
Compared the results of the Tables 13, 1 (or 15), and 14, we can find that with the depth of the backbone network increasing, the performance of each loss weighting strategy is improved. In addition, the proposed RLW method can always outperform EW on all backbones. On the other hand, the performance of those SOTA methods could degrade on deeper backbones. For example, the best relative improvement (i.e., ∆p) of SOTA methods is 1.58%, 0.48%, and 0.37% for ResNet18, ResNet-50, and ResNet-101 backbones, respectively. Compared with those baseline strategies, the RLW method can always achieve a significant improvement and even outperform the SOTA baselines on all the backbones.
Except the relative training speed ∆t, we also provide the difference of training speed for one epoch between each method and the EW method as ∆dt in Tables 13, 14, and 15 to better show the computational efficiency of the proposed RLW strategy. It can be observed that most of the SOTA methods, especially MGDA, PCGrad, and GradVac, need more training time (i.e., ∆dt ) when the backbone becomes deeper. One main reason is due to the computation of all task gradients. Compared with those baseline methods, the proposed RLW strategy brings negligible and backbone-agnostic computational cost, which indicates RLW is as efficient as EW. Even, RLW with Bernoulli and constrained
Bernoulli distributions are even more efficient than EW since only some tasks are involved in the loss computation. Moreover, although DWA and UW are efficient since DWA computes loss weights via the loss ratio and UW updates loss weights by gradient descent methods, those two methods do not perform well in all cases. For example, the performance of DWA is 1.02% lower than EW on the NYUv2 dataset with the ResNet-101 backbone (in Table 14). Differently, the proposed RLW strategy can not only keep the efficiency but also outperform EW and achieve comparable performance with SOTA methods in all cases.

heading: C.8 LEARNING TO COMBINE THE DISTRIBUTIONS FOR RLW
text: In this section, we proposed a method, called RLW-L2C, to learn a combination of those six different distributions for RLW. Specifically, in each iteration, we firstly sample loss weights from each of six distributions, then sum those weights elementwisely weighted by learnable distribution weights, and finally normalize the aggregated weights by the softmax function. The results of RLW-L2C on the NYUv2 dataset are shown in Table 15. Although the performance of RLW-L2C is slightly inferior to the best performance of RLW with single distribution (i.e. 0.09% performance degradation), this method still significantly outperforms those SOTA baselines and it can be adopted to any datasets and architectures by adaptively learning how to combine different distributions. Moreover, RLWL2C is computationally efficient by only increasing about 0.01% computational cost over EW. Thus, those empirical results show that RLW-L2C is an adaptive, effective, and efficient loss weighting strategy for MTL.

Review summaries:
This paper conducted an empirical study of existing loss weighting based multi-task learning techniques. By comparing them in both CV and NLP multi-task benchmark datasets, the authors also identified a simple random loss weighting method achieving similar performance. This paper provided both empirical and theoretical analysis of why random loss weighting can achieve similar performance efficiently.

Authors propose a dynamic loss weighting algorithm that chooses random loss weights per step from a chosen distribution in a multitask setting. Model with these dynamic weights is then trained until convergence. Theoretical analysis on convergence is provided and experimental results on NYUv2, PASCAL-Context, and XTREME are given. The method seems to perform relatively on par or a bit worse compared with other SOTA methods but does not require computation of gradients. 

Here, the authors present a novel method for multitasking loss weighting called Random Loss Weighting (RLW). The method is straightforward and clearly described; RLW considers the loss weights lambda as random variables. In practice, it works by sampling the loss weights from a distribution (authors evaluated six different distributions) with some normalization and then minimizing the aggregated loss weighted by the normalized random weights. The authors provided an extensive empirical evaluation including eight state-of-the-art loss weighting strategies on six computer vision datasets and the XTREME benchmark (four multilingual problems). RWL results are comparable to previous strategies, but with a lower computational burden. The authors also provided a theoretical analysis on how the extra randomness from the sampling affects the convergence and effectiveness of RLW compared with fixed loss weights optimized via SGD.