Scalarization for Multi-Task and Multi-Domain Learning at Scale
Authors: Amelie Royer
Tijmen Blankevoort
Babak Ehteshami Bejnordi
Venue: NeurIPS 2023 poster (2023)
Keywords: multitask
multidomain
optimization
population based training
TL;DR: A large-scale study of scalarization for multi-task and multi-domain training which harnesses population-based training for efficient search.

Abstract: Training a single model on multiple input domains and/or output tasks allows for compressing information from multiple sources into a unified backbone hence improves model efficiency. It also enables potential positive knowledge transfer across tasks/domains, leading to improved accuracy and data-efficient training. However, optimizing such networks is a challenge, in particular due to discrepancies between the different tasks or domains: Despite several hypotheses and solutions proposed over the years, recent work has shown that uniform scalarization training, i.e., simply minimizing the average of the task losses,  yields on-par performance with more costly SotA optimization methods. This raises the issue of how well we understand the training dynamics of multi-task and multi-domain networks. In this work, we first devise a large-scale unified analysis of multi-domain and multi-task learning to better understand the dynamics of scalarization across varied task/domain combinations and model sizes. Following these insights, we then propose to leverage population-based training to efficiently search for the optimal scalarization weights when dealing with a large number of tasks or domains.

Scalarization for Multi-Task and Multi-Domain Learning at Scale
Amélie Royer, Tijmen Blankevoort, Babak Ehteshami Bejnordi
Qualcomm AI Research"
Amsterdam, The Netherlands
(aroyer, tijmen, behtesha)@qti. qtt.tii..all.oo..ooomm
Abstract
Training single model on multiple input domains and/or output tasks allows for compressing information from multiple sources into unified backbone hence improves model efficiency. also enables potential positive knowledge transfer across tasks/domains, leading to improved accuracy and data-efficient training. However, optimizing such networks challenge, in particular due to discrepancies between the different tasks or domains: Despite several hypotheses and solutions proposed over the years, recent work has shown that uniform scalarization training, i.c. simply minimizing the average of the task losses, yields on on-par performance with more costly SotA optimization methods. This raises the issue of how well we understand the training dynamics of multi-task and multi-de multi-domain networks. In this work, we first devise a large-scale unified analysis of multi-domain and multi-task learning to better understand the dynamics of scalarization across varied task/domain combinations and model sizes. Following these insights, we then propose to leverage population-basel training to efficiently search for the optimal scalarization weights when dealing with large number of tasks or domains.
Introduction
Learning a unified architecture that can handle multiple domains and/or tasks offers the potential for improved computational efficiency, better generalization, and reduced training data requirements. However, training such models proves a challenge: In particular, when the tasks or domains are dissimilar, practitioners often observe that the learning of one task or domain interferes with the learning of others. This phenomenon is commonly known as interference or negative transfer. How best resolve this interference an open problem spanning numerous bodies of literature, such as multi-task learning, domain adaptation and generalization, or multi-modal learning.
In fact, there is currently unanimous understanding of what causes, how predict, task interference in practice: For instance, classical generalization bounds on training from multiple data sources [3, 48, 56, 44] involve measure of distance between the respective tasks/domains distributions, which can be intuitively thought of as measure of interference. In more practical applications, task clustering methods [65, 58, 16] propose various metries to measure task affinity, and train tasks with low affinity separately, with separate backbones, thereby circumventing the interference issue at the architecture level. Finally, in the multi-task optimization (MTO) literature. the prevailing view is that gradients from different tasks may point in conflicting directions and cause interference, which has led to line of work on reducing gradient conflicts by normalizing gradients or losses statistics [37, 7, 25, 64]. While MTO have established new state-of-the-an art for training multi-task models in the past few years, recent work [63, 30] shows that, surprisingly, minimizing
Quilcomm Al Research initiative of Qualcomm Technolagies, Inc.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

simple average of the tasks/domains losses, also known as scalarization, can yield performance trade-off points on the same Pareto front as more costly MTO methods.
Nevertheless, there still remains some unexplored areas which we aim to shed lights on in this paper. First, most previous experiments have only been conducted multi-task settings, on benchmarks with either few tasks or few training samples (e.g multi-MNIST, NYU-v2), and for a fixed architecture. In particular, the link between model capacity and the observed MTL performance is often overlooked. Secondly, it is not clear how the choice of scalarization weights links to and/or impacts the hypothesis of gradient conflicts as a key underlying signal of task interference. Finally, scalarization can become prohibitively expensive for a large number of tasks as the search space for the optimal loss weights grows exponentially; This raises the issue of how efficiently browse this search space. In this work, we investigate these questions to further motivate scalarization as a simple and scalable training scheme for multi-task and multi-domain problems.
Our key contributions. We perform a large-scale analysis of scalarization for both multi-task (MTL) and multi-domain learning (MDL). We cover a wide range of model capacities, datasets with varying sizes, and different task/domain combinations. Our key conclusions are as follows:
(C1) When compared a model trained on each task/domain individually, MDL/MTL performance tends to improve for larger model sizes, showing that the benefits of MTL/MDL frameworks should be put into perspective with respect to the backbone architecture capacity.
(C2) Tuning the scalarization weights for specific tasks/domains combination is crucial to obtaining the optimal MTL/MDL performance in settings with high imbalance; Nevertheless, the relative performance of different scalarization weights is often consistent across model capacities inside a family architecture. This suggests that searching for optimal scalarization weights for a lower model depth/width is also relevant for the full model size, while using less compute.
(C3) Gradients conflicts between tasks/domains naturally occur during MTL training: They behave differently across layers and learning rates, but are scarcely impacted by model capacity and scalarization weights choice. These observations give new insights into the practical implications of avoiding conflicting gradients throughout training.
(C4) We leverage fast hyperparameter search methods such as population-based training [24] to efficiently browse the search space of scalarization weights as the number of tasks grows.
2 Related work
Multi-Task Optimization (MTO) and scalarization. MTO methods aim to improve MTL training by balancing the training dynamics of different tasks. Prior research can be split into two categories: On the one hand, loss-based methods propose to align the task losses magnitudes by rescaling them through various criteria, e.g., task uncertainty [26], task difficulty [21], random loss weighting [34], or gradients statistics [7, 14, 60]. On the other hand, gradient-based methods directly act on the per-task gradients rather than the losses. For instance, [55, 13] tackle MTL as a multi mujti-obeecctvv optimization problem using the Multiple Gradient Descent Algorithm to directly locate Pareto- Pareto-optimal solutions. Another major line of work considers conflicting gradient direction to be the main cause of task interference. Consequently, these works [64, 37, 38, 62, 8, 25] aim mitigate conflicts across per-task gradients. For instance, PCGrad [64] suggests projecting each task's gradient onto the normal plane of other gradients to suppress confficting directions, while GradDrop [8] ensures that all gradient updates are pure in sign by randomly masking all positive or negative gradient values during training. Unfortunately, gradient-based techniques require per-task gradients, leading to substantial memory usage and increased runtime. Furthermore, recent work [63, 30] has shown that these methods often perform on-par with the less costly approach of directly optimizing the average of the task losses, also known as uniform scalarization. Building off these insights, we further investigate the benefits of scalarization and the practical dynamics of gradient conflicts in this work.
Architectures for MTL. Orthogonal to our work, another line of research focuses on designing multi-task architectures that mitigate task interference by optimizing the allocation of shared versus task lask-specific parameters. Hard parameter sharing methods [27, 26, 6, 59, 61] build off a shared backbone and carefully design task-specific decoder heads optimized for the tasks of interest. In contrast, soft parameter sharing methods [45, 53, 20, 39] jointly train a shared backbone while learning sparse binary masks specific to each task/domain which capture the parameter sharing pattern across tasks. To select which tasks should share parameters or not, prominent line of work

---

focuses on defining measures of task affinity [2, 58, 22, 16]: Tasks with high affinity are jointly trained, while low affinity ones use different backbones, in the hope of reducing potential interference by fully separating conflicting tasks. This results in multi-branch architectures where each branch handles specific subset of tasks with high affinities. The scope of our study complementary to this line of work, as we focus on how to best train a given set of tasks, without changing the architecture.
Multi-Domain Learning (MDL). MDL refers to the design and training of single unified architecture that can operate on data sources from different domains. Domain adaptation and generalization methods often aim to align the learned representations of the different domains [19, 4, 66]: These are in particular targeting the setting where one of the domain lacks supervisory signals. In the fully-supervised scenario, several methods have been proposed to use unified backbone, which captures common features across all tasks, while allowing the model to learn domain -specifcc information via lightweight adapter modules [49, 51, 50J. Similar to soft parameter sharing method, these methods bypass task interference by keeping the shared backbone frozen and only training domain-spp senain- modules. Alternativlyy, common practice when training with multiple input datasets is to use over- or undersampling techniques [1, 28, 43], in particular to handle class imbalance. As we describe later, resampling can be seen as the MDL counterpart of scalarization in MTL.. Both enable a simple and lightweight training scheme, yet finding the optimal resamplingsscllaivatoinn weights is nontrivial. This motivates empirically verify our findings in the MDL well MTL setting.
3 Motivation and experimental setting for MTL/MDL
3.1 Notations
We start by describing the MTL/MDL setup: Given T' supervised datasets with respective inputs X and outputs Y, our goal is to find model parameters 0 that minimize the datasets' respective losses: (C, (X1, (1),...,TT(,,
Y))). In practice, this can be achieved either by solving multi-objective problem on the task losses [13, 42] or, more commonly, by minimizing (possibly weighted) average of the losses [27]. In both cases, most training methods for MTL/MDL can be phrased as updating the model parameters 8 using a weighted average of the individual tasks' gradients:
T
0i+1 =0 0, +1
Σ
Pi E (( Vz Ce Mt). where 7j is the learning rate
where pii) = (pi.....,P) ..T))) captures the dynamic importance scaling weights for each dataset at timestep i, and q(x,, Y)) is the underlying distribution the t-th dataset is drawn from. We also follow the common assumption that is a probability distribution, i.e., >, = 1 and Vt, E {0, 1}.
We further distinguish between (i) multi-task learning (MTL) where every input is annotated for every task (i.e., Vi,j, X, = Xj). and (ii) multi-domain learning (MDL) where the goal is to solve a common output task across multiple input domains modalities (vi,j, Y, Yj). In the MDL setting, also rephrase (I) as resampling the data distributions according to p. Both formalisms are equivalent, but resampling often performs better in practice when using stochastic gradient methods [1]:
T Σ EPEBgtx..N)) v) (xx where (x () = Σ 1Lex, ((x,))p t=1
Reweighing formalism of (1)
Resampling formalism
Resampling distribation
3.2 Motivation
Current state-of-the-an methods for training objectives such as (1) can be roughly organized into three categories, based on how the datasets' importance weights p defined:
Scalarization defines the weights Pt constant hyperparameters: The update rule of (1) reduces to computing the gradient of the weighted average loss Vo (E,,,t P.C.). the MDL formalism of where pt is defined either as
(2), scalarization is similar to classical weersamlinggunddesaman scalar (per-domain) or vector (to handle both per-class and per-domain biases). both settings, the key difficulty lies in tuning the weights p, as the search space grows exponentially with T.
3

---

Loss-based adaptive methods [38, 26] dynamically compute Ps for every batch, aiming to uniformize training dynamics across tasks by rescaling the losses. For instance. IMTL-L [38] dynamically reweighs the losses such that they all have the same magnitud..
Gradient-based adaptive methods [7, 64, 38, 25, 8, 37, 13, 55, 36, 42] also dynamically compute weights for every batch. While this line of work usually outperforms loss-based methods, they also incur higher compute and memory training cost, they require T backward passes to obtain each individual dataset gradient, which in turn need to be stored in memory.
While gradient- gradient-based approaches are generally considered SotA across MTO methods, they present certain practical challenges: They come with higher computational and memory cost, which increases with T, as illustrated in Figure 1(b). The implicit weights p: computed by these methods are also hard to extract and transfer to other training runs since they are tied to other hyperparameters (training length, batch size, gradient clipping, etc.). In addition, recent work [63, 30, 52] has shown that simple scalarization with uniform weights actually often performs on-par with both loss- and gradient-basee methods. Furthermore, scalarization is highly practical: It does not incur additional costs during training. and scalarization weights are easily interpretable "task importance" in the optimization problem. However, some experiments in [58, 52] also suggest that the benefits of uniform scalarization MTL are impacted by model capacity in some settings.
measure of
Motivated by these insights, we aim to better harness scalarization for scalable and practical MTL/MDL training. In particular, we focus on the following two remaining gray areas: First, we perform a large-scale analysis investigating the underlying effects of model capacity on MTL and MDL (C/) in Section 4. We then discuss the impact of model capacity on the choice of optimal scalarization weights (C2) in Section 4.1 and on the presence of conflicting gradients (C3) in Section 4.2. Secondly, browsing the search space of p to tune the scalarization weights becomes increas- ingly costly as the number of tasks T grows. As an alternative to the usual grid- or approaches, we propose to leverage population-baseed training as cost-efficient parameter search for scalarization, and report our conclusions (C4) in Section 5.
MDI.
MTL.
pcgrad
cagrad
uncertainty
gradnomm
CI [29.9]
DemainNet [47]
CelchA [40]
Taskonomy
graddrop
Imtl
Dataset
[65] (iny split)
Peak memory (GB)
20
# tasks or domains, T
40
7
15
Training size. [X]
55k
162k
275k
10
ResNet-26
Backhone
VIT./44
ResNei- 101
VIT-s/4
encoder/eennder
5
Depth
(4, 0) shared
sweep
63 9)
[r26, r101)
6,
decoder layers
0
2
5
10
20
Width
[0.25, 0.5.
[0.25, 0.5, 1.0]
(0.25, 0.5. 0.75, 1.0]
(0.5, 1.0]
Number of tasks
sweep
0.75, 1.0)
(b) Memory usage of various multi-task
(a) Summary of our experimental setting
optimization (MTO) methods
Figure 1: Table (a) summarizes the experimental setups used throughout the paper. Figure (b) reports profiling results for popular multi-task optimization (MTO) methods in small-scale (ResNet18 8, batch size 16) and large-scale (ResNet50, size 128) setting, for 224px inputs: In practice, we find that the main bottleneck for gradient-basee training with reasonable batch sizes requires either high parallelism (compute demand) or gradient accumulation (slower runs). In contrast, loss-based methods have much better-constraiied costs. However, they usually underperform their gradient-basee counterpart in the literature.
methods is their high memory usage. Consequently,
3.3 Experimental setting
Here briefly describe the experimental setup that is used throughout the paper. We aim to cover wide range of model sizes, datasets with varying sizes, and different task/domain combinations for
MDL and MTL. A summary is given in Figure 1(a) and further details can be found in Appendix 1.
For MDL, we first consider a two-domain example composed of CIFARIO [29] and STL.10 [9] with VIT-S backbone optimized for small datasets [18]. We then expand the results to the DomainNet

---

benchmark [47] containing 345 output classes and 6 input domains. We use ResNet as our main backbone, following previous works [47, 32]. In both cases, the backbone parameters are fully shared across all domains; We use cross-entropy as training loss and top-1 accuracy as our main metric.
For MTL, use CelebA [40] with the same ViT-S backbone as for CIFAR/STI: We split the 40 attributes into semantically-cohere groups to form 8 output classification tasks, as described in the appendix. The transformer backbone is fully shared across tasks, while the last linear classification layer is task-specific. Then, we experiment on a larger MTL setting for dense prediction: Most traditional benchmarks, such as Cityscapes [10] and NYU-v2 [57]. are rather small, as they only contain 2-3 dense tasks (segmentation. normals, depth) and ~ 5k fully annotated images. Instead, we perform our analysis on the challenging Taskonomy dataset [65]. To keep experiments scalable, we filter the 26 tasks in Taskonomy down to 7 (e.g. by removing self-supervised tasks or clearly overlapping ones such as 2D edges and 3D edges). For training. we follow the setup described in [58]: We normalize every dense annotation to have zero mean and unit variance across the training set, and use the L loss as our training loss and common metric for all tasks. The backhone is composed of a (shared) ResNet encoder, followed by task-specific decoders built with upsampling operations and Ixl convolutions. To control model capacity, we vary (i) the width of the model, (ii) the depth of the encoder, well as (iii) the number of layers in the decoder(s) which are shared across tasks.
Finally, in all settings, we sweep over the domains/tasks' weights under the common assumption that p is a probability vector (i.e. Vt, Ps E {0. 1 and Σ, Pt = 1). All reported results are averaged across 2 random seeds for DomainNet and Taskonomy, and 3 for the smaller benchmarks. Unless stated, every experiment is conducted on single NVIDIA V100 GPU.
4 Benefits of MDL/MTL under the lens of model capacity
We first investigate the behavior of scalarization for MDL/MTL training with two tasks, while varying model capacity and for different tasks/domains combinations. To quantify the benefit or harm of joint training with multiple datasets, we compare the model performance on each dataset with that of a same-sized model, trained on a single dataset at once. We refer to this baseline as "SD" (single dataset). Since we use SD as a reference point, we ensure that its training pipeline is well tuned: We sweep over the hyperparameters (learning rate, weight decay, number of training steps, ett) of the baseline SD and keep the same values for training the MDL/MTL models, while varying the task weights (P1.P2 1 - p1) from P1 0 (SD baseline of the first dataset) to p1 1 (SD baseline of the second dataset). Further details on the training hyperparameters can be found in Appendix 1. Finally, we summarize our analysis results in Figure 2: For each task/domain pair, we first plot the weight pi yielding the best accuracy averaged across both tasks/domains, then we report the accuracy difference between the corresponding MDL/MTL model and the associated SD baseline.
Impact of model capacity. We first observe that MDL/MTLL performance greatly varies across model capacities, and tends to increase with model size (CI): In some cases, the trend even inverts from MDL/MTL underperforming to outperforming SD (c.g. DomainNet's 'ss real sketch pairing).
Selecting optimal importance weights p. Secondly, the best-performing p at inference are rarely the uniform P1 P2 = 0.5, even in the Taskonomy scenario where the uniform training objective is identical to the average of task metrics which we aim to maximize at inference. We further discuss the effects of tuning scalarization weights for the accuracylefficieccy trade-off (C2) in Section 4.1.
MTL/MDL primarily improves generalization. As we show in Appendix 2), even when MTL/MDL outperforms the SD baseline at inference, it is generally not the case at training time. In other words, tuning the respective tasks/domains weights in MTL/MDL can be seen as a regularization technique that improves generalization over the SD basclines by balancing the two datasets' training objectives. In particular, in the MDL setting, this insight is also closely connected to data augmentations: Each new domain can be seen a non- non-parametric and non-trivial augmentation function, while the associated resampling weight Pt is the probability of applying the augmentation on each sample.
Interestingly, recent work [30] suggests that many MTO methods which aim to avoid gradient conflicts can also be interpreted as regularization techniques, and that simpler tricks such as early stopping are often competitive. To better understand the link between these two insights, we analyze the natural emergence of conflicting gradients during MDL/MTL training, and how they are impacted by model capacity (C3) in Section 4.2.
5

---

real domain
clipart
infograph
painting
quickdraw
sketch
=|=53525
X 33023
X 50416 -X|=120750
X 4
+CIFAR10
depth depth +
real
depth
P57
Aacc reat
1,39
STL10
13:99
0.1
05 0.75 20
depth
-2.30
-100
-390
Acc CIFAR10
-10
Dac dod.dln
4.50
depth
0333
depth
3.68
-2442
20
30
width
width
width
width
width
width
(a) Analysis of MDL when pairing natural images (real domain) with each of the five remaining DomainNet domains (left), and the CIFAR- 10 STL-10 toy example (right)
gender
gender
haircolor
hairstyle
hairstyle
normal
Seam2a
Segmbd
+age
+clothes
+hairstyle
+clothes
+mouth
thared
depei
ceoder
Lo
Decc, book]
tacks
shared
depth
La a0100
Dacc, tasid
Dacc tock2
shared
depth
decodurinn
width
width
wiltth
-
-iitth
wwitt
wllth
-width
wiitth
(b) Analysis of MTL on CelebA (left) and Taskonomy tiny (right) for different task pairs and model sizes.
Figure 2: Performance of scalarization for MDL/MTL relative SD under different model capacities; Each column corresponds to a different task/domain pair (T = 2). The first row each plot contains heatmap of the best performing scalarization weights p" wrt. the average test accuracy both tasks. Each of the following rows contains the difference in metrics between the MTL/MDL model and its counterpart SD baseline, where green indicates positive changes and purple, negative one. Note that the colormaps' ranges are defined per row, and visualized as color bars at the beginning of each row. We observe the general trend that the performance improvement of MDL, relative the corresponding SD bascline, tends to increase with model capacity.
4.1 Selecting optimal scalarization weights p*
So far, we have discussed MDL/MTL improvement over the SD baseline when selecting the weights p" that maximize average accuracy. However, real-world settings, many criteria come into play when evaluating MTL/MDL models: For instance, one task may be more critical than others hence the model should be evaluated via weighted accuracy. Second, even the MTL/MDL model underperforms the corresponding SD baseline at the same model capacity, it may still improve afficiency-to-accuraay trade-off when both tasks need to solved at once. To give a clearer overview of how these considerations affect the choice of weights p. we report results for all scalarization weights in Figure 3; We also highlight points where MTL/MDL outperforms SD on both domains/tasks, and represent each model's 's number of parameters as the marker size.
First, observe clear asymmetry in terms of performance across the T 2 datasets, even when using uniform weighing P1 = 0.5: For instance Taskonomy (Figure 3), several MTL models outperform SD on the depth prediction task, but none on the semantic segmentation task. Nevertheless, MTL models are very appealing when taking model efficiency into account, as they contain roughly half as many parameters. Secondly, tuning some settings: For instance in DomainNet, training with Pı E [0.65, 0.75) is more advantageous than uniform scalarization. Furthermore, the relative ranking of the weight Pr, with respect model performance, does not change significantly across model capacities, for both the Taskonomy and
scalarization weights is crucial in

---

DomainNet examples, This suggests that optimal weights p" for given model may be good search starting point for another architecture of the same family (see Appendix 4).
Nevertheless, the search space for p grows exponentially with 7, making the search computationally prohibitive, for architecture. In Section 5, we propose scalable approach optimize scalarization weights and investigate its performance on DomainNet and CelebA
painting domain accuracy
r26
depth L11 distance
SD
r50
MTL shr
r101
MTL shr
SD
Py=0.25
P.50.25
P1-0.35
P:=0.35
P1=0.5
P1=0.65
P.5
Py=0.75
0.50.65
P:m0.75
54
15 58 68
64
66
66
segm2d
L1 distance
real' domain accuracy
(b) Taskonomy normals + depth (lower is better).
(a) DomainNet- real + painting (higher is better)
"shr" refers to the number of shared decoder layers.
Figure 3: MDL/MTL performance when varying tasks weight p1 for different model capacities. High opacity markers represent models that outperform their respective SDL baseline (black markers) on both tasks/domains. Dashed lines connect models with the same architecture. The marker size is proportional to the number of parameters each model.
4.2 Conflicting gradients in practice
A widely spread explanation for task interference in the literature is that individual task gradients may point in conflicting directions, hampering training. To investigate this behavior, we measure the percentage of conflicting gradients pair encountered in each epoch, when training with uniform scalarization. Following [64], we define gradients as conflicting if and only the cosine of their angle is negative. Finally, we provide further details and figures for this section in Appendix 3.
In Figure 4, we first illustrate an asymmetric characteristic of gradient conflicts: A high number of gradient conflicts typically translates to poorly performing models (e.g. early training), but the lower conflicts regime does not correlate well with MDL/MTL performance. In particular, it is common to encounter more gradient conflicts towards the end of training, while the loss steadily decreases. This suggests that, in practice, identifying and removing conflicts at every training iteration may be superfluous, with respect to the compute and memory cost occurred. We can also put this observation in perspective with Theorem of [64], which states that in the case of two tasks, a parameter update of PCGrad leads lower loss than the uniform scalarization update if the tasks conflicting enough; however, this assumption may not hold true across every training iteration.
project-True
project- False
% gradient Confiict
0.50
0.50
0.45
5.40
6.35
0.35
(r=0.06)
0.3
30
20
42
30
5.22
0.25
0.50
0.25
0.40
2,50
0.52
0.44
2.89
0.28
0.30
Training epoch
Training loss
Validation metric
(a) Per-epoch proportion of gradient conflicts during training
(b) Correlation plot with
(c) Correlation plot with per-epoch test metric
per-epoch average training loss
Figure 4: Proportion of conflicting gradients during uniform scalarization training with and without PCGrad on the 40 attribute classification tasks of CelebA while varying learning rate ([5e-4, 5e-3, le-2)), model depth ([3, 9]) and width ([0.5, 0.75, IJ). While the overall number of encountered conflicts differs, the trend is consistent across both settings in that the higher number of conflicts encountered towards the end training does not harm training final model performance.

---

confticting gradients
sketch
edges_
variance (x 10-4)
DomainNet
Taskonomy
quickdraw
real
segm 30
segm_26
Batch size
2.40
n/a
infograph
peinting
cee
keypoints,
depth
Learning rate
Model width
0.40
2.25
47.6
1.73
clipart
normal
nome
2d
1on 22
0000 22
Encoder depth
1.09
2.63
Decoder depth
n/a
5.06
(a) We measure the variance across learning rates while keeping other parameters fixed (model capacity, batch size, etc.), then report the median variance across all set- tings. We repeat this process for each axis of variation.
(b) Proportion of pairwise task conflicts (for a fixed model size and learning rate) across training epochs (median), for DomainNet (6 domains) and Taskonomy (subset of 6 tasks)
Figure 5: (left) Variance of observed gradient conflicts when sweeping over different training hyperparameters (high impact) as well as model capacity (low impact), and (right) illustrating the median proportion of pairwise gradient conflicts measure of task affinity
Secondly, we analyze the effect of different factors of training variations on the observed proportion of gradient conflicts in Figure 5(a). On the one hand, hyperparameters directly related to weights updates, such as the learning rate of batch size, have a clear impact. In particular, since memory consumption is often a bottleneck of gradient- based MTO methods, this means that naively decreasing batch size to reduce memory usage may severely impact such methods in practice. On the other hand, model capacity has lesser effect on the pattern of gradient confficts, despite influencing task interference as highlighted in Figure 2. Nevertheless, we also observe that the overall magnitude of conflicts encountered for different pairs of tasks reveal interesting task affinity patterns, illustrated in Figure 5(b): For instance, the qui ckdraw domain of DomainNet appears as a clear outlier (across all model capacities), which is also the case in terms of MDL performance in Figure 2.
In summary, we observe intriguing properties of gradient conflicts practice, in partieular suggesting that the extra cost of measuring, storing and correcting conficctig gradients at every training iteration can be superfluous. As an alternative, [30] shows that less costly regularization methods (c.g., early stopping) can be competitive with MTO. In this work, we consider an orthogonal direction, by tuning the acalarization//esaapling weights hyperparameters to regulate the training speed of the different tasks/domains. However, the computational cost of browsing this large search space becomes practical caveat as T grows. the next section, we leverage a scalable hyperparameter tuning algorithm to tackle this problem and apply it on the DomainNet and CelebA datasets.
5 Population-based training for scalarization weights selection
Tuning the weights with classical parameter search methods, such as grid search random search, becomes extremely costly when increases. To address this high computational demand, we propose to leverage the cquulation-based training (PBT) [24] framework which has been used for efficient hyperparameter search in reinforcement learning [24] and for data augmentation pipelines [23].
Population- training (PBT). PBT is an evolutionary algorithm for hyperparameter search: N models are trained in parallel with different starting hyperparameters, Every Fready epochs, the models synchronize: The Q% worst models in the population are stopped, and their model weights and hyperparameters are replaced by the ones of the Q% best models (exploit step); Then, the newly copied hyperparameters are randomly perturbed reach new part of the hyperparameter search space (explore step). Finally, training resumes until E' epochs are reached. In other words, PBT enables dynamic exploration of the hyperparameter search space with fixed computational cost (cost of training N models + potential overhead from synchronization). A follow-up work, Population- based Bandits (PB2) [46] proposes leverage Bayesian optimisation [17] to better guide the explore step. In contrast to PBT, PB2 also offers theoretical guarantees on the convergence rate.
Using PBT to tune scalarization weights. PBT tuning relies on three important characteristics that may conflict with the standard scalarization MDL/MTL training pipeline:
Models are trained with dynamic schedule of hyperparameters. While this contrasts with standard scalarization in which p fixed, we do not expect this to an issue as recent work has shown that scalarization with dynamic random weights performs well in practice [35].
8

---

Secondly, models are compared against one another after a few epochs of training (Eready epochs) in contrast to e.g. random search where models are usually trained until convergence, or reaching a certain stopping criterion. Consequently, tuning Fready can significantly impact the search's ss stability and outcome, which we further discuss in Appendix 5.
Finally, models in the population are compared using a single objective (e.g. average task/domain metric) during training. This may be an issue if the task metrics' have widely different ranges and the PBT scheduler may simply learn favor short term improvement by giving higher weight to tasks with high metrics; While we do not observe this issue in our settings, recent work [15] also proposes a multi-objective variant of PBT which may be better fitted for MDL/MTL applications.
Nevertheless, the key advantage of PBT is its computational efficiency with regard to search space exploration: For constant cost of training N models, and some minor overhead related to check- pointing, PBT explores up to N (1 + Q x Étotal/ Eready) possible hyperparameter configurations throughout training of the population.
Table 1: Results of MDL when jointly training all 6 domains of DomainNet for scalarization (uniform and PBT-found weights) and MTO methods. PBT is with population size of N 12 models, such that every Eready = 5 epochs, Q = 25% of the population triggers an exploit/explore step.
DomainNet (ResNet26 with 0.25 width)
average
clipart
infograph
painting
quickdraw
real
sketch
Scalarization
Uniform
46.78 # 0.10 56.31 # 0.04
20.46 = 0.15
40.95 + 0.45
60.69 + 0.07
55.64 * 0.01
46.64 # 0.32
PBT
48.01 + 0.08
58.31 + 0.15
19.45 + 0.08
41.32 + 0.11
63.58 + 0.31
60.43 + 0.05
45.00 + 0.26
MTO Loss-bused
Uncertuinty [26]
45.12 = 0.07
59.24 0.09
17.14 = 0,25
37.75
= 0,16
59.85 = 0,16
$2.35 = 0.20
44,42 # 0.10
IMTL-L [38]
44.22 = 0.10
58.05 = 0.22
16.41 = 0.16
37.53 + 0.21
59.22 = 0.29
51.27 = 0.19
42.83 # 0.38
MTO Gradient-based
CAGrad [37]
42.82 # 0.06
54.08 0.03
18.26 + 0.04
36.79 + 0.14
56.46 + 0.28
49.52 + 0.15
41.78 + 0.01
GradDrop [8]
42.52 = 0,05
53.34 = 0.03
18.16 0.07
37.25 # 0.02
55.09 = 0.13
49.94 # 0.26
41.35 + 0.09
PCGrad [64]
42.78 = 0.14 53.55 = 0.04
18.29 = 0.34
37.31 # 0.38
55.60 = 0.08
50.41 = 0.13
41.52 # 0.67
DomainNet (ResNet26 with original width) clipart
average
infograph
painting
quickdraw
sketch
Scalarization
Uniform
48.83 + 0.08 58.69 + 0.05
21.58 + 0.39
42,83 + 0.03
62.51 + 0.21
58.31
0.06
49.05 0.15
PBT
49.27 + 0.12
58.41 + 0.50
19.30 0.19
44.53 + 0.16
63.08 0.42
60.08 0.16
50.23 + 0.01
MTO Loss-based
Uncertainty [26]
46.96 + 0.10 60.71 # 0.36
18.74 + 0.03
40.22 + 0.40
60.92 + 0.04
54.37 + 0.12
46.79 * 0.23
IMTL-L [38]
46.04 + 0.21
59.76 + 0.78
18:21 + 0.15
39.12 + 0.76
60.24 + 0.39
53.06 + 0.38
45.87 # 0.29
MTO Gradient-bused
CAGrad [37]
44.91 = 0.18
56.56 0.38
19.63 + 0.32
38.84 0.447
58.06 0.41
51.80 0.58
44.58 0.45
GradDrop [8]
45.15 : 0.08
56.22 = 0.43
19.89 = 0.16
39.70 + 0.03
57.55 + 0.12
52.81 0.04
44.76 0,18
PCGrad [64] 44.96 + 0.14
55.79 + 0.24
19.82 + 0.20
39.65 + 0.29
$7.30 + 0.30
52.57 + 0.11
44.65 # 0.65
In Table and Table 2, we report results for searching optimal scalarization weights p when training for all domains of DomainNet and for 8 tasks (attribute subsets) of CelebA. For PBT results, we first run the search algorithm using the implementation from Raytune [33]. We use 70% of the training set for training, and use the remaining 30% rank models in the population by measuring their average accuracy on this set. Once the search is done, we retrain a model on the full training set using the scalarization weights found by PBT. For comparison, we also report results for uniform scalarization and MTO methods, using the implementation from [31]. All final models are trained for three different learning rates and the best metric is reported, averaged across 2 random seeds.
On the DomainNet example, we observe that the scalarization weights found by PBT outperforms all methods, confirming our insights that tuning weights p can further enhance scalarization. Note that MTO methods were not designed or employed for multi-domain settings such as DomainNet, which may explain why gradient-baat based MTO methods all underperform their loss-based counterparts in the results of Table 1, while they do exhibit good performance on CelebA MTL. In fact, on CelebA, while PB2 search reaches the highest overall average metric, we find that results across tasks exhibit more variance, with PB2 and CAGrad yielding the best, comparable, performance. Overall, the very narrow differences accuracy makes it difficult to highlight one specific method, which also raises the issue on whether CelebA is robust enough MTL benchmark. Nevertheless, both experiments

---

show that scalarization can outperform more complex optimization methods when its weights are tuned properly. which can be done efficiently using scalable hyperparameter search methods like PBT or PB2. We discuss further insights and compute details in Appendix 5.
Table 2: Results of MTL when training on all 8 tasks (subset of attributes) of CelebA defined in Appendix 1. For PBT and PB2 we use slightly different parameters than DomainNet account for the fact that CelebA contains more tasks, and hence has a larger search space: All PBT runs use population size of N 12 models, such that every Eready = epochs, Q 40% of the population triggers an exploit/explore step. For PB2 runs we use population size of N = 8 and otherwise the same Q and Eready hyperparameters. For the sake of space, we omit standard deviations in CelebA in the main text (in the range le- 1), and only report results for the four best performing MTO baselines.
ViT-S/4, 6 layers, full width
face structure
facial hair
hair color
hair style
average
age
clothes
gender
mouth
Scalarization
Uniform
91.23
86.70
92.79
85.16
95.45
98.10
92.99
91.68
86.95
PBT
91.21
86.79
92.77
85.18
95.42
98.02
92.92
91.66
86.95
PB2
91.28
86.96
92.87
85.17
95.47
98.10
92.90
91.81
86.94
MTO
IMTL-L [38]
91.19
86.60
92.80
85.18
95.46
97.97
92.93
91.66
86.94
CAGrad [37]
91.27
86.92
92.89
85.11
95.49
98.17
92.93
91.74
86.89
GradDrop [8]
91.27
87.05
92.73
85.27
95.48
98.11
92.93
91.64
86.94
PCGrad [64]
91.18
86.71
92.74
85.09
95.37
98.12
92.90
91.66
86.84
VIT-S/4, 9 layers, full width
face structure
facial hair
hair color
hair style
average
age
clothes
gender
mouth
Scalarization
Uniform
91.17
87.33
92.50
85.10
95.45
97.93
92.85
91.39
86.80
PBT
91.15
87.43
92.51
85.18
95.46
97.78
92.51
91.36
87.00
PB2
91.25
86.83
92.85
85.12
95.49
98.19
92.90
91.78
86.81
MTO
IMTL-L [38]
91.16
87.40
92.44
85.09
95.44
97.92
92.87
91.39
86.75
CAGrad [37]
91.22
87.37
92.66
85.13
95.40
97.92
92.92
91.61
86.74
GradDrop [8]
91.07
87.41
92.36
85.01
95.41
97.73
92.75
91.21
86.70
PCGrad [64]
91.14
87.53
92.42
85.00
95.38
97.87
92.84
91.37
86.68
6
Limitations
Since we are building off linear scalarization, our study suffers from the same issue: It has been shown, for instance in [5], that scalarization only finds solutions the convex parts of the Pareto front. Same as previous studies 52]. we do not observe this to be an issue in practice, but there may be some methods. In addition, our results and observations relies on an experimental study. While we do attempt experiment over diverse set of benchmarks and model capacities to get many data capace
where scalarization can simply not perform well
advanced MTO
points as possible, we can not guarantee the generality of our claims across all MDL/MTL settings.
Conclusions
This work presents comprehensive evaluation scalarization's 's effectiveness in multi-domain and multi-task learning, spanning diverse model capacities and dataset sizes. Our analysis reveals that larger-capacity models often benefit more from joint learning across diverse settings. In addition, tuning scalarization weights is key to reach optimal performance at inference and improve the MTL/MDL model generalization. Nevertheless, given specific of specific tasks/domains, the optimal weights are rather robust to changes in model capacities within the same architecture family. We then investigate the impact of model capacity on gradient conflicts observed during training and observe low correlation with MTL/MDL performance. Finally, to tackle the large search space of tuning scalarization weights, we propose to leverage population-based training as a scalable, efficient method for tuning scalarization weights the number of tasks/domains increases.
10

---

References
[1] Jing An, Lexing Ying. and Yuhua Zhu. Why resampling outperforms reweighting for correcting sampling bias. In International Conference on Learning Representations (ICLR), 2021. 3
[2] Shai Ben-David and Reba Schuller. Exploiting task relatedness for multiple task learning. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COI.1/Kernll 2003, Washington, DC, USA, August 24-27, 2003. Proceedings,
pages 567-580. Springer, 2003. 3
[3] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando C Percira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine Learning. 79: 151-175, 2010.
[4] John Blitzer, Koby Crammer, Alex Kulesza, Fernando C Pereira, and Jennifer Wortman Vaughan. Learning bounds for domain adaptation. In Conference on Neural Information Processing Systems (NeurIPS), 2007.
[5] Stephen P. Boyd and Lieven Vandenberghe. Convex optimization. IEEE Transactions Automatic Control, 11:1859-1859, 2004. 10
[6] Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1385-1394, 2019. 2
[7] Zhao Chen, Vijay Badrinarayanan, Chen-Ye Yu Lee, and Andrew Rabinovich. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. ArXiv, abs/1711.02257, 2017. 2,
[8] Zhao Chen, Jiquan Ngiam, Yanping Huang. Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick sign: Optimizing deep multitask models with gradient sign dropout. In Conference Neural Information Processing Systems (NeurIPS), 2020. 2, 4
[9] Adam Coates, A. Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011. 4, 15
[10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Conference on Computer Vision and Puttern Recognition (CVPR), pages 3213-3223 2016. 5
[11] Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Au- toAugment:
Learning augmentation strategies from data. Conference on Computer Vision and Pattern Recognition (CVPR), pages 113-123, 2019. 17
[12] Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. RandAugment: Practical automated data augmentation with reduced search space. 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops (CVPRW), pages 3008-3017, 2019, 17
[13] Jean-Antoine Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective opti- mization. Comptes Rendus Mathematique, 350:313 -18, 2012. 2, 3,4,
[14] Yunshu Du, Wojciech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.0222: 2018. 2
[15] Arkadiy Dushatskiy, Alexander Chebykin, Tanja Alderlicsten, and Peter A.N. Bosman. Multi- objective population based training. In International Conference on Machine Learning, pages 8969-8989 8989. PMLR, 2023. 9
[16] Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Effi- ciently identifying task groupings for multi-task learning. In Conference on Neural Information Processing Systems (NeurIPS), 2021. 1,3, 16

---

[17] P. Frazier. A tutorial on Bayesian optimization. ArXiv, abs/1807.0281 2018. 8, 19
[18] Hanan Gani, Muzammal Naseer, and Mohammad Yaqub. How train vision transformer on small-scale datasets? In British Machine Vision Conference (BMVC), 2022. 4, 15
[19] Yaroslav Ganin, E. Ustinova, Hana Ajakan, Pascal Germain, H. Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. ArXiv, abs/1505.07815 2015. 3
[20] Yuan Gao, Jiayi Ma, Mingho Zhao, Wei Liu, and Alan L Yuille. NDDR-CNN: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3205-3214, -214,, 2019. 2
[21] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In European Conference on Computer Vision (ECCV), pages 270-287, 2018. 2018. 2
[22] Lei Han and Yu Zhang. Learning multi-level task groups in multi-task learning. In Conference on Artificial Intelligence (AAAI), 2015. 3
[23] Daniel Ho, Eric Liang, Ion Stoica, P. Abbeel, and Xi Chen. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learing (ICML), 2019. 8, 17, 20
[24] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv: 2017. 2, 8, 19
[25] Adrián Javaloy and Isabel Valera. RotoGrad: Gradient homogenization in multitask learning.
In International Conference on Learning Representations (ICLR), 2021. 1, 2,
[26] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 7482-7491, 2017. 2, 4
[27] lasonas Kokkinos. UberNet: Training universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 3,
[28] B. Krawezyk. Learning from imbalanced data: open challenges and future directions. Progress in Artificial Intelligence, 5:221 2232, 2016. 3
[29] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 4, 15
[30] Vitaly Kurin, Alessandro De Palma, llya Kostrikov, Shimon Whiteson, and M. Pawan Kumar. In defense of the unitary scalarization for deep multi-task learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022. 1, 2, 5, 18
[31] Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representations: A unified look at multiple task and domain learning. arXiv preprint arXiv:2204.02744, 2022. 21
[32] Yunsheng Li. Lu Yuan, Yinpeng Chen, Pei Wang, and Nuno Vasconcelos. Dynamic transfer for multi-source domain adaptation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 10998-1 11077, 2021. 5, 16
[33] Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training. ArXiv, abs/ 1877051 18, 2018. 9, 20
[34] Baijiong Lin, Feiyang Ye, and Yu Zhang. A closer look at loss weighting in multi-task learning. arXiv preprint ar arXiv:2111. 2021. 2
12

---

[35] Baijiong Lin, Feiyang Ye, Yu Zhang, and Ivor Wai-Hung Tsang. Reasonable effectiveness of random weighting: A Litmus test for multi-task learning. In IEEE Transactions on Machine Learning Research (T-MIRR), 2022. 8
[36] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Tak Wu Kwong. Pareto multi-task learning. In Conference on Neural Information Processing Systems (NeurIPS), 2019. 4
[37] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Conference on Neural Information Processing Systems (NeurIPS), 34: 18878-18890, 2021. 1,2,4
[38] Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue. Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations (ICLR), 2021. 2, 4, 18
[39] Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. Conference on Computer Vision and Pattern Recognition (CVPR), pages 1871-1880, 2018. 2
[40] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision (ICCV), 2015. 4,5
[41] Raphael Gontijo Lopes, Sylvia J. Smullin, Ekin Dogus Cubuk, and Ethan Dyer. Tradeoffs in data augmentation: An empirical study. In International Conference on Learning Representations (ICLR), 2021. 17
[42] Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in Pareto optimization. International Conference on Machine Learing (ICML), 2020. 3, 4
[43] Lincy Mathews and Seetha Hari Learning from imbalanced data. Advances in Computer and Electrical Engineering, 2019. 3
[44] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes The benefit of multitask representation learning. Journal of Machine Learning Research (JMLR), (bs//505500679) 2015.
[45] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3994-4003, 2016. 2
[46] Jack Holker-Holder, Vu Nguyen, and Stephen J. Roberts. Provably efficient online hyperparam- eter optimization with population-bant based bandits. arXiv: Learning, 2020. 8, 20
[47] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In International Conference on Computer Vision
(ICCV), pages 1406-1415, 2018. 4, 5, 15, 16
[48] Anastasia Pentina and Christoph H. Lampert. Multi-task learning with labeled and unlabeled tasks. In International Conference on Machine Learing (ICML), 2016.
[49] Sylvestre-Alviss Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Conference on Neural Information Processing Systems (NeurIPS), 30, 2017. 3
[50] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi- domain deep neural networks. In Conference Computer Vision and Pattern Recognition (CVPR), pages 19-81127 2018. 3
[51] Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMII), 42(3):651 663, 2018. 3
[52] Michael Ruchte and Josif Grabocka. Multi-task problems are not multi-objective. ArXiv, abs/2110.07301, 2021. 4, 10
13

---

[53] Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent multi-task architecture learning. Conference on Artificial Intelligence (AAAI), 2019,
[54] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 155:211-2525 2015. 15
[55] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objectiive optimization. In Conference Neural Information Processing Systems (NeurlPS), 2018. 2,
[56] Changjian Shui, Mahdieh Abbasi, Louis-Emile Robitaille, Boyu Wang, and Christian Gagné.
A principled approach for learning task similarity in multitask learning. In International Joint Conferences on Artificial Intelligence (IJCAI), volume abs/1903.09100 2019. 1
[57] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. European Conference on Computer Vision (ECCV), 7576: 746-760, 2012. 5
[58] Trevor Scott Standley, Amir Roshan Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? ArXiv, abs/ abs/1905.07553, 2019. 3, 4, 5, 16, 17
[59] Gjorgji Strezoski. Nanne van Noord, and Marcel Worring. Many task learning with task routing. In International Conference on Computer Vision (ICCV), pages 1375-1384, 2019.
[60] Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal gradients. arXiv preprint arXiv:1912.000
2019. 2
[61] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3614-3633 2021. 2
[62] Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In International Conference on Leurning Representations (ICLR), 2021. 2, 18
[63] Derrick Xin, B. Ghorbani, Ankush Garg, Orhan Firat, and Justin Gilmer. Do current multi- task optimization methods in deep learning even help? In Conference Neural Information Processing Systems (NeurIPS), 2022. 1, 2, 4, 10, 18
[64] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Conference on Neural Information Processing Systems (NeurIPS), 2020. 1, 2, 4, 7, 18
[65] Amir Roshan Zamir, Alexander Sax, Bokui (William) Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 1, 5, 16
[66] Youshan Zhang. A survey of unsupervised domain adaptation for visual recognition. ArXiv, abs/2112.06745, 2021. 3
14

---

Supplemental Material to "Scalarization for Multi-Task and Multi-Domain Learning at Scale"
1 Experimental settings and training hyperparameters
Multi-Domain
We refer benchmarks as "multi-domain" when they contain multiple input visual domains with shared set of output classes (i.e., Vi * j. X; X, and Y; -)))
CIFAR- 10 and STL-10. CIFAR-IO [29] is a classical benchmark for image classification containing 50k training samples uniformly distributed across 10 classes. STL-10 [9] is semi-supervised dataset which was designed to resemble CIFAR-10. 10. Specifically, we only use the 5000 annotated images in STL-10, which also uniformly distributed across the same 10 classes CIFAR. In STL-10, the images themselves are from the ImageNet [54] dataset, and cropped/resized to 96 pixels. We further resize them to 32 pixels to align with CIFAR. In summary, the key difficulties (i) the input distribution shift between the two datasets and (ii) the high imbalance in data availability.
We use vision transformer backhone (ViT-S) optimized for small-scale datasets [18] (compared to the original VIT-S, this backbone contains smaller patch sizes, fewer transformer layers and narrower embeddings, but higher number of heads). To control model capacity, we vary the depth (number of transformer layers) {3, 6, 9} and the width (token dimension) in {48, 96, 144, 192]. Finally, we train each model from scratch 300 epochs (including 30 epochs of linear learning rate warmup), using learning rate 0.001 and weight decay of 0.05 with the AdamW optimizer and cosine learning rate decay.
single Nvidia 100 GPU with batch size of 256 images for
DomainNet. DomainNet [47] is a classification dataset of 6 visual domains annotated for 345 classes, for total of roughly 410k training samples. DomainNet was initially introduced for the problem of multi-source domain adaption, which one or more of the domains does not have training annotations; the key difficulty is thus learn representations that are aligned across domains. In contrast to the CIFAR+STL example, DomainNet exhibits distribution shifts across both the input domains and output classes, visualized in Figure
painting [X| = 50416
infograph [X] = 36023
quickdraw [X] 120750
clipart |X| 33525
sketch XX| 48212
real XX|
120906
Illustrating the data imbalance in DomainNet with contour plot of the number of samples per class and domains in DomainNet. Each of the corners of the hexagon represents one of the six domains in DomainNet, and the lines (levels of the contour plot) represent the number of samples, drawn every 15 classes, For comparison, a uniformly distributed dataset would yield perfect hexagons.
Figure A:
15

---

Following previous literature [47, 32], we use a ResNet- 101 as our main backbone. There is no domain-dependent layer in the architecture: the final classifier layer is shared across all domains. We first perform a training sweep over the largest domain (real) to select the best-performing learning rate from (0.3, 0.03, 0.003, 0.0003) and the number of epochs from {30, 60, 90). Following these results, we use a learning rate of 0.03 and train for 30 epochs with a batch size of 512 in subsequent
experiments. We train with the Adam W optimizer with a weight decay of 1e - 4. We also apply linear learning rate warm-up during the first five training epochs and use cosine schedule learning rate decay for the rest of the training. Finally, to control model capacity, we vary the depth (backbone) in [ResNet-26, ResNet-50, ResNet- 101) and the width factor.
Multi-domain, resampling and training length In the multi-domain setting, scalarization weights become resampling probabilities for each dataset, shown in (2). Consequently, the notion of "epoch" is hard to define compare to the standard mono-dataset setting. To resolve this, we always define epochs with respect to one of the domains. For instance, in the CIFAR+STL case, use STL as our reference. Therefore, "one epoch" translates to seeing as many samples as in the original STL dataset (5000) using the current batch size (256), i.e. roughly 20 training steps. In the DomainNet case, we define epochs relatively to the real domain. This definition has the advantage of not being impacted by the sampling weights p: In particular, this means that both the MDL models and the single dataset (SD) baselines are trained for as many training steps, and see the same amount of training samples, only sampled from different data distributions.
1.2 Multi-task
We define multi-task benchmarks as datasets where every image is fully annotated for multiple output tasks (i.e., Vi + X;; Xj and Y, * Yj). This setting is particularly popular for scene understanding problems where every scene is labelled with multiple dense predictions (c.g. depth, normals, segmentation mas, edges, etc.)
Celeba. CelebA is a binary attribute classification dataset containing 40 attributes and roughly 162k training images. To turn CelebA into a multi-task problem, it is common to consider each attribute as binary classification task: More specifically, we use fully shared backbone with final lincar layer of 40 outputs, outputting logits for every task. The model is then trained using 40 binary cross censspy losses, one for each attribute. To make our comparative analysis more scalable, we define several tasks as subsets of attributes, grouped based on semantic similarity (c.g. all hair colors are in the same subgroup). The 8 resulting subsets of attributes are described in Table A. In the scalarization setting, this simply means that some of the attributes share the same importance weight.
As backbone, we use the same ViT-S/4 based architecture as for CIFAR/STL. We train for 50 epochs with 5 epochs of learning rate warmup. We use a learning rate of 0.0005 with cosine schedule decay anf train with the AdamW optimizer with a weight decay of 0.05. We use input images of size 32 (with tokens of size 4), batch size of 256, and RandAugment data augmentations.
Table A: The eight tasks defined as subsets from CelebA attributes used in our main analysis.
Attributes in the same subset share a common importance weight p
Hair color
Hairstyle
Facial Hair
Mouth
Clothes
Face Stuuture
Gender
Age
Black Hair
Bald
S'o'Clock Shadow
Big Lips
Eyeglasses
Big Nose
Male
Young
Blond Hair
Bangs
Mustache
Mouth Slightly Open
Heavy Makeup
Chuhby
Brown Hair
Receding Hairline
No Beard
Smiling
Wearing Earrings
Double Chin
Gray Hair
Sideburss
Gontee
Wearing Lipstick
Wearing Hat
High Cheekbones
Straight Hair
Wearing Nocklace
Oval Face
Wavy Hair
Wearing Necktic
Pointy Noise
Taskonomy. Taskonomy [65] is a large dataset containing a variety of dense prediction tasks for indoor scenes. We use the tiny split of Taskonomy which contains roughly 275k images. Taskonomy was originally introduced for the problem of task clustering: The original work [65] proposes a task affinity metric to define a taxonomy of tasks. This taxonomy structure is then used to determine which tasks should be trained from scratch and which tasks could benefit from others via transfer learning. Closer to our setting, follow-up works [58, 16) propose to use this taxonomy to determine which tasks should be grouped not in multi-task learning. Once the groupings are determined,
16

---

separate backbone is trained for each group of tasks. Instead, for our analysis, we use Taskonomy-tin tiny in more standard multi-task framework, where a backbone shared across tasks.
For training, we follow the methodology of [58]. We use a ResNet-26 backbone (with varying bottleneck width) with mirrored decoder; By default, only the encoder is shared across tasks and each task receives its own decoder. To vary model capacity, we add the option to share more or fewer layers of the decoders across tasks. We use the same learning rate of 0.1 and training for 100 epochs using a batch size of 256, We train with SGD with a momentum of 0.9 and a weight decay of le - 4. Following [58], all output prediction maps are rescaled to have zero mean and unit variance on the training set, and all dense tasks are trained with L1 loss.
2 Additional analysis results
2.1 Complete results and methodology for Figure 2
In Section 4, we perform MDL and MTL experiments on several pairs of datasets, each time comparing to the single dataset (SD) baseline trained for the same model capacity and training length. All results are run for three random seeds on CelebA and CIFAR+STL, and two random seeds for DomainNet and Taskonomy. To present these results in condensed form in Figure 2, we first find the scalarization weights p' = (pi, 1 - pî) that yield the best average accuracy across both datasets. Then report the difference in metries between MDL trained with weights p* and the corresponding SD baseline, for each dataset. Note that for the Taskonomy case, where the tasks are evaluated via LI loss, we measure the negative difference instead keep the same interpretation as the other settings where a positive value means MDL improves over SD.
For completeness, we report all results for the CIFAR/STL case trade-off plots (accuracy dataset 1 versus accuracy on dataset 2) in Figure B (CIFAR/STL) and in in Figure C (segmentation 2D and depth tasks of Taskonomy). We observe the same trends summarized in the main paper: First, when increasing model capacity, the MDL performance over the SD baseline increases; This is best seen when width increases (across columns). Second, the optimal weights vary across model sizes: At low width, the best performance is reached for a ratio in the range of [0.3, 0.4). While larger models prefer PšTL, E 10.1, 0.3). Finally, it is interesting to note that these weights also differ from heuristics commonly used to set scalarization weights: such as uniform scalarization PSTL = 0.5 or for instance setting the weights to match the number of samples in each dataset PSTL- = 0.09. This further highlights the fact that tuning scalarization weights can make scalarization into a stronger baseline for MDL/MTL.
2.2
MDL helps generalization
When comparing the MDL/MTL and SD performance, we often observe that MDL/MTL improve- ments over SD are visible at inference but not at training time (as illustrated for instance in Figure D). This indicates that MDL/MTL training helps generalization of the model compared to training on a single dataset. In particular, in the MDL setting, this draws an interesting parallel with data augmen- tations: In fact, MDL training can be seen as adding additional input data from a new distribution, with probability given by the scalarization weights p. while sharing the same semantic classes. And similarly to data augmentations, adding this extra data source makes the training distribution harder to fit (hence the SD baseline outperforming MDL at train time) but can greatly benefit generalization performance (hence the inverse trend at inference).
To push the analogy further, the experimental study of [41] suggests that good data augmentation should be one with a high affinity to the original data distribution (i.c., the distribution shift between the original data and augmented one should not be too significant) as well as high diversity (i.c., the added data should be complex enough, which can be measure e.g. with magnitude of the training loss). This hypothesis also matches our observations: For instance adding infograph data to the real domain leads to low affinity pairing but with high diversity and yields weaker MDL performance on the real images compared to the SD baseline (see Figure 2).
Finally, we note that the problem of finding optimal scalarization weights mirrors the one of finding data augmentation hyperparameters, which has been extensively explored for the task of image classi- fication. It has led to now commonly-adopted augmentation strategies such as RandAugment [12], AutoAugment [11], or PBA [23], which also uses Population- based training to tackle this problem.
17

---

Ps71 cA
depth width 0.25 depth 3: width depth width 0.75 depth
3; width
domain cifar10
top-1 1pp11 ace
depth 6: width = 0.25 depth = 6; width 0.5 depth 6; width = 0.75 depth = 6; width 1.0
domain cifarlo
top- 1 acc
1-4
depth 9; width 0.25 depth = 9; width
0.5 depth
9; width = 0.75 depth = 9; width 1.0
domain cifar10
top-1 1 acc
8
top-1 acc domain stl10
top-1 domain st10
top-1 acc domain stl00
top-1 acc domain sti10
Figure B: Complete analysis results for the CIFAR+STL scenario. Each row corresponds to different model depth and each column a model width, in increasing order. each plot. we plot the model's test accuracy on CIFAR-10 10 versus the test accuracy on STI.-10. The single dataset baseline (SD) drawn in black and corresponds to the accuracy obtained when training two separate networks, one on each dataset independently. The circle markers correspond to the MDL model trained for different scalarization weights p. The value of PSTL is represented as the color of the marker, while the remaining weight is always set to PCTFAR
PSTL.
Methodology for measuring gradients conflicts
In this appendix, briefly describe methodology for Section 4.2. We use the same definition of gradient conflicts as PCGrad [64]: Two task/domain specific gradients are conflicting if and only the cosine of the angle between them is strictly negative. We train model using standard uniform scalarization, measure the number conflicting pairs of task/domain gradients over one epoch of training. and report it as percentage (of all pairs), for each epoch during training.
We report these results in the main text in Figure and in Figure E below. Our main observation is that the presence absence of gradient conflicts does not correlate well with actual MDL/MTL performance throughout trainined. This challenges the assumption underlying many multi-task optimization (MTO) methods [64, 38, 62] that reducing gradient conflicts leads to improved MTL performance. This also aligns well with recent results of [63, 30] showing that MTO methods that reduce gradient conflicts do not outperform simpler scalarization approaches in practice, and with the hypothesis [30] that many MTO methods can reinterpreted as regularization techniques.
4 Consistency of optimal scalarization weights
As noted in the analysis from Section 4, the optimal weights p rather consistent across model depths and widths. For instance, the CIFAR/STL case, p" always falls in the range of [0.2, 0.4].
18

---

Psegmad
%/
o/a
width 0.5;shr
width 0.5;shr =
width = 0.5;shr
6285
domain depth
0.28
top-1 acc
0.223
228
width 1.0;shr 0
width 1.0;shr =
width 1.0;shr 4
6.255
domain depth
6.30
top-1 acc
0.23
6383
0.55
1.44
5:45
0.46
5.47
0.43
0.44
0.45
0.08
0.67
5.43
2.44
0.45 0.45
2.47
top-1 acc
top-1 acc
top-1 acc
domain segm_2d 2d
domain segm_ __
domain segm_
Figure C: Complete analysis results for the segmentation 2D and depth prediction tasks in the Taskonomy scenario. Each row corresponds to different number of layers shared in the decoder (shr) and each column model width. In each plot, plot the model's test Le loss each task on either axis. The single dataset baseline (SD) is drawn in black and corresponds to the accuracy obtained when training two separate networks, one on each dataset independently. The cirele markers correspond to the MDL model trained for different scalarization weights p. The value of Psegmentation22 represented as the color of the marker.
This can also be from the qualitative results of the population- based training search of scalariza- tion weights in Section 5.2: While the history of hyperparameter changes during the search differ, PBT tends to converge similar distribution for the scalarization weights p across different model depths and widths. This suggests that the thcoretical search space for may be reduced in practice leading a more computationally efficient search: Performing rough initial search on smaller model from the same architecture family can provide promising range for p, and can then be refined by searching with the larger target architecture.
5 PBT results
Because the search space for scalarization weights grows exponentially with the number of tasks, classical hyperparameter search methods such grid search or random search would struggle to scale as the number of tasks increases. Bayesian optimization (BO) [17] allows for faster results by browsing the search space in smart way by building and following probabilistic model of the hyperparameters. However, BO still requires training models to convergence (or until an early stopping criterion is met) which can be computationally expensive. Instead, we experiment with the Population-be hppuee Training framework [24] for searching for the optimal scalarization weights. PBT relies on the assumption that the "goodness" of certain hyperparameter choice can be evaluated in few epochs during training, rather than having to finish full run of training.
5.1 Compute resources
As mentioned in the main paper perform all training runs on single NVIDIA V100 machine with 32GB of memory. However, both the PBT and MTO models require higher compute resources than single normal run of training, which we discuss below.
19

---

depth 9; width = 0.25 depth = 9; width
0.5 depth
9; width = 0.75 depth = 9; width 1.0
domain cifar10
top- acc
8
top-1
top-1
top-1
top-1
domain = st10
domain st10
domain st10
domain st10
depth = 9; width 0.25 depth 9; width 0.5 depth = 9; width = 0.75 depth 9; width 1.0
domain cifar10
top- acc
top-1
top-1
top-1
top-1
domain stil0
domain stil0
domain stl10
domain stl10
Figure D: Train-test Discrepancy when comparing MDL/MTL improvement over the SD baseline visualized the CIFAR+STL example. In particular, in the MDL setting, this matches the classical interpretation data augmentations: Adding additional semantically relevant data from an input distribution may be harder to fit at training time but leads improved generalization performance inference.
bs:
bs=256
bs=512
of gradient conflict
5.52
0.502
6.45
0.422
0:42
0.402
0.400
0.53
0.375
0.52
0.20
linear fit (r=-0.48)
Enear fit (r=0.47)
22
16
Training epoch
Training loss
Validation metric
(a) DomainNet. Individual curves colored by batch size.
Figure E: Additional result measuring conflicting gradients when training with standard uniform scalarization for DomainNet (6 domains).
PBT requires training a population of N models that are regularly synchronized; then fol- lowed by a final training with the found optimal weights. Following previous works [23], we perform the hyperparameter search on smaller subset of the data (in our experiments 0.7 fraction of the training set). In summary, the expected computational cost is roughly Nr times higher than standard training. On the CelebA example, we also observe that using PB2 [46], which combines the benefits of Bayesian Optimisation and PBT, yields better hyperparameters using a smaller population size. terms of memory usage, PBT is the same standard training run: Synchronization is handled via checkpoints saving and loading, such that only one model lives in memory. Finally, we use the publicly available PBT implementation from Raytune[33] which handles all synchronization operations across the population. The implementation would also scale well to more compute resources, as the Ray API allows for easy parallelization.
MTO. As shown in Figure 1, the bottleneck in most gradient-based based methods is memory usage. Consequently, this requires us to decrease the batch size to meet memory requirements and compensate with gradient accumulation (or parallelism if multiple devices are available).
20

---

For instance, on the DomainNet experiments with all 6 domains, we need to decrease from batch size of 512 to a batch size of 128 with 4 steps of accumulation still in memory requirements. This also raises the question of how to handle synchronization across batches: For instance, in PCGrad, the gradient conflicts (and projections) can be computed either (i) per local batch, before accumulation: this may lead to noisier updates; or (ii) after gradients are accumulated: However this is more memory-intensive as this requires to store the previously accumulated per-task gradient well as the one being currently computed. In practice we use the implementation of [31] for all MTO methods.
When comparing different hyperparameter searches for scalarization, PBT allows for much faster exploration of the search space than classical techniques such as grid search. However, comparing PBaascalaiination with MTO is less straightforward as the computational cost depends on many factors (e.g. population size, number of tasks, and impact memory usage, etc.), but generally, the "scalarization + hyperparameter search" " approach is more favorable case of low memory requirements as it does not change memory costs compared to standard training. However, soTA gradient-basee methods are not very costly for a low number of tasks (c.g. 2-3) as shown in Table of our key takeaways is
which makes them appealing in settings with few tasks. Nevertheless, that allocating extra resources for tuning scalarization weights, to mirror the extra resources needed for MTO training, makes scalarization into much stronger baseline, on-par even outperforming MTO methods as shown in Section 5. Finally, another important difference is that hyperparameter search methods directly optimize for the target objective: the optimal hyperparameters
found by maximizing the average task/domain accuracy on a hold-out validation dataset. In contrast, MTÔ methods optimize for a proxy metric (such as reducing gradients conflicts) that may not always correlate with final performance as shown in Section 4.2.
5.2 Qualitative results
In this section, report and PB2. At the end of PBT search, we select the model with the highest validation performance and backtrack its history backtrack its choices of hyperparameter values during training: This yields the policy of optimal weights found by PBT which is then used to retrain a model on the full training set. We also experimented with retraining a model using the last weights of the policy, but this usually slightly underperform using the whole history of weights in the majority of cases.
qualitative results of the hyperparameter scheduled found by PBT
In Figure F, report examples of the policy of weights found by PBT and PB2 search for the parameters Eready 3, Q = 40% and = 12 for PBT (respectively = 8 for PB2).
5.3 Quantitative results
Here we report additional results for the CelebA and DomainNet experiments of Section 5, in the style of Table and Table the main text: We include results using ResNet-50 on DomainNet in Table B, and results for additional widths in CelebA in Table C and Table D.
21

---

Task weights
weights
pender
gerder
hai enya
thyle
lack Par
bold hur
moum
63
width
width 0.25
(a) Comparing Population-based training results for depth of 9 layers with full width (left) and smaller model with a quarter of the width (right). While both policy are quite different across training epochs, they converge towards similar distribution: For instance the weights for tasks "hair style", "gender" and "age" are significantly smaller than the one for the "mouth" and "hair style" tasks.
1.2
0.8
6.8
Task weights
0.6
6.4
0.4
Talr 20
641
0.2
cstro
8.2
This muature
Soe rmatum
0.5
0.
depth
depth 6
(b) Comparing Population-bas based training results for depth layers with full width (left) and smaller model with a depth of 6 layers (right). Similarly to the results varying width in (a), both search converge to similar distribution in task weights
1.0
3,0
Taak weights
prider 3in
0.4
gorder
Thisk
6.4
mash
catter
dottes Soos rETaTtLr
20
10
50
Population- Training (PBT)
Population- based Bandit (PB2)
(c) Comparing Population- based training results for depth of 9 layers with full width (left) and the same search with Population- based bandit (right). The two search algorithms converge to significantly different results particular regarding weights for the "mouth" and "facial hair" tasks. This suggests that (i) there may be multiple good local minima in the search space of p and (ii) the heuristic used the explore step has significant impact on how the resulting policy.
Figure F: Qualitative results for Population based training search CelebA. The x-axis represents training epochs. The y-axis represents the policy scalarization weights for each task as a cumulative histogram for the run of the population with highest validation accuracy
22

---

Table B: Results of MDL when jointly training on all 6 domains of DomainNet for scalarization (uniform and PBT-found weights) and MTO methods with ResNet50 backbone. PBT run with population size of N = 12 models, such that every Eready = 5 epochs, Q 25% of the population triggers an exploit/explore
DomainNct (ResNet550 with 0.25 width)
average
clipart
infograph
painting
quickdraw
real
sketch
Scalanzation
Uniform
49.69 0.05
59.90 0.15
22.45:0000
4.14
63.13
58.95
49.79
0.10
PBT
50.69 = 0.10
61.69 = 0.45
21.27 11.2700.0 0.10
44.72
63.96
0.13
50.06
0.17
MTO Loss-based
Uncertainty [26]
40.51
15.70
0.02
34.44
54.27
37.45
IMTL-L [38]
37,04 0.17
48.85 0.59
13.93
0.42
30.64
0.36
$1.60 0.34
44.12 0.21
33.12 0.50
MTO Gradient-based
CAGrad [37]
39.82 0.10
50.68
0.05
16.94
34.37
52.16
46.59 = 0.00
38.20 = 0.07
GradDrop
39.18 0.15
49.80
0.04
16.77 0,65
33,95 0.52
51.18 0.06
46.04
0.28
37.36 0.20
PCGrad [64]
39.48 + 0.31 50.42 0.97
16.83 0.31
34.63 0.92
$1.14 0.53
46.37
0.51
37.49 0.98
DomainNet (ResNet50 with original width) clipart
average
infograph
painting
quickdraw
real
sketch
Scalarization
Uniform
51.53 + 0.06
61.89 + 0.12
23.63 * 0.01
45.87 * 0.01
64,50
0.08
61.46
51.83 # 0.33 +
0.09
PBT
51,83 + 0.06
62.22 # 0.06
22.61 = 0.20
46.61 # 0.29
64.71 0.05
61.91
0.05
52.93 # 0.10
MTO Loss-Losaeee based
Uncertainty [26] 42.90 0.20
56.24 0.44
17.36 0.12
36.91
0.82
56.11
0.08
50.33
0.48
40.49 = 0.51
IMTL--I [38]
39.69 0.13 52.51
0.59
15.51
0.14
33.45
0.14
53.54
0.05
46.37
0.48
36.75 0.17
MTO Gradient-bused
CAGrad [37]
41.90 0.13
0.43
17.94
36.72 0.39
54.15
0.03
48.31
40.94 0.14
GradDrop [8]
42.15 0.14
53.38
0.54
18.68
0.33
37.00 0.45
53.85
13.8520.. 0.11
49.04
0.27
40.95 # 0.04
PCGrad [64]
41.94 0.20
$3.46 0.69
18.20 0.28
36.95
0.66
53.29 0.13
48.88
0.48
40.87 + 0.49
23

---

Table C: (Table best seen zoomed in PDF) Results of MTL when training on all tasks (subset of attributes) of CelebA for a depth of 6 layers For PBT and PB2 we use slightly different parameters than DomainNet to account for the fact that CelebA contains more tasks, and hence has a larger search space: All PBT runs use a population size of N = 12 models, such that every Bready = 3 epochs, Q = 40% of the population triggers an exploit/explore step. For PB2 runs we use a population size of N = 8 and otherwise the same Q and Fready hyperparameters.
VIT-S4, 6 layen,
factal
averuge
age
gender
styl
Stalomization
Unifem
90.2 = 1c404 86-96
Sc-14
02.25
84
6hodl4
95.30
97.61
7.1c-04
92.55
3.4044
91
86 23
4:-04
PBT
90.9 = 1.8e
86.94
7.5e
92.37
10332-330 33
4.5e
91.22
+ 4.3e
91 3.5-00
36
PR2
6 20
10.10
1.2e-00
90. 13
33e-0
84.1
64
95.02 2.860
97.41
4.8e-4
91.61
5.560-0
91.1842.3e00
36 30
1e
MTO Lewsshoued
Unaortainy (26) 9882 19c004
84.33
30c-44
2.722
9762=22:00 28:-44
91.57
a 3.064
91.00
4.1o00
86.22
9c-04
IMTL L (58)
90.2 e 2.0e 04 86.9
1.2e/00
73
84.63
90.3222.76 Os
92.57
3.8e
91.02
4.
86.22 + 1.6e 04
92.35
2:7004
GnalDopp
1-76-11
02.08
97.41
1-4-00
5.-05 36 06:22-2
PCGind [04]
90.86
I.le
92.32
84.03
9
92.64
MTTSSH wilt
clothes
facial
bair
styl
Iuir
91.25 28004 $7.33 =: 100-03
92.71
85.15
28:04
95.49
98.3
6:0.044
92.99
3.80-04
R:-04
3.3--04
7.2e-0
MTO Lose-louul
Unacstaiit [26] 91.30 3.1044
87.52 2.0.03
02:72 1.40-44
4.:04
98.88
5.5-04
92.07 3.5-04
91.65
1.0-004
$6.85
= 6-4-04
21.30 57,49
144-03
02.04 Oibeltt
$5.16
1:0e-04
95.51
1..e-4 1.Je-M4
98.06
92.98 Ne-04
91.00
86.85 1le-3
CACirad [37]
91.32 3.204
67.47
2.3665
92.86
65.26
3.1044
05.51 i 1.-34
08.3
(10c000))
03.00 e 9c-04
91.73
1.5p-04
56 + 2:3-44
GndDDoo [5]
91.19 D4
8.5-2
92.00
+ 5.20 04
9.5e
a 2.9e 04
97.96
de 04
92.80
1.8e
91.54
3:35e 04
26.70:5::00 04
POGrdd (64)
91,30 4.000 $7.45 2.8e-05
92.72
5:0e-0
$5.10 8.9e-4
53.50 5.30--4 04
98.08 2.1e
92.97 e: 1.0e-04
91.00 24100
36.82 3.3e
VIT-S% 6 loyers, full wolth
average
age
clothes
fooe stndcum
facial
hsir
gonder
color
hair
style
horr
mootth
Uniorm
91.25 04
10 2.
44-04
16
04-04
98 98.00
92.99
3.0e-04
91
91.88
4.7e-05
36
Re-04
PET
91.21 a 22-04
86
1-le-
92.77 a 586-0
85 18
16-04
95.42
98
02
4-5000
92.91
6--004
91
1.0-05
86 95 a Re-04
PR3
91.28 a 255004
86
15-01
90.37 2.5rb
85 17
24-00
95.47
Spitt
98
36.10
3.92-44
92.90
5.80-04
91.51
86 94 1c-03
Uncenaimy [ib]
91.22
53e-00
91.72 2:le-0
CAGrud
91.25 11225222.0
8 1.7-00
02.58
7:lc-05
65.12 1.c414
95.46 4.56004
98.17
1.1-04
92.01
2.1444
91.74
3.6c-25
86.83
1.5:-04
GnddWrop [3]
91.29 23
83.01
4.3.le
52,00 13e-03
8
2.7e0l
98.15
03.00 9.
91.03
1.5e-00
36.97 4.0e
POCrd (64]
91.21 e 38c-04
86.55
4.3ee-0
52.11 293e-4
15.15 844e00
29.4419
98:16
500-04
92.92 2.5o-04
91.75
1.3e-03
36.87 4:8e-04
24

---

Table D: (Table best seen zoomed in PDF) Results of MTL when training on all tasks (subset of attributes) of CelebA for depth of 9 layers. For PBT and PB2 we use slightly different parameters than DomainNet to account for the fact that CelebA contains more tasks, and hence has a larger search space: All PBT runs use a population size of N = 12 models, such that every Bready = 3 epochs, Q = 40% of the population triggers an exploit/explore step. For PB2 runs we use a population size of N = 8 and otherwise the same Q and E Pready hyperparameters.
VIT-S4, layen,
factal
averuge
age
gander
styl
Scalomizatinn
Unifem
01.00 2.5044 87.24
Lic-1
02.40
766405
84.85 x So414
95.34 92.27
Si0-04
92.66
3.5od4
91.31 5.a00s
86 40-40 Kik--5
PBT
90.7 e 1.7e
86.00
1e
92.34
04
+
9
2.9
91.55
1.
91.15
36
PR2
81,21
1.9-00
92
4 255-44
$4.9 4.2204
95.15 3.5e0l
97:20
6.0e-04
91.87
1.4e-04
91.36 3.1e-00
36
4e
MTO Lensshoued
Unaortainy (26)
91.11 =.00::23:04 233044
87.18
1.400t
4.540
84.87 x 3.504 6.554
92.70
3.904
91.30 4.5p-ti
86
Ic-04
LMTL
91.00 04
87.18
1.5e (0
92.40
84.87 + 4:0e
6.
97.81 5.0e 04
92,70
3,5e
27730 0z
36.44
6.20
92.55
3.7004
GndlDop
02.13
92.66
2.50044
PCGind [04]
90.95
92.29
84.77
95.32
97.72 3.5e
92,70
91.21
36 04
MIT-SM, wilh
clotbes
facial
Isie
color
syle
Jair
Stalonizatiun
3.004 87.26 1.34t
92.51 244-04
3.004
98.15
7.1.-04
8.9c-16
91.66 4.0-04
7.3c-04
91.36=23--04 87.45
92.56
9
2.3-04
91.70
87
MTO Lewdhhed
Uneottiny [26] 91.33 14c404
02.54 4.300
85.10
1c414
08.15
7:8-04
92.06
2.0c444
91.60
6.5
86.93 2.5-004
157L
91.20 5.025
92.70
85.17 T.le-15
05.50
4.99-04
98.10
8:5
02.07
1.4c-14
91.66
4.8e-0i
86.92
9.76-5
MTO Grodieen based
CACiai [37]
91.37
18004
02.00
85.24
1.3c433
95.51
2.3604
98.20
1.c-04
92.05
x 2.60-444
91.78
2.7004
86
1-4:-04
GnalDrop [8]
91.27 18c-04
92.71 3.5ce45
85.10
1.le-44
95.46
6.26405
98.04
1.3x-68
92.04
2.342-443c-tt 4.3c-04
91.57
3.2e-04
86.85
POGrnd [64)
91.29 16x-04 87.10 3.20-14
24.63
85.15 6.c.044
95.47
1.1004
98.11
6:42044
93.01 8.9c-11
91.67 6.36-04
86 06.96: 2.6-044
VIT-S%, 9 layers full widh
meergg
clothes
Eacd strecture
facial bair
gender
color
hair
hair stle
mouth
Scaltrication
UniSon
91.17 1.7044 37.33
92.50
k.e.04
$5.10 4::044
95.45
1.0-04
97.93
22.85
1.2x-44
36 6.9c.4
PBT
57435311004
92.51
55.18
95.46
2
04
97.75
92.51
$7.00 5.8c
PRO
91.25 2.500
6683 12668
92.85 4 4o-004
65.12
7.3604
95.49
5.5o-44
96 06.19-
6:0e-04
92.90
56-04
91.78 Sp-04
36.81
Unoettinty [2]
91.17
1.8-04
02.50
1.6e-44
3.-04
91.39
5.3c-Id
IMILA (58)
MTO Gindiennden
CAGrad
91.21 14c04
47344433:04
02.64
45.16::0:004
05.43
24,664
07.91
1.1c04
02.80 3::-04
91.58
3.6e-04
= S.5:-04
GnalDop [8]
91.20 1.4-04
2
02.81
246-04
45-09
05.40 8.1004
98.09
2.66--44
2:-01
91.71
3-3.c-044
=6.00 8:9--66
POGrnd [64]
91.13 3.5e044
67.5 e 244--5
92.50
1.1e-t0
$5.04
5.3-04
95.40
9137
000+00
92.82 2
91.37 2: 1:2e-04
36.71
25


Reviews:
[{"review_id": "vspVDBHFXB", "reviewer": "Reviewer_iXKX", "summary": "This paper analyzes multi-task learning (MTL) and multi-domain learning (MDL) setting. The paper observes several points: 1. MDL/MTL improvements are more significant with bigger network capacity, 2. Gradient conflicts are not necessarily well correlated with MDL/MTL performances, 3. Tuning scalarization weight is important for MDL/MTL performance, 4. population-based training (PBT) can be an efficient way to tune the scalarization weights when there are many tasks. The experimental results support each of their claims, and lastly this paper show that PBT can be even competitive with memory-expensive gradient-based methods, such as PCGrad.", "questions": "See the comments above. Overall, I don't think this submission is above the acceptance bar. Most of the observations seem obvious and not very informative. In order for such an analysis-style paper to be accepted, the analysis should be 1. better organized with clear insight, 2. providing novel and useful insights that have not been found by other researchers.\n\n------------------------------------------------------------------\n[After rebuttal]\nI read the author's rebuttal and other reviewer's comments. Unfortunately, I'm still not convinced of the rebuttal, thus I maintain my current score.", "limitations": "The authors have properly addressed the limitations of this paper in Sec 6.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "- Paper was easy to read.", "weaknesses": "[major comments]\n\n- overall, almost all observations found by this paper seem quite trivial to me. Currently, I don't think this paper provides very useful insights or something that has not been investigated before. \n- For instance, in page 2, (C1) is already quite trivial - we already know that larger network capacity can mitigate negative interference because larger capacity means it can accommodate more diverse information. See [1] for the reference.\n- (C3) is also trivial - we already know that tuning scalarization weights is important and they should be tuned differently for each task/domain/architecture, and so on.\n- The conclusion of (C2) is misleading, in my opinion. The authors observed that MTL/MDL performance improves with bigger network capacity while the degree of gradient conflict remains the same, and they conclude that gradient conflict do not correlate well with the actual MTL performance in practice (L219-220). This conclusion sounds weird because the network architecture is different. What if the network architecture remains the same and we resolve the gradient conflict, which is the usual assumption of other papers, such as PCGrad? \n- (C4) is simply an application of the existing technique (PBT) to scalarization weights, which is not very surprising. And the authors did not provide other baselines than Uniform, although there should be many existing methods that allows to carefully tune the scalarization weights.\n- The same for the conclusion in section 4.1. All (C1), (C2), and (C3) sound obvious to me.\n- (C1): we already know that larger network capacity can mitigate negative interference effect, as mentioned above.\n- (C2): Of course the best-performing scalarization weight would not be p1=p2=0.5.\n- (C3): Of course MTL/MDL has a regularization effect, so the training loss converges slower but the test accuracy is higher.\n\n[minor comments]\n\n- in (1), the notation $\\frac{\\nabla}{\\nabla}$ looks very weird. It should be either $\\frac{\\partial}{\\partial}$ or simply $\\nabla_\\theta \\mathcal{L}$.\n- in (2), the definition of $f$ is missing. What is it? (I assume that it's $\\nabla_{\\theta_i} \\mathcal{L}_t(x_t,y_t)$?)\n- missing baseline - Sequential Reptile [2], which can resolve gradient conflict issue without heavy memory overhead.\n\n[reference]\n\n[1] Wang et al., On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment, 2020\n[2] Lee et al., Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learning, 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Intuitiveness is not a weakness:** While some of the observations in our paper may align with intuitive understanding, intuition does not always translate into empirical evidence. Our work aims to provide a rigorous and systematic analysis of multi-task learning (MTL) and multi-domain learning (MDL), and we believe that our findings contribute novel insights to the field. In particular, insights such as C1 and C3 are rarely taken into account or emphasized in previous multi-task optimization works, hence labeling them as \"obvious\" and \"trivial\" does not reflect current literature. Nevertheless, if reviewer iXKX would like to suggest additional references we should examine, we would be happy to further address those during the discussion period.\n\n**(C1) [Model size]**\nWe are not aware of a previous reference analyzing and generalizing the link between model size and MTL/MDL performance. In fact, many MTO works (e.g. [a, b] for recent references) are still evaluated on very specific architecture/dataset combinations, omitting the effect of model size that may affect model comparison.\n\nWe have reviewed the suggested reference [1], and it focuses primarily on the impact of dataset size on negative interference in multi-lingual models. While this is an important aspect, it does not directly address the link between model size and MTL/MDL performance that our paper explores.\n\n**(C2) [Gradient conflict]**\nOur conclusion (C2) does not undermine the effectiveness of existing gradient conflict resolution methods in multi-task learning (MTL) or multi-domain learning (MDL). Indeed, as highlighted in reference [c], multi-task optimization methods such as IMTL or PCGrad can have a beneficial regularization effect.\n\nHowever, our analysis shows that minimizing gradient conflict does not necessarily lead to optimal MTL/MDL performance. We believe this perspective adds valuable insights as it challenges the common assumption that reducing gradient conflicts to zero should be the de-facto way to solve task interference in MTL/MDL (additional figures and results can be found in the PDF attached to the global response). \n\n**(C3) [Tuning scalarization weights]** \nWe do not think it is obvious that uniform weights p1=p2=0.5 are never the best-performing solution: Uniform weighing can be a reasonable assumption as MTL models are usually evaluated by taking the uniform average of their respective task metrics, in particular when the training losses and test metrics coincide (e.g., in the Taskonomy example we consider in the main paper). In fact, the use of uniform weighing for task losses remains a prevalent scalarization method in both vanilla MTL and gradient-based multi-task optimization methods.\n\nMoreover, the claim of (C3) is not solely on the benefits of tuning scalarization weights, a concept that has indeed been explored, for example, in [d]. Our work also investigates how and whether the optimal scalarization weights evolve across different model capacities (lines 56-60 and section 4.3). This exploration adds a new dimension to the understanding of scalarization in MTL/MDL. and provides insights that extend beyond existing literature.\n\n**(C4) [Application of PBT]**\nTo the best of our knowledge, there does not exist a well-established efficient method to tune scalarization weights in large-scale settings: Classical search algorithms such as grid search or Bayesian optimization do not scale well to larger number of tasks/tunable parameters. While PBT itself is not a novel search algorithm, showing that it can be successfully applied to the context of scalarization is a novel contribution of our work; We additionally show that the learned schedule of dynamic task weights through PBT can compete with automatic task weighing from state-of-the-art MTO methods. \n\nIn conclusion, we believe that (C4) offers valuable insights and an efficient practical solution for tuning scalarization weights.\n\n\n**(Minor comments)**\n  *  $f$ refers to an arbitrary function of $(x, y)$ to illustrate the link between the reweighing and resampling formalisms; but indeed in the context of equation (1), it can be replaced with $\\nabla_{\\theta_i} \\mathcal{L}_t(x_t, y_t)$\n  * The suggested reference **[2]** tackles the problem of transfer learning/catastrophic learning in multilingual learning which differs from our setting (multi-task learning from scratch). It also does not incur memory overhead compares to other MTO methods, but it trades it off for additional computations (each parameter update necessitates K gradient steps in Equation (7) of **[2]**).\n\n\n**References**\n  * [a] RotoGrad: Gradient Homogenization in Multitask Learning, Javaloy et al\n  * [b] Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout, Chen et al\n  * [c] In Defense of the Unitary Scalarization for Deep Multi-Task Learning, Kurin et al\n  * [d] Do Current Multi-Task Optimization Methods in Deep Learning Even Help?, Xin et al \n"}, {"review_id": "bV73VRJXsK", "reviewer": "Reviewer_wCAe", "summary": "This work is interested in analyzing the extent to which scalarization is an effective strategy against negative transfer in multi-task and multi-domain learning. Scalarization focuses on selecting an adequate weights for a convex combination of task losses, rather than employing expensive or complex conflict mitigation strategies. Although the search space for scalarization weights grows exponentially with the number of tasks, during training minimizing a weighted sum of task losses is fast and simple compared to many complex optimization methods and has recently been shown to be just as good.\n\nThis work therefore attempts to better understand the dynamics of scalarization in multi-task models by studying multi-task generalization under scalarization alone. They find a few findings which appear consistent across the settings they consider: the first is that scalarization is more effective as model capacity grows; the second is that uniform scalarization is rarely optimal, implying that for each MTL setting the scalarization weights must be tuned; the final observation is that while gradient conflict can predict e.g. task affinity, it does not predict generalization because model capacity does not observably affect gradient conflict.\n\nUsing these observations, the authors posit that a strong scalarization approach to MTL can be extremely effective, but the search for the optimal scalars makes it more expensive than other proposed methods. To this end, the authors leverage population based training to efficiently explore the parameter space of task weights. They find that models trained with scalarization weights from PBT outperform the uniform scalarization baseline, as well as several other sota optimization methods.", "questions": "- Do you know how PBT scalarization compares to random scalarization?\n- How does the entire procedure of PBT compare to other optimization methods w.r.t. total compute time or flops?\n- Finally, I’m particularly interested in whether or not the trends w.r.t. model capacity and gradient conflict hold as the number of tasks increases. Do you happen to know if they do?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is well written. It conveys its core ideas and motivation clearly.\n- The in-depth, rigorous exploration into multi-task learning dynamics is important as recent work has shown that most prior optimization work is not actually beneficial for standard MTL problems.\n- The observation that uniform weighting is rarely optimal is useful, even if not surprising.\n- The findings w.r.t. task conflict and generalization are very interesting, and helpful to consider in further development of MTL methods.\n- In total, the analysis of section 4 could be helpful for the design of future methods which aim to target scalarization, and they serve to motivate the proposed method in section 5.\n- The proposed method is clearly useful empirically, and demonstrates the effectiveness of scalarization vs. other, much more complex, optimization methods.", "weaknesses": "- All 3 key conclusions come from experiments which study only 2 tasks at once. While it is not unreasonable to extrapolate some conclusions from this setting, some conclusions could be at least verified for larger task settings, even up to 3 or 4 simultaneous tasks just to ensure the trends still hold. For example, does model capacity really not affect gradient conflict levels when considering all 40 tasks of CelebA?\n- The models all use the optimal single-task parameters but this might be unfair to the MTL models, e.g. [1] suggested that the learning rate should scale with the number of tasks if all else is fixed.\n- The final results should probably use random scalarization [2] as an additional baseline. I find this to be especially true given that the uniform models almost uniformly outperform the other optimization methods, so it is not extremely surprising that additional tuning of the task weights will result in the best performance on the tables.\n- The comparison to previously considered SOTA methods only goes up to 7 or 8 tasks, whereas many of the methods were tested on e.g. up to all 40 tasks on CelebA. It’s not clear if PBT can efficiently scale up to 40 tasks. To that end, a comparison of the overall compute used by the tested methods would be really helpful.\n\n[1] the importance of temperature in multi-task optimization, Mueller et al., 2022\n\n[2] reasonable effectiveness of random weighting: a litmus test..., Lin et al., 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Gradient conflicts:** In response to reviewer wCAe's inquiry about gradient conflicts, we have expanded our analysis to include experiments that encompass all 40 tasks/attributes of CelebA and all 6 domains of DomainNet, when training a uniform MTL model. These additional experiments are illustrated in Figure 1 of the attached PDF. Generally, we observe that while the proportion of gradient conflicts may increase with the number of tasks, the overarching trend remains consistent: Low gradient conflict does not necessarily correlate with optimal MTL/MDL performance. This observation holds true across different model sizes and aligns with the conclusion C2 of the submission.\n\t\n**2. Population Based Training (PBT)**: \n\n**a. Computational cost:** A nice feature of Population-based Training search is that the computational cost can be easily decorrelated from the number of tasks. In fact, the main computational cost comes from the number of models in the population ($N$): While increasing $N$ does increase the search space coverage, it is also possible to control the exploration/exploitation trade-off through other parameters; namely the number of epochs before models in the population pause and are compared against one another ($E$) and the proportion of the population killed in each exploration step ($Q$); both $E$ and $Q$ have an impact on the computational cost which is often negligible in practice as it only incurs a few additional checkpointing/writing operations. In other words, even for a large number of tasks, we can use a low value of $N$ which introduces a natural trade-off between the compute budget allocated to the scalarization weight search and search space coverage. \n\n  To better highlight this trade-off, we report additional results of PBT search while varying the hyperparameters $N$ and $E$ in the global response (Table 1a); As suggested by reviewer  wCAe, we scale the experiments to cover all 40 tasks on CelebA. In that setting, while increasing population size generally improves the search result, we observe that *(i)* we can get good performance even when the population size is significantly smaller than the number of tasks and *(ii)* in the regime of a large population, the cost of going from e.g. N=24 to N=40 is not worth the gain in accuracy.\n\n**b. Comparison to multi-task optimization methods (MTO):** In the attached PDF (Table 1b), we summarize the computation costs of PBT and some MTO methods. The key difference between gradient-based MTO and PBT lies in the memory bottleneck: PBT requires training $N$ models, but independently, hence the algorithm does not require additional storage compared to training a single model. In contrast, gradient-based MTO methods only train a single model, but require storing each per-task gradient to compare them against each other in every training iteration. \n\nNevertheless, the efficiency comparison between MTO methods and PBT search is highly dependent on available resources. For scenarios with ample RAM/GRAM and a reasonable number of tasks, MTO methods may still be computationally viable. Conversely, PBT's natural parallelism is favorable in scenarios with many memory-constrained devices. \n\t\n\n**3. STL vs MTL learning rate**\n\n**a. PBT experiments**: For the experiments of section 5 in the paper (PBT and MTO comparison), we did train all models for different learning rates and report results for the best performing one (specifically `[5e-5, 5e-4, 5e-3]` for CelebA and `[3e -3, 3e-2]` for DomainNet)\n\n**b. Analysis**: Indeed, further tuning the learning rate for MTL models could lead to better performance with respect  to the STL baseline; in our setting, this would be roughly equivalent to dropping the constraint that the scalarization weights have to sum to 1 (line 115) but would also greatly increase the search space / number of experiments. \n\tIn our analysis, we mainly use the single task as a reference baseline with the primary goal to compare the relative improvement of the different MTL/MDL against each other (for different task ratios and model sizes). This is why we chose to select the learning rate as to favor the STL baseline rather than a specific MTL setting (e.g. the uniform weights).\n\tNevertheless, after reviewing the suggested reference [1], it would indeed be interesting to investigate how their conclusions on optimal learning rate/task temperature evolve across different model sizes.\n\n**Random scalarization baseline**. We added results for the suggested random scalarization baseline (RLW) of **[2]** for Table 2a in the attached  global response PDF for the CelebA setting with 40 tasks. Following the reference, in every training iteration we sample task weights from N(0, 1) and normalize them via softmax function. In that setting, we observe that RLW yields a slightly stronger baseline than uniform weighing on average, and that the configuration found by PBT outperforms both when searching with more than $N=6$ population size."}, {"review_id": "CJ5WCppaCX", "reviewer": "Reviewer_ytBJ", "summary": "This paper seeks to better understand the complexities of unitary scalarization for multitask and multi-domain learning. The paper explores the impact of model size, degree of gradient conflict and variations in scalarization weights in order to derive a set of guiding principles for MTL/MDL. \nThe authors also propose to leverage existing population based HP optimization procedures to efficiently search for the best set of scalarisation weights", "questions": "Questions \n1. Is the sweep to find the best single dataset HP performed for each model size (For figure 2) or is it performed for 1 model size and then used across all sizes ?\n2. For figure 3, how do the hyper-parameters like learning rate differ from for model size\n    1. Do you use a fixed learning rate across all model sizes ? Or use a pre-determined best learning rate for each model size ?\n3. Are the resnets in Figure 3 pre-trained resnets ? \n    1. It would be interesting to see if the effect still holds for resnets trained from scratch vrs pre-trained on say Imagenet\n", "limitations": "Authors have addressed limitations.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an expansive set of experiments to understand the impact of model size, multi-task setting (MTL vrs MDL) and task affinity on the performance of the unitary scalarization approach\n2. The paper is actionable -- it proposes to leverage pre-existing population based HP optimization approaches to search for the best scalarization weights.\n3. The paper is well written and the experimental methodology is described in sufficient (reproducible) detail\n", "weaknesses": "My main issue with the paper is that I am hesitant about specific parts of the experimental methodology.\n\nPrimarily, in  section 4, the experimental procedure is described as tuning HP for single task and then using the best single task HP for all followup multitask experiments. This creates an unfair comparison since it assumes that the HP setting that is best for the single task is also optimal for the MT setting thus bringing the robustness of the results to question. (This is especially considering that, forcing the scalarization weights to sum to 1, means that the effective per-task learning rate is always smaller than for the single task setting)\n\nAlso, there exists confounding variables for the the experiments on **model capacity and  gradient conflict**  that are not addressed. \n1. Are the models used ResNet models used for Figure 3  pre-trained or learned from scratch ?\n2.  How were the learning rates for this experiment chosen ?  In general, I suspect that there is also a dependence of the degree of conflict (after a reasonable number of epochs like 1 in your case) on the learning rate used. It would be important to see if the degree of conflict (at a fixed model size) -- varies substantially with learning rate and whether this variance is smaller or larger than the variance that comes changing model size at fixed LR. \n\n\n----- Update ------\n\nUpdated score after rebuttal. Thanks for the responses @ Authors\n\nI am willing to raise my score if these concerns are addressed. \n\nMissing relevant citation\n1. Exploration of model capacity and scalarization weight https://arxiv.org/pdf/2302.09650.pdf \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Gradient conflict and impact of learning rate:** Following reviewer ytBJ's suggestion, we added gradient conflict measurements experiments with varying learning rates, as illustrated in Figure 1 of the attached PDF (global response). The results show that the variance of gradient conflict measurements tends to be higher across learning rates than across the different model sizes we considered. Notably, we observe that when the learning rate is excessively high, leading to divergence in training loss, there are distinct peaks in gradient conflicts. Nevertheless, when the loss is well-behaved (does not diverge), the general trend of gradient conflict measurements across training epochs remains similar across learning rates.\n\n**Learning rate choice:**\n  *  **For the different model sizes in the analysis:** We do conduct a sweep across different learning rates for each single task architecture (e.g. `[3e-1, 3e-2, 3e-3, 3e-4]` for DomainNet). However, in practice, we found that the optimal single-task learning rate remained consistent across different model sizes. While this might vary with a more fine-grained learning rate grid search or different family of architectures, our analysis led us to a specific learning rate for one model with different depths/widths for practical purposes. In practice, the only parameter we adapt for different model sizes is the batch size and the number of gradient accumulation steps.\n  * **PBT experiments:** For the experiments of section 5 in the paper (PBT and MTO comparison), we train all models for different learning rates and report results for the best performing one (specifically the sweep is done over `[5e-5, 5e-4, 5e-3]` for CelebA and `[3e -3, 3e-2]` for DomainNet)\n* **Normalizing scalarization weight to sum to 1:**  We acknowledge that removing the constraint of normalizing scalarization weight to sum to 1 could be akin to further tuning the learning rate for each MTL model, potentially leading to improvements over the single task baseline. Since our primary goal was to compare the performance of the different MTL/MDL against each other (mainly using single-task performance to compute relative improvement), we chose to keep the normalization constraint to keep the search space at a reasonable size and make the analysis more scalable.\n\n**Pretrained vs from scratch models:** In all experiments, we train the models from scratch. It is very likely that different pretraining strategies would impact task interference and MTL/MDL performance, but we didn't investigate this angle in this work.\n\n**Missing citation:** Thank you for the suggestion. We will add the citation on designing scaling laws for multi-lingual language models in the related work section\n"}, {"review_id": "q68ihHY5Z0", "reviewer": "Reviewer_kG9J", "summary": "The paper studies scalarization for multi-task and multi-domain learning, which is a method to combine the losses of different tasks/domains. The authors conduct substantial experiments to draw insights into the effect of scalarization weights on multi-task/domain learning. They also propose an efficient method to search for a good set of weights.\n", "questions": "1. In section 4.1, to obtain the optimal scalarization weight, do you perform grid search? I wonder how you are able to select a single “optimal” weight since firstly, it is impossible to search for the weight exhaustively, and secondly, it is likely that several sets of weights have undistinguishable results. Could you clarify this?\n2. In fig. 4(b), it seems that training on a single task is always better than training the two tasks jointly regardless of model selection and weight tuning. Any insights on that?\n3. For sec. 5, if possible, it would be interesting to see how much gap PBT closes compared to some oracle (more exhaustive grid search), even on datasets with few tasks/domains. It is unclear how good a performance a static weighting strategy can achieve because the only static baseline is uniform.\n", "limitations": "The authors have adequately addressed the limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors make a valuable attempt to understand the scalarization of multi-task/domain learning. It is an important problem in the literature as despite the existence of various automatic weight selection methods, it is unclear under what circumstances these methods will outperform a static scalarization. The insights are critical and novel, especially the one about conflict gradient and scalarization v.s. dynamic weight update, which is different from common belief.\n", "weaknesses": "The writing can be improved. The goal of this work is ambitious because the authors attempt to study the effect of scalarization from a wide range of aspects. However, such ambition also makes the paragraphs very condensed and jumping. In many cases, the authors bring up several insights in one paragraph with little logical flow (e.g. sec 4.1). It is better to use latex paragraph if many parallel information needs to be conveyed.\n\nSome conceptual analysis is lacking. Since this paper is purely empirical, the generalizability of the insights is questionable if proper conceptual analysis is missing. For instance, in fig. 3, based on the two plots, it is difficult to reach any conclusions about the influence of conflicting gradient on performance because the two experiments themselves have conflicting results. For this part, an important question remained to be answered is that how conflicting gradient affect training and why in some cases it has less effects. There are a few “jump to conclusions” situations especially in sec. 4.1 and 4.2.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Improved writing:** We will carefully revise the manuscript to ensure that the logical progression of ideas is clear and coherent, and emphasize the most important insights.\n\n**Generalizing the conclusions of Figure 3:** We expanded the results of Figure 3, as detailed in the attached PDF to the global response (Figure 1). This extended analysis includes more tasks and varying learning rates to convey our insight more clearly: the MDL/MTL models with the highest generalization performance are not necessarily the ones with the least amount of gradient conflicts, and vice versa. We also observe that, while the global amount of gradient conflicts tends to increase with a higher number of tasks, the overall trend of each curve is rather consistent across model sizes.\n\n**Question 1 (optimal scalarization weight in analysis)** Indeed, for Figure 4.1, our approach essentially involves a grid search: For each setting, we perform a sweep for the task weight over $p_{task_1} \\in \\{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9\\}$ and then report the relative improvement of the MTL/MDL model over the corresponding single task baseline for the best ratio (where the best ratio is defined as the one that yields maximum accuracy averaged across the two tasks/domains): This can essentially be interpreted as having an oracle to select the task weight. In Section 2.1 of the supplemental material we also report some plots illustrating the results for all the ratios $p_{task_1}$ we sweep over.\n\n**Question 2 (Figure 4b):** Indeed, on the Taskonomy dataset we sometimes observe that the STL baseline outperforms the MTL models still. While this could simply be a symptom of the specific architecture/hyperparameters we considered, we also hypothesize that this might show the limits of forcing all tasks to share a common encoder: In fact, the Taskonomy dataset was initially introduced as a benchmark to uncover groups of tasks that can benefit from training together, from ones that should be kept fully separate to avoid interference [f, g]. Building on this orthogonal line of work, we posit that some task interferences may only be resolved through explicit architectural modifications. For example, allocating dedicated encoders for specific tasks might mitigate the observed performance discrepancy. \n\n\n**Question 3 (PBT vs static grid search oracle)**\nWe address this question from both theoretical and practical perspectives:\n  * **In theory**: While PBT itself does not provide any guarantee, its more recent variant PB2 [e] (which uses Bayesian Optimization to guide the exploration) provides a theoretical regret bound as a function of the population size $N$.\n  * **In practice**: In Table 2 of the attached global response PDF, we performed an experiment for comparing grid search (with uniformly distributed grid points) to PBT in a small setting with only three tasks/attributes of CelebA (`Five_o_Clock_Shadow`, `Arched_Eyeberows` and `Attractive`). Due to time constraints, we were only able to run experiments up to using 5 (uniformly distributed) grid points per task weight for the grid search, resulting in 125 models to run. In Table 2a, we observe that grid search can outperform the dynamic search of PBT, but with a much higher computational budget. \nWe further illustrate the search space covered by PBT during its dynamic search in Figure 3b: each point corresponds to a 3-scalarization weights configuration encountered during the dynamic PBT search. This visualization contrasts PBT's exploration pattern with the regular uniform grid used in a classic grid search, providing insights into how the two methods differ in navigating the search space.\n\n**References**\n  * [e] Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits, Parker-Holder et al\n  * [f] Which Tasks Should Be Learned Together in Multi-task Learning? , Standley et al\n  * [g] Disentangling Task Transfer Learning, Zamir et al\n"}]

Author rebuttal:
We thank all the reviewers for their insightful and detailed feedback. We have addressed each reviewer's concerns separately through comments, and we would like to use this global response to highlight the additional experiments we mention in these responses, which are illustrated in the attached PDF. Finally, we would be happy to address any further questions during the discussion period.

**Gradient conflict (Figure 1):** To reinforce the conclusion *(C2)* (Figure 3 of the main submission), we follow the reviewers' suggestion to investigate how gradient conflict measurements evolve with respect to the number of tasks (reviewer wCAe) and the learning rate (reviewer ytBJ).  In these experiments, we follow the same protocol as in Section 4.2, by measuring the percentage of gradient conflict pairs in each training epoch. We treat each attribute in CelebA as a separate task (40 tasks total).  We vary the learning rate (in `[5e-4, 5e-3, 5e-2]`) and model size (depth in `[3, 9]`; width in `[0.5, 1]`). We believe these additional experiments reinforce our observations from Section 4 demonstrating that while high gradient conflict is indeed indicative of bad MTL/MDL performance, low gradient conflict does not correlate well with best MTL/MDL performance.

**Population-based Training:**
  * **a) Computational cost (Table 1):** To address reviewer wCAe's question, we scale the PBT search to all the 40 tasks of CelebA and report the results in Table 1a. We report results across different population sizes (N), illustrating that the population size (hence the computational cost) does not have to scale linearly with the number of tasks. In addition, in Table 1b, we present a brief overview of the theoretical computational/memory cost of PBT compared to uniform MTL and state-of-the-art MTO methods.

  * **b) Comparison to grid search (Table 2):** To address reviewer kG9J's question, we report results for classic grid search on a small-scale MTL setting consisting of three attributes/tasks of CelebA. We also illustrate how the search space covered by PBT differs from the classic uniformly distributed grid search space.

Meta-review: The paper studies the intricacies of scalarization in the context of multi-task learning (MTL) and multi-domain learning (MDL). Through empirical analysis, several interesting albeit intuitive observations are made.

The paper tackles an essential problem in the ML domain and presents a rigorous empirical analysis to derive several crucial insights. Although some reviewers have raised concerns, the findings worth to be shared with the community. The authors are strongly encouraged to address the mentioned concerns before the camera ready version.
Decision: Accept (poster)