Can Optimization Trajectories Explain Multi-Task Transfer?
Authors: David Mueller
Mark Dredze
Nicholas Andrews
Venue: Accepted by TMLR ()

Abstract: Despite the widespread adoption of multi-task training in deep learning, little is understood about how multi-task learning (MTL) affects generalization. Prior work has conjectured that the negative effects of MTL are due to optimization challenges that arise during training, and many optimization methods have been proposed to improve multi-task performance. However, recent work has shown that these methods fail to consistently improve multi-task generalization. In this work, we seek to improve our understanding of these failures by empirically studying how MTL impacts the optimization of tasks, and whether this impact can explain the effects of MTL on generalization. We show that MTL results in a generalization gap (a gap in generalization at comparable training loss) between single-task and multi-task trajectories early into training. However, we find that factors of the optimization trajectory previously proposed to explain generalization gaps in single-task settings cannot explain the generalization gaps between single-task and multi-task models. Moreover, we show that the amount of gradient conflict between tasks is correlated with negative effects to task optimization, but is not predictive of generalization. Our work sheds light on the underlying causes for failures in MTL and, importantly, raises questions about the role of general purpose multi-task optimization algorithms.

Published in Transactions on Machine Learning Research (December/2004
Can Optimization Trajectories Explain Multi-Task Transfer?
David Mueller
dam& edu
Department of Computer Science
Johns Hopkins University
Mark Dredze
mdredze
Department of Computer Science Johns Hopkins University
Nicholas Andrews
noa@cs.jhue edu
Department of Computer Science Johns Hopkins University
Reviewed on OpenReview: https: // openreview. net/ pet/forree forum? id- QQE5
Abstract
Despite the widespread adoption multi-task training in deep learning, little understood about how multi-task learning (MTL) affects generalization. Prior work has conjectured that the negative effects of MTL are due optimization challenges that arise during training, and many optimization methods have been proposed to improve multi-task performance. However, recent work has shown that these methods fail to consistently improve multi-task generalization. In this work, we scek to improve our understanding of these failures by empirically studying how MTL impacts the optimization of lasks, and whether this impact can explain the effects of MTL on generalization. We show that MTL results in a general- ization gap a gap in generalization at comparable training loss between single-task and multi-task trajectories early into training. However, we find that factors of the optimization trajectory previously proposed to explain generalization gaps in single-task settings cannot explain the generalization gaps between single-task and multi-task models. Moreover, show that the amount of gradient conflict between tasks is correlated with negative effects to task optimization, but is not predictive of generalization. Our work sheds light on the under- lying causes for failures in MTL and, importantly, raises questions about the role of general purpose multi-task optimization algorithms. We release code for all our experiments and analysis here: https: //tp::////thtuu gighhu....... con/ wwwriidnndm///ulta. Task Optimization
1 Introduction
Multi-task learning (MTL) the simultaneous optimization of multiple related tasks has long history in machine learning (Caruana, 1993). By learning from additional related signals during training, multi-task learning yield models with stronger generalization than single-task models; however, these additional training signals may not always benefit one another, and MTL can also lead to models which generalize worse than single-task models (Figure I; Zhang et al., 2023). Prior work has conjectured that the negative impacts of multi-task training on task generalization occur due to optimization challenges that arise during joint training of multiple objectives simultancously. Consequently, number specialized multi-task opti- mizers (SMTOs; Kurin al., 2022) have been proposed address these optimization challenges in order improve the generalization of multi-fask models (Chen et. al., 2018; Sener & Koltun, 2018; Yu et al., 2020,

---

Published in Transactions on Machine Learning Research (December/ 2ecember/22244 22024)
inter alia). However, recently Kurin et al. (2022) and Xin et al (2022) found that these SMTOs actually fail to consistently improve the performance of MTL models over the baseline of uniformly aggregated SGD.
SMTOs are developed on the hypothesis that large differences between the gradients of tasks (often termed gradient conflict) gives rise to certain optimization challenges that naive SGD will not overcome. As a re- sult, many SMTOs are motivated by proving convergence in simplified settings (e.g. Yu et al., 2020; Chen et al., 2020) or demonstrating the method's superiority on toy optimization problems (e.g. Liu et al., 2021a) where such optimization challenges exist. However, in deep learning, it is not clear that these optimization challenges, lack thereof, can explain the mechanisms that drive transfer (the impact of MTL on gen- cralization). For instance, in single-task learning, large-batch training may result in worse generalization than small-batch training despite large-batch training leading to better training loss optimization (Smith et al., 2018). The disconnect between our understanding of how MTL impacts optimization and how MTL impacts generalization may explain the recent claims that SMTOs frequently do not improve MTL perfor- mance (Kurin et al., 2022; Xin et al., 2022). In this work, we aim to bridge this disconnect by asking how the impact of MTL on a task's optimization can explain its effect that task's generalization.
We approach this question first by comparing multi-task and single-task training loss trajectories to one another, studying the trade-off between tasks-both with respect to optimization and generalization- wwitin an MTL. setting, to understand why MTL benefits some lasks while simultaneously hurting others. Then, by studying the trajectories of a few target-tasks as the set of auxiliary tasks changes, we study how the amount of task conflict in MTL impacts task optimization trajectories, and whether this impact is predictive of transfer. Specifically, our research questions and contributions are:
What can training loss minimization tell us about multi-task transfer? In $3 we compare multi-task and single-task generalization by the total task training loss at each epoch across 5 multi-task settings. We find that transfer (both positive & negative) is observable across comparable training losses as early as a few epochs into training and is often maintained as a generalization gap between comparable training loss throughout the rest of training (e.g. Figure 1).
Can transfer be explained by factors of the training trajectory beyond training loss? In $4 we study whether certain factors of optimization trajectorics, previously connected to generalization in deep learning, are correlated with the impact of MTL generalization. We find that, while MTL does impact these factors (gradient coherence, early-stage Fisher information, and loss surface sharpness), the effect of MTL on task performance is not explained by these factors. Moreover, we show that, when SMTOs impact task performance, their effect is also not explained by their impact to these factors.
How does the amount of conflict impact optimization and generalization? Finally, in $5 we study how varying the gradient conflict that a task experiences impacts the factors that we study above. We find that high amount of gradient conflict. is correlated with negative impacts to all of the factors we study in $4. However, we simultaneously find that the amount of conflict has little-to-no correlation with task generalization and the effect of gradient conflict on optimization does not predict the benefit of different auxiliary tasks.
Our findings demonstrate a (current) inability of optimization trajectories and the impact of MTL on them to explain multi-task transfer. Importantly, our results make clear that (a) our current understanding of optimization and generalization in deep learning is not capable of predicting transfer from MTL and, as a result, (b) it is not clear what optimization challenges must be overcome in MTT, or what general purpose optimization algorithms need to tackle, to broadly improve the performance of multi-task models.
2 Background and Preliminaries
2.1 Multi-Task Optimization and Transfer
In a canonical multi-task setting, there are K tasks, where each task, k, consists of a dataset, Sp = (k) (k)
{xx .34 Na drawn from some task distribution, Sy ~ Dz. 1 Given a network f with parameters
In this work we will focus the settings where assume that St is drawn LLd. from Dj and in which we are interested in generalizing over Dr- This the setting where transfer and MTL optimization most commonly studied.
2

---

Published in Transactions on Machine Learning Research (December/200) 22024)
θ (0. @k........................ P............................... 0xx) .k.....,x........... the following minimization problem:
where 0 are shared across all tasks and Φe are specific to task k, our goal is to solve
C - EuxCx(0,0s) E Cx (k) fx
min
(1)
REK
where & some (potentially task-specific) loss function and wi are task weights that are typically set wa to reflect a priori no preference on distinct task objectives. The hope of multi-task learning is that solving equation 1 will yield solution OuT that results in better generalization for each task than the solution found by solving each task individually (0zT. ØST). More formally, let (0, Φx) = B(yy) a(fo,ox (4), v) be a measure of generalization on unseen samples of the task distribution Dr for task k. given some metric a; our hope is that multi-task transfer is positive, which occurs when (f% (hrr) - (03T. ØsT) > 0.
A naive solution to optimizing Equation to leverage the uniform multi-task gradient (UMTG) for 8:
We = - 1 Σ us (o.C (O, - Σ
KEK
where D is the loss of task k over a randomly sampled batch B, and C is the scaling factor. 2 The UMTG is thought to frequently result in poor performance for multi-task learning, often yielding worse generalization than single-lask learning, phenomenon named negative transfer (Zhang el. al., 2023). Negative transfer is often attributed to gradient conflict in the parameters of 0, which is typically thought to arise in two different manners (Liu et al., 2021b; Javaloy & Valera, 2022): directional conflict occurs when the angle ween task gradients is high, preventing any single direction from locally optimizing all tasks jointly (e.g. Yu et al., 2020; Wang et al., 2021); 1); separately, magnitude conflict can arise when the magnitude of task gradients are disparate, resulting in the inder-optimization of certain tasks (e.g. Chen et al., 2018).
To improve the performance of multi-task learning, prior work has focused on developing specialized multi-task optimizers (SMTOs, Kurin et al., 2022)
Generalization Trajectory of Fashion MTL
90
Test Accuracy (%)
which aim to improve multi-task optimization by mitigating task conflict during training. Although
88
these methods hope to improve the generulization of multi-task models, they directly target ways to
86
improve minimization of Equation 1 by address- ing conflict between tasks during training (Navon et al., 2022; Liu et al., 2021a; Chen et al., 2020; Javaloy & Valera, 2022, inter a.ia.). However, in deep learning better training loss minimization does not always lead to better generalization (e.g. Smith et al., 2020) and indeed Kurin et al. (2022) and Xin et al. (2022) recently demonstrated that SM- TOs often do not improve multi-tast performance over the UMTG. In this work we empirically study how gradient conflict impacts task optimization, and whether this impact can explain the effect of multi- task learning on task generalization.
84
Setting
82
Single-Task Learning
MTL: RashiorMNIST (Positive Transtrr
MTL: Fashsyaashion (Negative Transfer)
80
10:
10-a
10-2
10-1
10-4
Total Training Loss
Figure 1: Fashionl ($2.2) training loss by generaliza- tion for the single-task setting (blue curve) and two multi-task settings (red and green curves). The im- pact of multi-task training on test accuracy (positive and negative) is detectable early into the training tra- jectory, at comparatively high training losses.
2.2 Experimental Setup
Following work on multi-task learning and gradient conflict, we consider MTL architectures which leverage the shared-encoder architecture, i.c. fo.e. = ha, o Go where go is an encoder model which maps inputs into
2C typically set to [K], yielding uniform average gradient, 1, yiclding a uniformly summed gradient. Unless otherwise noted, we set C = 1 in our experiments, following the best practice suggested by Mueller ct al. (2022).
Foor example, methods like PCGrad (Yu al., 2020; Wang et al., 2021) are motivated by aligning task gradient directions such that convergence all tasks faster and avoids local minima.
3

---

Published in Transactions on Machine Learning Research (December/2000 22024)
representation space shared across all tasks and has is a task-specific head that maps representations to task-specific predictions. We consider the following MTL settings:
FashionMTL: FashionMTT. is a synthetic MTL setting that we construct from the Fashion task (Xiao et al., 2017) in which we have a target-task, an ideal auxiliary task that yields positive transfer, and an uninformative auxiliary task that yields negative transfer. We split the original FashionMTTL task in two (with equal class balance in both halves) and treat each half as a separate task (Fashion1 and Fashion2), creating two tasks of size 25, 000 samples each. Additionally, we create third task, which we call "NoisyFashion, " in which we randomly permute the labels of the Fashion2 data. We expect the Fashion1 data to observe positive transfer when trained with the Fashion2 task, and negative transfer when trained with the NoisyFashion task.
The MNISTS multi-tase setting (Hsieh & Chen, 2018) consists of 3 MNIST-like tasks: MNIST (LeCun et al., 1998), a 10-class digit classification task; FashionMNIST (Xiao et al., 2017), 10-class clothing classification lask; and NotMNIST, 26-class English letter classification task. All tasks contain 50, 000 training samples and 10, 000 test samples and all imputs are 28 x 28 greyscale images.
MNISTS:
CIFAR-100: CIFAR-100 (Krizhevsky, 2012) is a hierarchical 100-class image classification dataset; these class hierarchies can be separated into 20 individual 5-class classification tasks, e.g. Household Electronics classification or Aquatic Mammals classification, cach consisting of around 2, 500 samples.
CelebA: CelebA (Liu et al., 2015) is an image classification dataset consisting of celebrity images; each of the 160, 000 images is labeled with 40 binary attributes, which each constitute a classification task.
Cityscapes: The Cityscapes (Cordts et al., 2016) dataset consists of 60, 000 images of urban streets and we follow the setup of Sener & Koltun (2018) and cast it as image segmentation problem with two tasks: per-pixel 7-class semantic segmentation and pixel-wise depth estimation.
GLUE: The GLUE dataset (Wang et al., 2018) is a benchmark of 8 NLP tasks. 7 tasks are classification tasks, ranging from Natural Langnage Inference to Grammatical Correctnoss, and one task is a regression lask (Semantic Similarity). The amount of data per lask can vary significantly.
For every training trajectory we study, we consider 3 random seeds after selecting hyper-parameters based the best validation performance out of an initial wyrer-parameter sweep. To maintain comparability of individual task trajectories, single-task and multi-task models within a single MTL setting are trained for the same number of steps, with the same optimizer and C' (the scaling factor) set equal to 1.4
3 What Does the Training Loss Trajectory Tell Us About Transfer?
Although many multi-task optimization methods operate on the theoretical assumption that achieving lower training loss will lead to improved peteraiiiationppeffrrraa on held-out data from the training distribution it is well established that, in deep learning, distinet trajectories can generalize very differently at identical training loss (Hochreiter & Schmidhuber, 1997; Jastrzebski et al., 2018; Huang et al., 2020). For instance, due to underspecification, different solutions may latch onto different features of the data, leading to overfitting or poor robustness despite achieving low training loss (D'Amour et al., 2022). Morcover, a significant amount of prior work has posited that many properties of the final solution of a training run, including generalization, are determined early into training (Leclere & Madry, 2020; Jastrzebski et al., 2020; Frankle et al., 2020a; Fort et al., 2020; Frankle et al., 2020b; Juneja et al., 2023). However, it is not known whether, in practical deep learning settings, the failures of multi-task training are due to how gradicnt con- flict affects convergence on task training loss (i.e. how the loss is optimized towards the end of training) or how gradient conflict alters properties of optimization throughout training.
3.1 Multi-Task Transfer Occurs Early Into Training
We begin by empirically demonstrating that the value of the training loss near convergence is incapable of explaining positive and negative transfer in the 5 MTL settings consider. For each setting in $2.2 we compare the generalization trajectories of single-task and multi-task training as they pass through regions of
4See Appendix B for more details around models, datasets, and hyperparameters.
4

---

Published in Transactions on Machine Learning Research (December/ 2iccmber/22024 22024)
(a) CIFAR-100
(b) Cityscapes
Fruits & Vegetables
Outdoor Scenes
Large Omnivores
Depth Estimation (4)
Test Accuracy: Ba
Test Accuracy
Test Accuracy:
Test L1 Loss
Traring
Single Task
10
Must-Task
Setting
Setting
Setting
060
Single Task
55
Single Task
12
Single Task
655
Multillask
Multi- las
Multi- lask
050
10t1
10-1
30-1
10-4
1014
10
10t1
13
10-4
107
15-1
10-1 33m
20
20
20-1
10-i/
Training Loss: Lx
Training Loss: Cx
Training Loss: Cx
Train Loss: Cx
(c) CelebA
(d) GLUE
Eyeg lasses
Gray Hair
RTE
STS-B
Test Accuracy:
8.8
Test Speammnn
68
Test
Setting
Setting Single
Setting Single
Setting Single Task
TingleTask
8.6
Mult-Taak
56
Multi-Task
Multi-Tak
MultiTask
10
45
2.
20-1 Training Loss: Cx
Training Loss:
Training Loss: Cx
Training Loss: Cx.
Generalization (Ex) versus Loss (Lx) curves for tasks which exhibit positive or negative multi- task transfer in 4 multi-task settings (for more tasks, see Appendix E). In general, multi-task trajectories converge to a higher training loss than single-task trajectorics, meaning gradient conflict stops optimization early. However, trunsfer (positive and negative) is exhibited as a generalization gap ween single-task and multi-task trajectories at comparably high training losses, i.c. transfer can be observed carly into training. In other words, multi-task transfer is a property of how gradient conflict impacts the early phase of learning, rather than a property of how well the task training loss is minimized. Therefore, negative transfer must be explained by higher order factors of the optimization trajectory than the training loss.
Figure 2:
similar training loss. More specifically, we evaluate the full training loss (Cx) and generalization (Ex) for every epoch and we plot generalization by training loss, allowing us to study how MTL generalization differs from STL generalization across comparable truining loss throughout training. In Figure we plot these tra- jectories for single-task and multi-task learming in the FashionMTL setting. We see that multi-task training with additional MashionMNIST data leads to positive transfer, whereas training with noisy Fashion data leads to negative transfer. However, in both cases, this transfer occurs as a generalization gap a comparable training loss-carly into training and is
difference in gencralization between two trajectories then maintained throughout the rest of optimization." 5 In this FashionMTL setting, this gap is observable in trajectories at a training loss over 3 orders of magnitude higher than the eventual training loss at convergence (~ 10-4). This result is surprising if we hold the assumption that negative transfer in MTL arises because cradient-conflict stops learning early; however, it matches the intuition put forth by prior work on single-task learning suggesting that certain properties of a training trajectory are determined very early into training (c.g. Leclere & Madry, 2020; Frankle et al., 2020b, etc.).
In Figure 2, we compare generalization versus loss curves for lasks 4 additional Multi-Task settings (CIFAR-100, Cityscapes, CelebA, & GLUE), focusing on tasks which either benefit or suffer significantly from MTL in terms of their final model accuracy. 6 Across all scttings, we that tasks which exhibit high amounts of positive or negalive transfer also incur generalization gaps between STL and MTL trajectories comparatively early into training, well before either trajectory converges. These generalization gaps are exhibited as significant differences (outside of 2 standard deviations, demonstrated by error bars) between multi-task and single-task trajectorics for loss values as carly as 10 or 10-1, in some cases over 6 orders of magnitude higher than the eventual training loss. Importantly, while prior work on multi-task learning has operated on the assumption that the key to improving multi-task performance is tackling challenges in
We adopt the term "generalization gap" from literature surrounding the larch-batch generalization gap (Keskar al., 2017), the phenomenon where large-batch models generalize worse than small-batch models across identical training losscs. For more complete task comparisons, see Appendix Nate that not all tasks see significant transfer (either positive or negative) from multi-task training; we are primarily interested in those tasks which do see significant changes to generalization.
5

---

Published in Transactions on Machine Learning Research (December/ 2oceeber/22044 22024)
Sharpness by Training Loss
Grad. Covariance by Training Loss
FIM Trace By Training Loss
1402
Sharpness: 0.001
0.4
1202
18
2.
Task Generalization
1002
6.5
30
3.0
0.2
600
Setting
Single- Leaming
600
1.0
MTL: RisT (Positive Transfer)
202
10-
10
Nosooaasiin (Negative Transteri
10
109
10-1
10-1
10
10
10
10'
10-1
10-1 10-2 10 10 109
15-1
10-
16-
10
Total Training Loss:
Total Training Loss: Cr
Total Training Loss: 4x
Figure 3: Factors of the optimization trajectory are unable to simultaneously explain negative and positive transfer. We plot the trajectories of factors of the loss surface (sharpness, gradient covariance, and Fisher information) for FashionMTL, corresponding to the generalization trajectories in Figure (similar plots for the other multi-task settings are shown in Appendix E). We expect to see the red trajectory, which yields negative transfer, exhibit worse optimization properties than the single-task trajectory (blue curve) and vice-versa for the green curve (positive transfer). Regardless of whether multi-task training resulted in negative or positive transfer, multi-task trajectories (green and red curves) exhibit better optimization properties (e.g. lower sharpness or carly-phase FIM "cxplosions") than single-task trajectorics (bluc curve).
minimizing the training loss, here we see instead that multi-task performance is driven by factors that are implicit to the zeroth order training loss (the value of Lx) and are determined early into training.
When multi-task training has a significant impact on task generalization, this impact arises as a gen- eralization gap between single-task and multi-task trajectorics early into training. In other words, transfer must be explained by factors of optimization that go beyond the training loss minimization.
4 Can Factors of the Optimization Trajectory Explain Transfer?
In $3 we find that tasks which experience positive or negative transfer from MTL exhibit transfer as a gencralization gap carly into training. Thus, any theory of (and subsoquent method to improve) the trade- off between lasks in multi-task problem must explain how MTL impacts factors of optimization which are implicit to the zeroth order training loss but connected to generalization. Prior work generalization in single-task models has proposed several factors of the training trajectory to explain generalization gaps between models (e.g., surface sharpness (Hochreiter & Schmidhuber, 1997)). Here we ask whether certain factors of the loss surface along the optimization trajectory trajectoy--specifieeeee specifically, sharpness, Fisher information, and gradient coherence are capable of explaining positive and negative transfer in multi-task settings.
The sharpness of the region around where solution lies has long been associated with its generalization in deep learning, both empirically and theoretically (Hochreiter & Schmidhuber, 1997: Keskar et al., 2017; Dziugaite & Roy, 2017; Huang et al., 2020). We follow Andriuscchenko et al. (2023) and adopt the elementwise- worst-case-fxx [B] 1Bl-sharpsess measure:
Loss Surface Sharpness:
Sharpness (0) = EB
max 1el
-0+
where p is upper bound the adaptive foe norm of the perturbation, [ is batch-size, and S& is the training datasct. Intuitively, this metric measures the maximum change in loss in a radius around the solution 0. We set [B] = 128 and 10-3, and we truncate Sş to be of size 2048.
To explain the large- generalization gap, Smith et al. (2021) leverage backwards analysis to derive an implicit bias term which biases optimization towards regions of the loss surface where the gradient covariance is low (i.e. where gradients are "coherent"; Chatterjee & Zielinski, 2022). More specifically, Smith et al. (2021) show that large learning rate and small-batch training may implicitly optimize
Gradient Coherence:
6

---

Published in Transactions on Machine Learning Research (December/20)) 22024)
Factors of Optimization
Dataset
Task
Λ& (Transfer)
Sharpness
A FIM Trace
A Coherence
Fruits And Vegetables
-8.81
0.17
8.11
789.95
Large Carnivores
-3.39
0.04
-212.92
-0.26
Outdoor Scenes
-5.11
0.13
-45.61
697.43
CIFAR-100
Medium Mammals
-2.89
0.05
30.69
10.85
Fish
3.49
0.05
19.33
10.79
Large Herbivores
3.92
0.05
49.69
11.06
Invertebrates
3.53
0.07
19.80
1.42
Bluryy
-2.73
0.31
2.20
90.39
Eyeglasses
-1.20
0.32
0.31
7.90
CelebA
Double Chin
5.30
0.33
0.51
33.78
Gray Hair
4.75
0.28
0.73
31.50
Semantic Segmentation
0.99 x 10-3
6.93
-5.32
-0.39
Cityscapes
Depth Estimation
-0.05
0.43
-6.10
2.73
STS-B
-0.01
189.82
-73.51
-1.54
MRPC
0.02
-0.07
8.63
2.55
10
GLUE
RTE
0.07
0.12
-0.10
1.27
10-5
SST-2
0.01
0.16
14.14
211.80
Table 1: Multi-task transfer (A&& = (0T) & (OsT)) for the tasks in each setting that experience significant impact to generalization, along with the change to factors detailed in $4. Shading indicates a canonically positive negative insignificant delta, where insignificance determined by overlap of 2 standard deviations. For a factor of optimization to potentially explain multi-task transfer, we must see positive Iransfer connected to a positive change, and negalive transfer connected to a negative change for that factor across an MTL setting. However, we instead see that, across many multi-task settings, multi- task learning results in insignificant, generalization. This shows that the trade-off in generalization between tasks in given MTL setting is not explained by a corresponding trade-off in any factor of optimization trajectories.
worse, change to factors of optimization regardless of its effect
the trace of the gradient covariance matrix:
Coherence (0) (Vol(fe(z)))) VoLx(0)) (x.g)ss*
(4)
Geiping et al. (2022) and Novack et al. (2023) reformulate this bias penalty the gradient norm of small batches, and show that explicitly optimizing small-batch gradient norms during large-batch training can recover the gemeralization gap between large- and small-batch models.
Early Phase Fisher Information: Finally, Jastrzebski et al. (2021) found that (in single-task learning) the trace of the Fisher Information Matrix (FIM)
FIM Trace (0)
E nES fo.
(5)
in the early stages of training is correlated with final solution generalization. They show that optimization trajectories which maintain a low FIM trace in the early phase of training yield much better generalization than trajectories whose FIM trace "explodes" at the boginning of training. This finding corroborated by Novack et al. (2023), who show that directly optimizing amall-batch FIM norm leads to better generalization for latge-batch training (similar to gradient coherence).

---

Published in Transactions on Machine Learning Research (December/ 2oceeber/22244 22024)
4.1 Factors of the Optimization Trajectory are Not Correlated with Trade-Offs in Generalization
In Figure 3 we plot each factor by the total training loss for FashionMNIST in the FashionMTL setting, where the color of each curve corresponds to the generalization trajectory shown in Figure 1. Because we observe that MTL with additional FashionMNIST data (green curve) leads to positive transfer while MTL with NoisyFashion data (red curve) leads to negative transfer, we expect to see that one of these factors is minimized worse by the red curve than the single-task curve (blue), and minimized better by the green curve. However, we instead find that both MTL trajectorios exhibit either better or comparable optimization of each factor when compared to single-task trajectories, despite significant. differences to generalization. In other words, the impact of MTL on these factors does not explain negative and positive transfer.
In Table 1, for the remaining MTL settings, compare change in generalization to change in optimization factor for each task that experiences significant transfer, computing an aggregale value of each attribute." The A of each term is computed using the average of all multi-task trajectories (averaged over random seeds) minus the average of all single-task trajectories and the significance of a A is determined by whether the confidence intervals (using 2 standard deviations) of the multi-task and single-task values overlap. Our hope is that one of the factors we study will explain the trade-offs in generalization between tasks, i.e. that positive transfer will correspond to a decrease in one or more factor while negative transfer corresponds to increase. Such result would suggest how multi-task learning is impacting generalization, and would potentially provide path towards developing optimization methods that are "right for the right reasons".
However, we find that no factor is capable of explaining negative and positive transfer in any MTL setting we study. For instance, in CIFAR-100, MTL consistently leads to sharper solutions for all tasks; however, sharpness cannot explain MTL transfer because even tasks which experience positive transfer find sharper solutions. Other factors not only experience deltas which are not significant, but which are also inconsistent. More generally, we see that multi-lask learning tends to result in worse values for the factors we study, regardless of whether task generalization is improved or harmed by multi-task training (with the exception of the PashionMTL setting, where MTL seems to improve some factors). This negative result indicates that our current understanding of generalization gaps in single-task models is not capable of explaining the generalization gaps between multi-task and single-task models. More importantly, while there is clearly a trade-off in generalization between tasks within some multi-task settings, we cannot explain why that trade- off occurs from the perspective of optimization. As a result, it is not clear how we should alter optimization to improve or balance transfer across tasks or, more specifically, what optimization challenges SMTOs should be overcoming to improve multi-task transfer.
We find that factors of the optimization trajectory previously shown to explain generalization gaps between single-task training runs are not capable of explaining the generalization gaps between single- task and multi-task models. In other words, our current understanding of how propertics of opti- mization dictale generalization in deep learning cannot explain how multi-task learning improves generalization for some tasks while harming others.
4.2
Can Factors of the Optimization Trajectory Explain the Impact of SMTOs?
Although we find, in $4.1, that differences between single-task and multi-task generalization are not explained through the lens of optimization, this does not imply that SMTOs have no role to play deep multi-task learning. Namely, SMTOs do not necessarily seek align multi-task trajectories with single-task trajectories, but rather aim to address optimization challenges within a fized multi-task problem; in other words, SMTOs aim to improve optimization relative to the uniform multi-task gradient (UMTG, Equation 2), rather than single-task learning. In this scction, we compare the training trajectorics of SMTOs to those of the UMTG
"To compute a single-value of each term for single training trajectory we average values across the training trajectory, using the following heuristios: generalization is computed as the average kest-performance of the top-10 validation checkpoints; sharpness is computed as the average sharpness of the last 20 checkpoints at the end of training; gradient covariance is computed as the average covariance of the last 20 checkpoints at the end of training; finally, the FIM Trace computed as the average of the max 20 values of FTM Trace value (capturing the "explosion"). Full trajectories are shown in Appendix E.
8

---

Published in Transactions on Machine Learning Research (December/ 2oceeber/22044 22224)
(a) MNISTS
Minimum Loss
Sharpness
Coherence
FIM Trace
0.5
0.5
0.5
%6 Generalization
0.0
-0.5
-0.5
-0.5
Gracient
-1.0
MGDA
-1.4
MMST
-1.0
-1.0
Greahiorm
8
MNIST
POGred
-2.0
-20
-2.0
-10"-10² -100 Lop
10 %6 Min Loss (+- better)
10"
10t
10²
-10t
-10"
10f
10t
10
%4 Sharpness ( better)
%A Coherence (+ better)
%4 FIM Trace (+ better)
(b) CIFAR-100
%6 Generalization (+)
Minimum Loss
Sharpness
Coherence
FIM Trace
+
4
Vegetables Electrical Devices
Gradhiom
PCGrad
Vericles (2)
-10 1n"
1et 18 10 10*
-10-106 -100 t0f -125 -10.9 -75 -5.0 -2.5
10t 1e 10 %4 Coherence better)
-10² -18F -1cf o 10t
1et 10
%6 Min Loss
better)
%4 Sharpness (+ better)
%4 FIM Trace (- better)
(c) Cityscapes
Generalization (+)
Minimum Loss
Sharpness
Coherence
FIM Trace
0.0
0.2
-.44
Gredient Method
-.4
0.4
0.4
-066
-0.6
PCGreed
Depth
-0.8
-08
-0.8
-0.
-107
-10F
10a
1oa
-20
-15
-10
-5
0
10
-101
-10P
100
10a
10²
0 10f
10'
102
102
%6 Min Loss (e- better)
%6 Sharpness (- better)
%6 Coherence (+- better)
%6 FIM Trace better)
Figure 4: The impact of SMTOs on generalization vs. their impact on optimization trajectories, as their %A over the UMTG trajectory. SMTOs aim to impact task generalization by affecting optimization, so we expect see positive (negative) changes to task generalization are corroborated by positive (negative) changes to at least one factor of optimization. In other words, for factor to explain how an SMTO impacts generalization, all of an SMTOs points should exist within the shaded quadrants of plot. However, there is no SMTO whose impacted tasks exist solcly in the shaded regions, suggesting that the mechanisms by which SMTOs improve or harm task performance are not explained by task optimization trajectories.
and ask whether, when SMTOs improve (or harm) lask performance compared to the UMTG, that their effect corroborated by an improvement (or harm) to aspects of task optimization.
We select 3 SMTOs and compare their training trajectories to the trajectorics of the UMTG: MGDA (Sener & Koltun, 2018). PCGrad (Yu et. al., 2020), and GradNorm (Chen et al., 2018). We focus on 4 factors of optimization trajectories: in addition to sharpness, early-stage FIM, and gradient coherence (as in $4.1), we also compare the minimum training loss achieved by each trajectory, a factor that is classically used to mo- tivate many SMTOs. For each SMTO and factor, we measure the percentage change (%A) over the UMTG for each task that experiences a significant. impact to generalization; we calculate the percentage change keep all factors on the same scale across tasks, which may otherwise exist on different orders of magnitude.
8 Each of these methods is representative of class of SMTOs: MGDA, similar to CAGrad (Liu al., 2021a) and Mus (Navon al., 2022), is motivated by Pareto-Optimality and alters the gradient of each step such that conver- gence Pareto-Stationary point is guarantced; PCGrad, similar to GradientVacciue (Wang et al., 2021) and IMTL-G(Lin et al,, 2021b) directly alters the gradient direction of cach step such that the resulting step is sufficiently aligued with all task gradients; finally, GradNorm, similar to IMTL.-I (Liu al., 2021b) and RotoGrad-Seat (Javaloy Valera, 2022) attempts scale the at which tasks are learned, such that all tasks minimized an equal
9

---

Published in Transactions on Machine Learning Research (December/2000 22024)
In each plot, we shade the two quadrants which correspond to either simultancous improvement to opti- mization and generalization (yz or a simultaneous degradation to both optimization and generalization (+z,-y) SMTOs aim to influence generalization by improving or balancing properties of optimization, so we expect that, for cach SMTO, there is some factor of optimization for which all points (all tasks that the SMTO positively or negatively impacts) exists within the shaded regions, implying a connectiion ween that factor and the mechanisms by which the SMTO improves or trades-off task generalization.
In Figure 4 we plot these comparisons for the FashionMTL, CIFAR-100, and Cityscapes settings." 9 We find that factor has points which exist solely within the shaded quadrants, i.e. it is not clear what aspects of optimization are actually effected by SMTOs to impact multi-task performance. Of particular note is the minimum training loss, which is of primary concern to many multi-task optimizers; not only do SMTOs often lead to higher minimum training loss than the UMTG, but tasks whose minimum training loss is improved by SMTOs do not necessarily generalize better. For example, PCGrad in CIFAR-100 results in worse minimum training loss, for the tasks it impacts, and better sharpness and FIM Trace than the UMTG regardless of whether lask performance is improved or harmed by PCGrad oplimizalion. Kurin et al. (2022) empirically and theoretically demonstrate that many SMTOs induce early-stopping behavior and suggest that this may explain why SMTOs sometimes outperform the UMTG; here, we do corroborate the notion that SMTOs can result in an early-stopping for several tasks. However, we also show that the early-stopping behavior of SMTOs has little correlation to task improvement, i.e. it does not explain the benefits of SMTOs.
When SMTOs do have an impact on generalization (either positive or negative), that impact is not explained by corresponding improvement (or harm) to a task's optimization trajectory. Thus, it is not clear what aspects of optimization SMTOs are impacting to have an effect on multi-task transfer.
5 Does Gradient Conflict Explain Impact to Optimization or Generalization?
So far, we have focused our analysis on the trade-offs between different tasks within a fixed multi-task problem. While we often observe a trade-off in generalization between tasks in an MTL problem, we show that it is not clear from an optimization perspective why that trade-off occurs. However, despite being unable to explain the impacts to generalization, multi-task learning and consequently, gradient conflict- has a significant impact on the optimization trajectories of all tasks. One potential explanation of our result is that multi-task learning pulls optimization into regions of the loss surface that single-task trajectorics do not explore, such that comparisons between single-tast and multi-task trajectories uninformative.
In this section, we ask whether the amount of gradient conflict between a given task and the multi-task gradient drives the impact of MTL on the lask's optimization trajectory, and whether this impact is correlated with task generalization. Intuitively, the higher the gradient conflict-iee, -iee the lower the similarity between the target task gradient and the multi-task gradient-the higher the impact of MTL on optimization may be, as the optimization path diverges more severely from the single-task gradient. We focus our analysis on a few target-tasks from CIFAR-100, which has a large number of auxiliary tasks to choose from; we randomly select sets of auxiliary tasks for 2, 5, 10, and 20-task settings (covering a range of multi-task sizes) and train models with each target-task and auxiliary task set. Changing the set of auxiliary tasks impacts both the amount of gradient conflict that our target-task experiences during training, as well the ultimate generalization of the model on target-task. To measure gradient conflict at each epoch of training, we measure the cosine similarity between the target-task gradient and the UMTG gradient:
Gradient Similarity (k, K, 0) -cosine-siml V VOLk VTT (e, (TT B))
(6)
where B is the batch-size used during training, VM is the multi-task gradient (Equation 2), and 0 is all parameters of the shared encoder concatenated into single vector. To smooth out the impact of noise due to small batch-sizes, we compute the mean gradient similarity over 200 randomly sampled batches. If an
9 As in Xin ct al. (2022) and Kurin et al. (2022), we find that most tasks do not experience a significant shift generalization from SMTOs over the UMTG. CIFAR-100, we show only tasks that experience significant change. For Cityscapes, we show both tasks Finally, for MNSSi we show the 3 tasks that have the largest impact.
10

---

Published in Transactions on Machine Learning Research (December/ 2occmber/2202/ 22024)
auxiliary task gradient has high amount of conflict with the target-task, the UMTG will be pulled away from the target-task gradient. Thus, the similarity between the target-task gradient and the UMTG captures the average amount of conflict between the target-task and all auxiliary tasks.
5.1 Conflict Has a Predictable (Negative) Impact to Optimization Trajectories
We begin by asking whether the similarity of the larget-lask gradient to the MTL gradient (Equation 6) is indicative of how multi-task training will impact the optimization trajectory. While it is clear that high gradient conflict will have a negative impact on the minimization of the training loss (Nocodal & Wright, 2006; Yu et al., 2020), we have seen in $3 that the multi-task transfer cannot. be solely explained by the impact of gradient conflict on the training loss. Thus, we ask here whether a high amount of conflict between the target-task and the multi-task gradients has negative impact on the factors of optimization trajectories that we study in $4. For each target-task, we compare these factors (carly-stage FIM, sharpness, and gradient coherence) with the average gradient similarity between the target-lask gradient and the UMTG throughout training, and plot the results in Figure 5a.
Overall we find that there is a positive correlation between the amount of gradient conflict target-task experiences, and the value of the factors we study; the similarity between the target-task and multi-task gradient decreases, sharpness, carly-stage Fisher information, and gradient covariance all increase corre- spondingly during training, While prior work has shown that high conflict can create problems for zeroth order training loss minimization, e.g. poor local minima (Yu et al., 2020) or failure to optimize all tasks si- multancously (Chen et al., 2018), here we show that high gradient conflict negatively impacts more than just the training loss trajectory, including factors which have been previously tied to generalization. However, while it is clear that gradient. conflict has a predictable impact on optimization, it not clear whether this is a problem from the perspective of transfer; in other words, it is not clear whether the impact to target-task optimization is related to the target-task's generalization.
High gradient conflict negatively impacts task optimization in more ways than just training loss minimization. We find that the impact of MTL on task optimization-inclddiin factors such as the sharpness, Fisher information, and gradient coherence-is correlated with the amount of gradient conflict that the task experiences during the early stages of training.
5.2 Conflict Does Not Have a Predictable Effect on Generalization
While some amount of conflict between tasks is intuitively necessary for MTL to improve generalization over single-task learning (Du et al., 2020), the intuition behind many SMTOs is that a high amount of gradient conflict can negatively impact task optimization and generalization (Yu el al., 2020; Chen et al., 2020; Liu et al., 2021a, inter alia). Indeed, in $5.1 we see that higher amounts of gradient conflict can have a negative impact training trajectorics, worsening several factors associated with generalization gaps. However, we have shown in $3, MTL having a negative impact on training loss optimization does not necessarily imply negative impact on task generalization. To that end, we ask whether the amount of conflict that a target-task experiences during training is predictive of its generalization after training.
We plot target-tast task test accuracy by gradient similarity for our CIFAR-100 target-tasks in Figure 5b. Unlike factors of the optimization trajectory, we see that the gradient similarity between the target-task and the multi-task gradient has very little correlation with target-task generalization. In fact, for the target-task Fish, the auxiliary setting with the lowest gradient similarity to the target-task (ie. for which optimization consistently moves further away from the steepest single-task direction of descent) yields the strongest generalization of all auxiliary scttings, despite leading to higher FIM explosion, higher gradient covariance, and higher sharpness than the other settings. Based on the results of $5.1, intuition would suggest that we should mitigate gradient conflict (i.e. maximize the similarity of our gradient step with the gradient of our target-task) to improve training loss minimization, sharpness, gradient coherence, and minimize FIM explosion. However, our results in Figure 5b demonstrate that while mitigating conflict can improve many aspects of task oplimization, it will not necessarily improve task generalization.
11

---

Published in Transactions on Machine Learning Research (December/ 2occeberr22024 22024)
(a) Auxiliary Task Gradient Conflict vs. Impact to Optimization
Target Task: Flowers
Target Task: Vehicles 2
Target Task: Fish
1,660
0.594, p = 0.001
1.67
0.832, p = 0.000
1.660
0.732, p 0.000
Sharpness
1.655
1.66
1.655
83
1.650
1.645
Num Aux Thakks
1.65
Num
1.645
1.640
1.640
1.64
1.635
19
19
1.635
19
1.63
1.630
0.4
0.5
0.6
0.7
0.8
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.2
0.3
0.4
0.5
0.6
0.7
Gradient Coherence
r= 0.774, p 0.000
1.50
0.635, p 0.000
0.744, p 0.000
125
12.5
1.25
10.0
1.00
0.75
7.5
50
8
0.50
25
0.25
2.5
60 O
0.00
0.0
-25
0.7
0.2
0.6
0.2
0.5
22.5
40
r= - 0.750, p = 0.000
25
r= 0.725, p 0.000
r= 0.695, p 0.000
33
FIM Trace
20
17.5
15.0
g
15
20
10.0
10
83
7.5
5.0
0.4
0.5
0.6
0.7
0.8
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.2
0.3
0.4
0.5
0.6
0.7
Gradient Similarity
Gradient Similarity
Gradient Similarity
(b) Auxiliary Task Gradient Conflict vs. Impact to Generalization
Task: Flowers
Task: Vehicles 2 0.210, 0.265
Task: Fish
68
0.181,
0.339
87
0.761,
0.000
87
Test Accuracy
86
85
X
84
Num Aux Tasks
8
83
69
85
81
0.4
0.5
0.6
0.7
0.8
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.2
0.3 0.4 0.5
0.6 0,7
Gradient Similarity
Gradient Similarity
Gradient Similarity
Figure 5: The impact of gradient conflict on factors of the target-task optimization (a) and generalization (b) across auxiliary task settings. Pearson-r correlation coefficient and p-value are shown at the top. Gradient similarity is negatively correlated with each of the optimization factors that we study, implying that high gradient conflict negatively impacts many factors of task optinization; however, gradient conflict is not negatively correlated with target-task generalization. In other words, while gradient conflict has a consistent, negative impact to optimization, this effect does not predict or explain transfer.
Although early-stage gradient conflict has a predictable impact on target-task optimization ($5.1), this correlation does not hold for generalization. It is therefore not clear why mitigating gradient conflict- for instance, via specialized optimizers that manipulate the task gradients-shiee should improve multi-task transfer.
12

---

Published in Transactions on Machine Learning Research (December/ 2occeberr22024
6 Conclusion
We empirically show, across a number of multi-task settings, that both positive and negative transfer are determined early inlo Iraining, implying that negative transfer is a result of gradient conflict impacting factors of optimization that go beyond the zeroth order training loss. To understand how gradient conflict causes negative transfer for some tasks while benefiting others, we study factors of optimization that have previously been shown to explain generalization gaps in single-task learning. Our goal is to show how trade- ofls in transfer can be explained as trade-offs in lask optimization. However, we find that no factors we st study can adcquately explain the impact of multi-task learning on task generalization, i.c. it is not clear how the impact of task conflict on optimization is related to negative (and positive) transfer. Moreover, we find that when SMTOs impact which tasks benefit and which are harmed within a multi-task setting, this impact is not explained by a corresponding improvement or degradation in the tasks' optimization, including how well the task losses are minimized. Finally, we study how the amount of gradient conflict impacts transfer and we find that high amounts of gradient conflict are highly correlated with h, while having almost no correlation with task generalization.
Overall, we show that predicting which tasks will experience positive transfer and which will experience neg- alive transfer, through the lens of optimization alone, is currently not leasible. This result not only explains why current SMTOs motivated by gradient conflict and training loss minimization fail to generally help in practice, but also rules out several additional factors of training trajectorics that might, in theory, be used to predict transfer. Our results also have implications for the development of future SMTOs, such as narrowing the focus of future SMTOs to the early phases of training where the generalization gap occurs.
Future Directions Our work raises several open questions around transfer in deep learning: Can the benefits and failures of MTL be explained through the lens of optimization? Can multi-task transfer be understood through single mechanism, many SMTOs claim, or does it vary across model and setting? And, can multi-task transfer within given model be improved, over the UMTG, by addressing specific aspects of optimization? Answering these questions is crucial towards not only improving multi-task models in practice, but also towards understanding how models generalize in deep learning in general. Our work also suggests that, without answers to the above questions, the most principled approach to addressing failures in multi-task learning may not be to focus on optimization in deep learning, but to instead focus on properties of the tasks themselves, c.g. meta-learning transfer from task distributions in Song et al. (2022).
Limitations As in any empirical analysis such ours, our takeaways are limited by the number of settings and the breadth of models that we consider. While we mitigate this by considering 5 unique MTL settings and performing extensive hyperparameter searches, there remains the possibility that our results do not generalize to other tasks or domains. Additionally, our takeaways are naturally limited by the factors of optimization that we study, and their capacity to explain generalization deep learning. While we can rule out the potential for certain aspects of optimization to explain transfer, it remains an open question as whether not some aspect of optimization can predict. the benefits and failures of MTL in deep learning.
Acknowledgements
We would like to sincerely thank Steven Reich for his contributions to, and helpful discussions of, an earlier version of this work. We would also like to thank Michael Crawshaw, Neha Verma, Desh Raj, and Suzanna Sia for their feedback on recent drafts of this paper, well as Sophia Hager, Rachel Wicks, Aleem Khan, Rafael Rivera-Soto, Andrew Wang, Ashi Garg, and Cristina Aggazzotti. Finally, we would like to thank the anonymous reviewers who's helpful feedback improved our work. This work was supported, in part, by the Human Language Technology Center of Excellence at Johns Hopkins University.
13

---

Published in Transactions on Machine Learning Research (December/ 2occeberr22024 22024)
References
Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flammarion. A modern look at the relationship between sharpness and generalization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 840 902. PMLR, 23 29 Jul 2023. URL https: /ppor/////occeeneseee mlr. p.e..press//2//// andriushchenko23a. html..
Rich Caruana. Multitask learning: a knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on International Conference on Machine Learning, ICML'93, pp. 41 41-48, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc. ISBN 1558603077.
Satrajit Chatterjee and Piotr Zielinski. On the generalization mystery in deep learning, 2022. URL https: //arxiv. .twg wwrggabs/2203... 20002200...0......
Slang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation, 2017. CRL https://arxiv http://axxxvv..r/bbbea/a/ www./bs/////////oooo /ttp//176..05558
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normal- ization for adaptive loss balancing in deep multitask networks. volume 80 of Proceedings of Machine Learning Research, pp. 794 803, Stockholmmsmässan, Stockholm Sweden, 10 15 Jul 2018. PMLR. URL http://prroceedings mlr, press/ //0//////////// chen18a html.
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep mullitask models with gradient sign dropout. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2050. Curran Associates, Inc., 2020. URL http:////.../.0.0.0.00/aaeeeeee.
.df.
Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations, 2021. URL //pppr/////eeeeiee.../...
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiclc. The cityscapes dataset for semantic urban scene understand- ing. In Proceedings of the IEEE conference on compuler vision and patlern recognition, pp. 3213 3223,
2016.
Michael Crawshaw. Multi-task learning with deep neural networks: A survey, 2020. URL hat///bb//00///
Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eiscnstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingan,, Mario Lucie, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(26)):166 61, 2022. URL http:// jmlr. org/ wwwp///grg///oeee/o/// papprss/23/20-1333 1tm..
Jacob Devlin, Ming- Wen Chang, Kenton Lee, and Kristina Toutanova. BERT: Pro-training of deep bidirec- tional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Min- ncapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10.18653 1088553vv1/N99-.. 1423. URL https: https://aclanne http://cllanttoooooorr////////./
14

---

Published in Transactions on Machine Learning Research (December/ 2occeberr/202/ 22024)
Yunshu Du, Wojeiech M. Czarnecki, Siddhant M. Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Takshminarayanan. Adapting auxiliary losses using gradient similarity, 2020. URL https: //rxiv.. prg/abs/1814 02224..
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data, 2017.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (ods.), Advances in Newral Information Processing Systems, volume 33, pp. 5850-5861. Cur- ran Associates, Inc. 2020. URL https: https://prooeeedings. wwwpeeedddinnnnnnnii............ www.nees.ccpppppeee.eee wwtpc/ppeee.leeeppeeeeeeeeeeeeee filles/paper/2220//iil 46601699106644581777777888888838 par.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3259 3269. PMLR, 13 18 Jul 2020a. URL https: https://procceeings .oigg.s......... mwr. pttppr/evv1///rallleeee. html.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. Tn International Conference on Learning Representations, 2020b. URL 1d=Hk111RNF%
Jonas Geiping, Micah Goldblum, Phil Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. In International Conference on Learning Representations, 2022. URL https::/openreevie net//oruum1dd=2BBse11a
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Comput., 9(1):1-42, Jamuary 1997. ISSN 0899-7667. doi:
URL https://doi (ealddi..rgg//// org//0.1162/nee. 199..9..99... 9.1.1.
Po-Chen Hsich and Chia-Ping Chen. Multi-task learning on MNIST image datasots, 2018. URL https:
W. Ronny Huang, Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K. Terry, Furong Huang, and Tom Goldstein. Understanding generalization through visualizations. In Jessica Zosa Forde, Francisco Ruiz, Melanie F. Pradier, and Aaron Schein (eds.), Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops, volume 137 of Proceedings of Machine Learning Research, pp. 87-97. PMLR, 12 Dec 2020. URL ptatts::/proceeeee .ml...ng..m.......mmmmmmmmm....m www..l.ee./v........ ppwps/vve3 prees/v137/huang220 http//uaan00a... html.
Stanislaw Jastrzebski, Maciej Saymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho and Krzysztof Geras* The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2020. URL https: wwweeerrvieeneet//eeeee net/forum?id= r1g87C4KWB.
Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand, Giancarlo B Kerg, Huan Wang, Caiming Xiong, Richard Socher, Kyunghyun Cho, and Krzysztof J Geras. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4772 4784. PMLR, 18 18-24 Jul 2021. URL https: http:////occeegn...//// mlr. www.:mmww...mm...mmmmmmmmmm.. /ptps///////srrresttt222 jastzebsski2aa. html.
Stanislaw Jastrzebski, Zac Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Amos Storkey, and Yoshua Bengio. Three factors influencing minima in SGD, 2018. URL rJma2bZCW.
Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In International Conference Learning Representations, 2022.
15

---

Published in Transactions on Machine Learning Research (December/2000 22024)
Junguang Jiang, Baixu Chen, Junwei Pan, Ximei Wang, Dapeng Liu, jie jiang, and Mingsheng Long. Fork- merge: Mitigating negative transfer in auxiliary-task learning. In Thirly-seventh Conference on Neural mett/foruniddvv2Hk1000 Information Processing Systems, 2023. URL https://openreviee
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectivity reveals generalization strategies. In The Eleventh International Conference on Learning Representations, 2023. URL https:://ppereetteee// http:eerrevieeeeeeeeeeeeeeeeeeee htt.://orummidnhYM0000 JuL
Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton. On the maximum hessian eigenvalue and gen- eralization. In Javier Antorán, Arno Blaas, Fan Feng, Sahra Ghalebikesabi, Ian Mason, Melanie F. Pradier, David Rohde, Francisco J. R. Ruiz, and Aaron Schein (eds.), Proceedings on 'I Can't Believe It's Not Better! Understanding Deep Learning Through Empirical Falsification' at NewrIPS 2022 Work- shops, volume 187 of Proceedings of Machine Learning Research, pp. 51-65. PMLR, 03 Dec 2023. URL fttps://procceeingg wwwpeediiinnm........mmmmmmmm..e wwt.smm.....ssseeee wwwpppresvvvv.. /aaur23a.. html.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Con- ference on Learning Representations, 2017. URL https://openreview www.:nee/frruuee toyooorr
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Vitaly Kurin, Alessandro De Palma, Πya. Kostrikov, Shimon Whiteson, and M. Pawan Kumar. In defense of the unitary scalarization for deep multi-task learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2022. URL https: / //openreview
///pparrrrioo.nneffooroo/ooooooe net//orrmmid
Guillaume Leclere and Aleksander Madry. The two regimes of deep network training, 2020. URL https:
//arxiv. .rg/ /ttp/abb/202..0///7/
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradicnt-based learning applicd to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Dongyue Li, Huy Nguyen, and Hongyang Ryan Zhang. Identification of negative transfers in multitask learning using surrogate models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:// net//oorm//eeeg/AASSSS
Featured Cortification.
Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling SGD with stochastic differential equations (SDFs). In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021. URL https://openrevier www.:nee//eeeee...
__nVQI.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 18890. Cur- ran Associates, Inc., 2021a. URL https: ptttps:/pproceee http:////occec...../// www.igg..eriis..cooooooo wwtpp/ppeeeleeeeppeeeeeeeeeeeeee http///////a/e////////eee pdf.
Liyang Liu, Yi Li, Zhanghui Kuang, Jing- Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations, 2021b. URL https:// //tp::////ppereiiee net/forum?? In-
Shikun Liu, Stephen James, Andrew Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https:
//openreview. wwteenrvveeenntteeeeeeee hetf/foruum/dd-tttcCnssss
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. ]erbert]a: A robustly optimized (bert) pretraining approach, 2020.
URL https:: https://openreviier aett/fforummiieeSyyS7744..
16

---

Published in Transactions on Machine Learning Research (December/2000 22024)
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training, 2018. URL
06162.
David Mueller, Mark Dredze, and Nicholas Andrews. The importance of temperature in multi-task op- timization. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022. URL
https://openreviert http://ppeareeeeee...eeetrrtrddd
Maximilian Mueller, Tiflany Joyce Vlaar, David Rolnick, and Matthias Hein. Normalization layers are all that sharpness-awan aware minimization needs. In seerthh seventh Conference on Neural Information Processing Systems, 2023. URL
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning
bargaining game. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (cds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16428 16446. PMLR, 17-23 Jul 2022. URL https::/sse httpss////ocecenn...mom www.mmrr..eeeeeee v162/navon22aa hmmmm.
Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer, New York, NY, USA, 2c edition, 2006.
Zachary Novack, Simran Kaur, Tanya Marwah, Saurabh Garg, and Zachary Chase Lipton. Disentangling the mechanisms behind implicit regularization in SGD. In International Conference on Learning Repre- senlations, 2023. URL http:p/s//pgeeerrreeeeeee////eap
Amelie Royer, Tijmen Blankevoort, and Babak Ehteshami Bejnordi. Scalarization for multi-task and multi- domain learning at scale. In T'irtt--eventhh Conference on Neural Information Processing Systems, 2023. URL http://pppaerreeeeeeeeeooaaaaaaa
Sebastian Ruder. An overview of multi-task learning in deep neural networks, 2017. URL httth::t/////tt/tt prg/abs/1706.055 05098.
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Biancha and R. Garnett (eds.), Advances in Neurol Information Processing Systems 31, pp. 527 538. Curran Associates, Inc., 2018. URL http: http:://papprsamnee cc/ /appr//// pdf.
7334-multi- task- learning- as mess-muliime wwwpctiie-oppiieeeee......
Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9058 9067. PMLR, 13 18 Jul 2020. URL https: httpss://proeeee //tp://////ce/nss... alr.press/ v119/s/mm //mmi th20a. html.
Samuel L. Smith, Pieter-Jan Kindermans, and Quoe V. Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018. URL https: //ppr//v/ee........mmmmmme net/
B1yyB1Y..
Samuel T, Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In International Conference on Learning Representations, 2021. URL
https://openreviev.. http:p///pppoooooooo.eeeeettrrrr
Xiaozhuang Song, Shun Zheng, Wei Cao, James Yu, and Jiang Bian. Efficient and effective multi-task grouping via meta learning on task combinations. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: t //openreview. net/foormmidd-Re----r
17

---

Published in Transactions on Machine Learning Research (December/ 2occebeer2202/ 22024)
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In Hal Daumé IIT and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9120-9132. PMLR, 13-18 Jul 2020. URL https: htps://pproee http:///rccceenn..///// mlr. html.
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Mickk Subhransu Majī, and Mohit. lyyer. Exploring and predicting transferability across NLP tasks. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7882-7926, Online, November 2020. Association for Computational Linguistics. doi: 10. 10/553/v1/2200emmm// mamolpppmain.35... URL https: batps://acee //tp:aaallannoooyyy../oooo org/2020. emnle-meeeeee..ee main.m3in 635.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackborNL.P: Analyzing and Interpreting Neural Networks for NLP, pp. 353- 355, Brussels, Belgium,, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ W18-5446. URL https://aclanthology org/w18- org//11-5446.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural machine translation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Länguistics, pp. 8526 8537, Online, July 2020. Association for Computational Linguistics. doi: 10. (0.18653/v1/2022 acl-mmm..m..nnn.......mm..m..aa. 754. URL https:// aclanthology. org/2020/ acl -main. 754.
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improv- ing multi-task optimization in massively multilingual models. In International Conference on Learning Representations, 2021. URL attps://openrewee www.rrvveee.ne/foouuii...... mett/forum?dd=FLVEji------
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: a novel image datasct for benchmarking machine learning algorithms, 2017. URL attps://arxiv/ www.aawwwo./b////////oo 07747..
Derrick Xin, Behrooz Ghorbani, Justin Gilmer, Aukush Garg, and Orhan Firat. Do current multi-task optimization methods in deep learning even help? In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: / //openreview www.:/eentonnneee..e d1
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gra- dient surgery for multi-task learning. In Н. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 5824-5836. Cur- ran Associates, Inc., 2020. URL stttps//procceeingg www.:neweu.c/neneeeeeeee httpe//apppppeeeeeeppeeeeeeeeeee wwt.ss/papee//22///tt/.. pdf.
Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu. A survey on negative transfer. IEFE/CAA Journal of Automatica Sinica, 10(2):305 329, 2023. doi: 10.110//ASS.0002000000.
A Discussion of Limitations, Broader Impact, and Future Work
Future Directions Our work raises several open questions around transfer in deep learning: Can the benefits and failures of MTL be explained through the lens of optimization? Can multi-task transfer be understood through single mechanism, many SMTOs claim, or does it vary across model and setting? And, can multi-task transfer within given model be improved, over the UMTG, by addressing specific aspects of optimization? Answering these questions is crucial towards not only improving multi-task models in practice, but also towards understanding how models generalize in deep learning in general. Our work also suggests that, without answers to the above questions, the most principled approach to addressing failures in multi- learning may not be to focus on optimization in deep learning, but to instead focus on properties of the tasks themselves, e.g. meta-learning transfer from task distributions in Song et al. (2022).
18

---

Published in Transactions on Machine Learning Research (December/ 2occeberr22024 22024)
Broader Impact The goal of this paper is to deepen our understanding of how neural networks learn to generalize when learning jointly from diverse signals. Understanding the mechanisms by which multi-task learning impacts generalization has both practical implications (for a broad array of real-word settings that use multi-task training) as well as theoretical implications (by uncovering, corroborating, or contradicting explanations of how generalization is connected to optimization in deep learning). Furthering this under- standing has important socictal implications, such as understanding what types of distributions are harmful for models or lead to certain behaviors, or allowing the construction of more interpretable ML systems.
Limitations As in any empirical analysis such ours, our takeaways are limited by the number of settings and the breadth of models that we consider. While we mitigate this by considering 5 unique MTL sottings and performing extensive hyperparameter searches, there remains the possibility that our results do not generalize to other tasks or domains. Additionally, our takeaways are naturally limited by the factors of optimization that we study, and their capacity to explain generalization deop learning. While we can rule out the potential for certain aspects of optimization to explain transfer, it remains an open question as to whether not some aspect of optimization can predict the benefits and failures of MTT. in deep learning.
B Dataset & Model Details
FashionMTL: FashionMTL is a synthetic setting where consider the impact of training a task jointly with (a) an ideal task where we see positive transfer versus (b) an uninformative task which yields negative transfer. Our (arget-task in this setting is the FashionMNIST task (Xiao et al., 2017), which is a 10-class image classification task where the goal is to classify images of clothing articles. We randomly split: the FashionMNIST dataset into two splits of 25, 000 samples cach, and consider the first split to be our target-task training dataset. The second half of the dataset is used for the auxiliary task training: for setting (a)-the the ideal auxiliary task we the remaining 25, 000 FashionMNTST samples as the auxiliary task with no perturbations. In this setting, our auxiliary task is additional data from the exact same distribution of our target-task training data. For setting (b), we use the same remaining 25, 000 FashionMNIST samples, but we randomize the labels. In setting (b) our auxiliary task is now uninformative, as it amounts to fitting noise, and conflicts with the objective of our target-task. For each task we have 5, 000 validation and test samples. Our architecture is a simple LeNet CNN architecture whose penultimate representation is passed to task-specific linear classification layers.
MNISTS: MNISTS (Hsieh & Chen, 2018) is 3-task multi-task setup involving 3 MNIST-like dalasets (all 28 x 28 greyscale images). The 3 datasets are MNIST (LeCun et al., 1998), a 10-class handwritten digit rocognition task; FaShionMNIST (Xiao et al., 2017), a 10-class clothing article classification task; and NoIMNINST, a 26-class English character classification task. Each task has exactly 50, 000 training samples and 5, 000 validation and test samples. Our architecture for this setting is a simple LeNet CNN architecture whose penultimate representation is passed to task-specific linear classification layers.
CIFAR-100: CIFAR-100 (Krizhevsky, 2012) is hierarchical 100-class classification dataset; these class hicrarchies can be scparated into 20 individual 5-class classification tasks, c.g. Houschold Electronics classification or Aquatic Mammals classification. Each task consists of 5, 000 training samples (roughly 1,000 samples per-class), and 500 validation and test samples. We use a ResNet 8 architecture whose penultimate representation is passed to task-specific linear classification layers.
CclebA: ColebA (Liu et al., 2015) is a 40-way binary attribute prediction task. It consists of images of celebrity faces, each of which is labeled with 40 binary attributes. We treat these attributes as separate, binary classification tasks, yeilding 40-task problem. The dataset is large, Each task consists of 162, 700 training samples with 19, 867 validation and 19, 962 test samples. We use 18 architecture whose penultimate representation is passed to task-specific linear classification layers.
Cityscapes: The Cityscapes dataset consists of images of urban streets and is cast as a two-task multi- lask setting: 7-class semantic segmentation and depth estimation. The datasel consists of 2, 975 train images, 500 validation images, and 1, 525 test images; we follow the same pre-processing steps used in Sener & Koltun (2018). To model these tasks we use the DecpLabV3 architecture (Chen et al., 2017), which consists of a ResNet101 backbone, pre-trained on ImageNet, and task-specific Atrous Spatial Pyramid Pooling modules.
19

---

Published in Transactions on Machine Learning Research (December/ 2ocember/20224 2222)
GLUE: Finally, consider the GLUE dataset (Wang et al., 2018), benchmark of 8 NLP tasks. 7 tasks are classification tasks, ranging from Natural Language Inference to Grammatical Correctness, and one task is a regression task (Semantic Similarity). We use a pre-trained RoBERTa-Beas (Liu et al., 2020) backbone, with lincar task heads on top of the penultimate representation of the [CLS] token, as is standard for BERT line-tuning (Devlin et al., 2019).
For all settings, we conduct 1001,5001 10-2, 20-2, 10-3, 50-3, 10-1,0004,, 11105 and batch size: {4, 16, 32, 32, 64, 128, 256)- We use the Adam optimizer over all settings, we found it to yield the best generalization in our settings out of { Adam, SGD, SGD with Momentum). In all settings we use constant learning rate (no decay).
hyperparameter sweep over the learning rate:
C Calculating Trajectory Sharpness, Fisher Information, and Gradient Coherence
C.1 Sharpness
We calculate loss surface sharpness using the worst-case Lo sharpness measure over a batch-size, [B], of 128, following the details and implementation of Andriushchenko et al. (2023). At any given position, 0, in the loss surface, the equation for worst-case fo sharpness
L Sharpness (0) = Ba-s 1 <p
- -) (0)
(7)
where [B] is the batch-size that the loss is calculated over, Ss is the size of the dataset (which we truncate al. (2023)) and p is a hyperparameter which dictates the
be of size 2048, following Andriushchenko maximum size of the perturbations over which the worst-case loss is calculated. We experiment with several different values of p {10-3 3.5 3-, 10 1,-22100111 10-"), but we find that this parameter does not change the relative ordering between our different trajectories. Tn general, we default to using p 10-3
C.2 Fisher Information
The trace of the Fisher information matrix of given neural network is given as the average l norm of the gradient of 8 w.r.t. the distribution over given by the model:
FIM Trace (0) .ES..j fo.o, (x)
[[Vol((()))))))
(8)
In practice, we calculate the FIM Trace over a mini- batch of size [B] = 16, as Novack et al. (2023) demonstrate empirically that mini-batch FIM explosion is correlated with the large-batch gencralization gap. Additionally, truncate the size of the datasets S₂ to be of size 2048. In most cases, we calculate the expectation by sampling ~ fo.ox (x:) many times from fo.d, many times for each x. In cases where the model does not give a natural distribution to exactly sample i, approximate instead with the empirical fisher information matrix, using the true label (for instance, regression tasks such as STS-B or Depth Estimation).
C.3 Gradient Coherence
Motivated by McCandlish et al. (2018); approximate the gradient covariance using several small batches and an aggregated large-batch. We compute an aggregated large batch gradient of size [L] by first computing gradients of batch-size [B] [4] and averaging them.
VC = Σ B.
(9)
During the computation of each gradient, compute and store the gradient norm, giving samples of a batch-size B gradient norm; we denote the average of these gradient norms as Gg. Additionally, after the computation of the aggregated gradient of batch-size L, we compute its norm which we denote by GT.-
10 exclude the WNLI task, is well-documented extremely prone overfitting (Devlin al., 2019).
20

---

Published in Transactions on Machine Learning Research (December/ 2occmberr/202/ 22024)
Conveniently, as noted by McCandlish et al. (2018), access to G and Gz provides method to approximate the gradient covariance. Namely, we can write the expected gradient norm of batch-size [2] as
+ Az tr()
V (0)1 1 |V
(10)
1Z
where Az is the FPC for a batch-size of 2), We can therefore write
18-1-12 -
because examples are sampled without replacement.
AB BI
AL tr (r(E())
(11)
B
[M
and from this we see that
tr(20)) = V - (B
(12)
Ag A
[T]
2 (Gn Gr.) A Σ
(13)
An [
which gives us an approximation to BsGn- In many cases, the FPC factors Az are ignored because |2| < [Sx] and thus Ag s 1. However, in some settings, [L] approaches |S and there we do not ignore the FPC.
C.4 Aggregating Values Over Training Trajectory
In general, we are interested in the value of these factors over specific points of the training trajectory. For instance, we are interested how sharpness and gradient covariance are minimized towards the end of training, near convergence; alternatively, we are interested in how the FIM trace explosion is minimized at the initial phase of training. However, these factors are also connected to the magnitude of the training loss (o.g. in practice covariance). Therefore, when comparing these factors between different training trajectories, we compare values over similar loss by comparing points that fall within a loss bin. For instance, for early-stage FIM Trace explosion, we first compute the values of training loss that constitute the "early phase" of training by creating 20 geometrically spaced bins for a training trajectory, and selecting bins 2 - 5. This gives us a loss bin which captures the early phase of training, but ignore the very early points that typically have abnormally high loss; on tasks like MNISTS or CIFAR-100, this often constitutes a bin with max value of 10 and minimum value of 10-1, which is where we typically see the FIM explode.
that low training loss typically has smaller gradients and therefore lower gradient
For sharpness and gradient coherence, which we measure towards the end of training, we allow the bin to be set by the training trajectory that has the highest minimum loss. For example, when comparing the sharpness of a multi-lask and single-task trajectory, the multi-task trajectory will typically converge to a training loss that is an order or magnitude (or more) higher than the single-task trajectory. In this case, we select a loss bin that is computed to capture the end of the multi-task trajectory, e.g. bins 18 out of 20 geometrically spaced bins over the multi-task trajectory. Then, we compare the values of the single-task and multi-task sharpness that. fall within those bins, so that we always comparing values at similar loss values. We emphasize this because we know, from $3, that multi-task transfer can be seen as gap in generalization for comparable training loss (often from the carly phase of training all the way to convergence). Therefore, it is important to focus our analysis on comparing and contrasting aspects of training trajectories at comparable training loss.
Discussion of Related Work
D.1 Multi- Transfer and Optimization
The focus of our work is on undorstanding how, deep multi-task learning (specifically, in the sharod encoder setting: Ruder, 2017; Crawshaw, 2020), generalization impacted by the joint optimization of many tasks. In the recent past, many multi-task optimization methods have been proposed to mitigate negative
21

---

Published in Transactions on Machine Learning Research (December/200) 22024)
transfer, balance generalization across tasks, or even improve positive transfer in multi-task training: these methods all operate by directly attempting to tackle a variety of optimization problems caused by gradient conflict (see $2.1). However, while the consequences of gradient conffict on optimization are well established in theory, the impact of gradient conflict on generalization in deep learning is much murkier. For instance, in the related area of auxiliary task selection, methods often rely on signals from validation data to select related tasks (Standley et al., 2020; Wang et al., 2020; Jiang et al., 2023), as it has been shown that gradient conflict is not necessarily indicative of auxiliary task benefits (Du et al., 2020; Jiang al., 2023). Moreover, recent work has also found, in MTL, that the amount of conflict towards the end of training has relevance to generalization, while conflict at the beginning of training is only partially correlated with transfer (Royer et al., 2023). Finally, the efficacy of SMTOs has recently been called into question by Xin et al. (2022) and Kurin et al. (2022), who empirically demonsttate that many SMTOs do not actually improve generalization over the UMTG. These findings all call into question the ability of gradient conflict to explain transfer.
By demonstrating that multi-task transfer occurs as a generalization gap early into training ($3) and studying factors previously used lo explain generalization gaps in prior deep learning studies ($4.1), we hope to close the gap in our understanding on the connection between optimization and generalization in deep MTL. However, the negative results of our work instead largely highlight the inability of optimization to explain the impact of multi-task learning on generalization. Overall, our results offer a potential explanation to the findings of Xin et al. (2022) and Kurin et al. (2022), e.g. that specialized optimizers do not improve multi-task transfer because it is still unclear how multi-task transfer is impacted by optimization.
An alternative approach to predicting task relationships or transfer is to instead model task relationships in model-agnostic, at least optimization-agnostic way. For instance, Vu et al. (2020) look to generate task- embeddings from pre-trained model and use those embeddings to compute task similarity. Li et al. (2023) propose to train a surrgate model to predict the relevance of a set of auxiliary tasks given many examples of model performance across auxiliary samples. Meta- Mlta-llarning has also been proposed as an approach to learn the relationships between a group of tasks, using validation data to meta-learn which auxiliary tasks will be beneficial to the large-lask Song et al. (2022); Liu et al. (2022). These works often focus on predicting target-task transfer given a set of source-tasks rather than improving the performance of multi-task model given a fixed set of tasks or leverage validation data to prodict task relevance.
D.2 Generalization Gaps in Deep Neural Networks
Generalization gaps between training trajectories of identical architectures are perhaps most popular in the "large-batch generalization gap" literature, i.e. the observation that small-batch models generalize better than large-batch models at similar training loss (first noted by Keskar et al., 2017; Smith et al., 2018). Early attempts to explain the large-batch generalization gap relied on the width, or sharpness, of the solutions found by large versus small-batch training (Hochreiter & Schmidhuber, 1997), and prior work speculated that the level of noise in the gradient estimates dictated the flatness of the final solution (Jastrzębski el al., 2018; Smith et al., 2020; Li al., 2021). However, the connection between sharpness and generalization has recently been called into question, with several works finding that metrics for sharpness are not necessarily correlated with generalization (Kaur et al., 2023; Andriushchenko et al., 2023; Mueller et al., 2023) and that sharpness is instead correlated with the learning rate used during optimization (Cohen et al., 2021; Kaur et al., 2023). In response, recent work has explored alternative explanations for generalization gaps; notably, Jastrzebski et al. (2021) find that generalization may be dictated by the maximum trace of the Fisher Information Matrix (FIM) in the early stages of training, for which small-batch models have lower explosion. Separately, Smith ct al. (2021) leverage backwards analysis to show that mini-batch SGD optimizes a modified loss that contains an implieit bias towards small gradient covariance. Both the Fisher Information Trace and Gradient Covariance have been empirically shown to improve generalization when explicitly optimized in large- or full-batch training (Geiping et al., 2022; Novack et al., 2023).
Notably, attempts to explain the latch-batch generalization gap have focused on how aspects of oplimiza- tion trajectories-ed e.g. how the surface sharpness is impacted, or how well gradient coherence is implicitly minimizod may explain why trajectorics generalize differently. In this work, we demonstrate that multi-task transfer (both negative and positive) elicits a generalization gap, similar to the large-batch generalization gap in single-task learning ($3). However, we find that the aspects of optimization previously tied to gener-
22

---

Published in Transactions on Machine Learning Research (December/ (etember/2002. 2024)
alization gaps unable to explain the generalization gaps observe in multi-task learning, both between single-task and multi-task trajectories ($4.1, $5.2) and between the trajectories of different optimization methods in a fixed MTL problem ($4.2). Our findings not only empirically demonstrate the difficulty of explaining transfer in doep learning, but also emphasize a disconnect between how gradient conflict impacts optimization versus how gradient conflict impacts generalization in deep multi-lask learning.
E
Full Task Results
23

---

Published in Transactions on Machine Learning Research (December/ 1atnmmberr2222/ 22024)
Thu And Vegetables Genera wearatttttoooo on by Loss
Fruits And Vegetacien bes Sharpposs
Pruits and Vegetab in FNN Tace
Truts And Vegetables Gradert Caherence
Sing
120
Jank
3044
13-4
104
10-1 Large
384
Large
Setting
22ng w-3kk
H.B Tast
Setiig
Sing clikk
232
M Jast
23th
Tarl
Teb Cutccor Scores Gtedken Coherence
Ousdoor Scores Concroliatton by Loss
Ouodber Sconcs Shaprees
Outdoor Scencs PIM Tuce
Sectng
TOAA P311
19-4
Apm1
Setting
Sing lolsk
Mil Tosk
Sattrg Sing el
Figure 6: CIFAR Tasks which experience negative transfer.
24

---

Published in Transactions on Machine Learning Research (December/ 1atinnber/222/ 22024)
040
Settin
Gro w-/bkk
120
e-bckk
86
134
13-4
30
12-4
13-1
30-4
30
12-
10-4
Larpe
Shorpness
Large
Seting
2ing w-3bkk
M.B Tast
Task
Setting
Sin w-3kkk
Mu Tosk
Settrg e-bck
M.B.Tesk
154
39-1
10"
30-2
18-41
30.
35-7
30-4
30
13
Figure CIFAR Tasks which experience positive transfer.
25

---

Published in Transactions on Machine Learning Research (December/ (enemberr/222// 22024)
Big Lip Genera laadon by Loua
Sig. pi Shargreas
Dg Usa RN Tacs
5gg Liu Cradd lett Coherance
Tock
Tacc to aare S,o.Back Shadow Stadirt Caterrence
L.J_hoou_hharoe Gencrolizaion by
Stock_Staddowww Stadow Starpress
5,0, Shardw
Suting
5.000
Sctting
5 Shee
8440
Singl-Taek
Nute-Tee
The
150l
500
1:5
100
164
Attractive Gereratmation
Altractive Sharpeace
Astractive
librctive Goad ant Coherence
my
Setting
Ttak
Bingii-Takk
Tuacr sana tane
Teachre
Taxx The lass
and, far Gencra Aausion by Loss
Bland Mall Sharpinsss
Blond, Max FBN Trace
Slend Hair Crackent Consrracce
5.00
Nust-Tosc
8410
$130
$100
som
8005
Shorpness
Sect
Sctting
Tbak
Single-Teer Yy to lock
Mult- lask
1041
30P
1.5
22:1
Too
109
$0.4
Dcubre Osin Conemtizationn Lors
Double Chis Starprans
Dcuble Chin FIM Tacc
Double Ch Gradient Coharatce
Setting
Setting
Singlo taak
Single-Taa
0.15
Tec. Leai
Tax Th
Figure 8: CelebA Tasks which experience significant transfer.
26

---

Published in Transactions on Machine Learning Research (December/ (atnmbber/2220 2022)
Gostes Gantrnizztion
Gastee Sharsnwes
Gostre TIN
Gontee Gradenk Conenence
Senting
Sett
orak
Sinplo Taak
task
AU7
Tec
Cray.Hair Mar Guner alout Loss
Dray, Hatl Starpnes
Crayy Mall PAR
Giay Grod ent Ceherance
Setting
ng
A3
Tas High, Cheekboren Starsnens
Tactranag Lou.
.Egp Chwwchonee Conersization by Loos
High Cheekiornes RN Trace
High Cheechonee Gradient Coherence
Thct
Sasa
8/8
Triiiigg
Thaten Blnyy FI Tore
Text Theneelonn Stury Grddler Cehource
Bibryy Gerersitndion by Lons
(Rurry Sharpseens
Siyple
Text
Eyog asaaa Gencraliaaton by
Eyoplaseus prezes
Eyeglazuca Trace
Eyoglasses Graclent Catarence
Gerting
e-akk Mosk
Sinple/Tuik V4.1 back
Figure 9: CelebA Tasks which experience significant transfer.
27

---

Published in Transactions on Machine Learning Research (December/ 1atnnbber/22200 2024)
5.02
NEPC Genera tratton by Look
MBCC Sharpnnee
375
HBFC FIM TAcH
VAPC Gradient Coherence
Setting
6.75
2.92
Bvcc
18:00
1.00
12 Tan mannin
0.83
XTT Ceneenateenn Braton by Lone
RTT Shorpheee
RTE FTE Paace
RTC Geadent Coheeeeee
0.82
0.22
0.1
Selt
Taele-Taet
MLF: lact
0.4
Tran
71400
3.002
53T22 Gonsrelizttion by Lou
2572 Sharomoo
257.2 PIN Thacs
1572 Crudietf Conderrss
2.95
6.93
2.105
6.83
Ter
Taon Frain Em01
2.80
S75 Comera lsation by L:30
402
STS 5 Sharpnos
STS 8 PIN Thacc
STS wrw.lentCohoeeeeeooooo
2.99
6.93
Tark Twwking
Texc Tramseption
Tran Train
Traking
Figure 10: GLUE Tasks which experience significant transfer.
28


Reviews:
[{"review_id": "ftG5egNR3g", "reviewer": "Reviewer_KAUz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**[Strength]**\n1. The paper addresses an essential aspect of multi-task learning: identifying which combinations of tasks create synergy and how to predict these interactions, although a clear conclusion on this remains open.\n2. Table 1 effectively illustrates the relationship between generalization performance changes and various optimization factors.\n3. The authors conduct a wide range of experiments.\n\n\n**[Weakness]**\n\n1. The proposed FashionMTL dataset requires a more detailed explanation, and it may not be well-suited for analyzing multi-task learning. First, the authors mention splitting FashionMNIST into two parts (Fashion1 and Fashion2) while maintaining class balance. If each split includes all classes and has the same distribution, it’s unclear whether this setup can genuinely be considered “multi-task” — it seems more like a simplified example. Additionally, the test set should be clarified: in NoisyFashion, are the test labels also shuffled?\n\n2. FashionMTL, MNISTS, and CIFAR-100 are not standard multi-task datasets, which typically involve predicting different components (e.g., depth, segmentation, and attributes) from a single input. Instead, these datasets comprise multiple single-task settings that share the same backbone network and use ground-truth task labels to assign inputs to each classifier. Using datasets like Multi-Digit MNIST [2] or Taskonomy [3] would be more suitable for this context.\n\n3. While “generalization” in multi-task learning is often compared to single-task learning (e.g., [1]), the term “generalization gap” requires further clarification, especially regarding the comparison at the same training loss level. For example, achieving the same training loss does not necessarily guarantee equivalent performance.\n4. The description of gradient similarity (Eq. 6 and Eq. 1) needs elaboration. For instance, are gradients from all layers concatenated, or is a specific layer, such as the second-to-last, used?\n5. There is no explanation provided for why each task shows a different trend in Figure 6. Additionally, the choice of auxiliary tasks (1, 4, 9, and 19) should be further explained, including the rationale behind selecting these particular numbers.\n\n\n[1] Hu, Ziniu, et al. \"Improving multi-task generalization via regularizing spurious correlation.\" Advances in Neural Information Processing Systems 35 (2022): 11450-11466.\n\n[2] Sun, S.-H. Multi-digit mnist for few-shot learning, 2019. URL https://github.com/ shaohua0116/MultiDigitMNIST.\n[3] Zamir, Amir R., et al. \"Taskonomy: Disentangling task transfer learning.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates why certain multi-task settings enhance performance over single-task learning. The authors revisit existing analyses, particularly from an optimization perspective, and conclude that none of these approaches directly predict the generalizability of models trained with multi-task learning. However, they empirically observe that the positive or negative impacts of multi-task learning are established in the early stages of training.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**[Experiments]**\nAs mentioned in Weakness, it would be nice to emphasize CelabA, Cityscapes, and GLUE results and please elaborate more on the results.\n\n\n**[Writing]**\n* The phrase “Beyond the zero’th order” is used too frequently (5 times), and the word “beyond” appears 8 times; consider rephrasing to avoid repetition.\n* Change “zero’th” to “zeroth.”\n* In Figure 5, add “and p-value” to the background color setting.\n* In Appendix B, correct “FashionMNISt” to “FashionMNIST” and “lenet” to “LeNet.”\n* In Figure 4, the legend locations vary by row, and the notation for the Gradient Method differs in the second row. Please standardize these, for example, by having a single legend per row. Additionally, the straight lines from (0,0) to each data point don’t add value and could be removed.\n* Figures 5 and 6 could be combined into a single figure (or arranged as subfigures), or at least placed on the same page. Currently, readers must switch between pages to follow the authors’ claims effectively.\n\n**[Question]**\n* In Figure 2, what do the green and red bars represent, and what does each dot signify?\n* Why does the last column in Figure 11 have an extremely large error bar?", "broader_impact_concerns": "There are no broader impact concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FrRGG80Yb0", "reviewer": "Reviewer_oXuy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well-written and presents its findings with strong experimental support and a clear takeaway, which I really appreciate. Below, I offer some minor comments in the order they appear in the paper:\n\n1. The paper covers multiple aspects of MTL generalization, including training dynamics and gradient conflicts, yet the current title only emphasizes optimization trajectories. To better reflect the paper's scope, I would suggest adopting a more general title that reflects the full range of topics discussed.\n\n2. The references are inconsistent in formatting—some use abbreviated conference names while others do not; and quite a few references completely miss the conference/journal names. Additionally, the 7-class Cityscapes dataset citation might be incorrect; the dataset was constructed in MTAN, not MGDA.\n\n3. The paper concludes that “multi-task transfer is a property of how gradient conflict impacts the early phase of learning.” However, why is this limited to the early phase? While Fig. 1 shows that gradient conflict can be identified early on, this does not necessarily imply that conflicts are **only exisiting** in the early phase.\n\n4.  The paper convincingly argues that modified optimization trajectories introduced by SMTOs do not fully explain generalization behaviour in MTL, which I agree with. However, this observation may extend beyond MTL and apply to single-task learning scenarios as well, such as those involving data augmentation, loss regularization, or neural architecture variations. This actually suggests a broader phenomenon in deep learning, where generalization in high-dimensional parameter spaces remains poorly understood.\n\n5. At the end of Section 5, the paper concludes that mitigating gradient conflict does not necessarily improve generalization. Does this imply that MTL or auxiliary learning optimization strategies which were designed with a meta-objective directly optimized for generalization performance are inherently better? For instance, approaches like AuxiLearn (ICLR 2021), Auto-Lambda (TMLR 2022), and Task Weighting in Meta-learning with Trajectory Optimization (TMLR, 2023) could be relevant here. Looking foward to hearomg authors' comments on this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper offers a thorough analysis of the generalization challenges in multi-task learning, exploring its connections to training losses, optimization trajectories, positive/negative transfer, and gradient similarity. Through extensive experiments and ablation studies, the authors conclude that current multi-task optimization strategies do not significantly differ in their effects on optimization trajectories and its relationships with generalisation are still unclear. This finding may reinforce concerns about the limited generalization of various multi-task optimization methods, as previously highlighted in other studies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Fix references. Address the weaknesses listed above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rMtMR5OmPQ", "reviewer": "Reviewer_yNT4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n\n- This paper conducts extensive experiments on six multitask datasets (FashinMNIST, MNIST, CIFAR100, CelebA, Cityscapes, GLUE), and provides abundant results on the task transfer results and the evaluations of model optimization trajectories, such as sharpness, Fisher, and gradient covariance. \n- This paper shows that no single factor from the optimization trajectory (sharpness, Fisher, gradient covariance, gradient cosine similarity) can predict the multitask transfer results. \n\n### Weaknesses\n\n- This paper does not have a principled explanation for the empirical results. There are many empirical results of the evaluations in optimization trajectories. Yet, the conclusion from the results is that no single factor can predict the transfer results. The practical implication is limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper conducts extensive experiments on six multitask datasets (FashinMNIST, MNIST, CIFAR100, CelebA, Cityscapes, GLUE), and provides abundant results on the task transfer results and the evaluations of model optimization trajectories, such as sharpness, Fisher, and gradient covariance. \n\n- This paper shows that the generalization gap between single-task and multitask optimization trajectories emerges in the early phase of training. \n\n- This paper shows that no single factor from the optimization trajectory (sharpness, Fisher, gradient covariance, gradient cosine similarity) can predict the multitask transfer results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- For Figure 1, is the x-axis the training loss on a single target task or multiple tasks? If it is the latter, what does it mean when it reaches the same total training loss for STL and MTL? It would be better to explain more in the figure's title. \n\n- In Section 3, the paper claims that negative transfer incurs generalization gaps between STL and MTL trajectories comparatively early into training. How early does the negative transfer start to appear in training? It would be helpful to show figures of training/text losses w.r.t. number of epochs or iterations. \n\n- In Section 4, the paper observes that no factor is capable of explaining negative and positive transfer in any MTL setting we study. What is the correlation between the factors to negative transfers? Can we also measure the factors at the checkpoint at each epoch to study the correlation? How about considering various subset combinations of tasks (such as in GLUE), and how do the factors correlate with the transfer results (it would be better to report correlation scores)? \n\n- I also have similar questions about the results in Figure 4. It seems that there is some correlation between the factors and the transfer results. It remains unclear **how strong/weak the correlation is**. For example, can one fit a linear or polynomial model using the values of factors to predict the transfer results? How significant would that prediction be? It would be better to show similar figures like Figure 5 or 6. \n\n- It would be better to provide a section to describe the practical implications of the observations: such as how can we choose MTL optimization algorithms given the evaluated factors in the early phase of training (sharpness, Fisher, gradients)? \n\n- There are prior works on predicting negative transfers in multitask learning, which should be discussed in the paper:\n\n[1] Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., ... & Iyyer, M. (2020). Exploring and Predicting Transferability across NLP Tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)\n\n[2] Li, D., Nguyen, H., & Zhang, H. R. (2022) Identification of Negative Transfers in Multitask Learning Using Surrogate Models. Transactions on Machine Learning Research.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates when and how multi-task learning might improve generalization, by looking at learning trajectories during optimization.  The reviewers all found that the claims were well supported by the comprehensive empirical evidence.  They also felt that the paper and the presented takeaways would be of interest to the TMLR community.  Overall, the reviews were very positive, and the reviewers applauded the comprehensiveness of the empirical analysis and found the presented discussion and conclusions interesting and convincing.  The authors did well to address the reviewers' concerns during the discussion and the reviewers appreciated the clarifications and proposed revisions.  The paper presents a clear empirical study that addresses a highly relevant hypothesis in ML.  I found it interesting that the authors found a generalization gap between single task and multi-task early on the training process, emphasizing the importance of early training dynamics.  There was a bit of a negative result in terms of whether the various metrics studied were indicative of positive task transfer (I.e. none of them were), but this is well supported by the experiments and also an interesting takeaway.
Decision: Accept (poster)