NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective
Authors: Xiaohan Qin
Xiaoxing Wang
Ning Liao
Junchi Yan
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Task Learning
Task Imbalance
Primary area: general_machine_learning

Abstract: Multi-Task Learning (MTL) enables a single model to learn multiple tasks simultaneously, leveraging knowledge transfer among tasks for enhanced generalization, and has been widely applied across various domains. However, task imbalance remains a major challenge in MTL. Although balancing the convergence speeds of different tasks is an effective approach to address this issue, it is highly challenging to accurately characterize the training dynamics and convergence speeds of multiple tasks within the complex MTL system. To this end, we attempt to analyze the training dynamics in MTL by leveraging Neural Tangent Kernel (NTK) theory and propose a new MTL method, NTKMTL. Specifically, we introduce an extended NTK matrix for MTL and adopt spectral analysis to balance the convergence speeds of multiple tasks, thereby mitigating task imbalance.
Based on the approximation via shared representation, we further propose NTKMTL-SR, achieving training efficiency while maintaining competitive performance. 
Extensive experiments demonstrate that our methods achieve state-of-the-art performance across a wide range of benchmarks, including both multi-task supervised learning and multi-task reinforcement learning. Source code is available at https://github.com/jianke0604/NTKMTL.

NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective
Xiaohan Qin 122 Xiaoxing Wang's Ning Liao', Junchi Yan 12.4
School of Artificial Intelligence & School of Computer Science, Shanghai Jiao Tong University
Shanghai Innovation Institute
Abstract
Multi-Task Learning (MTL) enables single model to learn multiple tasks simulta- neously, leveraging knowledge transfer among tasks for enhanced generalization, and has been widely applied across various domains. However, task imbalance remains a major challenge MTL. Although balancing the convergence speeds of different tasks is an effective approach to address this issue, it is highly chal- lenging to accurately characterize the training dynamics and convergence speeds of multiple tasks within the complex MTL system. To this end, we attempt to analyze the training dynamics in MTL by leveraging Neural Tangent Kernel (NTK) theory and propose new MTL method, NTKMTL. Specifically, an extended NTK matrix for MTL and adopt spectral analysis to balance the con- vergence speeds of multiple tasks, thereby mitigating task imbalance. Based on the approximation via shared representation, we further propose NTKMTL-SR, achieving training efficiency while maintaining competitive performance. Ex- tensive experiments demonstrate that our methods achieve formance across wide range of benchmarks, including both multi-task super- vised learning and multi-task reinforcement learning. Source code is available at https:// //g thub. com/ comm/iaank00004//TTAT
introduce
per-
Introduction
Multi-task learning (MTL) [9, 20, 56, 45, 61] involves training single model to address multiple tasks concurrently. This approach enables sharing information and representations across tasks, enhancing the model's generalization capabilities and boosting performance individual tasks [6, 50, 38]. MTL is particularly advantageous in scenarios with limited computational resources, as eliminates the need to maintain separate models for each task. Its utility spans wide range of domains, including computer vision [1, 63,33,35, 35], natural language processing [10, 34, 40]. and robotics [16, 55). Despite these benefits, MTT. encounters significant challenge known as task imbalance, where certain tasks dominate the training process while others suffer from insufficient optimization. Previous studies [4, 48] have indicated that achieving more balanced optimization across tasks often leads to improved overall performance. Addressing such a task imbalance issue necessitates the development of sophisticated optimization strategies ensure that all tasks benefit equitably from the shared model parameters.
One widely adopted perspective to address the above issue is to balance the convergence speeds of different tasks [32, 60, 35]. Nevertheless, it is highly challenging to accurately analyze the training dynamics and convergence speeds of multiple tasks within the complex MTL. system. Most prior methods [35, 31] approximate the convergence speeds based on the difference or ratio between consecutive loss values. However, different tasks exhibit vastly different loss scales and heterogeneous
Corresponding author. This work is partly supported by NSFC (92370201).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ultimate loss minima, thus such a simple approximation fails to accurately capture task's s convergence capability or potential at a specific training stage. As demonstrated by experimental results on widely used benchmarks such as NYUv2, these methods still exhibit considerable task imbalance. Therefore, there is a pressing need for tool to characterize MTL. training dynamies and task convergence properties with robust theoretical foundation.
To this end, we attempt analyze the training dynamics within MTL systems by leveraging Neural Tangent Kernel (NTK) theory[17, 2, 7], which provides insights into the optimization trajectory of deep neural networks and has demonstrated its theoretical efficacy in single-task learning (STL) scenarios. From the perspective of NTK theory, the convergence speed of a neural network can be characterized by the eigenvalues of its corresponding NTK matrix. Specifically, lower frequenyy components of the target function typically correspond to larger NTK eigenvalues, which converge components often correspond to smaller NTK eigenvalues, which converge more slowly (or are harder to learn). This phenomenon is known as "spectral bias" in the context of single-task learning [22, 43, 53], which bears a strong resemblance to task imbalance in MTL. As explained by the NTK theory, such distinction in the training dynamics provides foundational understanding of why certain aspects of the target function are prioritized over others during the training process. However, despite its potential relevance, the application of NTK theory to the field of MTL has been scarcely explored by prior work.
faster [7, 52]. contrast,
Building upon the above motivation, this work applies the NTK theory to MTL scenarios and introduces an extended NTK matrix to jointly characterize the training dynamics of multiple tasks. Specifically, the target function in MTT. is explicitly decomposed into multiple distinct components, each associated with different task. Our theoretical analysis shows that, the convergence speed of the overall training error is jointly influenced by the NTK matrices corresponding to each task, and tasks associated with larger NTK eigenvalues can be learned more rapidly by the network, subsequently dominating other tasks and resulting in unsatisfactory performance the remaining tasks. To address this issue, we propose new MTL approach, NTKMTL, which assigns appropriate weights during training based on the NTK analysis of each task. This method effectively balances the convergence speeds of different tasks and, consequently, reduces task imbalance. In summary, our contributions can be outlined as follows:
1) We introduce new perspective on understanding task imbalance in MTL by leveraging NTK theory for analysis. Under this perspective, multiple tasks are viewed as distinct components of the training objective, each characterized by unique NTK spectral properties. These inherent differences in spectral characteristics lead to significant disparities in convergence speeds across tasks, subsequently causing task imbalance.
2) Based on the NTK spectral analysis of different tasks in MTL, we propose a new MTL method, NTKMTL. Both theoretical analysis and experimental results demonstrate that NTKMTL effectively addresses the ill-conditioned distribution of NTK eigenvalues during MTL training, thereby alleviating task imbalance.
3) To enhance the practical applicability and computational efficiency, we further introduce NTKMTL-SR, efficient approximation that leverages the Shared Representation in MTL. NTKMTL-SR requires only a single gradient backpropagation per iteration for shared parameters, which not only provides computational efficiency but also maintains competitive performance.
4) Extensive experiments validate that our methods achieve state-of-the-art performance across wide range of benchmarks. The experimental scenarios include both multi-task supervised learning and multi-task reinforcement learning, with task numbers ranging from 2 to 40.
2 Related Work
Multi-Task Learning. In multi-task learning (MTL), previous approaches can be broadly clas- sified into two categories: loss-oriented and gradient-oriente methods. Loss-oriented methods primarily aim to address convergence discrepancies arising from differences in loss scales, thereby mitigating task imbalance. These methods often exhibit training efficiency as they only require one backpropagation on the aggregated loss. Approaches include Linear Scalarization (LS), Scale- Invariant (SI), homoscedastic uncertainty weighting [24], dynamic weight averaging [35], self- pelf-paced learning [37]. geometric loss [14], random loss weighting [29], impartial loss weighting [33], fast
2

---

adaptive optimization [31], and multi-task grouping for alignment [48]. On the other hand, gradient- oriented methods [11] focus on resolving the gradient conflicts issue among shared parameters. seeking the most favorable updating vector to alleviate task imbalance. These types of methods often achieve better performance due to directly obtaining the gradients for all tasks for optimization. Notable approaches in this category include Multiple Gradient Descent Algorithm [46], gradient normalization [12], gradient conflicts projection [57], gradient sign dropout [13], impartial gradient
weighting [33], conflict-averse gradients [32], gradient similarity regularisation [5 11. dual balanc-
ing [27], stochastic direction-orienter update [54], smooth tchebycheff scalarization [30], independent
gradient alignment [47], Nash bargaining solution [39], fair resource allocation [4], performance
informed variance reduction approach [41], and consistent multi-task learning with task-specific
parameters [42].
Neural Tangent Kernel theory. Recent theoretical advancements have modeled neural networks in the limits of infinite width and infinitesimal learning rate as kernel regression using the Neural Tangent Kernel (NTK) [17, 2, 7, 22]. Specifically, analyses by [26] and [2] demonstrate that, during gradient descent, the outputs of a neural network remain close to those of linear dynamical system, with the convergence speed governed by the eigenvalues of the NTK matrix [5, 52, 53J. The NTK's function of frequency,
eigendecompossition reveals that its eigenvalue spectrum decays rapidly which explains the well-documented "spectral bias" of deep networks towards learning low-frequency functions [22, 43, 53]. Such spectral bias phenomenon bears strong resemblance task imbalance in MTL, which is an effective application of NTK theory in explaining neural network training dynamics in STL scenarios. Building on this observation, this paper extends NTK theory to MTL and proposes a solution to address the task imbalance issue.
3 Method
3.1 Preliminaries
To establish the foundation for our theoretical investigation, we first review the traditional Neural Tangent Kernel theory that explores the training dynamics of deep neural networks. In the subsequent sections, we extend these theoretical frameworks to the multi-task learning context and develop specific methodologies accordingly.
Neural Tangent Kernel. For deep neural network f with parameters 0 and training dataset yx, = [(xi, W:)] the Neural Tangent Kernel (NTK) is defined as
18(00, x.)) 0f
Kuu
00
00
Prior works [22, 26, 2] have demonstrated that under certain conditions (e.g., when the learning rate 1 approaches zero), the training dynamics of the neural network be characterized via gradient flow, which is governed by the following ordinary differential equation (ODE) [44]:
d0()) = (())).
(2)
dt
This leads to the following theorem:
Theorem 3.1. Let Ott) = (f xx) be the outputs of the neural network at time t. x = Y is the input data, and {yr 1 1 is the corresponding label, Then O(t) follows this evolution:
dO(t) dl
-C (0(1) y).
Detailed analysis can be found in the Appendix. Eq. 3 enables to utilize the neural tangent kernel to analyze the training dynamics of neural networks. Prior work [22] demonstrated that as the network width approaches infinity, the NTK K. remains approximately constant during training. A more widely used result is that during the training of deep neural networks, the kernel function is updated much more slowly than the network's 's output [52, 62]. Therefore, Eq. can also be interpreted as an ODE and provides the following approximation:
O(t) ~ (I my
(4)
3

---

Spectral analysis in neural network training, Let consider the training error O(t) y. Since the NTT matrix must be positive semi-definite, we can take its spectral decomposition K. == =QQ", where Q is orthogonal matrix and A is diagonal matrix whose entries are the eigenvalues of K. Then, since e gAa Qe Qe - we have
Q (O(()) y) -Q" T - Q'yy..
(5)
This implies that, when considering the convergence of training in the NTK eigenbasis, the i-th component of the absolute error Q (O(t) v) decays at an approximate exponential rate of na In other words, the components of the target function corresponding to kernel eigenvectors with larger eigenvalues are learned more rapidly, resulting in the high-frequence components the target function converging exceedingly slowly, to the extent that the neural network is difficult to learn these components.
3.2 Extended NTK in Multi-Task Learning
In multi-task learning, a neural network with shared parameters is trained simultaneously learn distinct tasks. In the general case, the overall loss function is defined as
(()) Σ (())
(6)
1=1
In this context, by leveraging the gradient flow defined Eq. 2, Theorem 3.1 can be further extended as follows:
Theorem 3.2. Let (t,(t), Ox(t)}
denote the outputs of the neural network function {f1, f2..... f& for the k tasks ar time t, and let 12.........3 represent the corresponding labels. Then, the ordinary differential equation in Eq. 2 gives the following evolution:
(1)
K11
KIk
(t) - yy
dt
(Oa(()
KA1
Kbk
Ox(t) ya
de
where Kij E Ruxn and Ki = for 15 siij S k. The (u, v)-th entry of Kig defined as (KCi)uu (ff( x.. (f, x.))
(8)
00
00
Detailed proof can be found in the Appendix. We define the large NTK matrix, formed by the training dynamics of the k tasks in Theorem 3.2, the extended NTK matrix in MTL, denoted K. It straightforward to observe that Ka for s k. as well as IC itself, are positive semi-definite matrices. In fact, let J, denote the Jacobian matrix of fi with respect to 0 then we can observe that:
J1
Ku = J.J, and Kc =
: JR
[
M
Analogous to the analysis presented in Sec. 3.1, we adopt the approximation derived from the theoretical framework of [22]. Consequently, Eq. 4 is extended to the multi-task learning scenarios as follows:
[t)(
y1
2 (I ---e- Ti-e-nky
;
(10)
(x)()
yk
As mentioned above, the extended NTK matrix positive semi-definite. Therefore, similarly to previous steps, we perform its spectral decomposition as K = QÂQT. Consequently, the training error MTL can be approximated by the following evolution:
[t)(
y1
~
-
: yk
(11)
yk

---

Eq. 11 illustrates that in the MTL scenario, where the learning objectives are explicitly partitioned into components, the training dynamics can still be interpreted through spectral analysis of the extended NTK matrix. The substantial disparities in the distribution of NTK eigenvalues across different tasks give rise to bias in convergence speed, causing the network to be dominated by certain specific tasks and hindering the effective simultaneous learning of all tasks.
3.3 The Proposed NTKMTL
Theoretical analysis in Sec. 3.2 and experimental results extensive benchmarks indicate that the traditional multi-task optimization objective in Eq. 6 yields unsatisfactory performance, exhibiting significant task imbalance across various task scenarios. Therefore, existing MTL methods often consider obtaining a weight w (w1, W2, ... to optimize the weighted objective instead:
(()) - Σ (0). 1=1
(12)
Specifically, the final gradient is ys Li=1 Wift. where yi represents the gradient of the i-th task's loss l, with respect to the shared parameters, and 9s is the final aggregated gradient. Under this formulation, the analysis of the extended NTK matrix for MTL systems presented in Sec. 3.2 is correspondingly modified. Therefore, we further introduce the following proposition:
Proposition 3.3. (Extension of Theorem 3.2) Let {O,( {y, (y;);-n. and {Kaghsassk be defined as in Theorem 3.2. We now replace the MTL optimization objective with the weighted as presented in Eq. 12. Consequently, the ordinary differential equation governing the MTL training dynamics in Eq. becomes:
wFKK11
Wi
TO((1) - Y1
dt
(13)
dOx(t)
WkW1KKk
KRRR
(x(() Y
dt
OR
Detailed proof can be found in the Appendix. The NTK matrix derived from the weighted optimization objective in Eq. 12 remains positive semi-definite, and thus, the analysis in Sec. 3.2 is still applicable. Proposition 3.3 shows that the cigenvalues of the new NTK matrix are strongly correlated with w, allowing w be used balance the relative magnitudes of the eigenvalues of the NTK across different tasks, thereby balancing the convergence speeds of different tasks.
Motivated by this, method is designed based the following strategy: In each training iteration, we compute the maximum eigenvalue A; of the NTK matrix Kj for task i, which serves as a good indicator of its current convergence speed. We then derive the task weights J by normalizing these eigenvalues } Proposition 3.3 provides the most intuitive physical interpretation: With the introduction of wi. the new NTK matrix for each task becomes w Kg,, and its maximum eigenvalue is accordingly scaled to wf\,, X,- Therefore, achieve eigenvalue normalization across tasks, we enforce the condition that:
1
W.
1...,...
(14)
VA
Directly employing wi = Vx. disregards the original scaling inherent in the task eigenvalues. Consequently, to retain the scale information reflecting the tasks' current convergence speeds, utilize the average maximum eigenvalue, X ES= >j. scale (w:)k-- This motivates the following definition for w,:
W
V x i i
(15)
The overall algorithm flow is shown in Algorithm 1. Our design of w enables the effective balancing of convergence speeds across different tasks by balancing the maximum eigenvalues of their NTK matrices, while preserving the scale information of the original eigenvalues. Nevertheless, computing the NTK matrix can be time-consuming since it requires dividing a batch into n mini-batches and computing their gradients. This limitation has prompted us to find way to compute the NTK with minimal cost, thereby extending the applicability of our method to more scenarios. The following section provides the solution.
5

---

Algorithm NTKMTLL
Algorithm 2 NTKMTL-SR
1: Input: Initial model parameters Oo: Learn-
1: Input: Initial model parameters Oo: Learning
ing rate [7):): number n of mini batches.
rate {mk}: number n of mini batches.
2: for t= 0 to T do
2: for == to T do
3: Compute [Ji.-... ... JE for n mini-batches,
3: Compute
;..")...,,,'( with respect
and obtain the gradients ..i. -%1-
to for mini batches.
4: Obtain the NTK { and through
4: Obtain (z) and K²(2) through Eq.9 and 17.
Eq. 8 and 9.
5: Compute {ws Jiz through Eq. 15.
5:
Compute (wi through Eq. 18.
6: Obtain the update vector 9
6:
Obtain aggregated loss L - -i w.C, Back propagate and update parameters O
Σi=1 wigh
7:
7: Update De+1 O2 O.n-..
8: end for
8: end for
3.4 Approximation via Shared Representation
In MTL, the model typically consists of shared parameters 0 and task-specific parameters for & tasks, where the number of parameters in the task-specific components (typically 1-2 layers of linear or convolutional layers) is much smaller than the number of shared parameters. We define the shared representation z as the output of the input x after passing through the shared parameters 0. Then, by applying the chain rule,
derive the following:
(f, x)
(f, x) dz
(16)
00
dz
00
Note that 80 is the same for all tasks and acts on all (ft)-- As result, it further impacts the overall NTK K. This implies that K can be viewed as projection of the NTK matrix K(2) of onto the feature space through 0z These analyses indicate that, similar to some previous works [23, 47], NTKMTL has the ability to accelerate computation using shared representation, and we name this approximation algorithm NTKMTL-SR. Specifically, consider approximating the original method using the NTK analysis of 2, i.e. by replacing Eq. 8 with the following expression:
(Ki, (=))ue (0, (0,
(17)
O=
0z
which further leads to the subsequent formulation for w:
i(z)
i 1,...,k.
(18)
(x(2)*
where (=) represents the maximum eigenvalue the NTK matrix K2+(z), and X(z) is the average of these eigenvalues across all & tasks, defined as A(z) E Az(z). Since computing the gradient of (: x) with respect to only requires backpropagation through the task-specific parameters, it incurs little additional time and memory cost. After obtaining w, instead computing the gradients separately for each of the & tasks and then weighting them, we can directly compute the aggregated loss using Eq. 12 and perform one backpropagation, meaning that we only need to compute the gradient for shared parameters 0 once per iteration.
The overall algorithm is outlined in Algorithm 2. Fig. illustrates the training speed the CelebA benchmark with up to 40 tasks, showing that NTKMTL-SR exhibits nearly the same training speed as traditional loss-oriented Linear Scalarization. This further enhances the generalizability of our method under various constraints.
Experiments
4.1 Protocols
We evaluate the performance of our proposed NTKMTL and NTKMTL-SR across a wide range of MTL scenarios, including multi-task supervised learning and multi-task reinforcement learning, For

---

Table 1: Results NYU-v2 (3-task) dataset. Each experiment is repeated 3 times with different random seeds and the average is reported. The detailed standard error is reported in the Appendix. The best scores are reported in gray.
SEGMENTATION
DEPTH
SURFACE NORMAL
METTOO
MIoUt 1
PIx
Ans
REi
ANGLE DISTANCE
WITHIN ("
MR Am% 4
MEDIAN
STI.
38.30
63.76
0.6754
0.2780
25.01
19.21
50.14
$7.20
69 15
LS
39.29
65.
0.5493
0.2263
28.15
23.96
22.09
47.50
61.08
17.44
5.59
RLW [28]
37.17
63.77
0.5759
0.2410
28.27
24.
22.26
47.05
62
20.22
7.78
DWA [55]
39.11
65.31
B. 1.5510
0.2285
27.61
23. 18
24 17
50.18
62. 39
16.35
3.57
UW [25]
36.87
63.17
0. 0.5444
0.2266
27.04
22. 61
23 54
49.05
63 65
16,00
4.05
MGDA
30.47
0.6070
0.2535
24.88
29.18
56,88
36
[58]
GRADDROP [13]
39.39
65.12
0.5155
0.2279
27.48
22.96
23.38
49.44
62.87
15.56
3.58
CAGRAD [32]
39.79
65
0.5486
0.2250
26.31
21.58
25.61
$2.36
65 58
11.56
0.20
IMTL-G [S3]
39.35
65.60
U.5426
0.2256
26.02
21.
26 20
$3.13
66 24
10.89
-0.76
MoCo [18]
83
NASH-MITL [39]
40.13
55.47
8.00
ALRINED-MIE [47]
40.15
66.05
0.5520
0.2291
25.37
19.89
28.30
55.29
67.95
10.44
-3.12
FAMO [3]]
38.88
64.90
0,5474
0.2194
25.06
19.57
29.21
$6.61
68.99 98
9.00
-4.10
SDMGRAD [54]
40.47
65.90
0.5225
0.2084
25.07
19.99
28.
55.74
68.53 53
6.11
-4.84
DB-MTL
STCH [30]
41.35
66.07
26.55
$1.39
8.22
FAIRGRAD [4]
39:74
66.01
0.5377
0.2236
24.84
19.60
29.26 26
56.38
69. 16
6.44
-4.66
G04ALION [48]
40.42
65.37
0.5492
0.2167
24.26
18.94
30.54
57.87
69 84
5.00
-6.08
NTKMTL
0.5296
NTKMTI--SR
63.28
0.5261
0.2136
24.88
19.58
29.53
56.67
08
5.56
-5.35
multi-task supervised learning, experiments are conducted on several benchmarks, including dense prediction tasks on the NYUv2 [49] and CityScapes [15] datasets, regression tasks on the QM9 [8] dataset, and image-level classification on the CelebA [36] dataset. For multi-task reinforcement learning, experiments are performed the MT10 environment from the Meta- World benchmark [59]. Meta-World
Due to the memory and time cost associated with fully computing the gradients of mini-batches for shared parameters to construct the NTK matrix, we set when comparing with other methods. In this case, it is consistent with other gradient-oriented methods, requiring gradient backpropagations per iteration. As for NTKMTL-SR, since its cost of constructing the NTK matrix is minimal, we set 4 by default based on experimental validation.
for NTKMTL to ensure fairness
Baselines. We comprehensively compare the proposed NTKMTL and NTKMTL-SR with the following methods: Single-task task learning (STL), Linear Scalarization (LS). Scale-Invariant (SI), Dynamic Weight Average (DWA) [35], Uncertainty Weighting (UW) [25],
Descent Algorithm (MGDA) [46], Random Loss Weighting (RLW) [28], PCGrad [58], GradDrop [13],
CAGrad [32], IMTL-G [33], Nash-MTL [39], Moco [18], Aligned-MTL. [47], SDMGrad [54],
DB-MTL [27], STCH [30], FAMO [31], FairGrad [4] and GO4Align [48].
Metrics. We follow previous works [39, 4] and use two overall performance metries for MTL: (1) Am% the average performance drop relative to the STL baseline: Am% = -E=((-1)) (-1)%; (Ma, My..) 100%, where S is the number of metrics. My.; is the baseline STL Ma.,
metric and Mm. in.: is the metric from the MTL method. &, if higher value is better for Mg. and otherwise. (2) Mean Rank (MR): MR reports the average rank of a method across all tasks, where lower value indicates better performance. A method with the top rank in all tasks has an MR of 1.
4.2 Multi-Task Supervised Learning
Dense Prediction. Widely used benchmarks this domain include NYUv2 [49] and CityScapes [15]. The NYUv2 dataset includes three tasks: semantic segmentation, depth estimation, and surface normal prediction, while SityScapes includes semantic segmentation and depth estimation tasks. The experimental results are presented in Table I and 2 in the main text, and Table 7 in the Appendix.
On the NYUv2 dataset, the difficulty levels of the three tasks show significant variation. Previous methods generally outperform the Single Task Learning (STL) baseline in the tasks of semantic segmentation and depth estimation, but almost all of them consistently underperform STL on the surface normal prediction task, leading to a significant task imbalance in the overall results. In contrast, by leveraging NTK theory to balance convergence speed of each task during training, both NTKMTL and NTKMTL-SR show strong performance in the surface normal prediction task. Notably,
7

---

among all existing methods, only NTKMTL and GO4Align consistently outperform the STL baseline across all three tasks, achieving a more balanced optimization. Moreover, NTKMTL achieves SOTA on this benchmark with an impressive mean rank of 4.33 and the best performance drop of -6.99%
On the CityScapes dataset, NTKMTL. and NTKMTL-SK also perform exceptionally well, as shown in Table 2. More detailed results can be found in Table 7 in the Appendix. Compared to existing methods that tend to prioritize the op- timization of semantic segmentation, NTKMTL and NTKMTL-SR achieve more balanced re- sults across both segmentation and depth esti- mation tasks, with NTKMTI. also obtaining the best performance drop.
Table 2: Results CityScapes (2-task) and CelebA (40-task), Each experiment is repeated 3 times with different random seeds and the aver- age is reported. Detailed standard error is reported in the Appendix. Best scores reported in gray
CITYSCAPES
CELEBA
METHOD LS
MR
Am%
MR
Am% +
8.25
22.60
7.85
4.15
SI
11.50
14.11
9.75
7.20
RLW [28]
10.25
24.38
6.90
1.46
DWA [35]
7.75
8.72
3.20
Image-Level Classitication. We also evalu- ated the performance of our proposed NTKMTL and NTKMTL-SRR on the CelebA dataset. CelebA [36] is large-scale facial attribute dataset consisting of over 200K images, each labeled with 40 attributes, such as smiling, wavy hair, and mustache. This task represents a 40- task MTL classification problem, where each task is designed to predict one binary attribute. This benchmark tests both the collaborative op-
UW [25]
7.75
5.89
7.38
3.23
MGDA [46]
12.00
12.97
PCGRAD [58]
8,50
18.29
8.53
3.17
CAGRAD [32]
7.25
8.10
IMTL- G [33]
5,50
11.10
6.25
0.84
WASH-MTL [39]
3.50
6,50
FAMO [31]
7.75
8.13
6.45
1.21
FAIRGRAD [4]
2.25
5.18
6.92
NTKMTL
7.00
1.92
4.35
-0.77
NTKMTL-SR
6.25
3.84
4.33
0.23
timization capability and efficiency of MTL methods when dealing with a large number of tasks.
The results are shown in Table 2. In this challenging setting, NTKMTL achieves better average performance than the STL baseline, as indicated by the negative Am% which was not achievable by previous methods. NTKMTL also achieved state-of-the-art performance in both MR and Am%.
Fig. 1 visualizes the training time per epoch for various loss-oriented and gradient-oriente methods. On the CelebA dataset with up to 40 tasks, NTKMTL-SR maintains a speed compa- rable to loss-oriented methods.
Loss-oriented
Gradient-orienter
Too (ne0)
5.58k
4.33k
Multi-Tash Regression. QM9 [8] is a widely used benchmark for multi-task regression, con- taining over 130K organic molecules repre- sented as graphs. It includes quiring the prediction of a molecular property. Due to the large number of tasks, as well as the significant differences in task difficulty and convergence speeds, existing MTL. methods exhibit substantial performance drop compared the STL baseline.
1.00x 1. 1.00x 1.08x 1.03x
FMMO
PEBI Cabredd
Redi M/z
tasks, each re-
Figure 1: Training time per epoch for various meth- ods on CelebA (40-task) dataset.
On this benchmark, previous methods were implemented using a shared MPNN codebase [39, 31, 19]. However, we find that the hyperparameter settings in this codebase are suboptimal (specifically, the improper learning rate scheduler makes the learning rate decay too quickly, resulting in incomplete convergence), potentially leading to unfair comparisons. Therefore, we adjusted specific hyperpa- rameters and reproduced all baselines for a fair comparison. To ensure accurate evaluation, we also reproduced the 1-task STL baselines under the same settings.
Experimental results are presented in Table 3. More detailed explanations and settings are provided in Appendix C. As both STL baselines and MTL methods show significant performance improvements under the new settings, we can observe different behaviors among previous methods. The final Am% of some methods (e.g., LS, RLW, PCGrad, CAGrad) were largely consistent with their originally reported values. For other methods (e.g., SI, UW, FAMO, GO4Align), the Am% significantly improved compared to the reported results. This indicates that the previous hyperparameter settings fail to fully exhibit the capabilities of these methods. Under the new settings that better ensure convergence, their performance shows further improvement. On this benchmark, NTKMTL shows competitive results. NTKMTL-SR surpasses NTKMTL in performance and achieves state-of-the-
8

---

Table 3: Results on QM9 (11 (11-task) dataset. All baselines are reproduced under optimized hyperpa- rameter settings. The best scores are reported in gray More details are reported in the Appendix.
METROD
CROWG FANSE
(NP)
ZPVE:
Up
U
MR.
Am% 4
MAE 4
0.060
51.22
0.419
3.08
42.9
41.7
43.1
0.061
L.S
0.077
0.253
55.95
68.39
4,163
11.08
109.2
109.8
110.1
106.7
0.099
9.82
179.8
SI
0.159
0.242
109.6
96.80
0.732
3,308
34.35
34.37
34.33
35.16
0,081
5.27
39.7
RLW [28]
76.39
4,948
12.54
124.9
122.1
DWA[35] [35]
0.078
0.239
55,17
67.10
3,992
10.92
107.4
108.1
108.3
105.4
0.098
8.73
173.0
UW [25]
0.194
0.274
120.6
102.8 8
0.763
3.698
41.11
41.13
41.16
41.75
0.089
8.27
58.4
MGDA [46]
0.154
0.266
95.20
67.51
3.068
4.468
49.38
49.21
49.62
49.69
0,087
8.36
101.4
PCGRAD [58]
59.14
67.82
2.937
$8.24
88.65
0.081
118.6
CAGRAD [32]
0.083
0.234
57.80
70.98
2.718
5.352
76.47
76.93
77.05
76.32
0.089
8.27
102.4
NASH MTL [39]
0.086
0.218
69.78
66.18
2.153
4.679
59.63
59.94
59.98
59.97
0.082
6.91
72.9
FAMO
0.128
0.230
98.09
84.42
0.859
3.541
40.24
40.57
40.62
40.21
0,081
38.9
FAIRGRAD [4]
0.109
0.208
81.74
72.82
1.669
3.418
51.31
51.67
51.72
51.97
0.079
GOLALIGN [48]
0.113
0.314
74.46
91.04
0.912
3.632
36,06
36.38
36.41
36.58
0.104
6.64
40.5
NTKMTL
0.091
0.212
70,97
70.81
2.113
3.835
44.18
44.56
44.53
14.38
0.077
5.91
56.7
NTKMTL-SS
0.081
0.207
75.95
69.10
1.176
3.689
40:14
40.46
40.48
40.49
0.074
30.7
results, which we attribute to the natural alignment of the L.2 loss used in regression tasks as analyzed in Appendix B.1. As result, larger n produces a more accurate estimate of convergence speed.
4.3 Multi-Task Reinforcement Learning
We further assess our method using the MT10 benchmark, which consists of 10 robotic manipulation tasks from the MetaWorke environment [59]. The goal in this setting is to train single policy that can generalize across a variety of tasks, including mick-and-place and door-opening. Specifically, we follow the setup outlined in previous works [39, 31] and use Soft Actor-Critic (SAC) [21] as the core
algorithm. Our implementation builds upon the MTRL codebase from [39, 4], training the model for 2 million steps with batch size of 1280.
In contrast to the multi-task supervised learning networks where shared and task-specific parameters can be easily distinguished, the MTRL problems commonly involve learning single pol- icy. Consequently, it is difficult to partition the parameters into shared and task-specific components in the same manner. Therefore, while applying NTKMTL-SK to this scenario is chal- lenging, we primarily validate the performance of NTKMTL. We compare NTKMIL with several -tate-of-thheart methods, including Multi-task SAC (MTL. SAC) [59], Multi-task SAC with Task Encoder (MTL SAC + TE) [59], Multi-headed SAC
Table 4: Results on MT10 bench- mark across 10 random seeds.
METHOD
SUCCESS (MEAN + STDERR)
0.90
MTL SAC [AC[59]
0.49 + 0.07
MTL SAC [59]
+ 0.
MH SAC [59]
0.61 = 0.04
PCGRAD [58]
0.72 0.02
CAGRAD [32]
0.83 4: 0.05
MoCo [18]
0.75 + 0.05
NASH-MMTI [39]
0.91 0.03
(MH SAC) [59], PCGrad [58], CAGrad [32], MoCo [18], Nash-
FAMO [3]]
0.83 J. 0.05
MTL [39], FAMO [31], FairGrad [4] and Aligned-MTL. [47]. The experimental results are reported in Table 4. In the MTRL scenario, NTKMTL continues to demonstrate strong performance with a competitive success rate.
FAIRGRAn
0.84 + 0.07
ALIGNED-MTL (47)
0.97 1 0.05
NTKMTL
0.96 0.03
5 Conclusion, Limitations and Future Work
In this paper, we introduce a new perspective on understanding task imbalance in MTL by leveraging NTK theory for analysis. and propose a new MTL method, NTKMTL. Specifically, we conduct spectral analysis of the NTK matrix during training, adjust the maximum eigenvalues of the task- specific NTK matrices to balance their convergence speeds, thereby mitigating task imbalance. Furthermore, we present NTKMTL--SR, an efficient variant based on approximation via shared representation, which achieves competitive performance with improved training efficiency. Extensive experiments have shown the strong performance of both NTKMTL and NTKMTL-SR, further demonstrating the applicability and generalization of our method across wide range of scenarios.
Limitations and Future Work. In this work, we proposed the extended NTK IC for MTL as tool for more comprehensive analysis of training dynamics across tasks. The weight design of our current method mainly focuses on the analysis of the task-specific NTK matrices CC.. The full structure

---

of the extended NTK matrix K offers further analytical opportumities. For example, analyzing the off-diagonal K,, blocks could reveal novel insights into task interactions or guide the development of task grouping strategies. These potential avenues are left for future investigation.
10

---

References
[1] Idan Achituve, Haggai Maron, and Gal Chechik. Self-supervised learning for domain adaptation on point clouds. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 123-133, 2021.
[2] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparametterized two-layer neural networks. In International Conference on Machine Learning, pages 322-332. PMLR, 2019.
[3] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12):2481-2499 2017.
[4] Hao Ban and Kaiyi Ji. Fair resource allocation in multi-task learning. arXiv preprint a4X2 15638, 2024.
[5] Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias neural networks for input of non- con-uniform density. In International conference on machine learning. pages 685-694. PMLR, 2020.
[6] Jonathan Baxter. A model of inductive bias learning, Journal of artificial intelligence research, 12:149-198, 2000.
[7] Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems, 32, 2019.
[8] Lorenz C Blum and Jean-Louis Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database gdb- 13. Journal of the American Chemical Society, 131(25))8732-773 8733, 2009.
[9] Rich Caruana. Multitask learning. Machine learning, 28:41 1997.
[10] Shijie Chen, Yu Zhang, and Qiang Yang. Multi-task learning in natural language processing: An overview. ACM Computing Surveys, 56(12):1-32, 2024.
[11] Weiyu Chen, Xiaoyuan Zhang, Baijiong Lin, Xi Lin, Han Zhao, Qingfu Zhang, and James Kwok. Bradient-based multi-objective deep learning: Algorithms, theories, applications, and beyond. arXiv preprint arXiv:2501.10445 2025.
[12] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794-803. PMLR, 2018
[13] Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039-2050, 2020.
[14] Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh. Multinet+:: Multi-streas stream feature aggregation and geometric loss strategy for multi-tas learning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pages 0-0, 2019.
[15] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223 2016.
[16] Coline Devin, Abhishek Gupta. Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learn- ing modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE international conference on robotics and automation (ICRA), pages 2169-2176. IEEE, 2017.
[17] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arX1::1810.02054, 2018.
11

---

[18] Heshan Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. Mitigating gradient bias in multi-objective learning: provably convergent approach. International Conference on Learning Representations, 2023.
[19] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International conference on machine learning. PMLR, 2017.
pages
[20] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision (ECCV), pages 270 287, 2018.
[21] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[22] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[23] Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In International Conference Learning Representations.
[24] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1482-7491, 2018
[25] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482-7491, 2018.
[26] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
[27] Baijiong Lin, Weisen Jiang, Feiyang Ye, Yu Zhang, Pengguang Chen, Ying-Cong Chen, Shu Liu, and James Kwok. Dual-balancing for multi-task learning. arXiv preprint arXiv:2308.1 12029, 2023.
[28) Baijiong Lin, Feiyang Ye, and Yu Zhang. A closer look at loss weighting in multi-task learning. 2021.
[29] Baijiong Lin, Feiyang Ye, Yu Zhang, and Ivor W Tsang. Reasonable effectiveness of random weighting: A litmus test for multi-task learning. arXiv preprint arXiv:2111.10600 2021.
[30] Xi Lin, Xiaoyuan Zhang, Zhiyuan Yang, Fei Liu, Zhenkun Wang, and Qingfu Zhang. Smooth tchebycheff scalarization for multi-objective optimization. arXiv preprint 1977 2024.
[31] Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization. Advances in Neural Information Processing Systems, 36, 2024.
[32] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Confliet-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34: 1888888888880 2021.
[33] Liyang Liu, Yi Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. iclr, 2021.
[34] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classification. arXiv preprint arXiv: 1oXXv:1704.080 1704.05742, 2017.
[35] Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1881-1880, 2019.
12

---

[36] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
[37] Keerthiram Murugesan and Jaime Carbonell. Self-paced multitask learning with shared knowl- edge. arXiv preprini arXiv: 1eto:0003777777 2017.
[38] Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. arXiv preprint arXiv:2007.02693, 2020.
[39] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game, arXiv preprint arX10:2202.01017, 2022.
[40] Jonathan Pilault, Amine Elhattami, and Christopher Pal. Conditionally adaptive multi-task learning: Improving transfer learning in nlp using fewer parameters less data. arXiv preprint arXiv:2009.09133 2020.
[41] Xiaohan Qin, Xiaoxing Wang, and Junchi Yan. Revisiting fairness in multitask learning: performance-driven approach for variance reduction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20492-20501, 2025.
[42] Xiaohan Qin, Xiaoxing Wang, and Junchi Yan. Towards consistent multi-task learning: Un- locking the potential task-specific parameters. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10067-10076, 2025.
[43] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. International conference machine learning, pages 5301-5310. PMLR, 2019.
[44] Daniel Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory, volume 46. Cambridge University Press Cambridge, MA, USA, 2022.
[45] S Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 2017.
[46] Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. Advances in neural information processing systems, 31, 2018.
[47] Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent component alignment for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20083-20093, 2023.
[48] Jiayi Shen, Cheems Wang, Zehao Xiao, Nanne Van Noord, and Mareel Worring. Go4align: Group optimization for multi-task alignment. arXiv preprint arXiv:2404. 2444022404006686, 2024.
[49] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rghd images. In Computer Vision-ECCV 2012: 12th European Conference Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746-760. Springer, 2012.
[50] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? International conference on machine learning, pages 9120-9132, PMLR, 2020.
[51] Vincent Szolnoky, Viktor Andersson, Balázs Kulesár, and Rebecka Jörnsten. On the inter- pretability of regularisation for neural networks through model gradient similarity. Advances in Neural Information Processing Systems, 35:16919-16630, 2022.
[52] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. Advances neural information processing systems, 33:7537-7547, 2020.
13

---

[53] Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics. 449:110768, 2022.
[54] Peiyao Xiao, Hao Ban, and Kaiyi Direction-oriented multi-objectiiv learning: Simple and provable stochastic algorithms. Advances in Neural Information Processing Systems, 36:4509 4530, 2023.
[55] Caiming Xiong, SHU Tianmin, and Richard Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning, January 24 2023. US Patent 11,562,287.
[56] Enneng Yang, Junwei Pan, Ximei Wang, Haibin Yu. Li Shen, Xihua Chen, Lei Xiao, Jie Jiang. and Guibing Guo. Adatask: A task-aware adaptive learning rate approach to multi-task learning. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 10745-10753, 2023.
[57] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836. 2020.
[58] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 5836, 2020.
[59] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100, PMLR, 2020.
[60] Hayoung Yun and Hanjoo Cho. Achievement-basee training progress balancing for multi-task learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16935-16944, 2023.
[61] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions on knowledge and data engineering, 34(12):5586-560) 2021.
[62] Zelin Zhao, Fenglei Fan, Wenlong Liao, and Junchi Yan. Grounding and enhancing grid-based models for neural fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19425-19435, 2024.
[63] Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang. Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. Deep learning- human pose estimation: A survey, ACM Computing Surveys, 56(1):1-37, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: The main claims are clearly presented in the abstract and introduction, and further elaborated and substantiated throughout the subsequent sections of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Refer Sec. 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
Refer to Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In paper, we provide detailed descriptions of each step of the algorithm, the pseudocode, and all settings required to replicate the experiments. Additionally, we commit to making the code publicly available by the time of the camera-ready submission.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Source code is provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
sooooiiiccsooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wttps.ccpppblccc//cccccccpcppppp geiic//gguee...ee CoodeSummii agionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Refer Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Refer to Appendix. C.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were performed on single NVIDIA RTX 4090. We also compared the training time in Fig. 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///nuripppe cttp:blicc/cccccccccc.cccccccclt hett//tiiissaeeeeeee cesSuuddeiiness
Answer:
[Yes]
Justification: This paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Refer to Sec. D.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators and original owners of all assets used in the paper are properly credited, and the licenses and terms of use are explicitly mentioned and fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https : http://ies...//eeeeeeeeee http///rrs..c//ooeeeeeeeeeeeeeee / for what should or should be described.
21

---

A Definitions of Notations
Due to the numerous concepts and theoretical derivations presented in the paper, we provide detailed definitions of the notations in Table assist readers better understanding the content.
Table 5: Definitions of notations in this paper.
Variable
Definition
the number of tasks
the mapping function of a deep neural network
fi
the mapping functtin of i-th task in MTL scenario
7
the number of data points the dataset, or the number of mini-batches in batch
(x,y)
training dataset {xx; yo) T in Single- Taake--Taskka Learning (STL) setting
(y
multiple labels for the tasks in MTL setting
the network parameters, specifically referring to the shared parameters in MTL
t
the current time step
K
the Neural Tangent Kernel matrix defined in Eq. 1
the eigenvalues of the NTK
C
the overall loss function, with different definitions in different scenarios
(4))-
the losses with respect to k tasks
O(t)
the output [2, (())) of the network in STL setting and time t
{O.())
the output of the network for tasks MTL setting and time t the Identity matrix
the learning rate at time
K
the extended NTK matrix defined in Eq. 9
J
the Jacobian matrix of f, with respect to 0
w
{w.) representing the weights the different tasks
the shared representation of the input data MTT setting
B Theoretical Analysis
B.1 Proof of Theorem 3.1
Theorem 3.1. Let O(t) x1)) be the outputs of the neural network time x (x:) is the input data, and = Hi is the corresponding label, Then Ott) follows this evolution:
do(I) K.. (0(1) y)- di
Proof. Following previous works [44, 52, 62], we build up the analysis framework for NTK in supervised regression setting. The overall loss L(0) is defined as:
m
((0) = Σ (S(0,z)) y2)2
(19)
22

---

Through the gradient flow in Eq. 2, we can obtain:
d0 dt
V&L(0)
12
OC 0f(0,z))
df(0,xz)
09
(20)
0f(0,z:)
(f(0, -y)).
00
In fact, for the commonly used cross cross-entropy loss, the above equation also holds. In this case, considering a data point 2y, we have
df (0, df (0,xz)) x.y)
do
dt
dt
df (8,0,)) 23)) d0
0f(0, xi)
(f(0, xz) -8))
00
(21)
df (0, xf,
(f(0, yy)).
d0
Given that (tt) {ffx,, (tt)) and y f we can express Eq. 21 in vector form:
dO(t) K (O(t) y),
where K. is defined
Kuv (0f(0.z.)
(f x.))
00
00
Q.E.D.
Discussions. Another equivalent definition of NTK given the Jacobian matrix is:
JJ
(22)
This theorem shows that NTK connects the error term O(1) y the changing rate of the output. Therefore, the theory can be used to analyze the training behaviors of neural networks.
B.2 Proof of Theorem 3.2
Theorem 3.2. Let (2()) (t)} denote the outputs of the neural network function ffr,fe.... f2..... } for the task at time t, and let N2.......3 represent the corresponding labels. Then, the ordinary differential equation in Eq. 2 gives the following evolution:
d dt
K11
Kik
OO - -y
dOa(t) dt
KN
KR
Ox(t) ya.
where Kij E Raxn and Kij = Kor Si.j S k. The (u, v)-th entry of Kig is defined as
(0.(0,,z.) x.) (0,
(KGz)uv
00
00
Proof. In multi-task learning, neural network with shared parameters 0 is trained to simultancously leam distinct tasks. In the general case, the overall loss function is defined
((0)
(())).
(23)
23

---

Similar to the derivation in Theorem 3.1, by utilizing the gradient flow in Eq. 2, we can derive:
d0
dt VoC(0)
&
n
0l
0f(
if.(0, x.)
09
(24)
@f. x.)
(fi σn)
Vi.u),
00
where Vi represents the ground truth label of the u-th element in the label set yı. For data point T, and the function [j of task j. we have:
df, df,(0,zz))
(6,(6,z))
de
dt
d6
dt
(0,
x.)
0f((0xz))
-(f(0, -Vh,u)
d0
00
(25)
u=1
(0, Tee)
0f((0, zu)
(fi(0. -Vi.a).
d0
00
M=1
Rewriting Eq. 25 the form of high dithessinnsii vectors gives:
do, (0)
Kn
KI
Ot - y1
de
dOa(t)
KK1
Ksk
Ox(t) YA
de
where Kig Ruxn and Kij -c.. for < i,j S k. The (u, wi-th entry of Kij is defined as
(Kii)uu (0f(00 x.. (0,
00 00
Q.E.D.
Discussions. It can be observed that Theorem 3.2 essentially extends the foundational NTK theory from Theorem 3.1 the MTL scenario. In this case, the input, output, and NTK matrix each acquire an additional dimension, corresponding to the task-oriented dimension.
B.3 Proof of Proposition 3.3
Proposition 3.3. (Extension of Theorem 3.2) Let (O,(1) (y:) and (Kis ) Jggigs be defined as in Theorem 3.2. We now replace the MTL optimization objective with the weighted form as presented in Eq. 12. Consequently, the ordinary differential equation governing the MTL training dynamics in Eq. becomes:
(0,(t)
WIWaKin
TOi()) yi
dt
(26)
dO
KRR
Ox(1) YE
dt
Proof. Consider the weighted loss
2(0) = Σ w,l,(0),
(27)
1=1
24

---

the analysis in Appendix B.2 becomes
d0
VoL(O)
dt
k
Ow:Ci
0f:(0,
0f((0,zw) x.))
(28)
0f((0,z)) 00
(fi(0, (.) Vi.u).
Under the action of w, the network output {O1..... O transformed to (wifi......w wefk]. Con- sider the output of the network for the j-th task at data point z.:
dweg (,,55(00x,)) = wwj dt
(f, x.))
d0
d0
dt
(0, x.2)
0f((0, F.))
(f((0, - -Bhu) (29)
Wi
d0
00
u=1
(f, x.)
(0,
WjW/
(fi(0.zu) Bi,.).
08
Then we have
(t)(t)
TO((1) - yy
de
do, (1) WawniKa
KRKKK
On(t) Yk
di
OR
Q.E.D.
Discussions. Proposition 3.3 actually provides an intuition: by adjusting the relative magnitudes of [wi)e_ one can alter the eigenvalue distribution of the NTK matrix, thereby balancing convergence speeds. This serves as the foundation for the design of NTKMTL and NTKMTL-SR.
C Detailed Experimental Results
C.1 Detailed Results on QM9
For QM9 benchmark, previous methods were all implemented based on a shared codebase [39, 31, 4], utilizing the message-passing neural network (MPNN) architecture [19]. All the methods were trained for 300 epochs. However, we found that the hyperparameter settings in this codebase were suboptimal (specifically, the improper learning rate scheduler makes the learning rate decay too quickly). Consequently, the reported results of most previous methods had not fully converged, potentially leading to unfair comparisons. Therefore, we readjusted the hyperparameters: changed the batch size from 120 to 60 and the learning rate scheduler's patience from 5 to 10. The remaining hy ayperparameters were kept unchanged, consistent with the original codebase. Subsequently, we reproduced all baselines that had reported results on QM9, and their performance on various tasks significantly improved under these revised settings. To ensure accurate evaluation, we also reproduced the 11 1-task STL. baselines under the same settings.
Experimental results are presented in Table 6. For each baseline, the upper row shows results taken from its original paper, while the bottom row (highlighted in green) ) presents results reproduced by us under new hyperparameter settings. It can be seen that without changing the model architecture, solely by adjusting the learning rate schedule, the performance of all baselines on the 11 tasks has significantly improved, notably extending the Pareto front.
Under the new settings, with both STL baselines and MTL methods showing significant performance improvements, we observed different behaviors among previous methods. The final Am% of some methods (e.g., LSS RLW, PCGrad, CAGrad) were largely consistent with their originally reported values. For other methods (e.g., SI. UW, FAMO, GO4Align), the Am% significantly improved
25

---

Table 6: Results QM9 (11-task) dataset. For each baseline, the upper row shows results taken from its original paper, while the bottom row (highlighted in green) presents results reproduced by us under new hyperparameter settings.
METHOD
J
CHOMO
GUMO
(R))
ZPVE
Un
U
6
MRJ
Am%
MAE
STL
0.067
0.181
60.57
53.91
0.502
4.53
58.8
66.2
NTT.
0.060
0.156
60.54
51.22
0.419
3.08
19.3
42,9
41.7
43.1
0.061
1.8
0.106
0.325
73.57
89.67
5.19
14.06
143.4
144.2
144.6
140.3
0.128
177.6
0.077
0.253
55.95
68.59
4.163
11.08
109.2
109.8
110.1
106.7
0.099
9.82
SI
0.309
0.345
149.8
135.7
1.00
4.50
55.3
55.75
55.82
55.27
0.112
77.8
SI
0.159
0.242
109.6
0.732
3.308
34.35
34.37
34.33
35.16
5.27
39.7
RIW [28]
0.113
0.340
76.95
92.76
5.86
15.46
156.3
155.1
157.6
1.53.0
0.137
203 8
RLW [25]
0.090
0.277
62.49
76.39
4.948
12.54
124.8
124.9
125.0
122.1
0.135
12.09
222.6
DWA [35]
0.107
0.325
74.06
90.61
5.09
13.99
142.3
143.0
143.4
133.33
0.125
175.3
DWA [35]
0.078
0.2339
55.17
67:40
3.992
10.92
107.2 4
108.1
10%. 3
105.4
0.099
8,73
173.0
UW [25]
0.386
0.425
166.2
4.99
66.4
66.78
66.80
66.24 24
0.122
UW [25]
0.194
0.274
120.6
102.8
0.763
3.698
41.11
11.13
41.16
11.75
0.089
8.27
58.1
MGDA [16]
0.217
0.368
126.8
104.6
3.22
5.69
88.37
89.4
89.32
88.01
0.120
120.5
MGDA [46]
0.154
0.266
95.20
3.088
49.38
49.21
49.62
49.69
0.087
8.36
PCGRAD [58]
0.106
0,293
75.85
88.33
3.94
9.15
116.36
116.8
117.2
114.5
0.110
125.7
PCORAD [58]
0.078
0.221
59.14
67.82
2.937
6.691
88.21
88.65
88.85
87.36
0.084
7.91
118.6
CAGRAD [32]
0.118
0.321
83,51
94.81
3.21
6.93
113.95 99
114.3
114.5
112.3
0.116
112,8
CAGRAD [32]
0.083
0.234
57.80
70.98
2.718
5.352
76.47
76.93
77.05
76.32
0.089
8.27
102.4
MTL [39]
0.102
0.248
82.95
2.42
5.38
75.02
75.10
74.16
62.0
NASH--MII (39)
0.086
0.218
69.78
66.18
2,153
4.679
59.63
59.94
59.98
59.97
0.082
6.91
72.9
FAMO [31]
0.15
0.30
94.0
95.2
1.63
4.95
70.82
71.2
71.2
70.3
0.10
58.5
FAMO [31]
0.128
0.230
98.09
40.57
40.62
5.91
38.9
PAIRGRAD [4]
0.117
0,253
87.57
84.00
2.15
5.07
70.89
71.17
71.21
70.88
0.095
$7.9
FAIRGRAD
0.109
0.208
81.74
72.82
1.669
3.418
51.31
51.67
51.72
51.97
0.079
6.64
57.0
GOLALIGN [48]
0.17
0.35
102.4 4
119.0
1.22
4.94
53.9
54.3
54.3
53.9
0.11
52.7
GO4ALIGN [48]
0.113
0.314
74.46
91.04
0.912
3.632
36.06
36.38
36.41
36.58
0.104
6.64
40.5
NTKMTL
0.091
0.212
70.97
70.81
2.113
3.835
44.18
44.56
44.53
44.3% 38
0.077
5.91
56.7
NTKMTTI-SS
0.081
0.207
75.95
69.10 10
1.176
3.689
40. 14
40.46
40.48
40 49
0.074
4.00
30.7
compared to the reported results. This indicates that the previous hyperparameter settings fail fully cxhibit the capabilities of these methods. Under the new settings that better ensure convergence, their performance shows further improvement. Notably, the traditional Scale-Invarian Linear Scalarization (SI) demonstrated extremely superior performance, surpassing the vast majority of recent baselines. This suggests that in cases where differences in loss scales among different tasks are too large, directly eliminating scale differences through logarithmic methods may be an effective solution.
In summary, we identified that the parameter settings in the previous code implementation hindered the smooth convergence of both STL and MTL methods, rendering comparisons made under these conditions unfair. Our experiments verified that this issue can be resolved by simply adjusting the batch size and learning rate scheduler parameters. Under the new settings that better ensure convergence, the performance of both STL methods and numerous MTL methods on the 11 tasks has significantly improved. We believe that fair and transparent reproduction of all previous baseline methods under such parameter settings enables more reasonable comparisons on this benchmark, providing new results that are valuable to the MTL community.
C.2 Detailed Results with Standard Errors
In this section, We provide the detailed experimental results with standard errors for our method, and the results for the baseline methods are taken from their original papers. Since results for CelebA are not reported by some methods [47, 54, 48], these methods are excluded when presenting combined CityScapes and CelebA results in Table 2. Table 7 provides detailed results solely on the SctySe dataset, including these methods. Consequently, the mean rank (MR) in Table 7 slightly differs from that in Table 2.
On NYUv2 and CityScapes, we follow the training settings of [39, 4], including data augmentation for all compared methods. Training runs for 200 epochs, with the learning rate initialized at 10-1 and reduced x 10-5 after 100 epochs. The architecture the beg [3] Multi-Task, Attention Network (MTAN) [33]. Batch sizes are 2 (NYUv2) and 8 (CityScapes), and the hyparameter
26

---

Table 7: Detailed results on CityScapes (2-task) dataset. Each experiment is repeated 3 times with different random seeds and the average reported. The best scores are reported in gray.
CITYSCAPES
SEGMENTATION
MR
Am +
MIOU
PIX Acc
ABS ERR
REL ERR
STL
74.01
93.16
0.0125
27.77
LS
75.18
33.49
0.0155
46.77
10.25
22.60
70.95
91.73
0.0161
14.00
RLW [28]
74.57
93.41
0.0158
47.79
12.25
24.38
DWA [35]
75.24
93.52
0.0160
44.37
9.75
21.45
UW [25]
72.02
92.85
0.0140
30.13
10.00
5.89
MGDA [46]
68.84
91.54
0.0309
33.50
14.75
44.14
PCGRAD [58]
75.13
93.48
0.0154
42.07
10.50
18.29
CAGRAD [32]
75.16
93.48
0.0141
37.60
9.25
11.64
IMTL-G [33]
75.33
93.49
0.0135
38.41
7.25
11.10
NASH-MIL [39]
75.41
93.66
0.0129
35.02
4.75
6.82
[31]
0.0145
ALIGNED LLGNNED-MTL.O [47]
75.77
93.69
0,0133
32.66
3.00
5.27
[54]
GO4ALIGN [48]
72.63
93.03
0.0164
27.58
10.75
8.13
FAIRGRAD [4]
75.72
93.68
0.0134
32.25
3,25
5.18
NTKMIL
92.71
0.0136
NTKMTL.SR
72.58
92.93
0.0124
31.65
8.00
3.84
Table 8: Results on CityScapes (2 tasks) and CelebA (40 tasks) datasets. Each experiment is repeated over 3 random seeds and the mean and stderr are reported.
CiaySs
CelebA
Method
Segmentation
Depth
Amr +
Am 4
mloU
Pix Acc f
Abs 1
Rel Err
NTKMTL. (mean)
73.71
92.71
0.0136
27.21
1.92
-0.77
NTKMTL (stderr)
10.17
40.15
10,0005
10.23
+0.30
10.37
NTKMTL- SR (mean)
0.0124
0.23
NTKMTL-SC (stderr)
10.32
40.23
10.0004
20.35
+0.37
10.46
Table 9: Results NYU-v2 dataset (3 tasks). Each experiment is repeated over 3 random seeds and the mean and stderr are reported.
Segmentation
Depth
Surface Normal
Method
mloU*
Acc
Abs Err
Rel
Angle Dist
Within 1
Am%
Mean
Median
11.25
22.5
30
NTKMIL (mean)
0.5296
0.2168
24.24
18.63
30.74
58.72
70.78
-6.99
NTKMTL (siderr)
=0.51
#0.21
5000008
+0.0011
40.07
+0.09
20.14
10.19
10.19
=0.38
NTKMTL-SR (mean)
40.23
65.28
0.5261
0.2136
24.88
19.58
29.53
56.67
69.08
-5.35
NTKMTL- (stderr)
10.42
1..26
100.0133
40.0014
10.13
for NTKMTL-SR on NYUv2 is set to 2. Our setup for the CelebA benchmark aligns with the configuration detailed in [31]. We employ 9-layer CNN as the network backbone, coupled with separate linear layers for each task. The method is trained for 15 epochs; optimization is carried out using Adam with a batch size of 256.
C.3 Visualization experiments for the NTK eigenvalues during training
To support the proposed theory, we conducted experiments on NYUv2, visualizing the change in the maximum eigenvalue of the NTK matrix for the three tasks during training under linear scalarization (i.e., equal weighting). On the NYUv2 dataset, difficulty levels of the three tasks show significant variation. Previous methods generally outperform the Single Task Learning (STL) baseline in segmentation and depth estimation tasks, but almost of them consistently underperform STL on the surface normal prediction task, leading to significant task imbalance in the overall results.
27

---

The largest eigenvalue of NTK
10
5
1
0.5
nm
0.25
0.25
Segmentation
Depth
0
Normal
2000
3000
4000
5000
Iteration
Figure 2: Visualization for the NTK eigenvalues during training.
As shown in Fig. throughout training, the largest eigenvalue corresponding surface normal prediction remains substantially smaller than those of segmentation and depth estimation. Given that the maximum NTK cigenvalue reflects task's convergence speed, this observation aligns with the empirical finding that many existing MTL algorithms struggle to converge on surface normal prediction. Although some prior methods (e.g.. MGDA) attempt to prioritize the most difficult tasks, their performance in surface normal prediction tasks remains unsatisfactory due to the challenge in accurately quantifying the "difficulty" and "convergence speed" different tasks. In contrast, by leveraging NTK theory to accurately characterize and balance the convergence speed of each task during training, NTKMTL delivers SOTA results on surface normal prediction and one of only two methods that outperform single-task learning all three tasks.
C.4 Ablation Study on the Hyperparameter n
For NTKMTL-SR, the computational cost of calculating the NTK is minimal, allowing us to further investigate the impact of varying mini- batch sizes the results. Therefore, set n to 2, 3, 4, 6] and conduct an ablation study on the QM9 (11-task) benchmark. For each value of n, we conduct repeated experiments with different random seeds and calculate the mean and variance for Am%. The results are shown in Fig. 3. When n = 1, the performance of NTKMTL-SR is comparable to that of NTKMTL. However, when increasing from 1 to 2 or more, NTKMTL-SE shows noticeable improvement in performance, accompanied by a reduction in the variance of performance across repeated experiments. We attribute this to the fact that increasing the number of mini-batches leads to larger NTK matrix dimension, which in turn reduces stochastic error and allows our method to more accurately characterize the convergence speed of the tasks.
Am1%
30
20
Hyperparameter
Figure 3: Ablation study on hyperparameter on QM9, Each experiment is repeated over 3 random seeds, and the mean and stderr are reported.
28

---

However, we also find that the results for n 4 and n 6 are almost identical, and the performance differences observed were potentially weaker than the inherent variability stemming from different random seeds. Concurrently, Fig. 4 visualizes the training time per epoch for various n. values on QM9. Despite only requiring the computation of the maximum eigenvalue of the NTK matrix with respect to training a single epoch when n 6 already approached 1.7 times the duration of the LS method. Overall, we posit that the selection of hyperparameter n is trade-off between performance and training speed, and n 4 generally presents favorable compromise.
3.0
1.69x
2.5
1.40x
1.27x
Time (min)
2.0
1.14x
1.05x
1.5
0.5
0.0
n=1
n=2
n=4
n=6
Figure 4: Training time per epoch for LS and NTKMTL-SR (with different hyperparameter n) on QM9.
D Impact Statement
This work aims to employ NTK theory to investigate the training dynamics multi-task learning and address the task imbalance issue. We acknowledge the potential implications of our findings on fairness machine learning systems. Our proposed methods aim to reduce task imbalance and enhance performance equity across different tasks, thereby mitigating biases that may arise in multi-task learning frameworks. We are committed to transparency and responsible dissemination of results, and we encourage further exploration of the ethical implications of our methodologies.
29


Reviews:
[{"review_id": "IOJDGHFPIc", "reviewer": "Reviewer_Gi8p", "summary": "This paper addresses the common issue of task imbalance in Multi-Task Learning (MTL) and proposes a novel optimization framework called NTKMTL based on Neural Tangent Kernel (NTK) theory. The method analyzes the training dynamics of multiple tasks by extending the NTK matrix, utilizes spectral analysis to evaluate the convergence speed of each task, and designs a task weight allocation strategy based on the maximum eigenvalue of the NTK to balance the training speeds of different tasks, thereby alleviating task imbalance. Furthermore, the authors introduce an approximate variant, NTKMTL-SR, which leverages Shared Representation to reduce computational overhead. Experiments were conducted on multi-task supervised learning, multi-task regression, and multi-task reinforcement learning across several mainstream datasets, achieving better performance.", "questions": "1. What are the reasons why the proposed method achieves better results compared to other approaches?\n2. Are there any modules or parameter choices in the proposed method that are particularly critical?\n3. What are the reasons for the varying performance of the proposed method across different tasks?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method is reasonable, and the task weighting strategy based on the maximum NTK eigenvalue has a clear meaning. The weights are inversely proportional to the convergence speed of the tasks, effectively adjusting the learning speed between tasks.\n2. The experiments are comprehensive and diverse, covering multiple supervised and reinforcement learning tasks such as NYUv2, Cityscapes, CelebA, QM9, and MT10. The evaluations are thorough, showcasing the broad adaptability of the proposed method.\n\nWeaknesses:\n1. This paper lacks more in-depth analysis and comparison. It mainly describes the rationale of the proposed method and the final accuracy achieved. In practice, balancing tasks based on gradients is a common approach, and other works also have their own validity. When compared with other studies, the paper lacks deeper theoretical or experimental analysis to explain why the proposed method is superior.\n2. This paper lacks ablation studies. Although extensive experiments were conducted on various datasets, the results only compare the overall performance of the proposed method. It lacks ablation studies to validate the contributions of specific modules and the choices of certain parameters.\n3. Although this paper shows improvement on certain datasets, the improvement is not particularly significant, and the method does not achieve the best performance on many subtasks. A deeper analysis of the underlying reasons behind these experimental results would make the work more robust and comprehensive.", "was_revised": true, "final_justification": "Most of my concerns have been resolved, and I will maintain my rating.", "rebuttal": "We appreciate your recognition of our method's rationality, the comprehensiveness and diversity of our experiments, and the broad adaptability of the proposed method. Thank you for your careful and valuable comments! We hope to address your concerns by answering your questions below.\n\n**Q1: Why does the proposed method achieve better results compared to other approaches (e.g. other methods that balance tasks based on gradients)?**\n\n**A1:** The superior performance of the proposed NTKMTL can be attributed to its ability to effectively balance the convergence rates of individual tasks, which fosters more equitable optimization and ultimately yields enhanced overall performance.\n\n**Why balanced optimization counts:** Recent works (e.g., FairGrad[1] and GO4Align[2]) have demonstrated that more balanced cross-task performance typically yields better overall results (See Fig.2 in [2]).  To more clearly elucidate the reason for NTKMTL's superior performance, we analyze it using the NYUv2 benchmark as an example. On the NYUv2 dataset, the difficulty levels of the three tasks show significant variation. Previous methods generally outperform the Single Task Learning (STL) baseline in the tasks of semantic segmentation and depth estimation, but almost all of them consistently underperform STL on the surface normal prediction task, leading to a significant task imbalance in the overall results. Although some prior methods (e.g., MGDA) attempt to prioritize the most difficult tasks, their performance in surface normal prediction tasks remains unsatisfactory due to the challenge in accurately quantifying the \"difficulty\" and \"convergence speed\" of different tasks. In contrast, by leveraging NTK theory to accurately characterize and balance the convergence speed of each task during training, NTKMTL demonstrates strong performance in the surface normal prediction task. Notably, among all existing methods, only NTKMTL and GO4Align consistently outperform the STL baseline across all three tasks, achieving a more balanced optimization.\n\n[1] Ban H, Ji K. Fair resource allocation in multi-task learning[J]. International Conference on Machine Learning, 2024.\n\n[2] Shen J, Wang Q, Xiao Z, et al. Go4align: Group optimization for multi-task alignment[J]. Advances in Neural Information Processing Systems, 2024.\n\n\n**Experimental validation:** To further substantiate this, we conducted additional experiments and visualized: **1)** the maximum eigenvalue of the NTK matrix for different tasks during NYUv2 training, and **2)** the relationship between cross-task performance variance and $\\Delta m\\\\%$ for various methods on NYUv2. As images cannot be uploaded, we present these findings in the tables below.\n\n**1). The maximum eigenvalue of the NTK matrix for different tasks during NYUv2 training.**\n\n| Task    | avg over 1000 | avg over 2000 | avg over 3000 | avg over 4000 | avg over 5000 |\n|--|-|--|-|--|-|\n| segmentation          | 2.52   | 2.35          | 2.92          | 3.50          | 3.76          |\n| depth prediction      | 2.71          | 2.27          | 2.41          | 2.68          | 2.87          |\n| surface normal prediction | 0.38      | 0.30          | 0.33          | 0.37          | 0.39          |\n\nIn the table above, we report the evolution of the largest eigenvalue of the NTK matrix for each task when utilizing conventional linear scalarization method. ‘Avg over xxxx’ denotes the mean of the maximum NTK eigenvalues during the first xxxx iterations. Throughout training, the largest eigenvalue corresponding to surface-normal prediction remains substantially smaller than those of segmentation and depth estimation. Since the maximum NTK eigenvalue reflects the convergence speed of a task, this observation aligns with the empirical finding that many existing MTL algorithms struggle to converge on surface-normal prediction. By accurately quantifying these differing convergence rates and re-allocating weights accordingly, NTKMTL attains more balanced performance, delivers state-of-the-art results on surface-normal prediction, and is one of only two methods that outperform single-task learning on all three tasks.\n\n**2). The relationship between cross-task performance variance and $\\Delta m\\\\%$ on NYUv2.**\n\n| Method   | $\\Delta m\\\\%\\downarrow$ | Var[$\\Delta m\\\\%$] |\n| -------- | -| -- |\n| LS       | 5.59         | 259.13            |\n| SI       | 4.39         | 247.77            |\n| UW       | 4.05         | 190.73            |\n| PCGrad   | 3.97         | 173.66            |\n| DWA      | 3.57         | 191.93            |\n| CAGRAD   | 0.20         | 137.94            |\n| Nash-MTL | -4.04        | 108.03            |\n| FAMO     | -4.10        | 74.66             |\n| FairGrad | -4.66        | 71.59             |\n| GO4Align | -6.08        | 60.78             |\n| NTKMTL   | -6.99        | 62.85             |\n\nThe table above reports $\\Delta m\\\\%$ (average performance drop) and Var[$\\Delta m\\\\%$] (variance of the performance drop across tasks) on NYUv2 for NTKMTL and other 10 existing MTL methods. Notably, the performance $\\Delta m\\\\%$ of different MTL methods exhibits a striking positive correlation with cross-task performance variance, and methods with smaller performance variance tend to achieve better overall performance. This is precisely why “mitigating task imbalance” is of paramount importance in the field of multi-task learning, and it also supports why NTKMTL can achieve a superior $\\Delta m\\\\%$ through more balanced optimization.\n\n\n**Q2: Are there any modules or parameter choices in the proposed method that are particularly critical (Have any ablation studies been conducted)?**\n\n**A2:** Since the weights are derived by computing the maximum eigenvalue of the NTK matrix for different tasks, the samples within a batch need to be split into n mini-batches. \nWhile a larger n can lead to a more precise NTK representation, it also incurs greater computational overhead. \n\nWe also noticed that exploring the impact of n as a hyperparameter on experimental results is meaningful, especially for NTKMTL-SR. Therefore, we conducted a comprehensive ablation study on n for NTKMTL-SR, with the detailed results presented in the supplementary material. We set n to $[1, 2, 3, 4, 6]$ and conduct an ablation study on the QM9 (11-task) benchmark. For each value of n, we perform 3 repeated experiments with different random seeds and calculate the mean and error bar for $\\Delta m\\\\%$. Detailed results and analysis are available in the supplementary material, and we have also excerpted them here for convenience. \n\n| n | $\\Delta m\\\\%\\downarrow$ | time per epoch (min) |\n|---|---|---|\n| 1 | 48.4 | 1.57 |\n| 2 | 33.5 | 1.71 |\n| 3 | 35.4 | 1.90 |\n| 4 | 30.7 | 2.10 |\n| 6 | 29.8 | 2.54 |\n\nThe results indicate that increasing n from 1 to 2 or more shows a noticeable improvement in NTKMTL-SR's performance, accompanied by a reduction in the error bar of performance across repeated experiments. We attribute this to the fact that increasing the number of mini-batches leads to a larger NTK matrix dimension, which in turn reduces stochastic error and allows our method to more accurately characterize the convergence speed of the tasks. However, since further increasing n (i.e., from 4 to 6) does not yield significant additional benefits and introduces further computational overhead, we posit that the selection of hyperparameter n is a trade-off between performance and training speed, and n=4 generally presents a favorable compromise.\n\nFor NTKMTL, due to the memory and time cost associated with fully computing the gradients of n mini-batches for shared parameters to construct the NTK matrix, we set n=1 to ensure fairness when comparing with other methods. In this case, it aligns with other gradient-oriented methods, requiring k gradient backpropagations for shared parameters per iteration. Additionally, during the rebuttal period, we conducted further ablation experiments on the hyperparameter n for NTKMTL. You can refer to A1 in our response to Reviewer 2 (yMmP) for details.\n\n\n**Q3: Why does the proposed method exhibit varying performance across different tasks?**\n\n**A3:** In the field of MTL, achieving optimal performance across all subtasks on a given benchmark simultaneously is challenging, as it essentially means directly extending the Pareto frontier. **While our method's performance varies across different benchmarks, a consistent pattern emerges: it consistently optimizes more difficult tasks more effectively, leading to more balanced optimization, effectively mitigating task imbalance, and achieving superior average performance.**\n\nFor instance, on NYUv2, NTKMTL achieved  SOTA results on the most challenging surface normal prediction task. It also obtained the optimal MR and $\\Delta m\\\\%$. On the CityScapes benchmark, most prior methods excelled at the segmentation task but performed significantly worse than the Single-Task Learning (STL) baseline on depth prediction. NTKMTL, however, better balanced the performance of these two tasks and achieved the optimal $\\Delta m\\\\%$.\n\n&nbsp;\n\nThese discussions have further enriched the content of this paper. We will integrate the rebuttal content and update the manuscript accordingly. Specifically, the visualizations from A1 regarding the maximum eigenvalue of the NTK matrix for different tasks during training and the relationship between cross-task performance variance and $\\Delta m\\\\%$ on various benchmarks will be added as figures to the Appendix. The corresponding analysis from A1 will also be incorporated into the main experimental section. The results of the ablation studies will be integrated into both the main text and the Appendix. The discussion from A3 will be added around lines 252-270 in Section 4.2 to enhance the analysis of the experimental results.\n\nIf you have any further questions, please do not hesitate to contact us. Thank you again for your valuable time and effort during the review process!"}, {"review_id": "K6IMVJwkHb", "reviewer": "Reviewer_yMmP", "summary": "This paper introduces NTKMTL, a novel Multi-Task Learning (MTL) method designed to mitigate task imbalance, a prevalent challenge where certain tasks dominate the training process. The core contribution is the application of Neural Tangent Kernel (NTK) theory to analyze and balance the convergence speeds of multiple tasks within an MTL system. The authors propose an extended NTK matrix for MTL, showing that tasks associated with larger NTK eigenvalues converge faster and can dominate others. To address this, NTKMTL assigns weights to tasks based on the maximum eigenvalues of their respective NTK matrices, aiming to normalize their convergence speeds. An efficient variant, NTKMTL-SR, is also introduced, which leverages shared representation approximation to improve computational efficiency, especially for scenarios with many tasks. Extensive experiments across multi-task supervised learning (NYUv2, CityScapes, QM9, CelebA) and multi-task reinforcement learning (MT10) benchmarks demonstrate that NTKMTL and NTKMTL-SR achieve state-of-the-art performance.", "questions": "Impact of n on NTK Estimation and Performance: The paper uses different n values for NTKMTL (n=1) and NTKMTL-SR (n=4), noting that larger n yields \"a more accurate estimate of convergence speed\". Please provide an ablation study on how varying n affects the accuracy of the NTK eigenvalue estimation and, consequently, the overall performance of both NTKMTL and NTKMTL-SR.", "limitations": "The authors have adequately discussed the limitations of their work. They acknowledge that their current weight design primarily focuses on task-specific NTK matrices ($K_{ii}$) and that further analysis of the full extended NTK matrix ($\\tilde{K}$), particularly the off-diagonal ($K_{ij}$) blocks, is an area for future investigation to reveal insights into task interactions. They also discuss the computational efficiency of their methods, contrasting NTKMTL with the more efficient NTKMTL-SR. Furthermore, they explicitly mention the challenge of applying NTKMTL-SR to multi-task reinforcement learning due to the difficulty in partitioning parameters into shared and task-specific components in single policy learning scenarios. Finally, the paper includes an \"Impact Statement\" addressing potential positive societal impacts regarding fairness and performance equity in machine learning systems, reflecting a commitment to responsible dissemination.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- Novelty: This paper presents a novel approach to address a significant problem in Multi-Task Learning, namely task imbalance. The application of Neural Tangent Kernel (NTK) theory to MTL is a significant and original contribution, providing a new lens through which to understand and mitigate this issue.\n- Strong Theoretical Foundation: The paper rigorously extends the foundational NTK theory to the MTL context, formally defining an extended NTK matrix and deriving its evolution equation. This theoretical framework allows for a clear characterization of convergence speeds based on NTK eigenvalues.\n- Effective Task Imbalance Mitigation: NTKMTL's strategy of weighting tasks based on their NTK eigenvalues effectively balances convergence speeds, leading to demonstrably improved overall performance and more balanced optimization across tasks.\nState-of-the-Art Performance: The experimental results are comprehensive and show impressive state-of-the-art performance across a diverse range of benchmarks, including computer vision, molecular property prediction, and reinforcement learning.\n- Computational Efficiency with NTKMTL-SR: The introduction of NTKMTL-SR is a crucial practical improvement. By leveraging shared representations, it significantly reduces computational cost, making the method feasible for tasks with a large number of components (e.g., 40 tasks on CelebA) while maintaining competitive performance.\n- Commitment to Reproducibility: The authors provide detailed algorithmic descriptions, pseudocode, and experimental settings. Their effort to reproduce and re-evaluate baselines on QM9 under optimized settings for a fair comparison is commendable and enhances the credibility of their results. The authors also commit to making their code publicly available with camera-ready submission.\n\nWeaknesses:\n- Limited Deep Dive into Extended NTK (K̃): While the paper introduces the extended NTK matrix K̃ and acknowledges its potential for deeper analysis (e.g., off-diagonal Kij blocks for task interactions), the primary weight design focuses only on the task-specific diagonal blocks Kii. This leaves a significant portion of the newly introduced theoretical framework underexplored in the current method. For a theoretically driven paper, this feels like an incomplete utilization of the proposed extended kernel.", "was_revised": true, "final_justification": "I acknowledge that i have read the authors' rebuttal to all reviewers. The response by the authors and the additional clarifications provided further solidifies my initial assessment of the work. I will keep my original high rating of 5.", "rebuttal": "We appreciate your recognition of our method's novelty, strong theoretical foundation, effectiveness, and efficiency. Thank you for your careful and valuable comments! We hope to address your concerns by answering your questions below.\n\n**Q1: Ablation Study on Hyperparameter n**\n\n**A1:** We also noticed that exploring the impact of n as a hyperparameter on experimental results is meaningful, especially for NTKMTL-SR. Therefore, we conducted a comprehensive ablation study on n for NTKMTL-SR, with the detailed results presented in the supplementary material. We set n to $[1, 2, 3, 4, 6]$ and conduct an ablation study on the QM9 (11-task) benchmark. For each value of n, we perform 3 repeated experiments with different random seeds and calculate the mean and error bar for $\\Delta m\\\\%$. Detailed results and analysis are available in the supplementary material, and we have also excerpted them here for convenience. \n\n| n | $\\Delta m\\\\%\\downarrow$ | time per epoch (min) |\n|---|---|---|\n| 1 | 48.4 | 1.57 |\n| 2 | 33.5 | 1.71 |\n| 3 | 35.4 | 1.90 |\n| 4 | 30.7 | 2.10 |\n| 6 | 29.8 | 2.54 |\n\nThe results indicate that increasing n from 1 to 2 or more shows a noticeable improvement in NTKMTL-SR's performance, accompanied by a reduction in the error bar of performance across repeated experiments. We attribute this to the fact that increasing the number of mini-batches leads to a larger NTK matrix dimension, which in turn reduces stochastic error and allows our method to more accurately characterize the convergence speed of the tasks. However, since further increasing n (i.e., from 4 to 6) does not yield significant additional benefits and introduces further computational overhead, we posit that the selection of hyperparameter n is a trade-off between performance and training speed, and n=4 generally presents a favorable compromise.\n\nAdditionally, at your request, we also conducted an ablation study on the hyperparameter n for NTKMTL. Due to limited rebuttal time, these experiments were performed on the smaller CelebA benchmark, and the results are presented in the table below. We maintain consistency with the settings of other MTL methods discussed in the main text, training for 15 epochs on CelebA. For each value of n, we conduct three repeated experiments and report the mean of $\\Delta m \\\\%$.\n\n| Method                     | $\\Delta m \\\\%\\downarrow$    | Time per epoch (min) |\n| -------------------------- | ---------------- | -------------------- |\n| NTKMTL (n=1)               | -0.77  | 15.6                 |\n| NTKMTL (n=2)               | -0.95  | 29.3                 |\n| NTKMTL (n=3)               | -0.89  | 42.6                 |\n| NTKMTL (n=4)               | -1.09  | 55.3                 |\n\n\nThe table shows that increasing n does yield modest performance gains; however, the computational overhead grows almost linearly with n, in sharp contrast to the negligible cost incurred by NTKMTL-SR. Because the results with n=1 are already highly competitive, and to ensure a fair comparison with other MTL methods, we recommend setting n=1 as the default for NTKMTL. In this case, it aligns with other gradient-oriented methods, requiring k gradient backpropagations for shared parameters per iteration.\n\n**Q2: Further analysis of the full extended NTK matrix $\\tilde{K}$, particularly the off-diagonal $\\tilde{K}_{ij}$ blocks.**\n\n**A2:** Thank you for showing this point. Our motivation is to balance the convergence speeds of different tasks, and thus we mainly require the maximum eigenvalue of the respective NTK matrix of each task to capture the convergence speed. Therefore, we primarily used the diagonal blocks of $\\tilde{K}$ for analysis.\n\nWe also recognize that the off-diagonal blocks of $\\tilde{K}$, specifically $\\tilde{K}_{ij}$, might contain rich information about task similarity. These information can potentially lead to new MTL methods based on task grouping. However, as this represents a technical direction orthogonal to our paper's focus of mitigating task imbalance by balancing convergence speeds, we did not elaborate extensively on it here. Instead, we included it in the limitations section for future work.\n\n**Q3: About typographical errors.**\n\n**A3:** Thank you for your careful review! We've corrected these typos, and they will be eliminated in the latest version of the paper.\n\n&nbsp;\n\nThese discussions have further enriched the content of this paper. We will integrate the relevant ablation studies into the main text and Appendix, while also carefully reviewing the manuscript to eliminate any typos. Thank you again for your high appreciation of our work and for the valuable time and effort during the review process!"}, {"review_id": "OqU5l1Hk0Y", "reviewer": "Reviewer_9Qpw", "summary": "This paper analyzes the task dominance phenomenon in multi-task learning based on NTK and proposes a method based on NTK singular value balancing to alleviate the task imbalance issue. Experiments were conducted on multi-task supervised learning and multi-task reinforcement learning to validate the effectiveness of the method. However, the paper still lacks evidence to support many of its analyses, and the effectiveness of the proposed method in specific scenarios needs further discussion.", "questions": "- On the CITYSCAPES dataset, the MR metric of NTKMTL is significantly worse than the baseline methods (with an average ranking of 7th). How do different datasets affect the effectiveness of NTKMTL? The scenarios in which NTKMTL may fail need further discussion.\n- NTKMTL-SR  is an approximation of NTKMTL, but why does the former perform better than the latter in certain scenarios (e.g., Tables 2 and 3)? This is quite confusing.\n- How does the task-relatedness/similarity influence task dominance and NTK singular values?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- This paper introduces a new theoretical perspective by analyzing the task dominance problem in the MTL training process from the NTK viewpoint.\n- The paper proposes two multi-task balancing methods, NTKMTL and NTKMTL-SR.\n- A significant number of experiments are conducted on multi-task supervised learning and reinforcement learning to validate the effectiveness of the proposed methods.\n\n**Weaknesses:**\n\n- The goal of the paper is to analyze the phenomenon of learning speed/NTK singular value dominance in MTL training; however, no visualization experiments are provided to support these claims.\n- In line 34, the authors mention \"However, different tasks exhibit vastly different loss scales and heterogeneous ultimate loss minima,\" but this part lacks supporting argumentation or analysis, especially regarding the heterogeneity of the minima.\n- The paper mentions that very few works apply NTK to MTL, but the key challenges or difficulties of extending NTK theory from single-task to multi-task need further elaboration.\n- Some works in the MTL field have also discussed the task dominance issue in multi-task model training, which should be addressed and referenced, such as Dynamic Priority[1] and AdaTask[2].\n- Figure 1 shows the time for training one epoch. Do all methods train for the same number of epochs in the experiment? If not, comparing the time for the entire training phase would be more meaningful.\n\n[1] Guo, Michelle, et al. \"Dynamic task prioritization for multitask learning.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\n[2] Yang, Enneng, et al. \"Adatask: A task-aware adaptive learning rate approach to multi-task learning.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 9. 2023.", "was_revised": true, "final_justification": "I'm not an expert in this field. I'm considering the decisions of other experts.", "rebuttal": "We appreciate your recognition of our method's new theoretical perspective and extensive experimental validation. Thank you for your careful and valuable comments! We hope to address your concerns by answering your questions below.\n\n**Q1: Visualization experiments for the NTK eigenvalues during training.**\n\n**A1:** Thanks for your suggestion. We agree that experimental visualization is essential to support the proposed theory. Hence, we conducted experiments on NYUv2, visualizing the change in the maximum eigenvalue of the NTK matrix for the three tasks during training under linear scalarization (i.e., equal weighting). As images cannot be uploaded, we present the results in the table below.\n\n|Task|avg over 1000|avg over 2000|avg over 3000|avg over 4000|avg over 5000|\n|-|-|-|-|-|-|\n|segmentation|2.52|2.35|2.92|3.50|3.76|\n|depth|2.71|2.27|2.41|2.68|2.87|\n|surface normal|0.38|0.30|0.33|0.37|0.39|\n\n'Avg over xx' denotes the mean of the maximum NTK eigenvalues during the first xx iterations. On the NYUv2 dataset, the difficulty levels of the three tasks show significant variation. Previous methods generally outperform the Single Task Learning (STL) baseline in segmentation and depth estimation tasks, but almost all of them consistently underperform STL on the surface normal prediction task, leading to a significant task imbalance in the overall results.\n\nAs evident from the table, throughout training, the largest eigenvalue corresponding to surface normal prediction remains substantially smaller than those of segmentation and depth estimation. Given that the maximum NTK eigenvalue reflects a task's convergence speed, this observation aligns with the empirical finding that many existing MTL algorithms struggle to converge on surface normal prediction. Although some prior methods (e.g., MGDA) attempt to prioritize the most difficult tasks, their performance in surface normal prediction tasks remains unsatisfactory due to the challenge in accurately quantifying the \"difficulty\" and \"convergence speed\" of different tasks. In contrast, by leveraging NTK theory to accurately characterize and balance the convergence speed of each task during training, NTKMTL delivers SOTA results on surface normal prediction and is one of only two methods that outperform single-task learning on all three tasks.\n\n**Q2: Question about \"different tasks exhibit vastly different loss scales and heterogeneous ultimate loss minima\".**\n\n**A2:** In MTL, simple equal weighting fails to balance task losses precisely because different tasks inherently possess distinct natures and may utilize distinct loss calculation methods (e.g., cross-entropy loss for classification vs. L2 loss for regression). This leads to vastly different loss scales and different ultimate loss minima, a fact widely acknowledged by many MTL works (e.g., UW, FairGrad, GO4Align). Directly applying equal weighting often results in the training process being dominated by tasks with larger loss scales.\n\nFor instance, on the NYUv2 benchmark, the losses for semantic segmentation and depth prediction tasks are typically an order of magnitude higher than that for surface normal prediction. On the even more heterogeneous QM9 dataset, the loss discrepancies among different tasks can span up to three orders of magnitude. \n\nThanks for your suggestion. We will rephrase this statement in lines 32-39 of the manuscript to enhance clarity.\n\n**Q3: Further elaboration for the key challenges of extending NTK theory from STL to MTL.**\n\n**A3:** In STL, the NTK expression can be directly derived from the gradient flow. However, MTL explicitly divides the objective function into k components, each associated with a specific task. Since all tasks share parameters, the gradient flow of one task also influences the losses of other tasks, which explicitly increases the complexity of the analysis. In MTL, analyzing only the overall loss using STL's NTK theory (Theorem 3.1) fails to capture fine-grained, per-task training dynamics. To fully understand the differences in convergence speed among tasks, it's essential to extend STL's NTK theory, which poses a new theoretical challenge.\n\nWe analyzed the specific scenario of MTL and proposed Theorem 3.2 and Proposition 3.3 (proofs detailed on pages 15-17). The most intuitive physical interpretation these theories provide is that with the introduction of weighting coefficients $w_i$, the new NTK matrix for each task becomes $w_i^2K_{ii}$, and its maximum eigenvalue is accordingly scaled to $w_i^2 \\lambda_i$. This ultimately led to our final method design.\n\nThanks for your suggestion. We will further clarify the necessity of our proposed theory in lines 175-197 of the method section and provide a smoother transition to the method design.\n\n**Q4: Missing References.**\n\n**A4:** Thank you for the additions! We have previously read both of these two papers. Dynamic Task Prioritization (DTP), in particular, is a significant work in the MTL field. It was an oversight on our part to have omitted these two papers from the related work section. We've now included them in the manuscript.\n\n**Q5: Do all methods train for the same number of epochs in the experiment?**\n\n**A5:** Yes, to ensure fairness, all methods are trained for the same number of epochs across all benchmarks. Therefore, the training time per epoch can essentially represent the comparison of total training duration.\n\n**Q6: On CITYSCAPES, the MR of NTKMTL is worse than some baseline methods. How do different datasets affect the effectiveness of NTKMTL?**\n\n**A6:** While our method's performance varies across different benchmarks, a consistent pattern emerges: it consistently optimizes more difficult tasks more effectively, leading to more balanced optimization and achieving superior average performance.\n\nWhen evaluating MTL methods, $\\Delta m\\\\%$ is typically the most intuitive metric as it relies solely on the objective performance of the MTL method on a specific benchmark. Mean Rank (MR), however, depends on the number and performance of other MTL methods used for comparison, and thus usually serves as a supplementary criterion when $\\Delta m\\\\%$ values are similar.\n\n**Why does NTKMTL exhibit suboptimal MR performance on the CityScapes dataset?** We report the detailed performance of 17 different MTL methods on the CityScapes in Appendix Table 7 (page 19). The CityScapes task comprises two subtasks: segmentation and depth prediction. Due to the disparity in task difficulty, most prior methods excelled at the segmentation task but performed significantly worse than the STL baseline on the depth prediction task, exhibiting severe task imbalance. Our method effectively balances the performance of different tasks, achieving significant progress in the depth task. However, this also implies a slight decrease in performance on the segmentation task, placing our method at a disadvantage in MR calculation.\n\nIn summary, NTKMTL's suboptimal MR performance on CityScapes does not indicate a failure of the method on this dataset. In fact, it effectively mitigates task imbalance and achieves the optimal $\\Delta m\\\\%$.\n\n**Q7: Why does NTKMTL-SR perform better than NTKMTL in certain scenarios?**\n\n**A7:** In Table 2, NTKMTL-SR does not outperform NTKMTL in terms of $\\Delta m\\\\%$ (0.23 vs. -0.77), achieving only a slightly better Mean Rank (MR) (4.33 vs. 4.35). As stated in A6, the MR metric is susceptible to changes based on the inclusion or exclusion of other comparative MTL methods. \n\nHowever, on the QM9 dataset, NTKMTL-SR indeed surpasses NTKMTL and achieves state-of-the-art performance. We also analyzed this phenomenon, attributing it to the relatively small number of model parameters used in this regression task; consequently, even with approximation, the training dynamics of different tasks can be captured effectively. As a result, a larger n produces a more accurate estimate of convergence speed. To further validate this, we conducted detailed ablation experiments in the supplementary material, where we set n to [1, 2, 3, 4, 6] for NTKMTL-SR and performed three experiments for each setting, reporting the mean and error bars. As shown in Fig. 2 of the supplementary material, when n=1, the performance of NTKMTL-SR is comparable to that of NTKMTL. However, when increasing n from 1 to 2 or more, NTKMTL-SR shows a noticeable improvement in performance, accompanied by a reduction in the variance of performance across repeated experiments.\n\n**Q8: How does the task-relatedness/similarity influence task dominance and NTK singular values?**\n\n**A8:** During our experiments, we observed that tasks exhibiting clear similarities (e.g., tasks 7-10 on QM9) not only have similarly scaled maximum eigenvalues of their NTK matrices but also show consistent trends in their evolution throughout training. Consequently, NTKMTL assigns essentially equal weights to these tasks to facilitate balanced optimization. (As shown in Table 3, these highly correlated tasks exhibit very close final MAE values.) We have also visualized this phenomenon, and the images and corresponding analysis will be included in the latest version of the appendix.\n\nFurthermore, we acknowledge that the off-diagonal blocks of $\\tilde{K}$, specifically $\\tilde{K}_{ij}$, contain rich information. These blocks can, to some extent, represent task similarity and potentially lead to new MTL methods based on task grouping. However, due to the orthogonality of the task grouping technical approach to our current method, we include it in the limitations section for future work.\n\n&nbsp;\n\nThese discussions have further enriched the quality of the manuscript. We will integrate all rebuttal content, adding visualizations as figures to the appendix, and simultaneously revise and refine certain statements in the main text. If you have any further questions, please don't hesitate to contact us. Thank you again for your valuable time and effort during the review process!"}, {"review_id": "yuTYaAKXTt", "reviewer": "Reviewer_k73J", "summary": "This paper addresses the persistent challenge of task imbalance in multi-task learning (MTL) by proposing a novel method grounded in Neural Tangent Kernel (NTK) theory. The authors introduce an extended NTK framework to model MTL training dynamics and leverage spectral analysis to derive task-specific weighting. This culminates in two methods: NTKMTL, which explicitly balances task convergence via NTK eigenvalue normalization, and NTKMTL-SR, a computationally efficient approximation using shared representations. Extensive experiments across supervised and reinforcement learning tasks validate the proposed methods, demonstrating competitive or superior performance compared to strong baselines.", "questions": "1. Aside from Table 1, most other tables do not report the results of all baseline methods, which makes it difficult to objectively assess the general effectiveness of the proposed method across different scenarios.\n2. Are there more ablation studies on hyperparameter n for NTKMTL?", "limitations": "1. Limited Analysis of Off-Diagonal NTK Components: The off-diagonal blocks of the extended NTK matrix (which capture inter-task interactions) are not exploited, potentially missing opportunities to guide task grouping or influence.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. Theoretical Innovation: The application of NTK theory to MTL and the formulation of an extended NTK matrix for spectral task analysis are original and theoretically grounded.\n3. Clear Algorithmic Design: The derivation of task weights from NTK eigenvalues is logically sound and well-motivated.\n2. Broad Experimental Evaluation: The paper evaluates on diverse benchmarks (NYUv2, CityScapes, CelebA, QM9, MT10), with both supervised and reinforcement learning settings, providing strong empirical support.\n\n**Weaknesses:**\n1. Incompatibility of NTKMTL-SR with Non-Task-Specific Architectures: The NTKMTL-SR method relies on the presence of task-specific parameters to compute gradients with respect to shared representations. This limits its applicability to architectures where all parameters are shared (i.e., no task-specific heads), making it incompatible with reinforcement learning settings (e.g., MT10) or even large language model scenarios.\n2. Unfair Comparison with Other MTL Methods: The weight computation in Eq. 15 does not strictly ensure that the task weights sum to k (the number of tasks). Without a formal guarantee, the sum of weights is generally ≥ k, potentially leading to unfair comparisons with other MTL methods that explicitly constrain weight normalization. **I believe it is critically important to include a fair comparison based on properly normalized weights; otherwise, the strong empirical results presented in the paper lose much of their significance.**", "was_revised": true, "final_justification": "The author has resolved most of my main concerns, including those regarding the incompatibility of NTKMTL-SR (Q1), the weight sum in NTKMTL (Q2), and ablation studies on hyperparameter n (Q4).\n\nThe results of all baseline methods (Q3) and further analysis for off-diagonal NTK (Q5) are absent, but this will not particularly affect the contribution and effectiveness of this work.\n\nTherefore, I will maintain my score of borderline accept for this paper.", "rebuttal": "We appreciate your recognition of our method's theoretical innovation, clear algorithmic design, and broad experimental evaluation. Thank you for your careful and valuable comments! We hope to address your concerns by answering your questions below.\n\n**Q1: Incompatibility of NTKMTL-SR with Non-Task-Specific Architectures.**\n\n**A1:** We acknowledge that NTKMTL-SR is an approximate algorithm specifically designed for models with task-specific layers, which might limit its direct applicability to settings like MT10 or large language model (LLM) training. However, NTKMTL  supports non-task-specific architectures and has demonstrated superior performance in multi-task reinforcement learning. Additionally, NTKMTL achieves a speed advantage of over 2x compared to some gradient-oriented MTL methods like PCGrad and Nash-MTL, which require additional gradient projection or iterative optimization.\n\nFurthermore, our proposed extended NTK theory in MTL still holds potential for generating new algorithms in these scenarios. For instance, a promising direction involves balancing training losses from different data sources (e.g., code, math) during LLM pre-training by computing the NTK matrix with respect to deep-layer parameters for each data source. We welcome any insights you may have on this!\n\n**Q2: The weight in NTKMTL does not strictly ensure that the task weights sum to k (the number of tasks).**\n\n**A2:** Yes, early MTL methods (e.g. DWA [1]) often constrained the sum of weights to k (the number of tasks). However, the vast majority of recent MTL algorithms (e.g., Nash-MTL [2], FAMO [3], FairGrad [4], GO4Align [5]) do not adhere to this constraint. They operate under the premise that the scale information inherent in the calculated weights is beneficial for training, and that the optimal weight scales differ across various training stages. Forcibly normalizing these weights to k would lead to a loss of this crucial information.\n\nEspecially for algorithms like Nash-MTL and FairGrad, the task weights are derived by solving an optimization problem. In such cases, forced normalization would invalidate them as solutions to the optimization problem. Therefore, most contemporary MTL algorithms only require that the weights be bounded. Their specific scale is determined by the algorithm's hyperparameters, which are often tuned to find the optimal settings.\n\n[1] Liu S, Johns E, Davison A J. End-to-end multi-task learning with attention[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019: 1871-1880.\n\n[2] Navon A, Shamsian A, Achituve I, et al. Multi-task learning as a bargaining game[J]. arXiv preprint arXiv:2202.01017, 2022.\n\n[3] Liu B, Feng Y, Stone P, et al. Famo: Fast adaptive multitask optimization[J]. Advances in Neural Information Processing Systems, 2023, 36: 57226-57243.\n\n[4] Ban H, Ji K. Fair resource allocation in multi-task learning[J]. International Conference on Machine Learning, 2024.\n\n[5] Shen J, Wang Q, Xiao Z, et al. Go4align: Group optimization for multi-task alignment[J]. Advances in Neural Information Processing Systems, 2024.\n\n**Q3: Aside from Table 1, most other tables do not report the results of all baseline methods.**\n\n**A3:** NYUv2, featured in Table 1, is one of the most widely used benchmarks in the MTL field, so nearly all MTL methods conduct experiments on it. Except for QM9, where we re-implemented almost all methods for a fair comparison, results for other MTL methods on other benchmarks were obtained from their original papers.\n\nSince some works have conducted experiments on only two or three benchmarks, we cannot obtain the results from their papers. Many of them overlook CelebA (40-task), a benchmark crucial for evaluating MTL performance with a large number of tasks. This explains why the number of methods shown in Table 2 is fewer than in Table 1. For the CityScapes benchmark, due to space limitations in the main text, we presented its results alongside CelebA in Table 2, showing only 14 MTL methods. We report the detailed results for this benchmark, encompassing 18 methods, in Appendix Table 7.\n\nThank you for your suggestion. We will consider to re-implement the missing results for some MTL methods on CelebA and integrate them into the latest version of the manuscript.\n\n\n**Q4: Are there more ablation studies on hyperparameter n for NTKMTL?**\n\n**A4:** For NTKMTL-SR, we have conducted a comprehensive ablation study on n , with the detailed results presented in the supplementary material. At your request, we also conducted an ablation study on the hyperparameter n for NTKMTL. We performed experiments CelebA benchmark, and the results are presented in the table below. We maintain consistency with the settings of other MTL methods discussed in the main text, training for 15 epochs on CelebA. For each value of n, we conduct three repeated experiments and report the mean of $\\Delta m \\\\%$.\n\n| Method                     | $\\Delta m \\\\%\\downarrow$    | Time per epoch (min) |\n| -------------------------- | ---------------- | -------------------- |\n| NTKMTL (n=1)               | -0.77  | 15.6                 |\n| NTKMTL (n=2)               | -0.95  | 29.3                 |\n| NTKMTL (n=3)               | -0.89  | 42.6                 |\n| NTKMTL (n=4)               | -1.09  | 55.3                 |\n\n\nThe table shows that increasing n does yield modest performance gains; however, the computational overhead grows almost linearly with n, in sharp contrast to the negligible cost incurred by NTKMTL-SR. Because the results with n=1 are already highly competitive, and to ensure a fair comparison with other MTL methods, we recommend setting n=1 as the default for NTKMTL. In this case, it aligns with other gradient-oriented methods, requiring k gradient backpropagations for shared parameters per iteration.\n\n**Q5: Further Analysis for Off-Diagonal NTK Components.**\n\n**A5:** Thank you for showing this point. Our motivation is to balance the convergence speeds of different tasks, and thus we mainly require the maximum eigenvalue of the respective NTK matrix of each task to capture the convergence speed. Therefore, we primarily used the diagonal blocks of $\\tilde{K}$ for analysis.\n\nWe also recognize that the off-diagonal blocks of $\\tilde{K}$, specifically $\\tilde{K}_{ij}$, might contain rich information of task similarity. These information can potentially lead to new MTL methods based on task grouping. However, as this represents a technical direction orthogonal to our paper's focus of mitigating task imbalance by balancing convergence speeds, we did not elaborate extensively on it here. Instead, we included it in the limitations section for future work.\n\n&nbsp;\n\nThese discussions have further enriched the quality of the manuscript. We will incorporate the ablation studies for both NTKMTL and NTKMTL-SR into the appendix, and simultaneously refine parts of the main text for enhanced clarity. If you have any further questions, please don't hesitate to contact us. Thank you again for your valuable time and effort during the review process!"}]

Author rebuttal:

Meta-review: Thank you for your submission to the NeurIPS 2025. This paper analyzes the training dynamics in MTL by leveraging Neural Tangent Kernel (NTK) theory and propose a new MTL method, NTKMTL. The method is reasonable, equied with theoretical guarantee and comprehensive experiments. Although everyone has pointed out some merits, they also have raised some concerns, such as missing in-depth analysis. During the rebuttal period, the authors’ feedback has helped on clarifying the reviewers’ concerns. Thus, the average score 4.25 is both above the average levels. Based on the current reviews and closed-door reviewer discussions, every reviewer seems to be fine with an acceptance.
Decision: Accept (poster)