Exploring Tradeoffs through Mode Connectivity for Multi-Task Learning
Authors: Zhipeng Zhou
Ziqiao Meng
Pengcheng Wu
Peilin Zhao
Chunyan Miao
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Task Learning
Mode Connectivity
Primary area: general_machine_learning
TL;DR: We propose a new paradigm to explore the best trade-off through mode connectivity for multi-task learning.

Abstract: Nowadays deep models are required to be versatile due to the increasing realistic needs. Multi-task learning (MTL) offers an efficient way for this purpose to learn multiple tasks simultaneously with a single model. However, prior MTL solutions often focus on resolving conflicts and imbalances during optimization, which may not outperform simple linear scalarization strategies~\citep{xin2022current}. Instead of altering the optimization trajectory, this paper leverages mode connectivity to efficiently approach the Pareto front and identify the desired trade-off point. Unlike Pareto Front Learning (PFL), which aims to align with the entire Pareto front, we focus on effectively and efficiently exploring optimal trade-offs. However, three challenges persist: (1) the low-loss path can neither fully traverse trade-offs nor align with user preference due to its randomness, (2) commonly adopted Bézier curves in mode connectivity are ill-suited to navigating the complex loss landscapes of deep models, and (3) poor scalability to large-scale task scenarios. To address these challenges, we adopt non-uniform rational B-Splines (NURBS) to model mode connectivity, allowing for more flexible and precise curve optimization. Additionally, we introduce an order-aware objective to explore task loss trade-offs and employ a task grouping strategy to enhance scalability under massive task scenarios. Extensive experiments on key MTL datasets demonstrate that our proposed method, *EXTRA* (EXplore TRAde-offs), effectively identifies the desired point on the Pareto front and achieves state-of-the-art performance. *EXTRA* is also validated as a plug-and-play solution for mainstream MTL approaches.

Exploring Tradeoffs through Mode Connectivity for Multi-Task Learning
Zhipeng Zhou', Ziqiao Meng", Pengcheng Wu', Peilin Zhao
Chunyan Miao
Nanyang Technological University
National University of Singapore
School of Artificial Intelligence, Shanghai Jiao Tong University
zzpustcml @gmail com, zq-mengênus. eew.enuuuueuueeeeeeeeeeeeeeeeee. edu.sg. eg, meilinzhaode,ju.. edu. {pengchenguu, ascymiao) edu.
Abstract
Nowadays deep models are required to be versatile due to the increasing realistic needs. Multi-task learning (MTL) offers efficient way for this purpose learn multiple tasks simultaneously with a single model. However, prior MTI solutions often focus resolving conflicts and imbalances during optimization, which may not outperform simple linear scalarization strategies [Xin et af.) [022]. Instead of altering the optimization trajectory, this paper leverages mode connectivity efficiently approach the Pareto front and identify the desired trade-off point. Unlike Pareto Front Learning (PFL), which aims align with the entire Pareto front, we focus on effectively and efficiently exploring optimal trade-offs. However, three challenges persist: (1) the low-loss path can neither fully traverse trade-offs nor align with user preference due to its randomness, (2) commonly adopted Bézier curves in mode connectivity are ill-suited navigating the complex loss landscapes of deep models, and (3) poor scalability to large-scale task scenarios. To address these challenges, we adopt non-uniform rational B-Splines (NURBS) to model mode connectivity, allowing for more flexible and precise curve optimization. Additionally, we introduce an order-aware objective to explore task loss trade- offs and employ task grouping strategy to enhance scalability under massive task scenarios. Extensive experiments on key MTL datasets demonstrate that our proposed method, EXTRA (EXplore Tofde-offs), effectively identifies the desired point on the Pareto front and achieves state-of-the-ar performance. EXTRA is also validated as plug-and -lay solution for mainstream MTL approaches. Code is avaliable at https:://// //gi htt///ggtttuuuu thub. com/ http///ppsccEETTTTA
Introduction
Deep models are widely deployed across various scenarios, with an increasing demand for efficiency, such reducing computational and storage costs. This growing need necessitates that deep models be versatile. In this context, multi-tae task learning (MTL) has been extensively explored (Zhang and Yang 2018 8 Wang et al. [2013] Zhang al. 2018). MTL aims to achieve strong performance across multiple tasks using a single model, in contrast to traditional deep learning approaches where the number of parameters models often scales with the number of tasks.
In recent years, significant attention has been devoted to optimization based MTL [Sener and Koltun 2018 Liu et al.) a. Zhou al.) [2025a], which assumes fixed model architecture comprising a task-shared backbone and multiple task-specific branches. The primary goal in this setting is to
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

develop optimization algorithms that enable the backbone to effectively extract task- features. However, this approach faces two major challenges: conflict and imbalance issues, wherein task gradients may exhibit conflicts or significantly imbalanced norms.
To address these challenges, various solutions have been proposed, including gradient orthogonal projection (Yu et al. 2020], Nash negotiation [Navon et al., 2022], and fair resource allocation mechanisms [Ban and Ji 2024), etc. Unfortunately, recent studies [Xin al., 2022 Mueller et al.) 2024, Zhou et al. 2025b] have argued that altering the optimization trajectory through combinations of task gradients may be less effective than anticipated. Notably, when dedicated tuning is applied across all methods, these advanced approaches often fail to outperform a simple linear scalarization strategy. These findings suggest the need to explore alternative paradigms for optimization in MTL.
Another research line is Pareto Front Learning (PFL., which aims to directly explore the entire Pareto front, enabling the attainment of Pareto optimality based on user preferences. For instance, inspired by mode connectivity, PaMal. [Dimitriadis et al.) [2023] attempts to approach the Pareto front through ensembles of single tasks. Several efficient variants (Tang al., [024 Dimitriadis al.) 2024 Chen and Kwok 2024] have since been proposed, incorporating techniques such as low-rank approximations mixtures of experts. However, challenges in manifold learning and the scalability of PFL as the number of tasks increases remain unresolved and inapplicable, as we will empirically verify in Section
In this paper, rather than leveraging mode connectivity [Garipov et al. 2018 to achieve PFL, we utilize it address MTL. Unlike PFL., which optimizes endpoints and attempts to align with the Pareto front in the manifold space, our approach optimizes the curve connecting the endpoints. This requires only two endpoints and few control points, ensuring that memory and storage costs do not scale with the number of tasks. However, three challenges remain: (1) The low-loss path identified by mode connectivity for MTL is random, lacking both trade-off traversal and alignment with user preferences. (2) Commonly adopted curves (e.g.. Bézier curves) lack local dynamics, making it difficult connect points with differing trade-offs. (3) Poor scalability of this framework to massive task scenarios. To address these issues, we adopt non-un non-unnform rational B B-Splines (NURBS) for mode connectivity, enabling the search for more intricate curves. Additionally, we introduce an order-aware objective to encourage task loss trade-offss during optimization. Moreover, we employ a task grouping strategy to enhance scalability under massive task scenarios. In a nutshell, we summarize our contributions as three-fold:
We conduct MTL through mode connectivity by optimizing the curve connecting endpoints rather than the endpoints themselves, offering a more efficient way to leverage mode connectivity compared to PFL.
To traverse trade-offss and approach the Pareto front, we adopt flexible NURBS curves for enhanced local adjustments. Additionally, we propose an order-aware objective further encourage trade-off traversal. Besides, a task grouping strategy is further employed to address the scalability issue.
Extensive experiments on key MTL datasets demonstrate that our method effectively iden- tiñes the desired point on the Pareto Frontier and achieves state-of- the--ar (SOTA) perfor- mance.
2
Related Work
This paper primarily bridges the gap between MTL and mode connectivity, exploring their intersection. To provide context, we also introduce key developments in both domains. Additionally, we include an overview in PFL to clarify the distinctions between our approach and PFL..
2.1 Optimization- MTL
Optimization- based MTL focuses on developing optimization algorithms rather than architectural changes to facilitate learning multiple tasks simultaneously. A significant body of work (Liu et al.
2024| Chen et al.) 2018 Zhou et al. 2024) has explored re- re-weighting task losses to balance their learning progress. For instance, GradNorm defines metric to quantify the learning progress of each task and dynamically adjusts task weights during training. Similarly, FAMO [Liu et al. 2024 refines this idea by providing more precise estimation of progress.

---

Another paradigm in optimization- based MTL involves combining task gradients to address conflict and imbalance issues. MGDA [Sener and Koltun 2018 employs the Frank-Wolfe algorithm to find the least norm combination of gradients. PCGrad Yu et al. 2020 projects task gradients onto orthogonal directions to prevent conflicts. CAGrad [Liu et al., al. 202 balances global convergence and conflict-averse goals by designing compromise objectives. Nash-MTL. [Navon et al. 2022], adopting a game-theoretic perspective, enables tasks to negotiate parameter updates for balanced progression.
Recently, FairGrad [Ban and Ji 2024] built upon Nash-MTL by introducing finer-grained constraint to ensure balanced learning progress across tasks.
2.2 Mode Connectivity
Mode connectivity [Garipov et al.) 2018 is recently discovered phenomenon asserting that low-loss path can always be found to connect two local minima. It introduces practical methods using simple curves (e.g., polygonal chains and Bézier curves) to identify such paths. Similarly, [Draxler et al. 2018] employs nudged elastic band (NEB) method to search for low-loss connections. Expanding on this concept, SPRO [Benton al. 2021] explores code-connecting volumes, extending connectivity beyond simple paths. CBFT [Lubana et al.) 2023] leverages mode connectivity for efficient fine- tuning in downstream tasks. Additionally, applications such as continual learning (Mirzadeh et al.) 2020 Wen et al., 2023) utilize mode connectivity to transfer models along low-loss paths for specific objectives. Most existing works foeus optimizing the connected curve, which categorize as curve-based mode connectivity for clarity.
2.3 Pareto Front Learning
Unlike MTL., which seeks a single optimal point on the Pareto front, Pareto front learning (PFL) aims to directly construct the entire Pareto front, allowing for user-defined trade-offs at test time. PHN Navon et al. 2020] em- ploys a hyper-network to generate Pareto-optimal mod- els conditioned on preference vector inputs. In contrast, cosMos(RRuhte and Grabockal 2021] eliminates the parameter overhead of hyper-networks by conditioning the model on preference vectors in the feature space. Pa- Mal. falm(Diiissiadis et al., 22023) learns the Pareto front in the manifold space by optimizing endpoints, each dedicated single task. To improve efficiency, subsequent works have explored techniques such as low-rank approxima- tions Domittriiii et al. 2024 Chen and Kwok 2024 and mixture-of of offexperts (MoE) [Tang et al. [024]] to reduce the parameter burden of endpoints. These manifold-bag based PFL approaches can be categorized as endpoint- based mode connectivity.
COSMOS [Ruchte
baaseddd-basedddd (a) Manifold- based PFL
(b) Ours
Figure 1: Comparison of manifold PFL and our work.
Connection and Difference with Counterparts: Inspired by mode connectivity, we aim to explore different trade-off's by aligning the low-loss path with the Pareto front, specifically for MTL rather than PFL. In other words, our focus is on obtaining desired points on the Pareto front for MTL by optimizing the curve (path), rather than constructing the entire Pareto front by optimizing endpoints. This distinction is illustrated in Figure using simple two- and three-task instances. As shown, (manifold-basee PFL seeks to cover the Pareto front by optimizing endpoints and deriving the desired point through a linear combination of these endpoints. The limitations of this paradigm are discussed in Section 1. In contrast, our approach focuses solely on identifying the best trade-off for MTL by optimizing the curve. Moreover, our framework introduces a novel perspective for MTT, distinct from previous paradigms that rely on re-weighting task losses or gradients.
3 Preliminary
3.1 Setup of Optimization-based MTL
Optimization- multi-task learning (MTL.) methods typically assume that the model consists of a task-shared backbone network and task-specific branches. These methods aim to develop
3

---

gradient combination strategies that optimize the backbone network to benefit all tasks simultaneously. Consider a scenario with K 2 2 tasks, each associated with differentiable loss function (.)... where / represents the model parameters. The objective of optimization-ban MTL is to find the optimal 0* E T that minimizes the losses across all tasks.
3.2 Pareto Concept
Let define the weighted loss Lu E WiLi(0), where E W and W represents the probability simplex over []] - point 0' is said to Pareto dominate and only if Vi, (. ()) Pareto optimality occurs when no /' exists such that Vi, (x)))) (()). Points satisfying this condition form the Pareto set, and their corresponding solutions are called the Pareto front.
3.3 Mode Connectivity for MTL
For ease of explanation, we introduce the our application of vanilla mode connectivity a 2-task MTL setting. As illustrated in Figure 2 the training process consists of two stages:
Curve
[0, 1]
region
[1.0]
(I) First stage: Train the single-task models to obtain opti- mized weights, O₁ and 22, each tailored for its respective task.
Control point
Endpoint
(2) Second stage: Initialize the endpoints with the pre- trained single-task weights, O₂ and G2 Using these fixed endpoints and trainable control points (e.g., O, and 02). we construct curve function C(I) (e.g., Bézier curve, Polygonal Chain) that connects the endpoints.
Figure 2: Illustration of mode connectiv- ity for MTL.
During training, we uniformly sample points along the curve and minimize their corresponding losses to obtain low-loss path:
(r( = Ex-Mt.. (C(1; 0)) + C2(C((, + Rregl
(1)
where 10.1) denotes the uniform distribution over the interval [0, 1]. and Rreg the (L.2 norm) regularization item, and À is the hyper-ypaaaae parameter.
Motivation and Observation
Since our work is closely related the manifold-base PFL series-which also leverages mode connectivity for multi-tas learning- we draw motivation for our choice by analyzing their limitations. In addition, highlight the challenges of using curve optimization an alternative approach for MTL, which further motivates the designs introduced in the next section.
4.1 Limitations of Manifold-basel PFL
0.35
To facilitate clear discussion, we categorize the scenarios into two-task and multi-task settings. We adopt PaMaL as the baseline for our exper- iments, since most existing manifold-basee based PFL approaches that utilize mode connectivity are either derived from or closely related to PaMal.
0322
24
0366 SOM2 sulde eate 0n13
03
0.2
04
05
03
10
(a) Task Losses
(b)
Am %
along
the
scarched path
Degeneration of 2-Task Scenario: In the 2- task scenario, PaMaL reduces to linear mode connectivity, as illustrated in Figure [I] In other words, regardless of how the models (endpoints) are trained, their final weighted combination, t 01 + +(1 1 f) * 62, lies along a linear path. However, is generally challenging to approximate the Pareto front using a linear path. To verify this, we re-implemented PaMaL and evaluated it on the CityScapes dataset, presenting the results in Figure 3 As shown in Figure 3 a). task losses along the path fall within a narrower range, forming a relatively flat region rather than well-defined Pareto front. Additionally, Figure 3 illustrates the changes in Am% (defined Eqn. [ along the path. Notably, Am% exhibits a monotonically decreasing trend, which deviates from the expected characteristics of the Pareto front.
Figure 3: The results of PaMal. evaluated on the CityScapes. (a) is the task losses while (b) is the Am% along the searched path (MC path).

---

(a) Loss Landscape: Poly- (h)
Loss
Landscape: (c) Loss Landscape: Bézier
(d)
Loss
Landscape:
Chain curve (random)
Vanilla (random)
Bézier
curve curve (order-aware)
NURBS
curve
(order-
aware)
6m5 (Log Scae)
Gmk (Log Scali)
Scale)
Paint CASTOR
0.00
0.25
0.50
0.75
1.50
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.50
0.00
0.25
0.50
0.75
1.00
(e) MTL Ability: Poly- (f) MTL Ability: Vanilla
(g) MTL Ability: Vanilla (h) MTI Ability: NURBS Bézier curve (order-aware) curve (order-aware)
Chain curve (random)
Bézier curve (random)
Figure 4: The results of employing PolyChain, Bézier and NURBS curve for mode connectivity on CityScapes. (a)-(d) are the loss landscapes (average task loss) and searched curves with respect to different curves and objectives. and are endpoint and control point, respectively, while is the searched curve. All endpoints are well trained single task weights. (c)-(h) are the mtl ability (Am % evaluation along the corresponding searched curves.
Scaling Problem Many Task Scenario: Since manifold- based PFL optimizes endpoints to approx- imate the Pareto front, the number of endpoints must scale with the number of tasks. Consequently. scenarios involving many tasks (>2), this approach demands significant computational and storage resources, making it challenging to optimize. We present the memory cost the number of tasks increases in Figure 5fa). As expected, PaMaL exhibits an almost linear increase in memory cost with respect to the number of tasks P While subsequent methods, e.g., PaLoRA [Dimitriadis et al. 2024). employ techniques like LoRA or MoE to address these scalability issues, they still encounter optimization difficulties. Despite introducing additional parameters, their performance often lags behind advanced MTL approaches. As illustrated in Figure (bb), PaMaL and its efficient variant, PaLoRA, achieve SOTA performance the CityScapes dataset (2-task scenario). However, they less competitive on NYUv2 (3-task scenario), demonstrating the difficulty of such manifold based optimization 3
4.2
Challenges of Curve-Based Mode Connectivity
To circumvent these issues intrinstic lie in the endpoint- mode connectivity, we opt to the curve-based ones. However, it will naturally encounters the following challenges:
4.2.1 Challenge 1: Randomness
-PAIE
The original mode connectivity approach seeks low-loss curve (c.g., Bézier Polygonal Chain) by minimizing task losses but lacks designs specifically tailored for bandling multi-task com- petition. As a result, effectiveness in MTL exhibits randomness.
Diydeaps
(a) Memory cost of PaMaL
(b) Performance compar- ison.
We verify this by visualizing the curve on the loss landscape and computing its MTL ability, quantified by Am% (defined in Eqn 3 along different points of the curve, as shown in Fig- ure 4 (b)) Although most points along
Figure 5: Analysis on PaMaL. (a) is the memory cost of PaMaL with the increasing number of end- points, (b) is the performance compared with ad- vanced MTL on CityScapes and NYUv2 datasets.
2 Mooe memory cost comparisons are presented in the Appendix (See. B.3).
We calculate Am% according the results provided in PaLoRA.
5

---

the curve lie within a low-loss region, their MTL abilities fluctuate significantly, indicating failure align with the Pareto front. This suggests that mode connectivity alone does not inherently promote stable trade-offss and may struggle to identify the optimal trade-off point (i.e., the best MTI ability) along the curve.
To address this randomness issue, we introduce an additional objective (see Section 5.1) to promote a more structured exploration of trade-off'. Figure (g) provides a preview of the benefits of our approach. As shown, the proposed objective guides the curve through a broader low-loss region and successfully identifies a better and more distinct optimal trade-off compared to Figure (f)))
4.2.2 Challenge 2: Flexibility
Another challenge concerns the choice of curve. The commonly used Bézier curve in mode connec- tivity is often too smooth [Draxler et al., 2018] effectively explore diverse trade-offs. As shown in Figure 4 b))c), each control point exerts a global influence on the curve, limiting its ability to capture local dynamics. In contrast, NURBS offers greater flexibility for local adjustment through segmental modeling. As illustrated in Figure dd, even with uniformly sampled points, most lic within low-loss basin (Point Cluster) and demonstrate strong MTL performance, underscoring NURBS's advantage in modeling local variations.
4.2.3 Challenge 3: Scalability
The last challenge arises in terms of scalability when addressing scenarios involving a large number of tasks. As shown in Figure 2 curve-based mode connectivity is primarily applicable to settings involving two endpoints (tasks), which limits its utility massive task scenarios a critical aspect of MTL. Although some prior works have attempted to extend mode connectivity to multiple endpoints [Fott and Jastrzebski, 2019) or adopt manifold- -based strategies [Benton et al. 2021], these approaches are often difficult to optimize require extensive pre-training of multiple endpoints. These limitations are key reasons why we do not adopt endpoint-bee based mode connectivity in our framework.
Remarks: This scalability issue shared by both curve- and endpoint-basee mode connectivity approaches. However, they pursue different goals: endpoint-basede mode connectivity aims to approximate the entire Pareto front, while our curve-based approach focuses on exploring the optimal trade-off point for MTL. This key difference allows us to apply endpoint clustering when facing massive tasks-a strategy that is not feasible for the endpoint-based approach.
Principal Design
5.1 Injecting Order to Mode Connectivity
To address the Challenge 1, we begin by recalling a key property of the Pareto front: the ability to traverse trade-offs. Building on this property, we propose the following order-aware objective. Assume we sample two variables, L and l2, and obtain the corresponding weights, C(4,) and C(12). The objective is then defined as:
R, 2 [ [claaa---
where (t,)) the abbreviate of (C(t;; 0)), and represent the losses of task i at C(t,; 0). This constraint explicitly encourages the task losses to exhibit different tendencies with respect to t. The overall objective can then be defined as follows:
[(0) Lr + a R.
where Lr represents the mean task losses, and a a peree--prrmeter. This objective ensures the low-loss property while encouraging the searched curve gradually approach the Pareto front.
5.2 Enabling Complicated Path: NURBS
As Challenge have mentioned, Bézier curves often suffer from an over-smoothing issue, making challenging capture long-dis -iittace trade-offs. This limitation arises because the order of the curve

---

is inherently tied to the number of control points, causing adjustments to any control point to shift the entire curve (Piegl and Tiller, 2012]. To provide the curve with greater flexibility, we adopt NURBS, which decouples the number of control points (knots) from the curve order. This approach enables more localized dynamics without significantly affecting the global structure of the curve. Formally, the adopted NURBS equation is defined as follows:
K
(t) UE
C(L)
Σ I=
0:
E (t,
2i=1
where (4) represents the p-th B-spline basis function, and 0g is the i-th control point (knot), while uy is the learnable weight factor. And the basis function (c) is derived using the Cox-de Boor recurrence formula [Boor 1971). as follows:
(1) {1 & Stt< tit1.
(5)
otherwise
1- 4
(1) + litn - L. No+1.m-((t)
(t) fi-n-1 ti
4i4n li41
These segmentations of enable the NURBS curve capture local dynamics without significantly altering the entire curve, making it more effective for exploring the complex loss landscape of this task. We also present the corresponding visualization results of employing NURBS in Figure|ddya d)- Compared to the Poly Chain and Bézier curves, NURBS exhibits greater flexibility, longer low-loss path, and improved MTL performance.
5.3 Scale to Many Task Scenarios
Equations 2 and 3 are specifically designed for two-task scenarios, facilitating trade-offs between the two tasks. To address Challenge
propose warmup-base based task grouping strategy that scales to many-task settings by dividing tasks into three clusters.
Prior to the main training phase, we perform a warmup stage for epochs to collect gradient statistics. For mini-batch, we obtain the set of task gradients (91,.... 9k]. Based on these, we compute
cosine similarity matrix A: A.j
Using A, we construct graph g (V,8), where each node U E V represents task and each edge Eij E corresponds to As for < j S K. We then apply spectral clustering [Ng et al. 2001] to partition G into three clusters P = {E1, E3}
g. g;
After pretraining on clusters ε and εg, we obtain their respective weights θ₁ and θ2, which are used to initialize the endpoints of the connectivity curve. To align the trade-offs across all clusters, we introduce the following alignment objective:
Rulign = -1- L.,
(6)
where C, denotes the average loss of tasks in cluster Eg. The full training objective is then defined as:
((0) Lr RR, + Rolign
This formulation ensures that L, reaches its minimum at = 1/2 while maintaining mode connec- tivity between ε, and Eg. This enables aligned trade-offss across all task clusters the center of the curve. Note that K = 3 is special case where grouping is not required.
Performance Evaluation
In this section, we first evaluate our method using mainstream MTL benchmarks and compare
it with the following baselines: Linear Scalarization (LS). Scale-Invarian (SI), RLW [Lin et al.
2021], DWA [Liu et al.) 2019]. UW [Kendall et al.] 2018). MGDA [Sener and Koltun [018]] GradDrop [Chen et al., 2020]. PCGrad Yuu et al. 2020]. CAGrad [Liu et al. 2021a]. IMTL [Liu
al. 2021b), Nash-MTL [Navnn et al. 2022), FAMO [Liu et al.. 2024] J and FrirGrad (Ban and Ji
2024 We also compare with several PFL approaches, including COSMOS [Ruchte and Grabocka
2021]. HPN (Navon et al. 2020]. and PaMal. (Dimitriadis al.) 2023), to demonstrate EXTRA's
ability to approach the Pareto front. Additionally, we provide further analysis, e.g., ablation study
7

---

Table 2: Scene understanding (NYUV2, 3 tasks). We report MTAN model performance averaged over
random seeds. The best scores are provided gray and the second scores are underlined.
Segmentation
Depth L
Surface Normal
Method
MR Am%
Angle Distance
Within (
mloU Pix. Acc. Abs. Err.
Rel. Err.
Mean
Median
11.25
22.5 30
Independent
38.30
63.76
0.68
0.28
25.01
19.21
30.14 57.20 69.15
LS
39.29 65.33
0.55
0.23
28.15
23.96
22.09
47.50 61.08
10.67
5.46
RLW
37.17 63.77
0.58
28.27
24.18
22.26
47.05 60.62
13.22
7.67
DWA
39.11 65.31
0.55
0.23
27.61
23.18
24.17
50.18 62.39
9.78
3.49
Uncertainty
36.87 63.17
0.54
0.23
27.04
22.61
23.54
49.05 63.65
9.67
4.01
MGDA
30.47 59.90
0.61
0.26
24.88
19.45
29.18
56.88 69.36
7.56
1.47
GradDrop
39.39 65.12
0.55
27.48
22.96
23.38
49.44 62.87
10.11
PCGrad
38.06 64.64
0.56
0.23
27.41
22.80
23.86
49.83
63.14
10.33
3.83
CAGrad
39.79 65,49
0.55
0.23
26.31
21.58
25.61
52.36
65.58
7.56
0.29
IMTL
39.35 65.60
0.54
0.23
26.02
21.19
26.20
53.13
66.24
6.67
-0.59
Nash-MTL
40.13 65.93
0.53
0.22
25.26
20.08
28.40
55.47
68.15
4.11
-4.04
FAMO
40.30 66.07
0.56
0.21
26.67
21.83
25.61
51.78 64.85
6.56
0.16
FairGrad
39.74 66.01
0.54
0.22
24.84
19.60
29.26
56.58 69.16
3.56
.4666
EXTRA-L
40.90
66.67
0.54
25.16
19.88
28.79
55.94
68.50
3.67
.4.41
EXTRA-F
41.89 67.60
0.53
0.22
24.64
19.30
29.66
57.30
69.76
1.56
-6.30
(Sec. B),1,, grouping strategy comparison (Sec. B.2), plug-and-play verification (Sec. B.4), analysis on control point number (Sec. B.5), and deeper insights into the performance. All experiments analysis (Sec. B.6), in Appendix to offer refer to Appendix (Sec. A) for more implementation details.
conducted on Tesla V100 GPUs. Please
Evaluation Metric. In addition to reporting individual performance, we also incorporate widely used metric, Am % [Maninis et al., 2019], which evaluates the overall degradation compared to independently trained models that considered the reference oracles. The formal definition of Am% is given as:
K Σ
Am %
1-1) 1-1)0O0Mm.. Mo,x)/Mo.. x 100
(
K
where Mm, and Mo represent the metric M₁ for the compared method and the independent model, respectively. The value of ôş is assigned 1 if higher value is better for M₁. and otherwise. Besides, we also report another popular metric named Mean Rank (MR), which computes the average ranks of each methods across all tasks.
Table 1: Results on CelebA.
6.1 Overall MTL Evaluation
CelebA
Method
In this section, we evaluate our method on two image classification tasks, CelebA and MultiMNIST, well datasets, CityScapes and NYUv2. CelebA serves as a multi-task dataset assess the model's ability handle large number of tasks, while MultiNNIST is toy dataset used to demonstrate the model's ability cover the Pareto front. Notably, 'EXTRA-L' represents equips with L.S trained endpoints, while FEXTRAFF' represents equips with both LS and FairGrad trained endpoints.
MR |
Am %
$
understanding
LS
8.00
4.15
SI
9.75
7.20
RLW
6.80
1.46
DWA
8.88
2.40
UW
7.48
3.23
MGDA
12,95
14.85
PCGrad
8.60
3.17
CAGrad
8.05
2.48
6.1.1 Scene Understanding
IMTL-G
6.45
0.84
Nash-MTL
6.53
2.84
CityScapes is large-scale dataset designed for scene un- derstanding, which we use as an MTL benchmark focusing on the tasks
CityScapes:
FAMO
6.35
1.21
FairGrad
7.05
1.15
of semantic segmentation and depth estimation. The results, presented in Table demonstrate that EXTRA achieves SOTA performance based on Am,, key metric widely regarded one of the most important in
EXTRA-L
4.15
0. 0.10
EXTRA-F
3.98
-0. -0.11

---

MTL. Additionally, it ranks highly in terms of the MR metric, which evaluates the ranking position of EXTRA among various MTL approaches. Notably, our method is competitive with advanced gradient-based MTI. approaches, highlighting that our novel framework is effective as the alter- native optimization trajectory paradigm and offers a fresh perspective for advancing MTI. research.
NYUv2: NYUv2 is another indoor scene understanding dataset com- monly used for MTL benchmarking, comprising three tasks: semantic seg- mentation, depth estimation, and sur- face normal prediction. We conducted experiments on this dataset, and the results are presented in Table 2 As shown, EXTRA significantly outper- forms counterparts both Am% and MR. Moreover, it achieves top performance on nearly every single task, demonstrating EXTRA's capabil- ity to search for the optimal trade- offs. We attribute this effectiveness to our design, which facilitates travers- ing trade-offs while maintaining low losses along the searched path. Even EXTRA-L exhibits competitive perfor- mance though their endpoints are naively trained with LS.
Table 3: Scene understanding (CityScapes, tasks).
Segmentation f
Depth +
Method
MR 1 Am % +
mloU Pix. Acc. Abs. Err. Rel. Er:.
Independent
74.01 93.16
0.0125
27.77
LS
75.18 93.49
0.0155
46.77
9.25
22.60
RLW
74.57 93.41
0.0158
47.79
12.00
24.37
DWA
75.24 93.52
0.0160
44.37
9.25
21.43
Uncertainty
72.02 92.85
0,0140
30.13
8.25
5.88
MGDA
68.84 91.54
0.0309
33.50
12.00
44.14
GradDrop
75.27 93.53
0.0157
47.54
8.75
23.67
PCGrad
75.13 93.48
0.0154
42.07
9.50
18.21
CAGrad
75.16 93.48
0.0141
37.60
8.50
11.58
IMTL
75.33 93.49
0,0135
38.41
6.75
11.04
Nash-MTL
75.41 93.66
0.0129
35.02
4.00
6.82
FAMO
74.54 93.29
0.0145
32.59
9.00
8.13
FairGrad
75.72 93.68
0.0134
32.25
2.50
5.18
EXTRA-L EXTRA-F
75.53 93.63
0.0127
33.45
3.25
4.93
77..11 93.58
0.0126
30.20
2.00
1.63
6.1.2 Image Classification
CelebA: CelebA Liu et al. 2015] is a widely used facial attributes dataset containing over 200,000 images annotated with 40 binary attributes. Recently, it has been adopted as a 40-task multi-task learning (MTL) benchmark to assess model's capacity to handle large number of tasks. Following the experimental setup of prior work [Liu et al. 2024| Ban and Jil 2024]. ). we evaluate EXTRA on CelebA, with results presented in Table FairGrad results are reported from three random runs using the same seeds as EXTRA, based on its official implementation. As shown, EXTRA achieves SOTA performance in both Am% and MR. These results indicate that EXTRA can enhance all individual tasks simultaneously and underscore the effectiveness our grouping strategy in large- scale task scenarios.
CorTeNaNEE Mehoob
MultiMNIST: Following the experimental setup in Pa- MaL. [Dimitriadis et al., [2023]. we evaluate EXTRA on MultiMNIST, a widely used dataset for assessing how MTL/PFL methods approach the Pareto front. The sults are presented in F As shown, the best trade- off identified by EXTRA is competitive with SOTA MTL methods. Additionally, EXTRA achieves broad coverage of trade-offss without sacrificing performance, demonstrating its superior ability to approach the Pareto front compared to its counterparts the 2-task scenario
6559
JMM
MIDA
Sach HITL
4.53
Apto
0320
6335
DCTRA
ale
Figure 6: Results on MultiMNIST.
Conclusion
In this paper, we approach MTL from the perspective of mode connectivity, in contrast traditional gradient-based methods. Specifically, our focus is on exploring the optimal trade-off's that deliver competitive MTL performance, rather than aiming to directly approach the Pareto front. By designing an order-aware objective and employing a more flexible curve (NURBS), we show that optimizing the curve, rather than just the endpoints, both more efficient and effective for MTL. This approach
*We report the results of PFL approaches based on runs of PaMaL's official implementation, while MTL results taken from PaMaL's paper.

---

is empirically validated across multiple mainstream datasets. We hope that our work offers valuable insights for future research in the field of MTL.
Acknowledgement
This research is supported, in part, by the WeBank-NTU Joint Research Institute Fintech, Jinan- NTU Green Technology Research Institute (GreenTRI), and Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY), Nanyang Technological University, Singapore. It's also supported, in part, by the NTU-PKU Joint Research Institute, a collaboration between Nanyang Technological University and Peking University that is sponsored by a donation from the Ng Teng Fong Charitable Foundation. The first author also would like to thank the insightful discussion with Dr. Shuaicheng Niu, Shibo Feng, and Haochen Li.
References
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE Iransactions pattern analysis and machine intelligence, 39(12):2481-249) 2017.
Hao Ban and Kaiyi Ji. Fair resource allocation in multi-task learning. arXiv preprint 17Xi::2402.15638, 2024.
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference Machine Learning, pages 769-779. PMLR, 2021.
C de Boor. Subroutine package for calculating with b-splines. In Los Alamos Sci. Lab. Los Alamos, NM, USA, 1971.
Weiyu Chen and James T Kwok. Efficient pareto manifold learning with low-rank structure. arXiv preprint arXiv:2407.20744 2024.
Zhao Chen, Vijay Badrinarayanan, Chen Yu Lec, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 903. PMLR, 2018.
Zhao Chen, Jiquan Ngiam, Yanping Huang. Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout, Advances in Neural Information Processing Systems, 33:2039-2050, 2020.
Nikolaos Dimitriadis, Pascal Frossard, and François Fleuret. Pareto manifold learning: Tackling multiple tasks via ensembles of single-task models. In International Conference on Machine Learning, pages 8015-8052. PMLR, 2023.
Nikolaos Dimitriadis, Pascal Frossard, and Francois Fleuret. Pareto low-rank adapters: Efficient multi-task learning with preferences. arXiv preprint 20Xi:22407.00056 2024.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309-1318, PMLR, 2018.
Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes. Advances in Neural Information Processing Systems, 32, 2019.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew Wilson. Loss surfaces, mode connectivity, and fast ensembling of dmns. Advances in neural information processing systems, 31, 2018.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482-7491, 2018.
10

---

Baijiong Lin, Feiyang Ye, Yu Zhang, and Ivor W Tsang. Reasonable effectiveness of random weighting: litmus test for multi-task learning. arXiv preprint arXiv:2111.10603 2021.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu, Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878- 2021a.
Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization. Advances in Neural Information Processing Systems, 36, 2024.
Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations, 2021b.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1871-1880, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.
Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mecha- nistic mode connectivity. In International Conference on Machine Learning, pages 22965-23000 PMLR, 2023.
Kevis-Kokitsi Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multiple tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1851-1860, 2019.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning, arXiv preprint arXi::2010.04495, 2020.
David Mueller, Mark Dredze, and Nicholas Andrews. Can optimization trajectories explain multi-task transfer? arXiv preprint arXiv:2408. 14677, 2024.
Aviv Navon, Aviy Shamsian, Gal Chechik, and Ethan Fetaya. Learning the pareto front with hypemnetorkss arXiv preprint arXiv:2010.04104 2020.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. In International Conference on Machine Learning, pages 16428-16448 PMLR, 2022.
Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems, 14, 2001.
Les Piegl and Wayne Tiller. The NURBS book. Springer Science & Business Media, 2012.
Michael Ruchte and Josif Grabocka. Scalable pareto front approximation for deep multi-objective learning. In 2021 IEEE international conference on data mining (ICDM), pages 1306-1311 IEEE, 2021.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
Anke Tang, Li Shen, Yong Luo, Shiwei Liu, Han Hu, and Bo Du. Towards efficient pareto set approximation via mixture of experts based model fusion. arXiv preprint arXiv:2406.09770, 2024.
Jialei Wang, Steven CH Hoi, Peilin Zhao, and Yong Liu. Online multi-task collaborative filtering for on-the-fly recommender systems. In Proceedings of the 7th ACM conference on Recommender systems, pages 237-244, 2013.
Haitao Wen, Haoyang Cheng, Heqian Qiu, Lanxiao Wang. Lili Pan, and Hongliang Li. Optimizing mode connectivity for class incremental learning. In International Conference on Machine Learning, pages 36940-36957. PMLR, 2023.

---

Derrick Xin, Behrooz Ghorbani, Justin Gilmer, Ankush Garg, and Orhan Firat. Do current multi-task optimization methods deep learning even help? Advances in neural information processing systems, 35:13797-13609 2022.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
Chi Zhang. Peilin Zhao, Shuji Hao, Yeng Chai Soh, Bu Sung Lee, Chunyan Miao, and Steven CH Hoi. Distributed multi-task classification: A decentralized online learning approach. Machine Learning, 107(4):727-747, 2018.
Yu Zhang and Qiang Yang. An overview of multi-task learning, National Science Review, 5(1):
30 43, 2018.
Zhipeng Zhou, Liu Liu, Peilin Zhao, and Wei Gong. Pareto deep long-tailed recognition: A conflict- averse solution. In The Twelfth International Conference on Learning Representations, 2024.
Zhipeng Zhou, Liu Liu, Peilin Zhao, and Wei Gong. Injecting imbalance sensitivity for multi- task learning. In Proceedings of the Thirty-Fourth International Joint Conference Artificial Intelligence, IJCAI-25, pages 7236-7244, 2025a.
Zhipeng Zhou, Ziqiao Meng, Pengcheng Wu, Peilin Zhao, and Chunyan Miao. Continual optimization with symmetry teleportation for multi-task learning. arXiv preprint urXiv:2003.04046 2025b.
12

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide the discussion of limitations in the Appendix (Sec. C).
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [NA]
Justification: Code will be released once this paper is accepted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
13

---

Justification:
This paper presents work whose goal is to advance the field of Machine There are many potential societal consequences of work, none which we feel must be specifically highlighted here.
Learning.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
16. Declaration of LLM usage
Question: Does the paper describe the usage of L.MSs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
14

---

A Implementation Details
A.1 Evaluation Protocol
For scene understanding benchmarks such CityScapes and NYUv2, mainstream multi-task learning (MTL) approaches typically report the final performance by averaging results over the last 10 epochs, due to the absence of a validation set. However, this evaluation protocol is not directly applicable to our method, which generates a solution curve during training and evaluates sampled points along the curve to demonstrate superior trade-offss. Specifically, save the trained model at the final epoch and evaluate MTL performance at }. which represents the midpoint of the curve.
2
For image classification benchmarks such as CelebA, we adopt the same training procedure as used for the scene understanding benchmarks. However, we utilize validation set to select the optimal value of i. and report the corresponding MTL performance.
A.2 Experimental Setting
In line with the implementation and training strategy of FairGrad [Ban and Jil 2024). we construct model using SegNet (Badrinarayanan al. 2017]. with MTAN [Liu et al. [219]] employed as the backhone. The model is trained using the Adam optimizer for 200 epochs. The initial learning rate is set le-4 and decayed by a factor of 2 after 100 epochs. The batch size is set 2 for NYUv2 and for CityScapes. The control point numbers are 4 and 5 on CityScapes and NYUv2, respectively, with and trainable in the second stage.
For the CelebA dataset, we adopt a 9-layer convolutional neural network (CNN) as the backbone, with task-specific linear heads appended. The model is trained with the Adam optimizer for 15 epochs using an initial learning rate of 3e-4 and a batch size of 256. The control point number is 3, with 1 trainable in the second stage.
Regarding the MultiMNIST dataset, we follow the protocol described in PaMal. (Dimitriadis et al. 2023]. Each MultiMNIST image is formed by sampling (with replacement) two MNIST digits (28x28), which are placed at the top-left and bottom-right of 36x36 grid. This composite image is then resized to 28x28 pixels. The resulting dataset comprises 60,000 training. 10,000 validation, and 10,000 test samples. The model uses LeNet-style shared-botton architecture: the encoder contains two convolutional layers with 10 and 20 channels (kernel size 5), each followed by max pooling and ReLU activation. The encoder outputs a 50-dimensional embedding. Each decoder consists two fully connected layers, with the final output layer producing predictions over 10 classes. The model trained using Adam with a learning rate of 0.001, no learning rate scheduler, a batch size of 256, and total of 10 training epochs. The control point number is 3, with trainable in the second stage.
B
Additional Experiments
B.1
Ablation Study
Our system comprises multiple components, including the order-aware objective (Ro). alignment objective (Ralign in Eqn.7), and curve selection. To evaluate the effectiveness and rationale behind each component, we conduct an ablation study on the NYUv2 dataset, with the results presented in Table As shown, without Raiign. EXTRA still outperforms the baseline but falls short of the complete system's s performance. This suggests that, while the model can minimize loss, it fails to properly calibrate the remaining tasks / 3. Additionally, EXTRA shows slight improvements without R.. demonstrating the strong capability of NURBS in capturing diverse trade-offs. This observation is further corroborated by the comparison between NURBS and Bézier, which reveals substantial performance gap in MTL.
B.2 Grouping Strategy Comparison
To further demonstrate the effectiveness of the proposed grouping strategy, we compare with two alternative approaches: random grouping and K-means clustering. The corresponding results are shown in Figure 7 In this experiment, the 40 tasks are grouped into 3 clusters using each of the three strategies, followed by multi-task learning (MTL) training. As illustrated, our strategy achieves
15

---

Table Ablation study of EXTRA on NYUv2 (3 tasks).
R
Ralign
Bézier
NURBS
Am %
-4.66
-6,05
-4.97
-4.36
-6.30
the best overall MTL performance. outperforming both random and K-means grouping. Notably, K-means clustering fails to deliver satisfactory results due imbalanced cluster sizes. Specifically, we observe that two of the three K-means clusters contain only a single task, which hinders the exploration of trade-offs under our training framework.
0.9
EXTRA-L
0.8
EXTRA-F
0.7
dml
0.4
0.3
0.2
0.1
Random
K-means
Ours
-0.1
Figure 7: Comparison of grouping strategies on CelebA. 'Random' represents the uniform division for 40 tasks, while K-means represents leveraging K-means to cluster the warmup gradients of 40 tasks.
B.3 Memory Cost Analysis
To evaluate the efficiency and scalability our method compared endpoint- approaches, we statistically analyze their memory consumption during training across various MTL benchmarks, as shown Figure S As depicted, in the 2-task (CityScapes) and 3-task (NYUv2) settings, EXTRA incurs slightly higher memory usage due to its two-stage training paradigm. However, in the large-scale scenario with 40 tasks (CelebA), endpoint-based methods such as PaMaL require substantial memory resources. resulting in out-of- of-memory (OOM) issue during training. In contrast, EXTRA maintains a manageable memory footprint, demonstrating better scalability.
B.4 Plug-and-PP -layy Verification
In addition LS and FairGrad, further incorporate another mainstream MTL approach, CAGrad, to demonstrate the plug-and-play capability of EXTRA, with the corresponding results shown in Table 5 As illustrated, EXTRA also provides significant improvements when applied to CAGrad, following the same trend observed with LS and FairGrad. Specifically, EXTRA not only enhances overall MTL performance but also improves each individual metric, thereby verifying its plug-and-play effectiveness.
B.5 Analysis on Control Point Number
We further analyze the effect of the number bends in the NURBS representation on the CityScapes dataset, with results presented in Figure 10 As shown, EXTRA-L achieves the best performance in terms of Am% when the number of bends is set to 4. Notably, reducing the number of bends to 3 leads to substantial drop in performance, likely due to the limited expressive capacity of the
16

---

200
PaMaL
175
EXTRA
EXTRA-1
Memory Cost (GB)
150
EXTRA-2
125
100
CityScapes
NYUv2
CeiebA
Figure 8: Memory Cost Comparison. 'EXTRA-1' and 'EXTRA-2' represents the first and second training stage of EXTRA. Note that the memory consumption of PaMaL the CelebA dataset is estimated, as its actual training raises an out-of-memory (OOM) issue under our experimental settings.
Table Scene understanding (CiryScapes, 2 tasks). indicates outperforms/underpeefonn. their vanilla versions. 'EXTRA-L', 'EXTRA-C', and "EXTRA-F' EXTRA augmented LS, CAGrad, and FairGrad versions.
Segmentation
Depth |
Method
Am % +
mloU Pix. Acc. Abs. Er.. Rel. Emr.
Independent
74.01
93.16
0,0125
27.77
LS
75.18
93.49
0.0155
46.77
22.60
EXTRA-L
75.53
93.63
0,0127
33.45
4.93
CAGrad
93.48 75.50
75.16
0.0141
37.60
11,58
EXTRA-C
93.55
0.0135 a 35.73
8.61
FairGrad
75.72
93.68
0.0134
32.25
5.18
EXTRA-F
76.11 93.58
0.0126 30.20
1.63
does not yield performance gains, which is somewhat counterintuitive. To investigate this further, we visualize the corresponding loss landscapes in Figure g As illustrated, EXTRA-3 degenerates into Bézier-like curve, exhibiting excessive smoothness. Meanwhile, the curves produced by EXTRA-5 and EXTRA-6 appear irregular and less stable, likely due to the increased difficulty optimization, which in turns highlights the superiority of choosing curve-based rather than endpoint-based mode connectivity for MTL.
NURBS curve. Conversely, increasing the number bends
3
(b) 4
(c)
(d) 6
Figure 9: The loss landscape of employing NURBS with different control point number (including endpoints), We abbreviate them EXTRA-3, EXTRA-4, EXTRA-5, and EXTRA-6.
B.6 Hyper-parameter Analysis
We evaluate the impact of the hyperparameter on the final performance and present the results in Figure [T] As shown, the performance remains relatively stable across different values of a. with the highest average performance achieved at = 0.3. However, 0.5 offers a better trade-off between average performance and variance. Therefore, we adopt a 0.5 the default setting in our experiments.
17

---

10
Amk
3
Number of Endpoints
Figure 10: Analysis of number of bends on CityScapes, tasks.
2.0
1.8
4m%
1.6
1.4
1.2
0.2
0.4
0.6
0.8
a
Figure 11: Analysis on
C Limitation and Discussion
Although EXTRA achieves state-of-the-art performance on mainstream MTL benchmarks, it faces challenges in delivering user-preas pper-preeeeence MTL results when the number of tasks exceeds two. While a Bézier surface can align mode connectivity with the Pareto front for three tasks, this approach becomes progressively more difficult the number of tasks increases, which represents a key limitation of our method. Addressing this limitation will be focal point of future work. Additionally, might raise concerns regarding the faimess of evaluation, given that EXTRA utilizes multiple points (endpoints and control points) to construct the curve, thereby increasing the model's capacity. While we acknowledge this concern, we emphasize that our work introduces novel perspective to MTL, potentially offering an alternative to gradient-based MTL approaches, which have been debated for their effectiveness [Xin et al. 2022
18


Reviews:
[{"review_id": "zZ1QkyEhCC", "reviewer": "Reviewer_uJP2", "summary": "This paper proposes EXTRA (EXplore TRAde-offs) for multi-task learning (MTL) that leverages mode connectivity to find optimal trade-offs between multiple task objectives. Unlike most optimization-based MTL methods that adjust the gradient updates for shared parameters, EXTRA models the parameter space as a curve connecting solutions for individual tasks, efficiently exploring the Pareto front of multi-objective trade-offs.\n\n\nSpecifically, it introduces Non-Uniform Rational B-Splines (NURBS) for flexible curve modeling in the parameter space, improving on previous methods like Bézier curves.\nIt proposes an order-aware objective to structure the search along the curve, promoting better trade-offs. The method achieves SOTA performance across several MTL benchmarks (CityScapes, NYUv2, CelebA, MultiMNIST).", "questions": "1. How well would this approach generalize to other tasks/objectives, model architectures with different loss landscape properties?\n\n2. How does the method compare with existing ones in terms of computational complexity?", "limitations": "- Limited theoretical guarantees on Pareto optimality or convergence.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. Extensive experiments on MTL benchmarks are performed to show the effectiveness of the proposed method.\n\n2. The paper is clearly written.\n\n3. The proposed method seems to provide a new perspective to explicitly consider the landscape of the multi-objective optimization problem.\n\n\n### Weaknesses\n\n\n1. The proposed method with curve optimization seems to add computational overhead. See Questions-2.\n\n\n2. Some of the motivation for the methodology is not clearly discussed. It is unclear why the objective in Section 5.1 is designed in this way. More explanation should be given. \n\n\n3. Mode connectivity and curve optimization is not new. The paper seems to combine and apply existing methods to multi-task learning. It would be better to state the difference of these two in multi-task compared to single-task learning.\n\n\n4. There are some important missing references in multi-task learning/multi-objective optimization and mode connectivity which the authors should discuss. I list a few below as examples.\n\n4.1 The following papers on multi-objective optimization should be discussed.\n\n- Steepest descent methods for multicriteria optimization\n\n- Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance\n\n\n4.2 The following papers on Pareto front approximation with MultiMNIST experiments should be discussed and compared.\n\n- Pareto Multi-Task Learning\n\n- Multi-Task Learning with User Preferences: Gradient Descent with Controlled Ascent in Pareto Optimization", "was_revised": true, "final_justification": "Most of my concerns are addressed in the rebuttal.", "rebuttal": "We thank the Reviewer uJP2's constructive and insightful comments. To address your questions, we provide pointwise responses below.\n\n>* Q1：The proposed method with curve optimization seems to add computational overhead. \n\n  A1: We understand the reviewer’s concern and acknowledge that our method introduces additional computational overhead due to its multi-stage training paradigm. However, we would like to emphasize that this overhead is affordable and justified by the significant benefits it brings.\n\n  First, our approach offers a novel research perspective for MTL, departing from conventional gradient manipulation techniques whose effectiveness has recently been questioned by emerging literature [1,2]. Second, the proposed paradigm enables explicit exploration of task trade-offs, which leads to significant improved performance. Finally, our method is compatible with mainstream MTL approaches, as demonstrated by our plug-and-play experiments, underscoring its flexibility and broad applicability. \n\n  \n\n>* Q2: Some of the motivation for the methodology is not clearly discussed. It is unclear why the objective in Section 5.1 is designed in this way. More explanation should be given.\n\n  A2: This objective is designed to imbue the learned curve with trade-off properties similar to those of the Pareto front. During training, by jointly optimizing both the overall task loss $L_T$ and the regularization term $R_o$, we not only ensure a low aggregated loss across tasks but also promote traversal along different trade-offs, thereby facilitating the discovery of optimal task balances. Specifically, we achieve this by encouraging $L_1$ to monotonically increase and $L_2$ to monotonically decrease as $t$ varies from 0 to 1, while maintaining a low overall task loss $L_T$ along the curve.\n\n>* Q3: Mode connectivity and curve optimization is not new. The paper seems to combine and apply existing methods to multi-task learning. It would be better to state the difference of these two in multi-task compared to single-task learning.\n\n A3: Thank you for your valuable question. We acknowledge that mode connectivity and curve optimization are not novel concepts, and we do not claim them as our contributions. Instead, we provide an overview of related work on these two topics and explicitly clarify the connections and differences between our approach and prior studies in Section 2.\n\n  While mode connectivity has been applied in manifold-based PFL to approach Pareto front, it still faces several challenges. For example, it suffers from degeneration in two-task scenarios and does not scale well to settings involving many tasks. These limitations make manifold-based PFL—what we refer to as **endpoint-based mode connectivity**—less practical for MTL. Therefore, we adopt a **curve-based mode connectivity** approach that directly optimizes the entire curve rather than just the endpoints.\n\n  However, applying curve-based mode connectivity to MTL is not straightforward due to three major issues discussed in Section 4.2: randomness, lack of flexibility, and poor scalability. To address these, we propose three key innovations: an order-aware objective, the adoption of NURBS to define the curve, and a task grouping strategy. Together, these components enable practical and scalable trade-off exploration, leading to improved MTL performance.\n\n  In summary, our main contribution lies in identifying the limitations of applying traditional mode connectivity to MTL and in developing tailored solutions—including new objectives, curve formulations, and task grouping methods—that successfully adapt this framework to the MTL setting and yield strong empirical results.\n\n  \n\n>* Q4: Some references should be compared or discussed.\n\n  A4: Thank you for your valuable question. Similar to the principles underlying CAGrad and FAMO, *Steepest Descent* [3] aims to optimize for the worst-case task, thereby aligning with the notion of adversarial task alignment. *MoDo* [4], on the other hand, can be viewed as a variance-reduced variant of MGDA. Both approaches fall within the category of gradient-based MTL  methods and are conceptually orthogonal to our framework. This distinction is further supported by the plug-and-play verification results provided in Appendix B.4.\n\n  We have additionally conducted comparisons with the referenced methods [3, 4] on the MultiMNIST benchmark, reporting task-wise performance under different preferences. As shown, although Pareto MTL and EPO exhibit broad trade-off capabilities, they often struggle to achieve favorable trade-offs, leading to suboptimal overall MTL performance—similar to COSMOS, as illustrated in Figure 6. These comparative results and analyses will be included in the next version of the paper.\n\n  | **EXTRA**          | 0.912, 0.960 | 0.937, 0.957 | 0.940, 0.952 | 0.958,0.949  | 0.963, 0.925 |\n  | -------------- | ------------ | ------------ | ------------ | ------------ | ------------ |\n  | EPO            | 0.640, 0.950 | 0.861, 0.948 | 0.902, 0.880 | 0.933, 0.850 | 0.944, 0.550 |\n  | Pareto MTL | 0.864, 0.900 | 0.892, 0.887 | 0.907, 0.882 | 0.909, 0.862 | 0.918, 0.848 |\n\n  \n\n>* Q5: How well would this approach generalize to other tasks/objectives, model architectures with different loss landscape properties?\n\n  A5: Thank you for your valuable question. Mode connectivity is a well-established property, as explored in prior works [5,6,7], and has been investigated across various tasks and architectures. In our evaluation, we further verify its applicability across diverse scenarios, including image classification and scene understanding tasks, as well as architectures such as CNN and MTAN. Intuitively, whenever two or more objectives exhibit differing optimization behaviors and trade-offs are desired between them, our proposed approach remains applicable and effective.\n\n  \n\n>* Q6: Limitation: Limited theoretical guarantees on Pareto optimality or convergence. \n\n  A6: We acknowledge that this paper does not provide a formal theoretical analysis of our approach. This is primarily due to the complex loss landscape and the fundamentally different learning paradigm compared to gradient manipulation-based methods. Currently, there is no existing research that offers theoretical guarantees for the convergence of mode connectivity.\n\n  Nonetheless, we would like to offer some intuition in this regard. Consider a two-task scenario as an example. At $t = 0$, we have $\\nabla \\mathcal{L}_1(\\mathcal{C}(0;\\theta)) = 0$, and at $t = 1$, $\\nabla \\mathcal{L}_2(\\mathcal{C}(1;\\theta)) = 0$. This property arises from using pre-trained single-task weights as the endpoints. Suppose we reasonably assume that these two endpoints are the closest local minima to each other, which is supported by empirical evidence suggesting that mode connectivity works well when endpoints are nearby [8]. This also aligns with the intuition in MTL that tasks should be correlated. If this assumption holds, then it follows that $\\nabla \\mathcal{L}_2(\\mathcal{C}(0;\\theta)) \\cdot \\nabla \\mathcal{L}_1(\\mathcal{C}(1;\\theta)) < 0$.\n\n  Now define $F(t) = \\nabla \\mathcal{L}_1(\\mathcal{C}(t;\\theta)) + \\nabla \\mathcal{L}_2(\\mathcal{C}(t;\\theta))$. Due to the $C^1$ continuity of NURBS and the differentiability of $\\mathcal{L}_1$ and $\\mathcal{L}_2$, $F(t)$ is continuous on the interval $[0, 1]$. By the Bolzano–Cauchy Theorem, there exists $t' \\in (0,1)$ such that\n  $\n  F(t') = \\nabla \\mathcal{L}_1(\\mathcal{C}(t';\\theta)) + \\nabla \\mathcal{L}_2(\\mathcal{C}(t';\\theta)) = 0,\n  $\n  which implies the existence of a Pareto stationary point.\n\n  We emphasize that this is not a rigorous theoretical analysis, but rather a conceptual insight. Nevertheless, our contribution lies in introducing a novel perspective to MTL, distinct from gradient manipulation-based approaches. This is particularly meaningful given the ongoing debate over the practical effectiveness of gradient-based MTL methods [1,2]. Furthermore, we properly adapt and apply mode connectivity to MTL, differing significantly from its prior use in PFL.\n\n  \n\n  **Reference**:\n\n  [1] Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglement, arXiv 2024;\n\n  [2] Can Optimization Trajectories Explain Multi-Task Transfer?, arXiv 2024;\n\n  [3] Steepest descent methods for multicriteria optimization. Mathematical methods of operations research, 2000.\n\n  [4] Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance. NeurIPS 2023.\n\n  [5] Loss surfaces, mode connectivity, and fast ensembling of dnns. NeurIPS 2018.\n\n  [6] Mechanistic mode connectivity. ICML 2023.\n\n  [7] Optimizing mode connectivity for class incremental learning. ICML 2023.\n\n  [8] Optimizing Mode Connectivity via Neuron Alignment. NeurIPS 2020."}, {"review_id": "l3ownqmCxs", "reviewer": "Reviewer_KFeN", "summary": "This paper presents a novel multi-task learning (MTL) framework based on mode connectivity, where the goal is to optimise the curve that connects task-specific endpoints (i.e., task-specific weights), rather than optimising the endpoints themselves as in conventional MTL methods. This leads to a curve-based optimisation strategy for Pareto front approximation, offering an alternative to standard gradient- and weighting-based MTL methods. The authors also provide a thoughtful discussion of the challenges specific to curve-based Pareto optimisation, along with design improvements to mitigate them. The proposed method is evaluated on multiple multi-task benchmarks across scene understanding and image classification tasks.", "questions": "See weaknesses.", "limitations": "No. would be great if the authors could discuss some remaining challenges that need to be solved.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper is well written and clearly structured. The proposed approach is both unique and insightful, and the connections drawn to existing MTL paradigms are well articulated. Visualisations effectively support different design insights, particularly in how the design addresses different optimisation challenges.\n\n[Weaknesses and Discussion Points]\n- Given the stochastic nature of curve-based optimisation, how sensitive is the final learned control point to different choices of hyperparameters, network architectures, or initialisations? Some empirical evidence or discussion on training stability would be helpful.\n- Can we extract any structural insights from the learned curves themselves? For example, if the final Pareto curve exhibits low curvature (i.e., nearly linear), might that imply the tasks are closely aligned or share similar representations? Conversely, does high curvature suggest more divergent task optima? \n- Some MTL baselines, particularly those using second-order gradients or iterative local optimisation (e.g., Auto-Lambda, PCGrad variants), can be computationally expensive. How does the runtime of the proposed method compare to such baselines? An analysis of training time or convergence speed would provide a more complete evaluation.\n- Figure 6 is difficult to interpret and contributes little beyond what simpler metrics already show. Since Multi-MNIST is a relatively easy dataset with small performance differences, it may not be the best case for visualising Pareto cruves. I recommend including similar visualisations for more complex datasets, where task trade-offs are more prominent and visual differences are more informative.", "was_revised": true, "final_justification": "The authors have sufficiently resolved my concerns, and therefore I am happy to keep the original rating.", "rebuttal": "We thank the Reviewer KFeN's constructive and insightful comments, and recognize our contributions. To address your questions, we provide pointwise responses below.\n\n> * Q1: Given the stochastic nature of curve-based optimisation, how sensitive is the final learned control point to different choices of hyperparameters, network architectures, or initialisations? Some empirical evidence or discussion on training stability would be helpful.\n\nA1: Thank you for your valuable question. We have evaluated our method across various tasks (e.g., image classification, scene understanding), architectures (e.g., CNN, MTAN), and hyperparameters (see Figures 9–11). To further address the reviewer's concern, we have conducted additional experiments focusing on initialization strategies. Specifically, the initialization of the curve is determined by the initialization of control points. In our current design, the control points are randomly initialized. For comparison, we consider a linear initialization scheme, where control points are initialized via linear interpolation between two pre-trained endpoints. The results on the Cityscapes dataset are presented below.\n\nAs observed, although linear initialization yields slightly worse performance compared to random initialization, it still achieves SOTA MTL performance. This may be attributed to the complex nature of NURBS curves, which differ significantly from linear curves, suggesting that linear initialization might not be optimal for our framework.\n\n  |                        | mIoU  | Pix. Acc. | Abs. Err. | Rel. Err. | $\\Delta$ m % |\n  | ---------------------- | ----- | --------- | --------- | --------- | ---------- |\n  | Linear initialisations | 73.91 | 93.42     | 0.0129    | 29.40     | 2.25       |\n  | Random initialisations | 76.11 | 93.58     | 0.0126    | 30.20     | 1.63       |\n\n> * Q2: Can we extract any structural insights from the learned curves themselves? For example, if the final Pareto curve exhibits low curvature (i.e., nearly linear), might that imply the tasks are closely aligned or share similar representations? Conversely, does high curvature suggest more divergent task optima?\n\nA2:  Thank you for your valuable question. Intuitively, low curvature may indicate a high degree of correlation between tasks, whereas high curvature suggests lower task correlation. Although, to the best of our knowledge, no existing work directly establishes this connection, some studies offer relevant insights. For instance, [1] demonstrates that if two models (i.e., local minima) learn similar functions, a low-loss connecting path between them is more likely to exist. Nevertheless, this observation is not yet conclusive, and we plan to explore this relationship more thoroughly in future work.\n\n  \n\n> * Q3: Some MTL baselines, particularly those using second-order gradients or iterative local optimisation (e.g., Auto-Lambda, PCGrad variants), can be computationally expensive. How does the runtime of the proposed method compare to such baselines? An analysis of training time or convergence speed would provide a more complete evaluation.\n\nA3: Thank you for your valuable question. As suggested, we compare the training cost of our method, **EXTRA-L**, with **PCGrad**, and present the results below. All training times are measured using a single A100 GPU. As observed, although EXTRA-L incurs additional cost compared to its baseline (LS) due to the multi-stage training paradigm, it remains slightly more efficient than PCGrad.\n\n  |         | Time COST (minus) |\n  | ------- | ----------------- |\n  | LS      | 140.2             |\n  | PCGrad  | 508.8              |\n  | **EXTRA-L** | **475.6**             |\n\n   \n\n> * Q4: Figure 6 is difficult to interpret and contributes little beyond what simpler metrics already show. Since Multi-MNIST is a relatively easy dataset with small performance differences, it may not be the best case for visualising Pareto cruves. I recommend including similar visualisations for more complex datasets, where task trade-offs are more prominent and visual differences are more informative.\n\nA4: Thank you for your valuable suggestion. Due to policy constraints, we are unable to upload any figures at this stage. Therefore, we provide the task-wise performance at each preference in tabular form instead. According to the listed results, all methods achieve certain trade-offs, but EXTRA obtains the best overall performance due to its effective exploration of the trade-off space.\n\nCurrently, the widely used two-task PFL datasets (e.g., MNIST variants, Census, etc.) are all small in scale and thus face the same limitation you pointed out. We will explore more complex datasets as alternatives for a more meaningful evaluation of Pareto fronts in future work.\n\n  | EXTRA  | 0.912, 0.960 | 0.937, 0.957 | 0.940, 0.952 | 0.958,0.949  | 0.963, 0.925 |\n  | ------ | ------------ | ------------ | ------------ | ------------ | ------------ |\n  | PaMaL  | 0.927, 0.933 | 0.932, 0.936 | 0.940, 0.939 | 0.945, 0.930 | 0.944, 0.915 |\n  | COSMOS | 0.825, 0.912 | 0.898, 0.908 | 0.910, 0.900 | 0.921, 0.890 | 0.923, 0.770 |\n\n  \n\n> * Q5: Limitation: No. Would be great if the authors could discuss some remaining challenges that need to be solved.\n\n  A5: We would like to respectfully point out that the discussion of our method's limitations is explicitly provided in Section C of the appendix accompanying the main submission. We chose to include it in the appendix to maintain a more focused presentation of experimental results in the main text. In future revisions, we will consider moving this discussion into the main body for greater visibility.\n\n**Reference**:\n\n[1] Git Re-Basin: Merging Models modulo Permutation Symmetries. ICLR 2023."}, {"review_id": "jUBT79GeP4", "reviewer": "Reviewer_XCec", "summary": "The authors propose a novel curve-based optimization for locating the MTL tradeoffs, in contrast to previous end-point-based approaches. Discover and tackle 3 challenges of curve-based mode connectivity. Extensive experiments show superior performance of the proposed methods compared to various baselines, including single-point MTL as well as pareto front learning.", "questions": "1) In Figure 4 (a)-(d), it is not very clear what the x- and y-axes represent in the loss landscape plots.\n\n2) In Line 217, the authors state “Figure 4(c)(g) provides a preview of the benefits of our approach”, where (c)(g) correspond to the order-aware Bézier curve. It is confusing if the order-aware Bézier curve is the authors' approach or if it is the NURBS curve (corresponding to (d)(h)).\n\n3) In Equation 2, there is no assumption on the relative magnitude of losses at t1 and t2. Is this made on-purpose? If the loss at t1 is made smaller than that at t2 significantly the objective is still minimized.\n\n4) The purpose of the additional objective defined in Equation 2 seems still unclear. By minimizing the objective, it seems that we wish the losses at two sampled points t1 and t2 be similar in magnitude?\n\n5) In Equation 2, why does the first term contain (t2-t1) while the second term has a reversed (t1-t2), that is t1 comes first? What is the underlying principle and how does it extend to three or more tasks?\n\n6) How does the method work in a two-task scenario, where it is impossible to make three clusters?\n\n7) In ablation (e.g. Section B.3), does EXTRA correspond to EXTRA-L or EXTRA-F? This is important in terms of training efficiency as to my knowledge and experience FairGrad requires significantly larger training time due to per-task gradient sampling as well as its quadratic optimization.\n\n8) In Line 591, the authors state “we observe that two of the three K-means clusters contain only a single task”. If a cluster contains only a single task, how does MTL training method such as FairGrad apply? As it degenerates to a STL on this particular cluster. I suppose no MTL is done in such a case?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1) Proposing a curve-based mode connectivity approach tailored for MTL is novel, as vanilla methods (such as Bézier curves) have significant limitations. This brings a new view to the MTL field, which I personally think is valuable.\n\n2) Identifies three key challenges of curve-based mode connectivity for MTL, with experimental evidence, bringing insights to the community. Presents three designs tailored to resolve the three challenges, demonstrating a systematic and well-motivated algorithmic design. \n\n3) Extensive experiments show superior performance of the proposed algorithm, setting a new SoTA for MTL.\n\n\n### Major Weaknesses\n1) The authors state that the additional objective (Equation 2) is introduced to resolve challenge 1. While from ablations we see the importance of this additional objective for performance, it is unclear if this objective really tackles challenge 1. Specifically, a plot similar to Figure 4, which is used to illustrate challenge 1, is desirable, with and without the additional objective. Anyhow the claim that Equation 2 resolves challenge 1 requires careful analysis to be backed.\n\n2) The motivational examples (Figure 4) to illustrate challenge 1 and 2 should be conducted on more than one dataset to show the generality of the observation. This is crucial if a scientific claim is to be made.\n\n3) The effect of the number of control points shall be studied more carefully on more settings (such as another dataset or another model). It is unclear if the number 4 is an in general best setting and the phenomenon of irregular curves with higher number of control points hold for other settings.\n\n\n### Minor Weaknesses\n1) Line 97, missing space after SPRO, and some others such as in Line 107. This is inconsistent with e.g. Line 98, CBFT, I recommend being consistent.\n\n2) Line 146, “we introduce the our application …” is a grammar error.\n\n3) Baselines such as MOCO, AlignedMTL, STCH, are all recent representative MTL algorithms, what is the particular reason that they are not included in the comparison to facilitate evaluation that is more up-to-date?\n\n4) The axis labels in Figure 4 and 9 are very small, making them a bit hard to read.\n\n5) In Line 578, the symbol “L_ast” is never stated, and I am not sure what it corresponds to. Do you mean L_o? Also L_align and L_o in this section is inconsistent with R_align and R_o in the main paper.", "was_revised": true, "final_justification": "The authors carefully responded to each of my questions and concerns, and indeed cleared most of my doubts. The experiments are extended, which convinces the reader that the proposed method indeed works great. The revision is needed to include extended results and clarify points referred to in our discussion.", "rebuttal": "We thank the Reviewer XCec's constructive and insightful comments. To address your questions, we provide pointwise responses below.\n\n>* Q1: The authors state that the additional objective (Equation 2) is introduced to resolve challenge 1. While from ablations we see the importance of this additional objective for performance, it is unclear if this objective really tackles challenge 1. Specifically, a plot similar to Figure 4, which is used to illustrate challenge 1, is desirable, with and without the additional objective. Anyhow the claim that Equation 2 resolves challenge 1 requires careful analysis to be backed.\n\nA1: As shown in Figures 4(f) and 4(g), the points along the curve generally demonstrate strong MTL performance, which can be attributed to mode connectivity. However, due to randomness (Challenge 1), Figure 4(f) also exhibits fluctuations in the trade-off performance (reflected in $\\Delta m%$). To address this, our order-aware objective $R_o$ enables the trade-off to be more evenly traversed along the curve, thereby mitigating this challenge. An ablation study of $R_o$, presented in Table 4 (Appendix), further highlights its contribution to our system. We apologize for the mislabeling of $R_o$ as $L_o$ in Table 4. Lastly, we would like to clarify that Figure 4 presents experimental results on the Cityscapes dataset, which we believe holds strong practical relevance.\n\n>* Q2: The motivational examples (Figure 4) to illustrate challenge 1 and 2 should be conducted on more than one dataset to show the generality of the observation. This is crucial if a scientific claim is to be made.\n\nA2:  Due to policy constraints, we are unable to upload these figures at this time. Therefore, we list and compare the MTL performance on **NYUv2** between **vanilla Bézier** and **Bézier + order-aware**, and present their sampled results below. As shown, a similar trend is observed to that on the CityScapes dataset.\n\n  | t                    | 0.1   | 0.3   | 0.5   | 0.7   | 0.9   |\n  | -------------------- | ----- | ----- | ----- | ----- | ----- |\n  | Vanilla Bezier       | -1.22 | -0.67 | -3.60 | -4.00 | -3.42 |\n  | Bezier + order-aware | -1.01 | -3.52 | -4.36 | -3.81 | -3.35 |\n\n>* Q3: The effect of the number of control points shall be studied more carefully on more settings (such as another dataset or another model). It is unclear if the number 4 is an in general best setting and the phenomenon of irregular curves with higher number of control points hold for other settings.\n\nA3: Thank you for your valuable question. In Section A.2 of the Appendix, we provide the control point settings for each dataset: 4 for Cityscapes, 5 for NYUv2, and 3 for both CelebA and MultiMNIST. To further address your concern, we conducted additional experiments on NYUv2 with varying numbers of control points, and present the results below. Overall, all variants outperform the baseline, showing a consistent trend with Cityscapes (Figure 9 in the Appendix): using fewer control points reduces flexibility, while using too many introduces optimization difficulties. Both extremes lead to suboptimal performance.\n\n  | Control Number | $\\Delta m %$ |\n  | -------------- | ---------- |\n  | 4              | -6.15      |\n  | 5              | -6.30      |\n  | 6              | -5.57      |\n\n  \n\n>* Q4: Why don't include more update-to-date approaches, e.g., MoCo, AlignedMTL, and STCH, etc.?\n\nA4: Thank you for your valuable suggestion. We have included these methods in our comparison, with the results presented below (* indicates values reported in the original papers). As shown, EXTRA continues to demonstrate superior performance. It is important to note that the performance of STCH on the CityScapes dataset is based on our re-implementation, as the original paper did not report results for this dataset. Therefore, this result may not accurately reflect its true performance. Due to time constraints, we were unable to perform thorough hyperparameter tuning. However, the result for STCH on NYUv2 is taken directly from their original paper.\n\n  |           | CityScapes | NYUv2 |\n  | --------- | ---------- | ----- |\n  | MoCo*     | 9.90       | 0.16  |\n  | AlignMTL* | -          | -4.93 |\n  | STCH      | 12.37      | -1.35 |\n  | EXTRA-F   | 1.63       | -6.30 |\n\n  \n\n>* Q5: In Line 578, the symbol $L_{ast}$ is never stated, and I am not sure what it corresponds to. Do you mean $L_o$? Also $L_{\\text{align}}$ and $L_o$ in this section is inconsistent with $R_{\\text{align}}$ and $R_o$ in the main paper.\n\nA5: We apologize for the typos. In this paragraph, $L_o$ and $L_{\\text{align}}$ should be $R_o$ and $R_{\\text{align}}$, respectively. Additionally, $L_{\\text{ast}}$ refers to the same objective as $R_{\\text{align}}$.\n\n  \n\n>* Q6: In Figure 4 (a)-(d), it is not very clear what the x- and y-axes represent in the loss landscape plots.\n\nA6: Our visualization of implementation is adapted from [1]. Suppose we have three weight vectors (control points) $w_1$, $w_2$, $w_3$. We set $u = (w_2 - w_1)$, $v = (w_3 - w_1) - \\langle w_3 - w_1, w_2 - w_1 \\rangle / \\\\| w_2 - w_1 \\\\|^2 \\cdot (w_2 - w_1)$. Then the normalized vectors $\\hat{u} = u / \\\\| u \\\\|$, $\\hat{v} = v / \\\\| v \\\\|$ form an orthonormal basis in the plane containing $w_1, w_2, w_3$. To visualize the loss in this plane, we define a Cartesian grid in the basis $\\hat{u}, \\hat{v}$ and evaluate the networks corresponding to each of the points in the grid. A point P with coordinates $(x, y)$ in the plane would then be given by $P = w_1 + x \\cdot \\hat{u} + y \\cdot \\hat{v}$.\nWe will explicitly involve these explaination in our next version.\n\n  \n\n>* Q7: In Line 217, the authors state “Figure 4(c)(g) provides a preview of the benefits of our approach”, where (c)(g) correspond to the order-aware Bézier curve. It is confusing if the order-aware Bézier curve is the authors' approach or if it is the NURBS curve (corresponding to (d)(h)).\n\nA7: Figure 4(c) and 4(g) illustrate the effect of incorporating the order loss into the Bézier curve, highlighting its benefits for trade-off exploration. We will revise the text to explain this point more explicitly.\n\n  \n\n>* Q8: In Equation 2, there is no assumption on the relative magnitude of losses at t1 and t2. Is this made on-purpose? If the loss at t1 is made smaller than that at t2 significantly the objective is still minimized.\n\nA8: Currently, we have not explicitly addressed the issue of loss magnitude. However, we fully understand the reviewer's concern. Prior studies have indeed highlighted the issue of scale differences between task losses, which can cause imbalance during the optimization of $R_o$. Nonetheless, based on our evaluation results, the current formulation already achieves the intended trade-off effect. We appreciate the reviewer’s valuable suggestion and will consider investigating this issue in future work.\n\n  \n\n>* Q9: The purpose of the additional objective defined in Equation 2 seems still unclear. By minimizing the objective, it seems that we wish the losses at two sampled points t1 and t2 be similar in magnitude?\n\nA9: We would like to clarify that this objective is designed to endow the learned curve with trade-off property between tasks similar to the that owns by Pareto front. During training, by optimizing this objective, we can not only maintain the low loss of overall task loss, but also encourage the traverse trade-offs, thereby assisting explore the best trade-off. Specifically, we achieve this by encouraging $L_1$ to be monotonically increasing (i.e., $L_1(t_1) < L_1(t_2)$, if $t_2 > t_1$) and $L_2$ monotonically decreasing (i.e., $L_2(t_1) > L_2(t_2)$, if $t_2 > t_1$) when t ranges from [0, 1]. \n\n  \n\n>* Q10: How does Equation 2 extend to three or more tasks? How does the method work in a two-task scenario, where it is impossible to make three clusters?\n\nA10:  We would like to clarify that we employ Eqn. (3) to tackle two-task scenarios, while employ Eqn. (7) (with additional designed $R_{\\text{align}}$ and grouping strategy if encounter massive tasks) to tackle many-task scenarios. \n\n  \n\n>* Q11: In ablation (e.g. Section B.3), does EXTRA correspond to EXTRA-L or EXTRA-F? This is important in terms of training efficiency as to my knowledge and experience FairGrad requires significantly larger training time due to per-task gradient sampling as well as its quadratic optimization.\n\nA11:  The EXTRA implementation discussed in Section B.3 specifically refers to EXTRA-F, which is included to highlight its memory-efficient property in comparison to endpoint-based mode connectivity approaches. We acknowledge the reviewer's concern regarding the potential inefficiency of EXTRA-F in terms of training time, given that it adopts FairGrad as its baseline. This is a valid concern. However, we would like to provide further clarification from two perspectives: (1) EXTRA-L already achieves SOTA performance, demonstrating the strength of our proposed method. (2) The EXTRA framework is orthogonal to existing MTL approaches. Even when combined with strong SOTA baselines, it consistently yields significant improvements, as confirmed by our plug-and-play experiments presented in Section B.4 (Appendix).\n\n  \n\n>* Q12: In Line 591, the authors state “we observe that two of the three K-means clusters contain only a single task”. If a cluster contains only a single task, how does MTL training method such as FairGrad apply? As it degenerates to a STL on this particular cluster. I suppose no MTL is done in such a case?\n\nA12: Under this scenario, clusters containing only a single task do not participate in the first training stage. Instead, they are solely used to compute the overall task loss $L_T$ and the alignment regularization term $R_{\\text{align}}$ in Equation (7).\n\n  \n>* Q13: Typos or unclear figures.\n\nA13: Many thanks to your reminder, we will fix them and proofread the paper.\n\n  **Reference**:\n\n  [1] Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. NeurIPS 2018."}, {"review_id": "bAxRD9MQLc", "reviewer": "Reviewer_5udz", "summary": "The paper introduces EXTRA, a method that explores optimal trade-offs in multi-task learning by learning a parameter-space curve connecting task-specific models, such that all points along the curve achieve low losses across tasks. Unlike Pareto Front Learning approaches that optimize multiple endpoints, EXTRA focuses on the loss-loss path itself, enabling efficient discovery of Pareto-optimal solutions. The authors identify that Bézier curves lack local flexibility, making them insufficient for modeling nuanced trade-offs in deep loss landscapes. Separately, they highlight that endpoint-based methods suffer from poor scalability as the number of tasks grows. To address both issues, EXTRA employs non-uniform rational B-splines (NURBS) for mode connectivity, which provide finer local control over the curve while keeping memory usage efficient. Experiments on diverse multi-task benchmarks demonstrate that EXTRA achieves state-of-the-art performance and remains compatible with existing MTL methods.", "questions": "Please refer to the weaknesses section of the paper. I’m particularly interested in how the reviewers will respond to these points.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Novel use of mode connectivity for MTL Optimization: Unlike prior Pareto Front Learning (PFL) approaches that optimize multiple endpoints, EXTRA leverages mode connectivity by directly optimizing the path between endpoints, enabling efficient discovery of optimal trade-off points.\n2. Increased flexibility through NURBS curves: The paper addresses the global smoothness limitation of traditional Bézier curves by adopting Non-Uniform Rational B-Splines (NURBS), which allow for localized adjustments along the curve and better adaptation to complex loss landscapes.\n3. Plug-and-play compatibility: EXTRA can be seamlessly integrated with existing multi-task learning methods (e.g., Linear Scalarization, FairGrad, CAGrad), demonstrating strong modularity and practical applicability across different optimization strategies.\n\nWeaknesses\n1. The use of a single continuous curve assumes a smooth task relationship in parameter space, which may not hold in complex real-world MTL settings with nonlinear or conflicting task interactions. This is related with EXTRA is primarily designed for two-task scenarios by learning a curve between two endpoints. Extending to settings with three or more tasks or modeling high-dimensional Pareto fronts (e.g., via surfaces) remains challenging. The paper avoid this by task grouping, I'm not sure it is enough.\n2. The method requires separate training of single-task endpoints followed by curve optimization, introducing additional computational cost and implementation complexity compared to end-to-end approaches.\n3. The effectiveness of the proposed grouping strategy relies on gradient similarity computed during the warm-up stage. Inaccurate grouping due to noisy gradients or ill-defined task similarity may negatively affect performance.\n4. The paper omits evaluation against recent state-of-the-art optimization-based MTL methods such as Go4align [1] and Selective Task Group Updates [2], which provide finer-grained control over task gradients and group-level optimization. Including such baselines would better position EXTRA in the evolving MTL landscape.\n\n[1] Go4align: Group optimization for multi-task alignment\n[2] Selective Task Group Updates for Multi-Task Optimization\n\n5. Despite introducing a novel empirical approach, the paper lacks formal theoretical guarantees. There is no analysis of convergence, approximation quality to the Pareto front, or how the order-aware loss formally shapes the learned curve. This leaves the method’s success purely justified by empirical performance, without insight into when or why it should be expected to succeed.", "was_revised": true, "final_justification": "The paper’s contributions merit acceptance, provided the authors address my concerns in a future revision.", "rebuttal": "We thank the Reviewer 5udz's constructive and insightful comments. To address your questions, we provide pointwise responses below.\n\n>* Q1: The use of a single continuous curve assumes a smooth task relationship in parameter space, which may not hold in complex real-world MTL settings with nonlinear or conflicting task interactions. This is related with EXTRA is primarily designed for two-task scenarios by learning a curve between two endpoints. Extending to settings with three or more tasks or modeling high-dimensional Pareto fronts (e.g., via surfaces) remains challenging. The paper avoid this by task grouping, I'm not sure it is enough.\n\n  A1: We appreciate the reviewers' concerns but believe some misunderstandings have arisen. We address these concerns point by point: (1) Achieving a completely smooth task relationship on a single curve is indeed challenging. As shown in Figure 4 of the main text, not all points on the trained curve lie within the low-loss region. Without the curve generated by our method, a satisfactory trade-off cannot be obtained. This is why we employ an order-aware objective and NURBS. The order-aware objective encourages exploration of multi-task learning (MTL) trade-off points, which may be widely separated, while the flexibility of NURBS compensates for this. It is important to clarify that we do not expect the trained curve to fully realize the MTL trade-off; rather, we aim to optimize the curve to approximate these trade-off points as closely as possible. Experimental results demonstrate the effectiveness of this approach for MTL. (2) Extending this paradigm to settings with many tasks is indeed challenging. However, our insights are consistent with the two referenced papers [1, 2], both of which group tasks based on inherent correlations and achieve balanced learning across groups.\n\n  \n\n>* Q2: The method requires separate training of single-task endpoints followed by curve optimization, introducing additional computational cost and implementation complexity compared to end-to-end approaches.\n\n  A2: We understand the reviewer’s concern and acknowledge that our method introduces additional computational overhead due to its multi-stage training paradigm. However, we would like to emphasize that this overhead is affordable and justified by the significant benefits it brings.\n\n  First, our approach offers a novel research perspective for MTL, departing from conventional gradient manipulation techniques whose effectiveness has recently been questioned by emerging literature [3,4]. Second, the proposed paradigm enables explicit exploration of task trade-offs, which leads to significant improved performance. Finally, our method is compatible with mainstream MTL approaches, as demonstrated by our plug-and-play experiments, underscoring its flexibility and broad applicability. \n\n  \n\n>* Q3: The effectiveness of the proposed grouping strategy relies on gradient similarity computed during the warm-up stage. Inaccurate grouping due to noisy gradients or ill-defined task similarity may negatively affect performance.\n\n  A3: Thank you for the valuable question. We appreciate the reviewers’ concerns. In this work, we simply propose a grouping strategy to demonstrate the effectiveness of our framework. As shown in Figure 7 (Appendix), we compare multiple grouping methods, and the results indicate that even random task division can yield competitive performance. Compared to the random strategy, our method groups more similar tasks together, which leads to improved outcomes.\n\n  \n\n>* Q4: The paper omits evaluation against recent state-of-the-art optimization-based MTL methods such as Go4align [1] and Selective Task Group Updates [2], which provide finer-grained control over task gradients and group-level optimization. Including such baselines would better position EXTRA in the evolving MTL landscape.\n\n  A4: Thank you for the valuable question. We agree that it is important to include these methods in the comparison. Below, we present the results (* indicates values reported in their original paper). It is worth noting that SelectiveMTL appears to perform poorly on CityScapes and NYUv2. This is likely because the original SelectiveMTL paper evaluates on different datasets than ours. Therefore, we re-implemented the method based on their official code. However, this comparison may not be entirely fair. Due to time constraints, we were unable to conduct thorough hyperparameter tuning, but we plan to include a more carefully tuned comparison with SelectiveMTL in a future version of this work.\n\n|              | CityScapes | NYUv2 |\n| ------------ | ---------- | ----- |\n| SelectiveMTL | 15.28      | -0.22 |\n| GO4Align*    | 8.11       | -6.08 |\n| EXTRA-F      | 1.63       | -6.30 |\n\n  \n\n>* Q5: Despite introducing a novel empirical approach, the paper lacks formal theoretical guarantees. There is no analysis of convergence, approximation quality to the Pareto front, or how the order-aware loss formally shapes the learned curve. This leaves the method’s success purely justified by empirical performance, without insight into when or why it should be expected to succeed.\n\n  A5: We acknowledge that this paper does not provide a formal theoretical analysis of our approach. This is primarily due to the complex loss landscape and the fundamentally different learning paradigm compared to gradient manipulation-based methods. Currently, there is no existing research that offers theoretical guarantees for the convergence of mode connectivity.\n\n  Nonetheless, we would like to offer some intuition in this regard. Consider a two-task scenario as an example. At $t = 0$, we have $\\nabla \\mathcal{L}_1(\\mathcal{C}(0;\\theta)) = 0$, and at $t = 1$, $\\nabla \\mathcal{L}_2(\\mathcal{C}(1;\\theta)) = 0$. This property arises from using pre-trained single-task weights as the endpoints. Suppose we reasonably assume that these two endpoints are the closest local minima to each other, which is supported by empirical evidence suggesting that mode connectivity works well when endpoints are nearby [5]. This also aligns with the intuition in MTL that tasks should be correlated. If this assumption holds, then it follows that $\\nabla \\mathcal{L}_2(\\mathcal{C}(0;\\theta)) \\cdot \\nabla \\mathcal{L}_1(\\mathcal{C}(1;\\theta)) < 0$.\n\n  Now define $F(t) = \\nabla \\mathcal{L}_1(\\mathcal{C}(t;\\theta)) + \\nabla \\mathcal{L}_2(\\mathcal{C}(t;\\theta))$. Due to the $C^1$ continuity of NURBS and the differentiability of $\\mathcal{L}_1$ and $\\mathcal{L}_2$, $F(t)$ is continuous on the interval $[0, 1]$. By the Bolzano–Cauchy Theorem, there exists $t' \\in (0,1)$ such that\n  $\n  F(t') = \\nabla \\mathcal{L}_1(\\mathcal{C}(t';\\theta)) + \\nabla \\mathcal{L}_2(\\mathcal{C}(t';\\theta)) = 0,\n  $\n  which implies the existence of a Pareto stationary point.\n\n  We emphasize that this is not a rigorous theoretical analysis, but rather a conceptual insight. Nevertheless, our contribution lies in introducing a novel perspective to MTL, distinct from gradient manipulation-based approaches. This is particularly meaningful given the ongoing debate over the practical effectiveness of gradient-based MTL methods [3,4]. Furthermore, we properly adapt and apply mode connectivity to MTL, differing significantly from its prior use in PFL.\n\n  \n\n  \n\n  **Reference**:\n\n  [1] Go4align: Group optimization for multi-task alignment. NeurIPS 2024.\n\n  [2] Selective Task Group Updates for Multi-Task Optimization. ICLR 2025.\n\n  [3] Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglement, arXiv 2024;\n\n  [4] Can Optimization Trajectories Explain Multi-Task Transfer?, arXiv 2024;\n\n  [5] Optimizing Mode Connectivity via Neuron Alignment. NeurIPS 2020."}]

Author rebuttal:

Meta-review: This paper initially received mixed scores and ultimately reached the upper borderline, leaning toward acceptance. The discussion period was productive in addressing several concerns raised by the reviewers. This AC also shares the strengths highlighted by the reviewers: (1) the paper is well written and clearly organized; (2) it introduces a novel use of curve-based mode connectivity for MTL, which identifies challenges of curve-based mode connectivity and addresses them; and (3) it provides insightful takeaways supported by evaluations and analysis. While some reviewers noted weaknesses, this AC believes the paper’s strengths far outweigh these issues, which can be resolved in a revised version. Accordingly, this AC concurs with the reviewers’ positive views and recommends acceptance of the paper. As a minor note, this AC also observed small typos while going through the manuscript: line 143 (“.” should have spacing before it), line 153 (“staeg”), and several spacing inconsistencies before citations.
Decision: Accept (poster)