Continual Optimization with Symmetry Teleportation for Multi-Task Learning
Authors: Zhipeng Zhou
Ziqiao Meng
Pengcheng Wu
Peilin Zhao
Chunyan Miao
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Task Learning
Symmetry Teleportation
Primary area: general_machine_learning
TL;DR: We propose a new plug-and-play optimization method from the perspective of symmetry teleportation for mainstream multi-task learning approaches.

Abstract: Multi-task learning (MTL) is a widely explored paradigm that enables the simultaneous learning of multiple tasks using a single model. Despite numerous solutions, the key issues of optimization conflict and task imbalance remain under-addressed, limiting performance. Unlike existing optimization-based approaches that typically reweight task losses or gradients to mitigate conflicts or promote progress, we propose a novel approach based on Continual Optimization with Symmetry Teleportation (COST). During MTL optimization, when an optimization conflict arises, we seek an alternative loss-equivalent point on the loss landscape to reduce conflict. Specifically, we utilize a low-rank adapter (LoRA) to facilitate this practical teleportation by designing convergent, loss-invariant objectives. Additionally, we introduce a historical trajectory reuse strategy to continually leverage the benefits of advanced optimizers. Extensive experiments on multiple mainstream datasets demonstrate the effectiveness of our approach. COST is a plug-and-play solution that enhances a wide range of existing MTL methods. When integrated with state-of-the-art methods, COST achieves superior performance.

Continual Optimization with Symmetry Teleportation for Multi-Task Learning
Zhipeng Zhou', Ziqiao Meng", Pengcheng Wu', Peilin Zhao
Chunyan Miao
Nanyang Technological University
National University of Singapore
School of Artificial Intelligence, Shanghai Jiao Tong University
zzpustcml @gmail com, zq-mengênus. edu.sg. eg, peil weilinzhaodsjju.. edu. {pengchenguu, ascymiao) edu.
Abstract
Multi-task learning (MTL) is widely explored paradigm that enables the simulta- neous learning of multiple tasks using single model. Despite numerous solutions, the key issues of optimization conflict and task imbalance remain under-addressed, limiting performance. Unlike existing optimization-basee approaches that typi- cally reweight task losses or gradients to mitigate conflicts or promote progress, we propose novel approach based Continual Optimization with Symmetry Teleportation (COST). During MTL optimization, when an optimization conflict arises, seek alternative loss-equivalent point on the loss landscape to duce conflict. Specifically, we utilize a low-rank adapter (LoRA) to facilitate this practical teleportation by designing convergent, loss invariatt objectives. Addition- ally, we introduce historical trajectory reuse strategy to continually leverage the benefits of advanced optimizers. Extensive experiments multiple mainstream datasets demonstrate the effectiveness of our approach. COST is plug-and-pla so- lution that enhances wide range of existing MTL methods. When integrated with state-of-the-art methods, COST achieves superior performance. Code avaliable at https://gitthe cc0p
Introduction
Traditional machine learning typically requires sep- arate models for each task, leading to higher com- putational and storage demands as the number of tasks increases Zhou et al., 2023 Pan et al., 2024 Feng et al. 2024 To overcome this issue, multi- task learning (MTL) offers an efficient approach, enabling the simultancous learning of multiple tasks using single model (Qiu al. 2017 Hai et al.)
(a)
(b)
Figure 1: The illustration of symmetry telepor-
tation. (a) is the original gradient descent. (b) is
the gradient descent with a faster convergence rate after teleporting the point from (a).
2016 Yao et al. 2021]. Recent developments
be broadly divided into two categories: structure-based (Heuer al.) 2021 Ye and Xu 2023 Chen et al. 2023 and optimization-ben based [Sener and Koltun 2018 Yu et 2020 Liu
MTL methods
methods focus on designing architectures that enhance task learning by utilizing task relationships and promoting individual progress. On the other hand, optimization-based
al. 2024a] Structure-baase
methods prioritize the learning process by addressing challenges such as gradient conflicts and task imbalances. Since this paper concentrates on optimization-basee methods, our analysis and comparisons will primarily focus these approaches.
"Coreesooning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Optimization-basee MTL aims to resolve the aforementioned issues by re-weighting on various
aspects. For example, a series of studies [Sener and Koltun 2018 Yu al., 2020 Liu et al. 2021a
explore different gradient combinations to prevent improving some tasks while sacrificing others. Another group of works [Chen et al. 2018 Liu et af., 2024a] re-weight task loss to ensure fair progress for individual tasks and thereby address the imbalance issue. While the former group of works endeavors to balance conflict and imbalance issues, the latter focuses on attaining balanced individual progress with minimal concern for the conflict issue. As result, the latter is generally less robust in different scenarios compared to the former according to empirical observations [Chen] et al. 2018). However, the former also struggles to achieve proper balance. In this paper. distinet from these two paradigms and based on the definition of Pareto dominance, we approach MTL from new perspective. i.c., secking the less conflicting and more convergent point through symmetry teleportation during MTL optimization.
Unlike the traditional gradient descent regime, symmetry teleportation aims to accelerate the opti- mization process by seeking another point within the same loss level set, as depicted in Figure] [ Several recent works [Armenta et al., 2023 Zhao et al., 2022 2023 have explored optimization through symmetry teleportation. For example, [Zhao et al. 2022 introduces a simple teleportation algorithm for non-linear neural networks, based on the assumption that activation functions are bijective, and seeks the point of maximal gradient magnitude using gradient ascent. However, these methods do not provide practical algorithms for larger, modern neural networks, primarily due to their reliance on strict assumptions about non- -on-iiteyr and computational intensity (Section (33) These limitations especially pronounced in more complex tasks, e.g., MTL.
Therefore, in this paper, we aim to develop a practical symmetry teleportation method that is applicable for modern deep models, and addressing MTL issues. Specifically, we leverage the low-rank adapter (LoRA) to realize leleportation when encountered with the conflicts issue. By designing the objectives to ensure the invariant task loss and promote progress, individual task learning for MTL models in a balanced manner. Besides, we also design a historical trajectory reuse strategy continually benefit from advanced optimizer (e.g.. Adam). In nutshell, our contribution can be summarized as follows:
are able to further extend the boundaries of
We approach MTL. from a new angle, i.e., symmetry teleportation, and empirically verify its applicability for MTL (Section 3.2).
A new practical teleportation method COST proposed for mitigating the conflict and imbalance issue. To the best of our knowledge, we are the first to develop a practical teleportation method for non-small deep models, specifically for MTL.
By proposing a historical trajectory reuse strategy, we can continually benefit from the advanced optimizer (e.g., Adam and its variants).
Taking the advanced method as the baseline, our COST can well augment it to achieve state-of-the-ar (SOTA) performance across diverse evaluations. Besides, we also equip mainstream MTL methods with COST, and showing its plug-and-play property.
2 Related Work
2.1 Optimization- MTL
Optimizaation-based methods aim to optimize multiple tasks simultaneously by enhancing the gradient-
based learning process itself. For example, MGDA [Sener and Koltun 2018) reduces conflicts
between task gradients by combining them using the Trank-Wolfe algorithm [Jagii 2013 to generate
a gradient with minimal norm. PCGrad (Yu et al. [020] addresses gradient conflicts by projecting gradients from different tasks onto directions that minimize interference. CAGrad (Liu et al. 2021a
attempts to balance global optimization and task-specific performance, maintaining both Pareto efficiency and overall optimization with the assistance of hyperparameter. Nash-MTL [Navon et al. 2022] introduces a game-the -tenrette approach where tasks negotiate to update parameters in a manner that enables balanced progression across tasks. Additionally, MoCo [Fernando et al. 2023 focuses on correcting biases in gradient direction by tracking parameters during the learning process, improving gradient alignment and task performance. FairGrad [Ban and Ji 2024) is a pioneering MTL algorithm that puts forward fairness measurements to facilitate maximal loss reduction. It can

---

be considered as an advanced version of Nash-MTL, being capable of balancing task progress in more fine-ge -ga-ger manner.
2.2 Symmetry Teleportation for Deep Model
Before presenting some recent works on symmetry teleportation, we first provide its definition here per [Zhao al. 2022). Let C(0) be the loss function. Here, Rd denotes the model's parameter space, and represents the acting space on the parameters that leaves the loss value unchanged. Subsequently, we have the following definition:
( C(a 0), E A, VO E VRR.
0' a 0, a argmaax VL((a 0)||" aEA
(2)
observe that symmetry teleportation aims to find loss-invariant point (Eqn. 1 with maximum gradient norm (Eqn. 2 on the loss level set by acting with group element.
As recent research topic, symmetry teleportation has been explored in only few works [Armenta cet al. 2023 Zhao et al. 2022 2023]. [Armenta et al. 2023 first introduced the concept of 'neural teleportation' and investigated its impact on optimization. Zhao al.) 2022] proposed gradient ascent-basse teleportation algorithm for small neural networks (e.g., three-layer MLPs). And (Zhao et al., 2023) established the connection between symmetry teleportation and generalization through series of theoretical analyses and provided an alternative for enhancing the meta optimizer.
2.3 Low-Rank Adapter
LoRA is and parameter-efficien fine-tuning (PEFT). It operates by maintaining the pre-trained weights of large model in fixed state and incorporating small, trainable rank decomposition matrices. During fine-tuning, rather than modifying all the parameters of the model, only these low-rank matrices are subject to update.
gaining increasing popularity in tandem with the rapid advancement of foundation models
Moreover, LoRA has several variants that attain dynamic rank (Zhang et al.], quantiza- tion [Dettmers et al. 2024]. For instance, AdaLoRA (Zhang et al.) adaptively assigns dynamic rank different parameters, thereby enabling the capture important updates while preserving efficiency, In contrast, QLoRA [Dettmers et al., (024) introduces 4-bit NormalFloat, double quantization, and paged optimizers more effectively optimize LoRA, while significantly reducing the required memory.
Connection and Difference: Our work tackles conflict and imbalance issues in optimization- based MTL through symmetry teleportation. Specifically, we utilize LoRA to imple- ment practical teleportation. In contrast to previous studies, explore MTL from novel perspective and introduce a new teleportation algorithm for modern deep models. This algorithm scalable, casily integratable, and compatible with both PEFT and MTL.
Motivation and Observation
8
Weak Conflict
g
B
3.1 Preliminary
Ro
As mentioned, optimization- MTI. ap- proaches operate under the assumption that the model consists of task-shared backbone net- work alongside task-specific branches. Con- sequently, the primary objective of these ap- proaches is to devise gradient combination strategies that optimize the backbone network to yield benefits across all tasks. Let us consider scenario where there are K > 2 tasks available, each associated with a differentiable loss func- tion (0), where 0 represents the task-shared
Imby N-Con
Imb, Con: Dominated Conflict
R)
Bo
Figure 2: Illustration conflict and imbalance is- sues in MTI. 'Bal' and 'Imb' represent balanced and imbalanced, while 'N-Con' and "Con' repre- sent non-conflicting and conflicting.

---

parameters. The goal of optimization based MTL is to search for the optimal 0 R that minimizes the losses for all tasks.
Definition (Gradient Similarity). Denote dy as the angle between two task gradients gi and gj- and assume lglll < llgy|le. then we define the gradient similarity as cos Pi and the gradients as conflicting when cos Otj < (referred as Weak Conflict). When the mean gradient go is conflicting with 9t. we call it as Dominated Conflict (see Figure 2).
3.2 Applicability of Symmetry Teleportation
Before delving into the principal design of our method, it is necessary to verify the existence of parameter symmetries with differing con- flict statuses. To this end, we examine the optimization process of mainstream MTL. ap- proaches. We analyze the mean loss across all tasks and the associated conflict status during optimization from various initial points, with the results presented in F
0.5
Costre Simbartyy
0.5
0.0
0.0
-1.0
122
1.3
13 Valee)
ExpELoss Valuel
(a) CAGrad
(b) FairGrad
Figure 3: Dominated conflict vs. loss examina- tion. The pink backdrop designates the conflicting area, whereas the green backdrop indicates the non- conflicting area. The blue scatter points the indi- vidual recorded points throughout the optimization process. The red dashed line symbolizes the tele- portation occurring from conflict point to a non- conflict point. An exponential amplification has been applied to the loss values to enhance visual clarity.
As shown in Figure 5 it is often possible to identify con-conflict alternative at the same loss level when encountering conflict, demonstrating the potential of symmetry tele- portation. Additional statistical results from other MTL approaches are provided in the Ap- pendix (Sec. A.3).
3.3 Pitfall of Current Paradigms
While several works [Armenta et al., 2023, Zhao et al. 2022, 2023 have proposed symmetry teleportation algorithms for neural network- models, we demonstrate their limitations with current deep models. First, these algorithms require activation functions to be bijective, which poses a significant challenge for widely-used deep models (e.g.. ResNet-50) that use non-bijective activation functions, e.g., ReLU and Sigmoid. Second, they require calculating the pseudo-inverse of inputs layer by layer to ensure output and loss invariance. This process is computationally intensive and may be impractical for modern deep models. As a result, these approaches have only been tested on simple lhree-layer MLP networks and small-scale datasets (e.g. MNIST) for verification.
4 Principal Design
In this section, we present the detailed design of COST, incorporating the symmetry teleportation paradigm and historical trajectory reuse strategy. We also provide an analysis of convergence.
4.1 Continual Optimization with Symmetry Teleportation
The overall framework of COST is depicted in Figure At certain training stage I, we utilize LoRA to teleport the weight of the shared backbone to the non-conflict point (merge the trained LoRA into the backbone's weight) with the same loss level. Subsequently, the model (including both the backbone and branches) is continuously optimized by other MTL algorithms. In this framework. there are two questions that need to be answered:
When: The first question is, when should teleportation be triggered? Unfortunately, the previous solutions presented in [Armenta et al. 2023 Zhao et al.) 2022 2023 did not offer clear answer to this question. They merely triggered
random or intuitive manner. In contrast, our goal is to address two key challenges in MTL: conflict and imbalance, challenges that are not concurrently addressed by existing solutions [Zhou et al. 2025 2024). Moreover, a naïve linear scalarization (LS) strategy can effectively promote all tasks, illustrated in Figure and has been empirically verified in [Xin et al. 2022). Thus, the primary challenge lies in resolving conflict arising from imbalance, i..., dominated conflict. Therefore, establish the teleportation trigger condition based on the

---

Backbone
branch
branch
branch
branch
Merge
Loss Invariance & Balanced Gradient Maximization Lr yLg
Frozen
Teleportation with LoRA (A0)
Trainable
Gradient
A
Model Weight
Loss Landscape
Figure 4: The Illustration of COST. Here, we depict a one-time teleportation procedure by using a 2-task example for the sake of illustration. Note that LoRA is only applied to the shared backbone.
occurrence of dominated conflict
040 0, Φ10 L(g:.
where 9i and go represent the task gradient with the smallest norm and the mean gradient, respectively. However, when handling large number of tasks, dominated conflicts become inevitable, reaching 97% conflict ratio per epoch on CelebA [Liu et al. 2015), as shown in Figure 5ja). Then we still employ dominated conflict the trigger condition, frequent teleportation would occurs and results in inefficiency. Therefore, our objective shifts to mitigating dominant conflicts, balancing efficiency and effectiveness. To achieve this, adopt the following condition:
K
Σ
11 [cos Ø30
<
0J
>
(4)
2
Under this condition, the trigger frequency is sig- nificantly reduced (see Figure ((a)) while main- taining effectiveness, as demonstrated evaluation. Additionally, off between effectiveness and efficiency for this condition in the Appendix (Section A.4).
Doninnated Confli
J82 Condisce
the analyze the trade-
How: In the symmetry teleportation paradigm, there are two key objectives: loss invariance and gradient maximization, as outlined in Eqn. [ and Eqn. 2 Since finding group action y is infeasible for deep models, we instead use LoRA (A0) alternative, reformulating it
Cdla
400 (b) Loss examination
(a) Conflict ratio examina- tion
Figure 5: (a) Conflict ratio per epoch CelebA (40-task) and NYUv2 (3-task) and (b) loss exami- nations during single teleportation.
((0) L( + A0)
(5)
AO argmax
V V + 20)
(6)
Ae
With respect to the specific symmetry teleportation taking place during the optimization process, in order to ensure the task loss remains invariant, we undertake the minimization of the loss fluctuation in the following way:
1 K K
L.
Σ C.
C.1
where C, represents the individual task loss, and 2; is its loss before starting teleportation, which is unchanged during teleportation.
We have provided comparison between dominated conflict and weak conflict in the Appendix (Sec. A.4).

---

To maximize the gradient of the target point, simplistic solution would be to incorporate it into the objective LoRA optimization. However, two challenges arise when attempting to do so: (1) It is difficult to explicitly incorporate gradient maximization into the objective design. (2) Even it were possible, the computation of the Hessian matrix would be overly burdensome for LoRA optimization. On the other hand, upon closely examining Eqn. 6 we can observe that our intention is merely to identify the point with the maximal gradient, rather than precisely attaining the maximal gradient itself. Consequently, we choose select another metric for measuring the gradient norm, ie., Sharpness [Foret et al. 2021]. Since the negative direction of the gradient is locally the fastest direction of descent, thus estimate the gradient by seeking the sharpest direction at 0" as follow:
Sharpness
max |0 + - C(0))] |el
where 5 the radius the sphere. We further implement Eqn. | by randomly sampling e times from the sphere, and estimating sharpness by selecting the maximum one:
K
A
Lg max
Σ
R [0, + AO + cs) L. + A0)
R
(9)
K
j=1
k E 1
|695
K
R = K softmax
(10)
where R. is computed to facilitate the search for more balanced alternatives, mitigating imbalance issues. Consequently, the overall objective for LoRA optimization can be formulated as follows:
Liora L. - Y
(11)
where y is the hyper-parameter (set as 0)1), As depicted Figure ()), L. largely decreases while fg increases as expected during teleportation.
G.
To enhance understanding of our approach, provide trajectory illustration on toy exam- ples [Liu al.] 2021a] in Figure As shown, LS may fail to reach the Pareto front from cer- tain initializations due to conflict issues. How- ever, with the augmentation of COST, it success- fully explores alternative paths for continuous optimization rather than getting stuck.
0-
-2
-6
-00
2
41
-10- -
(a) LS.
(b) + COST.
Figure 6: Trajectory illustration on toy examples.
4.2 Convergence Analysis
In this section, we present a convergence analysis to further enhance the understanding of the applicability of our proposed method. By formulating theorem, it has been proven that our method converges to the Pareto stationary point with guarantee.
Theorem Assume task loss functions Li...., LK are differentiable and A-smooth (A>0) such that ((,) - (.22)) A | O2 for any points 01. 02 and symmetry teleportation property holds. Set the step size as r WTT-I' is the training iteration. Then, there exists a subsequence {0*>} of the output sequence (0*) that converges a Pareto stationary point 0*.
The proof of this theorem is provided in the Appendix (Sec. F).
4.3 Historical Trajectory Reuse Strategy
When training MTL models with advanced optimizers, a minor issue arises after reaching the loss-invariant point through our symmetry teleportation. Specifically, the teleportation process disrupts the continuous optimization flow, preventing the MTL model from leveraging its historical trajectory- one of the key advantages of advanced optimizers (e.g., Adam [Kingma 2014).

---

20
Taking the Adam optimizer an example. which commonly utilized in mainstream MTL ap- proaches [Liu et al. 2021a 2024a Navon et al. 2022]. It employs an exponentially weighted mov- ing average to estimate the momentum (12) and quadratic moments (8t) of the gradient (histori- cal trajectory). However, when the model is tele- ported to another point, the stored historical trajec- tory would supply misleading information for the current model optimization. To tackle this issue, we partially preserve the historical trajectory by computing the correlation between teleportation and previous updating (as depicted in Figure 77:
Figure The illustration of HTR strategy. The red star represents the teleported point. g is the gradient at the pre-teleportation point, and is the angle between g' and AO.
σ cos cos_sim(44 A0,g')
(12)
where cos_sim is the function of computing cosine similarity, and g' is the gradient at the pre- teleportation point. In this way, the historical trajectory
be modulated and reused according to Eqn. 13 After that, a is still set to 1, as is typically the case with the Adam optimizer.
v 0 0 + (
(13)
8t - 0,28,-11 + (1 o32)
The overall training algorithm is concluded in the Appendix (Section C).
5 Performance Evaluation
Table 1: Scene understanding (CityScapes, tasks).
In this section, we initially evaluate our method using mainstream MTL bench- marks and compare it with the fol- lowing baselines: Linear Scalarization (LS), Invariatt (SI), Random Loss Weighting (RLW) as described in [Lin jet al. 2021]. Dynamic Weight Aver- 2019], Uncertainty Weighting (UW) detailed
Segmentation f
Depth J
Method
MR Am
mioU Pix. Ace. Abs. Err. Rel. Em.
Independent
74.01 93.16
0.0125 27.77
LS
75.18 93.49
0.0155 46.77
8.25
22.60
RLW
74.57 93,41
0.0158
47.79
11.00
24.37
age (DWA) from [Liu et al.
DWA
75.24 93.52
0.0160
44.37
8,25
21.43
Uncertainty
72.02 92.85
0,0140 33.13
7.50
5.88
MGDA
68.84 91.54
0.0309
33.50
11.00
44.14
in Kendall et al.) 2018). MGDA from
GradDrop
75.27 93.53
0.0157
47.54
7.75
23.67
[Sener and Koltun 2018). GradDrop
PCGrad
75.13 93.48
0.0154
42.07
8.50
18.21
presented in [Chen al. 22020) PC-
CAGrad
75.16 93.48
0.0141
37.60
7.50
11.58
Grad as in
(Yu et al.) 2020). CAGrad
IMTL
75.33 93.49
0,0135
38.41
5.75
11.04
from
[Liu al. [0221a]
IMTL detailed
Nash-MTL
75.41 93.66
0.0129
35.02
3.00
6.82
in
Liu et al.
2021b]. Nash-MTL from
FAMO
74.54 93.29
0.0145
32.59
8.25
8.13
Navon et 2022]. FAMO described
FairGrad
75.72 93.68
0.0134
32.25
2.25
5.18
in
[Liu et al.] [2024a]. and FairGrad
COST 75.73 93.53
0.0133
31.53
2.00
4.30
(a= 1) from [Ban and Ji 2024). Sub-
sequently, we offer some additional analyses regarding conflict and gradient examinations, ablation verification, etc., further enhance understanding. We also provide additional analysis on conflict and gradient examination (Sec. A.1), plug-and-play verification (Sec. A.2), alternatives to PEFT (Sec. A.5), and time cost (Sec. A.6) in the Appendix. All experiments are carried out single Tesla V100 GPU. For more experimental details, please refer to the Appendix (Sec. B). Code will be released once this paper is accepted.
studies, and plug-and-pil
Evaluation Metric. In addition to reporting individual performance, we also incorporate a widely used metric, Am % [Maninis al., 2019], which evaluates the overall degradation compared to independently trained models that are considered as the reference oracles. The formal definition of Am% is given as:
D(- (1)8 Mm.k Mo.k
Am %
x 100
(14)
Mr.
where Mn. & and MA.E represent the metric M₁ for the compared method and the independent model, respectively. The value of 8y is assigned as 1 if higher value is better for Mg, and 0 otherwise. Besides, we also report another popular metrie named Mean Rank (MR), which computes the average ranks of each methods across all tasks.
7

---

5.1 Overall Evaluation
Dense Prediction. CityScapes [Cordts et al. 2016 and NYUv2 Silberman et al. 2012] are Iwo widely-used scene understanding datasets, which are employed for the evaluation of MTL. NYUv2 comprises 1449 annotated images and is utilized for three fine-grained tasks, i.e., semantic segmentation, depth estimation, and surface normal prediction. CityScapes consists of 5000 annotated scene images, which are readied for two tasks: semantic segmentation and depth estimation.
Table 2: Scene understanding (NYUv2, 3 tasks). We report MTAN model performance averaged over random seeds. The best scores are provided gray and the second scores are underlined.
Segmentation
Depth &
Surface Normal
Method
MR Am %
Angle Distance +
Within ( t
mloU Pix. Acc. Abs. Err.
Rel. Err.
Mean
Median
11.25 22.5 30
Independent
38.30
63.76
0.68
0.28
25.01
19.21
30.14 57.20 69.15
LS
39.29 65.33
0.55
0.23
28.15
23.96
22.09
47.50 61.08
9.44
5.46
37.17 63.77
0.58
0.24
28.27
24.18
22.26
47.05 60.62
12.22
7.67
DWA
39.11 65.31
0.55
0,23
27.61
23.18
24.17
50.18 62.39
8.56
3.49
Uncertainty
36.87 63.17
0.54
0.23
27.04
22.61
23.54
49.05 63.65
8.78
4.01
MGDA
30.47 59.90
0.61
0.26
24.88
19.45
29.18
56.88
69.36
7.11
1.47
GradDrop
39.39 65.12
0.55
0.23
27.48
22.96
23.38
49.44 62.87
8.89
3.61
PCGrad
38.06 64.64
0.56
0.23
27.41
22.80
23.86
49.83
63.14
9.33
3.83
CAGrad
39.79 65.49
0.55
0.23
26.31
21.58
25.61
52.36 65.58
6.33
0.29
IMTL
39.35 65.60
0.54
0.23
26.02
21.19
26.20
53.13
66.24
5.56
-0.59
Nash-MTL
40.13 65.93
0.53
0.22
25.26
20.08
28.40
55.47
68.15
3.11
-4.04
FAMO
40.30 66.07
0.56
0.21
26.67
21.83
25.61
51.78
64.85
5.44
0.16
FairGrad
39.74 66.01
0,54
0.22
24.84
19.60
29.26
56.58
69.16
3.00
.4.66
COST
38.06 64.71
0.54
0.23
24.47
18.80
30.84
58.25
70.30
3.22
5..39
In line with the implementation and training strategy of FairGrad [Ban and Ji] 2024]. we construct our model using SegNet (Badrinarayanan al. 2017] and employ MTAN Liu al. 2019] as the backbone within it. We train our model with the Adam optimizer for total of 200 epochs, setting the initial learning rate 1.0e-4 and reducing half after 100 epochs. The batch size set to 2 for NYUv2 and 8 for CityScapes, respectively.
The results obtained on these datasets are presented in Table [ and Table respectively. With FairGrad serving the baseline, our method not only successfully surpasses it but also attains the SOTA performance in terms of MR and Am%. Specifically, upon closely exam- ining the performance of each individual task, we can note that COST significantly enhances FairGrad on the CityScapes dataset and consid- erably improves the surface normal prediction task, while also showing some promise on the other tasks the NYUv2 dataset. These obser- vations clearly demonstrate the effectiveness of design, which aids in alleviating conflict and facilitating convergence.
Table 3: Results on CelebA and QM9 datasets with MR and Am%. The results of FairGrad-R are reported according the official implementation.
CelebA
QM9
Method
MR J
Am %
MR
Am %
LS
7.08
4.15
9.09
177.6
SI
8.80
7.20
5.64
77.8
RLW
5.98
1.46
10.64
203.8
DWA
7.78
2.40
8.91
175.3
UW
6.65
3.23
7.00
108.0
MGDA
11.98
14.85
8.91
120.5
PCGrad
7.58
3.17
7.36
125.7
CAGrad
7.13
2.48
8.09
112.8
IMTL-G
5.53
0.84
6.91
77.2
Nash-MTL
5.73
2.84
4.27
62.0
Image Classification. CelebA Liu et al. 2015] commonly utilized face attributes dataset [Wang et al. 2024) that contains over 200,000 images and is annotated with 40 at- tributes. Recently, it has been adopted as a 40-task MTL benchmark to assess the model's capacity to handle large number of tasks. In accordance with the setup of FairGrad, we utilize 9-layer convolutional neural network (CNN) as the backbone and linear layers the task-specific heads
FAMO
5.65
1.21
5.18
58.5
FairGrad-R R
6.35
1.15
4,82
59,9
COST
4.80
0.93
4.18
58.3

---

top of it. We train our model with the Adam optimizer for a total of 15 epochs, setting the initial learning rate 3.0e-4. Moreover, the batch size is set 256.
The evaluation results are shown in T Given that our method is mainly developed based on FairGrad, performance is thus highly associated with it. We conscientiously re- re-implemented FairGrad using the official code they provided and were able to achieve the reported performance on CityScapes, NYUv2. However, we were unable do so on CelebA and QM9. Consequently, we only report our re-implemented performance of FairGrad here (referred to as FairGrad-R), As can be observed, COST still significantly enhances its baseline and attains the SOTA performance according to MR, ranking second according to Am%. These results demonstrate COST's s remarkable ability to handle numerous tasks simultaneously.
Regression, QM9 [Ramakrishnan et al. 2014 is another widely used drug discovery MTL dataset specifically for regression tasks. It contains 130,000 organic molecules that are organized graphs with node and edge features. This task is designed to predict 11 properties having different mea- an evaluation scenario for MTL involving a large
surement scales and can also be considered number of tasks. Our approach is trained for 300 epochs with batch size of 120. The initial learning rate is set to 1. 1.0e-3, and learning rate scheduler is applied reduce the rate when the validation performance shows no further improvement.
method still achieves competitive performance this specific dataset. However, in comparison to other datasets, it exhibits fewer enhancements over its baseline. One crucial reason for these relatively less satisfactory results is that this task adopts graph model with only two layers supporting LoRA in current PEFT package, which reduces its effectiveness.
According to Table
5.2 Ablation Study
We consider COST as an integrated system, and thus each component ought to be evaluated to showcase its effec- tiveness. In our design, there are primarily three key com- ponents: the loss invariance objective (C₂), the gradient maximization objective (Cg), and the HTR strategy. Con- sequently, we carry out ablation studies for verification purposes and present the results in Table In the context of symmetry teleportation, L, and Lg serve as the founda- tion for seeking alternatives within the orbit. Hence, we exclude C, and C, during the LoRA optimization process, respectively. The results indicate that without L. or Lg. COST performs worse than its baseline (which FairGrad in this case). More specifically, COST would experience severe deterioration without L,, thereby underlining the crucial importance of loss invariance. These results might address another concern regarding COST, namely: Are the im- provements brought about by COST rooted in the capability expansion facilitated by LoRA? Without an appropriate objective design, LoRA is unable to effectively augment the base models.
Table 4: Ablation study of COST on CityScapes (2 tasks).
Le
bg
HTR
Am % 4
5.18
7.90
381.86
4.65
4.30
On the other hand, it should be noted that LoRA is incorporated into the base model after each teleportation. Thus, the model's capability remains unchanged during the inference time. All that we are doing is assisting in finding better convergence point. Furthermore, when the HTR strategy is excluded, Am% decreases from 4.30 to 4.65, which demonstrates the significance of benefiting from advanced optimizers.
6
Conclusion
This paper explores the MTL problem from a brand new perspective, i.e., alleviating the conflict issue through symmetry teleportation. Specifically, we utilize LoRA to achieve practical symmetry teleportation for contemporary deep models. Additionally, we design boss-invariant and gradient maximization objectives to assist in identifying non-conflict and more convergent points. We also devise historical trajectory reuse strategy to continuously benefit from advanced optimizers. Extensive experiments have demonstrated the effectiveness of our proposed method as well as its plug-and-ple characteristic. As scalable framework, anticipate that our method can offer some

---

valuable insights to researchers engaged in optimization based MTL. Currently, there are still rooms for improvement within this system, and our future work will focus these aspects.
Acknowledgement
This research is supported, in part, by the WeBank-NTU Joint Research Institute Fintech, Jinan- NTU Green Technology Research Institute (GreenTRI), and Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY), Nanyang Technological University, Singapore. It's also supported, in part, by the NTU-PKU Joint Research Institute, a collaboration between Nanyang Technological University and Peking University that is sponsored by a donation from the Ng Teng Fong Charitable Foundation. The first author also would like to thank the insightful discussion with Dr. Shuaicheng Niu, Shibo Feng, and Haochen Li.
References
Marco Armenta, Thierry Judge, Nathan Painchaud, Youssef Skandarani, Carl Lemaire, Gabriel Gibeau Sanchez, Philippe Spino, and Pierre-Mare Jodoin. Neural teleportation. Mathematics, 11 (2):480, 2023.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions pattern analysis and machine intelligence, (9(1))244112293 2017.
Hao Ban and Kaiyi Ji. Fair resource allocation in multi-task learning. arXiv preprint arXiv:2402. 15638, 2024.
Zhao Chen, Vijay Badrinarayanan, Chen-Ye Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 793-803. PMLR, 2018.
Zhao Chen, Jiquan Ngiam, Yanping Huang. Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout, Advances in Neural Information Processing Systems, 33:2039-2050. 2020.
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G learned-Milles and Chuang Gan. Mod-squad: Designing mixtures of experts as modular multi-task learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11828-11837, 2023.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances Neural Information Processing Systems, 36, 2024.
Jian Feng, Yajie He, Yuhan Pan, Zhipeng Zhou, Si Chen, and Wei Gong. Enhancing fitness evaluation in genetic algorithm-based architecture search for ai-aided financial regulation. IEEE Transactions on Evolutionary Computation, (8(3))623-631) 2024.
Heshan Devaka Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. Mitigating gradient bias in multi-objective learning: provably convergent approach. In The Eleventh International Conference on Learning Representations, 2023.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness minimization for efficiently improving generalization. 2021.
Zhen Hai, Peilin Zhao, Peng Cheng, Peng Yang, Xiao-Li Li, and Guangxia Li. Deceptive review spam detection via exploiting task relatedness and unlabeled data. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1817-1826, 2016.
10

---

Falk Heuer, Sven Mantowsky, Saqib Bukhari, and Georg Schneider. Multitask-centemme (men): Efficient and diverse multitask learning using an anchor free approach. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 997-1005, 2021.
Nam Flyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efiiien federated learning. arXiv preprint arXiv:2108.06090 2021.
Martin Jaggi. Revisiting trank-wolfe:: Projection-fiee free sparse convex optimization. In International conference on machine learning, pages 427-435. PMLR, 2013.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 7482-7491, 2018.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.988, 2014.
Baijiong Lin, Feiyang Ye, Yu Zhang, and Ivor W Tsang. Reasonable effectiveness of random weighting: A litmus test for multi-task learning. arXiv preprint arXiv:2111.10003, 2021.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878-18890, 2021a.
Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization. Advances in Neural Information Processing Systems, 36, 2024a.
Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations, 2021b.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1871-1880, 2019.
Xinyue Liu, Hualin Zhang, Bin Gu, and Hong Chen. General stability analysis for zeroth-order optimization algorithms. In The Twelfth International Conference on Learning Representations, 2024b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738. 2015.
Kevis-Kokits, Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multiple tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1851-1860, 2019.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. In International Conference on Machine Learning, pages 16428-16446. PMLR, 2022.
Yuhan Pan, Zhipeng Zhou, Wei Gong, and Yuguang Fang. Sat: A selective adversarial training approach for wifi-based human activity recognition. IEEE Transactions on Mobile Computing, 23 (12):12706-12716, 2024.
George Pu, Anirudh Jain, Jihan Yin, and Russell Kaplan. Empirical analysis of the strengths and weaknesses of peft techniques for Ilms. arXiv preprint arXiv:2304.1499 2023.
Minghui Qiu, Peilin Zhao, Ke Zhang, Jun Huang, Xing Shi, Xiaoguang Wang, and Wei Chu. A short-term rainfall prediction model using multi-task convolutional neural networks. In 2017 IEEE international conference on data mining (ICDM), pages 394-404. IEFE, 2017.
Zeju Qiu, Weiyang Liu, Haiwen Feng Yuxuan Xue, Yao Feng. Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 792299229-79.., 2023.

---

Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1-7, 2014.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746-760. Springer, 2012.
Sebastian U Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint arXiv:1907.04232, 2019.
Tianyi Wang, Mengxiao Huang, Harry Cheng, Xiao Zhang, and Zhiqi Shen. Lampmark: Proactive deepfake detection via training-free landmark perceptual watermarks. In Proceedings of the 32nd ACM International Conference on Multimedia, page 10515-10524, 2024.
Derrick Xin, Behrooz Ghorbani, Justin Gilmer, Ankush Garg, and Orhan Firat. Do current multi-task optimization methods in deep learning even help? Advances in neural information processing systems, 35: 15:13597-,,60, 2022.
Huaxiu Yao, Yu Wang, Ying Wei, Peilin Zhao, Mehrdad Mahdavi, Defu Lian, and Chelsea Finn. Meta-learning with an adaptive task scheduler. Advances Neural Information Processing Systems, 34:7497-7509, 2021.
Hanrong Ye and Dan Xu. Taskprompter: Spatial-channel multi-task prompting for dense scene understanding. In The Eleventh International Conference on Learning Representations, 2023.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
Q Zhang, M Chen, A Bukharin, He, Y Cheng, Chen, and T Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. preprint (2023). arXiv preprint 1055:
Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Symmetry teleportation for accelerated optimization. Advances in neural information processing systems, 35: 35:16679-16690, 2022.
Bo Zhao, Robert M Gower, Robin Walters, and Rose Yu. Improving convergence and generalization using parameter symmetries. arXiv preprint arXiv:2305. /3404, 2023.
Zhipeng Zhou, Lanqing Li, Peilin Zhao, Pheng-Ann Heng, and Wei Gong. Class-conditional charpness-aware minimization for deep long-tailed recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3499-3509, 2023.
Zhipeng Zhou, Liu Liu, Peilin Zhao, and Wei Gong. Pareto deep long-tailed recognition: A conflict- averse solution. In The Twelfth International Conference on Learning Representations, 2024.
Zhipeng Zhou, Liu Liu, Peilin Zhao, and Wei Gong. Injecting imbalance sensitivity for multi- task learning. In Proceedings of the Fhirth-Fourth International Joint Conference on Artificial Intelligence, pages 7236-7244, 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide the discussion of limitations in the Appendix (Sec. D).
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide the proof in the Appendix (Sec. F).
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification: Code will be released once this paper is accepted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pp....aao......//o//////a
Answer: [Yes]
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
13

---

Justification:
This paper presents work whose goal is to advance the field of Machine There are many potential societal consequences of work, none which we feel must be specifically highlighted here.
Learning.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
16. Declaration of LLM usage
Question: Does the paper describe the usage of L.MSs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
14

---

A
Additional Results
A.1 Conflict and Gradient Examinations
Although our method achieves competitive performance, it remains unclear whether it effectively resolves the targeted issues, i... conflict mitigation and greater gradient norm discovery. To investigate this, we analyze the training process by recording the results before and after teleportation, as shown Figure The findings indicate that conflict significantly alleviated, with task gradients becoming positively correlated in most cases after teleportation. Besides, teleportation consistently yields greater gradient norms, confirming the effectiveness of COST's design.
-
Before Teleportation
0.35
Teleportation After Teleportation
After Teleportation
0.15
0.30
Cosine Similrity
0.10
Gradlent Wome
0.25
0.20
0.15
-0.05
-0.10
0.05
200
200
400
500
600
Iteration
(a) Conflict status
(b) Gradient norm status
Figure 8: Examinations before/after teleportation.
A.2 Plug-and-Play Verification
Intuitively, our method is orthogonal to existing MTL approaches and is therefore plug-and-play. enabling augmentation when integrated. Here, take three baselines (i.e., CAGrad, Nash-MTL, and FairGrad) to demonstrate the effectiveness of COST, and present the results in Table As anticipated, method successfully brings considerable augmentation its baselines, with improvements ranging from 0.88 to 3.21 according to Am%. Specifically, CAGrad and FairGrad receive improvements on almost each individual metric.
We also conduct an additional verification experiment using CAGrad on the NYUv2 dataset. The results, presented in Table 6, show that COST consistently enhances CAGrad's performance across all metrics.
Table 5: Plug-and-play verification CityScapes (2 tasks) dataset. We adopt FAMO's implemen- tation for Nash-MTL (denoted as Nash-R) and augment with COST, since Nash-MTL does not provide the official implementation CityScapes.
Segmentation f
Depth +
Method
Am % |
mIoU Pix. Acc. Abs. Er. Rel. Err.
CAGrad
75.16
93.48
0.0141
37.60
11.58
CAGrad COST
775.46 93.57
0.0134
35.68
8.37
Nash-R
75.87
93.57
0.0135
37.29
9,89
Nash-R CUST
75.70
93.56
0.0134
34.34
7.15
FairGrad
75.72
93.68
0.0134
32.25
5.18
FairGrad + COST
75.73
93.53
0.0133
31.53
4.30
15

---

Table 6: Plug-and-play verification NYUv2 (3 tasks).
Segmentation
Depth
Surface Normal
Method
Angle Distance
Within t
Am
(Higher Better)
(Lower Better)
(Lower Better)
(Higher Better)
mloU
Acc.
Abs Err
Rel Err
Mean
Median
11.25
22.5
CAGrad
39.79
65.49
0.55
0.23
26.31
21.58
25.61
52.36
65.58
0.29
CAGrad COST
40.76
66.42
0.53
0.22
26.00
21.13
26.44
53.25
66.30
-1.61
FairGrad
39.74
66.01
0.54
0.22
24.84
19.60
29.26
56.58
69.16
-4.66
FairGrad COST
38.06
64.71
0.54
0.23
24.47
18.80
30.84
58.25
0.30
-5.39
A.3
Motivation
To further elucidate motivation, additionally carry verification experiment observe the dominant conflict status within Nash-MTL. Based on the results shown Figure Nash-MTL exhibits fewer dominated conflicts in comparison CAGrad and FairGrad, yet still encounters significant number. Moreover, it also possesses symmetry points that have the same loss level but with different conflict status.
1.0
Cosine Similarity
0.5
0.0
0.5
-1.0
1.1
1.2
1.3
1.4
1.5
Exp(Loss Value)
Figure Dominated conflict vs. loss examination of Nash-MTL.
A.4 Analysis Trigger Condition
As stated in the main text, the trigger condition is of importance for our symmetry teleportation process. To illustrate this, carry out comparison between system when triggered by dominated conflict and when triggered by weak conflict, and the results are presented in Table be observed that although COST surpasses it accomplishes this by sacrificing significant portion of the performance on the Rel. Err. metric. This is an unexpected outcome in the context of MTL. Consequently, does perform satisfactorily on the overall metric (Am%). These results further emphasize that addressing both imbalance and conflict issues crucial for MTL. This because the scenario of weak conflict only focuses on the conflict issue, overlooking the importance of handling imbalance as well.
in terms of most individual metrics,
Table 7: Trigger condition comparison on CityScapes (2 tasks) dataset.
Segmentation
Depth
Method
Am % 1
(Higher Better)
(Lower Better)
mIoU Pix. Acc. Abs. Err.
Rel. Em.
COST .oninated 75.73 93.53
0.0133
31.53
4.30
COST
75.92 93.64
0.0127
35.94
6.94
We also conduct an additional experiment on the trigger condition the presence of numerous tasks, following Eqn. in the main text. As previously stated, this condition is designed to balance
16

---

effectiveness and efficiency ielecincy-relaxinggii it would improve performance but at the cost of increased inefficiency. The results, presented in Figure I0, confirm this trade-off. As shown, Am% gradually decreases the trigger condition is relaxed, with improvements driven by more frequent teleportation. However, this comes at the cost of an almost linear increase in time complexity.
100
0.90
Time Cost (minates)
80
0.80
40
00775
0.70
20
0.65
K/2
K13
Trigger Condition
Figure 10: Trigger condition under the massive tasks scenario. Time cost is measured for one epoch on CelebA.
Table 8: PEFT alternatives comparison on CityScapes (2 tasks) dataset.
Segmentation
Depth
Method
Am % 4
(Higher Better)
LLower Better)
mloU
Pix. Acc.
Abs. Err.
Rel. Err.
FairGrad LoRA
75.72
93.68
0.0134
32.25
5.18
75.73
93.53
0.0133
31.53
4.30
COST Lolla Hyeon Woo
2021
75.52
93.43
0.0130
35.07
7.10
COST OFT
68.17
91.40
0.0151
45.25
23.39
A.5 Analysis on Alternative of PEFT
Currently, numerous PEFT alternatives are available for teleportation purposes. further examine the effect of PEFT on method, we employ additional PEFT options assess their impact MTL performance. Specifically. we evaluate a LoRA variant (LoHa) and another PEFT alternative, OFT. The results, presented in Table [S] show that neither PEFT option improves upon their baselines. This suggests that while advanced PEFT methods may enable more efficient tuning, their complex designs limit generalizability across various scenarios, aligning with some recent observations (Pu et al. 2023 Identifying suitable PEFT approach remains future direction for our framework.
The obtained results demonstrate that our framework can indeed benefits from certain other PEFT alternatives, e.g., LoHa. However, also encounters setbacks when using alternatives like OFT. This implies that the selection of the PEFT method warrants further investigation.
A.6
Time Cost
Applying teleportation every instance of conflict would significantly increase the computational burden and training time. To mitigate this, we introduce two strategies: delayed start and frequency control. The delayed start strategy postpones the application of teleportation until after E epochs. Meanwhile, frequency control limits the number of teleportation operations within each epoch, reducing overhead without much compromising the optimization process.
Here, we measure the running time of single epoch on CelebA, comparing the scenarios with and without the COST augmentation, and present the results in Figure IT As can be observed, our applied strategies introduce only an additional 30% of the training time compared to its baselines. Nonetheless, acknowledge this still constitutes one of our limitations.
17

---

40
With CDST
20
MGDA
CAGrad
FairCrad
Figure 11: Time cost comparison.
B
Implementation Details
B.1 Baseline Implementation
CAGrad: CAGrad strikes balance between Pareto optimality and globe convergence by regulating the combined gradients in proximity to the average gradient:
max min Iw d s.t. ||d - goll < e |oo
(15)
dER"WEW
In our experiments, we extend the official implementation to include COST. And all our training settings follow the same manner with its original one. For more comprehensive information, please consult the official implementation.
Nash-MTL: Nash-MTL. provides the individual progress guarantee via the following objective:
min Σ Bi(w) + (P(ww)
(16)
s.I.i, pp (w) 50,
O.
(17)
where (w) log(w;) log(g, Gw), G 99. 12.92..... gkl. As demonstrated, the individual progress is ensured through the projection, subject to the constraint B; 9; Gw wi,"
We utilize the FAMO's implementation of Nash-MTL. since provide the implementation on CityScapes (2 tasks). In the Table 4 the main text, we honestly report the results of this implementation, and show improvements brought by COST.
official code does not
FairGrad: FairGrad a pioneer MTL. algorithm that proposes fairness measurements to promote maximal loss decrease, and formulate the following objective to derive the combination of individual gradients:
Gw
1/a
(18)
where is the hyper-paraee parameter, which is set to 1. We regard FairGrad as the advanced version of Nash- MTL that is able to balance task progresses finer grained. adopt its official implementation for all our implementation through the paper.
B.2 PEFT Implementation
LoHa: As variant of LoRA, it approximates large weight matrices with low-rank ones through the Hadamard product. This approach has the potential to be more parameter-efficient than LoRA itself.
OFT: OFT draws inspiration from continual learning. It operates by ce-parameterizing the pre-trained weight matrices using its orthogonal matrix, thereby preserving the information within the pre-trained model. To decrease the number of parameters, OFT incorporates a block-diagonal structure into the orthogonal matrix.
We utilize the implementations of LoRA, LoHa, and OFT provided by Hugging Face's PEFT. The rank for each of these is set to 5, while other configurations are left at their default values. We apply LoRA to the backbone network across three datasets (CityScapes, NYUv2, and CelebA). For the QM9 benchmark, since employs a graph network which is currently not supported by PEFT, we only apply LoRA to input and output linear layers instead.
18

---

Algorithm COST for MTL
Model parameters 0
Initialize Initialize 0 randomly
I == to Compute task gradients [e]]==
Dominated conflict detected Freeze 0f and train LoRA (A0*) according Eqn.7, 10, and 11 in the main text;
Merge 0f and A0": 0f 0 + A and unfreeze 0f
Apply HTR on the optimizer according to Eqn. and 13 in the main text; Other MTL optimization
Have applied HTR Reset 1;
C
Algorithm
We conclude the learning paradigm of COST in Algorithm 1 It should be noted that since COST is a scalable framework, thus the other MTL optimization in Algorithm could be mainstream MTL approaches (e.g., CAGrad, Nash-MTLL, and FairGrad, etc).
D Discussion & Limitation
We offer a new framework to address the challenges of MTL, which is highly scalable and can be further improved by integrating more advanced components. For instance, LoRA could be substituted with an alternative PEFT method, and sharpness estimation can be substituted with some efficient gradient estimation methods (Liu et al., 20246). However, there are still some limitations. The current training paradigm requires additional training costs, and its performance on regression tasks less competitive compared the others. We have discussed some of these limitations in the Appendix, while others are left for future work.
E
Pareto Concept
Formally, let us assume the weighted loss as Lu = w;),(0), where E W and W represents the probability simplex on [K]. point is said to Pareto dominate if and only if (. s (0). Consequently, the Pareto optimal situation arises when no O' can be found that satisfies Vi, (0') < (0) for the given point 0. All points that meet these conditions are referred to as Pareto sets, and their solutions stationary, requires minwew ||g = O, where Iw represents the weighted gradient w G. and G is the gradients matrix whose each row is individual gradient. We also provide the definition of gradient similarity for case of description.
known Pareto fronts. Another concept, known as Pareto
Convergence Analysis
Lemma 3. Let E( be a A-smooth function, where & is the i.i.d sampled mini-batch data. It follows that:
E [VC(0,))
2A (c())) - + 21 (L(8") - inf C(O,E)
Proof. We have the following inequality according to the A-smooth property of L( L(0'.E) - C(O,E) <
(19)
A 0' 0 v0',
(VC(0,E). 0'
Rd
2
And 0' 0 VL E), thus we have:
L(e - (1/A)4VV(2..)ee ((0,E) 2A |VC(0,E)]
(20)
19

---

Assume 0- = arg min (()), then we can re-arranging the above inequality to have: OER
[( :g) [(
(21)
_(0",5) inf ((0,5) + inf E L(),5)
$ C .) inf (( + L -)) A
0
E L("", {) - inf () - 2A [V
where the first inequality holds because info (( & ((0,5), vO. Taking expectation on above gives:
E | =011"
s 21E A
[((0", .E) inf ((0, -(000) -(0 .2)
s 2AE
L("", inf ((0, L(0,5) -(())) .8)
s (C(O) =()))) 2A (C(O) - inf ((0,E)
Theorem Assume task loss functions Li...., ........ LK are differentiable and A-smooth (A>0) such that [[V - (6,)l) I A 1 02 for any two points 01, 02. and our symmetry teleportation property holds. Set the step size as rj ATT is the training iteration. Then, there exists
subsequence { of the output sequence {0t} that converges Pareto stationary point 0"
Proof. We have the following inequality according to the A-smooth property C(0):
A
C((') L()) - (VL(O), 0) + 1|0' 012
(22)
Let 0' = gt+1, 0t = 0F + A (Gradient maximization: A argmax VL
AOt)), and
C(O*) L( (Loss invariance), we have:
((B s L(0") (VL(0"),
(23)
A lort1 - 1
(24)
2
C(( V C(O)). VL(0" <<)) Ank V {t)l "c(o""ee)]""
(25)
Taking expectation conditioned on 0t, we have:
E, [[(0+t1]] < C((') -(00)-m 1/: VC(Or') ,|²
(26)
Ank EE,
VL(O"' :5>1²]
According to Lemma
we have:
(V(0, S
2A - ((((0))((0")) + 21 (C(0)) - E inf C(0,5)
(27)
Inserting Eqn. 27 into Eqn. 26 we have:
E, [[] < L( - m1 |C( ||?
(28)
+ +, [(00') L())) + -())) E inf ((0,5) ))
20

---

By taking full expectation and re- arranging terms, we have:
| [|]]
T
<(1 + A² 12 E L(0*) L*)
(29)
E [-(ot+)) - [*]
A2
where x² = _(0*) EE [info C(0,E)] Then we consider to introduce the re-weighting trick in |Stich
2019]. Let "tt (?t > 0) be sequence such that yn(1 + A² = N Assume 7-1 then
7t + A31 (++1) By multiplying Ys on both sides of Eqn. 29 we have:
[1r]
The E
S 7 1E [C( [-]
(30)
7. ( L*] + 76,311100
Summing up the above equation from = ..., T 1, we have:
Σ wel
[IV || SE [2(00) L*]
(31)
T-1
+ A²
Σ
t=0
Dividing both sides by Et Yen we have:
|
V
min
E
(32)
T
T-1
Σ T-- Ye7t Σ Wh |
1== Tt E [[((00) L-] + A20
1300>T00 T-
- 7
Assume 7/. = E), then we have:
T-1
T-1
No7 = 17 Σ (1 + 1277)--(41) 1=0
(33)
1=0
1- (1+ A3 T
1+ A2 1- 1-
1- A1+
A21
Note that (1+ 121
S and 1tx
I log(1 + x)), thus we have
log(2)
[o < ST
(34)
log(1 1 + A2y22
1372
From this, we can obtain:
Jem 2 2A2n'' for T 7 log(2)(1 + A21/2)
(35)
A277
Inserting the above equation into the Eqn. we have:
|
min
E
(36)
log(2)(1 + A2
< fn [C(00)
L+]
n2202 for T>
(37)
A2n2
21

---

Setting 17 AVT-T' we finally have:
||
|99]
min
E
(38)
T...T=- T-1
2A VT
Ao
B [C(00)
Cx]
(39)
Thus, our method can readily reach to the Pareto Stationary point.
22


Reviews:
[{"review_id": "EtsD5ZjMoO", "reviewer": "Reviewer_Zv8Z", "summary": "The paper proposes COST (Continual Optimization with Symmetry Teleportation) to tackle gradient conflict and task imbalance in multi-task learning (MTL). When conflicts arise, the model is “teleported” to an alternative point on the same loss level set (i.e., task losses unchanged) with better gradient conditions. Teleportation is implemented using LoRA modules, optimizing a loss-invariance and gradient-maximization objective. The authors introduce a historical trajectory reuse (HTR) strategy to preserve optimizer momentum (e.g., for Adam). COST is a plug-and-play framework tested on several datasets (CityScapes, NYUv2, CelebA, QM9), showing consistent improvements over strong MTL baselines.", "questions": "1. Please quantify how often teleportation is triggered in practice and its cumulative time cost. Could more dynamic or adaptive frequency strategies be explored?\n\n2. Can the authors better explain the $L_g$ surrogate used to encourage stronger gradients after teleportation. What metric is being maximized in practice?\n\n3. How sensitive is performance to LoRA rank? Why did LoRA outperform LoHa and OFT? Would dynamic rank tuning help?\n\n4. How might COST be adapted to models where LoRA/PEFT is less applicable (e.g., GNNs or Transformers)?\n\n5. Could imperfect teleportation or over-teleportation lead to optimization instability? Any examples observed?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality:\n- The paper shows Strong theoretical motivation: targets dominated conflict with formal definitions.\n- The method enjoys practical and scalable implementation using LoRA.\n- The authors perform extensive experiments on vision and graph tasks with solid baselines.\n- The paper performs ablations and auxiliary studies (e.g., trigger frequency, PEFT alternatives) enhance rigor.\n\nLimitation: COST introduces 30% training time overhead and added complexity.\n\nSignificance\n- The paper addresses a central MTL challenge in a novel way.\n- Plug-and-play nature means broad applicability across existing MTL optimizers.\n- The authors show strong empirical results on datasets with both few and many tasks.\n- The method is practical for many MTL settings, though overhead may limit adoption in large-scale training.\n\nOriginality\n- The paper is the first to propose a practical symmetry teleportation method for deep MTL.", "was_revised": true, "final_justification": "I have read the author response and have no further questions. I'll maintain my score.", "rebuttal": "We sincerely thank Reviewer Zv8Z for the constructive and insightful comments. Below, we provide detailed point-by-point responses to each of your concerns.\n\n>* Q1: Please quantify how often teleportation is triggered in practice and its cumulative time cost. Could more dynamic or adaptive frequency strategies be explored?\n\n  A1: As shown in Figure 5, teleportation is triggered infrequently when using the dominated conflict condition on NYUv2 and the $K/2$ condition on CelebA. We further report the per-epoch computational overhead of our method on CelebA in Figure 10 (Appendix), which confirms its efficiency. While our current design uses fixed conflict-triggering criteria, recent studies [1,2] suggest that it may not be necessary to apply MTL at every optimization step. In the same spirit, not every conflicting step may require teleportation. Designing dynamic triggering strategies could therefore be beneficial, though this remains an under-explored area. We thank the reviewer for this insightful suggestion and plan to investigate it further in future work.\n\n  \n\n>* Q2: Can the authors better explain the $L_g$ surrogate used to encourage stronger gradients after teleportation. What metric is being maximized in practice?\n\n  A2: In our symmetric teleportation framework, we optimize two objectives: loss invariance $L_t$ and balanced gradient maximization $L_g$. For the design of $L_g$, we first compute a re-weighting factor $R$ based on task gradient norms (Eqn. (10)) prior to teleportation. This factor is then used to re-weight task sharpness in Eqn. (9), thereby guiding the search toward a direction that maximizes balanced gradients during teleportation.\n\n  Given that the negative gradient direction represents the steepest local descent, we use sharpness as a surrogate to approximate this descent direction. Specifically, sharpness is estimated by sampling perturbations from the unit sphere multiple times and taking the maximum perturbed loss as the sharpness measure. This approach shares a similar motivation with point-wise gradient estimators commonly used in online optimization literature [3,4].\n\n  \n\n>* Q3: How sensitive is performance to LoRA rank? Why did LoRA outperform LoHa and OFT? Would dynamic rank tuning help?\n\n  A3: To address the reviewer’s concern, we conducted a comparison of LoRA ranks and present the results below:\n\n  |             | mIoU  | Pix. Acc. | Abs. Err. | Rel. Err. | $ \\Delta m \\\\% $ |\n  | ----------- | ----- | --------- | --------- | --------- | ------------ |\n  | COST (R=5)  | 75.73 | 93.53     | 0.0133    | 31.53     | 4.30         |\n  | COST (R=10) | 73.67 | 92.87     | 0.0135    | 30.00     | 4.16         |\n  | COST (R=15) | 73.96 | 92.92     | 0.0134    | 30.13     | 4.04         |\n\n  As observed, increasing the rank leads to marginal improvements in MTL performance; however, it also results in increased training cost. Therefore, we adopt the default setting (R=5) to maintain a balance between performance and efficiency.\n\n  While a deeper investigation into LoRA is beyond the scope of this work, we note that similar findings have been reported in recent PEFT studies [5,6], where LoRA consistently ranks among the top-performing approaches. Intuitively, dynamically adjusting the rank may be beneficial, as the difficulty of finding a desirable symmetric point during teleportation may vary across steps. We thank the reviewer for this insightful suggestion and will consider exploring this direction in future work.\n\n  \n\n>* Q4: How might COST be adapted to models where LoRA/PEFT is less applicable (e.g., GNNs or Transformers)?\n\n  A4: We would like to clarify that LoRA can indeed be applied to Transformer architectures. We understand and appreciate the reviewer’s concern regarding its applicability in graph-based models. Currently, most GNN frameworks offer limited support for LoRA and other PEFT techniques. To accommodate this, in our design, we apply LoRA only to the final linear head of the GNN.\n\n  For instance, in one of our evaluated datasets, **QM9**, which employs a GNN for regression tasks, we restrict teleportation and LoRA adaptation to the last linear layer. Despite this minimal intervention, our approach still achieves notable performance improvements, suggesting the effectiveness of even partial LoRA integration in non-Transformer models.\n\n  \n\n>* Q5: Could imperfect teleportation or over-teleportation lead to optimization instability? Any examples observed?\n\n  A5: In our current experimental setting, we did not observe optimization instability. However, to proactively address the reviewer’s concern, we manually constructed an imperfect teleportation scenario. Specifically, we performed early stopping halfway through the LoRA optimization phase, and present the resulting performance comparison below:\n\n  |                   | mIoU  | Pix. Acc. | Abs. Err. | Rel. Err. | $\\Delta m \\\\%$ |\n  | ----------------- | ----- | --------- | --------- | --------- | ------------ |\n  | COST              | 75.73 | 93.53     | 0.0133    | 31.53     | 4.30         |\n  | COST (Early Stop) | 74.40 | 92.96     | 0.0134    | 32.54     | 6.04         |\n\n  As shown, **imperfect teleportation—caused by premature termination of LoRA optimization—results in degraded MTL performance**. This observation supports the importance of fully optimizing teleportation steps to ensure the effectiveness of our method.\n\n  \n\n>* Q6: Limitation: COST introduces 30% training time overhead and added complexity.\n\n  A6: We acknowledge that our method incurs an approximately 30% increase in training time, which we recognize as a limitation and have explicitly discussed in the paper. However, we believe this additional cost is justified by the substantial benefits it offers: (1) Our approach explores a **novel perspective in MTL**, fundamentally distinct from existing optimization-based MTL methods, whose effectiveness remains a subject of debate [7,8]. (2) By incorporating and adapting symmetric teleportation into deep MTL, our method not only consistently achieves state-of-the-art performance across widely adopted MTL benchmarks, but also remains orthogonal and complementary to existing optimization-based approaches.\n\n  \n\n  If you have any further questions, please don’t hesitate to raise it.\n\n  \n\n  **Reference**:\n\n  [1] Multi-Task Learning as a Bargaining Game. ICML 2022.\n\n  [2] Injecting Imbalance Sensitivity for Multi-Task Learning. arXiv:2503.08006, 2025.\n\n  [3]  Introduction to Online Convex Optimization. Foundations and Trends® in Optimization, 2016.\n\n  [4] Online convex optimization in the bandit setting: gradient descent without a gradient. arXiv preprint cs/0408007, 2004.\n\n  [5] Empirical analysis of the strengths and weaknesses of peft techniques for llms. arXiv:2304.14999, 2023.\n\n  [6] Quantum-PEFT: Ultra parameter-efficient fine-tuning. ICLR 2025.\n\n  [7] Do Current Multi-Task Optimization Methods in Deep Learning Even Help? NeurIPS 2022.\n\n  [8] Can Optimization Trajectories Explain Multi-Task Transfer? TMLR 2024."}, {"review_id": "X08MywWen6", "reviewer": "Reviewer_G25M", "summary": "To handle the important issues of optimization conflict in MTL tasks, this paper proposes a plug-and-play solution COST, the key point of which is applying symmetry teleportation. The method employs LoRA to propose a practical and effective solution for symmetry teleportation, and adopts historical trajectory reuse strategy to keep the advantages of optimizers like Adam. Several experiments indicate the performance of COST.", "questions": "1、How can the design of R in Equ.10 mitigate imbalance issues and what’s the performance of this design?\n2、Does the delayed start strategy provide any beneficial effects on model effectiveness or convergence, in addition to saving training time?\n3、What’s the performance of COST-weak when relaxing from [K/2, K/3, K/4, K/5]？Does COST-dominated offer the best performance in most datasets?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1、This work mitigates optimization conflicts in MTL by applying symmetry teleportation and reformulating it through LoRA to obtain a practical implementation. Besides, the advantages of optimizers like Adam can be retained by the proposed historical trajectory reuse strategy.\n2、This method is evaluated with several mainstream MTL benchmarks and compared to a set of optimization methods, and performs competitively.\n3、This paper is well written with detailed description, proofs and visualization.\n\nWeaknesses\n1、It seems that the paper does not address the issue of task imbalance in MTL. While the model achieves good performance in most overall metrics, the model's performance varies significantly across different tasks, which raises concerns about the robustness and stability of the proposed approach.\n2、This method is time-consuming, especially when the trigger condition relaxes. Even with delayed start strategy and frequency control strategy, COST\nintroduces an additional 30% of the training time, making it hard to use in large-scale datasets.", "was_revised": true, "final_justification": "The authors provided evidence that the design of R effectively alleviates imbalance issues in multi-task learning scenarios and achieves state-of-the-art performance in overall metrics. However, COST shows a poorer performance on Segmentation task in NYUv2 (Table 2), indicating the seesaw phenomenon between different tasks. Considering the concerns on task-specific robustness and training efficiency, I maintain my original rating.", "rebuttal": "We sincerely thank Reviewer G25M for the constructive and insightful comments. Below, we provide detailed point-by-point responses to each of your concerns.\n\n>* Q1: It seems that the paper does not address the issue of task imbalance in MTL. While the model achieves good performance in most overall metrics, the model's performance varies significantly across different tasks, which raises concerns about the robustness and stability of the proposed approach.\n\n  A1: We would like to clarify that the designs in Equation (9) and Equation (10) are specifically tailored to address task imbalance. To facilitate balanced gradient exploration, we first compute the re-weighting factor $R$ based on task gradient norms (Eqn. (10)) prior to teleportation. This factor is then used to re-weight the task sharpness in Eqn. (9), guiding the search toward the most balanced descent direction during teleportation. Additionally, we emphasize that COST consistently achieves state-of-the-art performance across all evaluated datasets, with notable improvements on CityScapes, NYUv2, and CelebA.\n\n  \n\n>* Q2: This method is time-consuming, especially when the trigger condition relaxes. Even with delayed start strategy and frequency control strategy, COST introduces an additional 30% of the training time, making it hard to use in large-scale datasets.\n\n  A2: We acknowledge that our method incurs an approximately 30% increase in training time, which we recognize as a limitation and have explicitly discussed in the paper. However, we believe this additional cost is justified by the substantial benefits it offers: (1) Our approach explores a **novel perspective in MTL**, fundamentally distinct from existing optimization-based MTL methods, whose effectiveness remains a subject of debate [1,2]. (2) By incorporating and adapting symmetric teleportation into deep MTL, our method not only consistently achieves state-of-the-art performance across widely adopted MTL benchmarks, but also remains orthogonal and complementary to existing optimization-based approaches.\n\n  \n\n>* Q3: How can the design of R in Equ.10 mitigate imbalance issues and what’s the performance of this design?\n\n  A3: Please refer to **A1** for further clarification. As additionally suggested, we conducted an ablation study to assess the impact of the re-weighting factor $R$. The results are presented below:\n\n  |              | mIoU  | Pix. Acc. | Abs. Err. | Rel. Err. | $\\Delta m \\\\%$ |\n  | ------------ | ----- | --------- | --------- | --------- | ------------ |\n  | COST         | 75.73 | 93.53     | 0.0133    | 31.53     | 4.30         |\n  | COST w/o $R$ | 74.53 | 93.04     | 0.0137    | 33.78     | 7.58         |\n\n  \n\n  As shown, removing the re-balancing design leads to a notable performance degradation, indicating the effectiveness of our proposed re-weighting mechanism.\n\n  \n\n>* Q4: Does the delayed start strategy provide any beneficial effects on model effectiveness or convergence, in addition to saving training time? \n\n  A4: Currently, the delayed start strategy is adopted purely as a time-saving mechanism.\n\n  \n\n>* Q5: What’s the performance of COST-weak when relaxing from [K/2, K/3, K/4, K/5]？Does COST-dominated offer the best performance in most datasets?\n\n  A5：We would like to clarify that we did not evaluate COST-weak on multiple datasets for the following reasons:\n   (1) Defining weak conflicts becomes increasingly ambiguous and challenging in scenarios involving more than two tasks.\n   (2) Even under the dominated conflict condition, CelebA already exhibits frequent teleportation triggering, incurring considerable computational overhead—this would be exacerbated under the weak condition.\n\n  Given the limited rebuttal timeframe, it is infeasible for us to conduct such extensive additional experiments. Moreover, as illustrated in **Figure 2**, weak conflicts may not pose a significant issue when task gradients exhibit approximately equal norms. Finally, **Table 7** already presents a comparison between COST-dominated and COST-weak, showing that the former substantially outperforms the latter.\n\n  \n\n  If you have any further questions, please don’t hesitate to raise it.\n\n  \n\n  **Reference**:\n\n  [1] Do Current Multi-Task Optimization Methods in Deep Learning Even Help? NeurIPS 2022.\n\n  [2] Can Optimization Trajectories Explain Multi-Task Transfer? TMLR 2024."}, {"review_id": "tkAE68HB89", "reviewer": "Reviewer_sH9s", "summary": "This paper proposes Continual Optimization with Symmetry Teleportation (COST), which seeks a new point on the loss-invariant contour to reduce gradient conflict. Besides, the authors proposes Historical Trajectory Reuse Strategy to keep the optimizer history. Experiments show improvements over previous methods.", "questions": "1. Why do we use L1 norm in eq. 7? What would happen if we use L2 norm?\n2. What would happen if we relax the trigger to 0? By that I mean change the r.h.s. of eq. 4 to 0. Will it significantly improve the results?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The idea of transporting the parameters on the loss contour to resolve gradient conflict is reasonable and interesting. It seems a promising direction to push multi-task learning one step forward.\n2. The authors did extensive experiments to compare COST with previous methods. This is very informative for readers.\n\nWeaknesses:\n1. Why do we even need LORA? We can simply optimize the backbone parameters with Eq.11 when conflict occurs.\n2. In Section 5.2, I see no support on the statement 'These results might address another concern regarding COST, namely: Are the improvements brought about by COST rooted in the capability expansion facilitated by LoRA?' Could the authors elaborate that?\n3. To prove that the improvement is not from LORA, the authors should add the following ablation study: when conflict occurs, instead of optimize eq. 11 with lora, directly optimize the original objective eq. 5 with lora, and then compare the difference in performance.", "was_revised": true, "final_justification": "After the rebuttal, I will keep my ratings unchanged. I encourage the authors to add the experiment results of Q&A5 in the next revision.", "rebuttal": "We sincerely thank Reviewer sH9s for the constructive and insightful comments. Below, we provide detailed point-by-point responses to each of your concerns.\n\n>* Q1: Why do we even need LORA? We can simply optimize the backbone parameters with Eq.11 when conflict occurs.\n\n  A1: Yes, it is feasible to directly optimize the backbone for symmetric teleportation. In our work, we employ LoRA primarily for efficiency purposes.\n\n  \n\n>* Q2&Q3: In Section 5.2, I see no support on the statement 'These results might address ... by LoRA?' Could the authors elaborate that? To prove that the improvement is not from LORA, the authors should add the following ablation study: ...\n\n  A2&A3: Thank you for your valuable suggestion. We have conducted the comparison experiments accordingly, and the results are presented below:\n\n  |                | mIoU  | Pix. Acc. | Abs. Err. | Rel. Err. | $\\Delta m \\\\%$ |\n  | -------------- | ----- | --------- | --------- | --------- | ------------ |\n  | COST (Eqn. 5)  | 75.73 | 93.53     | 0.0133    | 31.53     | 4.30         |\n  | LoRA (Eqn. 11) | 75.70 | 93.66     | 0.0134    | 32.61     | 5.40         |\n\n  As shown, applying LoRA alone does not lead to performance gains. One possible explanation is that the paradigm of alternating between pre-training and LoRA-based fine-tuning may not be optimal for this setting.\n\n  \n\n>* Q4: Why do we use L1 norm in eq. 7? What would happen if we use L2 norm?\n\n  A4: Thank you for your valuable suggestion. We conducted an additional experiment by replacing the L1 norm in Eq. (7) with the L2 norm, and present the results below. As shown, the L2 norm performs slightly worse than the L1 norm, but still achieves SOTA performance.\n\n  |                | mIoU  | Pix. Acc. | Abs. Err. | Rel. Err. | $\\Delta m \\\\%$ |\n  | -------------- | ----- | --------- | --------- | --------- | ------------ |\n  | COST (L1 norm) | 75.73 | 93.53     | 0.0133    | 31.53     | 4.30         |\n  | COST (L2 norm) | 74.43 | 92.83     | 0.0128    | 32.13     | 4.46         |\n\n  \n\n>* Q5: What would happen if we relax the trigger to 0? By that I mean change the r.h.s. of eq. 4 to 0. Will it significantly improve the results?\n\n  A5：As illustrated in Figure 5, setting the right-hand side (RHS) of Eq. (4) to zero results in frequent teleportations, with a conflict ratio of 97% per epoch. Due to the limited rebuttal time, we were unable to conduct further experiments under this configuration. Nevertheless, as shown in Figure 10 (Appendix), reducing the threshold from $K/2$ to $K/3$, $K/4$, and $K/5$ yields results of 0.76, 0.82, and 0.65, respectively. These outcomes are generally consistent with our expectations and further support the rationale behind our design choices.\n\n  \n\n  If you have any further questions, please don’t hesitate to raise it."}, {"review_id": "luJng1hReV", "reviewer": "Reviewer_WWrv", "summary": "The paper 'Continual Optimization with Symmetry Teleportation for Multi-Task Learning' proposes to use symmetry teleportation [Zhao, 2022] to deal with optimization conflict and task imbalance, which are common problems in multi-task learning. As the authors point out, the original symmetry teleportation is not scalable to larger and general neural networks as it usually requires some invertibility assumptions. This paper proposes to implement symmetry teleportation on large scale neural networks using a LoRA method to circumvent some of the difficulties. They also developed historical trajectory reuse strategy (HTR) to mimic momentum-based optimizers like Adam. Some empirical experiments are presented to show the effectiveness of COST.", "questions": "Major: \n1. In eq (4), how do you decide the RHS? How does the model performance and running time vary when you vary the bound on RHS? \n2. eq (7) and and Fig 5 (b) seem inconsistent to me. Since $\\mathcal{L}^{\\star}_i $  is the loss before teleportation, shouldn't we have $\\mathcal{L}^{\\star}_i = \\mathcal{L}_i$ before teleportation? Then shouldn't $L_t$ in Fig 5(b) starts from $(0,0)$? Please clarify. \n3. eq (8) and eq (9) seem inconsistent to me. Why does $\\mathcal{L}(\\theta^*)$ disappear in eq (9)? It seems that you are suggesting $\\max_{\\varepsilon} |\\mathcal {L}(\\theta+\\varepsilon) - \\mathcal{L}(\\theta)| = \\max_{\\varepsilon} |\\mathcal {L}(\\theta+\\varepsilon)|$ which is not true. \n4. In line 212, how do you choose $\\tilde{n}$? Is there a systematic way to choose $\\tilde{n}$ (maybe in terms of the dimension of $\\theta$)? How does the model performance and running time vary when you vary $\\tilde {n}$? \n5. The results for experiments are averaged over 3 seeds, which seems not enough. Improvement in numerical experiments seems marginal when considering the error bar (e.g. in Table 1, $4.30 \\pm 1.3$ vs $5.18$). Averaging over more seeds might reduce the error bar and give a clearer result. \n6. minimizing $\\mathcal{L}_t$ in eq (7) is only an approximation to the loss invariance assumption. What is the stopping criteria? In addition, how to compare the performance under this approximation with the performance under a true loss invariance assumption? \n7. Is COST with HTR more sensitive to parameter tuning vs vanilla COST (lr, batch size, total iteration, etc)? Please clarify. \n\n\nMinor: \n1. Page 3 eq (2), $a = \\mathrm{argmax}_{a\\in A} \\ldots$  is not a good notation. Consider using $a^*$ or something else on the left hand side. Similar issue in eq (6). \n2. In Fig 5 (a), I cannot see the $K/2$ Condition bar for NYU v2. Is it 0? If so, please state it clearly. \n3. line 197-199 is awkward, please correct the English. There are other typos and sentences that need to be fixed. Please proofread the paper carefully. \n4. Please correct the format of Algorithm 1 in Appendix C.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1.The paper is well-organized and includes illustrations to clarify the ideas.\n2. The idea to use symmetry teleportation to reconcile gradient conflicts and imbalance seems novel. \n\n\nWeaknesses:\nOverall, the quality of the paper can be greatly improved (English, notations). There are several major inconsistencies that need to be clarified. Current numerical experiments are insufficient to justify the significance of this paper. Several important unanswered questions are raised below.", "was_revised": true, "final_justification": "The authors have addressed all my concerns with experiments. Thus I would like to increase my score.", "rebuttal": "We sincerely thank Reviewer WWrv for the constructive and insightful comments. Below, we provide detailed point-by-point responses to each of your concerns.\n\n>* Q1: In eq (4), how do you decide the RHS? How does the model performance and running time vary when you vary the bound on RHS?\n\n  A1: In Figure 10 (Appendix), we provide an analysis of the trigger condition on the CelebA dataset. As shown, when the condition is varied from $K/2$, $K/3$, $K/4$, to $K/5$, the corresponding MTL performance scores are [0.93, 0.76, 0.82, 0.65], while the associated training time costs are [40.5, 51.7, 80.5, 112.1] minutes, respectively. Based on this trade-off between performance and computational efficiency, we adopt $K/2$ as the default trigger condition.\n\n  \n\n>* Q2: Eq (7) and and Fig 5 (b) seem inconsistent to me. Since $L_i^\\*$ is the loss before teleportation, shouldn't we have $L_i^\\*=L_i$ before teleportation? Then shouldn't $L_t$  starts from (0,0)?\n\n  A2: We understand the reviewer's concern. Intuitively, if the full dataset or an identical batch order were used to compute both losses, we would expect $L_i^* = L_i$. However, in practice, a randomly shuffled batch-wise training paradigm is adopted, which leads to non-identical losses at the beginning due to inherent sampling variance.\n\n  \n\n>* Q3: Eq (8) and eq (9) seem inconsistent to me. Why does $L(\\theta)$ disappear in eq (9)? It seems that you are suggesting ....\n\n  A3: We apologize for the confusion — this was a typo introduced for ease of presentation. When applying the weighting factor $R_i$, the term $L(\\theta)$ cannot be omitted. However, we would like to clarify that in our actual implementation, we do preserve the subtraction of $L(\\theta)$. The relevant code snippet is provided below, and we are committed to releasing our code upon acceptance:\n\n  ```\n  sharp_loss.append(torch.stack([ratio_list[i] * torch.abs(lora_loss[i] - L0[i]) for i in range(n_tasks)]))\n  ```\n\n  Here, `ratio_list[i]` corresponds to $R_i$, `L0[i]` represents $L(\\theta^\\*)$, and `lora_loss[i]` is $L_i(\\theta^\\* + \\epsilon)$.\n\n  \n\n>* Q4: In line 212, how do you choose $\\tilde{n}$? Is there a systematic way to choose $\\tilde{n}$? How does the model performance and running time vary when you vary $\\tilde{n}$?\n\n  A4: For operations akin to zero-order optimization, the gradient approximation is theoretically unbiased in expectation. However, when the number of samples is small (e.g., 1), the variance of the estimate becomes large, resulting in a potentially significant deviation from the true gradient. In our work, we set the number of samples based on common practices in related literature [1]. To further address the reviewer's concern, we conducted additional experiments with 10 and 15 samples. The results are presented below:\n\n  |             | mIoU  | Pix. Acc. | Abs. Err. | Rel. Err. | $\\Delta m \\\\%$ |\n  | ----------- | ----- | --------- | --------- | --------- | ------------ |\n  | COST (n=5)  | 75.73 | 93.53     | 0.0133    | 31.53     | 4.30         |\n  | COST (n=10) | 73.86 | 93.14     | 0.0131    | 31.25     | 4.42         |\n  | COST (n=15) | 74.32 | 92.95     | 0.0125    | 32.52     | 4.19         |\n\n  As shown, the performance across $n=5$, $n=10$, and $n=15$ remains relatively stable, with the best results observed at $n=15$. However, a larger sample size also incurs greater computational overhead. Hence, we adopt $n=5$ as a trade-off between performance and efficiency.\n\n  \n\n>* Q5: The results for experiments are averaged over 3 seeds, which seems not enough. Improvement in numerical experiments seems marginal when considering the error bar. Averaging over more seeds might reduce the error bar and give a clearer result.\n\n  A5: We would like to clarify that nearly all mainstream MTL approaches report results averaged over 3 random seeds (e.g., CAGrad, Nash-MTL, AlignMTL, FAMO, FairGrad, etc.). To further address the reviewer’s concern, we conducted and reported the following additional experiments:\n   (1) As a baseline, we faithfully re-ran FairGrad’s official code over three random seeds (consistent with both their implementation and ours), and reported the performance as *FairGrad-R* in the table below.\n   (2) As suggested, we further evaluated our method using two additional seeds, and reported the performance as *COST (5 seed)*.\n\n  |               | mIoU         | Pix. Acc.    | Abs. Err.       | Rel. Err.    | $\\Delta m\\\\%$ |\n  | ------------- | ------------ | ------------ | --------------- | ------------ | ----------- |\n  | FairGrad*     | 75.72        | 93.68        | 0.0134          | 32.25        | 5.18        |\n  | FairGrad-R    | 75.14 ± 0.71 | 87.08 ± 0.11 | 0.0142 ± 0.0019 | 30.76  ± 1.60 | 5.58 ± 3.82 |\n  | COST (3 seed) | 75.71 ± 0.25 | 93.53 ± 0.12 | 0.0133 ± 0.0003 | 31.53 ± 1.85 | 4.30 ± 1.30 |\n  | COST (5 seed) | 75.12 ± 0.83 | 93.31 ± 0.31 | 0.0131 ± 0.0004 | 31.57 ± 1.53 | 4.32 ± 1.53 |\n\n  As shown, COST maintains stable and consistent performance when evaluated on more seeds, with only a slight increase in variance. Notably, under the same evaluation protocol, our method demonstrates substantially better stability compared to its baseline, *FairGrad-R*.\n\n  \n\n>* Q6: minimizing $L_t$ in eq (7) is only an approximation to the loss invariance assumption. What is the stopping criteria? In addition, how to compare the performance under this approximation with the performance under a true loss invariance assumption?\n\n  A6: Since training is performed in a batch-wise manner, it is inherently difficult to determine whether the teleported point precisely lies at the symmetric position. Nevertheless, we would like to clarify that even the original symmetric teleportation work [2] does not achieve perfect loss invariance, primarily due to the inexact nature of gradient descent-based optimization.\n\n  \n\n>* Q7: Is COST with HTR more sensitive to parameter tuning vs vanilla COST (lr, batch size, total iteration, etc)? Please clarify.\n\n  A7: To address the reviewer’s concern, we conducted the suggested comparison by varying batch sizes in the LoRA optimization and present the results below. As shown, HTR consistently enhances performance across different batch sizes. Furthermore, based on the observed standard deviations, COST with HTR exhibits greater robustness to batch size variation compared to COST without HTR.\n\n  |                | with HTR | w/o HTR  |\n  | -------------- | -------- | -------- |\n  | batch size = 4 | 4.68     | 4.92     |\n  | batch size= 8  | 4.30     | 4.65     |\n  | batch size= 16 | 4.97     | 5.62     |\n  | **std**        | **0.34** | **0.50** |\n\n  \n\n>* Q8: In Fig 5 (a), I cannot see the K/ 2 Condition bar for NYU v2. Is it 0? If so, please state it clearly.\n\n  A8: Since the K/2 condition is only applied to CelebA and not to NYUv2, it is not displayed in the corresponding results. We will revise the text to clarify this distinction.\n\n  \n\n>* Q9: Typos or inappropriate expression.\n\n  A9: Thank you for your reminder. We will correct these issues and carefully proofread the paper to ensure clarity and consistency.\n\n  \n\n  If you have any further questions, please don’t hesitate to raise it.\n\n  \n\n  **Reference**:\n\n  [1] ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization. NeurIPS 2018.\n\n  [2] Symmetry Teleportation for Accelerated Optimization. NeurIPS 2022."}]

Author rebuttal:

Meta-review: This paper proposes COST, a continual optimization framework using symmetry teleportation to mitigate gradient conflict and task imbalance in multi-task learning. Reviewers highlighted the method’s novelty, strong empirical performance across multiple benchmarks, and its plug-and-play nature. Concerns were raised about training overhead, task imbalance, and ablation coverage, but the authors provided detailed responses and additional experiments that addressed most of these points. During the rebuttal phase, all reviewers actively engaged in the discussion phase, and several updated their scores positively after rebuttal. Overall, the contribution is solid, with clear motivation and extensive experiments. The AC encourages the authors to incorporate reviewer feedback and clarifications into the final version.
Decision: Accept (poster)