Selective Task Group Updates for Multi-Task Optimization
Authors: Wooseong Jeong
Kuk-Jin Yoon
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Task Learning
Multi-Task Optimization
Proximal Inter-Task Affinity
Primary area: learning theory

Abstract: Multi-task learning enables the acquisition of task-generic knowledge by training multiple tasks within a unified architecture. However, training all tasks together in a single architecture can lead to performance degradation, known as negative transfer, which is a main concern in multi-task learning. Previous works have addressed this issue by optimizing the multi-task network through gradient manipulation or weighted loss adjustments. However, their optimization strategy focuses on addressing task imbalance in shared parameters, neglecting the learning of task-specific parameters. As a result, they show limitations in mitigating negative transfer, since the learning of shared space and task-specific information influences each other during optimization. To address this, we propose a different approach to enhance multi-task performance by selectively grouping tasks and updating them for each batch during optimization. We introduce an algorithm that adaptively determines how to effectively group tasks and update them during the learning process. To track inter-task relations and optimize multi-task networks simultaneously, we propose proximal inter-task affinity, which can be measured during the optimization process. We provide a theoretical analysis on how dividing tasks into multiple groups and updating them sequentially significantly affects multi-task performance by enhancing the learning of task-specific parameters. Our methods substantially outperform previous multi-task optimization approaches and are scalable to different architectures and various numbers of tasks.

Published as a conference paper at ICLR 2025
SELECTIVE TASK GROUP UPDATES FOR MULTI-TASK OPTIMIZATION
Wooseong Jeong & Kuk-Jin Yoon
Korea Advanced Institute of Science and Technology (atk14570, k kjyoon)
@aaonnkkairr @ka lkkiss sss..a ac.kr kr
ABSTRACT
Multi-task learning enables the acquisition of task-generic knowledge by training multiple tasks within a unified architecture. However, training all tasks together in a single architecture can lead to performance degradation, known as negative transfer, which is a main concern in multi-task learning. Previous works have addressed this issue by optimizing the multi-task network through gradient ma- nipulation weighted loss adjustments. However, their optimization strategy focuses on addressing task imbalance in shared parameters, neglecting the learn- ing of task-specific parameters. As a result, they show limitations in mitigating negative transfer, since the learning of shared space and task-specific information influences each other during optimization. To address this, we propose different approach to enhance multi-task performance by selectively grouping tasks and updating them for each batch during optimization. We introduce an algorithm that adaptively determines how to effectively group tasks and update them during the learning process. To track inter-task relations and optimize multi-tase networks simultaneously, we propose proximal inter-task affinity, which can be measured during the optimization process. We provide a theoretical analysis on how dividing tasks into multiple groups and updating them sequentially significantly affects multi-task performance by enhancing the learning of ssscispccfiic parameters. Our methods substantially outperform previous multi-task optimization approaches and are scalable to different architectures and various numbers of tasks.
INTRODUCTION
Multi-task learning (MTL) stands out as key approach for crafting efficient and robust deep learning models that can adeptly manage numerous tasks within a unified architecture (Caruana) 1997). By training related tasks within a single network, MTL facilitates the acquisition of universal knowledge spanning multiple tasks, thereby enhancing generalization and accelerating convergence. Additionally, MTL reduces the need for expensive computing and storage resources by employing a shared network across tasks, making it a favorable choice for future generalized networks across many applications.
The goal of MTL is to mitigate negative transfer (Crawshaw 2020) between tasks. Since each task possesses its own distinct objective function, improving performance one task can potentially impede the performance of others. This phenomenon, characterized by a trade-off among tasks' negative transfer. To mitigate negative transfer during optimization,
performances, is referred
task-specific gradients are manipulated (Désidéri) 2012) Sener & Koltun 2018 Yu et al. 2020 Liu
al. 2021a b. Navon et al. 2022 Senushkin et al.) (003), tasks' losses are adaptively weighted
(Kendall et a 2018 Guo et al. 2018: Liu et al.) 2019; 2024), or combination of both approaches used (Liu et al. (02 Previous research focused on balancing the influence of different tasks by considering unbalance in gradients of shared parameters. However, these analyses overlooked the role of task-specific parameters. Since the learning of shared space and task-specific information influence each other during optimization, balanced shared space across tasks cannot be achieved without learning task-specific information in the ssecispe parameters.
Therefore, we take a fundamentally different approach to mitigate negative transfer in MTL (see Figure .T. Our experiments reveal a notable difference in multi-task performance depending on whether task losses updated collectively sequentially. By grouping tasks and updating them

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
Determine Loss weights
Gredient Manipulation
Selective Task Group Updates
w
" weLt
wrLi
Va, 8: Fo
8:
Backpropagation
Backpropaggtion
Grouping &
62
Forward Pass
Forward Pass
Forward Pass
Backpropagation
Update
Sequential Update
Updarte
Figure 1: Comparison of multi-task optimization methods. θ represents the network parameters, and {c) denotes the task-specific losses for K. tasks. (a) Loss-based approaches balance the loss by adjusting the weights {w; during optimization. (b) Gradient-based approaches modify the task- task-specific gradients {g: f=11 with respect to O. (c) Our method divides the tasks into M groups (in this case, M = 2) and updates them sequentially for each batch during optimization.
sequentially, the network can focus on specific task groups in turn, facilitating the learning of task- specific parameters. Thus, our main objective is to identify optimal strategies for grouping and updating tasks during optimization. Inter-task affinity (Fifty et al. (021), which evaluates the loss change of one task when updating another task's 's gradients in the shared parameters, is useful metric for understanding task relations. However, directly incorporating inter-task affinity into optimization presents several challenges: (i) It only considers updates to shared parameters, ignoring the influence of task-specific parameter updates. (ii) Inter-task affinity varies significantly throughout optimization, so relying on the average across time steps may not fully capture task relations. (iii) Measuring inter-task affinity is computationally intensive because it involves recursive gradient updates, tracking changes in loss, and reverting to previous parameter states. To address these issues, we introduce proximal inter-task affinity (Section 3.1. which concurrently explains the updates of shared and task- sesk--pccific parameters. This metric can be tracked over time to approximate inter-task relations during optimization without significant computational overhead.
By capturing task relations, we introduce an algorithm that dynamically updates task groups using proximal inter-task affinity to effectively mitigate negative transfer between tasks (Section (3.2). Additionally, we present practical methods for inferring proximal inter-task affinity during task group updates. In the theoretical analysis (Section 4. we explain how this sequential update strategy can improve multi-task performance from optimization standpoint. Previous approaches have demonstrated convergence to Pareto-stations -tatto-stationary points, where the sum of all task-specific gradients equals zero (Désidéri) 2012 Sener & Koltun, 2018 Yu et al. 2020 Liu et al. 202 b Navon et af. 2022) Senushkin et al.] (023). However, conventional convergence analysis cannot explain why sequential updates yield better multi-task performance with comparable stability, as they handle the learning of shared and task-specific parameters separately. Instead, we provide a theoretical explanation of the relations between the updates of shared and task-specific parameters and how the update sequence affects this. This perspective is not addressed in traditional multi- task optimization, which typically deals with the learning of shared and task-specific parameters independently. As result, our approach facilitates the learning of task-specific parameters with stability similar to other optimization methods, while achieving faster convergence compared to previous gradient-based approaches. Experimental results demonstrate that adaptively partitioning the task set and sequentially updating them during the optimization process significantly enhances multi-task performance compared to previous works. Our main contributions are threefold:
We propose algorithm that dynamically groups tasks for updating each batch during multi-task optimization. To do this, we introduce proximal inter-task affinity, which we can track throughout the optimization process to group tasks accordingly.
We provide theoretical explanation of the advantages of sequentially updating task groups based on proximal inter-task affinity. We discuss how sequential updates can enhance the learning of task-specific parameters and improve multi-task performance, result that traditional multi-task optimization analyses do not explain.
Our methods demonstrate superior performance across various benchmarks when compared to pre- vious multi-task optimization techniques, including both loss-based and gradient-based based approaches.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Optimization methods in MTL can be broadly categorized into those that manipulate task-specific
gradients (Chen et al.] 2018: 2020; Javaloy & Valera 2021 Désidéri 2012 Sener & Koltun 2018 Yu
cct al.
2020
Liu et al. 2021a la b Phan et al., 20,2)2 et al. 2022 Senushkin et al. 2002 Jeong &
Yoon 2024) and those that adjust the weighting of task-specific losses (Kendall al. [01 8 Guo
it al. 2018 Liu et al. [0199] 2024). Addressing the imbalance in task influence, normalized gradients
are utilized (Chen et al., (011)) to prevent task spillover. Introducing stochasticity the network's s
parameters based on gradient consistency, GradDrop (Chen et al. (020) drops gradients. RotoGrad
(Javaloy & Valera 202T) rotates the network's feature space to align tasks. MGDA (Sener & Koltun 2018) treats MTT. as multi-objective problem, minimizing the norm point in the convex hull. CAGrad (Liu et al., 202 la) minimizes multiple loss functions and regulates trajectory using worst local improvements of individual tasks. Aligned-MTL (Senushkin al., 2023) stabilizes optimization by aligning principal components of gradient matrix.. The weighting of losses significantly impacts multi-task performance tasks with higher losses can dominate training. Addressing this, tasks'
losses are weighted based on task-dependent uncertainty (Kendall et al. 2018). Prioritizing tasks according difficulty by evaluating validation results (Guo al. [018)) or balancing multi-task loss reflecting loss descent rates (Liu et al. 2019), are also proposed strategies. Previous works on multi-task optimization that directly modily task-specific gradients Chen et al. (2018) 2020 Javaloy
& Valera (2021): Désidéri
(2012) Sener & Koltun (2018); Yu et al. (2020)] Liu et al. 202 b Phan
jt al. (2022); Navon al. 2022); Senushkin et al. (2023): Jeong Yoon (2024) focus on learning the
shared parameters of the network without considering their interaction with task speci--ccificc parameters
during the optimization step. In contrast, we propose fundamentally different strategies to mitigate
negative transfer by sequentially updating task groups which lead to better multi-task performance. phenomenon not explained by conventional multi-task optimization analysis.
3 SELECTIVE TASK GROUP UPDATES FOR MULTI-TASK OPTIMIZATION
3.1 THE GOAL OF MULTI-TASK LEARNING AND INTER-TASK AFFINITY
In the context of MTL, we deal with multiple tasks with its own objective function denoted as {C.)K= where 1C represents the number of tasks. The network parameters Q are partitioned into (O., θ1, 02,00.--000 ... ....... θx), where O, refers the shared parameters while θ represents the task- specific parameters for the task i. To simplify the discussion, we assumed that θ, is shared across all tasks. However, the following discussion can still apply even when it is only partially shared for specific task sets. The goal of MTL is to find Pareto-optimal parameters (Sener & Koltun 2018)). denoted as O", that minimize the weighted sum of task-specific losses. This can be expressed as O" arg ming Σ (.,, Q2), where w; represents the weight for the loss of task i.
The concept of inter-task affinity (Fifty et al. 2021) aims to identify tasks that enhance each other's performance when learned together on shared networks. It measures the extent to which the gradient of specific task with respect to the shared parameters O.. affects the loss of other tasks.
Definition 1 (Inter- Task Affinity). Consider a multi-task network shared by tasks i and k. For a data sample z' and a learning rate T, the task-specific gradients from L, are applied to update the shared parameters of the network as follows: O²+1 = O ve Cx(2' Or,, O²). The inter-task affinity from task i to task lo at time step t is then defined as:
Lx(2", 1 O
Ai = 1-
Le(z', 0','",,',,0) 0
In the context of task grouping (Fifty et al. 202T), task affinity is computed and utilized in two stages. Initially, task affinity within shared network is assessed by monitoring how the loss for each task changes when updating the loss of the other task at each step. Subsequently, using the averaged task affinity, task groups are organized, and each group is trained in separate networks. However, directly incorporating inter-task affinity for multi-task optimization presents several challenges. First, it only tracks the learning of shared parameters, O,, and their influence task losses, without considering task spec-sse parameters, {e,) In practice, both shared and task-specific parameters need be optimized simultaneously. Second, since inter-task affinity fluctuates significantly during
3

---

Published as a conference paper at ICLR 2025
optimization, it must be tracked continuously capture evolving task relations. This process is computationally expensive, requiring recursive gradient updates, monitoring loss changes, and reverting to previous parameter states.
3.2 PROXIMAL INTER TASK AFFINITY FOR MULTI-TASE OPTIMIZATION
To address the issues mentioned above, we propose practical methods based on the concept of proximal inter-task affinity to track task relations and use this information for concurrent multi-task optimization. Firstly, we integrate the learning of task-specific parameters into the estimation of inter- task affinity to incorporate this affinity into ongoing optimization. Secondly, we extend the concept of inter-task affinity to elucidate relations between task groups rather than individual pairs of tasks. This allows us to monitor these affinities when a set of tasks is collectively backpropagated. Practically, proximal inter-task affinity serves as an approximation of inter- task affinity; however, it also plays a crucial role in theoretical analysis, discussed in Section 4 explaining why updating task groups sequentially yields distinct multi-task performance. To be specific, we lay out guidelines for updating proximal inter-task affinity by distinguishing between inter-group relations and intra-group relations. We then incorporate the concept of proximal inter-task affinity to explore the task update strategy, grouping tasks based on their affinity, which has a significant impact on multi-task performance.
Before delving into the algorithm, let's establish some notations building upon those introduced in Section 3.1 Our objective is to partition tasks into several groups, denoted as {G,, G2,... G.M } where M is the number of task groups. Each group set contains tasks' indices as their components, for example, G2 = j. When selecting tasks to form these groups, we require the task groups to be mutually exclusive, meaning that G, inn G5- = 0 for any i and (i $ j).
Tracking inter-task affinity during optimization, as defined eq. (T). impractical. This is because it is difficult to assess the influence of individual tasks on shared network parameters when multiple tasks are updated simultaneously. Additionally, task-specific parameters {0.) must also be updated concurrently, which is essential for MTL settings. Therefore, we practically approximate inter-task task affinity by utilizing proximal inter-task affinity, taking into account updates of multiple tasks with task susk-specific parameters. The difference between inter-task affinity and proximal inter-task affinity is difficult to detect, so we provide a more detailed discussion in Appendix A.
Definition 2 (Proximal Inter-Tast Affinity.. Consider a multi-task network shared by the task set G, with their respective losses defined as La- For data sample 2' and a learning rate т}, the gradients of task set G are updated to the parameters of the network as follows: O+11
O Ve; La(z O',, O')) and 6% = 0 - Ve (x(22,00 e%) for l E G. Then, the proximal inter-task affinity from the task set G to Tk at time step is defined as:
B B = Lx(=", 1
Lx(2', O
As it's not feasible to simultancously track inter-task affinity and optimize networks, we instead track the proximal inter-task affinity and utilize it for multi-task optimization. We explain the benefits of integrating inter-task affinity into optimization in the view of conventional multi-task optimization. Theorem [ Juggests that grouping tasks with high inter-task affinity leads to better alignment of their gradients compared grouping tasks with lower affinity. In analysis, we use the extended version of A for multiple tasks (e.g., AG-sk) for ease of notation.
Let gk denote the task-specific gradients backpropagated from the loss function Lx with respect to the parameters O, At given time step I, the inter-task affinity from lask group {ú, k} to task k is greater than or equal to the inter-task affinity from group {j.k} to task k. denoted as Af 2 Ali Then for sufficiently small learning rate rj 4 1. it follows that gi Ih 2 9j 9k-
Theorem 1.
The results also apply to proximal inter-task affinity B instead of A, as the only difference is the inclusion of task-specific parameters for task k.
We analyze how this alignment in task task-specific gradients reduces the loss of task l in Theorem 2
Theorem 2. Let gl denote the task-specific gradients backpropagated from the loss function Lx with respect the parameters O'. Let O represent the updated parameters after applying the
4

---

Published as a conference paper at ICLR 2025
gradients. Assume that for tasks i, 3. and k. the inequality 9: 9h 9; 9k holds. Then, for sufficiently small learning rate 1) < 1, the inequality Lx(=*, O O fx(=*, O O% holds.
The result indicates that when the gradients 9i align better with Ie than with gj. the loss on the reference task k using the updated gradients 9: + 99 is lower than that using the updated gradients 9, + 9k- Theorems I and 2 suggest that updating tasks jointly with high proximal inter-task affinity at each optimization step a reasonable approach.
3.3 TRACKING PROXIMAL INTER-TASK AFFINITY IN SELECTIVE GROUP UPDATES
From now on, we'll explain how we track proximal inter-task affinity in the sequential updates of task groups. For ease of explanation, we start with an already clustered (or initialized) task set {G M obtained from the tracked proximal inter-task affinity in the previous time step. MM is the number of task groups. We'll Il provide more explanation at the end of this section on how the tracked proximal inter-task affinity is utilized in clustering the task set.
Inter-Group Relations. We begin by forwarding the sample z' to the shared multi-task network, from which we obtain the multi-task loss {C,(=", O @=) Based on the task grouping {G. J:m. we divide each time step into M steps to update each task set. For simplicity of discussion, we update the first group G1 in steps from t to + 1/M, but the following discussion can be extended to all sequential updates between t and + 1.
+
1+1/M s
O 1 Σ w/ Li 0,,
IEG
e 1+1/M a represents the resulting parameter after updating the gradient from task group G1 to the parameter Og. Upon forwarding 2' once more, we can access (L,(2', 6/11/M, For tasks that do not belong to G1, their task task-specific parameters remain unchanged, satisfying 6 @++//M = O. Consequently, we can assess the proximal inter-task affinity from group Gl to other tasks. In case where target task Tj does not belong to G1, we refer to this inter-group relations as follows:
s|G] ),
C2 1411/M s/G, .00)
BBo = = B,,, = Al., = =
where TV G1 and Ty
Gi G
(4)
1,(z,003 0))
To be precise, eq. (4) represents the proximal inter-task affinity from G₁ to Tj. However, since т₃ belongs G1, which is group of tasks similar тi. approximate the relations between t; and T, using G1 and Ty. We explain how approximation B ~ Bi can be justified in task group updates from the results of Theorem 3 We can repeat this process iteratively M times, with each
step updating each task group in {G, J.M.
Intra-Group Relations. Inter Inter-group group relation addresses inter-task relations where the source task and the target task T, do not belong to the same group. Conversely, intra- group relation refers to the influence between tasks within the same task group, and it follows a slightly different process. Since both the source and target tasks are updated simultaneously, we must infer inter-task relations based on how the tasks' losses change. Let's suppose we'r re updating the task set G₁- The proximal inter-task affinity between tasks within the same group can be calculated as follows:
BG ~ B =1 Cz(2", I1/ 1/MM G (2' (1 09)
where Ti E G1 and Tj E G1
(5)
Similarly to the inter-group case, we approximate the relations from т; to T, using the relations from Gj Ty and this also can be justified by Theorem when we use cluster task sets. Since the target task T, is included in G,, we infer the inter-task relations by dividing cases into two. If both Bis and _ have positive signs, we infer that the inter-task affinity between 7; and TΙ is positive. Otherwise, if at least one of Bi or B B has a negative sign, the inter-task affinity between T and Ty is negative. This inference is intuitive, as increase in task's 's loss during the decrease of the other task's 's loss implies that they transfer negative influences. However, results in faster convergence compared to previous gradient-based optimization methods, which require number
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: Tracking Proximal Inter- Task Affinity for Task Group Updates
Require: Loss function {Li [_,]i=r- Dataset {2: )=) Total iteration T,
Proximal inter-task affinity (w/ decay) B (B"), Task group set {G.j/z Decay rate 3 E (0,
Initialize: Proximal inter-task affinity 8 = OxxX,
Task group G, = {T:} for 1,.., (M K. for initialization)
for i ,, do
Randomly mix the sequence of task group {G and calculate (L,(=',0) ) for k = : do
Update gradients Er,e, wy VL; backpropagated from losses in G3z
Forward 2 and calculate (x 14 @.ttt/MM for all 1...., K
Calculate Bi-,) (1/(k- and 3-
(M=- 1//M where T; GR for all j 1,,.,KK
TJ $ Gk bi+j
then (k-1k-1)/M = (1 (1) (k (k- 2)/M + | (k- (-)
g2+(k-1)/M 1B++(k-1)/M
else
1 (1 33 (, )
max (k-++/11)// 1,
10
Divide task groups {G: based on proximal inter-task affinity B3(+M-1)/
of hackpropagations equal the number of tasks, along with gradient manipulation. This is further analyzed in Section In the initial phase of learning, we initialize task groups by assigning each task to different group to facilitate the learning of proximal inter-task affinity. For stable tracking, we employ a decay rate 3. Using the tracked proximal inter-task affinity, the next task groups {G, ) are formed by grouping tasks with positive affinity, where MM' is the number of task sets for the next step. During optimization, the number of clustered task sets MM' continuously fluctuates along with their composition. The entire process is outlined in Algorithm
We explain how it effectively approximates inter-task affinity in selective task group updates in Theorem 3 To be specific, compare the inter-task affinity when the target task is included in the source task group versus when it is not.
Theorem 3. The affinity between {i, + and + k satisfies Ai, 2 Ai
When task i and k are within the same task group, we can only access B i,K rather than B during the optimization process. If - < 0, the inter-task affinity also satisfies Ai, S as A Rook S R (see the end of appendix A. According Theorem this condition implies
A S 0. The proposed algorithm separates these tasks into different groups when B which justifies our grouping rules.
Conversely, when tasks т; and belong to separate task groups, we only have access to B k instead of Bi,,--k- In this scenario, the proposed algorithm merges these tasks into the same group Bi 2 0. This incquality also implics Af 2 as B + = A (see the end of appendix A.1) justifying the merging of tasks Ti and based on B during optimization.
THEORETICAL ANALYSIS
We outline the benefits of grouping tasks with positive inter-task affinity and updating them in multi-task optimization in Theorem and Theorem[] We validate the feasibility of tracking proximal inter-task affinity and forming task groups based on this in our grouping updates strategy in Theorem 3 In this section, we clarify why, from the viewpoint of standard convergence analysis in Theorem it's difficult demonstrate that updating task groups sequentially leads to better performance than joint learning. We then explore how sequentially updating grouped task sets can improve multi-task performance, discussed in Theorem 5

---

Published as a conference paper at ICLR 2025
4.1 CONVERGENCE ANALYSIS
We perform conventional convergence analysis to determine the convergence points of the sequential updates of task groups, as outlined in Theorem 4
Conventional analyses treat the learning of shared and task-specific parameters separately, which fails explain why sequential updates result in better multi-task performance while maintaining comparable stability.
Theorem 4 (Convergence Analysis). Given a set of differentiable losses {C:) and Lipschitz continuous gradients with a constant > O, - S we sequentially update task groups {G, based inter-task affinity, employing sufficiently small step size y S min( HR. HIGmT) where |Gml is the number of tasks in Gun then the following inequalities are satisfied for any task group Gm-
all 1, 1.2,.... ...
M
M
La, (2' (2t 6|Gm
Mttm/M
Σ
La. (2', 1 0%:z)
(6)
o/Gm
k=1
M
.m
(1m
Gz
(m
1)/M
++(m- (1-m-111MM
77 2 I1gta.c..
112
(7)
where Gm and Its.G., represent the gradients the shared parameters and task-specific parameters, respectively, for group Gm. Previous approaches, which handle updates of shared and task-specific parameters independently, failing to capture their interdependence during optimization. The term, 9s,G..
1+(m-11/M (E Ia,G, (m-MM) fluctuates during optimization. When the gradients of group Gm
align well with the gradients of the other groups {G&) (=. their dot product yields positive value, leading to decrease in multi-task losses. In practice, the sequential update strategy demonstrates similar level of stability in optimization, which appears to contradict the conventional results. Thus, assume correlation between the learning of shared parameters and task task-specific parameters, where the learning of task-specific parameters reduces gradient conflicts in shared parameters. Under this assumption, the sequential update strategy can guarantee convergence to Pareto-stationary points. This assumption is reasonable, as task-specific parameters capture task-specific information, thereby reducing conflicts in the shared parameters across tasks.
4.2 ADVANTAGES OF SELECTIVE TASK GROUP UPDATES
Nonetheless, sequentially updating task groups facilitates the learning of task-specific parameters, concept not covered by conventional analysis. We directly compare the loss of joint learning with the loss of the sequential group updates strategy. do this, we introduce two-step proximal inter-task affinity, expanded concept of proximal inter-task affinity multiple steps, defined in Appendix(A.6 We analyze the benefits of sequential updates using this two-step proximal inter-task affinity based on the plausible assumptions delineated in Theorem S
Theorem 5. Consider three tasks {i.j, (i.jkk. k}, where task groups are formed with positive inter-task affinity as k and {j}. Assume all losses are convex and differentiable, and the change in affinity during a single step from + (m - 1)/M to t + m/M is negligible. The affinity, which learns all tasks jointly, is denoted as b.j.k-k (1((m- The affinity for the updating sequence ((i,k), (i)) is represented B (m 11/M while for the sequence (Li), ki), it is represented as B g (mm11//MM jiik-
skkij=k
Then, for sufficiently small learning rate v < 1, the following holds:
B1+(m-1)/ (m-)mM B.kks
BC (m-mMM 13
< (m-
(8)
bijk-
jüik-
This suggests that grouping tasks with proximal inter-task affinity and subsequently updating these groups sequentially result in lower multi-task loss compared to jointly backpropagating all tasks. This disparity arises because the network can discern superior task-specific parameters to accommodate task-specific information during sequential learning. Notably, the order in which tasks updated impacts multi-tase task outcomes within a single batch. However, as the optimization progresses, this influence diminishes, as demonstrated in the following experimental results.
7

---

Published as a conference paper at ICLR 2025
Table I: Comparison of time complexity and memory consumption between our optimization methods and other multi-task optimization approaches, including loss-based and gradient-based methods.
Method
Forward Pass
Backgropagation
Gradient Manipulation
Optimizer Step
Aflinity Update
Memory
boss-b Methods
O
O(1)
O(1)
O11)
Gradient-haod Methods
O()
O(CC
O(1)
O(K)
Ours
(M))
O(M)
((M)
0(1)
Forward Pass
Backpropegation
Gradient Naripulation
Optimizer Stop
Clusterigg Aftiity Update
60
0W
40
Gra0Dop
100ml
P0Gnd
Agrad477
HM0
Teeebo
Figure 2: Comparison of the average time required by each optimization process to handle a single batch for tasks on PASCAL-Context (left) and 11 tasks on Taskonomy (right).
EXPERIMENTS
Experimental settings. We assess the proposed techniques using three datasets: NYUD-V2 for indoor vision tasks (Silberman et al. 2012), PASCAL-CConeet for outdoor scenarios (Mottaghi et al. 2014),
and Taskonomy (Zamir et al. 2018) for large number of tasks. Multi-task performance is compared using the metric introduced by (Maninis et al. 2019). This metric calculates the per-task performance
by averaging relative the single-task baseline b: A. = (1/K.) Σ 11 Ma.s)/M.. Mb. where 4 = 1 if lower value of measure Mj means indicates better performance for task Tú, and 0 otherwise. More details are introduced in Appendix C
Baselines. We selected conventional multi-task optimization methods as our baselines: 1) Single-task learning, where each task is trained independently; 2) GD, where all task gradients are updated jointly without manipulation; 3) Gradient- based optimization methods that include gradient manipulation, such as GradDrop (Chen et al. 2020). MGDA (Sener & Koltun 2018). PCGrad (Yu et al., (2020). CAGrad (Liu et al., 2021a), la). A Aigned-MTTL (Senushkin et al. [023)) and Nash-MTL. (Navon et al. 2 4) Loss-based optimization methods, including UW (Kendall et al. 2018), DTP (Guo et al. 2018 DWA (Liu et al. 2019), and FAMO (Liu et al. (024); and 5) A hybrid approach combining gradient and loss-based methods, specifically IMTL (Liu et al., 2021b). Each experiment was conducted three times with different random seeds ensure fair comparison.
Architectures. We use the most common multi-task architecture, employing a shared encoder and multiple decoders, each dedicated to specific task. For our encoder, mainly use ViT Dosovitskiy et al. 2020), coupled with a single convolutional layer as the decoder.
We conduct experimental analysis address several key statements and questions. For additional and comprehensive experimental results, please refer to Appendix [
Computational cost and memory consumption. In Table memory consumption of previous approaches with our methods. Additionally, Figure 2 presents the average time required by each optimization process to handle a single batch. Loss-based approaches, such as FAMO, DWA, DTP, and UW, exhibit the lowest computational cost and memory usage because they do not require multiple forward passes or backpropagation. In contrast, gradient- based approaches, including Nash-MTL, Aligned-MTL. CAGrad, PCGrad, and GradDrop, incur significantly higher computational costs due to the iterative backpropagation required for each task (AC), even though they have the potential for better performance. These approaches also need to store task-specific gradients, which demands times more memory. Our methods, however, require M (number of groups) forward passes and backward passes, where M < K. As shown in Figure 3c the number of groups changes during optimization on Taskonomy, with our methods maintaining an average M = 1.8 with ViT-L. much lower than K. 11 for the Taskonomy benchmark. Despite the need for multiple forward passes, our methods remain computationally competitive because the majority of the computational load is concentrated on backpropagation and gradient manipulation.
compare the time complexity and
8

---

Published as a conference paper at ICLR 2025
Table 2: Experimental results on the Taskonomy dataset using ViT-L. The best results in each column are shown in bold, while convergence failures are indicated with a dash.
$2.5
Tisk
L1 Dis.
L1 Dot,
Dist.
Doc.
A.,
Siugle Tiak
0,0155
0.010
0,1012
0,1713
0,1630
0.083
0.2160
0.7105
0.1357
0,1700
0,1435
GD
0.0163
0.0167
0.1211
0,1742
0,3715
0.883
0.2333
0.7527
0,1625
0.1837
0,1487
8.55 10:2
GradDrop
0,0168
0.0122
0.1229
0.1744
0.1727
0.091
0.2562
0.7615
0.1656
0.1862
0.1511
10.11 10A0
MGDD
UW
0.0167
0.011
0.1213
0.1728
0.1212
0.089
0.2500
0.471
0.1677
0,1829
0,1538
-7.65 mm
DTTP
0.0169
0.1211
0.1720
0.120
0.089
0:2517
0.241
0. 1610
0.1814
0,1503
-8.16 sm
DAVA
0.0147
D.ISS
0,1210
0.1725
0.1711
0.889
0:3619
0.7486
011613
0.1845
0.1543
-7.92 pm
PCGrad
0.0161
0.0165
0.1206
0.1735
0.3696
0.090
0.2301
0.7540
0.1625
0.1830
0.1483
-7.72 6m
CAfindl
0.0162
0.0165
0.1201
0.1766
0.3651
0.691
0.2565
0,7653
0.1661
0.186]
0.157)
-1805
IMTL
0.0162
0.0165
0.1206
0.1741
0.3710
0.000
0.2256
0.7497
0.1617
0.1832
0.1543
-805 m0um
Aligned-Me MTL
0.0150
0.0155
0.1135
0.1725
0.1660
0.086
0.2513
0.8039
0.1645
0,1500
0.1435
-6.22 1030
Nash MTL
EAMO
0.0157
0.0155
0.1211
0,1730
0.1700
0U090
0.2433
0,7479
0.1610
0,1823
0.1527
7.58 1021
Ours
0.0140
0.0145
0.1195
0.1753
0,1679
0.087
0,2029
0.7166
6.1500
0.1769
0.1419
-1.43 10:39
0.0000
-1.000
-2.334
VIT0
-13300
200C3
2006
(a) (G) with VIT-L
(b) GG with ViT-T
(c) Change of M
(d) Change of B
Figure 3: The averaged grouping results {G),M, on the Taskonomy benehmark are shown for ViT-L in (a) and for ViT-T in (b). (c) illustrates how the number of task groups M, changes during optimization. (d) shows the change in proximal inter-task affinity from DE to C.
This trend becomes more advantageous as the number of tasks increases. Our methods achieve computational cost that falls between loss-based and gradient-basse based methods, while maintaining similar memory consumption to loss-based methods.
Optimization results comparison. We compare the results of multi-task optimization on Taskonomy in Table 2 and on NYUD-v2 and PASCAL-Context in Table 3 Our methods achieve superior multi-task performance across all benchmarks, demonstrating their practical effectiveness. One key observation is that the tasks showing the greatest performance improvements differ between our methods and previous approaches, reflecting the distinct motivations behind cach optimization design. Recent methods like Nash-MTIL, IMTL, and Aligned-MTL. focus on improving edge detection, often at the expense of other tasks. This is due to their emphasis on balancing task losses or gradients, which are affected by unbalanced loss scales. Since edge detection shows the lowest loss scale across the optimization process, these methods prioritize enhancing its performance. Loss-based approaches such as FAMO, DWA, DTP, and UW share a similar motivation but exhibit limited performance compared to gradient-basse compared to GD by optimizing task-specific parameters in a grouping update, which benefits all tasks. Furthermore, our methods achieve stable convergence with many tasks, unlike MGDA Nash-MTI., which struggle to converge in such scenarios on Taskonomy.
methods. In contrast, methods enhance performance across all tasks
Grouping results of selective task group updates. In Figure we illustrate how the grouping strategy evolves during the optimization process. Specifically, in Figures 3a and 33b we show the frequency with which different tasks are grouped together throughout the entire optimization process. Similar tasks, such as depth euclidean (DE) and depth zbuffer (DZ), tend to be grouped more often, whereas depth-related tasks are less frequently grouped with tasks of other types. Overall, the trends in task grouping are consistent across different backbone network sizes. However, one notable observation is that the overall level of grouping increases as the backbone size grows. This suggests that larger models are better at extracting generalizable features across multiple tasks, leading to higher inter-task affinity. In Figure we present the average number of task groups AM throughout the optimization process. During optimization, M tends to increase as proximal inter-task affinity decreases, suggesting that task competition grows more intense over time. Moreover, M grows as the backbone network's s capacity increases, which aligns with the grouping patterns observed for and and We provide an example of proximal inter- task affinity in Figure 3d with all task pairs shown in Figure D.2
different backbone sizes in Figures

---

Published as a conference paper at ICLR 2025
Table 3: We assess the proposed method alongside previous multi-task optimization approaches both NYUD-v2 and PASCAL-Context. The best results are highlighted in bold.
NYUD-2
PASCAL-Cooter
Methol
Semorg miol
Depti RMSE
Nornal
Edge
m %
Semseg mloU
Pumiin rilol.
Saliecyy
Nomal mirr
Bége
Ann
miir
ods
Single Task
39.35
0,561
22.14
39.7
67.96
58.90
53.76
15.65
47.7
GD
38.51
0.641
25.64
53.0
5.54 10.171
67.48
55.45
$1.36
18.41
39.0
9:05 10,095
GrdDDrop
38,42
0.638
25.75
55.0
6.60 10.264
67.18
55.35
$1.32
18.53
39.0
99.5 10.022
MGDA
20.13
0.570
+0.117
38.20
0.631
-399 +-0.96
67.11
51.33
18.44
9.46 +-0113
DTP
38.52
0.633
1-0.114
67.09
$1.35
18,40
9.41 1:0,00
DWA
38.56
+0220 220
67.11
$1.34
-0.46.20.118
PCGuu
38.25
-5.70 +0.033
66.62
CAGrad
38.31
+0.243
66.31
51.28
+00007
Aligned MTL
36.20
0.555
4.12 1-0121
60.7%
17.44
-3.20 +-0.07
IMTT.
37.19
82.23
-7.07
Mash-MITL
36.94
+0.11
75.63
FAMO
36.57
2.46-+0.115
40.12
-2.58
68.14
82.52
4.24
5e11555 detth
Table 5: Ablation stud- ies on Taskonomy ex- ploring the impact of group order and type of grouping.
0863
0001
0.6
0:32
9:23
0.79
0:42
0.45
10:5
44A
0:79
0.54
0:56
0.00
D-t
0:54
0.54
-0492
0.54
0.44
0.46
0.52
0.22
0:04
0.64
6/0
0:42
0.54
0:50
Method
Am (t)
0.23
0.54
0:54
0.50
0.50
Order of Group
Forwad
#0.211
Buckwand
-1,44-20.200
(a)
(b)
(c)
Grouping Methods
Figure 4: The averaged grouping results {G) are shown for NYUD-v2 in (a) and PASCAL-Comext in (b). (c) illustrates how the decaying factor B
Juin! (GD)
-865 20.222
Separate
-2.45 -0.65
Dars
-1.42
influences the stable tracking of proximal inter-task affinity.
we also report the variance in multi-task performance. Our methods demonstrate similar level of variance compared to previous optimization approaches. Additionally, we assess how the order of task group updates and the type of grouping affect perfor- mance and optimization stability in Table [ Using the task groups identified from tracked proximal task affinity, we compare performance when updating in forward order, backward order, and randomly selected order. The results show that the order of task group updates does not significantly impact the final multi-task performance or algorithm stability. When comparing our method to gradient descent (GD), where all tasks are grouped together (M = 1), and a scenario where each task is placed in separate group (M KC), our approach demonstrates better performance and greater stability by grouping tasks with high proximal inter-task affinity.
Stability of optimization. In Tables 2 and 3
The effect of the decay rate on tracking affinity. The affinity decay rate inffuences the tracking of proximal inter-task affinity as shown in Figure 4c] A larger /3 results more dynamic fluctuations in affinity, resulting in more frequent changes in task groupings. In contrast, smaller 3 leads to slower adjustments in task groupings. We observe that the affinity decay rate 3 does not significantly impact multi-task performance, which enhances the method's applicability. In our experiments, we set 3 to 0.001. We anticipate that B can be easily adjusted by monitoring the variation of proximal inter-task affinity. The overall results are shown in Figure D.5
CONCLUSION
We propose a novel approach to mitigate negative transfer by dynamically dividing task groups during optimization. Through empirical experimentation, we observed significant differences in multi-task performance depending on whether task losses are backpropagated collectively or updated sequentially. Building on this insight, we introduce an algorithm that adaptively separates task sets and updates them within single shared architecture during optimization. To facilitate simultaneous tracking of inter-task relations and network optimization, we introduce proximal inter-task affinity, which can be measured throughout the optimization process. Our analysis highlights the profound impact of sequential updates the learning of task-specific representations. Ultimately, our methods substantially enhance multi-task performance compared to previous multi-task optimization.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference computer vision, pp. 6430-6439, 2019.
Alessandro Achille, Giovanni Paolini, Glen Mbeng, and Stefano Soatto. The information complexity of learning tasks, their structure and their distance. Information and Inference: A Journal of the IMA, 10(1):51-72, 2021.
Deblina Bhattacharjee, Tong Zhang, Sabine Süsstrunk, and Mathieu Salzmann. Mult: an end-to-end multitask learning transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pp. 12031-12044 2022.
David Bruggemann, Menelaos Kanakis, Stamatios Georgoulis, and Luc Van Gool. Automated search for resource-efficient branched multi-task networks. arXiv preprint arXiv:2008. 10292, 2020.
Rich Caruana. Multitask learning. Machine learning, 28:41-75, 1997.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning. pp. 794-80488 804.800..PM PMLR, 2018
Zhao Chen, Jiquan Ngiam, Yanping Huang. Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 14-2039-20500 2020.
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G Learned-Min Milleer, and Chuang Gan. Mod-squad: Designing mixtures of experts as modular multi-task learners. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp-
11828-11837, 2023.
Michael Crawshaw. Multi-task learning with deep neural networks: A survey, arXiv preprint arXiv:2009.09796 2020.
Jifeng Dai, Kaiming He, and Jian Sun. Instance- aware semantic segmentation via multi-task network cascades. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3150-3158, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jean-Antoine Désidéri. Multiple-gradiens descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6):113-318, 2012.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture, Proceedings of the IEEE international conference on computer vision, pp. 2650-2658, 2015.
Mark Everingham and John Winn. The pascal visual object classes challenge 2012 (voc2012) development kit. Pattern Anal. Stat. Model. Comput, Learn.. Tech. Rep, 2007:1-45, 2012.
Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang, et al. vit: Mixture-of-exppee vision transformer for efficient multi-task learning with accellaacc co-design. Advances in Neural Information Processing Systems, 35:28441- 28457, 2022.
11

---

Published as a conference paper at ICLR 2025
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXtv:101.088334 2017.
Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34: 27503-27516, 2021.
Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-enn:
Layerwise feature fusing multi-task cnns by neural discriminative dimensionality reduction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3205-3214, 2019.
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task priori- tization for multitask learning. In Proceedings of the European conference on computer vision (ECCV), pp. 270-287, 2018.
Pengsheng Guo, Chen Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference Machine Learning, pp. 3854 3863. PMLR, 2020.
Ziniu Hu, Zhe Zhao, Xinyang Yi, Tiansheng Yao, Lichan Hong. Yizhou Sun, and Ed Chi. Improving multi-task generalization via regularizing spurious correlation. Advances in Neural Information Processing Systems, 35:1 1450-1 1466, 2022.
Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. arXiv preprint 1XXix:2003.02631/ 2021.
Wooseong Jeong and Kuk-Jin Yoon. Quantifying task priority for multi-task optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 363-372, 2024.
Junguang Jiang, Baixu Chen, Junwei Pan, Ximei Wang, Dapeng Liu, Jie Jiang, and Mingsheng Long. Forkmerge: Mitigating negative transfer in auxiliary-task learning. Advances in Neural Information Processing Systems, 36, 2024.
Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share in multi-task feature learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-II), pp. 521-528, 2011.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1482-7491, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X1v:1412.6990, 2014.
Abhishek Kumar and Hal Daume III. Learning task grouping and overlap in multi-task learning. arXiv preprint arXiv:1206.66 2012.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Sequential learning for domain generalization. In European Conference on Computer Vision, pp. 603-619. Springer, 2020.
Bo Liu, Xingchao Liu, Xiaojic Jin, Peter Stone, and Qiang Liu, Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878-18899 2021a.
Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization.
Advances in Neural Information Processing Systems, 36, 2024.
Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. iclr. 2021b.
Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1881- 2019.
12

---

Published as a conference paper at ICLR 2025
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision. pp. 10012-10022, 2021c.
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5334-5343, 2017.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi task learning with multi- -ate mixture-of offeuurr---expertss.... In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 1930-1939, 2018.
Kevis-Kokitsi Maninis, Ilja Radosavovic, and lasonas Kokkinos. Attentive ingle-tasking of multiple tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1851-1860, 2019.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam- Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEFE conference computer vision and puttern recognition, pp. 891-898, 2014.
Goong-Whan
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenattonn, and Neil Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems, 35:9564-9555 9576, 2022.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-tas Multi-task learning as a bargaining game. arXiv preprint arXiv:2202.01017. 10000,.0017,,00000 2022.
Hoang Phan, Lam Tran, Ngoc N Tran, Nhat Ho, Dinh Phung, and Trung Le. Improving multi-task learning via seeking task-based flat regions. arXiv preprint arXiv:2211.1372. 2022.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. Advances in neural information processing systems, 31, 2018
Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent component alignment for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20083 2023.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization, arXiv preprint arXiv:2104.09937, 2021.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rghd images. In Computer FisionFECCy 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pp. 746-760. Springer, 2012.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint VrXiv:1409..15 2014.
Ayan Sinha, Zhao Chen, Vijay Badrinarayanan, and Andrew Rabinovich. Gradient adversarial training of neural networks. 2018.
Samuel Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176 2021.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International conference on machine learning, pp. 9120-9132. PMLR, 2020.
13

---

Published as a conference paper at ICLR 2025
Guolei Sun, Thomas Probst, Danda Pani Paudel, Nikola Popović, Menelaos Kanakis, Jagruti Patel, Dengxin Dai, and Lue Van Gool. Task switching network for multi-task learning. In Proceedings of the IEEE/CVF international conference computer vision, pp. 8291 2021.
Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: deciding what layers to share. arXiv preprint arXiv: 1et.i.9044009999 2019.
Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Mti-net: Multi-scale task interaction networks for multi-task learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part IV 16, pp. 527-543. Springer, 2020.
Wenhai Wang, Enze Xie, Xiang Li. Deng Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 568-578, 2021a.
Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Crossformer: versatile vision transformer hinging on cross-scale attention. arXiv preprint arXiv: 100.2000800054 2021b.
Yongxian Wei, Zixuan Hu, Li Shen, Zhenyi Wang, Yu Li, Chun Yuan, and Dacheng Tao. Task groupings regularization: Data-free meta-learning with heterogeneous pre-trained models. arXiv preprint ar Xrxvv22005. /6560, 2024.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 10211277-12999 2021.
Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction- and-distillation network for simultaneous depth estimation and scene parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 675-684, 2018.
Xiaogang Xu, Hengshuang Zhao, Vibhav Vineet, Ser-Nam Lim, and Antonio Torralba. Mtformer: Multi-task learning via transformer and cross-task reasoning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII, pp- 304-321. Springer, 2022.
Yangyang Xu, Xiangtai Li, Haobo Yuan, Yibo Yang, and Lefei Zhang. Multi-task learning with multi-query transformer for dense prediction. IEEE Transactions on Circuits and Systems for Video Technology, 2023a.
Yangyang Xu, Yibo Yang, and Lefei Zhang. Demt: Deformable mixer transformer for multi-task learning of dense prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 3072-3080, 2023b.
Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00641, 2021.
Hanrong Ye and Dan Xu. Inverted pyramid multi-task transformer for dense scene understanding, In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII, pp. 514-530. Springer, 2022a.
Hanrong Ye and Dan Xu. Invpt: Inverted pyramid multi-task transformer for dense scene understand- ing. arXiv preprint arXiv:2203.0799) 2022b.
Hanrong Ye and Dan Xu. Taskprompter: Spatial-channel multi-task prompting for dense scene
understanding. In The Eleventh International Conference on Learning Representations, 2022c.
Hanrong Ye and Dan Xu. Taskexpert: Dynamically assembling multi-task representations with memorial mixture-of-expee -f-eeersss. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21828-21833 2023.
14

---

Published as a conference paper at ICLR 2025
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3712-3722, 2018.
Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. Mixture of attention heads: Selecting attention heads per token. arXiv preprint arXiv:2210.05144 144, 2022.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 94-108. Springer, 2014.
Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. Pattem-affinitivv propagation across depth, surface normal and semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4106 4115, 2019.
15

---

Published as a conference paper at ICLR 2025
A PROOFS OF THEORETICAL ANALYSIS
Before diving into the thcoretical analysis proof, we'll briefly introduce the basic concepts used in proof to ensure the paper is self-contained.
Definition (Lipschitz continuity). f is X-Lipschitz for any two points u, v in the domain of f. we have following inequality:
If(u) f(u)) & Al|u- ull
A.1 DIFFERENCE BETWEEN INTER-TASK AFFINITY AND PROXIMAL INTER-TASK AFFINITY
Firstly, let's reiterate the definitions of inter-task affinity and proximal inter-task affinity from the main paper.
In a typical SGD process for task i at time step with input 2², the update rule for O2 is as follows: O++1 0 nw, Vo; (i(2', Og θt) where 2f represents the input data and 17 is the learning rate, s/s + is the updated shared parameters with loss L. Then the affinity from task i to k at time step I, denoted as Ai
Le(=", 6x(=",0 Og)
A +/
(10)
For proximal inter-task affinity, let's consider a multi-task network shared by the task set G, with their respective losses defined as LG. For a data sample 2' and a learning rate n,, the gradients of task set G' are updated to the parameters of the network as follows: 0 O vi Ca Or 6%) and
@t1 Of Ve Cx(x', O%, O% for k E G. Then, the proximal inter-task aflinity from group G to task k at time step l is defined as:
Lx(=", G + Lx(2 O 0
B G
(11)
The primary distinction between the two affinities lies in the incorporation of the task set and the update of task-specific parameters (indicated by red letters). Proximal inter-task affinity is an expanded concept that integrates the task set rather than individual tasks as the source task. This difference is evident from the notation, where + k in Equation I10 and + in Equation (II).
The second main difference lies in the update of task-specific parameters. In the inter-task affinity in Equation (10), the denominator includes 0 while the proximal inter-task affinity Equation [1]. it includes e which is a subtle distinction that may not be noticed by readers. These two modifications allow to track proximal inter-task affinity while simultaneously optimizing multi- task networks.
When measuring affinity under the assumption of a convex objective, the proximal inter-task affinity is equal to or greater than the inter-task affinity. This also aligns well with real- real-world scenarios, as proximal inter-task affinity reflects updates to task-specific parameters, as shown below.
(x @+1 +1 a/i
Lx 0 0l+1) Lx O, 0, f
A
O O S
B
(12)
This inequality can also be applied an expanded setting that incorporates task sets. If we expand the concept of inter-task affinity from individual task to task set as Alc then the inequality A/ < B satisfied. If k & G then, A/c- = B holds.
For ease of notation, we use the expanded version of the affinity for multiple tasks throughout the proof, as follows:
A = ( 141 O
(13)
Cx(21, f
This differs from proximal inter-task affinity, as does not consider the update of tasc-spec parameters.
16

---

Published as a conference paper at ICLR 2025
A.2 PROOF OF THEOREM 1
Theorem 1. Let denote the task-specific gradients backpropagated from the loss function Lı. with respect to the parameters S At given time step t. if the inter-task affinity from task group {i, to task k is greater than or equal to the inter-task affinity from group {j. kj to task k. denoted as A k 2 A k Then for sufficiently small learning rate rj L. follows that 9i g 2 9j 9k-
Proof: Let's consider scenario where we update the network parameters θ with task-specific losses f and Ls simultaneously at time step with input 2². Applying the Taylor expansion, we obtain the following:
La O blikk O = Lu(=', 6 0%) + (6 s/i,6
6 0)) + 0 = Cx(2", 0%) - 19k (gi + 9x) + 0
(14)
(15)
where 9, and 9k represent the gradients backpropagated from the losses C, and Cr. respectively, with respect the shared parameters Og. For instance, 9i Vo: (= O O,).
Reorganizing the inequality to align with the format inter-task affinity, we obtain:
Cx(2', 0 sli,k O
A =1 Lx(=', O O
119k (g. + 9k) + 0((2)
(16)
6x(=',00, 0.]
Similar results can be obtained for A
A A/, Cx(2', 0t+1 01)
aljik*
119k (0) 9k) + 0
(17)
Lx(=t, O%, O
(x O 01)
From Aie 2 Ri and by ignoring the O17²) term with a sufficiently small learning rate ?) < we can derive the result:
9: 9k 2 gj gk
(18)
The findings indicate that grouping tasks with positive inter-task affinity exhibits better alignment in task-specific gradients compared to grouping tasks with negative inter-task affinity, thereby validating the grouping strategies employed by our algorithm. Furthermore, we analyze how this alignment in task-specific gradients contributes to reducing the loss of task ie in Theorem
A.3 PROOF OF THEOREM
Let I6e denote the task-specific gradients backpropagated from the loss function La with respect to the parameters Ø) Let e +1 represent the updated parameters after applying the gradients. Assume that for tasks i, j. and k. the inequality 9k 2 9j 99 holds. Then, for
Theorem 2.
sufficiently small learning rate n < 1, the inequality Lx(2", 0t+1 O S Cx(2", 0++1 O) holds.
Proof. Let's consider scenario where update the network parameters Ø with task- losses L. and La simultaneously at time step f with input z'. Let yi denote the gradients backpropagated from the loss L; with respect to the shared parameters O expressed as 9i V. O Ot).
Using the first-order Taylor approximation LA for O,, we obtain:
[a @++1 klikk Of)) La(2', 0', O%) + (6'+1 + O Ve: 0 0
(19)
= Lx(2", o,, Of,) 7(: + gh) + 0
(20)
For task j. we can follow similar process as follows:
Cx(2", Ali 0%) = Lx(z', 0;, +%) - T7 + 9k) gk + 0(7²)
(21)
17

---

Published as a conference paper at ICLR 2025
With sufficiently small learning rate < subtract eq. 21 from eq. 20 then:
(x 6 eli,kr
O),) (x 0 6% - + 9k) 9s 7(g + 9h)
9s
(22)
= 9j) 9h
(23)
which proves the results.
The result indicates that when the gradients from task i align better with those of the reference task compared to task j. the loss on the reference task k tends to lower with updated gradients + compared to 9j 94, especially for sufficiently small learning rates 17.
A.4 PROOF OF THEOREM 3
Theorem 3.
The affinity between kt, - k and k satisfies Al, Z Al
Proof. Let's begin with the definition of inter-task affinity between {i.}} - k and i -> as follows:
Ale-ss Lx(=", alL Ø
(x 0 O0)
A/ =
(24)
Cx(2*, 0 01))
[x (%, O)
When updating and hs simultancously. we can derive the first-order Taylor approximation of L, for O follows:
Lx(2", 0 siikk O fu(z', 0', O1) + (6 s/2,k Voy 0', O1) + 0
(25)
= O 7(9, + 9k) gk + 0
(26)
Similarly, when updating alone, the first-order Taylor approximation of La for O as follows:
10 Lx(2", 1 ( = fx(=", 0 O + (0 Vo) o 0 + O(n²)
(27)
= Cx O' O n9; Ib + 0(72)
(28)
With a sufficiently small learning rate v), the difference between the two inter-task affinities in eq. 24
can be expressed as follows:
Ai Alis = Cx(x", 0
s/i,k 0%
-(1
L 00)
(29)
Lx O 0%)
Cxlet, O 0%)) Cx(2' 0 O
Lu(=', 0+11 O.)) Lx(2',
04+1
slink: 0%)
(30)
Ca(2', Og, 0% 19t Ib (Ca(2", Or,, 0%) n(g: + 9x) gk)
(31)
Cx O%
n|lga|
(32)
Lx(2', O², O1)
(33)
The inequality in eq.
(33 proves that A 2 Ai
When tasks i and l are within the same task group,
access B during the optimization process. If B B 0, the inter-task affinity also satisfies Ai S 0 as Ai S B k
According to Theorem 3 this condition implies A! S O. The proposed algorithm separates these tasks into different groups when , < which justifies our grouping rules.
Conversely, when tasks and j belong separate task groups, we only have access B instead of B B. In this scenario, the proposed algorithm merges these tasks into the same group B Ai > 0. This inequality also implies Ai 2 0, justiffyn the merging of tasks i and k based B during optimization.
18

---

Published as a conference paper at ICLR 2025
A.5 PROOF OF THEOREM 4
Theorem 4 (Convergence Analysis). Given a set of differentiable losses {Li and Lipschitz continuous gradients with constant H > O. If we sequentially update lask groups {G,) i=1 based on inter-task affinity, employing a sufficiently small step size & I min( 2-, inequalities are satisfied for any task group Gm.
S H all = 2..,...
HK. HGGml where |Gml is the number of tasks in G.n. then the following
M
M
S
Σ
LG (2', mmmm
11tmm-11/M O%,,)
(6)
11/M
1+(m- 1)/M
1-1fm-1//M/
1
Il 1|
mm
Gs,Gr
Gm
2
Let's represent the sum of losses of tasks included in Gn as Lo... defined as follows:
La (z', Oltm//M (Lmm/M)- Σ
s|Gm Gm
Lx
6
bmm/MM 0 m
(34)
REG...
where O.tmm//M denotes the shared parameters, while M represents the set of task-specific parameters within G. after updating tasks in Gm.
We begin by expanding the task task-specific loss fG.. in terms of the shared parameter O m/MM and the a(Gm.. task-specific parameters @Gm f- using a quadratic expansion. During this process, the task-specific
parameters O (m-1)m = 0%,... and @Ltm/M_ = @ since the task-specific parameters in G.n Gm
are updated only once from +m 1)/M
to @ttm/M
@
Gm
LG (2*, Oltm/M a(Gm
10+1)) S Lam (2*, 10tmmm1m//MM D%.,.)
(35)
a|Gm
V Vlllim nymLa (2", mm m16m-1
(m+(m-1)/M
10 06
2 (mtmm-))/M
mm
s/Gm-1
(36)
1 2 0 M4fm-1//MMMmm f/6m
(2', 1't, (1mM &Gm G.
0 10
0
(37)
+ Ver. CGm (z @0ttm-11///MMMM 0
0 (0 - 0%
(38)
Vg Lo (2', (24mmm11///)
6m
s(Gm-1
0% 10 0 )
(39)
<Lo (z', O (m-m-1)/MM (%mmm)
(40)
2/Gm
V 1// (z', @ (0
(0tt//m
-
-
(41)
5/Gm-1
1/mm
5mGm-1
10 10m=1
H Gm (0
044mm-1//MM
(42)
2/Gm
+ Ve Lom (2", @4tmm-)
(0+(m-1)//
4|Gm-1
0% (0 (l(m.)
(43)
H ( O'm.,)² )
(44)
where Gml represents the number of tasks in Gm- The inequality holds with the Lipschitz continuity of V L with a constant H.
For the shared parameters of the network, the update rule is as follows:
@.+m/MM (m- (m-1)/M 1)/M e/G..
V/
(-mm I/M La., (2', (m/(m-1)/M O%...)
(45)
2(G.m
e/G.
g'+(m-1)//M
s(Gm-1
19..Gm
(46)
19

---

Published as a conference paper at ICLR 2025
13/M the gradients of the shared parameters with respect loss of tasks in Gm-
where 9s.G.. Similarly, the task-specific parameters of the network, the update rule is as follows:
O = 6.m.. - Lam (=', O .6...) = =0G.. -Whe... Gmmm1
(47)
where G.,,G,. the gradients of the task-specific parameters with respect to the loss tasks in Gu-
If we substitute eq. (46) and eq. 47 into the result of eq. (35 it become follows:
La (z', O 5(Gm
, SLa., (+', O .6%m)
(48)
Gm-
- -nllgamm ( 12+ H
|1
(49)
2
17 Hm
-nllgte.cm 112
|1.is.
(50)
2
We can derive similar results for the loss of task group G,, where the index is not the same as the updating group sequence (i of m). This process follows similarly the one described above. For the step from t + (m 1)/M to t + m/M, the task-specific parameters in G; remain unchanged.
g.+m/M (0.)) S (x, @2t(m-1)/M ,0%.) (z, gg
(51)
dGmm-1
Vettm--n// Lo, (2", 1t 5|Gm-1 0% ( M
(
(0
-
(0 (52)
5/6m-1
22 Mn (z', 2 (m- 16m-1
3|mm-1 0 (0
2/Gm
(53)
SCo, (2t, 1
(l+(m-1)/M 0%,,)
(54)
Vottmmnym Lo, (2', ( (m-1)M AGG
0 (0)
(0 2z
(2
1m)-)
(55)
H 4m/MM
+
m/mm
=|Gm-1
(56)
(La, gg
1)/mM1 (%..)
(57)
(H(m-1)/M 79,G.
(+(m-1)/4 J2
+
72 H²H
l19.,Gm
(m 1|2
(58)
2
(59)
20

---

Published as a conference paper at ICLR 2025
Then the total loss of multiple task groups can be expressed follows:
M
M
LG (z', (z',0 W
LG, (2", 10tmm-1m//// (%,,)
S
(60)
r2H 1tmm-1)/MMp
M
Gs
(m-1)/M
Is.Gm
(+mm-1)MM
2 1l
Σ
Gk
(61)
k=1
H22
17
-nllgi
112+
(62)
2
M
LG (z', , 0//mmm
(63)
Gs
1-mm-)/MM
Gn,Gm
1+ 1+mm-1)/M
1 G.
1/ !
2
IIlss, 11
(64)
M
(z' @ 2c
(65)
a/0m
79s.Gm
(1/M
++ c+(m 1)MM Gg
Dm.Gm
(40m-11/M) "1 112
(66)
The inequality between eq.
(6 and the first term in eq. 64 requires S
H-E G+ HR while
2
the inequality between eq. (62 and the second term in eq. 664 requires 1j MGN Therefore,
the inequality in eq. 63 holds when 17 S. min( 7 HIGH Previous approaches, which handle updates of shared and task-specific parameters independently, failing to capture their interdependence ++ 1+ (111MM/M fluctuates during optimization. ++ (m-1M/ (EM 9r.G,
during optimization. The term, Ia.G..
When the gradients of group Gm align well with the gradients of the other groups {G& (G&)M..see im their dot product yields a positive value, leading to a decrease multi-task losses. However, in practice, the sequential update strategy demonstrates similar level of stability in optimization, which appears to contradict the conventional results. Thus, we assume a correlation between the learning of shared parameters and task-specific parameters, where the learning of task-specific parameters reduces gradient confficts in shared parameters. Under this assumption, the sequential update strategy can guarantee convergence to careto-stationary points. This assumption reasonable, task-specific parameters capture task-specific information, thereby reducing conflicts in the shared parameters across tasks.
A.6 TWO-STEP PROXIMAL INTER-TASK AFFINITY
Before delving into the proof of Theorem 5, let's introduce the concept of two-step proximal inter-task affinity, which extends the notion of proximal inter-task affinity over two update steps.
Definition (Two-Step Proximal Inter-Task Affinity). Consider a multi-taat task network shared by the tasks i, k. with their respective losses denoted as L., L,, Lx. Sequential updates of (Ai), { k)) result parameters being updated from Ø' O++1 O++ and 0 - *+*+* 0 Then, the sji,k
alj two-step proximal inter-task affinity from sequential update (Li), (i, kz) to k at time step t is defined as follows:
B B) = - (1 (1BB)).))) B) BitL
(67)
Ca(e', 0+1 1 1
Cx(2 0++2 0442)
Lx(2', 0442 e2+2)
ski,k
glikk*
= 1 Cx(z,, O O
=
(68)
(x(2
0+ 6
Lx(x1, O
di
A.7 PROOF OF THEOREM 5
Theorem 5. Consider three tasks {i.j.k) k), where task groups are formed with positive inter-task
affinity {i, k) and (j). Assume all losses convex and differentiable, and the change affinity
21

---

Published as a conference paper at ICLR 2025
during a single step from + (m M to t + m/M is negligible. The affinity, which learns all tasks jointly, is denoted as B The affinity for the updating sequence (fi, k}, {i}) is B
+.j.k=k
represented as b.kkjjkk B1+(m-1)/MM while for the sequence ({j), {i, kt), it is represented as j.i.k4
Then, for a sufficiently small learning rate 1 < 1, the following holds:
Bt+(m-11/44 B3+(m-1)/M b.j.k-t
skkjjk
B f 1)/M
<
B1+(m-1/M
jit.
(8)
Proof. We compare the loss after jointly updating three tasks ji k) with the loss after sequentially updating the task sets fi, k) and {j}. To assess the impact of the updating order of task sets, we also conduct the analysis on the reverse order of task set {i}, ki,
(i) Let's begin with the definition of proximal inter-task affinity between {i, kj + and j + k.
taking into account the updates of task-specific parameters as follows:
Cx O
siij ,0% Ot
B + 1j/M =1 k.j.k
(69)
Lx(2*, Lu(2",0)+tm (0
Cx(2', ① Mm/m/ 6 sli,k
B Bf+(m-1)4)) 11/M 1
1% ( A
(70)
Cx(2t,
(m++)MM Cx(*", 6
0 f w1=+11/
B
(71)
(x gmm/MM 6% (
(0+(m+ 6 f/-m/M Lx(=', 41
(72)
fx(2*, @ (ttm/M)
where 6ftm/MM
represents the resulting task-specific parameters of k immediately after updating
the task set {i.j}: This notation is used to differentiate it from the task-specific parameter 6 t obtained after jointly updating all tasks.
The two-step proximal inter-task affinity with the sequence (i, k} and { can be represented as follows:
(1+(m-1)/MM = 1 - (1 (1+(m-1)/M 1)/M). (1 bi
B B/tt//MM
(73)
(x attm/M 6 G
La(2*, 201
O/(mm+1//MMM 6
G4tm/M)
(x O 1 1)/M O (mmm1mmMM)))
Cu(2', 44m/M 6
eli,k
(74)
Cx(2", ( I)/M G
= Cx(2', 0% (t-((m-i (m-1)/M (m 1)/M)
(75)
(m-1mM from eq. 75 assess Our objective is compare B (m-1)/ from eq. 699 with B j.jkkok each update's effect on the final loss. Since both equations share a common denominator, we only need to compare the numerators of each equation. Using the first-order Taylor approximation of
Lx(2*, (0 (m++M @ttm/M in eq. 75) we have:
Lx(2' 1 La(=",0 (1+(m1 Bftm/M)- Ca(z', gtm//M ,
.6, (.tm/MM
(76)
(0
@ttm/MM V
oftm//M Lx(2', @ttm/M ,6 @ftm//M + 0(7?)
(77)
slick
slihh
=24(2", ( @mm//M /6ftm//M, -195
utm/M
4tm/M + O(n2)
g
(78)
22

---

Published as a conference paper at ICLR 2025
The subscript in gradients indicates that it represents the gradients of the shared parameters of the network. Conversely, we will use the subscript ts for gradients of the task specific network in the following derivation. Similarly, Cx(z', grtm/M 3t4m/M in eq. (76 can also be further expanded using Taylor expansion follows:
fttm//M @ [x(2", 10 0 ( C=
(79)
sli,k
x
(tttm/M (m (11m))1MMJ V f. 1y (2 0 (0 1)/M 0 (0 siikk
x+ (m (m--)//
+
(80)
(6ttm/M Oc 1 V + Nm 1/ 0 +m (m 11/M 1 (m- 0(1")
(81)
=Lx(z',
, (0 11/M 0 g (m 11mm-11/MMa- - 19 (t(m-1)/M + Yack
1)/M
(t(mm-1)/M) (1m-11/M
Is.k
(82)
191sck
(m- M) (+ 1)/M + 0
Ftack
(83)
By substituting eq.
76 with the results of eq.
82 and eq. 83
we can obtain the following results:
Lx(z' 215
(0t(mM 6 = Cx(2", gt
gm+(m-1)/a 1)M o (m
(84)
19as
ttm/M Mtm/M - - (m-11/1 1)/M ++ (/ttm-1)/MM
+
Jo:k
Iask
++
1)/M
(85)
19ts;
(m-1)/
-2sck
(4mm-1)/M 0(1²²
(86)
For the scenario where all tasks {i.j,k] are jointly updated, the numerator of eq. expanded as follows:
(69) can also be
Cx(2* 6 MmmMM M La(e", t 11/M wtt.////mttmmmm////// ( slic
(87)
+ 10
65 g (m Cyle (0+(m-1))))))))0))) 1)/M , (m((m-1)/M)
(88)
slijkh
+ (0 Hm/MM 0
1+(m-1)/M
@ La(z', 2
(m 11mm-11pm 11/M 1 (m+(m11)/M,, +
(17²)
(89)
=La(2', (00)),,)((mmm)//) 1)/M
( M)/M
(90)
(41m-11))) 11/M
++ 1)/M
(+(m= (m--1)/MM) (+(m-1)/M
(9
9s.
+ ge.k
go.k
(91)
19t..k
(+1m- (mm1M)/M
g1a.
017²)
(92)
Finally, compare B 1m with B² (m-1)) M)/M by comparing the losses obtained: kkijj>k
i.j.-
Lx(z', (f+(m+)/ e (m with (a(,", sli.j.k 6 6f((m-1)/MM We assume sufficiently slj
LMMM
small learning rate n that allows us ignore terms larger than order two with 1).
Lx(2², (((2,,,tt((mm)))) 6/4m/M) Lx(2', O.m//M O
(93)
aliuik
Hmm/M - 110
(4(m-1)/M + Fo,k
1+(m-1)/MM ga,k
2+(m-
= 1995
Jz,k
(94)
(1(m-1))MM
(1mm-1)/M
19ta.
-
(95)
+ n99.. (+(m (m- M1/M
+
92.j
++ (m-1mmm/ + 9
14/m-1)/MM e+ +m- (1-mm11/// 1)/M
9a.k
(96)
+ 19tt.
(+m M
Its,k
c+ 1)/M
(97)
(--m111/4
(+(m-1)/AM
1+m/M 14m/M
7 (99.
Ja,k
9
Is,k
(98)
~0
(99)
The approximation in eq. 99 holds as we assume inter-task affinity change during a single time step from /. ++ (m M)/M to m/ M is negligible.
23

---

Published as a conference paper at ICLR 2025
(ii) In case update task groups in reverse order the results would differ with (i). Similarly, we begin with the definition of proximal inter-task affinity with reverse order between j - k and {t,j} - as follows:
Cx(x', O.tm/MM 06 (/tm/MM
By 1 Mi/M
a/i
(100)
Lx(=*,
@m 1)/M 1 1)/M
Cx(z', H
1
=1
LE(2*, + M)/M O + (m- 1)/M)
(101)
Lx(=', (m+m+1)/MMMM
6 - (m
Bt+m/M 1
=1
sli,k
(102)
LA (x O
O m/
Cu(2 0x(24,00-(mtt)) (0 66- (m+1)/M
Hiih
(103)
Lx(2*, 1 M s|s
@ft(m-1)//M
The two-step proximal inter-task affinity with the sequence {i} and (i, k} can be represented as follows:
B 1)/M
= =1 -1 (1- (1-BB j=b (1 11/MM (1
(104)
Jiik-b
Cx(2', O ttm/M 0 (
(m (m-1)/M
Cx(x', Sliik
(d+(m+1//M (ft(m+t//M)
sli.k ,6
Cx(2t, 0 +m 11mm-11 11/M (m+t(m11////
Cx(zt, (0(2t,00tmmm 0 13/M)
Ottm/M
(105)
Le(2', HIA
@l+(m)1/MMM ()t()// sliAk
(106)
Cx(2t, C,(2t,0'++ O 11ammm1p I)/M e (m-mmm))
Our objective is to compare 5 (m-(m-) from eq. (69 with (mm)/M from eq. (106) to assess k.j.k-sk each update's 's effect on the final loss. Since both equations share a common denominator, we only
jo.kok
need to compare the numerators of each equation. Using the first-order Taylor approximation of Lx(2', (m++)) ()+( sli.k
in eq. (106), we have:
La Cx 10 (m I)/M , (/ Lx 21 @ttm/M Mtt//MM
(107)
(0
(0 slik
@ttm/M V
Cx 6
tt/m/M o @ftm//M
(108)
a/j
0 gttm/MM VV
La (=', (
@.tm/MM , @ftm//M + 0(1))
(109)
LC'('' Othm/MM 1/
Mt
tm// M + gzck
s+m/MM 1+m/M
19
Izck
(110)
t+m/M
t+m/M + 0(1))
Drek
(111)
Similarly, Cx(2", @ttm/M gttm//M in eq.
107 can also be further expanded using Taylor expan-
sion as follows:
Lu (z (2//0 M @//tm/MM- = La(=', / Lm e (0ttmm1)/AM
(112)
( (2+(mmmmmm))))))) 1)/M 30/ (m- (m M)
(113)
+ (0ttm/M - 0 (m (m- 1)/M V f, 1//L "
@ltm/M o (1-m-11/MM
(114)
=Lu(z', 12al2"/0tt0m (2 1)/M @ (m(m-1)//
(115)
11/m-11/M 14m-1)/M + 0
(116)
24

---

Published as a conference paper at ICLR 2025
By substituting eq. (107) with the results of eq. results:
(115) and eq. (16). obtain the following
Cx(2',
O (m 1//M 6.t(mett/M)) = Cx(2", (
(/
(117)
Ltm/M + Da:k
Satk )tm//M)- Ltm/M - 111s; -tsc
Ltm/M +tm/M
I5;k
(118)
++ (mmm-1)/M 11mm-1)/M + O(T²²
19sg
Dack
(119)
Finally, compare B (m (m-1)/M with B (m M)/M by comparing the losses obtained: joik-
(f+mm+ e 1)/M) with Cx(2", O f ( (m)1M)))) We assume sufficiently Lx(2', alj small learning rate 7 that allows us ignore terms larger than order two with 1).
slij. ,6 (
- Cx(2", O'tm/M .0ttm/MM
(120)
s/i,k
sli.j.
(9aci
t+m/M
Is:k
Mtm/M) 1tm/M
Is:k
19ts;
#tm/M
ttm/M Itack
(121)
195g
14 (m- 1)/M
Dzzk
(-
(122)
+ +n(sst
1-mm-1)/
+ 9s.s
++(m 11m-1 +
Ja,k
1-(m-1)/MM) ++(m
In,k
(m-11)/M 1)/M
(123)
+ 19ro.k
(m M)/M (m-1))
I1s,k
(124)
t+m/M
Do:k
Is:k
1+m/M
n(9,tt
++1m- 1)/M
Je.k
ge.k
(m (m-
1)/M
(125)
191k
M+m/M
Dre:
1+m/M
+ 191.
(+(m-1)/M
-18.k
[+(m-1)/AM
++(m-
(126)
19
t+m/M
L+m/M
+ 191.,
++ (m 1)/M
-ts,k
++ 1)/M
(127)
(128)
The approximation in eq. 127 holds under the assumption that the change in inter-task affinity during a single time step from + (m - 1)/M to m/M is negligible. Since we assume convex loss functions, the magnitude of task-specific gradients Stack would increase after updating the loss of j. which exhibits negative inter-task affinity with is (A, < 0). Therefore, the inequality in eq. 128 holds.
This suggest that grouping tasks with proximal inter-task affinity and subsequently updating these groups sequentially result lower multi-task loss compared to jointly backpropagating all tasks. This disparity arises because the network can discern superior task-specific parameters to accommodate task-specific information during sequential learning.
25

---

Published as a conference paper at ICLR 2025
B ADDITIONAL RELATED WORKS
Task Grouping. Early Multi-Task Learning research is founded on the belief that simultaneous learning of similar tasks within a multi-task framework can enhance overall performance. Kang et al. KKng et al. (2011) identify tasks that contribute improved multi-task performance through the clustering of related tasks with online stochastic gradient descent. This strategy challenges the prevailing assumption that all tasks are inherently interrelated. In parallel, Kumar et al. Kumar & Daume III (2012 present framework for MTL designed to enable selective sharing of information across tasks. They suggests that each task parameter vector can be expressed as a linear combination of finite number of underlying basis tasks. However, these initial methodologies face limitations in their applicability and analysis, particularly in scaling to deep neural networks. Finding out related tasks is more dynamically explored in the transfer learning domain Achille et al. (2019; 2021). They find related tasks by measuring task similarity which can be comparing the similarity of features extracted from the same depth of the independent task's network directly measuring the transfer performance between tasks. Recent research has concentrated on identifying related tasks by directly assessing the relations among them within shared networks. This focus stems from the recognition that the measured inter-task relations in transfer learning fail to fully elucidate the dynamics within the MTL domain Standley et al. (2020): Fifty et al. (202T).
Multi-Task Architectures. Multi-task architectures can be classified based on how much the parameters or features are shared across tasks in the network. The most commonly used structure is shared trunk which consists of common encoder shared by multiple tasks and a dedicated decoder for each task Dai et al. (2016): Ma et al. (2018); [Simonyan Zisserman (2014): Zhang al. (2014). A tree-like architecture, featuring multiple division points for each task group, offers more generalized structure [u et al. (017)); Vandenhende et al. (2019); [Buuggemann et al. (2020); Guo et al. (2020). The cross-talk architecture employs separate symmetrical networks for each task, facilitating feature exchange between layers at the same depth for information sharing between tasks Gao et al. (2019); Xu et al. (2018). The prediction distillation model Eigen & Fergus (2015) [uu et al.
(2018): Vandenhende et al. (2020): Zhang et al. (2019) incorporates cross-task interactions at the end of the shared encoder, while the task switching network (Sun et al. (202T): ) (Sinha et al. (2018); Fernando et al. (2017); Maninis et al. (2019) adjusts network parameters depending on the task.
MTL in Vision Transformers. Recent advancements in multi-task architecture have explored the integration of Vision Transformer Dosovviskiy et al. (2020); Liu et al. (2021c); Wang et al. [ (2021a);
Yang et al. (202I): Xie et al. (2021): Wang al.J (2021b) into MTL. MTFormer Xu et al. (2022)
adopts a shared transformer encoder and decoder with a cross-task attention mechanism. MulT
Bhhatacharjee et al. (2022) leverages shared attention mechanism to capture task dependencies. inspired by the Swin transformer. InvPT Ye & Xu (2022a) emphasizes global spatial position and
multi-task context for dense prediction tasks through multi-scale feature aggregation. The Mixture of Experts (MoE) divides the model into predefined expert groups, dynamically shared or dedicated to specific tasks during the learning phase Riquelme et al. (202T); Zhang et al. (2022): Fan et al. (2022); ;; Mustafa et al. (2022); Chen et al. (2023); Ye & 2003). Task prompter Xu et al. (2023a b); Ye & (202cc) employs spec-ipe tokens to encapsulate task-specific information and utilizes cross-task interactions to enhance multi-task performance.
Multi-Task Domain Generalization. Task grouping based on their relations has also been explored in the field of domain adaptation. In partieular, (Wei et al.) 2024) proposes grouping heterogeneous tasks to regularize them, thereby promoting the learning of more generalized features across domain shifts. (Smith et al. 2021) explores generalization strategies the mini-batch level. (Li al. 2020) addresses diverse domain shift scenarios by incorporating all possible sequential domain learning paths to generalize features for unseen domains. (Shi et al.) (202I) focuses on generalization to unseen domains by reducing dependence on specific domains through inter-domain gradient matching. Additionally, (Hu et al., 2022) analyzes the problem of spurious correlations MTL and proposes regularization methods to mitigate this issue. The effect of gradient conflicts, which are considered the primary cause of negative transfer between tasks, is thoroughly examined in (Jiang et al., 2024). This work also proposes combining task distributions to identify better network parameters from a generalization perspective. The objectives of conventional multi-task optimization and domain generalization differ fundamentally. Conventional multi-task optimization typically assumes that the source and target domains share similar data distributions. In contrast, domain generalization focuses on scenarios involving significant domain shifts. This distinction leads to different approaches in
26

---

Published as a conference paper at ICLR 2025
leveraging task relations to achieve their respective goals. In multi-task optimization, simultaneously updating heterogeneous tasks with conflicting gradients results in suboptimal optimization. On the other hand, domain generalization leverages task sets as a tool to extract generalized features applicable to various unseen domains. Overfitting to similar tasks can harm performance on unseen domains, making it advantageous use heterogeneous tasks form of regularization.
C EXPERIMENTAL DETAILS
We implement our experiments on top of publically available code from Ye & Xu 2022b). We run our experiments on A6000 GPUs.
Datasets. We assess our method on multi-task datasets: NYUD-v2 Silberman et al. (011 PASCAL- Context Mottaghi et al. (204)), and Taskonomy Zamir et (2018). These datasets encompass various vision tasks. NY NYUD-v2 comprises 4 vision tasks: depth estimation, semantic segmentation, surface normal prediction, and edge detection. Meanwhile, PASCAL-Contes includes tasks: semantic segmentation, human parts estimation, saliency estimation, surface normal prediction, and edge detection. In Taskonomy, we use 11 vision tasks: Depth Euclidean (DE), Depth Zbuffer (DZ), Edge Texture (ET), Keypoints 2D (K2), Keypoints 3D (K3), Normal (N), Principal Curvature (C), Reshading (R), Segment Unsup2d (S2), and Segment Unsup2.5D (S2.5).
employed widely used metrics across different tasks. For semantic segmentation, we utilized mean Intersection over Union (mloU). The performance of surface normal prediction output and ground truth. Depth estimation task performance was evaluated using Root Mean Squared Error (RMSE). For saliency estimation and human part segmentation, over Union (mloU). Edge detection performance was assessed using (odsF). For the Taskonomy benchmark, curvature was evaluated using RMSE, while the other tasks were evaluated using LI distance, following the settings in Chen et al. (203)).
Metrics. To assess task performance,
gauged by computing the mean angle distances between the predicted
utilized mean Intersection --eauuee
To evaluate multi-task performance, we adopted the metric proposed in Maninis al. (2019) This metric measures per-task performance by averaging it with respect to the single-task baseline b, as shown in the equation: A. (1/T) 7 (-1)² (Mm, Mo.i//Mo, where 4 if lower value of measure M, means better performance for task i, and 0 otherwise.
Table C.1: Hyperparameters for experiments.
Hyperparameter
Value
Optimizer
Adam Kingma & Ba (2014)
Scheduler
Polynomial Decay
Minibatch size
Number of iterations
40000
Backbone (Transformer)
ViT Dosovitskiy et al. 2020
Learning rate
0,00002
Weight Decay
0.000001
Affinity decay factor 3
0.001
Implementation Details. For experiments, we adopt ViT Dosovitskiy al. (2020) pre-trained on ImageNet-22K| [Dng et al. (2009) as the multi-task encoder. Task-specific decoders merge the multi-scale features extracted by the encoder generate the outputs for each task. The models are trained for 40,000 iterations on both NYUD Silberman et al. (2012) and PASCALLEveringhan & Winn (2012) datasets with batch size We used Adam optimizer with learning rate 2x 10 and xx 10-6 of weight decay with polynomial learning rate schedule. The cross-entropy loss was used for semantic segmentation, human parts estimation, and saliency, edge detection. Surface normal prediction and depth estimation used LI loss. The tasks are weighted equally to ensure fair comparison. For the Taskonomy Benchmark Zamir et al. (2018), we use the dataloader from the open-access code provided by Chen et al. (2023). while maintaining experimental settings identical to those used for NYUD-v2 and PASCAL CASCAL-Coonext... We use the same experimental setup for the other hyperparameters as in previous works Ye & Xu 2002a detailed in Table C.1
27

---

Published as a conference paper at ICLR 2025
D ADDITIONAL EXPERIMENTAL RESULTS
Comparison of optimization results with different backbone sizes. We compare the results of multi-task optimization on Taskonomy across various sizes of vision transformers, as shown in Tables D.1 to D.3 Our method consistently achieves superior performance across all backbone sizes. Unlike previous approaches that focus on learning shared parameters. our optimization strategy enhances the learning of task-specific parameters. This leads to significant performance improvements, especially with smaller backbones, where competition between tasks is more intense due to the limited number of shared parameters. How tasks are grouped, as visualized in fig. D. depends on the backbone size.
Visualization of Proximal Inter-Task Affinity. In Figure D.2, we present the tracked proximal inter-task affinity for each pair of tasks Taskonomy. The changes in proximal inter-task affinity depend on the nature of the task pair, but the backbone size increases, the affinity tends to become more positive. This trend is more noticeable in NYUD-v2 and PASCAL-Context, where there are fewer tasks, as shown in Figures D.3 and D.4 This pattern also aligns with the number of clustered tasks in Figure 3c where the number of groups increases as the backbone size decreases.
visualize the proximal inter-task affinity tracked during optimization with various decay rates 8, ranging from 0.01 to 1e-5 on a logarithmic scale. The decay rate 3 helps stabilize the tracking of proximal inter-task affinity as fluctuates during optimization. Additionally, it aids in understanding inter-task relations over time, independent of input data. For vision transformers, a decay rate of B 0.001 demonstrates stable tracking. In real-world applications, multi-task performance is not highly sensitive to the decay rate B. In Table D..4 we evaluate how 3 impacts multi-task performance on the Taskonomy benchmark. The results demonstrate that the proposed optimization method consistently improves performance across various values, minimizing the need for extensive hyperparameter tuning in practical scenarios.
Effect of the Decay Rate with Visualization. In Figure D.5
The Influence of Task Grouping Strategy. In [.5 we present results comparing different task grouping strategies. These include randomly grouping tasks with a predefined number, grouping het- erogeneous tasks, and grouping homogeneous tasks (our approach). The results clearly demonstrate that grouping homogeneous task sets yields superior performance under the proposed settings. This contrasts with the multi-task domain generalization approach, which groups heterogeneous tasks form of regularization to enhance generalization to unseen domains. This difference arises from the fundamentally distinct objectives of conventional multi-task optimization and domain generalization. Conventional multi-task optimization typically assumes that the source and target domains share similar data distributions, while domain generalization addresses scenarios with significant domain shifts. Consequently, the approaches to leveraging task relations differ to meet these distinct goals. As demonstrated in Theorems and 2 of our work, in multi-task optimization, simultaneously updat- ing heterogeneous tasks with low task affinity leads to suboptimal optimization and higher losses compared to updating similar task sets with high task affinity. This observation aligns with findings from previous multi-task optimization studies referenced in the related works section.
Influence of Batch Size. In Table D.6 we compare our method with mangle-gradien descent (GD) to evaluate its robustness in improving multi-task performance across varying batch sizes. The proposed optimization method consistently demonstrates performance improvements (Am (% t)) of 5.27%, 5.71% %. and 6.13% across different batch sizes. These results highlight the robustness and adaptability of the proposed algorithm across diverse scenarios.
28

---

Published as a conference paper at ICLR 2025
(a) {G]== with ViT-L
(b) (G) with VIT-B (c) (G) with VIT-S (d) (G) with VIT-T
Figure D.I: The averaged grouping results {G) on the Taskonomy benchmark are shown for (a)
ViT-L, (b) ViT-B, (c) ViT-S, and (d) ViT-T, respectively.
Table D.I: Comparison with previous multi-task task optimization approaches Taskonomy with VIT-B.
ED
Task
Li Dst.
Dist
Dit.
L1 Dist. 4
Dis.
Li J
L1
Dist 1
A.. ff>
Single Task
0.0183
0.0166
0.1089
0.1713
0.1630
0.0863
0.2453
0,7522
0.1534
OL1738
0.1530
0.7911
GradDvop
0.0195
0.0066
0.1318
0.1748
0.1735
0,0945
0.3018
0.8060
0.3866
0L1920
0.1607
MGDA
0,0188
0.1285
0.1745
0.7915
0.1800
LYTP
0,0187
00198
0.1283
0.1745
6.1720
0.7912
0.1884
DWA
0,0183
0.1247
0.1745
0.1719
0.7927
0.1857
0.1632
PCGml
0.0155
00188
0.1285
0.1738
0.1703
0.0925
02257
0.70664
111810
0.1822
0.1569
CAGind
0.0193
0.196
0.1306
0.1733
0.0930
0.3871
0.8147
111901
0.1966
0.169
.8.34
0.0180
0.200
0.1387
0.1745
0.1720
0.0934
0.3618
0.7008
0.1858
0.1655
Migmed-MTL.
0.019L
0.0202
0.1263
0.1729
0.1663
0.0984
0.3051
0.8560
0.1936
0.1872
0.1585
-.95
Nash-MTL
0.0175
00152
0.1200
0.1730
0.1663
0.090
0.2696
0.7958
0.1707
0.1839
0.1577
-2.79
FAMO
0.0189
0,0000
0.1285
0.1745
0.1720
0.0934
0:2715
0,7979
1808
0.
0.1540
-7.31
Ozes
0.0167
0.0159
0.1228
0,1739
0,1095
0,0910
0.2344
0,7600
0.1691
0.1836
0,1571
0.64
Table D.2: Comparison with previous multi-tas optimization approaches on Taskonomy with ViT-S.
52
Tiok
L. Dise
Dist
L1 Dist
1.1 Diet.
Diot
E.1 Dist
L. Dist.
1
RMSE
1.1 Dist
L. Dist.
1
L. Dist.
A.. (t>
Single Task
0.0264
0.259
0.1348
0,1740
0.1667
0.0973
0.3481
0.8598
0.1905
0.1857
0.1891
0.0264
0.1574
0.1775
0.1838
0.1038
0.4570
0.9237
0.3475
0.3176
GradDvap
0.0274
00250
0.1009
0.1779
0.1856
0.1042
0.4422
0.9366
0.2106
-13.14
MGDA
0.0261
00248
0.1570
0.1725
0.1832
0.1037
0.4363
0.9302
0.3465
0.3055
0.856
1YTP
0.0263
00273
0.1568
0.1778
0.1831
0.1037
0.4884
0.9307
0.3466
0,3173
0.1899
DWA
0.0264
00271
0.1572
0.1776
0,1834
0.1038
0.8336
0.9215
0.2469
0.2075
0.1856
11.20
PCGmd
0.0271
00274
0.1570
0.1766
0.1784
0.1034
0.4522
0.9343
0.2525
0
01811
-11.78
CAGmnd
0.0260
00232
0.1611
0.1769
0.1706
0.1082
0.4723
0.9557
0.2689
02122
0.192
-15.09
JMTL-L
0,0255
00258
0.1510
0.1748
0.1715
0,1001
0.4339
0.9459
0.2856
0.2036
0.1825
8.90
Aligned MTL
0.0285
0.0200
0.1605
0.1744
0.1711
0,1033
0.4596
1,0022
0.27K3
0.2090
0.1854
15.06
Mash-MTL
0.0255
0.025
0.1510
0.1744
0.1716
0.10RS
0.4330
0.9459
B.2466
0.2033
0.1825
-5.79
FAMO
0,0263
0.0272
0.1573
0.1774
0.1835
0.1035
0.4326
0.920X
0.2464
0.2077
0.185%
-11.25
Oues
0.0225
0.0229
0.1444
0.1762
0.1775
0,0895
0.3953
0,8620
0.2156
0.1997
0.1774
-2.33
Table D.3: Comparison with previous multi-task optimization approaches on Taskonomy with ViT-T.
DE
D2
BO
ET
K2
K3
N
R
52
$2.5
Tiik
LI Dist. 1
Dist.
L1 Dist.
L. Dist. 1
L. Dist.
L.1 Dist
L1 Dist. 1
RMSIE
L. Dist_
L1 Dist.
LI Dist.
Ove (t>
Single Task
0.0289
0.0200
0.1405
0.1724
0.1682
0,0970
0.3837
0.8968
0.2096
0.1901
0.1729
0.0279
0.1004
0.1780
0.1860
0.1041
0.4704
0.9488
0.7613
0.3086
GradDrop
0.0267
00252
0.1630
0.1795
0,1868
0.1052
0.4795
0,9621
0.2697
022118
0.1878
10.68
MGDA
LW
B.0279
00255
0.1604
0.1789
0,1859
0.1043
0.4699
0.948
0.2613
0.2055
0.1914
.921
DTP
0.0278
00258
0.1605
0.1790
0,1859
0.1042
0.4697
0.9438
0.2614
0.2053
0.1915
.427
DWA
0.0279
00255
0.1604
0.1789
0,1859
0,1043
0.4693
0,9489
0.2613
0.2056
0.1913
9.20
POGinal
0,0233
00250
0.1604
0.1769
0.18013
0.1036
0.4230
0.9645
012663
0.2000
0.1866
.0.28
CAGira
0,0300
00504
0.1644
0.1745
0.1721
0.1033
0.4838
0.9518
0128447
0.2143
0.1974
-1212
IMTL
B10276
00252
0.1553
0.1754
0.1743
0.1018
0.4621
0.9403
012623
0.2151
0.1878
.7.55
Aligea--MIT.
0.0096
00318
0.1633
0.1765
0.1757
0.1150
1,0170
0 0.7035
0.300
011877
-13.20
Nash-MT.
0.0276
00233
0.1553
0.1754
0.1743
0.1018
046611
0.9429
0.2623
0.2051
0.187
-7.46
FAMO
0.0279
00285
0.1604
0.1789
0.1859
0.1043
0.4718
0.9488
0.2613
0.2085
0.1913
-9.33
Oxrs
0.0503
00257
0.1526
0,1774
0,1827
0.1019
0.4337
0,9100
0.2400
0.2026
0.1845
-3.67
29

---

Published as a conference paper at ICLR 2025
R--
Figure D.2: Changes in proximal inter-task affinity during the optimization process of ViT-I with Taskonomy benchmark.
30

---

Published as a conference paper at ICLR 2025
C.0
Figure D.2: Changes in proximal inter-task affinity during the optimization process with Taskonomy benchmark.
31

---

Published as a conference paper at ICLR 2025
Cepith
0.004
0,3055
0.000
0.000
2.333
2006
20000
20000
20030
30000
6X002
10000
20020
43002
15:.00
20033
30000
43000
deptth MCd
DOUTTEE
TRrmeaas depth
0.34
0.0005
0.300
0.300
3003
-1.30
adge
SARTSAO
edge desth
sdgg
WTL
WTL
3.000
9.300
0,005
3.000
00C
0.903
30000
20000
40003
200066
2000
40000
10000
23000
20000
40000
12300
23009
Figure D.3: Changes the proximal inter-task affinity during the optimization process of different sizes of VIT with NYUD-v2.
Table D.4: Results Taskonomy with different affinity decay rates 3.
DE
D2
BO
ET
K3
$2.5
Task
L1 Dist.
Li Dist
1.1 Dia. 1
L1 Dist. 1
Doc. J
Dist.
L1 Dist
RMSE L
L.1 Det. J
1.1 Dist. 1
Dist.
A.. (1)
Single Taok
0.0183
0.1366
0.1089
0.1713
0.1630
0.8863
0.2053
0.7522
0.1504
0.173
0.1550
GD
0.0188
80197
0.1283
0.1745
0.1718
0.0933
0.2599
0.7911
0.1799
0.188
0.1631
-6.35
3-0,0001
0.0165
00168
0.1224
0.1739
0.1693
0.0907
0.2301
0.788]
0.1683
0.1831
0.1571
-0.18
9:0.001
0.0167
00189
0.1228
0.1739
0.1695
0.0910
0.2344
0.7600
0.1691
0.1836
0.1571
-0.64
7-0.03
0.0167
0.0711
0.1232
0.1739
0.1698
0,0912
0,2362
0.7623
0.1705
0.1834
0.1576
-1.01
3-0.1
0.0167
0.0111
0.1231
0.1739
0.1695
0.0912
0.2355
0.7631
0.1697
0.1831
0.1575
0.87
Table D.5: Comparison of different grouping strategics on the Taskonomy benchmark.
DE
DZ
ED
ET
K2
X3
N
52
52.5
Task
L. Dist.
Li Doa. 1
Dist.
1.1 Dist, 1
L1 Dia. 1
Dist.
L1 Doa.
RMSE L
Dist.
1.1 Dos.
L. Dist
Ave (i)
Hecerogeneous
B.0122
0.0176
0.1252
0.1241
0.1700
0.9200
0.2475
0.718
0.1745
0,1849
0L1660
-3,10
Kandom (N (222
0.0177
0.0180
0.1299
0.1241
0.1707
0.2663
0/7807
0.1757
0.1871
0.1617
-4.44
Randem (N[Atee))
0.0122
0.0177
0.1550
0.1703
0.3619
0.1886
Randoma (1)
0.0157
0.1706
0.1636
Randoma (M)5)
Randors (M
0.121
E ALGORITHM COMPLEXITY AND COMPUTATIONAL LOAD
We also provide detailed time comparison previous multi-task optimization methods on Taskon- omy. As shown in Table F. our approach effectively optimizes multiple tasks with more efficient training times. Our method converges faster than gradicnt- based approaches, as the primary bottleneck in optimization lies in backpropagation and gradient manipulation.
32

---

Published as a conference paper at ICLR 2025
Scrseeg human part
Safroog 591
S011200 D000
457- MTL
0.000
0.000
MM
MTT.
-0.05
-0.31
20.333
0.735
-1.00
-0.22
-0.05
-1.04
Faumarr parts
put nOTTTTa
FAXTRATI pArto Mrdg
0.00
W.L.
0.300
0.300
-0.11
-1.003
-1 2015
30.30
-2.333
-0 2110
0.313
12306
30004
-0000
20030
30000
20006
20009
40000
10300
23009
normals
5.00
0.000
NTB
WTD.
0.100
8.000
-0.00
-2.303
-0713
-1.30
-0.003
0.724
-0312
V1T
5114
30006
3000
3009
40000
10008
20000
30009
4000
10300
20000
20009
40003
-1.304
10300
20000
33008
4000
0.014
normals samsog
nommars
TUTOO POrTS
0.300
Eesmaals val
nomalss coge
0.004
0.3033
WIH
MET
8.00
0.305
-0.35
0,300
$.200
-3.31
-23322
-0.24
-3.000
-1.003
Eandon
Bevdoon
edge serrcog
edge human parts
edge
edge nommal
0.33
MET VTL
8.32
0.21
0.00
-023
2.31
-0.0
Figure D.4: Changes in proximal inter-task affinity during the optimization process of different sizes of ViT with PASCAL-Context.
Table D.6: Results on Taskonomy with varying batch sizes using ViT-B (batch sizes in brackets).
DE
D2
DO
ET
K2
K3
N
R
S2
$2.5
Tack
Li Dist.
Li Dis
L1 Dist.
LI Dit. L
L1 Dix.
L1 Dist.
L1 Dist
RMSE L
L. Dist
L1 Dist. J
LI Dist.
A, <
Single Tock
0,0183
00186
0. 0.1089
0.1713
0.1630
0.0863
0,2953
075522
0.1504
0,1738
0.153
GDD4)
0.1208
0,0244
0.1323
0.1747
U.3723
0J952
0,2768
0.8214
0.1936
0.1921
0.1677
-1088
Qumsc4)
0,0185
00190
0.1373
0.1741
0.1709
0.0988
03230
0.997
0.1809
0.1888
0.1632
-619
GD
0.0188
0.197
0.1283
0.1745
0.1718
0.0913
02509
0.791
0.1799
6.1885
0.1631
-.35
Ourse8)
0.0167
00189
0.1228
0.1739
0.1695
0.0910
0.2344
0.7600
0.169]
0.1836
0.1571
GDX16)
0.0122
0.1300
0.2260
Oustt6)
00133
0.0144
0.0893
33

---

Published as a conference paper at ICLR 2025
SRITECO degth
100AAg COUTON
SAMTON 200e
death TEMOON
0.03
0.03
6.2L
00091
0.300
0.00
16455
6.300
0.00
0.303
0.00
0.00
0.51
-1.27
-0324
-0.05
-3.315
0.0011
3.23
-133
0.000
0.30
0.00
-0.30
-0.00
-0.00
0.33
00031
DUCTI
0.001
-1300
5000L
10306
30005
3000
0000
1030
2006
30009
8000
10306
20000
30009
10003
11:30
2000
3000
0000
Tarmah udge
deptt
0.3075
0.50
0.000
-0.31
222
325
000L
8.303
0.000
0.000
2333
16:00
3000
Seatinn
Swndon
Figure D.5: Changes in proximal inter-task affinity during the optimization process with different decay rates, B.
Table E.I: Comparison of the average time required by each optimization process to handle a single batch for 11 tasks Taskonomy.
Process (sec)
Forward Pass
Backpropagation
Gradicat Manipulation
Optimizer Step
Clustering Allinity Update
Total
0.198(59.26%)
0.106(31 (9663
UW
0.030S. 82%)
0.198(58.24%)
0.112(32.949)
0.34
DTP
0.03088 70%)
0,199157, 68%)
0.116(33 6266))
0.34
DWA
0.031(9.019)
0. (.198)57.665)
0.115(33.474
0.34
GradDrop
0.0301_13%)
2.05(80.54%)
(.411(1).02))
0.059) 2.3%
2.57
MGDA
0.033(0.086%)
2.06(5.36%)
36.29/91.17%)
0.031(0.0819) (1)
38.42
PCGrad
0.0300.635%)
2,07744.0958
2.57(54.625)
0.031(0.669
4,70
0.08000.576)
2.06(39.399 39%)
3.11(59.44%)
Aligned MTL
0.027(0.86%)
2.07/64. 99%)
1.06(33.20%)
0.030X0..95%
3.19
FAMO
(8.729)
Ours
0.072 (7.13%%
0,576156.728 72%)
0.323(31.82'9)
0.04-1(4.33%
1.02
34


Reviews:
[{"review_id": "xTvGEur7Nf", "reviewer": "Reviewer_KUiJ", "summary": "This paper presents a multi-task learning method that adaptively groups tasks based on proximal inter-task affinity and then sequentially updates each group. It provides a theoretical explanation of the benefits of sequentially updating task groups and the role of incorporating task-specific parameters in reducing conflicts. Experimental results demonstrate the method's superior performance across various benchmarks.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Solid analysis from a theoretical perspective: The paper provides theoretical insights to explain the effectiveness of the proposed method, including (i) the benefits of sequential updating of groups, and (ii) the role of incorporating task-specific parameters in reducing conflicts.\n\n2. The paper is well-written and well-organized.", "weaknesses": "1. Based on proximal inter-task affinity, what principle do we use for task grouping? Discussion on other principles should be included. For example, in [1], they use the Fisher Information Matrix, grouping the most heterogeneous tasks to mitigate conflicts.\n\n2. The motivation for introducing proximal inter-task affinity: After reading Appendix A.1, I still find it difficult to understand the motivation for introducing proximal inter-task affinity.\n\n3. Sequential learning on tasks [1], domains [3,4], and mini-batches [2] for alignment has been studied previously. It would be beneficial to compare the different theoretical perspectives between the proposed method and these prior studies.\n\n[1] Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models, ICML 2024  \n[2] On the Origin of Implicit Regularization in Stochastic Gradient Descent, ICLR 2021  \n[3] Sequential Learning for Domain Generalization, ECCV 2022  \n[4] Gradient Matching for Domain Generalization, ICLR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xKb6KmgTnK", "reviewer": "Reviewer_hDHX", "summary": "The author proposes a method for addressing multi-objective problems by grouping objectives.\nIn the process of considering relations between tasks (objectives), the concept of inter-task affinity was introduced, but additional computation was reduced by focusing on the update of task-specific parameters.\nAdditionally, introducing the concept of affinity to group objectives is the author’s original idea and has been thoroughly analyzed both theoretically and experimentally.", "questions": "As I mentioned in the weaknesses section, I do not interpret the author’s theoretical analysis as indicating that the proposed multiple-objective method can be effectively solved on a global scale.\nHowever, I am not suggesting the necessity of a stringent theoretical foundation.\nI would like to see evidence that the author’s method can consistently provide an optimal point.\nDemonstrating that the proposed method is robust across different batch sizes, numbers of groups, and optimization hyperparameters would effectively support its consistency and reliability.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Experimental Analysis\nIn the field of deep learning, the analysis of batch sequences has not been extensively explored. The author argues that grouping certain objectives in multi-objective problems can be significantly beneficial from a global perspective and has demonstrated this experimentally. In cases where the multi-task learning (MTL) results outperform those of single-task learning (STL), the author’s method consistently achieves the highest performance, which serves as strong empirical support for the validity of the proposed approach.\n\n2. Logical Approach\nThe author’s approach to deriving proxy task affinity is reasonable.\nBy utilizing the loss after task parameter updates, the author effectively reduced additional computations\nAlso, the proposed method has been theoretically proven to remain useful.\nAdditionally, the explanation of the benefits of grouping derived from Theorem 2 is clearly written and easy to understand.", "weaknesses": "In my understanding, some questions remain regarding the actual utility of certain theoretical approaches.\nThe author addresses the utility of multiple objectives in a local context, but optimization in the field of deep learning is far more complex.\nIn practice, grouping the same classes together for optimization in classification tasks may be optimal for the currently updated classes locally; however, it is challenging to reach a global optimum.\nI would like to see additional experimental evaluation on this matter.\nI will leave detailed suggestions in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6TZ5QmJeOZ", "reviewer": "Reviewer_akRc", "summary": "This paper discusses the challenges of multi-task learning, where training multiple tasks together in one architecture can lead to negative transfer or performance degradation. Traditional solutions focus on optimizing shared parameters but neglect task-specific ones. The proposed solution involves grouping tasks selectively and updating them in each batch, along with an algorithm that adapts to determine effective task grouping. The concept of proximal inter-task affinity is introduced to track task relations during optimization. This approach is said to improve multi-task performance by enhancing the learning of task-specific parameters and is shown to outperform previous methods, being scalable to different architectures and task numbers.", "questions": "Please refer to the weaknesss section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper investigates an important problem of multi-task learning. \n2. This paper is well-written and easy to follow.\n3.  Realizing that traditional solutions focus on optimizing shared parameters but neglect task-specific ones, the authors delve into the concept of proximal inter-task affinity, making this paper well-motivated.\n4.  The proposed method is new to me and gives a fresh perspective to further improve the performance of MTL.\n5. This approach is said to improve multi-task performance by enhancing the learning of task-specific parameters and is shown to outperform previous methods, being scalable to different architectures and task numbers.", "weaknesses": "1. The task grouping result in Figure 3c seems out of converge. Will the number of groups further increase as the iteration becomes larger?\n2. Why Nash-MTL is not reported in Table 2?\n3. In the theoretical analysis (Section 4), the authors explain how this sequential update strategy can improve multi-task performance from an optimization standpoint. What about the generalization standpoint? I think the generalization of a model is more important.\n4. In real-world applications, a typical MTL problem may have only a few tasks (e.g., 3). Will the proposed method work in such a circumstance? What is the task grouping result if there are only three tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mEa1c62iT5", "reviewer": "Reviewer_HAqj", "summary": "This paper proposes a novel MTL method in which tasks are dynamically grouped according to the conflict of different tasks during the optimization process and the model is optimized based on these grouped tasks. The method introduces only a limited computation cost and the experiment results show strong performance of their method compared to several existing MTL methods", "questions": "1. Why is the number of groups in Figure 3c not an integer?\n\n2. Line 523, \"Table 4c\" means \"Figure 4c\"?\n\n3. Can you compare the performance of random grouping during optimization?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes a novel optimization method to optimize the multi-task learning process from a new perspective.\n\n2. The method demonstrates promising results on the NYUv2, PASCAL-Context, and Taskonomy datasets.\n\n3. The paper provides a theoretical explanation of the advantages of sequential optimization of task groups and provides an analysis of convergence.", "weaknesses": "1. There are some incorrect statements in the article. For example, in Line 092, “This perspective is not addressed in traditional multi-task optimization, which typically focuses solely on the learning of shared parameters.” is wrong, because the learning of task-specific parameters is considered in IMTL [1].\n\n2. The conclusion in Line 373-374, “This suggests that grouping tasks with proximal inter-task affinity and subsequently updating these groups sequentially result in lower multi-task optimization. sequentially result in lower multi-task loss compared to jointly backpropagating all tasks.” does not seem relevant to the theorem above. Can you give a more detailed explanation?\n\n3. For the conclusion in Line 525-526, “We observe that the affinity decay rate ... within a reasonable range.”, there is a lack of experimental results on the performance of models with different $\\beta$.\n\n[1] Liu, L., Li, Y., Kuang, Z., Xue, J.-H., Chen, Y., Yang, W., Liao, Q., & Zhang, W. Towards Impartial Multi-task Learning. ICLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)