GO4Align: Group Optimization for Multi-Task Alignment
Authors: Jiayi Shen
Cheems Wang
Zehao Xiao
Nanne Van Noord
Marcel Worring
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-task learning
multi-task optimization
task grouping
dense prediction tasks
Primary area: optimization_for_deep_networks

Abstract: This paper proposes **GO4Align**, a multi-task optimization approach that tackles task imbalance by explicitly aligning the optimization across tasks. To achieve this, we design an adaptive group risk minimization strategy, comprising two techniques in implementation: (i) dynamical group assignment, which clusters similar tasks based on task interactions; (ii) risk-guided group indicators, which exploit consistent task correlations with risk information from previous iterations. Comprehensive experimental results on diverse benchmarks demonstrate our method's performance superiority with even lower computational costs.

G04Align: Group Optimization for Multi-Task Alignment
Jiayi Shen', Qi (Cheems) Wang", Zehao Xiao'; Nanne Van Noord', Marcel Worring'
Univeesity of Amsterdam, Amsterdam, the Netherlands Department of Automation, Tsinghua University, Beijing, China
Abstract
This paper proposes GO4Align, multi-tast optimization approach that tackles task imbalance by explicitly aligning the optimization across tasks. To achieve this, we design adaptive group risk minimization strategy, comprising two techniques implementation: (i) dynamical group assignment, which clusters similar tasks based on task interactions: (ii) risk-guided group indicators, which exploit consistent task correlations with risk information from previous iterations. Comprehensive experimental results on diverse benchmarks demonstrate our method's performance superiority with even lower computational costs.
Introduction
Multi-task learning is a promising paradigm for handling several tasks simultaneously us- ing unified architecture. It can achieve data efficiency, improve generalization, and reduce computation costs compared with addressing
RLW (1.0k]
Gradert oviette methods Lons-eresttes methods Tris paper
(1.0%)
(1.0x)
PCGrad [2.23x]
MTL Performance 5m
(1.0x)
GradDrap (2.06x)
bwA (1.02x)
each task individually [1]. Due these ben- efits, there a growing surge of applications with multi-task task learning in several domains, e.g., natural language processing [2-4], computer vi-
MGDA (2.32x] A
CAGrad (2.09X)
IMTL-G (2.1x)
-2
sion [1, 5, 6] and reinforcement learning [7, 8].
FAMO (1.19x)
NashMTL (2.34x)
The crux of multi-task learning is to enable pos- itive transfer among tasks while avoiding nega- tive transfer, which usually exists among irrel- evant tasks [9-11].
-6
GO4Align (1.02x)
1.5
2.5
Relative Training Time
Figure 1: Performance and computational effi- ciency evaluation for MTO methods evaluated on NYUv2. Each method's 's training time is relative to baseline method, which minimizes the sum of spec-fp empirical risks. Left-bottom marks comprehensive optimal results.
Existing Challenges: avoiding the nega- tive transfer, numerous multi-task optimization (MTO) methods [5. 9, 12-14] have emerged and attracted rising attention in recent years. A last- ing concern in MTO is the task imbalance issue. It describes phenomenon where some tasks worse overall performance with larger convergence differences across tasks.
severely under-optimized [1], which lead to
To deal with the task imbalance issue, scaling methods are proposed for MTO. According to differ- ences scaling manipulations, we roughly divide MTO methods into gradient-orienter [12, 15-18] and loss-oriented [5, 6. 19. 20]. The former tends to exhibit impressive results at the expense of higher computational or memory requirements during training time due to the assessment of per-task
Correspondence to: Zehao Xiao <zxiao4ai eziiao4ai@ggaiil.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

gradients [19].2 In contrast, the latter preserves training-time efficiency but usually suffers from unsatisfactory overall performance. As shown in Fig. I, most existing methods cannot simultaneously achieve superior performance and computational efficiency.
Proposed Solution: To improve the overall performance and maintain computational and memory efficiency, we propose Group Optimization for multi-task Alignment (GO4Align), novel and effective loss-oriented method in MTO. As shown in Fig. 2, this work identifies multi-task alignmens as crucial factor in solving task imbalance, which means learning progress across tasks should synchronously achieve superior performance over all tasks. The proposed model dynamically aligns learning progress across tasks by exploiting group-based task interactions for multi-task empirical risk minimization. The rationale behind this is that groupings can implicitly capture task correlations for more effective multi-task alignment and thus help multi-task optimizers benefit. from positive interactions among relevant tasks. The primary contribution is two-fold:
As a new member of the loss-oriented MTO branch, GO4Align recasts the task imbalance issue to a bi-level optimization problem, yielding an adaptive group risk minimization principle for MTO. Such a principle allocates weights over task losses at a group level to achieve learning progress alignment among relevant tasks.
We develop a heuristic optimization pipeline in GO4Align to tractably achieve the principle, involving dynamical group assignment and risk-guided group indicators. The pipeline incor- porates beneficial task interactions into the group assignments and exploits task correlations for multi-task alignment, improving overall multi-task performance.
Experimental results show that our approach can outperform existing state-of -frt----rr baselines in extensive benchmarks. Moreover, does not sacrifice computational efficiency.
2 Preliminary
Notations. This work considers a multi-task problem over an input space X' and a collection of task target spaces ()>T m== where M 2 denotes the number of tasks. A composite dataset for multi-task learning is (I=n; 1/2 x=11 where N is the number of training samples. Let O and 6Th respectively be the shared and task-specific parameters in a given multi-task model, thus have a parametric hypothesis class for the m-th task as f( 0- (' X ymm Then the empirical risk for the m-th task can be written as (m" @Tm) = N ΣN ('((f(z 0", aT) ). y where 1d (.-- : x ym + R. denotes the task task-specific loss function. The ultimate goal of MTO is to achieve superior performance over all tasks.
Scale Empirical Risk Minimization (Seale-ERMM, As preliminary, we recap representative and related strategy in MTO, Scale-ER.M, through the lens of risk minimization. Scale-ERM introduces task-specific weight X 2 0 scale the corresponding empirical risk. For con- A.]' ciseness, this principle is formulated using vector notations. Here, = [A'',A....,AA) e
RM represents M-dimensional vector comprising all task-specific weights. And i()) [E', ,)), (0r,0"))))) 8²).. L.MM
LM (0", (0" A denotes the corresponding vector of empirical risks, where 0 = {0", 0² 0 ... @M) represents all learnable parameters in multi-task backhone network. Thus, we obtain the objective of Scale-ERN as follows:
min X L()) + +(()) 8,2
(1)
where ((A) is a regularization term over task weights, designed to prevent the rapid collapse of these weights to zero, discussed in Kendall et al. [5]. When all task weights are the same in scale, Scale-ERM will degenerate to the most simple strategy in MTO, where each task is treated equally during the joint training.
In Scale-ERM, each task weight controls task-specific learning progress, either by adapting the task-specific weights with loss information (loss-oriented) or by operating task-specific gradients (grantent-oriented)). As previously indicated, there still remains a research gap in MTO to improve multi-task performance without affecting computational efficiency.
In MTO, computational efficiency refers training time efficiency.

---

3 Methodology
In resolving the task imbalance issue effectively and efficiently, we develop GO4Align in this section. Our approach relies on grouping-basee task interactions to align learning progress across tasks. As a new member of the loss-oriented branch, GO4Align holds the advantage of computational efficiency without the requirements of per-task gradients.
Motivation of Multi- Alignment. Empir- ically, we can observe that the overall perfor- mance is worse when there is larger conver- gence difference between tasks. The conver- gence difference is measured by the standard deviation of the task-specific epoch numbers to reach convergence. As shown in Fig. 2, UW [5] over-
converge differnnce:
converge difference:: 26.77
converge diffrrence:
Desth
100
200
202
Epech
Epoch
E2och
underperforms MOMO [14] in terms all MTI. metric Am% (lower is better); while UW has larger convergence difference than FAMO. Intuitively, a larger convergence differ- ence means that per-task training dynamics are more asynchronous, usually leading to worse overall performance.
UW
An 4.06
FAMD Am
4.10
GO4Align Am
6.08
Figure 2: Multi-task alignment and effeets on performance. We visualize relative task perfor- mance curves (lower is better) over training epochs. Better overall performance usually occurs with lower convergence differences. Our method ef- fectively reduces the convergence difference and achieves a better overall performance.
To perform MTO, we consider aligning tasks with group information in the multi-task risk minimization. We first present an adaptive group risk minimization principle for MTO, which targets the alignment of tasks' learning progress from grouping-haad based task interactions. Then, in tractable problem-solving we decompose the whole optimization process into two entangled phases: (i) dynamical group assignment and (ūi) risk-guided group indicators. The pseudo-code of GO4Align is provided in Appendix A.
3.1 Adaptive Group Risk Minimization Principle
Recent advances [21, 22] have explored incorporating multi-task grouping in feature sharing and revealed its benefit of aligning the learning progress through task interactions. Nevertheless, their grouping mechanisms ignore monitoring the learning progress, e.g. failure to capture variations in loss scales among tasks, weakening the effectiveness of multi-task alignment.
As result, we design new grouping mechanism for MTO with task-specific learning dynamics, which directly impacts the convergence behaviors in optimization. This induces the adaptive group risk minimization principle suitable for multi-task alignment. The hypothesis is that the dynamical grouping tends to implicitly exploit task correlations [21] and encourages beneficial task interactions from empirical risk information along the learning progress. Meanwhile, such principle retains computational efficiency as avoids the computations per-task gradients.
Adaptive Group Risk Minimization (AGRM). We first achieve beneficial task interactions by producing task weights with a grouping mechanism, which is adaptive to various loss scales and their time. Then, recast the task imbalance issue with the grouping mechanism
learning dynamics into a bi-level optimization problem: (i) In the lower-level optimization, the model aims to cluster M tasks of interest into K groups This implicitly exploits task correlations, where similar tasks should be clustered into one group, yielding more beneficial task interactions. With group assignments, group weights are designed to conduct learning progress alignment at the group level. The group weights in the multi-task objective are motivated as an extension of the observation that similar tasks benefit greatly from training together through parameter sharing [21]. (ii) In the upper-level optimization, the proposed principle updates model parameters from the grouped empirical risks, which implicitly relies on the lower-level optimized results. We illustrate GO4Align's 's bi-level optimization with grouping-based task interactions in Fig. 3.
Given K as the number of groups, denote the assignment matrix as G, E RRKMKKMM where G. (k,m) m) equals 1 if the k-th group contains the m-th task and 0 otherwise. Note that 9, is updated in the optimization, with t-th indexing iteration step. The group number generally is smaller than the task number, e.g., 1 < K & M. To balance different groups, we place weights over groups Wi [wtt wp w.. "., "
IR where wt 0 is specific to the k-th group the t-th iteration.
3

---

Lower leval opdimizaiion
Lower lavel opiiizaiion
Looer level optiiiztion
Took
Loss
Group
Teak 3
Grou
Grea
Task
sask
Modill
Task 8
paramaters
0g ag
09.0
Upper level optimiatioon
Upper level opinisaiion
Uppar Weni optimizatioon
Figure 3: GO4Align using adaptive group risk minimization the bi-level optimization frame- work. In the lower-level optimization, the model assigns tasks to groups with different group weights, encouraging task interactions and aligning learning progress. Such group information is nested into the upper-level optimization for updating the multi-task model's parameters.
Formally, we formulate the bi-level optimization problem as:
min wi (OL s.t. {we, 9.} = arg min J(w, w.g
where W, and G, reflect adaptive group information in the lower-level optimization. J J(w, G; is the corresponding optimization objective for aligning the learning progress across tasks at the group levels, which is further explained in Sec. 3.2.
To be specific, we perform the bi-level optimization in Eq. (2) according to the following steps. For the 1-th iteration, we first compute the group information w; and G, in the lower-level optimization; and then we update the model's parameter in the upper-level optimization. As a result, we obtain the updated parameter 0,+1, which is used compute task task-specific risk information at the next iteration. In the proposed principle, intra-group tasks share the same scaling weight w 2 0 to prevent similar tasks from inconsistent learning progress, improving knowledge sharing among similar tasks.
Important AGRM is to accommodate the grouping during training dynamically. In practice, the proposed principle is compatible with any gradient-bases optimizer, such as SGD and Adam [23], yielding dynamical training for each task. As new member of the loss-oriented branch, the grouping assignment matrix and group weights in GO4Align can sufficiently utilize the loss information over time to adaptively assign tasks and weight groups.
Unlike prior works on multi-task grouping [21, 22], which require group-specific architectures, our proposed principle focuses on spoup-seecific scaling and adaptively executing grouping operations. Considering the differences in architecture and optimization within grouping mechanisms, we provide two insights: (i) the criteria for grouping tasks should take both learning dynamics and loss scales into consideration so that similar tasks can benefit from each other's 's intermediate feature information and boost performance; (ii) adaptive grouping aligns learning progress across tasks and provides a more effective way for tasoss-task information transfer.
3.2 Dynamical Group Assignment
In solving the optimization problem in Eq. (2), the main challenge lies in the involvement of discrete and continuous variables, which are implicitly entangled in the objective. In detail, the lower-level optimization requires adaptively adjusting the discrete variable 9, and the continuous variable w; for the learning progress alignment such that the model's parameter 0 updates from the lastest grouping information.
Before executing the lower-level optimization, we need to introduce task-specific group indicators (0,) Ind (0) 0:) > "",, TM (0 )) E M In general, this involves the entan- glement of the model's parameters, which is obtained from high-level optimization. These group indicators work for exploiting cross-task correlations along the learning progress and provide group information to enable task interactions in the lower-level optimization. We will further discuss the design of the group indicator in Sec. 3.3.
Intuitively, we conduct the group assignment as a clustering process based on these group indicators 7:: (0.). In this case, each cluster represents group, and the cluster center is set to the group weight. Many clustering algorithms are available to achieve this. this work, we take the K-means clustering algorithm [24-26] as practical clustering implementation. Thus, we specify the optimization

---

objective of the dynamical group assignment as:
min J[w, w/9
gll
(3)
where w Y2 (0:) indicates that the cluster center closely relates the group assignment generalized inverse, especially one-sided right inverse.
matrix. It is worth noting that g;¹ g ("(g The designed dynamical group assignment plays an important role in the lower- level optimization of the proposed AGRM, and it tends to cluster similar tasks into the same group while scattering dissimilar ones in clusters.
By integrating the dynamical group assignment in Eq. (3) and the group indicators into Eq. (2), we provide instantiation for the AGRM's optimization objective:
m (i s.t. (wi, 9e) arg min (0,, w g/gl""
(4)
wgg
Moreover, the dynamic group assignment heuristically clusters tasks from the group indicators, which avoids the exhausted search of appropriate task combinations for performance gains like previous works [21].
3.3 Risk-guided Group Indicators
This subsection discusses the appropriate design of the group indicators for dynamic group assignment introduced in Sec. 3.2.
The misalignment of learning across tasks can usually be attributed to various risk scales and asynchronous learning dynamics among tasks over time. To address this, we take two operations with risk information, scale-balance and smooth-aligument, into consideration and then obtain the risk-guided group indicators by combining them. The role of the group indicators is to use the risk information to explore the relationships among tasks without incurring the expensive computational cost associated with gradients. Compared with other loss-oriented methods, our group indicator can capture the differences the per-task risk scale and fully utilize the learning dynamics over time, yielding better representations of risk information.
Scale-balance. To alleviate the misalignment caused by differences in per-task risk scales, we intro- duce crale-balance, which enlarges the importance of tasks with smaller risks in optimization. Given task-specific risks iteration we normalize them to their average risk for efficient scale balancing. In each iteration, the scale vector for all tasks is denoted as P. = (pt)). (0.), (0.), (0.)] e RA, which can be calculated as:
P. diag( M'
(5)
where diag(-) constructs a diagonal matrix with the elements of the vector placed on the diagonal. L(e,) is scalar to represent the average risk, and [.]a represents the construction of M- dimentional vector whose elements are all equal to the average risk. To avoid the upper-level optimization degenerating into fixed scalar, the gradients of empirical risks in the lower-level optimization are not being computed through them. However, in practice, the learning dynamics over time tend to make the scale vector inconsistent over iterations [12, 16), which is not conducive to aligning learning progress. We therefore introduce smooth--liggmeent to update the scale vector with historical information from previous iterations.
Smooth-alignmen To avoid sudden fluctuations of scale vectors over iterations, we introduce the smoothness vector (6:).:) [g] ( n.h (0.::) R.M. which smooths the updating of the scale vector with previous risk information. Thus, the smoothness vector can deal with asynchronous learning dynamics over time, which helps the model reduce the imbalance of training across tasks. To be specific, we compute the smoothness vector by a normalized exponential moving average as follows:
(0r.) of o (xp
where O denotes the element-wise multiplication and o.:] normalizes the sum of all smoothness elements to be 1. 3 is temperature hyperparameter to control the influence of current risk information.
5

---

Note that when 3 is close to zero, each element in the smoothness vector will degrade to a fixed value 1 which does not capture any historical information group indicators.
Risk-guided Group Indicators. By element-wis multiplying the scale vector in Eq. (5) and the smoothness vector in Eq. (6), we obtain the group indicators with sufficient risk information
(e())) = P:(O,) (11..).
Then, with the group indicators 7 (0.), optimize the dynamical group assignment in Eq. (3) to assign tasks into groups in the lower-level optimization.
For each group indicator, the role of P..) in Eq. (5) and (1(..)) in Eq. (6) differs in optimization: the smoothness vector requires the accumulated loss information from previous iterations, while the scale vector are independent of iterations. Thus, the smoothness vector can iteratively exploit more consistent task correlations to better align learning progress across tasks. The experimental section will show that the risk-guided group indicators empirically boost the proposed adaptive group risk minimization in aligning learning tasks.
4 Related Work
Multi-Task Optimization. Multi-task optimization addresses the task imbalance issue in multi- task learning, where each task usually influences shared network differently. Task-imbalance in MTO [1. 27] refers to imbalanced optimization rather than uneven data distributions in the task space. According to different manipulations in optimization, we roughly divide MTO methods into two branches: (i) grudient-oriented methods, which solve the task balancing problem by fully utilizing the gradient information of the shared network from different tasks. Some studies report impressive performance based on Pareto optimal solutions [18], gradient normalization [27]. gradient conflicting [12], gradient sign Dropout [17], conffict-averse gradient [16]. Nash bargaining solution [13]. However, most gradient manipulation methods usually suffer from high computational cost [19]. (ii) loss-oriented methods, which reweight task-specific losses with the help of inductive biases from the loss space, e.g., using homoscedastic uncertainty [5], task prioritization [28], self- paced learning [29], similar learning paces [6, 14], random loss weight [20]. Although ooientrieteed methods are more computationally efficient, they often underperform gradient-oriented ones in most multi-task benchmarks. This paper tries to trade off the overall performance and computational efficiency.
Recent work [30] weights tasks under the meta-learning setup but has lower training-time efficiency for large-scale systems with high dimensional parameter space, such as deep neural networks, limiting their applications for dense prediction tasks in MTL. The closest method to ours the recent work FAMO [14], which balances task-specific losses by decreasing task loss approximately at an equal rate. However, GO4Align proposes new MTL optimizer that dynamically aligns learning progress across tasks by introducing group-based based task interactions.
Multi-Task Grouping. Multi-task grouping [11, 21, 21J assigns tasks into different groups and trains intra-group tasks together in a shared multi-task network. Previous work [11] first evaluates the transferring gains for Md candidate multi-task networks (M is the number of tasks) and then conducts the brute-force search for the best grouping. Some works follow high-order approximation (HOA) [11] reduce the prohibitive computational cost. However, they also suffer from inaccurate estimations due to non-linear relationships between high-order gains and corresponding pairwise gains [31]. Meanwhile, Yao et al. [32] represents a clustered multi-task learning method, which clusters tasks into several groups by learning the representative tasks. The benefit of multi-task grouping performance gains by training similar tasks together, and this inspires us to capture helpful group information in multi-task optimization. Also, rather than employing different multi-task networks, GO4Align introduce group-based task interactions scaling for multi-task alignment. Moreover, we share high-level idea of task clustering with [33]. However, task clustering in [33] limited to pairwise relationships among tasks; meanwhile, our work allows grouping- based task interactions, thus capturing more complex relationships among tasks.

---

Table 1: Results NYUv2 (3 tasks). The upper and lower tables categorize baseline methods into gradient-oriented and loss-oriented types, respectively. Each experiment is repeated over 3 random seeds, and the mean is reported. The best average result marked in bold. MR and A % the main metrics for overall MTL performance. Metrics with & denote that the lower the better.
Segmentation
Depth
Surface Normal
Method
Angle Dist
Within (
MR J
A Am% %
mIoU
Pix Acc
Abs Err 4
Rel Er
Mean
Modian
11.25
22.5
STL
38.30
63.76
0.6734
0.2780
25.01
19.21
30.14
57.20
69.15
MGDA
30.47
0.6070
0.2555
24.88
19.45
29.18
56.88
69.36
PCGRAD
38.05
64.64
0,5550
0.2325
27.41
22.80
23.86
49,83
63.14
9.00
3.97
GRADDROP
39.39
65.12
0.5455
0.2279
27.48
22.96
23.38
49.44
62.87
7.89
3.58
CAGRAD
39.79
65.49
0.5486
0.2250
26.31
21.58
25.61
52.36
65.58
0.20
IMTL-G
39.35
65.60
0.5426
0.2256
26.02
21.19
26.20
53.13
66.24
4,56
-0,76
NASHMTL
40.13
65.93
0,5261
0.2171
25.26
20.08
28. 28.40
55.47
68.15
2.89
-4,04
LS
39.29
65.33
0,5493
0.2263
28.15
23.96
22.09
47.50
61.08
9.89
5.59
SI
38.45
64.27
0.5354
0.2201
27.60
23.37
22.53
48.57
62.32
8.78
4.39
RLW
37.17
0.5759
0.2410
22.26
60.62
12.22
7.78
DWA
39.11
65.31
0,5510
0.2285
27.61
23.18
24.17
50.18
62.39
8.67
3.57
UW
36.87
63.17
0.5446
0.2260
27.04
22.61
23.54
49.05
63.65
8.33
4.05
FAMO
0.2194
29.21
68.98
GO4Aligg
40.42
65.37
0,5492
0.2167
24.76
18.94
30.54
57.87
69.84
2.11
+6.08
L5
3.00
1.00
1.00
1.00x
MGDA
6.76x
2.05
6.28x
PCGrad
3.30
6.70
CAGrad
2.60
3.33
1.61
3.45
MTs
2.34x
6.08x
2.07
12.69
FAMD
1.13x
1.10
L07
G04Allgn
(a) NYU-v2 (3 tasks)
(b) QM9 (11 tasks)
(c) CityScapes (2 tasks)
(d) CelebA (40 tasks)
Figure 4: Efficiency comparisons on training time. Each method's training time is relative to simple baseline method with Eq. (1), which minimizes the sum of task-specific empirical risks.
Experiments
5.1 Comparisons on MTL Benchmarks
Datasets and Settings. We conduct experiments four benchmarks commonly used in multi-task
optimization literature [6, 13, 14, 16]: NYUv2 [34], CityScapes [35], QM9 [36], and CelebA [37]. For all benchmarks, we follow the training and evaluation protocol in [13, 14].
Baselines. We compare GO4Align with a single- task learning baseline, six gradient-oriented methods, and six loss-oriented methods. Note that single-tasi task learning (STL) trains independent deep network for each task. The gradient-orient
methods include MGDA [18], PCGRAD [12], CA- GRAD [16], IMTL--G [38], GRADDROP [17]. and NASHMTI. [13]. As for the loss-oriented methods, they are Linear scalarization (LSS) Scale-invariant (SI), Dynamic Weight Average (DWA) [6]. Un- certainty Weighting (UW) [5], Random Loss Weighting (RLW) [39], and FAMO [14]. Detailed information about datasets and baselines in Appendix B.
Evaluations. Following previous work [13, 16, 40], we report two MTL metrics that demonstrate the overall performance over various task-specific metrics: (1) A % is the average per-task performance drop relative to STL. We assume there are S metrics for all tasks. M denote the s-th metric value of a multi-task method, while 8 is the corresponding metric value of the STL baseline. Thus, we formulate the average relative performance drop as: Am% E (-1) (M*-B*) where 8* higher values for the s-th metric are better and otherwise. (2) MR 1/ E rank is the average rank of all task-specific metries, where rank(M'* denotes the ranking of the s-th metric value of the model among all comparison methods. Note that in practice the lower Am% and MR. the better overall performance.
7

---

Effectiveness Comparison. We provide perfor- mance comparisons on NYUv2 in Table In this benchmark, our method achieves the best overall MTL performance among both gradient- oriented and loss-oriented methods. We observe that our work is the only one that improves each task's performance relative to the corresponding STL performance. This suggests that grouping- based task interactions adequately alleviate the imbalance of learning progress across tasks.
Table 2: Comparisons on QM9 (11 tasks),
CityScapes (2 tasks) and CelebA (40 tasks).
Detailed results are in Appendix C.
Mithed
QMP
CityScapes
CdebA
MR.
Ami.
MR.
Amh.
MR
Amm 1
MODA
7.73
120.5
10.00
44.14
10:05
14.85
PCORAD
6.09
125.7
6.05
3.17
CAGRAD
7.89
112.5
11.64
5.65
2.48
IMTL-G
5.91
77.2
11:10
4.08
0.54
NASHMITL
3.64
62.0
2.50
4.53
2.54
177.6
5.55
5.09
77.8
7.30
7.30
9.36
203.5
24.38
4.50
1.46
The experimental results on QM9, CityScapes and CelebA are reported in Table GO4Align obtains the lowest Am% on QM9. It also shows comparable performance with FAMO on CityScapes, one possible reason could be that this dataset only contains 2 tasks, which limits the potential of the grouping mechanism in our method. In CelebA, even though our work does not achieve the lowest average performance drop, it outperforms all loss-oriented methods, which further verifies the effectiveness of the proposed method.
DWA
7.64
175.3
21.45
3.20
0.64
5.38
3.23
58.5
1.21
GOVARRR
4.55
52.7
7,00
3.10
0.58
Efficiency Comparison. To show the computational efficiency, in Fig. 4, we report the average training time per epoch over epochs for each method. We choose LS as the relative bascline for training time (cf. RLW [39] and FAMO [14]) as it is commonly used MTL baseline with equal weights for each task, and it does not require additional loss-oriented or gradient-oriented techniques. We note that we run all experiments on an NVIDIA A100 and the code of baseline methods comes from Liu et al. [14] and Navon et al. [13].
As shown in this figure, the proposed method GO4Align, as a new member of the loss-oriented branch, can perform more efficiently than most gradient-oriented methods. Moreover, when the number of tasks scales up from 2 to 40, the reduction in computational cost between our method and other gradient-oriented methods becomes increasingly significant, e.g., NashMTL (2.07) versus Ours (1.01) with 2 tasks, NashMTL (12.49) versus Ours (1.01) with 40 tasks. The main reason is that the training time of gradient-oriented methods is proportional to the number of tasks, but our work can avoid this. More experimental results are provided in Appendix C.
5.2 Ablation Study
The effectiveness and efficiency of our proposed method are shown in Sec. 5.1. Next, we answer the following questions with our ablation study: (1) Can we quantify the contributions of each phrase? (2) Can we disentangle the roles of the group assignment and group weights? (3) Can the proposed AGRM principle seamlessly integrate with existing MTO methods? (4) Are there practical ways to appropriately configure hyperparameters, e.g., group number K?? (5) Why do we choose the K-means clustering in the proposed method?
Contributions of Each Phase. To quantify the contributions of each phase in achieving the pro- posed AGRM principle on NYUv2, we report the detailed performance of our method in each phase. As the grouping is performed by (4), the first two rows in Table 3 are the variants of our method without grouping. and the last row is method with grouping. Table 3 empirically examines the performance gains of the variant with task grouping over without grouping.
Table 3: Effectiveness of each phase in GO4Align lign on NYUv2. / denote whether the component joins the pipeline.
Ea15) Eq/6)
Eqg0)) &.. %
Anrree Si
Am/
4.02
-21.76
13.14
2
14.22
-15.27
2.52
1.16
4.03
.2.37
4.18
-6.88
In detail, compared with the scale vector in Eq. (5), the smoothness vector in Eq. (6) can compromise the performance of the "Normal" and "Depth" tasks, however, scarifying that of "Seg.". Based on the scale and smoothness vectors, the proposed method employs dynamical group assignment in Eq. (4) to exploit the grouping based task interactions, thus well aligning the learning progress of similar tasks "Depth" and "Seg."". We also observe that our method with both phases can improve the task-specific performance relative to STL. This demonstrates that each phase in the method complements each other, resulting in more balanced performance across tasks.
8

---

1.5
135
035
235
(4) o² 0.8,af
Pepth, Normll)
1.00
(9 0.5,0* 0.2, 9
(Nomo:: Seg.
1.09
3og.
8eg
Dept
Deph
120
Normal
Norma
[B] 0.8,0' = 0.1: [Depéh: Seg. Vema]
(6) al 04,0'' A.,,a===0 0.2 f
(Nome: Seg, Depfi
Seg
275
Seg
135
Deph
Deph
Nomre
Noraa
035
0. 0.1; g
(Nomral Seg, Dupti
(h) 0.2, 0.4; (Nommal Sog, Degii
1.03
5eg
Seg
259
Daphi
Normal
Normal
(d) m² 0.8,u² /
Random
( Random
Nomml: Seg, Deprh
Deph
Dopti
Narma
3.25
Narmol
025
(6) GO4Aggn with dynamical group infomatoonea, (3)
GO4Algg dynamical group information (ot,, 9+) Equ(3).
Figure 5: Comparative analysis of the influence of the group assignment matrix and group weights NYUv2. The x-axis in the subplots denotes the epoch, and the intensity of the color indicates the weight value. (a-d) have fixed group weights w = [w", w but various group assignment matrices G. (f-i) have various group weights but fixed group assignment matrix G. (e) is method that dynamically exploits a group assignment matrix and group weights for each iteration. The right side of each method shows relative performance drops on each task and their average one.
Influence Group Assignment Matrix. To explore the influence of the group assignment matrix G, we assume the group number is 2 on NYUv2 and make comparisons with several variants, which have various group assignments with fixed group weights w [0.8, 0.1]. As shown in Fig. 5 (a-c), grouping "See." " and "Depth" outperforms other options. The main reason could be that these two tasks are very similar and far away from the "Normal" task [21]. We observe that variant (d) with random grouping strategy shows lower performance than the fixed grouping options (a) and (c), which further implies the importance of appropriate group assignment.
worth mentioning that the proposed method in (e) without the prior information of the appropriate group assignment also captures such task correlations and each task can get performance gains compared with STL. This demonstrates that group assignment plays important role in exploring task correlations over time in the proposed method.
Influence of Group Weights. To study the influence of group weights, we conduct another visualiza- tion in Fig. 5 (f-i), where we focus on various group weights w with the "optimal" group assignment matrix G [Normal; Seg, Depth}. We observe that with the fixed group assignment matrix, group weights have effects on the extent of compromising among different groups. On the NYUv2 dataset, lower weights for the first groups obtain better overall performance. The variant method (i) with random group weights achieves surprising performance, 1.75%, in terms of the average relative performance drop. (e) shows that method also tends to dynamically weight the first group with a high value. This demonstrates that group weights are necessary to align the learning progress of different groups over time.
Effect of Adaptive Group Risk Minimization Principle. Empirically, the proposed adaptive group risk minimization principle (AGRM) in
Table 4: Comparisons of existing MTO methods
with the proposed AGRM on NYUv2.
Methods
A.., %1
Ango :
%1
A.
See. 3.1 can be seamlessly integrated with isting MTO methods, taking their updated task weights group indicators. As detailed in Ta-
MGDA
13.25
083
1.38
MGDA AGRM
-1.14
-1.89
NASHMILL
-409
-22.01
3.15
-404
NASUMIL AGRM
-7.75
-20.44
3.60
-4.0
ble 4, MTO methods combined with AGRM con- sistently show improved performance. MGDA with adaptive group risk minimization achieves the biggest improvement gap. Moreover, our method still outperforms others. The reason could be that the designed risk- risk-guided group indicators are more suitable for AGRM by balancing risk scales and exploiting historical information from previous iterations.
FAMO
-1.65
-2002
1.20
-4.00
FAMO AGRM
GOAARge
4.03
-20.7
-1.18
-6.88

---

Table 5: Comparisons between different clustering methods in the proposed framework.
Methods
%
A depth
normal %
%,%
Relative runtime 1
SDP-based clustering [41]
-2.97
-18.76
-1.09
-5.44
1.20x
Spectral clustering [42]
-1.78
-18.58
-0,06
-4.56
1.17x
Ours
-4.03
-20.37
-1.18
-6.08
1.02x
Influence of Group Number. In the proposed method, group number K is an important hyper- parameter, especially when we instantiate the clustering process in dynamical group assign- ment with K K-means. In this case, there are many different techniques for choosing the right K. To be visualizable, here we apply the con- ventional elbow method. As shown in Fig. 6. overall performance (lower better) of our method in (a) and (b) drops at 2 and 5, respec- tively, after that both reach a plateau when the group numbers increase. Thus, in this paper set K 2 and 5 for NYUv2 and QM9.
140
Am PiLOO
120
K
(a) NYU-v2 tasks)
(b) QM9 (11 tasks)
Figure Identification of "elbow" points on NYUv2 and QM9. According to the conventional elbow method, we set the group number of the two datasets and 5, respectively.
Effeet of different clustering methods. In our main experiments, we employed standard K-means for instantiation. K-means is widely used clustering approach. investigate the effect of different clustering methods, we evaluate the impact of using alternative clustering algorithms.
Specifically, tested our proposed method NYUv2 by substituting K-means with SDP-based clustering [41] and spectral clustering [42]. As demonstrated in Table 5, these alternative clustering methods also outperform oate-of-thheart approaches (FAMO, -4.10%) particularly by enhancing the performance of each task over STL. Interestingly, our experiments show that the K K-means clustering algorithm outperforms spectral and SDP-based clustering methods.
6 Conclusion
Technical Discussion. This paper focuses on the task imbalance issue in MTO. Previous MTO methods suffer from either intensive computations or non-competitive performance. Our proposed G04Align addresses the issue by aligning learning progress actoss tasks with the help of the AGRM principle. In groblem-solving we present tractable optimization pipeline, which incorporates grouping-based task interactions into the loss scaling of MTO.
Limitation. The main limitation of this work the heuristic configuration of the group numbers. Although the search space is significantly smaller than some grouping multi-task methods [31], it still needs maximum M runs find the best number. Some related techniques [43] automatically set the group number can be added to avoid this limitation in future work.
Broader Impact. This paper is the first consider task grouping in multi-task optimization with deep multi-task models. We propose simple and principled way to fasten multi-task optimization with better training-time efficiency, which has many potential societal impacts, especially in dense prediction tasks. We provide the code for our method to encourage follow-up work."
Acknowledgment
This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
10

---

References
[1] Simon Vandenhende, Stamatios Georgoulis. Wouter Van Gansbeke, Mare Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task lcarning for dense prediction tasks: survey, IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
[2] Shijie Chen, Zhang, and Qiang Yang. Multi-task learning natural language processing: overview. arXiv preprint 0rXiv:2109.09136 2021.
[3] Jonathan Pilault, Christopher Pal, Conditionally adaptive multi-task learning: Improving transfer learning in nip using fewer parameters & less data. In International Conference on Learning Representa- tions, 2020.
[4] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classification. arXiv preprint arXiv: 7704.05442, 2017.
[5] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-tast learning using uncertainty to weigh losses for scene geometry and semantics, In Proceedings of the IEEE conference on computer vision and paitern recognition, pages 1491, 2018.
[6] Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1871-1888 2019.
[7] Lasse Espeholt. Hubert Soyer. Remi Munos, Karen Simonyan, Vlad Mnih. Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407-1416. PMLR, 2018.
[8] Shagun Sodhani, Amy Zhang, and Joelle Pineau, Multi-task reinforcement learning with context- context-based representations. arXiv preprint arXiv:2 2021.
[9] Derrick Xin, Behrooz Ghorbani, Justin Gilmer, Ankush Garg, and Orhan Firat. Do current multi-task optimization methods in deep learning even help? Advances in Neural Information Processing Systems, 35: 15:17797-13609 2022.
[10] Shikun Liu, Stephen James, Andrew J Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:2202.0309) 2022.
[11] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should he learned together in multi-task learning? In International Conference on Machine Learning, pages 9132-9132. PMLR, 2020.
[12] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint vrXiv:2001.06788 2020.
[13] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning hargaining game. arXiv preprint arXiv:220..01017 2022.
[14] Bo Liu, Yihao Feng. Peter Stone. and Qiang Liu. Famo: Fast adaptive multitask optimization. Advonces in Neural Information Processing Systems, 33, 2023.
[15] Dmitry Semushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent component alignment for multi-task lcarning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20083, 2023.
[16] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, (11118878889999 2021.
[17] Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. arXiv preprint arXiv:2010.06808, 2020.
[18] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. arXiv preprint arXiv: 1810:018000045500 2018.
[19] Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and Pawan K Mudigonda. In defense of the unitary scalarization for deep multi-task learning. Advances in Neural Information Processing Systems, 35:12169 15:12169-12183, 2022.

---

[20] Baijiong Lin, Feiyang Ye, Yu Zhang, and Ivor W Tsang. Reasonable effectiveness of random weighting; A litmus test for multi-task learning. arXiv preprint arXiv:21 2021.
[21] Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-tast leaming. Advances in Neural Information Processing Systems, 34:27503-27511 2021.
[22] Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share in multi-task feature learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-I1), pages 521-528, 2011.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[24] K Krishna and M Narasimha Murty. Genetic k-means algorithm. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybemetics)) 29(3)=333-39, 1999.
[25] Trupti M Kodinariya, Prashant R Makwana, al. Review on determining number of cluster in k-means clustering. International Journal, 1(6))90-95, 2013.
[26] Mohiuddin Ahmed, Raihan Scraj, and Syed Mohammed Shamsul Islam. The k-means algorithm: comprehensive survey and performance evaluation. Electronics, 98): 2020.
[27] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normal- ization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pages 793. PMLR, 2018.
[28] Michelle Guo, Albert Haque, De-An Huang. Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference computer vision (ECCV), pages 270-287, 2018.
[29] Keerthiram Murugesan and Jaime Carbonell. Self-paced multitask learning with shared knowledge. arXiv preprint arXiv:1703.0099) 2017.
[30] Cuong Nguyen, Thanh- Toan Do, and Gustavo Carneiro. Task weighting learnnng with trajectory optimisation. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
[31] Xiaozhuang Song, Shun Zheng, Wei Cao, James Yu, and Jiang Bian. Efficient and effective multi-task grouping via meta learning on task combinations. Advances in Neurui Information Prucessing Systems, 35: 37647-37659, 2022.
[32] Yaqiang Yao, Jie Cao. and Huanhuan Chen. Robust task grouping with representative tasks for clustered multi-task learning. In Proceedings of the 25th ACM SIGKDD International conference knowledge discovery & data mining, pages 1408-1417, 2019.
[33] Sebastian Thrun and Joseph O' Sullivan. Clustering learning tasks and the selective cross-task transfer of knowledge. Learning to learn, pages 235-257, Springer. 1998.
[34] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
[35] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3213-3223, 2016.
[36] L. C. Blum and J.-L. Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. J. Am. Chem. Soc., 131:8732, 2009.
[37] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[38] Liyang Liu, Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations, 2020.
[39] Baijiong Lin, Feiyang Ye, and Yu Zhang. A closer look at loss weighting in multi-task learning. arXiv preprint arXiv:2111.10605 2021.
12

---

[40] Kevis-Kokitsi Maninis. Iija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multiple tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1851-1860, 2019.
[41] Mariano Tepper, Anirvan M Sengupta, and Dmitri Chklovskii. The surprising secret identity of the semidefinite relaxation k-means: manifold learning. arXiv preprint arXiv:1706.0000 2017.
[42] Anil Damle, Victor Minden, and Lexing Ying. Simple, direct and efficient multi-way spectral clustering. Information and Inference: A Journal of the IMA, 8(1):181 2033, 2019.
[43] Kristina P Sinaga and Minn-Shen Yang. Unsupervised k-means clustering algorithm. IEEE access, 8: 807 80716-80727, 2020.
[44] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.024 022290903.022... 2019.
[45] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE trunsactions on pattern analysis and machine intelligence, 39 (12):2481-2495, 2017.
[46] lasonas Kokkinos. Ubernet: Training a universal convolutional neural network low-, mid-, and high- level vision using diverse datasets and limited memory. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 6129-6133 2017.
[47] Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5334-5343, 2017.
[48] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi- task learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3994 4003, 2016.
[49] Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille. Nddr-enn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3205-3214, 2019.
[50] Simon Vandenhende, Stamatios Georgoulis, Bert Brabandere, networks: deciding what layers to share. arXiv preprint arXiv:1904.00000 02920, 2019.
Van Gool. Branched multi-task
[51] Pengsheng Guo, Chen-Ye Lee, Daniel Ulbricht. Learning to branch for multi-task learning. In International Conference on Machine Learning, pages 3854-3863 PMLR, 2020.
[52] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems, 33:8728 8740, 2020.
[53] Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Philip S Yu. Learning multiple tasks with multilinear relationship networks. Advances in neural information processing systems, 30, 2017.
[54] Jiayi Shen, Xiantong Zhen, Marcel Worring, and Ling Shao. Variational multi-task learning with gumbel- softmax priors. Advances in Neural Information Processing Systems,
2021.
[55] Jiayi Shen, Xiantong Zhen, Qi Wang, and Marcel Worring. Episodic multi-task learning with heterogeneous neural processes. Advances in Neural Information Processing Systems, (6.75214-75222 2023.
13

---

Appendix
A Algorithm of GO4Align
The pseudo-code of GO4Align is provided in Algorithm 1. For clarity, we also illustrate the optimization process in Fig. 7.
Algorithm Group Optimization for Multi-Task Alignment (GO4Align)
Input: Maximum iteration number T: Batch size Nbzi Learning rate a; Temperature hyperpa- rameter 3; Task number M; Group number (1 < K S M);
2: Initialize model parameters 0.- (0 0g 0% 00
3: for 0 do
4: Randomly sample a batch of training samples:
{(In, Vn Nbz where 2n E & y." E <>m;
5: Compute the empirical risk for each task the 8-th iteration: (0r 0,mh)=
1 ΣA'n 1m (f(xn, 0,") "."))."))
Nbx
// Lower-level optimization for grouping- based task interactions. Compute the scale vector (0,),) in Eq. (5);
6:
7: Compute the smoothness vector (0 in Eq. (6);
8: Obtain the group indicators 7: (01a) with Eq,());
9:
Update the group information W, and 9, with Eq. (3) and the obtained 7 (0:)).
// Upper-level optimization for model parameters.
10: Update the model parameters 0 with Eq. (4): 0.11 + argming w ())
end for
level aotmization
actimiztion
Oesl
0sis
Batche-11
Batchit)
Batchg++1
oplimization
optimication
opimication
Optimization process of the proposed adaptive group risk minimization principle. At each iteration, given the randomly sampled mini-hatch data, we first compute the group information and G in the lower-level optimization and then update the model's parameter 0 in the upper-level optimization.
Figure 7:
B Experimental Set-Up & Implementation Details
B.1 Benchmark Descriptions
NYUv2 [34] an indoor scene dataset consisting of 1449 RGBD images and dense per-pixel la- beling with 13 classes. The learning objectives include 3 different dense prediction tasks: image segmentation, depth prediction, and surface normal prediction based on any scene image.
CityScapes [35] contains 5000 street-view RGBD images with per-pixel annotations. It needs to predict 2 dense prediction tasks: image segmentation and depth prediction.
QM9 [36] a benchmark for group neural networks to predict 11 properties of molecules. consists of > >30K molecules represented graphs annotated with node and edge features. We use 110K molecules from the QM9 example in PyTorch Geometric [44], 10K molecules for validation, and the rest of 10K molecules test set.
14

---

CelebA [37] contains 200K face images of 10K different celebrities, and each face image is provided with 40 facial binary attributes. As the protocol provided in previous work [14], each attribute corresponds to one task. Thus, we consider CelebA as a 40-task MTL problem.
This work follows the same experimental setting used in NashMTL. [13] and FAMO [14], including the dataset partition for training, validation, and testing. The benchmark partition is shown in Table 6. We also note that NYUv2 and Cityscapes do not have validation sets. Following the protocol in [13, 14], we report the test performance averaged over the last ten epochs.
Table 6: Benchmark partition for training, validation, and testing.
Datasets
Total
Training
Validation
Test
NYUv2
1449
795
N/A
654
CityScapes
3475
2975
N/A
500
QM9
~130k
~110k
10k
10k
CelebA
202,599
162,770
19,867
19,962
B.2 Compared Multi-task Learning Baselines
From the gradient manipulation branch, (1) MGDA [18] that finds the equal descent direction for each task; (2) PCGRAD [12] proposes to project each task gradient to the normal plan of that of other tasks and combining them together in the end; (3) CAGRAD [16] optimizes the average loss while explicitly controls the minimum decrease across tasks; (4) IMTL-G [38] finds the update direction with equal projections on task gradients: (5) GRADDROP [17] that randomly dropout certain dimensions of the task gradients based on how much they conflict; (6) NASHMTL [13] formulates MTL as bargaining game and finds the solution to the game that benefits all tasks.
From the loss scaling branch, (1) Linear scalarization (LS) is the sum of empirical risk minimization; (2) Scale-invariant (ST) is invariant to any scalar multiplication of task losses; (3) Dynamic Weight Average (DWA) [6]. a heuristic for adjusting task weights based on rates of loss changes; (4) Uncertainty Weighting (UW) [5] uses task uncertainty proxy adjust task weights; (5) Random Loss Weighting (RLW) [39] that samples task weighting whose bogabobbabilities follow the normal
distribution: (6) FAMO [14] decreases task losses approximately equal rates.
B.3 Neural Architectures & Training Details
For NYUv2 and CityScapes, we follow the training and evaluation protocol in [13], which adds data augmentations during training for all compared methods. We train each method for 200 epochs with initial learning rate of le 4 and reduce the learning rate 5e 5 after 100 epochs. The architecture is Multi-Task Attention Network (MTAN) [6] built upon SegNet [45]. Batch sizes for NYUv2 and CityScapes are set as 2 and 8 respectively. To make fair comparison with previous works [6. 12, 14, 16], we report the test performance averaged over the last 10 epochs.
We follow the protocol in Navon et al. [13] to normalize each task target at the same scale for fairness. We train each method for 300 epochs with a batch size of 120 and search for the best learning rate in {le - 3, 5e 4, 1e -4. 4). We take ReducconPPlaeean [13] the learning rate scheduler to decrease the Ir once the validation overall performance stops improving. The validation set is also used for early stopping.
Following [14], we use a neural network with five convolutional and two fully connected layers as the shared encoder. The decoder of each task is implemented by another fully connected layer. We train the model for 15 epochs with a batch size of 256. We adopt Adam as the optimizer with fixed learning rate of 3. Similar to QM9, we use the validation set for early stopping and hyperparameter selection, such as the number of groups K and the step size of smoothness value 3. We conduct all experiments single NVIDIA A100 GPU.
15

---

Table 7: Detailed results on QM9. Each experiment is repeated over 3 random seeds and the mean is reported. The best average result is marked bold. MR and A Am% are the main metrics for MTL performance.
Method
P
THUMO
"LUMO
(R")
7PVE
Uo
U
G
Gu
MR
Am
MAE 4
STL
0.07
0.18
60.6
53.9
0.50
4.53
58.8
64.2
63.8
66.2
0.07
MGDA
0.22
0.37
126.8
104.6
3.23
5.69
88.4
89.4
89.3
88.0
0.12
7.73
120.5
PCGRAD
0.11
0.29
75.9
88.3
3.94
9.15
116.4
116.8
117.2
114.5
0.11
6.09
125.7
CAGRAD
0.12
0.32
83,5
94,8
3.22
6.93
114.0
114.3
114.5
112.3
0.12
7.09
112.8
IMTL-G
0.14
0.29
102.4
100.1
NASHMTL
0.10
0.25
82.9
81.9
2.43
5.38
74.5
75.0
75.1
74.2
0.09
3.64
62.0
LS
0.11
0.33
73.6
89.7
5.20
14,06
143.4
144.2
144.6
140.3
0.13
8.00
177.6
SI
0.31
0.35
149.8
135.7
1.00
4,51
55.3
55.8
55,8
S5.3
0.11
5.09
77.8
0.34
76.9
15.47
156.3
157.1
157.6
153.0
0.14
9.36
203.8
DWA
0.11
0.33
90.6
13.99
142.3
143.0
139.3
0.13
7.64
175.3
UW
0.39
0.43
166.2
155.8
1.07
4.99
66.4
66.8
66.8
66.2
0.12
6.64
108.0
FAMO
0.15
0.30
94.0
95.2
1.63
4.95
70,82
71.2
71.2
70.3
0.10
4.73
58.5
G0dAlig
0.17
0.35
102.4
119.0
1.22
53.9
54.3
54.3
53.9
0.11
4.55
52.7
Table 8: Detailed results on CityScapes. Each experiment is repeated over 3 random seeds and the mean is reported. The best average result is marked in bold. MR and Am are the main metries for MTL performance.
Segmentation
Depth
Method
MR 1
Am% +
mloU 1
Pix Acc
Abs Err
Rel Err 4
STI.
74.01
93.16
0.0125
27.77
MGDA
68.84
91.54
0.0309
33.50
10.00
44.14
PCGRAD
75.13
93.48
0.0154
42.07
6.25
18.29
CAGRAD
75.16
93.48
0.0141
37.60
5.00
11.64
IMTL-G
75.33
93.49
0.0135
38.41
4.00
11.10
NASHMTL
75.41
93.66
0.0129
35.02
2.50
6.82
LS
70.95
91.73
0.0161
33.83
8.50
14.11
SI
70.95
91.73
0.0161
33.83
8.50
14.11
RLW
74.57
93.41
0.0158
47.79
7.75
24.38
DWA
75.24
93.52
0.0160
44.37
6.00
21.45
UW
72.02
92.85
0.0140
30.13
5.75
5.89
FAMO
74.54
93.29
0.0145
32.59
5.50
8.13
G04Aligh
72.63
93.03
0.0164
27.58
7.00
8.11
Additional Experimental Results
C.1 Detailed Results on QM9 and CityScapes
We provide task-specific performance on QM9 Table 7. The proposed GO4Align obtains the best performance in terms of the average performance drop Am%. And its average rank MR is lower than all loss-oriented methods, which demonstrates the proposed method can get a more balanced performance for each task.
As shown in Table 8, our method achieves competitive performance on CityScapes with other alternatives except for NASHMTL and UW. The main reason can be that there are only two tasks in the datasets, which constrains the effectiveness of the grouping mechanism in our method.
C.2 Training Time Comparisons
In Fig. 4, we show the training time of MTO methods relative to a baseline method (LS). Here we provide real training time (seconds) Table 9, where we compute the average training time epochs. From this table, we observe that loss-oriented methods in general use less time for one epoch than gradient-oriented methods. GO4Align requires the second lowest time cost during training on all four datasets, demonstrating its good computational efficiency.
16

---

Table 9: Detailed training time (seconds) for one epoch of different methods. In addition to LS, our method requires the lowest time cost during training on all datasets.
Method
NYU-v2
Q.999
CityScapes
CelebA
MGDA
199.73
601.52
136.69
902.53
PCGRAD
192.41
352.05
100.21
1486.71
CAGRAD
180.37
296.36
107.84
590.05
IMTL-G
180.67
306.11
113.51
522.22
NASHMTL
201.30
585.37
138.26
2134.75
LS
86.21
88.95
66.63
170.81
FAMO
102.87
100.40
78.70
183.46
G04Align
87.78
93.50
67.30
171.69
C.3 Visualizations on Risk Ratios
To investigate the influence of different scaling methods on the training of tasks, we illustrate the ratios between task sasc-fpec empirical risk and the sum of all empirical risks before and after scaling on NYUv2. In each small figure, the three shadows from top to down represent the ratios "Normal", "Seg." and "Depth", respectively. The x-axis represents epochs ranging from 1 to 200. Note that MGDA and NASHMTL scale task-specific gradients. For direct comparisons, here we provide the scaled loss ratios, which equivalent gradient scaling for the shared network.
As shown Fig. 8, methods have similar ratios over epochs on the unscaled risks but perform differently on the scaled risks. We observe that MGDA have significantly larger ratio the "Normal" task, which means MGDA prefers to optimize the "Normal" task. Compared with related works, our method has more stable ratios of different tasks. The possible reason could be that our method benefits from historical information, which avoids training instability among tasks.
UW (Am 1.05)
MGDA (Am 1.38)
NashMTL (Am -4.04 FAMO (Am -4100)
GO4Align( Am = -6008)
Unscaled
Scaled risk
of
Risk of Seg.
Risk of Depth
Figure 8: Analysis on risk ratios. Compared with other works, the proposed GO4Align shows more stable ratios among tasks epochs, indicating that GO4Align maintain better alignment throughout the training process.
D Additional Related Work
In MTL, we group multi-task learning methods into three categories: multi-task optimization, multi- task grouping, and deep multi-task architecture. The first two categories are most related to this paper and are mentioned in the main paper. To be self-contained, here provide detailed discussions about deep multi-task architecture.
Our method G04Align builds on multi-task grouping and multi-task optimization, inheriting advan- tages from both to balance different tasks joint learning. the following, we discuss each category of mulli-task learning methods and explain how they relate proposed method.
17

---

Deep Multi-Task Architecture. Multi-task architecture design can be roughly categorized into either -ard-paraeeter sharing design [46, 47] or soft-parameter sharing design [6, 48, 49]. The harr-parameter sharing design generally contains a shared encoder and several task-specific decoders. Branching points between the shared encoder and decoders are determined in an ad-hoc way [1, 50], resulting in a suboptimal solution. Some work [47, 51, 52] automatically learns where to share or branch with a network. The soft-parameter sharing design considers all parameters task-specific and instead learns feature-sharing mechanisms to handle the cross-task interactions [48, 533. Some work [54, 55] are probabilistic multi-task learning methods, which introduce latent variables to encode task task-specific information and enable knowledge sharing in the latent spaces. Soft patamearrettr sharing methods usually struggle with model scalability as the model size grows linearly with the number of tasks. GO4Align focuses on the optimization of multi-task learning, which adopts simple hard-parameter sharing architecture as the backhone and subsequently reduces task interference in the gradient space of the shared encoder the loss space.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of this paper claimed in the abstract. Detailed information can be found in the introduction section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide "limitation" subsection in the conclusion section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the details of the experiment the experiment section. The algorithm and Python codes of our method can be found in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: We provide all experimental details in the experiment section. The algo- rithm and Python codes of our method can be found in Appendix A. Details about MTL benchmarks in this paper are provided in Appendix B.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //niss wttpssccccpcccccccccccccccccccct gwwd/gu. Seeniddee/ooeeumiiiii CoodeSubmis asi psssoPoolcyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We follow the standard experimental setup in MTO, where the data splits, hyperparameters, and optimizer set as the same previous works (nashMTL, CAGrad, and FAMO). Detailed information can be found in Appendix B.3.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Following the standard experimental setup, we repeat each experiment over random seeds and report the mean of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the computing resources Appendix B.3. The training timming-time efficiency is provided in Figure 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neett httpar//////....//eee cc/ cc/ppbllc//thieee ctGiideenneennee
Answer: [Yes]
Justification:
We reviewed and followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the potential broader impacts in the conclusion section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The data and models pose no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers that produced the code package and datasets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com aamdaatseess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The provided Python code cannot be used without the authors' permission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "uTU18f6pMh", "reviewer": "Reviewer_FqZ5", "summary": "This paper presents an approach for multi-task learning aiming to reduce interference among tasks by adaptive loss weighting. Instead of task-specific weights as in existing works, the authors proposed to group similar tasks and all tasks in the same group share the same weight. The paper also includes a new derivation of task level weights (referring to group indicator), which contains two components: scale-balance and smooth-alignment. Evaluation was done using four benchmark datasets, three in computer vision and one in chemistry (predicting property of molecules)", "questions": "Refer to weakness", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tThe manuscript is very well written and easy to follow\n•\tThe proposed idea is neat. The proposed task grouping is flexible, which can easily work with other task-weight deriving methods.\n•\tAblation study was done to evaluate the contribution of ingredients in the proposed method.", "weaknesses": "•\tI see that the technical contributions of this work are twofold. One is the derivation of the group indicator (essentially task specific weights) and the other one is to use the same weight to weigh all tasks in a group instead of just using task specific weights. I do not see clear motivation of both. More discussion is needed why both are expected to be better than existing approaches.\n•\tFigure 5 needs more description to make it easier to read. It took me quite some effort and time to figure the x-axis in the plots are epochs and the intensity of the color indicates the weight value (if I am correct).\n•\tThere are results from only one dataset which contains just three tasks. Results from more datasets especially those with larger number of tasks are desired, given the proposed method aims at task grouping. \n•\tThe reported results are not strong. (1) Based on the results from CityScapes (table 2), which has only two tasks (meaning there is not much grouping involved and difference in performance should be due to that in weight calculation), GO4Align is not the best performing one, implying the proposed weight calculation could be inferior to existing one. (2) All numbers in Table 2 are positive, implying models perform worse than those in single-task learning, which defeats the major advantage of MTL, enhancing the model generalizability and performance. \n•\tI do not see an ablation study that compares with grouping to without grouping at all. \n•\tMore details in experiment setup is needed to enhance reproducibility, especially, for each of the experiment how partition of the dataset was done for training, validation, and testing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank # Reviewer FqZ5 for their insightful comments. The following addresses their concerns and provides answers to their questions.*\n\n---\n**1. Motivations.**\n\nThank you for acknowledging the two technical contributions to our paper. We will clarify the motivations for both in the main manuscript as follows:\n\n**Group indicator**:\nThe motivation of the group indicator derivations is to fully utilize the risk information to explore the relationships among tasks, where applying risk information can avoid the computation cost of gradients. Compared with other loss-oriented methods (such as RLW, DWA, UW, and FAMO), our group indicator can capture the differences in the per-task risk scale and fully utilize the learning dynamics over time, yielding better representations of risk information. \n\nEmpirically, **Table 4 in Line 340** shows comparisons with *other task-specific weights (MGDA, NashMTL and FAMO) + the proposed AGRM principle*; our group indicators result in superior performance over other alternatives.\n\n**Group-specific weights versus task-specific weights**:\nThe motivation for group-specific weights is that tasks with close group indicators have similar behaviors on risk, and similar tasks can benefit from training together by sharing parameters as much as possible [21], even weights in the multi-task objective. With the group-specific weights, our method can align tasks with similar risk behaviors during joint training, alleviating the task imbalance issue.  \n\nAs shown in **Table 3 in Line 307**, our work with group-specific weights outperforms its variants with task-specific weights (without Eq.4). This further demonstrates that our group-specific weights can effectively alleviate the under-optimization of some tasks.\n\n---\n**2. Caption of Figure 5.**\n\nThanks for your suggestion. We've taken your advice and added the description to the caption of **Figure 5**.  \n> The x-axis in the subplots denotes the epoch, and the intensity of the color indicates the weight value.\n\n---\n**3. Datasets with a larger number of tasks.**\n\nIn MTL literature, it is common to evaluate datasets with a small number of tasks, such as NYUv2 (3 tasks) and CityScapes (2 tasks) [6, 13, 14, 16]. As task grouping may be beneficial for larger numbers of tasks, we also included QM9 (11 tasks) and CelebA (40 tasks). On these last two benchmarks, our method achieves better overall performance than other loss-oriented methods, with higher training efficiency than gradient-oriented baselines. The overall results are in **Table 2 in Line 276** and details in Appendix **Table 5 & 6**.\n\n---\n**4. Experimental analysis on Table 2.**\n\nWe appreciate the reviewer's careful observations of Table 2. We want to provide clarifications on two points:\n\n**Weight calculations on CityScape**:\nGO4Align achieves comparable performance, securing the third position ($\\Delta m$) on CityScape. It’s critical to note that this benchmark, comprising only two tasks, offers limited grouping options, which constrains the effectiveness of the proposed weight calculation (risk-guided group indicator). Furthermore, in contrast to other weight calculation methods on NYUv2, such as **Table 4 in Line340**, the proposed risk-guided group indicator surpasses alternative weight calculations.\n\n**MTL methods underperform STL in terms of the overall evaluation**:\nThis is a common phenomenon in MTL literature (CAGrad, NashMTL, and FAMO), caused by the task-imbalance issue, where some tasks are under-optimized during joint training. In particular, we note that: \n(i) In Appendix **Table 5 & 6** with detailed task-specific performance, we can observe that most baselines achieve comparable performance over STL on some tasks but significantly sacrifice the performance on the other tasks. \n(ii) MTL still offers advantages such as improved computational efficiency and reduced training time, which significantly contribute to real-world systems.\n\n\n---\n**5. Our model with grouping versus without grouping.**\n\nWe refer the reviewer to the ablations in **Table 3 in Line307**. As the grouping is performed by Eq.(4), the first two rows in Table 3 are the variants of our method without grouping, and the last row is our method with grouping. Table 3 empirically examines the performance gains of task grouping over models without grouping. We will make this more clear in **Line 307**.\n\n\n---\n**6. Detailed experimental setup and open-source plan.**\n\nThis work follows the same experimental setting used in NashMTL [13] and FAMO [14], including the dataset partition for training, validation, and testing. The benchmark partition is attached. We will add this table in Appendix Sec.B.1. we also note that NYUv2 and Cityscapes do not have validation sets. Following the protocol in [13-14], we report the test performance averaged over the last ten epochs. ***Importantly, we will release our code to facilitate MTO research after the final decision.***\n\n| Datasets   | Total    | Training | Validation | Test   |\n|------------|----------|----------|------------|--------|\n| NYUv2      | 1449     | 795      | N/A       | 654    |\n| CityScapes | 3475     | 2975     | N/A       | 500    |\n| QM9        | ~130k    | ~110k    | 10k        | 10k    |\n| CelebA     |  202,599 | 162,770  | 19,867     | 19,962 |\n\n---\n*Thank you for your feedback. We greatly appreciate the time and effort you put into reviewing our work. We have carefully considered your comments and made improvements based on your suggestions. We hope you will reconsider your evaluation of our work. Thank you once again.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s40JAGKEZ8", "reviewer": "Reviewer_AXsM", "summary": "The paper proposes GO4Aligh, a multi-task optimization approach, which targets the task imbalance issue. Specifically, the authors devise two objectives in multi-task optimization: 1) the first is the dynamical group assignment, which can attribute similar tasks to the same cluster and distinguish different ones. 2) The second is the risk-guided group indicators, which include scale-balance and smooth-alignment. These two objectives are incorporated into the AGRM principle for multi-task optimization.", "questions": "- Given the existence of Eq.4, I’m unsure whether Eq.5 or Eq.6 would have a significant impact. How would the performance change if only Eq.4 is included?\n- The paper asserts that GO4Align has even lower computational costs, yet it is 1.02x that of STL. Does this mean that GO4Align is more efficient compared to the baselines (not the STL)?", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "From an overall view, this paper is quite well-structured, with clear writing and an organized layout. The primary motivation, which is to optimize task imbalance, is clearly articulated and valuable.  The methodology is understandable and easy to reproduce. Also, the experiments show that GO4Align can reduce 6.08% metrics with no further training time.", "weaknesses": "While I do not have major concerns about this paper, it’s worth noting that the performance improvement of GO4Align over FAMO is marginal, which may limit its broader impact. The concept of grouping-based task interactions is innovative, but not necessarily groundbreaking. Besides, I have some questions. Please refer to the next part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank # Reviewer AXsM for their insightful comments. The following addresses their concerns and provides answers to their questions.*\n\n\n---\n**1. GO4Align versus FAMO.**\n\nWe thank the reviewer for the comment. GO4Align and FAMO are both strong candidates for handling task imbalance in the MTO literature. However, on NYUv2, our work is the only method that improves each task’s performance relative to the corresponding STL results, especially for the challenging surface normal estimation task. This demonstrates that our method performs better in alleviating task imbalance.\n\n---\n\n**2. Questions.**\n\n**Performance change with only Eq.4**:\nEq.4 is a bi-level optimization process, where lower-level optimization takes the group indicators (Eq.5 and Eq.6) as inputs to update the assignment matrix and group weights. Thus, the optimization process can not be performed individually without specifying the group indicator. More details can be found in **Code Line 6-10** of Algorithm 1.\n\n**Computational cost**:\nEach method’s training time is computed relative to a common MTL baseline (LS, linear scalarization), the same as used in representative MTO works (RLW[39] and FAMO[14]).  LS weights each task equally without extra loss-oriented or gradient-oriented techniques. Moreover, our method has the advantage of computational efficiency over STL. STL takes o(M) the space or time cost, where M is the number of tasks. In contrast, our work uses o(1) space and time due to the shared architecture and loss-oriented mechanism. \n\t\t \t \t \t\t\n---\n*Thanks again for your time and efforts. We hope this has answered your questions. If you have any other questions, we are happy to discuss them and provide further clarification.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7CQba9MFtt", "reviewer": "Reviewer_9PzS", "summary": "This paper proposes a multi-task optimization method designed to address task imbalance by aligning optimization processes across different tasks. To accomplish this, the authors developed an adaptive group risk minimization strategy, which includes two key techniques: (i) dynamic group assignment, clustering similar tasks based on their interactions, and (ii) risk-guided group indicators, leveraging consistent task correlations and risk information from previous iterations. Extensive experimental results across various benchmarks show that the proposed method outperforms others while requiring even lower computational costs.", "questions": "See \"Weaknesses\".", "limitations": "No.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and easy to follow.\n2. The proposed method outperforms several newly proposed MTO algorithms.\n3. Extensive experimental results across various benchmarks show that the proposed method outperforms others while requiring even lower computational costs.", "weaknesses": "1. To help readers better understand the application scenarios of the proposed method, the author can provide examples illustrating the phenomenon of task imbalance in MTO tasks.\n2. In Eq. (4), what is the optimization method for $w$ and $G$? Is it performing standard k-means on $\\gamma$?\n3. In Line 172, why is it possible to invert the $K\\times M$ matrix $G$?\n4. In the introduction of the dataset, there is a lack of explanation regarding the imbalance phenomenon in the relevant learning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank # Reviewer 9PzS for their insightful comments. The following mainly addresses the concerns and answers questions.*\n\n---\n**1. The phenomenon of task imbalance and application scenarios.** \n\nThanks for your kind suggestion. This phenomenon of task imbalance describes that some tasks are severely under-optimized during multi-task training. This can be observed in **Fig. 2 and Table 1**. \n(i) In the left subplot of Fig. 2, task “normal” is under-optimized, leading to convergence earlier than other tasks. \n(ii) In Table 1, most baselines achieve comparable performance to STL on the segmentation and depth estimation tasks but significantly sacrifice the performance on the surface normal estimation task. \nWe note that on NYUv2, our work is the only method that improves each task’s performance relative to the corresponding STL results, especially for the surface normal estimation task. This demonstrates that our method performs better in alleviating task imbalance.\n\nMeanwhile, we would like to specify one typical application scenario of GO4Align. It lies in vision-based autonomous driving, such as Autopilot, which needs to simultaneously handle several tasks (instance segmentation and depth estimation). Significant differences in the improvements across tasks could exist due to diverse scales, different task difficulties, or asynchronous learning dynamics. In this case, the proposed method can be deployed to balance their learning process, particularly without extra training costs. \n\n---\n**2. What is the optimization method for $\\omega$ and $\\mathcal{G}$?**\n\nWe use standard K-means with “Euclidean” distance (*import kmeans_pytorch*) on $\\gamma$ to optimize the $\\omega$ and $\\mathcal{G}$. \n\n---\n**3. Why is it possible to invert the matrix $\\mathcal{G}$?**\n\nSpecial thanks for pointing out this. The assignment matrix $\\mathcal{G}$ is a full-row rank. For simplicity, we formulate its generalized inverse, especially one-sided right inverse, $\\mathcal{G}_{R}^{-1}=\\mathcal{G}^{\\top}(\\mathcal{G}\\mathcal{G}^{\\top})^{-1}.$ We will add a detailed description in **Line 172**.\n\n---\n**4. Imbalance phenomenon in datasets.**  \n\nThe task-imbalance phenomenon in the MTO literature [1, 27] mostly refers to the imbalanced optimization process rather than data distributions in the task space. We will clarify this in the paper, and specifically, we will add the description of task imbalance in **Line 257**.  \n\n---\n*Thank you for your time and efforts. We hope this has addressed your concerns and answered your questions. Please don’t hesitate to reach out if you have further questions or need more information.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S94LdJTwig", "reviewer": "Reviewer_sLBS", "summary": "The paper designs a multi-task optimization method, namely GO4Align, to address task imbalance by aligning optimization processes across tasks. It proposes an adaptive group risk minimization strategy, formulated as a bilevel optimization problem where the lower-level optimization is a task grouping optimization and upper-level optimization is a weighted optimization over task group losses. Then, this method optimizes the problem by alternating two steps: (1) Dynamical Group Assignment where tasks are clustered using a dynamic group assignment process, implemented via K-means clustering, to capture beneficial task interactions. (2) Risk-guided Group Indicators: indicators are designed to balance task risks and align learning progress by combining scale-balance and smooth-alignment operations.\n\nGO4Align is evaluated on benchmarks, including NYUv2, CityScapes, QM9, and CelebA. The results show that it outperforms existing gradient-oriented (MGDA, PCGRAD, CA-GRAD, IMTL-G, GRADDROP, and NASHMTL) and loss-oriented methods (Linear scalarization, Scale-invariant, Dynamic Weight Average, Uncertainty Weighting, Random Loss Weighting, and FAMO.), achieving lower performance drops and better computational efficiency. Lastly, the study explores the contributions of each component of GO4Align, the influence of group assignments, and the role of group weights. It shows that the proposed AGRM principle can integrate with existing MTO methods, further improving their performance.", "questions": "- How is the convergence difference evaluated in Figure 2? How is the $\\Delta m$ metric defined?", "limitations": "This work has discussed its limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes an adaptive group risk minimization strategy to address task imbalance, formulated as a bilevel optimization problem where the lower-level optimization is a task grouping optimization and upper-level optimization is a weighted optimization over task group losses.\n- GO4Align is evaluated on benchmarks, including NYUv2, CityScapes, QM9, and CelebA. The results show that it outperforms 12 existing gradient-oriented and loss-oriented methods, achieving lower performance drops and better computational efficiency.", "weaknesses": "- There are many existing clustering algorithms that can be potentially used for task grouping, such as spectral clustering and SDP-based clustering [1]. What is the rationale for choosing K-means clustering in the method? It would be better to discuss and ablate the clustering algorithms, as it is an important component of the proposed method. \n- Another important component is the choice of the group indicators for clustering tasks. This work uses indicators based on task loss trajectories. How about using gradients and model features as group indicators? Would it be worse than the proposed indicators in Section 3.3? \n- It would be better to explain why the proposed method uses the comparable time as linear scalarization. How does the proposed method scale the number of tasks? It would be better to provide a comparison of the additional computation across each method?\n\n\n[1] Relax, no need to round: integrality of clustering formulations. https://arxiv.org/abs/1408.4045", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank # Reviewer sLBS for their insightful comments. The following mainly addresses the concerns and answers questions.*\n\n---\n**1. Clustering methods.**\n\nIn our main experiments, we employed standard K-means for instantiation. K-means is a widely used clustering approach that worked well in our experiments, so we did not explore this part further (as it was not the main focus). \n\nWe appreciate your suggestion and have conducted additional experiments to evaluate the impact of using alternative clustering algorithms. We thank you for bringing the reference [a]; however, due to the absence of open-source code for [a], we explored another SDP-based clustering method [b].\n\nSpecifically, we conduct the experiments on NYUv2 by substituting the K-means clustering in our method with SDP-based clustering [b] and spectral clustering [c]. As demonstrated in the table below, these alternative clustering methods also outperform state-of-the-art approaches (FAMO, -4.10%), particularly by enhancing the performance of each task over STL. For training efficiency, we implemented K-means using the >kmeans_pytorch package, which offers GPU support.\n\nInterestingly, our experiments show that the K-means clustering algorithm we deployed outperforms both the spectral and SDP-based clustering methods. This is because the hyperparameters for the latter algorithms have yet to be thoroughly investigated. We have presented promising initial results and will include this ablation study and related discussions in the main manuscript.\n\n| Method               | Package   |  GPU support of clustering | Relative runtime $(\\downarrow)$    | $\\mathbf{\\Delta seg.} (\\downarrow)$ | $\\mathbf{\\Delta depth} (\\downarrow)$ | $\\mathbf{\\Delta normal} (\\downarrow)$ | $\\mathbf{\\Delta m} (\\downarrow)$ | \n|----------------------|-----------------|-------------|--------------|---------------|----------|-------------|------------------|\n| Ours w/ SDP-based clustering [b] | sdp_kmeans     |  NO          | 1,20X             | -2,97       | -18,76       | -1,09         | -5,44    | \n| Ours w/ Spectral clustering [c]  | sklearn       | NO          | 1,17X             |-1,78       | -18,58       | -0,06         | -4,56    | \n| Ours w/ K-means clustering (in the paper)               | kmeans_pytorch | **YES**         | **1,02X**             |**-4,03**       | **-20,37**       | **-1,18**         | **-6,08**    | \n\n---\n**2. Choice of the group indicators.**\n\nIt is also plausible to use the gradient information as clustering indicators. As shown in **Table 4**, we tried to replace the proposed risk-guided group indicator with gradient-guided task weights(MGDA and NashMTL). Our work outperforms these alternatives. The main reason could be that our group indicator yields better representations of learning information by capturing the differences in the per-task risk scale and exploring the learning dynamics over time. \n\nMoreover, from the perspective of training efficiency, the task-specific gradients need to back-propagate the shared architecture for $M$ times, where $M$ is the number of tasks, which increases computational cost linearly with the number of tasks. \n\n---\n**3. Comparable time with linear scalarization and scale-up number of tasks.**\n\nWe follow representative MTO works (RLW[39] and FAMO[14]) in choosing linear scalarization (LS) as the relative baseline for training time. This is a good choice because LS is a common MTL baseline, where each task has equal weights without extra loss-oriented or gradient-oriented techniques.\n\nAs shown in **Fig. 4**, when the number of tasks scales up from 2 to 40, the advantages of reducing computation cost with our method become increasingly significant compared to other gradient-oriented methods, e.g., NashMTL=2.07 versus Ours=1.01 with 2 tasks, NashMTL1=2.49 versus Ours=1.01 with 40 tasks. We will add this discussion in **Line 299**.\n\n---\n**4. Questions.**\n\n**Convergence difference in Figure 2**:\nWe evaluate the convergence difference by the standard deviation of the task-specific epoch numbers to reach convergence. We will polish this description in **Line 90**.\n\n**Definition of $\\Delta m$**:\n$\\Delta m$ is the average per-task performance drop relative to STL as representative MTO works (NashMTL and FAMO). We will polish this in **Line 268**.\n\n---\n**Reference**:\n\n[a] Awasthi, P., Bandeira, A. S., Charikar, M., Krishnaswamy, R., Villar, S., & Ward, R. (2015, January). Relax, no need to round: Integrality of clustering formulations. In Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science (pp. 191-200).\n\n[b] Tepper, M., Sengupta, A. M., & Chklovskii, D. (2017). The surprising secret identity of the semidefinite relaxation of k-means: manifold learning. arXiv preprint arXiv:1706.06028.\n\n[c] Damle, A., Minden, V., & Ying, L. (2019). Simple, direct and efficient multi-way spectral clustering. Information and Inference: A Journal of the IMA, 8(1), 181-203.\n\n---\n*Thank you for your time and efforts. We hope our experimental results and clarifications have addressed your concerns. Please don’t hesitate to reach out if you have further questions or need more information.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and Area Chair:

We sincerely thank you for your time, insightful suggestions, and valuable comments. We are encouraged by your support and positive reviews of our work:
+ Neat/innovative idea of adaptive task grouping in MTO **[# Reviewers FqZ5/AXsM]** ;
+ Manuscript well-written/structured and easy to follow **[# Reviewers FqZ5/AXsM/9PzS]**;
+ SOTA performance with no further training time **[# Reviewers AXsM/9PzS]**;
+ Extensive experiments with sufficient ablation studies **[# Reviewers FqZ5/9PzS/sLBS]**.

To address your concerns, we have been working diligently on improving the paper in several aspects. We summarize the major changes that we will update in the main manuscript:

+ Add additional conceptual analysis to clarify motivations for our method **[# Reviewer FqZ5]** and the computation efficiency over STL **[# Reviewer AXsM]**;

+ Add additional experimental analysis for the main results and the ablations, such as "Experimental analysis on Table 2" **[# Reviewer FqZ5]**, "Our model with grouping versus without grouping" **[# Reviewer FqZ5]**, "GO4Align versus FAMO" **[# Reviewer AXsM]**, "The phenomenon of task imbalance and application scenarios" **[# Reviewer 9PzS]**, "Choice of the group indicators" **[# Reviewer sLBS]**, and "Comparable time with linear scalarization and scale-up number of tasks" **[# Reviewer sLBS]**;

+ Provide additional experimental results to investigate the effects of different clustering methods. For more details and discussions, please refer to the response to **[# Reviewer sLBS]**;
| Method               | Package   |  GPU support of clustering | Relative runtime $(\downarrow)$    | $\mathbf{\Delta seg.} (\downarrow)$ | $\mathbf{\Delta depth} (\downarrow)$ | $\mathbf{\Delta normal} (\downarrow)$ | $\mathbf{\Delta m} (\downarrow)$ | 
|----------------------|-----------------|-------------|--------------|---------------|----------|-------------|------------------|
| Ours w/ SDP-based clustering | sdp_kmeans     |  NO          | 1,20X             | -2,97       | -18,76       | -1,09         | -5,44    | 
| Ours w/ Spectral clustering  | sklearn       | NO          | 1,17X             |-1,78       | -18,58       | -0,06         | -4,56    | 
| Ours w/ K-means clustering (in the paper)               | kmeans_pytorch | **YES**         | **1,02X**             |**-4,03**       | **-20,37**       | **-1,18**         | **-6,08**    | 
|

+ Clarify descriptions of figures **[# Reviewer FqZ5]**, experimental setups **[# Reviewers FqZ5/9PzS]**, generalized inverse of the assignment matrix **[# Reviewer 9PzS]**, and evaluation metrics **[# Reviewer sLBS]**.

Once again, we thank all reviewers and area chairs. Your efforts and suggestions helped us improve this paper. Please see the **reviewer-specific response** for more detailed information.

Meta-review: This paper proposes a multi-task optimization approach to alleviate the task imbalance problem. It adopts an adaptive group risk minimization strategy that consists of two parts: dynamical group assignment  and risk-guided group indicators. Dynamical group assignment aims at grouping similar tasks into the same cluster, while risk-guided group indicators are to balance task risks by utilizing consistent task correlations. Ablation experiments well demonstrate the effectiveness of each part. After rebuttual, most of issues are solved 
and all reviewers give positive scores.  Reviewers acknowledge its good-orgianization and sufficient experiments. Desipte for Reviewer FqZ5 the explanation why grouping helps being still not clear,  his/her other comments have been largely addressed and the rating is also increased. The authors are encourated to further enhance this part in the final version. Based on these, I recommend accepting this manuscript.
Decision: Accept (poster)