Auto-Lambda: Disentangling Dynamic Task Relationships
Authors: Shikun Liu
Stephen James
Andrew Davison
Edward Johns
Venue: Accepted by TMLR ()

Abstract: Understanding the structure of multiple related tasks allows for multi-task learning to improve the generalisation ability of one or all of them. However, it usually requires training each pairwise combination of tasks together in order to capture task relationships, at an extremely high computational cost. In this work, we learn task relationships via an automated weighting framework, named Auto-Lambda. Unlike previous methods where task relationships are assumed to be fixed, i.e., task should either be trained together or not trained together, Auto-Lambda explores continuous, dynamic task relationships via task-specific weightings, and can optimise any choice of combination of tasks through the formulation of a meta-loss; where the validation loss automatically influences task weightings throughout training. We apply the proposed framework to both multi-task and auxiliary learning problems in computer vision and robotics, and show that AutoLambda achieves state-of-the-art performance, even when compared to optimisation strategies designed specifically for each problem and data domain. Finally, we observe that Auto-Lambda can discover interesting learning behaviors, leading to new insights in multi-task learning. Code is available at https://github.com/lorenmt/auto-lambda.

Published in Transactions on Machine Learning Research (05/2022)
Auto-X: Disentangling Dynamic Task Relationships
Shikun Liu
shiken. Biu wwt.d//mmaaaaaaccmm uk
Dyson Robotics Lab, Imperial College London
Stephen James
sterjom@berkeeley edu
University of California, Berkeley
Andrew J. Davison
davisaas Cempiiiooe@mmmmmraaa..o uk
Dyson Robotics Lab, Imperial College London
Edward Johns
e.johns hhopera@iierial.ccc uk
Robot Learning Lab, Imperial College London
Reviewed on OpenReview: https: openrevieepr........mm net/ forum? id= KKeCMim5VI
Abstract
Understanding the structure of multiple related tasks allows for multi-task learning to im- prove the generalisation ability of one or all of them. However, usually requires train- ing each pairwise combination of tasks together in order to capture task relationships, at an extremely high computational cost. In this work, we learn task relationships via an automated weighting framework, named Auto-A. Unlike previous methods where task re- lationships are assumed to be fixed, i.c., task should either be trained together or not trained together, Auto-A explores continuous, dynamic task relationships via task-specific weightings, and can optimise any choice of combination of tasks through the formulation of a meta-loss; where the validation loss automatically influences task weightings through- out training. We apply the proposed framework both multi-task and auxiliary learning problems in computer vision and roboties, and show that Auto-λ achieves stale-of-the- art performance, even when compared to optimisation strategies designed specifically for each problem and data domain. Finally, we observe that Auto-A can discover interesting learning behaviors, leading to new insights in multi-task learning, Code is available at httts://github.coo htwpstuub.cco/looont//aaaaaeeeee http://oreenttutoollaaaae lambda!
1 Introduction
Multi-task learning can improve model accuracy, memory efficiency, and inference speed, when compared to training tasks individually. However, it often requires careful selection of training tasks, to avoid negalive transfer, where irrelevant tasks produce conflicting gradients and complicate the optimisation landscape. As such, without prior knowledge of the underlying relationships between the tasks, multi-task learning can sometimes have worse prediction performance than single-task learning.
We define the relationship between two tasks to mcan what entent these two tasks should be trained logether, following similar definition in (Zamir al. 2018 Standley et al. 2020 Fifty et al. 2021). For example, say that task A is more related to task B than task C, if the performance of task A is higher when training tasks A and B together, compared to when training tasks A and C together.
To determine which tasks should be trained logether, we could exhaustively search over all possible lask groupings, where tasks a group are equally weighted but all other tasks are ignored. However, this requires training 2/7 multi-task networks for a set tasks T, and the computational cost for this search can be intractable when [T] is large. Prior works have developed efficient task grouping frameworks based on heuristics to speed up training, such using an early stopping approximation (Standley et al. 2020) and

---

Published in Transactions on Machine Learning Research (05/2022)
Semantic Segmentation
Depti Prodiction
Normal Prodiction
Multi-task Network
Auto-A
Start of Training
Start of Training
Framework
Midall of Training
Middle of Thiining
Weighting
End of Training
Eand Training
Primary Tasks
Auxibaary Learning Semmantic Segmentation
These
Multi-task Learning
All Tasks
Time
Figure 1: In Auto-A, task weightings are dynamically changed along with the multi-task network parameters, in joint optimisation. The task weightings can be updated both the auxiliary learning sctting (one task is the primary lask) and the multi-task learning setting (all tasks are the primary tasks). In this example, in the auiiliary learning setting, semantic segmentation is the primary task which During training, task weightings provide interpretable dynamic task relationships, where high weightings emerge when tasks are strongly related (e.g. normal prediction to segmentation) and low weightings when tasks are weakly related (e.g. depth prediction to segmentation).
optimising for.
computing a lookahead loss averaged across few training steps (Fifyy et al. 2021 However, these task grouping strategies are bounded by two prominent limitations. Firstly, they are designed to be two-stage methods, requiring a search for the best task structure and then re-training of the multi-task network with the best task structure. Secondly, higher-order task relationships for three or more tasks are not directly obtainable due high computational cost. Instead, higher-order relationships are approximated by small combinations of lower-order relationships, and thus, the number of training tasks increases, evaluating these combinations may become prohibitively costly.
In this paper, instead of requiring these expensive searches or approximations, we propose that the relation- ship between tasks is dynamic, and based on the curvent state of the multi-task network during training. We consider that task relationships could be inferred within a single optimisation problem, which runs re- currently throughout training, and automatically balances the contributions of all tasks depending on which tasks we optimising for. In this way, we aim unify multi-task and auxiliary learning into a single framework - whilst multi-task learning aims to achieve optimal performance for all training tasks, auxiliary learning aims to achieve optimal performance for only subset of training tasks (usually only one), which we call the primary tasks, and the rest of the training tasks are included purely to assist the primary tasks.
To this end, we propose a simple meta-learning algorithm, named Auto-A. Auto-A explores dynamic task relationships parameterised by task-specific weightings, termed X. Through meta-loss formulation, we use the validation loss of the primary tasks to dictate how the task weightings should be altered, such that the performance of these primary lasks can be improved in the next iteration. This optimisation strategy allows to jointly update the multi-task network well as task weightings in fully end-to-end manner.
We extensively evaluate Auto-A in both multi-task lcarning and auxiliary lcarning scttings within both computer vision and robotics domains. We show that Auto-A outperforms not only all multi-task and auxiliary learning optimisation strategies, but also the optimal (but static) task groupings we found in the selected datascts. Finally, we take deep introspection into Auto-X's learning behaviour, and we find that the dynamic relationship between tasks is consistent across numerous multi-task architecture designs, with the converged final relationships aligned with the fixed relationships found via brute-force search. The simple
2

---

Published in Transactions on Machine Learning Research (05/2022)
and efficient nature of our method leads to promising new insight towards understanding the structure of tasks, task relationships, and multi-task learning in general.
2 Related Work
Multi-task Architectures Multi-Tass Learning (MTL.) aims at simultaneously solving multiple learning problems while sharing information across tasks. The techniques used in multi-task architecture design can be calegorised into hard-parameter sharing (Kokkinos) 2017; Heuer et al. 2021 J, soft-paraneter sharing (Misra)
et al. 2016 Xu al. 2018 Liu et al. 2019c Maninis et al. 2019 Vandenhende et al. 2020 and neural
architecture search (Rosenbaum et al.
2018 Gao et al. 2020
Sun et al. 2020
Multi-task and Auxiliary-task Optimisation In an orthogonal direction to advance architecture de- sign, significant efforts have also been invested to improve multi-task optimisation strategies. Although
this is a multi-objeetive optimisation problem (Sener & Koltun 2018 Lin et al., 2019 Ye et al., 2021). a single surrogate loss consisting of linear combination of lask losses are more commonly studied practice. Notable works have investigated finding suitable task weightings based on different criteria, such as task uncertainty (Kendall et al. 2018), task prioritisation (Guo et al., 2018) and task loss magnitudes (Liu al.
2019ch. Other works have focused on directly modify task gradients (Chen et al., 2018 2020 Yu et al. 2020
Javaloy & Valera
2022
Liu et al. 2021a Navon et al. 2022
Similar to multi-task learning, there is challenge in choosing appropriate tasks to act as auxiliaries for the
primary tasks. Du et al. (2018) proposed to use cosine similarity as an adaptive task weighting to determine
when a defined auxiliary task is useful. Navon et al. (2021) applied neural networks to optimally combine auxiliary losses in a non-lincar manner.
Auto-A is a weighting- weighttig--ased optimisation framework by parameterising these task relationships via learned task weightings. Though these multi-task and auxiliary learning optimisation strategies are encoded to each problem, Auto-A is designed to solve multi-task learning and auxiliary learning in a unified framcwork.
Understanding Task Grouping and Relationships Prior optimisation methods typically assume all training tasks are somewhat related, and the problem of which tasks should be trained together often overlooked, In general, task relationships are often empirically measured by human intuition rather than prescient knowledge of the underlying structures learned by a neural network. This motivated the study of task relationships in the transfer learning setting (Zamir et al., 2018 Dwivedi & Roig, 2019). However, Standley et al. (2020) showed that transfer learning algorithms do not carry over to the multi-task learning domain and instead propose a multi-task specific framework to approximate exhaustive search performance. Further work improved the training efficiency for which the task groupings are computed with only a single training run (Fifty el al. 2021). Rather than exploring fixed relationships, our method instead explores dynamic relationships directly during training.
Meta Learning for Multi-task Learning Meta learning VVilltta & Drissi 2002: Hospedales et al. 2020
has been often used in the multi-task learning setting, such to generate auxiliary tasks in a self-supervine
manner LLu et al.] 2019b Navon et al. 2021) and improve training efficiency on unseen tasks (Finn et al.
2017 Wang et al.] 2021). Our work is also closely related to Kaddour et al.] 2020)) : Liu ct al. (020)) which
proposed a task scheduler to learn a task-agnostic representation similar to supervised pre- pre-training, whilst ours learns a reprosentation that can adapt specifically to the primary task; Ye al. (2021) which applied meta learning to solve multi-objective problems, whilst ours focuses on single-objective problems; [Michel et al. (2021) which applied meta learning to balance worst-performing tasks, whilst ours balances multi- task learning by finding optimal task relationships. Related to meta learning, our framework is learning to generate suitable and unbounded task weightings as lookahead method, optimised based on the validation loss of the primary lasks, as a form of gradient-based meta learning.
Meta Learning for Hyper-parameter Optimisation Since Auto-X's design models multi-task lcarning optimisation as learning task weightings dynamically via gradients, we may also consider Auto-A as meta learning-aased hyper-parameter optimisation framework (Maclaurin et al. 2015 Franceschi et al. 2018 Baik
3

---

Published in Transactions on Machine Learning Research (05/2022)
et al. 2020 by treating À as lyper-parameterss Similar to these frameworks, also formulate a bi-level optimisation problem. However, different to these frameworks, we offer training strategies specifically tailored to the problem of multi-task learning whose goal is not only to obtain good primary task performance, but also explore interesting learning behaviours of Auto-A from the perspective of task relationships.
3 Background
Notations We denote multi-task network to be [(: 0), with network parameters 0, consisting of task- shared and K task-specific parameters: 0 = (Oan, 01.k) Each task is assigned with task-specific weighting A = (A1:k]. }. We represent a set of task spaces by a pair of task-specific inputs and outputs: T {Tı:k) where T = (X,
The design of the task spaces can be further divided into two different settings: single-domain setting (where all imputs are the same X; X,, + J. i.e., one-to-many mapping), and multi-domain setting (where all inputs are different: X; * X3, i # i.e., many-to-many mapping). We want to optimise 0 for all tasks T and obtain a good performance in some pre-selected primary tasks T ST. If T = T, we are in the multi-task learning setting, otherwise we are in the auxiliary learning setting.
The Design of Optimisation Methods Multi-task or auxiliary learning optimisation methods are de- signed to balance training and avoid negative transfer. These optimisation strategics can further be cate- gorised into two main directions:
(i) Single Objective Optimisation:
K
min Σ A La (21; Boh (,),
(1)
where the task-specific weightings À are applied for a lincarly combined single valued loss. Each task's influence on the werw parameters can be indirectly balanced by finding a suitable set of weightings which
can be manually chosen, or learned through a heuristic (Kendall al.) 2018 Liu et al. 2019c) which we called weighting-based methods; or directly balanced by operating on task-specific gradients (Du et al. 2018
Yu et al.] 2020 Chen et al., 2018 2020 Javaloy & Valera 2022 Liu et al. 2021a Navon et al 2022
which we called grudient-based methods. These methods are designed exclusively to alter optimisation.
On the other hand, we also have another class of approaches that determine task groupings (Standley et al., 2020 Fifty et al.) 2021), which can be considered as an alternate form of weighting- based method, by finding fine and binary task weightings indicating which tasks should be trained together. Mixing the best of both worlds, Auto-A is an optimisation framework, simultancously exploring dynamic task relationships.
(ii) Multi-Obie ObjecivOee Optimisation:
min [L. (f (zz Osh, 0:) ()))"
(2)
a vector-valued loss which is optimised by achicving Pareto optimality - when no common gradient updates can be found such that all task-specific losses can be decreased (Sener & Koltun| 20118 Lin et al.) 2019 Note that, this optimisation strategy can only be used in a multi task- learning setup.
4 Auto-A: Exploring Dynamic Task Relationships
We now introduce our simple but powerful optimisation framework called Auto-A, which explores dynamic task relationships through task-specific weightings.
The Design Philosophy Auto-A is a gradient-based meta learning framework, a unified optimisation strategy for both multi-task and auxiliary learning problems, which learns task weightings, based on any combination of primary tasks. The design of Auto-A borrows the concept of lookahead methods in meta
4

---

Published in Transactions on Machine Learning Research (05/2022)
2017 Nichol et al, 2018). to update parameters at the current state of learning, based on the observed effect of those parameters on a future state. A recently proposed task grouping method (Fifty et al. 2021) also applied a similar concept, to compute the relationships based on how gradient updates one task can affect the performance other tasks, additionally offering the option to couple with other gradient-based optimisation methods. Auto-λ however is a standalone framework and encodes task relationships explicitly with a set of task weightings associated with training loss, directly optimised based on the validation loss of the primary tasks.
learning literature (Finn et al.
Bi-level Optimisation Let us denote P the set of indices for all primary tasks defined in Ter;; (xtta), ,paa) and (azrasn,, uytrain) are sampled from the validation and training sets of the ith task space, respectively. The goal of Auto-A is to find optimal task weightings A which minimise the validation loss on the primary tasks, as a way to measure generalisation, where the optimal multi-task network parameters are obtained by minimising the X weighted training loss on tasks. This implies the following bi-level optimisation problem:
En vol, 02r: 07), v
min
iEP
(3)
s.t. 0- arg min
> (atrin, ; 0.). yf (rain).
Approximation via Finite Difference Now, may rewrite Eq. with simple approximation scheme by updating 0 and À iteratively with one gradient update each:
K A L. Oah, 0.), vi r
0' = 0 - a
(4)
i=1
A+ - BV X
Σ
L,(f(at' mill 1 0%). vi
(5)
EEP
K
0 0 V Σ
A L,(f(x
Ssh. 0.), 11 m
(6)
for which a, 3 are manually defined learning rates.
The above optimisation requires computing second-order gradients which may produce large memory and slow down training speed. Therefore, we apply finite difference approximation to reduce complexity, similar to other gradient-based meta learning methods (Finn et al., 2017 Liu et al., 2019a). For simplicity, let's denote C( X), ( A) represent weighted loss produced by all tasks and primary tasks respectively. The gradient update À can be approximated by:
Vx 1) ~ VACTM aVo(((, 1...
a VAC(0+ A)
VAC(O
(7)
2E
where 0' 0 aVo denotes the network weights for a one-step forward model, and 0= 0 + Lori 1), with small constant. are constants indicating that all primary tasks of equal importance, and may also apply different constants based prior knowledge.
Note that, À is only applied on the training loss not validation loss, otherwise, we would easily reach trivial solutions A = O. In addition, assuming 0' 0* is also not applicable, otherwise we have VX = 0.
Swapping Training Data In practice, instead of splitting training data into training and validation sets as in the standard meta learning setup, we sampled training and validation data be the different batches in the same training dataset. We found that this simple swapping training data strategy can learn similar weightings comparing to sampling batches in different datasets, making Auto-A single-stage framework.
5

---

Published in Transactions on Machine Learning Research (05/2022)
Stochastic Task Sampling Eq. requires to compute gradients for all training tasks. This may lead to significant GPU memory consumption particularly when the task-shared parameters are accumulating gradients in a multi-domain setting. To further save memory, we optimise À in multiple steps, and for each step, we only compute gradients for K' < K tasks sampled stochastically. This design allows Auto-A to be optimised with a conslant memory independent of the number of training lasks. In practice, we choose the largest possible K' each dataset that fits in GPU to speed up training, and we observed that the performance is consistent from a wide range of different K'.
5 Experiments
To evaluate the generalisation of Auto-A, experimented both single and multi-domain computer vi- sion and robotics datasets, in multi-task and auxiliary learning settings, with various choices of multi-task architectures.
Baselines In multi-task experiments, we compared Auto-A À with state-of-the-art weighting-based multi-task
optimisation methods: i) Equal: all task weightings are 1, ii) Uncertainty (Kendall et al. 2018): p: task weightings are optimised via Homoscedastic uncertainty, and iii) DWA (Liu et al. 2019c): task weightings are optimised via the rate of change of training losses. auxiliary learning experiments, we only compared with GCS (Gradient Cosine Similarity) (Du et al., 2018) due to the limited works for this setting. Additional experiments comparing to gradient-based methods are further shown in Additional Analysis (Section 7.2
Optimisation Strategies By default, we considered each single task as the primary task in the auxiliary learning setting, unless labelled otherwise. In all experiments, Auto-X's task weightings were initialised to 0.1, a small weighting which assumes that all tasks are equally not related. The learning rate to update these weightings is hand-selected for each dataset. For fair comparison, the optimisation strategies used all baselines and our method are the same with respect each dataset and in each data domain. Detailed dyper-parameet are listed in Appendix
5.1 Results on Dense Prediction Tasks
First, we evaluated Auto-À with dense prediction tasks in NYUv2 (Nathan Silberman & Fergus 2012) and
CityScapes (Cordts et al. 2016), two standard multi-task datasets in single-domain setting. In NYUv2, trained on 3 tasks: 13-class semantic segmentation, depth prediction, and surface normal prediction, with the same experimental setting as in Liu et al. (2019c). In CityScapes, we trained on 3 tasks: 19- class semantic segmentation, disparity (inverse depth) estimation, and a recently proposed 10-class part sogmentation de Geus et al. 2021), with the same experimental sctting as in Kendall et al. (2018). In both datasets, we trained on two multi-lask architectures: Split: the standard multi-task learning architecture with hard parameter sharing, which splits at the last layer for the final prediction for each specific task; MTAN [Liu et al.] (2019c]: stato-of-tho-art multi-task architecture based on task specific feature-level attention. Both networks were based ResNet-50 (He et al. 2016 the backbone architecture.
Evaluation Metrics We evaluated segmentation, depth and normal via mean intersection over union (mloU), absolute error (aErr.), and mean angle distances (mDist.), respectively. Following Maninis et al. 2019 ; we also report the overall relative multi-task performance AMTL of model m averaged with respect each single-task baseline b:
1 K
ATTI
(M Mo.)//Mbi.
(8)
where 4 = if lower means better performance for metric M2 of task i, and 0 otherwise.
Noise Prediction as Sanity Check In auxiliary learning, we additionally trained with a noise prediction
task along with the standard three tasks defined in a dataset. The noise prediction task was generated assigning a random noise map sampled from a Uniform distribution for each training image. This

---

Published in Transactions on Machine Learning Research (05/2022)
NYUv2
Method
Sem. Seg.
[uloU
Depth [aEr..
Normal JinDist. 4]
AMTI
CitySeappa
Method
Semn. Seg.
[mioOU
Part Seg.
[mloUU
[aEr.. 4]
Disp. AMrI T
Single-Task
43.37
62.24
22.40
Single-Taok
56.20
52.74
0.84
Equal
41.64
43.32
24. 48
+3.57%
Equal
54.03
50.18
0.79
-0.92%
Split
DWA
45.14
43.06
24.17
+4.08%%
Split Multi Task
DWA
34.93
50.15
0.80
-0.80%
Multi Task
Uncertainty
45.96
41.26
+6.50%
Uncertainty
56.06
Auto-A
47.17
40.97
23.68
+8.21%%
Auto-A
56.08
51, 88
0.78
+2.56%
Uncertainty
45.26
42.25
24.36
+4.011%
Uncertainty
36.72
52. 62
0.83
+0.04%
Split
GCS
45.01
42.06
24.12
+-..00%
Split
G08
56.76
52.19
0.80
+0.98%
Auxiliary-Tad Dusk Auto-A [3 Tusko]
48.04
40.G1
23.31
4.9.66%
Auxiliary- Taaiiaary-Takk Auto-A [3 Taaka]
56.42
62. 42
0.78
+.2.31%
Auto-A
Task]
47.80
+1.02%
Auto-A Task]
57.89
+4.30%
Equal
44.02
42.64
24.
444.2%%
Equal
50.00
60.74
0.78
+0.43%%
MTAN
DWA
45.04
42.81
+4.89%
MTAN Malti-Task
DWA
34.71
51.07
0.80
-0.33%
Multi- Taak
Uncertainty
46.41
40.94
23.65
47.60%
Uncertainty
56.28
53.24
0.82
+1.16%
Auto-A
47.68
40.37
23.28
+9.64%
Auto-A
66.67
5/2.67
0.76
+ +..6%%
Uncertainty
44.56
42.21
24.26
+-4.55%
Uncertainty
56.13
52.78
0.83
+0.38%
MTAN
GCS
44.26
4.1.07
24.03
+3.49%%
MTAN Auxiliary-Tass Auto-A
GCS
$5.47
52. 75
0.76
+2.75%
Auxiliary-aask
Auto-A
[3 Tacke]
47.35
40.10
23.
+99.00%
[3 Tasks]
37.64
52.77 77
0.78
+3.25%
Tusk)
47.70
+ +1.699
Auto-A Taskk]
58.39
54.00
44.48%
Table 1: Performance on NYUv2 and CityScapes datascts with multi-task and auxiliary learning methods in Split and MTAN multi-task archilectures. Auxiliary learning is additionally trained with a noise prediction task. Results are averaged over two independent. runs, and the best results are highlighted in bold.
task is designed to test the effectiveness of different auxiliary learning mothods in the presence of useless gradients. We trained from scratch for a fair comparison among all methods in our experiments, following prior works (Kendall al. 2018 Liu et al., 2019c Sun et al. 2020
Results Table showed results for CityScapes and NYUv2 datasets in both Split and MTAN multi-tase architectures. Our Auto-A outperformed all baselines in multi-task and auxiliary learning settings across both multi-task networks, and has a particularly prominent effect in auxiliary learning setting where it doubles the relative overall multi-task performance compared to auxiliary learning baselines.
We show results for two auxiliary task settings: optimising for just one task (Auto-A [I Task]), where the other three tasks (including noise prediction) are purely auxiliary, and optimising for all three tasks (Auto-A [3 Tasks)), where only the noise prediction task is purely auxiliary. Auto-A [3 Tasks] has nearly identical performance to Auto-A in a multi-task learning setting, whereas the best multi-task baseline Uncertainty achieved notably worse performance when trained with noise prediction as an auxiliary lask. This shows that standard multi-task optimisation is susceptible negative transfer, whereas Auto-À can avoid negative transfer due to its ability to minimisc À for tasks that do not assist with the primary task. We also show that Auto-A [1 Task] can further improve performance relative to Auto-A [3 Tasks], at the cost of task-specific training for each individual task.
5.2 Results on Multi-domain Classification Tasks
We now evaluate Auto-A on image classification tasks in a multi-domain sctting. We trained on CIFAR- 100 (Krizhevsky) b009) and treated each of the 20 'coarse' classes as one domain, thus creating a dataset with 20 tasks, where each task is a 5-class classification over the dataset's 'fine' classes, following Rosenbaum et al. 2018)); Yu et al. (2020). For multi-task and auxiliary learning, trained all methods VGG-16 network (Simonyan & Zisserman task-specific prediction layer.
2015) with standard hard-parameter sharing (Split), where each task has
Results In Table 2 we show classification accuracy on the 5 most challenging domains which had the lowest single-task performance, along with the average performance across all 20 domains. Multi-task learning in this dataset is particularly demanding, since we optimised with a x20 smaller parameter space per task compared to single-task learning. We observe that all multi-task baselines achieved similar overall performance to single-task learning, due to limited per-task parameter space. However, Auto-A was still able to improve the overall performance by a non-trivial margin. Similarly, Auto-} can further improve performance in the auxiliary learning setting, with significantly higher per-task performance in challenging domains with around 5 7% absolute improvement in test accuracy.
7

---

Published in Transactions on Machine Learning Research (05/2022)
Aquatic Animals
Small Manmals
CIFAR-100
Method
People
Trees
Reptiles
Avg.
Single-Task
55.37
68.65
72.79
75.37
75.84
82.19
Equal
57.73
73.59
74.41
74.64
76.69
82.46
Uncertainty
54.14
70.62
74.08
74.62
75.62
82.03
Multi-Task
DWA
55.25
71.54
74.12
75.68
76.26
82.26
Auto-A
57.57
74.00
75.05
75.15
77.55
83.92
GCS
56.45
71.05
72.93
74.45
76.29
82.58
Auxillary-Task
Auto-A
60.89
75.70
75.64
77.38
81.75
84.92
Table 2: Performance of 20 tasks in CIFAR-100 dataset with multi-task and auxiliary learning methods. We report the performance from 5 domains giving lowest single-task performance along with the averaged performance across all 20 domains. Results are averaged over two independent runs, and the best results are highlighted in bold.
5.3 Results on Robot Manipulation Tasks
Finally, to further emphasise the generality of Auto-A, we also experimented on visual imitation learning tasks within multi-domain robotic manipulation sctting.
To train and evaluate our method, we selected 10 lasks (visualised in Fig. 2 from the robot learning environment, RLBench (James et al. 2020). Training data was acquired by first collecting 100 demonstrations for each task, and then running keyframe discovery following James & Davison (2021). to split the task into a smaller number simple stages to create a behavioural cloning dataset. Our network takes RGB and point-cloud inputs from 3 cameras (left shoulder, right shoulder, and wrist camera), and outputs continuous 6D pose and discrete gripper action. To distinguish among each of the tasks, learnable task encoding is also fed to the network for multi-task and auxiliary learning. Full training details are given in Appendix B
Figure 2: A visual illustration of 10 hand-selected RLBench lasks from the front-facing camera. Task names are: reach torget, push button, pick and lift, pick up cup, put knife on chopping board, take money out of sofe, put money in safe, take umbrella out of umbrella stond, stack wine, slide block to target.
Results In Table B we reported success rate of cach and averaged performance over 10 RLBench tasks. In addition to the baselines outlined in Section 5 we also included an additional baseline based on Priority Replay (Schaul al. 2016):: a popular method for increasing sample efficiency in robot learning systems. For this baseline, prioritisation applied individually for each task. Similar to computer vision tasks, Auto-X achieved the best performance in both multi-task and auxiliary learning sctup, particularly can improved up to 30- 40% success rate in some multi-stage tasks compared to single-task learning.
R.B.Bnch
Method
Reach Target Button
Push
And Lift
Up Cup
Knife on Chopping Board
Money Out Safe
Money Safe
Pick Umbreela
Stack Block Wine
To Targes
Avg.
Single- Task
95
82
31
36
Equal
92
57
44
16
40
Uncertainty
100
85
75
78
70
16
65
63.5
Multi-Tusk
DWA
100
90
88
82
35
68
57
61
16
66
66.1
Priority
100
96
78
78
36
46
15
34
56.2
Auto-A
95
Auxiliary-Taeis
GCS
97
58
60
14
77
93
90
75
74
20
78
Table 3: Performance RLBench tasks with multi-task and auxiliary learning methods. We reported the success rate with 100 evaluations for each task averaged across random seeds. Best results are highlighted in bold.

---

Published in Transactions on Machine Learning Research (05/2022)
6 Intriguing Learning Strategies in Auto-À
In this section, we visualise and analyse the learned weightings from Auto-A, and find that Auto-X is able to produce interesting learning strategies with interpretable relationships. Specifically, we focus on using Aulo-A to understand the underlying structure of lasks, introduced next.
6.1 Understanding The Structure of Tasks
Task relationships are consistent. Firstly, we observe that the structure of tasks is consistent across the choices of learning algorithms. As shown in Fig. 3 the learned weightings with both the NYUv2 and CityScapes datasets are nearly identical, given the same optimisation strategics, independent of the network
archilectures. This observation is also supported by the empirical findings in Zamir el. al. (018)) 3 Standley et al. 2020) in both task transfer and multi-task learning settings.
Task relationships are asymmetric. We also found that task relationships are asymunetrie, i.e. learning task A with the knowledge of task is not equivalent to learning task with the knowledge of task A. A simple example is shown in Fig. 4 Right, where the semantic segmentation task in CityScapes helps the part segmentation task much more than the part segmentation helps the semantic segmentation. This also follows intuition: the representation required for semanlic ic segmentalion is subset of the representation required for part segmentation. This observation is also consistent with multi-task learning frameworks Lee
ctt al.
2016
2018 Zamir et al. 2020 Yeo et al. 2021
Tran Thoks
Train Tades
Train
Train Tasks
Seg.
Depth
Normal
Sem. Sog.
Depth
Normal
Seg.
Noise
Seg.
Dtap.
Seg.
3Taaks Norma Depth Sem Seg:
Port Seg. Seg.
37aak Dag. Part Seg. Scm Seg.
1.00
0.52
0.54
0.13
1.6
0.5
0:78
0:14
L.48
0.92
0.84
0.13
1.46
0.94
0,51
0.11
0.57
Primayy Toke
1.04
0.61
0.12
Piloary Tasks
0.12
Primary Tadls
0,50
Drimary
Noral
0.70
0113
1.M
0.5
Split- NYUv2
MTAN NYUv2
Split CityScapes
MTAN CityScapes
Figure 3: Auto-A explored consistent task relationships in NYUv2 and CityScapes datasets for both Split and MTAN architectures. Higher task weightings indicate stronger relationships, and lower task weightings indicate weaker relationships.
1.5
Seg,
Sem. Seg.
Sem. Sin:
Sem. Seg
Depth
Dopth
Port Seg.
Fart Seg.
Nocmal
Normal
Disp.
Disp.
Wateltiag
Noise
Weighting
Noice
Woightng
Nuise
Wrightng
Noise
0.5
1.5
0.5
0.5
100
200
200
200
Ppoch
Epoch
Epoch
Epoch
NYUv2
NYUv2
CityScapes
CityScapes
Auto-A [3 Tasks]
Uncertainty
Auto-X [Part Seg.]
Auto-A [Sem. Seg.]
Figure 4: Auto-A learned dynamic relationships based on the choice of primary tasks and can avoid negative transfer. Whilst Uncertainty method is not able to avoid negative transfer, having a constant weighting on noise prediction task across the entire training stage. [+] represents the choice of primary tasks.

---

Published in Transactions on Machine Learning Research (05/2022)
Task relationships are dynamic. A unique property of Auto-A is the ability to explore dy- namic task relationships. As shown in Fig. g Left, we can observe a weighting cross-over appears in NYUv2 near the end of training, which can be considered as a learning strategy of automated curricula. Further, in Fig. 5 we verify that Auto- À achieved higher per-task performance compared to every combination of fixed task groupings in NYUv2 and CityScapes datasets. We can also observe that the lask relationships inferred by the fixed task groupings is perfectly aligned with the relationships Icarned with Auto-A. For ex- ample, the performance of semantic segmentation trained with normal prediction (+6.6%) is higher than the performance trained with depth predic- tion (-6.0%), which is consistent with fact that the weighting of normal prediction (0.84) is higher than depth prediction (0.52) as shown in Fig. 3 addition, we can observe that the Uncertainty method is not able to avoid negative transfer from the noise prediction task, having a constant weighting across the entire training stage, which leads to a degraded multi-task performance as observed in Table These observations confirm that Auto-A is an advanced optimisation strategy, and is able to learn accurate and consistent task relationships.
The Performance of
The Performance of
Sem, Sog
Depth
Nornd
Sen Seg.
Part Seg.
Dig:
Deg. Part Sqg. Sam Sop.
+8.98%
5.01%
+0.99%
47.14%
Trained with
Trained
Depth
6.06%
4..7%
+1.8%
+0.62%
+6.02%
+18-2%
-4.54%
-6.73%
Auauk MTL
+2.95%
+17.05%
9.29%
Anc-A ATEL
3,80%
-1.83%
+5.93%
+10.1%
+22.91%
-3.00%
+3.01%
+1.55%
18.33%
NYUv2
CityScapes
Figure 5: Auto-A achieved best per-task performance compared to every combination of fixed task groupings.
7 Additional Analysis
Finally, we present some additional analyses with Split, multi-task architect understand the behaviour of Auto-A with respect to different syper-parameed and other types of optimisation strategies.
7.1 Robustness on Training Strategies
Here, we evaluate different prper-parameters trained with Auto-A [3 Tasks] in the auxiliary learning setting, As seen in Fig. G we found that Auto-λ optimised with direct second-order gradients offers very similar task weightings compared to when optimised with approximated first-order gradients. In addition, we discovered that using first-order gradients may speed up training time roughly x2.3. In Table we show that initialising with small weighting and suitable learning rate is important achieve a good performance. A larger learning rate leads to saturated weightings which causes unstable network optimisation, and larger initialisation would not successfully avoid negative transfer. In addition, optimising network parameters and task weightings with different data is also essential which otherwise would slightly decrcase performance.
0.10
Task Weightings
AMTL
Sem. Seg.
Depth
Normal
Noise
Init = 0.01
0.97
0.95
1.1
0.02
+8.98%
0.05
I
Triit = 1.0
2.00
2.11
2.08
1.00
+1.4%%
LR = 3 10
0.43
0.37
0.46
0.11
1853%
LR = 3 30-4
3.10
3.34
3.26
0.15
18.56%
= = 10-3
10.5
10.5
10.3
0.23
+5.04%
0
No Swapping
2.67
2.76
2.98
0.20
+8.17%
Sem. Seg.
Depth
Normal
Noise
Our Setting
1.11
1.06
1.26
0.12
49.66%
Figure 6: Mean and the range of per-task weighting difference for Auto-A [3 Tasks] opti- mised with direct and approximated gradients in NYUv2 dataset.
Relative multi-task performance in NYUv2 dataset trained with Auto-A [3 Tasks] with different The default setting is Init = 0.1.
Table 4:
LR = 1. 10-4 and with training data swapping.
10

---

Published in Transactions on Machine Learning Research (05/2022)
7.2 Comparison to Gradient-based Methods
Since Auto-A is weighting-baser based optimisation method, it naturally be combined with gradient-based methods to further improve per- formance. We evaluated Aulo-À along with the other weighting-bassee based baselines described in Soc. []] when combined with recently proposed state-of thete-of-theeatt the-art gradient- based methods designed for multi-task learning: GradDrop (Chen et al.) 2020), PCGrad (Yu et al., 2020) and CA- Grad (Liu et al. 2021a). We trained all methods in NYUv2 dataset with standard 3 tasks the multi-task learning setup.
Equal
DWA
Uncertainty
Auto-A
Vanilla
+3.57%
+4.58%
+6.50%
+8.21%
GradDrop
+4.65%
+5.93%
+6.22%
+8.12%
PCGrad
5.00%
14.37%
16.20%
+8.50%
CAGrad
7.05%
+88.08%
+9.65%
+11.07%
Table 5: NYUv2 relative multi-task performance trained with both weighting-bast based and gradient gradient-based methods in the multi-task learning sctting.
In Table we can observe that Auto-A remains the best optimisation method even compared to other gradient-basel methods the vanilla setting (with Equal weighting). Further, combined with a more advanced gradient-basee method such as CAGrad, Auto-A can reach even higher performance.
7.3 Comparison to Strong Regularisation Methods
Finally, recent works (Lin et al., 2021 Kurin et al. 2022 suggested that many multi-task optimisation methods can be interpreted as forms of implicit regularisation. They showed that when using strong regu- larisation and stabilisation techniques from single-tase task learning, training by simply minimising the sum of task losses, or with randomly generated task weightings, can achieve performance competitive with complex multi-lask methods.
As such, now evaluate Auto-A, along with all multi-task baselines evaluated our Experiments section, well as all multi-task methods included in the original work of (Kurin et al. 2022). coupled with this
Task Mean Accuracy (%)
91.25
91.00
90.75
90.50
90.25
Unit. Scal. Uncert.
DWA
RLW
RLW
IMTL
MGDA GradDrop PCGrad
CAGrad
Auto-A (MTL)
Auto-A (AL)
(Diri.)
(Norm.)
(a) Mean and the range (3 runs) for the averaged task test accuracy
Training Time (sec.)
10
10a
Unit.
Scal.
Uncert.
DWA
RLW (Diri)
RLW
IMTL
MGDA
GradDrop
PCGrad
CAGral
Auto-
(Norm.)
(b) Mean per-epoch training time (10 reptitions)
Figure 7: All multi-task methods perform the worse than Unit. Scal. the CelebA dataset trained with strong regularisation, except Auto-X. Part of the results directly borrow from Kurin et al. (022)
11

---

Published in Transactions on Machine Learning Research (05/2022)
strong regularisation on CclebA dataset (Liu et al. 2015). for challenging 40-task classification problem. We trained these multi-task methods with the exact same experimental setting in Kurin et al. (2022) for a fair comparison. To conclude, we compared with: Unit. Scal. (Kurin et al. (022), DWA @Liu et al. 2019c
RLW (with weights sampled from a Dirichlet and a Normal Distribution) (Lin ct al. 2021). IMTL (Liu et al.
202Tb), MGDA (Sener & Koltun 2018), GradDrop (Chen et al. 2020). PCGrad YYu et. al. 2020
CAGrad
Liu et al. 2021a), for a total of 10 multi-task optimsiation methods.
To our surprise, though most methods achieve similar performance, which is consistent with the findings in (Kurin et al. 2022), Auto-A is still able to improve performance (marginally in the multi-task learning setting, and significantly in the auxiliary learning sctting) with clear statistical significance. The improvement is especially pronounced in the auxiliary learning mode, which is the unique learning mode of Auto-,, showing the multi-task network's generalisation imposed from Auto-A is more than implicit regularisation.
In addition, we also compared training time across these multi-task mothods, and we ro-scaled the training time in our implementation Kurin et al. (2022)'s 's setting for fair comparison. We can observe that Auto-A À requires three times longer the training time than Unit. Scal. (Equal weighting) (Kurin et al. 2022). in consistent with its theoretical design, since Auto-A needs to compute additional two forward and two backward passes to approximate the sccond-order gradients. Though Auto-A requires longer training time, it can perform other multi-task methods, and still an order of magnitude faster than some gradient-based methods such as PCGrad Yu et al. 2020 and CAGrad (Liu et al. 2021a).
8 Conclusions, Limitations and Discussion
In this paper, we have presented Auto-A, a unified multi-task and auxiliary learning optimisation framework. Auto-A operates by exploring task relationships in the form of task weightings in the loss function, which are allowed to dynamically change throughout the training period. This allows optimal weightings to be determined at any one point during training, and hence, a more optimal period of learning can emerge than if these weightings were fixed throughout training. Auto-A achieves state-of-the-art performance in both computer vision and robotics benchmarks, for both multi-task learning and auxiliary learning, even when compared to optimisation methods that are specifically designed for just one of those two settings.
For transparency, we now discuss some limitations of Auto-A that we have noted during our implementations, and we discuss our thoughts on future directions with this work.
Hyper-parameter Search To achieve optimal performance, Auto-A still requires hyper-parameter search (although the performance is primarily sensitive to only one parameter, the learning rate, making this search relatively simple). Some advanced training techniques, such as incorporating weighting decay or bounded task weightings, might be helpful to find a general set of hyper- papemeerrseeeers which work for all datasets.
The design of Auto-A requires computing second-order gradients, which is computation- ally expensive. To address this, we applied a finite-difference approximation scheme to reduce the complexity, which requires the addition of only two forward passes and two backward passes. However, this may still be slower than alternative optimisation mothods.
Training Speed
Single Task Decomposition Auto-A can optimise on any type of task. Therefore, it is natural to consider a compositional design, where decompose a single task into multiple small sub-tasks, e-g. to decompose multi-stage manipulation tasks into sequence of stages. Applying Auto-A on these sub-tasks might enable to explore interesting learning behaviours to improve single task learning efficiency.
Open-ended Learning Given the dynamic structure of the tasks explored by Auto-A, it would be inter- esting to study whether Auto-A could be incorporated into an open-ended learning system, where tasks are continually added during training. The ficxibility of Auto-A to dynamically optimise task relationships may naturally facilitate open-ended learning in this way, without requiring manual selection of parameerrssssers for each new task.
12

---

Published in Transactions on Machine Learning Research (05/2022)
References
Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Meta-learning with adaptive hyperparameters. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normaliza- tion for adaptive loss balancing deep multitask networks. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
Zhao Chen, Jiquan Ngiam, Yanping Huang. Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick sign: Optimizing deep multitask models with gradient sign dropout. In Advances in Neural Informalion Processing Systems (NewrIPS), 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene under- standing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Daan de Geus, Panagiotis Meletis, Chenyang Lu, Xiaoxiao Wen, and Gijs Dubbelman. Part-aware panop- tic segmentation. In Proceedings of the TEEE Conference on Computer Vision and Pateern Recognation (CVPR), 2021.
Yunshu Du, Wojeiech M Czarnecki, Siddhant M Jayakumar, Razvan Pascamu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. orXiv preprint Axiv:1812.0222. 2018.
Kshitij Dwivedi and Gemma Roig. Reprosentation similarity analysis for efficient task taxonomy & transferr learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic mela-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel program- ming for hyperparameter optimization and meta- leeanng. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task-agnostic neural archi- tecture search towards general-purpose multi-task learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Michelle Guo, Albert Haque, De-An Huang, Screna Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Falk Heuer, Sven Mantowsky, Saqib Bukhari, and Georg Schneider. Mullitass--entternel (men): Efficient and diverse multitask learning using an anchor free approach. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. Xir preprint or arXiv:2004.05442 2020.
Stephen James and Andrew Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEFF Robolics and Automation Letters, 2021.
13

---

Published in Transactions on Machine Learning Research (05/2022)
Stephen James, Andrew J Davison, and Edward Johns. Transferring end-to-end visuomotor control from simulation to real world for multi-stage task. Conference on Robot Learning (CoRL), 2017.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.
Adrián Javaloy and Isabel Valera. Rotograd: Dynamic gradient homogenization for multi-task learning, In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Jean Kaddour, Steindór Sæemundsson, et al. Probabilistic active meta-lcarning. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Compuler Vision and Paltern Recognition (CVPR), 2018.
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogrition (CVPR), 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https: http///wwc..a toronto. httpp////krii/llaaarrr-eaaaaeree
Vitaly Kurin, Alessandro De Palma, Nyy Kostrikov, Shimon Whiteson, and M Pawan Kumar. In defense of the unitary scalarization for deep multi-task learning. arXiv preprint arXin:2201.04122, 2022.
Giwoong Lee, Eunho Yang, and Sung Hwang. Asymmetric multi-task learning based on task relatedness and loss. In Proceedings of the International Conference on Machine Learning (ICML), 2016.
Hae Beom Lee, Eunho Yang, and Sung Ju Hwang. Deep asymmetric multi-task feature learning. In Pro- ceedingg of the International Conference Machine Learning (ICML), 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 2016.
Baijiong Lin, Feiyang Ye, and Yu Zhang. A closer look at loss weighting in multi-task learning, arXiv preprint 1rXio:2111.1000 2021.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. Advances in Newral Information Processing Systems (NeurIPS), 2019.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multitask learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021a.
Chenghao Liu, Zhihao Wang, Doyen Sahoo, Yuan Fang, Kun Zhang, and Steven C. H. Hoi. Adaptive task sampling for meta-learning. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In Proceedings of the International Conference Learning Representations (ICLR), 2019a.
Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In Proceedings of the International Conference on Learning Rep- resentations (ICLR), 2021b.
Shikun Liu, Andrew J Davison, and Edward Johns. Self-supervised generalisation with meta auxiliary learning. In Advances in Neural Information Processing Systems (NeurIPS). 2019b.
Shikum Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEFE Conference on Computer Vision and Patlern Recognition (CVPR), 2019c.
14

---

Published in Transactions on Machine Learning Research (05/2022)
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the International Conference Computer Vision (ICCV), 2015.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. moudient-based hyperparameter optimization through reversible learning. In Proceedings of the International Conference on Machine Learning (ICML), 2015.
Kevis-Kokitsi Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multiple tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Paul Michel, Sebastian Ruder, and Dani Yogatama. Balancing average and worst-case accuracy in multitask learning. arXiv preprint arXiv:21 10. 100.08888, 2021.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for mulli-task lcarning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Proceedings of the European Conference on Computer Vision (ECCV), 2012.
Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit diffcrentiation. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as bargaining game. arXiv preprint orXiv:2202.0011 2022.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. orXiv preprint 1xXx:1803.02090 2018.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In Proceedings of the International Conference on Learning Representa- tions (ICLR), 2018.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Trevor Standley, Amir Zamir, Dawn Chen, Lconidas Guibas, Jitendra Malik, and Silvio Savarese. Which lasks should be learned together in multi-cask learning? Proceedings of the International Conference on Machine Learning (ICML), 2020.
Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Mti-net: Multi-scale task interaction net- works for multi-task learning. Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence. review, 2002.
Haoxiang Wang, Han Zhao, and Bo Li. Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation. In Proceedings of the International Conference on Machine Learning (ICMT), 2021.
15

---

Published in Transactions on Machine Learning Research (05/2022)
Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction-and- distillation network for simultaneous depth estimation and scene parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Feiyang Ye, Baijiong Tin, Zhixiong Yue, Pengxin Guo, Qiao Xiao, and Yu Zhang. Multi-objective meta learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Teresa Yeo, Oguzhan Fatih Kar, and Amir Zamir. Robustness via cross-domain ensembles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-lask learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the TEEE Conference on Compuler Vision and Pattern Recognition (CVPR), 2018.
Amir Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas J Guibas. Robust learning through cross-task consistency. In Proceedings of the IEEE Conference Computer Vision and Pallern Recognilion (CVPR), 2020.
16

---

Published in Transactions on Machine Learning Research (05/2022)
A Detailed Training Strategies
For dense prediction tasks, we followed the same training setup with MTAN based on the code that was made publicly available by the authors (Liu et al. 2019c). We trained Auto-A with learning rate 10-4 and 3 10-5 for NYUv2 and CityScapes respectively.
For multi-domain classification tasks, we trained each and all tasks with SGD momentum with 0.1 initial learning rate, 0.9 momentum, and 5-10-4 weight decay. We applied cosine annealing for learning rate decay trained with total 200 epochs. We set batch size 32 and trained Auto-A with 3 10-1 learning rate.
For robot manipulation tasks, trained with Adamn with constant learning rate 10-3 for 8000 iterations. We set batch size 32 and we trained Auto-A with 3 10-6 learning rate.
B Detailed Experimental Setting for Robotic Manipulation Tasks Naively applying behaviour cloning (e.g. mapping observations to joint velocities or end-effector incremental poses) for robot manipulations tasks often requires thousands of demonstrations (James et al.) 2017 To circumvent that, we first pre-processed the demonstrations by running keyframe discovery {James & Davison 2021); a process that iterates over each of the demo trajectories and outputs the transitions where interesting things happen, e.g. change in gripper state, or velocities approach zero. The results of the keyframe discovery is small number of end-effector poses and gripper actions for each of the demonstrations, essentially splitting the task into set of simple stages. The goal of our behaviour cloning setup is to predict these end-effector poses and gripper actions for new task configurations. Training data was then acquired by first collecting 100 demonstrations for each task, and then rumning keyframe discovery, to split the task into a smaller number of simple stages to create our behavioural cloning dataset.
We optimisod an encoder-docoder network which takes the inputs of RGB and point-clouds captured by three different cameras (left. shoulder, right shoul- der and wrist camera), and outputs continuous 61) pose and discrete gripper action. The 6D pose is composed of a 3- 3-dimensional vector encoding spa- tial position and a 4-dimensional vector encoding rotation (parameterised by a unit quaternion); the gripper action is represented by a binary scalar in- dicating gripper open and close. The position and rotation are learned through two separate decoders. The position decoder predicts allention maps based on RGB images, then we apply spatial (soft) argmax [evine et al.] 2016) on the corresponding point cloud to output a 3D spatial position of the attended pixel. We additionally optimised a position off-set for each stage of the task, so the predicted position will not be bounded by the position only available in the images. The rotation encoder predicts quaternion and gripper action via direct regression. A learnable task embedding is fed to the network bottleneck for multi-task and auxiliary learning.
Riist Ciaud
Position Dacoder
+
Spuial
RGB Inage
Attention Klpp
Rotaionn Decoder
Quaternion
Theb-Speelfic Stage Embedding
Gripper Opas/Close
Networ's Parunetess
Paruneters
Chrpute
Irputs
Figure 8: Visualisation of the network design for RL- Bench.
17

---

Published in Transactions on Machine Learning Research (05/2022)
C Auto-À Learned Weightings for NYUv2 and CityScapes
We found that the relationships in NYUv2 and CityScapes datasct are usually static from the beginning of training (except for NYUv2 [3 lasks] where we can observe a clear weighting cross-over).
1.5
Semn. Seg.
1.5
Sem.
Seg.
1.5
Sem. Seg.
1.5
Sem. Seg.
Deptlt
Degth
Depth
Depth
Normal
Normal
Normal
Norraal
Weightiin
Notse
Welgheiigg
Naise
Welglming
Noise
Welghebngg
Notse
0.5
0.5
0.5
0.5
0
0
100
200
100
200
0
100
200
0
100
200
Epoch
Epoch
Epoda
Epoch
NYUv2 [Sem. Seg-]
NYUv2 [Depth]
NYUv2 [Nrmal]
NYUv2 [3 Tasks]
Senn, Seg,
Som. Seg,
Sem. Seg.
Sem. Sey,
Part Sog.
Part
Part
Part
1.5
Diep
1.5
Disp.
1.5
Dieg.
1.5
Dieg.
Welgining
Notao
Wetgining
Noiso
Wetgining
Noisn
Wetgining
Nofan
0.5
0.5
0.5
01
0-
100
200
0
100
200
0
100
200
0
100
200
Rpoch
Fpods
Egods
Epoch
CityScapes [Sem. Seg.]
CityScapes [Part Seg.]
CityScapes [Dspp.]
CityScapes [3 Tasks]
Figure 9: Learning dynamics of Auto-A optimised on various choices of primary tasks in the auxiliary learning setup with Split architecture.
D Auto-À Learned Weightings for RLBench
The relationships vary more wildly in RLBBench tasks, where we can observe multiple weighting cross-over in different training stages.
Target Pst Monry Sofo
Buttio Unchwlls Pick Andi Lift
Pies And Piri Unibenla Picz Up Oup
Pirk Up
Pirk Up Uldeerll
0.15
Caoyping Pvt Up Unleelaa
0.36
Slale Blenk To Tazget
0.18
Slide Blods To Tanget
Wighhigg
Weaghtin
Waggttts
Stvn llbek To Taree
0.11
0.12
0.12
0122
0.12
0.22
0.5
0.1
153
11
9000
2.00 Derstions
300
Beenikas
Reach Target
Push Button
Pick And Lift
Pick Up Cup
Knife Chopping Board
0.11
Pat Money Safo
Pask Upp Undarlls
0J8
Piza Up Unaleela
Pui Mancy Tn Beft
335
Put Up Uededdla
Stzek Wine
Pok Undorlls
St2b libek To Svrae
Weighiig
0.16
Sbile Block Tacar:
Weighting
D.16
Do Larget
2.36
Kark Shipping
Wegghing
Puak Bulton.
0.35
lip Gaterla
Chopping Danl
0.14
4000
5000
0000
2002
011
4330
400
4000
5000
2000
100
Tientiins
Docationn
Take Money Out Safo
Put Money Safe
Pick Up Umbrella
Stack Wine
Slide Block Target
Figure 10: Learning dynamics of Auto-A optimised on each individual task in the auxiliary learning sctup for 10 RLBench tasks. We list 3 tasks with the highest task weightings in each setting.
18

---

Published in Transactions on Machine Learning Research (05/2022)
E Auto-À Learned Weightings for CIFAR-100
Interestingly, in the multi-task learning setting of multi-domain classification tasks (last row of Fig. 11 we can see a clear correlation between task weighting and single task learning performance, where the higher weighting is applied for more difficult domain (with low single task learning performance). For example, Peoplee and Vehicles 2', which have the lowest and highest single task learning performance respectively, were assigned with the lowest and the highest task weightings.
Aquatic Mammals
Food Containers
ID
ID 2
Fish
ID 3
Flowers
ID 4
Fruit and Vegetables
Household
Household furniture
TD 6
TD 7
ID
Insects
Electrical Devices
Large Carnivores
Large Man-made
large natural outdoor scenes
Large Omnivores
ID
ID
ID 12
Outdoor Things
and Herbivores
Modium-sized
Non-insect Invertehrates
ID 13
TD 14
TD 15
People
ID 16
Reptiles
Mammals
Small
ID 17
ID 18
Trees ID 19
Vehicles 1
ID 20
Vehicles 2
Mammals
Table 6:
The description of each domain ID in multi-domain CIFAR- 100 dataset.
CIFAR-100
Method
ID 1
ID 2
ID 3
ID 4
ID 5
ID 6
ID 7
ID 8
ID 9
ID 10
Singlo-TTast
68.65
81.00
82.34
83.71
89.10
88.72
84.75
85.88
87.07
90.15
Equal
73.59
82.36
79.78
83.94
89.14
87.03
83.73
85.87
86.67
89.86
Uncertainty
70.62
81.01
80.46
83.59
88.06
86.83
82.96
86.46
87.40
89.58
Multi-Task
DWA
71.54
82.12
81.60
23:22
89.70
86.64
82.57
17
87.34
90.19
Auto-A
74.00
83.96
81.30
83.57
88.69
87.85
84.57
87.75
88.04
92.03
GCS
71.05
82,27
80.31
83.36
07.07
85.94
83.05
86.80
87.54
89.34
Auxiliary- Task
Auto-A
75.70
84.39
82.71
84.64
90,23
88.02
85.52
87.36
89.04
92.20
Method
ID 11
ID 12
ID 13
ID 144
ID 15
ID 16
ID 17
ID 18
ID 19
ID 20
Single-Taase Task
89.76
84.88
90.33
84.41
55.37
75.84
72.79
75.37
91. 91.48
94.69
Equal
89.21
86.40
89.45
85.52
57.73
76.69
74.41
74.64
90.64
94.21
Uncertainty
89.80
87.07
89.76
85.64
54.14
75.62
74.08
74.62
90.83
89.54
Multi-Task
DWA
89.08
85.91
80.39
85.15
55.25
76.26
74.12
75.68
90.95
94.33
Auto-A
90.05
88.00
91.25
84.98
57.57
77.55
75.05
75.15
91.87
95.19
GCS
89.80
85.59
89.41
85.70
56.45
76.29
72.93
74.45
90.31
93.98
Auxiliary-Task
Auto-A
90.82
87.32
90.76
86.56
60.89
81.75
75.64
77.38
91.58
95.87
Table 7: The complete performance of 20 tasks in multi-domain CIFAR-100 dataset with multi-task and auxiliary learning methods.
19

---

Published in Transactions on Machine Learning Research (05/2022)
Train Tasks
ID
I
ID
ID4
ID5
ID6
ID7
ID8
ID9 10 I1
12
13 14
15
I6
ID
18
19
ID 20
ID ID
0.85
0.35
0.29
0.30
0.28
0.28
0.29
0.31 0.32
0.21
0.32
0.30
0.29
0.30
0.34
0.30
0.35
0.39
0.24
0.28
0.36
0.83
0.31
0.27
0.25
0.34
0.26
0.33
0.27 0.26 0.26
0.28
0.25
0.23
0.30
0.30
0.25
0.26
0.23
0.24
ID3
0.32
0.30
0.89
0.24
0.34
0.33
0.31
0.33
0.28
0.25
0.27
0.30
0.27
0.30
0.34
0.27
0.30
0.25
0.24
0.23
ID4
0.28
0.20
0.30
0.84
0.25
0.27
0.25
0.23
0.26
0.28
0.26
0.26
0.27
0.29
0.36
0.22
0.30
0.26
0.29
0.26
ID>
0.25
0.29
0.36
0.31
0.71
0.25
0.24
0.27
0.23
0.23
0.23
0.23
0.24
0.26
0.29
0.24
0.28
0.28
0.20
0.24
ID' ID6
0.22
0.26
0.25
0.25
0.23
0.68
0.25
0.22
0.21
0.22
0.25
0.28
0.26
0.22
0.30
0.28
0.25
0.29
0.26
0.23
0.29
0.33
0.25
0.25
0.24
0.30
0.72
0.26
0.24
0.28
0.25
0.27
0.27
0.25
0.28
0.24
0.26
0.31
0.23
0.27
ID
0.35
0.31
0.28
0.23
0.26
0.32
0.26
0.87
0.26
0.23
0.23
0.27
0.24
0.32
0.32
0.26
0.33
0.25
0.23
0.28
ID 11 ID 10 ID9
0.29
0.29
0.31
0.27
0.28
0.25
0.21
0.24
0.66
0.23
0.24
0.28
0.27
0.24
0.24
0.27
0.34
0.27
0.23
0.27
Primary Tasks
0.31
0.28
0.24
0.25
0.23
0.29
0.31
0.25
0.16
0.78
0.27
0.27
0.24
0.24
0.23
0.26
0.28
0.26
0.27
0.29
0.31
0.27
0.30
0.29
0.22
0.29
0.22
0.24
0.24
0.26
0.80
0.28
0.27
0.26
0.28
0.25
0.22
0.27
0.23
0.29
0.29
0.26
0.30
0.27
0.31
0.29
0.27
0.31
0.27
0.23
0.28
0.74
0.32
0.29
0.24
0.30
0.30
0.25
0.25
0.26
ID
0.30
0.28
0.23
0.24
0.25
0.26
0.22
0.24
0.30
0.21
0.17
0.26
0.63
0.22
0.26
0.31
0.23
0.32
0.22
0.25
14
ID
0.25
0.27
0.27
0.28
0.25
0.22
0.25
0.31
0.23
0.23
0.22
0.25
0.23
0.65
0.27
0.25
0.31
0.24
0.22
0.23
15
ID
0.32
0.31
0.38
0.30
0.34
0.27
0.27
0.37
0.32
0.26
0.26
0.31
0.30
0.29
1.13
0.28
0.32
0.31
0.25
0.27
16
0.31
0.30
0.24
0.27
0.21
0.27
0.29
0.29
0.22
0.28
0.22
0.28
0.21
0.28
0.33
0.80
0.30
0.23
0.23
0.24
ID 17 ID
0.30
0.32
0.31
0.29
0.27 0.30
0.32
0.30
0.32
0.22
0.27
0.29
0.29
0.31
0.28
0.30
0.85
0.31
0.20
0.19
0.33
0.27
0.26
0.28 0.23
0.27 0.26
0.29
0.27
0.23
0.28
0.28
0.22
0.29
0.31
0.25
0.25
0.90
0.21
0.27
ID
0.31
0.18
0.26
0.26
0.23
0.24
0.28
0.20
0.25
0.16
0.26
0.27
0.23
0.23
0.24
0.22
0.26
0.22
0.72
0.23
All ID 20 ID
0.27
0.28
0.22
0.28
0.22
0.23
0.19
0.28
0.23
0.24
0.22
0.23
0.19
0.22
0.20
0.22
0.21
0.23
0.28
0.63
0.86
0.81
0.91
0.77
0.77
0.82
0.89
0.89
0.77
0.83
0.87
0.76 0.76
0.85
1.14
0.95
1.03
0.94
0.71
0.68
0.2
0.4
0.6
0.8
1.0
Figure 11: Visualisation of learned weightings in Auto-A in auxiliary learning and multi task learning setup.
20


Reviews:
[{"review_id": "JxycOXM2dM", "reviewer": "Reviewer_P8PB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n- Learning \"dynamic task weighting\" using bi-level optimization seems new, although I did not extensively check for existing works that aim to achieve the same goal. \n- The proposed technique seems like a practical method to automatically find the weighting across the tasks in a multi-task learning scenario, given its strong performance over existing weighting methods.\n- The additional analyses of the relative scale of the weights over the course of training, and the task relationships are helpful in understanding how the method works. \n\nCons\n- The proposed method could be seen as a simple hyperparameter optimization method based on bi-level optimization. In fact, the proposed approach is a technique or trivially known to most researchers and practitioners working on multi-task learning. Thus, the paper does not provide any new insights for researchers working on multi-task learning. \n- Weighting task-specific loss is only one of many problems of multi-task learning, and what affects the performance of a multi-task learner the more, are other factors such as how to share the knowledge across tasks (for deep networks, how to construct the architecture). \n- The proposed method is validated against only a few existing works, missing comparison against relevant works such as [Lee et al. 16] and [Lin et al. 19].\n\n[Lee et al. 16] Asymmetric Multi-task Learning Based on Task Relatedness and Loss, ICML 2016\n[Lin et al. 19] Pareto Multi-Task Learning, NeurIPS 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a technique to automatically find the optimal weights to dynamically balance between the task-specific losses during multi-task learning. Specifically, the authors propose a bi-level optimization framework that aims to find the weight lambda for each task to minimize the validation loss of the primary task. Moreover, to avoid inefficiency resulting from the computation of second-order gradients, the authors further propose a finite different approximation method to approximate them. The proposed framework is validated on multiple benchmark datasets for dense prediction, multi-domain classification, and robot manipulation performance, and is shown to largely outperform baselines that adjust the task-specific weights using uncertainty or changes the weighting based on the relative scale of the loss.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Please provide comparison against [Lee et al. 16] and [Lin et al. 19].\n- Please describe the baselines more in detail. \n- Please apply the proposed loss weighting scheme to state-of-the-art MTL frameworks, that have specially designed architectures for sharing. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fx9B2QMJH0", "reviewer": "Reviewer_kiR7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "As I summarized above, the main strength of this work is empirical, and its applicability to the two mentioned setting of MTL. The paper is also well-written and easy to follow. However, conceptually, I have some concerns about the technical contribution of this work, and more generally, I feel there are several important claims made in the paper not well justified or clearly defined. \n\nSince the central claim of this work is a method used to dynamically discover the relationships between different tasks, and then adjust the weights accordingly, I think it is essential to first properly define the meaning of ``task relationship''. For example, in [1] the authors defined each task to be a vector that corresponds to the optimal linear classifier of a task, and then formally define the task relationship to be the correlation scores between two different task vectors. I am not saying that the authors need to follow the exact definition, but there should be a definition about the task relationship in order to (empirically) justify the claims made in the previous sections. \n\nFrom this perspective, it seems that the $\\lambda$ in Eq. (3) is implicitly referred to as the task relationship. However, there are several issues with this formulation. See my comments below. \n-   For the current optimization problem in Eq. (3), it is ill-defined: $\\theta$ should also appear in the optimization variables.\n-   Furthermore, it is unclear to me the following degenerate case could be avoided: one could make $\\lambda \\to 0$ and the problem is still feasible. However, when $\\lambda \\to 0$, the minimization problem becomes vacuous and it is unclear solving this optimization problem will lead to any meaningful model parameters. \n-   In each iteration of the optimization algorithm, the $\\lambda$ will be different. But if the authors implicitly treat $\\lambda$ to be the task relationship, this implies that the notion of task relationship is a local one, and is subject to change according to different model parameters. However, conceptually, the task relationship should only depend on the distributions of these tasks, and should be a global concept inherent to the tasks themselves, rather than the model being used to train these tasks. This distinction is important, as it can render all the claims made in the abstract as well as the introduction section non-justifiable. \n\n[1].    A Convex Formulation for Learning Task Relationships in Multi-Task Learning, Zhang et al.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contribution of this work is a method on learning the task combination weights dynamically in the setting of multitask learning (MTL). The proposed method could be applied in two variants of MTL. The first one is the classic setting where one hopes to obtain the optimal weighted loss; the second one is a special case of the first one, where only a subset of tasks are of main interest, whereas the others are treated as auxiliary tasks used to further improve the performance of the first one. The main strength of the proposed approach is empirical -- the authors demonstrated the effectiveness of the proposed approach on several benchmark datasets, and also did a good job in visualizing the learned task weights. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please see my comments above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G22PHp63PX", "reviewer": "Reviewer_BnS4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Guideline:**\n\nFollowing the TMLR guideline, I mainly check:\n\n(a) Are the claims made in the submission supported by accurate, convincing and clear evidence? and\n\n(b) Would some individuals in TMLR's audience be interested in the findings of this paper?\n\n**Strengths:**\n\n- This work is generally well-written and easy to follow.\n- Finding efficient optimization strategies is an important research topic in the current MTL community. This work has made a timely contribution in this direction.\n- The proposed method is simple and easy to implement. It is quite promising to see it can robustly outperform other methods on both CV and robotics problems (but there are also some concerns listed below).\n- The findings on task relationships obtained by Auto-\\lambda are also interesting.\n\n\n**Weaknesses:**\n\nAccording to the strengths listed above, I have no doubt that this work could be interesting to many TMLR audiences. However, there are also some concerns about the claims and design choices made in this work.\n\n**1. Lookahead Adaption during Training:** This work correctly mentions (Fifty et al., 2021) as a closely related work for learning the task relation, of which one limitation is its two-stage mechanism (e.g., search for the best task structure first, and then retrain the model). However, to my understanding, (Fifty et al., 2021) is one of the two ideas proposed in their previous work [1] for task relationship measuring. In [1], they also proposed a method to adjust the optimization dynamics among tasks with a lookahead strategy directly during *a single training stage*. The proposed approach in [1] can also be combined with other MTL methods.\n\nI do see the many differences between the method proposed in [1] and Auto-\\lambda, such as [1] selects gradient from a fixed set of candidates while Auto-\\lambda learns the weights. But since the core design concept of Auto-\\lambda is also the lookahead method, I think a more detailed discussion and comparison (similarity/difference) with the method proposed in [1] is needed.\n\n[1] Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Measuring and harnessing transference in multi-task learning. arXiv:2010.15413, 2020.\n\n**2. Missing Discussion on Gradient-based Hyperparameter Tuning:** If we treat the weights \\lambda as hyperparameters, the proposed auto-\\lambda method is closely related to the gradient-based hyperparameter adaptation approaches [2,3,4,5]. Especially, [2,5] also use gradient-based updates to adapt the hyperparameter (e.g., learning rate) at each iteration in an online manner, [6] considers the gradient-based hyperparameter adaption with the bi-level optimization formulation. What is the relation between auto-\\lambda and these methods?  \n\nIn addition, what if we allow different tasks to have different learning rates (might with fixed weights), and then use these methods to adapt the learning rate? Will they perform similarly with Auto-\\lambda?\n\n[2] L. B. Almeida, T. Langlois, J. D. Amaral, and A. Plakhov. Parameter adaptation in stochastic optimization. In D. Saad (ed.), On-Line Learning in Neural Networks. Cambridge University Press, 1998.\n\n[3] Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural computation 2000.\n\n[4] Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. ICML 2015.\n\n[5] Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. ICLR 2018.\n\n[6] Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back-propagation for bilevel optimization. AISTATS 2019.\n\n**3. Weight-based Methods v.s. Gradient-based Methods:** This work divides the multi-task optimization strategies into weight-based methods (e.g., Uncertainty/DWA/Auto-\\lambda) and gradient-based methods (e.g., GradDrop, PCGrad, CAGrad). However, the current gradient-based methods (such as CAGrad and RotoGrad[7]) usually claim they consider both the scale (weight) and direction for each task's gradient. In this sense, are the weight-based method also a gradient-based method (which only considers the scale)? \n\nWill it be more general to use the bi-level optimization and lookahead adaption to adjust the gradient (not just the weight) for each task?  \n\n[7] Adrián Javaloy, and Isabel Valera. RotoGrad: Gradient Homogenization in Multitask Learning. arXiv:2103.0263.\n\n\n**4. Single Gradient Update:** In Auto-\\lambda, the lookahead for model parameters \\theta and weight \\lambda are both updated by a single gradient update step. Will it be too myopic for a single step lookahead, and not sufficiently adjusted with only a single step update for \\lambda? How about conducting multiple gradient updates for both?\n\n**5. Comparison with Simple Method:** Some current works show that a simple MTL algorithm with random weight [8] or fixed linear scalarization with regularization [9] can robustly achieve comparable performance with the more sophisticated optimization approach (e.g., the weight-based and gradient-based methods), and they do not require any computation/memory overhead. These works also test their methods on computer vision (e.g., NYUv2, Cityscapes, PASCAL-Context, CelebA) and reinforcement learning (e.g., Meta-World) problems. It could be beneficial to clearly discuss and compare auto-\\lambda with those simple approaches.   \n\n[8] Baijiong Lin, Feiyang Ye, and Yu Zhang. A Closer Look at Loss Weighting in Multi-Task Learning. arXiv:2111.10603.\n\n[9] Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and M. Pawan Kumar. In Defense of the Unitary Scalarization for Deep Multi-Task Learning. arXiv:2201.04122.\n\n**6. More Analysis with Uniform Weighting:** According to Table 5, all weight-based/gradient-based methods (and their combination) can always outperform the vanilla approach with equal weights. However, the results in other works (e.g., [8][10]) show that many weight-based/gradient-based methods might indeed have poorer performance. Where does this inconsistency come from? Will the relative performance be very sensitive to the dataset/model/hyperparameter setting?\n\n[10] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. TPAMI 2021.\n\n**7. Weight-based/Gradient-based Method Combination:** It is quite interesting to see this combination can further improve the MTL performance (except for PCGrad + DWA). In the combination, is the gradient-based approach applied for both lookahead (eq 4) and the actual parameter update (eq 6)? Will it actually violate/change the assumption/setting for the gradient-based methods (e.g., the scale adaption step in CAGrad and RotoGrad [7]). \n\n**8. Computational Overhead:** This work explicitly discusses its limitation on the training speed. I appreciate the clear discussion and wonder if allowing other methods to train longer (e.g., the same wall-clock time or forward/backward passes with Auto-\\lambda) can close the performance gap.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes auto-\\lambda, a simple meta learning based approach, to adaptively adjust the weights for different tasks in both the multi-task and auxiliary learning setting. It formulates the weight adjustment as a bi-level optimization problem, where the upper level is to optimize the weights \\lambda with respect to the validation loss, and the lower level is to optimize the model parameters \\theta to the training loss. At each update step, the proposed algorithm first calculates the weights for each task by solving the bi-level optimization problem (with a simple one-step gradient update), and then updates the model parameters with the obtained task weights.\n\nIn practice, Auto-\\lambda simply samples data from a different batch in the same training set (rather than a truly held-out validation set) to calculate the validation loss for weight adjustment. This work uses a simple finite difference approximation to avoid computing the higher-order gradients, and uses stochastic task sampling to save memory for problems with a large number of tasks. Experimental results on CV and robotics applications show auto-\\lambda can outperform other well-known methods for both multi-task learning and auxiliary learning problems. The dynamic weights obtained by Auto-\\lambda can also be used for analyzing the relationship among tasks in multi-task learning.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I believe the critical changes will mainly depend on how the authors tackle the **concerns raised above**. Generally, I will expect:\n- more discussion/comparison with related works;\n- deeper analyses of the proposed Auto-\\lambda method (might with additional experiments);\n\nto support the claims made in this work.\n\nIn addition to the critical changes, I am also interested to see more discussions or experiments/analyses on:\n\n- The trade-off (performance v.s. training speed) between exact the second-order gradient and the approximate finite difference to solve the bi-level optimization problem at each step;\n- The trade-off for stochastic task sampling. Will taking K' << K significantly slow down the training process (in terms of #epochs)? Will the sampling make the training unstable and indeed hurt the final performance (even with the same computation budget)?\n\nMinor Issues:\n\n- The code is not attached although it says so in the introduction. Correct me if I miss it.\n\n- The year of CAGrad (Liu et al 2018) should be 2021?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hrULt1jgLu", "reviewer": "Reviewer_QXX5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is my view that this is an extremely strong paper. The method itself is well explained and draws heavily from successes in gradient based meta learning to improve multi-task learning performance. It also represents a possible paradigm shift from \"how can we maximize performance on all tasks training together\" to incorporate greater flexibility in automatically deciding which tasks would be helpful for the training of one or more primary tasks. \n\nThe empirical analysis is robust, and although I am unable to locate their code in the supplementary material, I would like to ask if the authors will commit to open sourcing it following conference reviews. Evaluating their method on 4 separate (and somewhat robust/realistic datasets -- i.e. not just MNIST/CIFAR-10), in addition to their comparison with accepted and popular MTL task weighing methods (i.e. Uncertainty Weights, DWA, GCS, etc.) is highly appropriate. \n\nThe ablation experiments were also very compelling. The comparison with a \"noise task\" highlights their method's capabilities in \"tuning out\" tasks which are unhelpful in the training of others in the network; a formulation which presents difficulty to other task weighing algorithms. The authors also motivate their intuition that task relationships are dynamic, and while I think many mtl researchers/practitioners would agree with this statement given prior work investigating this topic, Auto-$\\lambda$ put a new spin on this concept by explicitly comparing against optimal task-grouping methods in the auxiliary task learning domain. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose Auto-$\\lambda$, a method which dynamically adjusts the weights of loss functions in a multi-task learning paradigm via an optimization framework similar to gradient based meta learning. In contrast with prior work which focuses on balancing all tasks equally, Auto-$\\lambda$ considers the setting where auxiliary tasks may be used to augment the performance of one or more primary tasks. This change of perspective greatly increases the number of real-world problems Auto-$\\lambda$ may be applied to, and in my view, represents an important (and perhaps necessary) shift in the development of future multi-task learning methods. The author's proposed method appears well motivated -- grounded in meta learning, multi-task training augmentation, and multi-task grouping algorithms -- and the empirical analysis on 4 separate datasets is convincing. Finally, the authors offer several ablation experiments (i.e. capacity to handle a \"noise\" task, asymmetry in task relationships, etc.) in attempting to contribute the community's knowledge of multi-task learning training dynamics and underlying structure. \n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "It seems like there may be an issue with Eq. 7?\n\nRepresenting the gradient w.r.t. $\\lambda$ after first updating the parameters $\\theta$, we have (line 1):\n\n$\\nabla_{\\lambda}\\mathcal{L}^{pri}(\\theta - \\alpha \\nabla_{\\theta} \\mathcal{L}(\\theta, \\lambda), 1)$\n\nWith substitution $\\theta' = \\theta - \\alpha\\nabla_{\\theta}\\mathcal{L}(\\theta, \\lambda)$, line 2 becomes:\n\n$\\nabla_{\\lambda}\\mathcal{L}^{pri}(\\theta - \\alpha\\nabla_{\\theta}\\mathcal{L}(\\theta, \\lambda), 1) - \\alpha\\nabla^2_{\\theta, \\lambda}\\mathcal{L}(\\theta, \\lambda)\\nabla_{\\theta'}\\mathcal{L}^{pri}(\\theta', 1)$\n\nand then $- \\alpha\\nabla^2_{\\theta, \\lambda}\\mathcal{L}(\\theta, \\lambda)\\nabla_{\\theta'}\\mathcal{L}^{pri}(\\theta', 1) = 0$?\n\nIt also may be helpful to make it explicitly clear that the second argument to $\\mathcal{L}^{pri}$ is a \"1\" representing that the $\\lambda$'s are set to a constant value in the outer loss function when you define the notation above Eq. 7. \n\nThis is a personal nitpick of mine -- and you can entirely ignore this feedback if you disagree with it -- but under section 5 Experiments, you have \"To validate the generalization of ...\". This phrasing may sound like you've come to the conclusion that Auto-$\\lambda$ works before evaluating the method on experiments. Simply changing \"validate\" to \"evaluate\" may change the tone of this opening paragraph to be more inquisitive (i.e. we think this could work, but let's evaluate it on several datasets to find out). ", "broader_impact_concerns": "I have no Broader Impact concerns related to this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dAUzQdlsQa", "reviewer": "Reviewer_EZCk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nThis paper proposed to learn dynamic task relationships considering the different primary tasks. Then the MTL and auxiliary learning can be unified., which is interesting and efficient in MTL. Extensive experimental results have been conducted on two domains with different settings. The results demonstrate the effectiveness of the proposed method.\n\nCons:\n1. The position of this paper is not clear.\n\na). Plain MAML takes equal importance for all tasks involved, which may not be true in real applications. However, there are many works towards learning the task scheduler into meta-learning, e.g., [1,2]. It would be better to provide a comprehensive review on this part.\n\nb). The dynamic task relationship is not well defined and discussed. This paper aims to learn the dynamic task relationships. However, this assumption is not well proved in this paper, either in theoretical or experimental. What are the disadvantages when using the fixed task relationships, and what are the challenges when addressing this problem?\n\nBesides, In the introduction, considering the time consumption in the existing task clustering approaches, the Auto-λ is proposed to learn the dynamic task relationships with the gradient descent. There lacks an efficiency comparison between these methods, i.e., gradient-based methods and the task grouping approaches, and the proposed Auto-λ in the experiments. Moreover, it is unclear why these gradient-based methods cannot be applied to learn the dynamic λ.\n\n2. The technical contribution is limited.\n\na). All components in this paper are well studied, e.g., single-objective optimization in MTL, gradient-based task relationship design, bi-level optimization, etc. Combining these techniques is straightforward and easy to obtain. The insights of combining them are not clear.\n\nb). The optimization of task relationships λ and the architecture parameters θ are disentangled, and then it is transformed into a bi-level optimization problem, which is trivial and easy to be designed. In section 2, there lacks a comparison between Auto-λ and the existing gradient-based methods mentioned, and the comparisons of the single objective in Eq. (1) and the bi-level optimization mechanism in Eq. (3) are ignored. It is unclear what is the key contribution of the proposed algorithm.\n\n3. Correctness of the algorithm. Existing methods [3,4] point out the efficiency and reliability issues in the bi-level optimization. Therefore, it would be better to provide the theoretical analysis on the correctness of the gradient calculation.\n\n\nDetailed Comments:\n1. As introduced in Section 2, the multi-task architectures can be classified into three classes. While the NAS split is ignored in Tb. 1. \n2. It would be better to make a comparison with the existing few-shot learning methods in Section 2.\n3. In Table 2, the uncertainty method in the Auxiliary task is missed compared with Tb. 1.\n4. In Fig 3, it is not accurate to say that the task relationship is consistent based on the same optimization strategy, since only the proposed optimization algorithm is used in this figure. It would be better to provide the task relationships learned by different methods.\n5. The results of Fig. 5 cannot be aligned with the learned relationships in Fig. 3 except for the used example.\n\n\nRefs\n[1] Meta-learning with an Adaptive Task Scheduler.\n[2] Probabilistic Active Meta-Learning\n[3] A Single-Timescale Method for Stochastic Bilevel Optimization.\n[4] A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singleton\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed one method Auto-λ to learn the dynamic task relationships in the multi task learning. It assumes the task relationships should be dynamic w.r.t. the primary tasks, then MTL and auxiliary learning are unified into one framework.  A meta-loss formulation is designed to optimize the parameters in this framework, i.e., multi-task architectures and the task relationships λ. The experimental results demonstrate that both MTL and auxiliary learning problems can benefit from this optimization framework.\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please check Cons and \"Detailed Comments\" in \"Strengths And Weaknesses\".", "broader_impact_concerns": "not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to automatically and dynamically find optimal weights for tasks in the multiple task learning setting. A bi-level optimization framework has been used to update the weights. The proposed method has been empirically verified on several datasets and applications. Outstanding empirical performance is the main strength of the proposed method. 
Overall, the claims are supported by clear empirical evidence. The good empirical results will interest many practitioners and some researchers in the community.

However, there are concerns raised regarding the soundness of claims. Several important claims made in the paper are weak or do not stand. For example, the proposed gradient-based hyperparameter tuning of task-specific losses in a multi-task learning objective is not new and very similar approaches have been exploited in existing works on MTL such as [Alesiani et al. 20], and meta-learning [Lee et al. 20, Kilamsetty et al. 22], and the related work mentioned by Reviewer BnS4. The authors claim that all existing MTL methods consider the task relationships as static, which is not true. [Lee et al. 16, Lee et al. 18, Nguyen et al. 21] also learn knowledge transfer weights in a dynamic manner. Reviewer KiR7 points out that the definition of the task relationship should be more rigorously discussed. The authors’ reply that the learned task relationships are global based on an observation of Fig. 3 is not acceptable or convincing.

We think the concerns would be addressed via a minor revision and would provide an opportunity for the authors to carefully revise the paper.

Reference:
[Alesiani et al. 20] Towards Interpretable Multi-Task Learning Using Bilevel Programming, ECML PKDD 2020 [Lee et al. 20] Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks, ICLR 2020 [Killamsetty et al. 22] A Nested Bi-level Optimization Framework for Robust Few Shot Learning, AAAI 2022 [Lee et al. 16] Asymmetric Multi-task Learning Based on Task Relatedness and Loss, ICML 2016 [Lee et al. 18] Deep Asymmetric Multi-task Feature Learning, ICML 2018 [Nguyen et al. 21] Clinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Learning, AAAI 2021.


Decision: Accept (poster)