Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models
Authors: Nikolaos Dimitriadis
Pascal Frossard
François Fleuret
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-Task Learning
multitask learning
mode connectivity
loss landscape
pareto optimal
pareto frontier

Abstract: In Multi-Task Learning, tasks may compete and limit the performance achieved on each other rather than guiding the optimization trajectory to a common solution, superior to its single-task counterparts. There is often not a single solution that is optimal for all tasks, leading practitioners to balance tradeoffs between tasks' performance, and to resort to optimality in the Pareto sense. Current Multi-Task Learning methodologies either completely neglect this aspect of functional diversity, and produce one solution in the Pareto Front predefined by their optimization schemes, or produce diverse but discrete solutions, each requiring a separate training run. In this paper, we conjecture that there exist Pareto Subspaces, i.e., weight subspaces where multiple optimal functional solutions lie. We propose Pareto Manifold Learning, an ensembling method in weight space that is able to discover such a parameterization and produces a continuous Pareto Front in a single training run, allowing practitioners to modulate the performance on each task during inference on the fly. We validate the proposed method on a diverse set of multi-task learning benchmarks, ranging from image classification to tabular datasets and scene understanding, and show that Pareto Manifold Learning outperforms state-of-the-art algorithms.


23 as well, c.g., lower inference times and memory requirements. However, building machine learning models presents multifaceted host of decisions for multiple and often competing objectives, such 25 as model complexity, runtime and generalization. Conflicts arise since optimizing for one metric of- ten leads to the deterioration of other(s). A single solution satisfying optimally all objectives rarely Under review as a conference paper at ICLR 2023
PARETO MANIFOLD LEARNING: TACKLING MULTIPLE TASKS VIA ENSEMBLES OF SINGLE-TASK MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
In Multi-Task Learning, tasks may compete and limit the performance achieved on each other rather than guiding the optimization trajectory tion, superior to its single-task counterparts, There is often not a single solution that is optimal for all tasks, leading practitioners to balance tradeoff's between tasks" performance, and to resort to optimality in the Pareto sense. Current Multi- Task Learning methodologies either completely neglect this aspect functional diversity, and produce one solution in the Pareto Front predefined by their op- timization schemes, or produce diverse but discrete solutions, each requiring a separate training
reviewersl: We one
2
common solu-
"actes" to make the changes more vis- ible. For mincr things such dightty changing plot da not write anything.
6
8
In this paper, we conjecture that there exist Pareto Sub- spaces, i.e., weight subspaces where multiple optimal functional solutions lie. We propose Pareto Manifold Learning, ensembling method in weight space that is able to discover such a parameterization and produces a continuous Pareto Front single training run, allowing practitioners modulate the performance on each task during inference on the fly. We validate the proposed method on di- verse set of multi-task learning benchmarks, ranging from image classification tabular datasets and scene understanding, and show that Pareto Manifold Learning outperforms state-of of-the-art algorithms.
10
11
12
13
14
15
16
17
18
INTRODUCTION
19 In Multi-Tast Learning (MTL), multiple tasks are learned concurrently within a single model, striv- ing towards infusing inductive bias that will help outperform the single-task baselines. Apart from 20
21 the promise superior performance and some theoretical benefits (Ruder, 2017), such as generaliza-
22 tion properties for the learned representation, modeling multiple tasks jointly has practical benefits
27 exists and practitioners must balance the inherent trade-offs.
In contrary single-task learning, where metric governs the comparison between methods (e.g.,
29 top-1 1 accuracy in Imag)Net)), multiple models can be optimal in Multi-Task Learning; e.g., model
30 yields superior performance on task A compared to model Y, but the reverse holds true for task B;
31 thus, there is not single better model among the two. This notion of tradeoff's is formally defined
32 as Pareto optimality. Intuitively, improvement on an individual task performance can come only at
33 the expense of another task. However, there exists no framework addressing the need for efficient construction of the Pareto Front, i.e., the set of all Pareto optimal solutions.
34
35 Recent methods in Multi Task Learning casted the problem in the lens of mujtti-beeective optimiza-
36 tion and introduced the concept of Pareto optimality, resulting different mechanisms for comput-
37 ing the descent direction for the shared parameters. Specifically, Sener & Koltun (2018) produce a biased to- 38 single solution that lies on the Pareto Front. As an optimization scheme, however, wards the task with the smallest gradient magnitude, as argued in Liu et al. (2020). Lin et al. (2019) expand this idea and, by imposing additional constraints on the objective space to produce multiple solutions the Pareto Front, each corresponding to different user-specified tradeoff. Finally, the work by Ma et al. (2020) proposes an orthogonal approach that can be applied after training and starts with discrete solution set and produces a continuous set (in weight space) around each
40
42

---

Under review as a conference paper at ICLR 2023
Prone
Stepent Stepesttno
Stegin
Step=200000
4.
Figure 1: Illustrative example following Yu et al. (2020); Navon et al. (2022). We present the optimization trajectories in loss space starting from different initializations (black bullets) leading to final points (crosses). Color reflects the iteration number when the corresponding value is achieved. To highlight that our method (PML) deals in pairs of models, we use blue and red differentiate them. Dashed lines show intermediate results of the discovered subspace. While baselines may not reach the Pareto Front or display bias towards specific solutions, PMI. discovers the entire Pareto Front in single run and shows superior functional diversity.
44 lution, while the overall Pareto Front is continuous only in objective space as the union of the local (weight-space continuous) Pareto Fronts. As consequence, the memory requirements grow linearly 46 with the number of models stored. Navon et al. (2021); Lin et al. (2021) use hypemnttorks to pro- duce a Pareto Front in single training run, but this approach has limited scalability and introduces additional design choices.
48
[All reviewerst: Added shovt clarifictioon about wodk. More details are also provided in "Related week"" soction.
In this paper, we conjecture that we can actually produce a subspace with multiple Pareto stationary points in the Multi-'Task Learning setting with the hypothesis that local optima (produced by different runs or sharing training steps) can be found in close proximity and are connected by simple paths. This is motivated by the recent advancements in single task machine learning that have explored the geometry of the loss landscape and shown experimentally that local optima are connected by simple paths, even linear ones in some cases (Wortsman et al., 2021; Garipov et al.,
50
51
52
53
54
55 2018; Frankle et al., 2020; Draxler et al., 2018). We assume that, when the problem has multiple objectives, it acquires new dimension relating to the number of tasks. Concretely, there are multiple loss landscapes and a solution that satisfies users' performance requirements must lic in the intersection of low loss valleys (for all tasks).
56
57
59
Building upon our conjecture, we develop a novel method, Pareto Manifold Learning, which casts Multi-Task problems as learning ensemble of single-task predictors by interpolating among (en- semble) members during training. By operating in the convex hull of the members' weight space, each sangle-task model infuses and benefits from representational knowledge to and from the other members. During training, the losses are weighted in tandem with the interpolation, i.e., a mono- tonic relationship is imposed between the degree of single-task predictor participation and the weight of the corresponding task loss. Consequently, the ensemble as a whole engenders a (weight) subspace that explicitly encodes tradcoff's and results in continuous parameterization the Pareto We identify challenges in guiding the ensemble to such subspaces, designated Pareto sub- spaces, and propose solutions regarding balancing the loss contributions, and regularizing the Pareto properties of the subspaces and adapting the interpolation sampling distribution.
61
63
85
86
67
Front.
68
69
70
Experimental results validate that the proposed method is able to discover Pareto Subspaces, and out- performs baselines on multiple benchmarks. Our training scheme offers two main advantages. First, 72 enforeing low loss for all tasks on a linear subspace implicitly penalizes curvature, which has been linked to generalization (Chaudhari et al., 2017), benefitting all tasks' performance. Second, the al- gorithm produces subspace of Pareto Optimal solutions, rather than a single model, enabling prac- titioners to handpick during inference the solution that offers the tradeoff that best suits their needs.
71
74
76
RELATED WORK
Multi-Task Learning. Learning multiple tasks in the Deep Learning setting (Ruder, 2017; Craw- shaw, 2020) is usually approached by architectural methodologies (Misra et al., 2016; Ruder et al., 2019), where the architectural modules combined in several layers to govern the joint repre- sentation learning, or optimization approaches (Cipolla et al., 2018; Chen et al., 2018), where the architecture is standardized to be an encoder-decoder()) for learning the joint and task-specific rep-
78
80
81
2

---

Under review as a conference paper at ICLR 2023
resentations, respectively, and the focus shifts to the descent direction for the shared parameters. We focus on the more general track of optimization methodologies fixing the architectural struc- ture to Shared-Bottone (Caruana, 1997). The various approaches focus finding suitable descent direction for the shared parameters. The optimization methods can be broadly categorized into loss- balancing and pradient-balanciin (Liu et al., 2020). For the former, the goal is compute an appropriate weighting scheme for the losses, e.g., the losses can be weighted via task-dependent ho- moscedastic uncertainty (Cipolla et al., 2018), by enforcing task gradient magnitudes to have close norms (Chen et al., 2018). The latter class of methodologies manipulate the gradients so that they satisfy certain conditions; projecting the gradient of (random) task on the normal plane of another so that gradient conflict is avoided (Yu et al., 2020), enforcing the common descent direction to have equal projections for all task gradients (Liu et al., 2020), casting the gradient combination as a bargaining game (Navon et al., 2022).
83
85
87
88
89
reviewers]: Removed dis- cussion about Siner Koltun (2018) from this point, since the is also discussed intoduation the pangraph.
90
91
92
93
94
Multi- Muski-Task Learning for Pareto Optimality, The authors (Sener & Koltun, 2018) were the first to view the search for a common descent direction under the Pareto optimality prism and employ the Multiple Gradient Descent Algorithm (MGDA) (Désidéri, 2012) in the Deep Learning context. However. MGDA did not account for task preferences and the solutions yielded for various initial- izations in a synthetic example resulted in similar points in the Pareto Front (Lin et al., 2019). By solving slightly different formulation of the multi-objective problem, they are able to systemati- cally introduce task trade-offs and produce a discrete Pareto Front. However, this approach requires as many training runs as the stated preference combinations and the optimization process for each training step of each run introduces a non-negligible overhead. The work in (Ma et al., 2020) pro- poses an orthogonal approach for Pareto stationary points; after model fitted with any Multi-Task Learning method and has converged to point (seed) in parameter space, separate phase seeks other Pareto stationary points in the vicinity of the seed. The convex hull of these points guar- anteed to lie in the Pareto Front. But training still needs to occur for every seed point, the separate phase overhead grows linearly with the number of additional models, and the Pareto Front is not continuous across seed points in parameter space. Navon et al. (2021) and Lin et al. (2021) employ hypernetworks to continuously approximate the Pareto Front in a single run, which introduces ad- ditional design choices. Ruchte & Grabocka (2021) address the scalability issues of hypernetworks by augmenting the feature space with the preference vector. Raychaudhuri et al. (2022) employ a second hypernetwork to also modulate the architecture of the target network addressing.
95
96
98
99
100
101
102
103
104
105
106
107
109
110
111
112
[All reviewers]: Added peior
work.
113
Ensemble Learning and Mode Connectivity. Apart from Multi- Multi-Task Learning, our algorithm is methodologically tied to prior work in the geometry of the neural network optimization landscapes. The authors in (Garipov et al., 2018: Draxler et al., 2018) independently and concurrently showed that for two local optima 07, 0 produced by separate training runs (but same initializations) there exist nonlinear paths, defined as connectors by Wortsman et al. (2021), where the loss remains low. The connectivity paths can be extended include linear in the case of the training runs sharing some part of the optimization trajectory (Frankle et al., 2020). These findings can be leveraged to train a neural network subspace by enforcing linear connectivity among the subspace endpoints (Wortsman
114
115
116
117
118
119
120
et al., 2021). Appendix J discusses more related work regarding ensemble learning and flat minima.
122
3 PROBLEM FORMULATION
123
Notation.
We use bold font for vectors I, capital bold for matrices X and regular font for scalars T is the number of tasks and m is the number of ensemble members. Each task t [T] has a loss
L. The overall multi-task loss is L = [C,...., Lr) w E Ar C IR is the weighting scheme for the tasks, i.c., the overall loss is calculated as L w" L ΣT 1 a,C, Each member & [m] is associated with parameters 02 E N and weighting
125
AT.
Preliminaries. Our goal lies in solving an unconstrained vector optimization problem of minimiz- 128
ing (yy 9) [Cr (v1, Φ1) ....... Cr (r)) where L, corresponds to the objective function for the th task, e.g., cross- loss in case of classification. Constructing an optimal solution for all tasks is often unattainable due to competing objectives. Hence, alternative notion of optimality
130
131
132
is used, as described in Definition 1.
Definition 1 (Pareto Optimality). A point dominates point y if C, (x) S Le (w) for all tasks t E T and L (x) + L (y). Then, a point x is called Pareto optimal if there exists no point y that dominates it. The set of Pareto optimal points forms the Pareto front Pt.
134
135
3

---

Under review as a conference paper at ICLR 2023
4/2
00.6
0.2
01
[0.2]
0. 10g
Figure 2: A representation of encoding parameter space for T tasks. Each node cor- responds to tuple of parameters and weighting scheme (0... w,) RN x Ar. The blue dashed frame shows the model, c.g., shared-bottom architecture, implemented by the parameters 6, of each node. For each training step, sample Ar and construct the weight combination = θ 0.6 0 + 0.2 0 0.2 03.
The vector loss function is scalarized by the vector [0, 1]" to form the overall objective w L. Without loss of generality, we assume that u lies in the T-dimensional simplex År by imposing the constraint || = =i= Wt = 1. This formulation permits to think of the vector of weights as an en- coding of task preferences, c.g., for two tasks letting
[0.8, 0.2] results in attaching more impor- tance to the first task. Overall, the Multi-Task Learning problem can be formulated within the Empir- ical Risk Minimization (ERM) framework for preference vector w and dataset D {(x, }= as:
min ((.yy>D LL (y. f (æ:; 0))
(1)
136
Our overall goal is to discover a dinenimensional parameterization weight space that yields (continuous) Pareto Front in functional space. This desideratum leads us to the following definition:
137
138
Definition 2 (Pareto Subspace). Let T be the number of tasks, X the input space, y the multi- task output space, R C R.N. the parameter space, f X R - y the function imple- mented by a neural network, and
140
x y + -R be the vector loss. Let {θ, E R 1 [T]} be a collection of network parameters and S the corresponding conver envelope, i.e., and 2 0, Vi Consider the dataset D (Dx, Dy). Then, the subspace is called Pareto if its mapping to functional space via the network architecture f forms Pareto Front P Lff(Dx;; (), {l: L(f(Dx: Dy)) VO S).
S = DT {
a,,O; Ei=
143
METHOD
145
We seek to find collection of m neural network models, of identical architecture, whose linear combination in weight space forms a continuous Pareto Front in objective space. Model i corre- sponds to tuple of network parameters 0 and task weighting w; and implements the function [:: Oz). We impose connectivity among models by modeling the subspace in the convex hull of the ensemble members. Section 4.1 presents the core of the algorithm, and in Section 4.2 discuss various improvements that address Multi-Task Learning challenges.
147
149
150
152
4.1
PARETO MANIFOLD LEARNING
Let [,,, 02. mm be an m N matrix storing the parameters of all models, W [w1, wm]" be T matrix storing the task weighting of ensemble members. By designing the subspace as a simplex, the objective now becomes:
B(r.))~D [Ea~P [a W (v. (x: ae))]]
153 where P the sampling distribution placed upon the simplex. In the case where the en- semble members single-task predictors (w is one-hot) and the number of tasks coin- 155 cides with the number of ensemble members (m = T), the matrix of task weightings W
196 an identity matrix and Equation simplifies to EEzyy> [Eap (a (y, f (æ; a+)]]] =
157 E(a.y)D Ea-p Σn CeLe (y.s f Er an0,))] By using the same weighting for both
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Forlttoola
Input :
matrix of model parameters e = 0 02 0r vector loss function L, train set D,
network f. distribution parameters window W e N, regularization coefficient (A > 0)
Initialize each 0, independently
2 for batch (æ: y) S do
3
V + 2
for i E {1, 2, ..... W) } do
sample a; Dir(p)
V Vua;
0: a a
construct network in convex huil ensemble members [ai)) <- criterion (f(æ; 0:), // compute losses
L(a.) (L)( end
10
construct multi-forward graphs G. = EV) for all tasks 1 E [T]
1f see Section 4.2
[RCVLeио texk]: changed line 11 link to section
R R+ log (1²) Eoos,yees exp exp (C:(a)) (.(a)))+)
Ei=,, Lual 4 EE= a. L(a:) R
11
multiforward regularization
13
Backpropagate Local Gradient descent on θ
14
end
the losses and the ensemble interpolation, explicitly associate models and task losses with to-one correspondence, infusing preference towards one task rather than the other and guiding the learning trajectory to a subspace that encodes such tradeoff'.
159
160
161
Algorithm presents the full training procedure for this ensemble of neural networks, containing modifications discussed in subsequent sections. Figure showcases the algorithm toy example. Concretely, at each training step a random is sampled and the corresponding convex combination of the networks constructed. This procedure is shown in Figure 2. The batch is forwarded through the constructed network and the vector loss is scalarized by a as well. The procedure is repeated W times at each batch (see Section 4.2) and regularization term penalizing non-Pareto stationary points is added (line 11).
162
(Reviewer qetX): addressing
wenimess mentioned Fig.
163
in main
164
(Reviewer qetX]: addresing
165
werkness - mentione Fig.
test
186
[Reviewer qexX; Addressing weakness 4: Changed citabon from Figure
Claim 3. Let (0r E R : t E [T]} be the optimal ensemble parameters retrieved at the end of training by Algorithm I and let S be the their convex hull. Then S is a Pareto Subspace.
168
170 Note that we have chosen a convex hull parameterization of the weight space, but there are other options, such Bezier curves other nonlinear paths (Wortsman et al., 2021; Draxler al., 2018). However, the universal approximation theorem implies no loss of generality for our design choice. In practice, Claim is validated by uniformly sampling the discovered subspace and the definition of a Pareto Subspace is relaxed to conform to the nonconvex settings of Deep Learning, i.c., points are called Pareto optimal if the characterization holds in an open neighborhood rather than globally.
172
173
174
175
(Reviewer qetX): Addressing weakness about Pareto opt- mality (wo- Jonmoxity
178
4.2
REGULARIZATION AND BALANCING
177
Loss and gradient balancing schemes. A common challenge in Multi-Task Learning is the case where tasks have different loss scales, e.g., consider datasets with regression and classification tasks such UTKF Then, using the same weighting a for both the losses and the weight ensem- bling, as presented in Equation 2, the easiest tasks are favored and the important property of scale invariance is neglected. To prevent this, the loss weighting needs to be adjusted. Hence, we pro- pose simple balancing schemes: one loss and one gradient balancing scheme, whose effect is to warp the space of loss weightings. While gradient balancing schemes applied on the shared parameters, loss balancing also affects the task-specific decoders, rendering the methodologies can be complementary. To avoid cluttering, balaneing schemes are not presented in Algorithm
178
180
182
184
185
196
In terms of loss balancing, we use a lightweight scheme of adding a normalization coef- ficient each loss term which depends on past values. Concretely, let W Σ, be positive integer and (m(7)) be the loss of task m in step To- Then, the regularization coef-
187
188
ficient is I(TO: W) W I Lm +1 7) for To 2 W resulting in the overall loss Ce(To) For gradient balancing. let gr be the gradient of task
Lintal TT, L = Σt= Or (ToW))
180
5

---

Under review as a conference paper at ICLR 2023
L2
L2
(aaa))
fala
a
αq
(i(ss)
L2(e2)
L
Visual explanation of multiforward regularization, presented in Equation 3. The subfigures depict the loss values for various weightings a; = 1 04,2] Optimal lies in the origin. We assume that 01,1 > ... > 05,1. Green color corresponds to Pareto optimality, (Left) all sampled weightings in the Pareto Front and the regularization term is zero. (Right) The red points and, therefore, the regularization term penalizes the violations of the monotonicity constraints for the appropriate task loss: Or and a4 violate the L1 and L2 orderings w.r.t. 03 since 0221 0x,,1 / Li (022) < Li (ara) and 0x4,2 >> 03,2 + (2((4)) < (2(3s).
Figure 3:
optimal
191 1 E [T] w.r.t. the shared parameters. Previously, the update rule occurred with the overall gradient Grotat a G a [ grl. We impose unit f--norm for gradients and perform the update with Gtotal = a TG =
193
Ørl where gr
St 11
194 Improving stability by Multi-Forwart batch regularization. Consider two differentt weightings 0n and Q2 Ar-1- Without loss of generality [01]0 Q1 [a=goo dg- Then, ideally, the interpolated model closer to the ensemble member for task 1 has the lowest loss on that task, i.e., we would want the ordering L1 (aa) < (az)), and, equivalently for the other tasks. Furthermore, if a = c, s/r-r/...... /1] only one member essentially reaps the benefits of the gradient update and moves the ensemble towards weight configurations more suitable for task but, perhaps dele- terious for the remaining ones. Thus, we propose repeating the forward pass W times for different random weightings {a, [e allowing the advancement of all ensemble members concurrently in a coordinated way. By performing multiple forward passes for various weightings, we achieve lower discrepancy sequence and reduce the variance of such pernicious updates.
196
198
199
200
201
202
203
We also include regularization term, which penalizes the wrong orderings and encourages the subspace to have Pareto properties. Let be the set of interpolation weighs sampled in the current batch V {ox (amm,1 Ow..2,.... Cm,T) E Ar (rr-)we[wv wE[W]- Then each task defines the directed graph G. = (V, &g) where & {(a,, az) E x V aia aj,tt. The overall loss becomes:
W
T
Liotal Σ
a [(a)) +X Σ log
Σ (a:.a))E&
(3)
204
The current formulation of the edge set penalizes heavily the connections from vertices with low values. For this reason, we only keep one outgoing edge per node, defined by the task lexicographic order, resulting in the graph GLEX (V, ELEX) and |LEE = W 1, Vi E [T'] Note that the regu-
205
206
207
larization term is convex as the sum of log-sum-exy, terms. If no violations occur, the regularization term is zero. Figure offers a visual explanation of the regularization term.
208
209
The role of sampling. Another component of Algorithm is the sampling imposed on the convex hull parameterization. During training, the sampling distribution dictates the loss weighting used and, hence, modulates the degree of task learning. A natural choice is the Dirichlet distribution Dir(p) where p E R²", are the concentration parameters, since its support is the T-dimensional simplex AT. For p plr, the distribution is symmetric; for concentrated near the ensemble members, for p > 1 ii near the centre and for p = it corresponds
210
212
the sampling is more
214
to the uniform distribution. In contrast, for P1 # P2 the distribution is skewed. In our experiments,
216 we use symmetric Dirichlet distributions with 2 1 to guide the ensemble to representations best
suited for Multi-Task Learning.

---

Under review as a conference paper at ICLR 2023
Single Tick PML (ous)-0
PML (ours)-1
L8
MGDA DWV
POGra IMTL
CAGral Nich-MTL
FML (ours)-2
UW
MulSMNIST
Ceraus
000
beston-right accurice
000-
educalan DASUrAyy
0
002
3.02
0.00
E:04
0.7
0.99
0.22
0.924
EUCS
top-left accuracy
sccurary
Figure 4: Experimental results on MultiMNIST and Census. Top right is optimal. Three random seeds per method. Solid lines correspond to our method (PML) and thick lines the Pareto Front. We have used different color for each seed of PML. Baselines are shown in shades of gray: scatter plot for MTL baselines and dashed lines for single task. both datasets, Pareto Manifold Learning discovers subspaces with diverse and Pareto-optimal solutions and outperforms the baselines.
5 EXPERIMENTS
219
We evaluate our method on several datasets, such MultiMNIST, Census, MuMM UTKFace and CityScapes, and various architectures, ranging from MultiLayer Perceptrons (MLPs) Convolutional Neural Networks (CNNs) and Residual Networks (ResNets). Each ensemble member is initialized independently. In all experiments, the learning rate for our method m-fold the learning rate of the baselines counteract the fact that the backpropagation step scales the gradients by m- in expectation. The detailed settings used for each dataset and additional experiments are provided in the appendix. Our overarching objective to construct continuous weight subspaces which map to Pareto Fronts in the functional space. However, our method produces a continuum results rather than a single point, rendering tabular presentation cumbersome. For this reason, (a) for tables we present the bssapll--samplee) srunppeed--subspacee results, (b) we experiment on numerous two-task datasets where plots convey the results succinctly, (c) present qualitative results three-task datasets. The source code will be released after the review process.
220
222
223
224
225
227
229
230
231
We explore various algorithms from the literature: 1. Single- Task Learning (STL), Linear Scalarization (LS) which minimizes the average loss + E L,, 3. Uncertainty Weighting (UW, Cipolla et al. 2018), 4. Multiple-gradien descent algorithm (MGDA. Sener &
Baselines.
232
2.
233
234
Koltun 2018). 5. Dynamic Weight Averaging (DWA, Liu et al. 2019), 6. Projecting Conflicting
235
Impartial Multi-Task Learning (IMTL., Liu al. 2020), 8. Conflict-Aver Averse Gradient Descent (CAGrad, Liu et al. 2021) and 9. Bargaining Multi-Task Learning (Nash-MTL, Navon et al. 2022).
Gradients (PCGrad, Yu et al. 2020),
237
238
5.1
EXPERIMENTS ON DATASETS WITH TWO CLASSIFICATION TASKS
239
In this section, we focus on datasets with two tasks, both classification. This setting allows for rich visualizations that we use to draw insights on the inner workings of the algorithms.
241
MMNTT. We investigate the effectiveness Pareto Manifold Learning on digit classification
242
using LeNet model with a shared-bottom architecture. The ensemble consists of two members with single task weightings. To gauge the performance the models lying in the linear segment between the nodes, we test the performance the validation set the ensemble members well
243
245 as for models uniformly distributed across the edge, resulting in 11 models in total. We use this evaluation/plotting scheme throughout the experiments. We ablate the effect of multi-forward train- ing on Appendix D; we use a grid search on window W E {2, 3, 4, 5) and strength À E {0, 2, 5, 10) along with the base case of (W, A) () and present in the main text the setting that achieves the
247
7

---

Under review as a conference paper at ICLR 2023
Manker
Meaken
Meater
Sknder Sonder
Masherr Manzbo
Alander
AccursAy Tork
Ausunay Tlokk
Accuracy
Tack
(a) MuItiMNIST-3 Accuracy Hcatmap and Pareto Front for all tasks.
X
Minber
Co
Ridee San
Negative Iuaber Loss Age
Aosmney Gender
Colmaiity
UTKFace: Objective Heatmap and Pareto Front for all tasks.
Figure 5: Application of Pareto Manifold Learning on datasets with 3 tasks. Each triangle depicts the performance task, using color, a function of the interpolation weighting, i.e. each hexagon corresponds to a different weighting a [0r, 02 03] E A3. The closer the interpolated member single-task predictor, the higher the performance on the corresponding task. The 3D plot, on the right, show the performance of the model in the multi-objective space.
249
highest mean (across seeds) HyperVolume score on the validation set. Figure 4 shows the results on Mult tiNNIST using multi-forward regularization with window W and strength À 0. We observe that most baselines are characterized by limited functional diversity: their predefined op- timization schemes lead the differently seeded/initialized training runs to final models with similar performance (same markers are clustered in the plots). This lack of functional diversity, as well as
250
251
252
253
254
inability to consistently outperform the Linear Scalarization baseline, are also noted by Kurin et al. (2022); Xin et al. (2022). In contrast, all Pareto Manifold Learning seeds find subspaces with diverse functional solutions. This statement is quantitatively translated to higher ElyperVolume compared to the baselines, shown in Table 4 of the appendix, and can be attributed to the observation that Equation2 generalizes the Linear Scalarization method.
[Reviewer NiGo): esees [5.6] from the review
256
258
Census. We explore the method the tabular dataset Census (Kohavi, 1996) using Multi- Layer Perceptron. We focus on the task combination of predicting age and education level, similar Ma et al. (2020). We perform the same ablation study before and present the results Figure for the best setting (W = 3 and À = 10). In the case of Mu MNTST, there exists symmetry between the tasks, both digits are drawn from the same distribution and placed in the pixel grid in a symmetric way, resulting in equal pace learning. However, in the case of Census, tasks differ in statistics and, yet, the proposed method recovers Pareto subspace with diverse solutions.
280
262
254
285
296
5.2
BEYOND PAIRS OF CLASSIFICATION TASKS: MULT IMNIST 3 AND UTKFACE
287
We expand the experimental validation to triplets of tasks, consider regression and more complex ar- chitectures, graduating from MLPs and CNNs to ResNets (He et al., 2016). For three tasks, we create
269 a 2D grid of equidistant points spanning the three single-task predictors. n is the number of inter- polated points between two (out of three) members, the grid has (n+1) points. We use n 11, result- ing in 66 points. For visual purposes, neighboring points are connected. For three tasks, it would be visually cluttering to present the discovered subspaces with multiple seeds and baselines. Hence, we
270
272
opt for more qualitative discussion in this section and present quantitative findings in the appendix.
274
Mult MALIiNNITT3 3. First, we construct an equivalent of Mult iMNIST for tasks. Digits are placed left top-right and bottom-centre. Figure 5a shows the results MultiMMIST-- As argued previously, MNIST variants are characterized by task symmetry and Figure 5a reflects this. For this reason, do not employ any balancing scheme. The 3D plot in conjunction with the
276
8

---

Under review as a conference paper at ICLR 2023
Table 1: Test performance on SityScapes. 3 random seeds per method. For Pareto Manifold Learn- ing, we report the mean (across seeds) best results from the final subspace.
Segmentation
Depth
mloU
Pix Acc
Abs Err +
Rel Emr]
STL
71.79
92.60
0.0135
32.786
LS
70.94
92.29
0.0192
117.658
UW
70.97
92.24
0.0188
118.168
MGDA
69.23
91.77
0.0138
51.986
DWA
70.87
92.23
0.0190
113.565
PCGrad
71.14
92.32
0.0185
117.797
IMTL
71.54
92.47
0.0151
65.058
CAGrad
70.23
92.06
0.0173
100.162
Nash-MTL
72.07
92.61
0.0148
62.980
PML (ours)
70.28
91.94
0.0140
52.559
278
simplices reveal that the method has the effect of gradual transfer of learned representation from one member to the other, and offers succinct visual confirmation of Claim 3.
280
aTkFace. The UTкFace dataset (Zhang et al., 2017) has more than 20,000 face images and three tasks: predicting age (modeled as regression using Huber loss similar to (Ma et al., 2020)), classifying gender and ethnicity. The introduction of a regression task implies that losses have vastly different scales, which dictates the use of balancing schemes, discussed in Section 4.2. We apply the proposed gradient- balancinnn scheme and present the results in Figure 5b. For visual unity and to remain in the theme of "higher is better", the negative Huber loss is plotted. Despite the increased complexity and the existence of a regression task, the proposed method discovers a Pareto Subspace. Additional experiments and qualitative results provided in Appendix G.
282
284
286
287
5.3
SCENE UNDERSTANDING
289
We also explore the applicability of Pareto Manifold Learning for CityScapes (Cordts et al., 201 6), scene understanding dataset containing high-resolution images of urban street scenes. Our experimental configuration is drawn from Liu et al. (2019); Yu et al. (2020); Liu et al. (2021); Navon al. (2022) with some modifications. Concretely, we address tasks: semantic segmentation and depth regression. We use a SegNet architecture (Badrinarayanan et al., 2017) trained for 100 epochs with Adam optimizer (Kingma & Ba, 2015) of initial learning rate 10 which is halved after 75 ppochs. The images are resized to 128 x 256 pixels. In the initial training steps any sampling results in a random model, due to initialization, and the algorithm has warmup period until the ensemble members have acquired meaningful representations. Hence, to reduce computational overhead and help convergence, the concentration parameter of the Dirichlet distribution is set to Po 5. We use gradient balancing, window W 3 and À 1. The results presented in Ta- ble 1. In Depth Estimation and out of MTL methods, Pareto Manifold Learning is near-optimal with
281
293
294
295
296
296
300
MGDA narrowly better. However, the performance compared to the other algorithms is superior. In
302
Semantic Segmentation, our method outperforms MGDA, but is worse than other baselines. Overall no multi-task method dominates Pareto Manifold Learning. It is remarkable that, despite our goal of discovering Pareto subspaces, the proposed method is on par in performance on Semantic Segmen- tation with the state-of-the-art algorithms, and better than the vast majority on Depth Estimation.
[Reviewer qexX): Added sbort
304
comment addressing weakness
305
308
6
CONCLUSION
307
In this paper, we proposed a weight beight-ensembling method tailored to Multi-Task Learning; multiple single-task predictors are trained conjunction to produce a subspace formed by their convex hull, and endowed with desirable Pareto properties. We experimentally show on a diverse suite of bench- marks that the the proposed method is successful in discovering Pareto subspaces and outperforms
308
309
310
some state-of-the-art MTL methods. An interesting future direction is to perform hierarchical
312
weight ensembling, sharing progressively more of the lower layers, given that the features learned
at low depth similar across tasks. An alternative exploration venue is to connect method to
314
the challenge of task affinity (Fifty et al., 2021; Standley et al., 2020) via a geometrical lens of the
loss landscape.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: A Deep Convolutional Encoder- Decoder Architecture for Image Segmentation. IEEE transactions on pattern analysis and ma- chine intelligence, (12):44812449)) 2017.
318
Rich Caruana. Multitask learning. Machine Learning, (1):41-75, 1997. URL https://doi.
321
322
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer T. Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD. Biasing gra- dient descent into wide valleys. In International Conference on Learning Representations. Open- Review.net, 2017.
324
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. In International Con- ference Machine Leurning, Proceedings of Machine Learning Research, pp. 793-802, PMLR, 2018.
327
328
329
330
Roberto Cipolla, Yarin Gal, and Alex Kendall. Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics, In Conference on Computer Vision and Pattern Recognition), pp. 7482-7491. IEEE, June 2018.
331
332
333
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Conference on Computer Vision and Pattern Recognition), 2016.
334
325
336
Michael Crawshaw. Multi-Tass Task Learning with Deep Neural Networks: A Survey. arXivv2009.09996 [cs, stat], September 2020.
337
338
Yean-Antoing Désidéri. graddent descent algorithm (MGDA) for multiobjective optimiza- tion. Comptes Rendus Mathematique, (5-6):313-318, 2012.
339
340 L Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th international conference on machine learning, ICML 2017, sydney, NSW, australia, 6-11 august 2017, vol- ume 70 of Proceedings of machine learning research, pp. 1019-1028. PMLR, 2017. URL URLhttp! http: / //proceedings .m... / vv00////////// dinh1777 1e.17b.htee htmll. tex. bixbbibso dblp computer sci- ence bibliography, https://dblp.org tex.biburl: http:///bbb.oggeccccco/////////p tex.timestamp: Wed, 29 May 2019 08:41:45 +0200.
341
342
343
345
347 Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially No Bar- riers Neural Network Energy Landscape. International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 1308-1317. PMLR, 2018
348
Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. In Advances Neural Information Processing Systems, 2021.
351
Pierre Foret, Ariel Kleiner, Hossein Mobahi. and Behnam Neyshabur. Sharpness- aware Minimiza- tion for Efficiently Improving Generalization. In 9th International Conference on Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenKeview.net, 2021.
354
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear Mode Connectivity and the Lottery Ticket Hypothesis. In International Conference on Machine Learn- ing, Proceedings of Machine Learning Research, pp. 3269. PMLR, 2020.
357
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wil- son. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. In Advances in Neural Information Processing Systems, pp. 8803-8812, 2018.
380
10

---

Under review as a conference paper at ICLR 2023
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshmi- narayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In 9th international conference learning representations, ICLR 2021, virtual event, austria, may 3-7, 2021. OpenReview.ned 2021. URL https: / ew nnt/ / forum? d==GG d==0Gg9XnKxF3 A tex. tex.bibsource: dblp computer science bibliography, https://dblp.org bib tex.timestamp: Wed, 23 Jun 2021
363
365
366
387
tex.biburk
http:///bb...gecccc/////////t//a 17:36:39 +0200.
388
369
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Conference on Computer Vision and Pattern Recognition), pp. 770-778, 2016.
370
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wil- son. Averaging Weights Leads to Wider Optima and Better Generalization. In Proceedings of the Fhirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, Cal- ifornia, USA, August 6-10, 2018, pp. 876-885. AUAI Press, 2018. URL http:/ http://aaaaaaa .org/ usi2018 wwtppeeinggs/ppppoepppppppp wwwpnnggs/ppppppppeeaea wwwppsppppppsaaaaaa 31...31....... pdf.
372
373
374
375
378
Yiding Jiang*, Behnam Neyshabur*, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. Mar 2020. URL /ttp://////eeee......
377
378
forum? i d= IBFFVV...
379
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
381
382
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree tree hybrid. In Proceed- ings of the Second International Conference on Knowledge Discovery and Data Mining (KDD- 96), Portland, Oregon, USA, pp. 202-207. AAAI Press, 1996.
383
384
Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and M. Pawan Kumar. In defense of the unitary scalarization for deep multi-task learning. (arXiv:2201.04122) Oct 2022. doi: 10,488 10.48550/arxiv.2200..8 ar arXiv:2201.04122 [cs].
386
URL http:// http://aaxxvvv.// wttg://aggg/....////// 04122.
388
389
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In Advances in Neural Information Processing Systems. Curran Associates. Inc.. 2017. URL https: / /papers nips............................ cc/ htcp/apper/211/ hash/ (1t.:444277222118TTTTT4.333
380
391
392
wtw.:asbbbbee-baaaaa........aa.. .tmml.
393
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Kwong, Pareto Multi-Task Learning. In Advances in Neural Information Processing Systems, pp. 12037-12047, 2019.
Xi Lin, Zhiyuan Yang, Qingfu Zhang, and Sam Kwong. Controllable pareto multi-task learning. (arX (arXiv:2010.06311), Feb 2021. doi: (0.485550/arxi222000033333 httphtt:///rxxi org/ abs/ /2s00 06313. arXiv:2010.06313 [cs, stat].
396
387
398
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-Averse Gradient Descent for Multi-task Learning, October 2021.
389
Liyang Liu, Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards Impartial Multi-task Learning. In International Conference on Learning Representations, September 2020.
401
402
Shikun Liu, Edward Johns, and Andrew J. Davison. End-To-End Multi-Task Learning With Atten- tion. In Conference on Computer Vision and Pattern Recognition), pp. 1871-1880, IEEE, June 2019.
404
405
406
Pingchuan Ma, Tao Du, and Wojciech Matusik. Efficient continuous pareto exploration in multi-task learning. In International Conference Machine Learning, pp. 6522-6531. PMLR, 2020.
407
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-Stitch Networks for Multi-task Learning. In Conference on Computer Vision and Pattern Recognition), pp. 3994- 4003. IEEE, June 2016.
409
410
11

---

Under review as a conference paper at ICLR 2023
Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the Pareto Front with Hy- pernetworks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
412
414
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. In International Conference on Machine Learning, 2022.
416
417
Dripta S Raychaudhuri, Yumin Suh, Samuel Schulter, Xiang Yu, Masoud Faraki, Amit K Roy- Chowdhury, and Manmohan Chandraker. Controllable dynamic multi-task architectures. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10955- 10964, 2022.
418
419
420
421
Michael Ruchte and Josif Grabocka. Scalable pareto front approximation for deep
422
learning. In Proceedings of the IEEE International Conference on Data Mining (ICDM), 2021.
423
Sebastian Ruder. An Overview of Multi-Tast Learning in Deep Neural Networks, 1706.0700.000000 /cs, stat], June 2017.
425
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. Latent Multi-Task Architecture Learning. In AAAI Conference Artificial Intelligence, pp. 4822-4829, AAAI Press, 2019.
427
Ozan Sener and Vladlen Koltun. Multi-Task Learning Multi-Objective Optimization. In Ad- vances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 525-536, 2018
429
431
Trevor Standley, Amir Roshan Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Which Tasks Should Be Learned Together in Multi-task Learning? In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learning Research, pp. 9120-91322 PMLR, 2020.
433
434
435
436
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach efficient ensemble and lifelong learning. In 8th international conference on learning representations, ICLR 2020, addis ababa, ethiopia, april 26-30, 2020. OpenReview.net, 2020. URL https:
437
438
439
/ openreview. net/ for d=ssk ww..skffllryeee.. YDr. bibsouue dblp computer science bib- tex.timestam::
440
liography, https://dblp.oo tex. biburl: https://bbblooogoscccc///t///t/t
Thu, 07 May 2020 17:11:47 +0200.
442
Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning Neural Network Subspaces. In International Conference on Machine Learning, Pro- ceedings of Machine Learning Research, pp. 11217-11227. PMLR, 2021.
444
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Lud- wig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International conference on machine learning, ICML 2022, 17-23 july 2022, baltimore, maryland, USA, volume 162 of Proceedings of ma- chine learning research, pp. 23965-23998. PMLR, 2022. URL https: / /ttp///ccceingss.....///// mlr. www.press/vve 1 62, htt::/wortttttttt sman22a. a.tmm.22................... tex.bibsource dblp computer science bibli- ography, https://dblp.or tex. bex.biburl:
446
447
448
450
451
452
453
tex.timestamp: Tue, 12 Jul 2022 17:36:52 +0200.
455
Derrick Xin, Behrooz Ghorbani, Ankush Garg, Orhan Firat, and Justin Gilmer. Do current multi-task optimization methods in deep learning even help? (arXiv:2209. 11379), Sep 2022. doi: 10.48550/ 11379. arXiv:2209. [cs].
457
wttp////gga/aa/////e/aae arXiv.2209.11379 URL WtRLtttpp/aaxi..coo
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems,
459
pp. 5836. Curran Associates, Inc., 2020.
481
Zhang, Yang Zhifei, Song, and Hairong Qi. Age Progression/RRerrssion by conditional adversarial autoencoder. In Conference on Computer Vision and Pattern Recognition). IEEE, 2017.
462
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX OVERVIEW
464
As reference, we provide the following table of contents solely for the appendix.
485
A. Appendix Overview
Appendix A
486
B. Details on experimental configurations.
Appendix
467
C. Details on Illustrative Example
Appendix
468
D. Ablation Multi-Forward Regulaiization..................
Appendix D
E. HyperVolume Analysis... ................................ ................................
Appendix E
F. Additional experiments on Mu MultiMNTST-3
Appendix
471
G. Additional experiments on UTKFace
Appendix G
472
H. NEW! Details on sumpling........................
Appendix H
473
NEW! Connection between Pareto Optimality and multiple valley AitersectionnsAppeddii I
J. NEW! Additional Related Work
Appendix
K. NEW! Additional experiments
Appendix K
13

---

Under review as a conference paper at ICLR 2023
B EXPERIMENTAL DETAILS
477
MultiMNIST MUItiMNIST is a synthetie dataset derived form the samples of MNIST. Since there is no publicly available version, we create our own by the following procedure. For each Mult t iMNIST image, we sample (with replacement) two MNIST images (of size 28 x 28) and place them top-left and bottom-right on a 36 x grid. This grid is then resized to 28 28 pixels. The
478
479
480
procedure is repeated 60000 times, 10000 and 10000 times for training. validation and test datasets.
482
We use LeNet shared- bottom architecture. Specifically, the encoder has two convolutional layers with 10 and 20 channels and kernel size of 5 followed by Maxpool and a ReLU nonlinearity each. The final layer of the encoder is fully connected producing an embedding with 50 features. The decoders are fully connected with two layers, one with 50 features and the output layer has 10. We
484
486
use Adam optimizer Kingma & Ba (2015) with learning rate 10-3 no scheduler and the batch size set to 256. Training lasts 10 epochs.
488
The original version of the Census (Kohavi, 1996) dataset has one task: predicting whether person's 's income exceeds $50000. The dataset becomes suitable for Multi-Task Learning by turning one or several features to tasks (Lin et al., 2019). We focus on the task combination of predicting age and education level, similar Ma al. (2020). The model has Multi-Layer Percep- tron shared- bottom architecture. The encoder has one layer with 256 neurons, followed by a ReLU nonlinearity, and two decoders with 2 output neurons each (since the tasks are binary classification). Training lasts 10 epochs. We use Adam optimizer learning rate of 10-3
Census
489
490
491
482
495
495
Mul MMNTiMNIST-3 The configuration of Mul LiMNIST ST is used. Now, the model has three decoders and training lasts 20 epochs.
497
The UTKF ace dataset has more than 20,000 face images of dimensions 200 x 200 pixels and 3 color channels. The dataset has three tasks: predicting age (modeled regression using Huber loss similar to (Ma et al., 2020)), classifying gender and ethnicity (modeled as classification tasks using Cross-Enttae Entropy loss). Images are resized 64 64 pixels, age normalized and 80/20 train/test split is used. We use shared-bottom architecture; the encoder is a ResNetl8 (He et al., 2016) model without the last fully connected layer. The decoders (task-specific layers) consist of one fully connece layer, where the output dimensions are 1, 2 and 5 for age (modeled as regression), gender (binary classification) and ethnicity (classification with classes). Training lasts 100 epochs, batch size is 256 and we use Adam optimizer with learning rate of 10-3 No scheduler is used.
aTKFace
498
499
500
502
504
505
506
CityScapes Our experimental configuration is very similar to prior work, namely Liu et al. (2019); Yu et al. (2020); Liu et al. (2021); Navon al. (2022). All images are resized to 128 x 256. The tasks used are coarse semantic segmentation and depth regression. The task of semantic seg- mentation has 7 classes, whereas the original has 19. We use SegNet architecture (Badrinarayanan
507
508
509
510
et al., 2017) and train the model for 100 epochs with Adam optimizer (Kingma & Ba, 2015) of an
511
initial learning rate 10-4 We employ scheduler that halves the learning rate after 75 epochs.
14

---

Under review as a conference paper at ICLR 2023
Pareto Front
Final Point
Step=10000
Initial Point
Step=0
Step=20000
LS
CAGrad
Nash-MTL
PML-gb (ours)
-00
-15
h
MGDA
PCGrad
PML (ours)
PML-Ib (ours)
-10
-15
-20
-10 -15 -115--0------- -05
0. 0. 100 f
-20 -15 -1. -ás
82 -2.0 -is -10 -0.5
0.5 1.0 4 -20 -is -10 -as do as
Figure 6: Optimization trajectories in objective space in the case different loss scales. Similar to
Figure 1, 5 initializations are shown for baselines and pair of initializations for Pareto Manifold Learning (PML), in color for clarity. Dashed lines show the evolution of the mapping in loss space for the subspace at the current step. We also show the initial subspace (step= 0). All baselines, ex- cept Nash-MTL, and MGDA of the Pareto Front, namely minimizing the task with high loss magnitude. The same observation applies naïvely applying the proposed algorithm PML, because using the same weighting for both the interpolation and the losses attaches too much importance on the task with large loss magnitude. However, simple balancing schemes palliate this issue; gradient balancing (PML-gb) discovers superset of the Pareto Front and loss balancing (PML-Ib) discovers the exact Pareto Front.
lesser degree, are characterized by trajectories focused on a subset
512 C DETAILS OF THE ILLUSTRATIVE EXAMPLE
513
The details of the illustrative example are provided in this section. We use the configuration pre- sented by Navon al. (2022), which was introduced with slight modifications by Liu et al. (2021) and Yu al. (2020). Specifically, let 0 (01,02 E IR2 be the parameter vector and L (8,, 82) 518 be the vector objective defined as follows:
( c) (c(()))))))) and = c0 c1 (2()g))
where
(0) log (max (0.5(-6)) 7) tanh (-0z) 5e 6)) + 6,
f2( = log (max (00.5(-0) + 3) tanh (-02) + 2] 5e 6)) + 6,
(0) ( (00) + 7)2 + (-02 -8)) /10 - 20,
-7² + 0.1 (-02 8)2) /10 - 20, (2) (-0) - 7)2
C1 (0) = max (tanh (0.502) 0) and C2 (0) = max (tanh (-0.50g) .0)
517 We use the experimental setting outlined by (Navon et al., 2022) with minor modifications, i.c., Adam optimizer with a learning rate of 2e 3 and training lasts for 50K iterations. The overall objectives are & E and lz &g where we explore two configurations for the 520 scalar c, namely E [0.1,1). For = 1. the two tasks have losses at the same scale. For e 0.1, the difference in loss scales makes the problem more challenging and the algo-
15

---

Under review as a conference paper at ICLR 2023
rithm used should be characterized by scale invariance in order to find diverse solutions span- ning the entirety of the Pareto Front. The initialization points are drawn from the following set (9, -1.0)}. In the case of Pareto Manifold Learn- ing with two ensemble members there are 52 = 25 initialization pairs. In the main text we use the initialization pair with the worst initial objective values.
623
{(- (.-8.5, 7.5), (0.0, 0.0), (9.0, (-7.5,
525
526
527
Figure 6 presents the results for the case of different loss scales, i.e., c = 0.1. We plot various baselines and three versions of the proposed algorithm, Pareto Manifold Learning or PML in short. We focus on the effect of the balancing schemes, introduced in Section 4.2, resulting in the use of no balancing scheme (denoted as PML), the use of gradient balancing (denoted as PML-gb) and the use of loss balancing (denoted as PML-Ib). We dedicate two figures for each version of the algorithm and we present all initialization pairs for completeness. Figure and Figure correspond to no balancing scheme in the case of equal loss scales c = 1.0, i.e., they complement Figure 1 of the main text. The subsequent figures focus on the case of unequal loss scales where c 0.1; Figure Figure 10 correspond to no balancing scheme, Figure 11 and Figure 12 correspond to the use of gradient balancing, Figure 13 and Figure 14 correspond to the use loss balancing. The first figures of each pair show the trajectories for each initialization pair, with markers for initial and final positions. The other figures each pair dispense
528
529
530
531
532
533
534
536
538
the visual clutter and focus on the subspace discovered in the final step of training. which is plotted with dashed lines along with the analytical Pareto Front in solid light blue. Hence, they provide a suceinct overview of whether the method was able or not to discover the (entire) Pareto Front.
539
540
541
542
For = 1.0, the proposed method is able to retrieve the exact Pareto Front with no balancing scheme for most initialization pairs, can be seen in Figure 8. In three cases (out of 25), the method fails. In our experiments, we found that allowing longer training times or higher learning rates resolve the remaining cases.s For 0.1, the problem more challenging and the vanilla version of the algorithm results in a subset of the analytical Pareto Front. Figure 10 shows that this subset is consistent across initialization pairs, excluding the ones the method fails, and focuses on the task with higher loss magnitude. Applying gradient balancing, shown in Figure 11 and Figure 12, allows the method to retrieve (a superset of) the Pareto Front for all initialization pairs. Similarly, loss balancing, shown in Figure 13 and Figure 14, results in the exact Pareto Front. Hence, the inclusion of balancing schemes endows scale invariance in the proposed algorithm. Balancing schemes are used for the more challenging datasets, such CityScapes.
543
544
545
548
547
519
551
16

---

Under review as a conference paper at ICLR 2023
TAL (N8A
15
-20
hg
6g
-15
E2
45
-10
Figure 7: Illustrative example. Optimization trajectories in objective space for all initialization pairs in the case of equal loss scales (c = 1.0) and application of the proposed method with balancing scheme. Blue and red markers show each ensemble member's loss value, dots and "X"s correspond the initial and final step, accordingly. In all but four cases, Pareto Manifold Learning retrieves the entirety of the Pareto Front (can be sen clearly in Figure 8). Allowing longer training times or higher learning rates solves the remaining initialization pairs.
17

---

Under review as a conference paper at ICLR 2023
Supmi0900
-15
42
hg
15
15
-10
-15
-20
h2
-20
-00
-15
-10
-5
-20
-36
-3 -55 -0
81
-00 -15 -10
41
-50 15
-30
Figure 8: Illustrative example. Mapping in objective space the weight subspace discovered by the proposed method with balancing scheme, in the case of equal loss scales (c 1.0). The analytic Pareto Front is plotted in light blue. In all but four cases, the dashed line (our method) coincides with the full analytic Pareto Front.
18

---

Under review as a conference paper at ICLR 2023
hg
4g
-10
0g
bg
1.2
-13
bg
bg
4 -42-18-(0--is
6
4. -48-15-1a-05
Figure 9: Illustrative example. Optimization trajectories in objective space for all initialization pairs in the case of unequal loss scales (c = 0.1) and application of the proposed method with no balancing scheme. Blue and red markers show each ensemble member's loss value, dots and "X"s correspond to the initial and final step. accordingly. For the vast majority of initialization pairs, the lack of balancing scheme guides the ensemble to subset of the Pareto Front, influenced by the task with higher loss magnitude (can be sen clearly in Figure 10).
19

---

Under review as a conference paper at ICLR 2023
-10
4g
15
-3
-13
hg
1.2
15
-13
hg
15
-2
-15
-20 -15 -10
-66
45
-85 -5.5 -1. -65
66
-50 -15 -ia -15
1. -20 -15 60
4
-16 -10 -i as
Figure 10: Illustrative example. Mapping in objective space of the weight subspace discovered by the proposed method with no balancing scheme, the case of unequal loss scales (c 0.1). The analytic Pareto Front is plotted in light blue. The lack of balancing scheme renders optimization difficult; the method either completely fails or retrieves narrow subset of the analytic Pareto Front. Applying balancing schemes resolve these issues.
20

---

Under review as a conference paper at ICLR 2023
hg
1
bg
-18
-30
hg
bg
1r
bg
-13
hg
0g
-20
6
4:
Figure 11: Illustrative example. Optimization trajectories in objective space for all initialization pairs in the case of unequal loss scales (c = 0.1) and application of the proposed method with gradient balancing scheme. Blue and red markers show each ensemble member's loss value, dots and "X"s correspond to the initial and final step, accordingly. The proposed method discovers a subspace whose mapping in objective space results in superset of the Pareto Front. This can be clearly seen in Figure 12.
21

---

Under review as a conference paper at ICLR 2023
Supm-0900
-3
10
-18
-10
hg
1.2
15
-10
-20
hg
hg
-15
-28
-20
-15
-Co
-05
80
33
-05
-15
-14 -05 0s
-50 -15 -i -6s na nigg -10 -16 -in -55
-20 -15
-1s -as
Figure 12: Illustrative example. Mapping in objective space of the weight subspace discovered by the proposed method with gradient balancing scheme, in the case of unequal loss scales = 0.1). The analytic Pareto Front is plotted in light blue. The proposed method consistently finds the same subspace, which a superset of the analytic Pareto Front.
22

---

Under review as a conference paper at ICLR 2023
hg
1g
1.2
4g
-18
hg
bg
1.2
-13
hg
bg
-20
51
6
4:
41
Figure 13: Illustrative example. Optimization trajectories in objective space for all initialization pairs in the case unequal loss scales (c = 0.1) and application the proposed method with loss balancing scheme. Blue and red markers show each ensemble member's 's loss value, dots and "X"s correspond to the initial and final step, accordingly. For all but five cases, the proposed method discovers subspace whose mapping in objective space results in the exact Pareto Front. This can be clearly seen in Figure 14.
23

---

Under review as a conference paper at ICLR 2023
Nadoo Frect
-15
-20
1.
-10
bg
15
1r
15
-15
-4-
-10
-28
-20 -15 -18
-65
-28 -18 -6a
-20 -56 -io -is do as -20 -18 -Lo
065
-50 -15 -14
Figure 14: Illustrative example. Mapping in objective space of the weight subspace discovered by the proposed method with loss balancing scheme, in the case of unequal loss scales (c = 0.1). The analytic Pareto Front plotted light blue. Using loss balancing endows scale invariance and the solutions are more functionally diverse, in comparison with no balancing scheme in Figure 10. How- ever, the same initialization pairs continue to be problematic in the case of equal loss scales (see Figure 8). Allowing for longer training or higher learning rates solves the remaining initialization pairs.
24

---

Under review as a conference paper at ICLR 2023
0.8
0.6
0.6
0.4
0.4
022
0.2
0.2
11.0
0.0
0.5
1.0
0.2
0.4
0.5
11.1
1.0
Figure 15: Multi-Forward Graph: case of two tasks. We assume window of W 5. The nodes lie in the line segment a2 + aj 1, a1 02 E [0, 1]. (Left) Full graph and dashed edges will be removed. (Right) Final graph.
02
06
(2.4
06-
6.2
066
(2,
A 43
(4.5.0
(A,
02
022
6.
(...1.5)
(S....4)
022
0,4
0.0
0.2
0.4
0.5
01.4
0.0
Figure 16: Hurti-Foowaad Graph for three tasks. Left, middle and right present the of the first, second and third task, respectively. Each node is noted by its weighting, summing up to 1. Edges drawn if the nodes obey the total ordering imposed by the task. Dashed edges omitted from the final graph.
D ABLATION ON MULTI-FORWARD REGULARIZATION
553
Multi- Murti-Forward regularization, introduced in Section 4.2, penalizes the ensemble if the interpolated models' losses (sampled within a batch) are not in accordance with the tradeoff imposed by the corresponding interpolation weights. Simply put, the closer sample the member corresponding to task I, the lower the loss should be on task 1. The same applies to the other tasks. Equation the main text presents the of tasks, where the idea of the regularization is outlined in loss space. For completeness, we present the underlying graph construction for the cases of two and three tasks Figure 15 and Figure 16, respectively. The nodes of the graphs are associated with the sampled weightings and the edges for the graph G. of task are drawn w.r.t the corresponding partial ordering. the loss ordering violated for given edge, penalty term is added.
555
557
558
559
560
562
563
We ablate the effect multi-forward training and the corresponding regularization have on perfor- mance We explore the Mult 1MNI ST and Census datasets using the same experimental configu- rations in the main text. We are interested in two parameters:
564
586
W: number of a re-samplings per batch. This parameter is also referred as window.
567
X: the regularization strength as presented Algorithm For À = O, no regularization is applied but the subspace is still sampled W times and the total loss takes into account all the respective interpolated models.
569
570
Figure and Table present the results for MUNTIMIIST. Figure 18 and Table 3 present the re- sults for Census. It is important to note that Mult t iMNIST is symmetric, while Censua is not. As result, the features learned for each single-task predictor are helpful to one another and the case of A = 0, i... regularization and only multi-forward training, is beneficial for Mult iMNIST but not for Census. Intuitively, both digit classification tasks have the same difficulty and posterior distribution, which produces few violations of monotonicity constraints and renders the regulariza-
572
574
576
tion less applicable. On the other hand, severe regularization such as X = 10 can be harmful and
(Reviewer qexX]: expunded
hinder training. More details in table and figure captions.
commentary Hult1k9F15T.
25

---

Under review as a conference paper at ICLR 2023
Table 2: MultIMNIST: Ablation on multi-forward training and regularization, presented in Sec- tion 4.2. Validation performance in terms of HyperVolume (HV) metric. Higher is better, except for standard deviation (std). The visual complement of the table appears in Figure 17. For each configuration, track the Hypervolume across three random seeds and present Mean HV, max HV and standard deviation. We annotate with bold the best per column. the main text, we report the best result in terms of mean HV, i.e. W 4 and X = O.
Seed 0
Seed
Seed
Mean HV
Max HV
std
W 2 =
0.9205
0.9083
0.9100
0.9129
0.9205
0.0054
À = 2
0.9121
0.9105
0.9037
0.9088
0.9121
0.0036
0,9132
0.9016
0.8979
0.9043
0,9132
0,0065
À = 10
0.8766
0.8932
0.8470
0.8723
0.8932
0.0191
W = 3 0
0.9215
0.9141
0.9111
0.9156
0,9215
0,0044
= 5
0.9176
0.9150
0,9122
0.9149
0,9176
0.0022
0.9155
0.9138
0.9140
0.9144
0.9155
0.0008
= 10
0.9122
0.9050
0.8962
0.9045
0,9122
0.0066
W =
X
0.9220
0.9187
0.9143
0.9184
0.9220
0.0032
0.9213
0.9149
0.9157
0.9173
0,9213
0.0028
À 5
0.9158
0.9139
0.9132
0.9143
0.9158
0.0011
10
0.9177
0.9022
0.9102
0.9100
0.9177
0.0063
W= =5
0.9131
0.9180
0.9156
0.9156
0,9180
0.0020
2
0.9158
0.9203
0.9146
0.9169
0.9203
0.0024
À = 5
0.9138
0,9082
0.9140
0.9120
0.9140
0.0027
À 10
0.9165
0.9158
0.9121
0.9148
0.9165
0.0019
Table Census: Ablation on multi-forward training and regularization, presented in Section 4.2. Validation performance in terms of HyperVolume (HV) metric. Higher is better, except for standard deviation (std). The visual complement track the Hypervolume across three random seeds and present Mean HV, max HV and standard deviation. annotate with bold the best per column. In the main text, we report the best result in terms of mean HV, i.e., W and X 5.
the table appears in Figure 18. For each configuration,
Seed 0 Seed
Seed 2
Mean HV
Max HV
std
W 2 0
0.6517
0.6530
0.6532
0.6526
0.6532
0.0006
X = 2
0.6575
0.6564
0.6560
0.6566
0.6575
0.0006
0.6577
0.6574
0.6590
0.6581
0.6590
0.0007
A = 10
0.6548
0.6557
0.6554
0.6553
0.6557
0.0004
W 3 = 0
0.6517
0.6496
0.6501
0.6505
0.6517
0.0009
0.6540
0.6523
0.6544
0.6536
0.6544
0.0009
A 5
0.6552
0.6539
0.6536
0.6542
0,6552
0.0007
= 10
0.6574
0.6567
0.6566
0.6569
0.6574
0.0004
W = 4 A 0
0.6488
0.6516
0.6504
0.6503
0,6516
0.0011
2
0.6492
0.6522
0.6504
0.6506
0.6522
0.0012
A = 5
0.6499
0.6514
0.6525
0.6513
0,6525
0.0011
10
0.6529
0.6549
0.6558
0.6545
0.6558
0.0012
0.6497
0.6502
0.6484
0.6494
0.6502
0.0008
2
0.6478
0.6497
0.6495
0.6490
0.6497
0.0009
0.6492
0.6509
0.6489
0.6497
0.6509
0.0009
A 10
0.6507
0.6538
0.6508
0.6518
0.6538
0.0014
26

---

Under review as a conference paper at ICLR 2023
(a) W and 0
(b) W and
0
(c) W = 2 and = 2
(d) W = 2 and A 5
(c) W = 2 and À = 10
( W = 3 and = 0
gg) W 3 and = 2
(h) W = 3 and A =
ii W = 3 and À = 10
i W 4 and A 0
(k) W- 4 and A =
(I) W = 4 and A =
(m) W
4 and A = 10
(n) W and À 0
(o) W 5 and = 2
(p) W 5 and A =
(q) W 5 and À 10
Figure 17: MMNIST: Effect of multi-forward on the window and the regularization coeffi- cient À on the validation dataset. The case of no multi-forward (W = 1) is presented in the first row. Multi-forward regularization for higher W values is beneficial. Intuitively, attaching serious weight on the regularization formance since the update step focuses on an uninformed regularization term. The accompanying quantitative analysis appears in Table
(5, 10) while sampling few times W {2,3) leads to suboptimal per-
27

---

Under review as a conference paper at ICLR 2023
(a) W and
(b) W 2 and
(c) W 2 and
(d) W 2 and 10
(c) W and
(
3
(g) W 3
(h) W 3 and 10
ii) 1 and
0
ii
(k) 1 and 5
(I) and
(m) W = 5 and À (n) W 5 and 2
(0) W 5 and À = (p) W 5 and 10
Figure 18: Census: Effect of multiforward the window W and the regularization coefficient
X. The axes are shared across plots. Compared to Mult iMNIST, applying multiforward on the asymmetric Census dataset can improve accuracies and help significantly outperform the base- lines. However, widening the window W (e.g., last row for W = 5) can be hindering, since larger regularization coefficients are needed. The accompanying quantitative analysis appears Table 3.
28

---

Under review as a conference paper at ICLR 2023
nifestann jovid
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
optimal point
0.2
0.4 0.6
0.8
0 0.2 0.4
0.6
0.2
0.6
0.8
Figure 19: Visual Explanation of Hypervolume. The metric captures the union of axis-aligned rectangles defined by the reference point (star) and the corresponding sample points (red circles). This example showeases loss and the perfect oracle lies the origin. The point (1, used for reference. Hence, higher hypervolume implies that the objective space is better explored/covered.
E HYPERVOLUME ANALYSIS ON MULT IMNIST AND CENSUS
578
HyperVolume is metric widely used in multi-objective optimization that captures the quality of exploration. visual explanation of the metric is given in Figure 19. Table 4 presents the results of Figure 4 of the main text in tabular form. We present the best three results per column (higher is better) to succinctly and visually show that all Pareto Manifold Learning seeds outperform the baselines.
580
582
583
Table 4: Tabular complement to Figure 4. Classification accuracy for both tasks and HyperVolume (HV) metric (higher better). Three random seeds per method. For baselines, we show the mean accuracy and HV (across seeds). For PML, we show the results per seed; HV and max accuracies for the subspace yielded by that seed. We use underlined bold, solely bold and solely underlined font for the best, second best and third best results. We observe that the best results are concentrated in the rows concerning the proposed method (PMI... Note that the use of three decimals leads to ties.
MuItiMNIST
Census
Task
Task
HV
Task
Task
HV
LS
0.955
0.944
0.907
0.827
0.785
0.651
UW
0.957
0.945
0.913
0.827
0.785
0.650
MGDA
0.956
0.943
0.904
0.828
0.785
0.651
DWA
0.955
0.945
0.907
0.828
0.785
0.651
PCGrad
0.955
0.946
0.908
0.828
0.785
0.650
IMTL
0.958
0.944
0.908
0.828
0.786
0.651
Nash-MIL MTL
0.958
0.948
0.913
0.827
0.785
0.650
PMI.
0.968
0.951
0.92
0.830
0.789
0.655
PML
0.961
0.953
0.916
0.830
0. 08789
0.655
PML -
0.964
0.953
0.919
0.829
0.788
0.653
29

---

Under review as a conference paper at ICLR 2023
1.2.6
12.0221
199 y
97
99
Figure 20: Examples of samples and corresponding labels for the Mult iMNIST- - dataset.
584
F NULTIMNIST-2 ADDITIONAL. RESULTS
supplementary to Section 5.2 of the main text. MaltiMNIST-3 is a synthetic dataset generated by MNIST samples in a manner similar to the creation of the Mul t Si dataset, which is ubiquitous in the Multi-Task Learning literature. Specifically, each Mult t. iMNIST- sample is created with the following procedure. Three randomly sampled digits 588
This section
586
589
of size 28 28 are placed in the top-left, top-right and bottom middle pixels of 42 42 grid. For the pixels where the initial digits overlap, the maximum value is selected. Finally, the image is resized to 28 28 pixels. Figure 20 shows some examples of the dataset, which consists of three digit classification tasks.
591
582
593
Section compares the performance of baselines and the proposed method while Figure 21 presents visually the performance achieved on the discovered subspace.
Mult i MNTST-3: Mean Accuracy and standard deviation of accuracy (over random seeds). For the proposed method (PML), we report the mean and standard deviation of the best performance from the interpolated models in the sampled subspace. No balancing schemes and regularization are applied. Bold is used for the best performing multi-task method.
Table 5:
(Reviewer qexX): Table
Task
Task 2
Task 3
STL
96.97 0.06
96.10 0.17
96.40 0.22
LS
96.26 + 0.20
95.48 + 0.14
95.87 + 0.37
UW
96.48 0.08
95.42
0.30
95.77 0.06
MGDA
96.50 + 0.20
94.80 + 0.22
95.71 = 0.08
DWA
96.42 0.26
95.26
95.75 0.08
PCGrad
96.45 + 0.06
0.15
95.88 + 0.01
IMTL
96.58 = 0.22
9.11
96.08 0.31
CAGrad
96.70 0.13
95.20 1.20000.00
95.66 0.06
Nash-MTL
96.85 0.08
95.25
96.18 = 0.13
PML (ours)
96.85 + 0.43
95.72 - 0.22
96.27 + 0.32
30

---

Under review as a conference paper at ICLR 2023
Mentee
Monder
Mentler
Moulier
Menloo
Monles
Momborr
Ancunney Tiak
Aecurury Tick 2t
Acerarary
Task
Seed
Menlee
Monler
Mimiee
Munaber 5
Mandar
Mamber
Munioe
Manler
Maandor
Accuracy Tusk
Accurary
Tack
Acerrazy
Task
(b) Sced
Menthe 2
Monder
Menner
Mrnber'
Mamber Sambor
Meniter Monter
Momdor
Ancuracy
Accuraey
Aceemayy
375
(c) Sced
Figure 21: MuItiMNIST-3 results for all three seeds. Each triangle shows the 66 points in the convex hull and color is used for the performance on the associated task. The 3d plot shows the mapping of the subspace multi-objective space, balancing scheme is used.
31

---

Under review as a conference paper at ICLR 2023
G UTKFACE ADDITIONAL RESULTS
596
This section serves as supplementary to Section 5.2. Section 6 compares the performance of the baselines and the proposed method. We experiment without balancing schemes and with gradient- balancing, and present the results in Figure 22 and Figure 23, respectively. Together with the quan- titative results, we observe that for datasets with varying task difficulties, scales, etc. the lack of balancing can be impeding. On the other hand, its inclusion makes the subspace functionally di-
387
598
589
600
verse and boosts overall performance. For instance, Huber loss on the task of age prediction significantly improved.
Table 6: UтКFace: Mean Accuracy and standard deviation of accuracy (over 3 random seeds). For the proposed method (PML), we report the mean and standard deviation the best performance from the interpolated models in the sampled subspace. No multi-forward training is applied. We present Pareto Manifold Learning with no balancing scheme and with gradient balancing, denoted gb. Bold is used for the best performing multi-task method.
[Reviewer qetX): Table up- dasc.
Age 4
Gender 1
Ethnicity 1
STL
0.081 + 0.005
90.79 0.55
82.38 + 0.40
LS
0.086 + 0.003
91.66 0.55
82.78 0.60
UW
0.093 = 0.007
91.86 + 0.75
83.62 + 0.02
MGDA
0.075 + 0.003
91.17 0.59
74.06 2.66
DWA
0.093 + 0.008
91.65 + 0.46
82.85 + 0.20
PCGrad
0.101 = 0.018
0.90
83.57 0.43
IMTL
0.091 = 0.004
91.24
82.52 + 1.15
CAGrad
0.083 + 0.002
91.93 0.53
83.71 0.33
Nash-MTL
0.095 = 0.001
90.40 0.16
79.59 + 0.92
PML (ours)
0.096 + 0.002
90.97 0.63
81.78 0.14
pML-gb (ours)
0.086 = 0.003
91.61 + 0.52
81.77 + 0.86
32

---

Under review as a conference paper at ICLR 2023
Menter
Member
Menber
Ammrnas Grals
Miniberr S
Mexiler
Kambor
Monder
Munber's
Xomnle
Ca
Baber bas Age
Negativee Hluber Loss
Accurary Gender
Etanicity
(a) Sced
Bloc
Arcttre Goade
Mexther
Meniler Nember
Menber Maubee
Xomnler
Haben Day App
Negative Huber Lues Age
Accurary Gender
Accuruy Elinicity
(b) Seed
Dao
Arrinte Grnnkr
Mindeer 5
Menalor
Monber Mumlee's
Hober baw Ape
1Ag
Negative lluber Loss
Age
Aocuracy Gender
Accurayy
Ethnieity
(c) Sced
Figure 22: UTKF ace results with Linear Scalarization for all three seeds. Each triangle shows the 66 points the convex hull and color is used for the performance on the associated task. The 3d plot shows the mapping of the subspace to the multi-objective space. Applying balancing scheme for datasets with different loss scales, e.g., regression and classification tasks, may lead to limited functional diversity, such as for seed
33

---

Under review as a conference paper at ICLR 2023
Menter
Member
Menber
Atrora Gralss
Miniberr S
Mexilae
Kambor
Manber Mumber
Xamnle
Baber bas Age
190
Negatiive Hluber Loss
Accurary Gender
Etanicity
und
(a) Sced
Bloc
Artitre Gradke
U20
Mexther
Meniler
Nember
Menber Mamber
Nimnler
Haben Das Apo
Negative Huber Lues Age
Accurary Gender
Accuruy Elinicity
(b) Seed
Ammitd Grndk
Mindee 5
Monalor
Monber
Mumlee's
Nombor
Hober bew Age
Negative lluber Loss
Age
Aocuracy Gender
Accurayy
Ethnieity
(c) Sced
Figure 23: UTKFace results with Gradient- Gradient-Balancing Scheme for all three seeds. Each triangle shows the 66 points in the convex hull and color used for the performance on the associated task. The 3d plot shows the mapping of the subspace the multi-objective space. For datasets with tasks of varying loss scales, applying gradient balancing improves functional diversity and performance, shown Section 6.
34

---

Under review as a conference paper at ICLR 2023
300
4
200
2
100
0.5
0.5
(a) Beta 0.8)
(b) Beta(0.2,0.2)
1.5
30
1
20
0.5
10
0.5
1 x
0.5
(c) (2, 2)
(d) Beta(1000, 1000)
Figure 24: Dirichlet distribution in the case of two tasks. Top row: p 1 and the distribution is more concentrated towards the ensemble members. Bottom row: p > 1 and the distribution focuses more on the midpoint which corresponds all tasks having the same weight. Right column: extreme choices - 0 or + co. Left column: milder choices.
H DETAILS ON SAMPLING
603
604
[Reviewer SqPR). Added ap- pendix regarding sampling.
605
This appendix expands on Section 4.2 and, specifically, presents in greater detail the intuition behind the sampling distribution's 's parameters. Let p E R be the parameters of the Dirichlet distribution. Assuming no prior knowledge on the tasks, e.g., task difficulties or affinities, symmetric distribu- tion is used by setting p plr. This design choice results in three cases:
606
607
608
609
p 1: the distribution is uniform on the simplex. Intuitively this means that all tasks are equally important and we care about the diversity of solutions for all tradcoffs (reflected in the linear scalarization weights)
610
611
612
P E (0, 1):
the distribution more concentrated towards the ensemble members, as in the top of Figure 24. Assume extreme case of two tasks and pO. Then the distribution degenerates to a Bernoulli distribution. Effectively, at each iteration one of the ensemble members is selected and its weights are updated, which will result in two separate and inde- pendent single-task predictors with no common representation infused about the other task. Then, linearly interpolating in weight space will result in models with random predictions for both tasks, since the training procedure has not focused in retrieving a Pareto Subspace.
614
615
616
618
For milder cases (e.g. interpolation suffered in performance which can be attributed to the fact that the sampling focused more single-task rather than multi-task representations and performance.
0.7), we observed that the models in the middle of the linear
620
622
p >>> 1. Then the distribution more concentrated towards the midpoint of the simplex, as in the bottom row of Figure 24. Assume an extreme case of two tasks and - co. Then, the distribution becomes deterministic and outputs equal weights for all tasks. The randomly and independently initialized ensemble members will collapse to each other, resulting in duplicate ensemble members. Similarly, for very large values (e.g. p = 100), the functional diversity of the ensemble will suffer since the weights produced by the distribution will be almost equal for all tasks, resulting in a milder version of the aforementioned phenomenon. In contrast, found that small values such as p 2 p 3 can help convergence since
623
624
626
628
35

---

Under review as a conference paper at ICLR 2023
bottom mnttom-ndtt nottt a0curocy
69
65
top-let
Epoch
(a) Mul WultiMM SMISS:: Experimental results using three random seeds per method.
(b) MilISMIS Cosine similarities of ensem- ble members.
100
0.766
p
0297
07664
0.1
0.79
0.5
0.4
0.704
0.500
1.02
0.501
33009
0.05
0.500
Ag2 accuracy
Epoch
(c) Census: Experimental results using three random sceds per method.
(d) Census: members.
Cosine
similarities of ensemble
Figure 25: Experimental results on Mult ST and Census varying the concentration parame- ters p = plr of the sampling distribution. Three seeds depicted in shades of the same colors for the various p.
630
they put more emphasis towards common representation (compared p 1), but may limit functional diversity.
631
632
Figure presents experimental results on MultiMNIST and Census for various concentration parameters 0.1, 100} of the Dirichlet distribution. Let 0₁ and 0₂ be the parameters of the ensemble members. For = 0, the ensemble consists of two single-task predictors with no multitask learning representational knowledge, since their interpolation meets low accuracy/high loss barrier. We omit the case of representation is evident in the cosine similarities well, where for p cos(O) 02) R 0. On the other hand, forp = 0.1, common representations are infused into the ensemble and the experimental results show that the test performance is characterized by diversity. However, this comes at the expense of the interpolated models at the middle of the line segment, where the performance is suboptimal compared p 100 for Mult iittiMMNISS ST. This behavior is also illustrated in the cosine similarities, where for p 100 the ensemble weights are in an e-ball around the midpoint causing independently initialized models to progressively collapse. For Census, we also observe that this collapsing leads to very high cosine similarity cos(0,, 02) 0.9 and the ensemble is suboptimal compared to p = 0.1.
634
636
0 for Census for visual clarity. This lack of common
638
640
643
645
36

---

Under review as a conference paper at ICLR 2023
I CONNECTION BETWEEN PARETO OPTIMALITY AND MULTIPLE VALLEY INTERSECTIONS
647
648
[Reviewer qetX Added
pendix regarding
In this section, investigate the connection between the intersection of multiple loss landscapes, pareto optimality and the effect of the proposed algorithm Pareto Manifold Learning. We use the
For clirity during the rebuttal.
this discussion has been added
650
stundalone appendix, It
illustrative example, presented Figure Let be the parameter space of the model and
incorportaed appendix regarding the illustro
652
- + R, {1, be the losses of the problem. For a E [0, 1] and 0 E ④, the overall objective is [ 0:) = a (0) + (0))
tive example.
653
654
Figure 26 and the accompanying Figure 27 present the overall loss objective as varies from 0 to For the extreme values of the range, the loss landscape is inherently single-task. The subspace discovered by the method is depicted in blue, while black 'x' is used for the corresponding inter- polated model, i.e,, it corresponds L(a0) +
655
a0)0, a). Figure 28 presents the overall losses on the subspace by fixing as function of one of the parameters. In other words, the proposed method tracks the optimum in parameter space as the overall objective evolves and the various loss landscapes are weighted accordingly. While an acceptable multi-task solution lies in the intersection 681
657
659
of low loss landscapes, Pareto Manifold Learning focuses on the aforementioned dynamic scenario of loss weighting.
e=3.375
0.875
Figure 26: Illustrative example: (Overall) loss surface as function of the model's weights. The overall objective is L(O, a) aL, (0) + (1 (20)) and is shown for various values of a. The Pareto subspace discovered by the proposed method is depicted in blue. 'X' shows the solution of the method for the corresponding a.
37

---

Under review as a conference paper at ICLR 2023
0.075
Figure 27: Illustrative example: Alternate view of Figure 26, Refer the text for details.
38

---

Under review as a conference paper at ICLR 2023
10
-- a=0
a=0.25
a=0.5
a-0.75
a=1
-5
-15
-20
-10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
10.0
Figure 28: Illustrative example: Overall loss for various weightings [0, 1] as function of one of the parameters, denoted as x. Points corresponds to the loss achieved by the parameter vector (a:) = af, (1 02 The subspace discovered by the model spans the range [-7,7].
39

---

Under review as a conference paper at ICLR 2023
J ADDITIONAL RELATED WORK
[Reviewer NiGo); Added sp pendix discussing additional related work.
665
In this appendix, we further expand on prior work. Linear mode connectivity, as (Wortsman et al., 2021), encourages flatness and, therefore, is linked with methods explicitly enforcing flat minima (Chaudhari et al., 2017; Foret et al., 2021; Dinh et al., 2017: Jiang* el al., 2020). These approaches are applicable when designing single objective, e-g. average of losses in Multi-Task Learning, but do not allow for the infusion of Pareto properties and the inclusion of tradeoffs. Izmailov et al. (2018) produce flat minima by averaging multiple weight vectors discovered during the optimization
686
687
688
689
woeks "Eai minima"
670
trajectory, so that the final model lies in the middle of the low-loss basin. Wortsman et al. (2022) perform weight ensembling with fine-tuned models produced via different hyperparameter config- urations. Apart from the recent weight ensembling works, output ensembling has been of the staples machine learning literature. Lakshminarayanan et al. (2017) utilize deep ensembles for uncertainty prediction but inference scales linearly with the number of ensemble members. Wen et al. (2020) improve on the computational complexity of output ensembles by sharing the bulk of the parameters among members and differentiating them via rank-1 matrices, while Havasi et al. (2021) employ a multi-input multi-output network by accommodating independent subnetworks for each ensemble and allowing single-forward pass ensemble prediction. However, this results in subnetworks with incompatible architecture which does not allow for a continuous approximation of the Pareto Front.
672
674
676
678
(Reviewer NICio]: added poior
work ensemble learning
40

---

Under review as a conference paper at ICLR 2023
Single Tick
CAGra
EPN LN-10 spodie
PML (oun) 0
Nash-MI11.
HPN LN- epportooard 0
PMI (oun)-1
OUSMUS-10 spodie-eod
EPN LN- epochisaod
PML (ouns--2
00SMO6-10 quodi-sodd
HPN LN-150 epochs-oned
0USMO--10 spodimeod
HPN FP0010 eperloeeell
UW
00SMOS-110 spoclesoedd
HPN E20-10 opocla-aod
MCDA
epocha
EPN epocls
DWA
COSMOS spocla-eedd
HPN E2O-130 spodls
PCGua
HPN LN- 10 spedte seed
EPN EZO 150 spodis seed
IMTL
spedhesrol
FP0-130 spodisared
0.85
0.55
681
betton- 1ight aocuracy
6.3
0:9
0.89
0.86
0.55
0.55
3.55
6:90
0:92
0.51
0.96
top-left accuracy
Figure 29; MultiMNIST: Figure 4 with additional baselines.
682
K
ADDITIONAL EXPERIMENTS
reviewers]: up pendix with addinonal base- lines.
684
In this section, we supplement our experimental findings on MultiMNIST with additional base- lines, namely HPN-LN and HPN-EPO (Navon et al., 2021) and COSMOS (Ruchte & Grabocka, 2021)², use hyperparameters of Ruchte Grabocka (2021) for both methods. We provide two experimental settings:
686
688
Setting I: 10 epochs and no learning rate scheduler, i.e., the setting used for all other meth- ods Figure 4,
689
Setting //ttee experimental setting used by (Ruchte & Grabocka, 2021), i.e., 100 epochs for COSMOS and 150 epochs for HPN-LN/HPN-EPO with multi-step learning scheduler.
681
682
Figure 29 presents the results with the additional baselines, using three seeds each. We use dashed lines for setting I and solid lines for setting I/ and group the three methods in various color shades (blue, green, red) for visual clarity. We observe that in the original setting 10 epochs, all new base- lines are suboptimal compared all methodologies. For setting II, the hypernetwork methodologies are competitive with some baselines but are suboptimal compared to the proposed method. For COSMOS, only one seed is competitive with the proposed method. Moreover, HPN-LN, HPN-EPO employ hypernetwork of 1.6m parameters, while the target network has < 50k parameters.
694
696
688
We use the open source implementation provided by Ruchte & Grabocka (2021) making minimal changes. Our implementation of the Mu i iN IN IT dataset has images of size 28 x 28 rather than 36 36 resulting in slightly different models.
41


Reviews:
[{"review_id": "hZA9Bbt3Sjr", "reviewer": "Reviewer_NfGo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a Pareto manifold learning approach to produce a continuous Pareto front for a given multi-task learning (MTL) problem in a single run. The MTL problem is formulated as a multi-objective optimization problem, and the proposed method can be treated as an ensemble approach for multiple single-task models with linear weight interpolation. In the proposed method, each task (objective) has its own neural network model (with identical structure), and their linear weight combination will be explicitly associated with the corresponding trade-off losses (with linear scalarization). Experimental results show that this method can successfully find the Pareto subspaces for different MTL problems and outperforms other MTL methods.", "summary_of_the_review": "This paper is well-written, and it studies an important research topic that could further bridge the field of multi-task learning and multi-objective optimization. I personally enjoy reading this work. However, due to the major concerns on main contributions, relation to closely related work, and the experimental setting, it is hard for me to vote to accept the current manuscript.", "strength_and_weaknesses": "**Strengths:**\n\n+ This paper is well-written and easy to follow.\n\n+ Multi-task learning and multi-objective optimization are both important research topics, and this work tries to further bridge these two fields.  \n\n+ The proposed method has reasonable performance on different MTL problems (but with many concerns listed below).\n\n**Weaknesses**\n\nI have some major concerns on both the two claimed advantages of the proposed method, namely, a) learning the Pareto front for multi-task learning and b) better generalization performance.\n\n**1. Contribution and Closely Related Works**\n\nThis work proposes to learn the Pareto front for a multi-task learning problem. However, similar Pareto front learning approaches have already been proposed for multi-task learning [1,2,3,4]. Those methods all allow practitioners to choose their preferred solutions at inference time. All these closely related works are not discussed or compared in this paper. The proposed linear interpolation approach could be similar to using a simple linear hypernetwork in [1,2]. \n\nWithout a thorough discussion and comparison with these closely related works, it is very hard to judge the novelty and actual contribution of the proposed PML method.\n\n**2. Pareto Solution for Better Generalization Performance**\n\nThe motivation to learn the Pareto solution for better generalization performance is not strong. Recent works [5, 6] on multi-task learning have shown that a single fixed solution found by simple linear scalarization can have similar or even better performance than those found by multi-objective optimization methods. The effect of careful hyperparameters finetune rather than the Pareto properties could be crucial for good generalization performance.   \n\nIn addition, the experimental results on the found Pareto front and better per-task performance are somehow not solid. The major issue is that, for an MTL problem with m tasks, the proposed PML model is m times larger than the other baselines, which could lead to unfair comparison. For example, in Figure 4, some baseline models can have comparable or even better performance than the PML's Pareto front. If we can double their model capacity to match the one for PML, it could be likely that some of them (a single solution) can totally dominate the whole Pareto front by PML. In this case, the \"Pareto front\" could be useless since all tradeoffs are suboptimal and dominated.\n\nSimilarly, in Table 1, the overall per-task performance of PML is already nondominated with other MTL baselines. If the model capacity of all the baselines can be doubled, their performance could be significantly improved. On the other hand, the single-task learning baseline (STL) indeed has the same model capacity with PML. In this problem, it seems that simply having separate models for each task already significantly outperforms PML.   \n\n**3. Ensemble Learning Method**\n\nSince the proposed PML method requires learning multiple models and can be treated as an ensemble method, it should be compared with the closely related ensemble learning methods. A naive baseline could be to train m models with the same fixed weight among tasks (e.g., simple LS) or other multi-objective optimization (e.g., MGDA, CAGrad), and then a simple ensemble learning method [7] can be applied to achieve better overall MTL performance. An ensemble of models with different optimization methods, such as LS with different weights or MGDA + CAGrad, could be another choice.\n\nThe recent work on ensemble learning could have a smaller model size or faster inference time [8,9], and the weight average methods [10, 11] are also strong alternatives with a single final model. Some related works on mode connectivity have been briefly discussed but not compared in the paper. What is the advantage of the proposed PML model for generalization over those ensemble learning or weight average methods?    \n\n**4. Other Flat Minima Method**\n\nOne main motivation of this work is the connection between low-loss subspace (e.g., flat minima valley) and better generalization performance. From this viewpoint, it is interesting to know its relation to the entropy-sgd [12] and sharpness-aware minimization methods [13, 14, 15]. Can we simply use these methods to train an MTL model with fixed weights for each task? What is the advantage of PML over those methods? \n\n**Other Questions**\n\n1. In Algorithm 1, should the input size of \\theta be m rather than m?\n\n[1] Learning the Pareto Front with Hypernetworks. ICLR 2021.\n\n[2] Controllable Pareto Multi-Task Learning. arXiv:2010.06313.\n\n[3] Scalable Pareto Front Approximation for Deep Multi-Objective Learning. ICDM 2021.\n\n[4] Controllable Dynamic Multi-Task Architectures. CVPR 2022.\n\n[5] In Defense of the Unitary Scalarization for Deep Multi-Task Learning. arXiv:2201.04122, 2022.\n\n[6] Do Current Multi-Task Optimization Methods in Deep Learning Even Help? arXiv:2209.11379, 2022.\n\n[7] Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS 2017.\n\n[8] BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. ICLR 2020.\n\n[9] Training independent subnetworks for robust prediction. ICLR 2021.\n\n[10] Averaging Weights Leads to Wider Optima and Better Generalization. UAI 2018.\n\n[11] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML 2022.\n\n[12] Entropy-SGD: Biasing gradient descent into wide valleys. ICLR 2017.\n\n[13] Sharp Minima Can Generalize For Deep Nets. ICML 2017.\n\n[14] Fantastic generalization measures and where to find them. ICLR 2020.\n\n[15] Sharpness-Aware Minimization for Efficiently Improving Generalization. ICLR 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fmgT50KMSe", "reviewer": "Reviewer_qexX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel approach to seek a continuous Pareto Front for multi-task learning. The key idea is to train multiple single-task predictors and performs convex hull operations on different models in the weight space to produce a subspace with multiple Pareto stationary points. A continuous Pareto Front can be produced through only a single run in this method.", "summary_of_the_review": "This paper focus on an interesting problem and provides some new ideas. However, there are some technical flaws in this work.", "strength_and_weaknesses": "Strengths:\n\n1. This paper improves the method in Ma et al. (2020) and proposes a novel method to produce continuous Pareto stationary points, which is simple and easy to utilize. In addition, the method is efficient as it requires only a single run.\n2. This paper is inspired by the theory of single-task machine learning and the motivation is quite interesting. Meanwhile, the concept of Pareto manifold learning is insightful and worth exploring.\n3. The visual explanation provided in Figure 3 is easy to understand, while the visual analysis of the article (Figure 4 and Figure 5) is comprehensive.\n\nWeaknesses:\n\n1. It must be acknowledged that the conjecture about the loss landscape from the single-task to multi-task is insightful. However, it is better to give a toy example to explain it. More importantly, even though multiple valley intersections in the low loss region in the multi-task scenario, this observation is not strongly related to Pareto manifold learning. I suggest the author should explain it more clearly.\n2. In the last paragraph of the introduction, the author announces that “the algorithm produces a subspace of Pareto Optimal solutions”, It should be pointed out that multi-task learning methods can only ensure convergence to Pareto optimal when assuming it holds a convex loss, in other words, we can only approximate rather than obtain the Pareto optimal solutions.\n3. The proposed method fails to exceed the baseline methods such as Nash-MTL and PCGrad on several datasets(Table 1, Table 5, and Table 6), which makes the method less convincing. In addition, the ablation experiment demonstrated in Tabel 2 takes a value of $\\lambda$ equal to 0 on the MultiMNIST dataset, which may indicate that the regularization is not effective. Most importantly, there is a lack of detailed analysis of the experimental results.\n4. Paper writing should be polished, For instance, Figure 1 and Figure 2 are not mentioned in the context, which is not friendly to read.\n5. In addition, some typos exist in the paper. In line 11 of Algorithm 1, the citation of Figure 4.2 is wrong, the same problem is found in the corresponding text above Claim 3, I guess it should be Figure 3.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "En9sQnN7PEc", "reviewer": "Reviewer_SqFR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper assumes there exist Pareto Subspaces, i.e., weight subspaces where multiple optimal functional solutions lie, and develops a weight-ensembling method named Pareto Manifold Learning that casts multi-task problems as learning an ensemble of single-task predictors by interpolating among members during training.  Multiple single-task predictors are trained in conjunction to produce a subspace formed by their convex hull, and endowed with desirable Pareto properties. Each single-task model infuses and benefits from representational knowledge and the other members. The losses are weighted in tandem with the interpolation when training.  The ensemble as a whole engenders a weight subspace that explicitly encodes tradeoffs and results in a continuous parameterization of the Pareto Front.", "summary_of_the_review": "This paper produces a subspace with multiple Pareto stationary points in the multi-task learning based on the geometry of the loss landscape in single task machine learning where the local optimal are connected by simple paths. However, the representation and more illustrations are needed.", "strength_and_weaknesses": "Strength:\nThis paper castes multi-task problems as learning an ensemble of single-task predictors by interpolating among members during training, and designs weight subspaces where multiple optimal functional solutions lie. Based on this weight space, an ensembling method that can produce a continuous Pareto Front in a single training run is proposed. This allows practitioners to modulate the performance on each task during inference on the fly.  \n\nWeaknesses:\n\n1.Similar methods have already been proposed for multi-task learning and has not been disccussed in this paper [1].\n\n1.When sampling on the convex hull parameterization, authors choose to adopt the Dirichlet distribution since its support is the T-dimensional simplex. Does this distribution have other properties.   Why using this distribution?  If p≫1，how the ensemble will change. \n\n2.When training, a mono tonic relationship is imposed between the degree of a single-task predictor participation and the weight of the corresponding task loss. As a result, the ensemble engenders a subspace that explicitly encodes tradeoffs and results in a continuous parameterization of the Pareto Front.  Whether the mono tonic relationship can be replaced by other relationships? Explaining this point may be better.\n\n[1]Navon A, Shamsian A, Fetaya E, et al. Learning the Pareto Front with Hypernetworks[C]//International Conference on Learning Representations. 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject