Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences
Authors: Nikolaos Dimitriadis
Pascal Frossard
François Fleuret
Venue: ICLR 2025 Poster (2025)
Keywords: Pareto Front Learning
multi-task learning
low rank
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a novel Pareto Front Learning algorithm that augments any neural architecture with task-specific low-rank adapters and uses  deterministic preference sampling to better guide training.

Abstract: Multi-task trade-offs in machine learning can be addressed via Pareto Front Learning (PFL) methods that parameterize the Pareto Front (PF) with a single model. PFL permits to select the desired operational point during inference, contrary to traditional Multi-Task Learning (MTL) that optimizes for a single trade-off decided prior to training. However, recent PFL methodologies suffer from limited scalability, slow convergence, and excessive memory requirements, while exhibiting inconsistent mappings from preference to objective space. We introduce PaLoRA, a novel parameter-efficient method that addresses these limitations in two ways. First, we augment any neural network architecture with task-specific low-rank adapters and continuously parameterize the Pareto Front in their convex hull. Our approach steers the original model and the adapters towards learning general and task-specific features, respectively. Second, we propose a deterministic sampling schedule of preference vectors that reinforces this division of labor, enabling faster convergence and strengthening the validity of the mapping from preference to objective space throughout training. Our experiments show that PaLoRA outperforms state-of-the-art MTL and PFL baselines across various datasets, scales to large networks, reducing the memory overhead $23.8-31.7$ times compared with competing PFL baselines in scene understanding benchmarks.

Published as a conference paper at ICLR 2025
PARETO LOW-RANK ADAPTERS: EFFICIENT MULTI-TASK LEARNING WITH PREFERENCES
Nikolaos Dimitriadis EPFL
Pascal Frossard EPFL
François Fleuret
University of Geneva, Meta FAIR
ABSTRACT
Multi-task trade-off's in machine learning can be addressed via Pareto Front Learn- ing (PFL) methods that parameterize the Pareto Front (PF) with single model. PFL permits to select the desired operational point during inference, contrary traditional Multi-Task Learning (MTL) that optimizes for single trade-off decided prior to training. However, recent PFL methodologies suffer from limited scalability, slow convergence, and excessive memory requirements, while ex- hibiting inconsistent mappings from preference objective space. We introduce PaLoRA, a novel parameter-efficient method that addresses these limitations ways. First, augment any neural network architecture with task-specific low-rank adapters and continuously parameterize the PF their convex hull. Our approach steers the original model and the adapters towards learning general and task-specific features, respectively. Second, we propose a deterministic sampling schedule of preference vectors that reinforces this division of labor, enabling faster convergence and strengthening the validity of the mapping from preference objective space throughout training. Our experiments show that PaLoRA outperforms state-of-the-art MTI. and PFL baselines across various datasets, scales large networks, reducing the memory overhead 23.8 31.7 times compared with competing PFL baselines in scene understanding benchmarks.
INTRODUCTION
Building machine learning models with multi-task capabilities is becoming increasingly prevalent (Ruder, 2017: Crawshaw, 2020), pursuing the advantages of a shared representation coupled with the practical benefits of a single model, in terms of memory requirements and inference times. However, the construction of generalist agents (Reed et al., 2022) by solving simultancously multiple tasks introduces conflicts and there often does not exist single optimal solution. Multi-objective optimization (MOO) problems rather have a set of optimal solutions, formally known as the Pareto Front (PF), each corresponding different trade-off among the objectives.
Multi-Task Learning (MTL) algorithms optimize for a single trade-off, delivering one point in the PF (Mahapatra & Rajan, 2020; Lin et al., 2019a; Cipolla al., 2018; Chen et al., 2018; Sener & Koltun, 2018; Navon et al., 2022). but lack the flexibility of dynamic adaptation to unseen preferences during inference. In recommender systems, for instance, objectives such semantic relevance and revenue or content quality introduce trade-off's (Lin et al., 2019b). Similar competing objectives can arise in autonomous self-driving (Wang et al., 2018), multi-objective image or protein generation (Yao et al., 2024), and aligning LLMs to multiple preferences (Zhong et al., 2024). Pareto Front Learning (PFL) methodologies (Navon et al., 2021; Ruchte & Grabocka, 2021; Dimitriadis et al., 2023) address the inherent scalability issues of the MTL discrete solution set by directly parameterizing the entire PF. They learn a conditional model, where providing the desired preference as input generates the weights of neural network satisfying that trade-off.
Despite these advancements, the practical application of PFL methods remains constrained by several challenges, including high memory consumption and unreliable prefereece-to-objeecive
Contact: kolkaoc dimitriadissess .ch. Code available githh..ommnik-ddmyyllla..

---

Published as a conference paper at ICLR 2025
gr
g
Bg 3
W E 3
+
+
+
A Ra
Ag R
Original matrix
Semantic Segmentation
Instance Segmentation
Disparity Estimation
doli
Overall
drhy
Input x
A (o,, Asl ~
Figure 1: Conceptual illustration of the architecture, Fach layer consists of the base network's weight matrix W and low-rank adapters ((A,, B,) During training, we sample preference A= A2, Aa] As, each layer's weights and the tasks' low-rank adapters. The overall loss uses the same weigh the task losses, steering each adapter to learn task-specific features and the shared backbone to learn general representation.
formed by the weighted sum of the original matrix
mappings. Specifically, current methods rely training conditional model by sampling prefer- ences from a Dirichlet distribution, and using these preferences to weigh the objective losses. While this approach theoretically enables coverage of the entire PF, it introduces significant inefficiencies for PFL approaches compared to MTL ones. First, the weight-generating mechanism often requires parameters far exceeding those of the target network, leading substantial memory overheads (Navon et al., 2021; Dimitriadis et al., 2023). Second, the reliance on extensive training periods and complex learning rate schedules results slower convergence (Navon al., 2021; Ruchte & Grabocka, 2021). Finally, the inherent randomness in preference sampling reduces control over the optimization process and compounds on these issues. The stochasticity can lead to inconsistent map- pings, where adjusting the preference towards particular task may not yield improved performance in that task. Overall, these limitations hinder both the efficiency and the reliability of PFL models.
In this paper, introduce Pareto Low-Rank Adaptors (PaLoRA), novel PFL algorithm that tackles these limitations through two complementary strategies. First, we equip neural networks with task- specific low-rank adapters (Hu et al., 2022) and establish Pareto Set within their convex hull, in The core network parameters learn a general representation beneficial for all objectives, whereas the adapters acquire specialized features. Second, propose replace the random pref- erence sampling with a deterministic procedure, presented in Figure 2, that reinforces this division of labor. Initially, the focus lies on building general features and towards the latest stages of training on specialization, by selecting preferences in the middle and edges of the simplex, respectively.
Figure 1.
PaLoRA significantly reduces memory overhead by structured division of labor within the model, while the core neural network parameters remain fixed, responsible for learning shared representa- tions, only the participation of the lightweight low-rank adapters varies. The deterministic sampling schedule mirrors the natural learning process observed in neural networks: early training phases prioritize learning general features, followed by specialization to task-specific features strategy akin to curriculum learning (Soviany et al., 2022). By focusing on the center of the preference simplex during early training and progressively exploring its edges, PaLoRA achieves functionally diverse PF without sacrificing convergence speed. Finally, PaLoRA is inspired by pareameter- efficient fine-tuning methods (Hu et al., 2022) and, therefore, inherently supports Pareto expansion fine-tuning mechanism for extending the PF
(Ma et al., 2020), i.e., enabling the method to in the vicinity of pre-trained solutions. This opens up opportunities to leverage the representations learned by foundation models, making PaLoRA highly adaptable and scalable across tasks.
2

---

Published as a conference paper at ICLR 2025
Our experimental results demonstrate that PaLoRA significantly outperforms other PFL methods across multiple benchmarks using very small ranks (r I 4), has better Pareto aligmment and requires fewer parameters. For instance. when training occurs from scratch for CityScapes PaLoRA quires a 4.2% memory overhead compared to MTL baselines, a 23.8x overhead reduction compared to PaMal. (Dimitriadis et al., 2023), while showing superior performance. When applied as a fine- tuning mechanism, the proposed method is able to quickly align the adapters towards continuously modeling the PF locally, offering flexible Pareto expansion mechanism.
Our contributions are as follows:
We present PaL.ORA, novel PFL method that augments any architecture with task-specific per layer low-rank adapters and discovers a Pareto Set in the their convex hull.
PaLoRA uses annealed deterministic preference sampling to guide PFL training, and we show that it improves on training stability and the PF functional diversity.
Our experiments on multiple benchmarks and across model sizes show that PaLoRA out- performs PFL baselines, achieving higher hypervolume while incurring orders of magni- tude less memory overhead. For CityScapes and NYUv2, PaLoRA introduces an over- head of 4.2% and 6.3%, respectively, marking a 23.8x and 3 reduction compared Pa- MaL (Dimitriadis et al., 2023), while staying competitive or outperforming MTL baselines. We show that the proposed method can be easily adapted to fine-tuning mechanism to expand the PF continuously in the neighborhood of a pretrained checkpoint.
2 RELATED WORK
Crremeter-Efficienn Fine-Tuning (PEFT) strategies (Lester et al., 2021: Li & Liang, 2021;
Houlsby et al., 2019) have advanced the adaptation of pre-trained models to specific tasks without extensive re- training. Fine-tuning pre-trained models occurs in low-dimensional subspaces (Agha- janyan et al., 2021), and Hu et al. (2022) proposed fine-tuning only a low-rank decomposition of each layer's weight matrix, reducing computational and memory costs and sparking a series of subsequent methodologies (Huang et al., 2023; Valipour et al., 2023; Kopiczko et al., 2024; Hyeon-Woo et al., 2022; Yeh et al., 2023; Liu et al., 2024b; Tian et al., 2024; Wu et al., 2024). Low rank adapters have also been studied from a multi-task perspective. Feng et al. (2024) use a routing mechanism after fine-tuning several LoRAs, while Zhao et al. (2024) adaptively retrieve and compose multiple LoRAs depending on the input prompts. Compared to these works, we employ low-rank adapters throughout training and not just as a fine-tuning mechanism and towards approximating the PF instead of optimizing over single static objective.
Multi-Task Learning. The pursuit of learning multiple tasks within a single model has deep roots in machine learning (Caruana, 1997; Argyriou et al., 2008; Ruder, 2017; Crawshaw, 2020), evolving in the deep learning era through architectural innovations (Misra et al., 2016; Ma et al., 2018; Ruder et al., 2019) and optimization techniques (Cipolla et al., 2018; Chen et al., 2018; Yu et al., 2020; Liu et al., 2020). The methodologies focus on combining the task contributions, either directly on the loss level (Cipolla et al., 2018: Lin et al., 2022), on the gradient level (Chen al., 2018; Liu et al., 2022; 2021; Chen et al., 2020). However, these methods only produce a single point in the Pareto Front and do not offer users control during inference.
Pareto Front Learning methods parameterize the Pareto Front, allowing the construction of a model satisfying user's trade-off at inference and at cost. Sener & Koltun (2018) scale Multiple Gradient Descent Algorithm (Désidéri, 2012) to deep learning settings. Lin et al. (2019a) introduce constraints encoding trade-offss to steer solutions along the Pareto Front (Fliege & Svaiter, 2000), but require separate training runs per solution. Navon et al. (2021); Lin et al. (2021); Hoang et al. (2023) use HyperNetworks (Ha et al., 2017) to continuously approximate the Pareto Front, but even with chunking to reduce memory, they still double the GPU memory requirements due to simultancous deployment with the target network. Ruchte Grabocka (2021) offer a memory-effiiint approach via input augmentation, though network conditioning issues remain. Dimitriadis et al. (2023) use weight ensembles but require a copy of the model for each task, leading to excessive memory overhead. In contrast, our approach maintains a stable core network for general features while aligning task-specific adapters to improve convergence. We finally note that a concurrent work (Chen & Kwok, 2024) also employed LoRAs (Hu et al., 2022) towards approximating the Pareto Front. Beyond the architectural component, we also propose a deterministic preference schedule to improve the validity of the final Pareto Front and convergence.
3

---

Published as a conference paper at ICLR 2025
Algorithm 1: PaL.oRA
Input:
neural network f. tasl loss function L.
batch (x: 3) Dani
training dataset Duain
Limi +
Function Pr Preference (T, M, Q1:
A (Mr) + Preference(r, M, Q)
Distribute evenly in the f'-dimensional simplex ((M)
(2a), ,.... ((M) ( do
M points X
// rorm parameters via Equation
Anneal the rays with temperature Q via
0 w EE
deA B f=1
Equation 3 for timestep
return ( ....., A (M
Luaat Limat A (J
pazameterizes ray preference
12
Update Ow, Oa, Og via backpropagation 7 7
striibttion
13
++++1
T +
MULTI-OBIECTIVE OPTIMIZATION AND PARETO FRONT LEARNING
((x(1, y ( for samples ( E X and labels y 3 and T tasks. Each task & E [T] is associated with a loss L. 34 V/- - R4. The overall goal corresponds to the multi-objective optimization problem of minimizing the vector loss L - LT].
Consider a dataset
We assume general multi-task architecture comprising of encoder g( 0 Penc) X x + Z, mapping to embedding space 2 and multiple decoders fz(z, 0.)
x O. yr, [T]. learning model f X x + y that performs well on all tasks. However, in the setting of vector optimization (Boyd & Vandenberghe, 2004), single solution A can be optimal for all tasks simultaneously. In contrast, optimality is sought in the Pareto sense, i.e., a solution 0 is Pareto optimal if there is no other solution 0' that is better for all tasks. Formally:
where (c) represents network parameters. Multi-Task Learning focuses
Definition 1 (Pareto Optimality). Let L,(),, f(X, (). = (O). Assume two solutions 0, 0' E θ parameter space. point 0 dominates point 0' (0) S (.)))) VI [T] and L( + L(O"). Then, point 0 is called Pareto optimal if there exists no 0' that dominates it. The set of Pareto opti- mal points forms the Pareto Set Ps and its image in objective space known as the Pareto Front Py.
Let AT (A E EI Et= 2t=1 A 1} be the T T-dimensional simplex representing the set of all possible user preferences and e θ + R.T mapping from weight to objective space, i..., the vector loss (0) =
(0)). For a given preference vector objective of PFL is weight generating mechanism h ÅT - Q such that Ps h(Ar) and the function h is monotonic, i.e, increasing the importance of one task leads to its loss decrease. Our goal is discovering a Pareto Subspace (Dimitriadis al., 2023), i.c., low-dimensional subspace that approximates the PF in single training run.
År as input, the overall
4 PALORA: PARETO LOW-RANK ADAPTERS
PFL methodologies have two components: the conditional model h mapping from preference space the Pareto Set and the distribution used during training sample preferences. We improve both of these axes in Section 4.1 and Section 4.2, respectively, and present them jointly in algorithm 1.
4.1 WEIGHT GENERATING MECHANISM OF PALORA
Consider the case of single layer parameterized by W E [aaxm, We omit bias terms for simplicity. The output of the layer is y Wa for R Inspired by the weight ensembling approach of PaMaL (Dimitriadis et al., 2023), we propose that each task t E [T] augments the set of learnable parameters by introducing copy of the model:
y' W
for preference the T-dimensional simplex, i.e., Di...., AT E Ar. Given the similarity among ensemble members (Dimitriadis et al., 2023), the full-rank duplicates be replaced with
4

---

Published as a conference paper at ICLR 2025
Tioe 0.0
0.0
(a) Random (M -M=1 (b) Random (M IM=38 5)
(c) Fixed (M = 5)
(d) Anncaled (M 5)
Figure 2: Random vs deterministic preference schedules for two tasks a function time. Each dashed line corresponds to a different batch, bottom is beginning of training and top end of training. For each batch, preferences À [X, - X] are drawn; we only show Randomly sampling (a) M 1 ray per batch (b) multiple (M > 1) rays (Dimitriadis et al., 2023) can lead to poor mappings from preference to objective space due to lack exploration and tightly clustered sampled rays. In- stead, our (c) proposed deterministic schedule resolves these issues and (d) our temperature anneal- ing, focusing progressively more to learning task sask-specific features, can lead to wider Pareto Fronts.
low-rank matrices A, Raxr, B, E RRrxm , [T) and < min(m,,m)
W = A,B, = y = W + a Σ A.
(2)
where a scaling factor that tunes the emphasis on the low-rank residuals (Huh et al., 2024). Besides the above LoRÁ (Hu et al., 2022) formulation, the task-specific weights W, be constructed by any approaches in the low-rank family (Yeh et al., 2023; Zhang et al., 2023; Kopiczko et al., 2024).
During each training iteration, preferences À are drawn, and direct link between each task and its corresponding adapter is formed by defining the total loss as A = ΣT1 ArC.. This alignment ensures that the contribution of each adapter matches the loss weight of the corresponding task. Figure illustrates the method in the case of three tasks. PaLoRA establishes clear division of labor between the base network and the task-specific adapters. The former's 's involvement in the weight generation mechanism of Equation present, helping convergence by keeping the major part of the representation steady. Following the Dimitriadis et al. (2023, Theorem 4.2), Theoreom shows that PaLoRA can approximate the PF due to the universal approximation theorem (Cybenko, 1989). The proof is provided in the appendix.
not a function of the drawn preference but always
Theorem 2. Let ft X x θ + y be family of continuous mappings, where t = I....,T, T, and X IR." is compact. Then, Ve O, there exists ReLU multi-layer perceptron with three different weight parameterizations 0b, 01. 0₂ E ② such that Vt E [T], 3a E [0, 1]. Va X:
(x) f (x:; 0 + 00, + a:) 02) c.
PaLoRA is based parameter-efficiet approaches (Lester et al., 2021; Houlsby et al., 2019; Ma- habadi et al., 2021; Ben Zaken et al., 2022: Hu et al., 2022) and is, therefore, by design compatible with pre-trained models and able to take advantage of the knowledge embedded in foundation models (Bommasani et al., 2021). 1.. In contrast, previous PFL solutions, e.g., hypernetworks, are not easily adaptable to such settings and do not scale to the size of such large models. As consequence, our method can be launched as a second phase of training, similar to the setting studied by Ma et al. (2020), where once multi-task solution has been attained, e.g., using NashMTL (Navon et al., 2022) or linear scalarization, a second phase expands the PF in the neighborhood of the solution.
Memory Complexity For a linear layer with mn parameters, the PaLoRA layer requires mn + Tr(m + n2). Modulating the rank r controls the layer's expressivity and number of parameters, allowing for task- speciffc features to steered to the main backbone the adapters.
4.2 IMPROVED CONVERGENCE AND FUNCTIONAL DIVERSITY WITH DETERMINISTIC PREFERENCE SCHEDULE
Pareto Front Learning methods lead to an increase of trainable parameters and lower convergence rates compared MTL approaches. The preference distribution dictates the construction of the models during training, e.g., via Equation the case of PaLoRA or as an input to a hypernet- work (Navon al., 2021) 12 or mixing coefficients of weight ensemble (Dimitriadis al., 2023).
5

---

Published as a conference paper at ICLR 2025
500
5000
Tolk 3004005
Waltte
Cratose
(M-1),
DMS
200
PALARA Al
8.81
180
iRA (Mall) A)
Epoch
MUINIM
(b) UTK
(c) SARCOS
Figure 3: Experimental results. (a) PaLoRA outperforms MTL baselines and achieves higher Hypervolume while requiring less memory vs other PFL algorithms, (b) constructs wide Pareto Front for benchmarks with two classification and one regression tasks. (c) Even for 7 tasks, PaLoRA showcases fast convergence while PaMaL is slow due 7x increase in parameter count.
However, previous works assume the distribution to be (uniform) Dirichlet and suffer from slow convergence rates and inconsistent mappings from preference to objective space. Consider the case of drawing single random weighting À every batch as in Figure 2a. In the case of two tasks, for instance, where the ray (A, 1 A) for E [o, 1]. two consecutive rays can be A = (1-61, G1) and,
A2 (1 E2, <2) X/- (f2, 1 - e() for e1, e2, eg O. In the former case (À) - A2). starvation occurs for the second task and the representation drifts towards accommodating the first, while in the latter case (A1 X)) the representation abruptly changes leading instability. The lack of control introduces variance in the gradient updates, since À determines both the model's l's representation and the overall scalarized objective. Performing multiple forward passes (Dimitriadis al., 2023) with different rays, as in Figure 2b, can decrease the variance of the updates but the lack of exploration can still lead invalid mappings from preference space to weight space and, finally, objective space. Specifically, sampled rays from tightly clustered region of the probability space result in similar weight configurations, e.g. via Equation 2, and given the non-convexity of the loss landscape their small differences may not correspond to vector loss objectives that satisfy the Pareto properties de- fined in Definition 1. Overall, stochasticity in the updates can lead to decrease in loss but may violate the premise of Pareto Front Learning in creating valid mapping from preference to objective space.
We therefore propose to replace random sampling with a deterministic schedule. Assuming a budget of M forward passes and T tasks, we sample preferences (Rm, ) evenly spaced in the simplex Ar, as Figure 2c. Given that neural networks initially learn general features and specialize during the later training stages (Zeiler & Fergus, 2014; Kalimeris et al., 2019; Valeriani al., 2023), we propose to sample multiple preference vectors per batch with a schedule mimicking the training dynamics. For time r e [0, 1], where t = 0 and т = 1 correspond to the beginning and end of training, respectively, anneal the preferences (5m m=1 follows:
Af
Á
Am (T) = (A) =
for
T Q
m.t
where Q controls the temperature of the annealing. the initial training stages, the preferences are concentrated in the center and progressively more to the faces of the simplex, shown in Figure 2d for 2 dimensions and in Figure Il for 3 dimensions. The schedule deterministic in order to minimize the effect of pernicious updates due to potential imbalanced samplings and therefore we do not consider regularization terms (Dimitriadis et al., 2023; Ruchte Grabocka, 2021), also eliminating their associated hyperparameters.
EXPERIMENTS
We evaluate PaLoRA variety of tasks and datasets, ranging from multi-label classification to complex scene understanding benchmarks CityScapes and NYUv2. We include datasets ranging from 2 to 7 tasks and architectures from LeNet to SegNet (Badrinarayanan et al., 2017).

---

Published as a conference paper at ICLR 2025
Table CityScapes: Test performance averaged over seeds, Ap is the parameter count increase w.r.t. the MTL model. We highlight the best and second best results per task. PaLoRA outperforms PFL and MTL methods, while slightly increasing parameter count and allowing for user control.
Segmentation
Depth
4,% 4
Controllable
miloi ?
Ace
For
Rel
STL
0.01411
38. 38.6133
0%
LS (Caruana, 1997)
91.
0.01922
124.0615
0%
UW (Cipolla et al., 2018)
70,20
91:93
0.01899
125.9133
0%
MGDA (Sener & Koltun, 2018)
66.45
90.79
0.0141
53.1376
19%
X
DWA (Liu et al., 2019)
70.10
91.89
0.01922
127 6580
0%%
X
PCGrad (Yu al., 2020)
70.02
91.84
0,0188
126.2551
05%
IMTI. (1.i al., 2020)
70:77
0/0151
74-2300
0%
CAGrad (Lio al., 2021)
09:23
91.61
0,0168
110.1387
0%
Nash-MTL (Navon al., 2022)
71.13
92 23
0.0157
78. 1993
10%
X
RLW (Lin et al., 2022)
68.79
91.52
0.0213
126.9123
D%
Graddrop (Chen et 2020)
70:07
91.93
0.0189
127.
RomeGrd (lavaloy & Valem, 2022)
60.92
91.85
0.0193
127.2806
0%%
Auto-A (I.i et al., 2022)
70.47
92:01
0.0177
116-9504
0%
COSMOS (Ruchic& & Grabocka, 2021)
69.78
91.79
0.0539
136.614
42 1%
PaMal. (Dimiriadi et al., 2023)
70.35
91.99
0.014:1
54,520
100%
PaL.RRA [ours]
71.11
92.21
0.01:40
51.2672
4.2%
Baselines. We compare against a diverse set of algorithms, including Single-Task Learning (STL),
MTL methodologies based on loss balancing (Caruana, 1997; Cipolla et al., 2018: Liu et al., 2019;
Lin et al., 2022) and gradient balancing (Sener & Koltun, 2018; Yu et al., 2020; Liu et al., 2020;
Chen et al., 2020; Liu al., 2021; Navon al., 2022; Liu al., 2022; Javaloy & Valera, 2022). We consider PFL baselines in Pareto HyperNetwork (Navon et al., 2021, PHN), COSMOS (Ruchte & Grabocka, 2021) and Pareto Manifold Learning (Dimitriadis et al., 2023, PaMaL). For PFL methods, evaluation is performed on a grid of K evenly spaced points spanning the T -iddimeniional simplex.
5.1 MULTI-LABEL CLASSIFICATION
First, we test the effectiivnees of PaLoRA MultiMNIST, digit classification dataset based on MNIST and use a LeNet architecture (Lecun et al., 1998), trained for 10 epochs, and present the results in Figure 3a. Compared to standard MTL approaches that result in single point in objective space, PFL methodologies discover a subspace of solutions, parameterized by the user preference À in Equation 2. We use rank r 1, resulting in 12.5% additional trainable parameters compared to the base multi-task model, i.e., the scatter points in the plot. In contrast, PaMal. (Dimitriadis al.. 2023) doubles the trainable parameters, PHN (Navon et al., 2021) requires 100 memory overhead and COSMOS (Ruchte & Grabocka, 2021) has low hypervolume. Overall, our method achieves superior performance compared to other PFL methods, while requiring far less parameters, and showing that even rank of r = 1 suffices to cover the Pareto Front.
5.2 SCALING UP THE NUMBER OF TASKS
We explore benchmarks beyond two tasks and present the results in Figure 3 qualitatively for 3 tasks and in Figure 3c quantitatively for 7 tasks. We consider UTKFace (Zhang et al., 2017), a dataset with images and three tasks of gender and ethnicity classification and age regression, and we use ResNet (He et al., 2016) backbone following prior work (Dimitriadis et al., 2023). We observe in Figure 3b that our proposed method effective at discovering valid Pareto Front. Similarly, we include in the appendix results on Mult MMNIIMISS--SSS (Dimitriadis et al., 2023). a generalization of the previous dataset with three objectives, where PaLoRA is able to discover a wide and functionally diverse Pareto Front. Finally, we explore SARCOS (Vijayakumar & Schaal, 2000), a robotic dataset with T' regression tasks that predict joint torques based on joint positions, velocities, and accel- erations. Figure 3c shows that PaLoRA converges faster than Linear Scalarization, MTL method used as control, and considerably faster compared to state-of-the-ar PaMaL (Dimitriadis et al., 2023), due to the latter requiring the joint optimization of T 7 copies of the model. In contrast, use rank r = 4, leading to a parameter increase of 62%. Overall, PaLoRA scales to a larger number of tasks compared to PaMal., showcasing superior performance and lower memory requirements.
7

---

Published as a conference paper at ICLR 2025
Table NYUv2: Test performance averaged over 3 seeds. Ap the parameter count increase w.r.t. the multi-task model. We highlight the best and second best results per task. PaLoRA is superior to PaMaL (Dimitriadis al., 2023) while requiring 31.7x less parameters.
Segmentation
Depth
A,%
Controlibble
m
Pix Acc
Abs Err
Rel ErT
Angle Distance
Wittin
Mon
Median
11.25
22.5
30
STL
36.58
02
0.6538
0,2707
25.27
02:33
59.09
70,00
0%
(Canaan,, 1997)
36.33
0.5007
27:71
21.54
25:44
51.72
UW (Cipulla et 2015)
$1.68
00.11
0.6009
0.2242
25,80
22.50 50
25.74
00.33
62.09
05%
MGDA (Sener Koluun 2015)
91.19
90
32.
59.04
0%%
DWA 2019)
37.09
63.09
0.5663
0.2225
27.67
26.55
61.78
64.02
07%
PCGrad (Yu al. 2020)
36.83
63,43
0.0004
0,2182
27:44
21.57
25.50
12.25
64.40
05%
2020)
06.19
MTL (Nowen et 2002)
37:06
64.75
0.5279
0.2110
25.56
10:30
30.39
58.90
68.35
03%
REW (Lin sl., 2012;
35.86
61.49
0.5022
0,2271
29.06
25.08
21.06
48:27
00.67
Gradkrop (Chen 20200)
35.05
25:38
51.64
63.90
PaMal. (Dimiriadis et nl., 2023)
33.94
62.55
0.5002
0,2188
25.60
20.33
20.00
54:61
06.35
2005
Phll oRA (surs)
36. 27
64.79
0.5120
0.2150
25.66
19.59 34
30.47
56.90
68.56
6.3%
5.3 SCENE UNDERSTANDING
We now evaluate PaLoRA on large scale datasets of scene understanding. An example of input and label combinations is given in Figure I. Our experimental setup is based on previous MTL works (Liu et al., 2019; Yu et al., 2020; Liu et al., 2021; Navon et al., 2022; Dimitriadis et al., 2023).
CityScapes (Cordts al., 2016) contains high-resolution urban street images and focus on the tasks of semantic segmentation and depth regression. We train a SegNet (Badrinarayanan et al., 2017) for 100 epochs, using Adam optimizer (Kingma & Ba, 2015) with learning rate 10-1 that is halved after 75 epochs. The results are presented in Table for rank r = 4 and M = 5. In terms of PFL baselines, while COSMOS (Ruchte & Grabocka, 2021) increases slightly the number of train- able parameters, its poor performance on the task of depth estimation renders it non-competitive. We note that we do not consider PHN (Navon al., 2021), since it requires the ad hoc definition of hypernetorr architecture with chunking for the weight generating mechanism, introducing many hyperparameters. Additionally, the hypernetwork requires least as many parameters as the target network to match its expressiveness, effectively doubling the memory requirements. For this reason, Navon et al. (2021) only consider ENet (Paszke et al., 2016), a network of only 0.37M parameters compared to > 25M of the SegNet architecture. Compared to PaMaL (Dimitriadis et al., 2023), our method leads to improvements across tasks while reducing the memory overhead ~ 23.8 times.
NYUV2 Similar to previous MTL works, we consider the NYUv2 dataset (Silberman et al., 2012) for the tasks of semantic segmentation, depth estimation, and surface normal prediction, and report the results in Table 2. We reserve 95 images for validation and use a setup similar to CityScapes but train for 200 epochs. The full experimental details are provided in the appendix. To the best of our knowledge, and due to scalability, PaLoRA is the first PFL method to explore benchmarks as challenging as NYUv2. Additionally to PHN (Navon et al., 2021), we omit the COSMOS baseline (Ruchte & Grabocka, 2021) due to its poor performance on CityScapes. PaLoRA scales to the complexity of the benchmark without incurring large memory costs, while PaMaL (Dimitriadis et al., 2023) requires 3 times the memory of the original model, PaLoRA needs 6.3% more parameters.
5.4 CONTINUOUS PARETO EXPANSION
While previous sections focused on training models from scratch, PaLoRA can also be used as a second-stage fine-tuning approach, similar to the original use of low-rank adapters (Hu et al., 2022). Following Ma et al. (2020), we expand the Pareto Front locally around a pre-trained model Oo. using checkpoints trained with linear scalarization. Only adapters are fine-tuned, matching the final stage's learning rate, with training lasting 4 epochs for MultiMNIST and 5 epochs for CityScapes, consuming just 40% and 5% of the original training budgets. Results in Figure show PaLoRA effectively expands the Pareto Front, enhancing initial model performance. The scaling factor a controls the spread, while the middle plot of Figure shows that the annealed deterministic sampling gradually increases functional diversity. Unlike the costly checkpoint
8

---

Published as a conference paper at ICLR 2025
0.00
0.166
LS
0.194
0.154
Ezoch
Lea Bottre But
30
Ezoch 2
Ezech
Dept Megreeson
0.0050
2,0 m
0.190
0.180
Ezech
8.0078
0.188
0.135
0.156
0.154
0.077
0.122
0.132
0.125
0,150
0.125
0.385
0.50
0.201
0.218
Semanti: Segreoration
Figure 4: Pareto Front Expansion. Given checkpoint Op. marked as e, PaLoRA expands locally the Pareto Front in its neighborhood. N ((o).. (Left) The scaling a of Equation 2 determines the functional diversity of the final MultiMNIST Front. (Middle) The epoch-by-epoch progression of the Mult t iMNIST ST Pareto Front expansion. (Right) The final CityScapes Pareto Front.
Tee tme
(V-)) Foed)
Revee)
Pale (1-9)
(M-5, Raskee]
Paurtk (M-5, Rowd)
Stee
Casch
(a) PaLoRA converges faster to low loss basin
(b) PaLoRA maintains valid Pareto Front
Figure 5: PaLoRA satisfies both PFL goals: high controllability coupled with superior performance. PaLoRA converges faster than state-of-ther the-heerrttt PFL method in PaMaL and par better to MTL methods. It is also more consistent in terms of the validity of the Pareto Front across epochs, while the number of points in the Pareto Front in PaMal. varies lot.
storage of Ma et al. (2020), PaLoRA incurs only a 4.2% memory overhead, forming a continuous linear segment in weight space that maps the Pareto Front.
DISCUSSION
6.1 DETERMINISTIC SCHEDULE IMPROVES CONVERGENCE AND PARETO DIVERSITY
We evaluate the effect of the sampling procedure as function of the number of samples M E N, whether they are drawn from a random distribution deterministically and the effect of annealed schedule, Specifically, we consider the following distributions over preference rays:
Dirichlet(pl), Dirichlet(p1),
Random, M = 1 (Navon et al., 2021)
A, 10 A [ for {(m)
Random, M 1 (Dimitriadis al., 2023)
(2m) Deterministic [ours]
gr. (2m))
Deterministic Annealed [ours]
As 7 increases the sampled weightings shift focus from the center of the simplex towards the edges. We consider the setting outlined in Section 5.1 for the Mult iMNIST dataset and compare our proposed method with PaMaL (Dimitriadis et al., 2023) and Linear Scalarization (LS), a Multi-Task Learning method used as control. We present the results in Figure 5 in terms of two metries: the progression of the training loss captures how fast each method reaches a neighborhood of low loss, while the number of points" the validation Pareto Front measures the degree of Pareto alignment of said neighborhood. We observe that PaLoRA outperforms PaMal in both metrics. Specifically, PaMaL. has slower convergence since it has to simultaneously optimize two copies of the model, while the low-rank adapters of our proposed method are lightweight and efficient. PaLoRA also has faster convergence compared to a single-point algorithm LS, since sampling multiple rays
Wee sample 11 points in total {(A, >) for 0. ...,,....000 ..9.......9.11

---

Published as a conference paper at ICLR 2025
Top-Left accuracy (f)
88.11 91.66 95.56
96.1 95.31
94.83
Bottom-right accuracy (f)
57.19
58. 28.65
46.29
46.89
9.12 9.96
(1,0)
10.1)
(..5,0.5)
(0.0)
(1-11)
(-1,1)
(-1,-1)
22 (used in training)
Ag (not used in training)
Figure Adapters contain task-specific information. While selecting rays À Ag used in training maintains high performance on both tasks, psendopreferences X & A₂ reveal that adapters are task-spe:ific: negating the contribution one leads to forgetting the associated task.
acts as regularization effect (lzmailov et al., 2018; Foret et al., 2021). More importantly, the deterministic schedule of the proposed method maintains valid Pareto Front throughout training, as indicated Figure 5b. In contrast, both PaMaL and low rank adapters without the proposed deterministic schedule fluctuate in the number of validation points in the Pareto Front, highlighting that stochasticity undermines the control desideratum Pareto Front Learning.
We also conduct larger scale search on the effectiveness of deterministic sampling in constructing valid Pareto Fronts in Appendix D where we consider the cases of M E {3, 5,,5,77 7,99, deterministic and Dirichlet sampling, fixed annealed schedule and several temperatures for each sampling category and compare the HyperVolume (Zitzler & Thiele, 1999), which measures solutions quality, and the number of points in the Pareto Front. The results indicate that random sampling introduces instability: while it can achieve high HyperVolume, it often leads to a dominated Pareto Front. In contrast, deterministic sampling offers well-distributed Pareto Front, is more robust to hyperparameter variations, and enables effective performance with lower M, reducing memory overhead from multiple forward passes.
6.2 DIVISION OF LABOR BETWEEN CORE NEURAL NETWORK AND ADAPTERS
We investigate if the low-rank adapters contain task-specific information. While the rays used during training lie in simplex Ay, we examine the impact of escaping the simplex to highlight the separation of feature building objectives among the adapters. We evaluate psendopreferences E {0, 0), (1, - 1), 1)] & A2. corresponding to models with general features, and those forgetting the second and first tasks, respectively. For comparison,
also evaluated actual prefer- ences X E {(1,0), 0), (0, 1), (0.5, 0.5)) C A2, with results shown in Figure for the Mult MMISST ST benchmark. The findings reveal the specialized roles of adapters: removing the first task's contribution in À = (-1,1) notably impairs its performance, while the second task is less affected. The core network without adapter contributions for X (0.0) lacks discriminative features, and A = -1) results in random predictions. This aligns with task arithmetic evaluations (Ilharco al., 2023; Ortiz-Jimenez et al., 2023; Yadav et al., 2023; Wang al., 2024), where negating task contributions leads to forgetting. However, unlike independently trained models, our jointly trained adapters still allow knowledge transfer, avoiding a complete separation of task-specific knowledge.
CONCLUSION AND LIMITATIONS
In conclusion, PaLoRA addresses the limitations of existing PFL methodologies by introducing parameter-efficient approach that augments the original model with task-specific low-rank adapters coupled with deterministic preference sampling to guide PFL training. Our proposed method effectively separates the learning of general and task-specific features, facilitated by a carefully designed sampling schedule. Our extensive experimental results demonstrate that PaLoRA not only surpasses PFL baselines across benchmarks but also achieves scalability to larger networks with limited memory overhead. By providing a continuous parameterization the Pareto Front and ensuring efficient of memory, P offers promising solution for real-world applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like thank Guillermo Ortiz-Jiménez, Alessandro Favero, Ke Wang, Ortal Senouf and the anonymous reviewers for their valuable feedback.
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. In Association for Computational Linguistics (ACL.), 2021. URL https: / aa anthol waaeehhoooyyoooooooooooo wgw.orgg..oooo.o................ a22.1..llllonnn.... 568.
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Convex multi-task feature learn- ing. Machine learning, 73:243-2722 2008.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: Deep Convolutional Encoder- Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, (912))224811499)99 2017.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.
BitFit: Simple Parameter-efficient Fine- tuning for Transformer- based Masked Language-moder models. In Association for Computational Lin- guistics (ACL), 2022. URL https: / aacllal antho ho logy org 2rg//200 2222. acl short.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky. Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Hen- derson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee. Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim- itriou, Joon Sung Park, Chris Picch, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srini- vasan, Alex Tamkin, Rohan Taori, Armin W. Thomas. Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kait- lyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, 2021. URL https: / crr.. stanford. eewrerdddddeeee wdw.ssstts.aaeeaeeeeeeeeesssaaae http//aassssseepppeeeeeee
pdf.
Stephen P. Boyd and Lieven Vandenberghe. Convex Optimization. 2004.
Rich Caruana. Multitask Learning. Machine Learning, 28(1):41-75, 1997.
Weiyu Chen and James Kwok. Efficient pareto manifold learning with low-rank structure. In In- lernational Conference on Machine Learning (ICML), 2024. URL https:://////..t /pppnreveew.....eo net/forum?id=aa sHP...
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. GradNorm: Gradi-
ent Normalization for Adaptive Loss Balancing in Deep Multitask Networks. In International Conference on Machine Learning (ICML), 2018. URL http://arxiv.on wrw. atrg/bbs/1111. 02257v4.
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just Pick Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL http: //arxiv. wttpiiv.oggabaaaa/ae/eeee http:/ab//210.0060888. 06808v1.
11

---

Published as a conference paper at ICLR 2025
Roberto Cipolla, Yarin Gal, and Alex Kendall. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. URL httt:://axiiiii owg/ abo//005.. 07115v3.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bemt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), URL htt.tt/p////xii..
abs/ 604. 01 685v2.
Michael Crawshaw. Multi- Laaltin with Deep Neural Networks: A Survey. arXiv, 2020. URL http://arxxiv.org//bb//////
6v1.
George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics control, signals and systems, 2(4):303-314 1989.
Jean-Antoine Désidéri. Multiple-Gradee Descent Algorithm (MGDA) for Multiobjective Opti- mization. Comptes Rendus Mathematique, 35:(5-6):33--31, 2012.
Nikolaos Dimitriadis, Pascal Frossard, and François Fleuret. Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models. In International Conference on Machine Learning (ICML), 2023.
Wenfeng Feng. Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-LoRAs: An Effi- cient Multitask Tuning Method for Large Language Models. In Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), 2024. URL
https: / /awaaaaaaaaanoooo.aaaaaaaaaaaeat orgoorgg////ooo.oooo 2024 wree- main. 994.
Jörg Fliege and Benar Fux Svaiter. Steepest Descent Methods for Multicriteria Optimization, Math- ematical methods of operations research, 51(3):479 2000.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-Aware Minimiza- tion for Efficiently Improving Generalization. In International Conference on Learning Repre- sentations (ICLR), 2021. URL http: http//aaivo.rg///e..////// abs/ 2010 01412v3.
David Ha, Andrew M. Dai, and Quoe V. Le. HyperNetworks. International Conference on 609. 091 06v4
Learning Representations (ICLR), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Patlern Recognition (CVPR), 2016.
URL hwtp: http///xxi// org/ abs / /1222 03385v1.
Long P Hoang, Dung D Le, Tran Anh Tuan, and Tran Ngoc Thang. Improving pareto front learning via multi-sample hypernetworks. In AAAI Conference Artificial Intelligence, number 7, 2023.
URL http::/t////iii/ org/ abs/ 01130v7.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (ICML), 2019. URL http:
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang. and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In Interna- tional Conference on Learning Representations (ICLR), 2022. httppp:s//ppeeeevie... net./ forum?id=n2eeveeeFFY.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Chao Du, Tianyu Pang, and Min Lin. LoraHub: Efficient Cross-Tate Task Generalization via Dynamic LoRA Composition. In R0-FoMMoRRbbuteer of Few-shot and Zero-shot Learning in Large Foundation Models, 2023. URL https:// openreview net./ / forum? d=1 forumtidd=IyRy22ppph
Minyoung Huh, Brian Cheung, Jeremy Bemsstein, Phillip Isola, and Pulkit Agrawal. Training Neural Networks from Scratch with Parallel Row- Adapters. arXiv, 2024. URL http: arxiv. org/abs/ 222.. 6828v2.
12

---

Published as a conference paper at ICLR 2025
Nam Hyeon- Woo, Moon Ye-Bin, and Tae- Tae-Hyun Oh. FedPara: Low-rank Hadamard Product for Communication-Ffiiee Federated Learning. In International Conference on Learning Repre- sentations (ICLR), 2022. URL htttts:://penneviieeee wtt.e/iwwwweweeeeeeeeeeeeeeeeeee wwtp:/ooomiddddddnnntteeeeeeeeee
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing Models with Task Arithmetic. In International Conference Learning Representations (ICIR), 2023. URL http::tpt///aaa/ .rgg aa/g///b///// 2212 04089v3.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wil- son. Averaging Weights Leads to Wider Optima and Better Generalization. In Conference on Uncertainty in Artificial Intelligence (UAI), 2018. URL httttp://aaxxiwoorr/abbbaa 05407v3.
Adrián Javaloy and Isabel Valera. RotoGrad: Gradient Homogenization in Multitask Learn- ing. In International Conference on Learning Representations (ICLR), 2022. URL https:
/ //openreview... net./ forum? wtw.mmddddwwweeeemmmoo
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD Neural Networks Learns Functions of Increas- ing Complexity. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL https: https:: wwwiuingggnnuurrsseeeee neurips hepaper_filespppper///eeee File://22224c5339777777088888883
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), 2015. URL 1412. 6980v9.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based Random Ma- trix Adaptation. In International Conference on Learning Representations (ICLR), 2024. URL --NumLX---
https:/ wpwpr/eeeeeeeeeeeeeeeeeeeeeeeewe net / (orum? N--N LNf
Y. Lecun, L. Bottou, Y. Bengio. and P. Haffner. Gradient-Based Learning Applied Document Recognition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998.
Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficientt Prompt Tuning. In Empirical Methods in Natural Language Processing (EMNLP), 2021. URL http: //arxiv.org// org/ abs 2210 08691v2.
Xiang Lisa Li and Percy Liang.
Optimizing Continuous Prompts for Generation. In Association for Computational Linguistics (ACL), 2021. URL httttts:/accammmhol.gg/
org/2021. aaee1aacl-oaac llng. 353.
Baijiong Lin, YE Feiyang, Yu Zhang, and Ivor Tsang. Reasonable Effectiveness of Random Weight- ing: A Litmus Test for Multi-Task Learning. Transactions on Machine Learning Research, 2022.
URL http: /ttp///aavvvv//a////a//eee/aaaea org/ abs 2ts./2211.10..1222....1111111111
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Kwong. Pareto Multi-Task Learning. In Advances Neural Information Processing Systems (NeurIPS), 2019a.
Xi Lin, Zhiyuan Yang, Qingfu Zhang, and Sam Kwong. Controllable Pareto Multi-Task Learning, 2021. URL http: /tt///ixvvv/////////////////aavv orgg abss//////// 2010 0. 06313v2.
Xiao Lin, Hongjie Chen, Changhua Pei, Fei Sun, Xuanji Xiao, Hanxiao Sun, Yongfeng Zhang, Wenwu Ou, and Peng Jiang. pareto-efficient algorithm for multiple objective optimization in e-commerce recommendation. In ACM Conference on Recommender Systems (RecSys), 2019b.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2024a. URL http: / arxiv. org brg/abs/2an 2 03792v3.
13

---

Published as a conference paper at ICLR 2025
Liyang Liu, Li, Zhanghui Kuang, Jing-Hao, Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards Impartial Multi-task Learning. In International Conference on Learning Representations (ICLR), 2020.
Shih- Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. DoRA:
Low-Rank Adaptation. In In- ternational Conference on Machine Learning (ICML), 2024b. URL
net/forunm 3d55ScIG1122..
Shikun Liu, Edward Johns, and Andrew J. Davison. End-To-End Multi- Taski Learning With At- tention. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019. URL http:/ / arxiv. org/ abs/ babs/1803 1070442.
Shikun Liu, Stephen James, Andrew Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. Transactions on Machine Learning Research, 2022. ISSN 2835-
8856. URL https:://// httttp:://pperevveie http///pnnnneeeeeeeeeeeeeeeeeeea net/ www.::/wwwweee/o///t/aaaaa KKrcmimid-KKeMMMmS
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling Task Relation- ships in Multi-task Learning with Multi-gate Mixture-of-Expert In International Conference on Knowledge Discovery and Data Mining (KDD), 2018
Pingchuan Ma, Tao Du, and Wojeiech Matusik. Efficient Continuous Pareto Exploration in Multi- Task Learning. In International Conference on Machine Learning (ICML), 2020. URL http:
//arxiv.org/abee..... 2006. 6434v2.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient Low- Rank Hypercomplex Adapter Layers. In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL httpp:://ppen/eee
net forum? httr//ooommmmmmd/deeeeee id=bqckKSPII6--
Debabrata Mahapatra and Vaibhav Rajan. Multi-Task Learning with User Preferences: Gradient Descent with Controlled Ascent in Pareto Optimization. In International Conference on Machine Learning (ICML), 2020.
Ishan Mísra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-Stitch Networks for Multi-task Learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. URL http:://aaxxiii org/ abs 4044004000 C3539v1.
Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the Pareto Front with Hy- pernetworks. In International Conference on Learning Representations (ICLR), 2021. URL http://arxix org/ absa 04104v2
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. In International Conference on Machine Learning (ICML), 2022. URL https: / pttp///ioceeedn..oooooo//o/ mlr www.spressvvii html.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL http: arxiv. org/ abs/2305. 12827v3
Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. Enet: A deep neural network architecture for real-time semantic segmentation. arXiv, 2016. URL http: /arxiv. org//bs//1 60.66..... 6. 02147v 47v1.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin. Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, perf-paeformance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL http:://axaiiii org/ abs / 912 01 01703v1.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo. Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Ec- cles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A Generalist Agent. Transactions on Machine Learning Research, 2022. URL https: /ttp////pe//eeeeeeee/eeee net fowummirmmiooo d= kk0OH jvj.
14

---

Published as a conference paper at ICLR 2025
Michael Ruchte and Josif Grabocka. Scalable Pareto Front Approximation for Deep Multi- Mujett-beeeetive Learning. IEEE International Conference on Data Mining (ICDM), 2021. URL http://
arxiv.org/abs http///21....1111/..... 13392v2.
Sebastian Ruder. An Overview of Multi-Task Learning in Deep Neural Networks. arXiv, 2017.
URL hwtp::tt/aaiiiiee org/ abs/ 110.. 05098v1.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Sogaard. Latent Multi-Task Architecture Learning. In AAA/ Conference Artificial Intelligence, 2019.
Ozan Sener and Vladlen Koltun. Multi-Task Learning as Multi-Objective Optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2018. URL http://///rr abs/ 1810 0 6560v2.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In IEEE European Conference on Computer Vision (ECCV), 2012.
Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Currieulum learning: A survey. International Journal of Computer Vision, 130(6):1526-13)) 1565, 2022. URL http:://arxa org/abs 2211..11. 10382v3.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng zhong Xu. HydraLoRA: An Asymmetric LoRA Architecture for Efficient Tuning, In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https::/ /ttp://///eeeeeeeeeee
www.:eeetfforrmmee id=qipiiiiii 3NWX
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https:// openreview net wow.mniddccCaaaus....
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter- Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation. In Association for Computational Linguisties (ACL), 2023. URL org/2023. eacl-maala 239.
Sethu Vijayakumar and Stefan Schaal. Locally Weighted Projection Regression: An o (n) Algorithm for Incremental Real Time Learning in High Dimensional Space. In International Conference on Machine Learning (ICMI.), 2000.
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jiménez, François Fleuret, and Pascal Frossard. Localizing Task Information for Improved Model Merging and Compression. In International Conference on Machine Learning (ICML), 2024.
Peng Wang, Ruigang Yang, Binbin Cao, Wei Xu, and Yuanqing Lin. Dels-3d: Deep localization and segmentation with 3d semantic map. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018 URL http:// wttpaaaxvvvvr../.. www./rrg//////...oooo/o...o/..ss /1805.00. 04949v1.
Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong.
in Low-Rank Adaptation. In Empirical Methods in Natural Language Processing (EMNLP), 2024. URL
otgg//22//0000.//// emnlp ma main... 450.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-Merging: Resolving Interference When Merging Models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL http: /ttp://aaiivvv//ia///a/ org/ abs 2306. 01 01708v2.
Yinghua Yao. Yuangang Pan. Jing Li, Ivor Tsang, and Xin Yao. PROUD: PaRetO-gUided diffusion model for objti-obeective generation. Machine Learning, 1 113(9):511166538 2024. URL http::: //arxiv..a.///////// org. abs / /400 04493v1.
Shih- Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evalua- tion. International Conference on Learning Representations (ICLR). 2023. URL http: / //arxiv. 2a..bb../2999..1..2..2 14859v2.
15

---

Published as a conference paper at ICLR 2025
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient Surgery for Multi-Tash Learning. Advances Neural Information Processing Sys- tems (NeurIPS), 2020. URL http::/azxiv.....m.//// org/ absa 06782v4.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In IEEE
European Conference on Computer Vision (ECCV), 2014.
Zhang, Yang Zhifei, Song, and Hairong Qi. Age Progression/Regresiin by Conditional Adversarial Autoencoder. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Fine-Tuning. In International
Tuo Zhao. Adaptive Budget Allocation for Conference on Learning Representations (ICLR), 2023. URL /tattts://pppereeeee .eet/
forum? Scrum??id+1qg22uRRJ111
Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei
Wu. LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild. In Association for Computational Linguistics (ACL), 2024. URL https: /actps:/aalaattholl
org 2024. wtt.02.f.iiniigog. findings-acl.255 263.
Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for LLMs. In Advances in Neural Information Processing Systems (NeurlPS), 2024. URL https:// openreview net/ (=rrmm2.==g55114488..
Eckart Zitzler and Lothar Thiele. Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach. IEEE transactions on Evolutionary Computation, 3(4):257~ 271, 1999.
16

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL RESULTS
A.1 SARCOS
We further explore the setting for SARCOS dataset, presented in Section 5.2. The experimental setting remains the same except the number of epochs, which have been increased from 100 to 500. The results are presented in Figure 7 and show that PaLoRA converges faster to higher HyperVolume compared to PaMaL (Dimitriadis et al., 2023).
380
340
Validation HyperVolume
320
280
260
PaMaL (M (M-10)
PaLoRA (M (M-10)
PaMal (M=2)
PaLoRA (M-2)
220
PaMal (M=1)
PaLoRA (M=1)
100
400
Epach
Figure 7: Comparison between PaMaL and PaLoRA on the SARCOS dataset for longer training times.
A.2 ABLATION ON RANK r
Table 3 presents ablation the LoRA rank for CityScapes. Specifically, we consider ranks r E 21, 4, 8}. We also explore the axis of the number of forward passes M. We use 3 seeds per combination and report the mean.
Table 3: Ablation on rank for CityScapes.
M
Segmentation
Depth
mIOU t
Pix Acc
Abs Err +
Rel Err
71.56
92.35
0.0139
52.4928
1
5
71.55
92.32
0.0139
48.7060
2
71.44
92.37
0.0144
45.5715
2
71.47
92.26
0.0139
49.1715
4
3
71.58
92.37
0.0141
43.2215
71.44
92.29
0.0141
41.2195
8
3
71.44
92.41
0.0144
49.9970
71.64
92.38
0.0147
48.9209
A.3 DETAILED RESULTS FOR MULTIMNIST AND NYUv2
Table 4 and Table present the results for the Mu t MNTST and NYUv2 benchmarks, respectively, including standard deviations. Three seeds are used for each method in both benchmarks.
17

---

Published as a conference paper at ICLR 2025
Table 4: Detailed results for Mult IMNIST
Top-Left
Bottom-Right
LS
95.47 10.08
94.45
UW
95.70+0.34
+4.31
MGDA
90.
94.33+.0.1
DWA
95.52+0.07
+4.48
PCGrad
95.51+0.07
94.56110.37
IMTL
95.78
94.40..0.16
CAGrad
95.5510.11
94.17
NashMTL
95.84
94.78
RLW
95.41
94. 94.06=20.24
GradDrop
95.40+0.14
94. 94.24400.4
AutoL
95.94
94.57
RotoGrad
95.92+0.25
94.48
PHN
96.04
94.91 94.91 +0.46
COSMOS
94.08
93.9 20.35
PAMAL
96.17 95.32
PaLoRA
96.5510.13
15.39.0.24
Table 5: Detailed results for NYUv2.
Segmentation
Depths
Surface Noemal
Angle Distance
Moan
Median
11.25
22.5
30
0.2737 S00
25.27
32 g3.3
50.00guur
70.0guat
18
36.33
96
0.5507 0.5507gc.oee
0 0.2224
71
84
26. M.1.3
51.72
63 63.9
L/W
31.68....7
60.1 10.0
0.5609 ss86
0.2242 0093
28. 3.
22.50
25.74
50.33,am
62.40g100
31.19.0.m
2m
24.84
18.3
32 45.15
39.04.1m
70 28.25
DWA
37.09.2.50
63.00 63.60.1..
0.5463_cc4000
0.2225
27.57.006
79.79
26.50 26.50
61.78,1.0
64.02,1.1
POGra
43
0.5504
4.55044cc.coo sc.cco
0.2182:no0ms
47.4
41.57
26.0 Pants
52 25
6444
IMTL.
36.43.2.00
63.
0.5437
25.87
19:63g0.
06
67.00
NashMII.
37.66-2.02
64, 76
0.6279
U.5279
0:2100
26.56
19.3.ps
30 30.50amm
36.9
68 55
a3.66.
a..
0.5692,c.a1ze aceesto
100003
29.06
23.68
24.06
48 a7.2
60.67.11
GradDrrp
36.98_1.68
23.3)
0.5423_
0.2204_...0om
27.6420m
80
26 26.38
61.64g1.0
63.0511
PaMal
33.94.1.2
62.5 25
a.5592
26 15
20.33
29 10.09aaa4
5166an.r
66.34an
PaLoKAA
38.27
64.75 64.79
0.647
1.637_e-8006
0.210 0017
25.06gn
19:34
30.47a0
36.9
68.66
180
0.00
004
617
005
Toa
108
E54
100
Figure 8: Qualitative results iMMISTMM measuring accuracy three digit classification tasks. PaLoRA is able to cover the entire Pareto Front.
18

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS
All experiments are conducted with PPazke et al., 2019) in Tesla V100-SXM2-32GB GPUs. We use three seeds per method. Our source code extends the codebases of previous works (Dimitriadis et al., 2023; Navon et al., 2022; Liu et al., 2019). We use the Mult iMNI ST variant from Dimitriadis al. (2023). For CityScapes and NYUv2, we use the variants from Liu et al. (2019), which are also explored in Navon et al. (2022); Liu et al. (2021; 2024a; 2022).
B.1 FULL TRAINING
MultiMNIST We use the same settings as Dimitriadis et al. (2023). For all experiments the rank set to = 1. We perform a grid search on the scaling a E f1,2,5, 10) and annealing temperature and report the results in Figure and Figure 10 in terms of validation accuracy and loss. We use seeds per configuration and report the results that achieve the highest average hypervolume (Zitzler & Thiele, 1999). Training lasts 10 epochs.
M
The experimental settings are the same as Mult MultiMMIST, apart from the num- ber of epochs that has been increased from 10 to 20.
UTKFace We use the same settings as Dimitriadis et al. (2023).
SARCOS
We follow the experimental protocol presented by Navon al. (2021).
CityScapes We use the same settings (Dimitriadis et al., 2023). A modified version of the same experiment was also presented (Liu al., 2019; Yu et al., 2020; Liu al., 2021; Navon et al., 2022); the difference is the existence of validation set and training lasts 100 epochs. We use the SegNet architecture (Badrinarayanan et al., 2017), not the MTAN (Liu et al., 2019) variant for computational reasons. We train for 100 epochs.
NYUv2 We use the experimental settings from Navon et al. (2022). We use SegNet architecture (Badrinarayanan et al., 2017), the MTAN (Liu al., 2019) variant for computational reasons. We use 95 out of 795 training images for validation. We train for 200 epochs.
B.2 RUNTIME
The runtime depends on the number of forward passes, denoted as M. We also perform evaluation on held-out dataset every l epochs. Evaluation for PFL is expensive since we evaluate the perfor- mance for models formed for various preferences E Ar. Hence, evaluation time scales linearly with the number of sampled points. For instance. for two tasks, CityScapes, NYUv2 take min, a gradient balancing
11 evenly spaced points
in [0.1]. For M 1, the experiments on MultiMNIS 90 mins and ~ 4.5 hours. For the scene understanding benchmarks, algorithm similar to (Chen et al., 2018 8) and used in (Dimitriadis et al., 2023), since the results were superior. Gradient balancing algorithms have longer runtimes compared loss-balancing ones, e.g., (Caruana, 1997; Cipolla et al., 2018). The above runtime numbers refer to gradient balancing.
B.3 PARETO EXPANSION
We use exactly the same settings as in full training, the only difference is the number of epochs. For MultiMNIS checkpoint, denoted as 0g in the main text, corresponds to the first seed the Linear Scalarization (Caruana, 1997) baseline.
use 4 epochs and for CityScapes 5 epochs. In both cases, the initial
19

---

Published as a conference paper at ICLR 2025
(oad-
RateCred
PatARA (eard
0-1,-2
1=
Accuracy Bettam Right
1.10
Accuracy Bettom Right
Bottom
0.00
Accuracy Left
Accuracy
Accuracy Left 2.T
2.7 -
Accuracy Bottom Right
Accuracy Bottom Right
0.56
0.54
Accuracy
0100
Accuracy Top Left
Accuracy Top Left
Accuracy Top Left
3.T -
0-5,T-2 -
3 -
Acaracy Battom Right
Accurcy Bettom Right
0.34
Accoracy Bottom
0.66
0.91
Accuracy Top Left
Accuracy Top Left
Accuracy Top Left
a= =0
=0,77
10.77 7
Bottom Right
Bottom Right
Recuracy Bettom Right
0.58
0.68
0.34
0.50
0377
I:54
6000
0.44
Accuracy
Accuracy
Accuracy
Top
Figure 9: Ablation on the scaling parameter a, defined in Equation 2, and the annealing temperature T for the validation set of Mule SM.TS. We omit PFL baselines to reduce visual clutter and present 3 seeds for each configuration. We observe that higher temperatures and scales leads to larger Pareto Fronts. Figure 3a presents the test results for the configuration with the highest mean hypervolume.
20

---

Published as a conference paper at ICLR 2025
15
CAlirs
Graddrop Auto-
Pacatts (eend
POG.d
(owed-0))
(swed..
Lozs Bottom Right
0.24
Loss Bottom Right
Loss Bottom Right
0.50
0.16-
Loss Top
Top Left
Loss Top Left
2,7-
2,7-2
Loss Bettom Right
Loss Bottor Right
Loss Bottom Right
0.92
0.18-
Loss
Top Left
Loss Top Left
Lecs Bottom Right
0.54
Loss Bottom Right
Loss Bottom Right
0.20-
0.16
Loss Top
Top Left
Loss Top Left
10, 7-
1.055
Loss Betton Right
Loss Botton Hight
Loss Bottom Right
0.92
0.384
0.000
11:22
01150
5.135
11255
0.00)
14200
0.175
0.001
01225
u300
0.125
0.150
14375
11201
12225
Left
Left
Top
Left
Figure 10: Validation losses for the setting outlined in Figure 9.
21

---

Published as a conference paper at ICLR 2025
Figure 11: Examples of annealing in the case of tasks for 10 points. Similar to the 2d case, the points are initially set in the center of the simplex and gradually towards the edges.
C DETERMINISTIC SCHEDULE
Section 4.2 discusses the deterministic sampling mechanism. In Equation 3, the preference vectors for given timestep
function of the base preferences (mm) Jm=1- For tasks, we use torch. 1et..innsppe linseeee (0, 1, M) to produce X in [0. 1]. The vector preference then À = [X, A]. For 3 tasks, we the shoo library to produce the initial evenly distributed set in the simplex. Example for the 2d case are provided in Figure 2, while Figure 11 shows the case for
[0, 1]
three tasks.
D DETAILS OF THE ABLATION STUDY ON PREFERENCE SAMPLING
This section describes in greater detail the settings of the ablation study of Section 6.1. We ablate on the following dimensions:
1. number of forward pass M E 7.95,
2. scaling a E 21, 10). defined in Equation 2,
3. the following sampling schedules:
(a) Deterministic sampling with annealing, defined in Equation 3. We explore tempera-
ture parameters Q E {1, 2.5).
(b) Deterministic sampling without annealing, defined in Equation 3 if regardless
of training iteration. explore temperature parameters Q (1,2,5) 5).
(c) Random sampling with annealing using the Dirichlet distribution Dirichlet(p(1- -))1
for E 21,2,35 5).
(d) Random sampling without annealing using the Dirichlet distribution Dirichlet(y1) for E (1,2,5)
Prior works used solely schedule (a). We use 3 seeds per configuration and present the results as a scatter plot in Figure 12. Figure 12 shows the results focusing on the sampling mechanism only. Figure 13 and Figure 14 supplementary Section 6.1 of the main text, showing the effect of other hyperparameters as well.
22

---

Published as a conference paper at ICLR 2025
10
00
Anneaing
Food
correaatoon
0.5
Anneafing Random
Fandom
0.-
Optinal
-05
0.72
0.73
0.74
0.15
0.76
0.78
0.79
Hypervalume
Figure 12: Ablation Study the choice of preference vector sampling. Each scatter point corre- sponds to a different combination scaling a and number forward passes M E N. Deterministic schedules result in lower Spearman correlation reflecting more functionally diverse Pareto Fronts.
M
0.75
M
corrdation
0.50
9
Arneaing Random
Random
0.25
2
-0.00
Optimal
-1.00
0.73
0.75
0.77
0.79
Hypervolume
Figure 13: [Complement to Figure 12]: Ablation Study on the choice of preference vector sampling (in color) and the effect of scaling a (marker size) and number of forward passes M N (marker shape). Deterministic schedules result in lower Spearman correlation reflecting more functionally diverse Pareto Fronts.
Number nondiminated points (ouz of 11)
10-
M
M =.
8-
xXx
Annealigg
Fozd
Food
Anmuiing
Random
=
10
Optinal
0.72
0.73
0.74
0.75
0.76
0.77
0.78
0.79
Hypervolume
Figure 14: [Complement to Figure 13]: The y-axis changes to the number of nondiminated points.
23

---

Published as a conference paper at ICLR 2025
E PROOF OF THEOREM 2
Theorem 3. Let Jt X x O + y be a family of continuous mappings, where t = 1,...,T, X C DR. is compact. Then, Ve O, there exists ReLU multi-layer perceptron with three different weight parameterizations Oo, 0,, 02 E θ such that Vt E [T], 3ox E [0, 1]. Væ X:
,,, and
lf(()) f (x; Og + ab, + (1 (1-a)00)) c.
The following proof is based on the proof provided by Dimitriadis al. (2023).
Proof. Following the universal representation theorem, there exists E N, M E (DD+1)x B
RRQ M' RQxDD such that for single hidden layer perceptron with non-linearity a such that:
A 00 1 + R 2 + (M
Ya E A, {l,..... N1. fu(z)) 9 [11...,;27 DD. N 0-11 1 1) c.
For matrices
i= -
R
i= 2j otherwise
1
j = 2i
[S]
-1
2i
0
otherwise
(0
is 2D
[Up]s
(k i= 2D +1
For x E D,.,, we have
Va 2 0,
So +0 + aU, + (1 a)) = (T,,..,TD,,.))
For 0 (R, U, MS, B, M 0, U, MS, B, B, M') and 0, (R, MS, B.
f(ar,, u, m, b. m²) m'oom (ræ + +u) b),
then
f(r; - 00; + - (1-a)0,) = f(x; R, Up + aU1 (1- u2 MS B,M)
a)U2) + B) g(So Uo Uo (Q-
(M
Up u
= -(
24


Reviews:
[{"review_id": "Oyjn4odVHy", "reviewer": "Reviewer_vwtQ", "summary": "This paper introduces a new method called PaLoRA (Pareto Low-Rank Adapters) to address efficiency issues in multi-task learning. PaLoRA enhances any neural network architecture by adding task-specific low-rank adapters and continuously parameterizing the Pareto frontier within their convex hull. This approach encourages the original model and adapters to learn general features and task-specific features, respectively, thereby improving the model's flexibility and efficiency.\nAdditionally, the paper proposes a deterministic sampling schedule. This schedule reinforces the division of labor between tasks, accelerates convergence, and strengthens the validity of the mapping from preference vectors to the objective space. Specifically, the sampling schedule evolves over training time, gradually shifting the focus from learning general features to learning task-specific features, ensuring the effectiveness and stability of the training process.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-organized and clearly written.\n2. The idea of decouple a neural network into a general feature extrator and several task-specific low-rank adapters is reasonable.", "weaknesses": "1. The key equation (2) is quite similar to LoRA (Low-Rank Adaptation). And there are many multi-LoRA methods proposed for multi-task learning. However, the authors have missed. Please tell us the benefits of the proposed method compared with the multi-LoRA methods (e.g., [1-4]).\n2. Fig.2 is difficult to understand. First, how is $\\lambda$ changed over time? Second, how to judge the quality of the mappings from preference to objective space?\n3. For Fig.3, why are there fewer or even no compared methods in (b) and (c)?\n4. In Table 1 and Table 2, the proposed PaLoRA performs similarly to the other MTL methods. If the main purpose is accelerating convergence and reducing the memory requirements, then, I think it should be compared with the multi-LoRA methods [1-4].\n5. The effects of the hyper-parameters (e.g., $\\alpha$, $r$, and $Q$) on the results are not well studied in the experiments.\n\n[1] Mixture-of-Subspaces in Low-Rank Adaptation, EMNLP 2024.\n[2] Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild, ACL 2024.\n[3] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning, NeurIPS 2024.\n[4] Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models, COLING 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bdlRDWTdb4", "reviewer": "Reviewer_9e2E", "summary": "The paper presented a multi-task learning method using low-rank adaptors. The paper formulates the multi-task learning problem as Pareto Front Learning, where points on the Pareto Front are sampled and optimised in training. Different from previous work which uses random sampling for such point, the paper proposed to use fixed interval, sampled either evenly or using a pre-defined schedule. Across different benchmarks, the method outperforms other Pareto Front Learning methods, while sometimes underperforms conventional multi-task learning methods.", "questions": "1. Why does PaLoRA has a memory overhead compared against the MTL baseline? If I understand this correctly, only the LoRAs are trained, albeit one for each task. But training a LoRA only costs a fraction of the memory required for fine-tuning the whole model, which is needed for MTL if I didn't misunderstand the settings.\n\n2. For M>1, is it essentially doing gradient accumulation? Otherwise what does multiple forward passes refer to?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method uses low-rank adaptors to encode weight updates in multi-task learning, and is more efficient compared to previous work. Specifically, it achieves over $\\times 20$ reduction of memory usage. This is particularly impactful in real-world applications, where large neural networks are typically memory-intensive and costly to scale.\n\n2. The proposed deterministic sampling of preference, i.e., $\\lambda$, seems like a notable improvement over the existing random sampling strategy. The annealing strategy is fairly well motivated. By gradually moving $\\lambda$ from the centre of the simplex to the faces, the sampled point transition from general models to more task-specific ones. This was shown to increase the convergence speed and stability.\n\n3. The paper presented fairly comprehensive experiments across multiple benchmarks and demonstrates the superior performance of the proposed method over existing PFL methods, despite the occasional lower performance compared to MTL methods. The paper also includes rich visualisations that very well illustrates the ideas behind the method and the experimental results.", "weaknesses": "1. Being able to construct a Pareto Front of models is in itself an interesting result. But what is the practical value of this? The objective of multi-task learning is to produce one model that performs well on two or more tasks. By this definition, it seems sufficient to just have one model, or rather one point on the Pareto Front. I think it will make the paper much stronger if some practicality can be demonstrated, besides the nice theoretical result. Perhaps the Pareto Front can help in OOD scenarios. For instance, after a model is trained on cityscapes for segmentation and depth estimation, when it is tested on MS COCO for segmentation, I imagine MTL methods might be less effective due to the distribution shift, but the proposed method can still adjust the coefficients/preference to the new dataset.\n\n2. The contribution of the fixed schedule is not the most convincing. The disadvantage of random sampling according to the paper seems to be that it produces a tight cluster of sampled points. This may be related to the distribution used rather than the random sampling process. In addition, there doesn't seem to be any experiments showing the improvement of the annealed schedule compared against the even-sampling strategy.\n\n3. The motivation of the annealed strategy seems to be mimicking the training dynamics, where the network learn generic features in early stages while specialise in later stages. I don't think this is a homogeneous behaviour across the entire model. Usually the shallow layers of model extract generic, low-level features while the deeper layers extract task-specific features. Perhaps it makes more sense to apply this annealed schedule layer wise, instead of chronologically. There is in fact an increasing body of work on multi-task learning/model merging using learned layer-wise coefficients, such as Zhang et al. (2024) and Yang et al. (2024). I think the annealed schedule will make more sense in this context. \n\nRefs:\n\n- Zhang et al., Knowledge Composition using Task Vectors with Learned Anisotropic Scaling. NeurIPS 2024\n- Yang et al., AdaMerging: Adaptive Model Merging for Multi-Task Learning. ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ArNccLQt6l", "reviewer": "Reviewer_amQG", "summary": "This paper proposes Pareto Low-Rank Adaptors (PaLoRA), which are claimed to address the problem of limited scalability, slow convergence, excessive memory requirements, and inconsistent mappings from preference to objective space that existed in previous methods. The goal is achieved by adding multiple task-specific low-rank adapters and optimizing the weights with Pareto Front. Experiments on multiple tasks and variant datasets show the effectiveness of the proposed method.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes to address multi-task learning with multiple low-rank adapters equipped with Pareto Front Learning.", "weaknesses": "1. It lacks discussions between PaLoRA and MoE-like methods.\n2. It lacks discussions between PaLoRA and general adaptive weight learning methods. How is PaLoRA better and why?\n3. It seems that this work applies PFL on multiple Lora, any significant differences between general PFL applications?\n4. Comparisons between GFLOPS/Memory/Speed with other methods at the inference stage?\n5. What is \"Functional Diversity\" in Sec 4.2? Any definitions or quantitative evaluations?\n6. Any comparisons between PaLoRA and other LoRA-using methods on multi-task learning?\n7. It points out the problem of \" inconsistent mappings from preference to objective space\", but no experiments and explanations validate the improvements or the relationship between mapping consistency and good performance/generalization on multiple tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1vXahbhQSd", "reviewer": "Reviewer_EQhb", "summary": "PaLoRA presents a method for Pareto Front Learning (PFL) with two main contributions. First, the paper adds a low-rank adapter for each task, significantly reducing the overhead of the additional parameters added to the network. Second, they perform a deterministic (either fixed or annealing) preference sampling, leading to more stable learning and faster convergence.", "questions": "Please refer to the \"weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is more efficient regarding number of parameters, i.e., memory complexity, compared to PaMaL and outperforms it as well and they also scale up to larger number of tasks (as shown with the SARCOS).\n- Interesting discussion sections that shows the efficiency of their method in converging faster and also the multiple adapters learning different tasks.", "weaknesses": "- What happens if you train longer for the SARCOS dataset? I.e., Figure 3 (c) goes for more than 100 epochs (until convergence of each model)? How does PaLoRA compare to PaMaL?\n- Have you done a ablation over different ranks for the LoRA modules? It would be great if you could report the numbers for different ranks.\n- Please provide the standard deviation for the runs in Table 1 and also for the numbers for MultiMNIST (given the variance among the performances of PaLoRA on MultiMNIST and MultiMNIST-3).\n- Would be great to have another dataset added to the experiments, such as  UCI Census-Income.\n- Figure 3 (b) could improve and have the legends in the image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)