FedRAM: Federated Reweighting and Aggregation for Multi-Task Learning
Authors: Fan Wu
Xinyu Yan
Jiabei Liu
Wei Yang Bryan Lim
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Multi-Task Learning
Primary area: other

Abstract: Federated Multi-Task Learning (FL-MTL) enables clients with heterogeneous data to collaboratively train models capable of handling multiple downstream tasks. However, FL-MTL faces key challenges, including statistical heterogeneity, task interference, and the need to balance local learning with global knowledge sharing. Traditional methods like FedAvg struggle in such settings due to the lack of explicit mechanisms to address these issues.
In this paper, we propose FedRAM, a three-step framework that progressively updates two scalar hyperparameters: the task importance weight and the client aggregation coefficient. FedRAM introduces a reference-proxy-agent strategy, where the proxy model serves as an intermediate between the local reference model and the global agent model. This design reduces the need for repeated local training while preserving local performance.
Extensive experiments on six real-world FL-MTL benchmarks show that FedRAM improves performance by at least 3$\%$ over the most baseline on both in-domain and out-of-domain tasks, while reducing computational cost by 15$\times$. These results make FedRAM a robust and practical solution for large-scale FL-MTL applications.
The code is available at \url{https://github.com/wwffvv/FedRAM}.

FedRAM: Federated Reweighting and Aggregation for Multi-Task Learning
Fan Wu Xinyu Yan'
Jiabei Liu'
Wei Yang Bryan Lim
Nanyang Technological University
(fan009, xinyu020, S200131 .nt.ueeeeuu.......meee.eeeeeuee. edu.sg}
bryan www.wlitiimmeeeeeem
Abstract
Federated Multi-Task Learning (FL-MTL) enables clients with heterogeneous data collaboratively train models capable of handling multiple downstream tasks. However, FL-MTL faces key challenges, including statistical heterogeneity, task interference, and the need to balance local learning with global knowledge sharing. Traditional methods like FedAvg struggle in such settings due to the lack of explicit mechanisms address these issues. In this paper, we propose FedRAM, three- step framework that progressively updates two scalar hyperparameters: the task importance weight and the client aggregation coefficient. FedRAM introduces a reference-prox praxy-agnnt strategy, where the proxy model serves as an intermediate between the local reference model and the global agent model. This design reduces the need for repeated local training while preserving local performance. Extensive experiments on six real-worke FL-MTL benchmarks show that FedRAM improves performance by at least 3% over the most baseline on both in-domain and out- of-domain tasks, while reducing computational cost by 15x. These results make AM robust and practical solution for large-scale FL-MTL applications. The code is available at hitpps::.... /gg.ttu....... com /mmm/wfff//FedRA!
Introduction
Federated Learning (FL. key paradigm in distributed machine learning, enabling multiple clients to collaboratively train models while preserving data privacy. FL excels in scenarios where clients share similar domains and statistically homogeneous data distributions. However, in real-world multi-task learning (MTL) settings, significant data heterogeneity degrades global model performance and impedes effective knowledge sharing. Consequently, existing FL methods 24 struggle strike a balance between in-domain performance and out-of-domaad generalization in MTL contexts. To be specific, we refer the term domain as the data domain accessible clients.
key limitation of traditional FL [T] lies in its reliance on fixed aggregation coefficients, often proportional to local sample sizes, illustrated in Figure a)). While effective in homogeneous settings, this approach fails to capture task-specific variations in MTL scenarios, leading to subop- timal performance due to imbalanced client contributions. MTL settings commonly exhibit task heterogeneity and non-lID data characteristics, such as feature and label distribution skews, class imbalances, and quantity disparities (5) To address these challenges, FL-MTL methods (6) leverage task correlations to enhance local data representations. As illustrated in Figure (Tb), incorporating similarity-Bee adjustments, through representations, tasks, models, improves task alignment and local training efficacy.
Despite these advances, FL-MTL methods inadequately address imbalanced client contributions, where variations in data volume and relevance disproportionately impact the global model. This stems from their focus on in-domain performance. Recent work model merging 7 demonstrates the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Chant 1
FL
Ours
S S
of
L0
Clieni
QL1
1.0
Task
10
S S.
Task
A202
1.0
(a) Traditional FL
(1) FL-MIL
(c) FadRAM
Figure I: Comparison of aggregation strategies in federated learning. (a) Traditional FL aggregates local models using fixed coefficients X1 X2 (e.g., weighted by local sample sizes). (b) FL-MTL
methods incorporate task correlations and data distributions to enhance local training. (c) FedRAM employs adaptive aggregation coefficients a1, 02, dynamically adjusted by a proxy model to balance client contributions.
efficacy of adaptive aggregation weights for task-specific models, while g emphasizes quantifying client contributions to ensure aggregation fairness. These insights motivate us to propose a more adaptive reweighting mechanism from three perspectives: (1) effective knowledge sharing, (2) robust generalization, and (3) accelcrated convergence.
In this work, we propose FedRAM, a novel framework tailored for heterogeneous multi-task dis- tributed environments. FedRAM employs three-step architecture, with each model assigned distinct role: (1) compact local reference model 6,e that captures task- specifcc distributions and serves as a benchmark for proxy model training; (2) a compact federated proxy model (poxxy that dynamically adjusts task-specific weights and client-specific aggregation coefficients, as illustrated in Figure Te), and (3) large federated agent model 0 agent that performs reweighted aggregation to produce the final global model. This hierarchical design enables FedRAM to adapt to in-domain variations while achieving strong out-of-domain generalization. Our contributions are summarized as follows:
We propose FedRAM, novel FL-MTL framework that tackles multi-task heterogeneity through three-stage training process. Each stage employs tailored strategies aligned with the model's functional role.
We introduce task-specific and client-specific weights as key hyperparameters in FedRAM, along with their tuning processes. These mechanisms enable fine- line-grained control over task prioritization and client contributions to enhance adaptability and performance.
We conduct comprehensive experiments across diverse datasets, demonstrating that FedRAM significantly outperforms state-of-the-ar methods. Our framework achieves superior accuracy in both in-domain and out-of-domain evaluations, faster convergence, and reduced computational costs, validating its effectiveness and efficiency.
2 Related Work
Federated Learning. The seminal FL approach, FedAvg M. aggregates locally trained models into global model by averaging client updates, offering communication efficiency and privacy benefits. However, FedAvg assumes that client data are independent and identically distributed (IID). When
data distributions are non-III. FedAvg exhibits slower convergence and reduced accuracy 9 Subse- quent works relax this IID assumption and emphasize personalization. FedProx [10] incorporates proximal term reduce client drift. Ditto [I] introduces client- specific regularization to personalize models while still learning shared representation. More recently. methods like MOON [12] use a contrastive mechanism to guide local updates based on global representations. DBE tackles domain biases by preserving both generic and client-specific knowledge. Despite these advances, standard FL methods predominantly focus on learning a single global model or mildly personalized models, which is often insufficient for complex multi-task scenarios.
Federated Multi-Task Learning (FL-MTL). Addressing the limitations of single-model FL, FL- MTL tailors models to each client's 's specific tasks, while still exploiting task interdependencies 6 Early FL-MTL approaches, such as MOCHA O and MIFA [I3, jointly learn task-specific and global parameters but may suffer from high computational costs as the number of tasks grows. More recent frameworks adopt architectural strategies to improve scalability. FedDAT [1] employs multi-modal foundation models with dual adapters, enabling efficient sharing of representations for different data modalities. FedBone [15] decouples a shared backbone from task-specific layers, balancing communication efficiency and task-specific performance.

---

Client Domains
Server Domain
Mixed Tasks
Ssip TToil Barenics Modil Trintig
Notations
Quentonn Armozeing
Praxy
Lipcare
00b Frees
Agurecittion
it's strll and shordd model structure for thet Ter Beray and okwaep
Faderate Prony Tiining
Completon
8
Parallenso ldveiifratoon
2. Dedtriude
2 Update Prip
peany
2 Upood
medel structire gigeet
ligerr stand
Step Federated Agent Triinng
Cliest W Weights
and P
Carelenezao Reolction
3is Uplood
Bigent
33.d Acgrocation
Model
Model
: mmmmhhi Parameters
28 Uptite PERE
Clent/Serve Openaiin
Spent
Communication
Figure 2: An overview of FedRAM framework comprising of Reference, Proxy, and Agent models.
Despite these innovations, a key challenge remains in how to fairly aggregate client contributions and mitigate interference across tasks. Naive sizes) may overlook important factors like task complexity or relevance to the global objective (9. Several methods have proposed alternative aggregation schemes. Contribution-awat FL [16] weighs clients by measuring improvement to the global model, while [7] and 7 explore adaptive model merging to reduce parameter conflicts. Nonetheless, most existing work primarily targets either solution to non-IID distribution or coarse-grained multi-task modeling, without an explicit reweighting module that can adapt both client- and task-level contributions in unified federated framework.
weighting (c.g.. proportional to data
Unlike prior work focusing solely on single global model or static weighting heuristics, FedRAM introduces a framework that systematically adjusts task and client aggregation weights based on local performance improvements. This design alleviates inter-task conflicts and reduces computation and communication overhead by separating lightweight proxy updates from larger agent model training.
3 Problem Statement
Consider scenario with total K clients and each possesses local training dataset mixed by distinct T tasks. We use DE to present the data from the k-th client and 7-th task (D, denotes local data). The overall goal is train FL model that balances local task-specific distribution and adaptation to global knowledge. We consider two non-negative weights: task weights a [axi,... Qly, ... QTT, T== ar and client weights optimization goal of model 0 is formulated as:
[Wi,... .... Wky ... wk]. Ek= wk 1. Building this, the
T
min L(0; or; w)
Σ wk Σ (DD
8,a.w
k=1
We decouple this objective as a local loss Liocal and a global loss Leghbual:
Giccll Lo, (Dr.) Σ aLo (D7, Co, (DF))
(2)
We derive ar by two steps: Minimizing Le, (Dt); Minimizing the second term by co-optimizing Og and ar. We assign 0 and 0g with the same model architecture.
K
Legiobal
Wy
(3)
Our proposed framework follows a three-step sequential solution: a, w and 0g. respectively. In our implementation, we refer to the three distinct models O Opexxy. and Ougent-
4 Proposed Method
As illustrated Figure FedRAM employs distinct notations: @ and 05 differentiate models deployed on the server (Global model) and client devices (Local models), respectively; 0* and 6
3

---

Step 1 Local Training
Step
Step 3
Peat (0")-
Cexe
Tune
Resampling
Update
Oynney (0")-
FL Training
Cad
Tune
Aggregation
Pagens (0')-
FL Training
0
agmit
Figure 3: A functional overview of the three models in FedRAM.
distinguish between smaller and larger model architecture. Specifically, FedRAM consists of three main functional models (Figure 3). summarized as follows:
Reference. With previously training in the local client's domain in Step 1, the frozen 0şr (Step 2) is kept local and then used to calculate the excess loss Lexc. which helps train Proxx and adjust the task weights or-
Proxy. By applying smaller model architecture 0", the FL-trained proxx learns prior knowledge of task heterogeneity at lower cost. In Step 3, Oproxy tunes the client weights wk-
Agent. At the final stage AM, larger model Oageni trained FL for final evaluation. The training process incorporates the tuned parameters (for resampling local data) and wi (for weighted aggregation), which are obtained through O and praxy.
According to the Algorithm for FedRAM, we illustrate each step in the following sections.
4.1 Step 1: Local Reference Model Training
FedRAM benefits from local reference model training in two ways: set of locally adapted reference models t%r enables the adaptation process to local multi-tast distributions; 2. Applying lightweight model architecture lowers the computation cost. The key steps are as follows:
Model Initialization: Clients receive globally shared model structure d"
Local Optimization Objective: The reference models optimized to gain better represen- tation of local distributions:
1 [DL] I1sa (a)) -y112
min L(OL))
02
where (ar, y) denotes the data and label in local dataset Dr. [Dr.] is the dataset size.
Local Updates: Clients independently and locally update the parameters of d*, thus deriving the reference models gr.
0fr <- 0 AV oz. L(Dr; O%a)
where represents the learning rate, L represents the loss function.
Parameters Freezing: The well-trained reference models 8 freeze their parameters and are kept within the local domain. Specifically, Section 4.2 the inference outputs of 0L are utilized to compute excess losses for training the proxy models frowyy.
Once each client has updated their reference model and reported to the server, the server would embark the Step 2 proxy model training process.
4.2 Step 2: Federated Proxy Model Training
In this subsection, we introduce the excess loss updates the Local Training process and the lask-weights-baseee based aggregation scheme in the Global Aggregation process.

---

4.2.1 Local Training of Proxy Model
In local training round, the proxy model employs group distributionally robust optimization [18] to fine-tune the model parameters e provy. The loss update process consists of the following steps:
Initialization: To align with reference models, the proxy models O and 0 are randomly initialized with for lightweight training.
Task-wise Loss Computation: To better measure the contributions by distinct tasks, task-wise datasets DE E Dr are divided within the domain of client kt The split of different tasks aligns with the task weights or, (T = 1,2, 7). The task-wise loss denoted as l = [DD;; prowyy)
Excess Loss Calculation: Following the notation of task-wise losses ET, the excess loss caleu- lation involves the inference loss fee of frozen reference model and f (proxy of the updating proxy model. To quantify the potential improvement between the federated proxy performance and the reference headroom [9]. the task-wise excess loss is calculated as:
fx max
[D] (Crroyy
fer)) 0}
(6)
where [DF,] is the dataset size of Di Note that positive values mean that the proxy model represents the local distributions better than the reference model.
Dynamic Task Weight Adjustment: Task-specific weights (a E R=x1) are iteratively adjusted:
(7)
where 7r E Thask is an exponential scaling factor for task controlling the scaling span. For
hyperparameter study of the exponential scaling factor T/. please refer to Appendix [33 A smooth parameter is introduced to balance and the dynamically calculated a
sa (1 s)a+. Besides, normalization employed to bound and ensure Σ 1.
Local Optimization Objective: The proxy model's local optimization aims to minimize the combined weighted excess losses across tasks:
min [(Op a) g
Σ a.
(8)
This objective ensures that local training addresses the specifies of each task.
4.2.2 Global Aggregation of Proxy Models
FL aggregation facilitates global knowledge sharing related to task-specific adjustments. FedAvg is adopted as baseline in the aggregation of proxy models:
K
@ (t) +- prony (t - 1)
Σ R=
Agk
pruxy (t),
(9)
Drl
where denotes the i-th communication round, and represents the le-th client. the following sections, the symbol 0h instead of 05 is used to explicitly denote the k-th local model.
4.3 Step 3: Federated Agent Model Training
Agent models 0L Oggent and 0G agen act the decision-maker and perform primary evaluation subject FedRAM framework.
Resampling Client Dataset. By utilizing the task weights generated in Step 2, the start of each communication round, resampling each client domain ensures the heterogeneity in the MTL setting be contributed proportionally the global learning objective.
Optimizing Client Weights. FedRAM adjusts the merging weights un for each agent model based on the differential loss. Similar to Equation 6 aand]]
7
the differential loss lai is defined as below:
Edit
(prxxy
(10)
1D]
5

---

and the client merging weights are formulated by:
<- W ar.
(11)
where ?h is an exponential scaling factor. Here, a smoothing parameter and normalization can be introduced as in Equation] 7 A hyperparameter study of is provided in Appendix B.3
Local Optimization Objective: The optimization aims to minimize the local losses:
min L(D mggnn)) Spenn
(12)
Global Aggregation and Model Finalization. After optimizing the aggregation weight, the global model aggregation is performed using reweighted client weights:
g oegont (t)
8 Oegone (t 1)
Whe AAt agent (t).
(13)
where the 0% Oigemt instead of 6 agen is used provide a more fine-gratd fine-grained representation client k. To further improve FedRAM, various aggregation methods can be used as an alternative combination. We provide the convergence proof Appendix A
Algorithm I: FedRAM
Input: Base models 0* f (with initialization 0L BC Ofrwxy: Proxy f- 0" OL agent: BG
agent (-),
number of clients K, exponential scaling factors Thask and 7/ctient smoothing parameter 8, client & training data Dj, learning rate
Output: Global agent model agent
STEP 1: Train Reference Model
for client &s = 1, ..., K do 0 f- jet BV [(Ds; 0%te
Local Training
STEP 2: Train Proxy Model 5 for Communication round = 1.., ...,..Tr, do
for client & = 1, 2, .... K do
0g. Proxxy (f) +- G0 proxy ( 1);
a(t) c alt 1)
> Update Last Round Parameters
for lask T 1, 2, .... T do
FoR + max Vos L(DI; ; Oproxy)= - -Va VOLC(DE; 0ld)) /DEE1.03
10
(t)) t a(t) eltaaoo by Lexe = 1l Voxc lox....., [.c]
>> Update
11
o oprnxy( (t) + Prroay (t) 3 a(())))))))))))))))) Lexe
b Update Local Proxy Model
Last Epoch: Edit (Veg. pmny ((Da)) Vog L(Db)) /De
Lexe [ eexc: ..., ,xx ek
14
w(t w(t - 1) e Croe
Server Update w Server Aggregation
BG @ (t) <- o (t
Ex A Agk
15
STEP3: Train Agent Model
17
for Communication round = 1,2, do
for client l 1,
K do
ugee +- g mgenl (t- 1)
Resample the task datasets by size and
20
agent (t) + Bk 0 (t) B Vot L(Ds; 8l )
Update Local Agent Model
GG ageem
(t) +- gG aggem (t- 1) + Σx WK Agk ngee! (t)
Server Aggregation
Experiment
5.1 Experiment Settings
Datasets. Following the work in [7 and 77, we conduct experiments based on four diverse categories of NLP datasets, each corresponding to specific task types, including: (1) Question

---

Answering: QASC 20) W 21,, and QuaRTz [22]: (2) Paraphrase Identification: PAWS 23 (3) Coreference Resolution: Winogrande [24 and WSC 25 (4) Sentence Completion: Story Cloze 26 In constructing mixed local datasets, we especially focus on two settings as follows.
Setting /: The number of clients K' larger than the number of tasks T (K 2 T). We primarily consider K 10 and T in our main text. Thorough statistics of data distributions are provided in Figure i in Appendix B
Setting 2: An extreme heterogeneous case with cach client assigned one distinct task. We provide a case study under Setting Appendix B
Training Setup. In the experiments, employ global LoRA configuration to fine-tune the parameters. We adopt T5-small model as 0 and T5-base model as 0+, We assume equal values for exponential scaling factors (1/ and m7.) and smoothing parameter Our simulations on a cloud instance, equipped with 8 NVIDIA A10 GPUs (24 GiB of memory per GPU), 128 vCPUs (Intel Xeon Platinum 8369B), and 512 GB RAM. For the three-stage training, we employ cross-entropy loss with an Adam optimizer, setting the learning rate 3 to 1 x maximum global rounds to 50. For simplicity, we assume that all clients can participate in every communication round. More details about our experiment implementation and baselines can be found in Table in Appendix B
conducted
We set the
Evaluation Metrics. We evaluate model performance using both global and local held-out validation data. Specifically, we consider both the In- In-Domain (ID) and Out-of-Domaan (OOD) evaluation strategies to assess the generalization capabilities of the model:
In-Domain Evaluation: Evaluating the model on tasks that are locally accessible to individual clients. ID evaluation helps measure how well the model adapts to local data during training.
Out-of-Domain Evaluation: Assessing the model on tasks that were introduced by other clients and are locally inaccessible. OOD Acc reflects the model's robustness and ability to generalize.
ID Acc ΣJ Acc |PEL
ΣT E+er Acc
Σ [D]]
OOD Acc
Σ'= E++1 ID*|
(14)
where and itr denotes task and f, Acc(") denotes the accuracy for client trained on task while tested on task t, [D] is the dataset of task T and [D]] is the sample size of task T.
5.2
Results and Discussion
5.2.1
Main Results
Table I: Comparison of FL methods across diverse tasks using global, local, and in-domain/out- of-domain (ID/OOD) evaluation metrics. FedRAM achieves superior F1-scores in global and local validations, as well competitive ID/OOD performance. Scores in bold denote the best performance, and underlined scores indicate the second-best.
Tasks
Global
Local F1-Score
ID OOD
Methods
PAWS
Wino Grunde
Qun-
Story
Wiki
FI-Score
Bottom Decile
Evaluation
RTx
Cloze
QA
FedAvg D
83.36
77.90
75.99
32.44
78.30
82.20
79.78
72.71
76.34 54.79
71.68 75.66
FedProx
78.45
73.39
80.29
28.87
78.83
76.77
66.35
68.83
73.73 55.45
70.30 72.03
Ditoo [1]
83.25
77.69
77.65
30.04
77.97
80.28
79.79
72.26
77.09 62.37
70.83 71.75
FedRep 27
74.56
63.55
76.91
34.58
64.71
82.45
66.35
66.95
76.89/ 66.89
65 48 55.79
MOON [
74.45
73.39
78.88
31.91
79.21
78.24
79.79
70.31
72.71 /5.33
69.70 71.66
DBE: 8
81.22
76.44
76.25
30.82
78.40
81.56
79.79
72.08
75.56 58.53
68.09 73.16
FedMTL.
28
89.04
74.90
78.77
32.10
66.05
84.54
79.73
74.12
78.88 62.32
76.01 75.64
FedBone
85.31
75.50
78.32
35.48
55.24
81.24
79.79
71.19
71.85 61.60
67.97 70.39
FedRAM (Ours)
81.11
77.69
80.78
36.15
80.36
77.62
79.79
75.94
79.62 733.21
72.89/ 76.32
Table [resennt a comprehensive evaluation of FedRAM against SOTA FL methods across diverse tasks and metries. FedRAM consistently outperforms baselines global and local F1-scores, particularly in the bottom decile, while achieving competitive ID and OOD performance. Key observations include: (i) AM achieves the optimal global and task-wise performance, with global Fl-score of 75.94, achieving the highest performance in 4 out of 7 tasks. (ii On the QASC
7

---

task, all methods exhibit drop in Fi-score, while FedRAM maintains relatively high performance. This decline is attributed the inherent label imbalance the multiple-choice answer selection task. (iii) In terms of local evaluation, FMdRAM achieves the highest average score of 79.62 among clients, with bottom decile score of 73.21. Notably, the bottom decile typically occurs in the QASC- dominated client (possessing the largest proportion of QASC data), where severe label imbalance poses a significant challenge. While FedMTL and FedRep achieve competitive overall performance across clients, they fail to directly enhance the performance of the QASC-dominated client. (iv) For ID and OOD evaluation, FedRAM achieves scores of 72.89 (2nd) 76.32 (1st).
Compared to weight adjustment strategies such as DBE and non non-adjustment baselines (FedAvg, FedProx). FedRAM demonstrates a significant improvement of at least 3% across multiple metrics. This improvement validates FedRAM's ' improvement through adjusting task sample rates and client aggregation weights. FedRAM exhibits competitive performance correlation considered methods (including MOON, FedMTI,, and FedBone). FedRAM offers a well-balanced performance across both ID and OOD evaluations, though FedMTLL shows better II) performance (which is greatly skewed by the biased performance of PAWS and Story Cloze).
comparison to other task-
5.2.2 Convergence Analysis
Figure demonstrates that integrating FedRAM at Step 3 with established FL methods such as FedAvg, FedMTL, and FedBone, consistently accelerates convergence compared to these methods alone. As depicted in Figure 4 FedRAM yields a modest yet significant improvement in loss convergence when combined with FedAvg, though FedAvg achieves slightly lower final accuracy. Notably, AM improves the global and local F1-score by 3% over FedAvg (Table and reduces the rounds to convergence by 15% (Table 5). Figure 4b) highlights the substantial performance boost from integrating FedRAM with FedMTL, achieving faster and more efficient loss reduction compared to FedMTL alone. Similarly, Figure@e) shows that combining FedRAM with FedBone markedly enhances loss reduction. These results underscore FedRAM's effectiveness in dynamically adapting weights and learning rates to local data characteristics and task- requirements.
Feddug
FasMIL -ww-FFFFMMM-FFMM
121
FodBone FodRMM-FaodBore
FedRAM
104
3
1D
10
10"
20
Communicarion Round
Communication Round
Commanication Round
(a)
(c)
Figure 4: Convergence analysis of FedRAM integrated at Step 3 with (a) FedAvg, (b) FedMTL, and (c) FedBone, compared to baseline methods.
5.2.3 Computational and Communication Costs
Computational cost, defined as the total FL training time, depends on the number of communication rounds R and local computational complexity, while communication cost arises from the repeated transmission of considerable number of model parameters during training. Table DBE incurs the highest computational overhead (34,173s) due to its dual-module training mechanism. contrast, FedRAM achieves a competitive computational cost (12,060s) through the following optimizations:
Decoupled Training: Lightweight reference and proxy models (0*) requiring only 10-15% of total computation resources. As detailed in T Appendix B.4 the agent model O consistently accounts for 65.08%-88.0224 of total computation time across varying client scales (N 10 50).
Reweighted Aggregation Coefficients: FedRAM dynamically adjusts client update weights and learning rates during aggregation, optimizing convergence by prioritizing updates aligned with local data characteristics and task-specific requirements, as detailed in Section Consequently, FedRAM reduces convergence rounds by 15% relative to FedAvg. While FedGF and FedMTLL converge in fewer rounds, their computational costs are significantly higher than FedRAM's 12,060s. In contrast, FedRAM achieves superior balance of computational efficiency and model accuracy.
8

---

Table 2: Comparison of computational cost, convergence rounds, and communication cost across FL methods. denotes the convergence rounds for o+ denotes the convergence rounds for 0*
Computational Cost (s)
Convergence Rounds
Algorithm
Communication Cost
FedAvg
13774
21
I x K x 20+
FedProx 10
24195
25
R X K x 20
Ditto III
27782
27
R K
28+
DBE
34173
27
R x K x 28*
FedMTL. 28
24502
17
R. x x 20
FedBone
15
40989
30
R x K x 20
FedGF 29
15596
16
R x K x 20
FedRAM (Ours)
12060
18
S x K x 20"
20
5.2.4 Ablation Studies
We systematically evaluate the contributions of the functional models in FedRAM: feet, Pprxxy
Table 3: Ablation study of FedRAM main models.
Exp.
Method
Eval.
Global F1
Local F1
and 8agem- Table 3 presents the performance
of various ablated configurations, measured by
1
w/o Oier, Openxy
Orgent
72.71
76.34
2
w/o O
O Ogeet
71.17
73.58
Global and Local F1-Scores. Note that main-
3
w/o Brony
Biet
70.52
72.54
tain the three-step pipeline integrity, ablated components are replaced with randomly initial-
4
w/o H Ageen
Orenny
2.69
2.51
5
FedRAM
Ougen
75.94
79.62
ized surrogates, and evaluations are conducted across these different model configurations. In ablation study, we assess each model by comparing the complete FedRAM (Exp. 5) with configurations where specific models are excluded (Exp. 1-4).
(1) Reference Model: When assessing the only reference model (Exp. 3), both global and local metries exhibit a drop, demonstrating its limitations due to smaller parameter size. Comparing Exp. 2 and 5, excluding 6et (Exp. 2) results in notable performance decline relative to the baseline, underscoring the critical role of local training in addressing non-IID data challenges.
Combining ler and Paraay without Ougent (Exp. 4) leads to a significant performance drop, as praxy cannot independently update its parameters on local datasets. This behavior stems from Oproxy's 's design, which aligns closely with Ore locally trained benchmark. The proxy model first solves for model-agnostic hyperparameters a and w to avoid higher computational cost imposed by correlated minimax training across hyperparameter tuning and agent model training. Comparing Exp. 2 (without prrcxy. F1: 71.17) and Exp. (F1: 75.94), the proxy model's contribution to optimizing aggregation coefficients is evident.
(2) Proxy Model:
(3) Agent Model: Exp. 1 and 2 reveal the performance contribution of Oagents which serves as the baseline model. Although there is performance drop compared to Exp. 5, we can conclude that building upon the agent model's baseline performance, the reference and proxy models contribute by fine-tuning aggregation coefficients.
5.2.5 Extensions
Hyperparameter Sensitivity. Our empirical analysis reveals that the optimization of task and client weights is highly sensitive to two key hyperparameters, !uusk and Metieet- comprehensive sensitivity analysis of these hyperparameters is provided in Appendix B.3 revealing their impact.
Model Scalability. We leverage compact proxy model optimize client weights, which are then directly applied to enhance agent training at a significantly larger scale (up to 15x). The choice the reference/proxy model, when the agent model fixed, critically influences the client weights derived by FedRAM. To investigate the minimal viable size of the reference/proxy model, we introduce an Agent-to-Proxy (A/P) ratio, with detailed evaluations presented in Appendix [.4] we evaluate FedRAM in Setring with a client number set to be 10. To assess scalability real-world scenarios with larger client populations, we extend the analysis to client counts of 20 and 50. These experiments, detailed in Appendix B.5] confirm FedRAM's robustness and scalability across varying client numbers.
Client Scalability. In Section 5.2.1
Task Scalability. The heterogeneity in tasks and data distributions can significantly influence the task and client weights adjusted by FedRAM. For instance, task weights enhance performance by

---

mitigating proxy excess loss (Section 4.2 and enabling resampling of data distributions for agent model training (Section 4.3). Further analyses, including task size variations (e.g.. Heterogeneous Tasks under Setting 2) and alternative task types (e.g., Vision Tasks), are discussed in Appendix B.6
6 Conclusion
In this paper. we propose FedRAM, a novel FL-MTL framework. To address task heterogeneity. we decouple the primary learning objective into phased sub-objectives, leveraging three distinet functional models with tailored learning strategies. We validate FedRAM's effectiveness through comprehensive metrics, including and OOD performance, convergence, computational cost, and ablation studies. Additionally, provide convergence analysis and supplementary experiments in Appendices and B respectively.
Acknowledgement
This research is supported by the NTU startup grant and the RIE2025 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) (Award 12301 E0026), administered by A STAR,, as well supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL). This research project is supported by A*STAR under its Japan-Singapore Joint Call: JST-A 2024 (Project ID: R2416IR Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of
A*STAR.
10

---

References
[1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, "Commumication-<ffiiee learning of deep networks from decentralized data," in Artificial intelligence and statistics. PMLR, 2017, pp. 1273-1282.
[2] B. Liu, N. Lv. Y. Guo, and Y. Li, "Recent advances on federated learning: systematic survey," Neuro- computing, P. 128019, 2024.
[3] M. Scol and T. Kim, "Performance enhancement in federated learning by reducing class imbalance of non-iid data," " Sensors, vol. 23, no. p. 1152, 2023.
[4] B. Sun, H. Huo, Y. Yang, and B. Bai, "Partialfed: Cross-domain personalized federated learning via partial initialization." Advances in Neural Information Processing Systems, vol. 34, pp. 23 309-23 320, 2021.
[5] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, 7. Charles, G. Cormode, R. Cummings al., "Advances and open problems in federated learning," Foundations and trends in machine learning, vol. 14, no. 1-2, pp. 1-210, 2021.
[6] V. Smith, C.-K. Chiang. M. Sanjabi, and A. S. Talwalkar, "Federated multi-task learning," Advances in neural information processing systems, vol. 30, 2017.
[7] E. Yang, Z. Wang, L. Shen, S. Liu, G. Guo, X. Wang, and D. Tao, "Adamerging: Adaptive model merging for multi-task learning," arXiv preprint or ar.iv:2310002555. 2023.
[8] J. Zhang. Y. Hua, J. Cao, H. Wang. T. Song, Z. Xue, R. Ma, and H. Guan, "Eliminating domain bias for federated learning in representation space," Advances in Neural Information Processing Systems, vol. 36, 2024.
[9] Y. Zhao, M. Ii, I.. Lai, N. Suda, D. Civin, and V. Chandra, "Federated learning with non-iid data," arXiv preprint arxiv: 806. 00582, 2018.
[10] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, "Federated optimization in heterogeneous networks," Proceedings of Machine learning and systems, vol. 2, pp. 239-450, 2020.
[11] T. Li, S. Hu, A. Beirami, and V. Smith, "Ditto: Fair and robust federated learning through personalization," in International conference on machine learning. PMLR, 2021, pp. 6357-6368.
[12] Q. Li, B. He, and D. Song, "Model-contrasiie federated learning," in Proceedings of the IFFF/CVF conference on computer vision and pattern recognition, 2021, pp. 10713-10 722.
[13] A. Fallah, A. Mokhtari, and A. Ozdaglar, "Personalized federated learning: meta-learning approach." arXiv preprint arXiv:2002.07944 2020.
[14] H. Chen, Y. Zhang, D. Krompass. J. Gu, and V. Tresp, "Feddat: An approach for foundation model finetuning multi-modal heterogeneous federated learning," in Proceedings of the AAAI Conference on Artificial Intellignce, vol. 38, 2024, pp. 11 285-11 293.
[15] Y.-Q. Chen, T. Zhang, X.-L.. Jiang, Q. Chen, C.-L. Gao, and W.-L. Huang, "Fedhone: Towards large-scale federated multi-task learning," Journal of Computer Science and Technology, vol. 39, no. 5, pp. 1040-1057, 2024.
[16] D. Y. Zhang, Z. Kou, and D. Wang, "Fairfl: A fair federated learning approach to reducing demographic bias in privacy-sensitive classification models," in 2020 IEEE International Conference on Big Data (Big Data). IEEE, 2020, pp. 1051-1060.
[17] P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal, "Tres-merging: Resolving interference when merging models," Advances in Neural Information Processing Systems, vol. 36, 2024.
[18] S. Sagawa, W. Koh, B. Hashimoto, and P. Liang, "Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization," arXiv preprint 11110:1911.08731, 2019.
[19] S. M. Xie, H. Pham, X. Dong, N. Du, H. Liu, Lu, P. S. Liang, Q. V. T. Ma, and A. W. Yu, "Doremi: Optimizing data mixtures speeds up language model pretraining," Advances in Neural Information P Processing Systems, vol. 36, 2024.
[20] T. Khot, P. Clark, M. Guerquin, P. Jansen, and A. Sabharwal, "Qasc: A dataset for question answering via sentence composition," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 2020, pp. 8082 8090.
11

---

[21] Y. Yang. W.-t Yih, and C. Meek, "Wikiqa: A challenge dataset for open-domain question answering." in Proceedings the 2015 conference empirical methods in natural language processing. 2015, pp. 2013-2018.
[22] O. Tafjord, M. Gardner. K. Lin, and P. Clark, "Quartz: An open-domain dataset of qualitative relationship questions." arXiv preprint arXiv:1909003533, 2019.
[23] Y. Zhang. J. Baldridge, and L. He, "Paws: Paraphrase adversaries from word scrambling," arXiv preprint arXiv: (ttv:I904.011 190,,0113,,,,,,, 2019.
[24] K. Sakaguchi, L. Bras, C. Bhagavatula, and Y. Choi, "Winogrande: An adversarial winograd schema challenge scale," Communications of the ACM, vol. 64, no. 9, pp. 99-106, 2021.
[25] H. Levesque, E. Davis, and L. Morgenstern, "The winograd schema challenge," in Thirteenth international conference on principles of knowledge representation and reasoning. 2012.
[26] R. Sharma, Allen, O. Bakhshandeh, and N. Mostafazadeh, "Tackling the story ending biases in the story cloze test," Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018, pp. 752-757.
[27] L. Collins, H. Hassani. A. Mokhtari, and S. Shakkottai, "Exploiting shared representations personalized federated learning," " in International conference on machine learning. PMLR, 2021, pp. 2089-2099.
[28] P. Sen C. Borcea, "Fedmtl: Conference on Artificial Intelligence, ECAl 2024. IOS Press BV, 2024, pp. 1993-2002.
federated multi-task learning," 27th European
[29] T. and Yoon, "Rethinking International Conference on Machine Learning, 2024.
minima searching in federated learning," " Forty-first
[30] A. Vani, F. Tung, I.. Oliveira, H. Sharifi-Noghabi, "Frrett sharpness: perturbed forgetting model biases within sam dynamics," arXiv preprint 27X7022006.06700, 2024.
12

---

A Convergence Analysis
A.1 Theorem 1: Convergence of Standard FedAvg
We begin by recalling the standard convergence analysis for FedAvg under the standard assumptions of the main text.
Theorem A.1 (Convergence of FedAvg)- Let Cu(0) be the local loss function for each client satisfying the L-smoothness, bounded variance, and bounded gradient norm assumptions. Under these assumptions, the global model converges stationary point 0 with the following bound on the convergence rate after T rounds:
C T
where C' a constant that depends on the initial conditions and the smoothness parameter L.
Proof of Theorem 1. The proof follows from standard results in FL, in [10] The idea is to bind the decrease in the global loss function at each iteration by using the smoothness of the local loss functions and the bounded variance of the stochastic gradients. Specifically, we use the descent lemma:
L ( 8²) + 1100t1 1'11?
L( =((0) + (VL(0'), g++1
and show that the expected decrease is proportional 1/T, leading the convergence rate of (1/TT). For more details, see [10].
A.2 Theorem 2: Convergence of FedRAM with Dynamic Weights and Proxy Model
We now provide detailed proof of the convergence result for FedRAM, incorporating dynamic weight adjustments mediated by the proxy model.
(0) be the local loss function for each client k, satisfying the L-smoothness and bounded variance assumptions. Then, with weight adjustments og moderated by proxy model based on differential loss, the global model in FedRAM converges stationary point 0*, with the following bound on the convergence rate after rounds:
Theorem A.2 (Convergence with Adjusted Aggregation Weights in FedRAM). Let (o
Č T
σ²
1 Tik
K
- -
<
+
+ O(c),
P
K
k=
where Č a constant reflecting the weight adjustments and 8² represents the reduced variance in gradients influenced by the proxy model, and O(c) captures the error due task heterogeneity across clients.
Proof of Theorem 2. The proof for the convergence with adjusted weights now includes the influence of the proxy model, which acts as mediator to adjust the weights based on client-specific excess loss and task complexity.
Step 1:
Descent Lemma with Proxy Model Involvement. By the L-smoothness assumption and incorporating the proxy model's 's adjustments, we apply the descent lemma for any iterate A
L ((0)+1)+ < L(0') (VL(''), gh+1 8²) 100t1 1²11..
The global model update rule now reflects the proxy model's 'ss influence:
g++1 6t + Σ w.
where wa are the weights adjusted by the proxy model based on the assessment of each client's needs and contributions.
13

---

Step 2: Variance Reduction via Aggregation Coefficients Enhanced by Proxy Model. The proxy model enhances the weight adjustments wp. leading an optimized reduction in the variance of the aggregated gradient updates, thus improving the stability and effectiveness of the learning process. 'he adjusted variance now denoted by x², which accounts for the proxy's role in minimizing discrepancies in client updates.
Variance Decomposition of Federated Aggregation. Let the local client gradient be gf. (L), and the global aggregated gradient is:
g6
Wk (L))
where the adjusted weights satisfy Ex= k=1 wy = 1. The variance decomposes as the combination of the inter-client variance and cross-client variance:
9 Blgalll==
Σ B
-
w.w,BE EIVC,].V, BVVC,
kgj
Proxy-Driven Weight Adjustment. The proxy model optimizes weights via w e. where d La (0 (pxxy) (Oret)). Assuming local gradient variance V [ S of the total variance satisfies:
E 1996 E[gall|
Σ o Σ wk W.WDPttjKK0
where Pkj is the gradient correlation coefficient between clients k and j.
Optimal Variance Reduction via Proxy Weights. If the proxy adjusted weights satisfy we & कe we can construct the Lagrangian as:
C Σ of >
Σ wx -
k=1
take derivatives w.r.t wx, and solve for optimal weights:
1/0 Ej=1 //11//03
K
w
Σ
20
Σ k=
Though wy 1/oß is idealized, the proxy model approximates client reliability using th (assuming of x (tr)) leading to:
of 2
82 < max
K
(IK Ma
2'
This upper bound is strictly smaller than FedAvg's Peadnng
EK o when is properly tuned.
Step 3: Enhanced Convergence Bound. By combining the refined descent lemma and the en- hanced variance reduction facilitated by the proxy model, we derive more robust bound on the convergence rate, effectively reducing errors and speeding up convergence compared to traditional methods.
Č à² T P
K
B[c(0*) -
+
+ O(e).
K k=
Tik
This approach ensures a more adaptive and responsive FL environment, effectively addressing the complexities of real- world distributed learning scenarios.
14

---

A.3 Theorem 3: Convergence of Proxy Model with Excess Loss
Theorem A.3 (Convergence of Proxy Model), Under the assumptions of L-smoothness and bounded G. the proxy model with task weights og updated by af a e fex, satisfies:
gradient (eeell
- Sprasy
②2 praxy
E(ar))
EE(N (ar))
+ O(e).
T
P
where Oprnay is a constant reflecting the weight adjustments and omnn represents the reduced variance in the gradients inftuenced by the excess loss of proxy model and reference model, and (c)) captures the error due the heterogeneity of the tasks among clients.
Proof of theorem 3. The proxy model's convergence follows the same framework as the agent model (Theorem 2), with modified weights o that adapt different tasks f The key adaptation lies in the variance term:
o proxx P
T Σ (a.)²
of S max
e Leee
Proxy Variance
T
(IT=ee--aan eni-Eae
The full derivation parallels Theorem 2's steps, replacing client weights wh with task weights of, and differential loss lair with excess loss Lexe.
B
Supplementary Experiments
B.1
Experimental Setup
The hierarchical architecture of FedRAM comprises three model categories: (1) reference models (0rer) preserving task-specific knowledge through localized training, (2) proxy models (0paay) analyzing cross-task relationships via parameter (#*), and (3) agent models (Oagen) integrating global knowledge
through larger parameters (0+). Our default training environment involves K 10 clients handling N = 7 distinct tasks over T 50 communication rounds. The optimization process employs learning rate 3 = 10 with weight decay wd = 10-4 4 to prevent overfitting. Critical hyperparameters include an exponential scaling factor 17 0.1 for contribution reweighting and smoothing parameter
8 0.01 to stabilize gradient updates. Notably, each communication round completes local training step minimize client-side computation overhead. This default experimental setting validates through experiment results in Section 5.2 and Appendix
Table 4: Default experimental settings. 0" denotes the reference and proxy models, 0- the agent model.
Hyperparameters
Definition
Values
0xet 0 pausyy aggent
Reference Model
g"
Proxy Model
0"
Agent Model
0+
K
Number of Clients
10
T
Number of Tasks
7
Learning Rate
1E-3
wd
Weight Decay
1E-4
Communication Rounds
50
P
Training Steps Exponential Scaling Factor
1
Trask: Telient
0.1
Smoothing Parameter
0.01
B.2 Data Initialization.
Figure displays the heterogeneity in both task distribution and data availability across different federated clients, illustrating significant heterogeneity in the data handled by different clients. Panel
15

---

2000
2000
Dats Stare
000
1000
Clients
Clients
Figure 5: Heterogeneity Statistics Across Tasks and Clients. (a) Gray bars represent data size variations across different tasks. (b) Blue bars indicate the distribution of task categories per client, showing the diversity in tasks handled by different clients. (c) Green bars depict the data size within each client domain, where darker green represents the maximum dataset size, and lighter green indicates the total available data.
(a) highlights the imbalance in task sizes, where some tasks contain significantly more training samples than others, leading to disproportionate model updates during federated training. Panel (b) shows that clients engage with varying numbers of tasks, with some handling diverse set of task categories while others specialize in fewer domains. This discrepancy suggests the necessity for flexible model adaptation strategies. Finally, panel (c) visualizes the data size per client, revealing that certain clients have access to large datasets but may not fully utilize them. The overall heterogeneity observed in task distribution and data volume underscores the limitations of traditional FL. approaches, such as FedAvg, which assume homogeneous data distributions.
B.3 Hyperparameter Analysis
Effeet of Veelient on Client Weights Evoluation and Performance. Figure 6 depicts how client weights evolve over iterations in FedRAM, reflecting adjustments based on the parameter Veelicnt- The variation in 7elcnn values shows distinct convergence behaviours, which are critical for understanding how FedRAM adapts different client needs. For values of Melcnnt near O, weight changes gradual and uniform, indicating a balanced approach to weight adjustments among clients. As 7eeient moves away from O, either negatively positively, the weight trajectories show more pronounced disparities between clients, suggesting a more aggressive or conservative reweighting strategy. This particularly noticeable with 1ecieett values of 11 and 1, where weights converge more distinctly.
Figure 8 () further demonstrates how 7alien affects global performance, local adaptation, and out- of-domain generalization. Small values of Tjelient result in a more stable and gradual improvement in accuracy, while extreme values lead to larger performance variations, reflecting different aggregation strategies under heterogeneous conditions. As Jelient moves further from 0 (cither negative positive), weight updates become more aggressive, amplifying disparities in client importance. This explains the increasing trend in global performance seen in Figure
We analyze the impact hyperparameter 7/tiien based on Figure The results indicate that positive values of Netie lead to higher weights being assigned to clients with larger losses, encouraging the global model to prioritize the underperforming models for improvement. In contrast, negative values of Telieett allocate greater weight to clients with smaller global losses, reinforcing the influence of werl-peeforming models during global evaluation. Empirical observations in Figure ())b) suggest that positive Metien values yield more stable and consistent performance across all four evaluation metrics, making them a preferable choice for balancing model adaptation and generalization.
Effeet of Trask on Task Weights Evoluation and Performance. Figure 7 illustrates the dynamic evolution of task weights over different training iterations in FedRAM, showing how the method adapts task weights based on their contribution to global model performance. As training progresses, the task weights shift, with lighter blue shades representing earlier training rounds and darker shades indicating later stages. The red line reflects the initial task weight distribution, which is influenced by the dataset size ratio (data size to total data size). The gradual adjustments highlight that FedRAM has the ability to iteratively prioritize tasks based on their relevance and performance, allowing the model to better handle task heterogeneity. This dynamic weight adjustment process, driven by the proxy and agent models, ensures that tasks with smaller datasets or less initial contribution are given higher
16

---

weights in later stages, improving the overall performance. The task-wise reweighting mechanism is a key strength of FedRAM.
2.50
6111
6.5
Waight
Waggtts
Wesglts
6.155
20
631
230
0.108
6:14
Weight
Weightt
3.12
Weights
0:14
6:12
0284
herdion
1002
Figure 6: Evolution of Client Weights Over Training Iterations in FedRAM. The subfigures corre- spond to 'Jeieen - 005, 0..1, 0.5, 1, respectively. The parameter pelient results varying convergence behaviours while preserving the relative weight relationships among clients. Darker points indicate higher weight values, and blue points represent lower values.
Task weights (training round 10)
Task weights (training round 20)
Teak weights (trai (training round = 30)
Task weights (training round 40)
Task weights (training round 50)
Dataset size ratio
StoryCloze
StoryCloze
StoryCloze
WinggradiS
QASC
WinogradWSC
QASC
WinogradWSC
QASC
0
de
WinGGrende
QuaRTZ
WinoGrande
QuaRTz WinoGrande
QuaRTZ
WkkQA
PAWS
WKKDA
PAWS
WikiDA
Task Weights (y=0.01)
Task Weights (y=0.1)
Task Weights (1=1)
Figure 7: Evolution of Task Weights Over Training Iterations in FedRAM. The radar charts illustrate the dynamic adjustment of task weights under different Thssk settings. The red line represents the initial distribution of task weights (data size / total data size). The lines in blue tones present the evolution of task weights training progresses from early (lighter shades) to later (darker shades) stages.
B.4 Model Scalability
Effect of Agent-to-Proxy (A/P) Ratio on Model Performance. Figure presents the impact of the Agent-to-Proxy (A/P) ratio, a concept introduced in this study to evaluate the feasibility of using smaller proxy model optimize larger agent model. The A/P ratio is defined as:
Number of Agent Parameters
A/P Ratio Number of Proxy Parameters
(15)
A lower A/P ratio suggests a relatively larger proxy model, which may improve stability but increase computational overhead. Conversely, higher A/P ratios imply larger agent model trained with fewer proxy parameters, allowing more efficient adaptation while maintaining competitive performance.
17

---

Accuracy
0.8
9 Globel Purforannc
Guzel Performance
Local Avergg Perfermance
X Local Average Performance
Plrfarmmno
DD Performance
Performance
OOD Purforance
Min-Mox
Local Min-Max Range
0.8
-0.5
-0.1
0.1
0.5
AP Ratio
Exponential Scaling Factor
(b)
Figure 8: Impact of Hyperparameters on FedRAM Performance. (a) Effect of the agent-to-proxy (A/P) ratio, which controls the relative size of the agent and proxy models, on performance. (b) Effect of the exponential scaling factor, which influences weight adjustments during aggregation, on performance stability.
Table in the main text and Table 6 provide empirical results for different A/P ratios. Table T
corresponds to an A/P ratio of 15, while Table corresponds to an A/P ratio of 25. Comparing these two settings, observe that higher A/P ratio leads to improvement in global performance 162.16 vs. 79.74) and local adaptation (83.22 vs. 79.62), suggesting that FedRAM benefits from a well-tuned proxy-to-agent parameter balance. However, a larger A/P ratio can also lead slightly reduced stability in out-of-domain generalization, as seen in the marginal drop in OOD performance (82.89 82.67).
These results demonstrate that using a smaller proxy model to guide the reweighting of a larger agent model is entirely feasible. While increasing the A/P ratio improves performance, the effectiveness of FedRAM remains strong even when the proxy model is significantly smaller, highlighting the efficiency of our proposed approach in leveraging lightweight models for scalable optimization.
Computational Cost Analysis. As shown in Table the reference model, proxy model, and agent model exhibit significantly differentiated computational resources consumption. The agent model Ougent persistently dominates system resources, consuming 85.08% - 88. 88.02% of total computation time across client scales (N 10 to 50). Notably, the stability of component-wise proportions under- scores FedRAM's invariann resource allocation design. These findings quantitatively validate FedRAM's 'ss  architectural robustness for federated learning systems requiring adaptive resouree-aware coordination.
Table 5: Computational cost of FedRAM components.
N = 10
N
20
N
50
Algorithm
Model
Time (s)
Proportion (%)
Time (s)
Proportion (%)
Time (s)
Proportion (%)
g %
120
1.00
720
1.45
1093
1.10
prexy
1680
13.92
5231
10.49
10750
10.88
FedRAM
0 apent
10260
85.08
43903
88.06
87000
88.02
E A
12060
100.00
49854
100.00
98843
100.00
Performance of Larger Agent Models. We validate that the proposed FedRAM method signifi- cantly outperforms existing FL-MTL approaches in various evaluation metrics. Table presents a comparative analysis of FedRAM against several SOTA FL methods across diverse NLP tasks. We highlight the following key observations:
(i) Superior Global Average and Task-wise Performance: FedRAM achieves global F1-score of 12,16, outperforming all baseline methods. particular, FedRAM attains the highest F1-score in two critical tasks, Story Cloze (90.77) and wse (78.91), demonstrating its efficacy in handling various NI.P challenges. Compared to FedMTL, which excels in specific tasks but struggles with broader generalization, FedRAM achieves more balanced and robust performance across tasks.
18

---

Table 6: FI-Score Comparison of FL Methods Performed on Larger Models. This table shows the F1-scores for different FL. methods across multiple NLP tasks, including global Fi-scores and local Fl-scores within the bottom decile, alongside ID and OOD evaluations. Scores in Bold indicate the best performance, while scores underlined denote the second best.
Tasks
Global
Local F1-Score/ /
ID /OOD
Methods
PAWS
WSC
QASC
QuaRT>
Story Cloze
WikiQA
F1-Score
Bottom Decile
Evaluation
Grande
FedAvg ||
88.53
70.90
79.88
32.50
80.14
80.87
79.79
80.75
81.33 / 79.588
81.18/813
FedProx [00
71.27
78.70
38.26
80.37
78.65
65.73
74.79
78.55 74.89
78.06/ 78.97
Ditto [0
86.06
72.38
77.28
31.60
80.75
80.24
79.79
79.68
80,70 00.70/ 79.08
80.02 / 81
FedRep 27
88.79
72.11
80.71
32.78
79.12
82.79
63.87
80.72
81.25 79.06
81.59 77.991
DBE g
88.52
70.47
79.01
31.88
80.09
81.58
79.79
80,66
81.12 77.76
81.14/ 82.06
FedMTL.
28
90.26
73.67
79.21
33.06
69.72
88.85
79.79
78.00
82.33 77.55
81.92/ 82.67
FedBone
75.26
73.93
30.83
76.98
73.90
78.23 75.60
76.24 82.51
FodRAM (Ours)
90.00
78.91
78.36
33,03
80,70
90.77
79.79
82.16
83.22 80.63
83,35 82.89
(ii) Enhanced Performance in Low-performing Clients: In local Fl-score evaluation within the bottom decile, FedRAM reaches 83.22, surpassing the second-best method (FedMTL, 82.33) by 0.89. This highlights FedRAM's s ability to mitigate performance degradation in lower-performing clients. The reweighting mechanism introduced in FedRAM effectively balances task importance, ensuring robust personalization across heterogeneous data distributions.
(iii) Effective Handling of Statistical Heterogeneity: Compared to FedAvg and FedProx, which do not incorporate explicit mechanisms to address inter-task conflicts, FedRAM improves performance across multiple tasks by dynamically adjusting task sample rates and client aggregation weights. For instance, on the WSC task, FedRAM achieves 78.91, surpassing the second best score (FedProx, 74.90) by a significant margin, demonstrating its capability to address heterogeneity.
(iv) In-Domain Personalization and Dutmoi-Domain Generalization: FedRAM demonstrates superior performance in both in-domain personalization and out-of-domain generalization, outperforming baseline methods across various NLP tasks. For ID evaluations, FedRAM achieves the highest score (83.35), showing its ability to personalize well within the federated learning setting. Meanwhile, for OOD generalization, it also achieves the highest score (82.89) 89), demonstrating its ability to adapt to unseen data distributions better than other methods. Specifically, while FedMTLL exhibits competitive OOD performance (81.92), its overall global F1-score (78.00) is lower, indicating a trade- off between generalization and global performance. Similarly, FedAvg, which assumes homogeneous data distributions, underperforms in ID and OOD settings, reinforcing the importance of adaptive reweighting strategies.
(v) Performance Improvement with Larger Models:
Comparing Tables 1 and 6 we observe that using a larger model improves overall performance across all methods. Notably, FedRAM's global Fl-score increases from 75.94 to 16, and its local Fl-score (bottom decile) improves from 79.62 83.22. Additionally, its ID/ OOD evaluation scores rise from 72.89 / 76.32 83.35 / enhancements suggest that FedRAM benefits significantly from increased model capacity, further reinforeing its effectiveness in handling complex multi-task federated learning scenarios.
82.89. These
Overall, FedRAM establishes itself as a robust and efficient FL-MTL framework by effectively mitigating inter-task conflicts and optimizing both personalization and generalization. Compared to existing personalization strategies, including DBE and FedBone, FedRAM achieves at least a 3% improvement across multiple metrics, validating the effectiveness of its novel reweighting mechanism. This makes FedRAM well-rounded approach for federated learning scenarios with heterogeneous and multi-task data.
B.5 Client Scalability
As the number of participating clients increases, global performance (blue) shows a slight downward trend, likely due to increased data heterogeneity. However, local average performance (orange) and ID performance (yellow) remain relatively stable, indicating that FedRAM effectively adapts to the additional heterogeneity introduced by more clients. Particularly, OOD performance (green) remains consistently high, demonstrating strong generalization capabilities. The local min-man range (green
19

---

0.9
Asourazy
0.8
- Styaal Perfrmance
-Loenl Petfomence
0.7
Clubal Peffrmmaee
Pwrfomance
LoceelA Pariommenc
Perfemance
Porforaance
0.6
Local Min Surge
Gon Partomarce
Local Min-Mae
Local Min Max Range
0.5
Task Divursity Raão
Clients
(c)
(b)
(c)
Figure 9: Impact of Heterogeneity FedRAM Performance. (a) Performance trends across varying task diversity ratios. (b) Accuracy results across different numbers of tasks, (c) Model performance with increasing number of FL clients.
diamonds) widens slightly, suggesting some variability in individual client performance, but overall stability is maintained.
Table 7: Comparison of FL Methods with client K = 20 and 50. This table shows the Fl-scores for different FL methods across multiple NLP tasks, including global Fi-scores and local Fl Fl-sscres, alongside ID and OOD evaluations. Scores in Bold indicate the best performance.
Tasks
Global Local
ID /OOD Evaluation
Clients
Methods
Wino Grande
QASC
QuaRTz
Story Cloze
WikiQA
F1-Score
FedAvg 1
75.09
72.69
61.35
34.16
79.10)
72.20
71.13
67.98 68.67
70.07 67.31
Ditto
75.09
73.63
61.05
30.88
78.79
80.84
71.13
70.25 72.98
75.85 72.65
FedMTL 28
74.48
72.16
72.05
32.10
73.89
83.14
71.13
69.10/ 19.10/70.35 70.35
75.78 73.33
K
FedBone 15
69.54
66.90
70.11
33.05
58.88
76.22
71.13
66.91 66.85
69.43 68.55
FedRAM (Ours)
75.69
73.63
74.11
36.93
79.54
78.22
71.13
70.86 73.33
76.31 74.54
FedAvg 11
72.75
69.83
65.44
33.46
69.52
71.50
69.79
14.79/ 66.33
67. 65.53
Ditto 11
72.75
69.83
65.44
33.46
69.52
71.50
69.79
65.91 67.83
68.50 67.24
FedMTL 28
68.11
73.67
63.08
32.11
67.56
70.90
69.79
63.28 65.37
63 86.86/ 62.
K
FedBone
69.78
70.41
68.18
34.33
63.51
73.78
69.79
64.53 66.29
65.77
FedRAM (Ours)
73.88
74.27
68.76
35.78
73.90
75.53
67.52/ 68.85
70.89 68.90
B.6 Task Scalability
Building on the observations from Figure Figure Tigureepresents the impact of task and data heterogeneity on the performance of FedRAMM.
(a) Task Diversity Ratio. As the task diversity increases, global performance (blue) and local average performance (orange) both improve significantly, indicating that FedRAM benefits from more diverse task distributions. However, the performance (yellow) and OOD performance (green) maintain relatively stable trend, suggesting that task diversity does not compromise the model's generalisation ability. The local min-max range (green diamonds) shows relatively low variance, indicating consistent performance across different clients.
(b) Number of Tasks. As the number of tasks increases, global performance (blue) and local average performance (orange) gradually improve, but the improvement becomes marginal beyond five tasks. ID performance (yellow) follows similar increasing trend, while OOD performance (green) remains consistently high. The local min-max range shows steady spread, implying that FedRAM maintains robust client-wise performance even with more tasks.
(c) Extreme Heterogeneity. In Table 8 we present the results Setting and assign one client with distinct task. The improvements in Fi-scores derive from the cut down in client number K = T 7 in this setting. The results validate FedRAM's adaptive ability to extreme heterogeneous conditions.
(d) Computer Vision Tasks. We present a computer vision (CV) task compatibility study with FedRAM in Table 9 We base our experiments seven distinct CV datasets: MNIST, EuroSat,
20

---

Table 8: FI-Score Comparison of FL Methods Performed on Larger Models. This table shows the F1-scores for different FL. methods across multiple NLP tasks, including global Fi-scores and local Fl-scores within the bottom decile, alongside ID and OOD evaluations. Scores in Bold indicate the best performance, while scores underlined denote the second best.
Tasks
Global Local
/OD
Methods
Wino Grande
Story Cloze
PAWS
WSC
QASC
QuaRTz
WikiQA
F1-Score
Evaluation
FedAvg
88,45
90.32
84.33
76.11
79.53
84.30
90.29
82.67 87.96
90.29 84.78
Ditto [
89.51
88.86
90.05
76.13
83.42
83.44
89.70
83.85 88.67
90.45 87.03
FedMTL
28
90.48
82.37
80.07
82.68
89.54
83.14
81.88
83.88 88.48
90.53 86.51
FedBone
15
89.25
83.47
84.79
75.90
88.74
88.64
85.50
84.06 / 89.53
91.02 86.95
FedRAM (Ours)
90.36
90.32
87.94
78.48
90.40
90.50
90.42
84.29 90.29
91.53 87.26
Fashion-MNIST, SVHN, DTD, Stanford-Cars, and CIFAR-10. We reference models and proxy models as Swin Transformer, while the agent models are the CLIP-ViT. Other settings followed the condition Appendix B.1
Table 9: Comparison of FL Methods Performed on Vision Tasks. Scores in Bold indicate the best performance.
Tasks
Global Local
OOD Evaluation
Methods
Fashion -MNIST
Stanford -Cars
MNIST
EuroSat
CIFAR-10
F1-Score
FedAvg 1
97.22
38.13
81.88
66.14
20.08
80.62
77.47
78.09 76.36
68.11 / 62.57
FedProx
47.35
8.33
27.30
58.61
9.89
57.92
54.20
50.73 44.67
36.67 28.41
DBE 28
97.22
24.10
76.67
65.87
20.21
80.79
76.48
76.93 75.66
64.71 58.33
OBF
30
96.07
4.03
87.79
78.79
33.14
74.77
79.35
78.84/ 65.18
53.34 / 47.96
FedGF
29
72.73
5.19
75.03
30.08
27.40
71.72
75.79
75.08 62.88
52.54 / 44.04
AM (Ours)
97.22
50.55
87.70
78.49
43.28
80.28
80.91
82.05 / 79.23
70.94 / 65.26
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction in this paper accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification: In this paper, all proofs of theorems are provided and all assumptions are clearly stated or referenced in the statement of any theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yss]
Justification: All the results in this paper can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: We provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
/nniss.ccppbllc/guuii gwt.///ddee//eee// wwt.sSuumisssoopoooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details necessary in Section 5.1 and Appendix B.I understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report eтror bars suitably and correctly detined or other appropriate information about the statistical significance of the experiments in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: For each experiment, the paper provides sufficient information on the computer resources needed to reproduce the experiments in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps:/nneurips http:/p../c/pbblcctthcccccuudud
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the paper discusses potential positive societal impacts, such as improving model training efficiency and enhancing convergence performance in Section 5.2
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The research described in our paper does not involve data models that have a high risk for misuse, and thus does not require specific safeguards
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators original owners of assets used in the paper properly credited and the license and terms of use are explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not describe the usage of LLMs as an important, original, or non-standard component of the core methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
28


Reviews:
[{"review_id": "b5hvHoRFfi", "reviewer": "Reviewer_m3Lj", "summary": "This paper introduces a novel federated multi-task learning framework, FedRAM, which uses a proxy model to balance client contributions. The proposed framework shows at least 3% improvement in accuracy across metrics on diverse NLP benchmarks. It also shows up to 15X reduction in computational cost.", "questions": "* Although the paper is generally easy to follow, some of the notations can be simplified and cleaner\n   * For example, around line 100, what's the point of introducing $\\theta_1$, $\\theta_2$? They bring unnecessary confusions. Around line 106, $\\theta^+$ and $\\theta^*$ are defined but almost never used until very end of the paper. If the authors have to show the size of the model, why not directly use something like $\\theta_{\\text{ref}}^{+}$.\n* Regarding the rigorousness\n    * For example, around line 124, do we assume the loss is just MSE? In eq. (6), $\\ell_{\\text{proxy}}^{\\tau}$ and $\\ell_{\\text{ref}}^{\\tau}$  are not defined. Also I guess the notations here are also ambiguous since the losses are missing indices $k$ representing clients. In step 3 of Algorithm 1: do we just perform one step of gradient descent update? In line 14 of Algorithm 1, should the second term of the product be  $e^{\\eta_\\text{client}\\ell_\\text{dif}}$. Mixed use of $\\mathcal{L}$ and $\\ell$.\n\n* Do we really need the proxy model? Can we just train the agent model by comparing with the reference and dynamically adjust $\\alpha$ and $w$. I see the regression on performance in the ablation study section. However I wonder the detailed setup of the ablation experiments. Maybe I missed them?\n* The authors discuss the sensitivities on the hyper-parameter $\\eta_{\\text{client}}$ and $\\eta_{\\text{task}}$. What is the hyper-parameter generalization? What kind of approach the authors use to find the optimal ones. For example, if we apply the framework to another set of tasks, how to effectively find the optimal hyper-parameter? Could the authors discuss the effort finding them?\n* The balances of client contributions heavily depend on the quality of reference models. The authors use T5 family models for the small reference and large agent models. How sensitive the framework is with respect to the sizes and choices of reference model? Do we need them to be in the same family to have desired results. It would be more convincing to see performances of various pairs of reference and agent models.\n\nIn summary, I think in general the paper is good. Since the proxy model idea has been around, the novelty of the paper is relatively limited. I think the authors can extend their experiments a bit, in particular on the choices of reference mode. I am happy and willing to raise my scores if the authors can address mainly the last two question.\n\n---\n\nAfter reading the rebuttal from the authors, my questions and concerns have been addressed. As the authors promise to incorporate the rebuttal into their camera-ready version, I raise my score from 3 to 5. Thanks again the authors for their detailed responses.", "limitations": "See above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Quality:\n- The paper is generally well-written and easy to follow. However, in some paragraphs, the presentation, notations and the rigorousness could be better.\n\n### Clarity: \n- The paper is mostly clear. \n\n### Significance: \n- The framework is straightforward and can be easily applied to any federated multi-task learning tasks. \n- Experiments show the non-trivial performance improvement while using the least computational power compared with methods in the literature. \n\n### Originality:\n- The idea of using proxy models to evaluate the difficulties of different tasks is not new in multi-task learning but it seems that it is novel to apply it on the federated learning.", "was_revised": true, "final_justification": "After reading the rebuttal from the authors, my questions and concerns have been addressed. As the authors promise to incorporate the rebuttal into their camera-ready version, I raise my score from 3 to 5. Thanks again the authors for their detailed responses.", "rebuttal": "We sincerely thank you for the insightful comments and valuable suggestions, which have greatly contributed to enhancing our manuscript. Based on the given weaknesses and questions, the key issues we address in our response include:\n\n---\n\n### **Q1: Some of the notations can be simplified and cleaner** \n1. Around line 100, what's the point of introducing $\\theta_1$ and $\\theta_2$?\n2. $\\theta^+$ and $\\theta^*$ are defined but almost never used until very end of the paper\n\n**A1:** Thanks for pointing out. We will carefully adjust the use of notations.\n\n---\n\n### **Q2: Regarding the rigorousness:**\n1. Around line 124, do we assume the loss is just MSE? \n2. In eq. (6), $\\ell_{\\text{ref}}^\\tau$ and $\\ell_{\\text{proxy}}^\\tau$ are not defined. Also  the notations here are also ambiguous since the losses are missing indices $k$ representing clients.\n3. In step 3 of Algorithm 1: do we just perform one step of gradient descent update?\n4. In line 14 of Algorithm 1, should the second term of the product be $e^{\\eta_\\text{client}\\ell_\\text{dif}}$. Mixed use of $\\mathcal{L}$ and $\\ell$.\n\n**A2:**\n1. The loss function should be more flexible and will be replaced with $\\min_{\\theta^L_\\text{ref}} \\mathcal{L}(\\theta^L_\\text{ref}) = \\frac{1}{|D_L|} \\sum_{(x,y) \\in D_L} \\ell(f_{\\theta^L_\\text{ref}}(x), y)$\n2. We use $\\ell^\\tau = \\mathcal{L}(\\mathcal{D}^\\tau_{L};\\theta^L)$ to present task-wise loss. Thanks for pointing out the typo.\n3. No, we don't. Step 3 is a complete local training period, which consists of multiple gradient descent updates. \n4. Thanks for pointing out the typo. We will kindly adjust the use of $\\mathcal{L}$ and $\\ell$.\n\n\n---\n\n### **Q3: Do we really need the proxy model?** \n1. Can we just train the agent model by comparing with the reference and dynamically adjust $\\boldsymbol{\\alpha}$ and $\\boldsymbol{w}$?\n\n2. The detailed setup of the ablation experiments.\n\n\n**A3:** Yes, we do need the proxy model!\n\n1.**No, we cannot.** In this response, we will discuss your case first and then summarize the proxy model's functions.\n\n**In your pre-assumed case**, removing proxy model will cause two major problems:\n* **Non-IID Struggle**: Jointly optimizing [$\\boldsymbol{\\alpha}$, $\\boldsymbol{w}$ and $\\theta_{agent}$] makes it extremely **difficult to converge** to a stable performance, since it's continuing minimax process for independant training goals on $\\boldsymbol{\\alpha}$, $\\boldsymbol{w}$ and $\\theta_{agent}$, especially in federated non-IID settings. \n* **High Computation (in your pre-assumed case)**: Besides, training $\\theta_{agent}$ by comparing with $\\theta_{ref}$ **cannot disentangle the contributions** from multiple training goals. For instance, when client 1 shows a larger performance gap compared to all other clients, it remains unclear whether this stems directly from $\\boldsymbol{w}$ or $\\boldsymbol{\\alpha}$. The cost of re-adjusting $\\boldsymbol{w}$ and $\\boldsymbol{\\alpha}$ to make another FL training process is overwhelming.\n\n**Our proxy model**: \n\n* **Provides sequential solutions**: solve for model-agnostic hyperparameters [$\\boldsymbol{\\alpha}$, $\\boldsymbol{w}$] first, and then train $\\theta^*_{agent}$ in FL. \n* **Adaptation to Non-IID**: the locally trained $\\theta^L_\\text{proxy}$ learns prior knowledge of task heterogeneity; FL aggregation facilitates $\\theta^G_\\text{proxy}$ to share global knowledge.\n* **Low Computation**: according to Table 5 in our paper, the $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$, and $\\theta_\\text{agent}$ exhibit the computational proportion of around 1%, 14%, 85% (the number of clients is set to 10), so that saving compute in training $\\theta_{proxy}$ only results in a small compute benefit.\n* **Ablation Study** Our ablation study validates the effectiveness of $\\theta_{proxy}$. We present the detailed setup and results of ablation studies below.\n\n2.**The detailed setup of ablation studies** follow the same setup as in section 5.2.1, maintaining same datasets and client configurations. \n* **Experiment 5** **FedRAM**:\n\n$\\theta_\\text{ref}$ →  $\\theta_\\text{proxy}$  → $\\theta_\\text{proxy}$ and $\\theta_\\text{ref}$\n\n$~~~~~~~~~~~$  |   $~~~~~~~~~~~~~~~~~~~~~~~~$    |\n\n$~~~~~~~~~~$  $\\boldsymbol{w}$    $~~~~~~~~~~~~~~~~~~~~~~~$    $\\boldsymbol{\\alpha}$\n\n$~~~~~~~~~~$  └-----------------------└------→ $\\theta_\\text{agent}$ → Evaluation\n\n* **Experiment 1**: w/o $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$, we train and evaluate on the FL agent model $\\theta_\\text{agent}$ (FedAvg). Also, $\\boldsymbol{\\alpha}$ and $\\boldsymbol{w}$ are not involved, which equals to FedAvg and serves as the comparison baseline against Exp 6 and 7.\n\n$\\quad\\quad\\quad~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$ $\\theta_\\text{agent}$ → Evaluation\n\n* **Experiment 2**: w/o $\\theta_\\text{ref}$, we train $\\theta_\\text{proxy}$ in FL without the use of excess loss $\\ell_\\text{exc}$ while keeping differential loss $\\ell_\\text{dif}$. In this setting, we evaluate $\\theta_\\text{agent}$ solely based on client weights $\\eta_\\text{client}$:\n\n$~~~~~~~~~~~$  $\\theta_\\text{proxy}$  \n\n$~~~~~~~~~~~~$  | \n\n$~~~~~~~~~~~$  $\\boldsymbol{w}$\n\n$~~~~~~~~~~~$  └-----------------------------→ $\\theta_\\text{agent}$ → Evaluation\n\n\n* **Experiment 3**: w/o $\\theta_\\text{proxy}$. In this setting, we can not derive the task weights $\\boldsymbol{w}$ and client weights $\\boldsymbol{\\alpha}$, so that we directly evaluate on $\\theta_\\text{ref}$.\n\n$\\theta_\\text{ref}$ -----┐\n\n$~~~~~~~~~~~~$  |\n\n$~~~~~~~~~~~~$ └-----------------------------------------→ Evaluation\n\n* **Experiment 4**: w/o $\\theta_\\text{agent}$. We evaluate on $\\theta_\\text{proxy}$ in this case.\n\n$\\theta_\\text{ref}$ →  $\\theta_\\text{proxy}$  → $\\theta_\\text{proxy}$ and $\\theta_\\text{ref}$\n\n$~~~~~~~~~~~~$  | \n\n$~~~~~~~~~~~~$  └-----------------------------------------→ Evaluation\n\nWhen it comes to component-wise ablation study, we compare Exp 1, 3, and 4. We evaluate $\\theta_\\text{agent}$, $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$, respectively:\n1. $\\theta_\\text{agent}$ outperforms the other components both in global and local performance, for its larger model size and trainable parameters.\n2. $\\theta_\\text{ref}$ has a competitive performance on global evaluation but fail to improve on the local performance compared to $\\theta_\\text{agent}$.\n3. $\\theta_\\text{proxy}$ shows a giant gap among the components, because it does not necessarily learn the local and global knowledge in this system but to tune hyperparamters $\\boldsymbol{\\alpha}$ and $\\boldsymbol{w}$. \n\n| Exp  | Method  | Evaluation on | Global | Local  |\n|---------|---------|--------|-------|------|\n| 1 | w/o $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$ | $\\theta_\\text{agent}$ | 72.71   | 76.34  |\n| 2 | w/o $\\theta_\\text{ref}$ | $\\theta_\\text{agent}$ | 71.17   | 73.58 |\n| 3 | w/o $\\theta_\\text{proxy}$ | $\\theta_\\text{ref}$  | 72.54   | 70.52  |\n| 4 | w/o $\\theta_\\text{agent}$ | $\\theta_\\text{proxy}$     | 2.69   | 2.51  |\n| 5 | FedRAM | $\\theta_\\text{agent}$     | 75.94   | 79.62  |\n\n\n---\n\n### **Q4: Hyper-paramters $\\eta_\\text{client}$ and $\\eta_\\text{task}$ Generalization** \n1. Approach to find the optimal ones and the effort of finding them?\n\n2.What is the hyper-parameter generalization?\n\n3. How to effectively find the optimal Hyperparamters in another set of tasks?\n\n**A4:** \n1. We employed a **logarithmic grid search** strategy, ranging from 1E-2 to 1E2.  The optimal logarithmic value in our framework is 1E-1, with the effective range being $[-1, 1]$, as in Figure 8. In this range, the impact on global performance remains consistently stable.\n2. Take $\\eta_\\text{task}$ as an example. Theoretically, assuming $\\theta_\\text{ref}$ in Step 1 is well trained, and the loss $\\ell^\\tau_\\text{ref}$ equals to 0. The term $\\ell^\\tau_\\text{exc}$ equals to $\\frac{\\ell_{\\text{proxy}}^\\tau}{{|\\mathcal{D}^\\tau_L|}}$. Assuming $\\ell^\\tau_\\text{proxy}$ goes down per iteration. We make sure the first-iteration of $\\alpha^1_\\tau\\leftarrow\\alpha^0_\\tau e^{ \\eta_\\text{task} \\ell^\\tau_\\text{exc} } \\leq 1$ should not excess the natural upper bound 1 (note that $\\alpha^0_\\tau=\\frac{1}{K}$). Therefore, it is less risky to set  $\\eta_\\text{task}\\leq\\frac{\\text{ln}K}{\\ell^\\tau_\\text{exc} }=\\frac{|\\mathcal{D}^\\tau_L| \\times\\text{ln}K}{\\ell^\\tau_\\text{proxy}}$, where $|\\mathcal{D}^\\tau_L|$ is denoted as the training dataset size., $K$($K>1$) as the number of clients, $\\ell^\\tau_\\text{proxy}$ as the maximum value of first step proxy loss performed on task $\\tau$.\n3. We do not recommend extensive tuning for these parameters. For practical implementations, $\\eta = 1, 0.1$ or $0.01$ can be simple and effective default values. \n\n---\n\n### **Q5: Model Types** \n1.How sensitive is FedRAM with respect to the sizes and choices of $\\theta_{ref}$?\n\n2. Same family models? Performances of various pairs of $\\theta_{ref}$ and $\\theta_{agent}$.\n\n**A5:**\n1. We discussed Agent-to-Proxy (A/P) Ratio in Appendix B.4. For a fixed size of $\\theta_{agent}$, the size of $\\theta_{ref}$ (along with $\\theta_{proxy}$) can be 1/25 $\\times$ smaller model scales.\n2. No, we don't. In Appendix B.6 Task Scalability, we applied **Swin Transformer** as reference and proxy models, while **CLIP-ViT** as the agent models. Based on the results in Table 9, we extend the experiments to other model pairs as follows:\n\n| Exp | Modality| $\\theta_{ref}$ and $\\theta_{proxy}$  | $\\theta_{agent}$  | Global | Local | ID   | OOD  |\n|-|-|-|-|-|-|-|-|\n|1| **cv** |Swin Transformer | CLIP-ViT  | 82.05 | 79.23  | 70.94 | 65.26 |\n|2| **cv** |CLIP-ViT | Swin Transformer  | 79.39 | 80.59  | 73.52 | 62.67 |\n|3| nlp |T5-base | T5-large| 84.29   | 90.29  | 91.53 | 87.26 |\n|4| nlp |T5-large | LLama  | 86.77   | 92.03  | 92.24 | 88.19 |\n|5| nlp |bert | T5-large   | 84.80   | 90.41  | 91.64 | 85.36 |\n|6| nlp |bert | LLama   | 84.43   | 89.64  | 91.78 | 87.40 |\n\nTo conclude, experiments both in CV (Exp 1, 2) and NLP (Exp 3~6) can validate FedRAM's generalization to model types."}, {"review_id": "D2v93hoSRd", "reviewer": "Reviewer_dqPQ", "summary": "This paper proposes FedRAM, a novel federated learning framework for multi-task learning (FL-MTL). FedRAM introduces a three-step framework that updates two scalar hyperparameters: task importance weights and client aggregation coefficients. It employs a reference-proxy-agent strategy, where the proxy model serves as an intermediate between the local reference model and the global agent model. The framework aims to improve performance on both in-domain and out-of-domain tasks while reducing computational costs.", "questions": "1. What task was used to optimize θref in step 1? Is it one of the multi-tasks?\n\n2. Is there a clear guideline on selecting ηtask and ηclient beyond grid search?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tFedRAM introduces a novel three-step framework that effectively addresses the challenges of statistical heterogeneity and task interference in FL-MTL.\n\n2.\tThe use of excess loss to tune task weights and differential loss to adjust client weights is principled and empirically effective.\n\n3.\tConvergence proofs are provided and align with standard assumptions in federated learning literature.\n\nWeaknesses:\n\n1.\tThe three-step framework and the need to train multiple models (reference, proxy, and agent) may add complexity to the implementation process. \n\n2.\tPerformance depends on exponential scaling factors and smoothing parameters, but these are only lightly addressed in the main text.", "was_revised": true, "final_justification": "I read the rebuttal, and I will raise my score accordingly.", "rebuttal": "Thank you for your insightful feedback and helpful suggestions. dqPQ notes that “FedRAM introduces a novel three-step framework” with \"principled and empirically effective loss use\", and “Convergence proofs are provided and align with standard assumptions”. We address specific questions below:\n\n---\n\n### **W1: The three-step framework and the need to train multiple models (reference, proxy, and agent) may add complexity to the implementation process.** \n\n**A1:** We acknowledge that our three-stage framework introduces additional complexity compared to traditional federated learning approaches. However, this design choice provides significant flexibility by maintaining compatibility with existing FL methods, allowing practitioners to integrate FedRAM components selectively based on their specific requirements. The apparent complexity stems from our deliberate decoupling of three critical optimization objectives in FL-MTL scenarios: task weights ($\\boldsymbol{\\alpha}$), client weight distribution ($\\boldsymbol{w}$), and model parameter optimization ($\\theta_\\text{agent}$). This decomposition strategy substantially reduces computational overhead, as our reference and proxy models are approximately 1/25 the size of the agent model, contributing only ~15% of the total training time. Furthermore, in practical deployment scenarios, organizations often possess pre-trained smaller models already adapted to local datasets, which can serve directly as reference models. In such cases, the primary training effort focuses on the proxy and agent models.\n\n---\n\n### **W2: Performance depends on exponential scaling factors and smoothing parameters, but these are only lightly addressed in the main text.** \n\n### **Q2: Is there a clear guideline on selecting $\\eta_{task}$ and $\\eta_{client}$ beyond grid search?**\n**A2:** We appreciate the reviewer's concern regarding hyperparameter sensitivity. For **smoothing parameters**, we consistently use 0.01 across all experiments, chosen to balance convergence stability without significantly increasing communication rounds or overly influencing weight dynamics. This value ensures that both task weights $\\boldsymbol{\\alpha}$ and client weights $\\boldsymbol{w}$) converge within reasonable bound while maintaining stability.\n\nRegarding **exponential scaling factors** $\\eta_\\text{task}$ and $\\eta_\\text{client}$, while the main text prioritizes the core algorithmic contributions of our three-step architecture and the sequential solutions for $\\boldsymbol{\\alpha}$, $\\boldsymbol{w}$ and $\\theta_{agent}$, we provide $\\eta_\\text{task}$ and $\\eta_\\text{client}$ analysis in Appendix B.3.  We'd like to add how we find the $\\eta_\\text{task}$ and $\\eta_\\text{client}$ here:\n\n1. We employed a **logarithmic grid search strategy**, ranging from 1E-2 to 1E2.  The optimal logarithmic value is 1E-1, with the effective parameter range being $[-1, 1]$, as demonstrated in **Figure 8**. Within this range, the impact on global performance remains consistently stable.\n\n2. Take $\\eta_\\text{task}$ as an example. Theoretically, assuming $\\theta_\\text{ref}$ in Step 1 is well trained, and the loss $\\ell^\\tau_\\text{ref}$ converges to 0. The term $\\ell^\\tau_\\text{exc}$ equals to $\\frac{\\ell_{\\text{proxy}}^\\tau}{{|\\mathcal{D}^\\tau_L|}}$. Assuming $\\ell^\\tau_\\text{proxy}$ will effectively go down per iteration as is trained. We make sure the first-iteration of $\\alpha^1_\\tau\\leftarrow\\alpha^0_\\tau e^{ \\eta_\\text{task} \\ell^\\tau_\\text{exc} } \\leq 1$ should not excess the natural upper bound 1 (note that $\\alpha^0_\\tau=\\frac{1}{K}$). Therefore, it is less risky to set  $\\eta_\\text{task}\\leq\\frac{\\text{ln}K}{\\ell^\\tau_\\text{exc} }=\\frac{|\\mathcal{D}^\\tau_L| \\times\\text{ln}K}{\\ell^\\tau_\\text{proxy}}$, where $|\\mathcal{D}^\\tau_L|$ is denoted as the training dataset size., $K$($K>1$) as the number of clients, $\\ell^\\tau_\\text{proxy}$ as the maximum value of first step proxy loss performed on task $\\tau$.\n\n3. We do not recommend extensive tuning for these parameters. For practical implementations, $\\eta = 1, 0.1$ or $0.01$ can be simple and effective default values. \n\n### **Q1: What task was used to optimize $\\theta_\\text{ref}$ in step 1? Is it one of the multi-tasks?**\n\n**Response to Q1:**\nThe reference model is trained on all multi-tasks assigned to each client, using their local training data without any task weighting. During Step 1, each client performs standard multi-task learning with equal task importance, ensuring the reference model captures baseline local knowledge across all tasks.\n\nThe key difference from subsequent stages is that while the reference model uses uniform task treatment, both proxy and agent models incorporate learned task weights $\\boldsymbol{\\alpha}$ to dynamically adjust task sampling ratios during training. The task categories remain identical across all three steps—only the dataset sizes are changing based on computed task weights and resampling."}, {"review_id": "aNlQsEb1ti", "reviewer": "Reviewer_gkRF", "summary": "FedRAM proposes a federated multi-task learning (FL-MTL) framework to address statistical heterogeneity, task interference, and global-local trade-offs in distributed settings. Its main contribution includes: \n\n1. Propose a novel reference-proxy-agent strategy for decoupled optimization.\n2. Adaptive tuning of the task importance parameter and client contribution parameter.\n3. Experiments on 6 benchmarks show ​​the accuracy gains​​ in in-domain and out-of-domain tasks", "questions": "1. Is the assumption in Theorem 2 (Appendix A) about idealized weight updates ( $w_k \\propto 1 / \\sigma_k^2$ ) reasonable?\n2. Add experiments measuring gradient variance across clients with/without proxy adjustments.\n3. Include more recent FL-MTL baselines and report wall-clock time.\n4. Add metrics like task-specific variance or min-client accuracy.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "​​Strengths​​\n1. Hierarchical model design decouples local adaptation, weight tuning, and global aggregation\n2. The proposed proxy models reduce computation by guiding the agents\n\nWeaknesses​\n1. The baseline comparison is limited \n2. The Ablation study (Table 3) omits component-wise impact \n3. Experiments (Fig. 5) show severe task/data imbalance, but FedRAM's resampling lacks fairness guarantees for minority tasks.", "was_revised": true, "final_justification": "Many of the concerns have been addressed.", "rebuttal": "We greatly appreciate your thoughtful comments and constructive feedback, which have been invaluable in improving our manuscript. We have focused our responses on the following key issues:\n\n---\n\n### **W1: The Ablation study (Table 3) omits component-wise impact.** \n\n**A1:** **Ablation Study Explanation:**\nWe'd like to address the misunderstanding and the part we did not addressed enough in the main text.\n\n* **FedRAM (Experiment 5)**: \n\n$\\theta_\\text{ref}$ →  $\\theta_\\text{proxy}$  → $\\theta_\\text{proxy}$ and $\\theta_\\text{ref}$\n\n$~~~~~~~~~~~$  |   $~~~~~~~~~~~~~~~~~~~~~~~~$    |\n\n$~~~~~~~~~~$  $\\boldsymbol{w}$    $~~~~~~~~~~~~~~~~~~~~~~~$    $\\boldsymbol{\\alpha}$\n\n$~~~~~~~~~~$  └-----------------------└------→ $\\theta_\\text{agent}$ → Evaluation\n\n* **Experiment 1**: w/o $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$, we train and evaluate on the FL agent model $\\theta_\\text{agent}$ (FedAvg). Also, $\\boldsymbol{\\alpha}$ and $\\boldsymbol{w}$ are not involved, which equals to FedAvg and serves as the comparison baseline against Exp 6 and 7.\n\n$\\quad\\quad\\quad~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$ $\\theta_\\text{agent}$ → Evaluation\n\n* **Experiment 2**: w/o $\\theta_\\text{ref}$, we train $\\theta_\\text{proxy}$ in FL without the use of excess loss $\\ell_\\text{exc}$ while keeping differential loss $\\ell_\\text{dif}$. In this setting, we evaluate $\\theta_\\text{agent}$ solely based on client weights $\\eta_\\text{client}$:\n\n$~~~~~~~~~~~$  $\\theta_\\text{proxy}$  \n\n$~~~~~~~~~~~~$  | \n\n$~~~~~~~~~~~$  $\\boldsymbol{w}$\n\n$~~~~~~~~~~~$  └-----------------------------→ $\\theta_\\text{agent}$ → Evaluation\n\n* **Experiment 3**: w/o $\\theta_\\text{proxy}$. In this setting, we can not derive the task weights $\\boldsymbol{w}$ and client weights $\\boldsymbol{\\alpha}$, so that we directly evaluate on $\\theta_\\text{ref}$.\n\n$\\theta_\\text{ref}$ -----┐\n\n$~~~~~~~~~~~~$  |\n\n$~~~~~~~~~~~~$ └-----------------------------------------→ Evaluation\n\n* **Experiment 4**: w/o $\\theta_\\text{agent}$. We evaluate on $\\theta_\\text{proxy}$ in this case.\n\n$\\theta_\\text{ref}$ →  $\\theta_\\text{proxy}$  → $\\theta_\\text{proxy}$ and $\\theta_\\text{ref}$\n\n$~~~~~~~~~~~~$  | \n\n$~~~~~~~~~~~~$  └-----------------------------------------→ Evaluation\n\nComponent wise, to better assess the results, please refer to Exp 1, 3 and 4.\n\n* **Experiment 6**: w/o $\\boldsymbol{\\alpha}$. $\\theta_\\text{agent}$ is trained using client weights $\\boldsymbol{w}$:\n\n$\\theta_\\text{ref}$ →  $\\theta_\\text{proxy}$ \n\n$~~~~~~~~~~~~$  |  \n\n$~~~~~~~~~~~$  $\\boldsymbol{w}$ \n\n$~~~~~~~~~~~$  └-----------------------------→ $\\theta_\\text{agent}$ → Evaluation\n\n* **Experiment 7**: w/o $\\boldsymbol{w}$. $\\theta_\\text{agent}$ is trained using task weights $\\boldsymbol{\\alpha}$:\n\n$\\theta_\\text{ref}$ →  $\\theta_\\text{proxy}$  → $\\theta_\\text{proxy}$ and $\\theta_\\text{ref}$\n\n$~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$  |  \n\n$~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$    $\\boldsymbol{\\alpha}$\n\n$~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$ └------→ $\\theta_\\text{agent}$ → Evaluation\n\nWhen it comes to component-wise ablation study, we compare Exp 1, 3, and 4, and we evaluate $\\theta_\\text{agent}$, $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$, respectively. It is observed that: \n1. $\\theta_\\text{agent}$ outperforms the other components both in global and local performance, for its larger model size and trainable parameters.\n2. $\\theta_\\text{ref}$ has a competitive performance on global evaluation but fail to improve on the local performance compared to $\\theta_\\text{agent}$.\n3. $\\theta_\\text{proxy}$ shows a giant gap among the components, because it does not necessarily learn the local and global knowledge in this system but to tune hyperparamters $\\boldsymbol{\\alpha}$ and $\\boldsymbol{w}$. \n\n| Exp  | Method | $\\quad$ | Evaluation on | Global | Local  |\n|---------|--------|---------|--------|-------|------|\n| 1 | w/o $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$ | | $\\theta_\\text{agent}$ | 72.71   | 76.34  |\n| 2 | w/o $\\theta_\\text{ref}$ |  | $\\theta_\\text{agent}$ | 71.17   | 73.58 |\n| 3 | w/o $\\theta_\\text{proxy}$ |  | $\\theta_\\text{ref}$  | 72.54   | 70.52  |\n| 4 | w/o $\\theta_\\text{agent}$ |  | $\\theta_\\text{proxy}$     | 2.69   | 2.51  |\n| 5 | FedRAM |  |  $\\theta_\\text{agent}$     | 75.94   | 79.62  |\n\n**Ablation Study on Hyperparameters**: Based on the original ablation study, we extend to two more experiments on the hyperparameters' impact. Exp 6 and 7 both show improvements uppon Exp 1, demonstrating their effectiveness.\n\n| Exp  | Method  | Weights | Framework $\\quad\\quad\\quad$ | Evaluation on | Global | Local  |\n|---------|---------|--------|-------|------|------|------|\n| 1 | w/o $\\boldsymbol{\\alpha}$, $\\boldsymbol{w}$ | None  | $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$, $\\theta_\\text{agent}$ | $\\theta_\\text{agent}$ | 72.71   | 76.34  |\n| 6 | w/o $\\boldsymbol{\\alpha}$  | $\\boldsymbol{w}$ | $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$, $\\theta_\\text{agent}$ | $\\theta_\\text{agent}$ | 73.57   |  79.51  |\n| 7 | w/o $\\boldsymbol{w}$    | $\\boldsymbol{\\alpha}$ | $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$, $\\theta_\\text{agent}$  |$\\theta_\\text{agent}$  | 75.20   | 76.78  |\n| 8 | FedRAM | $\\boldsymbol{\\alpha}$, $\\boldsymbol{w}$ | $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$, $\\theta_\\text{agent}$ | $\\theta_\\text{agent}$     | 75.94   | 79.62  |\n\n### **W2: Experiments (Fig. 5) show severe task/data imbalance, but FedRAM's resampling lacks fairness guarantees for minority tasks.**\n\n**A2:** Task and data imbalance is indeed an inherent challenge in federated multi-task learning under non-IID conditions.\n\nOur resampling strategy inherently protects minority tasks through several key mechanisms:\n\n1. Task Weight ($\\boldsymbol{\\alpha}$): The proxy model learns individual importance weights $\\alpha_\\tau$ for each task $\\tau$, ensuring that minority tasks receive appropriate representation regardless of their data volume. As demonstrated in **Figure 7**, our task weights dynamically adjust from the initial data size distribution (**red line**) to more balanced configurations (**blue lines**). Notably, when $\\eta$=0.1, minority tasks like PAWS and WikiQA receive significantly increased weights compared to their original data proportions. \n\n2. Distributionally Robust Optimization: The resampling mechanism employs principles similar to group distributionally robust optimization [1], where the local training process of proxy model balances performance across all tasks. As demonstrated in Figure 7, all task weights are distributed within the range of [0.2, 0.4] compared to the dataset size range [0, 0.8].\n\n[1] Sagawa et al. Distributionally robust neural networks for group shifts. ICLR 2020.\n\n\n### **Q1: Is the assumption in Theorem 2 (Appendix A) about idealized weight updates ( $\\boldsymbol{w}_k \\propto 1/\\sigma_k^2$) reasonable?**\n\n**Response to Q1:** The assumption $w_k \\propto 1/\\sigma_k^2$ in Theorem 2 is practically reasonable for **Fairness**: FL clients with high $\\sigma^2$ (caused by noise or extremely heterogeneous data) can harm model convergence or suppress the contributions of the minorities if weighted equally. Downweighting them (as in the assumption) is to achieve worst-case robustness by prioritizing clients with more reliable data. In this case, dynamically **upweights high-loss data groups** during training to improve worst-group performance. Groups with higher loss variance (or higher loss) are upweighted to ensure robustness.\n\n### **Q2: Add experiments measuring gradient variance across clients with/without proxy adjustments.**\n\n**Response to Q2:**\nSetup: The experiment setup is aligned with **Section 5.1**.\nImplementation: Track variance metrics with/without proxy adjustments at communication rounds 1, 10, 20, 30 and 40.\n\n| Method  | proxy adjustments $\\quad$ |  Round 1  |  Round 10 |  Round 20 |  Round 30 |  Round 40 \n|---------|--------|-------|------|------|------|------|\n| FedAvg | No  | 4.33e6   | 8.46e5   | 1.13e4   | 8.72e2   |   2.26e-3   |\n| FedRAM | yes | 2.67e6   | 3.58e4   | 9.67e3   | 4.48e2   |   1.86e-3   |\n\n### **Q3: Include more recent FL-MTL baselines and report wall-clock time.**\n\n**Response to Q3:**\n\nThank you for the suggestion. As shown in Section 5.2.3 **Table 2**, we have already presented wall-clock time (Computational Cost) for all methods. Our FedRAM achieves the fastest training time of 12,060 seconds, which is 14.2% faster than FedAvg (the second).\n\nWe add more experiments here:\n\n| Methods | Global |  Local  |  ID |  OOD |  Computational Cost (s) |\n|---------|--------|-------|------|------|------|\n| pFedFDA [1] | 74.12 | 77.38  | 71.77  | 75.68  |  23603  |\n| OBF [2] | 69.22   | \t75.75   | 70.93   | 67.48    |  32497  |\n|FedHCA$^2$ [3] | 71.31   |  80.56\t  | 72.14   |  70.45   | 42506  |\n| **FedRAM** (ours) | 75.94   |  79.62  |  72.89  |  76.32  |   12060  |\n\n[1]. Mclaughlin, et al. \"Personalized federated learning via feature distribution adaptation.\" NIPS, 2024.\n[2].Vani, Ankit, et al. \"Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamics.\" ICML. 2024.\n[3]. Lu, Yuxiang, et al. \"Fedhca2: Towards hetero-client federated multi-task learning.\" CVPR/CVF, 2024.\n\n### **Q4: Add metrics like task-specific variance or min-client accuracy.**\n\n**Response to Q4:**\n\nThank you for the suggestion. As shown in main text **Table 1**, we have already presented min-client accuracy (shown as **Bottom Decile** in the table) for all methods. We will update a clearer description about the Bottom Decile.\n\nWe add task-specific variance of the Global F1-score metric here:\n\n| Methods | min-client accuracy |  task-specific variance  |\n|---------|--------|-------|\n| FedAvg | 54.79 |  1.27  |\n| FedProx |55.45  | \t0.93   | \n| Ditto |  62.37   | 0.56  | \n| Fedrep | 66.89 | 0.33  |\n| MOON |  56.30   | 0.78  | \n| DBE | 58.53   |  1.12\t  | \n| FedMTL | 62.32   |   0.82\t  | \n| FedBone | 61.60   |   0.50\t  | \n| **FedRAM** (ours) |  73.21   | 0.24  |"}, {"review_id": "jbC4lGRUoW", "reviewer": "Reviewer_9ir2", "summary": "This paper introduces FedRAM, a federated multi-task learning (FL-MTL) framework that addresses statistical heterogeneity, task interference, and imbalanced client contributions. FedRAM proposes a three-stage design involving:\n\nLocal reference models to capture client-specific distributions,\n\nLightweight proxy models to dynamically tune task and client weights based on excess and differential losses,\n\nAgent models to aggregate using learned weights.\n\nThe authors demonstrate strong performance across six NLP datasets under both in-domain (ID) and out-of-domain (OOD) settings, with improvements in convergence speed and communication efficiency. The paper includes convergence proofs and extensive experiments.", "questions": "See above", "limitations": "Yes, limitations are reasonably addressed in Section 5.2.5 and Appendix B. However, the main paper should bring some of these forward, especially:\n\nSensitivity to hyperparameters,\n\nDependency on accurate proxy training,\n\nLack of support for non-transformer tasks.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper offers a well-structured and modular solution (reference, proxy, agent) that aligns with real-world FL-MTL settings.\n\nExperiments are comprehensive, covering ablation studies, scalability, convergence speed, computational costs, and performance on both ID and OOD tasks.\n\nThe method is evaluated against a diverse set of recent and relevant baselines, such as FedMTL, FedRep, FedBone, and DBE.\n\nWeaknesses:\n\nThe exponential update rules for task and client weights (Equations 7, 11) are intuitive but not enough motivated. Why exponential over other update rules?\n\nThe authors decouple the optimization into three phases, but interactions among these models are not sufficiently validated. What happens when proxy and reference disagree? Is the proxy model actually learning meaningful generalization signals?\n\nAll experiments are based on T5-small/base models. While practical, it leaves questions on generality across modalities (e.g., vision, speech) or architectures (e.g., CNNs, GNNs).\n\nThe choice of η_task, η_client, smoothing factor s and their effects are not discussed in the main text. These seem critical to convergence and stability.\n\nWhile the related work section is informative, it would benefit from a more comprehensive coverage of relevant baselines, especially in the areas of personalized federated learning and personalized device-cloud collaboration. The experimental section could also be enhanced by including these baselines, if time allows.", "was_revised": true, "final_justification": "Thank for authors' rebuttal, which addressed my concerns. I will maintain my positive attitude and score.", "rebuttal": "We thank the reviewer for the feedback. 9ir2 notes that “Novel combination of proxy-mediated task weights and differential-loss-based client weights within a unified three-stage pipeline” and “well-structured”, with “Comprehensive results”. We address specific questions below:\n\n---\n\n### **W1: The exponential update rules for task and client weights (Equations 7, 11) are intuitive but not enough motivated. Why exponential over other update rules?** \n\n**A1:** The exponential update rules follow established principles in distributionally robust optimization, particularly the multiplicative weights framework used in group DRO literature [1]. Exponential updates provide convergence guarantees and numerical stability when dynamically balancing competing objectives across different groups or tasks, especially in the non-iid fl setting. \n\nUnlike additive updates, exponential scaling prevents poorly-performing tasks/clients from being completely ignored while avoiding dominance by any single component. The exponential form ensures that weights remain positive (essential for valid probability distributions) while providing responsive adaptation to performance changes.\n\n[1] Sagawa et al. Distributionally robust neural networks for group shifts. ICLR 2020.\n\n### **W2: The authors decouple the optimization into three phases, but interactions among these models are not sufficiently validated. What happens when proxy and reference disagree? Is the proxy model actually learning meaningful generalization signals?** \n\n**A2:** We appreciate the reviewer's concern about model interactions. To clarify, the proxy model in FedRAM is not designed to learn generalization signals but serves as a hyperparameter learning mechanism for the subsequent agent model training.\n\n**Proxy Model's Specific Role (Figure 3):**\nAs illustrated in Figure 3, the proxy model's sole purpose is to produce two sets of hyperparameters: task weights ($\\alpha_\\tau$) and client aggregation coefficients ($w_k$). After Step 2 completes, the proxy model itself is discarded, and only these learned weights are passed to Step 3 for resampling and aggregation in agent model training.\n\n**Handling Reference-Proxy Disagreements:**\nThis disagreement is constructively managed through our loss design:\n\n1. Reference as Training Headroom: The model $\\theta_\\text{ref}$ serves as a local performance benchmark, capturing task-specific distributions that the proxy should respect.\n\n2. Excess Loss Mechanism: When disagreements occur, $\\mathcal{L}_\\text{exc}$ ensures that the proxy model's training aligns with the reference's performance. This prevents the proxy from learning weights that would degrade local task performance.\n\n**Ablation Study Explanation:**\nWe'd like to address the misunderstanding and the part we did not addressed enough in the main text.\n\n* **FedRAM** (Experiment 5): \n\n$\\theta_\\text{ref}$ →  $\\theta_\\text{proxy}$  → $\\theta_\\text{proxy}$ and $\\theta_\\text{ref}$\n\n$~~~~~~~~~~~$  |   $~~~~~~~~~~~~~~~~~~~~~~~~$    |\n\n$~~~~~~~~~~$  $\\boldsymbol{w}$    $~~~~~~~~~~~~~~~~~~~~~~~$    $\\boldsymbol{\\alpha}$\n\n$~~~~~~~~~~$  └-----------------------└------→ $\\theta_\\text{agent}$ → Evaluation\n\n* **Experiment 1**: w/o $\\theta_\\text{ref}$, $\\theta_\\text{proxy}$, we train and evaluate on the FL agent model $\\theta_\\text{agent}$ (FedAvg). Also, $\\boldsymbol{\\alpha}$ and $\\boldsymbol{w}$ are not involved, which equals to FedAvg and serves as the comparison baseline against Exp 6 and 7.\n\n$\\quad\\quad\\quad~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$ $\\theta_\\text{agent}$ → Evaluation\n\n* **Experiment 2**: w/o $\\theta_\\text{ref}$, we train $\\theta_\\text{proxy}$ in FL without the use of excess loss $\\ell_\\text{exc}$ while keeping differential loss $\\ell_\\text{dif}$. In this setting, we evaluate $\\theta_\\text{agent}$ solely based on client weights $\\eta_\\text{client}$:\n\n$~~~~~~~~~~~$  $\\theta_\\text{proxy}$  \n\n$~~~~~~~~~~~~$  | \n\n$~~~~~~~~~~~$  $\\boldsymbol{w}$\n\n$~~~~~~~~~~~$  └-----------------------------→ $\\theta_\\text{agent}$ → Evaluation\n\n* **Experiment 3**: w/o $\\theta_\\text{proxy}$. In this setting, we can not derive the task weights $\\boldsymbol{w}$ and client weights $\\boldsymbol{\\alpha}$, so that we directly evaluate on $\\theta_\\text{ref}$.\n\n$\\theta_\\text{ref}$ -----┐\n\n$~~~~~~~~~~~~$  |\n\n$~~~~~~~~~~~~$ └-----------------------------------------→ Evaluation\n\n* **Experiment 4**: w/o $\\theta_\\text{agent}$. We evaluate on $\\theta_\\text{proxy}$ in this case.\n\n$\\theta_\\text{ref}$ →  $\\theta_\\text{proxy}$  → $\\theta_\\text{proxy}$ and $\\theta_\\text{ref}$\n\n$~~~~~~~~~~~~$  | \n\n$~~~~~~~~~~~~$  └-----------------------------------------→ Evaluation\n\nBased on the original ablation study, we extend to two more experiments:\n\n* **Experiment 6**: w/o $\\boldsymbol{\\alpha}$. $\\theta_\\text{agent}$ is trained using client weights $\\boldsymbol{w}$:\n\n$\\theta_\\text{ref}$ →  $\\theta_\\text{proxy}$ \n\n$~~~~~~~~~~~~$  |  \n\n$~~~~~~~~~~~$  $\\boldsymbol{w}$ \n\n$~~~~~~~~~~~$  └-----------------------------→ $\\theta_\\text{agent}$ → Evaluation\n\n* **Experiment 7**: w/o $\\boldsymbol{w}$. $\\theta_\\text{agent}$ is trained using task weights $\\boldsymbol{\\alpha}$:\n\n$\\theta_\\text{ref}$ →  $\\theta_\\text{proxy}$  → $\\theta_\\text{proxy}$ and $\\theta_\\text{ref}$\n\n$~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$  |  \n\n$~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$    $\\boldsymbol{\\alpha}$\n\n$~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$ └------→ $\\theta_\\text{agent}$ → Evaluation\n\n**Experimental Validation:**\nOur ablation studies (Section 5.2.4, Table 3) systematically validate the synergistic contributions of each component. The results demonstrate that:\n\n(1) **Reference model serves as the local training headroom**: By comparing **Experiment 1 and 3**, we compare the direct performance of $\\theta_\\text{agent}$ (In Exp. 1) and $\\theta_\\text{ref}$ (In Exp. 3). In general, while both $\\theta_\\text{ref}$ and $\\theta_\\text{agent}$ can generalize to global evaluation, $\\theta_\\text{ref}$ fail to do well on local evaluation for smaller model size than $\\theta_\\text{agent}$.\n\n(2) **Proxy model do not necessarily learn the global representations**:\nThe performance on $\\theta_\\text{proxy}$ (Exp 4) suffers a great drop compared to Experiment 1. This is because proxy models are trained to tune $\\boldsymbol{\\alpha}$ and $\\boldsymbol{w}$. The knowledge $\\theta_\\text{proxy}$ learned (tuning hyperparameters) fail to apply to the dataset.\n\n### **W3: All experiments are based on T5-small/base models. While practical, it leaves questions on generality across modalities or architectures.**\n\n**A3:** \n\n**For extending to vision tasks**, we already presented the results in **Appendix B.6 (d)**. In this setting, we choose swin transformer to be our architecture for $\\theta_\\text{ref}$ and $\\theta_\\text{proxy}$, and clip-vit for $\\theta_\\text{agent}$.  We base our experiments on seven distinct CV datasets: MNIST, EuroSat, Fashion-MNIST, SVHN, DTD, Stanford-Cars, and CIFAR-10.\n\n**For architectures**, In Appendix B.6 (4) , we applied **Swin Transformer** and **CLIP-ViT**, which are not from the same model family. Based on the results in Appendix Table 9, we extend the experiments as follows:\n\n| Exp | Modality| $\\theta_{ref}$ and $\\theta_{proxy}$  | $\\theta_{agent}$  | Global | Local | ID   | OOD  |\n|-|-|-|-|-|-|-|-|\n|1| **cv** |Swin-T | CLIP-ViT  | 82.05 | 79.23  | 70.94 | 65.26 |\n|2| **cv** |CLIP-ViT | Swin-T  | 79.39 | 80.59  | 73.52 | 62.67 |\n|3| nlp |T5-base | T5-large| 84.29   | 90.29  | 91.53 | 87.26 |\n|4| nlp |T5-large | LLama  | 86.77   | 92.03  | 92.24 | 88.19 |\n|5| nlp |bert | T5-large   | 84.80   | 90.41  | 91.64 | 85.36 |\n|6| nlp |bert | LLama   | 84.43   | 89.64  | 91.78 | 87.40 |\n\nTo conclude, experiments both in CV (Exp 1, 2) and NLP (Exp 3~6) can validate FedRAM's generalization to model types.\n\n\n### **W4: The choice of $\\eta_{task}$, $\\eta_{client}$, smoothing factor s and their effects are not discussed in the main text.**\n\n**A4:**\n1. We employ a **logarithmic grid search**, ranging from 1E-2 to 1E2.  Our systematic analysis identified the optimal value as 1E-1, with the effective parameter range being $[-1, 1]$, as in ** Appendix B Figure 8**. Within this range, the impact on global framework performance remains consistently stable.\n2. Take $\\eta_\\text{task}$ as an example. Theoretically, assuming $\\theta_\\text{ref}$ in Step 1 is well trained, and the loss $\\ell^\\tau_\\text{ref}$ equals to 0. The term $\\ell^\\tau_\\text{exc}$ equals to $\\frac{\\ell_{\\text{proxy}}^\\tau}{{|\\mathcal{D}^\\tau_L|}}$. Assuming $\\ell^\\tau_\\text{proxy}$ goes down per iteration. We make sure the first-iteration of $\\alpha^1_\\tau\\leftarrow\\alpha^0_\\tau e^{ \\eta_\\text{task} \\ell^\\tau_\\text{exc} } \\leq 1$ should not excess the natural upper bound 1 (note that $\\alpha^0_\\tau=\\frac{1}{K}$). Therefore, it is less risky to set  $\\eta_\\text{task}\\leq\\frac{\\text{ln}K}{\\ell^\\tau_\\text{exc} }=\\frac{|\\mathcal{D}^\\tau_L| \\times\\text{ln}K}{\\ell^\\tau_\\text{proxy}}$, where $|\\mathcal{D}^\\tau_L|$ is denoted as the training dataset size., $K$($K>1$) as the number of clients, $\\ell^\\tau_\\text{proxy}$ as the maximum value of first step proxy loss performed on task $\\tau$.\n3. We do not recommend extensive tuning for these parameters. For practical implementations, $\\eta = 1, 0.1$ or $0.01$ would be simple and effective default values. \n\n### **W5: FedRAM could be enhanced by including the areas of personalized federated learning and personalized device-cloud collaboration, if time allows.**\n\n**A5:** \n\nWe appreciate the suggestion to include experiments in personalized federated learning (PFL) . However, we would like to clarify that FedRAM primarily focuses on global robustness and performance in **FL**, rather than personalization (**pFL**). We believe adding pFL comparisons may be **misaligned**, as the objectives and evaluation metrics differ significantly. To achieve the similar training goals, pFL would greatly harm the global convergence and tend to take up greater computational resources."}, {"review_id": "AJWY73enbC", "reviewer": "Reviewer_7drK", "summary": "The paper presents FedRAM, a three-stage framework for heterogeneous federated multi-task learning.\n\nStep 1 – Local Reference Model ($\\theta^{ref}$): Each client trains a lightweight model on its own mixed-task data to capture the local distributions.\n\nStep 2 – Proxy model ($\\theta^{proxy}$): Clients and the server jointly train another lightweight global model with task-specific weights $\\alpha$ that are updated from an excess-loss signal relative to the frozen reference. The server also derives client aggregation weights $w$ from the differential loss between local and global proxy performance.\n\nStep 3 – Agent model ($\\theta^{agent}$): a larger shared model is fine-tuned with data resampled according to $\\alpha$ and aggregated according to $w$.\n\nExperiments on seven NLP tasks with 10 clients demonstrate that FedRAM improves global F1, reduces the number of convergence rounds, and reduces total training time while maintaining both in-domain and out-of-domain accuracy gains.", "questions": "1. **Scalability to larger/backbone models:** Can FedRAM handle $\\ge$ 1 B-parameter transformers or vision encoders? A single ImageNet-100 or GLUE-MNLI run would support significance.\n\n2. **Robustness to partial participation:** How do $\\alpha$ and $w$ adapt when only a random subset of clients join each round?\n\n3. **Hyperparameter guidelines:** Figures 6 & 7 show strong sensitivity to $\\eta_{client}$ and $\\eta_{task}$. Could you propose a heuristic (e.g., set $\\eta \\propto 1/\\sqrt{rounds}$) that works across datasets?\n\nCriteria for score change: convincing large-backbone results and answering all questions would raise the score to 5. Evidence of instability under partial participation would decrease to 3.", "limitations": "The authors acknowledge compute requirements and manual tuning of the energy parameters, but do not address: (i) privacy/fairness risks of task reweighting, (ii) runtime/energy overhead per stage, and (iii) lack of evidence on large-scale or non-NLP tasks. Clarifying these would strengthen the limitations section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths** \n\nWell-structured paper, intuitive Figures, and a running example showing the three roles. Detailed appendix with proofs and extra ablations.\n \nClear algorithm description with pseudocode and convergence proofs for the weight-adapted FedRAM and its proxy stage.\n\nComprehensive evaluation on six baselines, multiple metrics (global/local, ID/OOD), convergence curves, ablations, and hyperparameter studies.\n\nThis paper addresses a significant gap: the simultaneous task-level and client-level reweighting in FL-MTL, which leads to faster convergence and fairer performance. The hierarchical reference–proxy–agent idea could inspire other resource-aware FL designs.\n\nNovel combination of proxy-mediated task weights and differential-loss-based client weights within a unified three-stage pipeline; distinct from prior task-correlation or model-merging works.\n\n**Weaknesses** \n\nAll experiments use T5-small/base and only NLP tasks; vision or larger-scale models have not been tested. No robustness tests against adversarial or partially participating clients.\n\nImprovements over the FedMTL are modest on some tasks; gains in ID/OOD may diminish for very heterogeneous or large-task suites. Practical adoption needs validation on larger models and non-text modalities.", "was_revised": true, "final_justification": "The authors provide thorough and convincing answers and new evidence that addresses all my concerns.\n\nI consider revising the score as these evidence meaningfully improves the paper’s significance and supports acceptance.", "rebuttal": "We thank the reviewer for the feedback. 7drK notes that “Novel combination of proxy-mediated task weights and differential-loss-based client weights within a unified three-stage pipeline” and “well-structured”, with “Comprehensive results”. We address specific questions below:\n\n---\n\n### **Q1: Scalability to larger/backbone models:** \n**Can FedRAM handle $\\geq$ 1 B-parameter transformers or vision encoders? A single ImageNet-100 or GLUE-MNLI run would support significance.**\n\n**A1:** Thank you for this important question regarding the scalability of FedRAM to larger models and standard benchmarks.\n\n**Large Language Model Experiments**: We have conducted experiments with Llama-3.2-1B to demonstrate FedRAM's scalability to billion-parameter models. The setup aligns with **Section 5.1**.\n\n| Methods | $\\theta_\\text{ref}$ and $\\theta_\\text{proxy}$|$\\theta_\\text{agent}$| Global | Local | ID   | OOD  |\n|---------|---------|--------|------|------|------|------|\n| FedAvg |$\\textit{None}$| LLama 1.24B| 84.20   | 86.08  | 86.32 | 83.69 |\n| FedRAM |bert 110M| LLama 1.24B| 84.43   | 89.64  | 91.78 | 87.40 |\n| FedRAM |T5-large 770M | LLama 1.24B | 86.77   | 92.03  | 92.24 | 88.19 |\n\nDue to computational resource and time constraints, we focused on existing datasets and comparison to Fedavg. From the table, FedRam can handle $\\geq$ 1B models with consistent results across the four metrics (still **outperforms** FedAvg). \n\n**Vision Encoder Experiments**: For vision encoders, please refer to Appendix B.6, where we present experiments on several independent datasets using large-scale vision models. Specifically, we employed Swin Transformer as both $\\theta_{\\text{ref}}$ and $\\theta_{\\text{proxy}}$ and CLIP-ViT as $\\theta_{\\text{agent}}$.\n\n\n### **Q2: Robustness to partial participation:** \n**How do $\\boldsymbol{\\alpha}$ and $\\boldsymbol{w}$ adapt when only a random subset of clients join each round?**\n\n**A2:** \n\nSetup: We base the experiments on the same task dataset, clients and model setting in the main text **Section 5.1**. \nWithin one communication round, we set participation ratio of clients to be 70% and 50%, which equals to 30% and 50% dropout rate, respectively. Under different dropout rate, we randomly select the clients to drop. Note that the task weights and client weights of the dropout clients are frozen for no FL training performed.\n\nWe set the dropout rate to be 50% for Exp 1 and Exp 2. From Exp 1, it is observed that $\\boldsymbol{\\alpha}$ is robust. For instance, the task weights in Client 2 are stable from round 30 to 50.\n\nExp 1: $\\boldsymbol{\\alpha}$ adaptation, here we take client 2's adaptation as an example:\n| Task  | round 0  | round 10 | round 20 | round 30   | round 40  | round 50  |\n|---------|---------|--------|-------|------|------|-|\n| 1 | 0.155  |  0.155  | 0.155  | 0.155 | 0.154  | 0.154  |\n| 2 | 0.225  |  0.210  | 0.209  | 0.207 | 0.205 | 0.205 |\n| 3 | 0.120 | 0.111   |  0.111 |  0.111 | 0.112 | 0.111 |\n| 4 | 0.120 |  0.123  |  0.125 | 0.125 | 0.126 | 0.126 |\n| 5 | 0.120 |  0.123  | 0.124  | 0.124 | 0.124 |  0.124 |\n| 6 | 0.140 | 0.154   | 0.153  | 0.155 | 0.154 | 0.155 |\n| 7 | 0.120 | 0.124   | 0.123  | 0.123 | 0.124 | 0.124 |\n\nIn Exp 2, we present the client weights $\\boldsymbol{w}$ across communication rounds. It is observed that $\\boldsymbol{w}$ changes slowly at the beginning for suffering from client heterogeneity. The client weights begin to converge from round 30 and exhibit robustness. \n\nExp 2: $\\boldsymbol{w}$ adaptation:\n| Client  | round 0  | round 10 | round 20 | round 30   | round 40  | round 50  |\n|---------|---------|--------|-------|------|------|-|\n| 1 | 0.100  | 0.105 | 0.142  | 0.160 | 0.166  | 0.167  |\n| 2 | 0.100  | 0.099 | 0.092  | 0.089 | 0.088 | 0.087 |\n| 3 | 0.100 |  0.099   |  0.090 |  0.088 | 0.086 | 0.086 |\n| 4 | 0.100 |  0.099  |  0.095 | 0.093 | 0.092 | 0.092 |\n| 5 | 0.100 |  0.100  | 0.101  | 0.102 |  0.099 | 0.099 |\n| 6 | 0.100 | 0.100   | 0.097  | 0.095 | 0.094 |  0.094 |\n| 7 | 0.100 | 0.100   | 0.097  | 0.095 | 0.095 |0.095 |\n| 8 | 0.100 |  0.099  | 0.094  | 0.091 | 0.090 | 0.090 |\n| 9 | 0.100 | 0.099   | 0.092  | 0.089 | 0.090 | 0.090 |\n| 10 | 0.100 | 0.100 | 0.100  | 0.098 | 0.100 | 0.100 |\n\nExp 3: We compare the partial clients participation in FedRAM and FedAvg:\n| Methods  | Global | Local | ID   | OOD  | Convergence Round |\n|---------|---------|--------|-------|------|------|\n| FedRAM|  75.94 | 79.62 | 72.89 | 76.32 | 18 |\n| FedRAM (Dropout Rate = 0.3) |  75.03 | 79.65 | 72.97 | 76.93 | 20 |\n| FedRAM (Dropout Rate =0.5)  | 73.67  | 76.88 | 70.92 | 74.71 | 24 |\n| FedAvg|  72.71 | 76.34 | 71.68 | 75.66 | 21 |\n| FedAvg (Dropout Rate = 0.3) |  72.54 | 74.33 | 70.67 | 75.68 | 24 |\n| FedAvg (Dropout Rate =0.5)  | 70.39  | 73.40 | 66.58 | 70.33 | 30 |\n\nFrom Exp 3, FedRAM outperforms the FedAvg baseline under the same dropout rate, demonstrating the robustness of FedRAM. Especially, it is noticed that FedRAM with even 0.5 dropout rate outperforms FedAvg with no dropout rate (full participation).  The main reason is that FedRAM performs better when less clients are participated. To confirm this, we refer to **Appendix B5** Figure 9(c). FedRAM shows an increase as the FL client number goes down, demonstrating FedRAM performs better when focusing on a smaller group of clients and performs more fine-grained reweighting.\nBesides, from Exp 2, we can observe that FedRAM upweights client 1 by 67%, which is effectively reflected on the metrics.\n\n\n### **Q3: Hyperparameter guidelines:**\n**Figures 6 & 7 show strong sensitivity to $\\eta_\\text{client}$ and $\\eta_\\text{task}$. Could you propose a heuristic (e.g., set $\\eta \\propto 1/\\sqrt{rounds}$ ) that works across datasets?**\n\n**A3:**\n1. To determine the exponential scaling factors, we employed a logarithmic grid search strategy, ranging from 1E-2 to 1E2.  Our systematic analysis identified the optimal logarithmic value as 1E-1, with the effective parameter range being $[-1, 1]$, as demonstrated in Figure 8. Within this range, the impact on global framework performance remains consistently stable.\n2. Take $\\eta_\\text{task}$ as an example. Theoretically, assuming $\\theta_\\text{ref}$ in Step 1 is well trained, and the loss $\\ell^\\tau_\\text{ref}$ equals to 0. The term $\\ell^\\tau_\\text{exc}$ equals to $\\frac{\\ell_{\\text{proxy}}^\\tau}{{|\\mathcal{D}^\\tau_L|}}$. Assuming $\\ell^\\tau_\\text{proxy}$ goes down per iteration. We make sure the first-iteration of $\\alpha^1_\\tau\\leftarrow\\alpha^0_\\tau e^{ \\eta_\\text{task} \\ell^\\tau_\\text{exc} } \\leq 1$ should not excess the natural upper bound 1 (note that $\\alpha^0_\\tau=\\frac{1}{K}$). Therefore, it is less risky to set  $\\eta_\\text{task}\\leq\\frac{\\text{ln}K}{\\ell^\\tau_\\text{exc} }=\\frac{|\\mathcal{D}^\\tau_L| \\times\\text{ln}K}{\\ell^\\tau_\\text{proxy}}$, where $|\\mathcal{D}^\\tau_L|$ is denoted as the training dataset size., $K$($K>1$) as the number of clients, $\\ell^\\tau_\\text{proxy}$ as the maximum value of first step proxy loss performed on task $\\tau$.\n3. We do not recommend extensive hyperparameter tuning for these parameters. For practical implementations, we recommend using $\\eta = 1, 0.1$ or $0.01$ as simple and effective default values."}]

Author rebuttal:

Meta-review: The paper introduces a framework named FedRAM for Federated Multi-Task Learning (FL-MTL). It's designed to handle challenges such as heterogenous client data and different learning tasks. The key idea is a three-step strategy: A proxy model acts as a middleman to balance contributions from different clients and tasks. This approach dynamically adjusts two key parameters: the task importance weight and the client aggregation coefficient. Experiments show that FedRAM improves performance and cuts down computational costs significantl compared to existing methods. Reviewers agreed that the framework is straightforward, novel in its application to federated learning, and shows significant performance improvements.

Initially, the reviewers raised several concerns despite finding the paper generally well-written and significant. Common concerns included the clarity of some mathematical notations, the need for more justification for certain design choices, and the sensitivity to hyperparameters (ablation studies).  The most significant point, raised by multiple reviewers, was the limited scope of experiments, which focused only on NLP tasks and a specific models. This raised questions about the method's scalability and general applicability to other domains such computer vision. Reviewers also requested robustness tests, such as how the system handles partial client participation.  The authors' rebuttal addressed most of these issues. The reviewers' unanimously confirm that the authors' thorough response addressed their concerns and recommending acceptance of the paper.
Decision: Accept (poster)