Co-LoRA: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients
Authors: Minhyuk Seo
Taeheon Kim
Hankook Lee
Jonghyun Choi
Tinne Tuytelaars
Venue: ICLR 2026 Poster (2026)
Keywords: Collaborative Learning
Federated Learning
Continual Learning
Multi-modal Learning
Personalization
Distributed Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: As AI becomes more personal, e.g., Agentic AI, there is an increasing need for personalizing models for various use cases.
Personalized federated learning (PFL) enables each client to collaboratively leverage other clients' knowledge for better adaptation to the task of interest, without privacy risks. Despite its potential, existing PFL methods remain confined to rather simplified scenarios where data and models are the same across clients. To move towards realistic scenarios, we move beyond these restrictive assumptions by addressing both data and model heterogeneity. We propose a task-relevance-aware model aggregation strategy to reduce parameter interference under heterogeneous data. Moreover, we introduce Co-LoRA, a dimension-invariant module that enables knowledge sharing across heterogeneous architectures. To mimic the real-world task diversity, we propose a multi-modal PFL benchmark spanning 40 distinct tasks with distribution shifts over time. Extensive experiments shows that our proposed method significantly outperforms the state-of-the-art PFL methods under heterogeneous scenarios. Code is available at https://github.com/snumprlab/fedmosaic.

Published as a conference paper at ICLR 2026
Co-LORA: COLLABORATIVE MODEL PERSONALIZA- TION ON HETEROGENEOUS MULTI-MODAL CLIENTS
Minhyuk Seo 1....2. 12+ Taeheon Kim Hankook Lee
Jonghyun Choi Tinne Tuytelaars
KU Leuven Scoul National University "uungkyunkwan University
(minhyuk. seo, t tinne. tuytelaars } @kuleuven.be
hankook. lee@skku. (thkim0305, jonghyunchoi] ) @snu. ac. ac.kr
ABSTRACT
As AI becomes more personal, e.g., Agentic AI, there is an increasing need for personalizing models for various use cases. Personalized federated learning (PFL.) enables each client to collaboratively leverage other clients' knowledge for better adaptation to the task of interest, without privacy risks. Despite its potential, existing PFL methods remain confined to rather simplified scenarios where data and models are the same across clients. To move towards realistic scenarios, we move beyond these restrictive assumptions by addressing both data and model heterogeneity. We propose a task-relevance-awarr model aggregation strategy to reduce parameter interference under heterogeneous data. Moreover, we introduce Co-LoRA, a simension-invarian module that enables knowledge sharing across heterogeneous architectures. To mimic the real-world task diversity, we propose multi-modal PFL benchmark spanning 40 distinct tasks with distribution shifts over time. Extensive experiments shows that our proposed method significantly outperforms the state-of-the-art PFL methods under heterogeneous scenarios. Code is available //ttps:///iitee thub. com/ snumpr lab/ fedmosaic
INTRODUCTION
Multimodal Large Language Models (MLLMs) with billions of parameters often employ centralized training massive, heterogeneous datasets using giggh-erformncce computing resources (Hurst et al. 2024; Yang et al. 2024a] Team et al., 2024). Such centralized training raises significant concerns about data privacy and high transmission costs (Alvi et al.) 2022 Yang et al. (023). Moreover, as the demand for personalization grows, further fine-tuning is essential to adapt these centrally trained
general-purpose models individual user preferences (Lau et 2024 Zhang 2024).
To address both privacy and personalization, personalized federated learning (PFL) (Smith et al. 2017) has emerged as a decentralized alternative. Recent PFL methods demonstrate that collabora- tively leveraging knowledge from other clients significantly improves personalization to the task of interest (Scott et al.) [224]) Xie al.) (024).. Unfortunately. despite its potential, most PFL studies overlook the heterogeneity inherent in real-world clients (Zhang et al.) [2024b] Bujotzek et al. (225)) where clients typically have distinct models depending on their computational resources (i.e., model heterogeneity) and deal with highly personalized data (i.e., data helerogeneity), as in agentic AI.
Although some recent work addresses client heterogeneity, mostly considers either model hetero- geneity (Fang et al. |2033] Wu et al. 2024a Yi et al. 2023) or data heterogeneity (Chen al. 2024a] Xie et al. 2024: Tamirisa et al.] 2024). Some tackle both, but in less realistic, simplified setups, e.g., assigning each client model with different LoRA (Hu et al. 2022) adapter ranks while keeping the base architecture identical (Cho et al. 2024 Bai et al. (2024). or lacking data heterogeneity by simply
splitting single dataset into non-i.i.d. partitions (Li & Wang 2019 Alam et al. 2022).
As realistic setup tackling both challenges, illustrated in Fig. we consider (i) data heterogeneity, where clients tackle highly personalized tasks, and (ii) model heterogeneity, where clients employ models of different families (e.g., Llama-based (Grattafiori et al. 22024)- vs. Qwen-based (Yang et al.) 2024a) MLLMs) and scales (e.g., IB vs. 3B). To mimic the real-world data heterogeneity, we first
indicates equal contribution. indicates corresponding authors. JC is with ECE, IPAL, ASRI SNU.

---

Published as a conference paper at ICLR 2026
Data Heterogeneity
Server
Global Model
Model Heterogenelty
Distribute
Upload
Client
Client 2
Client N
In
L/
LN
Mulli-modal Ressoning Tasks
Visual-Relation Tasks
VQA Tasks
Figure I: Overview of the heterogeneous personalized federated learning scenarios. Le refers to the local model for the ith client. Clients focus on different tasks (i.e., data heterogeneity) where new data encountered continuously. In addition to data heterogeneity, model architectures may differ across clients (i.e., model heterogeneity) due to differences in hardware constraints.
introduce DRAKE, a comprehensive benchmark for multi-modal PFL with 40 diverse tasks. Unlike prior works that simulate heterogeneity via non-i.idd splits of a single dataset (Xie et al. 2024 Long et al. 2024] Morafah et al. 2024), our benchmark assigns each client a distinet multi-modal task (e.g., visual question answering or visual reasoning). while also incorporating temporal distribution shifts inherent in the real world. To the best of our knowledge, this is the first benchmark for multi-modal FL that considers data heterogeneity, as well as distribution shifts.
To address the real-world challenge, we propose ModMosaic that jointly addresses both data and model heterogeneity in the realistic scenario. Under data heterogeneity, naive model averaging (McMahan et at. often degrades performance (Wu et al. [024c] Yadav et al. 2023) due to interference between models trained unrelated tasks. To mitigate the interference, inspired by the fact that models trained on similar tasks have less conflict (Gurulingan et al., 2022). we propose RELevance- guided Aggregation (RELA), which constructs a customized global model for each client based on task-relatedness. This enables related clients to share knowledge more effectively.
Under model heterogeneity, aggregating model weights is infeasible due to architectural mis- match (Fan et al.) 2024). Although federated distillation (FD) has been proposed to aggregate heterogeneous models' knowledge using logits from public data (Li et al., al..| (2024)). domain discrepan- cies between public and client data hinder effective knowledge transfer (Wang et al. 2023), and logit extraction computationally expensive (Malladi et al. 2023), especially for large models. Instead, (Co-LoRA), which enables cross-architecture collaboration via dimension-invariin modules P P R xr and Q RY. Since their sizes depend only on low-rank size r, independent of hidden dimension, they directly shareable across heterogeneous models.
we propose Collaborative-Lode
We summarize our contributions as follows:
Proposing DRAKE, a comprehensive multi-modal federated learning benchmark. Proposing RELA, a model aggregation strategy that promotes selective knowledge sharing among models learning relevant tasks only, addressing data heterogeneity. Proposing Co-LoRA, shareable across heterogeneous models, addressing model heterogeneity.
RELATED WORK
Personalized Federated Learning. Federated learning aims to train a strong global model in a distributed manner while preserving privacy by sharing model weights instead of raw data (Yurdem) ct al. 2024 Hu et al. 2024). With the growing importance of model personalization. which allows large foundation models adapt individual user preferences, contexts, and needs (Zhang et al. 2024c), personalized federated learning (PFL) (Smith et al.) 2017) has emerged. PFL aims to train a personalized model on each client's local data by leveraging shared knowledge from other clients to enhance both personalization and generalization while preserving privacy (Xie et al. 2024).
Recent PFL methods, such as DITTO (Li et al. 2021). FedSim (Pillutla al. 2022 FedDAT (Chen et al. 2024a), and FedDPA (Long et al. 2024), adopt dual-adapter designs, maintaining separate local and global adapters. The local adapter captures client-specific knowledge to address data heterogene- ity, while the global adapter preserves client-agnostic knowledge and mitigates overfitting, Despite their effectiveness, these methods build the global adapter by averaging local adapter parameters, which restricts collaboration to homogeneous models due dimension and depth mismatches. In con- trast, FedMosaic enables knowledge sharing even across heterogeneous architectures by proposing a dimension-inee -nvaaiiinnnninnnt shareable module. Moreover, unlike prior work that simulates data heterogeneity
2

---

Published as a conference paper at ICLR 2026
through non-ii.d. splits of single dataset, explore a more realistic setting where clients focus on different tasks, reflecting the diversity of real-world deployments.
Data Heterogeneity. Prior work simulates data heterogeneity by partitioning single image classifi- cation dataset (e.g., .NNITT (Deng 2012) into non-i.id. subsets per client. However, such label-skew fails to capture real-world data heterogeneity (Borazjani et al. 2025), where clients tackle different tasks across vision and language domains (Madni et al.] (024). Recent work (Chen et al. 2024a) moves beyond label skew by assigning different VQA datasets to clients, but remains confined to a single task type and single-image inputs. In contrast, our proposed benchmark, DRAKE, spans a broader range of multi-modal tasks, including VQA, visual reasoning, and visual relation, covering single- and multi-image inputs, while also modeling temporal distribution shifts within each client, reflecting the evolving and non-stationary nature of woal-world data (Garg et al., 2024).
Model Heterogeneity. Federated distillation (FD) transfers knowledge across heterogeneous models by sharing logits on public data. FedMD (Li & Wang, 2019]. and PerAda (Xie al., 2024) average logits from local models, while FedMKT (Fan et al. 2025) uses those with the lowest loss. However, public-client domain gaps limit transferability (Wang et al., 2023), and logit sharing introduces both privacy risks (Lyu et al. 2022) and high computational cost for large models (Malladi et al. 2023
Recent works address the limitations of FD through direct model aggregation for heterogeneous models under LoRA-based fine-tuning. HETLORA (Cho et al. 2024) and FLEXLORA (Bai et al., 2024) handle varying LoRA ranks through zero-par -oood-ddiggruunaattioooo and SVD-based redistribution, re- spectively. Although they address the varying rank sizes, both assume (i) identical hidden dimensions and (ii) uniform depths across clients, limiting their applicability to heterogeneous architectures that differ in both dimensions and depths (Yao) 2024). In contrast, our proposed Co-LoRA accommodates both dimensional and depth heterogeneity, enabling more general heterogeneous setups.
PRELIMINARIES
Low-Rank Adaptation (LoRA). LoRA (Hu et al. 2022) assumes that fine-tuning updates lie in a low-rank space. Building on this assumption, LoRA constrains the weight update AW for pre-trained weight matrix W, E do through low rank decomposition using matrices A E Rdr and B E Rdoxx, where the rank r & min(do, dr). During training, only A and are updated, while W, remains frozen. With LoRA, the original output ho = Wph E Rdo for an input hidden state hy E Rd is modified by incorporating the low-rank update as ho (W p h = (W, BA)hy
Problem Statement of Personalized Federated Learning. We consider PFL setup with N clients, where each client i E [N] has a local dataset D; Reflecting real-world scenarios where data arrives incrementally (Seo et al. 2024), each client receives a continuous stream of samples (of%) (()),, (al U32 (6) ),... ,..... Given a set of model architectures W = (W1,.... Wx), }, each client i V(i), based on its hardware constraints, where is a
selects its local pre-trained model w mapping function V {1...., N) + W. For efficiency, each client freezes w/(") while training only its local LoRA adapter Li. At each round, the server computes the global adapter G by aggregating the local adapters {Li...., L. } and sends G back to the clients. Let fx: w L,, G) be the model output, and let l be the loss function. The empirical loss for client over its local dataset D; is J(Lii G) = DD E(awEDD, E(f(x; w/ L,, G), We then formulate the PFL as min (LI.... N EN JT,,, G), following Zhang et al. 2021 Tamirisa et al. 2024
4 PROPOSED METHOD
To address the real-world heterogeneities, i.e., data and model heterogeneity, that hinder client collab- orations in PFL, we propose FedMosaic, illustrated in Fig. 2 comprising of: RELA (RELevance- guided Aggregation) and Co-LoRA (Collaborative-Lot) LoRA). RELA mitigates data heterogeneity by restricting knowledge sharing to local models trained on related tasks, thus reducing interference during aggregation (i.e., merging by parameter averaging). Under model heterogeneity, model ag- gregation becomes infeasible. To enable arosttrcciee collaboration, we introduce Co-LoRA, which incorporates shareable modules P E RR"x", Q E IR' in LoRA, whose sizes depend only on low-rank size r, not the hidden dimension. We provide pseudocode in Sec. A.35
3

---

Published as a conference paper at ICLR 2026
Client
W.
Be/OW ast
EMA
W
g
Server
Relevance Guided Aggregation
Transformer Layer
91
IN
Clinnt Relevance matrix
Customized Global Co-LoRA
91-
9N
Local Training
L
L
P Q
91
Wia Mag
G
Q
P
IN
:
Time
A
B
-
GN
Qu
P Qh
Ps Qu
Frozen
Trainable
Upload
Distribute
Softmax
Compression Noise
Figure 2: Overview of proposed FedMosaic. On every round, the local Co-LoRA I, fine-tuned during local training and the sanitized last layer gradient
uploaded from the ith client server. The last layer gradient is extracted on every m iterations from the small pre-trained model W., which then EMA updated and sanitized to g. In server, the sanitized gradients g. are used measure client task relevance and to build customized global Co-LoRA G,, which is distributed and kept frozen. h and Wp denote the hidden state input and the pre- ure-trained weight, respectively. | is learnable gating parameter that balances the output from the global and local models.
4.1 CELLVANCE-GUIDERR AGGREGATION
Model aggregation builds a single model that excels multiple tasks without accessing raw data (Wei al. 2025). It is thus widely used to construct a shared global model in federated learning. However, naively averaging models trained on different tasks often causes parameter interference (Yaday al. 2023 Yang al. 2024b). Recent work shows that models solving similar tasks share more transferable knowledge with fewer conflicts (Gurulingan et al. (022). Motivated by this, we replace the uniform averaging with relevance- -egiidde strategy that assigns higher aggregation weight to clients with closer task relations, providing each client with customized global model.
Client-Wise Gradient 9:- To measure task relevance between clients, we calculate the similarity of client-wise gradients. Formally, we calculate Ji, the gradient for in client as follows:
9: EsCD, [Vw,(>))]
(1)
where D; is inh client's data stream, l refers to the loss function, C DD is mini-batch, and W, is small-scale frozen pre-trained model W.. For efficiency, pre-trained model W, that provides sufficient representativeness with reduced overhead (Lee et al. 2024), and (ii) compute only the last-layer gradient, as preceding layers' gradients are proportional to it based on the chain rule (Sco et al. 2025). Moreover, we compute the gradient every m batch iterations rather than every batch, thus incurring negligible additional cost, as shown in Sec, A.88 Note that we measure gradients from frozen pre-trained model, not the actual training model. This because in heterogeneous PFL, clients train on diverse tasks, and gradient similarities from models trained on heterogeneous data may not capture task similarity (Tang al.) 2020; Evans et al. 2024).
use gradients from small-scale frozen
Decayed Client Wise Gradient ÿt. However, 9i may not reflect learned task relevance under shifting data distributions, it is expectation over the entire data stream D; (Eq. []] i.e., unweighted average of gradients across all time points, ignoring forgetting the model over time. Consequently, client learning A -> B and client learning B A yield the same 9., despite retaining different knowledge due to catastrophic forgetting (McCloskey & Cohen 1989 |atcliff 1990). To reflect the model knowledge shifts under distribution shifts, we introduce decayed gradient gi, computed using the exponential moving average (EMA) of past gradients, inspired by the exponential decay of
knowledge in forgetting (Mahto al. 2021 Chien al. 2021 Seo al. 2025). Formally, for the in client, (t) with EMA ratio at timestep t is defined recursively as:
(1) (1 a) gt 1) + a: (c()).
where gt) = Vw, (2x) ) is the gradient vector for the given batch 2t C D; at timestep t.
Sanitized Client-Wii Gradient 9:. Transmitting EMA-aggregated gradients (i.e,, 9:) rather than per-sample gradients mitigates gradient- privacy attacks, as gradient mixing (i.e, aggregating) increases resistance to gradient inversion (Mo et al. (021). We further prevent privacy risks by transmitting sanitized gradient ğ; through: (i) adding Gaussian noise to gi and (ii) applying gradient compression (i.e., randomly selecting only N, % of the gradient vector dimensions from di IRd). as
4

---

Published as a conference paper at ICLR 2026
randomly sampled dimensions approximate full- full-gradient distributions (Li al. 2023b) while making gradient inversion substantially more difficult than using full gradients (Zhu et al. 2019) and
simultaneously reducing transmission costs (detailed in Sec. A.9.. Formally, is defined as:
(t(t) = M @ (á.(t) + pe), e~ (O, La)
where M (0, 1)d is the binary mask for random subsampling and μ denotes the noise scale.
Using the samitized wlient-wise gradients {ōı.... EN from N clients, we construct a client-relevente matrix S RNXN, where Syy cos( 9j The customized global module for the im client, G2, is then constructed by weighted aggregation of local modules {L...... L as follows:
N
G
Σ j=1
wiy Lis
Wi
(4)
where T denotes the softmax temperature and cos(-,)) denotes cosine similarity.
4.2 Co-LORA
ho
LoRA matrices A E RR'xd and B E Rdoox depend model-specific hidden dimensions d, and do, preventing direct aggregation across different models. To enable knowledge shar- ing among heterogeneous architectures, we in- troduce Co-LoRA, which inserts dimension- invariant modules P E IR"x" and Q I'' be- tween A and Their dimensions depend only on the low-rank making them shareable across heterogeneous models. We illustrate compar- ison of Co-LoRA with conventional LoRA in Fig. 3 Formally, Co-LoRA outputs ho as:
B Rdo
Pretrsined Weights
QUR
WP
PER
W, R Ruod
A
(a) Conventional LORA
Co-LoRA
Figure 3: Illustration of (a) Conventional LoRA and (b) Co-LoRA. While A and B are trainable in conventional LoRA, Co-LoRA freezes both, updating only the dimension-invariant modules P Re and R² during training.
ho Wphs + B(PAN, +Q),
(5)
for input hidden states fı and pre- pre-trained weight Wp. After local training on each client, the P and Q modules aggregated and shared between heterogeneous clients for knowledge sharing.
Although and Q are shareable across heterogeneous architectures having different dimensions, two main challenges hinder their aggregation: (i) depth heterogeneity heterogeneous architectures often differ depth, making it non-trivial to determine which layers across models should aggregated; and (ii) weight misalignment interpolating weights with different optimization trajectories degrade performance (Jordan et al. 2023 Stoica et al.) 2025). We address these challenges with two strategies: (i) block-wise aggregation, which aligns layers the relative depth, and (ii) weight alignment, ensuring heterogeneous models share the same initialization. For simplicity, we consider aggregating heterogeneous models, though naturally extends to multiple heterogeneous models.
4.2.1 BLOCK- WISE AGGREGATION
Llama-1B vs. Llama-3B
To decide which layers of heterogeneous models should share knowledge, we measure layer-wise representation alignment between two MILMs, W, and W,, of different depths using CKA (Kornblith et al. 2019). We observe high similarity between layers at similar relative depths.
10
Llama-18
i.., approximately linear alignment (Fig. 4. See Sec. A. A0
for broader analysis beyond Llama- IB/3B. Accordingly, divide each model into Ng blocks linearly and attach Co-LoRA to each block's 's final layer, enabling cross cross-model sharing at relevant depths. The attachment layer index Te of the k-th Co-LoRA in W -lyyrr model defined as:
0.4
Llama-38 layers
Figure 4: Layer-wise similarity be- tween Llama-1B/33 measured with CKA. The diagonal brightest band shows the strongest alignment between layers at similar relative depths (e.g., Llama- IB layer 8 Llama-3B layer 14).
W Nn
for
k: 1, ..., NB k = NB-
1
(6)
W
for

---

Published as a conference paper at ICLR 2026
4.2.2 WEIGHT ALIGNMENT IN Co-LORA
We then align the NB number of Co-LoRA modules at layer indices I, = (Ti.... [2 } in model
M, and I; = [K,,.... .............................. 13 } in model Mj. For simplicity, we describe the alignment for a single pair x= ((P:, Q. A,, B.). (P). Qj: Aj. B2)), which generalized across all NB pairs.
Motivated by findings that models fine-tuned from the same initialization share optimization paths and can be merged without interference (Wortsman et al. [022a b) Yadav et al. 2003), we set the pair X to share common initialization. Dimension-invariin modules, P., Pj E R and Qi, Qj E R", can share the same weight directly, but dimension-depeeden modules, A, E grxdd A. E RRxx4() and B; E
R xr, Bj E TR xx, cannot due to dimension mismatches. Although only P and Q are shared between heterogeneous models M, and M,, A and B should also be aligned, they affect the output ho (Eq, (5) and optimization trajectories. Therefore, we align A,, Aj and B;, Bj. ensuring aligned representations for the same input, even in parameter spaces with different dimensions.
Aligning A Matrices. We first align A, E IR"x and A, E Rrxad" using L2 loss in the shared r-dimensional space with publicly available data D,. For Pp. we use a subset of the MLLMs' pre-
training data, as detailed in Sec. A.17 In particular, we use the smaller model A; (where d{²) df²) as the pivot model by freezing and updating the larger model A, to minimize the L.2 loss on r- dimensional representations obtained from p, as min [251 E(a,))E,, (I(A(()) (a0(x))AA00 (x) where
A (z) denotes r-dimensional feature extracted by A for input r. Please see Sec. A.29 for detailed justification of our choice of smaller model as the pivot model.
Aligning B Matrices. We then align B; and Bj. Since their output dimensions differ (d + L2 loss is inapplicable. Instead, we employ canonical correlation analysis (CCA) (De Bie al. 2005). which finds projection matrices maximizing the correlation between two feature sets. Using m public data {(T:: m== C Dp. we extract output features H₃ E R.mxd and Hj E Rmxdd from B3 and By. then find projection matrices IT E RR xr and IT, E R xr. which project them to the maximally correlated space of dimension r (i... H,ll; & H,l1,), We project B; to the shared maximally correlated space, i.e., (IIa)² B. and subsequently bring it to B,'s space by inverse projection, as B- (T, (II,)². B,, where II, 1 denotes the pseudo-inverse of II,.
During the alignment of and B, we enforce orthogonality in A and to maximize the expressive capacity (i.e., span) of Co-LoRA weight updates, following Theorem [ We provide details of enforcing orthogonality in See. An5 and the proof of Theorem I in Sec. A.2
Theorem 1. If the column vectors of matrix B E Rdoxr are orthogonal and the row vectors of matrix A E Rxx are orthogonal, then the span of the weight update space of Co-LoRA, span (AW), ), has x2 dimension, which the maximum possible dimension under frozen B and A.
In addition, our method does not incur much computational cost since (i) Co-LoRA aligning is performed once before federated training to establish a shared initialization, (ii) it is required only for heterogeneous model-type pairs (since some clients may share the same architecture). After aligning A and B, we update shareable modules P' and Q, while freezing A and B during local training to preserve alignment. We provide theoretical justification for this freezing approach in Sec. A This freezing design also reduces the communication cost of Co-LoRA relative to conventional LoRA, as it requires communication of only P, Q modules. See Sec. A.9 for details of communication costs.
After receiving the aggregated Co-LoRA (i.e., global Co-LoRA) by RELA at each communication round, clients freeze it during training to preserve global knowledge and update only the local model for personalization. Specifically, at the 4n layer, given an input hidden state h1, we combine the output of the local LoRA (i.e.. h&), the frozen global LoRA (i.e., hG), and the pre-trained weights (i.e., Wp w,,,), by adaptively balancing them using learnable gating parameter B. With sigmoid- ngmoii-normalized balancing parameter 3 o(3), the output hidden state ho is computed as follows:
ho = Wphr + (1 - 3)hr + BhG.

---

Published as a conference paper at ICLR 2026
Malti Datn Sources
Dieriterann
Mush Imago Suppor:
Matti Medaliies
Unsere Evalaation
NodIID 2001)
Corbeeeeeeeeer Miail
LEAF FCL
MINIST-Sharlle (C1
Yoall Uroamanding
Mull-mdda Reasoning Tasks
Irage Quallt D4
HC-FMT) (CVPR 3051)
Fed-SNI (NW 1077)
FEDE RGAL (ACT) 2007
X
Fed (NcullS 2004
Gavera
Fed FLAN (Nou 2024)
Unddersanding
DRAKE
VQA Tooks
(AAA)
Visual
DRAKE (Ours)
Relation
Tisiks
Table Comparison of FL benchmarks across key di-
Nelaton
Unsean
QA
mensions: Multi-Data Sources (using diverse datasets vs. non-i.i.d. splits of a single dataset), Distribution Shifts (evolving client data distributions), Multi- Image Support (handling multiple images per input), and Unseen Evaluation (testing tasks unseen during training). See for the detailed comparisons.
Tiskk
A
Parmin PoRlttoo
Conteetrrah ON
New Moull
Sec. A.28
Figure 5: Overview of DRAKE.
Dual Image Relation QA
'Remove the women from the image" Is the sentence refers to change to 3. B
Fashion Relation QA
Visual Relation Tasks
Image has blus color and longer slevves
True b. False
b. to
Visual Figurative Understanding
Context-dependent Visual Reasoning
Multi-modal Reasoning Tasks
What phrase this image shows? a. moves on b. off the wall C. turn the corner
Positive:
Negalive:
What common action Positive set?
b. ride c. turn wash
go
icon-domain
Image Quality QA
What the focus like in the image?
VQA Tasks
Blurry with visible noise b. Clear with no noticeable noise Completaly
What image?
Moon
Star
noise
Figure 6: Data samples of DRAKE.
DRAKE BENCHMARK
We propose novel multi-modal FL. benchmark, called DRAKE, with three key merits: (i) Task heterogeneity: Each client handles distinct multi-modal tasks (e.g., visual reasoning, VQA), while existing benchmarks merely assign non-i.id. subsets from a single dataset. (ii) Dynamic distribution: Client datasets contain progressive tasks (e.g., encountering new visual concepts), simulating real- world temporal distribution shifts. To the best of our knowledge, DRAKE is the first benchmark supporting multi-modal federated learning under distribution shifts. (iii) Generalizability Evaluation: DRAKE incorporates unseen task data to evaluate the generalizability of FL models. We summarize the comparison with existing FL benchmarks in Tab. Specifically, scale multi modai benchmarks that differ in distribution or properties from LLa'A's pre-training data and provide clear metadata or instructions, enabling task-wise splits for emulating distribution shifts. DRAKE consists of three training task subgroups, i.e., VQA, visual relation, and multi-modal reasoning. and two unseen task subgroups, comprising 40 heterogeneous tasks sourced from 19 different multi- modal datasets, totaling 375k images and 274k questions. illustrated in Fig. 5 We provide data samples of each sub-group Fig. Please refer to Sec. A.27 for more details.
curate DRAKE using large-
Visual Relation Tasks. Visual relation tasks focus on capturing relationships among visual objects. They comprise 12 distinct tasks grouped into three splits: fashion-relation, dual-image relation, and spatial/temporal relation. The fushion-relatio split targets fine-grat -oeegrained attribute understanding in clothing images, the dual-image relation split evaluates comparative reasoning across separate images, and the spatial/temporal split assesses understanding of positional and temporal dependencies.
Multi-modal Reasoning Tasks. Multi-modaa reasoning tasks require integrating visual information with language and common-sena knowledge. We group 12 tasks into three splits: visual figurative understanding, which inspects interpretation of non-literal, figurative, and idiomatic descriptions of images, depeededee visual reasoning, which tests discrimination among highly similar yet
7

---

Published as a conference paper at ICLR 2026
contrastive visual sets, and malti-image-ter understanding, which evaluates reasoning over multiple images or image-text pairs provided as contextual evidence to understand.
VQA Tasks. The VQA sub-group contains 16 tasks that differ from LLa's's pre-training data and is organized into four splits: image quality, icon-domain, dommai, and general VQA. The image quality split evaluates photographic properties such as focus brightness. The icon-domain split requires interpreting iconic images. The cuisine-domain split involves fine-grained food recognition combined with geographic cultural knowledge. The general VQA split covers standard settings, including chart and diagram understanding, visual instance identification, and counting.
Unseen Tasks. To assess generalization capability, we add 7 additional tasks that are disjoint from the clients' tasks in DRAKE. These benchmarks are intentionally challenging and require substantial training. spanning novel task types and novel visual domains with familiar task formats.
EXPERIMENTS
6.1 SETUPS
Models.
To simulate model heterogeneity in federated MILM training, we employ LLa (Lu
et al. 2023b) variants. For LLM of LLaVA, we employ various sizes of Llama-3 Grrttafioni et al.
2024 (IJamma3..--B, Llama-3.2-3B, Llama-3.1 1-8B) and Qwen-2.5 (Yang et al., 2024a) (Qwen2.5-
0.5B, Qwen2.5-1.5B, Qwen2.5-3B). We use the Llama-3 series for text-only benchmarks.
Metrics. We report Atast, the accuracy at the end of training, and AAUC (Koh et al. 2022), which computes the area under the accuracy curve by measuring aceuracy at each evaluation period to capture intermediate performance. All experiments use five rounds of evaluation intervals and are averaged over three different random seeds, with standard deviations reported.
Benchmarks. We evaluate FedMosaic on our proposed DRAKE and the benchmark used in Chen ct al. (0024a), which we refer to the Heterogeneous Federated Learning Benchmark (HFLB). Our evaluation covers PFL-Static (i.e., i.i.d. intra-client data distributions) and the more realistic PFL- Dynamic setup (i.e., intra-client distribution shifts with four tasks). We also evaluate on text-only PFL benchmarks, i.e,, Fed-Scope (Kuang et al. (022)) Fed-Aya (Singh et al. (2024), and Fed-LLM-Large. which combines Fed-LLM (Ye et al. 2024) and Fed-FLAN (Long et al. 2024).
Baselines. We compare FedMosaic with SOTA PFL methods. We also compare with supervised fine- tuning (SFT), where clients train independently without knowledge sharing and often outperforming PFL methods under data heterogeneity (Ghari & Shen 2024).
See Sec. for the details of benchmarks, baselines and implementation details.
6.2 QUANTITATIVE ANALYSIS
In all experiments, we evaluate each client's model on its own task (i.e., 'Self") and on other clinets' tasks (i.e., 'Others'), ) reporting average performance across clients. 'Self" performance shows personalization, while 'Others' indicates generalizability. Although personalization is PFL's main goal. generalization ability is also crucial for continual personalization. as it enables rapid adaptation to new tasks in the future during training (Finn et al. 2017 Rao et al. 2023).
Heterogeneous Multi-Modal Clients. We evaluate FedMosaic in multi-modal heterogeneous PFL.- Dynamic and Statiic setups, where clients employ different architectures (LLaVA-Llama3- or 3B). As shown in Tab. and Tab. 3 FedMosaic consistently outperforms baselines on both clients' own tasks ( ('Self") and others' tasks ('Others") under static and dynamic distributions. HFLB's 's single VQA task with angle-image inputs converges faster than DRAKE, yielding smaller gaps among baselines.
Note that Tab, reports the average performance across all clients. Local client training (SFT) can suffice for simpler tasks (Mosbach et al. 2021 Wožniak et al. 22024), e.g., single-image VQA (clients 2, in Tab. 5) making average improvements appear small. In contrast, FedMosaic significantly enhances personalization for complex multi-image tasks (clients 5, 7, 9 in Tab. (5) through effective knowledge sharing. See Sec for additional per-client results and Sec. 7.27 for task descriptions.
As shown in Tab not only do clients using smaller models (i.e., LLaVA-Llama3.. BB) benefit from knowledge sharing through PFL., but larger models (i.e., LLL.VA-Llama3.2-000 also see significant
8

---

Published as a conference paper at ICLR 2026
DRAKE- -Dnnamic
HFLB-Dyren BFFB--Dyamic
Self
Others
Self
Others
Method
Amse
Aare
Atuct
Aare
Auast
AAtr
Ansa
Amuc
SFT
0..79
26
47.66
46.67=0.11
79.9
77. 07.14=0.33
61.6610.56
61.43200.14
DITTO (CML) 2021)
59.91=0.118
54.16-00.06
47.45=0.36
46.70=0010
19.10+0.11
76.05-0.08
61,86++0.82
61.43:+-0.08
FedSim (CML 2022)
56.42.10.70
47.0410.116
46.15_0.17
79.90.0.099
76.47-0.0
59.92.1.544
59.56100.13
FedIT (ICASSP 2024)
66.11+00:2
57.86++0.27
47.63+0.16
46.62-0.04
79.87+0.50
77.04-0:38
61.73+0055
61.45+0.18
TAKFL (NeurIPS 2024)
64.54=0..5
56.19=0..88
47.38=0.06
46.4==0.44
79.76+0.06
76.86=0.04
61,42+00.4
61.22=00110
Fed1>PA (NerPPS 2024)
63.34+0:23
55.24++0.38
47.61+0.21
46.64-0.20
70.06+0.22
17.10-0.36
61.58+0.27
19+0..0
FedDAT (AAA] 2024)
58.47=1.10
54,38:00.28
48.91=0042
47.78=0.18
79.61
77.43=1002
64.155=0.08
64.51=0.13
PerAda (CVPR
3.075::1.05
54.48.0..13
47.30
46.72-10.3
79.03.1..01
75.92-0001
61.7610.80
61.3910.066
FodMKT (COLING 2025)
61.38=0.18
55.45:0.31
47.50=0.07
46.68:00088
79.48:0.01
76.57=0006
61.54:0.61
61.1120.19
FedMosaic (Ours)
67.86-0.55
59.83-00116
51.16-0.M
49.36-9.000 0.08
80.80 0.26
78.43-
0.14
67.07
0.10
Table 2: Quantitative comparison in heterogeneous PFL. 'Self" denotes evaluation
client's
own data, while 'Others' denotes evaluation on data from other clients. In DRAKE, 4 clients use LaaVA-Llama3.2111 and 6 clients use LLaVA-Llama3.2-33 while in HFLB, 3 clients use LLaVA- Llama3.2-1 and 6 clients use LLaVA-Llama? -Lama3 2.3B. SFT refers to supervised fine-tuning on each client's data without cross-client knowledge sharing.
Self
Others
Self
Others
Method
Awst
Aauc
Ausst
Awa
Mothod
Ator
Awx
Ascat
Awac
SPT
68.50=0.42
0372770.27
07.91=0399
0:
68.60+0.57
61.33
48.3440.15
47.53::0222
DITTO
63.67=1.50
59.17:+1225
48 48.22+00.9
47.65+00:05
DITTO
66.77-0.66
60.67+0777
49.04+0.65
48 33-+0.55
FedSim
66.55-0.66
61.20+0060
46.93+0.25
46.76+00.02
FedSim
66665+0.26
59.49+0.17
46 46.77+0.299
46.42+0.11
FedIT
68.71:0.04
47.91:0.22
47.6010.17
FellT
68.7299997
60.88:081
48.22:0.17
47.25:t0.22
TAKFL
67.28.00.01
62.42.10.36
47.551:0233
47.5110.07
TAKFL
67.771:0.46
60.13...32
48. 18:10.07
47.5111.14
FedDPA
66.09=1.51
61.36:10.11
17.93:10.00
47.6910.11
FodDFA
67.38:10.58
60:20:10:10
18.10110.22
47.32:10.33
FedDAT
61.28:00.07
57.92100022
49.3710.22
48.78110.44
FedDAT
66.08100.95
60.00:10:2
50.0511.002
19.1011.111
PerAda
63.67-1.01
59.03+1.19
48.13+0.03
47.60+0.55
PerAda
86+00.511
58. 38-33+048
47,80 47.8900770
47.45-+0.45
FeJMKT
65.81=1.099
60.37:11.16
47.19220.86
46.4411.49
FolMKT
63.4-120991
59. 19.16:2062
48 48.0920.41
47,51
FedMossic
0..10:0.53
64,64:00.00
51:57:0.24
50.42::0112
FodMosaic
70.67:0.661
63:1::0.4
52.31:10.15
50.60:10.66
Table 3: Quantitative comparison in PFL on Table 4: Quantitative comparison in cross-
DRAKE-static. 4 clients use VA-Laaa-33 family PFI. on DRAKE-dynanic. 3 clients use IB, clients use V.
LLaVA-Llama332-33 4 use -LaVA-QwenZ) 1.5B, 3 use LLaVA-Qwen2.5-2.
LLaVA-Llmaak--In
LLaVA-Lamm3.2-33
Mellad
Clicse 1
Clicst 2
Clicat
Clicsi 4
Clisnt 5
Client 6
Clian
Cliest 8
Client
Client 10
SFT
76-45/ 90
58 101 54 35
6663/ 99.57
63.85/51 16.16
62 62.00/ 51
76 56. 66.88
63 s0y 54.63
48 Sc/ 90.65
/ 51.93
54.66
FalMuesic
77 42) 60.45
/8.50/55.26
66.71/59.0
63.13 55.84
6837
63.60
77.61 69.30
69.19 56.66
70.54/ 58.93
78.6//4419
59.21 /5.32
Gail
+0.57
+0,407 +0.1
1008/ +0.44
+0.271220 22.7
16886/ +2.09
11.05 (2.93
/6.61/
12.00/ 0,72
+4.76/ +2.16 -0.91
Table 5: Per-client 'Self' performance of FedMosaacc vs. SFT in a heterogeneous PFL--Dnamic setup on DRAKE. The Gain row shows positive gain of FedMosaic in blue, negative in red against SFT. Clients 1-4 use -LaVA-Llama3.2-1B. while Clients 5-10 use LLaVA-Llama3.2-3..
gains. We attribute this to (i) RELA accurately measuring task relevance under distribution shifts, reducing interference during aggregation, and (ii) Co-LoRA enabling effective knowledge transfer between heterogeneous architectures, allowing smaller and larger models to mutually benefit.
We also emphasize that FedMosaic significantly outperforms baselines in generalization ('Others'), which is crucial under distribution shifts where new, unseen tasks continuously emerge. This enhanced generalizability enables faster adaptation and accelerates future personalization, as shown in Fig.
Cross-family Heterogeneity. Beyond varying model sizes within the same family (e.g., LLaVA- (lama-1/3BB) we further simulate cross-family heterogeneity (e.g., Qwen- vs. blsed MLLMs). Tab. 4 shows FedMosaic consistently outperforms baselines under cross-family het- erogeneity, highlighting its generality. See Sec. A. for additional cross-family experiments.
Fast Adaptation Evaluation. In real-world scenarios, unseen tasks continuously emerge, making rapid adaptation crucial for future personalization. We evaluate adaptation speed by initializing models with each PFL method's aggregated model and fine-tuning on unseen tasks for 200 iterations. Fig. shows that while sufficient training eventually yields similarly high performance regardless of initialization, models initialized with FedMosaic achieve high performance within few steps, highlighting enhanced generalizability through effective knowledge sharing in heterogeneous PFL.
Large-Scale Evaluation of Heterogeneous PFL in LLMs. Beyond multi modal PFL, we evaluate FedMosaic in heterogeneous PFL for LLM training (i.e., text-only NLP domain). Tab, 66 shows Fed significantly outperforms baselines in both 'Self' and 'Others', consistent with our

---

Published as a conference paper at ICLR 2026
Random Inlt
SFT
FedIT
FedSIM
FedDPA
FedMKT
FedMosalc (Ours)
DreamSim
RecipeQA: VisuaiCloze
Contrastive Caption:Unseen Img
ImageCoDe
80
Accuracy (%)
Accuracy (%)
Accuracy
60
Accuracy
30
60
40
80
25
20
20
100 Iterations
200
0
100
200
200
100
200
Iterations
Iterations
Iterations
Figure 7: Comparison of adaptation speed. We use DRAKE's unseen tasks as downstream tasks.
Random init starts from randomly initialized models, while other baselines are initialized from aggregated local models trained on DRAKE using each respective FL baseline.
Sell
Others
DITTO FedDPA
FedSIM
PerADA
Method
Absut
Aacc
Aauc
FedDAT @ FedMKT
FedMosaic (Ours)
SFT
18.03=0.52
17.11:0.56
14.18
13.70=00.35
FedSim (CML, 2022)
16.29=0.46
16.57=0.26
12.62.0.088
12.60=00.10
68
FedIT (CCSSPP 2004)
17.10=0.20
16.83=0.37
14.03=00.29
13.68=0.32
60% Less FLOPs
TAKFL (NcutPS
14.04=0.211
16.37=0.82
11.3620.13
12.57=0.06
67-
2.6 %
(NeurPPS 2024)
17.60-00225
16.66_0.12
13.66-0..07
FedDAT (AAAI 2004)
17.06-00113
15.83-0.03
13.72-00.04
13.02-0.01 14.66:10.05
PerAda (CVPR 20124)
18.83-0.80
17.32_0.88
13.79_0.00
65
FodMKT (COLING
17.31-0.11
17.03-10.08
14.38.0.011
13.89.0.02
1.0
1.5
2.0
2.5
FedMosaic (Ours)
20387-0.13
19.07-00300
Relative FLOPs
Figure 8: Accuracy and relative FLOPS in DRAKE-Dynamic.
Table 6: Large-Scale Experiments with 52 heteroge-
neous LLMs (Llama-1B/3B) on Fed-LLM-Large.
DRAKE
Fed- Scupe
Sclf
Others
Self
Others
Alo
Aucc
Aust
Awuc f
Alss
Aque
Abse
Aaue
Vanilla
66. 10:20.22
57.87110227
47.6320.16
46.62:10.04
25.06:2 1.36
28. 69
25.86:10.94
27.9410.4
Co-LLOA
66 99+0077
58 38.39+-0.38
51,33-0.06
48.78+0.05
0.527
31 64+0.21
3088++1.3
32.27+0082
Co-LeRA (Ours)
67.8610.551
59.8310.16
51.1620.11
49.36110.88
32.68:0.44
31,01::466
Table 7: Ablations for proposed components of FedMosaic in heterogeneous PFL. 'Vanilla'
aggregates homogeneous local models within each model type by averaging them with equal weights.
MLLM-FL benchmark results. This experiment involves 52 clients, highlighting the scalability of FedMosaic to large client populations and its applicability in real-world deployments.
Computation and Communication Cost Analysis. We compare accuracy and relative FLOPs in Fig. 88 FedMosaic demonstrates higher personalization performance with computation comparable to SFT. See Sec. A.8 and Sec. A.9 for detailed analyses of computation and communication costs.
Additional Experiments. Additional results include: heterogeneous PFL with LLAVA-L.lama3.2 1B/3B/8B (Sec I.L. LLLVVA-Qee2.55 variants (S1c text-only benchmarks (Sec. A.1)).
homogeneous PFL setup (Sec. (2.)), extended fast adaptation evaluations (Sec. AA.33 comparison of RELA with similarity-awart aggregation methods (Sec. A.22),, and client model selection details
(Sec. A.3) and configurations (Sec. (A7)). We further provide hyperparameter analysis Sec. A. (8).
effect of weight alignment in Co-LoRA (Sec. A.21), effect of client model assignment (Sec. A.4).
effect of decayed client-wise gradient ğ in RELA (Sec. A.19). and limitations (Sec. A.33
Ablation Study. We ablate FedMosaic investigate each proposed component's benefit, and summarize the results in Tab. 7 Our observations indicate that each component significantly enhances both the personalization ('Self") and the generalization ('Others') performance. Specifically, Co- LoRA enhances generalization by transferring knowledge across heterogeneous architectures, while RELA improves personalization by promoting knowledge sharing among relevant-task models only.
7 CONCLUSION
We address both data heterogeneity and model heterogeneity in PFL while prior works tackle only one challenge or both under simplified assumptions. For that, we propose FedMosaic, which handles both forms of heterogeneity through two components: RELA and Co-LoRA. Moreover, to better reflect real-world data heterogeneity in PFL scenarios, we introduce DRAKE, a comprehensive multi-modal FL benchmark capturing task heterogeneity and distribution shifts. Extensive evaluation shows FedMosaic achieves superior personalization compared to local training, while improving generalizability and few-shot adaptation capabilities essential for future personalization.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We propose a better learning scheme for heterogeneous federated learning for realistic federated learning scenarios. While the authors do explicitly aim for this, the increasing adoption of deep learning models in woaldw contexts with streaming data could potentially raise concerns, such as inadvertently introducing biases or discrimination. We note that we are committed to implementing all feasible precautions to avert such consequences, as they are unequivocally contrary to our intentions.
REPRODUCIBILITY STATEMENT
To further facilitate the reproduction, we provide open-source implementations of our proposed method (Sec. 4.. benchmark (Sec. (). along with data splits and baseline models used our experi-
ments (Sec. 6 available at https: /tt:://gitttt com/ snumpr ab/ fedmosa
THE USE OF LARGE LANGUAGE MODELS
We use large language models (LLMs) to support labor-intensive and mistake-prone work. Specif- ically, we use LLMs (e.g., GPT-4) categorize samples from single dataset into multiple tasks based on keywords topies, to assess generation quality in specific benchmarks using LLMs, and to detect grammatical errors during writing.
ACKNOWLEDGMENT
This work was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant Agreement No. 101021347), the IITP grants 1, RS- 2021-1121134 RS-2024-00437633
TRS-2022-11220077 RS-202 2022-1120113, RS-2022-11220952 (SNU AI), RS-2025-25442338 (AI Star Fcllowship-NNU, 10--0025-254422 funded by the Korea government (MSIT), grants 2SS-2025-2546289 (US-KOR BARI). RS-2025- 25453780) funded by MOTIR, grant of Korean ARPA-H Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (RS-2025-25424669)) and the BK21 FOUR program, SNU in 2025.
We also acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputers MareNostrunt at BSC, Spain; LEONARDO at CINECA, Italy; VEGA IZUM, Slovenia; Karolina at IT4Innovations. Czech Republic; MeluXina at LuxProvide, Luxembourg: Discoverer at Sofia Tech Park, Bulgaria; and Deucalion at Minho Advanced Computing Centre, Portugal, under project IDs EHPC-DEV-2025D04 134, SHPC--BEN-2225066-00 EHPC-DEV. 2025D09-044 and EHPC-DEV.2025509-052 through EuroHPC Development and Benchmark Access calls.
REFERENCES
Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. Fedrolex: Model-heterogeeneous federated learning with rolling sub-model extraction. NeurIPS, 2022.
Mohammed Aljahdali, Ahmed M Abdelmoniem, Marco Canini, and Samuel Horváth. Flashback:
Understanding and mitigating forgetting in federated learning. arXiv preprint 2005582402.05558
2024.
Sheeraz A Alvi, Yi Hong, and Salman Durrani. Federated learning cost disparity for iot devices, In ICC Workshops, pp.
IEEE, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pp. 2425-2433, 2015.
Haoping Bai, Shancong Mou, Tatiana Likhomanenko. Ramazan Gokberk Cinbis, Oncel Tuzel, Ping Huang. Jiulong Shan, Jianjun Shi, and Meng Cao. Vision datasets: A benchmark for vision-based industrial inspection. arXiv preprint arXiv:2306.07800 0769098000 2023.

---

Published as a conference paper at ICLR 2026
Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In NeurIPS, 2024.
Mislav Balunovic, Dimitar Dimitrov, Nikola Jovanović, and Martin Vechev. Lamp: Extracting text from gradients with language model priors. Advances in Neural Information Processing Systems, 35:7641-7654, 2022.
Alberto Bietti, Chen Yu Wei, Miroslav Dudik, John Langford, and Steven Wu. Personalization improves privacy-accuracy tradeoff' in federated learning. In International Conference on Machine Learning, pp. 1945-1962. PMLR, 2022.
Kasra Borazjani, Payam Abdisarabshali, Naji Khosravan, and Seyyedali Hosseinalipour. Redefining non-iid data in federated learning for computer vision tasks: Migrating from labels to embeddings for task-specific data distributions. arXiv preprint arXiv:2503. 14553, 2025.
Markus Ralf Bujotzek, Unal Akünal, Stefan Denner, Peter Neher, Maximilian Zenk, Eric Frodl, Astha Jaiswal, Moon Kim, Nicolai R Krekiehn, Manuel Nickel, et al. Real-world federated learning in radiology: hurdles to overcome and benefits to gain. Journal of the American Medical Informatics Association, 2025.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024.
Haokun Chen, Yao Zhang, Denis Krompass, Jindong Gu, and Volker Tresp. Feddat: An approach for foundation model finetuning in multi-modal heterogeneous federated learning. In AAAI, volume 38, pp. 1 1185-11223,, 2024a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pp. 24185-24198, 2024b.
Hsiang- Yun Sherry Chien, Javier Turek, Nicole Beckage, Vy Vo, Christopher Honey, and Ted Willke. Slower is better: revisiting the forgetting mechanism in Istm for slower information decay, arXiv preprint arXiv:2 1eee21155505954 2021.
Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heterogeneous ensemble knowledge transfer for training large models in federated learning. IJCAI, 2022.
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous lora for federated fine-tuning of on-device foundation models. In EMNLP, pp. 12903-12913, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110 14168, 2021.
Liam Collins, Shanshan Wu, Sewoong Oh, and Khe Chai Sim. Profit: Benchmarking personalization and robustness trade-off in federated prompt tuning. In NeurlPSW, 2023.
Tijl De Bie, Nello Cristianini, and Roman Rosipal Eigenproblems in Pattern Recognition, pp. 129-167. Springer Berlin Heidelberg, Berlin, Heidelberg, 2005. ISBN 978-3-540-28247-L. doi: 10.1007/3-540- 28240-2222255555 URL /
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, pp. 141-142, 2012.
Chuntao Ding, Xu Cao, Jianhang Xie, Linlin Fan, Shangguang Wang, and Zhichao Lu. Lora-c: Aarameter-efficieee tin-ttnnig of robust cnn for iot devices. arXiv preprint arX1v:2410.16954, 2024.
12

---

Published as a conference paper at ICLR 2026
Nathaniel J Evans, Gordon B Mills, Guanming Wu, Xubo Song, and Shannon Mc McWeeney. Data valuation with gradient similarity. ArXiv, pp. arXiv 2405, 2024.
Boyu Fan, Chenrui Wu, Xiang Su, and Pan Hui. Fedtsa: A cluster-based two-stage aggregation method for model-heterogennous federated learning. In ECCV. Springer, 2024.
Tao Fan, Guoqiang Ma, Yan Kang, Hanlin Gu, Yuanfeng Song, Lixin Fan, Kai Chen, and Qiang Yang. Fedmkt: Federated mutual knowledge transfer for large and small language models. In COLING, 2025.
Xiuwen Fang, Mang Ye, and Xiyuan Yang. Robust heterogeneous federated learning under data corruption. In ICCV, 2023.
Tiantian Feng, Digbalay Bose, Tuo Zhang, Rajat Hebbar, Anil Ramakrishna, Rahul Gupta, Mi Zhang, Salman Avestimehr, and Shrikanth Narayanan. Fedmultimodal: A benchmark for multimodal federated learning. In Proceedings of the 29th ACM SIGKDD conference knowledge discovery and data mining, pp. 4035-4045, 2023.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. -gdel-agmostic meta-learning for fast adaptation of deep networks. ICML, pp. 1126-1133 PMLR, 2017.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprini urXiv:2306. 2tt.:23366.33394 13394, 2023a.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning dimensions of human visual similarity using synthetic data. arXiv preprint urXiv:2306.0995 09344,, 2023b.
Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mitamura. A dataset and baselines for visual question answering on art. In ECCV Workshops, pp. 12--08..0 Springer, 2020.
Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. In ICLR, 2024.
Pouya M. Ghari and Yanning Shen. Personalized federated learning with mixture of models for adaptive prediction and model fine-tuning. In NeurlPS, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pp. 6904-6913, 2017.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, er The llama herd of models. arXiv preprint arXiv:2407.21788 2024.
Pengxin Guo, Shuang Zeng, Yanran Wang. Huijie Fan, Feifei Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation in federated learning. In ICLR, 2025.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pp. 3608-3617, 2018.
Naresh Kumar Gurulingan, Elahe Arani, and Bahram Zonooz. Curbing task interference using representation similarity- multi-task feature sharing. CoLLAs, pp. 937-951. PMLR, 2022.
Xintong Han, Zuxuan Wu, Phoenix Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Automatic spatially- aware fashion concept discovery. In ICCV, pp. 1463-1471, 2017.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Kerg-Kiikpatrick, and Graham Neubig. Towards a unified view of parfmeter-efffiient transfer learning. In ICLR, 2022.
13

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.000 00t::2000000300,, 2020.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Kai Hu, Sheng Gong, Qi Zhang, Chaowen Seng, Min Xia, and Shanshan Jiang. An overview of implementing security and privacy in federated learning. Artificial Intelligence Review, 57(8):204, 2024.
Wei Hu, Lechao Xiao, and Jeffrey Pennington. Provable benefit of orthogonal initialization in optimizing deep linear networks. In ICIR, 2020.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real- rorl-world visual reasoning and compositional question answering. In CVPR, pp. 7700-6709, 2019.
Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Cihang Xie, and Yuyin Zhou. HQ-edit: A high-quality dataset for instruction-bassee image editing. In ICLR, 2025. URL
https::/gppnreviiee
net/ htt.::eteooooooooodddtmtttttt
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Phillip Isola, Joseph Lim, and Edward H Adelson. Discovering states and transformations in image collections. In CVPR, pp. 1383-1391, 2015.
Mohit lyyer, Varun Manjunatha, Anupam Guha, Yogarshi Vyas, Jordan Boyd-Grabe Hal Daume, and Larry S Davis. The amazing mysteries of the gutter: Drawing inferences between panels in comic book narratives. In CVPR, pp. 7186-7195, 2017.
Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In ICLR, 2022.
Divyansh Jhunjhunwala, Shiqiang Wang. and Gauri Joshi. Fedfisher: Leveraging fisher information for one-shot federated learning. In AISTA7S, pp. 1612-1620. PMLR, 2024.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved image instruction tuning. TMLR, 2024. ISSN 2835-8856. URL https
//openrevi ew net./ forum? dtw.ktt
Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard -oi: Benchmarking few-shot visual reasoning for human-objes interactions. In CVPR, pp. 19056- 19065, 2022.
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renor- malizing permuted activations for interpolation repair. In ICLR, 2023.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualiza- tions via question answering. In CVPR, pp. 8664-5656, 2018.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In CVPR, pp. 4999-5007, 2017.
Hyunseo Koh, Dahyun Kim, Jung- Woo Ha, and Jonghyun Choi. Online continual learning class incremental blurry task configuration with anytime inference. In ICLR, 2022.
Hyunseo Koh, Minhyuk Sco, Jihwan Bang. Hwanjun Song, Deokki Hong, Seulki Park, Jung-War Woo Ha, and Jonghyun Choi. Online boundary-free continual learning by scheduled data prior. In ICLR, 2023.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICMI., pp. 3519-3529. PMLR, 2019.
14

---

Published as a conference paper at ICLR 2026
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image retrieval from contextual descriptions. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), ACL, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federateessope-lllm: A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 5260-526715 2024.
Allison Lau, Younwoo Choi, Vahid Balazadeh, Keertana Chidambaram, Vasilis Syrgkanis, and Rahul Krishnan. Personalized adaptation via in-context preference learning. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning. 2024.
Jaewoo Lee, Boyang Li, and Sung Ju Hwang. Concept-skill transferability-basedd data selection for large vision-language models. In EMNLP, 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Beed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Bohao Li, Yuying Ge, Yixiao Ge. Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan.
Seed-bench: Benchmarking multimodal large language models. In CVPR, pp. 13299-13308, 2024a.
Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv 2019.
preprint
Hongyang Li, Caesar Wu, Mohammed Chadli, Said Mammar, and Pascal Bouvry. Trustworthiness of
stochastic gradient descent in distributed learning. arXiv preprint arXiv:2410.2149)) 2024b.
Juncheng Li, Kaihang Pan, Zhiqi Ge. Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal LLMs to follow zero-shot demonstrative instructions. In ICLR, 2024c. URL /ttpss://ppeerereww..ee
id= 7c31
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal LLMs to follow zero-shot demonstrative instructions. In ICLR, 2024d.
Lin Li, Jianping Gou, Baosheng Yu, Lan Du, and Zhang Yiand Dacheng Tao. Federated distillation: A survey. arXiv preprint 24Xx::2404.08564, 2024e.
Sheng Li, Geng Yuan, Yue Dai, Youtao Zhang, Yanzhi Wang, and Xulong Tang. SmartFRZ: An efficient training framework using attention based layer freezing. In ICLR, 2023b. URL
https://openrevia
net wtt./foorouuiioii 1t1.ii----191A11111
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In ICML, pp. 6357-6368. PMLR, 2021.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023c.
Zexi Li, Tao Lin, Xinyi Shang, and Chao Wu. Revisiting weighted aggregation in federated learning with neural networks. In ICML, pp. 19767-19788. PMLR, 2023d.
Chin Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Yang Lin, Xinyu Ma, Xu Chu, Yujie Jin, Zhibang Yang, Yasha Wang, and Hong Mei. Lora dropout as a sparsity regularizer for overfitting control. CoRR, 2024.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 1::65-651,, 2023a.
15

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36:34892 -4991,, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, pp. 216-233. Springer, 2024.
Zheyuan Liu, Cristian Rodriiguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on 2021.
real-life images with pre-trained vision-and-language models. In ICCV, pp.
Guodong Long. Tao Shen, Jing Jiang, Michael Blumenstein, et al. Dual Durl-peersn adapter for federated foundation models. Advances Neural Information Processing Systems, 37:39409- 39433, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chul Zhu. Iconqa: new benchmark for abstract diagram understanding and visual language reasoning. In NeurlPS 2021, 2021.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurlPS, 35:2507-2521, 2022.
Chun Zhu, Oyvind Tafjord,
Yuxiang Lu, Suizhi Huang. Yuwen Yang, Shalayiding Sirejiding, Yue Ding, and Hongtao Lu. Fedhca2: Towards hetero-elient federated multi-tast learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5599-5609, 2024.
Chenxiang Luo, David KY Yau, and Qun Song. Svdefense: Effective defense against gradient inversion attacks via singular value decomposition. arXiv preprint arXiv:2510.033/9. 2025.
Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and Philip S Yu. Privacy and robustness in federated learning: Attacks and defenses. IEEE transactions neural networks and learning systems, 2022.
Hussain Ahmad Madni, Rao Muhammad Umer, and Gian Luca Foresti. Exploiting data diversity in multi-domain federated learning. Machine Learning: Science and Technology, 5(2):025041, 2024.
Shivangi Mahto, Vy Ai Vo, Javier Turek, and Alexander Huth. Multi-timescale representation learning in Istm language models. In ICLR, 2021.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. NeurlPS, 2023.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume of Psychology of Learning and Motivation, pp. 109-165. Academic Press, 1989.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-eficee learning of deep networks from decentralized data. In AISTATS. PMLR, 2017.
Fan Mo, Anastasia Borovykh, Mohammad Malekzadeh, Soteris Demetriou, Deniz Gündüz, and Hamed Haddadi. Quantifying and localizing usable information leakage from neural network gradients. arXiv preprint arXiv:2 13929, 2021.
Mahdi Morafah, Vyacheslav Kungurtsev, Hojin Matthew Chang, Chen Chen, and Bill Lin. Towards diverse device heterogeneous federated learning via task arithmetic knowledge integration. In NeurIPS, 2024.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning (bert): Misconceptions, explanations, and strong baselines. In ICLR, 2021.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya Multi-task learning bargaining game. In ICML., pp. 16428-16446. PMLR, 2022.
16

---

Published as a conference paper at ICLR 2026
Behnam Neyshabur, Hamie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 2020.
Aleksandra Nowak, Lukasz Gniecki, Filip Szatkowski, and Jacek Tabor. Sparser, better, deeper, stronger: Improving static sparse training with exact orthogonal initialization. In ICML. PMLR, 2024.
Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. Federated learning with partial model personalization. In ICML, pp. 17716-17758, PMLR, 2022.
Ameya Prabhu, Zhipeng Cai, Puneet Dokania, Philip Torr, Vladlen Koltun, and Ozan Sener. Online continual learning without the storage constraint. arXiv preprint 0at.233550095533 2023.
Daiqing Qi Handong Zhao, and Sheng Li, Better generative replay for continual federated learning. In ICLR, 2023.
Jiaxing Qi. Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, and Depei Qian. Fdlora: personalized federated learning of large language model via dual lora tuning. arXiv preprint MrXiv:2406.07925 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140)1-67, 2020. URL http: http:/mmrr.rrroo http://mm..oo//pppa///////211111
html
Shuzhen Rao, Jun Huang, and Zengming Tang. Adaptive regularized warped gradient descent enhances model generalization and meta-leaming for few-shot learning. Neurocomputing, 537: 271-281, 2023. ISSN 0925-2312.
Roger RatclifT. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990.
Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. NeurIPS, 28, 2015.
Jonathan Scott, Hossein Zakerinia, and Christoph H Lampert. Pefll: Personalized federated learning by learning to learn. In ICLR, 2024.
Minhyuk Seo, Hyunseo Koh, Wonje Jeung, Minjae Lee, San Kim, Hankook Lee, Sungjun Cho, Sungik Choi, Hyunwoo Kim, and Jonghyun Choi. Learning equi-angular representations for online continual learning. In CVPR. pp. 23933-23942, 2024.
Minhyuk Seo, Hyunseo Koh, and Jonghyun Choi. Budgeted online continual learning by adaptive layer freezing and frequency- -bssd sampling. In ICLR, 2025.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pp. 8317-8326, 2019.
Shivalika Singh, Freddic Vargus, Daniel D'souza, Börje Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O' O'Mahony, al. Aya dataset: An open-access collection for multilingual instruction tuning. In ACL, 2024.
Virginia Smith, Chao-Kai Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. NeurIPS, 30, 2017.
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with SVD to tie the knots. In ICLR, 2025.
Alane Suhr. Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), ACL, pp. 6418-6433 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 18.156 10//33/11/119114// URL https: / aaaaaaaann aclanthology
arq/P19-11444
17

---

Published as a conference paper at ICLR 2026
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in learning. ICLR, 2024.
federated
Rishub Tamirisa, Chulin Xie, Wenxuan Bao, Andy Zhou, Ron Arel, and Aviv Shamsian. Fedselect: Personalized federated learning with customized selection parameters for fine-tuning, In CVPR, 2024.
Jiahao Tan, Yipeng Zhou, Gang Liu, Jessie Hui Wang, and Shui Yu. pfedsim: Similarit- model aggregation towards personalized federated learning. arXiv preprint arXiv:2305. /5706, 2023.
Shuai Tang, Wesley J Maddox, Charlie Dickens, Tom Diethe, and Andreas Damianou. Similarity of neural networks with gradients. arXiv preprini arXiv:2003. 1498, 2020.
Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.055300 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019. URL [RKL/ttts////eeezeeeee
Jiaqi Wang, Xingyi Yang, Suhan Cui, Liwei Che, Lingjuan Lyu, Dongkuan DK Xu, and Fenglong Ma. Towards personalized federated learning via heterogeneous model reassembly. Advances in Neural Information Processing Systems, 36:29515 2023.
Penghao Wang, Qian Chen, Teng Zhang, Yingwei Zhang, Wang Lu, and Yiqiang Chen. Fhbench: Towards efficient and personalized federated learning for multimodal healthcare. arXiv preprint arXiv:2504. 10817, 2025.
Shaowen Wang. Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. NeurlPS, 37:54905-5444 54931, 2024.
Yongxian Wei, Anke Tang, Li Shen, Chun Yuan, and Xiaochun Cao. Modeling multi-task model merging as adaptive projective gradient descent. arXiv preprint arXiv:2501.0/2.00, 2025.
Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Yutong Wang, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afif Amriani, et al. Worldcuisines: A massive-scale benchmark for multilingual and multicultural visual question answering on global cuisines. arXiv preprint arXiv:2410. 12705, 2024.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML, pp. 23965-23998 PMLR, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, 2022b.
Stanistaw Wožniak, Bartlomiej Koptyra, Arkadiusz Janz, Przemyslaw Kazienko, and Jan Kocoñ. Personalized large language models. arXiv preprint arXiv:2402. 09269, 2024.
Feijie Wu, Xingchen Wang, Yaqing Wang, Tianci Liu, Lu Su, and Jing Gao. Fiarse: Model- heterogeneous federated learning via importance-awee aware submodel extraction. In NeurlPS, 2024a.
Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang. Wenxiu Sun, Qiong Yan, al. Towards open-ended visual quality comparison. In ECCV, pp. 360-377. Springer, 2024b.
Hui Wu. Yupeng Gao, Xiaoxiao Guo. Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. Fashion iq: A new dataset towards retrieving images by natural language feedback. In CVPR, pp. 11307-11317, 2021.
18

---

Published as a conference paper at ICLR 2026
Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A Rossi, Franck Demoncourt, et al. Personalized multimodal large language models: A survey. arXiv preprint 4:X1v:2412.02142, 2024c.
Rujie Wu, Xiaojian Ma, Zhenliang Zhang, Wei Wang, Qing Li, Song-Chur Chun Zhu, and Yizhou Wang. Bongard-openworld: Few-shot reasoning for free-form visual concepts in the real world. In ICLR. net/ /ere/foorrmt forum? dd=hss4Meey2e
2024d. URL htttps::/opperee////
Abudukelimu Wuerkaixi, Sen Cui, Jingfeng Zhang, Kunda Yan, Bo Han, Gang Niu, Lei Fang, Changshui Zhang, and Masashi Sugiyama. Accurate forgetting for heterogeneous federated continual learning. In ICLR, 2024.
Chulin Xie, De-An Huang, Wenda Chu, Daguang Xu, Chaowei Xiao, Bo Li, and Anima Anandkumar. Perada: Parameter-efffcient federated learning personalization with generalization guarantees. In CVPR, 2024.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706 2019.
Binqian Xu, Xiangbo Shu, Haiyang Mei, Guosen Xie, Basura Fernando, and Jinhui Tang. Fedmilm: Federated fine-tuning milm on multimodal heterogeneity data. arXiv preprint arXiv:2411.14717, 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. NewrIPS, 2023.
Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii (eds.), EMNLP. Association for Computational Linguistics, October-November 2018.
An Yang. Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1511v:2412.15115, 2024a.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, milms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.0766 2024b.
Qiang Yang, Anbu Huang, Lixin Fan, Chee Seng Chan, Jian Han Lim, Kam Woh Ng, Ding Sheng Ong, and Bowen Li. Federated learning with privacy-preserving and model ip- right-protections Machine Intelligence Research, 20(1):19-37, 2023. ISSN 2731-538X. doi:
10.10077/111333-02- 1343-2. URL https: www.://ww....reerr.......... doi/10 1ttp:/000000//1111111 02222221633302--11---112222222 022-1333------------- 1343-2
mi research htw.:/aaccc..eeeeeeaatat.......e
Dixi Yao. Exploring system- heterogeooosss federated learning with dynamic model selection. arXiv preprint wxXiv:240.008858 2024.
Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Du Yaxin, Yang Liu, Yanfeng Wang, and Siheng Chen. Fedllm-bench: Realistic benchmarks for federated learning of large language models. NeurIPS, 2024.
Shih- Shih-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning model evaluation. In The Twelfih International Conference on Learning Representations, 2023.
Liping Yi, Gang Wang, Xiaoguang Liu, Zhuan Shi, and Han Yu. Fedgh: Heterogeneous federated learning with generalized global header. In ACM Multimedia, pp. 8686-8696, 2023.
Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In ICML, pp. 12073-12086. PMLR, 2021.
19

---

Published as a conference paper at ICLR 2026
Ron Yosef, Yonatan Bitton, and Dafna Shahaf. IRFL: Image recognition of figurative language. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), EMNLP. pp. 1044-1058, Singapore, December 2023. Association for Computational Linguisties. doi: 10. 10.18653/w1/223, https: /tttps://aclaae antho ogy. wwwp://.gg222.... f
http://w/222...nnnnneemmme/miee findi ndi ngs emn 1. pp.
URL
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917. 2022.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.024444 2023.
Betul Yurdem, Murat Kuzlu, Mehmet Kemal Gullu, Ferhat Ozgur Catak, and Maliha Tabassum. Federated learning: Overview, strategies, applications, tools and future directions. Heliyon, 2024.
Jianyi Zhang, Saced Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. ICASSP. IEEE, 2024a.
Jiarui Zhang. Guided profile generation improves personalization with 1lms. In ACL, 2024.
Kai Zhang. Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems, 36:31428-31422 2023a.
Kaiyuan Zhang. Siyuan Cheng, Guangyu Shen, Bruno Ribeiro, Shengwei An, Pin- Yu Chen, Xiangyu Zhang, and Ninghui Li. Censor: Defense against gradient inversion via orthogonal subspace bayesian sampling. In NDSS, 2025a.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In ICLR, 2021.
Yuanhe Zhang, Fanghui Liu, and Yudong Chen. Lora-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently. In ICML, 2025b. URL https
/oopenrvieww.oo net www.::.nwtt....m...t.. Kwd=KwI 1
Yukun Zhang, Guanzhong Chen, Zenglin Xu, Jianyong Wang, Dun Zeng, Junfan Li, Jinghua Wang, Yuan Qi, and Irwin King. Fedovd: The first worldworld federated learning benchmark on cardiovascular disease data. arXiv preprint arXiv:2411.07055 2024b.
Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, et al. Personalization of large language models: A survey. arXiv preprint arXiv:2411.0002 2024c.
Zhuo Zhang, Xiangjing Hu, Jingyuan Zhang, Yating Zhang, Hui Wang, Lizhen Qu, and Zenglin Xu. Fedlegal: The first real-world federated learning benchmark for legal nlp. In ACL, pp. 3492-3507, 2023b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging Ilm-as-a aan-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-46623, 2023.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. NeurlPS, 32, 2019.
20

---

Published as a conference paper at ICLR 2026
A TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL
This document provides a comprehensive overview of the technical appendices
Theoretical Foundations
Section A.1; T'eoretical justification for freezing matrices A and B in Co-LoRA, proving that this approach achieves zero aggregation error in federated learning.
Section :.2; Proof Theorem ! showing that orthogonal initialization and B maximizes the representational capacity of Co-LoRA.
Implementation Details
Section
A.3
Details client model selection methodology in heterogeneous PFL scenarios.
Section A.4
Consistency of FedMosaic over diverse random heterogeneous PFL scenarios.
In-depth explanation of Co-LoRA, including initialization with orthogonal sets, weight alignment, and orthogonality -enforiiign post- post-processing.
Section A.5
Section A.6;
Comprehensive description of experimental configurations, including model architecture details, benchmark specifications, evaluation metrics, and hyperparameter settings.
Section A.7;
Summary of client model configuration all heterogeneous PFL experiments.
Section A.8;
Comparative analysis of memory and computational costs across various
baselines.
Section A.9
Comparative analysis communication (transmission) cost of FedMosaic.
Section A.10
Empirical analysis supporting block-wise aggregation through CKA similarity measurements across heterogeneous models.
Justification and empirical analysis for the choice of pivot model during Co-LoRA alignment.
Section A.29
Detailed algorithms for the FedMosaic framework, including initialization, alignment, and training procedures.
Section A.35
Extended Experimental Results
Results with more diverse heterogeneous PFL scenarios, including experi- ments with three different model architectures.
Section A.11
Section A.12
Experimental results in homogeneous PFL setups.
Section A.13
Experimental results heterogeneous PFL scenarios using Qwen-based
LLaVA models.
Section A.14
Additional experiments various cross-family heterogeneous PFL. setups.
Section A.15 Results on the text-only benchmark, complementing our Led-L.LM-L.arg experiments.
Section A.16:
Detailed client-wise accuracy analysis for both homogeneous and heteroge- neous PFL setups.
Section A.23:
Extended evaluation of fast adaptation capabilities on additional unseen tasks.
Section A224
Experimental results in LoRA rank heterogeneous PFL scenarios.
Section A.25
Experimental results in task-wise non-i.i.d. heterogeneous PFL scenarios.
Section A.26
Experimental results in heterogeneous PFL setups using small T5 models.
Experimental results in heterogeneous PFL-Dynamic setups with asyn- chronous data distribution-shif among clients.
Section A.32
21

---

Published as a conference paper at ICLR 2026
Ablation Studies
Section Section/..17- Analysis of the choice (i.e., type and size) of public data Dp for Co-LoRA alignment.
Section A.18
Analysis of how the change of various hyperparameters introduced in
FedMosaic affects performance.
Section A.11: Ablation study on RELA, comparing different strategies for measuring task similarity.
Section A.20
Analysis on the size of W, for RELA in terms of computational cost and performance in PFL setup.
Section A.2T:
Investigation of the effects of weight alignment in Co-LoRA
Discussion of RELA and existing similarity- bbasdd model aggregation in federated learning and multi-task learning.
Section A.22
Section A.30:
Privacy analysis of the sanitized gradient of RELA under gradient inversion
attack.
Section Visualization task relevance matrix of DRAKE.
Limitations/Future Work and Benchmark Details
Comprehensive description of our proposed DRAKE benchmark, including task categories, dataset sources, client configuration, and comparison with existing FL. benchmarks.
Section A.27;
Section A.28
Comparison between DRAKE and existing FL benchmarks.
Section A.33
Discussion of limitations and potential directions for future work.
Section
A.34
Impact statement addressing the broader implications of our research.
A.1
THEORETICAL JUSTIFICATION FOR FREEZING AND IN Co-LORA
Recap of Co-LoRA. Co-LoRA consists of LoRA matrices A E Igrxd,, B E xr along with shareable modules E IR Rxrr Q E R". Co-LoRA outputs ho for the given input hy as:
ho = W,hs + B(PAh, + Q).
(8)
where Wp refers to the pre- pre-trained weight.
For simplicity, assume that Q 0 IR', i.e., ho BPAh, and prove under the homogeneous Fl. setup, but can easily extend to Q o O and the heterogeneous FL. setup.
Definition (Aggregation Error 8). We first define the aggregation error 8. In FL setup with N clients, the server aggregates client updates AW; P.P.A for i E [N]. With FedAvg aggregation, the 'ideal' aggregation AW* (Sun et al. 2024 Guo et al. 2025) should be:
AW
N (BIPIAI + B,P24, +
By BN
However, cannot perform 'ideal' aggregation in FL, since W cannot be decomposed into trainable parameters, i.e., B, P, and A, on client sides (Guo et al.) 2025). In other words, although W can be computed the server, it cannot be redistributed as trainable parameters the client side. As result, instead of directly averaging AW, for E [N]. we average the trainable parameters separately and obtain the practical update A as:
AW"
Σ
B
P
A
(10)
To this end, we define the aggregation error 8 AW shareable modules P and Q. while freezing A and B to preserve aligument. This way, we minimize the aggregation error 6, following Theorem 2
AW" During local training, update
Theorem 2. If we freeze A &Ry and B E R aggregation error 5 0.
22

---

Published as a conference paper at ICLR 2026
Proof of Theorem 2
Proof. If A is frozen after alignment, then Ai A AN. Similarly, freezing B
implies B = B1 = B2 = Bv. Under this condition, both the ideal update AW* (Eq. 9 and the practical update AW* (Eq. 10 can be simplified to:
AW* AW B
Σ P A.
(11)
iml
Thus, 8
A.2 PROOF OF THEOREM
Proof. For simplicity, we assume that Q = R"
The weight update for Co-LoRA is given by:
AW BPA,
(12)
where B E Ro Pe A grxd1
We can decompose the matrix P using scalar values and basis vectors as follows:
p=
ΣΣ Poj e,ee.
(13)
i=ij=1
where P. refers the scalar value (i,j) position of matrix and e; and ej refers to the standard basis vector in IR".
Substituting Eq 13 into Eq 12 get:
AW BPA
Σ 1.
Pis c,e
P. (Be,) (c)
(14)
Since Be; = b; E Rdo (fn column of B), of A a.. E Rddr (in row of A), we can simplify Eq. 14
as:
AW
Σ i.j
Pu b,a..
(15)
Therefore, representation space of AW, denoted SpQ. is defined as:
Spq = bi,n i.j = I.....r..
(16)
Since both {b). .g...., b,} c de and {af ah..... a.) c gd form orthogonal sets, their outer products [b:a Mi are linearly independent.
Therefore, the representation space SpQ has the maximum possible dimension:
dim(Srq) 22
(17)
To satisfy Eq 17
is sufficient for the column vectors of and the row vectors A to form linearly independent sets, even if they are not strictly orthogonal. However, compared to initialization with linearly independent vectors, orthogonal initialization offers additional advantages, such as faster (Hu al. 2020) and stable (Nowak et al. 2024) convergence. Therefore, motivated by both the practical advantages and the theoretical justification provided in Theorem I we initialize and B with orthogonal sets.
23

---

Published as a conference paper at ICLR 2026
A.3 DETAILS OF CLIENT MODEL SELECTION IN THE HETEROGENEOUS PFL SCENARIO
In the heterogeneous PFL scenarios, we assign each client's model based on supervised fine-tuning (SFT) performance on their respective tasks. Specifically, clients are assigned larger models when these models exhibit significantly better SFT performance than smaller ones. Conversely, when the performance difference between model sizes is negligible, we allocate smaller models maximize computational efficiency, as larger models require substantially more computational resources. We summarize the SFT performance of each LLaVA VA-mmama3 variant model and the resulting client-wise model allocation in Tab. g
DRAKE
Model
Client
Client
Client
Client
Client
Client
Client
Client
Client
Client
LLaVA- L.VA-Llama3.-11
78.43
57.98
65.05
62.53
63.15
63.35
47.56
51.68
$3.07
50.95
LLaVA Llamm3.2-38
79.33
59.01
66.75
63.33
66.00
76.15
63.40
68.21
63.08
57.32
Allocated Model
LLaVA-Llamas L----lamm3222 3-2 1B
LA LLaVV-LlamssS 2-3B
HFLB
Model
Client
Client 2
Client 3
Client 4
Client
Client 6
Client 7
Client 8
Client
LLaVA-Llama3.2 2-1B
79.78
98.18
51.23
80.73
71.58
80.48
71.64 74.47
76.45
89.45
LLaVA-LJama3.2-BB
80.33
97.32
52.85
81.85
73.84
82.75
79.50
91.50
Allocated Model
LLaVA- 2-11B
LLaVA-Llama3.
Table 8: Per-client SFT performance of different L.La VA-V model variants. We assign the larger model (i.e., LLaVA-LLamaa-3.2-333 to clients whose SFT performance shows substantial gain, while the smaller model (i.e., L.LaVA-Llam-33.2-- B3 is assigned otherwise.
A.4 EFFECT OF CLIENT MODEL ASSIGNMENT
To validate whether FedMosaic consistently achieves strong performance regardless of the specific model assigned to each client, we conduct experiments with varying ratios of LLaVA- 1B and LLaVA- 3B models, where model assignments are randomly determined in each run. Table g we summarize the PFI. results of supervised fine-tuning (SFT) and FedMosaic across diverse heterogeneous PFI. scenarios, each corresponding to a different configuration of client model selections in the DRAKE- dynamic benchmark. We observe that FedMosaic consistently outperforms SFT with minimal performance variance across heterogeneous setups, demonstrating its robustness to the composition of client models.
Self
Others
Scenario
Method
Atast
AAUC
Atsst
Aauc
SFT
65.57+0088
0..46
47.80+0.117
46.80-t0.1
3x Clients 7x3B Clients
FedMosaic
67.0240.34
58.85+00.15
51.55=0.31
49.62=0.04
(FedMosaic- SFT)
+1.45
+1.39
+3.75
+2.82
SFT
64.08+0.77
56.61 +6.42
46.82+00.44
46.24-10.23
4x Clients 6x3B Clients
FedMosaic
66.28:10.23
58.4710.177
51.20::0.40
49.26:10.29
A (FedMosaic- SFT)
+2.20
+1.86
+4.38
+3.02
SFT
63.72-10.77
56.61:10116
46. 46.70-0.07
45.99-10.06
5x1B Clients 5x3B Clients
FedMosaic
65.34:10.21
58.25:10.17
51.08
49.07:1.25
A (FedMosaic- SFT)
+1.62
+1.64
+4.38
+3.08
SFT
63.49-+0.93
56.02+0.28
46.63+0.07
40.63+0.02
6x Clients 4x3B Clients
FedMosaic
65.75+0.55
57.9611018
50.57=0.23
48.65-00118
(FedMosaic- SFT)
+2.26
+1.94
+3,94
+3.02
SFT
62.72-+0.19
55.54+0.02
45.54+0,22
44.83+0.07
7x1B Clients 3x3B Clients
FedMosaic
65.40+0.53
57.56+1000
49.41=0.18
47.54-0.08
(FedMosaic- SFT)
+2.68
+2.02
+3.87
+2.71
Table 9: Effect of heterogeneous PFL scenarios on DRAAKE-Dyname We change the number of VA-L-A-a-...... and LLaVA-Llama3. 2-3 models and randomly assign the model size of each client. The 64x IB Clients/ 6x Clients' scenario here is different with the client model selection
in Tab. g
24

---

Published as a conference paper at ICLR 2026
A.5 MORE DETAILS OF Co-LORA
As discussed in Sec 4.2.2 we enforce orthogonality the A and B matrices to maximize the capacity of Co-LoRA, following Theoren[] This achieved through (i) initializing and B with orthogonal sets, (ii) weight alignment, and (iii aattogonnlity-enfooriii post-processing.
Initialization with Orthogonal Set. of B E Rdo kr with an orthogonal set, as follows:
We initialize row vectors A E grxdr and column vectors
AA Lrs
BBB=L.
(18)
where I, refers to the x r identity matrix.
Weight Alignment. For B, and By, align them using canonical correlation analysis (CCA), as follows:
(1, -II)) Bi.
(19)
where II; and I are projection matrices that project B; and B3 into to the maximally correlated space, respectively. During alignment, Bj remains orthogonal because B₁ is initialized with orthogonal vectors, and the projection in [9 preserves this orthogonality.
However, during the alignment of A; and A; in Sec. 4.2.2 orthogonality be disrupted due L2 loss training. To prevent significant deviation from orthogonality, we add a regularization term to the objective function of A alignment as follows:
min A
Σ
((A())) | I1
(20)
D
(R.N)ED,
where || || denotes the Frobenius norm. Despite the regularization, A, may not strictly satisfy orthogonality. Therefore, to ensure exact orthogonality, we perform an additional post-processing step.
Orthogonality-Ennorrii Post-processing To enforce orthogonality in Aj. we apply orthogonal projection. Specifically, we aim to find the closest orthogonal matrix Aj from A, by minimizing the Frobenius norm, as follows:
min A A
|4 - -,
(21)
This optimization can be efficiently solved using the Singular Value Decomposition (SVD). We compute the SVD of A:
A, =EV
(22)
where U RR"x, E Rexd, and V Rdixd,
The closest orthogonal matrix Aj is then given by:
A =
(23)
As result, we get A, and B, initialized with orthogonal sets and kept frozen during alignment, and A3 and Bj, which are aligned with A, and B, while maintaining orthogonality.
A.6 DETAILS OF EXPERIMENTAL SETUPS
Models. To employ LLa variant models (i.e, LLaVA--Laam3.2--11 LLaVA-Lama3 - 3B, LLaVA-Llama3, 1-8)) and LLaVA-Qwen2.5 variant models (i.e., LLaVA-QWen2.5-00555 LLaVA- Qwen2.5-1.55 and LLaVA ILLaVA-Qeen.... Qwen2.5-3B in heterogeneous PFL scenarios, we instruction- them on 158K (Liu et al.) 2023b), following the original LLaVA training setup (Liu et al.) 20236). We summarize the performance of the instruction-tuum tunded variants in Tab. an Tab. 1 base models for text-only benchmarks.
25

---

Published as a conference paper at ICLR 2026
VOAv2
GOA Hudson & Manning 2019)
VizWiz (Gurari ct
ScIOA (Lu at. 2022
TextVOA Singh ml. 2019
Model
Goyal al. 2017
2018
LLaVA-1.5-BB
78.5
62.0
50.0
66.8
58.2
75.3
59.7
45.3
67.3
49.0
LL Llammmaa33
78.7
62.6
45.3
75.7
55.8
LLAVA- LLAVa-Lamm331-88
79.8
64.2
47.6
76.4
57.9
Va--aen2-nneeeee 5-0.52 5-0.5B
71.8
56.5
37.3
58.1
43.4
LLAVA-Qwenn2 5B
75.3
59.3
40.9
69.5
51.8
MLaVA-Qwen2.5.3
77.2
50.9
Table 10: Zero-Shot performance of instruction-tuned I.L.AA-1.5 variants academic-task- oriented benchmarks. We use the CLIP ViT-L/336pp model as the vision encoder for all variants.
POPE
MME
MMBennc
SEED-Bench
LLaVA-Wild Lin 2023h
Me-Vet
Model
Ji 20236
Fu
2023a
Liu
2024
Lietal
2023a3
Ya
2023
LLaVA-1.5-7B
85.9
1510.7
64.3
58.6
65.4
31.1
LLaVA-Llama3,2-1B
85.0
1338.1
61.3
59.6
LLVVA-Lamm3.2-35 2-338
86.1
1446.5
70.9
62.5
67.0
34.5
ALaaa-L 1-SB
86.3
1486.1
72.5
64.1
73.7
32.7
LLaVA LLaVA-Qwen2. 5.0.5B
86.2
1251.9
53.6
53.1
55.0
23.1
LLavA- Qwen2.5-1.55 1-1.--11....
86.4
1376.4
66.8
60.6
60.0
27.9
1IaVA-Qwen2.5.33
66.7
Table 11: Zero-Shot performance of instruction-tuned VA-1.5 variants instruction-
following benchmarks. We use the CLIP ViT-L/336px model as the vision encoder for all variants.
Baselines. We compare FedMosaic with SOTA PFL methods: DITTO (Li al. 2021) FedSim (Pil- lutla et 2022). FedIT (Zhang al.) 2024a). TAKFL (Morafah al. 2024). FedDPA (Long et
2024), FedDAT (Chen et al. 2002aa. PerAda (iee et al. 2024), and FedMKT (Fan et al. 2025) Federated distillation baselines (i.e., TAKFL,, PerAda, and FedMKT) share knowledge between het- erogencous models via logits. while FedMosaic uses Co-LoRA. For the other baselines, combine them with Fed-ET (Cho et al., 2022). which aggregates models among homogeneous clients. We also compare with supervised tine-tuning (SFT), where clients train independently without knowledge sharing and often outperforming PFL methods under data heterogeneity (Ghari & Shen 2024).
Benchmarks. We evaluate FedMosaic on Multi-modal FL benchmarks, such our proposed DRAKE and HFLB, and I.L.-FI. benchmarks, such as Fed-Scope and Fed-Aya, and Fed-L.LM- Large. We summarize the details of benchmarks in Tab. 12] As shown in the table, unlike HFLB, which focuses solely on VQA tasks, DRAKE covers diverse task types and involves more images per sample due to its multi-image inputs. In DRAKE-Dynamic each client learns four tasks, where all data samples are randomly mixed in PFL-Static, while the samples from each task are introduced incrementally in PFL-Dynamic. We provide more details of DRAKE in Sec. A.27
HFLB (Heterogencous Federated Learning Benchmark) is multi-moda PFL benchmark used in FedDAT (Chen al. 2024a) that includes VQA datasets for heterogeneous tasks
clients, where we assign them into 9 clients by partitioning GQA (Hudson & Manning, 2019) into three clients based on the QA types (e.g., Yes/No, 2 Multi-choice, and 4 Multi-choice). We further split each assigned dataset into four tasks to allow experiments on the PFL-Dynamic setup. We use keywords in questions or GPT categorize each sample, where the details are provided in Tab.
To simulate PFL in NLP, we modify existing sets (i.e., Fed-Aya and Fed-Scope) and compose larger increase task variety and heterogeneity across clients. Specifically,
mixtures (i.e., Feed-LLM-Large Fed-Aya is composed of 12 different languages equally sampled from different language families. We further separate the tasks based on the topic the QA using GPT-4. Fed-Scope (Kuang al., 2024) includes coding, mathematics, and general capability datasets, where each client learns either one of the three datasets. Led-LLM-Large is a large-scale text-only benchmark designed for personalized federated learning. To diversify the personal tasks among clients, we combined tasks from three sources, Tod-Chabbotl' and Fed-aya from Fed-LLM (Ye et al.) 2024). and Fed- Fed-FLAN (Long et al. 2024). This results in 52 clients and 2 tasks per client, where each client fine- tine-tunesse either instruction tollowing tasks from different sources or different languages (note that all other benchmarks have 4 tasks per client).
26

---

Published as a conference paper at ICLR 2026
Renchmark
# Images
I of Questions
# of clients
# rounds
local steps
Tisk types
DRAKE
357K
274K
10
20
100
Visual Relation, Mult-modal Ressoning, VQA
HPLB
217K
314K
9
20
100
VQA
Fed-Scope
30K
General Capabilitt, Matheratics, Coding
Fed-Aya
36K
20
Question-Answern Answering Languages Instruction Following
Ped-Li Large
31K
52
4
10
Table 12: Benehmark details and federated learning configurations.
Client 1
Client 2
Client 3
GQA (Hudson Manning, 2019
GQA (Hudson & Manning 2019
GQA (Hudson & Manning 2019
Yes/No
Four-Chowe
Two-tnorce
Attribute
Attribute
Attribute
Relation
Relation
Relation
Object
Category
Category
Global
Global
Global
Client 4
Client 5
Client 6
Abstract VQA (Antol al. 2015
SNLI-VE (iee al. 2019
COCOQA (Ren et al. 2015
Attribute
Action
Object
Number
Scene Cuntext & Obj Relations
Number
Yes/No
Object-Centric
Color
Others
Commonsense
Location
Client 7
Client 8
Client 9
NLVR2 (Suhr 2019
VizWiz (Gurari et al. 2018
AQUA (Garcia et 2020
Presupposition Negation Universal
Food, Brand, and Label Identification
"standing" keyword
Spatial Comparative
Color and Type Identification
"sit" keyword
Cardinality Existential
Optical Character Recognition
"wear". "hold", "walk", "talk" keywords
Coordination Coreference
General Object Identification
Other keywords
Table 13: Per-client task configuration of HFLB. Datasets are partitioned into question-type subsets using keyword rules or GPT judgement.
Metrics. We measured the accuracy metrics, Atast and AAUC, based on the correct choice for multi- choice questions and the correct tokens compared to the ground-truth answer for the open-ended questions. For Fed-Aya, we used GPT rate the generated response compared to the given ground- truth response. We use the same prompt template shown in Fig. 9 from the original paper (Singh al. 2024).. For Fed-Scope, we follow the evaluation process of the original paper (Kuang et al. 2024 and use MMLU (Hendrycks et al., 2020), GSM8K (Cobbe et al. 2021). and HumanEval (Chen al. 202T) benchmarks to evaluate general capability, math and coding skills, respectively. For Fed- Led-LLLMLLarge. we use Rouge-L (Lin 2004) metric as it is commonly used metric to assess LLM
generation quality compared to ground-truth (Long et al. 2024
et al. 2024c).
Implementation Details and Hyperparameters. We set the batch size to 4, the learning rate for Co-LoRA to 5 x 10 55,5,,m and for other parameters to x 10 15- We use the Constant LR scheduler and the AdamW optimizer (Loshchilov & Hutter, 2019) for all datasets. We use the LoRA rank r of 128 attached to all linear layers in the LLM backbone. For LLM experiments, we set the learning rate to 3 10-4 and the LoRA rank to 16. For federated learning, we set the total number of communication rounds R to 20, with the local training step of 100. For a fair comparison, we adjust the local training step for each baseline and FedMosaic use the same computational cost as SFT, following (Seo et al. 2025). For the PFL-DDnamic setup, we employ memory-only training, where newly encountered samples are added to episodic memory. and training batches are retrieved solely from memory, following (Koh et al., 2023 Seo et al. 2024 2025). For episodic memory, we adopt a memory-infnitte setup (Prabhu et al.) 2023) Seo et al. 2025), assuming all data can be stored in memory, reflecting that memory cost is not a bottleneck in real- scenarios. For т, the softmax temperature parameter used RELA, we use 0.5 in all experiments. For a, the EMA ratio of client-wise gradient in Eq. 2 we use 0.5 in all experiments. For X, the loss balancing coefficient between the L.2 loss and the regularization term in Eq. 20 we use 0.5. For No, gradient sampling ratio for sanitized gradient S. use 40%. For μ. the noise scale applied to set μ 10 We set N B. the number of layers employing Co-LoRA, to 4 in all experiments. See Sec. A.18 detailed hyperparameter analysis.
27

---

Published as a conference paper at ICLR 2026
[Intruccioo]]
Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant the user question displayed below. A good answer should follow these rules:
1. It should be in the same language as the question.
should answer the request in the instruction.
3. It should be factually and semantically comprehensible. should be grammatically correct and fluent.
Begin your evaluation by providing short explanation. Be as objective possible. After providing your explanation, you must rate the response on a scale 1 to 10 by strictly following this format: "[ [rating]]]] for example: "Rating: [[5]]". A human-annotated answer given for reference.
[Question] (question)
[The Start of Assistant's Answer]
(answer)
[The End of Assistant's Answer]
[Reference] (reference)
Figure 9: Prompt template used in GPT-4 GPT--judgge.
Federated distillation baselines (i.e., TAKFL, PerAda, and FedMKT) share knowledge between heterogeneous models via logits, while FedMosaic employs Co-LoRA. For the other baselines, we combine them with Fed-ET (Cho et al. 2022), which aggregates models among homogeneous clients. For W.. a small-scale pre-trained MLLM used for calculating client-wise gradients in RELA, we adopt the smallest model among all client models. For example, in Table 16 where clients have three different types of heterogeneous architectures (t.e., LLaVA-Llama3.22 1-1 L.VA--Llamm 3---3.. and 1-a---Lama3..111883 1-BB), we employ LLaVA-Llama3.2-11 as W, for all clients. This choice ensures computational efficiency and maintains gradient dimension consistency for cosine similarity calculation. Note that we can employ a lighter W, (e.g., reduced size or lower-bit quantization) to further decrease the computation overhead of gradient calculation in RELA, as detailed in Sec. Sec.A..00 [A22 All experiments are executed in Python 301 on four Ubuntu 20.04 machines, with NVIDIA RTX A6000 GPUs each. Each experiment runs single RTX A6000 GPU in day.
A.7 CLIENT MODEL CONFIGURATION
We summarize the model configurations for each PFL experiment, including model types and their counts, in Tab. [4 For experiments on multi-modal benchmarks, we use LLaVA-Llama3 or LLaVA-Qwen22 varinns, while for text-only benchmarks, we use Llama-3 variants.
A.8 COMPARISON OF COMPUTATIONAL AND MEMORY CoSTS
We compare the computational cost C and memory cost M of various baselines and summarize the results in Tab. 15 Following (Seo et al., 2025). measure computational cost in FLOPs per iteration and memory cost in Bytes. Specifically, ly, for each baseline, we report the relative FLOPSs and relative Bytes in comparison supervised fine-tuning (SFT), which only requires a single forward and backward pass without any extra computation and memory overhead.
Comparison of Computational Cost C. We first compare the computational cost FL methods. PerAda (Xie al.) 2024) incurs approximately twice the computational cost compared to other baselines, since it sequentially updates both the personalized adapter and the local adapter. Similarly, FedDAT (Chen et al. 2024a) independently optimizes the local adapter and the dual adapter teacher
28

---

Published as a conference paper at ICLR 2026
LLa Va-ama3
LLaVA-QwenzZ Qwen2.e
Llama-3
Benchmark
Experiment
IB
3B
8B
0.5B
1.5B
IB
3B
Tab.
DRAKE- AAK--Dyamm
6
0
0
Tab.
HFLB. LLB-Dynamic
6
0
0
Tah.
DRAKE-Static
6
0
0
Tab.
DRAKE-DDV AAAK--Dyamicc
0
Tab.
DRAKE-Dynamic
6
0
T6
DRAKE
Multi-modal
Tah.
DRAKE-Homo
10
17
HFL.B-Homo
Tab.
18
DRAKE-Homo
0
0
2
0
10
Tab.
T8
DRAKE-Hetero
0
3
2000
Tab.
19
Llama 3B Qwen 1.5B
6
Tah.
19
Jama IB/ /BBB / Qwen 1.5B
5
All other analysis ablation experiments
Fed-LLMM-Lrre-Dyyee
26
Fed-Scope-Static
Text-only
Tab.
20
Fed-aya-Dynamic
4
Tab.
Fed-Scope-Stutic
3
Table 14: Client model configuration details for each experiment. Counts per model family/size.
Memory Cost M
Computational Cost
Methods
Overhead Type
Relative A SFT
Overhead Type
Relative C. SFT
SFT
1,000
1.000
DITTO (CML 2021)
Daal Adapter
1.052
Double Forward/Buckwand
2,000
FedSim (IMM)
Daal Adapter
Double Forward
FedIT (CASSP 2024)
1,000
1.000
TAKFL (NeutPS 2014)
1,000
Distill Logit Extract
1.294
PerAda (CVPR
Daal Adapter
1,052
Double Forward/Backwwan
Extraction
2.294
FedDAT (AAAI 2024)
Triple Adapter
1,104
Double Forward/ Backwaar
2.564
FedDPA (NeudPS 2024)
Dual Adapter
1.052
Double Fowward/Backward
2.051
FedMKT (COLLING
Public data Logit Share
1,002
Extraction
1.574
Last Layer Compute
FedMosaic (Ours)
CosLoRA
1.053
Client-wisc Similarity Compute
1.161
Table 15: Comparison of memory and computational costs. FedMosaic incurs additional memory and computational costs compared to SFT, but only by approximately 5.3% and 16.1%, respectively.
(DAT), which also results in double computational cost. FedMKT, PerAda, and TAKFL perform knowledge distillation and transfer using logits (e.g., from public data), which introduces additional forward computation for logit extraction.
Our proposed FedMosaic incurs a minor additional computational cost due to (i) gradient computation from the frozen pre-trained model used for measur- ing task similarity (ii) dual adapter structure, and (iii) Co-LoRA aligmment, but this overhead amounts only approximately 9.8% more FLOPPs compared to SFT. Specifically, gradient computation and the dual adapter structure add about 9.8% overhead dur- ing training, while Co-LoRA alignment adds 6.25% overhead before federated learning to align heteroge- neous models, totaling 16. 1.11% % additional cost. Here, the Co-LoRA alignment overhead is computed the ratio between the number of batch iterations used for Co-LoRA alignment and those used for SFT training across all clients.
Relative Computational Cost Num Clients
Relatve to SFT
SFT
(W, LLLVAALLlaa. Llama-1B)
1.3
FedMosaic (W, -LLaVA-Qeen-..5 4bil)
12
0.9
100
500
1000
Number of Clients
Figure 10: Relative Computational Cost to SFT vs. Number of clients of FedMosaic.
The reason for the small increase in computation during training is that (i) we apply 60% gradient compression for similarity calculation in RELA, (ii) we only use the last layer gradient of the small- scale pre-trained frozen model, and (iii we perform gradient computation once every 10 batch iterations rather than on all batches for computational efficiency, as mentioned in Sec. 4.1 Similarly, the small overhead of Co-LoRA alignment stems from the fact that (i) it is performed only once before federated training to establish shared initialization, and (ii) it is required only between heterogeneous model-type pairs, not across all clients, mentioned in Sec. 4.2.2 Note that while
29

---

Published as a conference paper at ICLR 2026
FedMosaic also introduces dual adapter structure, i.e., maintaining the local adapter and the global adapter separately, similar to PerAda, FedDAT, and FedDPA, we optimize the sual-adapter at once using learnable balancing parameter in Co-LoRA, which adaptively balances local and global output. In contrast, the baselines separately optimize each adapter, thus incurring a double computational cost. Thanks to these designs, the relative overhead remains approximately at 10% even under large-scale client setups (e.g., 1000 clients), as shown by the blue line in Fig. I0 Finally, the cost can be further reduced by adopting smaller or quantized Wa. as discussed in Sec. A. For example, using LLaVA-Qenn 0.5b--bt model as W, reduces the overhead to approximately 5% with 1000 clients (yellow line in Fig. 10 while achieving performance comparable to LLaVA-Llams IB (Sec. A.20).
Comparison of Memory Cost We then compare the memory cost of FL methods. FedMKT incurs a marginal additional memory overhead, as it requires storing logits from clients for knowledge aggregation. DITTO, FedSim, PerAda, FedDAT, FedDPA, as well FedMosaic employ a dual adapter structure, which maintains both local and global adapters separately, to preserve global knowledge. It incurs additional memory cost, but since we adopt the LoRA adapter (Hu et al. 2022). which occupies significantly less memory compared to pre-trained weights (Qi et al.) 2024). the dual adapter only consumes about 5% additional memory cost compared using a single LoRA adapter.
A.9 DETAILS OF COMMUNICATION COSTS IN FEDMOSAIC
The only additional transmission in FedMosaic compared Vanilla in Tab. (i.e., FedMosaic w/o RELA and w/o Co-LoRA) is a single EMA-updated gradient vector per client, adding 8.6% overhead compared sending only local LoRA parameters. However, mentioned in Sec. 4.1 apply gradient compression by randomly selecting only N2% of the client-specific gradient vectors, thereby reducing communication costs. With N, 40%, the communication overhead drops to 3.4%.
Moreover, while baselines transmit full LoRA mod- ules (i.e., A E grxd and B Rdxr) across all layers, FedMosaic freezes A and B in Co-LoRA layers and transmits only P T and Q e R", significantly reducing 14.3% communication cost. Combining the marginal overhead from client-specific vectors with the reduction from Co-LoRA, FedMosaic con- sequently achieves 10.9% lower communication cost than even the most efficient baseline (i.e., FedAAg), regardless of the number of clients, ensuring scal- ability and communication efficiency (Blue line in Fig.
Communication Costs vs. Num Clients
FedAvg
6000
FedMosaic (Nr- 100%)
FedMosaic (N,= 40%)
Smaller
5000
Costs
4000
3000
200
300 Number of Clients
400
Figure 11: Communication Cost vs. Num- ber of clients of FedMosaic.
10 EMPIRICAL ANALYSIS OF BLOCK-WISE AGGREGATION
To identify layer-wise correspondencee between depth--eteerggeneou models, we analyze representa- tion alignment using CKA (Kornblith et al. 2019). Specifically, we measure similarity across layers within the Llama-3 family (IB, 3B, 8B) and the Qwen-2.5 family (0.5B. 1.5B, 3B), as illustrated in Fig. 12 As shown in the figure, layers with the same relative depth exhibit strong alignment, indicating approximately linear alignment within both the Llama-3 and Qwen-2.5 families. More- over, we observe near-lined linear alignment even across families, i.e., between Llama-3 and Qwen-2.5, despite weaker linearity than intra-family alignment. Moreover, to demonstrate that this layer-wise correlation trend generally holds across different models, not just between Llama and Qwen, we have additionally included the layer-wise correlation analysis between InternLM (Cai et al. 2024 and Llama in Fig. [3] which shows the same trend as our previous findings. This empirical analysis supports our block-wise aggregation of Co-LoRA. We provide an illustration of the biock-wise
Co-LoRA in Fig. 14
A.11 EXPERIMENTAL RESULTS IN MORE DIVERSE HETEROGENEOUS PFI. SCENARIOS
We evaluate FedMosaic in heterogeneous PFL using three different heterogeneous architectures, i.e., LLVA--Llama3.. 1BB, LLaVA-Llama3.2-00 and L.I. VA-Llamm.3.... 3.1 1-8B, and summarize the results in
30

---

Published as conference paper at ICLR 2026
Liama Liama
0 32-
31
Janp
34
11 12 Llama
Llama layers
Liama layers
layers
(a) Llama-3 family (1 B, B, B)
Owen 0.5B VS. Owen 1.58
Owen 0.58 VS. Owen 3B
Owen 1.5B VS. Owen 3B
Qwen 0. 58 layers
Owen 0.50 leyers
-04
36
Qwen 1.58 layers
Qwen
Qwen 3B layers
(b) Qwen-2.5 family (0.5 B, 1.5 B, 3 B)
Liama vS. Owen 1.58
Liama 18 vs. Owen
Liama 38 VS. Qwen 1.58
Liama 1B
1.80
6.5
layers
layers
(c) Llama Qwen mixed
Figure 12: Layer-wise similarity across model scales measured with CKA. Each heat-map cell reports the semteredd-eenel-llinnnee of heterogeneous multi-modal LLMs at every pair of layers. Here, lighter colors indicate higher similarity. (a) For the 1aama-3-based heterogeneous models and (b) for the 1wen-2.5-bbsee heterogeneous models, the brightest (i.e., the highest similarity) band appears roughly along the main diagonal, indicating that layers with relative depth align most strongly. The near-linear trend supports our proposed block-wise aggregation, which transfers knowledge from smaller to larger models that have the same relative depth within the same architectural family.
(CKA) similarity between the hidden representations
M2-28 Llama
InternLM2.5 vs.
M2.55-7 vs.
31
0.95
21
0.95
g
0.90
23
0.90
8
IntemLM?
13
0.85
0.85
15
0.80
0.80
0.75
0.75
0.65
0.70
11
15
15
1 411399977992229997999
Llama-18 layers
Liama-18 layers
Llama-88 layers
Figure 13: Layer-wise similarity across models measured with CKA. Each heat-map cell re- (CKA) similarity between the hidden representations from InternVL (Chen et al.) 2024b) using InternLM as base LLM and LLaVA using Llama as base LLM. The layers with relative depth align most strongly even between models
ports the seieeredd-eeel-lllinne
31

---

Published as a conference paper at ICLR 2026
Blockwise Co-LoRA
D
Co-LoRA
Conventional LORA
Block
Block
Illustration of blockwise Co-LoRA When a model has NB Co-LoRA modules, each block employs Co-LoRA its last layer, while the remaining layers adopt conventional LoRA. Each block contains the same number of layers.
Figure 14:
Tab. 16 Consistent with the previous heterogeneous PFL scenario with two different types of archi- tectures (i.e., LLa ALLaVA-Llama32-11 3-m..B..-1BBBBBB and LLa LVVA-LLama3.2-33 FedMosaic consistently outperforms the baselines in both PFL-Dynamic and PFL-Static settings. These results demonstrate that Co-LoRA effectively enables knowledge sharing across various heterogeneous architectures, highlighting its applicability real-world scenarios where each client possesses individual heterogeneous models.
DRAKE-Dynamic
DRAKE
Self
Others
Sell
Others
Method
Axue
Atust
AxcC
Ausst
SET
20.81
59.17:10.64
47.941:0.18
47.10::0.19
68.86:12.16
62 62.30-3.12 3.12
47.98:10.27
47.44:.0.20
DIT TO (ICM) 2021)
61.56=00.01
56.36++0.20
48 19:40.66
47.41=0011
65.48=0.97
60.34-0.32
48.49++0.03
48.09=00.01
FodSim (CMML 2022)
65.00:20:41
58.18:0.46
47.47=0.10
46.61=0044
67.39:0.32
62.42-0.255
47.10220.03
FedIT (CCSSPP 2024)
66. 18:0022
58.95=+0.02
47.54=0.02
46.97=0001
68.78:00.2
64.07=0.92
47.34:20.5
47.33=20.29
TAKFL (NeurPSS 2020)
64.732:0.49
58.33-1.533
47.85=0.:6
47.16=0008
68.1710.75
63.19=1.21
47.28.1014
46.90111.00
FedDPA (NeutPS 2024)
57.31
47.21:0007
67.71
FedDAT (AAAI 2024)
60.43:10.54
56.82:10.29
49.79110.21
48.25=0.255
61.9710.58
$7.64=0445
49.76.1046
47.9910.21
PerAda (CVPR
38.10:22774
54.84+ 1.68
46 46.90-0.22
46.69-0.21
61.57:0.97
56.22-0.38
45.01-10.37
47.06+00.71
FedMKT (COLING 2025)
63.25=00.05
57.23++0.18
47.88=0.29
47.25=20.15
65.50=0.37
61.12=0.56
48.27:40.4
47.70=0.31
FedMossic (Ours)
68.4=00.68
60.96:00.06
52.18:0.34
50.11=0.03
70.41=1.27
65.12-1.15
52.67=00.0
50.91=4.700
Table 16: Quantitative comparison in heterogeneous PFL with three different types of models. "Self" denotes evaluation on client's own data, while 'Others' denotes evaluation data from other clients. clients use LLaVA-Llama3.2. clients use LLaVA-Llama3. I-BB model. SFT refers to supervised fine-tuning on each client's data without cross-client knowledge sharing.
model, 5 clients use LLaVA- .233B model, and 2
A.12 EXPERIMENT RESULTS ON HOMOGENEOUS PFL SETUP
In addition to the heterogeneous PFL scenario (Sec. 6.2]. where both data distributions and model architectures vary across clients, we also evaluate FedMosaic in a homogeneous PFL setting, where clients share the same model architecture but have heterogeneous data distributions. We assume all clients use LLa VA-3B models and summarize the results in Tab Consistent with results in the heterogencous setup, FedMosaic significantly outperforms basclines in generalization ability (i.e., 'Others") and even surpasses SFT in personalization performance (i.e., 'Self").
A.13 EXPERIMENT RESULTS ON QWEN-BASED LLAVA
In addition to heterogeneous PFL scenarios with LLa VA-Llamm variants, we also compare FedMo- saic with baselines using Qwen-based LL models. Specifically, we LLaVA-Qwen2.5-44 for the small model and LLaVA-Qwen2.5-3 for the large model. For basclines, we select the top-3 well-performing baselines heterogencous PFL scenarios using LLaVA-Llama3 variants, as well as supervised fine-tuning (i.e., SFT). We summarize the results in Tab. 18
As shown in the table, FedMosaic significantly outperforms baselines in both homogeneous and het- erogeneous PFL scenarios, consistent with results from LLaVA-Llama variants. This demonstrates that our proposed Co-LoRA consistently facilitates knowledge sharing across heterogeneous models and RELA reduces interference during local model aggregation, regardless of architecture.
A. 14 EXPERIMENT RESULTS IN CROSS-FAMIN HETEROGENBOUS PFL SCENARIOS
In addition to Tab. we further evaluate FedMosaic under cross-family heterogeneity using clients with either Qwen- and Llama-basse LLa VAs TRAKE-Dynamin As shown in Tab. [99 FedMosaic consistently outperforms all baselines, demonstrating its generalizability and transferability beyond the same family heterogeneous models (e.g., LLLVA-Lllam--11 1B, Laaa
32

---

Published as a conference paper at ICLR 2026
DRAKE- DRAKE-Dyuamis
HFLB- BrLB-Dyaamic
Self
Others
Sclf
Others
Method
Atoe
AADe
Absst
Aare
Auast
AATr
Auao
Auu
SFT
67.57=00.14
59.18.1.266
49.70-0.02
48.04=1118
80.26.2:.14
77.98-0.49
63.24.10.23
62.90110.60
DO (CML.
64.09-0.55
57.22++0:11
49.92-0.27
49.05-0.07
79.34+0.41
76.72-0.84
64.10++0.40
63.57+00.06
FedSim (CML) 2022)
67.05=00.88
60.69:0.84
47.94=1.69
47.55=1.144
79.60=0.77
76.93=0.78
60.85:1.155
60,50=00.2
FedIT (ICASPP 2024)
67.62-0.449
60.35+0.69
49.84+0.19
48.95-0.13
79.34+0.59
1..40-1.221
62.51+1.30
62.38+0030
(NeurPS
67.02=0004
59.8620.36
50.03=00.6
48.85=0.09
79.45=0.69
17.15=0.99
63.01:00.66
62.70=0096
FedDPA INeutPS 2024)
66.39.10.86
58.68:10.71
49.92:10.24
48.81=0.133
80.5710.62
77.65_0.61
63.58.10.30
63.02.00.09
FodDAT [AAAI 2024)
61.95=00.00
57.22=00.45
51.50=0.13
50,28=0008
79.58=0.53
76.82=1.13
67.33=40.13
66.47=0.29
(CVPR 2024)
63.78:10.70
49.7910.25
49.10
76,96
76.31=1.24
63.35.11.26
63.091.0.56
FedMKT (COLLIGG 2025)
65.20 65.20-00.43
58.73+00.12
49.68-0..2
48.84-0.22
79.43--0.69
77.24-0.61
62,66++0.38
02.79+0.86
FedMossic (Ours)
70.80-0.23
62.18++0664
53.93-0.66
$1.72-0.28
80.75+0.14
78.87 0.20
68.30++0.69
67.46-+0.22
DRAKE-Sutie
HFLBB-ttatic
Self
Others
Self
Others
Method
Amos
AuDe
Atsst
AAu
Atext
AAUL:
Acuu
Awuc
SFT
70.56=00.27
66.31
50.37=0.04
49.99:0.16
80.89=0.85
80.01=0.12
63.40::0.41
63.09=20.17
DITTO (CML 2021)
06.19.8
62.05
50.59.10.05
49.95-0.088
80.3710.37
79.05-0.25
63.93.10.24
63.52.1:0.10
FedSim (CMI)
68.65-0.86
63.96-00.78
49.27+0.22
48.85-0.14
80.05+0.01
79.36-0.79
61.22:40.00
61.31+00.2
FedIT (CASSP 2024)
70.14:1..03
66.08.10.59
50.0910..15
49.87-10.26
80.63.1.22
79.97-0001
63.34.10.11
63.09110.04
TAKFL (NerIPS 2014)
68.75+0.04
64.39+00.07
49. 49.70-00:16
49.60-0.03
79.97+0.11
79.41-0811
63.67+00.9
63.30 63.30+-0.25
FodDPA (NeulPS 2024)
67.40=22.73
63.71#1.24
50.45=0.38
49.92=0030
79.
79.02=0.35
63.45:20.07
63.11=0.02
FedDAT (AAAT 2024)
64.09=1.116
61.10+00.0
52.20=0.266
5147
79 80+00.50
78.43-0.68
68.92:+0.44
67.66+00.27
PorAda (CVPR 2024)
65.64120.48
61,93:20.58
50.53=0.02
49.94=0.05
80.25=0.455
79.01=0.26
64.0010.13
63.58=0.31
FedMKT (CULING 2025)
67.85:10.92
62.93.11.11
1.52
48.77
79.85
79.72-0060
63.84110.20
63.39
FedMosaic (Ours)
72.02.00.37
67.18.10.13
54.21:10005
52.94.0.055
81.6910.11
8..54.00099
68.70.10.26
67.75
Table 17: Quantitative comparison in homogeneous PFL. 'Self' denotes evaluation on a client's s own data, while 'Others' denotes evaluation data from other clients. All clients use VL-3B models. SFT refers to supervised fine-tuning on each client's data without cross-client knowledge sharing.
DRAKE- BLAAK---Hoongggou
DRAKE Heterognccoon
Self
Others
Self
Others
Method
Aust
Aau:
Atost 1
Aau f
Ausst
AAue:
Alst
Aauc
SFT
68.79=1.36
59.06:20.18
47.87=0.40
47.10=:0.44
65,79=0.17
58.24=0.14
44,4020.86
44,06:20.44
FedIT UCASSP 2024)
68.40:10.07
60.58.1.441
49.7010.52
48.5420007
66.02.0.72
58.26=1.06 1.06
44.47.0.300
44.06110.43
FodDPA (NeudPP 2024)
66.73:00:4
60.21:0:47
49.39-0.07
48.6920.11
65.43=0.64
57.79_0.08
44.4820.43
44.53:00.26
FedMKT (COLING
65.95.00.35
58.8110.64
49.7310.09
48.52-1024
62.85.0.99
56.71-0.33
44.7311.35
44.4110.75
FedMossic (Ours)
70.38-0.34
62.84+0.23
54.25-0.33
51.85-0.20
67.36+0.21
59.95-0.71
50.43+0.01
48.51+0.299
Table 18: Quantitative comparison in PFL--Dynmicc using Qwen-based LLaVA. 'Self" de- notes evaluation on a client's own data, while 'Others' denotes evaluation on data from other clients. In DRAKE-Homogencous,, all clients use LLaVA-Qwen2.5-3B models, while in DRAKE- Heterogeneous, 3 clients use LLaVA-Qwen2.55 model, and client uses LLaVA 1VVA-wwe2.5--3 3BB model. SFT refers supervised fine-tuning on each client's 's data without cross-client knowledge sharing.
model, 2 clients use
A.15 EXPERIMENTAL RESULTS ON TEXT-ONLY BENCHMARKS
In addtion to Fed-LLM-Large (Sec 6.2 we evaluate FedMosaic on other text-only PFL benchmarks, Fed-Aya and Fed-Scope, and summarize the results in Tab, 20 As shown in the table, FedMosaic outperforms baselines in both personalization and generalization, consistent with results from other MLLM PFL benchmarks, such DRAKE and HFLB
A.16 CLIENT- WISE ACCURACY
In addition to reporting the average accuracy across all clients in Sec. 6.2 report client-wise accuracy in both homogeneous and heterogeneous PFL setups.
Homogeneous PFL Setups. We first compare client-wise performance in the homogeneous PFL- Dynamic setups on DRAKE and HFLB, and summarize the results in 21 and Tab respectively. As shown in the tables, personalization performance is improved for all clients compared to SFT (i.e., supervised fine-tuning
21
local data) except Client 8 in Tab. 22 We highlight this result because SFT is a strong baseline for personalization (Ghari & Shen 2024), as we also show in Tab. [7 where it outperforms all baselines except for FedMosaie. Moreover, supervised fine-tuning may be
33

---

Published as a conference paper at ICLR 2026
Llama 38/ / Qwen 1.5B
Llama Lllama 3B / /Qwen 1.5B
Self
Others
Sclf
Others
Method
Ateee
Aare
Amset
Aare
Auast
AArr
Auas
Auuc
SFT
68.4120.39
61.04
48.4820.15
47.75_00007
67.24.10007
60.42-0.35
47. 47.48.0.02
46.73:10.10
DO (CMM.
64.34-0.35
57.72++0.99
48.35+0.38
47.63-0:22
63.78+0.37
57.11-0.78
47.88+00.99
47.09+00099
FedSim (CML 2022)
66.40=0.94
59.08+0..33
47.32=0.59
46.74=0.33
65.24=0.21
58.33-0.24
46.73++0.64
46.07=0.41
FedIT (ICASSP 2024)
67.81+0.73
60.46+0.61
48.37+0.24
47.41-0.06
67.21+0.51
60:05-0.07
47.25++0.04
46.62+00.03
TAKFL (NeurPS
65.41=00277
58.52:00:07
47.51=0062
47.34=0.32
66.1341.23
58.47=0.44
47.70=0.83
46.7020.41
FedDPA INeutPS 2024)
65.92.20:49
58.73:10.01
48.30:10.20
47.41=0.19
65.93.1.200
58.42=0.07
47.67.10.08
46.6410.05
FodDAT [AAA] 2024)
63.86:21.35
58.4140.33
50.08=0.34
49.23=0.09
63.1621.65
57.74=0.78
49.13=20.33
48. 18.1610.15
PerAda (CVPR 2024)
64.11:10.73
57.89.10.54
48.5210.551
4..76.10.27
62.8711889
57.07-0.62
47.54.1004
46.8710.011
FedMKT (COLLIGG 2025)
65.38-00.99
58,57+0.11
47.60+0.49
47.29-0004
64.29 1.34
58.31 58.31-0.43
47.13400.3
46.71+0.37
FedMossic (Ours)
70.62
63.33++.26
52.56-0.00
50.76-0.01
69.64
62.09-15 0.51
51.69+0.115
49.85+0.13
Table 19: Quantitative comparison in cross-family heterogeneous PFL on DRAKE-dynandn
"Self" denotes evaluation on a client's s own data, while 'Others' denotes evaluation on data from other clients. 'Llama 3B / Qwen 1.5B' experiment is with 6 clients using LLaVA-Llama3.22 2-3B model and 4 clients using LLaVA SVV---wn2.5-1...
model, while 'Llama IB Llama 3B Qwen 1.5B' experiment 1B model, 5 clients using LLa LLaVA-Llama3. 2-3B model, and 3 clients using LLaVA- LLaVA-Qwen2.55 Swae-.eee22255. 1.5B model. SFT refers to supervised fine-tuning on each client's 's data without cross-client knowledge sharing.
is with 2 clients using LLaVA-Llamat
Fed-Ava
Fed-Soope
Self
Others
Sell
Others
Method
Atazt 1
Asur
Ause
Aaue 1
Auat 1
Aare
Asut f
Auur
SPT
2.34+-0.08
2.174-0.03
2.01+-0.02
1.99-0.01
1..3
29.35+0.61
25.83+11008
27.30-00.3
RedSim (ICML, 2022)
1,9910.03
1.9810.04
1.92.10.01
1.92.10.03
21.13.10.97
20.03
15.5710.87
18.5910.30
FedIT (ICASPP 2024)
2.27=0008
2.1810.02
2.1920.11
2.10:20:04
1..06:1.336
28.89:0.66
25.8622094
27.94=0.44
TAKFL (NeulPSS 2024)
2.34=0.12
2.17+0.06
2.19+0.10
2.08=20:33
26.18+1.05
27.84=0.57
28.53=0.33
28.27=0.19
FedDPA (NaulPS 20124
2.39+0.09
2.27+0002
2.20+0.05
2.13-0.001
24.93++100
27.51+0.55
26.72+0067
27.69-0.27
FedDAT (AAAI 2024)
2.1410.14
2.0910.04
1.98
1.96.10.04
27.5110.28
26.7210.53
27.69
FedMKT (COLING 2025)
2.3120.06
2.12:0.05
2.06:10.02
1.96:20.00
24.92:20.96
27.5120.63
29.55:0:43
30,57:00199
FedMosaic (Ours)
2.51+0.01
2.32+-0.3
2.25+0.01
2.17-00.02
30.58 0.84
22.68-0.64
3101+1146
32.50-0.62
Table 20: Quantitative comparison of heterogeneous LLM clients on the text-only benchmarks. In Fed-aya experiment, 4 clients use Llama-3.2-18 and 4 clients use Llama-3.2-3B. In Fed-Scope experiment, 3 clients use Llama-3.2-3B and 2 clients use Llama-3.1-8B.
sufficient for personalization some tasks (Wožniak et al. 2024 Mosbach et al. 2021). as seen with Client 6 in Tab. 21 where there are only marginal improvements compared to SFT. However, for more challenging tasks, properly leveraging knowledge from other clients significantly improves personalization, e.g., FedMosaic shows gain of 8.2% in Atast for Client 9 and 6.4% for Client 7, as well as a 7.1% % improvement in AxuC for Client 5. In summary, even though there is only a 1-3% gain in the average performance across clients, this is because in clients where SFT is sufficient for personalization, the gain from PFL seems smaller. Consistent with the PFL.-Dynamic setup, FedMosaic outperforms SFT in the PFL-Static scenario, as shown in Tab. 23
Anse Aunc
Method
Cheene
Cherrt 2
Cheet
Client 4
Client 5
Chirre
Chere
Chent 8
Chient 9
Client 10
SET
31 90/
59 28
67.3% 50.31
20.28 00.40
53.88 62 OT
76.05 66.
65 54.17
66 93 59.31
62, 52.20
5t. 16.66/55555 59.88 55.89
FedMoene
$2.38/ 66:11
67.31 63.76
6R 62.53
71.0/67.51
68477 /63.8
76.96/ 68.56
69.83/ 58.86
70.11 00.68
78,89 54.99
Difevennce
+0,37+401
+4.57+131
12.8/
+1.59/+2..3
+0,57 11.3
16.45/
H077
15.197
+1.71/+1.31
Table 21: Per-client performance of FedMosaic SFT in a homogeneous PFL-Dynamic setup on DRAKE. The Difference row shows the performance gain/loss of FedMosaic compared to SFT. All clients use the LLaVA-Llama3.2--3 model.
Atuu/ Asge
Method
Client
2
Client
Client
Client
Client
SFT
82.08/ 13
06.26/ 99.59
52.73/
6 81.52
72.79/ 0 73.93
184.44
76.04/ 73.79
79
VIL58/
FedMossic
82.77 79.79
/97.08
$2.99
50.88
81.89 / 81/98
74.31/74.9
88.20 86.55
76.78/ 74.74
81.94 80.33
90.00/ 83.25
Diffensnce
+0.49/ 20.67
40.34/ 40.48
+0.267 40.67
.0.7 .0.4
+1.52/-2.02
40.74/ +0.94
+0.51 / +0.50
4.58 40.23
Table 22: Per-client performance of FedMosaic vs. SFT in Homogeneous PFL-Dynamic setup on HFLB The Difference row shows the performance gain/loss of FedMosaic compared to SFT. All clients use the LLaVA-Llama3.7 2-BB model.
34

---

Published as a conference paper at ICLR 2026
Asus I Auee
Method
Client
Client
Client
Client
SPT
83.09/ 88,54
96.82/
96.70
71.375
80.62/ 80.32
72.34/ 73.06
92.11 190.2
72.56/ 75.12
82.66 88.74
91.04/80 889.83
FodMrestc
83.21 88.98
97.96 97.68
53,29 1 52.24
81.59/ /81.92
76.20/ 74.54
92.34/ 19113
77.77 75.89
81.00 80.15
91.08/ 89.99
Diffrrnce
+0.12/ +0.44
+1.33/ +0,98
+1.48 +0.96
+0,97/ +1.00
+3.86 +0.87
-0.3/700.3
+0.21/ +0.46
0,86 / 0..9
+0.05 -0.16
Table 23: Per-client performance of MedMo vs. SFT in a homogeneous PFI.-Static setup on HFLB. The Difference row shows the performance gain/loss of FedMosaic compared to SFT. All clients use the LLa VL-VA-L 3-BBB model.
Heterogeneous PFL Setups. Consistent with the results in the homogeneous PFL setups, FedMo- saic enhances personalization ability in most clients, shown in Tab 24b and Tab 25 As shown in the tables, not only do clients using smaller models (i.e., LLaVA-L.ama3..2 1B, LLaVA-Lllama3.2-33 benefit from knowledge sharing through PFI., but clients with larger models (i.e., L.aVA-Llama3..2 8B) also see significant improvements. e.g., a 6.5% improvement in Alast and 9.4% improvement in AAuC for Client 9 in Tab. 24 We believe this is due to Co-LoRA effectively sharing knowledge between heterogeneous architectures, allowing both smaller and larger models to provide meaningful information each other.
Alsa Auc
HaVe-l 2-1
Method
Cliers I
Clicst 2
Cliess
Clicas
Clicut 5
Cliest 5
Clien
Cliert 8
Clicut
Clieat 10
7.31/ 1//66855
41.22/55.00
63.22
77 7784/71
66
56.91
64
38.61
57.17
FalMousi:
61.35/ 99.93
67.61 63,08
78.56/ 7471
72.35
71.45 62.36
65.3
61,88
71.12/ 66.58
7366/7007
Diffeence
+0.95/11377
+0.13/2213
+0.5/+0885
+373//3370
1553/+314
15.99/55455
+1,4/+1.00
0,197-037
+6,99/49411
+1.00/11.06
Table 24: Per-client performance of FedMosaic vs. SFT in a heterogeneous PFL-Statt setup on DRAKE. The Difference row shows the performance gain/loss of FedMosaic compared SFT. Clients 4-8 use LLaVA-Llama3.2-0 and Clients 9-10 use
Clients 1-3 use
LaVAA-Llama31-88.
Alsst / Age
LLaVA-Llama3.22 2-1B
LLaVA-Llama3 2-38
Method
Client
Client 2
Client 3
Clbent
Clieni
Clieni 6
Client 7
Client 8
Clieni 9
SFT
50.69/ 77.18
97.41 195 25
$0.95 747.72
82 39 1.1 $1.49
73.99 /3.00
92.10 / 88
76.58/ 74 74.19
81.00/ 79.94
80.32 83.12
FedMossic
81.69 77.53
96.46 / 96.18
$1.38 / 47.85
82.70 /S241
74.49 75.09
10.00/ 88.36
76,95/ 74.92
82.24/ 80.25
89.94 83.62
Difionnnce
+1.00/
10.13
+0.32/0095
/2.06
40.15
40.36/
+1.34/
40.62 1.62//40.0
Table 25: Per-client performance of FedMosaic vs. SFT in heterogencous PFL-Dynamic setup on HFLB. The Difference row shows the performance gain/loss of FedMosaic compared SFT. Clients 1-3 use LLaVA-Llamal 2- 1B, while Clients 4-9 use LLaVA --VA-A---aam.3333
A.17 DETAILS OF PUBLIC DATA Dp
For the public data Dp. we use a subset of the MLLM's pretraining data, as briefly mentioned in Sec. 4.2.2 Specifically, since our experiments are based on LLaVA, we adopt its instruction tuning 158K (Liu et al. 2023b), as D,. For the alignment details, we randomly sample 5,000 examples from D, and align Co-LoRA for 1 epoch using a batch size of 4, a learning rate of 5 x 10- and the Adam optimizer with cosine scheduler. To further assess the effect of Dp. we explore the effect of D, 's size and distributional alignment with the client data.
dataset,
We first assess the effect of public data size in Tab 26 As shown, using too little data results in insufficient aligmment of Co-LoRA. leading to degraded performance. Interestingly, using too much public data also causes slight performance drops, likely due to overfitting to the D, distribution, which hinders generalizable alignment to clients" distributions. Considering this trade-offf, we selected moderate dataset size (i.e., 5,000 samples).
Next, we assess robustness under domain mismatch between D, and the clients' distribution. Specifi- cally, we use ChatbouT (Zheng et al. 2023) (text-only NLP benchmark) and Visual Storytelling (Li et al. 2024d) (multi-sentence outputs) as Dp. both differing from the clients' multi-modal, short- answer tasks. To ensure a fair comparison,
fix the size of Dp to 5,000 samples. As shown in Tab. 27 using Visual Storytelling shows comparable performance with LLa Similarly, using text-only NLP benchmark shows performance comparable to (and even surpassing)
158K.
35

---

Published as a conference paper at ICLR 2026
Self
Others
Size of Dp
Atest
AAuC f
Alast t
AAuC
625
67.25-0.08
59.17+0..05
51.02+0004
49.05=0.09
1250
67.97-0.04
59.86+0.16
51.33+0.03
49.58-0.07
2500
68.11-0.06
59.76+0.00
51.35++0.31
49.52-=0.13
5000
67.96-0.05
59.8340.115
51.46:10.04
49.56=00.66
10000
67.44-0.18
59.3440.16
51.04:0.116
49.37=00.33
Table 26: Effect of public data Dp size. FedMosaic performs best on moderate-sized datasets (i.e., 1,250-5,000)
Self
Others
Dataset D,
Atast
AAUC
Atast f
AAUC
LLaVA--nstrucc- 158K (1 Liu et al. 2023b
67.96-0.05
59.83++0.15
51.46+0.04
49.56+0.06
ChatbodIT (Zheng et al. 2023
68,42-00118
60.17+00.6
51.16+0.22
4..78+0.13
Visual Storytelling (Li et al. 2024d
67.90=00009
59.86:t0.12
51.06:00.05
49.41
Table 27: Effect of domain gap between Dp and clients' data. FedMosaic shows consistent performance across multiple benchmarks, even on text-only NLP benchmarks (i.e., ChatbotlT).
Mat.VAA-nstrut--55555 demonstrating resilience of FedMosaic to distribution misalignment between Dp and clients' data.
A.18 HYPERPARAMETER ANALYSIS
Since dataset- sataset-specific hyperparameter search undesirable under distribution shifts (i.e., Dynamic setup), where future data are unknown, we adopt single set of hyperparameters across all benchmarks and setups, determined from the DRAKE-Dynamic setup.
Effect of Communication Rounds R and Local Steps. We analyze the trade-off by varying the number of communication rounds and local steps under fixed training budget. As shown in Tab, [ more rounds (i.e., fewer local steps per round) degrade personalization ('Self"). We attribute this to insufficient local adaptation, which weakens the personalized models and reduces the quality of shared knowledge during communication, ultimately limiting improvements in generalizability. Con- versely, significantly reducing rounds (i.e., increasing local steps) improves personalization but harms generalization (' ('Others') ) due to limited inter-client knowledge sharing, reducing generalizability. To balance this trade-off we adopt moderate setting with 20 rounds and 100 local steps per round.
Effect of Low Rank in Co-LoRA. We study how varying the rank in Co-LoRA affects performance, and summarize the results in Fig. 15 As shown in the figure, while both excessively high and low values of lead to degraded performance, broad range of intermediate values shows stable performance. While a larger r increases shareable capacity (I"X)), it may cause overfitting (Lin al., 2024| Cho et al., 2024). In contrast, smaller helps mitigate overfitting but may introduce capacity limitations, resulting in suboptimal performance (He et al. 2022). By balancing the trade-off, we select adequate low rank r 128.
Efffect of Ng 1 in Co-LoRA. We study the effect of NB, the number of layers employing Co-LoRA, and summarize the results in Tab. 29 In model W with [W| layers, increasing Ng reduces the layers using conventional LoRA to W Ng. Since Co-LoRA has fewer trainable parameters (i.e., (d, + do)). increasing NB decreases the total number Rddo xr are frozen, with only P E TR and Q E R' being trainable. Despite the reduced number of parameters, shown in the SFT performance in the table, performance remains stable even as Ng increases. We attribute this to orthogonal and frozen A and B, which maximize and preserve the expressiveness of Co-LoRA (Theorem despite the lower trainable parameter count.
2 r) than conventional LoRA (i.e., r
of learnable parameters in W, as A RRrx and
However, there is trade-off in Ng in the PFL setup: increasing Ng improves generalizability (e.g., No = 2 vs. Nn = in AwuC of 'Others' in FedMosaic) but can degrade personalization (e.g.. NB vs. NB 8 in A and A AxUC of 'Self" in FedMosaic). This is because, while increasing
36

---

Published as a conference paper at ICLR 2026
Self
Others
Total Rounds
Local Steps per Round
Alast
AAuC 1
Atast
AAUC
5
400
66.66-+0.89
60.21+-0.61
47.36++0.13
46.65=0.01
10
200
05.6610.447
58.5810.229
47.44:10.03
46.63:0.26
20
100
0..86-0.51
59.83
51.36.10.04
49.46-1006
40
50
6..05=00554
59.49:0.11
51.10:0.11
49.42-10.02
66.83:10.38
59.23:10.06
51.16::004
49.47=10.21
Table 28: Effect of communication rounds and local steps on DRAKE under Fixed Total Training Cost (i.e, Total rounds Local steps per round).
Effect of Low rank r of Co-LoRA
60
70
58
Alast
AAuc
68
56
54
32
128 Low rank r
5i2
1024
52
32
128
512
1024
Low rank
Figure 15: Accuracy on different low rank values in Co-LoRA on DRAKE-Dynamic setup. Extremely low or high ranks degrade performance, but wide intermediate range maintains stable accuracy.
Na enables heterogeneous models share knowledge across layers, the number of trainable parameters for local training decreases due to the reduction in the number of conventional LoRA modules. As result, by balancing this trade-off, we employ a moderate value of NB, i.e. NB
Self
Others
Method
Aiuo
Aaue 1
Atust
AAuC
SFT (Ng 2)
66.14:10.22
$7.02++0.09
47.75 0.42
46.02:1-0.1
SFT (Ng 4)
66.24=0.68
57.71+0.27
47.63=0.15
46.62+0..1
SFT (Ng 8)
66.21+0.62
$7.87+0.20
47.86-0.29
46.76++0.20
FedMosaic (Ng
68.07=0.110
59.90=10668
51.07=0.06
48.78 48.78+20.1
FedMosaic (Ng 67.86.10.51
19.8310.116
51.26 0.04
49.36 10.08
FedMosaic (Np 8)
67.52+0.77
59.39++.223
51.27 0.27
49.26+
Table 29: Accuracy on different number of blocks (Ng) in Co-LoRA in heterogeneous PFL- dynamic setup on DRAKE. Given model W with [W] layers, using Ng blocks means that Ng layers adopt Co-LoRA, while the remaining WW - NB layers use conventional LoRA.
Effect of aaahogonalityyrgguuaizaaiie Scale À in Co-LoRA. We study how varying the reggooaaltt--ggullriatattooo scale in Co-LoRA alignment process (Eq. 20) affects performance, and summarize the results in Fig. [6) We observe that the 'Self' performance does not fluctuate on larger X, but reduces for small A. This indicates that insufficient weighting of the aathogonality--egulaizziaaa term weakens the enforced orthogonality among A matrices from heterogeneous models, thereby reducing the representational capacity (i.e., Span) of Co-LoRA, as mentioned in Theorem
Effeet of Temperature in RELA. We analyze the effect of the softmax temperature 7 in RELA 4). which controls the aggregation sharpness, i.e., lower focuses aggregation similar clients, while higher aggregates weights across more diverse ones. Interestingly, as shown in Fig. 17 neither decreasing nor increasing the temperature consistently improves personalization generalization Instead, we observe two trends: (i) Extremely low 7 degrades both personalization and generalization (ii) Increasing too much reduces personalization, while generalization gradually stabilizes.
This is because low T limits the diversity of aggregation, restricting knowledge sharing even among moderately relevant clients. In contrast, high encourages sharing across dissimilar tasks, which can
37

---

Published as a conference paper at ICLR 2026
Effect of Orthogonality regularization scale À
60.0
68.5
last
68.0
AAUC
59.75
67.5
67.0
59.5
66.5
2
À
Figure 16: Accuracy on different L2 regularization scale À values in Co-LoRA DRAKE- Dynamic setup. Extremely low À degrades performance, but a wide intermediate range maintains stable accuracy.
reduce personalizability due to the inclusion of unrelated knowledge in the global model. However, increasing 7 does not continuously improve generalization; converges after a certain point. We believe this is because, although a higher promotes aggregation of more diverse knowledge, also increases parameter interference (e.g., sign conflicts) when combining models trained on distinct tasks (Yeh al., 2023 Ding et al. 2024). Consequently, adopt moderate temperature of 7 0.5 for all experiments.
Effect of Temperature of RELA
69
61
Self Slast Alast
Self AAuc
60
58
16
Temperature T
Temperature T
52
50
Others Alast
Others AAUC
51
49
48
16
16
Temperature T
Temperature T
Figure 17: Accuracy under different temperatures on DRAKE-Dynamic setup. Extremely low or high temperatures degrade performance, but wide intermediate range maintains stable accuracy.
Effect of Gradient sampling ratio N₂ in RELA. We analyze the effect of gradient sampling ratio N, and summarize the results Fig. [8 As shown, sampling only a very small number of dimensions from the decayed client-wing gradient 9; to construct the sanitized gradient Ø; significantly degrades performance, as it undermines the representativeness and informativeness vector (Li al. 2024b). However, using up N, 40% of the gradient dimensions maintains comparable performance while reducing both communication costs and privacy risks from gradient inversion, Accordingly, consistently set N, 40% for RELA all experiments.
the compressed gradient
Effect of decaying EMA ratio a in RELA. We analyze the effect gradient deacying EMA ratio and summarize the results in Fig. 19 Too small ignores current task information, while too large
38

---

Published as a conference paper at ICLR 2026
Effect of Gradient sampling ratio Ns
Self Alast
68
Self AAuc
59
66 100
80
60
40
20
58
100
80
60
40
20
Gradient sampling ratio Ns (%)
Gradient sampling ratio N₂ (%)
Figure 18: Accuracy under different gradient sampling ratio NN on DRAKE-Dynamic setup. Extremely low gradient sampling ratio degrades performance, but wide range maintains stable accuracy.
a ignores previously learned tasks. Balancing the trade-off, we consistently set a = 0.5 for RELA in all experiments.
Effect of EMA ratio a of RELA
60
68
Alast
AAUC
66
0.1
0.3
0.5
0.7
0.9
59
0.1
0.3
0.5
0.7
0.9
EMA ratio a
EMA ratio
Figure 19: Accuracy under different gradient decaying EMA ratio on DRAKE-Dynamic
setup. Balance between current and old (a = 0.5) shows the best result.
Effeet of noise scale μ in RELA sanitized gradient. We also study the noise scale μ in sanitized gradient (Eq. 3 in RELA and visualize the results in Fig. 20 We clearly see the decreasing trend in the results when we increase the noise scale. While stronger random noise enhances privacy by better concealing sensitive information, excessive noise severely distorts the gradient signal, undermining the reliable estimation of task relevance based on gradient similarity.
Effect of Noise scale μ in Sanitized gradient
60.5
68.5
68.0
60.0
Aast
AAuC
67.5
59.5
67.0
66.5
59.0
1e-5
1e-4
1e-3
1e-2
1e-1
1e-5
1e-4
le-3
1e-2
1e-1
μ
μ
Figure 20: Accuracy under differrnt noise scale on DRAKE-Dynamic setup. Strong noise distorts the gradient signal, undermining the reliable estimation of task relevance based on gradient similarity.
Effect of Gradient computing interval m RELA. We analyze the effect of computing interval (i.e., computing every m batches) of client-wise gradient 9i during local training in both perfor- mance and computational cost, and summarizes the results in Fig. 2T As shown Fig. 2T computing gradients frequently (i.e,, smaller m) improves performance but increases computational cost,
39

---

Published as a conference paper at ICLR 2026
whereas computing them too infrequently (i.e., larger m) reduces cost but significantly degrades accuracy. Since wide range of m values (i.e., 1~20) maintains stable performance, we adopt 10 as balanced choice that offers comparable accuracy with reduced computational overhead.
Effect of Gradient Computing Interval m
69.0
68.5
m
Alast
68.0
m
67.5
m
m 20
m 25
67.0
50
66.5
1.2
1.3
1.4
1.5
Relative cost C to SFT
Figure 21: Accuracy ys. Computational Cost under different gradient computing interval m on DRAKE-Dynanm setup. Computing gradient every 10 batches during local training shows the best balance between the performance and computational cost.
Effect of choice of Gradient layer in RELA. We also analyze how the choice of model layers from which per-client gradient vectors are collected for RELA affects performance, computational cost, and communication cost. As shown in Fig. 22 (a), employing only the last layer achieves comparable performance to employing gradients of more layers (e.g., Last 8 layers, All layers) with incurring minimal computational cost. Moreover, utilizing gradients from more layers increases communication cost, as the dimensionality of the vectors transmitted from clients to the server grows accordingly. As shown in Fig. (b), this leads to higher communication overhead when more layers are included. Based on this empirical evidence, which demonstrates that the last-layer gradient provides a cost- efficient approximation to full- full-layer gradients while achieving nearly identical performance, we track only the last-layer gradient in RELA.
Effect of choice of gradient extracting layer
Last layer
Last 2 layers
Last 4 layers
Last 8 layers
All layers
68.5
68.0
68.0
Alast
67.5
67.5
67.0
67.0
1.1
1.12
1.14
1.18
Relative computational cost C SFT
Relative communication cost Last layer
(a) Accuracy vs. computational cost
(b) Accuracy vs. communication cost
Cost under different gradient extract- ing layers on DRAKE-Dynamic setup. Note that we use the last Linear layer parameter for 'Last layer' while using all parameters in Transformer layers for others. The results demonstrate that the last layer gradient provides a cost-efficient approximation to full-layer gradients, achieving nearly identical performance.
Figure 22: Accuracy vs. Computational/Commnieen
A. ABLATION STUDY RELA
Our proposed RELA measures task similarity among clients using client-wise gradients to construct similarity-wwart customized global model for each client. Specifically, for the i-th client, we maintain decayed gradient ý,, updated via exponential moving average (EMA) from the current gradient yi.
40

---

Published as a conference paper at ICLR 2026
which is computed using the current batch and small, frozen pre-trained model. Note that the last-layer gradient of the frozen model, not the training model, as mentioned Sec. 4.1
We ablate RELA by comparing four aggregation variants: (1) Equal-weight aggregation, which ignores task similarity and aggregates local models uniformly: (2) Training model gradients, using current batch gradients from the trainable model; (3) Frozen model gradients, using current batch gradients from the frozen pre-trained model; and (4) Decayed frozen gradient (i.e., RELA), which maintains an EMA of gradients from the frozen pre-trained model to capture model knowledge shifts. We summarize the results in Tab.
As shown in the table, model aggregation considering task similarity measured by client-wise gradients generally improves the "Self" accuracy. This demonstrates the loss of information in Equal- weight aggregation due parameter interference. While using gradients from the training model (" ("Training model gradient") shows improved performance compared to Equal-weight aggregation, it still suffers from data heterogeneity, as gradient similarities from models trained on heterogeneous data may not capture actual similarity (Tang al., 2020 [vans et al. (024). Using frozen model's gradients from the current batch ('Frozen model gradients') can address this limitation, but cannot reflect model knowledge shifts under distribution shifts, mentioned in Sec. 4.1 In contrast, RELA effectively captures task similarity under data heterogeneity with distribution shifts, thus outperforming other aggregation strategies.
Self
Method
Ajast
AAUC t
Eeuul-weight aggregation
66.99-40.77
58.39--0.38
Training model gradients
67.16++0.74
59.38
Frozen model gradients
67.24:11.27
5..45
Decayed frozen gradients (RELA)
67.86+0..51
59.83++0.16
Table 30: Ablation of components in RELA in Heterogeneous PFL-dynamic setup on DRAKE. -eight-weight aggregation' refers to aggregating local models with equal weight. 'Training model gradients' refers to using gradients from the current batch computed with the training model, 'Frozen model gradients' refers to using gradients from the current batch computed with the frozen pre-trained model, and Decayed frozen gradients' refer EMA-estimater gradients from the frozen model (i.e.. RELA).
A.20 RELA WITH LIGHTER W,
To extract per-elient gradients for RELA, we employ the smallest MILMM among clients' models (e.g., I.L.VA-1.5 ALaVA--15--BB) To further reduce the computational overhead of RELA, we can utilize an even smaller model (c.g., LLaVA-Qwen-O.5B or apply lower-bit quantization (e.g., 4-bit and 8-bit), as shown in Tab. 31 The results show that FedMosaic maintains consistent performance even with smaller models and lower-bit quantization, demonstrating that computational costs can be further reduced without sacrificing performance.
Self
Others
W,
Relative computational costs
Auast
AxuC
Ausst
AxuC 1
LLaVA- --ae-allam3221111 2-1B (16bit)
1.0
67.86:0.51
59.83:10.15
51.16:20.04
49.36:20.08
LLaVA- L.VA--Qwe--0.55 (16bit)
0.52
67.84=10.19
59.42:10775
51,40:0.12
49.37-0.19
LLaVA- M..VA-Qwen-O.5f5 (8bit)
0.26
67.3620.14
59.22:10.43
51.74.1.06
49.56:10.07
LLVVA-Qwen-O.5B (4bit)
0.13
67.2310.49
59.41:0.13
51.34:10.42
49.43:10009
Table 31: Effect of W, 's model size and quantization DRAKE-Dynamic Relative compu- tational costs denote the ratio of computation compared to LLaVA-1.5-1D (16-bit). Comparable performance is maintained with both a smaller model (i.e., 0.5B) and quantized models (i.e., 4-bit, 8-bit).
41

---

Published as a conference paper at ICLR 2026
A.21 DIFFERENT WEIGHT ALIGNMENT METHODS IN Co-LORA
To ensure shared initialization across heterogeneous architectures, we align A matrices from heteroge- neous architectures (i.e., A, E R Aj E Rr)d)) using L2 loss, and B matrices (i.e., B, E R4 B, IR xT) using canonical correlation analysis (CCA), as detailed in 4.2.2 We ablate the effects of aligning A and B matrices, and summarize the results in Tab. 32 Note that the results in the table report the average performance across all clients, which can make the overall average improvements appear small, as mentioned in Sec. 6.2
As shown, alignment improves both personalization ('Self") and generalization ('Others') perfor- mance. This occurs because aligning both A and B in Co-LoRA ensures heterogeneous models share initialization and follow consistent optimization paths, allowing aggregation without weight
interference (Wortsman et al.] 2022a b. Yadav et al. 2023). In contrast, misalignment in either matrix introduces initialization discrepancies between models, leading to weight interference during
aggregation and degrading performance (Jordan et al. 2023 Neyshabur et al. 2020 Stoica et al.
2025..
Initialization
Self
Others
B
Atast
AAUC
Aiast
AAUC
Random Random
Random
67.95:11.71
63.50:10.66
47.71=10008
47.46:10005
Aligned (CCA)
1..2
63.59:10.2
47.80:10.06
47.56:10.08
Aligned (L2)
Random
68.29:11.57
63.58.10.00
47.72.10114
47.41100.4
Aligned (L2)
Aligned (CCA)
68.73.11.22
63.82.10.59
48.04:10.04
47.57.1008
Table 32: Accuracy on different weight alignment in Co-LoRA under Heterogeneous PFL-Static setup on DRAKE. 'Random' initializes matrices randomly, while Aligned (L.2) and Aligned (CCA) use matrices aligned by L2 loss and canonical correlation analysis (CCA), respectively.
A.22 COMPARISON WITH AIMILARIT--AAKK MODEL AGGREGATION
Federated Learning We are not the first to try to aggregate client models based on task similarity, but most existing approaches are not applicable to large language model (LLM) federated learning scenarios. pFedSim (Tan et al., (023) measures the classifier similarity among clients and uses it for weighted model aggregation, which is infeasible for LLM fine-tuning where the pre-trained classifier remains frozen. pFedHR (Wang et al. 2023) computes similarity between client models based on the output logits and performs similarity-awar layer stitching. However, its logit extrac- tions relies public data, which (as acknowledged by the authors) raises public data sensitivity concerns: the discrepancy between private and public data may leads to inaccurate similarity estimate. Flashback (Aljahdali et al. 2024) conducts dynamic knowledge distillation where teacher logits are adaptively weighted by wlient-wise label count, but this method is limited to single-dataset classi- fication FL tasks. There is also another approach like FEDLAW (Li et al.) 2023d) that adaptively aggregates models based on the optimized learnable weights instead of client similarity. However, it incurs high computational costs and remains highly sensitive to the choice of public data.
RELA, on the contrary, measures the task similarity among clients using sanitized last layer gradients. These gradients from each private data effectively capture task characteristies with minimal additional computation and communication cost, while preserving privacy through EMA updates, noise injection, and gradient compression.
Multi-Task Learning Federated learning (FL) and personalized federated learning (PFL) are not the only paradigms that learn heterogeneous tasks concurrently. Multi-task learning (MTL) trains single model on multiple tasks simultaneously in a single compute node, not in separate private nodes as in FL. MTL methods, such as NBS (Navon et al. (022) and Rotograd (Javaloy & Valera, 2022), leverage inter-task relationships to dynamically weight gradients, thereby mitigating interference and promoting balanced update direction across tasks. This is similar to the motivation of our aggregation method, i.e., RELA, which aims to reduce the interference between multiple tasks. However, these techniques are not directly applicable to personalized federated learning due to differing objectives: similarity- aware MTL strategies target generalizability by reshaping task- specific gradients to improve single shared model, whereas RELA focuses on personalization. It
42

---

Published as a conference paper at ICLR 2026
provides each client with customized global model by down- weighting contributions from irrelevant tasks, rather than enforcing a single globally shared one, thereby providing knowledge beneficial for personalization, while still enhancing generalizability
byproduct.
We also empirically compare RELA with MTL methods (Navon et al., 2022). Note that Ro- tograd(Javaloy & Valera 2022) is incompatible with PFL because it aligns gradient directions by optimizing rotation heads using samples from multiple tasks, which not feasible under federated data isolation and privacy constraints. Therefore, only compare with NBS, which determines task weights via Nash bargaining objective that maximizes total loss reduction. In PFL setting, we apply NBS weighting to adapter parameters rather than raw gradients for shared global model, since FL aggregates parameters. As shown in Tab. 33 NBS shows lower "Self" performance, while maintaining similar generalization ("Others') performance. We believe NBS's weighting mechanism focuses on harmonizing parameters from multiple tasks to minimize the overall loss, but fails to preserve or share task-relevant information necessary for each client's personal tasks, resulting in poor 'Self" performance.
Self
Others
Method
Atost
AAUC
Atuet
AAUC
FedMosaic w/ RELA
67.8610.51 66.8910.43
59.8310.15 58.98.1..00
51.1610.04 51.0711.009
49.36:10.08 49.09110.06
FedMosaic w/ NBS (Navon et al. 2022
Table 33: Quantitative comparison with MTL method.
A.23 EXTENDED FAST ADAPTATION EVALUATIONS
In addition to the fast adaptation evaluation in Sec 6.2 we further assess fast adaptation on additional unseen tasks from DRAKE. We summarize the results in Fig/23] As shown in the figure, models initialized with FedMosaic adapt significantly faster than those with random initialization or other PFL baselines, demonstrating that FedMosaic enhances generalizability by effectively sharing knowledge in heterogeneous PFL setups.
Random Init
SFT
FedIT
FedSIM
FedDPA
FedMKT
FedMosalc (Ours)
RecipeQA: ImageCoherence
Contrastive Caption:Seen Img
HQ-Edit
80
90
Accuracy (%)
Accuracy (%)
Accuracy (%)
80
60
40
70
40
60
20
50
0
50
100
150
200
0
50
100
150
200
50
100
150
200
Iterations
Iterations
Iterations
Figure 23: Comparison of adaptation speed. We use FedMosaic's unseen tasks as downstream tasks. Random init starts from randomly initialized models, while other baselines are initialized from aggregated local models trained on DRAKE using each respective FI. baseline.
A.24 EXPERIMENTAL RESULTS RANK HETEROGENEOUS PFL SCENARIOS
In the main experiments, we assume that all clients use the same LoRA rank r, which ensures the consistent dimensions for P E R and Q E RT in Co-LoRA across heterogeneous models. enables knowledge sharing across heterogeneous models, but it also imposes the constraint that all clients must adopt the same rank r. To support knowledge sharing among heterogeneous architectures with both different parameter dimensions and different LoRA ranks, Co-LoRA be readily combined with prior heterogeneous FL methods (Cho et al. 2024) that allow clients to use LoRA modules with heterogeneous ranks but shared dimensions.
43

---

Published as a conference paper at ICLR 2026
HETLoRA (Cho et al. 2024). HETLoRA proposes to align the rank dimensions of heterogeneous local LoRA modules by proo-padiing aggregate them with sparsity-based weighting, and finally distribute the aggregated update using truncation. For example, assume two clients I = {i have heterogeneous LoRA adapters, A; Rd B, R and A, Rdxr; Bj E RR*x where ri < r. HETLoRA first zero-pads the smaller-rank LoRA module to Tmax = max (r;, ry) e.g..
zero-pad Asli, Ti ['ma]] and Bi(r; Tmax, :]. Then, it aggregates the LoRA modules as
A=
Σ Ak. RET
B
ag
(24)
KET.
where the aggregation weights 0g are defined proportional to the Frobenius norm ||W lIr of the reconstructed full-rank update AW; AR. e.g., 0; Eaez AF Finally, the aggregated global LoRA module distributed to each client via dimension truncation to match each client's LoRA rank. For example, for client having rank < Tmax and LoRA matrix B; E RRdr,, the client keeps the first ry columns of the aggregated B matrix and discards the remaining 7max r4 columns.
||wW,
Combining HETLoRA with Co-LoRA We integrate HETLoRA with Co-LoRA to enable knowl- edge sharing among clients with both model and rank heterogeneity, For example, consider two clients I = ji with heterogeneous P and Q modules in Co-LoRA, Le., P; E TR": Q, E Rr and P) E R Q, E R"; with ri <rz. Similar to HETLoRA's A's aggregation process, we zero-pad P. Qi to rank Fiax r;), i.e., zero-pad P.[ra mmax Ti : T Tmax] and [r(r: Tmax]. With the rank dimension aligned P and Q modules, we aggregate them obtain the global module P and Q by combining Eq. 24 and RELA as
P. P.
0új P3: Q
Σ
dis Qj
Wija
(25)
n= O/
where Waj is the client-wise task relevance weight in RELA in Eq. Finally, the P and Q are distributed to each client via dimension truncation to match each client's rank, egg. truncating the Tmax T; rows and columns of P and Tmax т; rows of Q
With the combination of two orthogonal components, i.e., Co-LoRA and HETLoRA, which enable knowledge sharing under both model and rank heterogeneity, we evaluate setting where clients different models with heterogeneous LoRA ranks. Specifically, clients adopt heterogeneous models or LLaVA-Llama3 2-3B) and three different LoRA ranks {64, 128, 256). For a fair comparison, we also compare against other PFL baselines combined with HETLORA. As shown in Tab. 34 FedMosaic combined with HETLoRA significantly outperforms all baselines with the same HETL integration. These results highlight that Co-LoRA is not limited to heterogeneous architectures with a shared rank, but can also handle heterogeneous ranks across heterogeneous architectures when paired with existing hetero-rank LoRA FL approaches, further extending its applicability to real-world clients.
A.25 EXPERIMENTAL RESULTS ON NON-L.L.D HETEROGENEOLE PFL SCENARIOS
We further validate the effectiveness of FedMosaic on standard non-i.i.d personalized federated learning setups, where clients have different distributions of the same dataset, rather than each client tackling distinct dataset. To this end, we randomly choose 10 sub-tasks from DRAKE, merge them into a single dataset, and split into 10 unbalanced subsets by allocating a portion of each task using Dirichlet distribution with concentration parameter We then assign each subset to one of the 10 clients. Specifically, we use MagicBrush, VSR, and NLVR2 from the visual relation sub-group, 2 IRFLs, Hongard-HOl, and Bongard- World from the multi-modal reasoning sub-group and 2 IconQAs and Co-Instruct from the VQA sub-group in DRAKE. Following [Xe et al.] (2024); Morafah et al. (2024), we evaluate E (1.0,0.5.0. 0.1), with 20 rounds and 100 local steps. As
44

---

Published as a conference paper at ICLR 2026
Self
Others
Method
Atost
AAuc 1
Alast 1
AADC 1
SFT
65.35=+0.62
57.89=0.27
46.98:0.255
46.26++0.20
DITTO (CML 2021)
60.6810.09
54.69-0.13
47.2010.11
46.46:10.16
FedSim (CML 2022)
63.46=00888
56.57=0.31
46.03-4:0.29
45.58+0.12
FedIT (ICASP 2024)
65.15:1.771
57.82=0.43
47.02:10.49
46.12.10.35
TAKFL (NeurPPS 2024)
63.3410.84
56.59=0.13
47.52440.20
46,83:10.34
FedDPA (NeudPS 2024)
63.66.10.880
56.12:10.6
46,9711044
46,22:10.23
FedDAT (AAAI 2024)
58.90:10,62
55.30=0.20
48.02:10.17
47.22:10.01
PerAda (CVPR 2024)
60.29::0.43
54.46+0441
47.25++0.23
46.36.+0.29
FedMKT (COLING 2025)
61.35::0.41
55.07:-0.12
46.47:10.67
46.06:10.33
FedMosaic (Ours)
6..71++0.33
59.2
0..4
48.07++0.15
Table 34: Quantitative comparison in rank heterogeneous PFL-dynamic on DRAKE. 4 clients use LLaVA-Llama3.2-11 model, and 6 clients use -iLaVA--lamall 1aama32--3B model, with each client randomly choosing rank from {64, 128, 256). resulting in 3 clients with r 64, 5 clients with r= 128, and 2 clients with 256. 'Self" denotes evaluation client's 's own data, while 'Others' denotes evaluation on data from other clients. SFT refers to supervised fine-tuning on each client's data without cross-client knowledge sharing.
shown in Tab. 35 FedMosaic shows superior performance compared to PFL baselines even on traditional non-i.i.d. PFL, demonstrating its strong generalization capability. This highlights that the performance gains of FedMosaic in the main experiments do not arise from our data heterogeneity setup, but from the general effectiveness of FedMosaic.
1.0
a
0.1
Method
Aisst
AAUC
Atast 1
AAUC t
Atast
AAUC
SFT
66.77:10.05
62.38:10588
68.2511:40
63.73=1.04
71.40=0.99
66.90+10.7
DITTO
62.00+1.11
58.37:1049
62.3410.12
59.5
66.80=0.95
63.46:10.02
FedSIM
63.73400.3
59.82++0.27
67.44+0.33
62.77=0078
69.90=1.97
65.03+1.51
FedIT
66.89:10.77
62.57:1.32
67.6510.58
63.46=1.27
72.07=0.36
67.17-0.31
TAKFL
65.18:10.46
63.57+0.59
66.65-1.37
62.08=0.29
69.88=1.69
6..21
FedDPA
64.3410.27
60.3410.35
65.5812.722
61.91=1.72
68.56-0.27
64,58:10.03
FedDAT
62.9910.01
59.1910.09
63.01
59.95-0.65
66.43-0.14
63.16:10.2
PerADAA
61.73+0.61
58.20+00.68
62.88+1.09
59.77-0.76
66.41-0.61
63.27++0.28
FedMKT
64.87+0..3
60.67+0.12
65.14+0.10
60.81=0.39
69.04-0.83
64.78++0.3
FedMosaic (Ours)
69.12:10.39
64.72:10.02
7..20=1064
64.68=0.20
73.12=11.32
68.39-10.99
Table 35: Quantitative comparison in heterogeneous non-i..d. PFL on DRAKE. We conduct
PFL on Dirichlet non-i.i.d. splits with a E (1.0, 0.5,0.1), 0.13. clients use L...VA-Llama3..- model, and 6 clients use LLaVA-LJama3.2-38 model. We report the average accuracy across all clients.
A.26 EXPERIMENTAL RESULTS HETEROGENEOUS PFL WITH SMALL-SCALE MODEL.
In addition to our large-scale experiments with billion-parameti models in heterogeneous PFL scenarios, we also evaluate heterogeneous PFL using small-scale models. Specifically, we consider heterogencous PFL with model heterogeneity. where two clients use T5-small (60M) and the remain- ing two use T5-base (220M) (Raffel et al. 2020). For data heterogeneity, each client independently learns a different sub-task from the GLUE benchmark (Wang et al. 2019). To simulate distribution shifts, each client learns two sub-tasks sequentially: client I: MNLI -NNII-QQP, client 2: MRPC-<TEE, client 3: SST2=*QLL client 4: COLA-
We use the Adam W optimizer with a learning rate 3 x 10 -4 and batch size 32 for local training in all clients, updating only the LoRA parameters while freezing all treitriined weights. Following prior T5 model training frameworks (Wang al. 2024 Zhang et al. (0255), we set the LoRA rank to 32, LoRA alpha to 8. We evaluate two combinations of Tederated learning rounds and local steps and summarize the results in Tab. 36 As shown in the table,
45

---

Published as a conference paper at ICLR 2026
6 mounds, 100 local steps
20 rounds, 50 local steps
Self
Others
Self
Others
Method
Atset
Auue
Atoet T
AAte:
Auaa
Aare 1
Abst 1
Auuc
SFT
60.94++1 1.35
50.96-0.57
26.19-0.80
24.49-0.32
62.88-00.36
51.20++0.22
26.94+1.34
25.26+0050
DITTO
59.22.11.08
47.89.0.64
24.6810.14
24.29110.05
1.33
48. 48.99110.39
25.36.10.41
24.0110.18
FedSIM
60.6610.41
50.70=0.12
25.33=0.78
24.24=0.05
61.88:10.91
49.55:2.111
26.3910.66
23.9010.09
FedIT
60,53+0.53
50.76=0.04
25.98=0.62
23,93=0.36
62.34+0.64
50.85++0110
26.92+1.58
24.40=0.52
TAKFI
60.49++0.54
49.88+0.16 15
25.37=0.24
23.60+0.35
61.05++1668
49.89++0.57
26.02+00.09
23.66++0.07
FedDPA
61.04:10.48
50.1910.113
25.78.0.08
23.8410.24
61.47:10.6
49.09.1.332
25.77.10.58
FedDAT
51.13:10.27
43.2510.08
24.29-10.09
24.41:1.001
53.32:10.10
44.65.10.388
23.6910.40
23.3911006
PerADA
59.53=11001
47.99=0.60
24.69=0.22
24.31=0.06
60. 00:903
49.05:00.16
25.33++..13
24.11+0.30
FodMKT
60.29:00.45
49.14=00.4
25.41=00.42
23.63=0:21
62:21+0008
50.2 50.25:+0.68
24.82+0.28
23.27+0015
FedMosaic
71.20:1.54
55,28:1.11
40.33:00.04
31.20:1.600
78.24:1..14
60.32.10.02
37.1410.82
31.1210.33
Table 36: Quantitative comparison in heterogeneous PFL-dynamic with small-scale models. We evaluate heterogeneous PFL using small-scale T5 models, where T5-small serves as the smaller model and T5-base as the larger model the GLUE benchmark. Two clients use T5-small and two clients use T5-base. We evaluate heterogeneous PFL under two settings: 6 rounds with 100 local steps, and 20 rounds with 50 local steps. 'Self" denotes evaluation on a client's own data, while 'Others' denotes evaluation on data from other clients.
FedMosaic significantly outperforms baselines in both PFL settings. These small-scaa experiments demonstrate that Co-LoRA and RELA can be applied effectively regardless of model sizes.
A.27 ADDITIONAL DETAILS OF DRAKE
We curate DRAKE using open-sourced datasets from the Internet. We refer to diverse multi-modal benchmarks, such DEMON (Li et al. 2024c). SEED-Bench-2 (Li al. 2024a), Co-Instruct (Wu 2024b). and HFLB (Chnn et al. 2024a).
et al.
DRAKE consists of 40 distinct tasks, where for each task, we subsample 10,000 training and 1,000 test samples, if the dataset is large: otherwise, we split the data into training and test sets with an 8:2 ratio, totaling 375k images and 274k questions. We illustrate the overall structure of DRAKE as a tree diagram in Fig. [4] As shown in the figure, DRAKE consists of three sub-groups, i.e., visual relation, multi-modal reasoning, and VQA, each comprising multiple fine-grained tasks. We also provide detailed per-client task configuration of DRAKE in Tab. 37
Visual Relation Tasks Visual relation tasks focus on capturing relationships among visual objects, consist of 12 distinct tasks. We categorize them into three splits: fashion-relation, dual-image relation, and spatiaa/temporal. The fashion-relation split includes the cloth-counting and three-/four- image variants of Fashion200K (Han al.) 2017) as well as the query- reference caption split of FashionIQ (Wu et al. (021). The dual-image relation split is comprised of on NLVR2 (Suhr et al.)
2019), CIRR (Liu et al. 200T), the two- and four-image true/false subsets of VISION (Bai al. 2023).
and MagicBrush (Zhang et al. 2023a). The spatial/temporal split combines single-frame spatial reasoning in VSR (Liu et al. 2023a) with temporal understanding tasks from SEED-Bent -BEEhhhhhh2 (Li
al. 2024a)
Multi-modal Reasoning Tasks Multi-modal reasoning tasks require the integration of visual cues with common-sense knowledge, comprising 12 distinct tasks. We include four tasks from IRFL. (Yosef) et al. 2023) that examine the figurative interpretation of images paired with non-literal language. We also include five additional tasks derived from the positive-negative image groups in Bongard- HOI (Jiang al. 2022) and Samgard-OpenWWee
(Wu et al. (024d). These tasks require fine-grained discrimination between highly similar but contrasting visual sets. We incorporate three reasoning challenges from COMICS (Lyyer al.) 2017). MIT-States (Isola et al. 2015). and VizWiz (Gurari
et al. 2018), all sourced from DEMON (Li et al. 2024c).
VQA Tasks The VQA sub-group includes 16 diverse VQA tasks that differ significantly from LLaVA's pre-training datasets. We include novel question types, such as image-quality queries in Co- co-Inntruct (Wu et al. 2024b), textbook-style diagram interpretation in TQA (Kembhavi et al. 2017). DVQA (Kafle et al. 2018), and Knowledge-groundee QA tasks from SEED-Bench-2 (Li et al.
20248
We also incorporate unfamiliar image domains, including IconQA (Lu et al. 2021) and
WCVQA (Winata et al. 2024). We partition Co-Instruct by question type (e.g., Yes/No, How/what
46

---

Published as a conference paper at ICLR 2026
questions). IconQA by difficulty and question category (e.g.. Multi-choice Short answering, kindergarten-level or grade 1-level questions), while we divide WCVQA using their original split.
Client
Client 2
Fashion200K Han al. 2017
ClothCombination
Co-Instruct (Wu et al. 2024b
ImgCompare
FashionlQ (W et a. 2021
SEED-Bench-2 e al 2024a KGQA
VISION (Ba et al. 2025 Img
IconQA ct 2021
ShortAnswerrassy
MagicBrush Zhang ct 2023a
WCVQA (iiata al,
2024
DishName
Clientt
Client
Co-Instruct (Wu al.
2024b
HowWhat
Co-Instruct (Wu al.
2024b):
ImgCompare
DVQA (affi at.
SEFD (DEID-Bennh-22 [
2024a
InstanceQA
IconQA La
2021
Answeranw
IconQA (Lu
al 2021
Muin ChoiceEasy
WCVQA Winala et al.
2024:
ContexIDishName
WCVQA Winala al
2024
AdCComtexDDisthNaee
Client
Client
Fashion200K (Han et al.
3017: ColorConsistency
Fashion200K (Han et al.
2017): StyleConsistency
NL.VR (Suhr el al
2019
Subsetl
NLVR2 (Suhr el at.
2019
Sunset2
VISION (Ba el mi
2123
4_Img
CIRR (iu el al
2021
VSR (Liu at.
2023a
SEED JEED-Bench-2 Li al.
2024a]: TemporalQA
Client
Client
IRFL Yoseef et al. 2023
MetaphorSimilMMcchinn
IRFL (Yosel et al.
2023
Tigurative
Bongard- (Jang di 2022 ActionDetection
IRFL (Yoscl ct al.
2023
IdiomMatching
VizWiz GGurn 2018
Bongurd- Open World
20248]
ConcepiDetecttion
MIT- [loola et al
2015
COMICS [yyer al.
2017
Dialogue
Client
Client 10
IRFL (Yosef
2023 PhraseSelection
Co-Instruct (Wu al.
2024b YesNO
Bongard- (Wu et al. 2024d] ConceptQuery
TQA (Kembhavi et al.
2017
Bongard-HOF HOl [iang of am 2022
ConoeptQuery
IconQA (Lu et al. 2021
MultihhoiccHard
Bongard- HOI
Diang el al. 20222
ActionIncoherenc
WCVQA (Winla ot al.
2024 Location
Table 37: Per-elient task configuration of DRAKE. DRAKE consists of clients, each with
distinct tasks. Client 1, 5, 6 tackle visual relation tasks, Client 7, 8, 9 handle multi-modaa reasoning tasks, and Client 2, 3, 4, 10 focus on VQA tasks, as illustrated in Fig.
Unseen Tasks The unseen tasks include novel task types (DreamSim (Fu al. 2023b), Image- CoDe Krojer et al. 2022)) and novel visual domains with familiar task format (RecipeQA (Yagcioguu et al. 2018) subsets proposed in DEMON (Li et al.) (202cc). and HQ- --diit (Hui al. (2225) In addition, we split the Contrast-Caption subset of Mantis (Jiang et al.) 2024 Yu et al. (022) into two tasks: one with images overlapping VA's pre-training distribution and another with previously unseen images.
A.28 COMPARISON WITH OTHER FL BENCHMARKS.
number of federated learning (FL) benchmarks exist, but most lack critical aspects required for FL of mudti-modal foundation models, which DRAKE addresses, with the main differences summarized in T
Benchmarks like NonIID-50 (Yoon et al. 2021). LEAF-FCL (Qi et al. 2023). and MNIST- Shuffle (Wuerkaixi et al., 2024) simulate distribution shifts under FL. However, they are single- task and single-moda (i.e., image classification) only, thus cannot reflect real-world task diversity. HC-FMTL (Lu et al.) 2024) spans multiple vision tasks, such as depth estimation and semantic segmentation, but still remains unimodal.
On the language side, several text-only benchmarks have recently been proposed to target LLM federated learning scenarios. Fed-SNI (Collins et al. (023) and Fed-FLAN (Long al. 2024) cover diverse NLP tasks, while Fed-Aya (Ye et al.) 2024) focuses on multilingual instruction following. FEDLEGAL (Zhang al. 2023b) curates FL benchmark for the privacy-sensitiv legal domain where federated learning is necessary. Despite their breadth, they are also limited to a single modality and assume static data distribution, while data distribution often shifts over time in real-world.
For modltl PFL, we find only one prior benchmark: HFLB (Chen et al. 2024a). Although HFLB includes 7 different datasets, the multi-modal task diversity limited, and they mostly
47

---

Published as a conference paper at ICLR 2026
Sub-group
Fashlon200K Cloth Counting
Fashion Relation
Fashion200K Style T/F
Fashion200K image Color
FashioniQ Query-Rolerence Capton T/F
NLVR2 Dual Image Relationship Check
CIRR Transition Caption T/F
Visual Relation Tasks
Dual image Relation
image
VISION 4 Image T/F
MagicBrush Image Edit Instruction Direction Check
Spatial/Temporal
Spalial Understanding SEED-Bench2
vSR
Understanding
Understanding
Sub-group
IRFL
Figurative Verilication T/F
Visual Figurative Understanding
IRFL
Imagee Selection for Idiom
IRFL
Image Selection for Metaphorl
Simile
Selection Image
Bongard-OpenWorid Posittee/Negative
Concept Query
Longdard-OpenWWern
Federated Training Tasks
Multi-modal Reasoning Tasks
Context-dependent Visual Reasoning
Concept Detection
Bongard-HOI Postlive/Neaalive
Concegt Cuery
Bongard-HOi Common Action Detection
Bongard-HOI Incoherence
Recognition
Mult--maag--exx Understanding
COMICS Dialogue prediction VizWiz Question-Answer Difference MIT-States State Recognition
Sub-group 3
Co-Instruct- How/What QA
Co-Instruct- Yes/No QA
Image Quality QA
Co-Instruct Image Quality Compare
Co-Instruct 3 Image Quality Compare
DRAKE
QA
TOA Texibook QA
General VOA
DVQA Barchart QA
SEED-Bench2 Instance-Leve QA KGDA
VQA Tasks
IconQA - Multi-Cholce QA (prek &
Kindergartan)
IconQA Mult-Chhize QA
grade
Icon-domain QA
IconQA Short Answering QA (prek
kindergarien)
IconQA Short Answering QA (grade &
grade
WCVOA Dish Name Prediction
WCVQA Contextualized Dish Prediction
Cuisine-domain QA
WCVQA Dish Name Prediction with
Achersarial Context
WCVOA Location Prediction
DreamSim - Similar Image Matching
Contrast Caption Capfion-man Iieoon-mmgee
Matching (Seen image domain)
New Task Type
ImageCoDe Sequential Image
Unseen Tasks
RecipeQA Incoherent Image Detection
RecipeQA Next Image Prediction
New Visual Domain
Contrast-Captie Catsticn Criptio-lmag
Matching (Unseen domain)
HQ-Edit Image Check
Direction
Figure 24: Detailed configuration of the proposed DRAKE benchmark.
single-image settings. Other multi modal FL benchmarks, e.g., FedMultimodal (Feng et al.) (023)..
FHBench (Wang et al., 2025), and FedMLLMM (Xu et al. 2024), primarily comprised of only
48

---

Published as a conference paper at ICLR 2026
classification task different modalities split two VQA datasets, which is insufficient for evaluation personalization under diverse heterogeneous tasks. Importantly, existing benchmarks generally lack an unseen task split test the generalization ability of the model.
In contrast, DRAKE offers 40 different multi-modal tasks (including multi-image tasks), which is suitable for personalization evaluation and temporal distribution shift. Moreover, DRAKE additionally provides 7 unseen tasks for out-of -iutt---iittibttiion generalization evaluation.
A.29 JUSTIFICATION FOR USING SMALLER MODEL AS THE PIVOT
During the Co-LoRA alignment, we freeze the smaller model as pivot because, over time, new models that become available are typically larger and stronger, rather than smaller and weaker ones. In this scenario, using the larger model as the pivot would cause the pivot to change whenever a new model introduced, requiring re- aleinnmmennt of all previously aligned models. For example, when aligning ILLaV-1B IB and LLaVA-3B, if we freeze the larger model (ie., LLaVA-3B) as the pivot, the introduction of ILLVV--8 would make the new pivot. Consequently, both L.LaVA 1B and LLaVB would need to be updated during alignment with LLaVA-8B, which would break the previously established alignment between VA- IB and LLaVA-3B.
Moreover, we note that this strategy does not affect performance. As shown in Tab. 38 employing the larger model as the pivot yields comparable performance to our strategy. In summary, we adopt this approach not because it offers better performance, but because it provides stable pivot for alignment when larger models are newly deployed.
Self
Others
Co-LoRA alignment scenario
Pivot model
Awax f
Aaue 1
Atuut f
AAUC 1
small (1B)
6..86:10.51 67.7110.27
59.83:10.15
51.16110.04
4..36:20.08
I.aVA-Llama3..2 IB aLL.V VA J.mma3.2-333
large (3B)
59.5410.05
0..46:0.37
49.29:20119
small (1.5B)
70.62++0.29
63.33++0.6
52.56:0008
50.76+0.01
LLaVA-Qwen2.5-1155/ Bttp://2........LLLLAAL-LL. LLLVAALLlaaa..225
large (3R)
7..83++0335
63.40++0.25
52.55+00221
5 0.01
Table 38: Efffec of pivot model selection for Co-LoRA alignment in heterogeneous PFL-dynamic on DRAKE, During the alignment process, we freeze the pivot model and update the other model to align with The accuracy remains consistent regardless of the chosen pivot model. 'Self' denotes evaluation on client's 's own data, while 'Others' denotes evaluation on data from other clients.
A.30 PRIVACY GUARANTRE OF SANITIZED GRADIENT ğ
FedMosaic requires transmitting per-client gradient vectors to the server for RELA, which may expose clients to privacy attacks such as gradient inversion (Mo et al. 2021). To mitigate these concerns, employ (i) using only the last-layer gradient, (ii) exponential moving average (EMA) updates, and (iii) gradient sampling with noise addition for sanitization. To assess the privacy protection of our sanitized gradient g. we evaluate its resistance under a strong gradient inversion attack, LAMP (Balunovic et al., 2022). LAMP is designed to reconstruct text inputs by searching for an text sequence that produces the same gradient the target gradient, alternating between continuous and discrete optimization steps. In the continuous optimization step. LAMP updates the text embedding so that its gradient matches the target gradient. the discrete optimization step, maps the continuous optimized text embedding back to tokens, generates multiple candidates by applying operations such as token swapping or moving, and then selects the best candidate with the lowest perplexity to update the text embedding. By iteratively alternating between these two steps, LAMP gradually recovers an input that is both close to the target gradient and linguistically natural.
Under this attack, we compare our sanitized gradient against several recently proposed defense approaches for shared gradients in the (personalized) federated learning domain, such as Clipping + Noise (Bietti et al.) 20022, SVD truncation (Luo al. (2025). Orthogonal projection (Zhang et al.) 2025a), and Fisher Information (Jhunjhunwala et al. 2024). Clipping + Noise clips the gradient norms and add Gaussian noise to guarantees differential-prrvacyy SVD truncation retains only the
49

---

Published as a conference paper at ICLR 2026
Defense Method
Rouge-L
Full layer gradient
0.2952
Clipping + Noise (Bietti et al. 2022
0.2720
SVD Truncation (Luo et al. 205
0.2688
Fisher Information wJhunhhnnaaa et al 2024
0.2128
Orthogonal Projection (Zhang et al. 2025a
0.1614
Last layer gradient
0.1442
Last layer gradient + EMA aggregated
0.0844
Last layer gradient EMA aggregated Noise
0.0699
Last layer gradient EMA aggregated Noise Compression (sanitized gradient g:)
0.0653
Table 39: Comparison of privacy-defense methods under gradient inversion attack. We report
the recovered text Rouge-L. scores, where lower Rouge-L. indicates better protection against privacy leakage from gradient inversion attacks. We set the batch size 4, noise scale ja to 10-A, and compression ratio to 40%.
Privacy-Accuracy Trade-off on noise scale μ
0.1
# 1e-6
Rouge-L4
0.08
A le-5
A le-4
0.06
P 1e-3
P le-2
0.04
0.04
67.25
67.5
67.75
68.0
Alast
Figure 25: Privacy-Accuracy trade-off on noise scale μ. Lower Rouge-L score indicates better protection against gradient inversion attack, and higher Atast indicates better PFL performance. Increasing noise scale leads to stronger privacy protection but incurs accuracy drop.
principal components of the gradient and Orthogonal projection projects normalized random vector on the gradient subspace, rather than full gradients. Jhunjhunwala al. (2024) use Fisher information to weight local modules while mitigating privacy concerns, approximating with the diagonal of the Fisher Information Matrix, i.e., the squared gradient, which discards the sign information.
We summarize the results of LAMP the Llama3.2- 3BBBBBBBBBBBBBB model using GLUE-COLA dataset in Tab. 39 reporting the Rouge-L score of recovered input texts. As shown in the table, our sanitized gradient effectively protects private information from the attack compared baselines. Specifically, using only the last-layer gradient significantly reduces reconstruction success compared to using full-layer gradients well as other defense baselines, demonstrating that last-layer gradients not only reduce communication and computation cost but also mitigate privacy risks. Moreover, EMA-based gradient accumulation provides additional protection by mixing gradients along the temporal axis. Finally, noise addition and compression for sanitization further obscure input information, making sanitized gradient ý substantially more resistant to privacy attacks.
Finally, we analyze the privacy-accurant trade-off noise scale As shown in Fig. 25 adding more Gaussian noise hinders text recovery, but increasing the noise level also degrades performance by reducing the representativeness and informativeness of the gradients. Based on this privacy-accuracy trade-off, we adopt moderate noise scale (1 10 10-4) in RELA.
A.31 CLIENT-WISE TASK RELEVANCE VISUALIZATION
To analyze task relevance within DRAKE, compute task-relevance matrix using the sanitized gradients 9 employed in RELA. For ease of visualization, tasks belonging to the same group are placed adjacently. Specifically, each task corresponds the clients listed in Tab. 37 as follows: tasks
50

---

Published as a conference paper at ICLR 2026
Task Relevance Matrix
1.0
0
0.8
Task Relevance
3
Task index
0.6
4
5
0.4
6
7
8
0.2
0
2
3
4
5
6
7
Task index
Figure 26: Task Relevance Matrix of DRAKE. The task relevance is measured by sanitized gradient from PRAKE-SStaii setup. Each task is the per-client task summarized in Tab. [37] where 0-2 tasks tackle visual relation tasks, 3-5 tasks handle multi-modal reasoning tasks, and the rest focus on VQA tasks. Tasks within the same task-category exhibit notably high mutual relevance.
0-2 (clients 1, 5, 6) correspond to visual relation, tasks 3-5 (clients 7, 8, 9) to multi-modal reasoning, and tasks 6-9 (clients 4, 10, 3, 2) to VQA.
As shown in Fig. 26 tasks within the same group exhibit noticeably higher mutual relevance than those across different groups. Moreover, tasks 0-5 show relatively high cross-group relevance, likely because they all involve multi-image inputs, whereas tasks 6-9 primarily use single-image inputs, and this shared input structure contributes to their similarity.
A.32 EXPERIMENTAL RESULTS ON HETEROGENEOUS PFL WITH ASYNCHRONOUS DISTRIBUTION SHIFT
We simulate data distribution shift in PFL (i.e., dynamic setup) by sequentially introducing new task for the fixed number of rounds. Thus, all clients experience the distribution shift simultaneously (i.e., at the same round) in the experiments. For instance, in both HFLB-Dynamic and DRAKE-Dynamic experiments, each client learns 4 tasks sequentially every 5 rounds. Although FedMosaic shows its effectiveness under syncrhonized distribution-shin can naturally be extended to an asynchronized distribution-shift setup, where distribution shifts occur at different times across clients. To demonstrate that FedMosaic works robustly regardless of the timing of distribution shifts, we evaluate FedMosaic under the asynchronized distribution shift setup. Specifically, instead of assigning every client fixed rounds per task, each client randomly adopts one three rask-round patterns: (5,5.5,5), (3.7,3,7), or (7,3,7.3), where each number denotes the number of rounds allocated to each task in order. As shown in Tab, 40) FedMosaic consistently outperforms other baselines even under this asynchronous distribution-shin scenario, demonstrating its stability to heterogeneity in the timing of distribution shifts.
20 rounds, with distribution shift occurring
A.33 LIMITATIONS AND FUTURE WORK
While evaluate FedMosaic across diverse molti-modal FL benchmarks (e.g., HFLB and our proposed DRAKE) and text-only NLP benchmarks (e.g., Fed-Aya, Fed-Scope, and Fed-LLLM-argee), Large), has yet been tested other modalities such speech and time series. Extending evaluations
51

---

Published as a conference paper at ICLR 2026
Self
Others
Method
Atost
AAuc
Alast 1
AAUC 1
SFT
66.23:40.26
08.35=0.28
46.9440.34
46.36:+0.28
DITTO (CML
60.1110.88
54.60-0.15
47.27:00228
46,43:10100
FedSim (CML 2022)
64.71=0.49
57.62=0.25
46.44:10.13
45,91
FedIT (ICASPP 2024)
66.77:10.25
58.71=0.24
46.92:10.19
46.32::0.15
TAKFL (NeurPPS 2024)
65.47:10.95
58.07=0.52
47.1440..3
46,46:10.02
FedDPA (NeudPS 2024)
62,40:10.82
56.26=0021
47.33:10.24
46.53:10.12
FedDAT (AAAI 2024)
59.06:10.35
54.79=0.21
48.90:10.09
47.83110.06
PerAda (CVPR 2024)
59.82++0.39
54.49+0.16
47.18-10.17
46.43-+0.06
FedMKT (COLING 2025)
61.76:10.06
55.79:0.15
46.76110.18
46.30:10.07
FedMosaic (Ours)
68.30--0.45
6.023+0.07
01.96
49.63-+0.10
Table 40: Quantitative comparison in heterogeneous PFL-dynamic on DRAKE with asyn- chronous distribution shift among clients. 4 clients use LLaVA--lamee VA-Llama3.22 model, and clients use LLaVA-Llama3.2 2-3B model, with each client randomly choosing one of three task-round patterns: (5,5,5,5), (3,7,377), or (7,3,7,3), where each number denotes the number of rounds allocated to each task order. 'Self" denotes evaluation on a client's 's own data, while 'Others' denotes evaluation on data from other clients. SFT refers to supervised fine-tuning each client's 's data without cross-client knowledge sharing.
these domains would better capture real-world FL scenarios and represent an important direction for future work.
Moreover, although LoRA is most commonly used in transformer- based models, it restricted to them. Specifically, prior studies (Ych et al.) [023] Ding et al.) (0224) have demonstrated its applicability convolutional layers, highlighting its sccitecture-agnossee Co-LoRA beyond transformers enable knowledge transfer across broader model families represents an important direction for future work.
nature. Therefore, exploring the use of
A.34 IMPACT STATEMENTS
This work focuses federated learning, which enables model training in distributed manner, eliminating the need to share raw data and thereby mitigating data privacy risks. However, since we assume that the model is trained under distribution shifts, there is a potential for the unintended amplification of issues such as model bias and ethical misalignment. We are committed to taking all necessary measures address these risks, although tackling these concerns is not the primary focus of this work.
A.35 DETAILED ALGORITHM OF FEDMOSAIC
Algorithm provides a pseudocode for the proposed FedMosaic framework. further provide detailed algorithms for initializing adapters (Algorithm 2) and aligning adapters across heterogeneous clients (Algorithms) and
52

---

Published as a conference paper at ICLR 2026
Algorithm I: FedMosaic
Input
Number of clients N, Number of cl client model types K, Set of frozen pre-trained models W (,,,,,, Wk.,
Mapping of client to model V [I,... - wW Number of zounds
R,
Batch size B, Learning rate " Gradient frequency f, EMA ratio a, Temperature T, Gaussian noise std H4, Gradient subsample ratio N,
Small-scale pre-trained model Ws, Regularization coefficient X
Training data streams of tasks for clients (D,, D,,..,, where D; (D), (....,, Number of Co-LORA blocks NBr Public dataset Dpr Conventional--ioRA, Co-LoRA
Iniriaallreedepterssfooeeee (W, NB, K) # (Alg. AlignAdapter (W, E, Ppr K, X) # (Alg. Alg.
Initialization for each client
for in range (N) do
M[i] = [] # Initialize episodic memory
Initialize local and global adapters with the aligned adapter
L[i] E[i]
G[i] E[i]
Initialize gradient vector with zeros
glil = torch.zerog (d_ (V
Initialize gating parameters for each layer in L[i] betali] [0.0 for in range (1en (I(i)))]
Set Random subsample indices for last layer gradient gubsample_index random_chon ce (last_layer_grad size, N,)
for task t in range (T) do
for round r in range (R) do
client-side
for client index in range (M) do
Wii = -(i)
g_t = []
for j, (x_j, 1) in D do
M[i]. append (x_j, y-5))
x, y = random_choice (M[il, B)
Compute loss with mixed local-global adapter
loss = CrossEntropy ( L[i], G[i], betali),
x), y)
L[i] - "* Grad(([)) loss)
i 8 f 0 then
loss = CrossEntropy (Forr (Ws, x), y)
gasa LastLayerGrad (Ws, loss)
# Subsample gradient and add gaussian noise
eps N(O, I plast_layar_gree g_sanitized q last eps tssuusammle_indde. append (g_sanitized)
g[i] - (1 a) gii]
(g_at)e(__t)
Server-side
Aggregate global adapters using SIMA
SIMA (L, g)
53

---

Published as a conference paper at ICLR 2026
Algorithm 2: Intttallcehdaeeeasoooobbttts....
7/ Initialize the list adapter lists for each model
= []
for k in range do
// Initialize adapter list for model W4
E_k []
// Block size per model (Co-LORA assigned)
B_k len (W[]] // NB
// Remaining layers not evenly divisible
B_r len (W[k]) NB
for 1 in range (1, len (W(k)) do
ir3 n in (1..NB) s.t. l= n* k Br then
// Append Co-LoRA
R_k.append Co-LORA)
else
// Append conventional LORA
k. append (Conventional-1034)
// Add adapter list to full collection E. append (E_k)
return E
54

---

Published as a conference paper at ICLR 2026
Algorithm 3: AlignAdapter (W, E, Dpr K, A) Part
Define hook functions to capture outputs during forward
pass
Function hook__n (module, input, ouput)) (output)
L wel...ora__uttuuss.apppp
I+# Combine Co-LoRA with the pre-trained model
E indices = []
for range (K) do
E __index = []
for in range (len (W(1))) do
[[i][j] =- Co-LoRA then
# Attach Co-LoRA to pre-trained red model layers
[[i][i] (Etach(B(i) (ii](i))
E E_index append (j)
E E_indices append (E [E_index)
### Align Co-LORA A matrices W_P w[0] Set pivot model
W_p. freeze() Freeze pivot model
for in range K) do
Wi W[i]
## Step 1. Attach hooks to LORA A and perform forward pass # Register hooks to capture the outputs of LoRA A matrix at Co-LoRA attached layers for in E [ndices do
L [_p[i]] rww.s....rr.rr.oooooooe..
(hook_ fn)
for in E_ []i do
L W_ilil (alii...ree wewg..rterrforwwaaeeeeeeeeeeeeee hook (hook_ fn)
#4 Step 2. Alignment of A using L2 loss total_loss
for (x, y) in D, do
W for.aorwrddwiee with W wtt.foowrrdwwth_oooa__eaaa total_loss
scale-0)
(x, scale=0)
Compute L2 loss for each layer using outputs captured by hooks
for in range (len outruts [i])) do Compute L2 between LoRA A outputs of W_i and W_p at
layer j
#Li.lora_outpe [i]) loss_12 = L2 (Mwp..oora
Compute Regularization loss to prevent deviation from orthogonality (Eq. 21
loss_reg reg Reg [i]il
total_loss loss_12 À loss_reg
i - Grad(WWi. total_loss,
## Step 3. for in L_indices [i] do
A to enforce Orthogonality
Use singular Value Decomposition to get the closest orthogonal matrix
U, S, Vt SVD (W_i [i]
W_i(i] A U+Vt
55

---

Published as a conference paper at ICLR 2026
Algorithm 4: AlignAdapter E, Dyr K, A) Part
#FF Align Co-LoRA B matrices
for in range (1, K) do
W_i W[i]
## Step 1. Attach hooks to LORA B and perform a forward pass
# Register hooks to capture the outputs of B in Co-LoRA attached layers
for in E []] do
L [_p[j]_B. for reg.ssegggstteefoorrddddd.o (hook_fn) fn)
F [1] do
L W_i[j] rww.sterrrooowarr.hoeeeeeeee (hook_fn)
Forward pass to collect outputs of the LoRA B matrices for CCA
for (x, y) in D, do A...forwrd. with with lotth_oorr_ccaeedd scale=0) Lo
scale=0)
Step Apply CCA to align matrices
for in range (len (W_1)) do
Extract B matrix outputs from hooks
X = W w.p.lora_outputst]]
X -d==__i.ooaee [_i.lora__ottuuss[]]]
# Compute CCA projection vectors
II, 1 CC X_i
Initialize using the CCA transformation
W_i[j].B = (1) (ILp) W_pli] .B
Lalignn
for in range (K) do
for in range (len (W(i))) do E[i][j] == Co-LoRA then L E[i]lil] W[il[i]
E_align (E
return E_align
56


Reviews:
[{"review_id": "wcYOy4PrIP", "reviewer": "Reviewer_GXyk", "summary": "The paper considers personalized federated learning (pFL) of multimodal large language models (MLLMs) under realistic scenarios that involve not only data heterogeneity, but also model architecture and model family heterogeneity, and task diversity. The paper designs a new method called FedMosaic that enables FL style collaboration across clients even in the simultaneous presence of all of the heterogeneities. FedMosaic has two important components - RELA (Relevance-guided Aggregation) and PQ-LoRA (Dimension-invariant Low-Rank Adaptation) that respectively address (data, task) and model heterogeneities. In an effort to make evaluation more realistic, the paper also introduces a new benchmark called DRAKE that incorporates all these heterogeneities and further includes aspects like dynamic distribution shifts and unseen tasks. Extensive experimental evaluation is provided in the paper for FedMosaic as well as several other state-of-the-art pFL baselines, which establish superior characteristics of FedMosaic.", "questions": "(Q1) Section 4.2.1 Figure 4, and Appendices A.12, A.17: Even though supporting empirical evidence is provided, I don't understand why layer correlations should exist across model families (Llama, Qwen, etc.). Is this exclusively caused by the common training data source from which $D_P$ is sampled? PQ-LoRA would only work if such correlation exists, right? How should one think about the system when common subset from pre-training/post-training data may be unknown/may not exist/may be inaccessible?\n\n(Q2) Line 73, 210: Does the system require a separate model instance on the server for each client? If yes, is that scalable to large number of clients? If no, what do experiments suggest about observed number of model instances on the server per client, across datasets of interest?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(S1) The writing and presentation of the paper are very clear in terms of both algorithm design and experimental results. Adequate intuitions are provided throughout the paper and appendices. The problem is well motivated, related work is well cited, and the contributions are contextualized appropriately.\n\n(S2) FedMosaic is an original and interesting solution to a very complex practical problem of multiple heterogeneities in pFL of MLLMs. This is a significant contribution to the field in terms of both ideas and solutions. RELA and PQ-LoRA would likely find use in other problems too.\n\n(S3) The supporting experimental evidence provided in the paper and appendices is quite exhaustive and impressive. The paper undertakes a wide diversity of studies to establish the characteristics of FedMosaic from several angles and shows competitive or improved performance w.r.t. all compared baselines.", "weaknesses": "(W1) Introducing a new benchmark in an algorithms paper is counterproductive. The benchmark would be difficult to discover for any reader. To a reviewer, the benchmark's design is impossible to evaluate when only 10 lines can be allocated to it in the main body. While DRAKE looks extremely useful, there are several nuances which can only be understood by carefully reading multiple sections in the appendices. My opinion is that DRAKE should be submitted as a separate datasets & benchmarks style paper for it to be properly peer-reviewed as such.\n\n(W2) There is no benchmark called HFLB in (Chen et al., 2024). The name/citation should be corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KDmxa8Pi7a", "reviewer": "Reviewer_pywM", "summary": "This paper aims to address the challenges of personalized federated learning in scenarios where both data and models are heterogeneous. The authors proposed a framework named FedMosaic, which consists of two core components: the RELevance-guided Aggregation (RELA) and PQ-LoRA. RELA is a task-relevance-based model aggregation strategy that constructs customized global models for clients. PQ-LoRA is a module shareable across heterogeneous models, addressing differences in model depth and dimensions through \"block-wise aggregation\" and \"weight alignment\". Additionally, the authors propose DRAKE, a comprehensive multimodal federated learning benchmark that covers 40 different tasks and simulates real-world task diversity and temporal distribution shifts. Experiments on both multi-modal and text-only benchmarks demonstrate that FedMosaic outperforms PFL methods in both personalization and generalization.", "questions": "Please clarify my concerns in the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, this paper represents a meaningful problem in personalized federated learning. The authors find that existing personalized federated learning methods are still confined to simplified scenarios with highly homogeneous data and models across clients, while real-world scenarios are more complex. They proposed FedMosaic, which addresses the simultaneous heterogeneity of data and models through a task-correlation-aware model aggregation strategy and dimension-invariant modules. Additionally, they introduced DRAKE, a comprehensive multimodal federated learning benchmark.", "weaknesses": "**Major Weaknesses:**\n\nOverall, this paper has some merits, but there are a few weaknesses that stop me from giving a higher rating. My major concerns are as follows.\n\n(1) The paper mentions that the FedMosaic method does not require high computational costs, and the authors' experiments indeed include sections related to computational costs. However, the process of weight alignment in PQ-LoRA seems to be relatively complex, and the paper does not provide information about the computational costs of this part.\n\n(2) Section 4.2.1 of the paper mentions using CKA to find relative depth alignment and demonstrates with Llama-1B and Llama-3B, but lacks sufficient explanation regarding the applicability of this method.\n\n(3) In the weight alignment in the PQ-LoRA section, it mentions freezing the smaller model as a pivot and updating the larger model. The paper lacks an explanation for why this strategy was adopted.\n\n(4) DRAKE is one of the contributions of the paper, but extensive details are provided in the appendix, with relatively limited space allocated in the main text.\n\n**Minor Weaknesses:**\n\n(1) Figure 2 provides an overview of FedMosaic, but the image is relatively dense and slightly lacking in readability, so it could be adjusted a bit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MMSxQbUiBQ", "reviewer": "Reviewer_UdQR", "summary": "This work addresses both data and model heterogeneity in Personalized Federated Learning (PFL). The authors propose FedMosaic, a framework that jointly mitigates these challenges through two core components: RELA and PQ-LoRA. RELA (Relevance-guided Aggregation) constructs client-specific global models by weighting updates based on task relatedness, enabling effective knowledge sharing among similar clients while reducing interference across unrelated tasks. PQ-LoRA introduces dimension-invariant low-rank adapters whose parameters depend only on rank $r$, allowing efficient and architecture-agnostic knowledge sharing across heterogeneous models. To more accurately capture real-world task heterogeneity and distribution shifts, they further introduce DRAKE, a comprehensive multi-modal federated learning benchmark.", "questions": "1. In the related work section, most PFL citations are listed without discussion. It would be helpful to briefly summarize the current state of the field: What approaches do recent state-of-the-art methods adopt, and what limitations does FedMosaic specifically address beyond them?\n\n2. The preliminaries conclude with a PFL objective, but the formulation of the global model objective is unclear. How does the given objective differ from the standard local objective, and why does it include terms dependent on other clients’ models?\n\n3. In Equation (1) and Figure 2, are the gradients computed with respect to the frozen weights $W_s$?\n\n4. The rationale for computing only the last-layer gradient (based on the proportionality of preceding gradients via the chain rule) requires further justification or empirical support. Are there results related to it in the appendix?\n\n5. The paper mentions that gradients $g_i$ are computed every $m$ batch iterations rather than every batch. What is the observed accuracy trade-off with and without this optimization?\n\n6. For PQ-LoRA, does the method assume that all clients use the same low-rank dimension $r$? If so, the approach still enforces a degree of architectural homogeneity. Given that the core challenge is model heterogeneity, how can we justify $P$ and $Q$ modules remaining dimensionally the same?\n\n7. How are the LoRA parameters $A$ and $B$ trained?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Experimental results demonstrate consistent improvements over strong baselines, indicating the effectiveness of the approach.\n\n2. The appendix further provides a thorough and extensive suite of experiments, supporting the validity and robustness of the reported findings.\n\n3. The paper is generally well-written and clearly organized, making the technical ideas easy to follow.", "weaknesses": "1.\tThe comparison against prior works using non-IID splits of a single dataset may not be entirely equitable. The contextual settings differ significantly (maybe the earlier studies targeted models specialized for single-domain or unimodal tasks, rather than fine-tuning billion-parameter foundation models). Moreover, the motivation for exploring multi-modal tasks in PFL requires further clarification. What are the practical or deployment-oriented use cases where clients naturally possess distinct modalities? At present, the setup appears somewhat hypothetical, with each client operating on different data and architectures. In such a scenario, the incentive for federated participation is not clearly articulated.\n\n2. The novelty of RELA is not fully evident. The client-wise gradient update formulation $\\hat{g}_i^{(t)} = (1 - \\alpha) \\hat{g}_i^{(t-1)} + \\alpha g^{(t)}_i$ closely resembles a first-order exponential moving average (EMA), similar to adaptive optimization methods such as Adam. Furthermore, the addition of a sanitization or noise component introduces privacy-related implications that warrant more rigorous analysis. If differential privacy–like noise is applied, the paper should evaluate its robustness against gradient-based privacy attacks and report accuracy trade-offs with and without the noise injection.\n\n3. The paper attempts to address multiple orthogonal challenges simultaneously (data heterogeneity, model heterogeneity, privacy), which can dilute the focus of the contribution. A clearer ablation or modular analysis could help isolate the effects of each component. Currently it's unclear how impactful the \"computing gradients at every $m$ batch is\" or how impactful (accuracy-wise) the sanitized gradients are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q69Slli89P", "reviewer": "Reviewer_oYUk", "summary": "The paper tackles personalized federated learning (PFL) under realistic heterogeneity: data heterogeneity where each client has distinct multi-modal tasks with temporal shifts, and model heterogeneity where clients use different model families and sizes. It introduces FedMosaic, which combines relevance-guided aggregation and PQ-LoRA to enable selective knowledge sharing and cross-architecture parameter sharing.\nThe authors also release DRAKE, a multi-modal PFL benchmark with 40 tasks spanning VQA, visual reasoning, visual relations, including multi-image inputs and unseen task evaluation under distribution shifts.\nEmpirically, FedMosaic outperforms strong baselines across heterogeneous/static/dynamic and cross-family settings on Self (personalization) and Others (generalization), and improves fast adaptation on unseen tasks; ablations show both RELA and PQ-LoRA contribute meaningfully.", "questions": "- I'm curious about the task relavance between the tasks in your DRAKE benchmark, like showing the relavance matrix.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Well-posed problem + realistic setup. The paper motivates that most PFL work oversimplifies heterogeneity; here, clients differ in both data and model (families and depths/sizes), which is closer to practice (agentic AI, device constraints).\n\n- Clear algorithmic design. The proposed method comes with clear motivation and design solution accordinly. For instance, RELA computes client-wise gradients on a small frozen model. It applies EMA decay to track shifting client knowledge and also adds Gaussian noise + random subsampling for privacy and bandwidth.\n\n- Benchmark contribution. DRAKE covers multi-modal, multi-image tasks, temporal shifts, and unseen evaluation; the table contrasts prior FL benchmarks along these axes.\n\n- Strong and granular evidence. The experiments are comprehensive with different settings like heterogeneous (same-family) PFL and cross-family heterogeneity, also analyize the performance from per-client view and fast adaptation possibility. Detailed ablations about adding PQ-LoRA improves Others, adding RELA further lifts Self/Others, are also provided.", "weaknesses": "While RELA applies EMA, noise, and subsampling to the last-layer gradients from privacy perspective, an explicit comparison to baselines with stronger privacy guarantees would strengthen the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)