Flick: Empowering Federated Learning with Commonsense Knowledge
Authors: Ran Zhu
Mingkun Yang
Shiqiang Wang
Jie Yang
Qing Wang
Venue: NeurIPS 2025 poster ()
Keywords: Federated Learning
Data Heterogeneity
Primary area: other

Abstract: Federated Learning (FL) has emerged as a privacy-preserving framework for training models on data generated at the edge. However, the heterogeneity of data silos (e.g., label skew and domain shift) often leads to inconsistent learning objectives and suboptimal model performance. Inspired by the data-driven approach, we propose Flick, a novel data generation framework for heterogeneous **F**ederated **L**earning w**i**th **C**ommonsense **K**nowledge from Large Language Models (LLMs). In Flick, the client performs the local data summary to capture client-specific knowledge in textual form. The central server then distills task-relevant, high-quality knowledge from the out-of-the-box LLM -- guided by cross-client-specific insights -- to generate informative text prompts. These prompts direct a generative model in producing synthetic data, enabling global model fine-tuning and local data compensation. This process gradually aligns the label and feature distributions across clients. Extensive results on three datasets demonstrate that Flick improves the global model accuracy by up to 11.43\%, and accelerates convergence by up to 12.9$\times$, validating its effectiveness in addressing data heterogeneity.

Flick: Empowering Federated Learning with Commonsense Knowledge
Ran Zhu', Mingkun Yang', Shiqiang Wang", Jie Yang', Qing Wang'
Delff University of Technology, 2IBM Research
[r. zhu-1, m. yang-3, j yang-3, qing.wang) wang) .n shiqiang. wang@ieee.co org
Abstract
Federated Learning (FI.) has emerged privacy-preserving framework for train- ing models on data generated at the edge. However, the heterogeneity of data silos (e.g., label skew and domain shift) often leads to inconsistent learning objectives and suboptimal model performance. Inspired by the data-driven approach, we propose Flick, novel data generation framework for heterogeneous Federated Learning with Commonsense Knowledge from Large Language Models (LLMs). In Flick, the client performs the local data summary to capture client-specific knowledge in textual form. The central server then distills task-relevant, high- quality knowledge from the out-of-the-box L.LM - guided by cross-client-speifie insights to generate informative text prompts. These prompts direct a generative model producing synthetic data, enabling global model fine-tuning and local data compensation. This process gradually aligns the label and feature distributions across clients. Extensive results on three datasets demonstrate that Flick improves the global model accuracy by up to 11 11.4%, and accelerates convergence by up 12.9x, validating its effectiveness in addressing data beterogeneity. The code can be found at wottttt::/// /i github...... com/ Nom/Raak Ran- ZHU/FLL Flick
Introduction
With the proliferation of mobile devices equipped with sensing capabilities, edge-generated data has unlocked new opportunities for cyber-physical services. Meanwhile, advances in computing and networking the edge have driven the adoption of edge computing, where deploying deep learning models near data sources enables agile and efficient Al applications within distributed systems (133). In this context, Federated Learning (FL) has emerged key paradigm that enables collaborative model training on decentralized data while preserving data privacy [4 FL proceeds in communication rounds, where participating devices (clients) perform several steps of Stochastic Gradient Descent (SGD) on local data and send the updated models to a central server for aggregation into global model. commonly used algorithm FedAvg [5], which alternates between local updates clients and global parameter aggregation among clients by a central server. However, the performance of the global model often hindered by the non-identically and/or non-independently distributed (non-IID) nature of data silos across clients. Specifically, when data distributions are heterogeneous due to class imbalance or domain shifts (e.g, sketch vs. photo) - clients may learn inconsistent objectives. This leads to slower convergence and reduced generalization in the resulting global model (Figure leeft).
To solve this problem, previous works proposed solutions such as 1) modifying loss functions for local models TO, 2) re-weighting central aggregation for the global model [k] or 3) using adaptive hyper-parameten for local training [1]. However, since the local objective is only computable on clients, those model-driven methods that align the global optimum with local (surrogate) objectives
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Local- saall-gnnration
94.89%
54. 49
Global Model Accuracy
92.47%
90
91.48
90.06
Accuracy
84
83.06%
Server- generation
26
Firk Genentiion Augmencation
Photo
Art Painting
Cartoon
Skitch
Dommat Labt- illew
1000 1500
2000
1300mman-ahe shit:
AVG acrous domarrr
#generated samples
Figure 1: Problem (left figure): Heterogeneous data across clients speciffiill label skew and domain shift-leads to cross-domain variance and inferior overall accuracy. rendering the degraded global model in FL.. Solution (middle figure): In data-driven approaches, most existing frameworks compensate for heterogeneous data by generating samples locally. However, a server-based generation framework offers the potential to run computationally intensive image generation models, thus providing high-quality synthetic samples. Flick's Gain (right figure): Flick improves the test accuracy by large margin while requiring fewer generated samples.
cannot fundamentally eliminate the model bias caused by heterogeneous local data. A few recent works aiming at addressing this issue are based on dlta--diieen methods by utilizing generative models either the server client side to produce pseudo samples or synthetic data for fine-tuning the global model or enriching local datasets, which requires sensitive information shared between clients and the server such as local data distribution, small portion of local data or large number of data features 27. These approaches raise severe privacy concerns. On the other hand, many frameworks 18 23 perform generative models directly the client side to avoid transmitting privacy information to the server. While this allows for efficient data augmentation tailored to locally observed data, only client-specific augmentation remains insufficient for addressing domain shift challenges. Also, the effectiveness and efficiency of local generation are inherently constrained by the limited scope of knowledge and computational resources of clients. We ask the following question:
Can we design a generative framework utilizing limited low-sensitivityy cross-client knowledge and task-relevant commonsense of LLMs to quickly promote FL's performance under data heterogeneity?
In this paper, we propose Flick, a novel server-side data generation framework to mitigate data heterogeneity, with particular focus on label skew and domain shift. Flick, in a nutshell, provides a novel design distill client-specific knowledge while leveraging Large Language Models (LLMs)- as an extra informative source to instill task-relevant commonsense knowledge into data generation. The synthetic data is then used to refine both local data silos and the global model. Specifically, at the beginning of each communication round, the participating client selectively captions the local samples using pre-trained image-to-text model (24) and reports the local summaries (in the form of token sequences) to the server. This process extracts and converts salient yet sensitive local information into a relatively low-sensitive textual format. The central server in Flick then employs a carefully designed prompt template to instruct an out-of-the-box LLM in analyzing local information (referred to as cross-client-specific knowledge) while incorporating the LLM's inherent commonsense knowledge produce informative text prompts. These text prompts are subsequently fed into an image-genen gengerainn model, benefiting from the powerful computational resources to run computationally intensive model on the server side, and thus can generate high-quality synthetic samples. These data points serve two key purposes: 1) the server sends the generated samples back to the clients compensate for heterogeneous data silos, and 2) the server fine-tunes the aggregated global model on the generated samples. These designs distinguish Flick from existing data augmentation methods. For the gain (Figure [right)), Flick improves the global model accuracy by up to 8. % compared to local augmentation with the same amount of generated samples, and the global model in Flick achieves comparable accuracy to local augmentation while the latter requires 6.3x generated samples. We attribute this performance gap to the effective distillation of task-relevant commonsense knowledge from LLMs, which further facilitated by cross-client- -seenn- insights. In summary, we make the following key contributions:
We design Flick, a framework to enhance FL's performance on non-lID data by blending local and server-generated samples. It integrates low-sensitive peass-lleett-seccff knowledge and I.LMs' commonsense knowledge, enabling clients to progressively acquire unbiased data in both label and domain aspects.
2

---

We design local summary method to encode client- siient-specific knowledge into token sequences to facilitate task-specific data generation the server. The generated samples serve two purposes local data compensation and global model fine-tuning to enhance FL's overall performance.
Extensive evaluations three datasets demonstrate Flick significantly enhances FL's perfor- mance, achieving up to 11.43% higher model accuracy and 12.9x faster convergence. Moreover, our study highlights Flick a flexible pipeline adaptable various LLMs and image-generation models.
2 Related Work
Existing works to solve heterogencous FL can broadly be classified as model-driven and data-driven.
Model-driven Heterogeneous FL. Model-driven methods seek to mitigate the diverse model updates caused by inconsistent local objectives through modifications to the loss function, robust aggregation methods, adaptive hyper-parameter settings. For the skewed label distribution, some works [7/9] introduce penalty terms into the local loss function. Alternatively, FedNova [12] re-weights the local updates during central aggregation, while FedOpt [13 and FedAvgM [[]] take aggregation as an optimization problem and apply various optimizers to average the updates. In terms of domain shift, FedHEAL to performance improvement, and FedBN [26] excludes the parameters of batch normalization layers from aggregation. Futhermore, concurrent frameworks also adapt hyper-parameter across communication rounds to solve the data heterogeneity issue, such as learning rate and weight decay for SGD solvers [6 or the proportion of participating clients [5].
25 selectively aggregates weight parameters based on their importance
Data-driven Heterogeneous FL. In data-driven paradigms, auxiliary data/pseudo samples are synthesized via generative models deployed on either the client server side, which are leveraged to refine the global model or augment local training distribution. Based on the entity generating data, works can be categorized into local generative and server generative approaches. In local generative methods, clients typically maintain a local generator that is periodically updated. For instance, the conditional Generative Adversarial Network (GAN) in FAug (17) and the Conditional AutoEncoder (CVAE) in FedDA 18. The Gen-FedSD 22 and ReGL 23] generate synthetic data more directly by feeding a local Stable Diffusion (SD) model with textual prompts from predefined templates. Alternatively, FRAug (1 performs augmentation in the feature embedding space rather than in the input space. In server generative methods, such as FedFTG 19 and DynaFed [27]. pheudo-samples are generated based on knowledge distilled from the global model. FGL [20] shares a similar motivation with Flick, as both use client-side information to guide server-side data generation. However, FGI. requires clients to caption all local images, incurring substantial overhead and exposing sensitive information such as local data distributions. It also relies on large volumes of synthetic data for global model fine-tuning, increasing computational cost. In contrast, Flick minimizes the client-side effort during the local summary phase, while achieving efficient data generation by integrating cross-client-specine insights with commonsense knowledge distilled from out-of-the-box LLMs.
Flick Design
In FL., the server searches for the optimal global model W by aggregating local updates from a set of clients J. iterating over N E N communication rounds. In round n E [N] ([]] = (1,.... .N)), a subset of clients J" C I perform SGD on their local data (Diliet in parallel where Dj ((af) w/)) E IRd x |C E [[D with d, C' (number of classes) representing the dimensions of input and output space. The global objective function L(W) is the weighted sum of local objectives:
W arg min L(W) arg min W
Σ Eo jEJ
W
where the weights for local objectives (LD, (W)bses satisfy EGET Pj = 1; the fraction of local samples py = |Dyl/ E5e (Dyl Specifically, the local objective of j-th client is the empirical loss over the local data, that is, LD, (W) EnwwED, ((W; v)/lD, where / is the loss function.
3

---

2.T 2------ Fine-tuning
E( |II. LTP)
Data Pool
Model Aggregation
Global
Fine-Tuned
Data Géneration
M
Commitin Knoaledge
A
B
LLM
Eransdedge
Text Prompts
Model
Samples
Local Summary
Local Compensation
Synthutic Data
Updates
Data Silo
Tokens
Original
Augmented
Figure 2: Flick overview: it bootstraps participating clients to summarize their local data, distilling client-speciff knowledge into low-sensitivity tokens. The central server uses an out-of-the-box LLM to analyze local summaries and instill inherent commonsense knowledge into generated text prompts. text-to-image model then produces synthetic samples, which are used in two ways: to compensate for the local datasets of clients and to fine-tune the aggregated global model.
In the context of data heterogeneity, the local data distribution D, (a y) differs between clients, meaning Pj (z, y) * (x. y). 12 E J. + j2. Following the taxonomy non- IID data in (28), rewrite (2 y) (t) In this paper, focus on two types of data heterogeneity: label skew and domain shift, where the marginal distribution (v) and the conditional distribution (xrly) may vary across clients, respectively. To address this problem, we propose Flick, data-driven approach where the central server generates new data points Dj for each participating client E J In this way, the local data is progressively updated over communication rounds, such that Dj = D ----u10 DJ For the stale client & J", the local dataset remains unchanged, i.e., Dj- DØ-1_ The data across clients progressively approaches an IID condition, where P,, (xly) (xly) and P,, (y) (v) with Vj1.22 E J.h # j2- All notations used throughout this paper are summarized in the Appendix A
3.1 Flick Overview
The overall framework of Flick is illustrated Figure 2 Flick introduces two additional phases into the FL workflow: local summary and data generation. Clients first summarize their local data by selectively captioning samples. The server then generates new data points by analyzing the collected captions using an LLM guided by designed prompt, fusing client-specific insights with commonsense knowledge embedded in the LLM. Generated samples are stored in a server- maintained data pool, which supports both global model fine-tuning and local data compensation. Flick is designed to ensure effective data generation and efficient utilization of synthetic data, aiming to enhance global performance and convergence. However, designing Flick non-trivial: 1) the server requires clients to report information about local data, raising potential privacy concerns; 2) data generation has to balance the sample quality and sample quantity within limited budgets (e.g.. maximum generation cost, generation latency), necessitating careful prompt design and generative database management; 3) two types of generated sample usage are complementary and equally important for improving global model performance: progressively providing clients with additional samples to approach the IID condition yields long-term benefits while fine-tuning the global model provides more straightforward yet slight improvement. Detailed designs are presented in the following sections.
3.2
Local Summary Phase
In round n, clients J initialize their models with the latest global weights mo!n. Wn-1 and per- T5 Vgg(n. (Wnfaul))
form Tj E - steps of SGD on the local data Dy-1; w/(n.Ts) 1 Σイ
where V giin (es (W/fa.))) is stochastic gradient over mini-batch B/ C DD-1 step [7,].

---

S Input Prompt
C LLM Response
Given of captions: (2")). select are relevant
("there colorful paiming of'a dog wearing surrounded by balloons" .m
subjjcct (f). Then, analyze the selected exptions, which depict various
"nn image of dog playngg fetch with a frisboe a sunny park".
3015
scenes and coutexts but consistently center around the main subject (E]-
"a cartoon dog dressed superhero flying through the sky with a cape",
Based analysis, generate {G]
prompts that still focus
the main subject of [E). You may flsse domains, styles, estities, or
black and white sketch of'a dog lying on cary rug nost to Gnephace") m
contexts from the provided captions. but ensure that any new introdiced do not detract from or obscure the main subject, which must remain the central focus. lntroducing elements encouraged, but they should cnrich the context rather than the focus away from main subject.
Data Generation
Similaityin ml) 0.8024
whimsical painting of
dog wearing colorful hat
surrounded by flowers'
[10g MIG Jn1 m
Your response should the following format: ["prcerpe""; "proenpe (G "]] If none of the captions are related to the category (0) onigly [].
LLM Text Prompts
(MC
D
SBERT Similarity
Figure 3: Illustration of the data generation phase: by feeding the designed prompt containing cross- client-specific knowledge T", the server obtains the text prompts for data generation in the i-th class (e.g., dog) from the LLM output. To reduce generation cost, Flick first retrieves historical samples from the server- heed data pool based on the Sentence- BERT (SBERT) embeddings [66 similarity between the LLM- --M-egnerttedd and historical text prompts. For remaining text prompts that fail to find matches in the historical data, the server uses a generative image model to produce synthetic samples.
is the learning rate. The post-training summary phase extracts client-specific knowledge from local data, providing the server with references for data generation. To achieve this, each participating client selects representative samples and uses image-to-text model caption generator to transform samples into low-sensitivity textual information, which is then offloaded to the server.
Sample Selection. To minimize the privacy risk and local captioning overheads, the representative samples D C Dy-1 of the j-th client for captioning should be limited in size, while still ensuring coverage of all classes held by the client. In this way, we propose a loss-based sample selection strategy where samples are chosen based on their informativeness for the updated model w. Specifically, client j calculates the class-wise loss of the local model across the holding classes Cy-
Σ
; v.,y)
1
D
(a.N)ED Jin
where Dia refers the set of local samples in the class e, and DJ-1 = Uoecg-1 Dj.t1 We define the average class-wise loss as 13 Ececy-
The sample selection follows the criteria: for each class e E Ch-1, if the class-wise loss 1n J.c S T we select the sample from D with the largest loss in that class. Conversely, if 17, > Ty. we select the sample with the smallest loss. The rationale behind this that, for classes where the updated model performs above average, we choose the sample with the largest loss, as it has a significant influence on model training (29]. For classes where the model has inferior performance, selecting the sample with the smallest loss ensures that the representative samples are less likely to be noisy or distorted.
Sample Captioning, Each clients extracts a set of representative samples D=-1 for captioning. As the text generation task, the learning-bass image captioning provides natural language descriptions for visual content using the incoder-decoder model structure [3))22] 32]. Among existing methods, pre-trained on large-scale image-text pairs for local captioning. Each client feeds images Dz into the VLP model for inference, generating set of output token sequences T7", which are captions for corresponding representative samples.
we employ VLP model 33 24
Privacy Consideration. Compared to frameworks where clients offload raw local data or latent features to the server [8], Flick has a privacy-consciors design for the local summary, adhering to the data minimization and anonymization principles defined in work (35). Specifically, it offers a minimally invasive method by selectively captioning small subset of representative local samples into token sequences, providing the server with essential yet low-sensitive information for data generation. We also conduct the privacy assessment of the local summary design in Appendix B
5

---

3.3 Data Generation Phase
In round n, the central server collects locally updated models (W/a herr" and token sequences of local representatives (je.. The server aggregates local updates for a global model W Heer..
Enegm [Dy] (D text captions 7," and pools them into T = U sc." 7j" Based on the analysis of T supported
In parallel, the server decodes the token sequences T," into the
by LLMs, the server provides clients with new data points (D)) beeg", where Dj = (x u²²) E Rd x NO l E [1D compensate for local data distribution by blending each local dataset D Simultancously, the server fine-tunes the aggregated global model with generated samples to update the model weights from W to W"
Generative Data Pool. The server in Flick maintains generative dataset G. = ((= v², m))lk E [[G.]]} to pool the pairs of data points (I.: Ms) along with corresponding text prompts m, used to generate each data point. Specifically, by feeding text prompts into a generative model fWa, for instance, latent diffusion model (LDM) [37]. the server generates synthetic sample f (m) in the class The server-held dataset G. is tightly coupled with the data generation process by using validation dataset for evaluating local updates task-specific classes,
it in three ways: database for retrieving historical samples, and as a fine-tuning dataset for global model enhancement. Compared to previous frameworks [38] constructing the public/auxiliary dataset residing in the server, G. differs in two aspects: 1) the public dataset typically requires either web-sourced data or soliciting data from paid anonymous workers, whereas G, in Flick is entirely self-contained and generated from scratch; 2) server-side datasets in previous work are usually static while G. is dynamic and evolves over the communication rounds by updating with new samples.
Synthetic Data Generation To gener-
Algorithm 1: Data generation and usage.
ate the task-required samples, the server first determines class-level data require- ments for each client based on valida- tion accuracy of local models on the cur- rent server-side dataset G--1, shown in Algorithm| (line 1-2). We use matrix D" E NCX to record decisions in n- th round, where the entry Daj (0, 1} in the i-th row and j-th column indicates whether supplementing client j with sam- ples of class i. The value of D.j de- pends on the performance of local model: Dij = when the validation accuracy of (o((.7s) on class is below the prede- fined threshold Tu, suggesting that client j would benefit from additional samples of class to improve local training, and D.j 0 otherwise
input :Captions T
Local updates Weeee Sever-held datasct gn Thresholds Te, Tu; Generative model f Budget G.
output: Fine-tuned global model weights W";
Compensated local dataset (Dz)jeJa.
Server Executes Data Generation: /* obtain decision matrix */
Valcx1J"1 f- validate (Wj)senn on g.
D cx|3"| + (Val T.)
3 M &- extract historical text prompts in G. for i 1,2, ...,....0 C' do if Σ Djezn Du, 0 then /* obtain text prompts */
M" <- feed LLM prompt (T", G.
Sim wisr similarity (MP. .M))
/+ retrieval-based samples */
(M", <- 1(im 2 T.)
D(nn)) +- samples from - by M /* generation-beet
samples */
The server then generates samples for classes 7 = {ili E [C]. EjenE D.j > 0}. The generation procedure follows
D(m.)) + feed ww text prompts M. \M
10
DP + (n.3) U UD(aa)
11
three steps: 1) Given a generation budget G (equal for each class), an out-of-the- box L.LM generates text prompts MM {m/lk E } following the designed prompt shown in Figure 3, The prompt template instructs LLM in extracting in-
/* local data compensation */
12 for j E J do
13
Dj + Ulecyy DT [DD,,=1]
DD + send DJ to client executing 277 U DJ
14
global model fine-tuning */ 1S G. + update server-held dataset by Uie D.
formation from the provided cross-client- specific knowledge 7 while instilling in- herent commonsense knowledge to pro- duce informative text prompts for the i-th class. 2) To reduce generation overheads, the server
16 W < fine-tune global model by G.'
retrieves historical samples from the data pool - by comparing the similarity between generated

---

text prompts M and pooled text prompts = {m m E -.-1). We calculate the cosine similarity of Sentence-BERT (SBERT) embeddings [66 for each pair of text prompts from M, and M For the text prompts M C M, where the highest pairwise similarity with historical text prompts M C M exceeds predefined threshold Txs the server includes the corresponding
((m.)) {(a5, ut) (at v². m²) E g m E _MM thereby avoiding the need to data points retr
generate new samples. 3) For the remaining text prompts M \ (M",, , the server employs an image
generator (-) to synthesize new data points DSmn {[S (mf)), uf Imm E MØ \ MM?); ). In this way, the server provides each class E I with samples obtained by either retrieval or generation:
D. = (m U " Algorithm (line 5-11) describes the overall data generation process. More
data generation details, including synthetic samples, are provided in Appendix G
3.4 Generated Data Usage
The server the generated samples in two ways: for global model fine-tuning and for local data compensation, as described in Algorithm [ (line 12-16). The server sends the corresponding generated samples to j-th elient: Dj = Uiez D. (Da,) = 1), where 1(-) is the indicator function.
Each client then compensates for its local data distribution by updating Dj- Dj u DJ. Vj J
In parallel, Flick updates the server-held data pool 9 by replacing the stale samples with latest generated samples U.ez D keeping the dataset size constant. The server then uses the updated dataset G" to fine-tune the aggregated global model, updating the weights from W to W
4 Experiments
4.1
Experimental Setup
Datasets and Models. We extensively evaluate Flick on the multi-domain image classification task, where data of different domains exhibit heterogeneous appearances but share the same labels. We use three datasets: (1) PACS [1]. consisting of 9,991 images in 7 classes across the following four domains: Photo, Art Painting, Cartoon, and Sketch; (2) CflicehCaltech [42]. containing 10
overlapping classes between the Office dataset (43 and Caltech256 dataset 44. with data from
four domains: Amazon, Caltech, DSLR, and Webcam; (3) DomainNet 45). large-scale benchmark
covering six domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch, each originally containing 345 object classes. Following prior work [20, [6] 46), we construct subset of DomainNet by selecting the top 10 most common classes for our experiments.
In our experiments, we simulate federated settings with heterogeneous data distributions. For the PACS dataset, we use 20 clients, and for Office-Caltech, we adopt 8 clients. Data from each domain is partitioned into 5 (PACS) or (Office-Caltechl subsets using Dirichlet distribution with concentration parameter a = 0.1 and a = 0.05, respectively. To evaluate the scalability of Flick in large-scale federated settings, we further conduct experiments on the DomainNet dataset with 100 clients, where 20% are randomly selected to participate in each communication round. Each domain is split into 15 or 17 subsets using a Dirichlet distribution with a 0.1. Across all three datasets, each client receives data from a single domain with a skewed label distribution, effectively simulating woal- scenarios characterized by both domain shift and label skew. Following prior works [21 25 47), we employ ResNet- 18 as the shared model architecture for PACS and DomainNet, and ResNett 10 48 for Office-Caltech across all compared methods.
Baselines and Metric. Our evaluation is based on four baselines: the vanilla FedAvg ( and three model-driven methods- methods-FeeProx 99, FedDyn [4), and FedNAR ddsinned for heterogeneous FL. Building upon each baseline, we incorporate Flick and compare it with seven counterparts:
methods tailored mitigate domain shift such FedBN 26 and FedHEAL 25 ; and generative
methods including DynaFed FedFTG [19) and FGL
20 (server-side), as well as FRAug (2
and
[50] (client-side). We run all methods three times per setup and report the average and standard deviation of Top-1 accuracy. Model convergence is assessed by the round-to-ace to-accuracy performance (#Round) defined as the number of rounds required to reach the target accuracy (i.e., the best accuracy of baseline methods). As a generative method, we also report statistics on generated data points, including the number of samples at the target accuracy (#Sample) and the total generated samples over all rounds (#Total Sample). Evaluation on the temporal scale is provided in Appendix
7

---

Table 1: Global model accuracy (%) under both domain shift and label skew. AVG denotes the average accuracy across all domains, and A indicates the accuracy gain compared with vanilla methods. Best in bold. Acronyms in the PACS dataset: Photo (P), Art Puinting (A), Cartoon (C), and Sketch (S); In the Office- Caltech dataset: Amazon (A), Caltech (C), DSLR (D), and Webcam (W).
PACS
Offic-Caltes
Methods
P
S
AVG
A
#Round 4
A
W
AVG
A
#Round
FedAwg
[5
73.54
86.37
84.82
82
87.49
83.0611.V
142
68.52
62.44
72.92
70.84_1.1
135
+FedBN 26
80.70
84.82
90.33
90.29
86,544445
3.48
81
66.58
62.89
82.14
80.83
13.11=00
2.27
135
+FedHEAL
25
85
86.9
93.45
85.96
87.88
4.82
50
73.23
63.56
79.76
80.56
74.28=00
3.44
48
+Dynalied
27
74.39
85.03
87.95
89.34
84.1811
1.12
79
78.91
62.45
72.10
72.34
71.45_
0.61
+FedFTG 19
75.61
89.81
88.10
88.64
85,54
2.48
90
72.69
62.89
72.10
64.00
67.92-00:
-2.92
+rGL. 200
88.96
87.69
97.62
85.09
89.84
6.78
22
67.36
62.22
85.71
80.00
73.82..1.31
2.98
45
91.99
94.59
97.74
93.91
11.43
75.82
64.00
96.43
77.78
78.51:055
7.67
FeutProxx
74.82
87.53
86,98
87.69
84,25a13v
142
65.93
60.11
79.46
76.25
70.44:104
135
+FodBN 26
77.43
87.05
88.39
89.02
85,47=025
1.22
113
64.77
62.22
82.14
78.33
71.87
1.43
59
+FedHEAI.
84.
86
94.79
84.20
17
3.23
50
76.68
64.89
78.57
68.33
1.68
+DynaFied
27
77.43
85.77
87.05
87.94
14.55
0.30
81
77.65
61.11
79.24
69.00
70.25.1.31
-0.19
+FedFTG 19
73.79
89.17
84,52
91.62
84.78aa50
0.53
11.5
70.47
62.89
75.67
65.67
68.68=1172
-1.76
+FGL 200
87.99
91.51
95.68
88.13
90.83
6.58
21
72.54
65.56
76.67
74.23..03
3.79
69
+Flick
91.87
94
98.07
93.27
94.34
10,09
19
74.96
64,59
96.43
79.44
78.86=0447
8.42
40
FeuDyy H9
75.12
83.76
91.07
67,50
84.362020
22
70.98
59.11
82.14
80.83
73:27-054
100
+FodBN 20
78.76
85
89.88
88.58
85,70=022
1.34
22
75.39
64.67
83.93
78.33
75.58-1145
2.21
97
*FedHEAT.
80.22
85.67
91.22
86.68
85,95
1.59
26
74.09
60.00
78.57
86.67
74.83__ns
1.56
50
+DynaFed
27
78.80
87.47
91.17
83,80
0.95
15
75.73
60.45
77.67
74.34
72.05
-1.22
+FedFTG
79.05
85.56
91.77
85.66
1.15
75.47
62.07
78.14
73.78
72.37_
-0,90
+FGL 200
81.96
87.33
91.87
84.05
86.30 36.30
1.94
18
72.80
60.89
82.14
80.83
74.17=025
0.90
37
+Flick
87,50
93:42
95.24
93:40
92.39 14
8.03
12
77.85
63.33
91:07
83.75
79.00=071
5.73
22
FeuNAR 16
81.19
87.47
93.60
80.52
85.70=446
142
67.88
60.89
76.79
78.33
70.97-044
135
+FedBN
82.77
89
94.49
88.96
88.85
3.15
35
69.95
60.22
78.57
81.67
72.60=044
1.63
129
+FedHEAL
25
83.37
83.35
95.39
84.77
87.22.14m
1.52
81
73.06
63.56
85.71
71.67
73.50_ _035
2,90
45
+DynaFed
27
86.08
87.90
94.74
86.93
88.91=
3.21
46
68.03
63.33
79.24
65.67
69.09-093
-1.88
+TedTTG
19
80.95
91.72
93.15
86.04
87.97
2.27
100
78.91
65.11
61.38
65.67
-3.19
+FGL 200
87.99
87.90
96.58
85.85
89.58
3.88
22
69.43
62.45
85.71
70.83
72.11=032
1.14
47
+Flick
92.35
91.83
97.47
92.83
93.62:5031
7.92
12
73.96
62.22
91.96
80.42
77.14-0.4
6.17
43
Implementation Details. For fair comparisons, all methods are implemented using the same settings. We use SGD as optimizer with learning rate of 0.01; the weight decay is 4c-5 and the momentum is 0.9. The batch size for local training is 64 and 32 for the two datasets, respectively, with four clients participating in each round. The communication rounds are to 150. We utilize "Salesforce/blip- umage-captioning-arye for the image generation, sourced from Hugging Face. We also use "gpt-4o-mini" from OpenAl
(24] for the image captioning and "xc--aacc/ssabeeddfffiio--o
[51]
API [52 to analyze offloaded captions. More details are given in Appendix B
4.2 Performance Evaluation
Flick Effectiveness. Table ! presents performance under the heterogeneous setting with both domain shift and label skew. We observe that Flick outperforms all counterpart methods across integrated baselines, consistently yielding substantial improvements in global model accuracy and requiring significantly fewer rounds to reach the target accuracy. Specifically, Flick improves the Top-1 accuracy by up to 11.43% on the PACS dataset and 8.42% on the Office-Caltech dataset while reducing #Round from 142 and 100 to 11 and 22, respectively. The performance of DynaFed and FedFTG heavily depends on the quality of generated pseudo samples, posing significant challenges when facing relatively high-resolution data such as 224 224x23 images adopted in our experiments, resulting in severe variance across the two datasets. FGL 200 constructs a large IID synthetic dataset before FL training using captions from all local data, with its size matched to Flick for a fair comparison, which is used to fine-tune the global model to boost FL performance. However, due to its reliance on local data side information, FGL underperforms compared to Flick. Additionally, creating large synthetic dataset in advance introduces substantial overhead, making it difficult to reach the target accuracy within a short time frame (see Appendix DD. Besides, we investigate the performance of Flick's under the domain shift-only setup where each client holds data from a single domain while maintaining balanced label distributions across clients. Figure 3 shows the superior performance of Flick compared to domain shift-specific methods (i.c., FedBN and FedHEAL) across two datasets. These results highlight that Flick effectively fuses cross-client-specfficc knowledge while installing commonsense insights from the LLMs into synthetic data, significantly mitigating the heterogeneity problem in FL. More experimental results can be found in Appendix C.2
8

---

beceency (e)
boo
Vanila
D + FodHEAL
FeddNN
tlik
76-
Fedling wit L.
wio OR wht
Fedlog
Techrox
Tedbyn
TeckAR
Ted/ug
TeaPaa
Tedbyn
54
PACS dataset
(h) Office--Clttttt dataset.
Figure 5: Performance across various baseline methods Figure 4: Learning curve of ablation study, under domain shift only.
50
20
Round
50
220
140
Comparison with Local Generative Methods. We also compare Flick with local generative methods: FRAug, Gen- FedSD, and a transformation-basee local augmentation method that locally gener- ates data points through random transfor- mations such as cropping, rotation, and image mirroring [50]. For Gen-FedSD and transformation-balt based methods, the number of samples per class depends on the local data distribution, with more samples generated for underrepresented classes. To ensure fair comparison,
Table 2: Performance across local generative methods.
PACS
Methods
P
8
AVG
#Round
FRAug 21
85.99
90.48
18
Transformation
50
87.94
90.16
95.04
90.57
90.93
FedD-FedSD 23
90.78
94.27
96.73
93.15
19
Flick
91.99
94.59
97.74
93.91
94
11
OfficeCCltech
Methods
W
AVG
#Round
FRAug 21
03
50.67
82.14
85.00
96
Transformation
75.00
77.22
11
138
Gen-FedSDD 22
77.98
64.44
83.93
75.00
75. 34
48
Flick
75.82
64.00
96.43
77.78
58.5
37
keep the consistent budget of generated samples across all methods with Flick. Table Table||shows shows that local generative methods, by leveraging their own determined data distributions, can perform precise augmentation to balance local datasets effectively. This leads to notable performance gains over other model-driven approaches and even some server-side methods. Among them, sample-wise methods (i.e., Gen-FedSD and Transformation) consistently outperform the feature-wise method (i.e., FRgug) across two benchmarks. Although Flick is server-side data-driven framework, it avoids introducing data generation overhead on clients while achieving superior performance by centralizing LLMs usage and leveraging pross-client-seee
knowledge.
Ablation Study. To assess the effee- tiveness of key components in Flick, conduct an ablation study on the PACS dataset with five breakdowns: Local Sum- mary (LS), Commonsense Knowledge (CK), Data Retrieval (DR), Local Com- pensation (LC), and Global Fine-tuning (GF). Results in Table 3 show that the variant of Flick without the local sum- mary phase leads to 2.67% accuracy drop, as the server generates the data solely based on class names. It highlights the importance of pooss-clint--spciffe knowledge in acquiring task-related information from LLMs. Flick without commonsense knowledge directly uses client-specific captions for data generation, relying solely on the local summary. As training progresses, local compensation depends primarily on retrieved samples rather than high-quality data generated through the integration of cross-client insights and commonsense knowledge, leading to fewer generated samples, slower convergence, and reduced model accuracy. The variant of Flick without local compensation brings more data generation over- head while introducing limited performance improvement. Besides, Flick without data retrieval does not significantly affect performance; however, it results in more samples being generated, introducing extra overhcad without proportional gains. Figure further showcases the benefits of synthetic data usage: global fine-tuning speeds up the model convergence at the beginning, and the combi- nation of local compensation and global fine-tume fine-tuning yields consistent and substantial performance improvements throughout training. More ablation studies can be found in Appendix C
Table 3: Ablation study on the PACS dataset.
LS
CK
DR
LC
GF
AVG
#Round #Sample #TotalSmmple
v
v
v
91 01.82
19
430
1218
V
92.92+a22
312
400
x
v
94.75
339
89. 11m00
3239
93.75:0004
19
504
1169
94.49
11
270
1083
Scalability Analysis. To further evaluate the scalability of Flick in large-scale federated learning, we extend our analysis to the DomainNet dataset 45 with 100 clients. As illustrated in Figure 6 Flick consistently delivers superior performance in both model accuracy and round-to-acoonn -o-cuury (#Round) across four different bascline methods, demonstrating its robustness under highly heterogencous and

---

Flick
FGL
FedFTG
DynaFed
FedHEAL
NSN
FedBN
140-
Accuracy
8.3
80-
11.35% higher
12.11% higher
5.02% higher
higher
5.%%
#ound
95
T
10.36Mr
faster
3.58k faster
50-
fasieer
74-
FedAvg
FedProx
FedDyn
FedNar
FedAvg
Fed Prox
Fed
FedNar
(a) Model accuracy.
(b) Round-to-accuracy
Figure 6: Model performance across various baseline methods on the DomainNet dataset.
large-population FL. scenarios. In particular, Flick improves accuracy by up to 11 11.35%, 12.1 1% 5.82%, and 6.50%, and accelerates convergence by up to 10.36x, 5.48 3.58 and 4.17x compared with the respective baselines. Notably, although FGL achieves fast convergence by fine-tuning the global model on large IID synthetic dataset, this comes at the expense of substantial computational and generative overhead, making Flick more appealing in resource-constraingg federated settings. Comprehensive configurations and detailed results are included in Appendix C.4
Discussion
We provide extensive quantitative analyses in the Appendix D D I offering deeper insights into Flick's system efficiency, privacy preservation, and practical deployability Flick improves global performance without exposing data, only lightweight and low-sensitivity token summaries are transmitted, inherently reducing privacy risks. Although server-side synthetic data generation introduces additional computation, this cost is fully offloaded from clients and is largely offset by the fast convergence enabled through rich LLM knowledge infusion. Moreover, directly deploying large models on clients is infeasible in real- world edge intelligence due strict resource constraints, and querying cloud-based L.MS could cause privacy leakage. In contrast, Flick allows compact client models inherit the knowledge of LLMs in cost-effective and communication-effiee manner. We believe Flick highlights that foundation models and lightweight client models play complementary roles: while large models provide rich knowledge, small models remain the only practical and privacy-preserving learners at the edge, making their collaboration essential in real deployments.
6 Conclusion
In this paper, we design a novel generative framework Flick to address the data heterogeneity, including both label skew and domain shift issues in FL. where the central server generates synthetic samples by integrating rross-clien-sspecfic knowledge with the commonsense knowledge embedded in out-of-the-boo LLMs. Flick allows clients selectively caption their samples, distilling client- specific knowledge into low- sensitivity textual information that is then explored by the server to generate data with the power of LLMs boost FL training. We also explore the generated data for local data compensation and global model fine-tuning, further enhancing the global model accuracy and convergence performance. Extensive experiments on three datasets have been conducted to validate the effectiveness of Flick. Specifically, Flick improves global model accuracy by up to 11 11.43% and facilitates round-to- to-accuracy performance from 142 11. Besides, Flick produces consistently superior performance against other counterparts on DomainNet with 100 clients, demonstrating its potential to enhance model performance large-scale heterogeneous scenarios.
Acknowledgement
This work is partly funded by the EU's Horizon Europe HarmonicAl project under the HORIZON- scheme with grant agreement number 101131117. The work is also part of the ICAI GENIUS lab of the research program ROBUST (project number KICHS.LTP..00006) partly funded by the Dutch Research Council (NWO). We also thank the support provided by Samenwerkende Universitaire RekenFaciliteiten (SURF) with their Snellius infrastructure.
10

---

References
[1] Giancarlo Fortino and Paolo Trunfio. Internet of things based on smart objects: Technology, middleware and applications. Springer, 2014.
[2] Xiaofei Wang. Yiwen Han, Victor CM Leung, Dusit Niyato, Xueqiang Yan, and Xu Chen. Con- vergence of edge computing and deep learning: comprehensive survey. IEEE Communications Surveys & Tutorials, 22(2):869-99) 90446999999209 2020.
[3] Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S Rellermeyer. A survey on distributed machine learning. Acm computing surveys (csur), 13(2):1-33, 2020.
[4] Jakub Konečný, H Brendan McMahan, Felix X and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv: 160.0546, 2016.
Yu, Peter Richtárik, Ananda Theertha Suresh,
[5] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. ceaptuniicationfffciieeeee learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 2017.
[6] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learning at scale: System design. Proceedings of machine learning and systems, 374-388, 2019.
[7] Xin-Chun Li and De-Chuan Zhan. Fedrs: Federated learning with restricted softmax for label distribution non-iid data. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 995-1005, 2021.
[8] Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhome Xu. Fedde: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10112-10121, 2022.
[9] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 4500, 2020.
[10] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference machine learning, pages 5132-5 5143. PMLR, 2020.
[11] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.00
2019.
[12] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 13:7611-7623. 2020.
[13] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295. 2020.
[14] Zhenguo Ma, Yang Xu, Hongli Xu, Zeyu Meng. Liusheng Huang, and Yinxing Xue. Adaptive batch size for federated learning in resource-conss cosstrre-constraiiee edge computing. IEEE Transactions on Mobile Computing, 22(1):37-53, 2021.
[15] Ravikumar Balakrishnan, Tian Li. Tianyi Zhou, Nageen Himayat, Virginia Smith, and Jeff Bilmes, Diverse client selection for federated learning via submodular maximization. In International Conference on Learning Representations, 2022.
[16] Junbo Li, Ang Li, Chong Tian, Qirong Ho, Eric Xing, and Hongyi Wang. Fednar: federated opti- mization with normalized annealing regularization. Advances in Neural Information Processing
Systems, (16774753-747633 2023.
11

---

[17] Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong- Lyun Kim. Communication-efico on-device machine learning: Federated distillation and augmen- tation under non-iid private data, arXiv preprint arXiv: /81 /.1 1479, 2018.
[18] Hui Wen, Yue Wu, Jingjing Li, and Hancong Duan. Communication-fffcieni federated data augmentation on non-iid data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3377-3386, 2022.
[19] Lin Zhang. Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10174-10183, 2022.
[20] Jie Zhang, Xiaohua Qi, and Bo Zhao. Federated generative learning with foundation models. arXiv preprint 2rX6v:2306.16600 2023.
[21] Haokun Chen, Ahmed Frikha, Denis Krompass, Jindong Gu, and Volker Tresp. Fraug: Tackling federated learning with non-iid features via representation augmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4849-4859, 2023.
[22] Mahdi Morafah, Matthias Reisser, Bill Lin, and Christos Louizos. Stable diffusion-basse data augmentation for federated learning with non-iid data. arXiv preprint arXiv:2405.0m 07925, 2024.
[23] Ziyu Yao. Recovering global data distribution locally in federated learning. arXiv preprint arXiv:2409.14063 2024.
[24] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMLR, 2022.
[25] Yuhang Chen, Wenke Huang, and Mang Ye. Fair federated learning under domain skew with local consistency and domain diversity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12077-12086, 2024.
[26] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXtv:2102.07623. 2021.
[27] Renjie Pi, Weizhong Zhang, Yueqi Xie, Jiahui Gao, Xiaoyu Wang. Sunghun Kim, and Qifeng Chen. Dynafed: Tackling client data heterogeneity with global dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 12177-12186, 2023.
[28] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends@ in machine learning, 14(1-2):1 10:1-22 12210, 2021.
[29] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. Advances neural information processing systems, 34:20596-20607, 2021.
[30] Oriol Vinyals, Alexander Toshev, Samy Bengio. and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156-3164, 2015.
[31] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. From show tell: A survey on deep learning- beadenggbbssed image captioning. IEEE Iransactions on paltern analysis and machine intelligence, 45(1) (511:55395599 2022.
[32] Taranch Ghandi, Hamidreza Pourreza, and Hamidreza Mahyar. Deep learning approaches on image captioning: A review. ACM Computing Surveys, 56(3):1-39, 2023.
[33] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 13041-13049, 2020.
12

---

[34] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language inguage-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[35] Kallista Bonawitz, Peter Kairouz, Brendan McMahan, and Daniel Ramage. Federated learning and privacy: Building privacy-preserving systems for machine learning and data science on decentralized data. Queue, 19(5)877-114, 2021.
[36] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019, URL hts/1908..00889
org/
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[38] Tao Qi, Fangzhao Wu, Chuhan Wu, Liang He, Yongfeng Huang, and Xing Xie. Differentially private knowledge transfer for federated learning. Nature Communications, 14(1):3785, 2023.
[39] Wenke Huang, Mang Ye, and Bo Du Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10143-10153, 2022.
[40] Mingkun Yang, Ran Zhu, Qing Wang, and Jie Yang. Fedtrans: Client-transparee utility estimation for robust federated learning. In The Twelfth International Conference on Learning Representations, 2024.
[41] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542 5550, 2017.
[42] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pages 2066-2073. IEEE, 2012.
[43] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Computer Vision-ECCC 2010: 1 1ith European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings. Part IV 11, pages 213-226. Springer, 2010.
[44] Gregory Griffin, Alex Holub, Pietro Perona, et al. Caltech-256 object category dataset. Technical report, Technical Report 7694, California Institute of Technology Pasadena, 2007.
[45] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi- domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406- 1415, 2019.
[46] Guogang Zhu, Xuefeng Liu, Shaojie Tang, and Jianwei Niu. Aligning before aggregating: Enabling cross-domain federated learning via consistent feature extraction. In 2022 JEEE 42nd International Conference on Distributed Computing Systems (ICDCS), pages 809 819. IEEE, 2022.
[47] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16312-16322 IEEE, 2023.
[48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[49] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference Learning Representations, 2021.
13

---

[50] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 113-123, 2019.
[51] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the- art diffusion models. https://gtt http///gtttubbb com/huggingface/ diffusers 2022.
[52] OpenAl. Openai gpt-4 api [gp--4o-minn-2224407-111181 2024. Available at: https://platfors: openai com/docs/
[53] Gary Mareus, Ernest Davis, and Scott Aaronson. very preliminary analysis of dall-e 2. arXiv preprint arXiv:2204. 13807, 2022.
[54] Umme Sara, Morium Akter, and Mohammad Shorif Uddin. Image quality assessment through fsim, ssim, mse and psnr- a comparative study. Journal of Computer and Communications, (3):8-18, 2019.
[55] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing. 13(4): 600-612, 2004.
[56] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[57] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[58] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXi: 13923, 2025.
14

---

Appendix
A Notations
All notations used in this paper are summarized in Table We omit the superscript of symbols indicating temporal information, e.g., J refers participating clients in communication round n.
Table 4: Symbols and notations in the paper.
Symbol
Explanation
J
Set of clients
Dj
Dj
Set of local/epressentatie samples of client
D
Set of generated samples for client j
T
T
Tokens sequences/text captions from client Data pool maintained by server Set of text prompts for class
G.
M,
W
Weights of updated local model Weights of auggregated/fine-tune funettundeee global model Decision matrix for data generation
W. W
D
I
Set of classes generated samples Number of local SGD steps for client j Number of communication rounds Number of task task-specific classes Budgel for the number of generated samples Threshold for validation accuracy/text similarity
N
G
T, T.
B Experimental Details
B.1 Visualization of the PACS, Office-Caltechh and DomainNet datasets
We evaluate the performance of Flick and its counterparts on three benchmark image datasets: PACS [41] Office- Office-Caltech Caltech [42), and DomainNet [45]. Each benchmark is multi-class classification task, where each class includes samples from different domains. The PACS dataset comprises seven classes: dog, elephant, giraffe, guitar, horse, house, and person. The Office-Calter dataset contains
Sketch
Photo
Cartoon
Art Painting
Dog
Elephant
Giraffe
Guitar
Horse
House
Person
Figure 7: Example samples from different classes and domains in the PACS dataset
41
15

---

Webcam
DSLR
Caltech
RC
with
Amazon
Backpack
headphones
Keyboard
Laptop
Mug
Projector
Figure 8: Example samples from different classes and domains in the Office-Caltech dataset
Clipart
BIROFI
Infograph
Painting
Quickdraw
Real
MEDIERA
Sketch
airplane
clock
basketball
bicycle
bird
strawberry
flower
Pizza
Bracelet
Figure 9: Example samples from different classes and domains in the DomainNet dataset
45
ten classes: backpack, bike, calculator headphones, keyboard. laptop, monitor, mouse, mug, and projector. Following prior work [20) [6 46), we construct a sub-dataset of the DomainNet dataset for our experiments by selecting the top ten most common classes: airplane, clock, axe, basketball, bicycle, bird, strawberry, flower, pizza, bracelet. Figures g
the diverse domains within individual classes. the PACS dataset, significant Teature variations across domains within the same class are clearly observable, making domain differences explicit. In contrast, domain differences in Office- Caltech are more latent, for instance, variations with background, view, and clarity of the object, posing additional challenges in effectively addressing the domain shift problem within this benchmark. DomainNet contains six highly heterogencous domains, where the appearance gap of the same class across domains be extremely large, making our experiments this dataset more realistic and representative of real-world scenarios. To simulate heterogeneous FL settings with domain shifts, partition the data and assign data from domain exclusively each client.
Office-Caltech
B.2 Hyperparameter Settings
Here, we provide more details about the experimental training parameter settings. All methods are implemented in Python, with neural networks developed using PyTorch. In local training, local epochs are set to 5, and the input data size is fixed 224 x 224. We set the server-held data pool size
16

---

[G.] to 25, and the data generation budget G to 5 for both datasets. We use "stable-diffusii saable-diffusion-vl. for the image generator and "gpt-4o-mini" and out-of-the-box LLMs for generating text prompts by analyzing the uploaded client-specific knowledge combined with embedded commonsense knowledge. comprehensive robustness evaluation of Flick under different image generators and out- out-of-the-box LLMs is provided in Appendix C.1 Additionally, for the PACS dataset, we set text similarity threshold T. = 0.8 and validation threshold T. 0.9, which guide the construction of the decision matrix D. For the Office-alltch dataset, we adopt T, 0.7 and T. 0.8 the default settings.
In baseline methods, FedProx uses a hyperparameter to control the proximal term'ss 's weight in the objective function. We tune this hyperparameter within the [0.001, 0.01, 0.1, 1] following [9] and report the best result. Specifically. the value of 0.1 is adopted in both datasets. For FedDyn, we provide its best performance by fine-tuning the weight of penalized risk within [0.001, 0.01, 0.1). We set this value 0.1 for PACS and 0.01 for Sffice-Caltechh For FGL., we adopt the multi-round-sy variant, where the global model is initially trained on a synthetic dataset generated in the first round and subsequently fine-tuned using synthetic data throughout the following communication rounds. To ensure fair comparison, we match the total amount of synthetic data to that produced by method, Flick, under each experimental setting. For other counterparts, we keep the default settings they provided.
C More Results and Analysis
C.1 Sensitivity Analysis
In this section, we demonstrate that Flick consistently maintains superior performance across a range of hyperparameter settings and model choices, indicating the robust generalization of Flick without reliance on extensive tuning.
shows the model performance improvements with larger text similarity threshold Tç. The number of generated samples required to reach the target accuracy (i.e., FedAvg's best accuracy) rises with a higher T, while #Sample decreases because fewer rounds are needed. When T3 reaches 0.9, no samples are retrieved. For validation accuracy threshold T,, lower value generates fewer samples per round but requires more rounds to reach the target accuracy. With a fixed number of rounds, the accuracy is lower at smaller values of T... As shown in Figure b)) T. 0.9 1.0 serves as effective threshold, yielding better performance with fewer required samples. Figure [0 shows that the accuracy performance does not change much across different sizes of the server-held data pool. Here, x-axis denotes storage capacity per class in the data pool; we can observe that a larger pool size significantly benefits the global model's fine-tuning, greatly reducing #Sample. However, larger data pool size requires more storage.
Impact of text similarity T., validation accuracy T., and data pool size. Figure 10
#Retrieval
#Generation
800
-95
#Sampl
600
600
600
600-
400-
50
400
200-
200
200
0.6
0/7
0.9
0.6
0.7
0.8
0.9
1.0
(a) Text similarity threshold 7;
(b) Validation threshold T.
(c) Data pool size
Figure 10: The performance of Flick under varying hyperparameters on the PACS dataset.
Impact of Data Generation Budget G. our designed prompt illustrated in Figure 3 the server hosting the LLM can generate text prompts of varying sizes for class under a specified hyperpa- rameter G. This design flexibility in our Flick allows the server adapt the practical deployment with the given data generation budget. In the experiments, we evaluate the performance of Flick under different data generation budgets E (5, 10. 15). We keep the experimental setup for the two benchmarks consistent with the experiments in Table I with FedAvg as the baseline method, except for varying G and the server-held data pool size fixed |G,| = 40. The results are shown in Table We can observe that a larger G indeed improves the global model accuracy in the long run by generating more samples enriched with commonsense knowledge. Additionally, = 10 and G = 15 speed up the model convergence the Office-Caltedh dataset compared to G 5, evidenced by
17

---

Table The performance of Flick under different data generation budget G both datasets.
PACS
G
A
S
AVG + #Round #Sample
G 5
91.99
91.51
97.62
93.27
93.60+0.17
188
G
10
92.72
94.06
96.43
93.65
94.21 +0.20
276
15
93.93
93.84
97.17
93.53
94.62+0.14
420
Office- Office-Caltech
G
D
AVG
#Round
#Sample
G 5
76.68
65.78
88.10
80.56
77.78
33
146
G 10
77.98
63.56
94.64
79.17
78,8 28.84
24
253
15
77.72
65.33
96.43
78.33
79.45
317
the enhanced round-to-accuracy performance (i.e., #Round). However, G = 5 requires the fewest generated samples (i.e., #Sample) reach the target accuracy, revealing trade-off between training performance (e.g., accuracy and latency) and data generation overheads. It underscores the impor- tance of carefully selecting the balance data generation costs and performance improvements in practical applications. In the rest of the evaluations in this paper, we use G = 5 as the default setting,
Impact of different image captioning methods. In Flick, clients perform image captioning small set of representative local samples during the summary phase. This summarization captures essential local insights that play crucial role in guiding the server extract task-relevant, high- quality commonsense knowledge from LLMs an effect validated in our ablation study. To assess the sensitivity of Flick different image captioning strategies, evaluate performance using poli-immageccaptionngg and gpt2-image cageeccattioiigg As shown in Figure [T] the results on both datasets indicate that Flick is agnostic to the choice of captioning model and consistently maintains superior performance. This demonstrates the flexibility and generalization of our approach.
three representative captioning models: hopp--aag-cappioingg-----
600
800
# Retrievaa
#Generation
#Retrieval
#Generation
80
#Samples
400-
Accuracy
@Samples
Sccura (%)
400-
200
91
200
88
70
blip-large
blip-base
vit-gpt2
blip-large
blip-base
vit-gpt2
(a) PACS dataset
(b) Office-Caltech dataset
Figure 11: The performance of Flick over different image captioning methods: blip-image-captionnng large, blip-image-captionnn---as and vi-gno2-magg-capioonng
600
#Retrieval
#Generation
#Retrieval
#Generation
60
#Samples
400-
Accuracy
M #samples
800
Accuracy (%)
600
400-
200
91
(gg))
200
4o-mini
40
4
3.5
40-mini
4o
3.5
(a) PACS dataset
(b) Office-Caltech datasct
Figure 12: The performance of Flick over different out-of-the-box LLMss gpt-4o-mini, gpt-4o, gpt-4, and gpt-3.5-turbo-0133
Impact of different out-of-the-box LLMs. The server in Flick employs LLM to analyze the collected local summary and generate text prompts for subsequent data generation. The designed prompt template instructs the LLM to extract class-specific information from collected local captions
18

---

and fuse it with inherent commonsense knowledge, thereby enhancing the quality of the output text prompts. To demonstrate the compatibility the Flick workflow with out-of-the-bon LLMs, we use the same prompt template as illustrated in Figure and evaluate the performance of Flick by configuring the L.M to "gpt-3.--uubo-01255 "gpt-4", "gpt-4o", and "gpt-4o-mini" (default LLM in this paper). The experimental results are given in Figure [2] We can see that the choice of LLM has negligible impact on the text prompt generation, evidenced by the similar global model performance and data generation decisions. These valuation results validate the compatibility of our designed prompt template across a wide range of LLMs.
Impact of Image Generator (.). As key component of the Flick workflow, a text-to-image model is employed generate synthetic samples taking input text prompts. Since the data generation runs on the server side, the availability of rich computational resources allows for extending the choice within advanced generative image models, thereby enabling the generation of high-quality data points for further usage in Flick. To this end, Figure [33 reports the performance of Flick when using various generative models. We investigate the LDMs publicly available in HuggingFace. including
(default model in this paper). "stable-diffusion-vl=""" and "stable- diffusion-xl-base- 1.0". We also incorporate the "DALL-E-2" embedded ChaiGPT/OpenAl API for comparison. The results indicate that synthetic samples provided by all four image generators significantly enhance FL performance compared to the baseline FedAvg that achieves 83.06% on PACS and 70.84% global model's performance better, at the cost higher computational demands. E.g. "DALL-E-2" model leads to higher accuracy, potentially due its ability to produce samples in diverse artistic styles 53
Cafice-Caltech. Note that the models with more advanced designs improve the
#Retrieval
#Generation
96
#Retrieval
#Generation
80
#Samples
400-
Accuree
#5 #amgles
800
Accuracy
600
200
400
92
76
200
v1.5
v1.4
xl-base-1
DALL-E-2
v1.5
v1.4
xl-base-1
DALL-E-2
(a) PACS dataset
(b) Office-Caltech dataset
Figure 13: The performance Flick over different image generators: stable-diftusion--l- stable-
diffusion-vl 11-4, sableediffusiion-xl-bass 1.0, and DALL-E-2.
C.2 Detailed Evaluation under Domain Shift
The data generation process in Flick considers the class-wise performance local models (i.e., through the decision matrix D) and the domain knowledge variations across clients (i.c., the collection of local summaries T used in LLM prompt). This highlights the capability Flick to solve domain shift and label skew in heterogeneous FL jointly and therefore distinguishes Flick from other model- driven [49 16 or data-driven 17 18 21 22] methods that solely address imbalanced local datasets across clients. To validate the performance gain by Flick attributed to mitigating domain shift, Table 6 reports experimental results where each client holds balanced local data label space but only from one domain. shows that Flick consistently yields significant improvement in model accuracy and reduces the number of communication rounds (#Round) to reach the target accuracy-defined
the best accuracy of the baseline methods (i.e., FedAvg, FedProx, FedDyn, FedNAR). Among generation-base counterparts, DynaFed and FedFTG have limited performance gain under the heterogeneous settings with domain shift alone and even show inferior performance to the baselines on the Office-Caltech dataset. For FGL, we match the synthetic data volume to that of Flick. For example, under the FedAvg baseline on the PACS dataset, Flick generates only 300 synthetic samples in total. This limited amount of auxiliary data constrains the performance of FGL a finding consistent with its released paper. In contrast, the same amount proves sufficient for Flick to deliver significant performance improvements. More importantly, compared to methods specifically designed to address domain shift (i.e., FedBN and FedHEAL), Flick further improves the global model performance and convergence by large margin. Combined with the results from Table 1 it is clear that Flick shows great performance in addressing data heterogeneity issues regarding label skew and domain shift.
19

---

Table Global model accuracy performance (%) under domain shift only. AVG denotes the average accuracy calculated across all domains, and A indicates the accuracy gain compared with vanilla methods. Best in bold. Acronyms in the PACS dataset: Photo (P), Art Puinting (A), Cartoon (C), and Sketch (S); In the Office- Cattech dataset: Amazon (A), Caltech (C), DSLR (D), and Webcam (W).
PACS
Office-Cie Office-Caltcch
Methods
S
AVG
A
&Round
W
AVG
A 1 #Round
FedAwg [5]
91.85
95.98
87.88
1247
84
77.47
63.33
70.37
74.931am
+FedBN 26
89.53
93.35
95.63
90.91
92.36401
-0.13
77.60
64.22
72.22
88.98
75.76
0.83
45
+FedHEAL
25
91:00
94.69
95.09
91.37
93 93.04
0.57
79
76.43
65.11
75.93
91.10
77. 14
2.21
38
+Dynalied 27
90.27
92.71
89.68
12.15
-0.32
79.69
65.33
70.37
84.75
15.03
0.10
+FedFTG 19
89.78
92.99
90.77
92.39
91.48=006
-0.99
82.29
65.00
67.96
78.03
73.32+000
-1.61
+FGL (20
89.90
90.13
95.09
89.97
91 21.27
-1.20
61.04
62.
85.19
81.19
72.41=000
-2.52
91.48
94
96.88
92.20
93.76
1.29
24
81.60
87.65
91.53
81.561651
6.63
FedProx
88.56
93.63
94,05
91.50
91. 25
84
78.52
62.56
75.00
90.68
76.69.031
59
+FedBBN 26
89.05
93.74
94.94
89.59
91. 1o.34
-0.11
74.35
64.11
78.70
88.56
76.43
-0.26
+RedHEAL
25
90.88
95.12
95.39
90.99
93.09=611
1.15
41
79.95
67.33
74.07
91.53
78.22+0071
1.53
66
+DynaFed
27
90.02
91.44
95.04
89.97
91.62 Lo44
-0.32
82.30
64.44
72.22
80.51
74.871147
-1.82
+FedFTG 19
87.23
93.95
91.52
89.21
90.48=0.34
-1.46
81.25
64.89
74.26
81.27
75. 75.42+122
-1.27
+FGL. 20
90.43
89.53
94,94
89.47
91 19.09 -0.85
68.77
63.78
85.19
86.27
76.00
-0.69
+Flick
91.36 94.27 95.83
93.46
93.73
1.79
16
80.21
65.33
85.19
94,92
81.4
4.72
16
PedDyya
83.94
91.93 92.11
91.75
89.93
22
79.17
62.44
83.33
88.14
78. 27.277
68
+FedBB
26
86.78
91.30
92.66
91.50
90.56_p000
0.63
22
81.64
65
84,26
90.68
80.51
2.24
79
+RedHEAL
87.59
91.56
91.67=8.0
1.74
16
94,92
79.13+8826
+DynaFed
87.75
91
94.25
90.61
92
1.19
20
79.67
64.67
76.89
91.11
78.09 28.09:
-0,18
+FedITG
86.62
93.10
93.45
91.62
91. _0ur
1.27
20
80.73
65
80.18
84.14 14
77.62...
-0.65
+FGL 20
93.63
92.01
91.79=6005
1.86
13
93.22
81 $1.52
3.25
+Flick
90.02
95.01
95.39
93.
93.39
3,46
16
84
65 56
87.03
94,07
82.69
4.42
23
FodNAR 16
89.86
92.92
96.03
89.81
92. 16
84
77.99
63.00
75.93
92.80
77.43..0
66
+FedBN
90.75
92.57
94,94
90.61
92.222:044
0.06
142
64.00
76.54
77.68:2000
0.25
+RedHEAL
25
94
95.12
95.24
90:61
93.05
0.89
30
80.21
63.56
77.78
91.53
78.27#004
0.84
59
DynaFed
27
89.90
92.94
94.72
89.88
91 86..mm -0.30
81.52
67.
74.04
88.81
77.98
0.55
101
+Fd//TG
88.81
93.84
91,96
91.08
-1.08
82.73
65.11
73,81
75.75
-1.68
+FGL 20
90.39
90.76
95.09
89.78
91. =0.12 -0.65
72.99
64.33
83.33
84.34
76 =5.10
-1.18
-Fliek
92.34
93.74
95.98
92.70
93.69.n11
1.53
16
38
65.33
88.89
88.70 70
80.83.0.54
3.40
25
Table 7: Extended ablation study on the components of client-side local summary and server-side data generation in Flick. We compare Flick with its variants: Flick with random local sample captioning (Random Captioning) and Flick with random historical sample retrieval (Random Retrieval).
PACS
Methods
P
A
C
S
AVG 1
#Round + #Sample +.
Random Retrieval
89.81
91.40
96.28
93.21
92.67=007
19
423
Random Captioning
91.87
91.61
97.32
93.08
93.47=0.33
19
430
Flick
91.99
94.59
97.74
93.91
94.49=1011
11
270
Office-Caltech
Methods
A
C
D
W
AVG #Round | #Sample L
Random Retrieval
78.24
65.78
89.29
73.33
76.6620.52
45
295
Random Captioning
75.13
61.33
92.86
79.17
77.12=020
43
235
Flick
75.82
64.00
96.43
77.78
78.51
150
C.3 Extended Evaluation on Flick Variants
we conduct the ablation study on Flick by evaluating its performance without the Local Summary (LS) phase or the Data Retrieval (DR) during the data generation phase. To further explore the effectiveness of our LS and DR designs- specifically the loss-based LS and similarity- based DR strategies we compare Flick with its variants: random local sample captioning (Random Captioning) and random historical sample retrieval (Random Retrieval). Table 7 summarizes the results under the setup of FedAvg as the baseline method for both benchmarks. For fair comparison, the server in the Random Retrieval retrieves the same number of samples as Flick but randomly selects from the server-held data pool. This ensures that Random Retrieval maintains the same proportion of retrieved samples Drel. and generated samples Dgen as Flick during the entire data generation phase across all communication rounds. The experimental results clearly reveal the benefits of selective local sample summarization and guided historical sample retrieval in Flick over their random counterparts.
In Table 3
20

---

Random local sample captioning results in inefficient extraction of client-specific knowledge, as it lacks the focus on captioning the representative local samples provided by the loss-based criteria. Furthermore, the negative impact of introducing randomness into data retrieval is even more severe in Random Retrieval, where the central server randomly sends diverse synthetic samples that fail to align with the specific requirements of local data compensation, thereby degrading the training efficiency.
C.4 Scalability Analysis DomainNet Dataset
To evaluate the scalability of Flick in large-scale federated settings, we experiment on the DomainNet dataset [5 with 100 clients, where 20% are randomly selected to participate in each communication round. The DomainNet dataset consists of natural images with six different domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. Following prior work [20 26 46 construct sub-dataset for our experiments by selecting the top ten most common classes: airplane, clock, axe, basketball, bicycle, bird, strawberry, flower, pizza, bracelet. We partition each domain into or 17 subsets using Dirichlet distribution with concentration parameter a = 0.1. Each client receives data from single domain with skewed label distribution, simulating a scenario with both label skew and domain shift. We fix the synthetic data volume for FGL at 1500, while the total number of generated samples (#Total Sample) in Flick is lower 1457. 1133, 1412, and 1299 for FedAvg. FedProx, FedDyn, and FedNova, respectively. Besides, the number of samples required to reach the target accuracy only 427, 414, 417, and 477, highlighting that Flick achieves superior performance with substantially less generative overhead.
Table 8: Global model accuracy performance (%) under both domain shift and label skew on DomainNet dataset. AVG denotes the average accuracy calculated across all domains, and A indicates the accuracy gain compared with vanilla methods. Best in bold.
DomainNet Real
Methods
Clipart Infograph Painting Quickdraw
Sketch
AVG
f
#Round
FedAvg
82.59
46.99
71.60
77.63
85.95
80.00
74.1320.8
144
+FedBN 26
83.64
$2.00
72.41
69.78
91.30
77.36
14.41
94
+FedHEAL
86.50
53.18
74.33
72.42
90.65
80.00
76.
2.05
53
+Dynaled 27
86.13
50.31
69.22
85.90
87.16
81.40
76.68
2.55
100
*FedFTG 19
84.31
50.00
71.20
80.87
87.69
74.69
74.79
0.66
114
+FGI, 20
91.09
55.56
79.71
79.07
92.21
84.80
80.41
6.28
5
+Flick
92.11
63.90
83.88
91.40
92.91
90.34
85.762000
11.63
11
FedPrrx
85.32
52.59
72.99
83.37
88.15
80.76
77.2020.0
144
+FedBN 26
83.48
52.73
72.34
66.63
91.49
77.66
74.06=0.2
-3.14
+FedHEAL
87.36
54.84
74.88
75.91
92.33
81.86
77. 77.86=0.12
58
+Dynaledd 27
86.94
53.94
73.18
82.60
87.56
81.79
77.67
115
#FedFTG
86.82
51.79
73.86
86.07
89.72
81.45
78.2810.30
114
+FGL. 20
91.40
54.98
80.98
81.00
92.79
85.03
81.03+0-48
14
+Flick
92.81
64.94
84.94
91.60
93.07
89.66
16.17
8.97
21
FedDyn 49
88.06
$5.39
77.08
75.60
90.32
81.03
77.91 20:22
34
+FedBN 26
87.22
54.79
78.82
77.90
89.72
80.48
78.16
0.25
45
+FedHEAL 25
86.51
56.97
74.66
75.03
91.89
82.68
77.96=021
0.05
93
+Dynaledd 27
84.41
53.53
74.11
81.73
89.46
81.03
17.3
-0.53
#FedFTG 19
86.13
54.05
74.50
82.20
88.80
81.93
77.94
0.03
31
+FGL.
87.35
$4.67
76.55
80.17
89,99
82.76
78.5820.13
0.67
31
+Flick
89.68
60.37
83.09
87.87
91.14
87.03
83.20
26
FedNAR 166
89.88
56.85
75.36
86.10
91.69
81.93
80,30
150
+FedBN 26
85.50
58.22
76.50
78.90
93.33
81.17
78. 28.94
-1.36
+FedHEAL 25
88.43
$7.05
77.37
77.87
93.87
83.69
79.71 +0,22
-0.59
Dynaleed
27
89.77
56.08
77.16
83.63
92.60
84.38
80.6010.21
0.30
150
#FedFTG
19
90.72
53.10
74.63
84,50
94.71
84.41
80.3410.10
0.03
114
+FGI, (00
90.38
57.47
81.11
83.83
92.65
85.24
81.78
1.48
79
+Flick
91.30
64.73
84.02
90.40
93.11
89.10
85.44
5.14
36
D Overheads
Compared the model-driven methods for heterogeneous Fl., the data-driven Flick introduces additional overheads to the clients, including extra training and captioning latency. However, the
21

---

94
Original data
Captioning
Time (seconds)
500
Synthetic data
Accuracy
76
400
58
FedAvg
+FedBN
Fliick (aurs)
300
+FGL
FedHEAL
0.0
0.6
1.2
1.8
200
5 17
18
19
23
39
49
Training time (hour)
Participating Round Index
Time-to-Accuracy performance
(b) Local latency rounds
Figure 14: Evaluation on the temporal scale: (a) Wall-clock training time of Flick and its model- driven counterparts on the PACS dataset, running on an NVIDIA A40 GPU; (b) Wall-clock local latency of a specific client, running on an NVIDIA Jetson AGX Orin, across participating rounds, including training on both original and synthetic data points, as well as local captioning.
Table 9: Resource usage comparison between model-driven FedProx and data-driven Flick across three benchmarks. We report per-round and accumulated-to-- to-rggett (with target defined as the best performance of FedProx, highlighted in gray) resource consumption using four metrics: Peak GPU Memory (PGM), GPU Memory Hours (GMH), Energy (E), and Running Time (RT). For GMH and E, we provide the dynamic range observed across the communication rounds. Lowest in bold.
PACS
DomainNet
Metrics
FedProx
Plick
FedProx
Plick
FedProx
Flick
PGM (GB)
8.33
14,53
4.77
10.50
13.74
20.47
GMH (GB-hour)
[0.049, 0.064]
[0. 152, 0.353]
[0.016, 0.022]
[0.084, 0.376]
[0.529, 0.681]
[1.407, 2.1
Bx10-4 (kWh)
[0.89. 80.62)
[5.64, 164.60]
[1.81, 52.47]
[1889, 90.90J
18.23, 273.60J
18.06, 1207 [00]
RT (hour)
0.96
0.55
0.50
2.11
GMH (GB-hour)
7.91
2.63
5.25
91.79
34.61
E (kWh)
0,149
0.075
0.063
0.050
1.329
0.444
performance gain from Flick remains significant: it facilitates the model convergence, as reflected in both reduced rouuc-to-accuracy (#Round) and training time, while also improving the top-1 accuracy of the global model in the long run.
Figure 14 shows the learning curves of Flick and its counterparts, integrated into the baseline method FedAvg, with time-to tnee-o-cccuracy performance evaluated on an NVIDIA A40 GPU. The faster convergence offered by Flick significantly counteracts the negative impact from additional overheads.
We also trace local latency variation over communication rounds by measuring the time consumption of single client running
NVIDIA Jetson AGX Orin, shown in Figure b).. We break down the additional overheads into training on synthetic data and captioning the representative wtwp://pppiiiaaaapppiiiiiiiiieie
samples, which employs the and text encoders. It is obvious that the local overheads Flick stabilize over time and constitute only small fraction of the overall local latency as training proceeds. In contrast, FGL the best-performing counterpart incurs substantial delay in the first communication round due to the extensive preparation of synthetic data, leading to longer time to reach the target accuracy.
with ViT and BERT image
We further investigate storage and GPU memory requirements across three VLP-based captioning models: BLIP-Large requires 1.88 GB of storage and 3.19 GB of GPU memory; BI.I--Base requires 0.97 GB of storage and 2.30 GB of GPU memory; ViT-GPT2 requires 0.96 GB of storage and 2.48 GB of GPU memory. Moreover, Flick shows consistent performance gains across these three image captioning models as shown in Figure II Such flexibility enables clients to choose models that are best suited to their available computational budgets.
Additionally, we compare the resource usage of data-driven Flick with the model-driven baseline FedProx from three aspects: 1) peak GPU memory (GB) during training, 2) GPU memory hours (GB (GB-hour), akin to GPU hours but weighted by runtime memory usage, and 3) energy consumption (kilowatts- We collect these statistics on an NVIDIA A40 GPU for both FedProx and Flick across the three benchmarks. The Table 9 reports the per-round resource usage and the accumulated usage required to reach the target accuracy (i.c., best accuracy of FedProx). Since GPU memory
22

---

hours and energy consumption vary across communication rounds, we provide both the dynamic range and the average value over all rounds. For Flick, GPU memory hours and energy consumption show a wider dynamic range because the overhead from key components changes across rounds. Specifically, Flick relies 'heavily' on L.M and L.DM for synthetic data generation during the early training stage, which takes the majority of the overall cost. As training progresses, both GPU memory hours and energy consumption per round Flick decrease. In contrast, the variation in these two per-round measurements for FedProx is mainly due to the randomness of participating clients in each round, where heterogeneous data silos introduce varying local training overhead.
From the results, we observe that for the Office-Caltech dataset, Flick requires more GPU memory hours than FedProx to reach the target accuracy. This is because the additional GPU memory for LLM and LDM far exceeds that required for local training, thereby counteracting the resource savings from faster convergence (i.e., fewer communication rounds). However, Flick still offers shorter training latency to the target and achieves higher best accuracy than FedProx. For more complex tasks, such as the DomainNet benchmark with large-scat clients and data, Flick outperforms conventional FL. across all metrics: it reaches the same performance as counterpart FL frameworks with lower training latency, GPU memory usage, and energy cost; furthermore, it achieves better accuracy in the long run. To conclude, although Flick introduces additional overhead, primarily on the server, it enables faster convergence of the global model compared to FedProx. In most cases, Flick further reduces resource consumption to achieve the same target accuracy, particularly in challenging settings with large-scale scale clients and datasets. Furthermore, with continued training, Flick offers substantial accuracy gains; for example, the PACS dataset, Flick achieves 11.43% higher accuracy than FedProx after the same number of communication rounds.
E
Privacy Analysis
Flick can ensure the e-DP guarantee by adding noise sampled from the Laplace distribution the token sequences, with the scale parameter in the Laplace mechanism set to l/e. It provides stronger privacy protection
decreases. Table [0 shows the trade-off between system utility (i.e.. Flick performance) and privacy level, with the ranging from 0,05 to 0.5. Note that the performance gradually degrades towards the Flick w/o Local Summary variant as the decreases. Even under strict privacy constraints, Flick maintains promising model performance.
Table 10: The performance of Flick across different values.
PACS
Office- CalieehCltechh
AVG f
A f
#Round
AVG t
A 1
#Round J
0.5
94.321021
11.26
11
78.24
7.40
38
0.1
93.78=0.24
10.72
25
77.61
6.77
43
= 0.05
92.99=0.11
9.93
31
77.05
6.21
43
Table 11: Privacy assessment of local summary phase in Flick. The server utilizes five image generation models reconstruct the data from collected local summaries. The similarity between original raw local data and reconstructed data is measured by three widely used metrics, with reference ranges provided to indicate the threshold below/beyond which pair-wise images can be considered dissimilar. PSNR (54): peak signal-to-noke ratio; SSIM (55% structural similarity index measure; LPIPS [56]: learned perceptual image patch similarity.
Metries
dall-c-2
dall-e-3
v1.5
v1.4
xl-base-1
PSNR (< 20)
7.77
8.11
7.79
7.80
9.37
SSIM << 0.3)
0.20
0.16
0.17
0.14
0.20
LPIPS (> 0.6)
0.72
0.74
0.78
0.77
0.72
Besides, we assess the potential information leakage in local summaries from a different angle: the difficulty for the server to infer raw data from collected token sequences. To this end, the central server feeds the local summaries (in the textual format) to text-to-image model, and we evaluate five different models. Table TT reports the reconstruction quality based on three widely used metrics-peak signal-to-noise ratio (PSNR) [54], structural similarity index measure (SSIM) (55). and learned
23

---

percepiual image patch similarity (LPIPS) [56] whicc measure the similarity between reconstructed images generated by server-held models and the original raw local images. For PSNR and SSIM, higher values indicate greater similarity, whereas for I.IPS, lower values indicate higher similarity. The results across all five image generators show that reconstructed images remain significantly dissimilar to the original raw data across three metrics, demonstrating that uploading local summaries in Flick presents a low privacy risk.
Practical Impact
Although versatile foundation models are rapidly advancing and becoming increasingly accessible, the design of Flick remains centered on training task-specific lightweight models a crucial requirement for edge deployment under privacy constraints and limited computational resources. To articulate the practical impact of Flick, we recognize it is important to clarify the following two questions:
Given the rupid udvancement of versatile foundation models, what role do conventional models (e.g..
ResNet) still play?
and the follow-up:
How such powerful foundation models be leveraged benefit the training of lightweight, task- specific models, especially in privacy-preserving federation way?
For the first question, despite foundation models offering broad generalization capabilities across a wide range of tasks, directly deploying them in real-world applications, especially on resource- constrained edge devices, is still challenging (exceeding resource budgets) and is overkill (leading to unacceptable latency) for specific tasks. Alternatively, relying on remote cloud-based inference via APIs introduces significant privacy risks, it requires users upload private sensitive raw data to third-party services. Together, these limitations indicate that both the versatile LLMs and lightweight models offer practical values depending deployment constraints such resource budgets, latency tolerance, and privacy preferences. Consequently, there remains a strong and persistent demand for lightweight task-specific models that balance efficiency and effectiveness in deployment.
Table 12: Performance comparison between Qwen-serice MLLMs and a lightweight ResNet- 18 on the DomainNet dataset. Note that ResNet-18 is trained under federated setting with both domain shift and label skew across silos. AVG denotes the average accuracy calculated across all domains.
DomainNet
Methods
Clipart
Infograph
Painting Quickdraw
Real
Sketch
AVG
Awen2-VL-777 57
96.97
91.08
93.53
63.13
97.05
98.07
89.97
Bwvn2-VL-22b [5
97.37
90.66
92.21
62.07
96.78
97.79
89.48
Qwen22.5VVL-7b
97.77
92.32
94.32
62.73
97.31
98.21
90.44
Qwen2.5-VL-3b
58
96.36
88.59
92.73
57.27
97.18
96.55
88.11
FGL 200
91.09
55.56
79.71
79.07
92.21
84.80
80.41
Flick
92.11
63.90
83.88
91.40
92.91
90.34
85.76
To support our claims, we investigate the runtime performance of Qwen-series 57 58 Multimodal Large Language Models (MLLMs) on the DomainNet benchmark, and report the inference latency on takes 7.05 seconds per sample, while the lighter Qwen2-VL. 2--22 still requires 5.94 seconds. The latency severely exacerbated under CPU-only mode: 239.43 49 seconds for the D0wen2.5-VL-3P and 169.93 seconds for the Qwen2-VI. VL.2BB We emphasize that the Jetson AGX Orin is already considered powerful hardware for edge computing. In contrast, ResNet-15 requires only 4.73 seconds per sample under the same CPU-only configuration. When using GPU acceleration, the inference time is further reduced to 0.4 seconds per sample.
an NVIDIA Jetson AGX Orin. Under GPU-mode, Qwen2.5-VL
We further compare the classification performance of Qwen-series MLLMs with lightweight ResNet- trained under Flick framework on the DomainNet benchmark, as shown in Table [2] We emphasize two points for fairly assessing the desNet-18 performance and Flick contributions:
1) training under heterogeneous FL settings inherently degrades model performance compared to centralized training; and 2) our data-driven Flick addresses such challenges more effectively than the similar M-sed FGL by efficiently incorporating commonsense knowledge from L.MSs. The results show that Flick achieves competitive global accuracy on DomainNet while maintaining edge
24

---

practicality. We also note an interesting observation from the DomainNet results, particularly on the two challenging domains, 'Infograph' and 'Quickdraw': all four MLLMs perform relatively well on the 'Infograph' domain, while showing generally inferior performance on the 'Quickdraw' domain. As described in DomainNet [45) 'Quickdraw' images are collected from Google's 'Quick, Draw!" game, where the drawings are very simple, abstract line sketches; 'Infograph' images are typically informational graphics such as posters, diagrams, and other structured visual layouts. This distinct performance gap between the two domains suggests that: 1) MLLMs inherently have a strong understanding of complex icons, symbols, and text arranged in a structured format; and 2) they still have limitations in handling certain tasks. This observation underscores the value of FL frameworks, which allow preaeering model training for such domain- sppaiin---eiicc tasks using task relevant, decentralized data.
By answering the follow-up questions, we aim to highlight the contribution of our data-driven Flick in improving the performance under heterogeneous FL. by effectively leveraging powerful foundation models. In the FL. context, conventional approaches focus on optimizing training over distributed data silos, which inherently limits the upper bound of accessible knowledge. This performance ceiling can be lifted by distilling external task-related knowledge embedded in foundation models, which is actually our target in Flick.
G Data Generation Details
In Section 3.3 we have introduced the data generation process in Flick. In this section, we provide some more details- specifically for the LLM usage and data retrieval - by showing the intermediate results.
The data generation phase bootstraps the server to obtain the class-wise decision matrix D. After that, an LLM is employed to generate text prompts for each class. Figure [5 provides a detailed case example to supplement Figure in the paper, illustrating how the LLM produces G = 5 text prompts for the class dog, using a collection of local captions T as reference. Recall that, for privacy considerations, our design in the local summary phase only requires participating clients to report captions of representative local samples while omitting class information. Consequently, the LLM has to first identify and select class-related captions from the pool T. From the LLM output in Figure [15] it evident that the LLM can effectively select captions relevant to the class dog, extending beyond simple keyword matching. For instance, it also identifies captions containing terms like
Input Prompt
Local Captions
Given lie copeption: (7'), select those that relevart stlyect (Sng). Then.
"there is black and waite duaking of dog with a stick is south""
analyze solected coprions, which depict varions around the main
but consiseeriyy
"dowing of mon sitting or beich wath his arns crossed"
frog):
"an image of's Mack and wine drawing of persont hand holding toothbeush"
Banod this aralysic grnenie (5) now test prompp that stll foom main abjest (dar), Yies fess domains, eylis, crntiss conteats from provited caplioe, bui cienents invodtasal
"them a black and white drwwin of hone on white badiground
"dhwiing of diphent with hoe
not detrect from obaure the main azbject. demeraa escounged, but they ssculd cresc the contect raher thas skift the focus away fiom die main suli Your respoase should use the followng foenuc: p'portpei",. "i. "poonst (5)") of the aggiins ae related to the catepacy (oog), vetust enjty lis:
"fuur centon dogs with wings and bones difinenni posot"
which rerain the centrdl fecto. Intodaiing
"an imags of cartaon hasec with helehhll koup foom of
"a imass bron with
head"
is elestaes
sandin
molde the pictue"
vileat parting geldes retticer lomaing on watay pato"
as painting group people siting the gras"
cD LLM Response
"there is gintte
painting cottaee with fhoched
standing
avtther giratl""
Selecting target class related captions
painting
canding stending the army park painting clepham
logs"
relevast that
subject of dog" include:
paylii prpery chisiig
black white drowing
moutt?
"hour
vihrr painting goiden veriner lonngng
with
botns ditioont poses"
patie"
picture
water"
'A playful pappy dosing hiteefiis a <Stcooning:
filld with flencrre
These exptiens depict varine scmo invlling dop, shuncasing amotiona playful sature, and diffcrent poics.
Based os dhis aralvso, will create sew cuptioa that maintais the Socas on dasp while intodacing sew dienents that cilianee dhe contest.
Ratrevol
-Genentting text promtts:
Outyet tie 5 new captioes stretly is the Sillowngg fornot: ("here a colorfil gainting of's dog wearing panty het sumoundedd balloons".
"In image of's dog plaing frack with frishe in anmy pars..
M
"s certion dog dimood os espetturo flying through the da cape".
Thhne a whimoral drowing of's day sitinng on doest with minbho the backgaandd
Generadon
There is back and white sketch of& dog bing or cory ong noi to fteppaoe"
Figure 15: Detailed illustration of LLLM-base text prompt generation: given cross-client-specfiic knowledge T", the central server obtains the text prompts for the class dog in the PACS dataset. The underlined words are recognized by LLM as related to class dog.
25

---

Input Prompt
Local Captions
Gives lsi of captions 17"),, select those that are oelerant the ssin wdyjet tleuiphhncc))
Taa imsage of laptep coupuar with anall scrcce if
Then, analyre the selected eaptors which depect vances sonee and contoca for conersenaty center cround woin subject (besiphones).
Tans image of vhine and hiw min akrach machins on ohinn hadk heddggeend
"elure iia adl phcee that ltying the groand carmended by laaved
(bcadsksees). You grveate
prorpe
provided oitjer.
"dane s like with a log en fre leck of and bog er dee beck ofir"
"ducne a pair headphane entng on with findk will bockgramne
which seatt conais the central loco. intodaainng dicness anouragol tiey should
bas ooeare that
dierants irmofaced to demes from otherne
"dao
is corigatel rivone les mecegge on book
cwich the contest ratker thanslill the focus Eor the sujjet
comppater
dekk white cheir"
opptions are reloed to the ontegery (hodfpooess] return as orpty lst.
Your ropane abould sac the tollwning fiematt pprorree"" o. "preeept 1o)') i. none the
"iki image kaptap with keyboard oe ofp
tdloro with harkyart woiking trall toemmnttionn
"ducn iis competer dedk with buyfword moses and mentior
close up of calosbore on white bockgronnd"
DDL Response
"ducre a pair of rod baudpheezwwitnng gray axtacc"
"darro i whate bicylle will besker of Dowes on the offi"
corpgater keyfoood mosee
SSllecting larget class-oelated captionss
plase"
that focts saleezt "eadpkone"
uakee"
thom of haudphones
tohtr with brick wall horkyrondd
"duom where with btack tryhoart
narthrr"
Maree suaa of les siliicssiiingg ot fave"
"dai innge of compaais musiter and kes lourd en dook"
"hhr in par of tlack heafphonc on woades savle"
"dais image of ponanic device siting on woodee oavie
"ducre a ca celaser exting on weeden table with caletitoois froet of
Those captions condieent dep hee tondllonesii vaskoss contests, sach as plicod on
"darro inn bike that loining against wall in rooe""
suriices wedeopece actting.
"dare a pair of hendplastz or weeden tabik"
gaventt captines
oo hoodphooes
cop ehiting on
introduce dillaeet
dessara
the dosiitio..
hackppriic
silling
weeden
coffee"
Ourpur the
etricly the fellowing fiemat:
cnlcutitiom sitingg nast ouch other white warfoo""
(% visaan pair of Deudjpooon ress a slock bads delk, ilassiatt by will natural light. a por of headphonee byny on uandy seacs with oceas waves in the desunce.
"dki priture of sossa and a boddge cooss pedf
"ducre a competter with exthhmeoase aest a layboard and mozee"
pur of nose canoding hadgliees with andi cadionn a cosy coads:
"duree a cokeraa bookguck sittag cn bench is the park""
por of wirkies hoxdyhoroe honos floring shooe ciyy iry umeet. pai chadpphnnes poadioned aps liptop, with coloctal woad wancs ilaatrated
"durro is plraue of liprop with a picter people servot"
2712
Ratrieval
mf
M Generation
Figure 16: Detailed illustration of the LLM-based text prompt generation for the class headphones in the Office-Caltcch dataset.
"golden retriever" "puppy". During the reasoning phase, the LLM analyzes the selected captions knowledge) with new elements (i.e.,
and recombines extracted features (i.e., pross-client-seee commonsense knowledge embedded in the LLM) to generate text prompts for the subsequent text-to- image model. Besides, Figure 16 illuststtsss the text prompts generation for the class headphones in the Office-Caltchh benchmark where the L.MM showcases similar capabilities in extracting class-related knowledge and instilling commonsense knowledge.
To reduce the data generation overheads, the server in our Flick does not directly feed all text prompts into a generative model for new data points. Instead, it first retrieves historically generated samples from the server-held data pool G.. The qualified historical samples are retrieved based on the pairwise similarity between the historical text prompt m E M, and the generated text prompt m E M; Specifically, each prompt pair (mg, m is transformed into SBERT embeddings used for calculating cosine similarity. In Figure 17] we provide more examples of the data retrieval step across two benchmarks and their respective four domains. For detailed illustrations, Figure 17 presents the text prompt pairs (mp, m?) ) and their SBERT similarity scores, as well as the sample corresponding to the historical text prompt my that the server retrieves.
Sindlaripten ont) 04814 wO vikvane amwork of sumponded by coloriul flawers an
Sinllartoon anl) 8.353
we" "A quirky certase harse bright blus mohawkk and chhekky grie" well to whine iaal castoon horee and a cheertul exprestion"
Siedwetyttee snfts of large window" unft man standing by conterpllatios
wigh la invssdhiaee dieeck of gallop tng scross a cammen
thaughttally
wdfh gicafle wandering with colortul
wift a mosss dhesess dkerck
Sindlrropen od's osasT
Skrdaarbppn inf) 82783
Sindarggtr snf) oaros
"s sook monitee diyboylogia
WeCrteere projectrr displaying
wed vibrent gantng kayboard Blandeated npon calore dark bnlegoondd"
mecra blea januporter buckpeck diding on villos salee rost
lasartlasdacpp modem
in Ik.somm ai collee reaelty
=ci thereis compater mosiitoe Image an deslc dedi""
-here is projector en table ou of coffee'
unf: there a mechunical
nn. "hhae la blue bedgaack
lighting on wedden table"
ptop and remelwoak
Figure 17: Illustration of the retrieved samples in two datasets: PACS (top) and Office-Caltech (bottom). In each example, my and my represent historical text prompt and generated text prompt, respectively. In Flick, the server retrieves samples from the data pool for which the corresponding text prompts show high similarity to the generated text prompts.
26

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly state our key contribution: our Flick greatly promotes global model performance in heterogeneous FL by distilling task-relevant, high-quality commonknowledge from LLMs, guided by gros--liiet-speccin information.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss and analyze the limitations of our work with respect to privacy (Appendix and computational overhead (Appendix D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detailed information required to reproduce the results is provided in Section 3 In particular, Algorithm outlines the specific implementation process.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully specific model and empirical evaluation, may
might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
28

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets involved in this paper are publicly available, and we have included link the code in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
htt.://oooooabsiissooooool...o.o for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
sow.suoooossssoooooocococooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify them in Section 4 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In the performance comparison, we report both mean accuracy with variance under each setup three times.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative eтror rates).
show in tables
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources detailed in Appendix B We report time cost in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
The research is conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
30

---

Justification: There is no societal impact of the work performed since our research focuses on the general studies in the FL.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: While our paper involves data generation using LLMs and generative models, we do not release any generated data. We have taken steps to ensure responsible use. The generated data are fully synthetic, guided by task-specific prompts, and used solely for experimental purposes.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original papers or websites that produce the code package or datasets.
Guidelines:
31

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi wwwrrw.ethcc.....m com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/coddemmodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
32

---

Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLMs are an integral part of our propose method. Specifically, we leverage them distill task-relevant, high-quality commonsense knowledge - guided by cross-client- specific knowledge to enable effective learning in heterogeneous FL.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy hbtp::///maauiioococc/o/foerrrer for what should or should not be described.
33


Reviews:
[{"review_id": "7juEe6Wdj3", "reviewer": "Reviewer_Bcqt", "summary": "The authors propose a novel framework named Flick, designed to address the heterogeneity of classic data silos in federated learning. It leverages the common-sense knowledge of LLMs to guide data generation, thereby resolving label and feature distribution shifts. Specifically, clients perform local data summarization to capture client-specific knowledge in textual form. The server then extracts high-quality, task-relevant knowledge from the LLM and uses it to generate informative textual prompts. These prompts guide the generative model in producing synthetic data, enabling global model fine-tuning and local data compensation.", "questions": "1. How can the generated summaries be evaluated?\n2. Are there more reasonable selection methods for sample choice?\n3. During sample annotation, the VLP model suffers from semantic loss when converting images to text. Could this lead to missing key features in the textual summaries?", "limitations": "yes, but it's given in supplementary material. it should be mentioned briefly in main text.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1. The experimental results show improvement, demonstrating the method's effectiveness.\n2. The combination of LLMs and federated learning (FL) is novel, with a reasonable and effective approach.\n3. Enhanced privacy: clients only share text summaries, preventing the leakage of raw data.\n4. The study includes comparisons with baseline models in terms of accuracy and convergence speed, as well as results in heterogeneous scenarios. Ablation experiments and a Performance across local generative methods comparison are also conducted.\n\nWeaknesses：\n1. The use of local summarization methods may lead to the loss of key features and insufficient representation of heterogeneity, thereby introducing new issues during the generation process.\n2. During sample selection, there is a risk of failing to capture intra-class diversity, as well as overfitting or selecting low-quality data.\n3. More evidence is needed to support privacy claims, and experiments under extreme data heterogeneity are lacking.", "was_revised": true, "final_justification": "Thanks for authors' rebuttal. most of my concerns have been addressed. I keep my score.", "rebuttal": "We thank the reviewer for the insightful comments. Below are our responses.\n\n`[1/4] The use of local summarization methods, i.e., sample annotation by VLP model, may lead to the loss of key features and insufficient representation of heterogeneity, thereby introducing new issues during the generation process. How can the generated summaries be evaluated?`\n\nWhen generating the local summaries, we aim at an underlying trade-off among informativeness, privacy, and overhead. Specifically, we aim to ensure that the textual summaries are low sensitive, thereby making it difficult for the server to reconstruct the original data (as discussed in Table 10 in Appendix E). Besides, the overhead of image captioning constrains the scale of local samples for summarization. Therefore, in our design, each client follows a loss-based rule to selectively caption one representative data point per class, thus providing the server with effective yet privacy-preserving local summaries at an acceptable cost. The effectiveness of the local summary is also resilient to the choice of the VLP model, as demonstrated by the results in Figure 10 in Appendix C. \n\nWhile designing a reasonable metric to directly and quantitatively evaluate the quality of the summaries is non-trivial, the ablation results in Table 3 (w/o local summary) and Table 6 (random local summary) show a clear accuracy drop, which strongly shows the effectiveness of our local summarization design. \n\n`[2/4] During sample selection, there is a risk of failing to capture intra-class diversity, as well as overfitting or selecting low-quality data. Are there more reasonable selection methods for sample choice?`\n\nOn one hand, each participating client selects one representative data point per class with minimal captioning overhead. We clarify that the selected sample varies across the communication rounds, even for the same class within the same client, which benefits capturing intra-class diversity in the long run. On the other hand, the loss-based selection strategy ensures that clients choose clean, undistorted samples for underperforming classes and impactful samples for well-performing classes, thereby avoiding captioning low-quality data. The ablation results shown in Table 6, where random sample selection leads to a clearly lower accuracy, further validate the effectiveness of our design. \n\nWhile methods such as hard-sample selection [1] could potentially improve the sample selection in Flick, directly adopting them is not suitable due to the overhead constraints and the limited, imbalanced nature of data silos in our case. We leave this as a promising direction for future work.\n\n[1] Angelos Katharopoulos and François Fleuret. \"Not all samples are created equal: Deep learning with importance sampling\", ICML, 2018.\n\n`[3/4] More evidence is needed to support privacy claims.`\n\nWe have already investigated the privacy guarantee of our selective sample captioning mechanism from two perspectives: 1) the formal $\\epsilon$-DP guarantee (Table 9 in Appendix E), and 2) the potential risk of reconstructing raw data from the textual captions (Table 10 in Appendix E). We would appreciate it if the reviewer could clarify the additional evidence that should be further considered to support our privacy claims.\n\n`[4/4] Experiments under extreme data heterogeneity are lacking.`\n\nIn response, we conduct additional experiments under a more extreme heterogeneous setting, where each client holds samples from only two random classes on the large-scale DomainNet dataset with 100 clients. We report the results in the following table.\n\n|Methods|Clipart|Infograph|Painting|Quickdraw|Real|Sketch||AVG|Δ↑|#Round↓|\n|-|-|-|-|-|-|-|-|-|-|-|\n|*FedAvg*|70.11|45.02|62.00|64.22|84.19|63.86||65.03±0.20|–|143|\n|+FedBN|71.28|42.11|61.22|65.93|83.26|61.59||64.23±0.46|-0.08|-|\n|+FedHEAL|70.24|41.29|62.48|75.40|81.55|62.34||65.55±0.32|0.52|84|\n|+DynaFed|71.93|42.71|67.49|70.23|82.93|61.41||66.28±0.27|1.25|95|\n|+FedFTG|71.99|45.95|62.73|65.87|81.09|63.31||65.15±0.40|0.12|90|\n|+FGL|76.52|49.72|71.86|57.98|90.98|74.16||70.20±0.79|5.17|19|\n|**+Flick**|84.01|53.63|76.68|81.73|90.12|82.34||**78.09±0.12**|**13.06**|**18**|\n\nCompared to the Dirichlet distribution with $\\alpha=0.1$ (Table 8 in Appendix C.4), we can observe that the accuracy of FedAvg drops from 74.13% to 65.03%, indicating an extreme heterogeneity under this setup. The experimental results show that our Flick consistently outperforms the baseline methods, and the performance gain is actually more significant as the data distribution across clients becomes extremely heterogeneous."}, {"review_id": "4Jb5xjfFyM", "reviewer": "Reviewer_yj5i", "summary": "The paper proposes Flick, a data-driven framework designed to address data heterogeneity across silos in federated learning. Flick leverages commonsense knowledge from large language models (LLMs) to generate synthetic data, enhancing data diversity and balance. To protect local privacy, clients share only textual captions describing their local data with the server, rather than the raw data itself. The server then uses these captions to create text prompts via an off-the-shelf LLM, which guides a generative model to produce synthetic data for both server-side augmentation and local use.", "questions": "- The retrieval mechanism seems to introduce duplicate generated samples across different clients or even on the same client. Could you provide quantitative insights on how frequently this duplication occurs in practice? Additionally, how does this duplication impact model performance, either positively (e.g., through regularization) or negatively (e.g., by reducing data diversity)?\n- The paper mentions that stale sample removal is applied only to the server-side dataset. Would applying a similar strategy to the client-side local datasets also be beneficial? Could you clarify the rationale for restricting this operation to the server dataset, and whether client-side stale samples could similarly degrade learning performance?\n- In line 126, the paper suggests that client datasets progressively approximate an IID distribution over time. However, this assumption may not hold if certain classes inherently present greater learning difficulty. Could you expand on how the proposed method addresses or accounts for such class-specific learning challenges in the non-IID setting?\n- In line 166, the sample selection strategy favors choosing samples with the smallest loss for underperforming classes, under the assumption that such samples are less likely to be noisy. This makes sense in early training stages. However, in later stages, prioritizing harder (i.e., higher-loss) samples might better facilitate learning. Could you clarify why your method consistently favors small-loss samples?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and easy to follow.\n- The evaluation is thorough, providing comprehensive results that help assess the effectiveness of the proposed approach.\n\nWeaknesses\n- Since federated learning often involves private or domain-specific datasets, off-the-shelf LLMs may lack the relevant commonsense knowledge to accurately represent or augment such data.\n- The motivation behind certain design choices and methodological components is not clearly explained. (See the detailed questions below for clarification.)", "was_revised": true, "final_justification": "Authors have properly addressed my concerns. I would like to maintain my positive score.", "rebuttal": "We thank the reviewer for the insightful comments. Below are our responses.\n\n`[1/5] Since federated learning often involves private or domain-specific datasets, off-the-shelf LLMs may lack the relevant commonsense knowledge to accurately represent or augment such data.`\n\nTypically, the off-the-shelf LLM (e.g., gpt-4o-mini used in our experiments) contains rich knowledge spanning a wide range of domains. We want to highlight that our proposed pipeline in Flick can effectively query task-relevant information from such an LLM and instill it into the data generation process. Furthermore, in practical deployments, the server in Flick can substitute the off-the-shelf LLM with a task-oriented LLM to better capture domain-relevant commonsense knowledge.\n\n`[2/5] The retrieval mechanism seems to introduce duplicate generated samples across different clients or even on the same client. Could you provide quantitative insights on how frequently this duplication occurs in practice? Additionally, how does this duplication impact model performance, either positively (e.g., through regularization) or negatively (e.g., by reducing data diversity)?`\n\nThere indeed exist some duplicate generated samples that can occur both within a client and across different silos. We define the intra-client duplication as the proportion of duplicated samples within a single client, and inter-client duplication as the proportion of duplicated samples shared across clients. For example, in our PACS experiment using FedAvg as the baseline (Table 1), the intra-client duplication across all clients ranges from 0% to 2.86% (with an average of 1.35%), while the inter-client duplication ranges from 4.89% to 19.30% (with an average of 12.79%). \n\nWhile such duplication may slightly reduce data diversity, it also brings benefits by reducing generation overhead through sample reuse. This reflects a trade-off between the efficiency and effectiveness of the data generation process. The degree of duplication is controlled by the hyperparameter $T_ s$, which defines the text similarity threshold for data retrieval. As shown in Figure 9(a), a higher $T_ s$ reduces duplication by favoring the generation of new samples over retrieval. We observe consistent performance gain by Flick across $T_ s$ ranging from 0.6 to 0.9, and we set $T_ s=0.8$ as the default in our experiment.\n\n`[3/5] The paper mentions that stale sample removal is applied only to the server-side dataset. Would applying a similar strategy to the client-side local datasets also be beneficial? Could you clarify the rationale for restricting this operation to the server dataset, and whether client-side stale samples could similarly degrade learning performance?`\n\nWe clarify that the update mechanism for the server-side data pool $\\mathcal{G}_s$ is tightly tied to its function. Removing stale samples helps prevent overfitting problems and ensures that the dynamic $\\mathcal{G}_s$ remains effective both as a validation set for evaluating local updates and a fine-tuning dataset for enhancing the global model. Moreover, this strategy keeps the retrieval and storage overhead constant, as the size of $\\mathcal{G}_s$ remains stable across communication rounds.\n\nIn contrast, applying such an update mechanism to the client-side local dataset $\\mathcal{D}_j$ is not suitable, since $\\mathcal{D}_j$ serves as the training dataset where every sample matters. After local compensation in Flick, we expect the local silos to become domain-rich and progressively closer to an IID distribution, making stale sample removal on the client side unnecessary.\n\n`[4/5] In line 126, the paper suggests that client datasets progressively approximate an IID distribution over time. However, this assumption may not hold if certain classes inherently present greater learning difficulty. Could you expand on how the proposed method addresses or accounts for such class-specific learning challenges in the non-IID setting?`\n\nWe clarify that the client in Flick progressively approximate an IID distribution through local compensation, which holds even in the presence of “hard” classes. Specifically, Flick tends to compensate for such hard classes across all participating clients via the decision matrix $\\mathbf{D}$ (lines 202-207), since such learning challenges are not client-specific but lead to generally underperforming local models on validation samples of these classes. In such cases, the local data still approximates an IID distribution. We want to highlight the distinction between an IID and a balanced distribution: Flick compensates the local data across clients towards an identical distribution (i.e., $\\mathcal{P}_ {j_ 1}(y)\\simeq$ $\\mathcal{P}_ {j_ 2}(y)$ with $ \\forall j_ 1, j_ 2 \\in \\mathcal{J}, j_ 1 \\neq j_ 2$), while the resulting label distribution ($\\mathcal{P}_ {j_ 1}(y)$ and $\\mathcal{P}_ {j_ 2}(y)$) may still be imbalanced.\n\n`[5/5] In line 166, the sample selection strategy favors choosing samples with the smallest loss for underperforming classes, under the assumption that such samples are less likely to be noisy. This makes sense in early training stages. However, in later stages, prioritizing harder (i.e., higher-loss) samples might better facilitate learning. Could you clarify why your method consistently favors small-loss samples?`\n\nWe clarify that our strategy does not consistently favor small-loss samples. Specifically, for well-performing classes, we intentionally select the sample with the largest training loss for image captioning, as such samples provide larger gradients and thus contribute more to model updates. Moreover, the choice between small-loss and large-loss samples varies across communication rounds, even for the same class within the same client, depending on the dynamic relationship between the class-wise loss and the averaged training loss (lines 161-166)."}, {"review_id": "j8goFvi4YS", "reviewer": "Reviewer_t7ek", "summary": "The paper introduces a novel framework aimed at addressing the challenges of data heterogeneity in Federated Learning (FL). Federated Learning allows for collaborative model training across decentralized data sources while preserving privacy. However, the heterogeneous nature of data distributions across different clients often leads to inconsistent learning objectives and suboptimal performance of the global model. Flick proposes a data-driven solution by incorporating commonsense knowledge from Large Language Models (LLMs) to generate synthetic data, which compensates for local data variations and enhances global model performance. The framework involves a local data summarization phase, where client-specific knowledge is captured in textual form, and a server-side data generation phase, where LLMs guide the generation of synthetic data used for fine-tuning the global model and aligning client data distributions. The paper presents extensive experimental results demonstrating improvements in model accuracy and convergence speed.", "questions": "Implementation Details:\nCould you provide more details on the prompt generation process with LLMs? Specifically, how do you ensure that the prompts effectively capture the necessary client-specific knowledge without introducing significant privacy concerns? Additional technical details or examples would enhance understanding.\nReal-World Applicability:\nThe reliance on synthetic data generation is a notable aspect of the framework. How do you envision Flick performing in real-world scenarios where data distributions may be more complex and nuanced? It would be useful to discuss potential strategies for adapting the framework to such environments.\nPrivacy Considerations:\nWhile the paper outlines a low-sensitivity approach to data summarization, could you elaborate on any privacy assessments conducted? Understanding how the framework complies with privacy regulations and mitigates risks would be beneficial for its practical adoption.\nComputational Resource Requirements:\nFlick leverages server-side resources for data generation. How does the framework perform in environments with limited computational capabilities? Discussing any optimizations or adaptations for resource-constrained settings would be valuable.\nThe potential impact of synthetic data on model bias and fairness is not fully explored. Could you provide insights into how Flick addresses these issues, or suggest directions for future work to ensure equitable model performance across diverse client data?", "limitations": "The authors have addressed some limitations of their work, particularly regarding data heterogeneity and privacy considerations. However, further discussion on computational constraints and the implications of synthetic data on bias and fairness would strengthen the paper's impact and applicability.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality:\nStrengths:\nThe methodology is robust and well-designed, integrating modern LLM capabilities with FL to address data heterogeneity.\nExtensive experimental validation is conducted across multiple datasets, showcasing significant improvements in both accuracy and convergence speed.\nThe paper provides a comprehensive analysis of the framework's performance, including comparisons with existing methods and an ablation study.\n\nWeaknesses:\nThe used benchmark dataset for evaluation may not fully capture the complexities of real-world FL deployments, where data distributions can vary unpredictably.\nThe implementation details of the LLM-guided prompt generation process could be expanded for greater clarity.\n\nClarity:\nStrengths:\nThe paper is generally well-structured, with clear explanations of the Flick framework and its components.\nFigures and tables effectively illustrate the problem, solution, and experimental results, aiding reader comprehension.\n\nWeaknesses:\nCertain technical aspects, such as privacy considerations and the specifics of text prompt generation, could be more thoroughly explained.\nSome sections may benefit from additional background information to assist readers less familiar with the domain.\n\nSignificance:\nStrengths:\nThe paper addresses a critical challenge in FL, offering a novel approach that could significantly enhance the applicability of FL in heterogeneous data environments.\nThe integration of LLMs for commonsense knowledge distillation introduces a new dimension to synthetic data generation, with potential applications beyond FL.\n\nWeaknesses:\nThe framework's reliance on server-side resources may limit its applicability in scenarios with constrained computational infrastructure.\nThe potential impact of the generated synthetic data on model fairness and bias is not fully explored.\n\nOriginality:\nStrengths:\nThe use of LLMs to guide synthetic data generation in FL is a novel approach that distinguishes Flick from existing methods.\nThe framework's design, which balances client-side and server-side responsibilities, offers a unique solution to data heterogeneity.\n\nWeaknesses:\nWhile innovative, the framework's dependence on LLMs could be seen as an extension of existing generative approaches, rather than a fundamentally new paradigm.", "was_revised": true, "final_justification": "I have checked out author’s responses and discussions, and decide to maintain my positive score.", "rebuttal": "We thank the reviewer for the insightful comments. Below are our responses. \n\n`[1/6] Implementation Details: Could you provide more details on the prompt generation process with LLMs? Specifically, how do you ensure that the prompts effectively capture the necessary client-specific knowledge without introducing significant privacy concerns?`\n\nIn the data generation phase, Flick instructs the LLM to generate a set of class-specific text prompts using a predefined prompt template. These text prompts are then passed to a text-to-image model (e.g., LDM) to synthesize data. The prompt template, shown in Figure 3, takes as input the local summaries $\\mathcal{T} ^{n}$, class $i$, and budget $G$ (number of text prompts). Appendix F also provides detailed examples of LLM responses for generating $G=5$ text prompts in the classes dog (Figure 15) and headphones (Figure 16), given the corresponding local captions.\n\nThe privacy concerns mainly arise from the local summaries $\\mathcal{T} ^{n}$, while these summaries are deliberately designed to contain low-sensitivity textual information. Extensive evaluation in Appendix E supports our privacy claims, and further discussion is provided in the response [3/6]. In a nutshell, $\\mathcal{T} ^{n}$ is resistant to image reconstruction and satisfies an $\\epsilon$-DP guarantee, making local summary a privacy-preserving design.\n\n`[2/6] Real-World Applicability: How do you envision Flick performing in real-world scenarios where data distributions may be more complex and nuanced?`\n\nWe clarify that our framework design, particularly the calculation of decision matrix $\\mathbf{D}$ (lines 200-207 in the paper), naturally enables its resilience to complex and heterogeneous real-world data distributions. Extensive experimental results support such a conclusion, including settings with both extreme label skew and domain shift, which are commonly used to simulate realistic non-IID scenarios.\n\nSpecifically, data compensation for each local silo follows the matrix $\\mathbf{D}$, where entries are obtained based on the performance of local models on a server-held, task-related validation dataset. Such a dataset is built from scratch and dynamically updated across communication rounds, making the data-driven compensation process fully automated and agnostic to the actual local distribution.\n\nTo validate the performance of Flick under heterogeneous data distribution, we report experimental results under the Dirichlet distribution with the concentration parameter $\\alpha=0.1$ for PACS dataset and $\\alpha=0.05$ for Office-Caltech dataset, where a smaller $\\alpha$ indicates stronger heterogeneity. We also evaluate Flick on the large-scale DomainNet dataset with 100 clients with $\\alpha=0.1$ (Table 7 in the Appendix). Furthermore, we evaluate Flick under a more extreme case where each client holds samples from only two random classes (please see our response [4/4] to the comment of Reviewer Bcqt for the detailed results). Overall, Flick consistently improves convergence speed and accuracy of the global model by a large margin compared to the baseline methods, especially under more heterogeneous settings. \n\n`[3/6] Privacy Considerations: While the paper outlines a low-sensitivity approach to data summarization, could you elaborate on any privacy assessments conducted?`\n\nThe local summary phase in Flick provides the server with essential client-specific information with minimal privacy risk by 1) selectively captioning only one representative sample per class and 2) using textual summaries that carry informative yet low-sensitivity content. We evaluate the privacy risk of such a data summarization approach from two perspectives: providing a formal $\\epsilon$-DP guarantee and investigating the risk of raw data reconstruction from local summaries. As shown in Table 9 (Appendix E), we add Laplace noise (parameterized by $\\epsilon$) to the textual summaries and observe that Flick maintains performance gain even under increasing noise levels. Moreover, Table 10 (Appendix E) shows image reconstruction results using a wide range of image generators. We observe that reconstructed images remain significantly dissimilar to the original raw data, as measured by three widely used image similarity metrics. It indicates that recovering the original data from local captions is highly challenging.\n\n`[4/6] Computational Resource Requirements: How does the framework perform in environments with limited computational capabilities?`\n\nWe highlight two features of our Flick that support deployment in real-world scenarios: zero training burden and flexible inference overhead. Specifically, the four models within the framework, including the captioning VLP model, LLM, SBERT embedding model, and LDM model, are all out-of-the-box models without any fine-tuning or prompt-tuning when applied to the Flick framework. As a result, they introduce zero training burden to Flick. In terms of inference overhead, only the VLP model for image captioning runs on the edge side, while the other three models run on the server, which typically has rich computing resources. Moreover, Flick offers strong flexibility in resource requirements through: 1) tunable hyperparameters such as generation budget $G$ and server-held data pool size $\\lvert \\mathcal{G}_ s \\lvert$, and 2) consistent performance gains across different choices of image captioning models, LLMs, and image generators, as shown in Figures 10, 11, and 12 in Appendix C, respectively. This flexibility enables both clients and the server to choose models that are best suited to their available computational budgets.\n\n`[5/6] The potential impact of synthetic data on model bias and fairness is not fully explored. Could you provide insights into how Flick addresses these issues, or suggest directions for future work to ensure equitable model performance across diverse client data?`\n\nThe bias and fairness of synthetic data primarily depend on the text prompts fed into the text-to-image model. In Flick, the server uses a predefined prompt template to instruct LLM in generating these text prompts, as discussed in our response [1/6]. Within the template, the local summaries $\\mathcal{T}^{n}$ capture the cross-client-specific knowledge. We emphasize that all participating clients have an equal opportunity to contribute to $\\mathcal{T}^{n}$ in Flick, as each client shares the same number of local image captions with the server. Moreover, designing a fairness-aware prompt template could further mitigate the potential bias during text prompt generation and the subsequent data synthesis. We leave this as a promising direction for future work.\n\n`[6/6] While innovative, the framework's dependence on LLMs could be seen as an extension of existing generative approaches, rather than a fundamentally new paradigm.`\n\nWhile the availability of versatile generative models has made data synthesis more accessible, effective data generation in the context of FL remains non-trivial, primarily due to privacy and resource constraints. We highlight that Flick is a novel data-driven framework tailored to the challenges of heterogeneous FL. We solve the problem of designing a generative pipeline leveraging limited, low-sensitivity cross-client knowledge and task-relevant commonsense knowledge of LLMs.\n\nRather than merely stringing together existing generative tools, Flick is carefully designed to address three core challenges: 1) distillation of client-specific knowledge in a low-sensitivity way; 2) prompt design that effectively guides the instillation of task-relevant knowledge from LLMs into the data generation process; 3) usage of the synthetic samples, balancing both effectiveness (long-term improvements in model performance) and efficiency (controllable and flexible overhead). The effectiveness of each key design module is thoroughly validated through the ablation study presented in Table 3."}, {"review_id": "5xWNsqc6Po", "reviewer": "Reviewer_28YM", "summary": "This paper proposes **Flick**, a novel data generation framework for heterogeneous Federated Learning with Commonsense Knowledge from Large Language Models.  \nWithin Flick, the client performs the local data summary to capture client-specific knowledge in textual form. \nThe server then generates new data samples by analyzing the collected captions using an LLM guided by a designed prompt. Generated samples are stored in a server-maintained data pool, which supports both global model fine-tuning and local data compensation.  \nFlick achieves superior performance, as demonstrated by experiments on two datasets, PACS and Office-Caltech.", "questions": "**Questions**:\n1. How many data samples are used in Table 1? This information is difficult to find, even after lookig through both the main text and the appendix. Specifically, please clarify the number of: \n\n    Original data samples, prompts generated (is G = 10?), generated images, and sampled images for training.\n***\n**Suggestions**:\n1. Make clear which LLM/Diffusion model you use in \"Datasets and Models\".\n***", "limitations": "I acknowledge that the authors have made a genuine effort to discuss the limitations of this paper, which primarily relate to the additional cost overhead and potential privacy risks. \nHowever, I remain concerned about the severity of these limitations and their impact on the practical applicability of the proposed method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n1. Extensive experiments with 9 baselines on two datasets.\n2. Clear structure and visualization.\n3. The loss-based sample selection method, aimed at minimizing privacy risks, is both critical and effective.\n4. Additional experiments in the Appendix to validate the choice of hyper parameters.\n\n***\n**Weaknesses**:\n1. I greatly appreciate the authors' efforts in presenting Flick. However, the method appears to rely on a major, and arguably unreliable, assumption: *leveraging powerful LLMs and diffusion models to assist in training ResNet models for relatively simple image classification tasks* and *via federated learning*. This approach seems misaligned with current community practices and **lacks clear practical applications**. To draw an analogy, it feels like hiring an astronaut to ride a bicycle.\n2. Building on the first point, although Appendix D provides some analysis, the proposed approach introduces substantial additional computational overhead—not all of which is adequately addressed in Appendix D. Some of these additional costs are non-negligble: \n    * the deployment cost of diffusion models which is significantly higher compared to that of standard ResNet models; \n    * the API fees associated with using GPT models; \n    * the inference time required for both LLMs and diffusion models to generate synthetic data; \n    * etc.\n\n    Have you measured the actual time required to generate synthetic data in Figure 14? Based on my estimates, the generation time probably far exceeds 500 seconds.\n\n    If the authors can provide a detailed breakdown of the time and resource consumption involved in the full Flick process—especially in comparison with traditional federated learning methods such as FedProx—and convincingly argue that the additional costs are justified by performance gains, I would be happy to raise my score.\n\n3. Lack of comparison with other straightforward approaches. \n    * If the server has already acquired summaries of the client data and can generate synthetic data accordingly, why not simply fine-tune a global model on all synthetic data directly at the server side? This would reduce communication to just a single round, which, according to Table 1, Table 2, and Figure 6, yields the best statistical performance. \n    * If powerful LLMs or diffusion models are assumed to be available, why not directly leverage them for the task? It is necesary to either compare the proposed approach with models such as GPT-4o or Qwen2-VL on PACS, or justify why these models are not suitable in the given context.\n4. Lack of experimental comparisons under standard IID settings, e.g. using datasets like CIFAR.\n***", "was_revised": true, "final_justification": "At least half of my concerns regarding the performance-efficiency trade-off have been addressed.\nWhile I still hold some reservations about the practical value of Flick, I believe—following the principle of \"focusing on merits\"—that its inherent limitations should not be too much criticized by me.\nTherefore, I am now leaning toward acceptance in respond to the authors' hard efforts.", "rebuttal": "We thank the reviewer for the insightful comments. Below are our responses.\n\n`[1/5] Lack practical applications, and why not leverage powerful LLMs for the task?`\n\nDespite the increasing availability of versatile foundation models, we emphasize that real-world deployment, especially inference at the edge, still requires task-specific lightweight models due to privacy constraints and limited resources. This also necessitates privacy-preserving training on data that remains distributed across users, thereby leading to the emergence of FL. On one hand, even if a strong pretrained model (e.g., ResNet-18 trained on ImageNet) is available, it still requires refinement using task-relevant, decentralized data residing on edge devices. On the other hand, while foundation models such as GPT-4o or Qwen2-VL may achieve superior performance on tasks like PACS classification, they are typically too large to run efficiently on edge devices. Relying on remote cloud-based inference via APIs introduces significant privacy risks, as it requires users to upload private or sensitive raw data to third-party services.\n\n`[2/5] Authors can provide a breakdown of time and resource consumption involved in the Flick process; API fees associated with using GPT.`\n\nThanks for the suggestion. Below, we provide a detailed breakdown of the additional computational overhead introduced by Flick, which varies across rounds.  For example, we report results from an experiment running on an NVIDIA A40 GPU, using the FedProx baseline on PACS dataset. The table below shows the wall-clock time (in seconds) for six components: extra local training, local summary,  LLM inference, sample retrieval, data generation, and global model fine-tuning. We observe that the server-side computational overhead (i.e., the last four components) takes the majority of the overall cost. However, server-side overhead decreases over time, as Flick no longer heavily requires data generation after 20 rounds. We also clarify that the data generation time using the LDM is approximately **1** second per sample, which is lower than the reviewer's estimation. This latency can be further reduced with a more powerful server.\n\n|Round|Extra Local Training|Captioning|LLM|Retrieval|Generation|Global Finetuning|\n|-|-|-|-|-|-|-|\n|1|0.00░░░░░|3.18▓▓▓▓▓|15.90▓▓▓▓▓░|0.00░░░░░░|31.18▓▓▓▓▓▓░░░░░░|0.35▓░░░░|\n|3|0.84▓▓░░░|3.05▓▓▓▓▓|18.55▓▓▓▓▓▓|0.97▓░░░░░|51.88▓▓▓▓▓▓▓▓▓▓░░|0.42▓░░░░|\n|5|1.10▓▓░░░|2.78▓▓▓▓░|15.90▓▓▓▓▓░|1.78▓▓░░░░|18.67▓▓▓▓░░░░░░░░|0.24▓░░░░|\n|7|0.81▓▓░░░|3.19▓▓▓▓▓|13.25▓▓▓▓░░|3.24▓▓▓▓░░|20.22▓▓▓▓░░░░░░░░|0.67▓▓░░░|\n|9|0.57▓░░░░|3.19▓▓▓▓▓|18.55▓▓▓▓▓▓|4.75▓▓▓▓▓░|20.97▓▓▓▓░░░░░░░░|0.71▓▓░░░|\n|11|0.70▓▓░░░|2.86▓▓▓▓░|15.90▓▓▓▓▓░|5.19▓▓▓▓▓▓|59.81▓▓▓▓▓▓▓▓▓▓▓▓|0.91▓▓▓░░|\n|13|0.48▓░░░░|3.15▓▓▓▓▓|18.55▓▓▓▓▓▓|4.27▓▓▓▓▓░|26.71▓▓▓▓▓░░░░░░░|1.09▓▓▓▓░|\n|15|0.52▓░░░░|2.96▓▓▓▓▓|7.95▓▓▓░░░|1.35▓▓░░░░|18.96▓▓▓▓░░░░░░░░|1.28▓▓▓▓░|\n|17|0.58▓░░░░|2.81▓▓▓▓░|18.55▓▓▓▓▓▓|3.19▓▓▓▓░░|28.86▓▓▓▓▓▓░░░░░░|1.44▓▓▓▓▓|\n|19|0.47▓░░░░|0.00░░░░░|0.00░░░░░░|0.00░░░░░░|0.00░░░░░░░░░░░░|0.00░░░░░|\n|21|1.17▓▓▓░░|2.82▓▓▓▓░|5.30▓▓░░░░|5.10▓▓▓▓▓▓|14.52▓▓▓░░░░░░░░░|0.58▓▓░░░|\n|23|0.86▓▓░░░|0.00░░░░░|0.00░░░░░░|0.00░░░░░░|0.00░░░░░░░░░░░░|0.00░░░░░|\n|25|1.07▓▓░░░|2.97▓▓▓▓▓|2.65▓░░░░░|0.68▓░░░░░|1.79░░░░░░░░░░░░|1.43▓▓▓▓▓|\n|27|1.21▓▓▓░░|0.00░░░░░|0.00░░░░░░|0.00░░░░░░|0.00░░░░░░░░░░░░|0.00░░░░░|\n|29|0.68▓▓░░░|0.00░░░░░|0.00░░░░░░|0.00░░░░░░|0.00░░░░░░░░░░░░|0.00░░░░░|\n|31|1.16▓▓▓░░|3.07▓▓▓▓▓|10.60▓▓▓░░|2.10▓▓░░░░|23.30▓▓▓▓▓░░░░░░░|0.58▓▓░░░|\n|33|1.24▓▓▓░░|2.78▓▓▓▓░|2.65▓░░░░░|0.42░░░░░░|5.22▓░░░░░░░░░░░|0.56▓▓░░░|\n|35|1.45▓▓▓░░|0.00░░░░░|0.00░░░░░░|0.00░░░░░░|0.00░░░░░░░░░░░░|0.00░░░░░|\n|37|1.44▓▓▓░░|2.53▓▓▓▓░|2.65▓░░░░░|0.60▓░░░░░|10.31▓▓░░░░░░░░░░|0.55▓▓░░░|\n|39|0.76▓▓░░░|2.65▓▓▓▓░|2.65▓░░░░░|1.03▓░░░░░|4.14▓░░░░░░░░░░░|0.58▓▓░░░|\n|41|1.37▓▓▓░░|3.27▓▓▓▓▓|7.95▓▓▓░░░|0.39░░░░░░|8.50▓▓░░░░░░░░░░|0.56▓▓░░░|\n|43|2.04▓▓▓▓▓|2.84▓▓▓▓░|2.65▓░░░░░|0.34░░░░░░|1.38░░░░░░░░░░░░|0.57▓▓░░░|\n|45|1.08▓▓░░░|0.00░░░░░|0.00░░░░░░|0.00░░░░░░|0.00░░░░░░░░░░░░|0.00░░░░░|\n|47|1.97▓▓▓▓░|0.00░░░░░|0.00░░░░░░|0.00░░░░░░|0.00░░░░░░░░░░░░|0.00░░░░░|\n|49|2.21▓▓▓▓▓|0.00░░░░░|0.00░░░░░░|0.00░░░░░░|0.00░░░░░░░░░░░░|0.00░░░░░|\n|1-19|11.53|54.06|291.50|51.73|598.05|14.11|\n|1-150|217.51|190.47|651.90|145.49|1139.48|31.47|\n\nFor a fair comparison, we report the wall-clock time required for both Flick and FedProx to reach the same target accuracy (i.e., best accuracy by FedProx): **0.48** hours for Flick (at round 19) and **1.24** hours for FedProx (at round 142). The accumulated additional computational overhead by Flick in the first 19 rounds is *0.28* hours, as shown at the bottom of the table. Moreover, after 150 rounds, Flick further improves the global model accuracy by 10.09% over FedProx, with only an additional *0.66* hours cost. These results highlight that the overhead introduced by Flick is well justified by its superior efficiency and accuracy improvements. \n\nFor real-world deployment, we also provide the wall-clock time to the target accuracy when local training and summary running on an NVIDIA Jetson AGX Orin: it takes **0.96** hours for Flick while **2.37** hours for FedProx. The accumulated additional computational overhead is *0.62* hours, with the breakdown across the six components being *57.3s, 1220.0s, 291.5s, 51.7s, 598.1s, and 14.1s*, respectively. We emphasize the strong flexibility of Flick for inference overhead by allowing clients and the server to choose models according to their available resource budgets. Please see our further response [4/6] to the comment of Reviewer t7ek for the details.\n\nIn terms of the API fees, we use the **gpt-4o-mini** model for local summary analysis. There are a total of 388 API requests across 150 rounds, at a cost of \\\\$0.05, with \\\\$0.03 for input tokens and \\\\$0.02 for output tokens.\n\n`[3/5] Why not fine-tune a global model on all synthetic data at the server? This would reduce communication to just a single round.`\n\nWe clarify that simply fine-tuning a global model on synthetic data, particularly in a one-shot way, is a suboptimal solution for two main reasons: 1) obtaining a rich task-related synthetic dataset, especially within a single round, is privacy-sensitive and resource-intensive; and 2) the performance gain from global fine-tuning alone, as shown in Tables 1 and 3, is limited. \n\nIn Flick, the fine-tuning dataset $\\mathcal{G}_s$ is built from scratch and dynamically updated over rounds. In contrast, one-shot fine-tuning requires collecting a rich and substantial dataset within a single round, which leads to privacy risk and significant latency. For example, the baseline method FGL bootstraps all clients to caption all local images and then uploads them to the server-side generator to construct a large-scale fine-tuning dataset in the first round. As shown in Figure 14(a) (Appendix D), this results in ~0.6 hours of warm-up latency. Although the global model achieves high accuracy after the first round, its performance over time is suboptimal. Moreover, collecting large-scale local summaries in a single round raises privacy risks and relies on full client participation, which is often unrealistic in real-world settings.\n\nMoreover, global fine-tuning alone is insufficient. Flick introduces dual usage of synthetic data: the local data compensation progressively provides clients with additional samples to approach the IID condition, yields long-term benefits, while fine-tuning the global model offers a more straightforward yet slight improvement. As shown in Table 3, ablation of local data compensation causes a larger accuracy drop than removing global fine-tuning. Table 1 further supports this: Flick, which combines both local compensation and global fine-tuning, consistently outperforms FGL that is only based on global fine-tuning.\n\n`[4/5] Lack of experiments under IID settings, e.g. using datasets like CIFAR.`\n\nWe clarify Flick is specifically designed for heterogeneous FL scenarios, which are far more realistic scenarios in life than ideal IID settings. In such heterogeneous settings, data silos are non-IID, exhibiting both label skew and domain shift. Such non-IID settings are practical and pose significant challenges to global model performance. In contrast, simple IID settings are idealized and offer limited space for improvement. Flick aims to bridge the gap between realistic non-IID distribution and the IID ideal cases through local data compensation. Besides, we emphasize that Flick explicitly alleviates the domain shift issue, which is not captured in datasets like CIFAR-10 or CIFAR-100. Therefore, evaluation on such datasets cannot reflect or validate the core capabilities and contributions of Flick.\n\n`[5/5] Questions and Suggestions.`\n\nWe have provided in Appendix B.2 the details of the data generation budget $G$, LLMs, and image generators used for the two datasets in Table 1. Specifically, we set $G$=5, use **stable-diffusion-v1-5** for image generation, and employ **gpt-4o-mini** for generating text prompts. We appreciate the suggestion and will move this information from Appendix to the main text in future revisions, to make the paper easier to follow. The table below summarizes the number of samples across three datasets -- PACS (7,984 images, 20 clients), Office-Caltech (2,003 images, 8 clients), and DomainNet (12,740 images, 100 clients) -- used in our experiments. For each baseline, we report the number of generated and training images under Flick over 150 rounds (before `/`) and up to reaching the target accuracy (after `/`).\n\n||PACS||Office-Cal.||DomainNet||\n|-|-|-|-|-|-|-|\n|Flick|#generated|#training|#generated|#training|#generated|#training|\n|w/FedAvg|1083`/`270|9344`/`8554|243`/`217|3008`/`2998|1127`/`292|16905`/`13080|\n|w/FedProx|1089`/`447|9384`/`8934|278`/`253|3233`/`3178|1133`/`407|16760`/`13305|\n|w/FedDyn|1166`/`413|9509`/`8554|252`/`192|3028`/`2983|1186`/`435|17110`/`13410|\n|w/FedNAR|1100`/`289|9330`/`8609|250`/`220|3318`/`3108|1299`/`492|16810`/`13845|"}, {"review_id": "OuPJcEIQvh", "reviewer": "Reviewer_FvW9", "summary": "To address the label skew and domain shift of data silos across clients in FL, this paper proposes Flick, a generative data augmentation framework for heterogeneous FL, which introduces three additional phases into the FL workflow, including local summary, data generation, and generated data usage. On the one hand, clients select representative samples from their local data and use an image-to-text model as a caption generator to transform samples into low-sensitivity tokens, extracting client-specific knowledge from local data, which is then offloaded to the server. On the other hand, the server uses an out-of-the-box LLM to analyze local summaries and instill inherent commonsense knowledge into generated text prompts, which direct a text-to-image model in producing synthetic data, enabling global model fine-tuning and local data compensation to gradually align the label and feature distributions across clients. Experiments on three multi-domain image classification datasets (PACS, Office-Caltech, and DomainNet) demonstrate the effectiveness of the proposed method.", "questions": "N/A", "limitations": "The authors partially addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Reasons to Accept:**\n\n1.\tThe method is innovatively designed, skillfully integrating image-to-text models, LLMs, sentence embedding models, and text-to-image models with synthetic data generation.\n2.\tThe synthetic data are used not only for fine-tuning the global model but also for local data compensation at the clients, creating a dual-benefit effect.\n3.\tThe experiment is thorough, and the implementation details are clear.\n\n**Reasons to Reject:**\n\n1.\tAlthough synthetic data is used to improve model performance, the paper lacks quantitative evaluations of image quality (e.g., FID, IS, CLIP score), making it difficult to establish a direct link between data quality and performance gains.\n2.\tThe applicability of Flick to non-vision tasks (e.g., text classification, speech, time series) or more complex structured tasks (e.g., object detection, segmentation) is not discussed.\n3.\tWhile Flick aims to alleviate data heterogeneity, its mechanism—relying on \"client feedback per round → server-generated data → client-side compensation\"—may introduce new forms of client imbalance (e.g., favoring clients with stronger descriptive capabilities), due to variations in the amount of feedback, description quality, and generated data quality. No fairness or consistency analysis is provided in this regard.\n4.\tToken length and communication cost statistics are not reported.\n5.\tCompared with traditional federated learning methods, the proposed method is very complex, introducing four additional large-scale models and multiple hyperparameters, including an image-to-text model (i.e., Salesforece/blip-image-captioning-large) for image captioning of local data, a large language model (i.e., gpt-4o-mini) for text prompt generation, a sentence embedding model (i.e., Sentence-BERT) for similarity calculation between generated text prompts and pooled text prompts, and a text-to-image model (i.e., sd-legacy/stable-diffusion-v1-5) to synthesize new data points from low-similarity text prompts. Therefore, it brings concerns on training burden and optimization difficulty.\n6.\tThe authors did not provide any code for review, so the reproducibility of this paper is open to question.", "was_revised": true, "final_justification": "After reading the authors' rebuttal,  most of my misunderstandings and concerns have been addressed, so I tend to keep the original score and maintain a positive score.", "rebuttal": "We thank the reviewer for the insightful comments. Below are our responses.\n\n`[1/6] The paper lacks quantitative evaluations of image quality (e.g., FID, IS, CLIP score), making it difficult to establish a direct link between data quality and performance gains.`\n\nThe data generation in Flick follows two steps: the server first obtains a set of text prompts from LLM, then feeds them into a text-to-image model (i.e., LDM) for synthetic images. We use an out-of-the-box image generation model (e.g., stable-diffusion-v1-5 used in the experiments) without fine-tuning. Therefore, metrics such as FID, IS, or CLIP score, which primarily evaluate the pretrained generator, are not the focus of our evaluation. Instead, as shown in Figure 12 in Appendix C, Flick consistently improves performance across different image generators, highlighting its resilience to the image generative backend.\n\nWe clarify that the performance gain mainly lies in the generation of informative, task-relevant text prompts, which is the core contribution of this work. To this end, we propose a prompt template that instructs LLM in extracting information from the collected cross-client-specific knowledge while instilling inherent commonsense knowledge to produce informative text prompts for each class. As shown in Table 2, integrating cross-client-specific knowledge leads to significant gains over local generation. Further, the ablation results in Table 3 confirm that the inclusion of commonsense knowledge improves text prompt effectiveness.\n\n`[2/6] The applicability of Flick to non-vision tasks (e.g., text classification, speech, time series) or more complex structured tasks (e.g., object detection, segmentation) is not discussed.`\n\nWe clarify that the Flick can be extended to other vision tasks with minimal modification, particularly in how to obtain the label $y_s$ of a generated data point $(x_ s, y_ s)$. In the image classification task investigated in this paper, the label $y_ s$ refers to the image class, which is known a priori to the server. When adapting Flick to more complex tasks such as object detection, $y_ s$ would include both the class label and the bounding box of the object. To this end, the server can leverage an automated annotator, such as SAM [1], to generate structured bounding boxes for synthetic images.\n\nWe further highlight that the core contribution of Flick lies in its data-driven pipeline, which integrates low-sensitive client-specific knowledge distillation, LLM-based commonsense knowledge instillation, and dual-use of generated samples to address data heterogeneity in federated learning. This pipeline is modality-agnostic and not limited to vision tasks. Nevertheless, we acknowledge two key challenges when applying Flick to non-vision modalities such as text, speech, or time series: 1) identifying suitable methods for converting raw data into meaningful yet low-sensitive text representations (akin to image captioning); and 2) finding the generative models with the capacity of producing realistic synthetic data from text prompts. We leave this as a promising direction for future work.\n\n[1] Alexander Kirillov, et al. \"Segment anything\". ICCV, 2023.\n\n`[3/6] While Flick aims to alleviate data heterogeneity, its mechanism—relying on \"client feedback per round → server-generated data → client-side compensation\"—may introduce new forms of client imbalance (e.g., favoring clients with stronger descriptive capabilities), due to variations in the amount of feedback, description quality, and generated data quality. No fairness or consistency analysis is provided in this regard.`\n\nWe highlight two essential fairness-enhancing features built into Flick. First, all clients have an equal opportunity to share their client-specific knowledge with the server. In each communication round, the server randomly selects a set of clients, and each participating client is allowed to report the same number of local summaries --  specifically, one representative local sample per class for image captioning. Second, the quality of generated data used for local compensation is consistent across clients, as the server centrally employs a text-to-image model (i.e., LDM) to synthesize data. This generation process is decoupled from individual client differences and thus not influenced by client diversity.\n\nWhile variations in description quality, mainly arising from different local captioning models across clients, may still introduce fairness concerns, we mitigate this by enforcing a fixed output length (default: 30 tokens) across all captioning models. This constraint helps limit variations in description quality. Moreover, designing a fairness-aware prompt template could further alleviate the negative impact of local summary disparities on LLM analysis, which we leave as a promising direction for future work.\n\n`[4/6] Token length and communication cost statistics are not reported.`\n\nFor the local summary phase, each client uses a VLP model to selectively caption one representative sample per class. We set the maximum token length of each image caption to 30 tokens, balancing the informativeness of the content with the risk of privacy leakage. \n\nWe break down the communication overhead in Flick into three parts: uploading local summaries, downloading synthetic samples for local data compensation, and exchanging model parameters, where the first two components represent additional overhead compared to FedAvg. Taking the PACS experiment in Table 1 as an example, each client reports local summaries with a minimal cost ranging from $1.01kb$ to $1.48kb$ per round, which is negligible compared to the $42.72Mb$ for exchanging model parameters. For a fair comparison, we report the overall communication overhead when Flick and FedAvg reach the target accuracy (defined as the best accuracy of FedAvg). Flick requires only 11 communication rounds with the server sending 510 synthetic images to clients. Thus, the total communication cost of Flick is: $42.72 (Mb/round) \\times 11 (round) \\times 2 + 0.14 (Mb/sample) \\times 510 (sample)=0.99Gb$ where the factor 2 refers to clients offloading the local updates and downloading the global model. For FedAvg, it takes 142 rounds to reach the same target accuracy, and the total communication cost is $42.72 (Mb/round) \\times 142 (round) \\times 2 =11.85Gb$. Moreover, after 150 rounds, Flick further improves the global model accuracy by 11.43% over FedAvg, by compensating 2,405 synthetic samples at an additional cost of only $0.33Gb$. It indicates that the performance gain by Flick justifies the additional costs.\n\n`[5/6] Compared with traditional federated learning methods, the proposed method is very complex, introducing four additional large-scale models and multiple hyperparameters. Therefore, it brings concerns on training burden and optimization difficulty.`\n\nWe clarify that the image captioning model (i.e., VLP model), LLM, SBERT embedding model, and image generation model (i.e., LDM) are all out-of-the-box models without any fine-tuning or prompt-tuning when applied in our Flick framework. As a result, they introduce zero training burden to Flick. Moreover, the inference overhead introduced by these models is controllable and flexible: 1) three of the four models (excluding the VLP model) are running on the server, which typically has rich computing resources; 2) the performance gain from Flick is consistent to the choice of image captioning model, LLM, and image generation model, as evidenced by the results in Figures 10, 11, and 12 in Appendix C, respectively. This flexibility allows both clients and the server to select models that are best suited to their available computational budgets.\n\nWe also investigate the sensitivity of Flick to the key hyperparameters: text similarity $T_ s$, validation accuracy $T_ v$, and data pool size $\\lvert \\mathcal{G}_ {s}\\lvert$ in Figure 9, and data generation budget $G$ in Table 5 of Appendix C. The extensive experimental results show that Flick generalizes well without requiring careful hyperparameter tuning.\n\n`[6/6] The authors did not provide any code for review, so the reproducibility of this paper is open to question.`\n\nWe appreciate the reviewer for pointing it out. Due to the policy prohibiting external links in the rebuttal, we are unable to share the code at this stage. We are committed to releasing the code once anonymity is no longer required."}]

Author rebuttal:

Meta-review: # Summary
The paper introduces Flick, a novel framework for federated learning addressing data heterogeneity through synthetic data generation guided by commonsense knowledge of Large Language Models (LLMs). Clients summarize local data to capture client-specific knowledge in textual form, which is used by the server to create informative prompts for generating synthetic data. This approach enables global model fine-tuning and local data compensation, enhancing model performance and convergence speed across diverse data sources in federated learning. Experiments show that Flick improves the global model accuracy by up to 11.43%, and accelerates convergence by up to 12.9x.

# Strengths
-  LLMs-based commonsense knowledge distillation introduces a novel dimension to synthetic data generation for both the global server and local clients in a federated learning setting.
- Clients share only text summaries, enhancing privacy protection by preventing the leakage of raw data.
- Extensive experiments with nine baseline models on two datasets provide a robust evaluation.
- The paper is well-written and easy to follow, with a clear structure and explanations of the Flick framework and its components.

# Weaknesses
- Local clients' textual summarization of local data might remove important details and information.  
- The motivation behind certain design choices and methodological components is not clearly explained.
- More analysis of the data selection method in terms of data diversity and quality is needed. It is also important to discuss and compare different data selection criteria and their effects on different training stages.   
- It is not clear if the commonsense knowledge of LLMs can cover sufficient domain-specific expert knowledge required in many heterogeneous settings for different clients. 
- Several important design choices and technical details are not discussed and justified thoroughly. 
- The focused image classification tasks might be too simple given the complexity and generalizability of LLMs and diffusion models. 
- Computational overhead caused by the proposed method is not analyzed.
- Lack of comparison with some important baselines.

# Reasons to Accept
- The paper studies a novel application of LLMs to address the data synthesis in a heterogeneous federated learning setting. 
- The proposed method leverages LLMs' textual summarization and prompting capabilities to reduce the privacy leakage risk.
- Comparisons with baselines and ablation studies are comprehensive for the focused image classification tasks. 

# Discussion Summary
- In the rebuttal, the authors provided various new experimental results to answer the questions from the reviewers and further clarified several key points and details of the proposed method. 
- In the responses by reviewers, though some concerns/questions remain, most reviewers confirmed that important concerns have been well addressed by the rebuttal, making them remain positive about the paper.
Decision: Accept (poster)