Federated Learning on Virtual Heterogeneous Data with Local-Global Dataset Distillation
Authors: Chun-Yin Huang
Ruinan Jin
Can Zhao
Daguang Xu
Xiaoxiao Li
Venue: Accepted by TMLR ()

Abstract: While Federated Learning (FL) is gaining popularity for training machine learning models in a decentralized fashion, numerous challenges persist, such as asynchronization, computational expenses, data heterogeneity, and gradient and membership privacy attacks. Lately, dataset distillation has emerged as a promising solution for addressing the aforementioned challenges by generating a compact synthetic dataset that preserves a model's training efficacy. However, we discover that using distilled local datasets can amplify the heterogeneity issue in FL. To address this, we propose Federated Learning on Virtual Heterogeneous Data with Local-Global Dataset Distillation (FedLGD), where we seamlessly integrate dataset distillation algorithms into FL pipeline and train FL using a smaller synthetic dataset (referred as virtual data). Specifically, to harmonize the domain shifts, we propose iterative distribution matching to inpaint global information to *local virtual data* and use federated gradient matching to distill *global virtual data* that serve as anchor points to rectify heterogeneous local training, without compromising data privacy. We experiment on both benchmark and real-world datasets that contain heterogeneous data from different sources, and further scale up to an FL scenario that contains a large number of clients with heterogeneous and class-imbalanced data. Our method outperforms *state-of-the-art* heterogeneous FL algorithms under various settings.

Published in Transactions on Machine Learning Research (01/2025)
Federated Learning on Virtual Heterogeneous Data with Local-Global Dataset Distillation
Chun- Yin Huang University of British Columbia Vector Institute
chunginh@ece.ube.cm ubc.c
Ruinan Jin
muinangjnn@aaammiii.ddo ubc..
University of British Columbia Vector Institute
Can Zhao NVIDIA
con:@nvidia.com
Daguang Xu NVIDIA
dagnangzrrvvidaa.com
Xiaoxiao Li University of British Columbia Vector Institute
riaoriao. Nicoriao.li@ece. ubc.
Reviewed on OpenReview:
Abstract
While Federated Learning (FL) is gaining popularity for training machine learning models in a decentralized fashion, numerous challenges persist, such as asynchronization, computa- lional expenses, data helerogeneity, and gradient and membership privacy altacks. Lately, dataset distillation has emerged as a promising solution for addressing the aforementioned challenges by generating compact synthetic dataset that preserves a model's training effi- cacy. However, we discover that using distilled local datasets can amplify the heterogeneity issue in FL.. To address this, propose Federated Learning on Virtual Heterogeneous Data. with Local-Global Datasct Distillation (FEDLGD), where scamlossly intograte dataset distillation algorithms into FL pipeline and train FL using smaller synthelic dataset (re- ferred as virtual data). Specifically, to harmonize the domain shifts, we propose iterative dis- tribution matching to inpaint global information to local virtual data and use federated gradi- ent matching to distill global virtual dala that serve as anchor points to rectify helerogeneous local training, without compromising data privacy. We experiment on both benchmark and real-world datasets that contain heterogeneous data from different sources, and further scale up FL scenario that contains large number of clients with heterogencous and class- imbalanced data. Our method outperforms stale-of-the-arl heterogeneous FI. algorithms under various settings. Our code is available at httss///gthubb.om. htw.b//oooobb-eee/eeee....
1 Introduction
Having a compatible training dataset is an essential de facto precondition in modern machine learning. However, areas such as medical applications, collecting such a massive amount data not realistic since it may compromisc privacy regulations such as GDPR (Voigt & Von dem Bussche 2017). Thus, rescarchers seek circumvent the privacy leakage by utilizing foderated learning pipelines or training with synthetic data.

---

Published in Transactions on Machine Learning Research (01/2025)
Federated learning (FL) (MMMahan et al. 2017 has emerged as pivotal paradigm for conducting machine learning on data. from multiple sources in a distributed manner. Traditional FT, involves a large number of clients collaborating to train a global model. By keeping data local and sharing only the local model updates, FL prevents the direct exposure of local datasets in collaborative training. However, despite these advantages, several research challenges remain in FL., including computational costs, asynchronization, data heterogencity, and vulnerabilities to deep privacy attacks (Wen et al., 2023
Another approach to GDPR compliance that has gained increased interest is using synthetic data in ma- chine learning model training to supplement replace real data when the latter is not suitable for direct use [Nikolenko] 2021). Among data synthesis methods, dataset distillation (Wang et al.) 2018 Cazenavette
et al. 2022 Zhao et al. 2021 Zhao & Bilen 2021 2023) has emerged as an ideal data synthesis strategy, as it is explored to enhance the efficiency and privacy of machine learning. Dataset distillation creates a
compact synthetic dataset while retaining similar model performance of that trained on the original dataset,
allowing efficiently training a machine learning model (Zhao al.) 2021 Zhao & Bilen
2023). The distilled
data usually remains low fidelity to the raw data but yet contains highly condensed essential information that makes the appearance of the distilled data dissimilar to the real data (Dong et al. 2022
In this work, we introduce an effective training strategy that leverages both FL and virtual data generated via dataset distillation, referred as federaled virtual learning, as the models are trained from virtual data (also referred as synthetic data) (Xiong et al. 2023 Goetz & Tewari 2020 Hu et al. 2022 y. In particular, we aim to find the best way to incorporate dataset distillation into FL under ordinary FL pipeline, where the only change is replacing real data with virtual data for local training. A simple approach is to generate synthetic data first and then use it for FL training; however, this could lead to suboptimal performance in data heterogeneous settings. We observe increased divergence in loss curves in early FL rounds if we simply replace real data with distilled virtual data synthesized by Distribution Matching (DM) (Zao & Bilen 2023). Thus, We perform a simple experiment: Measure the distances between a set of digits datasets (please refer to DIGITS in Soc. 5.2 for details) before and after distillation with DM. Statistically, we find that the MMD scores increase after distillation, resulting in an averaged 37% increment. Empirically, we visualize the tSNE plots of two different datasets (USPS Hull 1994) and SynthDigits (Ganin & Lempitsky 20150) in Fig. and distributions become diverse after distillation. This reveals that local virtual data from dataset distillation may worsen the data heterogencity issue in FL. Note that the data heterogencity referred here (also throughout the paper) is domain shift, which assumes variations in P(Xly) across clients, where X represents the imput data and y the corresponding labels. The concept differs from label shift (P(y)), which considers the heterogeneity on the labels.
To alleviate the problem of data heterogeneity, we enforce consistency in local embedded features using consensual anchors that capture global distribution. Existing works usually rely on the anchors yield from pre-generated noise (Tang et al. 2022 that cannot reflect training data distribution; or shared additional features from the client side (Zhou et al.)
D.D
3.0
2.0
3.0
clenro
Ye et al., 2023b), exposing more data leakage. To overcome the limitations, we propose an effective solution to address the heterogeneily issues using global virtual anchor for regularization, supported by our theoretical analysis. Without compromising privacy in implementation, our global anchors are distilled from pre-existing shared gradient infor- mation in FL to facilitate sharing global distribution information.
2023
clenzt
(e) Original datsects
(b) Distilled vinual datroets
Figure 1: Distilled local datasets using Distribution Matching (DM) (Zhao & Bilen 2023) can worsen hetero- geneity in FL. tSNE plots of (a) original datasets and (b) distilled virtual datasets of USPS (client 0) and SynthDig- its (client 1). The two distributions are marked in the dashed curves. We observe fewer overlapped o and x in (b) compared with (a), indicating higher heterogeneity be- tween two clients after distillation. Statistically, we find that the Maximum Mean Discrepancy (MMD) [Gretton| al. 2012 scores increase after distillation, resulting in an averaged 37% increment.
Apart from facilitating heterogencous FL, such fed- erated virtual learning further reduces computa- tional cost and offers better empirical privacy pro- tection. Specifically, we empirically demonstrate the
2

---

Published in Transactions on Machine Learning Research (01/2025)
reconstructed image from Gradient Inversion Attack (Geiping al. 2020 Huang et al. 2021 trained
distilled data obtain much lower quality. We also present that our virtual training can defend better against Membership Inference Attacks (Shokri et al. 2017 Please refer to Sec. - for more details.
To this end, we propose FEDLGD, a federated virtual learning method with local and global distillation. Data distillation gaining attention in centralized machine learning. Recognizing the need for efficiency in FL, we propose integrating two efficient dataset distillation methods into our FL pipeline. Specifically, propose iterative distribution matching in local distillation by comparing the feature distribution of real and synthetic data using an evolving global feature extractor. The local distillation results in compact local virtual datasets with balanced class distributions, achieving efficiency and synchronization while avoiding class imbalance. In addition, unlike previously proposed federated virtual learning methods that rely solely on local distillation (Goetz & Tewari 2020 Xiong et al.) 2023 Hu et al., 2022), we also propose a novel and efficient method, federated gradient matching, that seamlessly integrate dataset distillation into FI pipeline synthesize global virtual data as anchors on the server side using the uploaded averaged gradients. The global virtual data then serves as anchors lo alleviate domain shifts among clients.
We conclude our contributions follows:
This paper focuses on an important but under-explored FI. setting in which local models are trained on small virtual datasets, which we refer to as federated virtual learning, and we are the first to reveal that using distilled local virtual data using Distribution Matching (Zhao & Bilen 2023 may exacerbate the beterogeneity problem in the federated virtual learning setting.
We propose to address the heterogeneity problem by our novel distillation strategies, ileralive dis- tribution matching and federated gradient matching, that utilizes pre-existing shared information in FL, and theoretically show it can effectively lower the statistic margin.
Through comprehensive experiments on benchmark and real-world datasets, we empirically show that FEDLGD outperforms existing statc-of-the-art FL algorithms.
2
Related Work
2.1
Dataset Distillation
Data distillation (Wang al. 2018) aims to improve data efficiency by distilling the most essential fea- ture from large-scale dataset (e.g., datasets comprising billions of data points) into a certain terse and high-fidelity dataset. For example, Gradient Matching (Zhao et al. 2021 is proposed to make the deep neural network produce similar gradients for both the terse synthetic images and the large-scale original dataset. Besides, (Cazenavette et al., 2022) proposes matching the model training trajectory between real and synthetic data to guide the update for distillation. Another popular way of conducting data distillation is through Distribution Matching (Zhao & Bilen 2023). This strategy instcad, attempts to match the distri- bution of the smaller synthelic dataset with the original large-scale dataset latent representation space. It significantly improves the distillation efficiency. There are following works that further improves the utility of the distilled data {Li ct al.
Zhang et al. 2024). Morcover, recent studies have justified that data distillation defend against popular privacy attacks such as Gradient Inversion Attacks and Membership Inference Attacks (Dong et al.
2022 Carlini et al. 2022b which is critical in federated learning. In practice, dataset
distillation is used in healthcare for medical data sharing for privacy protection (Li et al. 2022). We refer the readers to (Sachdeva & McAuley
2023 for further data distillation strategies.
2.2 Heterogeneous Federated Learning
FL performance downgrading on non-iid data is a critical challenge Ye et al. 2023a A variely of FL algo- rithms have been proposed ranging from global aggregation to local optimization to handle this heterogeneous issuc, as echoed by (Sun al., 2024), data heterogeneity plays a critical role for model generalization. Global aggregation improves the global model exchange process for better unitizing the updated client models create powerful server model. FedNova. (Wang al. 2020 notices imbalance among different. local
3

---

Published in Transactions on Machine Learning Research (01/2025)
models caused by different levels of training stage (e.g,, certain clients train more epochs than others) and tackles such imbalance by normalizing and scaling the local updates accordingly. FedRod (Chen & Chao 2021) seeks to bridge personalized and generic FL by training separate global and local projection layers. Similarly, FodGELA Fan et al. (2024) also aims to bridge personalizod and generic FL, employing simplex equiangular light. frame (ETF) to address ilasaanm data. Meanwhile, Fed Avd (Hsu et al. 2019 applics the momentum to server model aggregation to stabilize the optimization. Furthermore, there are
strategies to refine the server model client models using knowledge distillation such as FedDF (Lin et al.
20200. FedGen (Zhu et al., 20211 FedFTG (Zhang et al. 20220, FedICT (Wu et al. 2023). FedGKT (He
ct al. 20200, FedDKC (Wu et al. 2022. and FedHKT) (Chen et al. 2023). However, we consider knowledge
distillation and data distillation two orthogonal directions to solve data heterogeneity issues. Local training
oplimization aims to explore the local objective to lackle the non-iid issue in FL system. FedProx (Li el al.
2020) straightly adds Le norm to regularize the client model and previous server model. Scaffold (Karim- ireddy ct al. 2020) adds the variance reduction term to mitigate the "clients-drift.. MOON (Li al. 2021
brings mode-level contrastive learning lo maximize the similarity between model representations to stable the local training. There is also another line of works (Ye et al. 2023b Tang et al. 2022 proposed to use global anchor to regularize local training, Global anchor be either set of virtual global data global virtual representations in feature space. However, in (Tang et al. 2022 , the empirical global anchor selection may not be suitable for data from arbitrary distribution as they don't update the anchor according to the training datasets. More recently, (Chen et al. 2024 propose to utilize communication compression to lacilitate heterogeneous FL training. Other methods, such as those rely on feature sharing from clients Zhouu 2023 Ye et al. 2023b), are less practical, as they pose greater data privacy risks compared classical et al.
FL settings.
2.3 Datasets Distillation for FL
Datasel distillation for FL an emerging topic that has attracled allention due to its benefit for elficient FL systems. It trains model on distilled synthetic datasets, thus we refer as federated virtual learning. It can help with FL synchronization and improve training efficiency by condensing every client's data into a small set. To the best of our knowledge, there are few published works distillation in FL. Concurrently
with our work, some studies (Goetz & Tewari 2020 Xiong et. al., 2023 Hu et al. 2022. Huang et al. 2024) distill datasets locally and share the virtual datasets with other clients/ servers. Although privacy is protected against currently existing attack modcls, we consider directly sharing local virtual data not a reliable strategy. It is worth noting that some recent works propose to share locally generated surrogates, such as prototypes (Tan et al. 2022). performance-sensitivee features (Yang et al. 2024), ; or logits (Huang et al., 2024) instead of the global model parameters. However, this work focuses on combining dataset distillation with pre-exxsting shared information in the classical FL setting to alleviate the data heterogeneity problem.
3 Method
3.1 Setup for Federated Virtual Learning
We start with describing the classical FL setting. Suppose there are N parties (clients) who own local datasets (Da,.... "D.,,..,,eyy Dow), and the goal of classical FL system, such as FedAvg (McMahan et al. 2017]. is to train a global model with parameters 0 on the distributed datasets (D = Uke[N] D The objective function is written as: L(0) EN IDd1 clients in FL may have variant amounts of training samples, leading to asynchronized updates. In this work, we focus on new type of FL. training method - federated virtual learning, that trains on virtual datascts for efficiency and synchronization (discussed in Sec 2.3 Federated virtual learning synthesizes local virtual data Do: for client for E [N] and form D
(D] (.))), where L,(0) is the empirical loss of client i. In practice different
UEIN D% Typically, [DS| < [ and [D&| = [D%). A basic setup for federated virtual learning is to replace D with D to train FL model on the virtual datasets.
4

---

Published in Transactions on Machine Learning Research (01/2025)
Selected Rounds Local-Global Data Distillation
Client 1
Server
D
Iterative Distribution Matching
Model Aggregation
fi-1
D
ft
Client
MMP
fl-
Federated Gradient Matching Lpist
Model Updating
D new
Do new
D
D old
L.CE
Unselected Rounds Local-Global Model Update
Client
Server
Client N
fC-1
Model Updating
f
f
Model Aggregation
B
LCF + AL.
Figure Overview of the proposed method FEDLGD. We split FL rounds into selected and unselected rounds. For the selected rounds, clients will refine the local virtual data and update local models, while the server uses aggregated gradients to update global virtual data and the global model. We term this procedure Local-Globar Data Distillation. For the unselected rounds, data while adding regularization loss local model updating. In the middle box, we also show the evolution of global and virtual data. Observe that although local virtual does not change visually, we found the local distillation steps are essential for improving model performance as shown in Fig. 30 and 3d
perform ordinary FL training with virtual
3.2 Overall Pipeline
The overall pipeline of our proposed method contains three phases, including 1) initialization, 2) local-global distillation, and 3) locol-global model update. We depict the essential design of FEDLGD in Fig. We begin with the inilialization of the clients' local virtual data De by performing distribution matching (DM) Zhao & Bilen 2023h. Meanwhile, the server will randomly initialize global virtual data De and network parameters 8% Then, we refine our local and global virtual data using our proposod local-global distilaation strategies. Among the selected iterations, we update 0, D9, and De in early training epochs, where the server and clients update their virtual data to match global information. For the unselected iterations, we train 0 using with additional regularization loss which penalizes the shift between local and global virtual data. The full algorithm is shown in Algorithm
3.3
FL with Local-Giobal Dataset Distillation
3.3.1 Local Data Distillation for Federated Virtual Learning
First of all, we hope to distill virtual data conditional on class labels achieve class-balanced virtual datasets. Second, we hope the virtual local data is best suited for the classification task. Last but not least, the process should be efficient due to the limited computational resource locally. To this end, we design Iterative Distribution Matching to fulfill our purpose.
Iterative distribution matching. The objective for this part to gradually improve local distillation quality during FL. Given efficiency critical for an FL system, we propose adapt one of the most efficient yet effective data distillation method that leverage distribution matching (DM) the representation space, DM (Zhao & Bilen 2023), in iterative updating form to be integrated with FL. To this end, we split
5

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm Federated Virtual Learning with Local-global Distillation
Require: f Model, w Feature extractor, 0: Model parameters, D: Virtual data, Original data,
Losses, G: Gradients.
2: Distillation Functions:
De + D fo)
Distribution Matching D Iterative Distribution Matching Federated Gradient Matching
4: Dę < MM(ra
5: D + FoderatedGM((D G
6:
7: Initialization:
8: Di + DM Frndd Frand)
Distilled local data for virtual FL training
10: FedLGD Pipeline:
11: for t t= L...,,, do
12:
Clients:
13:
for each selected Client do if 1 E then
14:
D Local-global distillation
Dr + IterativeDM(DE
15:
16:
G + (eCce(Di,
17:
clse
Dr + Di
18:
G. + Ve[Lce( Ge (C) + ALooN (D,, (()+()))
19:
20:
end if
21:
Uploads G. to Server end for Server:
22:
23:
24:
G + Aggregate(G Gi)
25:
L then
D Tocallglobal distillation
26:
DE + FederateddG GT)
27:
Send D to Clients
28:
end if
29:
f + MModeUUpaate(GF; f()
30: Send fft to Clients
31:
end for
model into two parts, feature extractor v and classification head h and the whole classification model is defined as fr = h o v. Given feature extractor & Rd - Rdd we want to generate Dr so that (D.) 2 (DD where Ps is the distribution in feature space. To distill local data during FL efficiently that best fits our task, we intend to use the up-to-date global feature extractor as our kernel function to distill virtual data with global information. Since we can't obtain ground truth distribution of local data, we utilize empirical maximum mean discrepancy (MMD) (Gretton et al. 2012 as our loss function for local virtual distillation:
DE
1 2 (E))).
LMMD
(za)
Σ
(1)
D
D
where 2 and Dest are the server feature extractor and local virtual data from the latest global iteration t. x; and of are the data sampled from Di and D respectively. K is the total number of classes, and we sum over MMD loss for each class k [K]. Thus, we obtain updated local virtual data for each FL round.
Although such an efficient distillation strategy inspired by DM, we highlight the key difference that DM uses randomly initialized model to extract features, whereas we use trained global feature extractor, as the ilerative updaling the clients' data using the up-to-date network parameters generate better task-

---

Published in Transactions on Machine Learning Research (01/2025)
specific local virtual data. Our intuition comes from the recent success of the empirical neural tangent kernel for data distribbttio learning and matching (Mohamadi & Sutherland 2022 Franceschi et al. 2022
Especially, the feature extractor of the model trained with FEDLGD could obtain feature information from other clients, which further harmonizes the domain shift among clients. We apply DM to the baseline FL methods and ret the effectiveness of our proposed iterative strategy Sec. E During distilling global information, FEDLGD only requires few hundreds steps for, which is computationally efficient.
Harmonizing local heterogeneity with global anchors. Data collected in different sites may have different distributions due to different collecting protocols and populations, which degrades the performance of FL. Even more concerning, we find that the issuc of data heterogencity among clients is exacerbated when training with distilled local virtual data in FL (see Fig, D To address this, we propose adding a regularization term in the feature space to the total loss function during local model updates, inspired by (Tang et al. 2022
Liotal = LCr D²:0)= ACcon(D)) (D;,
=
(2)
K
LCE =
1
Σ
ynlog(ūh) f(z;0)
(3)
D
e(ve(=s)) Tromp)
LCon
log
E (l
(4)
JEB
LCE is the cross-entropy measured on the virtual data D = (D=, Do} and K is the number of classes. Ccon is the supervised contrastive loss (Khosla et al.) 2020 for decreasing the feature distances between data from the same class while increasing the feature distances for those from different classes By represents a batch containing both De and De but without data j, " is subset of B only with samples belonging to class Vj, and Tremp is scalar temperature parameter. In such way, global virtual data can be served for calibration and groups the features of same classes together. At this point, a critical problem arises: What global virtual data shall we use?
3.3.2 Global Data Distillation for Heterogeneity Harmonization
We claim a 'good' global virtual data should be representative of the global data distributions. Therefore, we propose to leverage local clients' averaged gradients to distill global virtual data, and this process can be naturally incorporated into FL pipeline. We term this global data distillation method as Federated Gradient Matching.
Federated Gradient Matching. The concept of gradient-based dataset distillation is minimize the distance between gradients from model parameters trained by original data and virtual data. It is usually considered learning-to-leart problem because the procedure consists of model updates and virtual data updates. Zhao et al. (Zao et al., 2021) studies gradient matching in the centralized setting via bi-level optimization that iteratively optimizes the virtual data and model parameters. However, their implementation is not appropriate in our context because there are two fundamental differences in our set settings: 1) for model updating, the virtual dataset is on the server side and will not directly optimize the targeted task; 2) for virtual data update, the 'optimal' model comes from the optimized local model aggregation. We argue that these two steps naturally be embedded in local model updating and global virtual data distillation from the aggregated local gradients. First, we utilize the distance loss L (Zhao
et al. 2021 for gradient matching: Cpist Dist( (0), voLcE
(5)
D
where De and Do denote local and global virtual data, and VoLCE the average client gradient. Dist(S,T) is defined as
The
1 T Dist(S,T) =
(6)
||

---

Published in Transactions on Machine Learning Research (01/2025)
where L is the mumber of layers, St and Tj are flattened vectors of gradients corresponding each output node i from layer 1, and di is the layer output dimension. Then, our proposed federated gradient matching optimize as follows:
min (0) subject to 0 N Σ 8
where 0%0 arg ming (D))) is the optimal local model weights of client i. By doing federated gradient matching, we gradually distill global virtual data that captures local model information. It is worth noting that we do not need to perform this step for every FL communication round, instead, we find that only selecting a few rounds in the early stage of FL is sufficient to synthesize useful global virtual data, which shares similar insights as reported in (Feng al., 2023). We provide theoretical analysis to justify the effectiveness of our novel federated gradient matching in lowering the statistical margin in Appendix 1
4 Theoretical Analysis
In this section, we show theoretical insights on FEDLGD. Denote the distribution of global virtual data Po and the distribution of client local virlual data as Pe. In providing theoretical justification for the elficacy of FEDI.GD, we can adopt a similar analysis approach as demonstrated in Theorem 3.2 of VHT. (Tang et al. 2022), where the relationship between generalization performance and domain misalignment for classification tasks is studied by considering mazimizing the statistic margin (SM) (Koltchinskii & Panchenko 2002)
To assess the generalization performance of with respect to the distribution P(x, y), we define the SM of FEDLGD as follows:
E SMn
(7)
where is distance metric, and f FEDLGD((,, (z,y)) means that model f is optimized using FEDLGD with minimizing Eq. g Similar to Theorem A.2 of (Tang et al. 2022 we have the lower bound
Lemma 1 (Lower bound of FedLGD's statistic margin) Let f &op be neural network decompose of feature extractor o and a classifier The lower bound of FEDLGD's SM is
Ep- P, SM P)= 2 P Spm(p D) Pp [SMm (p, Pg) (a) SM D)] E,d 1)) v))
Proof 1 Following proof in Theorem A.2 of (Tang al. 2022 the statistical margin is decomposed as
Ep--, P, SM
2 Rp P, (M D)- BBp P, SM (p,Pg) - SMm(p. D)]1 F p, [SM (p, SMM (p. Ps)]|
PP 1) |pp-P [SM (p, Pg) (M D)]l Ryd (P(s v) Po(0 v))
Another component in our analysis building the connection between our used gradient matching strategy and the distribution match term in the bound.
Lemma 2 (Proposition of (Yu et al., 2023)) First-order distribution matching objective is approzi- mately equal to gradient matching of each class for kernel ridge regression models following random feature extractor.
Theorem 1 Due to the complexity of data distillation steps, without loss of generality, we consider kernel ridge regression models with a random feature extractor. Minimizing total loss of FEDLGD (Eq. 2 for harmonizing local heterogeneity with global anchors elicits a model with bounded statistic margin (i. upper bound of the SM bound in Theorem
Proof 2
The first and second term can be bounded by marimizing SM of local virtual training data and global virtual data. The large SM of global virtual data distribution Py(x, y) is encouraged by månimizing cross-entropy Tor (D9,. y) in our objective function Eq.

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Test accuracy for DIGITS under different images class (IPC) and model architectures. R. and C stand for ResNet18 and CouvNet, respectively, and we set IPC to 10 and 'Average' the unweighted lest accuracy average of all the clients. The best results are marked in bold.
DIGIIS
MNIST
SVHN
USPS
SynthDigits
MNIST-M
Average
IPC
10
50
10
50
10
50
10
50
10
50
10
50
R
73.0
92.5
20.5
48.9
83.0
89.7
13.6
28.0
37.8
72.3
45.6
66.3
FedAvg
C
94.0
96.1
65.9
71.7
91.0
92.9
55.5
69.1
73.2
83.3
75.9
82.6
R
72.6
92.5
19.7
48.4
81.5
90.1
13.2
27.9
37.3
67.9
44.8
65.3
FedProx
93.9
96.1
66.0
71.5
90.9
92.9
55.4
69.0
73.7
83.3
76.0
82.5
75.5
92.3
17.3
50.6
80.3
90.1
11.4
30.5
38.3
67.9
44.6 6
66.3
FedNova
C
94.2
96.2
65.5
73.1
90.6
93.0
56.2
69.1
74.6
83.7
76.2
83.0
R
75.8
93.4
16.4
53.8
79.3
91.3
11.2
34.2
38.3
70.8
44.2
68.7
Scaffold
94.1
96.3
64.9
73.3
90.6
93.4
56.0
70.1
74.6
84.7
76.0
83.6
R 15.5
80.4
15.9
14.2
25.0
82.4
10.0
11.5
11.0
35.4
15.5
44.8
MOON
C 85.0
95.5
49.2
70.5
83.4
92.0
31.5
67.2
56.9
82.3
61.2
81.5
R 13.5
56.7
9.3
7.8
39.6
79.7
10.0
10.6
10.0
11.2
16.5
33.2
FedProto
C
91.9
96.8
52.7
73.9
93.3
96.6
27.2
52.8
69.0
84.3
668
80.9
R 87.8
95.9
29.5
67.0
88.0
93.5
18.2
60.7
52.2
85.7
55.
80.5
VHI.
C
95.0
96.9
68.6
75.2
92.2
94.4
60.7
72.3
76.1
83.7
78,5
84.5
R
92.9
96.7
46.9
73.3
89.1
93.9
27.9
72.9
70.8
85.2
65.5
84.4
FEDLGD
C
95.8
97.1
68.2
77.3
92.4
94.6
67.4
78.5
79.4
86.1
80.6
86.7
The third lerm represents the discreponcyy of distributions of virtual and real dala. We denole this term (Pa) = E (Pel) yv) Pylá v)) and aim to show that De (Py((p)) can achieve small upper bound under proper assumptions.
Based on Lemma the first-order distribution matching objective DP (Pg) is approzimately equal gradient oly matching of each class, shown in objective Dist (Eq. D Namely, minimizing gradient matching objective Disst. in FEDLGD implies minimizing Do (Pg) in the selling. Hence, using gradient matching generaled global virtual data clicits the model's SM a tight lower bound.
Remark 1 The key distinction betwween FEDLGD and VHL. primarily lies in the fonal term, which is eraclly a distribution matching objective. It is important to note that in VHL, the global virtual data is generated from on un-pretie on-pretrained StyleGAN, originating from vorious Gaussian distributions, which we denote as Pg. The VHL paper only provided lower bound for DP ( but did not show how it is upper bounded. However, for the purpose of mazimizing SM to achieve strong generalization, bound. Therefore, upper bounded the last term is desired. In contrast, our approach employs the gradient matching strategy to synthesize the global virtual data. To proce our performance improvement, we can show that FEDLGD could achieve a tight lower bound for SM.
want to show SM has tight lower
5 Experiment
To evaluate FEDLGD, consider the FL setting in which clients obtain data from different domains with the same target task. Specifically, we compare with multiple baselines on benchmark datasets DIGITS, where each client has data from completely different open-sourced datasets. The experiment aims to show that FEDLGD can ellectively mitigate large domain shifts. Additionally, we evaluate the performance of FEDLGD on another large benchmark dataset, CIFAR1 OC (Hendrycks & Dietterich 2019), which collects data with different corruptions yiclding data distribution shift and contains a large number of clients, so that we can investigate varied client sampling in FL. The experiment aims to show FEDLGD's feasibility on large-scale FL environments. We also validate the performance under real medical datasets, RETINA.

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2: Averaged test accuracy for CIFARICC with ConvNet.
CTPAR1OC
FedAAv
FedProx
FedNava
Sraffold
MOON
FedProto
VHI.
FRDL.G
IPC
50
10
10
10
50
10
50
50
0.2
27.0
44.9
27.0
44.9
26.7
34.1
27.0
44.9
20.5
31.3
14.4
26.3
21.8
45.0
32.9
46.8
Client ratio
0.5
29.8
51.4
29.8
51.4
29.6
45.9
30.6
51.6
23.8
43.2
16.4
36.1
29.3
51.7
39.5
52.8
33.0
54.9
33.0
54.9
30.0
53.2
33.8
54.5
26.4
51.6
20.3
40.4
34.4
55.2
47.6
57.4
5.1 Training and Evaluation Setup
Model architecture. We adapt ResNet18 (He al. 2016 and ConvNet. (Zhao et al.
2021 (detailed in
Appendix C.4 in our study. To achieve the optimal performance, we apply the samee architecture to perform both the local distillation task and the classification task, as suggested in (Zhao et al. 2021
Comparison methods. We compare the performance of downstream classification tasks using state-of-the- art heterogeneous FL algorithms, FedAvg (McMahan et al., 2017), FedProx (Li et al. 2020). FeddNva (Wang
ctt al. 2020), Scaffold (Karimireddy et al. 2020 MOON (Li ct al., 2021 FedProto (Tan et al. 2022). ; and
VHL. Tang al.] 2022]. We use local virtual data from our initialization stage for FL methods other than
ours and perform classification on client's testing set and report the test accuracies.
FL training setup. We use the SGD optimizer to update local models. If not specified, our default sctting for learning rate is 10-2, local model update epochs is 1, total update rounds is 100, the batch size for local training is 32, and the number of virtual data update iterations ([=)) is 10. The numbers of default virtual data distillation steps for clients and server are set to 100 and 500, respectively. Since we only have few clients for DIGITS, we will select all the clients for each iteration, while the client selection criteria for CIFAR1 OC experiments will be specified in Sec. 5.3
Proper Initialization for Distillation. For privacy concerns and model performance, initialize local virtual data using local statistics for local data distillation. Specifically, each client calculates the statistics of its own data for each class, denoted as pf,of, and then initializes the distillation images per class, x ~ N(u,, where с and represent each client and categorical label. For privacy consideration, we use random noise as initialization for global virtual data distillation. The comparison between different initialization strategies can be found in Appendix B
5.2 DIGITS Experiment
Datasets. We use the following datasets for our benchmark experiments: DIGITS = {MNIST LeCun
et al. 1998). SVHN (Ntzerr et al. 2011p. USPS (Hull 1994). SynthDigits (aain & Lempitsky 2015
MNIST-M (Ganin & Lempitsky 2015 }. Each dataset in DIGITS contains handwritten, real street and synthetic digit images of 1, .9. As a result, we have 5 clients in the experiments.
Comparison under various conditions. To validate the effectiveness of FEDLGD, we first compare it with the alternative FL methods varying on two important factors: Image-per-class (IPC) and different deep neural network architectures (arch). We use IPC E {10, 50} and arch ResNet18(R), ConvNet(C)) to examine the performance of SOTA models and FEDLGD using distilled DIGITS. Note that we fix IPC = 10 for global virtual data and vary IPC for local virtual data. Tab. D shows the test accuracies of DIGITS experiments. One can observe that for each FL algorithm, NonvNet(OO (C) always has the best performance under all IPCs. The observation is consistent with (Zhao & Bilen 2023 as more complex architectures may cause over-fitting to virtual data. is also shown that using IPC = 50 always outperforms IPC 10 as expected since more virtual data can captures more real data distribution and thus facilitates model training. Overall, FEDLGD outperforms other SOTA methods, where average accuracy, F FEDLGD increases the best test accuracy results among the baseline methods of 2.1% (IPC =10, arch C), 10.4% (IPC =10, arch R), 2.2% (IPC = 50, arch C) and 3.9% (IPC arch R). VHL is the closest strategy to FEDLGD and achieves the best, performance among the baseline methods, indicating that the feature alignment solutions are promising for handling heterogeneity in federated virtual learning. However, the worse performance may result from the differences synthesizing global virtual data. VHL uses untrained StyleGAN (Karras al. 2019) to generate global virtual data without further updating. On the contrary, we gradually update our global virtual data during FL. training.
10

---

Published in Transactions on Machine Learning Research (01/2025)
5.3 CIFAR10C Experiment
Datasets. We conduct large-scale FL experiments on CIFAR1 0 where, like previous studies [Li et al. 20210, we apply Dirichlet distribution with a = to generate 3 partitions on cach distorted Cifar10- C (Hendrycks Dietterich 2019), resulting in 57 domain and label helerogeneous non-IID clients. In addition, randomly sample fraction of clients with ratio 0.2, 0.5, and 1 for each FL round.
Comparison under different client sampling ratios. The objective of the experiment is to test FEDLGD under popular FL questions: class imbalance, large number of clients, different client sample ratios, and domain and label heterogeneity. One benefit of federated virtual learning is that we can easily handle class imbalance by distilling the same number (IPC) of virtual data. We will vary IPC and fix the model architecture to ConvNet since it is validated
yield better performance in virtual training. One can observe from Tab. _ that FEDI.GD consistently achieves the best performance under different TPC and client sampling ratios. We would like to point out that when IPC=10, the performance boosts are signifi- cant, which indicates that FEDLGD is well-suited for FL when there is large group of clients with limited number of local virtual data.
5.4 RETINA Experiment
Dataset.
For medical dataset, we use the retina image datasets, RETINA = (Drishti Sivaswamy
ct
al. 2014 Acrima (Diaz-Pinto et al.) 2019), Rim [Batista et al. 2020), Refuge (Oraando
eet
al. 20201) , where each dataset contains retina
images from different stations with image size
96 x 96, thus forming four clients in FL. We
perform binary classification to identify Glau-
comatous and Normal. Example images and Comparison with baselines. The results for RETINA experiments are shown in Table g where D, A, Ri, Re represent Drishti, Acrima, Rim, and Refuge datasets. We only set IPC=10 for this ex- periment as clients in RETINA contain much fewer data points. The learning rate is set to 10-3 FEDLGD has the best performance compared to the other methods w.r.t the unweighted averaged accu- racy (Avg) among clients. To be precise, FEDL.GD increases unweighted averaged test accuracy for 3. 3.1%(versuu the best baseline) on ConvNet. The same accuracy for different methods due to the limited number of testing samples. We conjecture the reason why VHL (Tang et al., 2022) has lower performance improvement in RET experiments is that this dataset is in higher dimensional and clin- ical diagnosis evidence on fine-grat ine-grained details, e.g., cup-to-dise ratio and disc rim integrity (Schuster et al.
distributions can be found in Appendix C.3
Test accuracy for RETINA experiments under dif- ferent model architectures and IPC=10. We have clients: Drishti(D), Acrima(A), Rim(Ri), and Refuge(Ree,, respec- tively. We also show the average test accuracy (Avg). The same accuracy for different methods is due to the limited mumber of testing samples.
Table 3:
RETINA
Ri
Re
Avg
FedAvg
69.4
84.0
88.0
86.5
82.0
FedProx
68.4
84.0
88.0
86.5
81.7
FedNova
68.4
84.0
88.0
86.5
81.7
Scaffold
68.4
84.0
88.0
86.5
81.7
MOON
57.9
72.0
76.0
85.0
72.7
FedProto
C
73.6
86.0
54.0
77.5
72.8
VILL
68.4
78.0
81.0
87.0
78.6
FEDLGD
78.9
86.0
88.0
87.5
85.1
2020 Therefore, it is difficult for untrained Style- GAN Karras et al. 2019) to serve as anchor for this kind of larger images.
5.5 Ablation studies for FedLGD
The success of FEDLGD relies on the novel design of local-global data distillation, whore the selection of regularization loss and the number of iterations for data distillation play key role. Recall that among the total FL training epochs, we perform local-global distillation on the selected 5 iterations, where the server and clients will perform data updating for some pre-defined steps. Thus, we will study the choice of regularization loss and its weighting (A) in the total loss function, well the effect of iterations and steps. By default, we use ConvNet, global TPC=10, local IPC=50, |r|=10, and (local, global) update sleps=(100,
1Cifarl0-C C is collection of angmented Ci that applies different corruptions, resulting 6k 114k data points.
11

---

Published in Transactions on Machine Learning Research (01/2025)
100
Compuration Cost (M FLOPS)
DIGITS CIFARLOC FLOPs
DIGITS
CIFAR1OC
80
LAveraged Accuracy
Accuracy
60
60
56
DIGITS
CIFAR1OC
20
Contrastiveoours)
MND
0
20 124455 "12 2c0
1770 P1 Local
KCC
[c]
Local
(a) Vary Rog, loss
(b) Vary |r]
(c) Vary steps
(d) Vary steps
Figure 3: (a) Comparison between different regularization losses and their weightings(A)) (A). One can observe that LCon gives us better and more stable performance with different coefficient choices. (b) The solid curves describes the improved accuracy compared to |T| 0, and the dashed curve indicates the computation cost. The model performance improves with the increasing [r], which is a trade-off between computation cost and model performance. Vary data updating steps for (c) DIGITS and (d) CIFAR10C. FEDLGD yields consistent performance, and the accuracy improves with increasing number of local and global steps.
label
Aligned fea
Feature shift
Feature shift
clentl
(a) FedAvg
FedLGD without regularization
(c) FedLGD
Figure 4: tSNE plots feature space for FedAvg, FEDLGD without regularization, and FEDLGD.
500). We also discuss computation cost and privacy, two important factors in FL. Further ablation studies can be found in Appendix
Effect of regularization loss. FEDLGD uses supervised contrastive loss LCou regularization term to encourage local and global virtual data embedding into similar feature space. To demonst rate ils effective- ness, we perform ablation studies to replace LCon with an alternative distribution similarity measurement, MMD loss, with different X's ranging from 0 to 20. Fig. Ba shows the average test accuracy. Using Lcon gives us better and more stable performance with different choices. We select 1=10 and for DIGITS and CIFAR1 OC, respectively. It. worth noting that when O, FEDI.GD can still yield competitive accuracy, which indicates the utility of our local and global virtual data. To explain the effect of our proposed regu- larization loss feature representations, space using LSNE (Van der Maaten & Hinton 2008) shown in Fig. 1 For the model trained with FedAvg (Fig. 1,), features from two clients (x and o) are closer to their own distribution regardless of the labels (col- ors). In Fig. ,, we perform virtual FL training but without the regularization term (Eq. Fig. shows FEDLGD, and one can observe that data from different clients with the same label are grouped together.
embed the latent features before fully-connected layers to 2D
Analysis of distillation iterations (Irl). Fig. 3b shows the improved averaged test accuracy if we increase the number of distillation iterations with FEDLGD. The base accuracy for DIGITS and CIFAR1 OC are 85.8 and 55.2 when 7 0. We fix local and global update steps to 100 and 500, and the selected iterations (r) are defined as arithmetic sequences with d = 5 (i.e., 7 (...).. One can observe that the model performance improves with the increasing |r|. This because we obtain better virtual data with more local-global distillation itns,tions, which trade-off between computation cost. and model performance.
we vary (local, global) data updating steps. One can observe that FEDLGD yields stable performance (always outperforms baselines), and the accuracy slightly improves with an increasing number of local and global steps.
Robustness on virtual data update steps. In Fig. 3e and Fig. 3d
12

---

Published in Transactions on Machine Learning Research (01/2025)
Computation cost. We have shown the increased computation cost caused by increasing the number of selected rounds |T| in Fig. 3b Here, we discuss the overall accumulated computation cost for the 100 total FL training rounds, including both selected and unselected iterations in Fig. 5 The computation costs for FEDLGD in DIGITS and CIFAR1OC are identical since we use IPC=50 for training. For RETINA, since we apply I0,=10, FEDLGD has significant efficiency improvement. Overall, FEDLGD reduces the computation cost on the clients' side by training with virtual data compared to classical FodAvg that train on real datasets.
Privacy. We note that FEDLGD uses prestiis information, i.e., shared averaged gradients and global model, to distill virtual data, so there is no extra privacy leakage. Like the standard FL training, FEDLGD may be vulnerable to deep pri- vacy attacks, such as membership inference attacks (MIAs) (Shokri et. al.) 2017] and gradient inver- sion attacks(GlAs)) Zhuu et al. 2019 Huang et al.) 2021b. We empirically show FEDLGD can poten- tially defend both attacks, which is also implied by Xiong et al. 2023 Dong et al. 2022). Preserv-
Foddag
Fedlvg
PedMay PedLCO
150-
50-
50
100
50 TL Tordunds
160
TL: rounds
TL munds
(a) DIGITS
(b) CIFAR1OC
(c) RETINA
Figure 5: FEDLGG reduces the Accumulated computa-
tion cost on the clients' side compared to FedAvg.
ing identity-level privacy can be further improved by employing differential privacy (Abadi et al. 2016 in
dataset distillation, such applying DPSGD during local data distillation or applying DPSGD on the local gradients, but this goes beyond the main focus of our work.
MIAs (Shokri et al. 2017 aims to identify if a given data point belongs to the model's training data. We compare the performance of MIA directly on mod- cls trained with original data (FedAvg) and with the synthetic datasel (FEDLGD). If the MIA per- formance on the original images is worse than the FedAvg, we claim that the synthetic data helps with privacy. Here, implemented the Like- lihood Ratio MTA (Carlini et al. 2022a). where the gradients are collected for the server model on train- ing and testing data individually. The likelihood of the point belonging
FoiLGD Fed/vgg
FalLGD
Fedtivgg
Thue Peskove Bale
0.8
0.5
0.0 0.2
04
0.6
0.8
1.0
0.0
02
0.4
0.6
0.5
1.0
(a) MIA Synth-Digits
(b) MIA MNIST-M
Figure 6: MIA results models trained with FedAvg (us- ing original dataset) and FEDLGD (using distilled virtual dataset). If the ROC curve is the same as the diagonal line, it means the membership cannot be inferred.
the training set is then ob- tained using the Gaussian kernel density estimation (Fig. G. If the ROC curve intersects with the diago- nal dashed line (representing random membership classifier), it signifies that the approach provides stronger defense against membership inference compared to the method with a larger arca under the ROC curve. FEDLGD results in ROC curves that are more closely aligned with the diagonal line, suggesting that allacking membership becomes more challenging.
Using dataset distillation to synthesize virtual data can be shown to mitigate against gradient- gradient-based in- version altacks (GIAs) [Geiping et al. 2020 Huang
et al. 2021 Here, use Cifar10 (Krizhevsky
(a) Recoustructed raw Cifar10 imagee
ct al. 2009) as an example. We perform local train-
ing on ConvNet from one client in CIFARI OCand apply gradient inversion attack to reconstruct the raw images. Then, we evaluate the reconstruction quality using perceptual loss (LPIPS) (Zhang al., 2018). As result, the reconstructed distilled image is visually different from raw images, and it effec- tively alleviates the attack from perceptual perspective, by reducing LPIPS from 0.253 to 0.177. Note that
(b) Reconstructed distilled Cifar10 images
Figure 7: GTA and distilled Cifar10 images.
13

---

Published in Transactions on Machine Learning Research (01/2025)
in FEDLGD, the shared global virtual data is synthesized by the averaged gradients, which further improves the privacy guarantee.
6 Conclusion
In this paper, we introduce a new approach for FL, called FEDLGD, It utilizes virtual data on both client and server sides to train FL models. We are the first to reveal that FL on distilled local virtual data can increase heterogencity. To tackle the heterogeneity issue, we scamlessly integrated dataset distillation algorithms into FL pipeline by proposing iterative distribution matching and federated gradient matching to iteratively update local and global virtual data. Then, apply global virtual regularization to effectively harmonize domain shift. Our experiments on benchmark and real medical datasets show that FEDLGD outperforms current maate-of-theaat methods in heterogeneous settings. Furthermore, FEDLGD can be combined with other nodlelsynccroonzzaioon--sss FI, approaches to further improve its performance. The potential limitation lies in the additional communication and computation cost data distillation, but we show that the trade-off is acceptable and can be mitigated by decreasing distillation iterations and steps. data generation and utilizing the synthesized
Our future direction includes investigating global virtual data for federated continual learning training personalized models. We believe that this work sheds light on how to effectively mitigale data heterogeneity from dataset distillation perspective and will inspire future work to enhance FL performance, privacy, and efficiency.
Acknowledgement
This work is supported in part by the Natural Sciences and Engineering Research Council Canada (NSERC), CIFAR AI Chair Awards, CIFAR AI Catalyst Grant, NVIDIA Hardware Award, UBC Sock- eye, and Compute Canada Research Platform.
References
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang, Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference compuler and communications security, pp. 308 308-318, 2016.
Francisco José Fumero Batista, Tinguaro Diaz-Aleman, Jose Sigut, Silvia Alayon, Rafael Arnay, and Denisse Angel-Pereira. Rim-one dl: A unified retinal image database for assessing glaucoma using deep learning. Image Analysis & Stereology, 39(3):161-161) 2020.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp- 1897 1914. IEEE, 2022a.
Nicholas Carlini, Vitaly Feldman, and Milad Nasr. No free lunch in' privacy for free: How does dataset condensation help privacy". arXiv preprint arXiv:2209. 14987, 2022b.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu. Dataset distilla- tion by matching training trajectories. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
Hong-Yos You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. arXio preprint arXiv:2107.00778, 2021.
Huancheng Chen, Haris Vikalo, et al. The best of both worlds: Accurate global and personalized models through fodorated learning with data-free hyper-knowledge distillation. arXiv preprint orXiu:2301.08968 2023.
Sijin Chen, Zhize Li, and Yuejic Chi. Escaping saddle points in heterogencous federated learning via dis- tributed sgd with communication compression. In International Conference on Artificial Intelligence and Slatistics, pp. 2701-20 -7009. PML.R, 2024.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Andres Pinz-Pinto, Sandra Morales, Valery Naranjo, Thomas Köhler, Jose M Mossi, and Amparo Navea. Cnns for automatic glaucoma assessment using fundus images: extensive validation. Biomedical engi- neering online, 18(1):1-18 19, 2019.
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In International Conference on Machine Learning, pp. 5378-5396. PMLR, 2022.
Ziqing Fan, Jiangchao Yao, Bo Han, Ya Zhang, Yanfeng Wang, et al. Federated learning with bilateral curation for partially class-disjoint data. Advances in Neural Information Processing Systems, 36, 2024.
Yunzhen Feng, Shanmukha Ramakrishna Vedantam, and Julia Kempe. Embarrassingly simple dataset distillation. In The Twelfth International Conference on Learning Representations, 2023.
Jean-Yves Franceschi, Emmanuel De Bézenac, Ibrahim Ayed, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. A neural tangent kernel perspective of gans. In International Conference on Machine Learning. pp. 6660. PMLR, 2022.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pp. 1180 1189. PMLR, 2015.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-how easy is it to break privacy in federated learning? Advances in neural information processing systems, 33: 16937 16947, 2020.
Jack Goetz and Ambuj Tewari. Federated learning via synthetic data, arXiv preprint 2xXi:2200.04489, 2020.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723 773, 2012.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. Advances in Neural Information Processing Systems, 18:14068-14080, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.1 12261, 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint orXio:1909.003:0 2019.
Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, and Yue Liu. Fedsynth: Gradient compression via synthetic data in federated learning. arXie prepriut ar Ntv:2204.01210 2022.
Chun- Yin Huang, Kartik Srinivas, Xin Zhang, and Xiaoxiao Li. Overcoming data and model heterogeneities in decentralized federated learning via synthetic anchors. In forty-first International Conference Ma- chine Learning, 2024.
Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. Advonces in Neural Information Processing Systems, 34:7232- 7241, 2021.
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550 554, 1994.
Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Scbastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for foderated learning. In International PMIRR 2020.
Conference Machine Learning, pp.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE conference on computer vision and pollern recognilion, pp. 4401 4410, 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Newral Information Processing Systems, 33:18661- 18673, 2020.
Vladimir Koltchinskii and Dmitry Panchenko. Empirical margin distributions and bounding the generaliza- tion error of combined classifiers. The Annals of Statistics, 30(1):1-50, 2002.
Alex Krishevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Dataset distillation for medical dataset sharing. arXiv preprint ar Xiv:2200. 14608, 2022.
Hongcheng Li, Yucan Zhou, Xiaoyan Gu, Bo Li, and Weiping Wang. Diversified semantic distribution matching for dataset distillation. In ACM Multimedia 2024.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 10713 10722, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet. Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-455 2020.
Tao Lin, Lingjig Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2383 2263, 2020.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effan effficeet learning of deep networks from decentralized data. In Artificial intelligence and slatistics, pp. 1273 1282. PMLR, 2017.
Mohamad Amin Mohamadi and Danica J Sutherland. A fast, well-founded approximation to the empirical neural tangent kernel. arXiv preprint 2tf::2200.1255.. 2022.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. Tn NIPS workshop deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Sergey Nikolenko. Synthetic data for deep learning, volume 174. Springer, 2021.
José Ignacio Orlando, Huazhu Fu, João Barbosa Breda, Karel van Keer, Deepti R Bathula, Andrés Diaz- Pinto, Ruogu Fang, Pheng-Ann Heng, Jeyoung Kim, JoonHHo Lee, et al. Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Medical image analysis, 59:101570, 2020.
Noveen Sachdeva and Julian McAuley. Data distillation: A survey. arXiv preprint XrXiv:2301.04272, 2023.
Alexander K Schuster, Carl Erb, Esther M Hoffmann, Thomas Dietlein, and Norbert Pfeiffer. The diagnosis and treatment of glaucoma. Deutsches Ärzteblatt International, 117(13):225, 2020.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Jayanthi Sivaswamy, SR Krishnadas, Gopal Datt Joshi, Madhulika Jain, and A Ujjwaft Syed Tabish. Drishti- gs: Retinal image dataset for optic nerve head (onh) segmentation. In 2014 IEEE 11th international symposium biomedical imaging (ISBI), pp. 53 56. IEFE, 2014.
16

---

Published in Transactions on Machine Learning Research (01/2025)
Zhenyu Sun, Xiaochun Niu, and Ermin Wei. Understanding generalization of federated learning via stability: Heterogeneity matters. In International Conference. on Artificial Intelligence and Statistics, pp. 076-684. PMLR, 2024.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prolotype learning across helerogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8432 8440, 2022.
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Virtual homogencity learning: Delending against data heterogeneity in federated learning. arXiv preprint urXiw:2206. 02465, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Praclical Guide, 1st Fd., Cham: Springer International Publishing, 10(3152676):11 5555, 2017.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33: 1611- 2020.
Tongzhou Wang, Jun-Yar Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. orXiv preprinl arXiv:1811.10958 2018.
Jie Wen, Zhixia Zhang, Yang Lan, Zhihua Cui, Jianghui Cai, and Wensheng Zhang. A survey on federated learning: challenges and applications. International Journal of Machine Learning and Cybernelics, 14(2): 513 535, 2023.
Zhiyuan Wu, Sheng Sun, Yuwci Wang, Min Liu, Quyang Pan, Junbo Zhang, Zeju Li, and Qingxiang Liu. Exploring the distributed knowledge congruence in proxy-data-free lederated distillation. arXie preprinl arXiv:2204.07022 2022.
Zhiyuan Wu, Sheng Sun, Yuwei Wang, Min Liu, Quyang Pan, Xuefeng Jiang, and Bo Gao. Fedict: Federated multi-task distillation for multi-access edge computing. IEEE Transactions on Parallel and Distributed Systems, 2023.
Yuanhao Xiong, Ruochen Wang, Minhao Cheng, Felix Yu, and Cho-Jui Hsieh. Feddm: Iterative distribution matching for commuicationn-effieeee federated learning, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16323-16332, 2023.
Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, and Bo Han. Fedfed: Feature distillation against data heterogeneity in federated learning. Advances in Neural Information Processing Systems, 36, 2024.
Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogeneous federated learning: State- of-the-art and research challenges. ACM Computing Surveys, 56(3):1 44, 2023a.
Rui Ye, Zhenyang Ni, Chenxin Xu, Jianyu Wang, Siheng Chen, and Yonina C Eldar. Fedfm: Anchor-basod feature matching for data heterogeneity in federated learning. IEEE Transactions on Signal Processing, 2023b.
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. arXio preprint arXiv:2301.00014, 2023.
Hansong Zhang, Shikun Li, Fanzhao Lin, Weiping Wang, Zhenxing Qian, and Shiming Ge. Dance: Dual-view distribution alignment for dataset condensation. ar Xāu preprint ar urXio:2406. 01063, 2024.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Lin Zhang. Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fino-tuning global model via data-free knowledge distillation for non-iid lederated learning. In Proceedings of the IREF/CVE Conference Computer Vision and Pattern Recognition, pp. 10174 10183, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pollern recognilion, pp. 586-586660 595, 2018.
Bo Zhao and Hakan Bilen. Dataset condensation with diffcrentiable siamese augmentation. In International Conference Machine Learning, pp. 12674-12685 PMLR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514 6523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR, 1 (2):3, 2021.
Tailin Zhou, Jun Zhang, and Danny HK Tsang. Fedfa: Federated learning with feature anchors align features and classifiers for heterogencous data. IEEE Transactions Mobile Computing, 2023.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances neural information processing systems, 32, 2019.
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-frec knowledge distillation for heterogencous federated learning. In International conference on machine learning, pp. 12878 12889. PMLR, 2021.
18


Reviews:
[{"review_id": "XEx2zXRcat", "reviewer": "Reviewer_ye5S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The idea of harmonizing local heterogeneity with global anchors is an interesting one.\n2. Decent literature review.\n\n### Weaknesses\nNotational issues:\n1. In description of eq (1), $x_i$ and $\\tilde{x}_j$ are never introduced, including in the notation section of the appendix. The reader is left to assume it's meaning.\n\nClaims and explanations:\n1. Authors claim that they are \"the first to reveal that using distilled local virtual data can exacerbate the heterogeneity problem\". However, only one example is provided if I am not mistaken with USPS and SynthDigits. Does this hold in Cifar10 and other dataset the authors use? I think more is needed to justify this claim.\n3. Authors mention that their choice to use global feature extractors instead of randomly initialized models to extract features in turn generates better task specific local virtual data and that this is inspired by empirical neural tangent kernel works. Could you explain how are these related?\n\nExperimental Setup:\n1. Relatively simpler tasks and datasets are used (retina with binary classification, cifar10c, and digits). While this is not necessarily a critique, in does limit the claims in the paper, such as \"we evaluate the performance of FedLGD on another large benchmark dataset\"\n2. Although the authors compare with a range of FL methods in their baselines, none of the methods use distillation. Some distillation in FL works [1, 2]. Could the authors either compare to these works or provide arguments on why comparison is not necessary?\n\n\nWriting and Language:\n1. The writing fell short for me. While this might be acceptable in some cases, here it significantly reduces the clarity of the exposition.\n2. Too many adjectives are jumbled together to create noun phrases, often multiple times in single sentences, making the paper very hard to read - especially in crucial places. For example: \"... address the heterogeneity issues with the aid of *global virtual anchor distillation and regularization* ...\" This is a common theme across the paper.\n3. Too many typos. E.g. \n    - \"With compromising privacy in implementation, our global anchors ...\". Should it be *without compromising*?  \n    -  \"morden\" machine learning. etc.\n\nAdditional Comments and Questions:\n1. Is the idea of global anchors similar to global prototype distillation [2]? Does it (or not) differ?\n2. What does Images Per class (IPC) mean?\n\n**References:**\n1. Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments [2023].\n2. Global prototype distillation for heterogeneous federated learning [2024].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a pipeline that uses distilled virtual dataset, which represents a smaller and condensed form of the training data for Federated Learning (FL):\n1. Authors use distribution matching (DM) [zhao and Bilen, 2023] to match the model's feature distribution on the original dataset $D^c$ and distilled dataset $\\tilde{D}^c$ in an iterative manner.\n2. They propose adding a regularization term in the feature space to the total loss function to handle data heterogeneity among clients arising due to data distillation.\n3. Gradient matching is used to distill global virtual data to make the global virtual data representative of the global data distribution.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Fix typos and improve writing/presentation.\n2. Either CIFAR-100 or Tiny ImageNet would serve as better test cases to validate the ideas at a larger scale and with more classes. If this is not possible, please either:\n    1. Provide convincing arguments for why CIFAR-10-C qualifies as a \"large-scale dataset\" or,\n    2. Revise claims about the method's effectiveness on large-scale datasets.\n3. Please, either compare to related distillation works [1,2] or provide arguments why it's not necessary.\n4. Please provide more justification (e.g. experimental) to the claim: \"we are the first to reveal that using distilled local virtual data can exacerbate the heterogeneity problem\".", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PANrNpojUY", "reviewer": "Reviewer_SoTu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of local-global dataset distillation is interesting.\n- The authors conducted extensive experiments to demonstrate the effectiveness of their method\n\nWeakness:\n- The symbols are not consistent throughout the paper, i.e., $\\tilde{D}_i$, $\\tilde{D}^c$, $\\tilde{D}_g$\n- The definition of the small virtual dataset is unclear. More specifically, why does the equation $|\\tilde{D}_i| \\ll |D_i|$ need to hold?\n- The authors claim that `We cliam a ‘good’ global virtual data should be representative of the global data distributions`. If I understand correctly, in most existing FL literature, the global data is typically considered the test dataset. Could the authors elaborate on a few scenarios or works where the global data exists but is not the test dataset?\n- In the experimental results section, it is surprising to see that FedAvg outperforms other more advanced methods, which are specifically designed to address data heterogeneity, in most experimental setups. The authors are encouraged to explain the reasons behind this.\n- According to Figure 2, in addition to the local model and the global model being shared between clients and the server, the global virtual data $\\tilde{D}_{new}^g$ is also shared between the clients and the server. However, in the Privacy section, the authors mention that `We note that FedLGD uses pre-existing information, i.e.,, shared averaged gradients and global model, to distill virtual data, so there is no extra privacy leakage` . The authors are encouraged to explain the discrepancy between these two parts. \n- The authors are also encouraged to explain why FedLGD has better privacy protection than FedAvg.\n- The technical contribution is marginal.\n- The authors mention that `The global virtual data then serves as anchors to alleviate domain shifts among clients.` Could the authors indicate which experimental results support this statement? Additionally, the authors need to define \"domain shift\" as used in this paper prior to making this statement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the issue of data heterogeneity across clients in federated learning via local-global data distillation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- cliam -> claim \n- the consistency of the symbols\n- Big bracket for equation 5 and 6 \n- i.e.,, -> i.e., \n- What does `different images per class` mean?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qg63aJuygy", "reviewer": "Reviewer_Q8sp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- The paper is including a large number of experiments and comparing with many baselines (7).\n- The paper provides a good overview of the literature and cites generously\n\n**Weaknesses**:\n\nOne of the main claims is that standard FL with virtual datasets does not work and that this is because of increased data heterogeneity between clients. From Figure 1 and the introduction it appears that this claim is based on experiments on two datasets, one distillation technique and one FL algorithm. To include such a strong claim I think it is necessary to investigate the claim more thoroughly. Specifically:\n\n- The claims \"distilled virtual data increases heteogenity\" / \"the first to reveal that using distilled local virtual data can\" seems to only be based on Figure 1. Considering that this is the main motivation for the paper, I suggest providing more evidence for the statement.\n    - Does the statement hold up under change of the distillation technique, data, federated learning technique, etc?\n    - The evidence of heteogenity is based on tSNE, Wasserstein Distance (37.6%) and Maximum Mean Discrepancy (43.3%). The tSNE I would not trust by itself and WD and MMD are not too informative on high dimensional spaces. I would suggest computing FID instead (i.e. WD on a low dimensional feature space).\n- What is the virtual data distillation technique used for Figure 1? The results heavily depends on the process.\n- Is the poor performance of FL algorithms on virtual data a question of data heterogeneity or is the reduce performance just a consequence of virtual datasets being of poorer quality, since they are based on a (small) client dataset? \n- What prevents us from simply sharing the (smaller) virtual dataset and perform model updates on the server using the joint virtual dataset? This would not have any problems regarding heterogeneity. If this approach also fails it would suggests that data heterogeneity is at least not the only problem.\n\nHow the proposed method solves the above problem of virtual data is unclear:\n\n- It is not clear how the contrastive loss in (2) helps with data heterogeneity, since it treats the global/local dataset jointly. The contrastive loss acts across labels, and it is unclear how this relates to resolving the data heterogeneity issue. Is it because the type of heterogeneity considered is label shift, that the method helps?\n- Why is the local virtual dataset not directly shared with the server instead of reconstructing it through Federated Gradient Matching?\n\nThe presentation of the algorithm could be improved:\n\n- The algorithm is described across section 3.2-3.3 across paragraphs making it hard to follow and. Algorithm 1 in the appendix was critical for me to understand the structure of the proposed method. I suggest pulling it into the main text.\n- It is not clear from section 3.2 what is meant by \"selected iterations\" and why only \"early training epochs\" is considered. It would be good to reference a precise algorithm at this point.\n- Maybe explicitly state that (2) will be optimized over $\\theta$.\n- explicitly parameterize $\\psi$ by  $\\theta$ in (2) so it is clear what is being optimizing over.\n- \"only selecting a few rounds in the early stage of FL is sufficient to synthesize useful global virtual data\". Does this mean that the client update of $\\tilde D_t^c$ also only happens in the early epochs, since $\\tau$ governing the update frequency in Algorithm 1 (appendix) is shared between server and client?\n\nAdditional comments:\n\n- Figure 4 tSNE preserved local structures well (clusters) but cannot be used reliably to compare distances between clusters. So it is not too convincing for arguing about a feature shift.\n- \"In such a way, global virtual data can be served for calibration and groups the features of same classes together\". It is not entirely clear to me what this means. Both global and local virtual data plays the same role in the contrastive loss, so why is this comment only considering the global virtual data?\n\nTypos:\n\n- l. 1 morden -> modern\n- Section 3.3.2 cliam -> claim", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers using a distilled virtual dataset, that is smaller than the original client dataset, for federated learning (FL):\n\n- They observe on the USPS and SynthDigits dataset that the wasserstein distance increases between client distributions when distilled, and state that this leads to divergent behaviour in FL when simply replacing client datasets with their distilled version.\n- They propose the following method to mitigate the problem, which switches between two modes:\n    1. Updating the virtual data: i) the client updates their virtual data using the global feature extractor from the global model ii) the server updates it's global virtual data by using Federated Gradient Matching, where client gradients are used on the server to recover the local dataset\n    2. Updating the model: i) clients computes gradients on a cross entropy loss and a contrastive loss (across classes) – both computed on the join global/local virtual dataset ii) the server aggregates gradients and updates the model.\n- The communication involved in the algorithms is:\n    - clients sends gradients\n    - server sends the current model (which includes the feature extractor) and the global virtual dataset", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As stated under weaknesses in more details, I would suggest the following:\n\n- Include more evidence of data heterogeneity and issue for FL (or downtone the claim)\n- Motive the innovations in the algorithm more precisely\n- Present the Algorithm more clearly", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "27FLlddXAQ", "reviewer": "Reviewer_F8EA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths are listed as follows:\n1) Strong motivation. The insight that synthetic local data may exacerbate distribution shift is interesting and valuable.\n2) Provides a thorough evaluation with both experimental results and theoretical analysis.\n3) Well-written and easy to follow.\n\nWeaknesses are listed as follows:\n1) Figures 1 and 4 are not vector-based, which could affect clarity and scalability.\n2) The experiments are relatively small-scale compared to recent studies in the federated learning literature.\n3) The proposed method lacks a privacy analysis about the proposed method, given the authors emphasize the privacy problem in FL.\n4) Unlike simpler methods like FedAvg and optimization algorithms such as FedNova, FedProx, and MOON, this approach requires complex computation capabilities on the server.\n5) In the related works section, while global aggregation or local training are mentioned, other approaches also address data heterogeneity by combining both strategies. Relevant examples include FedRod, FedGELA, and FedHKD:  \n\nFedRod: \"On bridging generic and personalized federated learning for image classification\"  \n\nFedGELA: \"Federated learning with bilateral curation for partially class-disjoint data\"  \n\nFedHKD: \"The best of both worlds: Accurate global and personalized models through federated learning with data-free hyper-knowledge distillation\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents FedLGD, a method to mitigate data heterogeneity in Federated Learning. FedLGD integrates dataset distillation into FL, using compact synthetic data and aligning local and global data through distribution matching. Experiments show FedLGD outperforms current methods in handling heterogeneous FL settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Following changes are requested, if possible:\n1) Change Figure 1 and 4 in a vector form.\n2) Extend experiments to larger dataset with more classes (It is also fine, if it is not work in more complicated dataset).\n3) Privacy analysis(Optional, I understand it may be difficult).\n4) Provide computational cost and communication cost of both local and global training.\n5) Consider adding those related works like FedRod, FedGELA, and FedHKD.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new method for federated learning on virtual heterogeneous data, using local-global data distillation. All reviewers find the studied setting novel and the results provide new insights. The authors’ rebuttal has successfully addressed the major concerns of reviewers, and all reviewers are in favor of accepting this paper. The reviewers also provided some additional feedback to improve the final version, as listed below
- The main motivation that "distilled virtual data increases heterogeneity" has only weak evidence and needs more justification
- The method reconstructs the virtual data through Federated Gradient Matching, but the clients could simply share the virtual data directly. Neither privacy or dataset size are arguments against this direct approach. The motivation needs to be further discussed.

Overall, I recommend acceptance of this submission. I also expect the authors to include the new results and suggested changes during the rebuttal phase to the final version.
Decision: Accept (poster)