Foundation Models Meet Federated Learning: A One-shot Feature-sharing Method with Privacy and Performance Guarantees
Authors: Mahdi Beitollahi
Alex Bie
Sobhan Hemati
Leo Maxime Brunswic
Xu Li
Xi Chen
Guojun Zhang
Venue: Accepted by TMLR ()

Abstract: Adapting foundation models for downstream tasks via Federated Learning (FL) is a promising strategy for protecting privacy while leveraging the capability of foundation models. However, FL's iterative training and model transmission result in high communication costs and GPU memory demands, making large foundation models impractical for FL. This paper introduces a one-shot FL method with a server-side performance bound to enable foundation models by reducing communication costs and GPU memory requirements. Our approach, FedPFT (FL with Parametric Feature Transfer), involves clients learning and transferring parametric models for features extracted from frozen foundation models in a single round. Parametric models are then used to generate synthetic features at the server to train a classifier head. We evaluate FedPFT across eight vision datasets using three vision foundation models. Our findings demonstrate that FedPFT is agnostic to data heterogeneity and network topology and it enhances the communication-accuracy frontier up to 7.8\%. Finally, we show FedPFT's compatibility with differential privacy and its resilience against reconstruction attacks. Our work highlights the capability of private, feature-sharing methods for one-shot knowledge transfer using foundation models.

Published in Transactions on Machine Learning Research (04/2025)
Foundation Models Meet Federated Learning: A One-shot Feature-sharing Method with Privacy and Performance Guar- antees
Mahdi Beitollahi
mahdi, www.ollahiihuuuuueecccccmm
Huawei Noah's 's Ark Lab, Montreal, Canada.
Alex Bie
alex. bic www.aaalwweecommmmmmmm
Huawei Nooh's Ark Lab, Montreal, Canada.
Sobhan Hemati
sohha.kemmaii@haawus..ccm
Huawei Nooh's Ark Lab, Montreal, Canada
Leo Maxime Brunswic Huawei Nooh's Ark Lab, Montreal, Canada.
sohhon.hemattshaaaaee..cm
Xu Li
2u. ww.lica@huawei.com
Huawei Techeoooogies Canada Inc., Otlawa, Canada.
Xi Chen
zí. ci.chhnn wwwwwwi.cooooooomooooo
Huawei Nooh's Ark Lab, Montreal, Canada
Guojun Zhang
wuojun..hannghlaauss.cco
Huawei Noah's Ark Lab, Montreal, Canada.
Reviewed on OpenReview: https: // openreview. net/ forum? id= 55593myukGG
Abstract
Adapting foundation models for downstream tasks via Federated Learning (FL) a promising strategy for protecting privacy while leveraging the capability of foundation models. However, FT.'s iterative training and model transmission result in high communication costs and GPU memory demands, making large foundation models impractical for FL. This paper introduces one-shot FL method with a server-side performance bound to enable foundation models by reducing communication costs and GPU memory requireents. Our approach, FedPFTT (FL with Parametric Feature Transfer), involves clients learning and transferring paramctric models for features extracted from frozen foundation models in a single round. Parametric models are then used to generate synthetic features at the server to train classifier head. We evaluate FedPFTT across eight vision datasets using three vision foundation models. Our findings demonstrate that FedPFT is agnostic to data heterogeneity and network topology and it enhances the communication-accuuee compatibility with differential privacy and its resilience against reconstruction attacks. Our work highlights the capability private, fcature-sharing mothods for onc-shot knowlodge transfer using foundation models.
frontier up to 7.8%. Finally, show FedPFT's
1 Introduction
Federated learning (FL) (McMahan et al. 2017) is learning paradigm that can facilitate fine-tuning foundation models on downstream tasks across clients without sharing their raw data to adhere privacy concerns and regulatory guidelines such GDPR (European Parliament Council of the European Union). Traditional FL. takes place iteratively over multiple rounds, wherein each client frains local model with its

---

Published in Transactions on Machine Learning Research (04/2025)
data and subsequently transmits it to central server for aggregation. The frequent exchange or training of models, each containing hundreds of millions of parameters, imposes an intolerable burden of high communication costs and GPU memory requirement on each client, rendering the scalability of cross-device FL systems to large foundation models impractical (Konččný et al. 2016 Beitollahi Lu. 2023).
One-shot FL. (Guha et al., 2019) an approach address the high communication cost FL by requiring single communication round. Moreover, one-shot FL addresses several drawbacks of multi-round FL: a) coordinating a multi-iteration FL process across large number of clients is susceptible to failures, stemming from issues such as client dropout, resource heterogeneity, and real-world implementation challenges; b) support of scenarios where multi-round FL is impractical, c.g., dynamic environments (Zhou) 2022) where the global model is required to adapt to evolving environments; e) frequent communication poses a higher chance of being intercepted by outsider attacks such as ban-in-the-middee attacks (Bouacida & Mohapatra 2021
While onc-shot FL holds promise, current methods lack the accuracy of multi-round FI. counterparts (sce Figure D. Furthermore, convergence analysis of multi-round FL cannot be extended to one-shot FL due to its reliance large number of iterations; thus, current one-shot FL methods lack theoretical guarantees. Additionally, one-shot FL alone cannot mitigate the GPU memory demands.
Recent strides in foundation models [Bommasani et al. 2021 present novel opportunities for training- knowledge sharing among clients in FL. Foundation models such as CLIP Radford al., ) and GPT series (Radford ct al. 2019 Brown et al. 2020) demonstratc remark- able performance via lask-agnostic represenlations. Features from these models can be leveraged "as they are" without extensive fine-tuning, via probing or clustering, to achieve performance on downstream tasks surpassing that of complex, task-specific mod- els. Further, in well-trained foundation models, there exist lincar paths in reprosentation space that vary according to semantic axes (Mikolov et al. 2013 Härkönen et al.] 2020). A corollary is that fitting simple, smooth, parametric distributions in repre- sentation space can lead to realistic samples sharing semantic characteristics; on the other hand, this is certainly not true in imput space (i.e., Gaussians cannot model the distribution of natural images).
100
Centralized
free and communication--ffiite
90
FedPFT (Ours)
Accuracy (%)
DP-FedPFT
FedNCM
Centrafized
FodAvg
x
Ensemble
70
Ensemble
FedBE
FedNCCM
FedPFT (Ours)
60
FedBE
FedAvg
De-Fed (Ours)
Communication Cost (MB)
Figure 1: Comparison of different one-shot FL methods for image classification on Caltech- 101 with 50 clients. See Section 5.2 for experimental details. FedPFT and DP- FedPFT outperform other one-shot FL methods and are competitive with transmitting real features (Centralized). With more communication budget, multi-round FI. (i.e. FedAvg) performs better than one-shot methods.
Our contributions. In this paper, we propose
FedPFT (Federated Learning with Parametric Feature Transfer) a faiiiin--free, one-shot FL method with a server-side guarantee to enable foundation models in FL. In FedPFT, each client learns and transfers parametric models from extracted features from pre-trained, frozen foundation models in one round. These parametric models are used to generate synthetic features al the server to train classifier head (as illustrated in Figure 2 Therefore, FedPFT does not require training or fine-tuning of large foundation models. We evaluate FedPFT with different families of parametric models, including a mixture of Gaussians. Our evaluation of eight vision datasets with three vision foundation models shows that FedPFTT achieves performance close to centralized training and is agnostic to data heterogeneity. We show that FedPFT supports both centralized and decentralized FL. Finally, we provide theoretical server-side performance guarantees for FedPFT.
Our main contributions include;
We introduce FedPFT, training-free free, one-shot parametric feature-sharing framework that enables foundation models in FT. to enhance the communication-accuraayy frontier.
2

---

Published in Transactions on Machine Learning Research (04/2025)
We evaluate FedPFT using eight vision datasets and three vision foundation models, showing that FedPFT enhances the communication-accuuee frontier by up to 7.8% and is agnostic to various network topologies and distribution shifts including label shift and task shift.
We extend FedPFT to offer differential privacy guarantees, demonstrating favorable privacy-accuracy tradcoffs. Additionally, we conduct reconstruction attacks on various featuro-sharing schemes and demonstrate the privacy risks of sending real features.
We theorctically analyze the performance of FedPFT and provide server-side bound.
2 Related work
do not perform well in one-shot FL settings (Guha et al., 2019) (see Figure D. To overcome this limitation, researchers have explored diverse strategies, including knowledge distillation (KD) (Hinton et al. 2015
One-shot FL. Naive paramcter averaging methods such as FedAvg (McMahan et al.
2017)
ensemble learning
(Guha et al., 2019 Chen & Chao 2020), and generative method (Kasturi & Hota 2023 Zhang et al. 2022).
KD methods commonly rely on public dataset for distillation which is not always available in practice. Further, ensemble learning methods use client. models collectively as the global model, but we demonstrate, they perform poorly with extremely diverse data due to overfitting. Finally, generative methods are used to create or extract synthetic images at the server. Generative methods require exlensive Iraining on the client and can be challenging when clients access only a few samples. In our experiments, we were unsuccessful in utilizing methods like DENSE (Zhang et al. 2022 to extract synthetic images from frozen, pre-trained models (see Appendix for details).
Foundation models for FL. Foundation models can greatly enhance the training of FL, For instance, clients can directly conduct fine-tuning their local data without training from scratch (Chen et al. 2022 Nguyen et al.; Tan et al., 2022 to converge faster and achieve better performance. Further, foundation models allow novel sharing paradigm. For instance, clients can train and share prompts (Zhao et al.) 2023 Guo et al., 2023), instead of sharing high-dimensional model parameters. However, these methods require multiple rounds of sharing paramcters and training of the foundation models.
Sharing feature statistics in FL. Sharing feature statistics has been explored FL. For instance, CCVR (Luo et al., 2021) and Fedlmpro (Tang al., 2024), in addition to conducting multi-iteration FL, represent the features of each class Gaussian distribution and transmit both the model parameters and the distribution parameters to the server to sample and re-train the classifier. Similarly, FedNCM (Legate ct al. 2024 and FedPCL (Tan et al. 2022) consider sharing class prototypes for learning the classifier head using Nearest Class Means and Contrastive Learning, respectively. However, neither a Gaussian nor a class prototype can completely represent the features (see Section 6.1), FedPFT outperforms FedNCM and CCVR by up to 7% (sce Figure D.. Further, none of these methods provide any thcoretical performance or privacy analysis for the effect of sharing feature statistics. Our work is an extension of these methods to formally study and analyze feature-sharing in training-free free, one-shot FL settings. Table I highlights the differences between FedPFT and similar approaches that rely sending the statistics of features.
Table 1: Comparison between FedPFT and approaches that rely on sending the statistics of features.
Parameterizattin of features
Performance guarantee
Training-free foundation model
Privacy guarantee
Supports
Methods
Oue-shot
deccutralized FL
CCVR Luo et 2021
Gaussian Gaussian
X
X
FedImpro Tang et al.
2014
x
x
FedPCL Tan et al. 2002
Class prototypes
X
FedNCM Logate et al.
2004
Class prototypes
/
X
FedPFT (Ous)
Mixture of Gaussians
/ /
3
Preliminaries
Federated learning. The objective of one-shot FL learn a model w from data distributed across I clients, each commuicating only once. We represent D; as the local dataset for client i E {1,... of example-label pairs (x,y), and denote n4 [Dj the number of samples for local data of client. i.
3

---

Published in Transactions on Machine Learning Research (04/2025)
Server
1
E
Client
laan
D1
GMM
Synthetic Features
Client
1
Combined Features
Classiler Head
Dn
Jesza
Dateset
i Feature Estractor
Real
GMM
GWM
Synthetic Festures
Figure 2: Illustration of FedPFT in centralized FL. Each client learns GMMs for their distributions of extracted features for each class. Then, GMM's parameters transmitted the server, which then samples from these distributions to train a classifier head as the global model.
The model w is decomposed into w == h f. where f represents a feature extractor mapping input x to a d-dimensional embedding, and Rd + RCC the classifier head (i.e., linear laycr), with denoting the number of classes. In FL setups, the goal is minimize the objective function:
L(w) Σ E (x
where & is the cross-entropy loss function and EL P4.
Gaussian mixture models. We employ Gaussian mixture models (GMMs) as one of our chosen parametric models, given their concise parameterization and status as universal approximators of densitics (Scott 2015 Our approach relies on learning Gaussian mixtures over feature space Rd.. Let S+ denote the set of all d positive definite matrices. We denote by G(K) the family of all Gaussian mixture distributions comprised of K components over Rd. Each density function g G(K) is identified by set of tuples {(T M2. Σx) /==1 where each mixing weight Ta 2 0 with ΣK Th = 1, each mean vector μa E IRd, and each covariance matrix Σg E s+, satisfying:
g Σ TK N Ex)
where N (1, Σ) refers to the Gaussian density over Rd with mean l and covariance Σ. In addition, we denote Gring (K) to denote Gaussian mixtures comprising of diagonal Gaussians, i.e., with the additional constraint that all Σg are diagonal. We also denote Gspher to Gaussians mixtures with spherical covariances, i.e., each EN E {AIa À E Rzo). We may also refer to the family of full covariance G(K) as Itul(K), and use Goov (K) to denote different family types.
(K))(K)
4 Methods
In this section, We first describe FedPFT for the cantralized FL setting, assuming the presence of centralized server connected to all clients. We also describe the modifications required to adapt FedPFT for (a) the decentralized FL setting; and (b) differontial privacy requirements. Finally, We provide a performance guarantee for FedPFT.
4.1 Centralized FedPFT
In this conventional FL setup, a central server can aggregate the knowledge from all clients. In the centralized FedPFT scenario, as illustrated in Figure 2 each client i extracts class-conditional features from its local
dataset for each available class c {1,-... ...,. Cj:
pic (f(x); (x, E Di, c):
(3)
4

---

Published in Transactions on Machine Learning Research (04/2025)
using the pre-trained foundation model feature extractor f. Next, each client learns runs the Expectation Maximization (EM) algorithm (Dempster et al. 1977 on Fic to learn GMM gb E Ganv (K) for each class that approximates F*. Finally, each client sends its gi parameters {[T My " to the server. On the server side, the server samples class-conditional synthotic features Fi.c from cach received ge paramcters, i.e
K
fiie gi===)= Σ ie NN 2
(4)
with the size of [F Then, the server combines class caas-coonditinal synthetic features Fi.e from all the clients and classes to create synthetic feature dataset F as follows,
1 I D U..
(5)
Finally, the server trains a classifier head on F, minimizing ((v.y) w (e(h; v, where is the cross-cntropy loss. The trained, global classifier head then sent back to the clients, and clients can use w = h f as the global model. This process is described Algorithm
4.2 Decentralized FedPFT
Client i
Traditional FL relies on a central server for coordi- nation, which introduces a single point of failure and limits scalability. This contralized architecture also poses privacy and security risks by exposing model updates and reduces fault tolerance by making the system dependent on server availability. Decentral- ized FL mitigates these issues by enabling direct peer-to-peer communication, improving resilience, and eliminating the need for a trusted aggregator. Therefore, we extend FedPFT to the decentralized setting.
Classatee Head
D;
Dataset
Feature Extractor
Roal Features
Combined
Features
Reseived parameters from user
Received GMM
Synthelic Received Feetures
E
Updated local
GMM
In decentralized FL, there is centralized server, and clients are all connected in ad-hoc manner. clients update received GMMs with their local data. Therefore, each client has the responsibility of aggre- gating the knowledge of its dataset with other clients and passing the knowledge to the next client. FedPFT for decentralized FL is illustrated Figure 3 In this method, similar to centralized FL, each client i creates class conditional features from its local dataset for each label c, Le., FL.e, Then, client samples from the received GMMMs from client j to generate synthetic candiiooniiional features Fü.c, Next, client runs the Expectation Maximization (EM) algorithm Fi.e U Fiie to learn GMM gD E Gev)(K) for cach class that approximates the union of pice and Fri.c. The parameters of gi are sent to the next client. At the same time, each client can use the combined features F U Fii. to train its local classifier head hę. By passing GMMs between clients, knowledge of each client is accumulated and propagated between clients with just. one round of communication where the last client has the knowledge of all the clients.
Figure Illustration of decentralized FedPFTT where
4.3 Differential Privacy
FedPFT's goal is to transfer the parameters of GMMs without leaking clients' private information in their dataset. For formal privacy guarantees, we employ differential privacy (DP, Dwork et al. 2006). To make FedPFT differentially private, we use the Gaussian mechanism (Dwork et al. 2006; 2014] to privatize the release of all mean vectors and covariance matrices. The following thcorem along with the proof the Appendix E provides the (c, diffferrrlll privacy guarantee for FedPFT in the case of Gaussians (Gral(K 1))-
5

---

Published in Transactions on Machine Learning Research (04/2025)
Theorem 4.1 (Privacy Guarantee). Suppose the feature embedding f satisfies ll s 1. Let po(-) and {() be the eslimator of mean and covariance, respectively. Define the Goussian mechanism
((D)) ((D) - ( + Au, E(D) Projs. (D(f(D)) + AΣ),
(6)
(7)
(8)
where the elements of vector Au and matrir AΣ are sampled from independent N (0, (51m(4/)))
)
and Projs, the projection onto the set of positive semi-definile matrices. Then, the Gaussian mechanism M satisfies (c.8)-differente privacy.
Note that the assumption of normalized features in Theorem 4.1 does not limit the performance networks with soft-max loss function since both normalized and unnormalized features have the same expressive power shown in Proposition 3.A in (Zhang et al.) 2023). Further, note that Theorem 4.1 only assures the privacy of GMMss. However, FedPFT sends label counts along with the GMMs to the server, which may not be advisable from a privacy perspective. In Section to address this issue.
A.3 we discuss cryplographic adaptations that can be made
4.4 Performance Guarantee
FedPFT trains a classifier head on the server using synthetic features. The training loss at the server assures that the classifier can classify synthetie features effectively. But how can guarantee that this classifier works well for the real features of clients without accessing real features? The following theorem provides server-side bound on the performance of the classifier head on the real features:
Theorem 4.2 (Performance Guarantee). For any classifier head h, with g--1 and go the 0-1 losses of h on U fD;)) and U U Fi.e respectively,
2
E
RM s.c
(9)
Raw feature's 's 0-1 loss
Synthetic feature's 's 0-1 loss
FM toss of fitting GMM on class e of elsent
where LEM -
Hi,
8RM and (EM is the class-wise log-likelihood of the EM, and Hi,o is the self-entropy 4,e
of the distribution Fic.
This theorem states that the 0-1 loss on all the real features (centralized loss) is bounded by twice the 0-1 loss of the server on all the generated synthetic features plus the sum the expectation EM loss of fitting GMMs on all the classes across all the clients. Therefore, if the goal to have a centralized lized 0-1 loss of 0.1, the 0-1 loss on synthetic features should be at most 0.05 given perfect GMMs (i.e. FFM 0). Notably, this bound server-side guarantee since Ø-1 can be calculated at the server and E. EM [L.M] is already calculated during learning of GMMs at the clients and can be sent to the server. Using this theorem, the server can calculate the performance of clients without accessing their data. Complete proof of this theorem along with comparison to other bounds can be found in Appendix F
5
Experiments
We examine the knowledge transfer capabilities of FedPFT and DP-FedPFT in the one-shot sel comparing them with state-of-the-aae methods across various data heterogeneity settings and network topologies. Our
experiments support the claim that FedPFT: (1) compares favorably against existing one-shot FL mothods; (2)
succeeds in a variety of extreme client distribution scenarios challenging for FL; and (3) supports decentralizod network topologies.

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: Summary of datasets and foundation models
Dataset
Min size
[|]
Foundation Model
CIFARIO
32
50k
10
ResNet-50
2048
CIFAR100
32
50k
100
ResNet-50
2048
PACS (P)
224
1.3k
7
VIT-B/16
768
PACS (S)
224
3k
7
VIT-B/16
768
Office Home (C)
18
3.4k
65
VIT-B/16
768
Office Home (P)
63
3.5k
65
VIT-B/16
768
Caltech101
200
6k
101
CLIP, ViT-B/32
768
Stanford Cars
240
12k
196
CLIP, VIT-B/32
768
Oxford Pets
108
3.6k
37
CLIP, ViT-B/32
768
Food101
512
75k
101
CLIP, ViT-B/32
768
50 clients CIFAR100
50 clients Caltech101
80
Centralized
100
Centralized
(10)
Gaug (0)10) 10)
(p)ne(1)
70
90
(100
(pphe(1)
CCVR
Accuracy (%)
(1)
Accuracy (%)
60
CCVR
FaaNCM
Ensemisle
FadBE
70
Ensemble
Centralized
DSFL
50
FedFed
Ke
60
FedBE
FedFeed
FeaKT
FodBE
Ensemble
Fedrogl
FedProx
CCVR
40
FedNCM
FedAvg
FedPrex
FedYogl
FedPrex
FedAug
FodKT #DS
FooNCM
FedFed
Fedkug
FedPFT [Ours)
DP-FeuPFT (Ouns)
100
200
300
400
20
40
60
80
100
Communication Cost (MB)
Communication Cost (MB)
Figure 4: FedPFT vs existing one-shot and multi-round methods in Centralized setting with CIFAR100 (left) and Caltech 101 (right) dataset. FedPFT (G) and DP-FedPFT (9) surpass other one-shot FL. methods, and are competitive with sending raw features (Centralized).
5.1 Experimental setting
Datasets and foundation models. We use vision datasets including CIFAR10/100 (Krizhessyy el al. 2014h, PACS (Li et al. 2017), Office Home (Venkateswara al.) 2017), Caltech101 (Li et al.
2022), Stanford Cars Krause et al.) 2013). Oxford Pets (Parkhi et al. 2012), and Food101 (Bossard et al.
2014) and three foundation models including ResNet-50 (He et al. 2016 pre-trained on ImageNet, ViT-B 16 (Dosovitskiy
et al. 2020) pre-trained on ImageNet, and the CLIP (Radford et al. 2021 image encoder our feature extractor f. Table 2 provides a summary of the datasets and the corresponding feature extractors used. We keep the foundation models frozen for all the experiments except in Table where we analyze the effect of fine-tuning.
Implementation and Baselines. We use FedPFT to train the classifier head of foundation models. We provide a discussion on training other layers Appendix A.2 Also, We provide the code for implementing GMMs in FedPFT in Appendix D For transmitting GMM parameters, we use 16-bit encoding. We use Genv (K) and G.ov (K) to denote FedPFT and DP-FedPFT, respectively, where types. We set 1, 5 1/|D*=*), and use K mixture components for all DP-FedPFT experiments since our thcoretical analysis currently supports only this case. We run all the experiments for three seeds and report the mean and standard deviation of the accuracy on the hold-out Lest dataset, along with its communication cost. Further experimental details can be found Appendix G For baselines, we present the results of centralized training with raw, pro-trained features ( (Centralized) as the oracle. We also include the results of an Ensemble comprising locally trained classifier heads from clients, where the class with the highest probability across the models is selected.
represented covariance
7

---

Published in Transactions on Machine Learning Research (04/2025)
Table 3: FedPFTT in three extreme shifts two-client decentralized FL. We format first and oracle results.
Disjoint Label shift
Covariate shift
Task shift
Method
Caltech101 J
Oxford Pets
CIFAR-10
CIFAR-100
PACS (P-+S)
Office (C-P)
Stanford Cars
Food101
Centralized
90.85
0.08
73.97
0.00
89.15
0.17
82.00
0.10
81.88
0.00
88.48
# 0.00
Ensemble
80.18
= 0.30
57.94
0.22
79.59
0.88
71.36
= 0.46
58.33
= 2.01
83.54
# 0.21
Average
77.66
56.82
77.83
69.69
72.65
83.25
KD
74.22
0.42
55.62
0.20
67.69
2:47
72.15
0.75
40.46
0.31
43.67
1. 0.04
Geing (K 10)
86.19
69.97
89.12
80.56
81.74
88.22
Gding (K 20)
86.89
1: 0.02
70.31
0.10
89.00
0.15
80.94
0.2.6
81.75
0.04
88.25
0.08
5.2
Comparing to existing one-shot and multi-round FL methods
In Figure we compare the performance of FedPFT with stale-of-the-art one-shot FL methods in centralized settings with 50 clients for training the classifier head. We conduct tests on both CIFAR100 with the ResNet-50 feature extractor and the Caltech101 dataset with the CLIP VIT feature extractor, where the samples distributed across clients according to Dirichlet (3 = 0.1). The benchmarked multi-round methods include FedAvg (McMahan et al. 2017). FedProx (Li et. al. 2020b). DSFL (Beitollahi et al. 2022). FedFed
Yang et al. 2023). CCVR (Luo et al. 2021] and FedYogi (Reddi et al. 2020 along with single-shot methods such as FedBE (Chen & Chao 2020) FedNCM (Legate et al. 2024), FedKT
Li ct al. 2020a and Ensemble.
We were unsuccessful in running model inversion methods including DENSE
(Zhang et al. 2022 for frozen,
pre-trained models (See Appendix A.4 for further details).
Figure 4 shows that FedPFT and DP-FedPFT beat other one-shot FL methods; and are competitive with sending raw features (Centralized). With more communication budget, multi-round FL methods perform better than existing one-shot methods. Additionally, Figure 4 demonstrates different tradeoffss for varying numbers of mixtures K and covariance types. See Section 6. for more discussion on these tradeoffs.
5.3 Extreme shifts peer-to- peer-to-peer decentralized FL
By design, FedPFT is agnostic to data heterogeneity. To assess this claim, ex- amine decentralized FL settings with two clients: source and destination. These clients exlibit significantly different train- ing distributions. Specifically, we ex- plore three types of extreme shift scenar- ios label shift, covariate shift, and task betwenn clients, where the source can communicate only once to the destina- tion client. We report the performance of the destination's trained classifier head on both clients' test datasets in Table g Ta- ble demonstrates that FedPFT succeeds in various extreme client distribution sce- narios and highlights the limitations of vanilla averaging, ensembling, and KD methods. Below we provide the details of the benchmarks and distribution shifts. Further details of this experiment includ- ing the communication cost can be found
D2
Dg Client
Dg Client
D3 Client
Client
Client 2
90
Accuracy (%)
70
50
Centalized (Oracle)
Local training
GKi = 5)
Client
Client 2
Client 3
Client
Client
Figure (top) Five clients linear topology. Each client updates its received GMM with its local data and sends it to the next client. (bottom) Results of FedPFT with clients linear topology as illus- trated in Figure
in Appendix G.4
8

---

Published in Transactions on Machine Learning Research (04/2025)
1.6
Raw Features
Raw Features
3.0
144
Raw Features
100
Accuracy (%)
10
50
2.5
k=10
Indext
2.0
0.8
92
Diagonal
0.6
1.5
91
Full
0.4
Spherical
90
Raw Features
1.0
0.2
10P
10
107
-1.0
-0.5
0.0
0.5
1.0
1.5
0.0
0.0
0.5
1.0
1.5
2.0
Number parameters (log)
Index
Index
Figure 6: Comparing real and synthetic feature distributions using the Caltech101 dataset. (Left) Classifier accuracy on raw vs. synthetic features from various GMMs. (Middle) 2-dimensional and (Right) 1-dimensional distribution of random indexes of real features vs GMMs' counterparts with different covariance types and number of mixtures.
5.4 Linear topology
In this setup, we demonstrate the propagation and accumulation of knowledge using FedPFT in decentralized FL with five clients in linear topology where each client has access to 100 i.i.d. samples of the CIFARIO dataset, illustrated in Figure 5 (top). Using GMMs, transfer knowledge of client 1 to client 5 in four communication rounds and report the performance of each client's classifier head trained on its received GMM on the entire 5 client dataset in Figure 5 (bottom). We also compare FedPFT with centralized training and local training, where we train a classifier head for each client's dataset. Figure (bottom) shows that as GMMs pass through clients, they can accumulate and propagate the knowledge and achieve performance close to 1.8% of centralized training.
6 Analysis
In this section, we provide a comprehensive analysis of FedPFT, examining its accuracy, communication cost, and privacy characteristics. In particular, we: (1) examine the tradeoff between accuracy and communication cost for different families of GMMs; (2) prove theoretical guarantees on local client accuracy; (3) estimate communication costs; and (4) analyze privacy leakage in FedPFT, compared to sending raw features.
6.1 How well do GMMs model feature distributions?
FedPFT relies on learning the distribution of features of each class using GMMs. We aim to assess how effectively GMMs can estimate cons--onniiional features. We measure the accuracy gap between two classifier heads: one trained on real features and another on synthotic features generated using GMMs. Monitoring the accuracy gap helps us evaluate the discriminative power of synthetic features. We examine the effect of the number of mixtures K and the type of covariance matrix. For this experiment, pictured in Figure D use CLIP ViT-B/32 features the Caltech101 dataset. For various families of GMMs, we plot accuracy and the total number of statistical parameters.
Figure (left) shows that 10-50 Gaussians are sufficient to represent raw extracted features with less than a 1% drop in accuracy. Notably, GMMs with a spherical covariance matrix exhibit better tradeoffs between communccatio and accuracy compared to full/ /idiggaaa covariance matrices. Additionally, we illustrate the 1-dimensional (Figure 6 right) and 2-dimensional (Figure T middle) density of random indexes of raw features and compare them with GMMs.
6.2 Communication cost of FedPFT
Here, we estimate the communication cost of FedPFT and compare to sending the classifier head or sending the raw features. Denoting by d the feature dimension, K the number of components, and C the number of
9

---

Published in Transactions on Machine Learning Research (04/2025)
Attack on Raw Features
Attack on FedPFT
Attack on DP-FedPFT
Test Image
Example 1
Reconstruction source
Similarity measure
Example 2
PSNR f
LPIPS 4
SSIM
Raw features
16.5
.257
.535
FedPFT
14.7
.423
.508
DP-FedPIT
14.6
.511
.483
Example 3
Train set
14.7
.389
.500
Figure 7: (left) Results of reconstruction attacks on feature-sharing schemes using three random test images from CIFAR-10. Attackes can reconstruct raw features (middle-left) to generate images resembling real data (left). However, the same reconstruction on FedPFT (middle-right) and DP-FedPFT (right) does not resemble the real image. For more reconstruction ezamples see Figures 12 and 13 the Appendiz. (right) Quantitative metrics set-level reconstruction. We report image similarity the top 1% (n 90) of test set images, by their SSIM to member of the reconstruction set. As baseline, we report the results treating the train set as a reconstruction set, For threat model, Appendix H
further elaboration
classes, we calculate the communication cost of FedPFT for different covariance families:
d² d 1)KC~
Cost(Grun) (2d
(10)
2
Cost(GSiag) (2d 1)KC O(2CCK),
(11)
Cost(Gspher) (d 2)KC O(dCK).
(12)
12 indicate that the communication cost of FedPFT is independent of the number of samples of each client 74: Therefore, FedPFT can scale better compared to sending raw data or raw features when the number of samples large. More specifically, when n, > 2dCK, it is more communication efficient to send Gring (K) than send the features. This also shown in Figure t (left). Simílarly, equation (12) shows that the communication cost of Gspher (K- 1) is equal to the communication cost of sending the classifier head, which is (Cd + C). Therefore, GMMs can have the same communication cost FedAvg. Further, FedPFT supports heterogeneons communication resources, as each client can utilize a different K.
Equations 10
6.3 Evaluating against reconstruction attacks
We conduct reconstruction attacks on the various feature-sharing schemes described in this paper using the CIFARIO dataset. Figure 7 (left) verifies the vulnerability of raw feature sharing an attacker with access to in-distribution data (i.e. CIFAR-10 train set) can obtain high-fidelity reconstructions of the private data of clients (i.e. CIFAR-10 test set). This necessitates sharing schemes beyond sending raw features.
The attack we present involves training U-Net-based conditional denoising diffusion model on (extracted feature, image) pairs and then performing inference on received feature embeddings. We apply the same attack features sampled via FedPFT and DP-FedPFT. The reconstructions set-level, and we present the closest image among the entire reconstruction set by SSIM. The resulting reconstructions do not resemble the original image (Figure (left)). Quantitative results reporting image similarity metrics can be found in
Table 7 (right).
For full experimental details, further quantitative results, and samples, please see Appendix H We also include full description of our threat model and results on different backbones (e.g. our attacks are stronger MAFs), which could not be included in the main body due to space limitations.
10

---

Published in Transactions on Machine Learning Research (04/2025)
6.4 Effect of data heterogeneity
90
We analyze the elfect of data heterogeneity on the performance of FedPFT and compare it to other one-shot baselines in Figure S We utilize CLIP,
80
Accuracy (%)
ViT-B/32 as our feature extractor and distribute the Caltech101 dataset across 50 clients according to Dirichlet with different values of B. Figure C that FedPFT invariant to data heterogeneily and outperforms other baselines by up to 32%. Further, it shows the challenge of model merging in one- shot setting, since baselines such as averaging and FedBE do not perform as well as an ensemble with the decrease in data heterogeneity.
60
Centralized
40
FedProx
Fed
Ensemble
Fed (Ours)
Awerage
30
0.02
0.1
0.5
Data Heterogeneity (B)
Figure Effect of data heterogeneity (8) using the Cal-
tech101 dataset with 50 clients. We use K=1 for FedPFT.
7
Conclusion
We introduce FedPFT, a one-shot federated learning (FL) method leveraging foundation models for better accuracy and communication efficiency. FedPFT utilizes per-client parametric models of features extracted from foundation models for data-free knowledge transfer. Our experiments show that FedPFT improves the communication-acccaaayy frontier in a wide range of data-hetteogeeniity settings. Moreover, we showed that FedPFT is amenable to formal privacy guarantees via differential privacy, exhibiting good privacy-accuracy tradeoffs. Our theoretical analysis demonstrates that FedPFT has server-side guarantees on the local accuracy of clients. Additionally, we conduct reconstruction attacks on feature-sharing schemes and demonstrate the privacy risks of sending real features.
References
Mahdi Beitollahi and Ning Lu. Federated learning over wireless networks: Challenges and solutions. 1EEE Internet of Things Journal, 2023.
Mahdi Beitollahi, Mingrui Liu, and Ning Lu. Dsfl: Dynamic sparsification for foderated lcarning. In 2022 5th International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pp- 1-6. IEEE, 2022.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
Rishi Bommasani, et al., and Percy Liang. On the opportunitics and risks foundation models. ArXio, 2021.
URL https://crfm. ftttp:://crfm.stanforrt stanford......o edu/ http://ssstssreepr.s.oos .dff
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In Computer Vision Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446 461. Springer, 2014.
Nader Bonacida and Prasant Mohapatra. Vulnerabilities in foderated learning. IEEE Access, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877 1901, 2020.
Nicholas Carlini, Steve Chien, Milad NasT, Shuang Song, Andreas Terzis, and Florian Tramer, Membership inference attacks from first principles. In 49rd IEEE Symposium on Security and Privacy (SBP'22), 2022.
Hong- You Chen and Wei-Lum Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. Xiv preprint ar o0x::2009.0111 019974 2020.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Hong- You Chen, Cheng-Hao Tu, Ziwei Li, Han- Wei Shen, and Wei-Lum Chao. On the importance and applicability of pre-training for federated learning. aiviv preprint or arXim:2206.114 11488, 2022.
Imre Csiszár and János Körner. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press, 2011.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological)) 39(1):1 22, 1977.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition al scale. Xiv preprinl arxiv:2010. 1...1111. 11929, 2020.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to sensitivity in private data analysis. In TCC*06: Proceedings of the Third conference on Theory of Cryplography, 2006.
Cynthia Dwork, Aaron Roth, al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science, 9(3 4):211 407, 2014.
European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council. URL
Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated learning. arXiv preprint arXin:1902.11115, 2019.
Tao Guo, Song Guo, Junxiao Wang, Xucyang Tang, and Wenchao Xu. Promptfl: Let foderated participants cooperatively learn prompts instead of models-federated learning in age of foundation model. IEEE Transactions Mobile Computing, 2023.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable
GAN controls. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sobhan Hemati, Mahdi Beitollahi, Amir Hossein Estiri, Bassel Al Omari, Xi Chen, and Guojun Zhang, Cross domain generative augmentation: Domain generalization with latent diffusion models. arXio preprint arXiv:2312.05399, 2023.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXie preprinl arXiv:1503.0955 2015.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends@ in machine learning, 14(1 2):1 210, 2021.
Anirudh Kasturi and Chittaranjan Hota. Osgan: One-shot distributed lcarning using generative adversarial networks. The Journal of Supercomputing, pp. 1-21, 2023.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05499, 2016.
Jonathan Krause, Michael Stark, Jin. Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 10 dataset. online: http://www.. cs. toronto. edu/kriz/cifar. himal 55(5), 2014.
12

---

Published in Transactions on Machine Learning Research (04/2025)
Gwen Legate, Nicolas Bernier, Lucas Page-Caccia, Edouard Oyallon, and Eugene Belilovsky. Guiding the last layer in federated learning with pre-trained models. Advonces in Neural Information Processing Systems, 36, 2024.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542 5550, 2017.
Fei-Fei Li, Marco Andrecto, Marc' Aurelio Ranzato, and Pictro Perona. Caltech 101, Apr 2022.
Qinbin Li, Bingsheng Hc, and Dawn Song. Practical one-shot federated learning for cross-silo setting. arXiv preprint trXiv:2010.011 01017, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogencous networks. Proceedings of Machine learning and systems, 2:429-450, 2020b.
Mi Luo, Fei Chen, Dapeng Hu, Yilan Zhang, Jian Liang, and Jiashi Feng. No lear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-5984, 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282. PMLR, 2017.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word rep- resentations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Compulational Länguistics: Human Language Technologies, pp. 146-751, Atlanta, Georgia, June 2013. Association for Computational Linguistics.
Nguycn, J Wang, K Malik, M Sanjabi, and M Rabbat. Where to begin? on the impact of pre-training and initialization in federated learning. arxiv 2022. arxiv preprinl ur Xiw:2210008000
Mijung Park, James Foulds, Kamalika Choudhary, and Max Welling. DP-EM: Differentially Private Expectation Maximization. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Stalistics, volume 54 of Proceedings of Machine Learwing Research, pp. 896-904. PMLR, 20-22 Apr 2017. URL https://prooeeedngs mlr paress/v54/pate wwt.///kk7...ttttt..
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pp. 3498 3505. IEEE, 2012.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint orXiv:2003.00290 2020.
David W Scott. Multivariote density estimation: theory, practice, and visualization. John Wiley Sons, 2015.
Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Foderated learning from pre- trained models: contrastive learning approach. Advances neural information processing systems, 35: 10332-19344 2022.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xinmei Tian, Tongliang Liu, Bo Han, and Xiaowen Chu. Fedimpro: Measuring and improving client update in federated learning. Xi preprint or 0rxi:2402..07011, 2024.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pollern recognilion, pp. 5018 2018-5027, 2017.
Shuai Wang, Yexuan Fu, Xiang Li, Yunshi Lan, Ming Gao, al. Dfrd: Data-free robustness distillation for heterogeneous federated learning. Adcances in Neural Information Processing Systems, 36, 2024.
Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, and Bo Han. Fedfed: Feature distillation against data heterogeneity in federated learning. Advances in Neural Information Processing Systems, 36:60397-60428, 2023.
Guojun Zhang, Mahdi Beitollahi, Alex Bie, and Xi Chen. Understanding the role of layer normalization in label-skewed federated learning. Transactions on Machine Learning Research, 2023.
Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35: 11414-21420 2022.
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Commuicattion-cffciettt and privacy-preserving prompt tuning in federated learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5. IEEE, 2023.
Zhi-Hua Zhou, Onvn-environment machine learning. National Science Review, n(w):1 2022.
14

---

Published in Transactions on Machine Learning Research (04/2025)
A Discussions and Limitations
A.1 Why does FedPFT perform well with limited data?
The strong performance of FedPFT stems from its use of pro-trained foundational models to extract task- agnostic features. Additionally, GMMs require only a few parameters (centroids and variances) from these features, allowing them to perform well even with limited client. data. Furthermore, since FedPFT recreates features at the server, it remains agnostic to data heterogeneity. Increasing GMM complexity from spherical to diagonal and full covariance-cnalle better feature representation at the cost of higher communication overhead.
A.2 Can train any layer of the model using FedPFT?
We have explored training layers other than the classifier head using FedPFT. When training any layer of a network using FedPFT, we need to extract the outputs of the previous layer and learn a parametric model of those outputs. Two issues arise for layers that are not the classifier head: 1) the size of the output of hidden layers is very large and 2) simple parametric models cannot describe the output of hidden layers. For instance, consider using FedPFT for training the second to last layer (12th layer) of ViT-B/16 using FedPFT. In this case, we need learn the outputs of the 11th layer which is a tensor of size (soquence=197, hidden-dim=76s). In this case d = 197 768 151, 257 is very large for both computation and transmission. More importantly, it is not clear how well GMMs model intermediate layer activations. We attribute the success of FedPFT to the observation that last layer outputs are well modeled by GMMs (See Figure G). We believe the reason is that during the pre-training, last-layer activations of different classes were forced to be well-separated for classification using a linear probe. However, pre-last layer activations have positional information that does not aggregate well for calculating parametric statistics. We have tried modeling pre-last layers using GMMs but we were unsuccessful.
A.3 Does sending label counts along with the GMMs to the server violate privacy?
Indeed, FedPFT sends label counts along with the GMMs to the server, which may not be advisable from a privacy perspective. However, this is not an inherent limitation of FedPFT. For example, one adaptation that be made to address this is to employ secure shufflers (as used in private FL protocols,
(Kairouz et al. 2021 page 45). These are class of cryptographic primitives that would shuflle the (GMM, label count) tuples provided by each client before the server receives them, and therefore, the server does not know the classes present on specific client.
A.4 Comparing FedPFT with model inversion methods
Model inversion methods like DENSE (Zhang et al. 2022) and DFRD (Wang et al. 2024) attempt to recreate the client's data at the server to distill the ensemble of the client's models to a global model using the generated synthetic data. Therefore, the global model would adhere the performance of the ensemble the client's models (Zhang et al., 2022). Based on the results of the ensemble presented in Sections 5 and 5.3 we argue that the performance of the ensemble model is also very limited due the extreme data heterogeneity of clients. Further, these methods rely on training and transmitting the entire model and it is unclear how model inversion methods like DENSE can be extended to frozen, pre-trained foundational models as feature extractors. We were unsuccessful in our attempts to utilize DENSE for a pre-trained, frozen ResNet-50 to achieve competitive results. We believe one reason for our poor performance was the lack of access of DENSE to batch norm statistics of clients. Finally, inversion methods like DENSE and DFRD do not provide any privacy or accuracy guarantees.
A.5 Using fine-tuned features vs zero-shot features
In section 5 we are reporting the centralized bascline for zero-shot features to represent the upper bound of what any method can achieve using frozen, pre-trained feature extractor. In Table 0 we report the results when you can fine-tune feature extractor. Table demonstrates that fine-tuning hurts FedPFT performance,
15

---

Published in Transactions on Machine Learning Research (04/2025)
and we attribute this to the fact that this introduces mismatch between the client features computed with client feature extractors and the server feature extractor used at test time. For our baselines (with the exception being KD), fine-tuning and aggregating fine-tuned feature extractors improves performance. However, FedPFT with pre-trained feature extractors still outperforms them. Finally, an important caveat fine-tuning and sharing models the additional compulational resources (FLOPS and memory) and communication costs, which become especially relevant when dealing with large foundation models. We will include this table along with the discussion in the final version.
A.6 FedPFT for multi-shot FL
In this paper, our objective is to 1) bring and utilize large pre-trained foundation models FL while at the same time 2) improve the communication cost by focusing only on one-shot FT.. In the one-shot setting, FedPFT optimizes parametric distributions to capture feature representations. On the other hand, in the multi-round setting, it is not clear regarding the benefits: any approach that combines with feature fine-tuning, i.e., CCVR, will be bottlenecked by the communication of model weights of large foundation models. Further, the removal of the multi-round FI parts lets us prove formal accuracy and privacy guarantees.
A.7 Comparing FedPFT with CCVR using pre-trained model
CCVR, when using a pre-trained model, does not perform similarly to FedPFT (K=1). This is because, in FedPFT, we train the classifier head from *scratch* using Gaussian mixtures, whereas in CCVR, clients send trained classifier head to the server, which then post-calibrates (*fine-tuns*)) the classifier at the end of each round to mitigate bias and improve accuracy in multi-round FL. In Figure 4, we compare CCVR with a frozen pre-trained model against FedPFT, showing that FedPFT outperforms CCVR while also reducing communication costs by avoiding classifier head transmission. We believe that the bias in CCVR's local classifier heads prevents from achieving competitive performance compared FedPFT.
A.8 Other limitations
One of the main limitations of our work is that we have not explored non-vision tasks. We will leave non-vision tasks for future work. Further, we leave other attacks including membership inforence attacks for future work.
Societal Impact
This work promotes privacy by bringing foundation models to FL and using differential privacy. Unlike traditional approaches that require clients to send their raw data to server training, our method utilizes FL and differential privacy to allow users to send features instead. We show that clients do need sacrifice privacy achieve competitive performance. We further evaluate our method against privacy attacks and highlight the capability of private, featuro-sharing methods for knowledge transfer using foundation models.
C Algorithm
In this section, we provide the complete algorithm for FedPFT, Algorithm I describes the algorithm for FedPFT.
16

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm FedPFT for centralized, one-shot FL.
1: Input: Client datasets Di...., D1, pre-trained feature extractor f.
2: Parameters: Number of clients I, number of classes C, number of mixtures K, covariance type cov.
Output: Model w w==h
// Client side:
for each client E {1, ...., / do
6: for each class E (1,..., C) do
7: Let Ft [f(x): (x, y) E D,,y=C
8: Run the EM algorithm on Fie to learn a GMM g e (eor(K)
se ))) to the server. 9: Send gf parameters {(T) p
10: end for
11: end for
12: // Server side:
13: for each received ((*b) be 14 ))) set do
14: Sample synthetic features Fir gh Σ Ek=1 the NN o of size |F
15: end for
16: Let D -= -'
e pr
17: Train a classifier head h on F, minimizing ((v.))D(((h; v, v)] where e is the cross-entropy loss.
18: return model w == o.
Code
In this section, we provide code for the main component of FedPFT which is extracting the GMMs for class-conditional features. We consider the case k 1 and refer readers to the DP-EM (Park et al.) 2017 method for the general case. Also, in this section, we assume each client can access only one class for simplicity and without losing generality.
from sklearn. mixture import GausstanMixture as GMM import numpy as
def create_gam (features: array, labels: up array) >> Dict:
Creates diagonal GMM with 10 mixtures for each class in features and sanple from it
synthetic dataset
Args:
13
features training dataset labels (np. array): anp.ayy)): all associated labels features returns "syathetic_datteett
features (np. :nrrarry):
14
15
18
17
18
prathetic_dataace 0
19
for label in list (labels)):
20
aaditional_featuree features [labels== label]
21
# Create GMM for 'label' CMM(
components=10.
covariance_type- 'dag',
27
gmm fit
Sample GMK 'label' server
30
gmn feature. gmm www..aappeccccaiiioolll__eeeeeea shape [0])
17

---

Published in Transactions on Machine Learning Research (04/2025)
31.
32
Add the syath dict syathetic,datasee [label] gmm_ centresture
33
34
36
return ayathetic_datases
E
DP-FedPFT
This section provides detailed definitions and proofs for Theorem 4.1
Definition E.1. (Differential Privacy, (Dwork el al. 2006)). ). A randomized algorithm M:U + O is (e, differentially private if for every pair of neighboring datasets D, D' E u for all S " O, we have
Pr(M(D) E 8) S Pr(M(D") E +
(13)
Lemma E.2. (Gaussian Mechanism, (Doork et al. 2000 2014))
Let and let g D - IRd be
function with fg-sensilivity Ag. Then the Gaussian mechanism
( Ag 22n(2/ M(D) = (DD) (0.
2
Laxd
(14)
e
satisfies (c, ddiffferrreettia privacy.
Proof. (Theorem 4.1p We wish to apply Lemma E.2 with g mapping feature dataset to its Gaussian mixture approximation obtained via EM-method. The 20144) ensures that the projection onto the positive semi-definite symmetric matrices preserves differential privacy. For =, 1, denote by úi, E the mean and covariance matrix this Gaussian approximation; this instance they coincide with the mean and covariance matrix of the dataset. We may use the Ez-sensitivity of the function (j. Σ) apply the aforementioned Lemma.
property (Proposition 2.1 of (Dwork et al.)
To begin with, for any triplet of independent random variables (,) with X, Y with values in d matrices with real entries and in {0, 1} with IP(e 1) p, denote X == eX + (1 e)Y the mixture of X and Y:
B(X*) = (X) + () E(()); Cov (X. = Cov (X,X) (1 p) YY + p (B E(Y)) E(Y))*
(15)
(16)
If follows the uniform distribution on a dataset D' with n element, Y deterministic with value at some In Rd and p = 11-12 1/n then X follows the uniform distribution on the datasct D" D' {xx). Assuming D' and D" are in the ball of radius have:
|[(X') - E(X)22 = (1 (1-))(( - E(X))l2 S 2 |oov (X (x) Cov x), = (1 (1-p))/[Cov(e (Xx]]]((XX
+ +p²(1 - - (E(X) - (EY)) E(Y)) -(Y)) B(Y))²)
2p(1 - -p) 2r (X, - -X)
- -B(V))
= (1 - 11 Cov (X, x) + p² (IX)) - E(Y)
+pp(1 (E (ov)))((((X X) - E(Y))
1
(n - 1)2
8(n
< n
シ11 +
p( P(S)
n
n
1
(4 16 8 xx
n?
18

---

|Cov (X x9) Cov (X, 6 12. Published in Transactions on Machine Learning Research (04/2025)
Taking the square root of both sides we have:
where -|= is the Frobenius norm, return the largest eigenvalue. We used the bounds p(Σ) < | and || S 2; the latter may be obtained in similar fashion by developing Cov(X, X) using X = -==1 x; where each X, is deterministic at some and is uniform on the set of binary vectors of length satisfying E; =1
Finally; the fz-sensitivity of (fz. E) is (2) (0) yields the result.
2 n
Inserting the fz-sensitivity in equation 14
Remark E.3. Note that in (Park et al. 2017 they derive lz-sensitivity bound for Σ of 2/n instead of our 6/n. The reason is that one may replace (á, Σ) by (p, + faf²)) to reduce the éz-sensitivity of the covariance part. This leads to the improved total ez-sonsitivity of 2v2
On the server side, the Gaussian mechanism would return (f + Ap, fuft" + +AE). The server would thus roconstruct Σ by computing
server = Σ jupt + AΣ + (fr + Ap) + Ap)
= + puPp" App" ApAp" AE.
Therefore, for a given coefficient (i,j) of the reconstrueted covariance matrix Eserver, the error term is N4 A1j + Au An,An + AEig. Assuming a Gaussian noise of standard deviation 2v2 with C (21m(2/0) for both AE and Au the standard deviation the reconstruction error then
reconstruction = (0) + 13)02 2 + at + SoV2+02 2v20 2220V2+C2/m
Compared the reconstruction error of the Gaussian mechanism we used in our experiments recconstruction 2/10 C, the more sophisticated methods we described above may allow better reconstruction error without sacrificing the differential privacy for big enough datasets, ie if n 2
221m(2/8)
v/e
Another consequence is that the noises on different elements of the covariance matrix are not independent.
Remark E.4. The assumption of normalized features, i.e., [x)]2) S 1, in Theorem 4.1 does not limit the performance of networks with soft-max loss function since they both have the same exprossive power shown in Proposition 3.A in Zhang et al. 2023
F
Control over 0-1 loss
Since the projection from true feature distributions to mixture of Gaussian is lossy, one may control the consequence on the accuracy of the classifier: "Given a classifier X + f1,.... CC} trained on synthetic dataset, what guarantees do we have on the accuracy of used classify the true dataset?'.
We adress this question by proving a theoretical bound that my be rewritten using the loss of the local client training and the accuracy of the server model.
F.1 A Theoretical bound
The setting may formalized as follows:
finite set of classes C (1, ..... C}, feature space X' and approximate feature space ))
19

---

Published in Transactions on Machine Learning Research (04/2025)
true feature/label distribution C x &' and an approximate feature/label distribution B on
C x )
classifier ha X + C. trained and classifier hg : - C trained on B.
an feature approximator mapping x: -
We denote by ac) the accuracy of the predictor h evaluated using the distribution We prove bound adapted to the purpose of linking the EM loss to the accuracy.
Theorom F.l. Let a, be probability distributions on X x C and y x C with same marginal on C. Let h y - C be any prediclor and let u X + be any measurable map. We have
Acc(h oc 2 E. [Aco(h; (Aco(h; Blc) (llc)((+(a)c),
Proof. Consider any coupling IP of and 3 over ie distribution C x X y whose marginals on C X
and C y are and 8 respectively"
Given some
Aco(h o (alc)({z hor(z) =
(17)
(Plc)(((z,))) ho = =c]
(18)
(Plc)(((2,g) hol(z) = and ((x) y})
(19)
(P)c)(((z))) : h(y) cc) x (Plc,hly) = c)({(z))) d(x) =y))
(20)
= (Blc)((v : Mw) c)) x xPPl, h(y) c)(((()))) : (x) = =
(21)
Aco(h:mlc)
Since neither the left hand side of the first line nor the first term in the right hand side of the last line depend
on the coupling P we chose, we may choose it to maximize (Plc, h(y) = c)(((r v) ((x) - v)) 1 1 E(1zyr)
with Y drawn from (Ble, hty) = c) and Z drawn from (a)c). We recognize a characterization the total variation as the minimal value of B(1z,y) over couplings of Z and Y. Then:
(Plc, h(v) = ()((()))) (x) v)) 2 1 divrv(##aal
(Blc, = c))
(22)
2 1 - divrv (Blc) - ((3lc) (31c, -))
(23)
- divry(+#(acc): (8]c) - (1 - Acc(hly)) Blc))
(24)
Acc(h(y): Blc) divry (Blc)).
(25)
We use triangular inequality of total variation to get the second line. The third line is obtained by applying the general property that for any probability distribution Q and any event with Q(F) > we have
divrv (Q F)) = Q(F).
We thus have
Acc(h aaaec 2 Aco(h; Blc) x (Acc(h(y): Blc) divrv (alle)) (8)c)
The result follows by taking expectation distributed along 7)
F.2
Application to FedPFT
We now switch back to the notations used in the preliminaries. Using Pinsker inequality (Csiszár & Körner 2011 we obtain the following bound for the accuracy of the server model
Acc(h, F+) 2 E. Aco(h, Frto AAco(t, Frio) v (divk((Fheeleen
(26)
We define the conditioning of a probability distribution PP by an event E having P(E) > is the distribution 1gF still distribution the underlying measurable space.
so that it F(E]
20

---

Published in Transactions on Machine Learning Research (04/2025)
Since the EM algorithm maximizes the log likelyhood of the Gaussian mixture given dataset samples, we pi.c
may insert div ( (tte) = Hie LEM to obtain
Acc(h, F*) 2 Be Acc(h, (Acc(h, Frie) V22 HHee
EM
(27)
where 7²0 the self-entropy of the distribution Fic of features with label c in client and fEM is the oigliikeyhood of the Gaussian mixture Fite trained using the EM algorithm. The last equation yields incquation g by replacing accuracies by 0-1 losses. Beware that the feature distribution is priori discrete, in order to evaluate the entropy we need to dequantize the dataset, otherwise, H = +oo and the bound is useless.
F.3 Comparison to earlier bounds
(Ben-David Commonly known bounds on the accuracy of h may be found in (Ben- David et al. 2010). but these bounds are unfortunatcly not dircctly useful in our sctting. Indoed, they depend on an estimation on how much server/client class predictors differ.
More precisely, in the limit of perfect accuracy of the server class predictor, ie Vi, e, Acc(h, Fite) ~ 1, our bound yields
Aco(h, F²) 2 Ee 1
Hic (EM)
(28)
In this limit, the bound deduced from that of Con-David et al. 2010 contains an additional intractable negative term on the right hand side:
V 1
Acc(h, F*) 2 Ee 1
Hic -EE)) E. min Exn 11(z) (4) ;E Fr. hh()) (e)) (29)
where Welient is a hypothetical class predictor perfectly fine-tuned by the client with its own data I(D.). However, our bound may be less sharp in general as the positive term is in Acc(h, Fieyy. This leads to a quick degradation of the theoretical accuracy guarantee as the measured server accuracy drops. In general, we should have error on the server side at least twice lesser than the target error on the client side order to hope for achievable EM-loss target on client side.
21

---

Published in Transactions on Machine Learning Research (04/2025)
G Experiments
In this section, we provide the full details of all of our experiments and datasets.
G.1 Implementation details
We use the Pytorch library for the implementation of our methods. We cluster of 4 NVIDIA v100 GPUs for our experiments. All of our experiments can be run single v100 GPU.
Datascts. We provide the complete details of the datasct we used in this paper in Table
Learning rates and optimizers. We use Adam with a learning rate of le for training the classifier head for FedPFT, Ensemble, FedFT, and centralized training.
Knowledge Distillation. For the implementation of KD methods, we use Adam for training the local classifier heads with 1e-4 learming ratc. We locally train the models for 100 epochs and keep the best model the local test. dataset. Then, we distill from the source to the destination model in 50 epochs with the same optimizer and classifier head. We test three temperature values {1,5, 10) and report the best results
FedAvg. We Lest three local training epochs {50, 100, 200} and three learning rates (5e le 1e=2) and report the best result.
Yogi We set the server learning rate to 0.01, 31, 82, and r to 0.9, 0.99, and 0.001 respectively. We also initialize vector i to le 1e-6, For the rest of the hyperparameters, inchuding the clients' learning rates, we use the same hyperparamcters as FodAvg.
FedProx. We add regularizer with weight of 0.01 to the loss function of each client during local training to penalize divergence. We use the same hyperparameters as FedAvg for the rest of the hyperparameters.
DSFL. We set the top-K sparsification (K in the DSFL paper) half of the number of parameters of the classifier head.
FedBE. We use Adam with a learning rate of 1 for training the classifier head and we sample 15 models from the posterior distribution classifier heads.
CCVR For CCVR, we use Mc 100 as stated in (Luo et al. 2021 We also utilize a diagonal covariance matrix make the algorithm competitive terms of communication cost. The rest of the method similar to FedAvg.
Table Summary of datasets
Dataset
Image size
Train
# Testing
# Classes
Feature extractor
CIFAR1O
(32, 32)
50,000
10,000
10
ResNet-50
CIFAR100
(32, 32)
50,000
10,000
100
ResNet-50
PACS (P)
(224, 224)
1,336
334
Base ViT, 16
PACS (S)
(224, 224)
3144
785
7
Base ViT, 16
Office Home (C)
min:(18, 18)
3492
873
65
Base VIT, 16
Office Home (P)
min:(75, 63)
3551
888
65
Base ViT, 16
101
min:(200, 200)
6084
3060
101
CL.IP, ViT-B/32
Stanford Cars
min:(360, 240)
12948
3237
196
CLIP, ViT-B/32
Oxford Pets
min: 114)
3680
3669
37
CLIP, ViT-B/32
Food101
max:(512, 512)
75750
25250
101
CLIP, VIT-B/32
22

---

Published in Transactions on Machine Learning Research (04/2025)
G.2 Comparing to existing one-shot and multi-shot FL methods
In this section, we provide the details for the experiment in Section 5.2 Table 5 summarizzs one-shot methods results in Table format for Figure 4 Further, Figure I0 and shows the data partitions based on Dirichlet shift with 3 = 0.1 with 50 clients for Caltech101 and CIFAR100 Datascts, respectively. The magnitude of data samples for each class label in each client is represented by the size of the red circle. This figure shows the non-iidness of data, distribution among clients.
Table 5: one-shot methods results Table format for Figure
CIFARI00 (3 = 0.1)
Caltech101 (8 = 0.1)
Methods
Accuracy
Comm
Accuracy
Comm.
Centralized
73.50 +
97 MB
95.36 + 0.08
3.6 MB
Ensemble
53.97 + 0.49
19 MB
72.86 + 2.00
4.9 MB
AVG
43.51 + 0.26
19 MB
56.90 + 0.07
4.9 MB
FedKT
51.80 +
19 MB
46.64 + 0.3a
4.9 MB
FedBE
56.32 +:
19 MB
57.67 + 0.27
4.9 MB
Eapher (k=1)
56.96 + 0.28
22 MB
86.41 + 0.18
3 MB
Gdiing (0ing(k=10 (k-10) 10)
69.91 +: 0.11
0.3 GB
94.67 + 0.03
23 MB
Gdiag (k=50)
72.05 +
0.9 GB
94.74 + 0.18
33 MB
Oding (k=100)
72.29 +
1.2 GB
94.83 +
36 MB
Japher (k=1)
60.16 + 0.04
22 MB
93.46 + 0.18
3 MB
Gapher (k (k= 10)
69.74 +: 0.16
0.2 GB
94.59
+ 0.13
12 MB
Sspher (k=50)
71.88 + 0.03
0.5 GB
94.71 + 0.16
16 MB
G.3 Effect of fine-tuning feature extractor
In this section, we investigate the effect of fine-tuning the feature extractor. More specifically, in Table
which is an extension of label shift setup in Table 3 we also fine-tune the feature extractor as well as the classifier head. From Table G We see that fine-tume min-tuning hurts FedPFT performance, and we attribute
Table Knowledge transfer results for label-shift
CIFAR-10
CIFAR-100
Methods
Accuracy
Comm.
Accuracy
Comm.
Centralized (fine-tuned)
97.33 +
71.6 MB
85.18 + 0.08
71.6 MB
Centralized (frozen)
90.85 = 0.03
97.6 MB
73.97
- 0.08
97.6 MB
Ensemble (fine-tuned)
85.23 + 0.01
89.6 MB
57.94
0.22
89.7 MB
Ensemble (frozen)
80.18 + 0.30
80.0 KB
74.78
0.57
0.78 MB
Average (fine-tuned)
83.18 + 2
89.6 MB
68.63 + 1.06
89.7 MB
Average (frozen)
77.66 +-2 80.0 KB
56.82 = 0.22
0.78 MB
KD (fine-tuned)
58.79 +
89.6 MB
44.38 +
89.7 MB
KD (frozen)
74.22 +
80.0 KB
55.62 + 0.20
0.78 MB
FedPFT (fin-tuned, K=20)
76.23 + 0.50
0.8 MB
60.46 + 0.08
7.8 MB
FedPFT (frozen, K-20)
86.89 + 0.02
0.8 MB
70.31 + 0.10
7.8 MB
this to the fact that this introduces a mismatch between the client features computed with client feature extractors and the server feature extractor used at test time. For our baselines (with the exception being KD), fine-tuning and aggregating feature extractors improves performance. However, FedPFT with pre-trained feature extractors still outperforms them.
23

---

Published in Transactions on Machine Learning Research (04/2025)
50 Clients CIFAR100 Dirichlet(B-O... 0.1)
48
44
34
20
5 7
15
17
19
25
27
29
39
45 47 49
55 57
59
61
63
67
69
71
73
97 99 77
79
81
83
87 89 91 93
Figure 9: Data partitions based on Dirichlet shift with / = 0.1 with 50 clients for CIFAR100 datasets. The size of the red circle represents the magnitude of data samples for each class label in each client
Clients Caltech 101 Dirichlet((=0.1)
46
42
38
32
28
15 23
51533555 Label
59
69
79
05
07
99101
Figure 10:
Data partitions based Dirichlet shift with B 0.1 with 50 clients for Caltech101 datasets, The magnitude of data samples for each class label each client represented by the size the red circle
G.4 Details of experiments for peer-to-peer experiments
In this section, we provide the full details of experiments in Section 5.3 and Table 3 Table 7 is the complete version of Table 3 where we also include the communication cost each method.
For baselines in Table 3 we compare to KD, where: (1) each client locally trains classifier head; (2) the source client sends its local classifier head to the destination client; and (3) the destination client distills
24

---

Published in Transactions on Machine Learning Research (04/2025)
Table FedPFT in three extreme shifts two-client decentralized FL.
Dijooint
Covariate shift
shift
CIFAR-10
CIFAR-100
PACS (P :8)
Office Home
(C
sP?
Birds
Cars
Pots
Food
Methods
Accuracy
Comu.
Accuracy
Corum.
Accuracy
Commn
Accuracy
Commn.
Accuracy
Comun
Accuracy
Commn.
Centralized
90.85 ..
97
73.97 w..
0.4 MB
89.15 131
2.5 MB
12:00 t.n
6.3 MB
$1.88 x.4
6.7 MB
88.48 was
MB
Besemble
80.18
80 K35
67.94
0.7 MH
79.50
10.5 KH
71.36
96 KH
58.33
0.3 M1H
83.54
0.2 MB
Average
2.104
56.82 2030
0.7
77.83 1m
69:00 2..
96
72.65 aam
83.25 =0m
KD
74.22
80 KB
55,62
0.7 MI
67,09
10.5 KD
72.15
96 KB
40.48 4.3
0.3 MD
43.67
0.2 MB
RanglK=100
2018
80.56
Gang (K- 200
86.89
0.00
0.8 Mn
70.31
7,8 MTB
89.00
0.4 MB
80.94 2016
3.6 MB
81.76
2014
3.8 MB
88.25
1.4 MB
the received classifier head to its local classifier head. We report ensembling and averaging locally trained classifier heads as basclines.
In our disjoint label shift setups, our source client only has samples from the first half of labels (0-4 for CIFAR-10 or 0-49 for CIFAR-100), and the destination client has the other half. In our covariate shift sctups, the two clients have access to the two most distinctive domains of PACS and Office Home datasets according to (Hemati et al. 2023). Specifically, for PACS, we consider the scenario where the source has access only to Photo (P) images, while the destination has access to Sketch (S) images. For Office Home, the source has access to Clipart (C) images, and the destination has access to Product (P) images. In our task shift setups, the two clients have access to two different datasets with distinct tasks. First, we consider the scenario where the source has bird images from Caltech while the destination has car images from Stanford Cars. In the second experiment, the source has access pet images from Oxford Pets, and the destination has food images from Food101.
G.5 Effect of
In this section, we evaluate the effect of using the CIFAR-10 settings from the experiment in Table 3 As shown in Figure II increasing leads the performance of DP-FedPFT to approach that of FedPFT.
80
Accuracy (%)
60
Gypher (1)
Gypher (1)
40
Raw Features
10
10
10
10a
10a
(log)
Figure 11: Effect on the performance DP-FedPFT
H Reconstruction attack details
We conduct reconstruction attacks on the various feature-sharing schemes described in this paper. First, verify the vulncrability of raw feature sharing: we demonstrate that an attacker with access to in-distribution data can obtain high-fidelity reconstructions of privale training data. This necessitates sharing schemes beyond sending raw features. The attack we present proceeds by training a generative model on (feature, image) pairs, and then performing inference on received feature embeddings. Next, we apply the aforementioned attack on features sampled from GMMs received via our proposed scheme. We find that the resulting reconstructions do not resemble real training data.
25

---

Published in Transactions on Machine Learning Research (04/2025)
H.1 Threat model
We consider the setting where two clients (a defender and an allacker) collaborate via fealure-sharing scheme to train model to perform well on the union of their datasets. Both parties have:
Local datasets of (image, label) pairs, which we call Da and Da respectively, where D. = {1ri, D=)
Black-box access feature embedding function E X 2.
The defender produces an embedded version of their dataset E(Da) == {(E(a)), 1 and passes it along the altacker via. fealure-sharing scheme. We consider sending: the embedded dalaset. E(Da) directly (raw features), our proposedeettureshhriig scheme (FeilFFT) well as with differentially privacy (DP-FelPPT))
The attacker has a dataset Da, which is assumed to be in-distribution of the defender's dataset Da- Concretely: Du is the CIFAR-10 train set (50K examples) and Da is the CIFAR-10 test set (10K examples) in our experiments.
H.2 Attacker objectives.
We identify 3 attacker objectives representing varying levels of privacy violation, ordered by strictly decreasing attack strength.
The attacker is able to accurately reconstruct every example in E(Da). 2. (Partial reconstruction). The attacker is able to identify small subset of E(Da), which it accurately reconstruct.
1. (Total reconstruction).
(Set-level reconstruction). The attacker able to produce a set of reconstructions and small subset of them correspond to real training points.
Although weaker attack than (1), (2) still constitutes strong privacy violation, since privacy is worst-case notion: priori, user submitting their data does not know whether they part of the reconstructable set not (see (Carlini et al., 2022) for discussion). Success in (3) would imply an attacker can generate accurate reconstructions, but are unable to identify which candidates correspond to real training data. Note that success in (3) combined with a good membership inference allack implies success in (2).
H.3 Experimental details
We train U-Net-based conditional denoising diffusion models on the CIFAR-10 train set. During training, extracted features are added to time-step embeddings and condition the model via shift-and-scale operations inside normalization layers. We sample with DDIM and classifier-free guidance.
We find that there is a non-trivial overlap between the CIFAR-10 test and train sets; on which our models memorize and reconstruct perfectly. To account for this, we filter the CIFAR-10 test set for near-duplicates: remove the 1K test images with the highest SSIM score with member of the train set, leaving 9K images to evaluate on. We also manually inspect reconstructions and verify that they differ from the closest training image to the target.
H.4 Results
Raw feature reconstruction. We present the results of our reconstruction attack raw features in Figure for 3 selection methods (all, atlacker, and oracle). Each selection method is representative of performance in the corresponding attacker objectives of lotal, partial, and sel-level reconstruction. Quantitative image similarity measures are reported in Table
For sttacker-selection, we sample 10 reconstructions from each embedding, compute the average pairwise SSIM amongst reconstruction, and select the top 1% of reconstructions according to this metric. This is
26

---

Published in Transactions on Machine Learning Research (04/2025)
based the intuition that stronger determinism during sampling implics the model is more confident about what the information in the embedding corresponds to.
Similarity measure
Method
Selection
PSNR f
LPIPS +
SSIM
All
13.5
.384
.168
ResNet-50 reconstruction
Attacker
14.7
.297
.358
Oracle
16.5
.257
.535
All
15.6
.305
.266
MAE roconstruction
Attacker
17.9
.191
.579
Oracle
19.5
.181
.674
Train set
Oracle
14.7
.389
.500
Table Similarity metrics between original image and raw feature reconstructions on filtered CIFAR-10 test set. Figures are computed different selections of the test set, representing different attacker objectives. All: averaged result over entire test set; performance corresponds to total reconstruction objective. Attacker: attacker selects 1% (n 90) reconstructions without access to ground truth; corresponds to partial reconstruction objective. Oracle: top 1% (n = 90) reconstructions selected based on ground-truth SSTM; corresponds to set-level reconstruction.
Selection method
Random
Attacker
Oracle
Original
ResNet-50
Reconstruction
Closest frain
Original
MAE
Reconstruction
Closest train
Figure 12: Results comparing the original image and feature reconstructions. along with the original images' closest train set member by SSIM. We present results for backbones: ResNet-50 and Masked Autoencoder; and 3 selection methods (random, allacker, and oracte), which correspond performance total, partial, and set-level reconstruction.
Our main result is that an allacker is capable of producing reasonable reconstructions in all 3 settings, outperforming a baseline of selecting the closest training image. Furthermore, we find that: (1) The feature backbone affects reconstruction quality (MAE reconstructions are better than ResNet-50); (2) The attacker can effectively employs heuristics (intra-sample similarity) to identify which reconstructions are likely to be good, approaching the results of oracle selection based the ground-truth image.
27

---

Published in Transactions on Machine Learning Research (04/2025)
FedPFT reconstruction. Table 9 and Figure 13 show results for set-level reconstruction of GMM-sampled features, for both random and worst-case test images. We see that even when with ground truth similarity
oracle, most images fail to be reconstructed (Oracle-Randoos Random column of Figure 13 DP further diminishes attack effectiveness, in particular for worst-case sct-level reconstructions.
Similarity measure
Backbone
Selection
PSNR f
LPIPS 4
SSIM f
Oracle
14.7
.423
.508
0
Oracle-all
12.7
.546
.366
ResNet-50
Oracle
14.6
.511
.483
10
Oracle-all
12.7
.571
.322
Oracle
14.5
.466
.497
00
Oracle-all
12.5
.558
.323
MAE
Oracle
13.4
.553
.452
10
Oracle-all
12.2
.595
.306
Table 9: Similarity metrics between original images and set-level reconstructions on the filtered CIFAR-10 10 test set. For each test image, match it to its closest in terms of SSIM among all reconstructions. Oracle: the top 1% (n = 90) of matched reconstructions by ground-truth SSIM, corresponding to performance on sel-level reconstruction. Oracle-all: average similarity between lest images and their matched pairs.
Selection method
Oracle- Rracle-Random
Oracle
FedPF FedPFT(E=a) T(
Original
Reconstruction
DP-FedPFT
Original
Reconstruction
Figure 13: Results comparing the original image and set-level reconstructions with ResNet-50 backbone. We present results for two selection methods. Dracle-Randone the closest image in the reconstruction set by SSIM for random test images. Oracle: the closest image the reconstruction set by SSIM for worst-case test images.
28


Reviews:
[{"review_id": "qcQvo7TVQi", "reviewer": "Reviewer_bpfy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The writing is clear, and the extensive experiments conducted in diverse setups are convincing. Theoretical results further substantiate the research, giving it a concrete foundation.\n\nWeaknesses\n- However, the paper primarily lists experimental achievements and lacks deeper analytical insights. For instance, providing intuition on why FedPFT performs better with certain covariance types and why it remains agnostic to data heterogeneity would be beneficial.\n- It would also enhance the work to connect the high accuracy of FedPFT more directly with the low-level analysis presented in Theorem 4.2.—for example, by incorporating the design of the intended loss and presenting the corresponding accuracy graph.\n- I cannot find the architecture used for the attacker model.\n- The following sentence appears to be duplicated:\n\"Also, Figure 4 shows different tradeoffs for different numbers of mixtures K and covariance types. Additionally, Figure 4 demonstrates different tradeoffs for varying numbers of mixtures K and covariance types.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose FedPFT, which incorporates a one-shot federated learning approach based on transmitting Gaussian Mixture Model parameters. This method addresses the communication cost and GPU memory limitations typical of conventional FL in foundation models. FedPFT offers significant performance guarantees, including scalability (both decentralized and centralized), privacy protection, and robustness to data heterogeneity. The authors support these claims with both theoretical proofs and empirical results across a variety of experimental setups.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iTW6k8fZHc", "reviewer": "Reviewer_ncqR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is mostly well-written and easy to read. \n2. The method is well-motivated. The authors provided sufficient evidence to show that GMMs effectively approximate the distribution of the extracted features from foundation models (and the advantage of increasing $K$ in performance). Moreover, the true loss of the model can be estimated by the loss trained on the synthetic features.\n3. Variants of the method can adapt to decentralized FL and private learning settings, which is not possible for existing one-shot methods.\n4. Extensive experiments are provided to support the authors' claims, including high accuracy, comparable communication cost, and good performance against reconstruction attacks.\n\n**Weakness:**\n\n1. I think the comparison between FedPFT and CCVR in Section 2 should be more extensive to highlight the contributions and differences. In particular, CCVR learns the features and the Gaussian approximation simultaneously through multi-round communication and finally retrains the model on the learned Gaussian representations. FedPFT uses a pre-trained model to extract the features and learns the GMM approximation for server training. So I would expect CCVR using a pre-trained model to perform similarly to FedPFT ($K=1$), is that correct? Although I acknowledge that increasing $K$ can largely enhance the model performance.\n2. It would be better to motivate the choice of $K$ in the differentially private version (currently $K=1$). Is it due to the theory or the computational cost?\n3. In Figure 8, the author showed that FedPFT is robust to data heterogeneity, but it is not clear which $K$ is used and whether the choice of $K$ is sensitive.\n4. Some minor errors:\n* Equation 1, Using $\\frac{n_i}{n}$ as the coefficient is more common in the FL literature.\n* Section 4.1, line 5 \"Next, each client learns runs the Expectation ...\"\n* Page 14, \"A.2 Does sending sending label counts ... \"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel one-shot (i.e. one communication round) Federated Learning (FL) method FedPFT leveraging foundation models. In its centralized version, a task-specific frozen foundation model is employed to extract feature representations from client data. These features are subsequently modeled using Gaussian Mixture Models (GMMs) to capture class-conditional distributions. The clients then transmit the GMM parameters along with corresponding class information to the server, where the distributions are merged and utilized to train a classification head. Empirical studies have shown that centralized FedPFT achieves more accurate prediction than other one-shot FL methods while maintaining comparable communication costs. Moreover, private and decentralized learning can also be considered in this framework. The theory supports the proposed method by giving performance and privacy guarantees.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarification on the difference between FedPFT and CCVR.\n2. Include motivations on the choice of $K$ for DP-FedPFT.\n3. Provide more details in Figure 8.", "broader_impact_concerns": "I have no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qRki2w1wCu", "reviewer": "Reviewer_tydh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n+ The topic of distributing the foundation model is timely and interesting. As large-scale foundation models become increasingly central to various applications, exploring their downstream tasks for their efficient usage seems to be a reasonable need in the field. This focus aligns well with current trends toward scalable and adaptable model deployment.\n+ The usage of differential privacy (DP) can improve the privacy guarantee. By incorporating DP, the work safeguards sensitive client data, and the inclusion of experimental results demonstrating the reconstruction attacks further validate the feasibility and necessity of this privacy-preserving mechanism.\n+ The idea can reduce communication costs by avoiding transmitting the original client-side model. This optimization reduces communication and computational overhead, making the framework more viable for distributed systems where resource efficiency is critical.\n\n### Weaknesses\n\n- The experiments benchmark the proposed method against somewhat dated works, such as FedKT and FedBE, both published in 2020. To strengthen the evaluation, comparisons with more recent studies—such as \"FedFed: Feature Distillation against Data Heterogeneity in Federated Learning\"—would provide a more current and relevant context, especially given the rapid evolution of federated learning and feature sanitization techniques.\n\n- The experimental setup lacks diversity in parameter choices. For instance, Figure 4 considers only 50 clients for comparison. Including experiments with varying client counts (e.g., 5, 10, 20, or 100) could better demonstrate the framework’s scalability. Additionally, the impact of Gaussian mixture parameters on accuracy remains unexplored, as does the effect of varying DP parameters (e.g., different $\\epsilon$ or noise scale) Comparing these with other DP-based protections in federated learning could further clarify the method’s effectiveness. Moreover, Theorem 4.1 appears vague and would benefit from more precise articulation or empirical support.\n- The differential privacy analysis seems not to derive a new concrete conclusion of the proposed framework. In Section 4.2, the authors utilize DP for preserving privacy. However, it seems that the author did not give a new concrete conclusion of DP guarantee? It is unclear whether the proposed framework improves the DP-bound compared to existing methods. A clearer explanation of how (or if) this work advances DP guarantees would strengthen its theoretical contribution. Besides, the analysis to Gaussian mechanism seems to be not new. A better clarification is required to demonstrate its theoretical contribution.\n- The experiments rely on older foundation models like ResNet-50 (Section 5.1), which may not reflect the state-of-the-art. Incorporating more contemporary models, such as large language models (LLMs), could better align the work with current research trends and demonstrate its applicability to cutting-edge systems.\n- The rationale for distributing foundation models in a decentralized framework is not sufficiently justified. LLM is very powerful in a centralized setting. The more explicit motivation to distribute it should be explained more, particular in specific advantages or necessities of adapting them to distributed environments—such as privacy, scalability, or resource constraints.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to adapt large foundation models for downstream tasks in Federated Learning (FL) settings while maintain privacy preservation. To solve the challenges of FL’s iterative training and model transmission, this paper introduces FedPFT, a one-shot FL method with a server-side performance bound to reduce communication costs and GPU memory requirements.\n\nIn FedPFT, without requiring training or fine-tuning of large foundation models, each client learns and transfers parametric models from extracted features from pre-trained, frozen foundation models in one round. These parametric models are used to generate synthetic features at the server to train a classifier head. This work performs some experiments on accuracy, communication costs, and privacy validation for differential privacy against reconstruction attacks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Comparing FedPFT with recent works would align your evaluation with current advancements. Please update your experiments accordingly.\n\n2. The experiments (e.g., Figure 4 with 50 clients) lack parameter diversity. Test additional client counts (e.g., 5, 10, 20, 100), explore Gaussian mixture effects on accuracy, and vary DP parameters ($\\epsilon, \\delta$, noise scale) with comparisons to other DP protections. This will demonstrate scalability and robustness. Also, clarify the vague Theorem 4.1 with precise details or empirical support.\n\n3. Section 4.2’s DP discussion lacks a novel conclusion. Please specify if your framework improves the DP-bound over existing methods, providing a concrete theoretical or empirical outcome to enhance its contribution.\n\n4. Using ResNet-50 (Section 5.1) feels outdated. Incorporating modern models like large language models (LLMs) would better reflect state-of-the-art trends. It would be better to update your experiments with the recent models.\n\n5. It would be better to enrich the justification for shifting to distributed settings with specific motivations.\n\n6. Minor: the font sizes of Figure 6 and Figure 8 are different.", "broader_impact_concerns": "The authors discussed the social impact of distributing foundation models to FL and using differential privacy. It would be better to consider the up-to-date foundation models as baselines for broader impact in the real world.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel one-shot (i.e. one communication round) Federated Learning (FL) method FedPFT leveraging foundation models. It is well-written and motivated, providing a model featuring good performance and Differential Privacy (DP) guarantee. The reviewers suggest the authors to include a few more experiments in the final version of the paper: experiments with different differential privacy budget $\epsilon$, a comparison with the baseline FedFed.
Decision: Accept (poster)