Covariances for Free: Exploiting Mean Distributions for Training-free Federated Learning
Authors: Dipam Goswami
Simone Magistri
Kai Wang
Bartłomiej Twardowski
Andrew D. Bagdanov
Joost van de Weijer
Venue: NeurIPS 2025 poster ()
Keywords: Federated learning
Transfer Learning
Primary area: deep_learning
TL;DR: We estimate global class covariances at the server with a provably unbiased estimator requiring only local class means from clients, achieving performance competitive or superior to algorithms sharing second-order statistics

Abstract: Using pre-trained models has been found to reduce the effect of data heterogeneity and speed up federated learning algorithms. Recent works have explored training-free methods using first- and second-order statistics to aggregate local client data distributions at the server and achieve high performance without any training. In this work, we propose a training-free method based on an unbiased estimator of class covariance matrices which only uses first-order statistics in the form of class means communicated by clients to the server. We show how these estimated class covariances can be used to initialize the global classifier, thus exploiting the covariances without actually sharing them. We also show that using only within-class covariances results in a better classifier initialization. Our approach improves performance in the range of 4-26% with exactly the same communication cost when compared to methods sharing only class means and achieves performance competitive or superior to methods sharing second-order statistics with dramatically less communication overhead. The proposed method is much more communication-efficient than federated prompt-tuning methods and still outperforms them. Finally, using our method to initialize classifiers and then performing federated fine-tuning or linear probing again yields better performance. Code is available at https://github.com/dipamgoswami/FedCOF.

Covariances for Free: Exploiting Mean Distributions for Training-free Federated Learning
Dipam Goswami
Simone Magistri Kai Wang2556
Bartlomiej Twardowski 24 Andrew D. Bagdanov2
Joost van de Weijer!
DDpartment of Computer Science, Universitat Autònoma de Barcelona, Spain 'Computer Vision Center, Barcelona, Spain
Media Integration and Communication Center (MICC), University of Florence, Italy IDEAS Research Institute, Warsaw, Poland 'City University of Hong Kong Program of Computer Science, City University of Hong Kong (Dongguan) dgoswami @eooswami@vve.... waw.uua.eee.......e............. nab.es, simone ieriiistriumiif..im kai wang@cityu-dg............. edu. cn
Abstract
Using pre-trained models has been found to reduce the effect data heterogeneity and speed up federated learning algorithms. Recent works have explored training- free methods using first- and second-order statistics to aggregate local client data distributions the server and achieve high performance without any training. In this work, we propose a training-free method based on an unbiased estimator of class covariance matrices which only uses first-order statisties in the form of class means communicated by clients to the server. We show how these estimated class covariances can be used to initialize the global classifier, thus exploiting the covariances without actually sharing them. We also show that using only within- class covariances results in better classifier initialization. Our approach improves performance in the range of 4-26% with exactly the same communication cost when compared to methods sharing only class means and achieves performance competitive or superior to methods sharing second-order statistics with dramatically less communication overhead. The proposed method is much more communication- efficient than federated prompt-tuning methods and still outperforms them. Finally, using our method to initialize classifiers and then performing federated fine-tuning linear probing again yields better performance. Code is available at https: //github. com/ dipangoswami/ FFedCDF
Introduction
Federated learning (FL) is a widely used paradigm for distributed learning across multiple clients. In FL, each client trains their local model on their private data and then sends model updates to common global server that aggregates this information into a global model. The objective is to learn a global model that performs similarly
model jointly trained all the client data. A major concern in existing FL algorithms [77] is the poor performance when the client data is not identically and independently distributed (iid) or when classes imbalanced between clients [56] 32 [2]] In B5),
the authors showed that client drift in FL is mainly due to drift in client classifiers which optimize
local data distributions, resulting in forgetting knowledge from clients of previous rounds 33) 5
Another challenge in FL. is the partial participation of clients in successive rounds
(32]. which
becomes particularly acute with large numbers of clients 43 20 To address these challenges, recent
works have focused on algorithms to better tackle data heterogeneity across clients [35 47 29
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table I: FedNCM 29 shares only class means e and has minimal communication. Fed3R [T] requires sum of class features Bg and feature matrix G₁ from all clients, thereby increasing the communication cost by d K. We propose FedCOF, which shares only class means and esti- mates a global class covariance E. initialize the classifier weights. Note that only small subset of all classes are present in each client. For simplicity, we show the upper bound of communication cost here where C" denotes the maximum number of classes present a single client.
Accuracc (56
Fed34
(Dun))
29-
30000
20883
39000
aréss
sodee
6300
Communication (MB)
Figure I: Accuracy vs. communication cost using pre-trained MobileNetv2 on iNaturalist- -serisst-UUesss---ee 120K. FedCOF out- performs Fed3R while having the same communication cost as FedNCM.
Method
Client Shares
Server Uses
Comm, Cost
FedNCM
(A M Dh
4C'K (dC' + (P)
Fed3R
(Gr, B.
FedCOF
MAAn Me Ne ti (E
d
Motivated by results from transfer learning [15. several recent works on FL have studied the impact of using pre-trained models and observe that it can significantly reduce the impact of data heterogeneity [29 38
46 [5]. An important finding in several of these works that sending local class means to the server instead of raw features is more efficient in terms of communication costs, eliminates privacy concerns, and is robust to gradient- based attacks 6 57]. The authors of [47] used pre-trained models to compute and then share class means as the representative of each class, and in 29 the authors showed that aggregating local means into global means and setting them as classifier weights (FedNCM) achieves very good performance without any training. FedNCM incurs very little communication cost and enables stable initialization. Recently, the authors of Fed3R [1] explored the impact of sharing second-order feature statistics from clients to server to solve the ridge regression problem in FL and improve over FedNCM. The sharing of second-order statistics has also previously been explored for classifier calibration after federated optimization (35)
47
6 41
45
35
Although it is evident that exploiting second-order feature statistics results in better and more stable classifiers, it poses new problems. Notably, sharing second-order statistics for high-dimensional features from clients to the server significantly increases communication overhead and exposes clients to privacy risks [35 [T]. To reap the benefits of second-ordes client statistics, while at the same time mitigating these risks, in this paper we propose Federated learning with COvariances for Free (FedCOF) which only communicates class means from clients to the server (see Table TD. We show that, from just these class means and the mathematical relationship between their covariance and the class covariance matrices, we can compute a provably unbiased estimator of global class covariances on the server. FedCOF is training- the same communication budget as FedNCM while delivering performance comparable to or even superior Fed3R (see Figure TT. Additionally, only within-class covariances, setting the classifier weights based on aggregated class means and our estimated class covariances. To summarize, our main contributions are:
method that exploits pre-trained feature extractors. It uses
show how improve classifier initialization using
We propose FedCOF, a training free method that exploits a provably unbiased estimator of class covariances, which requires only class means from clients, thus avoiding the need to share second-order statistics, reducing communication costs and mitigating privacy concerns.
We show how FedCOF leverages within-class covariances, estimated solely from client means, initialize the global classifier, yielding more stable and better-conditioned solutions than using both within- and between-class scatter matrices.
We validate FedCOF across multiple FL benchmarks, including the non-iid iNaturalist- Users-120K 120K dataset, achieving state-of the-art results with lower communication cost than methods using second-order statistics, and showing superior performance to recent federated prompt-tuning approaches while also serving federated optimization methods such as fine-tuning and linear probing.
effective initialization for subsequent
2 Related Work
Federated learning. FL. focuses on neural network training in distributed environments 555 Initial works like FedAvg (37) proposed training by averaging of distributed models. Later works

---

focus more on non-iid settings, where data among the clients is more heterogeneous B2 20 51
В3]. Fedaova 52 normalizes local updates before averaging to address objective inconsistency.
Scaffold [22] employs control variates to correct drift in local updates. FedProx
31 introduces
proximal term in local objectives to stabilize the learning process. [42 proposed use of adaptive optimization methods, such as Adagrad, Adam and Yogi, at the server side. While CCVR [35] proposed a classifier calibration method that aggregates class means and covariances from clients, other recent works [34] 8 39 [3 proposed using fixed classifiers inspired by the neural collapse phenomenon. After federated training with fixed classifiers, FedBABU [39] proposed to fine-tune the classifiers and SphereFed [8] proposed closed-form classifier calibration.
FL with pre-trained models. While conventional FL methods start training from scratch without any pre- traininin, we focus on the FL setting using pre-trained models, FedFN 24 recently highlighted that using pre-trained weights can sometimes negatively impact performance. However, there has been increasing interest in incorporating pre-trained, foundation models into federated learning. Multiple works propose using pre-trained weights which reduces the impact of client data heterogeneity and 45 54]. Federated prompt-tuning methods [54]
achieves faster model convergence [38 47
6 41
proposed to reduce the communication cost by tuning only prompts in a federated manner using pre-trained VIT models. Recently, it has been shown that training-free methods using pre-trained networks, achieve strong performance without any training by exploiting feature class means [29] or second-order feature statisties [II]. Another recent work (2) proposed sharing Gaussian mixture models (multiple sets of means and covariances) for each class from clients to server for one-shot FL. In this work, we propose training- trining-frree method with pre-trained models that estimates class covariances from only client means for initializing the global classifier.
3 Preliminaries
In the FL setting we assume K clients, each with local dataset Dx = (Xx, Yk.), for k E {L...., K}. and denote by N the total number of images across all clients. The model is composed of a feature extractor f. parameterized by 0, which maps images to d-dimensional embeddings, and classifier h: IRd - LR parameterized by W, where C' is the total number of classes. In federated optimization, each client trains its local model on its private data D3 and transmits only intermediate information such as parameter updates or feature statistics while a central server aggregates these signals to minimize a global objective, without revealing raw data 225)
With the growing availability of high-quality pre-trained models, recent works have focused on scenarios in which all clients are initialized with the same pre-trained feature extractor 6 29 38 47
54]. Notably, training-free methods [T] 29 - in which only the global classifier is initialized using client feature statistics, without training the feature extractor - achieve strong performance. They often outperform federated full fine-tuning methods, such as FedAdam and FedAvg, which train client backbones and classifiers, starting from the same shared pre-trained feature extractor but with randomly initialized classifiers, at fraction of the communication and computation costs. Moreover, this training-free initialization can serve as an effective starting point, improving the performance of federated full fine-tuning, We now discuss recent training-free methods.
Federated NCM. FedNCM 29 employs a Nearest Class Mean (NCM) classifier, where the global linear classifier weights for class (denoted as Wc) are initialized as Pe/ll Relll Here, Poo represents the global class mean aggregated from the local client class means e as follows:
K
1
Σ S(x),
Ae
ThE.C 2EX2.c
where X1.c is subset of Xx having images of class e, ni,c refers to number of images in Xg,es and No = =i Tkkc is the number of images of class across all clients.
Federated Ridge Regression. While FedNCM exploits only class means, Fed3R (I]] recently proposed using ridge regression which requires second-order feature statistics from all clients to initialize the global classifier, leading improved performance compared to FedNCM. The ridge regression problem aims to find the optimal weights that minimize the following objective:
W arg min
||Y FT 211 + W|//
(2)
WEROXC
3

---

where F E RAxN Ñ is the feature matrix extracted using pre-trained model and Y E R contains one-hot encoded labels for the N features over C classes. The closed-form solution is given by:
W = =G -
where G FFT E Rdxd B FY E Raxc' Ia E IRddd is the identity matrix, and À IR is hyperparameter. In Fed3R, each client computes two local matrices G = F& E RRdxd and B₁ = F&Yx E R where F& and Y2 are the feature matrix and the labels of client k. and then sends them to the global server. The server aggregates these matrices as = Σ Gs. B =>> Bk to compute W which is normalized and used to initialize the global classifier.
4 Federated Learning with COvariances for Free (FedCOF)
In this section, we derive our FedCOF method which based on provably unbiased estimator of class covariances and requires transfer of only class means from clients to server.
4.1 Motivation
Communication cost. While Fed3R more effective than FedNCM, it requires each client send an additional d x d matrix, significantly increasing the communication overhead by d K compared FedNCM which only shares the class means (see Table TT. Fed3R scales linearly with number of
clients and quadratically with the feature dimension. FedPFT 2 shares multiple sets of (pi,e, Ek.e) from clients, further increasing communication costs. Considering cross-device FL settings [20], having millions of clients, the communication cost required for these methods would be enormous.
Recent works 26 54] focus parameter-efffcient federated fine-tuning where the per-round com- munication costs are significantly reduced, enabling applications in low-bandwidth communication settings. For example, recent work (26) shows that using pre-trained language models (RoBERTa- base) can yield performance similar to full fine-tuning while using rank-1 LoRA updates and thereby reducing communication cost by 99.8%. Sharing a similar goal reducing communication costs, we propose an unbiased covariance estimator without sharing client covariances for training-free FL.
Potential privacy concerns. Sharing only class means provides higher level of data privacy compared to sharing raw data, as prototypes represent the mean of feature representations. easy reconstruct exact images from prototypes with feature inversion attacks 35 As a result, sharing class means common in many recent works [47 46 45 29
is not
On the other hand, Fed3R shows that sharing second-order statistics improves performance compared sharing class means. However, this could expose the feature distribution of clients to the server since all clients employ the same frozen pre-trained model to extract features [T]. Sharing covariances makes clients more vulnerable to attacks if secure aggregation protocols are not implemented 3
4.2 Estimating Covariances Using Only Client Means
Our method leverages the statistical properties of sample means derive an unbiased estimator of the class population covariance based only on per-client class means (see Figure We model the global features of each class drawn from multivariate distribution with mean M4c and covariance Ec. and assume that the local features for class c computed by each client using the shared frozen pre-trained model are iid. Under this assumption, class features in a client form random sample from the class population.
For client kt let ( from class e assigned to client k. The sample mean of these features, ke
yau. denote the feature vectors of class c, where 7&.. is the number of samples
_1_ R4. F is
itself random variable with expectation and variance given by:
Σ
E F k.el = Pe: Far
(4)
Tkk.c
This well-known result whose proof we include in Appendix B for reference implies that the variance of sample means over subsets of size Ng.c reflects the underlying class covariance. In principle, by assigning multiple subsets of n4... features to a single client and computing the empirical covariance of their means, could recover E., since E. nkk Var [k.].

---

Ec
Client. 1
Plaes 13,,
FedCOF Server
fo
Data
Ee
De
X1
E1
Ee
(A)
(B)
Client k
Pluc Plkee
Inttatiza Linear
fo
Pey
X4
Unhinand Fetiimaton of Per Vie
Figure 2: Federated Learning with COvariances for Free (FedCOF). Each client communicates only its class means fice and counts Tkk.c- On the server side, (A) we use provably unbiased estimator Σ. (denoted by solid lines) of population covariance M. (denoted by dashed lines) based on the received class means (sce Section 4.2). (B) We initialize the linear classifier using the estimated second-order statistics and remove the between-class scatter matrix as discussed Section
However, in federated learning data assigned only once each client, and there are K clients in the federation, each with rk features and 74.cc * ng..c for + j. To estimate the population covariance Σe. we need an estimator that accounts for the contributions of all K clients. In the following proposition, we propose just such an estimator.
Proposition 1. Let K be the number of clients, each with ng.cc features, and let C be the total number of classes. Let Pr. - + Fí be the unbiased estimator of the population mean pee and Ne ===xx = nk.c be the total number of features for single class. Assuming the features for class e are tid across clients at initialization, the estimator
Σ Ê= K-12 ビ-=K1ī
(F (F) -Mc)
(5)
is an unbiased estimator of the population covariance for all I....,C.
Proof. To prove that Σ. is an unbiased estimator of the population covariance, we show that E[Σ,] = Σg. Under the iid assumption of client feature distribution with a frozen pre-trained model, the class features of each client
be considered as random sample of size nk.,o and the global class features as a sample of size No. By exploiting the result in Equation 4 each client class mean has E F k,c] = Me and Var [F k,c] = Ee, while the global class mean Pee has E[Ac] = Pe and N. Using this fact and applying the properties of expectation to E[Ex], we complete the proof. In Appendix C we provide the detailed proof.
Var[ūe] Ex.
stabilizes covariance estimation by adding a scaled identity matrix to the covariance matrices, which regularizes small eigenvalues and reduces estimator variance. This is particularly helpful when the number of samples is smaller than the feature dimensionality, leading to low-rank covariances. and has been recently adopted in continual learning methods leveraging feature covariances [14] 36]. In the FL setup, the covariance estimation using limited number of clients may poorly estimate the population covariance Σg. So, we perform shrinkage to better estimate the class covariances from the client means as follows:
Covariance shrinkage. Shrinkage|49) 12]
C (. PO) + 7ld=
(6)
K
where Po.c = FR. represents a realization client means and 7 > 0 is the shrinkage factor.
Impact of number of clients. The quality of estimated covariances depends on the number of clients. More clients will give more means and improve the estimate compared to fewer clients. While realistic settings have thousands of clients [18 20), there can be FL settings with fewer. In such cases, we propose to sample multiple means from each client to increase number of means used for covariance estimation. This can be done by randomly sampling subsets of features in each client and computing a mean from each subset. We validate this approach experimentally (see Figure 5) We discuss more sampling multiple class means in Appendix E
5

---

4.3 Classifier Initialization with Estimated Covariances
Having derived how to compute class covariances from client means, we now describe how FedCOF leverages the estimated class covariances to initialize classifier weights.
Proposition 2. Let F E RaxN be a feature matrix with empirical global mean Pg E Rd, and Y E NRC be a label matrix. The optimal ridge regression solution W = (G + XTa) where B E RdxC and E IRdxd can be written in terms of class means and covariances as follows:
B= (-NNeel G G=E(N, - 1)5. + ENNNla - - - -Pg) + NNP.g
(7)
c=1
c=1
where the first two terms Gwith- = EN - 1)$e and Gow-= =cE= Nd i -ig) represents the within-class and between class scatter respectively, while Pe.. Se and No. denote the empirical mean, covariance and sample size for class c, respectively.
Proof. The proof is based on the observation that G = FFT from ridge regression an uncentered and unnormalized empirical global covariance. By using the empirical global covariance definition and decomposing it into within-class and between-class scatter, we obtain the above formulation of G. In Appendix D we provide the detailed proof.
To analyze the impact of the two scatter matrices, first consider the centralized setting in Table 2 and empirically find that using only within-class scatter matrix performs better than using total scatter matrix in Equation 7 We then analyze their spectral prop- erties via the condition number, defined for a matrix G as K(G) Amax (G)/Att (G), where Ama and Aiin (G) are the largest and smallest non-zero eigen- value, respectively. For a SqueezeNet backbone in the centralized setting, we observe that Gotw is highly ill-conditioned, with condition numbers (C(G of 3.0 10", 2.5 10?, 2.2 10", 1.3 x 102 on CUB, Cars, ImageNet-R and CIFAR- respectively, while Gwntn is much better conditioned (KC(Gmitt): 4.3x 2.5 x 10a, 8.2 x 102, 6.3 x 10a). Including Gbrer can cause numerical instability due its poor conditioning, leading the classifier to overfit directions with small cigenvalues that capture noise dataset-specific artifacts. This is consistent with the results in Appendix where we show that using Gstw leads to higher overfitting on the training data.
Table Analysis showing improved accu- racy by removing between-clat scatter for classifier initialization in centralized setting using traitee SqueezeNet.
Dataset
Gotw
Guith
Avc.(t)
CIFAR-100
57.1
x
57.3
ImageNet-R
37.6
x
/
38.6
CUB200
/
50.4
X
53.7
Stanford Cars
41.4
X
44.8
As result, we propose to remove the between-class scatter and initialize the linear classifier at the end of the pre-trained network using the within-class covariances E, which are estimated from client means using Equation (6), as follows:
C W - G-IB - [ -N.- 1J + Naau.
(8)
c=1
Theoretically, we observe that a similar approach is used in Linear Discriminant Analysis 13). which employs only within-class covariances for finding optimal weights. By removing between-class scat- ter, we propose a more effective classifier initialization than Fed3R (which uses G from Equation 7 and considers both within- and between-class scatter matrices).
We summarize in Algorithm I how we estimate the covariance matrix for each class using only the client means and use the estimated covariances to initialize the classifier in Equation 8 The normalization the weights accounts for class imbalance in the entire dataset.
Impact of the iid assumption. FedCOF initializes the classifier with the class covariance estimator (Equation (). unbiased only under the assumption of iid pre-trained features per class (Section 4.2). In FL. each client has its own data, typically distributed in a statistically heterogeneous or class- imbalanced manner according to a Dirichlet distribution [17. As a result, each client has data from different set of classes, resulting in non-iid data distributions across clients. However, note that the

---

Algorithm 1 FedCOF: Federated Learning with COvariances for Free
Client-Side (Client k):
Server-Side:
Input: C: set of all classes. fe: pre-trained model, Xx,e: samples class in client k. Bx,c: number of samples in Xk.e
Input: jik,e Pk,r sent from K clients, 7 for c 1... do
Ac = No Ex ne.efueec No = Ex Tkke
for forc== C' do
AA.. RE,E xEXE,o
Σ fx(x)
E. = K-1 Σ
1
nn.elfix (o) T +yyaa Eq.
end for
end for
Pg # Eest Nefic N= No
Send the class means fúk,c and sample counts
Akk to the Server
B liaNeleem
G =E 1)2 Nig
W ="= G-1 B. Eq.
Normalize W": W. W//l 1.....,,
samples belonging to the same class in different clients are sampled from the same distribution. We exploit this fact in FedCOF. We later show empirically that our method can be successfully applied to non-iid FL scenarios involving thousands of heterogeneous clients on Naturalist-Users 120K (18]. We hypothesize that this can be attributed to the strong generalization capabilities of pre-trained models. We analyze the bias of the estimator under non-iid assumptions for the same class and also evaluate FedCOF in feature shift settings 33 in Appendix G
FedCOF in multiple rounds. While the proposed estimator requires class means from all clients in single round, this might not be realistic in settings which clients appear in successive rounds based on availability. For multi-roum classifier initialization (see FedCOF in Figure 3 before fine-tuning), the server uses all class means and counts received from all clients seen up to the current round and stores the accumulated means and counts for future rounds. As a result, FedCOF uses statistics from all clients seen up to the current round, similar to Fed3R. This can be easily implemented by ensuring on the client side that each client transfers statistics to the server only once, avoiding repeated transfer of statistics. FedCOF converges when all clients are seen at least once, with total communication cost equal to the single round-case (see Appendix H for details on communication cost).
Privacy aspects. FedCOF requires each client to send its class means and frequencies (see Eq. 5) similar to CCVR (35). While this avoids sharing feature-level data, it does not guarantee strong privacy and may introduce concerns, particularly because complicates the secure aggregation protocol applicable in Fed3R. Nonetheless, as discussed in Appendix [I, FedCOF can be combined with additional adapting the method to support secure aggregation. The latter preserves FedCOF's performance but increases communication overhead to the level of Fed3R.
mechanisms, such as adding noise to the shared statistics or
5 Experiments
Datasets. We evaluate FedCOF multiple datasets namely CIFAR- (28]. ImageNet-R [16]
(IN-R), CUB200 [50]. Stanford Cars (27) and iNaturalist (48). We distribute the first 4 datasets
to 100 clients using highly heterogeneous Dirichlet distribution (a 0.1) following standard practice 17 29). We also use real-world non-iid FL benchmark of NNtuuallsstUSsess 120K [18
(20K) having 1203 classes across 9275 clients. We discuss the dataset details in Appendix J
Implementation details. We use three models: namely SqueezeNet [99 following 22 and (88, MobileNetV2 44 following [8] and ViT-B/16 [9] All models are pre-trained on ImageNet- 1k (7]. We use the FLSim library. We use 7 = for all experiments with SqucezeNet and ViT-B/16, and 7 = 0.1 for all experiments with MobileNetV2 due to very high dimensionality d of the feature space. We compare to FedCOF Oracle in which real class covariances are shared from clients and aggregated in server instead of using we set the client participation in each round to 30%, and we show the training- faining-free methods in multiple rounds in Figures 3 and 4 We provide more implementation details in Appendix J We discuss computation of communication costs for all methods Appendix
estimated covariances (see Appendix[] For all experiments,
Evaluation for different training-fre methods. We compare the performance of existing training- free methods and the proposed method in Table using pre-trained Squeezenet, Mobilenetv2 and ViT- B/16 models. We observe that Fed3R [ using second-order statisties outperforms FedNCM (9
7

---

Table 3: Evaluation of different training-free methods using 100 clients for first four datasets and 9275 pre-defined clients on iNat-120K across 5 random seeds. We show the total communication cost (in MB) from all clients server. We also show the FedCOF oracle in which full class covariances are shared from clients to server. Best results from each section in bold.
SqueezeNet 512)
MobilaNctv2 1280)
VIT-B/16 (d
768)
Method
Acc (f)
Comm. ())
Ace (f)
Comm. (L)
Acc (t)
Comm. (J)
CIEAR- 100
FedNCM
41.5.101
5.9
55.610.11
$5.2.10.1
8.9
Fed3R 11
56.9110.1
110.2
62.710.1
670.1
73.910.1
244.8
FedCOF (Ours)
56.1+0.2
5.9
63.5+0.1
14.8
73.2+0.1
8.9
FedCOF Oracle (Full Covs)
56,4+0.11
3015.3
63.9+0.1
18823.5
73.8+0.1
6780.0
FedNCM
23.8++0.1
37.6+0.2
32.3+0.1
10.7
IN-R
Fed3R [1
37.610..
111.9
46.010.3
673.1
51.910.2
246.6
FedCOF (Ours)
37.810.4
7.1
47.4
17.8
51.8:0.3
10.7
FedCOF Oracle (Full Covs)
38.210.11
3645.7
48.010.3
22758.8
$2.7.10.1
8197.4
CUB200
FedNCM 22
37.8.10.3
4.8
58,310.3
12.0
75.7:10.1
7.2
Fed3R [1]
50.4+0033
109.6
58.6+00.2
667.3
77.7+0.1
243.1
FedCOF (Ours)
53.7.10.3
4.8
L..5
79,4:10.2
7.2
FedCOF Oracle (Full Covs)
54.4+0.11
2472.1
63.1+0055
15432.7
79.6+0.2
5558.6
FedNCM (29
19.8+0.2
5.4
30.0+0.11
13.5
26.2+0.4
8.1
Cars
Fed3R 11
39.9:0.2
110.2
41.620.1
668.8
47.921.33
244.0
FedCOF (Ours)
44.0++0.3
5.4
47.3+0.5
13.5
52.5+0.3
8.1
FedCOF Oracle (Full Covs)
44.6:10.1
2767.3
47 47.2.00.3
17275.7
53.110.11
6222.5
1Nat- 120K
FedNCM 29
21.21:0.1
111.8
36.010.11
279.5
53.910.1
167.7
Fed3R [
32.1::0.1
9837.3
41.5:10.11
61064.
62.5++0.1
22050.2
FedCOF (Ours)
32.5.10.1
111.8
44.1.00.1
279.5
63.110..1
167.7
FedCOF Oracle (Full Covs)
32.4+-0.1
57k
43.6+-0.1
358k
62.9+0.1
128k
significantly ranging from 0.3% to 21% across all datasets. However, Fed3R requires higher communication cost compared to FedNCM. In real-world iNat- benchmark, Fed3R needs 61 MB compared 280 MB for FedNCM (see Figure D. which is 218 times higher. FedCOF performs better than Fed3R in most settings despite having the same communication cost as FedNCM. FedCOF achieves similar performance as the oracle setting using aggregated class covariances requiring very high communication, which validates the effectiveness of the proposed covariance estimator.
FedCOF maintains similar accuracy with Fed3R CIFAR-100 and ImageNet-R, with improve- ment of about 1 % when using MobileNetv2. FedCOF outperforms Fed3R on CUB200 and Cars. On CUB200, FedCOF outperforms Fed3R by 3.3% 3.9% and 2.2% using SqueezeNet, MobileNetv2 and ViT-B/16 respectively. FedCOF improves over Fed3R in the range of 4.1% to 5.7% on Cars. On iNat-120K, FedCOF improves over Fed3R by 0.4%, 2.6% and 0.6% using different models. When comparing FedCOF with FedNCM both with equal communication costs and same strategy in clients can observe that the usage of second order statistics derived only from the class means of clients leads to large performance gains, c.g. 24% using SqueezeNet and 26% using ViT-B/16 on Cars, above 8% using all architectures on large-scale ANat-120K, We also adapt CCVR (35) for classifier initialization in Appendix and show that FedCOF outperforms CCVR.
Comparison with communication-effiies methods. We consider the recent Probabilistic Fed- erated Prompt-Tuning (PFPT) approach that aggregates trainable prompt parameters from clients and tunes them in federated manner while keeping the backbone fixed [54]. We compare PFPT, FedAvg-PT, and FedProx-PTT (prompt-tuning variants of FedAvg and FedProx) with the proposed FedCOF. Similar to PFPT 54 we use a pre-trained ViT-B/32, assign class samples clients using Dirichlet distribution (a = 0. 1), and use random seeds. We use the same training hyperparameters as PFPT. We show in Table that FedCOF is much more communication efficient compared PFPT and achieves higher performance without any training. We also observe that prompt-tuning methods perform poorly on fine-grad mne-grrined datasets. We discuss more on this Appendix H
Table 4: Comparison of FedCOF with federated prompt-tuning methods using VIT-B/32.
CIFAR-100
IN-R
CUB200
Cars
Method
Acc (f)
Comm. (1)
Acc (t)
Comm. ())
Acc (?)
Comm. ())
Acc
Comm. (L)
FodAvg-PT 54
74.5=0.5
884.7
47.6E1.3
1622.0
37.0=2.0
1622.0
13.6E17
1592.5
FedProx-PT
73.620.4
884.7
47.940.5
1622.0
38.5-10.8
1622.0
13.7415
1592.5
PFPT [44
75.1+00.5
846.5
50.7-40.2
1794.4
38.6+0.9
1765.5
12.9+1111
1736.1
FedCOF (Ours)
75.3=0.11
8.9
54.9410.2
10.7
65.0=20.1
7.2
50.410.1
8.1
8

---

ImageNict R SqueczeNet
CUB200 SqucezeNet
Cars SqueczeNet
54-
Accurncy th
RCCLE
Fedting
Tedlvg
Fedlug
Tedtidam
Tedkdom
Tooksam
12
FodNCM Fooldomn
FoBlCM FodAlam FeeB-Feed FooCDF-Feol.aam
Redhic Fodiddn
Fed3R+ Fedkkaa
Fe Fediddmm
Fed:OFF-Fechhne
FedCCF- Fedlldam
25
20
300
210
de
300
500
15
350
100
136
300
Rounds
Rounds
Rourdds
Figure 3: Performance comparison when initializing with different methods and fine-tuning with FedAdam [42 and FedAvg (37. We also compare with FedAdam and FedAvg using a pre-trained backbone and random classifier initialization. The training-free initialization stages are shown as dotted lines and stars represents the start of fine-tuning stages. Accuracies are averaged over 3 seeds.
Comparison with full fine-tuning methods. We compare training-free methods with FL. baselines like FedAvg and FedAdam with randomly initialized classifier and pre-trained backbone in Table [5] We use adaptive optimizer, FedAdam 42] since it performs better than most other optimizers as shown in [38. Without any training. FedCOF outperforms FedAvg in all settings and FedAdan by 7.3% on CUB200 and 2.2% on Cars, and achieves competitive performance in ImageNet-R. We show in FigureBhhow FedCOF starts from a very high aceuracy compared FedAdam and further improves fine-tuning. We provide more experiments with pre-trained ResNet-18 in Appendix K
Analysis of fine-tuning and linear probing. While FedCOF achieves high accuracy with- out training, we show in Figure that that further fine-tuning the model with FL achieves bet- ter and faster convergence compared to fed- erated optimization from scratch. We show the performance of fine-tuning after training- free classifier initialization in Figure 3 These training-free methods end after all clients ap- pear at least once to share their local statistics. We fine-tune the models after FedCOF and Fed3R for 100 rounds since they achieve fast convergence, while we train for 200 rounds for FedAdam, FedAvg, and fine-tuning after FedNCM which takes longer to converge. Fine-tuning after FedCOF starts at a higher accuracy and converges faster compared to FedNCM. Although FedCOF achieves better accuracy than
Table 5: Comparison with training- FI. base- lines (FedAvg and FedAdam) using pre-trained SqueezeNet. For training-basee methods, we con- sider 100 rounds of training for fair comparison and report accuracy random seeds.
Melbod
Traming
ImageNet-R
CUB200
Cars
FedAug
30.020.6
303316.7
24.91.6
FedAdam
38.84066
464408
418+06
FedNCM
X
23,8+0.1
378+033
198402
Fed3R
X
376602
50440.3
399402
FedCOF (Ours)
37.8+0.4
$3.740.3
44.0+0.3
FeaNCM+FaaAddmm
44.7+0.1
50.2+0.2
48.7+0.2
Fed3R-FedAdae
45.920.3
$1.24003
47.410.4
FedCOF FFddDFa
46.0=0.44
55.7+0.4
49.6+0066
and Fed3R converge similarly ImageNet-R, FedCOF Fed on CUB200 and Cars. We see in Table that all training- ffreinng--reeae approaches followed by fine-tuning outperform FedAdam and FedAvg with random classifier initialization.
Following 29 and 38), we perform federated linear probing (LP) of the models using FedAvg after classifier initialization with training-free -free methods. In FedAvg-LE LP,, we perform FedAvg and learn only the classifier weights of all client models. Linear probing requires much less computation compared to fine-tuning the entire model and were found to be effective with pre-trained models. We observe in Figure 4 that linear probing after FedCOF improves significantly compared to FedNCM and Fed3R using ViT-B/16 on Cars and SqueezeNet iNat- 120K. On the real-world dataset iNat 1 120K, FedAvg- LP with random classifier initialization achieves 27.3% after 5000 rounds while FedCOF achieves 34% in less than 1000 rounds. We plot accuracy versus communication in Figure 4 (center) to demonstrate the advantage of FedCOF over other methods. We discuss more in Appendix K
Impact of number of clients and data heterogeneity. We analyze in Figure 5 (left), the performance of FedCOF with varying number of clients and data heterogeneity. We observe that the performance of FedCOF improves with increasing number of clients and decreasing heterogeneity. This is due to the fact that more clients provides more class means and more uniform data distribution gives better representative local means. While more clients are favourable for FedCOF, it still performs well and outperforms FedNCM significantly in the setting with 10 clients and high data heterogeneity.
Multiple class means per client. We analyze FL settings with fewer clients ranging from 10 to 50 in Figure 5 (center) and show that sharing multiple class means from each client improves the accuracy. Using only 10 clients, sharing 2 class means per client improves the accuracy by 2.6%.

---

INat: 120K SqueezeNet
INet-120K -quesokkKSuccseeett
Cars SqueozeNet
Accuracy (83)
Accuracy (h)
Feddeg
FesAvg UP
1P
Fa
Fad3R= LP
Fod3R=Foolkgg LP
FosBR Faddeng LP Fedtcd uudtogg
Fedcor 4t
Fedcor 4tedhrg--
ablo
Rounds
log18(Communication (MB))
Rounds
Figure 4: Performance of different classifier initialization methods when linear-probing with Fe- dAvg 37. FedAvg-LP (in blue) uses random classifier initialization and pre-trained backbone. The
training- free initialization stage is shown in dotted lines, stars represents the start of linear probing.
CIFAR100 MobileNetv2
CIFAR100 SqueezeNet
ImageNet-R SqueezeNet
Accurscy (5)
FedNCM aipha=00.11.100
Accurscy (i
FedBi (elphamO: 1.1.10)
Fedcor (aipha=0.1)
Accuracy
(sipham1)
10 dients
FedCoF (alpha-10)
20 dients
Shrinkage Shrinlage Shrinkage maans)
40 dients
50
14 Number
Number of share per
clients
Figure 5: Ablation experiments: (left) Change in performance with varying number of clients and data heterogeneity. (center) Sharing multiple class means per client improves FedCOF performance. (right) Impact of shrinkage with varying number of clients and sampled means per client.
Impact of shrinkage. We show in (right) that using shrinkage improves the covariance esti- mates thereby improving the accuracy. We observe that shrinkage consistently improves performance, especially when the number of sampled means is small. When the number of means is low relative to the feature dimension (d = 512 in SqueezeNet), the covariance estimate becomes ill-conditioned, and shrinkage stabilizes it. However,
class means sampled per client the number of clients increases, the benefit of shrinkage diminishes, since the total number of means approaches the feature dimension, making the estimate more stable. We present more ablation studies in Appendix L
Conclusion
In this work we proposed FedCOF, a novel training-free approach for federated learning with pre- trained models. By leveraging the statistical properties of client class sample means, we showed that second-order statistics can be estimated using only class means from clients, thus reducing communication costs. We derived a provably unbiased estimator of population class covariances, enabling accurate estimation of global covariance matrix. By applying shrinkage to the estimated class covariances and removing between-class scatter matrices, the server can effectively use this global covariance to initialize global classifier. Our experiments demonstrated that FedCOF outperforms FedNCM [29] by significant margins while maintaining the same communication cost. Additionally, FedCOF delivers competitive or superior results to Fed3R [II] across model architectures and benchmarks while substantially reducing communication costs. Finally, we showed that FedCOF outperforms federated prompt-tuning methods and serves point for improving the convergence federated fine-tuning and linear probing methods.
more effective starting
The quality of our estimator depends on the number of clients, shown in Figure S where using multiple class means per client helps with fewer clients. Another limitation is the assumption that samples of the same class are iid across clients, which is, however, an assumption underlying most of federated learning. We discuss the bias in our estimator non-iid settings in Appendix G
Limitations.
10

---

Acknowledgements. We acknowledge project PID2022-143257NB- TTL.///E//10.1303////0 10001 1033 and ERDF/EU and FSE+, funding by the European Union ELLIOT project, and the Generalitat de Catalunya CERCA Program. Bartlomiej Twardowski and National Centre of Science (NCN, Poland) Grant acknowledges the grant Kai Wang acknowledges the funding from Guangdong and Hong Kong No. 2023/1/D//T
financed by
Universities I+1+1 Joint Research Collaboration Scheme and the start-up grant B01040000108 from CityU-DG, Andrew Bagdanov acknowledges financial support from the AVIOS project, financed by the Italian Ministry of Enterprises and Made in Italy (MiSE/MIMIT), We acknowledge all reviewers, especially anonymous reviewer (Md8N) whose constructive feedback restored our belief in the review system, which had been severely shaken in earlier review rounds.
References
[1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint 1rX1v:2111.04263, 2021.
[2] Mahdi Beitollahi, Alex Bie, Sobhan Hemati, Leo Maxime Brunswic, Xu Li, Xi Chen, and Guojun Zhang. Foundation models meet federated learning: A one-shot feature-sharing method with privacy and performance guarantecs. Transactions on Machine Learning Research, 2025.
[3] Keith Bomawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. arXiv preprint arXiv: /6/ 1.04822, 2016.
[4] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[5] Debora Caldarola. Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by seeking flat minima. In European Conference on Computer Vision, pages 654-672. Springer, 2022.
[6] Hong- Houg-You Chen, Cheng-Hao Tu, Ziwei Li, Han Wei Shen, and Wei-Lun Chao. On the importance and applicability of pre-training for federated learning. In The Eleventh International Conference on Learning Representations, 2022.
[7] Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 Icce, 2009.
[8] Xin Dong, Sai Qian Zhang, Ang Li, and HT Kung. Spherefed: Hyperspherical federated learning. In European Conference on Computer Vision, 2022.
[9] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers image recognition at scale, In International Conference on Learning Representations (ICLR), 2021.
[10] Cynthia Dwork, Frank McSherry, Kobhi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin, editors, Theory of Cryptography, pages 265-284, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
[11] Eros Fani, Raffaello Camoriano, Barbara Caputo, and Marco Ciccone. Accelcrating heterogencous federated learning with closed-form classifiers. Proceedings of the International Conference on Machine Learning, 2024.
[12] Jerome H Friedman. Regularized discriminant analysis. Journal of the American statistical association. 84(545)).65-175. 1989.
[13] Benyamin Ghojogh and Mark Crowley. Linear and quadratic discriminant analysis: Tutorial, arXiv preprint arXiv:1906.0 025900 2019.
[14] Dipam Goswami, Bartlomicj Twardowski, and Joost Van De Weijer. Calibrating higher-order statistics for few-shot incremenntee learning with pre-trained vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[15] Kaiming He, Ross Girshick, and Piotr Dollar. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
11

---

[16] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In International Conference on Computer Vision (ICCV), 2021.
[17] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribu- tion for federated visual classification, arXiv preprint 19X9:v:909..06330 2019.
[18] Tzu-Ming Harry Hsu, Hang Qi. and Matthew Brown, Federated visual classification with real-world data distribution. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28. 2020, Proceedings, Part X 16, pages 76 (6.9 Springer, 2020.
[19] Forrest N. landola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <0.5mb model size, 2016.
[20] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurdlien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems federated learning. Foundations and trends@ machine learning, 14(1 (1:1-2):1-1100 2021.
[21] Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Schastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Inter- national Conference on Machine Learning, pages 5132-5143, PMLR, 2020.
[22] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5132-5143. PMIR, 13-18 2020.
[23] Scongyoon Kim, Minchan Jeong, Sungnyun Kim, Sungwoo Cho. Sumycong Ahn, and Se- Young Yun. Feddr+: Stabilizing dot- dot-regression with global feature distillation for federated learning. arXiv preprint arXiv:2406.02233 2024.
[24] Seongyoon Kim, Gihun Lee, Jaehoon Oh, and Se- Young Yun. Fedfa: Feature normalization for alleviating data heterogeneity problem in federated learning. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurlPS 2023, 2023.
[25] Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distrihuted machine learning for on-device intelligence. arXiv preprint 11Xiv:1610.00527, 2016
[26] Jabin Koo, Minwoo Jang, and Jungseul Ok. Towards robust and efficient federated low-rank adaptation with heterogeneous clients. arXiv preprint arXiv:2410.22815, 2024.
[27] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In International Conference on Computer Vision (ICCV-W) Workshops, 2013.
[28] Alex Krizhevsky, Learning multiple layers features from tiny images. University of Toronto Technical Report, pages 32-33, 2009.
[29] Gwen Legate, Nicolas Bernier, Lucas Caccia, Edouard Oyallon, and Fagene Belilovsky. Guiding the last layer in federated learning with pre-trained models. In Advances in Neural Information Processing Systems, 2023.
[30] Gwen Legate, Lucas Caccia, and Eugene Belilovsky. Re-weighted softmax cross-entropy to control forgetting in federated learning. In Proceedings of The 2nd Conference on Lifelong Learning Agents, 2023.
[31] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith, Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
[32] Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data, arXiv preprint 110::1907.02188 2019.
[33] Xiaoxiao Li, Meirui JING, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. In International Conference on Learning Representations, 2021.
[34] Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages
2023.
12

---

[35] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang. Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 2021.
[36] Simone Magistri, Tomaso Trinci, Albin Soutif, Joost van de Weijer, and Andrew D. Bagdanov. Elastic feature consolidation for cold start exemplar- free incremental learning. The Twelfth International Conference on Learning Representations. 2024.
[37] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficl learning of deep networks from decentralized data. In Artificial intelligence and stafistics, pages 1273-1282. PMLR, 2017.
[38] John Nguyen, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where to begin? exploring the impact of pre-training and initialization in federated leamnigg. In The Eleventh International Conference on Learning Representations, 2023.
[39] Jachoon Oh, Sangmook Kim, and Se- Young Yun. Fedbabu: Towards enhanced representation for federated image classification. arXiv preprint arXiv:2106. 06042, 2021.
[40] Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEF/CVF international conference on computer vision, pages 1406-1415, 2019.
[41] Liangqiong Qu Yuyin Zhou, Paul Pu Liang. Yingda Xia, Feifei Wang. Ehsan Adeli, Li Fei-Fei, and Daniel Rubin. Rethinking architecture design for tackling data heterogeneity in federated learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2022.
[42] Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2020.
[43] Yichen Ruan, Xiaoxi Zhang. Shu-Che Liang, and Carlee Joe- Loe-Wong. Towards flexible device participation in federated learning. In International Conference on Artificial Intelligence and Statistics, pages 3403-3411. PMLR, 2021.
[44] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[45] Aliaksandra Shysheya, John F Bronskill, Massimiliano Patacchiola, Sebastian Nowozin, and Richard E Turner. Fit: Parameter efficient few-shot transfer learning for personalized and federated image classifica- tion. The Eleventh International Conference on Learning Representations, 2022.
[46] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[47] Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Federated learning from pre-trained models: A contrastive learning approach. Advances in Neural Information Processing Systems, 2022.
[48] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongic. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769-88 (7798877,,777 2018
[49] John Van Ness. On the dominance of non-parametric Bayes rule discriminant algorithms in high dimensions. Pattern Recognition, 1980.
[50] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
[51] Jianyu Wang. Zachary Charles. Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivar, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide federated optimization. arXiv preprint arXiv:2107.069/ 2021.
[52] Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective incon- sistency problem in heterogeneous federated optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 761 1623. Curran Associates, Inc., 2020.
13

---

[53] Jie Wen, Zhixia Zhang, Yang Lan, Zhihua Cui, Jianghui Cai, and Wensheng Zhang. A survey federated learning: challenges and applications. International Journal of Machine Learning and Cybernetics. 14(25551-5535, 2023.
[54] Pei- Yau Weng, Minh Hoang, Lam Nguyen, My T Thai, Lily Weng, and Nghia Hoang. Probabilistic federated prompt-tuning with non-iid and imbalanced data, Advances Neural Information Processing Systems, 37:81 77788193--88 81958, 2024.
[55] Chen Zhang. Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge- Based Systems, 216: 106775, 2021.
[56] Yue Zhao, Meng Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00588 2018.
[57] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances neural information processing systems, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We state main contributions in the abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the proposed estimator in Section 6 We exten- sively analyze them in Appendix G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide full derivations and assumptions for our propositions. including
proofs in Section
Section 4.3 Appendix
B
Appendix
[
Appendix D and Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all experimental details needed to reproduce the results Section 5 Appendix H and Appendix J
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code will made publicly available in Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/ubblicgguuueee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all training and test details of the datasets in Section 5 Appendix and Appendix L
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Section
We report the standard deviations of the accuracy values our experiments in
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention the GPU resources used for our experiments in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have carefully checked and ensured compliance with the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the impact of the proposed method in Appendix M
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work involves no release of models or data that pose a risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite existing assets and respect their usage licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This work does not involve the release of models datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing was involved in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects were involved and the work requires no IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM for what should or should not be described.
(https: ottp:ea///eeriipcccc/c//ofeerers
21

---

Appendix
A Scope and Summary of Notation
These appendices provide additional information, proofs, experimental results, and analyses that complement the main paper. For clarity and convenience, here we first summarize the key notations used throughout the paper:
N: total number of samples.
K: number of clients.
C: number of classes.
d: dimensionality of the feature space.
Nk,c: number of samples from class assigned to client At
=o= = =N=Eken Nk.c: total number of samples in class
Pg.Pe E Rd. empirical global mean and class mean for class c, respectively.
Me E RR:: population mean of class c.
Se TRdxd, empirical sample covariance for class c.
De E Rdxd, population covariance for class c.
De E Rdxd; our unbiased estimator of the population covariance Eg employing only client means.
E RdxN; feature matrix, where each column FJ IRd is a feature vector, for = I....,N.
Fv k,c Rd: j-th feature vector from class assigned to client
Fi.c E IRd: sample mean of the feature vectors for class e on client k; treated as a random vector. specific realization of this random vector is denoted by Pk.c-
Var = Cov(F Fr.c
F ke represents the covariance matrix of the random vector Fk.o-
B Expectation and Variance of the Sample Mean
Let ( 18k. random sample from a multivariate population with mean (4 and covariance Σ., where Pi,e is the j-th feature vector of class assigned the client k and ngg,o is the number of elements of class e in the client k. Assuming that the per-class features k,c. in each client are iid in the initialization, then the sample mean of the features for class
FR.e
F
(9)
Tkk.c
distributed with mean and covariance given by:
De
E k..] = Me
Var[Fu,c]
(10)
N.K.c.
Proof. To prove this, we fix the class and omit the dependencies on for simplicity. Thus, we write Ak..c = Ug. F)_ k.c F.. Fk.-- FR. and p. n, Σ. = Σ.
Since (F 1 is a random sample from a multivariate distribution with mean μ and covariance Σ, and the per-class features Fę in each client i.i.d at initialization, it follows that:
E E
Var[F]]= =, Vj
(11)
22

---

By computing the expectation of F& and using the linearity of expectation, we obtain:
1 T
1
1
E
B[F
+....
E
E
(naf) He
Tkk
n
Tkk
where the last equality we used Equation which completes the first part of the proof.
Thus the expectation of the sample mean is μr,
Next, we show that the variance of the sample mean is Σ, By computing the variance of F5 and using the fact that the variance scales by the square of the constant, we obtain:
nk
Var[Fu] Var
(Var[Fi)) +.... Varle
Cov(Fi F2).
j=1
j
By the independence assumption of {F 1 the cross terms Cov Fiz Fx = 0 for i + j. Apply-
ing Equation [T], have:
1 [Var[Fg] + Var[Fr"])
1
Σ
Var
(n₂
ngg
ng
71
C Proof of Proposition
Proposition 1. Let K be the number of clients, each with Tik.c features, and let C be the total number of classes. Let jic Er F5 be the unbiased estimator the population mean μe and Ne =E Du.c. be the total number features for single class. Assuming the features for class are iid across clients at initialization, the estimator
Σ. E--11
PF. (.)(FR -Ao)
(12)
1-
k=
an unbiased estimator of the population covariance Ee for all I....,C.
Proof. To prove this, we fix the class e and omit the dependencies on e for clarity. So we write Akk.c Fr.c = FR. Nº N, Ac f., E E, μ., and E. = E. By the definition of an unbiased estimator, we need to show that:
K n (F A) - p)) =
E
K
By the linearity of the expectation, the definition of sample mean F = 1 Ej=
F2. and the
definition of global class mean fi = N
E
Th j=
F
we have:
K Σ
BPJ
mE
E
BBE
+ Σ
ny BuEi
K
&
Rm
K
F.FFF.FF 2E
DOR Σ
K
1
Σ
-
n.F.FFFF
2NE
Σ nu EnEl
(13)
K
By applying the variance definition, along with the expectation and variance of the sample mean (see Equation (10)) we obtain:
Σ + 4u
E F Var F
=
(14)
7
23

---

Now, by considering the right term in Equation 13 since fi is an unbiased estimator of the population mean, then Elp] = μ.. Moreover, since we assume that the features for single class across clients are i.i.d at initialization, we can use re-use the result in Equation (TO) by considering the all class features as random sample of size N from population with mean y and variance Σ. Consequently, the global sample mean is has variance Varlid]
Σ Then
N
Σ
Blap
Var [
E
pp
(15)
By using Equation obtain: 14) and Equation 15 in Equation and recalling that N Ex=1 Tkk we
K
K
Σ + pt mF²) ) 2N( N
Σ + pt mTT)
T& 7
+ up pp)) Σ
ny
K
N
km
2
1
K (KE + up T N 2 - 2N + + um [N)
(K - 11= E.
k-
K
D Proof of Proposition? 2
Proposition 2.
Let E RaxN be a feature matrix with empirical global mean By E Rd and
E RNC be a label matrix. The optimal ridge regression solution W = (G Xaa
(a))-1 -B, where
B E RRdx and E Raaxd can be written in terms of class means and covariances as follows:
B= 3-IoNNle-1
(16)
C
G= G-Σ(N- (Ne - 1)S + Σ
NolPc Agg (b)) + N Nigh
(17)
c=1
c=1
where the first two terms Ee (N. 1)5. and SemiNolpee pg) Ag) represents the within-class and between class scatter respectively, while Peo. Se and No. denote the empirical mean, covariance and sample size for class respectively.
Proof. The first part, regarding Equation (16). follows directly. From the ridge regression solution, B FY, which is obtained by summing the features for each class and arranging them into the columns a matrix. This results in the product of class means and samples per class.
Now, for computing the matrix G, we proceed with the definition of the global sample covariance:
$ = N
(F
F) - F) N
[FF OFFT-FF - FF +FFF
where F = 1/2 E Fd) 1 = - E RdxN is the matrix obtained by replicating the global
mean N times in each column and 1 E Nx1 is column vector of ones. Recalling that G = FFT, we have:
S N - (G (G- F1 111 FT + 1111)) = N - (G - -F11 + NN
since FI B1 TFT and 1 N.
Now, since =1= = Eiz i Ps, we can obtain the matrix G as:
N
G = (N-1)
Σ Fd
No TN--NPpPT = (N- (N- - (N (N-1) N.SS+NNi 2g
(18)
It is well known result that the global covariance can be expressed as:
Σ (Ne - 1758 +
No - Ag) - kT) pa)²)
N
24

---

Replacing the global covariance S in Equation
18 we obtain the final expression for G as:
G= NN. 1)8. +
NelPe Pg) (g)) + Noglg
c=1
E
Sampling Multiple Class Means per Client
From a theoretical perspective, although the estimator Σ. is unbiased in Equation (5), this only ensures that expected value equals the true covariance Σg - not that any individual estimate is accurate. Its variance still depends on the number of independent means K sampled that is the number of clients in the federation K which contains class c.
Therefore, increasing K leads to tighter concentration of Σ. around its expectation, reducing the overall mean square error (MSE) the sum of variance and squared bias (which remains zero since the estimator is unbiased) - between the estimator and the true covariance matrix. Moreover, where the number of clients is small relative to the feature dimension the estimate may be ill-conditioned. The shrinkage term yLa in Equation (6) improves numerical stability in these settings, trading a small amount of bias for significant reduction variance even when only few clients are available.
Sampling strategy. We propose using a simple multiple mean sampling strategy following sampling without replacement. We consider the number of means M to sample fixed number which is a hyperparameter. For each class, we take disjoint random sets from fgg,c samples in a client and compute the mean for these subsets. We take disjoint The only condition we enforce is that clients use atleast samples to compute a mean. If client does not have atleast 2. samples, we send less than M sample means. For instance, if client has samples, we compute and share single mean.
to avoid computing similar sample means.
We observe that the proposed sampling approach improves performance. However, more sophisticated sampling approaches could be employed if the user is interested in improving performance FL settings with very few clients. One approach could be sampling means based on the number of samples ni,e instead of using fixed number means to sample from every client. If client has more samples, it could share more sample means and this would thus share more means overall and improve the covariance estimates. Future work could optimize this multiple mean sampling approach better suit fewer client FL settings.
F On Excluding Between-class Scatter
Intuitively, to represent the feature distribution of each class we do not really need to consider the relationships between different classes which represent the distribution of the overall dataset since our goal is to estimate the class-specific classifier weights using these covariances. Based on this intuition, we propose remove the between-class scatter from Equation and initialize the classifier weights using only the respective within-class scatter matrices.
We also analyze in Table Jublcclussig usiing the centralized setting how the different scatter matrices affect over- fitting of the model. We observe that, while both methods achieve similarly high training accuracies, Ridge Regression consistently underperforms on the test set. This suggests that incorporating Goo introduces an overfitting effect, as the classifier learns directions that do not transfer well to unseen samples.
Finally, we also empirically analyze in Table 7 the impact of removing the between-class covariances in Federated scenario using SqueezeNet. Here in the FL setup we again clearly see the negative effect of incorporating between-class scatter statistics.
G Bias of the Estimator with non-iid Client Features
In Appendix C we showed that, under the assumption that the per-class features are iid across clients, the proposed estimator is an unbiased estimator. In this section, we theoretically quantify the bias when the i.i.d assumption is violated.
25

---

Table Comparison of classifiers across datasets with and without Giww in the centralized setting
Classifier
Guith
Ghew
Dataset
Train Acc
Test Acc
Ridge Regression
CIFAR-100
60.0
57.1
Ours
CIFAR-100 100
60.4
57.3
Ridge Regression
7
Imagenet-R
52.8
37.6
Ours
x
Imagenet-R
53.4
38.6
Ridge Regression
CUB200
92.0
50.4
Ours
x
CUB200
91.3
53.7
Ridge Regression
Cars
85.9
41.4
x
Ours
Cars
86.2
Table 7: Performance comparison of different FL setups using Ghiw and Gwith across datasets.
HI. Setup
CIFAR-100
ImageNet-R
CUB200
CARS
Using Guu
52.8
34.8
49.7
33.7
Using Ghtw Gwith
56.3
36.8
51.6
42.4
Using Gwith (Ours)
56.3
37.2
53.5
44.6
Under the i..dd assumption, the single class features assigned to clients be treated as random samples from the same population distribution with mean ja and covariance Σz. For simplicity, focusing on a single class and dropping the class subscript c, the population distribution has mean μ and covariance E. As result, recalling Equation (I4), we can write:
Σ
F R
Var
+ u4
where ni the number of samples assigned client k, and Fx is the sample mean client k
Now, if the i.i.d assumption is violated the local features assigned to each client can be viewed as random samples drawn from different client population distributions, each characterized by a mean My and covariance Σg. with # μj and Σ; * Σg for i * j. and i,j 1...., K. In this case:
Eg n
E Var Var F
part
(19)
To compute the expectation of the estimator E[L) we follow the same procedure used to prove proposition in Appendix C up to Equation
E
Σ
TFEFF F 2NE
Σ ny m
(20)
K
Assuming the global feature dataset, regardless of client assignment, is a random sample from the population with mean y and covariance Σ, we can write:
Epp
Var M
EM
pp
(21)
26

---

Substituting Equation 21 and Equation we obtain:
(19) into Equation (0).
and recalling that N = -E k= 7/,
E Σn mKPEE 2N +
Σ + mnx))
B
Th
mM m
+
TAk
K
N
N
k=1
1
Σx
+ maMLTe
-E -E-NupT
Tol:
1kkk
K
K
K
(EN
21
Σ
The PEPERE - - napu
K
K
K
k=1
k=1
K
Σ
1
(2)
+
The MELUEME -mp
K
K
K
k=
k=1
K
Σ
1
E
Σ
nE(UE 10) 14)
K
K
where in the last step we used that Σ
Ek=l
K
nEjiK N₁.
The bias of the estimator is thus given by:
- ()T) Σ na -m)(u -
Bias ( -E[[] - E K
Σ k=1
E E Σ)
(22)
K
Note that if each client population covariance DA is equal to the global population covariance M, and the mean of each client [4 is equal to the population mean, then the bias zero (i.e., the estimator is unbiased). However, the bias formula reveals that when the distribution of a class within a client differs from the global distribution of the same class, our estimator introduces systematic bias. This situation can arise in the feature-shiff setting, in which each client is characterized by different domain. We next evaluate FedCOF under the feature-shift setting to quantify how this bias affects performance in this specific scenario.
G.1 Experiments on Feature Shift Settings.
we perform experiments with MobileNetv2 in non-iid feature shift setting on the DomainNet [40] dataset. DomainNet contains data from six different domains: Cli- part, Infograph, Painting, Quickdraw, Real, and Sketch. We use the top 10 most common classes of DomainNet for our experiments following the setting proposed by [3) We consider six clients where each client has i.i.d. data from one of the six domains. As result, different clients
Following 33
Table 8: Comparison of different training- free methods using MobileNetV2 on the feature shift setting on DomainNet. We show the total commu- nication cost (in MB) from all clients to server.
Method
Acc (f)
Comm, ())
FedNCM
65.8
0.3
Fod3R
81.9
39.6
TedCOP
74.1
0.3
FedCOP (2 class means client)
76.5
0.6
FedCOF (10 class meuns per client)
have data from different feature distributions. We show in Table Rable&hee hoo training- methods perform in feature shift settings and the accuracy communication trade-offs.
Fed3R achieves better overall performance then FedCOF, likely due to its use exact class covariance, avoiding the bias that FedCOF introduces. However, FedCOF achieves comparable results while significantly reducing communication costs. FedNCM perform worse than FedCOF at the same communication budget. When we increase the number of means sampled from each client, the performance of our approach improves. This due to the fact that our method suffers with low number of clients (only 6 in this experiments) and sampling multiple means helps.
While non-iid feature shift settings have been studied some papers not using pre-trained models, using this setting with pre-trained models works a bit differently. When using a pre-trained model, the generalization capabilities of the pre-trained model can help in moving the distribution of class features across clients towards an iid feature distribution even if the class distribution across clients
27

---

is non-iid at the image level. We believe that more comprehensive analysis of feature-shif shift settings when using pre-trained models requires more extensive benchmarks with higher number of clients and could be an interesting direction to explore in future works.
H Communication Costs
When computing communication costs we consider that the pre-trained models are on the clients similar to [1]] and do not need to be communicated. All parameters are considered to be 32-bit floating point numbers (i.e. 4 bytes) in all our analyses and experiments.
For training-free methods, each client sends its local statistics the server only once, in 29 In the multi-round federated learning setting, this can be efficiently implemented by ensuring on the client-side that each client only shares its local statisties during first participation round. This avoids repeated communication of statistics from clients for all training-free methods. Following [. the communication from server to client is not considered since the clients does not need to receive any updates from the server as the client models are not updated in the training- faiiing--free initialization of the global classifier. Thus, the communication cost is the same for a single round setting with full client participation and the setting with multiple rounds having partial client participation in each round.
Let Cg denote the number of classes present in client Due to the non-iid Dirichlet data distribution, C C. where C' is the total number of classes in the dataset. As result, the communication cost varies across clients. Defining M = Σ Ck the total number of class means shared from the K clients and assuming that each class mean is a d-dimensional feature vector, the communication cost of each evaluated approach is given by:
FedNCM shares total of M means from K clients. Cost Md-
Fed3R shares a total of M sum of class features and total of K matrices of dl dimension- ality from K clients. Cost = (Md + Kd)) -32.
FedCOF shares total of M means from K clients. Cost = -d-32.
CCVR and MaadOF--rrckke shares a total of M means and M covariance matrices of d2 dimensionality from K clients. Cost M (d + ) -32.
For the non-iid sampling in experiments reported in Table the values of M for CIFAR-100, ImageNet-R, CUB200, Cars and 1N0K are 2870, 3470, 2353, 2634 and 54590, respectively.
Instead, for federated finetuning, the communication cost is calculated as follows. Let E, C and H = d. C' denote the sizes of the feature extractor, the total number classes and the classifier head, respectively. Assuming that
K is the number of clients participating in each round and T is the total number of communication rounds, the communication cost for federated fullratedfull-fnnettunne full-finetuning is 2 (E + d C) T s- 32 and the cost for federated linear probing will be 2 d C- --
For federated prompt-tuning methods [54 considering the classifier weights = dC, T rounds of communication, and clients per round, the communication costs are:
FedAvg-PP PT and FedProx-PT share the classifier weights H = d.C and the fixed size prompt
pool P for each client of size d at each round. Cost 2. (d C + P..) .Pdd T 32.
PFPT shares the classifier weights H d C' and variable size prompt pool P, for each client each round Considering the sum of all P. across all T rounds as Σ" P. the communication cost is 2 (d C' T + ΣT_ P.) 32.
For the experiments reported Table 4 we follow the same training settings as and train for 120 rounds with 10 clients participating per round. We use a fixed prompt pool size P = 20 for FedAvg-PT and FedProx-PT and variable prompt pool size which is optimized over rounds for PFPT. The sums of the variable prompt pool sizes ΣT= P. across all rounds for CIFAR-100, ImageNet-R, CUB200 and Cars 1777, 5205, 4736 and 4736 respectively.
Using PFPT 54 on CIFAR-100, the prompt pool size starts from 20 and ends 10 after 120 rounds leading improved performance and reduced communication compared to FedAvg-PT and
28

---

Uniform w/ Real Classmise Statistics
+ w/ Noiey Elassw Statisties
Gaussian Noise
Laplacian Noise
Text Accurwcy
Figure 6: Performance of FedCOF with noisy class statistics CIFAR- using SqueezeNet. The number of clients is fixed at 100 and classes are distributed using a Dirichlet distribution with a = 0.1. Results averaged over five random seeds, each generating different noise in client statistics, and the standard deviation is reported. FedCOF demonstrates robustness to uniform, Gaussian, and Laplace perturbations in class statistics, with performance showing slight drop noise, parameterized by e, increases. Lower corresponds to higher noise levels in the class statistics.
FedProx-PP PT. This is consistent with the results from [4]. However, we observe in our experiments on out-of-distributin datasets like ImageNet-R and fine-grained datasets like CUB200 and Cars that the prompt pool size increases over rounds leading to increased communication costs with respect to FedAvg-PT and FedProx-PT. We also in Table that all existing prompt-tuning methods performs very poorly on fine-grained datasets like CUB200 and Cars.
I Improving Privacy Preservation in FedCOF
In this section we discuss additional privacy techniques to prevent the sharing of client class-wise count statisties and class means.
I.1 Adding Random Noise to Protect Class-wise Statistics.
Our method requires transmitting class-wise statistics to compute the unbiased estimator of the popu- lation covariance (Equation (12) and classifier initialization, similar other methods in federated learning [29 335) In general, transmitting the class-wise statistics may raise privacy concerns, since each client could potentially expose its class distribution. Inspired by differential privacy [TO), propose perturbing the class-wise statistics of each client with different types and intensities of noise, before transmission the global server. This analysis allows us evaluate how robust FedCOF is to variations in class-wise statistics and whether noise perturbation mechanisms can effectively hide the true client class statistics. Specifically, we propose perturbing the class-wise statisties follows:
Uk.c max (nk,c + 0 moise
(23)
where ofoiiee is noise added to the statistics, and e a parameter representing the noise intensity. The max operator clips the class statistics to zero if the added noise results in negative values, which is expected to happen in federated learning with highly heterogeneous client distributions. When clipping is applied, the client does not send the affected class statistic and class mean, and the server excludes them from the computation of the unbiased estimator.
We consider three types of noise:
Uniform noise: o u(-(1 e) +(1 e)ma,c), proportional to the real class
statistics.
Gaussian noise: de
gauss
NO0, 2), independent of the real class statistics.
Laplacian noise oe L(0, Ł), which is also independent of the real class statistics.
Japlace
Lower values correspond to higher levels of noise in the statistics.
In Figure 6 we show that the performance of FedCOR robust with respect to the considered noise perturbation, varying the intensity of e {0.1, 0.3, 0.5, 0.7, 0.9). These results suggest that
29

---

Uniform Noise
Laplacian Noise
120
Vatey Cesaay se Trequencies with
Sext Clessaice Trequencies
No Clecsotee Trequenctes with c 0.1
Real Clecs or frequences
80
60
20-
EP28
Clasass
Cosses
Figure 7: Class frequency distributions for a single client under different noise types: uniform noise (left) and Laplacian noise (right) on CIFAR-100. Both noise types are applied to the real class statistics with the highest noise intensity (c = 0.1). The bar heights represent the average class frequencies, and the error bars indicate the standard deviation across 5 seeds. Real class-wise frequencies and their noisy counterparts are shown for comparison.
differential privacy mechanism can be implemented to mitigate privacy concerns arising from the exposure of client class-wise frequencies. In Figure 7 we provide qualitative overview of how the proposed Laplacian and uniform noise perturbation affect class-wise distributions.
1.2 FedCOF with Secure Aggregation Protocols
To incorporate secure aggregation protocols with FedCOF, we propose use pairwise masking between clients following [3]. Each client i generates random noise vector 2i,j for every other client j. Then, each of these clients computes perturbation vector Pi,) = 2z,j 2j.1. Similarly, Pj.i Zj.a 20,j resulting in P5. Pi.j. These perturbation vectors are added to the client statistics before sharing them to the server. When all client statisties are added in the server, the noise cancels out across clients and the server can use the aggregate statistics.
To such a secure aggregation protocol with FedCOF, the communication cost will increase and become similar to Fed3R. We proceed in two steps:
Secure aggregation class means and counts. this phase, each client sends only class means and counts, adding perturbation vectors to the means and perturbation scalars to the counts to ensure privacy. The server then computes the global class means from the aggregated statistics and sends them back to the clients. Specifically each client l sends:
Ug.c + fkk.c
Ukec
Pk + Mk,c X Tkk.c
where The and Hkec are the original class counts and means; Ph = ΣieK Pk. is the total perturbation vector for client k and Ek Pk = 0 (similarly qu is perturbation scalar such that K 9h 0).
The server aggregates these quantities, and since the perturbations cancel out, it can compute:
Uk.c ΣR Uk.c
Me
No
uk.es
k=1
and sends (Pc, NN) back to all clients.
2. Secure aggregation of class-covariance terms. Each client computes
Sk.c (No 1)nk, Mc)(ue. Me)
30

---

and forms
Sk =
Σ sk.c Ma.
where Mr is random matrix such that Σ M& =
After receiving all Sk. the server sums them, and since the noise terms cancel out, obtains:
K
K
c
Σ
S
(N. - 1)mp,c(uu,e Me) - Mc)
k=
k=
The final global matrix is then estimated as:
K
Σ
Sk
K 1)7ta + NNsll
K
-
c=1
1
Σ (No
-Ma)
K
1 1)-a+ N
K-1
K-
\==
Σ (No - 1)
K nn.oluRo Mo) uo) + yla
+ NN
where N = -e No and Mg Lng Me This results in the exact formulation of the matrix G as computed in the proposed FedCOF.
Although this increases training-free communication costs which becomes equivalent Fed3R, FedCOF offers benefits in terms of accuracy and faster convergence which also reduces overall communication costs when fine-tuning after FedCOF initialization.
Dataset and Implementation Details
Datasets. We use the following five datasets in our paper:
CIFAR-100 has 100 classes provided in 50k training and 10k testing images.
ImageNet-R (IN-R) is composed of 30k images covering 200 ImageNet classes. ImageNet- R [16] is out-of-distribution dataset and proposed to evaluate out-of- -itt----istiiutionn gen- eralization using ImageNet pre-trained weights. It contains data with multiple styles like cartoon, graffiti and origami which is not during pre-training.
CUB200 is fine- grained dataset and has 200 classes of different bird species provided in 5994 training and 5794 testing images.
Stanford Cars 196 classes of cars with 8144 training images and 8041 test images.
[17 real-world, large-scale dataset 48 proposed by [17] for federated learning and contains 120k training images of natural species taken by citizen scientists around the world, belonging to 1203 classes spread across 9275 clients.
INaturalist-Users-
In datasets like ImageNet-R and CARS, we also face class-imbalanced situations where there is significant class-imbalance at the global level.
Implementation Details. Here, we provide details on learning rate (Ir) used for all fine-tuning experiments with FedAdam. For ImageNet-R and Stanford Cars, we a 0.0001 for both server and clients for FedNCM, Fed3R and FedCOF initializations. For CUB200, we use a server Ir of 0.00001 and client of 0.00005 for Fed3R and FedCOF, while for FedNCM, we use higher Ir of 0.0001 for clients. For random classifier initialization with all datasets, we use higher Ir of 0.001 for clients and Ir of 0.0001 for server. We use I local epoch, for fine-tuning experiments on datasets. After training-fiee free classifier initialization, we fine-tune the models for 100 rounds. When starting from random classifier initialization, we train more 200 rounds. When training with FedAvg and random classifier initialization, use client Ir of 0.005 for all datasets other than mnt-110KK
31

---

For the linear probing (LP) experiments for the 4 datasets other than iNat-120K, 120K, with FedAvg we train for 200 rounds with local epoch and use client Ir of 0.01 and server Ir of 1.0 for FedNCM. For Fed3R and FedCOF initializations, we use a client Ir of 0.001 and server Ir of 1.0. For LP experiments on iNat- 10K,, we use 3 local epochs, 30% client participation and train for 5000 rounds. For 1NK, we use client Ir of 0.001 for FedAvg-LP without classifier initialization, a client Ir of 0.0005 for FedNCM and client of 0.00001 for Fed3R and FedCOF.
We use one Nvidia RTX 6000 GPU for all our experiments.
The FedCOF Oracle (Sharing Full Covariances). Similar CCVR [35. aggregate the class covariances from clients as follows:
2C
TRe N
1k. N
.T Pk,c
No
PePe
(24)
N
For the oracle setting of FedCOF, use the aggregated class covariance from Equation (24) and apply shrinkage to obtain Eg yIa and use in Equation (8) instead of using our estimated covariances.
K
Additional Experiments
Adapting CCVR for Classifier Initialization.
Table 9: Comparison of FedCOF with CCVR across different datasets and models.
Dataset
Method
SqueezeNet (d=512)
MobileNetv2 (d=1280)
ViT-B/16 (d=768)
Acc (1)
Comm. (L)
Acc (f)
Comm. (J)
Acc (1)
Comm. (1)
CIFAR-100
CCVR
$7.540.2
3015.3
59 69..2
18823.5
72.3+0.2
6780.0
FedCOF (Ours)
56.1
5.9
63.5+0.1
14.8
73.2+0.1
8.9
IN-R
CCVR FedCOF (Ours)
36.420.2
3645.7
41.920.2
22758.8
49.3=0.2
8197.4
37.820.4
7.1
47.420.1
17.8
51.8.0.3
10.7
CUB200
CCVR
51.240.1
2472.1
61,6#0.2
15432.7
78.740.4
$558.6
FedCOF (Ours)
53.7=0.3
4.8
62.5+0.4
12.0
79.4=0.2
7.2
Cars
CCVR
40.9+0.4
2767.3
36,0+0.4
17275.7
49.440.4
6222.5
FedCOF (Ours)
44.0=0.3
5.4
47.3+0.5
13.5
52,5+0.3
8.1
Since CCVR B5 was originally proposed to calibrate classifiers after training, we adapt it our setting and use it as initialization method. CCVR is similar to the FedCOF Oracle in terms of communication cost as it shares class means and covariances from all clients to the server. CCVR aggregates the class covariances and means to obtain a global class distribution the server, and then trains linear classifier on Gaussian features sampled from aggregated class distributions from clients. We show Ta- ble 2 that the proposed FedCOF outperforms CCVR in most settings despite having significantly lower communication cost.
Table 10: Performance comparison of different
initialization methods followed by FedAdam opti- mization for 100 rounds training.
Method
ImageNet-R
CUB200
Cars
FcdNCM+FedAdam
44.7+0.1
50.2+0.2
48.720.2
CCVR+FedAddm
44.6+0.3
51.5+0.2
47.9=0.1
Fed3RRFedAddam
45.920.3
51.2+0.3
47.420.4
FedCOF=FedAdam
46.0s0.4
55.740.4
49.6:0.6
We also show in Table Table that FedCOF initialization is better for further fine-tuning using a pre-trained SqucezeNet that we finetune with FedAdam for 100 rounds after different initialization methods.
Linear probing after initialization experiments. We show in Figure g that linear probing after FedCOF classifier initialization improves the accuracy significantly compared to FedNCM and is marginally better than Fed3R initialization across three datasets using SqueezeNet.
Comparison of training- free methods with linear probing. We also compare with our approach with the training-based federated linear probing without any initialization (where we perform FedAvg and learn only the classifier weights of models) and show in Table T1 that FedCOF is more robust and communication-effecle compared to federated linear probing across several datasets. We follow the same settings as in Table 3 For first datasets, perform federated linear probing for 200 rounds
32

---

CIFAR1DO SqueseeNee
geaie-Net-- Squarzee Not
CUEZ0O SqueczeNet
Accuracy
Accuracy
Fedleg
Fndinc
TeStCC
TedtiCM Fediing
Potch
30-
Post
Poch Postkg 1L71
Fo83R-Fostig LLS
2P1
15
t2e Rounds
Rounds
Rounds
Figure 8: Analysis of the performance with federated linear probing using FedAvg
Table 11: Comparison of different training-free free methods using SqueezeNet with training-based Fed-LP (federated linear probing with FedAvg (37] starting with pre-trained model and random classifier initialization) across 5 random seeds. FedNCM, Fed3R and the proposed FedCOF does not involve any training. We show the total communication cost (in MB) from all clients to server. The best results from each section are highlighted in bold.
CIFAR-100
ImageNer-R
CUB220
CARS
iNat-120K
Method
(?)
Comm. (.)
(t)
Comm,
Acc
(t)
Comm, (1)
Acc
Fed-LP
59.910.2
2458
37.8.1..
4916
16.820.88
4916
33.1
4817
28.0.10.6
I.x10'
FouNCM
41.520.1
23.8:00.1
7.1
37.8:0.3
1988=0.2
21.2:00.1
111.8
Fed3R
$6.910.11
110.2
37.6.10.2
111.9
50.410.3
109.6
39.9.0022
110.2
32. 120.1
9837.5
FedCOF (Ours)
56.1+0.2
37.8+0.4
7.1
53.7+0.3
44.0-0.3
32.5+0.1
111.8
with 30 clients per round using FedAvg with a client learning rate of 0.01. For iNat-120k, we train more for 5000 rounds.
Impact of using pre-trained models. To quantify impact of using pre-trale pre-trained models performed experiments using a randomly initialized model and show in Table [22 that federated training using a pre-trained model significantly outperforms randomly initialized model using standard methods like FedAvg and FedAdam on CIFAR-10 and CIFAR-100.
Table 12: Impact of using a pre-trained SqueezeNet with different federated learning methods on CIFAR-10 and CIFAR-100. We show the total communication cost (in MB). We train a total of 100 clients with 30 clients per round for 200 rounds in non-iid settings with a Dirichlet distribution of 0.1. When starting from random initialization (no pre-training), we train for 400 rounds.
CIFAR-10
CIFAR-100
Method
Pre-trained
Acc (f)
Comm. (L)
Ace (t)
Comm. (()
FedAvg
X
37.3
74840
23.9
79248
FedAdam
X
60.5
74840
44.3
79248
FedAvg
V
84.7
37420
56.7
39624
FedAdam
85.5
37420
62.5
39624
Experiments with ResNet-18. We perform experiments with pre-trained ResNet-18 in Table I3 For FedAvg and FedAdam, we train for 200 rounds with 30 clients per round. For FedAvg, we train with a client learning rate of 0.001 and server learning rate of 1.0. For FedAdam, we train with client learning rate of 0.001 and a server learning rate of 0.0001. We show that tinettuning after FedCOF classifier initialization for 100 rounds outperforms competitive FL methods like FedAdam (which are trained for 200 rounds) by 2.5% on CIFAR- 100 and 5. on ImageNet-R. The improved performance with FedCOF initialization validates the effectiveness of the proposed method, as it reduces communication and computation costs by half compared to FedAdam and FedAvg and still outperforms them.
33

---

Table 13: Comparison of different training-free free methods using pre-trained ResNet- 18 for 100 clients with training- beining-based federated learning baselines FedAvg [37] and FedAdam [42] starting from pre-trained model. We train for 200 rounds for FedAvg and FedAdam which uses pre-trained backbone and random classifier initialization. FedNCM, Fed3R and the proposed FedCOF do not involve any training. We also show the performance of fine-tuning with FedAdam after classifier initialization. For fine-tuning experiments only train for 100 rounds after initialization. We show the total communication cost (in MB) from all clients to server. The best results from each section are highlighted in bold.
CIFAR-100
ImageNet-R
Method
Acc (t)
Comm. (4)
Ace (t)
Comm. (4)
FedAvg
67.7
538k
56.0
54lk
FedAdam
74.4
538k
57.1
541k
FedNCM
53.8
5.9
37.2
7.1
Fed3R
63.5
110.2
45.9
111.9
FedCOF (Ours)
63.3
5.9
46.4
7.1
FedNCM+FedAAdam
75.7
269k
60.3
271k
Fed3R=FedAdamm
76.8
269k
60.6
271k
FeddOF++FedAdam
76.9
269k
62.2
271k
Imagefiet-R- SqueezeNet
ImageNeet-R- SquesceNet
CIFAR: 0O VIT-8/16
20
20 enu
700-
Atcuracy
clents
dients
Doverea
Number of means shared per class per clint
Number of means shared per class per clent
Number of clinnts
Figure 9: (left) Analysis of FedCOF performance with multiple class means per client Ietge (center) Total number of means per class on average that are used to estimate the covariance for FedCOF in Figure (left). (right) Performance comparison of FedCOF with full, diagonal, and spherical covariance matrix communication.
L Additional Ablations
Sampling multiple class means. We perform multiple class means sampling per client using ImageNet-R and show in Figure 9 (left) that using FedCOF with more class means shared from each client improves the performance. We also show in SigureP[centerr used per class on average in Figure (left) to perform the covariance estimation. The number of means used estimate each class covariance less than the total number of clients due to the class-imbalanced or dirichlet distribution used to sample data for clients. This is due to the fact that not all classes are present in all clients.
the total number of means
In most settings we observe that the largest performance improvement occurs when increasing the number of class means per client in scenarios with fewer clients. For instance, with only 10 clients, accuracy improves from 33.1% % to 37.0% when increasing from 1 to 4 means per client. In contrast, when 50 clients are available, the improvement marginal (from 37.4% to 37.8%). This trend is consistent with the theoretical insights in Appendix E when fewer clients are available, sampling more means per client increases the number of independent statistics, thereby reducing variance and improving the quality of the covariance estimate.
Communicating diagonal spherical covariances. While communicating diagonal spherical covariances (mean of the diagonal covariance) from clients to server and then estimating the global class covariance from them can significantly reduce the communication cost, such estimates of global class covariance is poor compared to FedCOF. We show in Figure 9 (right) that FedCOF outperforms these covariance sharing baselines when communicating spherical or diagonal covariances.
34

---

Impact of Shrinkage. We analyze the impact of shrinkage on the estimated class covariances in FedCOF when us- ing a pre-trained SqueezeNet in Table for our
Table 14: Ablation showing the impact of using shrinkage in FedCOF using a pre-trained SqueezeNet.
Dataset
0,01
0.1
10
We use a shrinkage experiments with SqueezeNet and ViT- B/16. We observe that shrinkage yields marginal improvement for ImageNet-R and a bit more significant improvement in accuracy of 2.59 5% on CUB200. This can be attributed to the few-shot settings where the covariance estimation is not very good due to the lack of data and thus fewer clients having access to each of the classes. The use of shrinkage in FedCOF stabilizes and improves the covariance estimation leading to improved accuracy especially few-shot settings.
ImageNet-R
36.53
36.98
36.96
37.25
36.07
CUB200
51.08
51.07
51.81
53.57
53.50
Severe imbalance settings. We further evaluate FedCOF in settings with more severe class imbalance or heterogeneity across clients (Dirichlet distribution with a 0.05, 0.01) in Table 15 and show that the performance of FedCOF drops bit with very severe heteregeneity expected.
Table 15: Analysis of FedCOF performance across settings with varying heterogeneity.
Class means shared
ImageNet-R (100 clients)
Cars (100 clients)
per client
= 0.5
0.1
0.05
a = 0.01
a 0.5
a
0.1
& 0.05
0.01
38.4
37.3
36.4
33.5
45.0
44.5
43.1
39.9
38.2
37.8
37.4
35.9
44.9
44.5
43.8
42.5
38.1
37.5
37.5
36.7
44.9
44.7
44.2
43.2
38.3
37.7
38.2
37.4
45.1
44.9
44.4
43.6
We show the impact of class imbalance in different settings Table Using the most extreme setting (a 0.01), each class is present on 4 clients on an average. Although the extreme settings are less unrealistic, we analyze and evaluate FedCOF in those set- tings. For instance, the drop in performance of FedCOF on Cars from 44.5 to 39.9 is due to the fact that the global covariance is estimated from around 3.6 sample means instead of around 13.4 sample means. This scenario faces the same issue fewer clients due to the severe class imbalance. Here, we show that sampling multiple class means per client improves the performance in the extreme settings mitigating the bias in these settings.
Table 16: Total means shared per class average (the number clients each class is present in, on average).
Dataset ImageNet-R Cars
a 0.5
a
0.05
a 0.01
36.1
17.4
11.7
4.0
24.5
13.4
9.6
3.6
M Impact Statement
In this paper we propose a highly communication--eficinne method for federated learning which exploits pre-trained feature extractors. Reducing communication between clients and the central server is critical aspect of federated learning to enhance its application in practical scenarios. The proposed method training-fiee and thus does not require extensive training or incur excessive computational costs across all client devices like training-based federated learning methods. Our method drastically reduces communication while achieving similar or even better accuracy compared to existing approaches. The proposed initialization can be used with different federated fine-tuning approaches. We believe that our work will advance federated learning applications and make them more efficient.
35


Reviews:
[{"review_id": "ME0e7lS8Bh", "reviewer": "Reviewer_WU3D", "summary": "The paper addresses training-free federated learning by proposing FedCOF, a method that estimates per-class covariances using the variance of local class means. This approach leverages pre-trained models to extract features and uses these to derive unbiased global covariances, which are then utilized to obtain a ridge regression classifier. This method avoids sending local covariance matrices, thereby reducing communication costs and potential privacy risks. Experimental results demonstrate that the method excels in both performance and convergence efficiency.", "questions": "1. I would like to know if pre-trained models might have negative impacts in certain scenarios. If so, could you please analyze these impacts?\n2. The paper primarily discusses classification methods. I am interested in understanding whether the proposed method has broader implications and if it is applicable to other tasks, such as large language models (LLMs).", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The paper is well-structured and clearly articulated, providing a comprehensive exploration of training-free federated learning using feature extractors. It addresses a pertinent and intriguing problem in the field.\n\n2. The proposed estimator for covariances is both innovative and theoretically sound. The authors have conducted extensive experiments to validate the effectiveness of their approach, demonstrating significant improvements over existing training-free methods.\n\n3. The paper thoroughly considers aspects such as communication cost, convergence, and privacy, reflecting meticulous attention to detail. The derivations are presented in a clear and comprehensible manner, enhancing the paper's accessibility.\n\n\n**Weaknesses:**\n\n1. The method primarily focuses on classification tasks. Expanding the discussion to include its applicability to a wider range of tasks would enhance the paper's impact and demonstrate the broader feasibility of the approach.", "was_revised": true, "final_justification": "After carefully reading the authors' response, the other reviews, and the ensuing discussion, I have decided to update my recommendation. The authors have provided satisfactory answers, especially in response to my two main questions (Potential Negative Impacts of Pre-trained Models, and Generalizability of the Method). I believe the work is solid and meets the bar for acceptance. Therefore, I have raised my score to 5 (Weak Accept).", "rebuttal": "We thank the reviewer for appreciating our paper as well-structured and clearly articulated, with innovative and theoretically sound estimator for covariances, clear and comprehensive derivations, and extensive experiments. We address the reviewer questions below.\n\n> I would like to know if pre-trained models might have negative impacts in certain scenarios. If so, could you please analyze these impacts?\n\nIn our experiments across several datasets and architectures, we have not found any negative effects of using pre-trained models. While the vast majority of works in FL observed several positive effects of using pre-trained models like significantly reducing the impact of data heterogeneity and reducing the communication and computation budget by achieving early convergence, we found one recent work [a1] which shows that using a ResNet18 pre-trained on CIFAR-10 can sometimes have  negative effects in settings with high data heterogeneity. We show in Appendix M that using pre-trained weights is always better across several datasets.\n\n[a1] Kim et al., Fedfn: Feature normalization for alleviating data heterogeneity problem in federated learning. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023.\n\n> The paper primarily discusses classification methods. I am interested in understanding whether the proposed method has broader implications and if it is applicable to other tasks, such as large language models (LLMs).\n\nWe thank the reviewer for the question. Our method is designed for classification tasks, which is the primary focus of the paper. However, we believe that training-free approaches like FedCOF could be extended to LLMs when a classification head is incorporated—for example, in tasks such as multiple-choice question answering with models like LLaMA.\n\nThat said, applying training-free methods to non-classification tasks (e.g., generation or summarization) remains an open challenge. Exploring this broader applicability is a promising direction for future work, especially given the communication efficiency of training-free methods, which could make them appealing in resource-constrained settings."}, {"review_id": "TaXpuP8WZp", "reviewer": "Reviewer_Jmdu", "summary": "This paper introduces FedCOF (Federated Learning with COvariances for Free), a training-free method that leverages pre-trained models and an unbiased estimator of class covariance matrices to initialize the global classifier in federated learning (FL) settings. The key contributions are:\n- Proposing a method to estimate class covariances using only client means, which reduces communication costs and mitigates privacy concerns.\n- Demonstrating state-of-the-art performance on various FL benchmarks with reduced communication overhead compared to methods that share second-order statistics.\n- Showing notable improvements over federated prompt-tuning approaches and effectiveness as an initialization for federated optimization methods.", "questions": "- The paper mentions that FedCOF’s performance can degrade with fewer clients and suggests sampling multiple means per client as a remedy. Could the authors provide more detailed empirical results or theoretical analysis to support the effectiveness of this strategy? \n  - Are there specific guidelines for choosing the number of means to sample per client? \n  - How does the shrinkage factor affect the performance in this case?\n- How does FedCOF handle situations with severe class imbalance or heterogeneity across clients? Are there approaches to mitigate potential bias introduced in non-iid settings?\n- Are there plans to provide an open-source code to facilitate adoption?", "limitations": "The authors have discussed some limitations of the work in the conclusion. I think there is no potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n1. Overall, the paper is well written and provides a solid theoretical foundation with detailed proofs. Experiments are conducted across multiple datasets (CIFAR-100, ImageNet-R, CUB200, Stanford Cars, iNaturalist-Users-120k) to demonstrate the method's effectiveness.\n2. FedCOF eliminates the communication cost associated with sharing second-order statistics by requiring only class means from clients, making it suitable for large-scale FL settings.\n3. The method achieves competitive or superior performance compared to existing methods that share second-order statistics (Fed3R, FedNCM).\n4. FedCOF can serve as an effective initialization for federated fine-tuning or linear probing tasks, leading to better overall performance.\n\n### Weaknesses:\n1. While FedCOF performs well with a large number of clients, its performance may degrade in settings with fewer clients. The authors propose sampling multiple means from each client to mitigate this issue, but more empirical validation is needed.\n2. Although the method is designed to handle non-iid data distributions, the impact of extreme class imbalance and client heterogeneity is not fully explored.", "was_revised": true, "final_justification": "I have decided to maintain my current score and remain supportive of the paper.", "rebuttal": "We thank the reviewer for appreciating our paper as well-written, providing a solid theoretical foundation with detailed proofs, eliminating communication costs making it suitable for large-scale FL settings, with experiments across multiple datasets and an effective initialization method. We address the reviewer questions below: \n\n> The paper mentions that FedCOF’s performance can degrade with fewer clients and suggests sampling multiple means per client as a remedy. Could the authors provide more detailed empirical results or theoretical analysis to support the effectiveness of this strategy? How does the shrinkage factor affect the performance in this case?\n\nFrom a theoretical perspective, although the estimator $\\hat{\\Sigma}_c$ is unbiased (Eq. 5), this only ensures that its expected value equals the true covariance $\\Sigma_c$ -- not that any individual estimate is accurate. Its *variance* still depends on the number of independent means $K$ sampled: if each client contributes one mean, then $K$ is the number of clients in the federation; if multiple means are sampled per client, $K$ corresponds to the total number of means collected across all clients.\n\nTherefore, increasing $K$, for example by sampling multiple means per client, leads to a tighter concentration of $\\hat{\\Sigma}_c$ around its expectation, reducing the overall mean squared error (MSE) -- the sum of variance and squared bias (which remains zero since the estimator is unbiased) -- between the estimator and the true covariance matrix. Moreover, when the number sampled means is small relative to the feature dimension $d$, the estimate may be ill-conditioned. The shrinkage term $\\gamma I_d$ (Eq. 6) improves numerical stability in such settings, trading a small amount of bias for a significant reduction in variance -- even when only a few clients are available. \n\nFrom an empirical perspective, in the main paper in Fig. 6 for CIFAR-100 and Fig. 8 for ImageNet-R, we show FedCOF performance under high heterogeneity and class imbalance (non-iid Dirichlet distribution with $\\alpha=0.1$) when increasing the number of sampled means per client or the number of clients. Below, we report the detailed accuracy results for ImageNet-R: \n\n**ImageNet-R: Accuracy**\n\n|Class means shared per client|10 Clients|20 Clients|30 Clients|40 Clients|50 Clients|\n|--|--|--|--|--|--|\n|1|33.1|34.9|35.9|36.3|37.4|\n|2|35.6|36.7|36.6|37.0|37.4|\n|3|36.2|37.0|36.8|37.2|37.7|\n|4|37.0|37.3|37.3|37.2|37.8|\n\nIn most settings, we observe that the largest performance improvement occurs when increasing the number of class means per client in scenarios with fewer clients. For instance, with only 10 clients, accuracy improves from 33.1% to 37.0% when increasing from 1 to 4 means per client. In contrast, when 50 clients are available, the improvement is marginal (from 37.4% to 37.8%). This trend is consistent with the theoretical insights above: when fewer clients are available, sampling more means per client increases the number of independent statistics $K$, thereby reducing variance and improving the quality of the covariance estimate.\n\n### Impact of shrinkage in fewer client settings\n\nWe show in the table below that using shrinkage (Eq. 6) improves the covariance estimates thereby improving the accuracy:\n\n**ImageNet-R: Accuracy**\n\n|Class means shared per client|Shrinkage|10 Clients||20 Clients||30 Clients||40 Clients||50 Clients|\n|--|--|--|--|--|--|--|--|--|--|--|\n|1|❌|27.1||29.7||32.8||34.5||35.5|\n|1|✅|33.1||34.9||35.9||36.3||37.4|\n|2|❌|30.8||34.0||35.3||36.1||36.7|\n|2|✅|35.6||36.7||36.6||37.0||37.4|\n|3|❌|33.0||35.8||36.2||36.8||37.0|\n|3|✅|36.2||37.0||36.8||37.2||37.7|\n\n\nWe observe that shrinkage consistently improves performance, especially when the number of sampled means is small. This aligns with the theory: when the number of means $K$ is low relative to the feature dimension ($d = 512$ in SqueezeNet), the covariance estimate becomes ill-conditioned, and shrinkage stabilizes it. However, as more class means are sampled per client or the number of clients increases, the benefit of shrinkage diminishes, since the total number of means approaches the feature dimension $d$, making the estimate more stable.\n\n\n> How does FedCOF handle situations with severe class imbalance or heterogeneity across clients? Are there approaches to mitigate potential bias introduced in non-iid settings?\n\nWhile we already use high heterogeneity in all the main experiments of our paper (Dirichlet distribution with $\\alpha=0.1$, following several previous works), here we further evaluate with even more severe class imbalance, and thus heterogeneity, across clients (Dirichlet distribution with $\\alpha=0.05, 0.01$) and show that the performance of FedCOF drops a bit with very severe heteregeneity as expected: \n\n|Dataset (100 clients)|$\\alpha=0.5$|$\\alpha=0.1$|$\\alpha=0.05$|$\\alpha=0.01$|\n|-|-|-|-|-|\n|ImageNet-R|38.4|37.3|36.4|33.5|\n|Cars|45.0|44.5|43.1|39.9|\n\nBelow we show the severity of class imbalance in different settings. Using the most extreme setting ($\\alpha=0.01$), each class is present on 4 clients on average. This is an extreme and unrealistic setting, but still useful to understand the impact of heterogeneity on our approach. The drop in performance of FedCOF on Cars from 44.5 to 39.9 is due to the fact that the global covariance is estimated from around 3.6 class means instead of around 13.4. In the table below we summarize the average number of class means shared for varying $\\alpha$:\n\n**Total means shared per class on average:**\n|Dataset (100 clients)|$\\alpha=0.5$|$\\alpha=0.1$|$\\alpha=0.05$|$\\alpha=0.01$|\n|-|-|-|-|-|\n|ImageNet-R|36.1|17.4|11.7|4.0|\n|Cars|24.5|13.4|9.6|3.6|\n\nThis scenario faces the same issue as fewer clients due to the severe class imbalance. Our strategy for sampling multiple means per client helps mitigate this effect even in this extreme setting: \n\n**ImageNet-R (100 clients): Accuracy**\n\n|Class means shared per client|$\\alpha=0.5$|$\\alpha=0.1$|$\\alpha=0.05$|$\\alpha=0.01$|\n|-|-|-|-|-|\n|1|38.4|37.3|36.4|33.5|\n|2|38.2|37.8|37.4|35.9|\n|3|38.1|37.5|37.5|36.7|\n|4|38.3|37.7|38.2|37.4|\n\n**Cars (100 clients): Accuracy**\n\n|Class means shared per client|$\\alpha=0.5$|$\\alpha=0.1$|$\\alpha=0.05$|$\\alpha=0.01$|\n|-|-|-|-|-|\n|1|45.0|44.5|43.1|39.9|\n|2|44.9|44.5|43.8|42.5|\n|3|44.9|44.7|44.2|43.2|\n|4|45.1|44.9|44.4|43.6|\n\n> Are there specific guidelines for choosing the number of means to sample per client? \n\nWe propose to use a simple multiple mean sampling strategy following sampling without replacement. We consider the number of means $\\mathcal{M}$ to sample as a fixed number which is a hyperparameter. For each class, we take disjoint random sets from $n_{k,c}$ samples in a client and compute the mean for these subsets. We take disjoint sets to avoid computing similar sample means. The only condition we enforce is that clients use atleast 2 samples to compute a mean. If a client does not have atleast $2\\mathcal{M}$ samples, we send less than $\\mathcal{M}$ sample means. For instance, if a client has 3 samples, we compute and share a single mean. \n\nWe observe that the proposed sampling approach improves the performance. However, more sophisticated sampling approaches could be used if the user is interested in improving FL settings with very few clients. One approach could be clients sampling means based on the number of samples $n_{k,c}$ instead of using a fixed number of means to sample from every client. If a client has more samples, it could share more sample means and this would overall share more means and improve the covariance estimates. Future works could optimize this multiple mean sampling approach to better suit fewer client FL settings.\n\n> Are there plans to provide an open-source code to facilitate adoption?\n\nYes, we will provide the open-source code to enable reproduction of all our experiments including experiments on the real-world iNaturalist-Users-120K."}, {"review_id": "mY58tAQXk5", "reviewer": "Reviewer_DZVi", "summary": "This paper proposes a training-free FL framework called FedCOF. FedCOF seeks to compute a closed form ridge regression global solution \nshared across multiple clients, given their siloed data and a pre-trained foundation model. The straight-forward solution to this objective requires sharing second-order statistics that are bandwidth consuming. This paper presents an analysis showing that the closed form solution can be (unbiasedly) estimated using only feature mean from each class of each client. FedCOF is shown to achieve better performance and lower communication cost compared to other training-free and training-based FL solutions.", "questions": "- In the comparison with FedADAM & FedAvg, it was mentioned that these methods were initialized with a random classifier. What is the architecture of this classifier? Do they use a ridge regression head as well, or do they get a standard non-linear classifier (e.g., 2/3-layer feedforward net)?\n\n- Are the clients using the same random initialization? Otherwise, could it be that the gap in performance (i.e., between FedAdam and FedCOF+FedAdam) is caused by the misalignment of initializations?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This is a meaningful and timely problem. I'm quite surprised to see that training-free federated learning performs competitively with training-based approaches. I also recognize the significance of the communication bandwidth issue. Given these factors, I believe the paper makes a valuable contribution.\n- Derivation of the unbiased estimator is sound and novel.\n- Writing is generally clear and understandable.\n- The paper showcases FedCOF's strong performance across a well-chosen set of tasks, highlighting its potential.\n\nWeaknesses:\n- This method seems to require all clients to adopt a ridge regression prediction head, which is where it derives the closed form solution from. I guess this is the same for many previous training-free FL works such as Fed3R. Perhaps the authors could discuss/show empirically what is the upper limit of their method (e.g., is there a task complexity for which the RR head is no longer good enough?)\n- Please see the qn section for specific questions/suggestions regarding the experimental setup.", "was_revised": true, "final_justification": "This paper addresses a timely problem in federated learning and shows that a training-free approach can perform competitively with training-based methods. The derivation of the unbiased estimator is novel, and the experiments are well chosen to highlight FedCOF’s strengths. I am satisfied with the authors' discussion of the limitations in the rebuttal. Overall, the paper makes a relevant and useful contribution and I'm leaning towards an acceptance decision.", "rebuttal": "We thank the reviewer for appreciating our work as a valuable contribution solving a meaningful and timely problem, with sound and novel derivation of the unbiased estimator, clear writing and strong performance of FedCOF across a well-chosen set of tasks. We are pleased to see that the reviewer recognizes the significance of the communication bandwidth issues in FL applications where our method significantly reduces communication costs compared to existing methods. We address the reviewer questions below.\n\n> This method seems to require all clients to adopt a ridge regression prediction head, which is where it derives the closed form solution from. I guess this is the same for many previous training-free FL works such as Fed3R. Perhaps the authors could discuss/show empirically what is the upper limit of their method (e.g., is there a task complexity for which the RR head is no longer good enough?)\n\nThe extreme upper bound would be sharing the real features from clients to server and training a linear classifier (i.e. linear probing) or using a ridge regression head. However, such an upper bound is unrealistic in FL due to privacy concerns. We show another alternative in the form of the CCVR method (see Appendix H) which trains a global linear classifier by sampling features from global class distributions at the server instead of a ridge regression head. We also show in Appendix H how the FedCOF classifier outperforms the linear probing classifier from CCVR in most settings. To the best of our knowledge, among existing works, using a ridge regression head works the best. We believe future work could explore other classifier approaches. Our principal approach of estimating covariances could potentially be used with future methods that exploit second-order statistics.\n\n> In the comparison with FedADAM & FedAvg, it was mentioned that these methods were initialized with a random classifier. What is the architecture of this classifier? Do they use a ridge regression head as well, or do they get a standard non-linear classifier (e.g., 2/3-layer feedforward net)?\n\nFollowing FedNCM and Fed3R, the architecture of the global classifier at the server is a linear classifier (i.e. a single linear matrix) for all methods in the comparison. For training-free classifier initialization methods, we set the weights in the linear layer using the aggregated client statistics (see lines 104-106 for FedNCM; lines 118-119 for Fed3R; lines 214-216 for FedCOF).\n\n> Are the clients using the same random initialization? Otherwise, could it be that the gap in performance (i.e., between FedAdam and FedCOF+FedAdam) is caused by the misalignment of initializations?\n\nYes, all clients use the same random initialization to ensure that there is no misalignment due to difference in initialization."}, {"review_id": "8XxOXJd33D", "reviewer": "Reviewer_Md8N", "summary": "The paper proposes FedCOF, for training-free FL. The goal is, given a pre-trained feature extractor, to set the parameters of the classifier without resorting to standard multi-round gradient based training. Previous methods have done this using feature means or both feature means and outer-products. FedCOF uses both means and outer-products to initialize via a variant of the ridge-regression method proposed by Fed3R but does this without transmitting the outer-product matrices. Instead it estimates the covariance using the standard sample covariance matrix estimator which is unbiased under the assumption that the means are unbiased. It then uses the covariance matrix to recover the outer product matrix, in the process dropping the between class covariants, and computes the ridge regression solution. The paper provides experiments evaluating the performance of the training-free initialization as well as additional experiments examining further finetuning + ablation studies.", "questions": "Am I correct in saying that the server sees each individual client statistic $\\hat{\\mu_{k,c}}$ and $n_{k, c}$?", "limitations": "The authors include a valid discussion of the limitations. For me, however, the biggest limitation, namely the privacy issue of working with unaggregated client statistics, is not adequately addressed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well written. It is clear and easy to follow.\n\n- The methodological contribution is that the client to server communication cost is reduced compared to Fed3R. Additionally, the removal of the between class covariance appears to lead to an increase in empirical performance.\n\n- Theoretical results appear to be sound and the proofs appear to be correct and well written. I would not say the theoretical results are a major contribution, but nor are they claimed to be.\n\n- Experiments are thorough and well conducted. There are a good range of datasets, and test over 3 different realistic models. They additionally conduct experiments showing performance increases when combined with finetuning and linear probing. \n\n**Weaknesses**\n\nMy main concern with the paper is the similarity of the method to Fed3R and that the changes made actually make the method less suited to an FL setting in my opinion. I see three main areas in which FedCOF differs from Fed3R, namely Communication Cost, Privacy and Accuracy. FedCOF makes a tradeoff between privacy and communication that I find unfavourable.\n\nCommunication cost - loosely speaking FedCOF saves a factor of $d$ in the client to server communication compared with Fed3R. However, in this context I do not find this so critical. Firstly, the actual raw per client communication costs are not that high. For Fed3R in table 3 they are approx 1.1MB, 6.7MB and 2.4MB for the 3 different models. For reference the model sizes are approx 4.8MB, 14MB and 330MB, so significantly larger. Of course it does not hurt to reduce the communication cost in FedCOF, however, for these sizes of uploads it is unlikely to be the bottleneck in the first place, rather sampling devices, scheduling on device, waiting for stragglers etc, is far more impactful. Secondly, we are talking about a single round procedure, so optimizing communication is automatically less impactful than if we were running 1000s of rounds of training. Related to this, the paper shows that multi-round full finetuning of the FedCOF initialization leads to better performance. As a user of FedCOF then of course I would do this, and in this case my overall communication cost is anyway dominated by the model transmission in the fine-tuning phase.\n\nPrivacy - As far as I understand, FedCOF requires the server to access each client statistic $\\hat{\\mu_{k,c}}$ and $n_{k, c}$. This is a very significant privacy issue in my opinion and deviates from the normal FL paradigm of working only with across client aggregate statistics. It makes the method incompatible with client-level differential privacy and we cannot use secure aggregation. Essentially, all formal guarantees of privacy are off. In contrast Fed3R only uses aggregates of client statistics on the server and as such is fully compatible with all privacy addons that are crucial in FL. I see there is a brief discussion of this in Appendix P but I do not find this formal enough. There could be a possibility that FedCOF might work in the setting of data-point level DP in a cross-silo setting, but I do not believe client-level DP, as is the standard for cross-device FL, is possible. \n\nAccuracy - The results tables in the paper show that for some settings FedCOF outperforms Fed3R. It appears to me that Table 2 is the explanation for this, i.e. FedCOF benefits by dropping $G_{btw}$. While this is a nice insight, such a modification should also be simple to make for Fed3R as well, while still maintaining privacy by aggregating client statistics on the server.\n\n**Summary**\n\nI find the paper to be well written, with soundly presented (albeit rather minor) theoretical results and a thorough experimental evaluation. My main concern is that the novel contribution is minor and there exists prior work (Fed3R) that is very similar and that has better privacy characteristics in exchange for a slight increase in communication cost, which I would argue is not so important. FedCOF exhibits better accuracy performance in some, but not all settings, but the improvement that leads to this is independent of the main claimed contribution (sharing only the means) and could be easily implemented into Fed3R.", "was_revised": true, "final_justification": "The authors addressed my main concerns. In particular the argument about fast convergence when fine tuning was valid to me. My main concern was the privacy issue of acting on unaggregated client information (when there is a method that is similar and does not do this) but I think it is still a useful contribution to have the proposed method available as it sits on a different part of the communication vs privacy trade off curve. I find the promised addition of the variant of Fed3R that incorporates the papers proposal of removing $G_{btw}$ a useful one that would really strengthen the paper. This method would then have the added accuracy benefits of FedCOF with better privacy guarantees, but at the cost of higher communication. This would provide practitioners with a useful range of methods.", "rebuttal": "We thank the reviewer for appreciating our paper as well-written with soundly presented theoretical results and a thorough experimental evaluation. We address the reviewer concerns below.\n\n> FedCOF makes a tradeoff between privacy and communication that I find unfavourable.\n\nWe appreciate this observation and agree that FedCOF and Fed3R lie at different points on the privacy-communication trade-off curve. Our work explicitly focuses on reducing communication overhead, resulting in a more communication-efficient method that nonetheless achieves competitive—and often superior—performance across a range of experiments. Importantly, federated learning is not one-size-fits-all; practitioners should be able to select algorithms aligned with their specific system constraints and privacy requirements. We believe FedCOF adds valuable diversity to the suite of available methods by offering an effective solution for scenarios in which communication efficiency is critical.\n\n> Communication cost - loosely speaking FedCOF saves a factor of d in the client to server communication compared with Fed3R. However, in this context I do not find this so critical. \n\nNote that the focus of our paper is on *training-free* federated learning using pre-trained models and that our goal is to explore how to do federated learning without sharing client models. Our work aims to achieve similar or better performance with reduced communication without any training or sharing of model parameters. In this context, for the real-world iNat-120K using MobileNetV2, Fed3R has a per-client communication cost of 6.6 MB, while FedCOF reduces that to 0.03 MB. In the training-free setting, where model size does not play a role in communication costs, our proposed method is much more efficient and performs better.\n\nFor fine-tuning experiments, the reviewer is correct to point out that communication cost of initialization is relatively small compared to the cost of sharing models during fine-tuning. However, we show in Fig. 3 that FedCOF provides a better starting accuracy (better than Fed3R in most cases) which then significantly reduces the overall number of training rounds required for federated fine-tuning and linear probing. More importantly, our initialization method leads to higher final accuracies on several datasets after fine-tuning (see Fig. 3 where we achieve much faster convergence compared to Fed3R, FedNCM and FedAdam and achieve higher accuracy in far fewer training rounds, thereby saving significant communication and computation costs). For example, on the Cars dataset, FedCOF+FedAdam achieves around 50% accuracy after 50 rounds of training, while FedAdam does not reach this performance after 200 rounds and even Fed3R+FedAdam achieves only around 47% at convergence. \n\nAs a final point, we would like to point to the recent trend of parameter-efficient federated fine-tuning [a2,a3] where the per-round communication costs are significantly reduced. For example, recent work [a2] shows that using pre-trained language models (RoBERTa-base), performance similar to full fine-tuning can be achieved using rank-1 LoRA updates and thereby reducing communication cost by 99.8%. In this case, initialization communication costs do contribute to the overall communication costs. \n\n[a2] Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients. ACL, 2025.\n\n[a3] Probabilistic federated prompt-tuning with non-iid and imbalanced data. NeurIPS, 2024.\n\n> Q1. Am I correct in saying that the server sees each individual client statistic $\\mu_{k,c}$ and $n_{k,c}$?\n\nYes, The server receives individual client statistics $\\mu_{k,c}$ and $n_{k,c}$.\n\n> Privacy - FedCOF requires the server to access each client statistic $\\mu_{k,c}$ and $n_{k,c}$. This is a very significant privacy issue in my opinion and deviates from the normal FL paradigm of working only with across client aggregate statistics.\n\nWe understand the reviewer’s concern regarding the privacy implications of our method. However, we clarify that our paper does not focus on the privacy aspects of FL and claims to be more privacy preserving than Fed3R only in scenarios in which secure aggregation protocols are not implemented (lines 133-136) and client feature distributions are exposed to the server. While privacy is an important aspect, our work is not centered on enhancing privacy of existing methods. We will improve the discussion of the privacy aspects of our method (including the limitations with respect to secure aggregation) in any final version of the paper. \n\nWe discuss in Appendix P and empirically show how FedCOF performs when client class counts are perturbed inspired by differential privacy. We believe $(\\epsilon, \\delta)$-differential privacy approaches could be incorporated with our method for more formal privacy guarantees, which is beyond the scope of our paper and we leave for future work.\n\nFor incorporating secure aggregation protocols with FedCOF, we propose to use pairwise masking between clients following [a4]. Each client $i$ generates a random noise vector $z_{i,j}$ for every other client $j$. Then, each of these clients $i$ computes a perturbation vector $p_{i,j} = z_{i,j} - z_{j,i}$. Similarly, $p_{j,i} = z_{j,i} - z_{i,j}$ resulting in $p_{j,i} = -p_{i,j}$. These perturbation vectors are added to the client statistics before sharing them with the server. When all client statistics are summed at the server, the noise cancels out across clients and the server can use the aggregate statistics. \n\nTo use such a secure aggregation protocol with FedCOF, the communication cost will increase and become similar to Fed3R. We proceed in two steps:\n1. Sharing only class means and counts by adding perturbation vectors to compute the global class mean and send them back to clients:\n\n+ Each client sends the class counts as $u_{k,c} = q_k + n_{k,c}$ and class means as $v_{k,c} = p_k + \\mu_{k,c} \\times n_{k,c}$, where $p_k = \\sum_{i\\in K,i \\neq k}p_{k,i}$ and $p_{k,i}$ is the perturbation vector for client $i$ and $\\sum_k p_k = 0$. $q_k$ is perturbation number similar to $p_k$.\n\n+ The server aggregates the $u_{k,c}$ and $v_{k,c}$ to cancel noise vectors cancel out. The global class mean $\\mu_c = \\frac{\\sum_k v_{k,c}}{\\sum_k u_{k,c}}$ is computed using these statistics. The server sends $\\mu_{c}$ and $N_c = \\sum_k u_{k,c}$ back to each client. \n\n2. Each client computes $s_c =(N_c-1) n_{k,c} (\\mu_{k,c}-\\mu_c)(\\mu_{k, c} - \\mu_c)^{\\top}$ and then $S_k = \\sum_c s_c+M_k$, where $M_k$ is a random noise matrix similar to the perturbation vector $p_k$ so that $\\sum_{k}M_{k}=0$. The server receives $S_k$ from all clients and adds them thus canceling out the noise to obtain $\\sum_k S_k = \\sum_k \\sum_c (N_c-1) n_{k,c} (\\mu_{k,c} - \\mu_c)(\\mu_{k, c} - \\mu_c)^{\\top}$. Now these aggregates could be used to estimate the global matrix as $G = (\\sum_k S_k + K(N_c-1)\\gamma I_d)/(K-1)+N \\mu_g \\mu_g^T$ where $N = \\sum_c N_c$ and $\\mu_g = \\frac{\\sum_c \\mu_c}{N}$. We obtain the exact same formulation of the matrix $G$ as computed in the proposed FedCOF.\n\nAlthough this increases training-free communication costs (equivalent to Fed3R), we reiterate the benefits offered by FedCOF in terms of accuracy and faster convergence which reduces overall communication costs when fine-tuning after FedCOF initialization.\n\n[a4] Practical secure aggregation for federated learning on user-held data. arXiv:1611.04482 (2016).\n\n> Accuracy -  FedCOF benefits by dropping $G_{btw}$. While this is a nice insight, such a modification should also be simple to make for Fed3R as well, while still maintaining privacy by aggregating client statistics on the server.\n\nWe emphasize that removing the between-class scatter $G_{btw}$ is not possible in Fed3R due to the statistics aggregation method used which computes the second order statistics from features of all classes in a client.\n\nFed3R shares the matrix $G = (N-1) \\hat{S} + N \\hat{\\mu}_g \\hat{\\mu}_g^\\top$ (Eq. 18) where the global covariance matrix $\\hat{S}$ can be written as the sum of within-class and between-class covariances. Fed3R could be modified in two ways to incorporate our proposal from Proposition 2:\n1. To compute $G$ directly using only the within-class scatter, Fed3R has to share the $FF^T$ matrix for each class present in a client. That would significantly change the Fed3R method by increasing its communication budget to $(d+d^2)CK$ which is the same as the FedCOF Oracle and CCVR. For instance, using MobileNetV2 for iNat120K, per client communication of Fed3R increases from 6.6 MB to 38.6 MB, which is almost 3x the size of the model (14 MB). \n2. An alternative would be sharing the class counts per client in Fed3R (similar to FedCOF) which could be used to compute the class means $\\hat{\\mu_c}$ and $\\hat{\\mu_g}$ to compute between-class scatter and then remove it. This change in Fed3R results in a method with the same privacy properties as FedCOF but with higher communication costs.\n\nWe highlight that both modifications are possible only due to our derivation in Proposition 2 (see Appendix D), since in standard ridge regression (following the formulation in Eq. 3) the between-class and within-class scatter matrices are not explicit. Thus the accuracy of Fed3R could be improved only based on one of the contributions from our paper.\n\nEven though not the main contribution of this paper, we theoretically derive the ridge regression solution in the form of covariances to enable removal of the between-class scatter. We demonstrate how this change improves performance over standard ridge regression heads across several datasets in FL settings and also in centralized setting (see Tables 7, 8 in Appendix G). This improvement is also evident after fine-tuning for 100 rounds (see Table 5). This improvement is non-trivial and is a novel contribution on using second-order statistics for classification in FL and could definitely be adapted and incorporated in other FL works."}]

Author rebuttal:

Meta-review: In order to solve the linear regression problem perfectly, typically we need to have access to covariance matrices of the features. This paper proposes a communication-efficient unbiased estimator of the covariance, and leverages the estimated statistics to initialize the linear classifier on top of pre-trained feature extractors in federated learning. Empirical improvements are convincing, and the algorithmic modification is simple but well-justified. The author rebuttal addressed the reviewers’ major concerns, and all reviewers recommend acceptance. I’d encourage the authors to incorporate the feedback such as generalizing to other non-classification tasks and potential privacy risks into the revision of the paper.
Decision: Accept (poster)