Covariances for Free: Exploiting Mean Distributions for Federated Learning with Pre-trained Models

Authors: Dipam Goswami, Simone Magistri, Kai Wang, Bartłomiej Twardowski, Andrew D. Bagdanov, Joost van de Weijer
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Federated Learning, Transfer Learning

Abstract:
Using pre-trained models has been found to reduce the effect of data heterogeneity and speed up federated learning algorithms. Recent works have investigated the use of first-order statistics and second-order statistics to aggregate local client data distributions at the server and achieve very high performance without any training. In this work we propose a training-free method based on an unbiased estimator of class covariance matrices. Our method, which only uses first-order statistics in the form of class means communicated by clients to the server, incurs only a fraction of the communication costs required by methods based on communicating second-order statistics. We show how these estimated class covariances can be used to initialize a linear classifier, thus exploiting the covariances without actually sharing them. When compared to state-of-the-art methods which also share only class means, our approach improves performance in the range of 4-26\% with exactly the same communication cost. Moreover, our method achieves performance competitive or superior to sharing second-order statistics with dramatically less communication overhead. Finally, using our method to initialize classifiers and then performing federated fine-tuning yields better and faster convergence.

Decision: reject
Reviewer scores: 3, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730284337821

thread_type:
official_review

<thread_object>
creator: reviewer_6242

<content>
summary:
This paper proposes a novel training-free FL method called FedCOF, which approximates the covariance on the server side to eliminate the enormous communication overhead. Numerical results demonstrate that FedCOF achieves comparable performance to Fed3R by merely transmitting class means to the server.

strengths:
1. Building on theoretical guarantees, this paper introduces a novel algorithm that eliminates the need for transmitting covariances between the server and clients while maintaining performance levels. This represents a valuable step for training-free federated learning.

2. The authors have carried out extensive experiments to validate the effectiveness of the proposed method, demonstrating considerable effort in their research.

weaknesses:
While the motivation of this paper is clear, I have the following questions/discussions.

1. The algorithm necessitates the transmission of $n_{k,c}$ to the server, which introduces certain privacy concerns. Although other methods also require this information, it would be beneficial if the authors could discuss potential techniques to address or mitigate this issue.

2. As modern pre-trained models tend to be generative models (e.g., GPT), it would be interesting to explore the possibility of extending the proposed methods to handle generative models by initializing the decoding heads accordingly.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730366711602

thread_type:
official_review

<thread_object>
creator: reviewer_z1dd

<content>
summary:
The paper tackles training-free federated learning. It proposes to estimate the per-class covariances in a federated learning setting using the variance of local per-class means. This covariance can be used to obtain a ridge regression classifier, which outperforms a nearest-neighbor classifier based on class means. The proposed approach thereby avoids sending local covariance matrices which reduces communication and potential privacy risks. 

The derivation of the estimator for the per-class covariance is sound. The empirical evaluation is comprehensive.

strengths:
- Training-free federated learning using feature extractors is a relevant and interesting problem.
- The proposed estimator of covariances is sound and novel.
- Experiments show substantial improvement over existing training-free methods and potential for its combination with federated fine-tuning or linear probing.

weaknesses:
- The impact of the iid assumption on realistic scenarios is evaluated empirically. It would be great to quantify how heterogeneous distributions impact the estimator theoretically, e.g., under the assumptions that local distributions are Gaussians with different mean or covariance.
- The dimension of the feature space could impact the accuracy of the estimator. This impact should be evaluated, e.g., using a synthetic dataset with varying number of feature dimensions.
- The paper focusses on label shifts, i.e., a heterogeneous distribution of classes. It is unclear how the method performs in case of feature shift, i.e., a heterogeneous distribution of features, e.g., via locally different covariance structures [1].

questions:
- Why is this approach better in terms of accuracy that Fed3R? Shouldn't it perform slightly worse or en-par, since it only approximates the covariance matrix? Here it would be good to investigate the approximation of the covariance matrix and compare it to the one produced by Fed3R.
- It would be great to compare the results using a strong pre-trained feature extractor with a classical end-to-end federated learning baseline, e.g., training a ResNet-50 on CIFAR100. 
- For consistency I suggest to use $\widehat{\Sigma}$ instead of $\widehat{S}$ in Eq. 10.
- Please compare your approach also to distributed training of linear models (using standard FedAvg), since ideally the training-free approach should perform at least en-par in terms of model performance and should outperform them in terms of communication. Here, it would be particularly interesting to compare to communication-efficient approaches [2]. 
- Since ridge regression might not always be ideal approach given a fixed feature extractor, I wonder whether a kernel ridge regression could be applicable. This would require sending the kernel matrix, but also has a closed form solution. The communication cost in that case would be quadratic in the number of data points, rather then linear in the features, so for many scenarios communication might be higher. Once could employ compression techniques here, though, like the Nyström method.


[1] Li, Xiaoxiao, et al. "FedBN: Federated Learning on Non-IID Features via Local Batch Normalization." International Conference on Learning Representations, 2021.
[2] Kamp, Michael, et al. "Communication-efficient distributed online prediction by dynamic model synchronization." Machine Learning and Knowledge Discovery in Databases, 2014.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730497700262

thread_type:
official_review

<thread_object>
creator: reviewer_wumb

<content>
summary:
The paper proposes to use pre-trained models to perform federated classification. More specifically, Fed-COF uses pre-trained models to extract features of each example on the client, then averages the features within each class. The class-averaged features from clients are aggregated in the server to estimate the first-order and second-order statistics, which are then used in ridge regression to fit a classifier. The communication cost of Fed-COF scales only linearly with the size of the embedding.

strengths:
The paper is well written. The derivations are clear and easy to understand. The proposed Fed-COF achieves decent empirical performance. Fed-COF also seems to work well with fine-tuning.

weaknesses:
- The steps of Fed-COF + Fine-tuning can be made clearer. The description around line 357 is difficult to follow.

- The choice of ridge regularization parameter $\lambda$ seems important for the classification performance. Can authors give more empirical suggestions on how $\lambda$ should change with different numbers of clients/means per client?

questions:
- What is the difference between Fed-COF oracle and Fed3R? In Table 2, it is surprising to see Fed3R sometimes achieves lower accuracy with more communication. The authors should provide more explanations for the phenomenon.

- In line 472, atleast should be at least.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730825600626

thread_type:
official_review

<thread_object>
creator: reviewer_jbiq

<content>
summary:
The authors introduce FedCOF, a training-free federated learning approach that utilizes a pretrained model's feature extractor while updating only the classifier. Unlike previous methods that only aggregate global means, FedCOF improves performance by also deriving and leveraging unbiased global covariances from these means. Local clients send first-order statistics (class-wise feature means) to the server, which then uses these to estimate global covariances. This innovation allows for efficient communication while significantly enhancing the effectiveness of the global classifier updates.

strengths:
**Strengths of  FedCOF:**

FedCOF presents a timely contribution that leverages pretrained models in federated learning (FL), addressing the deep learning (DL) field's growing emphasis on foundation models. FedNCM is communication-efficient but suffers from limited performance, while Fed3R improves performance using both first- and second-order statistics but at a high communication cost. In contrast, FedCOF achieves strong performance with minimal communication overhead by deriving unbiased global covariances using only first-order statistics.

weaknesses:
***Concerns on Presentation***

**Related Works**
I recommend enhancing the Related Works section to include a broader range of studies that cover both the use of fixed classifiers and the potential limitations of pretrained models in federated learning settings.

In **Line L101**, the section discussing the application of **fixed classifiers** in federated learning could be expanded by incorporating recent and relevant studies. Specifically, it would be valuable to reference works such as FedBABU [1], SphereFed [2], and Neural Collapse-inspired approaches [3,4], which explore the impact of classifier freezing in federated scenarios.

Furthermore, the discussion in **Line L102** and beyond about **federated learning with pretrained models** should present a more balanced view. The current description highlights only the positive outcomes of using pretrained models. However, it is important to acknowledge that pretrained models are not always advantageous in federated settings. For example, findings from FedFN [5], particularly in Section 5.2, demonstrate situations where pretrained models can adversely affect the performance of the global model, especially under heterogeneous data conditions. Including this perspective would provide a more comprehensive understanding of the complexities involved in using pretrained models within federated learning frameworks.

**Preliminaries**

L117: D_k seems to refer to the local dataset rather than local data.

L127: There is no clarification on the type of loss function or how the loss is calculated (whether as a batch mean or batch sum).

L129-130: "After initializing \theta with pretrained weights, the models can be optimized in a federated manner" — In this paper, local clients do not perform local updates based on the pretrained model, and this information seems to hinder the understanding of the paper.

**Concerns on Privacy Discussion in Section 4:**

The algorithm sends the class-wise frequency of the data held by clients to the central server. I believe this information could also raise privacy concerns, yet there is no mention of this issue. In fact, there are many previous FL papers that have communicated class frequency information and provided justifications. Citing these studies would strengthen the discussion, but this type of content is entirely missing.

***Concern on Incremental Contribution***

The problem may seem incremental, as it combines existing methods' strengths, but it addresses the practical challenge of balancing communication cost and performance in FL.


[1]FedBABU: Toward Enhanced Representation for Federated Image Classification, ICLR 2022.

[2]SphereFed: Hyperspherical Federated Learning, ECCV 2022

[3]No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier, ICCV 2023

[4] FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning. FedKDD 2024

[5]FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning. NeurIPS Workshop 2023, Federated Learning in the Age of Foundation Models.

questions:
While I currently have several concerns that have led to a lower score, I am open to increasing the score if these issues are adequately addressed during the rebuttal period.

**Questions and Suggestions:**

The use of pretrained models in federated learning (FL) is a promising and timely research direction, especially given the current trend in the deep learning community toward leveraging foundation models effectively. However, as mentioned earlier in the Related Work section, using a pretrained model is not always superior to using a randomly initialized model. Specifically, findings from FedFN [1], Section 5.2, highlight scenarios where pretrained models can negatively impact global model performance in high heterogeneous settings.

Given this, it would be beneficial for the authors to include experimental comparisons between using a pretrained model and a randomly initialized model. These comparisons should cover various baselines and the proposed algorithm to provide a clearer understanding of whether the pretrained model genuinely improves performance.

Additionally, if the characteristics of the training data used for the pretrained model(e.g. ImageNet) are significantly different from the test data(e.g. SVHN) targeted by the global model, using a pretrained model could potentially be detrimental. It is important to clarify what specific test dataset the final foundation model and redefined classifier are targeting, as this information does not seem to be explicitly stated in the Preliminaries section.


Furthermore, a fundamental challenge in FL is the heterogeneity of client data, which often leads to **class imbalance** issues within each client’s local dataset. However, what differentiates FL from traditional class imbalance problems is the presence of **missing classes**, where certain classes are entirely absent from a client's dataset. This problem is especially pronounced as data heterogeneity increases, causing missing classes to occur more frequently across clients.

The proposed algorithm sends class frequency information from each client, but in the case of missing classes, this would simply convey a value of 0. I am concerned that the algorithm might be particularly vulnerable to the impact of these missing classes. Could the authors explain how their proposed algorithm is designed to mitigate this vulnerability in the context of FL, and why it might still perform well despite the challenges posed by missing classes?


[1]FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning. NeurIPS Workshop 2023, Federated Learning in the Age of Foundation Models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732181077894

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank all reviewers for their insightful feedback aimed at improving the quality of our work. The reviewers agree that the paper is well-written (**Wumb**), address a relevant and interesting problem (**z1Dd**), presents a timely contribution (**jBiQ**), with clear and easy to understand derivations (**Wumb**), propose a sound (**z1Dd**) and novel (**6242**,**z1Dd**) covariance estimator with theoretical guarantees (**6242**) and provide comprehensive (**z1Dd**) and extensive (**6242**) empirical evaluation.

Here we provide a summary of our responses and highlight new results presented during the discussion period. We have updated the paper with all the experiments detailed below and highlighted the additional clarifications. 

In response to the concerns raised by reviewer [jBiQ](https://openreview.net/forum?id=7NtAIghBsE&noteId=5ZSh899J9u), we updated the related works by citing FedBABU, SphereFed, FedDr+, neural collapse and FedFN. We also updated the preliminaries section to clarify notations. We performed experiments to show that using pre-trained networks for FedAvg and FedAdam significantly outperforms training with a randomly initialized network. We have updated the paper to discuss the privacy concerns raised by sharing class statistics and address that using perturbations. We also clarify that our method is not affected by the missing classes problem.

In response to the concerns raised by reviewer [Wumb](https://openreview.net/forum?id=7NtAIghBsE&noteId=NcBpn5ddrg) we updated the discussion of FedCOF with multiple rounds in the paper for more clarity. We also discuss the role of the ridge regression parameter and highlight the difference between the Fed3R and FedCOF-Oracle initializations. 

In response to the concerns raised by reviewer [z1Dd](https://openreview.net/forum?id=7NtAIghBsE&noteId=vWoj55YR9j), we theoretically analyze the bias of the proposed estimator in non-iid settings with locally different covariances. We also performed experiments on feature-shift settings using DomainNet. Finally, we performed experiments using ResNet-18 to compare with classical federated learning methods and to also compare our method with federated training of linear models. We have also clarified the difference between FedCOF from Fed3R initialization.

In response to the concerns raised by reviewer [6242](https://openreview.net/forum?id=7NtAIghBsE&noteId=qTJF4h1WaX), we now discuss the privacy concerns related to sharing local class frequencies and have proposed a perturbation strategy to address those concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an interesting train-free federated learning method which leverages (publicly available) pre-trained models to boost performance. The empirical results appear promising.

There has been multiple extensive discussions regarding the impact of the paper's contribution between authors and reviewers as well as between reviewers and AC. During the reviewer-author discussion, there is a debate among the reviewer on the benefit of initializing federated learning with pre-trained models. While this feels at first counter intuitive that initializing federated learning with a pre-trained model might not always help, this is empirically true in cases where there is significant distribution gap between local datasets and/or when local datasets are highly skewed/imbalanced in different ways -- see [*]. 

[*] https://openreview.net/pdf?id=nw6ANsC66 

Otherwise, in standard settings where there is no such imbalance or significant distribution gap across local datasets, there is a verified consensus that initializing federated learning with pre-trained models will help. Upon extensive discussion with the reviewers, I believe the focus of this paper is on the standard setting so the point raised by reviewer jBiQ does not affect the contribution of the paper. The authors however are encouraged to provide extra discussion around this point in their revised paper.

Note that [*] is cited here to reconcile the seemingly opposite points raised by the reviewers during the discussion. The paper is not penalized for not citing/comparing with this recent work (which also focuses on an orthogonal setting).

--

Having said that, the real concern here, however, is that using pre-trained backbone is not a new practice and so this paper should have compared its method with more direct baselines. For example, if we view local models as fine-tuned versions of the large model, we could easily apply existing FL methods (e.g., FedAvg, FedProx etc.) to aggregate the fine-tuning parameters -- see the baselines used in [*]. 

The authors should have then compared the performance of their proposed (one-shot) method with both multiple- and single-shot variants of those baselines to conclusively demonstrate its benefit. I would expect to see that the performance of the proposed method coming close to or even exceed the performance of the multiple-shot variants while incurring much less communication cost. I believe this is the main point here which needs to be demonstrated more thoroughly.

Appendix H + the main-text experiments currently fall short of achieving this.

--

Overall, I feel that this paper is somewhat below the acceptance bar mainly due to the aforementioned issue with its empirical studies. Otherwise, I agree that its technical idea (pending thorough impact assessment) is sufficiently novel. It essentially boils down to whether the contribution of this paper outweigh its flaws. I have asked the positive reviewers to see if anyone is willing to champion this paper given the above assessment. But, unfortunately, there is no indication that anyone is willing to do so and this paper remains in the borderline.

--

Regardless of the final decision of the PC, I hope the authors would seriously revise the paper to take into account all the key discussion points that I summarize above.