On the Importance of Language-driven Representation Learning for Heterogeneous Federated Learning

Authors: Yunlu Yan, Chun-Mei Feng, Wangmeng Zuo, Salman Khan, Yong Liu, Lei Zhu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: federated learning, language-driven representation learning, data heterogeneity

Abstract:
Non-Independent and Identically Distributed (Non-IID) training data significantly challenge federated learning (FL), impairing the performance of the global model in distributed frameworks. Inspired by the superior performance and generalizability of language-driven representation learning in centralized settings, we explore its potential to enhance FL for handling non-IID data. In specific, this paper introduces FedGLCL, a novel language-driven FL framework for image-text learning that uniquely integrates global language and local image features through contrastive learning, offering a new approach to tackle non-IID data in FL. FedGLCL redefines FL by avoiding separate local training models for each client. Instead, it uses contrastive learning to harmonize local image features with global textual data, enabling uniform feature learning across different local models. The utilization of a pre-trained text encoder in FedGLCL serves a dual purpose: it not only reduces the variance in local feature representations within FL by providing a stable and rich language context but also aids in mitigating overfitting, particularly to majority classes, by leveraging broad linguistic knowledge. Extensive experiments show that FedGLCL significantly outperforms state-of-the-art FL algorithms across different non-IID scenarios.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730268801965

thread_type:
official_review

<thread_object>
creator: reviewer_q1wc

<content>
summary:
This paper solves the heterogeneous federated learning (in the image classification field) via a pre-trained and fixed text encoder (BERT) on the server and learned image encoders on the clients. The classification is done by clip-style image-text embedding similarity comparison. The authors report superior performance over traditional heterogeneous federated learning methods (FedAvg, etc.), and provide scalability analysis, efficient analysis, and ablation study.

strengths:
1. The proposed method shows superior performance.
2. Comprehensive analysis is provided.
3. The usage of Bert embedding as a prototype vector for FL image classification is novel.

weaknesses:
1. There are few comparisons with prototype FL (like [1] and [2]).
2. The method proposed has some limitations. It is restricted to image classification and requires a pre-trained model that can give good embeddings of the classification labels. This seems to exclude many FL scenarios. (I am not to say that using Office-Caltech-10 and DomainNet is not enough. I just wonder if it may not be as versatile as FL methods like FedAvg. Yet I still appreciate this paper as a good method for image classfication.)


[1] Federated Learning from Pre-Trained Models: A Contrastive Learning Approach https://proceedings.neurips.cc/paper_files/paper/2022/file/7aa320d2b4b8f6400b18f6f77b6c1535-Paper-Conference.pdf
[2] FedProto: Federated Prototype Learning across Heterogeneous Clients https://arxiv.org/pdf/2105.00243

questions:
1. Comparison/Detailed discussion about prototype FL
2. Can it be adjusted for tasks besides image classification?
3. Can traditional word embedding such as Glove also achieve a performance comparable to Bert's? (just curious about this)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730441217385

thread_type:
official_review

<thread_object>
creator: reviewer_4piz

<content>
summary:
The paper presents FedGLCL, a novel federated learning (FL) framework designed to address the challenges posed by non-Independent and Identically Distributed (Non-IID) data. The framework leverages a pretrained VLM, CLIP, specifically through contrastive learning that integrates global language and local image features. The proposed method aims to harmonize feature learning across different clients, reducing variance in local representations and mitigating overfitting to majority classes. The paper includes extensive theoretical and empirical analyses, demonstrating the superiority of FedGLCL over state-of-the-art FL algorithms in various non-IID scenarios.

strengths:
1. The utilization of large pretrained CLIP into FL is a unique and innovative direction. By leveraging contrastive learning to align local image features with global textual data, the paper proposes a novel solution to tackle the Non-IID problem.
2. The paper provides a solid theoretical foundation for FedGLCL, explaining how the framework can mitigate issues related to Non-IID data. This theoretical backing strengthens the credibility of the proposed method.
3.The authors conduct a comprehensive set of experiments, comparing FedGLCL with multiple state-of-the-art FL algorithms across different non-IID scenarios. The results demonstrate significant improvements, validating the effectiveness of the proposed framework.

weaknesses:
1. Clarity and presentation issue: It is not clear which pretrained vision language models are used in the experiments. CLIP? Align? BLIP?

2. It is better to compare the performance of different text encoders from various pretrained VLMs, because the proposed method heavily relies on the pretrained text encoder.

questions:
Are the datasets used in the experiments part of the pretraining of VLMs? If this is the case, then your framework achieves better performance is due to the pretraining stage.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717223173

thread_type:
official_review

<thread_object>
creator: reviewer_bw7e

<content>
summary:
This paper proposes a novel approach, FedGLCL, to learn image models in non-IID federated settings. Their approach leverages a fixed frozen set of class embeddings (from a language model) and locally updates image representations in a CLIP-like contrastive manner. When tested on multiple variations of non-IID federated learning benchmarks (e.g. significant label shift/imbalance), this approach is shown to notably outperform other training/aggregation-based interventions without introducing additional computation/communication costs. Furthermore, the authors provide several additional ablations and theoretical analyses to deepen understanding of their results.

strengths:
**Originality and Significance.** Federated learning with non-IID client data is a well-motivated/challenging problem and the authors' approach, while simple, is novelly and effectively applied in this setting. As mentioned in Sec 5.4, this approach also importantly does not come with any decreases in efficiency. 

**Quality.**  Overall, the results for their method are quite strong compared to a variety of baselines and on several variations of the evaluations. The authors also did a great job ablating aspects of their method (particularly in the Appendix), addressing several of the questions that came up while I was reading the paper. In particular, I appreciated the ablations on the text embeddings, which show that it is important to both (1) keep the embeddings fixed; (2) use a pre-trained language model instead of custom class embeddings. These help establish that contrastive-style training itself is not the main explainer for good performance. 

**Clarity.** Generally, l found the paper to be well-written and easy to follow. Particularly, the authors did a good job contextualizing their methods/results with previous works.

weaknesses:
**More discussion about the impact of language supervision.** One result I found particularly curious (and which I believe should be highlighted more in the main paper) was that using embeddings for randomly mismatched class names results in comparable performance to using embeddings for the correct class names. Before seeing this result, I would have thought that the semantic information and relationships between classes was being usefully leveraged (e.g., similar classes being closer together in text embedding space). But based on this result, I'm not sure then to what extent the "broad linguistic knowledge" emphasized in the abstract is important in explaining FedGLCL's success? Perhaps the authors can expand upon their discussion of this in the Appendix and comment more. 

**Measuring the impact of FedGLCL-style training in centralized/IID-Federated settings** It would be interesting to see how well the contrastive style training from FedGLCL performs in the non-federated setting (and perhaps also the IID federated setting). This would shed light on whether the gains in the non-IID federated setting might also be explained by the contrastive learning approach in FedGLCL-being more effective generally on these tasks v.s. if it is uniquely helpful for overcoming challenges in non-IID federated settings. Relatedly, it would also be nice to discuss your results in relation to Fang et al., 2022 (https://arxiv.org/abs/2205.01397), which included results assessing the impact of language supervision on CLIP pre-training. in their case, they find that it does not have a major impact on the final robustness of CLIP models.

**Standard errors for results.** It would be nice if the empirical results came with standard errors. That being said, I can acknowledge that the current gaps between the proposed method and previous baselines seem to be large/consistent across evaluation settings and that repeated runs might be costly.

questions:
Most questions I had were covered in the Weaknesses section. The only other question I had was if you considered any settings involving pretrained image models. On the surface, this might better reflect practical deployments for image-based tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723265676

thread_type:
official_review

<thread_object>
creator: reviewer_ym3c

<content>
summary:
The paper introduces FedGLCL, a FL framework designed to improve the handling of non-IID data in FL environments. The authors leverage language-driven representation learning (e.g. CLIP, as opposed to label driven), incorporating a pre-trained text encoder to stabilize feature representations. This approach enables the alignment of local image features with a global text feature space, reducing feature disparity across clients. Experiments demonstrate that FedGLCL outperforms traditional label-driven FL methods in various non-IID scenarios, including label and feature distribution skew.

strengths:
1. The proposed method is novel - mitigating non-iid data issue in FL on image classification pre-training with text encoders.

2. The experiment results are good, the authors experimented on many datasets, comparing against many baselines, highlighting the generalizability of their method. 

3. The author provided theoretical guarantees to the generalizability of the proposed method.

weaknesses:
1. The use of image-text pre-training is prevalent, and the non-iid data issue is not limited to FL settings, so adding text encoders to mitigate the non-iid issue shouldn't be new. Honestly I am not familiar with the FL literature, so I don't know how much this would add to the FL community. 

2. The experiments are somewhat limited to small-scaled datasets. Performing experiments on larger scaled datasets (e.g. Imagenet) would make the claims stronger.

questions:
In this paper, federated learning is applied to a specific type of image-text pre-training. I suggest to make this clear in the abstract and throughout the paper - as there are also other types of federated learning as well (e.g. text only)

158 - 161: Can you explain more how your work differs from other works that uses CLIP in FL training (e.g. Lu et al., 2023, Guo et al., 2023, Shi et al., 2023)? In particular - what is "the philosophy of language-driven representation learning"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732108265320

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by Authors

comment:
Dear reviewers and meta-reviewers,

We appreciate all reviewers for their valuable comments and suggestions. We've revised our manuscript based on reviewers' comments as follows:

(1) For Reviewer yM3c, we have added the results for Tiny-ImageNet in Table 16, with detailed information about the dataset provided in Table 5.

(2) For Reviewer yM3c, we have revised the abstract (Line 017-018) to clarify the type of FedGLCL.

(3) For Reviewer yM3c, we have revised our writting in Line 160.

(4) For Reviewer bW7e, we have discussed the impact of language supervision in Appendix D.2 (Line 1069-1079).

(5) For Reviewer bW7e, we have reported the standard errors for results in Table 17.

(6) For Reviewer bW7e and Q1wc, we have added the results of the experiment using pre-trained image models and a new baseline, FedPCL, in Table 18.

(7) For Reviewer Q1wc, we have added the results on medical image segmentation in Table 19.

(8) For Reviewer Q1wc, we have added the results of our method with Glove in Table  13.

The changes have been highlighted in **blue** in the revised paper. Please see below for our responses to each reviewer. If you have any further questions or suggestions, please feel free to share them on OpenReview.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces FedGLCL, a language-driven federated learning (FL) framework that integrates global language and local image features via contrastive learning to address challenges posed by non-IID data. By harmonizing local image features with a pre-trained text encoder, FedGLCL ensures uniform feature learning, reduces variance in local representations, and mitigates overfitting to majority classes. Experiments demonstrate its superior performance over state-of-the-art FL algorithms in diverse non-IID settings.

This is a borderline paper. The reviewers have brought up a few issues with this paper, though I think the authors have addressed many of the issues. In my opinion, it would be good to see this paper in ICLR. I would encourage the authors to go through the reviews carefully and address it in the next version.