On the Importance of Language-driven Representation Learning for Heterogeneous Federated Learning
Authors: Yunlu Yan
Chun-Mei Feng
Wangmeng Zuo
Salman Khan
Yong Liu
Lei Zhu
Venue: ICLR 2025 Poster (2025)
Keywords: federated learning
language-driven representation learning
data heterogeneity
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Non-Independent and Identically Distributed (Non-IID) training data significantly challenge federated learning (FL), impairing the performance of the global model in distributed frameworks. Inspired by the superior performance and generalizability of language-driven representation learning in centralized settings, we explore its potential to enhance FL for handling non-IID data. In specific, this paper introduces FedGLCL, a novel language-driven FL framework for image-text learning that uniquely integrates global language and local image features through contrastive learning, offering a new approach to tackle non-IID data in FL. FedGLCL redefines FL by avoiding separate local training models for each client. Instead, it uses contrastive learning to harmonize local image features with global textual data, enabling uniform feature learning across different local models. The utilization of a pre-trained text encoder in FedGLCL serves a dual purpose: it not only reduces the variance in local feature representations within FL by providing a stable and rich language context but also aids in mitigating overfitting, particularly to majority classes, by leveraging broad linguistic knowledge. Extensive experiments show that FedGLCL significantly outperforms state-of-the-art FL algorithms across different non-IID scenarios.

Published as a conference paper at ICLR 2025
ON THE IMPORTANCE OF LANGUAGE-DRIVEN REPRE- SENTATION LEARNING FOR HETEROGENEOUS FEDER- ATED LEARNING
Yunlu Yan'*, Chun-Mei Feng"; Wangmeng Z4, Salman
Jalman Khan Yong Liu', Lei Zhu 1.71
The Hong Kong University of Science and Technology (Guangzhou), China
2 Institute of High Performance Computing (IHPC).
Agency for Science, Technology and Research (A (A"STAR), Singapore
3 Harbin Institute of Technology, China
4 Pengcheng Laboratory. China
5 Mohamed bin Zayed University of Artificial Intelligence (MBZUAT), UAE
6 Australian National University, Australia
7 The Hong Kong University of Science and Technology, China
ABSTRACT
Non-Independent and Identically Distributed (Non-IID) training data significantly challenge federated learning (FI.), impairing the performance of the global model in distributed frameworks. Inspired by the superior performance and generaliz- ability of language-driven representation learning in centralized settings, we ex- plore its potential to enhance FL for handling non-IlD data. In specific, this paper introduces FedGLCL, a novel language-driven FL framework for image- text learning that uniquely integrates global language and local image features through contrastive learning, offering a new approach to tackle non-IID data in FL. FedGLCL redefines FL by avoiding separate local training models for each client. Instead, it uses contrastive learning to harmonize local image features with global textual data, enabling uniform feature learning across different local mod- els. The utilization of a pre-trained text encoder in FedGLCL serves a dual pur- pose: it not only reduces the variance in local feature representations within FL by providing a stable and rich language context but also aids in mitigating overfit- ting, particularly to majority classes, by leveraging broad linguistic knowledge. Extensive experiments show that FedGLCL significantly outperforms state-of- the-art FL algorithms across different non-IID scenarios. Codes are available at https: //t////iii thub. com/ wwt.://w.....a....a.......
INTRODUCTION
Federated learning (Li et al., 2020a) (FL) serves as a powerful tool for distributed learning, allowing multiple clients to collaboratively train global model without privacy concerns. It widely applied in various applications (Niknam et al., 2020; Kang et al., 2020), especially in some privacy-sensitive fields such healthcare (Jiang al., 2022; Feng al., 2022; Wicaksana al., 2022; Yan et al., 2024a; Feng et al., 2023a; Yan et al., 2024b; 2023). The pioneering FL algorithm, FedAvg (McMa- han al., 2017), trains the model locally and averages the updated parameters of local models to update the global model. This process is effective and easy to implement, which establishes the fundamental framework for FL.
Since data is acquired from different devices in diverse environments, each local dataset inherently exhibits variations the underlying distribution, iee., non-IlI' (shifts due to weather conditions, night-time images, and various image degradations). Label distribution skew and feature distri- bution skew (see Definition
1) are two typical cases of non-III situations real-world
" "Equal contribution.
1 Lei Zhu (1 i zhu@ust. hk) is the corresponding author. We use "heterogenenus" and "non-IlI>" interchangeably.

---

Published as a conference paper at ICLR 2025
30
Gairo of Language-dien Regresotaiion Learing
100
Cartslooll
Federited
02.25
Apc
OffcoCallechB 30
CIFAR 100
Mijortt
Minoiity
Libe ofice
CCrrmddd Label Labelicher) ProdentonLieeei
Cartmlted 2.ngageehhert Fodeolest111 Inguogge ser)
LLangageehhhnn
Luyer Index
Figure I: Analysis of two types of training paradigm, i.e., label-driven representation learning and language-driven representation learning. (a) Accuracy comparisons on centralized and fed- erated scenarios, where the language-driven representation learning improves Aces on the federated scenario higher than it improves on centralized learning.
(b) Accuracy comparisons for both lo- cal majority and minority classes on CIFAR-100. Language-driven representation learning showed significant improvement over label-driven representation learning, especially for minority classes, indicating that language-driven representation learning effectively enhance the generalization performance of local models. (c) Variations in CKA similarity (Kornblith et al., 2019) reveal that language-die unnuege-drriven driven representation learning reduces the disparity in feature representations among local models.
applications, manifesting data imbalances at the label level and feature level (Li al., 2022c; Zhang et al., 2022). Such statistical heterogeneity incurs a significant decline in the performance of FedAvg (Zhang et al., 2022; Luo et al., 2021; Karimireddy et al., 2020; Li et al., 2020b; Zhao et al,, 2018), emerging as a fundamental challenge in FI.. This has attracted significant attention, and many studies et al., 2021; Zhang et al., 2022; Gao et al., 2022) or model aggregation (Li et al., 2020c; Wang et al., 2020; Li et al., 2023; Fallah al., 2020) of FedAvg on non-lID data. Despite the progress, most existing FL methods still follow label-driven representation learning during local training, such as cross-entropy loss in classification tasks.
dedicated to improving the local training (Li et al., 2021; Tan al., 2022a; Luo
In this work, we argue that it is not advisable to adopt the label-driven training paradigm, as it leads to significant drift in local models. Drawing on visual recognition as the running example, as mentioned in previous studies, induces model bias towards imbalanced local class distribution in terms of label distribution skew setting (Zhang et al., 2022; Diao et al., 2024). The local model seeks to fit the majority classes (see Definition 2 in 83.1) of the private dataset as much possible (see Figure (b)), incurring differences in both the feature representation layers (backbone) and the classifier. It exhibits greater bias in the classifier compared to other layers (Luo al., 2021). In terms of feature distribution skew setting, using cross-entropy as the local objective will result in the feature shift problem (Li al., 2020e; 2023; Zhou & Konukoglu, 2023), thereby degrading the performance of the global model (see Figure 1 (c)). The shortcomings of label-driven representation learning prompt us to explore a new training paradigm redefine local training.
With the emergence of CLIP (Radford et al., 2021), language-driven representation learning has gained considerable attention and achieved success in various applications (Cha al., 2023; He et al., 2023; Wang et al., 2022). Models trained with language showcase better generalization. Since language-driver driven representation learning is so effective in the centralized setting, we are thus interested in whether linguage-driver representation learning can be a better training paradigm for FI. particularly in alleviating the non-IID issue. By introducing it into FI., we surprisingly observed consistent accuracy improvements in FedAvg across two federated scenarios as shown in Figure (a). We also observed that these improvements are significantly more pronounced in FL compared to centralized learning. This suggests that language-driven representation learning provides additional benefits to FedAvg in the federated setting. With deeper analysis, we further found that: Language- driven representation learning benefits 1) learning unified feature representation across different clients (see Figure 1 (c)); 2) alleviating the overfitting to majority classes (see Figure (b)), more details can be seen in Appendix $A. This explains why language-driven representation learning is more effective compared to traditional label-driven representation learning in heterogeneous FL. scenarios.
Based on the above analyses, we propose Federated Global Language-image Contrastive Learning, termed FedGLCL. FedGLCL focused on redefining the local training phase of FedAvg, reeldig.yeelding
2

---

Published as a conference paper at ICLR 2025
language-drivee driven training manner for FL on non-lID data. It instantiates the idea of language-din drivee representation learning by aligning each local image feature space to a common global text feature space during the training phase. Different from CLIP, we use fixed pre-trained text encoder to encode the global class texts, thereby yielding a consistent text feature space. Subsequently, we use contrastive learning train the image encoder from scratch. In framework, there is no tradi- tional classifier. Instead, similar to CLIP, we use similarity measures between images and text for predictions. Accordingly, we argue that the bias across different local models in our method, no matter whether caused by the label distribution skew or feature distribution skew, can be interpreted as the difference in feature representation. This issue has been alleviated through the alignment of feature spaces. Therefore, FedGLCL can be applied to various non-llI scenarios. Extensive experi- ments on four benchmarks demonstrated the effectiveness of our approach, which shows significant improvements under different non-Ill) scenarios.
Contribution. In this paper, we systematically explore the importance of language-driven repre- sentation learning for heterogeneous FL, yielding language-driven FL framework, i.e. FedGLCL. While seems straightforward, we believe this study provides more insights into addressing the non- IID issue in FL. Moreover, offer comprehensive theoretical and empirical analyses to gain deeper understanding of our approach, providing valuable insights for future research work.
2 RELATED WORK
2.1 FEDERATED LEARNING ON NON-IID DATA
How to alleviate the impact of non-lII data on model training has always been a critical issue in the field of FL (Ye et al., 2023; Huang et al., 2023; Luo et al., 2021; Feng et al., 2023b). Variations in the distribution of data across different clients may result in optimization bias in the local models, thereby significantly degrading the performance of the model aggregation (Karimireddy et al., 2020; Zhao et al., 2018). To address this, type of solution (Durmus et al., 2021; Li et al., 2020b; Zhang et al., 2022: Li et al., 2021; Gao et al., 2022) is to minimize the optimization difference among local models during the local training phase. For example, FedProx (Li et al., 2020b) and FedDyn (Dur- mus et al., 2021) introduced additional regulation terms into local objectives to mitigate the bias between local models. In contrast to these approaches, another type of solution. (Li et al., 2020c; Hsu et al., 2019; Wang et al., 2020; Fallah et al., 2020; Ma et al., 2022; Lee et al., 2023) focuses on designing novel model aggregation strategies yield better global model. For instance, Fed- Nova (Wang et al., 2020) proposed normalized averaging method, which corrects local model parameters before averaging. Besides, a typical representative among these methods is personalized aggregation strategies (Li et al., 2020c; Arivazhagan et al., 2019; Collins et al., 2021; Lee et al., 2023), which mitigate the impact of data heterogeneity by allowing local models to retain portion of personalized parameters.
In this work, we focus on the local training phase. While previous studies have made progress on this type of solution, most of them still use label-driven representation learning the local training paradigm. They typically employ cross-entropy as the primary supervisory training loss and additionally incorporate some auxiliary losses to mitigate bias in local models, such regulation (Li et al., 2020b; Durmus et al., 2021), distillation (Yao et al., 2023; Lee et al., 2022) and prototype learning (Tan et al., 2022a;b). While effective, it is unable overcome the inherent disadvantages of label-driven training (see Figure (b) and (c)). Given this, this work redefines the process of local training by language-drives representation learning, which can effectively address non-IlI issues.
2.2 LANGUAGE-DRIVER REPRESENTATION LEARNING
Recently, language-driven representation learning (Gan et al., 2022) has been very hot topic in the field of vision, giving rise to numerous vasion-language models, such as CLIP (Radford et al., 2021) and Align (Jia et al., 2021). CLIP employs text and image encoders to learn rich semantic representations from extensive pairs of images and texts by contrastive loss. This grants CLIP more powerful feature representation capabilities and superior zero-shot learning performance, prompting lots of studies to directly apply it to various downstream tasks (Zhou al., 2022; Patashnik et al., 2021; Wysoczańska et al., 2024; Luo et al., 2022; Gao et al., 2021). Besides direct application, another is to modify the text-image feature alignment approach of CLIP for specific downstream
3

---

Published as a conference paper at ICLR 2025
Dog
photo of (clos]
Text Ercoder
Airplane
Dog
photo of (class)
Airplane
Glotall
Servers 5
Elibbal teat emberdings
Local wittweddingg
95
1.T% A:T.
1:Ta
Bactbone
Bookbond
7.9% Fethy FyPh
N
Dog
(a) Training Stage
(b) Inference Stage
Figure 2: Illustration of FedGLCL. (a) During the training stage, FedGLCL utilizes a frozen text encoder to embed the global class texts the server side, thereby yielding unique global text feature space. Then, each elient aligns its own local image feature space with the global text feature space. We omit the model aggregation procedure, it is identical to FedAvg. (b) During the inference stage, we make predictions based on the similarity between images and text.
tasks (Li et al., 2022a; Xu al., 2022: Wang et al., 2023). For example, RegionCLIP (Li et al., 2022b) and GLIP (Cha et al., 2023) proposed region-text alignment to learn region-level semantic information for object detection. The idea of region-text alignment has also been applied in semantic segmentation tasks (Cha et al., 2023; Liang et al., 2023).
However, little is understood about the importance and applicability of language-driven representa- tion learning for FL.. Previous studies have merely applied the pre-trained CLIP model directly FL as a foundational component (Lu et al., 2023; Guo et al., 2023; Shi et al., 2023), which unrelated language-driven representation learning. In this paper, thoroughly explored language-driven representation learning for non-IID FL settings and proposed a novel language-driven FL frame- work.
METHODOLOOGY
3.1 PRELIMINARY
Federated Learning. Suppose there are clients and each client E [K] has a set of supervised training instances {(2, yr 1:) [= The image I; X and label y/ E Y from device-indexed joint distribution (z,, y/) ~ (x y). The goal of FL is to collaboratively learn a model f without sharing local private data, which can be formulated the following optimization problem:
[(w) Σ (WL,(w) where Ykk
7
ΣK n
To solve this, FedAvg (McMahan al., 2017) minimizes each local empirical loss Lx(w) to par- allel train local models local training the clients and then average the updated local model parameters- - model aggregation at the server:
Cx(w
Σ (NF.
(ffz; ()) 1):
(2)
w/
Σ Tka
k=1
where the superseript t represents the model parameters after f-th round training, wa and we the parameters local model and global model. label-driven FL, the loss function is usually cross-entropy loss for the image classification task.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1: FedGLCIL
Input: Number of Clients K, communication rounds M, epochs E, learning rale v, temperature
Output: wi
Server-side Execution:
T < Embed global class texts
for round t 1, .... M do
for client k 1, 2, ...,, K parallelly do for (Ti, U1) P3 do
w/g Local Training (ki, (G .T)
end
end
wig + Σk= 7tw1
end
Local Training (ks, u0 1-1 Ty:
12 wi
1-1 w/6
for epoch 1, 2, ..., E do
for (Ti, V/) Fx do
15
I: + f(x,; w
8, I.T ||||T
exp(6i.;;/1)
8. log Lith 777)
wh. wi Vi
19
end
end
return w to server
Definition (Non-IID Data In Federated Learning) The local distribution (x, y) can be rewrit- ten (u PP)) or Px()))))(()). For label distribution skew, Ps(y) varies across clients while (x(gyy) is consistent for all clients. In contrast, for feature distribution skew, Px(y)z) keeps consis- tent for all clients but (x(x) is different.
Definition (Majority and Minority Classes) For label distribution skew, the number of instances for each class the client side is imbalanced. We partition the Y into majority classes Y and minority classes Y3 based on whether their quantities exceed a threshold T, where Y Y U Y,-.
Tradition FL methods adopt label-driven representation learning the local training paradigm, which struggles to exhibit robustness on non-IID data, leading to inconsistent local and global ob- jectives, i.e., the global model from Eq. (3) deviate from the solution of Eq. (1), termed client drift. Previous studies (Li et al., 2020b; Durmus et al., 2021; Li et al., 2021; Lee et al., 2022) attempt to introduce various forms of auxiliary loss terms into Eq. (2), but it remains challenging to fully over- come the inherent limitations of label-driven representation learning for client drift (Karimireddy al., 2020). In this work, we explore a novel federated global language-iman -mmge contrastive learning, namely FedGLCL, to address the non-llI issue in FL.
3.2 FEDERATED GLOBAL LANGUAGE- IMAGE CONTRASTIVE LEARNING
Our approach, FedGLCL, the first language-driven federated learning framework, which alleviates the non-lID from feature perspective. sanguage-driven training paradigm typically involves text encoder and an image encoder, mapping text and images to their respective feature spaces. In FedGLCC,, we have deployed text encoder on the server side, while each client has an image encoder and a projector. The key is to embed local images and global texts into common space, learning consistent feature presentation among clients. Mcanwhile, the learned metric between image and text features enables making predictions. We illustrate the training and inference of our framework in Figure and describe each part in detail below.
Global Text Embeddings. If training text encoders independently on each local client, the imbal- anced data distribution will lead to multiple skewed local text feature spaces, which runs counter to our objectives (see $5.5). To achieve consistent global text feature spaces, we use pre-trained text encoder embed the set of N texts, producing N text embeddings T (Ti, T2. ... Tv)
5

---

Published as a conference paper at ICLR 2025
RNXV The above process is carried out solely on the server side before training, and previous work (Li et al., 2021; 2020c; Zhou & Konukoglu, 2023; Diao et al., 2024; Zhang et al., 2022) typ- ically assumes that the complete set Y of classes is known. The class is embedded into context prompt such as "a photo of {class] where "class" is the class name. The global text embeddings will be transferred all clients for local training.
Local Training. On the k-th client, the local model f has an image encoder & serves as the backbone extract the image features, and lightweight projector H to align the dimensions of image and text features:
= wx = w U U
where uo and w are the parameters of backhone and projector. Given a training instance (ma, ys) ~ Pk. where Ha E RC*H*W is an image with spatial size (H x W) and C channels, ye E [N] the label, we can get the corresponding image embedding as follows:
I = faz wk)
ERV.
(5)
Following this, we can get the similarity between image embedding and global text embeddings as follows:
I, TTT | ITI
8:
ERY
(6)
To align the image and text into a common space, we introduce contrastive learning (Hadsell et al., 2006; Oord et al., 2018). Consider the image embedding I, as query, its positive key is the text embedding Ty, corresponding to class yı. The rest of the I text embeddings are considered negative keys for query, which should be dissimilar to L. Thus, the loss function of Eq. (2) can be rewritten as:
exp(8 /T)
- log Σ ]JE[N] exp(org/T)*
where 7 is the temperature parameter that can control the tolerance for feature differences (Wu et al., 2018; Zhang et al., 2021). Notably, only the backbone and projector are trainable our framework. At the end of local training in each round, clients upload their updated parameters of backbone and projector to the server, and we can update the global model by Eq. (3).
Algorithm 1 shows the detailed training procedure of FedGLCL. Before training, the server first encodes the class names using a pre-trained encoder to acquire global text embeddings, which are subsequently transferred to individual local clients. During the local training stage, each client learns the consistent image feature representations with the guide of global text embeddings. By this, the differences among local models will be decreased, leading aggregation process aligns with FedAvg (McMahan et al., 2017).
better global model. Our model
Language-driven Inference. After optimization through contrastive learning, the trained backbone and projector exhibit strong feature representation capabilities. Intuitively, their image embeddings are closer to the text embeddings of the corresponding class as far as others. Meanwhile, these text embeddings contain class-wiss information, which can be used to distinguish different classes. The learned metrics between image and text can be used for making predictions. Therefore, similar to CLIP, we can use the similarity between image and text embeddings for prediction. The process of prediction for given image æ can be described as follows:
I
TT
ÿ = arg max softmax(
and I H(G(x; w w H),
(8)
|
T
where wg and wiy are the optimized parameters of global backbone and projector.
4 THEORETICAL ANALYSIS
In this section, we provide deep insights into our framework by theoretical analysis. To understand the underlying working principle of our method, we provide the following three theorems.
Theorem 1. Consider FL system with K clients. and K training instances {(zg, {(z,, from each client, satisfy (Ik, σ) ~ Pg. Let = (T..... , } be a set of N text embeddings containing one positive sample and N negative samples, local image embedding Ts- [( ux) Then,

---

Published as a conference paper at ICLR 2025
the total mutual information between K local image embeddings and their positive sample T, has a lower bound
K
K
Σ M(IR: K (N)((N)
Σ L.
(9)
k=1
Jom
Theorem 2. In FedGLCL, the loss function La of client k can be expressed as:
L* = (2:5 Tuz wy.) + E (ar (Ts unx),
(10)
where E is the target objective, and 8 is the regulation term to prevent overfitting.
Theorem indicates that FedGLCL aims to increase the mutual information between local fea- ture representations and global text embeddings. This ensures that all local feature representations converge towards a common objective, reducing the disparity in feature representations. Besides, Theorem 2 further shows that FedGLCL introduces regulation effects into local training, thereby preventing overfitting to the majority classes.
Theorem 3. Let 1 21,22...,,PK be the empirical data distribution and P., ,P K be the true data distribution. Denote the projector H as the hypothesis from hypothesis space H and d be the VC-dimension of H. The number of text embeddings is N. With probability at least 1- 6,
K
max (w)) (W Wx)
nh Cr. ( (T; wk)
Σ
TkPp, (T; wk)
(11)
1=
YAA:
log
N
+
d
log
> k= T1A
2
Σ
k=
71k
Theorem provides the bound of our generalization error and indicates our method can achieve the expected performance. The detailed analysis and proof are provided in the Appendix $B.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets. We conduct experiments on two different non-lID scenarios, including label distribution skew: CIFAR-100 (Krizhevsky et al., 2009) and Fashion-MNIST (Xiao et al., 2017), and feature distribution skew: Office-Caltech-10 (Gong et al., 2012) and Do- mainNet (Peng et al., 2019). For label distribution skew, following (Li et al., 2021; Tan et al., 2022a; Diao et al., 2024), we randomly partition datasets according to the Dirichlet distribution (P& Dir(B)) select C classes for each client. The number of clients is 10 and all participate in training, ie., the sample fraction is 1. For feature distribution skew, we use the subsets from differ- domains clients (Zhou & Konukoglu, 2023; Li et al., 2020c), where the number of clients in Office-Caltechh 10 and DomainNet are 4 and 6, respectively.
Baselines. We compare FedGLCL with several state-of-the-an FL methods for non-IID data, includ- ing FedAvg (McMahan et al., 2017), FedProx (Li et al., 2020b), FedNova (Wang et al., 2020), Scaf- fold (Karimireddy al., 2020), MOON (Li et al., 2021), and FedProto (Tan al., 2022a), which popular baselines in heterogeneous FL. Moreover, we compared some FL methods that are designed for specialized scenarios, such as FedBN (Li al., 2020c) and FedFA (Zhou & Konukoglu, 2023) for the feature distribution skew scenario, and FedLC (Zhang et al., 2022) and FedConcat (Diao al., 2024) for the label distribution skew scenario.
Implementation Details. We implement our method and other baselines by Pytorch and conduct all experiments on an NVIDIA RTX 4090 GPU with 24 GB memory. We adopt AlexNet (Krizhevsky et al., 2012) for Office-Caltech-10 DomainNet and Fashion-MNIST, and ResNet-50 (He et al., 2016) for CIFAR-100. The models are trained with the SGD optimizer, with a learning rate of 0.0001 for Office-Caltech- 10 and DomainNet, and 1 for MaSHion-MNIST and CIFAR-100. Additionally, the batch size set to 64 for CIFAR-100, 32 for Office-Caltec--I1 and DomainNet, and 256 for Fashion-MNIST. The local epoch is set 5. For fair comparison, all methods the above

---

FedAvg Published as a conference paper at ICLR 2025
Test accuracy ( (%) on feature distribution skew scenario. For a detailed comparison, present the test accuracy of each client, i.e., Office-Caltech-10 (Gong et al., 2012): A(Amazon), C(Caltech), D(DSLR), W(Webcam), DomainNet (Peng et al., 2019): C(Clipart), I(Infograph), P(Painting). Q(Quickdraw), R(Real), S(Sketch). Avg. denotes the average accuracy of all clients.
Table 1:
Mffice-Caltech-11
DomainNet
Method
A
C
D
W
Avg.
C
P
Q
R
S
Avg.
FedAvg
50.00
49.77
53.12
88.13
60.25
75.47
37.44
64,45
73.10
73.04
74,18
66.28
FedProx
57.81
46.66
56.25
76.27
59.25
75.28
36.68
62.68
71.20
68.94
71.84
64.43
FodNova
50.00
42.22
62.50
88.13
60.71
75.85
35.00
64.94
70.60
70.66
72.56
64.93
Scaffold
45.83
39.55
67.75
76.66
58.45
69.20
36.37
57.67
55.20
66.88
60.10
57.57
MOON
53.64
44.88
53.12
89.83
60.37
72.24
35.15
65.91
58.10
72.47
64.07
61.32
FedProto
52.08
40.88
78.12
86.44
65.23
76.42
22.83
60.90
89.60
79,86
70.03
66.61
FedBN
60.93
44.44
84.37
86.44
69.04
81.36
29.68
63.97
89.60
82.82
73.10
70.09
FedFA
61.45
49.77
84.37
89.83
71.36
83.27
20.24
71.40
91.40
87.75
83.57
72.94
FedGLCL (Ours)
75.52
64.00
75.00
94.91
77.35
84.60
37.44
79.96
88.50
86.27
82.49
76.54
Test accuracy (%) on label distribution skew scenario. We evaluate all methods on the test of CIFAR-100 (Krizhevsky al., 2009) and Fashion-MNNIST (Xiao et al., 2017) with different partition strategies, where smaller 3 and C means higher heterogeneity.
Table 2:
CIFAR-100
Fashion-MNIST
Method
0.5
1.0
C
C= 3
3 0.3
B 0.5
B 1.0
64.52
66.67
67.75
42.69
74.50
89.46
92.35
92.65
FedProx
63.98
68.64
69.21
38.98
74.04
87.84
92.11
92.51
FedNova
63.19
68.34
68.78
50.07
60.87
90.13
92.49
92.80
Scaffold
59.61
68.77
70.89
29.44
57.84
91.45
92.44
92.95
MOON
66.30
70.91
71.85
51.95
74.40
88.96
91.00
91.36
FedProto
66.64
70.04
70.51
43,95
72.82
88.64
91.92
92.33
FedLC
63.34
65.70
66.91
37.28
71.06
86.49
91.66
92.08
FedConcat
67.50
71.82
72.49
52.29
75.65
89.83
92.20
92.47
FedGLCL (Ours)
68.58
73.52
74.85
56.63
84.47
93.04
93.82
93.98
FedGLCL (Ours) FedProx
68.95
74.02
75.15
58.69
85.28
93.34
93.86
93,93
FedGLCL (Ours) MOON
69.36
74.38
75.56
59.71
84.61
93,36
93.85
93.89
settings. We use the popular language model, i.e., Bert-base (Devlin et al., 2018) as text encoder and V is 640. The temperature is 0.07 as default (Wang & Liu, 2021). Notably, the image backbone is trained from scratch without loading any pre-trained models. More details and experimental results are presented the Appendix $C and $D.
5.2
MAIN RESULTS
In this section, present the overall results of all methods on four popular FL benchmarks under two different scenarios: feature distribution skew in Table 1 and label distribution skew in Table 2. The performance of the method is quantified using Top-1 accuracy.
Comparison with State-of-the-ards As we can see, FedGLCL consistently outperforms all base- lines across different scenarios, achieving new state-of-the-art art performance. In particular, by making lightweight modifications to the local training of FedAvg, our FedGLCL yields significant improve- ments over FedAvg across four different datasets, egg., Office-Caltech. 10: 60.25% - 77.35 %,, Net 66.28% + 76.54%, CIFAR-100: 66.67% - 73.52%, and Fashion-MNIST: 42.69% + 56.63%. This indicates the effectivvness of language-driven representation learning in mitigat- ing data heterogeneity in federated learning. Besides, the methods designed for specific scenarios demonstrate their advantages in the corresponding setting, achieving better performance than other general heterogeneous FL frameworks. For example, FedFA outperforms other baselines in the fea- ture distribution skew scenario while FedGLCL yields higher accuracy than other baselines in the label distribution skew. However, both of them achieve worse performance them our FedGL.C..
8

---

Published as a conference paper at ICLR 2025
which shows the robustness and generality of method for different non-IID scenarios. More- over, our FedGLCL also sigmificantly outperforms the other baselines focused on local training such as FedProx, MOON and FedProto. This indicates that global text embeddings possess stronger guiding capability than model relations (FedProx and MOON) or class prototypes (FedProto). worth noting that Fedl.C, which proposed calibrated cross-entropy loss, achieves worse per- formance than FedAvg, consistent with the results in the previous study (Diao et al., 2024). This highlights the inherent limitation of label-driven representation learning in non-IID scenarios. In contrast, language-drinn representation learning exhibits more generalized and robust performance.
Combined with Other Baselines. The core of FedGLCL lies in using language-drive represen- tation learning to replace the traditional label-driven training paradigm. This means that FedGLCL can substitute for the supervised loss component in the local objective of previous FL. methods, Le., cross-entropy loss. To explore this, we combine our approach with FedProx and MOON and the loss weight is unchanged. Their accuracy presented in Table Notably, after incorporating our language-representaaii
learning as the local training manner, the performance of FedProx and MOON has significantly improved compared to their use of traditional label supervision. Surpris- ingly, find the approximation term in FedProx and the model contrastive loss term in MOON also contribute to the performance improvement of FedGLCL, which provides additional guidance information for language-driven training.
5.3 SCALABILITY
To investigate the scalability of FedGLCL, we build up different scales of federation by splitting the dataset into more clients on CIFAR- 100 (Li et al., 2022c; Diao et al,, 2024). There are two different set- tings: (1) Full: we increase the num- ber of clients from 10 to 30 while keeping the sample fraction at 1. (2) Partial: we the number of clients and sample fraction to (20, 0.5) and (100, 0.1), respectively, keeping the number of clients participating in each round of training the same the default setting (10 training clients per round). Other experimental settings remain the same as the default con- figuration. As shown in Table 3, all methods exhibit decline in perfor- mance as the number of clients in- creases, but FedGLCL still outperforms other baselines. This suggests that language- driven training paradigm is resilient to changes in the number of clients and can be seamlessly deployed in larger federations, even under partial participation settings.
Test accuracy (%) with different number of clients (#K, #SF) CIFAR- 100 under two different partic- ipation setting, i.e., Full and Partial, where #K is the total number of clients, and #SF is the sample fraction of training clients per round. The / is 0.5.
Table 3:
Full
Partial
Method
(20, 1)
(30.
0.5)
(100, 0.1)
FedAvg
66.67
65.32
64.24
64.39
39.22
FedProx
68.04
66.02
65.30
65.32
40.78
FedNova
68.34
67.36
63.68
63.97
37.81
Scaffold
68.77
66.12
65.38
64.22
39.75
MOON
70.91
67.30
65.55 65.85
65.36
42.54
FedProto
70.04
69.44
66.61
63.24
38.32
FedLC
65.70
64.20
62.95
60.49
37.65
FedConcat
71.82
70.02
68.77
66.05
44.98
FedGLCL (Ours)
73.52
71.71
70.20
69.66
49.17
5.4 EFFICIENCY ANALYSIS
Communication Cost. Since the size of global text embeddings is N 640, which smaller the dimension of the classifier layer, ie.,, N x 2048 for ResNet-50 and N x 4096 for AlexNet. Therefore, FedGI.II does not incur additional communication overhead, and its communication overhead is even slightly lower than that of FedAvg. For example, FedGLCL reduces the commu- nication overhead per round by 0.12M on CIFAR-100 compared to FedAvg. This highlights the advantage of FedGLCL over traditional label-driven representation learning in terms of communi- cation efficiency.
Communication Efficiency. In Figure 3, show the test accuracy of several FL frameworks versus communication rounds on two scenarios. Our FedGLCL exhibits a faster convergence speed than FedAvg in two data heterogeneity scenarios, which further indicates the superiority of language-

---

Published as a conference paper at ICLR 2025
FedAvg
MOON
FedP rox
Ours
100
100
100
80
80
80
Accuracy (%)
Accuracy (%)
Accuracy (%)
60
60
60
40
20
0
20
40
60
50 100 200 300
50 100 150 200 250 300 Communication Rounds
Communication Rounds
Communication Rounds
(a) CIFAR-100
(b) Office-Caltech-10
(c) DomainNet
Figure Test accuracy of several FL frameworks versus communication rounds on three benchmarks. The 3 is 0.5 for CIFAR-100 (Krizhevsky et al., 2009).
driven representation learning, Compared with FedProx and MOON, which also emphasize improv- ing local training, FedGLCL achieves faster and more stable convergence on Office-Caltede 10 and DomainNet. It is worth noting that FedGLCL achieves high accuracy with just one communication round on DomainNet, a larger dataset for feature distribution skew. This suggests that the alignment of features between local images and global text in FedGLCL can effectively mitigate the feature shift issue.
5.5 ABLATION STUDY
Table 4: benchmarks. The is 0.5 for CIFAR-100 100 (Krizhevsky et al., 2009).
Ablation studies of FedGLCL on three
FedGLCI vs. FedGL.CL-L. Global text embedding is core insight of method. To further validate our insights, construct FedGLCL-L: each client has trainable text encoder optimized by Eq. (7), and after each round of local train- ing, their encoders are averaged to obtain globally trainable text encoder. The re- sults in Table indicate that skewed local
Variant
CIFAR-100
OfficeCCaltech110
DomainNet
FedAvg
66.67
60.25
66.28
FedGLCI--I
63.47
56.80
58.75
FedGICI
73.52
77.35
76.54
text feature spaces fail to assist clients learning consistent image feature representations, conse- quently resulting in significant performance degradation.
Hyper-parameter Analysis. FedGLCL involves only one pyper-parameter, i.e., temperature To explore its fluence for our method, we tune from {0.01, 0.07, 0.1, 0.5, 1} on CIFAR-
Accuracy (%)
100 with B 0.5 and shows the results in Figure 4. We observe that a smaller value for r leads to better performance, aligning with common belief about in contrastive learning. This is
because a smaller temperature coefficient places more emphasis on distinguishing challenging samples that are similar to the an-
chor, often resulting in more uniform representations (Wang & Liu, 2021).
0.01 Figure 4: Hyper-parameter analysis for т on CIFAR- 100 (Krizhevsky et al., 2009) under 0.5.
0.07 0.1 0.5
CONCLUSION
In this work, we focused on addressing the non-llI issue, a fun- damental challenge in FL. Due to inherent limitations in label-driven FI., this paper explores the applicability of language-driver representation learning in FL. We found that language-driver repre- sentation learning can benefit local models to learn consistent representations and prevent overfitting to majority classes, effectively addressing the non-lID problem, and yielding novel FL framework, representation learning allows to exhibit out-
FedGLCL. The advantage of language- standing performance across different scenarios. Finally, comprehensive theoretical and empirical analysis helps us better understand FedGLCL, thereby providing more insights into the applications of language-driven representation learning in FI.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by the Guangzhou-HKUUS((G2) Joint Funding Program (No. 2023A0310671), the National Research Foundation, Singapore under its Al Singapore Pro- gramme (AISG Award No: AISG2-TC-2021-0033 Agency for Science, Technology and Research (A*STAR) through its AME Programmatic Funding Scheme Under Project A20H4b0141, A*STAR Central Research Fund "A Secure and Privacy Preserving AI Platform for Digital Health", and Agency for Science, Technology and Research (A*STAR) through its RIE2020 Health and Biomedi- cal Sciences (HBMS) Industry Alignment Fund Pre-Positionis (IAF-PP) (grant no. H20C6a0032), and Guangdong Provincial Key Laa of Integrated Communication, Sensing and Computation for Ubiquitous Internet of 2tingg(NN.2223B1122200000
REFERENCES
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Fed- erated learning with personalization layers. arXiv preprint arXiv:19/2.008/8, 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pp. 531-540. PMLR, 2018
Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open-wordd semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11165-11174, 2023.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In International conference on machine learning, pp. 2089-2099. PMLR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-258225522 Icve, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint #rXiv:1810.04805, 2018.
Yiqun Diao, Qinbin Li, and Bingsheng He. Exploiting label skews in federated learning with model concatenation. Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
Alp Emre Durmus, Zhao Yue, Matas Ramon, Mattina Matthew, Whatmough Paul, and Saligrama Venkatesh. Federated learning based on dynamic regularization. International Conference on Learning Representations, 2021.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with the- oretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3568. 2020.
Chun-Mei Feng, Yunlu Yan, Shanshan Wang, Yong Xu, Ling Shao, and Huazhu Fu. Specificity- preserving federated learning for mr image reconstruction. IEEE Transactions on Medical Imag- ing, 2022.
Chun-Mei Feng, Bangjun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo Learning federated visual prompt in null space for mri reconstruction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8064-8073, June 2023a.
Chun-Mei Feng, Kai Yu, Nian Liu, Xinxing Xu, Salman Khan, and Wangmeng Zuo. Towards instance-adaptive inference for federated learning. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pp. 23287-23296, October 2023b.
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trends@ in Computer Graphics and Vision, (63-44):63--333 2022.
11

---

Published as a conference paper at ICLR 2025
Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhoue ZZong Xu. Fedde: Feder- ated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 101 10112-1012,, 2022.
Zijian Gao, Jingyu Liu, Weiqi Sun, Sheng Chen, Dedan Chang, and Lili Zhao. Clip2tv: Align, match and distill for video-text retrieval. arXiv preprint wrxi:2211.005110 2021.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference computer vision and pattern recognition, pp- 2066-2073. IEEE, 2012.
Tao Guo, Song Guo, Junxiao Wang, Xueyang Tang, and Wenchao Xu. Promptfl: Let federated learning in age foundation
participants cooperatively learn prompts instead of model. IEEE Transactions Mobile Computing, 2023.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference computer vision and pattern recognition (CVPR'00) volume 2. pp. 1735-1742. IEFE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Wenbin He, Suphanut Jamonnak, Liang Gou, and Liu Ren. Clip-s4: Language-guided self- supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11207-11216, 2023.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06330 2019.
Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with do- main shift: A prototype view. In 2023 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16312-16322. IEEE, 2023.
Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Lc, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916 PMLR, 2021.
Meirui Jiang, Zirui Wang, and Qi Dou. Harmofl: learning heterogeneous medical images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 1087-1095, 2022.
Harmonizing local and global drifts in federated
Jiawen Kang, Zehui Xiong, Dusit Niyato, Yuze Zou, Yang Zhang, and Mohsen Guizani. Reliable federated learning for mobile networks. IEEE Wireless Communications, 27(2) 2020.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learn- ing. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Confer- ence on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5132-5143. PMLR, 13-18 Jul 2020. URL https::/ karimi reddy20a aaa..... html.
https://ppocceede http:///occeeeiee ngs. mar..mlrr.err www..prres//vv11..
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMLR. 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
12

---

Published as a conference paper at ICLR 2025
Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, and Se- Ye-Young Yun. Preservation of the global knowledge by not-true distillation in federated learning. Advances in Neural Information Processing Systems, 35:38461-38474, 2022.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): (144):1234-1240) 2020.
Sunwoo Lee, Tuo Zhang, and A Salman Avestimehr. Layer-wise adaptive model aggregation for scalable federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8491-8499, 2023.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2022a. URL https: attps://openreviiie
net/ forum?id-R:i
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang. Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei
Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
10965-10975, 2022b.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10713-10722, 2021.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 965-96658888 978. IEEE, 2022c.
Qinbin Li, Bingsheng He, and Dawn Song. Adversarial collaborative learning on non-III features. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 2023. URL https:::........... : / /ttp/ppoeeenn/ee iig.ing..... Sal...pres/vv000
PMLR, 23-29 Jul
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50 60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 2:429 450, 2020b.
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learn- ing on non-iid features via local batch normalization. International Conference on Learning Representations, 2020c.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Quande Liu, Qi Dou, Lequan Yu, and Pheng Ann Heng. Ms-net: multi-site network for improving prostate segmentation with heterogeneous mri data. IEEE transactions on medical imaging, 39 (0):2713-2724, 2020.
Wang Lu, Xixu Hu, Jindong Wang. and Xing Xie. Fedelip: Fast generalization and personalization for clip in federated learning. arXiv preprint arXiv:2302. 13485, 2023.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei. Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing. 508: 293-304, 2022.
Mi Luo, Fei Chen. Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-5984 2021.
13

---

Published as a conference paper at ICLR 2025
Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for person- alized federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10092-10101, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficeen learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Solmaz Niknam, Harpreet S Dhillon, and Jeffrey H Reed. Federated learning for wireless commu- nications: Motivation, opportunities, and challenges. IEEE Communications Magazine, 58(6):
16, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1r0.vv/87..03772 2018.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclipp Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 2085-222 2094,, 2021.
Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1406-1415, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34: 16-11228-, 2021.
Xinyi Shang, Yang Lu, Gang Huang, and Hanzi Wang. Federated learning heterogeneous and long-tailed data via classifier re-training with federated features. International Joint Conference Artificial Intelligence, 2022.
Jiangming Shi, Shanshan Zheng, Xiangbo Yin, Yang Lu, Yuan Xie, and Yanyun Qu Clip-guided federated learning on heterogeneous and long-tailed data. arXiv preprint 23Xiv:3312.08644 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang, Fed- proto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1432-8440, 2022a.
Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Federated learning from pre-trained models: A contrastive learning approach. Advances in Neural Information Processing Systems, 10399199344999 2022b.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2495-2504, 2021.
Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-766 17776611762,,. 2020.
Mengmeng Wang, Jiazheng Xing. Jianbiao Mei, Yong Liu, and Yunliang Jiang. Actionclip: Adapt- ing language-img -maruegg--maeeeaeoe pretrained models for video action recognition. IEEE Transactions on Neural Networks and Learning Systems, 2023.
14

---

Published as a conference paper at ICLR 2025
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 11686-11695, 2022.
Jeffry Wicaksana, Zengqiang Yan, Dong Zhang, Xijie Huang, Huimin Wu, Xin Yang, and Kwang-
Ting Cheng. Fedmix: Mixed supervised federated learning for medical image segmentation. IEEE Transactions on Medical Imaging, 2022.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Monika Wysoczańska, Michaäl Ramamonjisoa, Tomasz Trzciński, and Oriane Siméoni. Clip-diy: Clip dense inference yields open- ppenvoocbuulary semantic segmentation for-free. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1403-1413, 2024.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnis a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong
Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1813418844, 2022.
Yunlu Yan, Chun-Mei Feng, Yuexiang Li, Rick Siow Mong Goh, and Lei Zhu. Federated pseudo modality generation for incomplete multi-modal mri reconstruction. arXiv preprint arXiv:2308. 10910, 2023.
Yunlu Yan, Hong Wang, Yawen Huang, Nanjun He, Lei Zhu, Yong Xu, Yuexiang Li, and Yefeng Zheng. Cross-modal vertical federated learning for mri reconstruction. IEFE Journal of Biomed- ical and Health Informatics, 28(1 1) 11:633446333 2024a.
Yunlu Yan, Lei Zhu, Yuexiang Li, Xinxing Xu, Rick Siow Mong Goh, Yong Liu, Salman Khan, and Chun-Mei Feng. A new perspective to boost performance fairness for medical federated learning. In International Conference Medical Image Computing and Computer-Assiste Intervention, pp. 13-23. Springer, 2024b.
Dezhong Yao, Wanning Pan, Yutong Dai, Yao Wan, Xiaofeng Ding, Chen Yu, Hai Jin, Zheng Xu, and Lichao Sun. Fed gkd: Towards heterogeneous federated learning via global knowledge dis- tillation. IEEF Transactions Computers, 2023.
Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogeneous federated learning: State-of oftte-of-the-re the-art and research challenges. ACM Computing Surveys, 56(3):1 44, 2023.
Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research. pp. 263 11 26329. PMLR, 17-23 Jul 2022. URL https: https:////ccceeeeeeeeeeee mmlr. press/v 62/6222222 / / wwwgg22nnnnppppc html.
Oliver Zhang, Mike Wu, Jasmine Bayrooti, and Noah Goodman. Temperature as uncertainty in contrastive learning. arXiv preprint a 10.04403, 2021.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint 1 2018.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pp. 696-712. Springer, 2022.
Tianfei Zhou and Ender Konukoglu. Fedfa: Federated feature augmentation. In The Eleventh International Conference Learning Representations, 2023.
15

---

Published as a conference paper at ICLR 2025
A EMPIRICAL ANALYSIS
In this work, we found that language-driven representation learning has two inherent advantages for FL under non-III data. First, effectively reduces the feature representation difference among local models. We employ the Centered Kernel Alignment (Korbblihh et al., 2019) (CKA) similarity to quantify the differences in feature representations among different local models. CKA a reliable technique for quantifying the similarity between pairs of feature representations, widely used in various applications as an analytical tool (Raghu et al., 2021). Specifically, we use the global model as the anchor and calculate the CKA similarity for each layer of the network between all local models and the anchor. We then visualize the average CKA scores of all local models in Figure 1 (c). The results indicate that the representation disparity among local models under non-IID is primarily concentrated in the deep layers of the network (A larger index means a deeper layer), and language- driven representation learning can effectively mitigate these differences. This is facilitated by the text-image feature alignment of language-driven representation learning. Second, it helps prevent local models from overfitting to the majority classes. Label- -riven representation learning can easily lead to overfitting on the majority classes due to a lack of supervised information on minority classes. In contrast, during the training process, language-driven representation learning not only brings the features of images closer to the corresponding textual features of their corresponding class but also pushes them away from the text features of other classes, thereby indirectly introducing supervisory information from other classes. This process effectively prevents the overfitting of local models and enhances their generalization performance (see Figure I (b)). Building upon the above two aspects, language- driven representation learning achieves consistent improvements in two different federated scenarios, and its enhancement for FedAvg is superior to that in centralized learning (see Figure (a)). Motivated by this, we suggest using language-driven representation learning as the local training paradigm to address the non-III issue.
B THEORETICAL ANALYSIS
B.I ANALYSIS OF WORKING PRINCIPLE
As stated in $1, language-driven representation learning exhibits two empirical phenomenons, i.e. mitigating feature difference and preventing overfitting, which benefit for addressing the non-III issue. In this section, we provide a theoretical analysis to understand the working principle of FedGLCL. We first explain why FedGLCL can reduce the feature representation differences among local models from the perspective of mutual information (Belghazi et al., 2018), which is a com- monly used method to quantify and optimize the differences feature representations. First, we introduce the following lemma from (Oord et al., 2018) for contrastive learning.
Lemma 1. Given set X (x1,..., ,NN of N random samples containing one positive sample and N - 1 negative samples for latent representation c, and optimized with the contrastive loss LN. Then, the mutual information M(c, between and its positive samples 24 X has lower bound
M(c,24)) 2 log(N) LN-
(12)
The above lemma illustrates that when we optimize the contrastive learning loss Lv it enhances the lower bound of mutual information M(c, In other words, it reduces the representation difference between x and c. Since our FedGLCL has the same form with the LN of (Oord et al., 2018), the above lemma can be directly applied in our setting. The set X can be considered as global text embeddings T and e is replaced with local image embeddings I. Therefore, we provide the following theorem:
Theorem 1. Consider FL system with K clients, and K training instances { k from each client, satisfy (Ik, d) ~ Ps, where a E [N] represents the class index. Let T = (T..... Tn) be a set of N text embeddings containing one positive sample and N negative samples, local image embedding In [(x;; wx). Then, the total mutual information between K local image embeddings and their positive sample Ty has lower bound
K
K
Σ M(I. 2 K (Nog(N)
Σ 1=
Cr.
(13)
16

---

Published as a conference paper at ICLR 2025
Proof. In FL, each local training is conducted independently and in parallel. Thus, for each client E [K], we have
M T,) 2 log(N) L*
(14)
Therefore, with sufficient updates of local models, I& converges to T. The total mutual information of clients satisfy
k
M(L To) 2 (log(N) Li) + (log(N) L2) +.... (log(N) Lx)
(15)
(Voog(N>
Lx
k=1
Theorem I implies that minimizing the global empirical loss (Eq. (1)) is equivalent to maximiz- ing the sum of mutual information between the feature representations K clients and the posi- tive text embedding. Due to the independent training of each client, Ex Ls is minimized and Σx MM T,) is maximized only if vL Te Since T, is consistent and fixed for all clients, FedGLCL ensures that all local embeddings converge to the same target, thereby mitigating the fea- ture difference. Notably, both the number of samples and the number of clients are K, so we use the same notation.
Furthermore, we offer additional insight into how our introduced loss function regulates local train- ing, aiming to prevent overfitting on the majority class of local datasets.
Theorem 2. In FedGLCI, the loss function La of client K can be expressed as:
Cx = et (z,; T wK) E (xg : wk),
(16)
where e is the target objective, and E- is the regulation term to prevent overfitring.
Proof. The local loss function can be transformed:
L E(zin))pP
exp(oi../7)
log EjE(N)itM exp(6rs/T)
Σ JE(N],j
01ghT - 8i.0eTT
I T.
I. TT
(17)
Σ
I |
| I
jE(N
f(z
TT
((;: T (z((=:))))l)
Σ
[lf(z w ) |||,
JE(N Jæ
- (TibeeINN w)) Et ( wp);
Then, we have
[xx;; wxx Ty
et (x; Ty. wx) = Ben.ee)PP T (2 wy) (ll)lll [l]
(18)
(mawwy) T.
(0x ( =E (
Σ
(19)
jE(N)Jj
Compared with traditional cross-entropy loss, local loss function introduces the information from all classes. By optimizing e (x;; ; wx). ). fw, wx) is forced to approach Ty,, Besides, regularization term to keep f(z;; ws) far away
e (ax;; (Tibse(N) wxx be considered from VT E (TihieiM)issss which can prevent the local model overfitting to majority class of local datasets.
17

---

Published as a conference paper at ICLR 2025
B.2 GENERALIZATION BOUNDS
In this section, we analyse the performance of FedGLCL from the perspective of generation error. Before giving our eтrоr bound, we provide the following lemma from (Tan et al., 2022b), which is a prototype-basee FL framework.
Lemma 2. Consider a FL system with mn clients. Let D1, D2...., Dm be the data distribution and D1, Di..-.. DK be the empirical data distribution. Denote the projector head h the hypoth- esis from H and d be the VC-dimension of H. The total number of samples overall clients is N. Then, with probability at least 1
.......
m
Dd N
m
De N
max 021,02 Pmn)
Le COSCC (Cp)
L.D,
(20)
N log
(m +
T)|C|
eN log d'
V
where (m + 1)C is the total number of local and global prototypes, e is the base of the natural logarithm.
Since text embeddings T contain the class- wise information, we find the above lemma can be di- rectly applied in our framework by considering T as type of global prototype. This leads to the below theorem.
Theorem 3. Let P1, PI,,P PK be the empirical data distribution and P, P2...... PP be the true data distribution. Denote the projector H as the hypothesis from hypothesis space H. and d be the VC-dimension of H.. The number of text embeddings is N. With probability least 1 5,
K
max (w) W........, W.
7k Cr. (T;
Th Tk. (T:wk)
(21)
K k=
T&k
log
+
d
log
e
7 >
k=
2
8
Σ
d
k=
14k
The above theorem indicates that FedGLCL can achieve the expected performance with appro- priate projection network and the number of classes.
C DETAILS
C.I DATASET
We illustrate the detailed information of four datasets: CIFAR-100 and Fashion- in Table 5, Office-Caltech. Caltech-10 10 in Table 6, and DomainNet Table 7. For CIFAR- 100 and Fashion-MNIST, the ratio of training and testing provided by the dataset, where there 50000 training instances for CIFAR-100 and 60000 training instances for Fashion-MNIST. Both of them have 10000 instances for testing. The training will be split into multiple clients for training by two different Non-IIID partition strategies (Diao et al., 2024; Zhang et al., 2022): 1) Sharding: we randomly allocate C classes each client and keep an equal number of samples for each client. 2) LDA; we utilize the Latent Dirichlet Allocation (LDA) strategy to divide the data among clients, where each local dataset sampled from a Dirichlet distribution (IP& ~ Dir(8)). C and 8 are the data heterogeneity of experiments, and the smaller value indicates higher data heterogeneity. After training, we evaluate the global model of all methods on the testing set (Diao al., 2024; Li et al., 1)2 For Sificc-Caltech1111 10 and DomainNet. following (Li al., 2020c; Zhou & Konukoglu, 2023), we partition the dataset each client into 8 for training: testing. We report the test accuracy of each client and their average result for all methods.
that control
C.2 METHOD
It's worth noting that there are some important can affect the performance of the method. For these hyper-parameters and experimental details,
the compared baselines, which
18

---

Published as a conference paper at ICLR 2025
Table 5: Detailed information of CIEAR-100 (Krizhevsky et al., 2009), Fashion-MNIST (Xiao et al., 2017), and Tiny-ImageNet.
Property
CIFAR-100
Fashion-MNIST
Tiny-ImageNet
# train samples
50000
60000
100000
test samples
10000
10000
10000
of classes
100
10
200
Image size
(32, 32.
(28, 28, 1)
(64, 64, 3)
Table 6: Detailed information of Office-Caltec--10 (Gong et al., 2012). total.
There are clients in
Office-Calteh-110
Property
Amazon
Caltech
DSLR
Webcam
# train samples
766
898
125
236
of test samples
192
225
32
59
# of classes
10
10
10
10
Image size
(256, 256, 3)
(256, 256, 3)
(256, 256, 3)
(256, 256,
Table 7: Detailed information of DomainNet (Peng et al., 2019). There are 6 clients in total.
DomainNet
Property
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
# train samples
2103
2626
2472
4000
4867
2213
of test samples
526
657
619
1000
1217
554
of classes
10
10
10
10
10
10
Image size
(256, 256, 3) (256, 256, 3) (256, 256, 3) (256, 256, 3) (256, 256, 3) (256, 256, 3)
Table Detailed structure the projector for ResNet-50. The projector a lightweight module, consisting of two fully connected layers (FC). We list parameters with sequence of input and output
dimensions.
Layer
Details
Layer1
FC(2048, 2048), ReLU, Normalize
Layer2
FC(2048, 640), ReLU, Normalize
we strictly adhere the descriptions provided by their paper. For example, the local objective function of FedProx, MOON and FedProto can be expressed as L L1 + μu22, where Li and La are supervised loss and auxiliary Loss, μ the Loss. For FedProx, tune μ from the range {0.0001, 0,001, 0.01, 0.1J. and empirically set to 0.001 for CIFAR-100 and Fashion-MNIST,, and 0.0001 for Office-Caltech- 10 and DomainNet. For MOON and FedProto, the is set to 1, and the temperature parameter of MOON is set to 0.5. As for FedConcat, the encoder and classifier are trained independently, with the encoder being trained first, followed by the training of the classifier. The training communication rounds for the encoder are 50 rounds, while for the classifier, they are 1000 rounds. To ensure a fair comparison, all baselines employ the same model structure, integrating the projector into ResNet-50 (backbone, projector, classifier). The projector is a two-layer Multi-Layer Perceptron (MLP), which is presented Table 8. However, FedGLCL differs by removing the classifier, retaining only the backbone and projector components. In the case of AlexNet, its classifier comprises three consecutive linear layers. FedGLCL discards the last linear layer of the classifier and utilises the remaining two linear layers as the projector while all baselines maintain the full classifier structure.
to control the influence of auxiliary
D ADDITIONAL EXPERIMENTS
D.1 PROMPTING ENGINEERING
To explore the impact of prompting engineering the method, build up variant Mj by moving prompt templates and directly using class names "[class]". The results in Table 9 show varying degree of performance decline across the three datasets after removing the prompts. This
19

---

Published as a conference paper at ICLR 2025
Table 9: Ablation studies about the prompting engineering FedGLCL on three benchmarks. The 3 is 0.5 for CIFAR-100.
Variant
CIFAR-100
Office-Caltech-10
DomainNet
FedAvg
66.67
60.25
66.28
FedGLCL + "(class)
72.42
75.91
75.29
FedGLCL + Prompt ensemble
74.30
77.92
77.02
FedGLCL + photo {class}"
73.52
77.35
76.54
Table 10: Test accuracy (%) with different text embeddings three benchmarks. The /3 is 0.5 for
CIFAR- 100
Variant
CIFAR-100
Office-Caltech-10
DomainNet
Random initialization tensors
38.99
35.09
42.13
Random orthogonal tensors
67.58
62.60
69.22
Proxy text embeddings
72.80
77.52
75.83
FedGLCL
73.52
77.35
76.54
Table 11:
Test accuracy (%) with vgg-11 CIFAR-100. The 3 is 0.5.
FedAvg
FedProx FedNova Scaffold MOON FedProto FedLC FedConcat FedGLCL
57.57
54.64
58.12
58.43
60.12
59.63
55.52
60.94
65.45
Table 12:
Test accuracy ( (%) with vgg- on Office- -aaicc--aatech--ae 10.
FedAvg
FedProx 73.54
FedNova
Scaffold
MOON
FedProto
FedBN
FedFA
FedGLCL
74.97
73.27
72.44
75.05
78.64
81.55
82.68
85.30
suggests that text prompts contribute to enhancing the semantic information of text embeddings. However, after removing textual prompts, FedGLCL still significantly outperforms traditional label-driven representation learning methods, demonstrating the superiority of language-driven rep- resentation learning for heterogeneous data.
Besides, following previous works (Zhou et al., 2022), we employ prompt ensemble by feeding prompt-engineered texts into the text encoder with 85 different prompt templates, and average 85 text embeddings of the same class. The results show that this ensemble strategy yields better text em- beddings, leading to improved performance. This further demonstrates the scalability of FedGLCL.
D.2 DIFFERENT TEXT EMBEDDINGS
In this section, we explore the impact of different types text embeddings on our approach. First, conduct experiments by replacing the global text embedding with randomly initialized and ran- domly orthogonal tensors implemented by Pytorch. The main idea of FedGLCL to leverage the semantic information contained in global text embeddings for global supervision. However, as shown in Table 10, randomly initialized feature embeddings struggle to provide effective semantic supervision, leading sharp decline in performance. Random orthogonal matrices can be type of supervision, but manually crafted feature embeddings significantly lower the quality of global text embeddings, which are learned from massive data. In addition, manually crafted feature embeddings can hinder the convergence of the model.
Besides. we use fake class names as proxies by swapping the class names between 10 and DomainNet, and randomly selecting 100 class names from ImageNet (Deng et al., 2009) for CIFAR-10M Then, we utilise the text encoder to encode them as the global text embedding for both training and inference. Surprisingly, we found that it achieved comparable performance to using text embeddings from real classes as shown Table 10. In FedGLCL, local image encoders are trained from scratch without any prior knowledge. Therefore, the critical factor is the consistency of text embeddings between training and testing. This ensures that the encoded text embeddings can effectively guide the learning process, regardless of whether the class names perfectly correspond to the images. While the proxy class names are not the real classes of images, they are still in the do- main covered by the pre-trained text encoder. Consequently, the global text embeddings still provide
20

---

CIFAR- Published as a conference paper at ICLR 2025
Table 13: CIFAR-100.
Test accuracy (%) with different text encoders on three benchmarks. The 0.5 for
Variant
CIFAR-100
Office-Caltech-10
DomainNet
Glove
68.95
71.51
70.48
CLIP/ResNet-50
73.30
77.26
76.41
CLIP/ViTB-16
72.88
75.99
76.09
Bert-base (default)
73.52
77.35
76.54
Table 14:
Test accuracy (%) with ResNet-50 on CIFAR- 100-LT (Shang et al., 2022).
FedAvg
FedProx
FedNova
Scaffold
MOON
FedProto
FedLC
FedConcat
FedGLCL
45.29
48.23
45.49
49.40
51.13
50.76
48.92
50.53
56.14
Table 15: Training efficiency with ResNet-50 on CIFAR-100.
Method
FedAvg
FedProx FedNova
Scaffold
MOON
FedProto
FedLC
FedGLCL
Time/Round (s)
12.58
21.69
13.68
18.83
21.14
15.05
13.13
14.13
Rounds
88
93
96
92
Total Time (S)
1107.04
1973.79
1272.24
1807.68
1966.02
1309.35
1037.27
1299.96
meaningful semantic supervision for training. This result highlights that the success of language su- pervision is not dependent the precise semantic relationships between class names and images rather on the overall structure and knowledge encoded in the global text embeddings. This not only offers effective supervision for training local image encoders but, more importantly, ensures the alignment of all local image feature spaces with the global text feature space for mitigating drift in local models. Besides, this also suggests that use proxy texts to deal with the domain shift of text and privacy encryption.
D.3 DIFFERENT IMAGE BACKBONES
We conducted experiments using "gg-11 (Simonyan & Zisserman, 2014) for all methods on CIFAR- 100 and Office-Caltecht Table 11 and 12. As one can after employing different network architectures, the improvement of FedGLCL compared to FedAvg remains notable, i.e., 7.88% and 10.33% on the two datasets, re- spectively. Besides, FedGLCL consistently outperforms all compared baselines under two different non-III settings. The above results demonstrate the generalizability of FedGL LCL across different network architectures
where the 3 is set to 0.5 for CIFAR-100. The results are as shown in
D.4 DIFFERENT TEXT ENCODERS
To explore the impact of different pre-trained language models, we used another popular vision- language model, CLIP (Radford et al., 2021). Specifically, we only use the text encoder of CLIP/ResNet-50 and CLIP/ViTB-16 as the text encoder of FedGLCL, and the results are presented in Table 13. After using different text encoders,
method still outperformed the best baseline. This demonstrates the robustness of FedGLCL to various text encoders.
Besides. compared Glove (Pennington al., 2014), traditional word embedding method. The results show that GloVe outperforms FedAvg, but achieves lower performance than Bert and CLIP. This is because Bert and CLIP employ deep neural networks to learn data representations through supervised learning on large-scale datasets, allowing them to capture semantic information more effectively.
D.5 EXPERIMENT ON LONG-TAILED SETTING
To further validate the effectiveness of FedGLCL in mitigating overfitting to the major classes, following previous work (Shang et al., 2022), conducted long-tailed FL experiments 100-LT. The laiget distribution is controlled by imbalance factors (IF), which is set to 10, and is set to 0.5. The results, illustrated in Table 14, show that the improvement of FedGI.CL over
21

---

Published as a conference paper at ICLR 2025
Table 16:
Test accuracy (%) with ResNet-50 on Tiny-ImageNet. The 3 is 0.5.
FedAvg
FedProx
FedNova
Scaffold
MOON
FedProto
FedLC
FedConcat FedGLCL
47.54
46.61
47.11
48.09
48.97
48.16
45.97
49.24
51.05
Table 17: Results of statistical significance. For accuracy, we report mean + std over three trials.
CIFAR-100
Office- 10fice-Caltech1100
DomainNet
Method
Accuracy
P-value
Accuracy
P-value
Accuracy
P-value
FedAvg
66.64 + 0.14
0.0016
60.57 0.85
0.0003
66.58 0.70
0.0007
FedConcat
71.09 + 0.51
0.0138
FedFA
71,50 + 1.03
0.0026
72.27 + 0.72
0.0031
FedGLCL
74.10-t 0.41
77.84 + 0.96
77.13 + 0.53
Table 18:
Test accuracy (%) with AlexNet on Office-Caltechs 10 (Gong et al., 2012), where the image backhone is initialized with pre-trained weights from ImageNet.
FedArg FedProx
FedNova Scaffold
MOON FedProto FedBN
FedFA
FedPCL
FedGLCL 81.81
64.99
62.46
65.50
61.33
66.78
69.37
71.87
74.91
72.46
FedAvg is more significant in long-tailed FL settings as large as 10.85% (higher than 6.85% in Table 2). This further demonstrates that FedGLCL can prevent overfitting the major classes.
D.6 TRAINING EFFICIENCY
We present the local training time per round of single client on CIFAR-100, 100, and the average result of all clients is reported. We also report the number of rounds for each method to reach the best accuracy. Notably, since FedConcat's backhone and classifier are trained independently, the number of rounds and training time for each stage differ, it is challenge to evaluate its time using the above strategy. The results are presented in Table 15. Compared to FedAvg, the additional training overhead in FedGLCL primarily comes from the similarity computation between image and text features. However, this overhead is minimal as it leverages Py TyToch'ss 's efficient matrix operations, resulting in only a slight increase in Time/Round. Additionally, we compared FedGLCL with other methods that incorporate auxiliary losses, such as FedProx, MOON, and FedProto. The auxiliary losses FedProx and MOON lead to significantly higher overhead compared to our method. In summary, both the training time per round and the total training time for FedGLCL are acceptable.
D.7 EXPERIMENTS ON
SCALE DATASETS
We conduct experiments using a larger-scale dataset, Tiny-ImageNet' and its detailed information presented in Table 5. We use the same experimental setup for CIFAR-100, and / is 0.5. The results in Table 16 show that FedGLCL consistently achieves the best performance compared all baselines, further demonstrating the effectiveness our method.
D.8 STATISTICAL SIGNIFICANCE
In this section, conduct two additional trials using different random seeds and report the mean and standard deviation (std) across all three trials. Furthermore, to evaluate the statistical signifi- cance of the performance improvements, we perform paired t-test between the baseline and our method, reporting the corresponding p-value. The above results of FedAvg, FedGLCL, and two best baselines, i.e., FedConcat and FedFA, presented in Table 17. It be observed that the p-values for all baselines are less than 0.05, indicating the statistical significance of the performance improvements achieved by our method.
22

---

Published as a conference paper at ICLR 2025
Table 19: Test accuracy (%) on ProstateMRI (Liu et al., 2020). For a detailed comparison, we present the test accuracy of each client. Avg. denotes the average accuracy of all clients
ProstateMRI
Method
BIDMC
HK
12CVB
BMC
RUNMC
UCL
Avg.
FodAvg
87.66
94.48
96.00
90.46
93.21
87.47
91.55
FedGI.CI.
91.81
94.89
95.95
92.08
93.34
90.68
93.12
D.9 EXPERIMENTS WITH PRE-TRAINED IMAGE MODELS
We conducted additional experiments on the Office-Caltech-10 dataset using pre-trained image backbone. In this experiment, the image backbone (AlexNet) was initialized with pre-trained weights from ImageNet, while all other settings remained unchanged. The results in Table 18 demonstrate that incorporating the pre-trained image backbone improved the performance of all methods, with our approach consistently achieving the best results. This confirms that under fair comparisons (where all methods either utilize the pre-trained image backbone or do not), FedGLCL significantly outperforms other methods, further highlighting its effectiveness and superiority.
Besides, we compared a new baseline, i.e., FedPCL (Tan et al., 2022b), a prototype-based FL method specifically designed for pre-trained image backbones. As cantly worse than our method, demonstrating the superiority of FedGLCL over prototype-based FL methods.
see, FedPCL performs signifi-
D.10 EXPLORATION ON MORE TASKS
In this section, we extend FedGLCL to more visual tasks. Inspired by (Li et al., 2022a), we extend the image-text alignment to pixel-text alignment for image segmentation. We conducted experiments on ProstateMRI (Liu et al., 2020), widely used federated medical image segmentation benchmark. This benchmark has six clients including BIDMC, HK, I2CVB, BMC, RUNMC, and UCL, each from a different domain. We used U-Net as the segmentation network and aligned the output of the final layer with the text features. Since this benchmark involves binary segmentation task, use "foreground" and "background" as the class names. The communication rounds are set to 200, with local rounds
1. The results presented in Table 19. It can be observed that FedGLCL. achieves certain improvements compared to FedAvg, indicating that our method has the potential to be applied to other tasks to address non-lID data issue.
E PRIVACY SECURITY
FedGLCL effectively utilizes language-driven representation without leaking privacy. (1) We share the encoded text embeddings instead of directly sharing the category text. Additionally, we conduct the encoding process on the server side, making invisible to individual clients, thus preventing reverse inference of the text. (2) Furthermore, the introduced prompts can increase the complexity of the text, thereby further enhancing privacy security. (3) Finally, we can further enhance privacy security by using proxy texts.
F LIMITATION
Domain shift in pre-trained models is fundamental challenge and can be mitigated in this work. Actually, FedGLCL is not limited by domain shift issues because language exhibits consistency in real-world applications. For example, the word "cat" holds the same semantics in both natural images and cartoon image tasks. However, for image backbones, the natural images and the car- toon images hold two entirely different data distributions. On the contrary, the image encoder of
23

---

Published as a conference paper at ICLR 2025
FedGLCL is trained from scratch without any prior knowledge and thus is irrelevant to the pre- trained CLIP image encoder. For the domains not covered by text encoder, we can utilize the corre- sponding text encoder for specific data domains, e.g., BioBERT (Lee et al., 2020) for medical image tasks. Besides, the ablation results in Table 10 indicate that we can use in-domain class names as proxies when deployed in domains not well-covered by pre-trained text encoder. This can broaden the application scenarios of FedGLCL. Although FedGLCL is not limited by domain shift, it may have limitations in larger vocabulary scenarios (e.g., 10,000 target categories). Text encoders may struggle to effectively encode rich class vocabulary to provide effective semantic information. In the future work, we will thoroughly explore it.
24


Reviews:
[{"review_id": "7JNY5ynkJn", "reviewer": "Reviewer_yM3c", "summary": "The paper introduces FedGLCL, a FL framework designed to improve the handling of non-IID data in FL environments. The authors leverage language-driven representation learning (e.g. CLIP, as opposed to label driven), incorporating a pre-trained text encoder to stabilize feature representations. This approach enables the alignment of local image features with a global text feature space, reducing feature disparity across clients. Experiments demonstrate that FedGLCL outperforms traditional label-driven FL methods in various non-IID scenarios, including label and feature distribution skew.", "questions": "In this paper, federated learning is applied to a specific type of image-text pre-training. I suggest to make this clear in the abstract and throughout the paper - as there are also other types of federated learning as well (e.g. text only)\n\n158 - 161: Can you explain more how your work differs from other works that uses CLIP in FL training (e.g. Lu et al., 2023, Guo et al., 2023, Shi et al., 2023)? In particular - what is \"the philosophy of language-driven representation learning\"?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is novel - mitigating non-iid data issue in FL on image classification pre-training with text encoders.\n\n2. The experiment results are good, the authors experimented on many datasets, comparing against many baselines, highlighting the generalizability of their method. \n\n3. The author provided theoretical guarantees to the generalizability of the proposed method.", "weaknesses": "1. The use of image-text pre-training is prevalent, and the non-iid data issue is not limited to FL settings, so adding text encoders to mitigate the non-iid issue shouldn't be new. Honestly I am not familiar with the FL literature, so I don't know how much this would add to the FL community. \n\n2. The experiments are somewhat limited to small-scaled datasets. Performing experiments on larger scaled datasets (e.g. Imagenet) would make the claims stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bg2GuOhiUm", "reviewer": "Reviewer_bW7e", "summary": "This paper proposes a novel approach, FedGLCL, to learn image models in non-IID federated settings. Their approach leverages a fixed frozen set of class embeddings (from a language model) and locally updates image representations in a CLIP-like contrastive manner. When tested on multiple variations of non-IID federated learning benchmarks (e.g. significant label shift/imbalance), this approach is shown to notably outperform other training/aggregation-based interventions without introducing additional computation/communication costs. Furthermore, the authors provide several additional ablations and theoretical analyses to deepen understanding of their results.", "questions": "Most questions I had were covered in the Weaknesses section. The only other question I had was if you considered any settings involving pretrained image models. On the surface, this might better reflect practical deployments for image-based tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality and Significance.** Federated learning with non-IID client data is a well-motivated/challenging problem and the authors' approach, while simple, is novelly and effectively applied in this setting. As mentioned in Sec 5.4, this approach also importantly does not come with any decreases in efficiency. \n\n**Quality.**  Overall, the results for their method are quite strong compared to a variety of baselines and on several variations of the evaluations. The authors also did a great job ablating aspects of their method (particularly in the Appendix), addressing several of the questions that came up while I was reading the paper. In particular, I appreciated the ablations on the text embeddings, which show that it is important to both (1) keep the embeddings fixed; (2) use a pre-trained language model instead of custom class embeddings. These help establish that contrastive-style training itself is not the main explainer for good performance. \n\n**Clarity.** Generally, l found the paper to be well-written and easy to follow. Particularly, the authors did a good job contextualizing their methods/results with previous works.", "weaknesses": "**More discussion about the impact of language supervision.** One result I found particularly curious (and which I believe should be highlighted more in the main paper) was that using embeddings for randomly mismatched class names results in comparable performance to using embeddings for the correct class names. Before seeing this result, I would have thought that the semantic information and relationships between classes was being usefully leveraged (e.g., similar classes being closer together in text embedding space). But based on this result, I'm not sure then to what extent the \"broad linguistic knowledge\" emphasized in the abstract is important in explaining FedGLCL's success? Perhaps the authors can expand upon their discussion of this in the Appendix and comment more. \n\n**Measuring the impact of FedGLCL-style training in centralized/IID-Federated settings** It would be interesting to see how well the contrastive style training from FedGLCL performs in the non-federated setting (and perhaps also the IID federated setting). This would shed light on whether the gains in the non-IID federated setting might also be explained by the contrastive learning approach in FedGLCL-being more effective generally on these tasks v.s. if it is uniquely helpful for overcoming challenges in non-IID federated settings. Relatedly, it would also be nice to discuss your results in relation to Fang et al., 2022 (https://arxiv.org/abs/2205.01397), which included results assessing the impact of language supervision on CLIP pre-training. in their case, they find that it does not have a major impact on the final robustness of CLIP models.\n\n**Standard errors for results.** It would be nice if the empirical results came with standard errors. That being said, I can acknowledge that the current gaps between the proposed method and previous baselines seem to be large/consistent across evaluation settings and that repeated runs might be costly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "46GRRM93Ju", "reviewer": "Reviewer_4piz", "summary": "The paper presents FedGLCL, a novel federated learning (FL) framework designed to address the challenges posed by non-Independent and Identically Distributed (Non-IID) data. The framework leverages a pretrained VLM, CLIP, specifically through contrastive learning that integrates global language and local image features. The proposed method aims to harmonize feature learning across different clients, reducing variance in local representations and mitigating overfitting to majority classes. The paper includes extensive theoretical and empirical analyses, demonstrating the superiority of FedGLCL over state-of-the-art FL algorithms in various non-IID scenarios.", "questions": "Are the datasets used in the experiments part of the pretraining of VLMs? If this is the case, then your framework achieves better performance is due to the pretraining stage.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The utilization of large pretrained CLIP into FL is a unique and innovative direction. By leveraging contrastive learning to align local image features with global textual data, the paper proposes a novel solution to tackle the Non-IID problem.\n2. The paper provides a solid theoretical foundation for FedGLCL, explaining how the framework can mitigate issues related to Non-IID data. This theoretical backing strengthens the credibility of the proposed method.\n3.The authors conduct a comprehensive set of experiments, comparing FedGLCL with multiple state-of-the-art FL algorithms across different non-IID scenarios. The results demonstrate significant improvements, validating the effectiveness of the proposed framework.", "weaknesses": "1. Clarity and presentation issue: It is not clear which pretrained vision language models are used in the experiments. CLIP? Align? BLIP?\n\n2. It is better to compare the performance of different text encoders from various pretrained VLMs, because the proposed method heavily relies on the pretrained text encoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2x0nkVgHCO", "reviewer": "Reviewer_Q1wc", "summary": "This paper solves the heterogeneous federated learning (in the image classification field) via a pre-trained and fixed text encoder (BERT) on the server and learned image encoders on the clients. The classification is done by clip-style image-text embedding similarity comparison. The authors report superior performance over traditional heterogeneous federated learning methods (FedAvg, etc.), and provide scalability analysis, efficient analysis, and ablation study.", "questions": "1. Comparison/Detailed discussion about prototype FL\n2. Can it be adjusted for tasks besides image classification?\n3. Can traditional word embedding such as Glove also achieve a performance comparable to Bert's? (just curious about this)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method shows superior performance.\n2. Comprehensive analysis is provided.\n3. The usage of Bert embedding as a prototype vector for FL image classification is novel.", "weaknesses": "1. There are few comparisons with prototype FL (like [1] and [2]).\n2. The method proposed has some limitations. It is restricted to image classification and requires a pre-trained model that can give good embeddings of the classification labels. This seems to exclude many FL scenarios. (I am not to say that using Office-Caltech-10 and DomainNet is not enough. I just wonder if it may not be as versatile as FL methods like FedAvg. Yet I still appreciate this paper as a good method for image classfication.)\n\n\n[1] Federated Learning from Pre-Trained Models: A Contrastive Learning Approach https://proceedings.neurips.cc/paper_files/paper/2022/file/7aa320d2b4b8f6400b18f6f77b6c1535-Paper-Conference.pdf\n[2] FedProto: Federated Prototype Learning across Heterogeneous Clients https://arxiv.org/pdf/2105.00243", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)