Resource-Aware Federated Self-Supervised Learning with Global Class Representations
Authors: Mingyi Li
Xiao Zhang
Qi Wang
Tengfei LIU
Ruofan Wu
Weiqiang Wang
Fuzhen Zhuang
Hui Xiong
Dongxiao Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated self-supervised learning
Heterogeneity
Distillation
Primary area: other

Abstract: Due to the heterogeneous architectures and class skew, the global representation models training in resource-adaptive federated self-supervised learning face with tricky challenges: $\textit{deviated representation abilities}$ and $\textit{inconsistent representation spaces}$. 
In this work, we are the first to propose a multi-teacher knowledge distillation framework, namely $\textit{FedMKD}$, to learn global representations with whole class knowledge from heterogeneous clients even under extreme class skew. Firstly, the adaptive knowledge integration mechanism is designed to learn better representations from all heterogeneous models with deviated representation abilities. Then the weighted combination of the self-supervised loss and the distillation loss can support the global model to encode all classes from clients into a unified space. Besides, the global knowledge anchored alignment module can make the local representation spaces close to the global spaces, which further improves the representation abilities of local ones. Finally, extensive experiments conducted on two datasets demonstrate the effectiveness of $\textit{FedMKD}$ which outperforms state-of-the-art baselines 4.78\% under linear evaluation on average.

Resource-Aware Federated Self-Supervised Learning with Global Class Representations
Mingyi Li Shandong University
Xiao Zhang" Shandong University
Qi Wang Shandong University
Tengfei Liu Hong Kong University of Science and Technology
Ruofan Wu Coupang
Weigiang Wang
Shanghai Jiaotong University
Fuzhen Zhuang
Hui Xiong Thrust of AI, HKUST(GGangghouu) Dep. of Com. Sci. and Eng.. HKUST
Dongxiao Yu"
Institute of Artificial Intelligence, Beihang University Zhongguancun Laboratory
Shandong University
Abstract
Due to the heterogeneous architectures and class skew, the global representation models training in addurce-adaptive federated self-supervised learning face with tricky challenges: deviated representation abilities and inconsistent representation spaces. In this work, we are the first to propose a multi-teacher knowledge distil- lation framework, namely FedMKD, to learn global representations with whole class knowledge from heterogeneous clients even under extreme class skew. Firstly, the adaptive knowledge integration mechanism is designed to learn better repre- sentations from all heterogeneous models with deviated representation abilities. Then the weighted combination of the self-supervised loss and the distillation loss can support the global model to encode all classes from clients into a unified space. Besides, the global knowledge anchored alignment module make the local representation spaces close to the global spaces, which further improves the representation abilities of local ones. Finally, extensive experiments conducted on two datasets demonstrate the effectiveness of FedMKD which outperforms state-of-the-ane baselines 4. 4.78 under linear evaluation average.
Introduction
The federated self-supervised learning (Fed-SSL) has emerged highly promising paradigm due to the extremely limited labeled data in real-world scenarios [4, 25, 30j. The Fed-SSL mechanism learn common representations collaboratively across all the clients without labeled data [7, 8), which could enable the aggregation of knowledge from diverse unlabeled data sources and overcome the limitations caused by the high cost and scarcity of labeled data [24. 28. 31].
Traditional Fed-SSL methods usually assume that each client should train the identical architecture model, such as FedU [33], FedEMA [34], FedCA [30]. But would not be easy in resource-limited scenarios, especially for the existing arsing large-scale models [1, 26). As shown Fig. client A, client B and client C. might train heterogeneous representation models due to the varying system resources. In addition, real-world data often exhibits skewed class distributions across clients.
Xiao Zhang and Dongxiao Yu are corresponding authors. Email: Xxiaozhhn,, dxyu @ sdu.edu.cn
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Therefore, how to learn global class representations under the heterogeneous architectures and class skew in resouree-are aware Fed-SSL paradigm is challenging, particularly comparing with existing FedU2 [17], FedX [5] with identical architectures.
Although both Hetero-S3FI. [20] and FedFoA [19] consider the heterogeneous client models, they can not learn global representation model. In order to aggregate the knowledge from the clients to form global class representations, some tricky challenges arise. (1) Deviated representation abilities. Even for the same data samples, the different models might encode them into different latent spaces with deviated representation abilities. For example, client and client B all have images with dog, cat, tiger, but client model A can encode cat, tiger well into different clusters, client model B can only learn better representations of dog. So how could global representation models take advantage of the best of both client models? (2) Inconsistent representation spaces. The skewed class distributions across clients lead to inconsistent representation spaces. For example Fig. 1, comparing with client A, client C has different kinds of images with dog, cat, airplane. Thus how to make global representation models encode the whole classes from all the clients well a unified space? Therefore, different from the existing works, our goal is to break the gaps caused by the hybrid heterogeneity, which can learn the high-quality global representation model in federated self-supervised learning.
Along this line, we propose FedMKD, a multi- teacher knowledge distillation based resource- adaptive Fed-SSL framework, which can learn global representations over all classes from het- erogeneous clients. First, an adaptive knowl- edge integration module is introduced learn high-quality representations from all the het- erogeneous models with deviated representa- tion abilities. Then in order to encode all classes from clients in unified space, the global model uses the weighted combination of self- supervised loss and distillation loss to update. Besides, the global knowledge anchored align- ment module is applied within the server to elim- inate the inconsistency in representation spaces and reduce the burden on the clients. uses global knowledge additionally update the lo- cal models, which can not only make the lo- cal representation spaces close to the global space but also improve the representation ca- pability of both local models and the global ones. Code is available at https: /github.. com/ hememmmme-eeee sareeee-dduFFeeeee sdu/FedMK] The main contribu- tions of this paper can be summarized as follows.
Server
Global
representationn
space
Deviated representation abilities
Inconsistent
representation
dog
dog
dos,
Figer
Representation Space"
Representation Space
Representution Space'
Cat
Dog
Dog
Dog
Tiger
Tiger
Tiger
Plane
Plane
Plane
Client A
Client B
Client C
Figure 1: Illustrations of main challenges in
resource-aware Fed-SSL.
In resource-aware Fed-SSL, we are the first to delve into global class representation learning through revealing the deviated representation abilities and inconsistent representation spaces caused by the heterogeneous architectures and class skew.
We design a multi-teacher knowledge distillation framework, namely FedMKD, to adaptively aggregate positive knowledge from heterogeneous models with deviated representation abilities. Through combining the self-supervised loss and the distillation loss, FedMKD can encode skewed classes into unified space.
Extensive experiments conducted on CIFAR-IO and CIFAR-100 show the representation abilities over all classes the FedMKD perform better than state-of-the-art baselines. Our algorithm can improve 4.22% and 5.31% % separately on the two chosen datasets.
2
Related work
The federated self-supervised learning aims to learn high-quality representations from clients without large labeled datasets [11]. From the beginning, several works [12, 23) simply combine federated learning with self-supervised methods. Besides, FedU [33] designs communication-effciinnt mech-

---

Table 1: Comparison of federated self-supervised learning methods.
Global model
Global model size
Model Heterogeneity
Deviated representation ability
Inconsistent representation space
Theoretical Analysis
Method
FedU [33]
Client
FedEMA [34]
Client
L-DAWA [21]
Client
FedX [5]
Client
FedCA [30]
Client
FLPD [29]
Client
FedU [17]
Client
X
FedFoA [19]
Hetero-SSFL. [20]
FourS
Client
anism by only aggregating the online encoders under non-lID data. FedUTN [16] is proposed to use the aggregated online networks for the target network updating in the self-supervised framework. L-DAWA [21] proposes the layer-wise divergence aware weight aggregation to mitigate the influence of client bias. FedEMA [34] considers the divergence-aware moving average updating in clients, measuring the divergence between local models and global model. FedX [5] proposes a unsupervised federated learning framework to learn representations through a two-sided distillation method. How- ever, all the above works intuitively gain the global model through parameters average due to the identical client models, which not be applied in the heterogeneous clients setting directly. In addition, although FedCA [30] address the misaligned and inconsistent representation challenges by gathering features from clients, inducing potential privacy problems. FLPD [27, 29] introduces distillation method based similarity between prototypes from a labeled public dataset to update the local model. F'dUU2 [17] focuses on mitigating representation collapse entanglement and obtaining unified representation spaces.
Considering heterogeneous client models in federated self superviee learning, Hetero-SSFL [6] introduces linear-CKi to align lower-dimensional representations between the local model and global model without architectural constraints. FedFoA [19] designs factorization-basee method to extract the cross-feature relation matrix from the local representations for aggregation. However, both Hetero-SSHi and FedFoA can not learn a global representation model considering the hybrid heterogeneity, which is the main focus of our work. The comparison details are shown in Table 1.
3 Preliminaries
The goal of federated unsupervised learning is learn the generalized representation for some downstream tasks from several distributed unlabeled data sources. A federated learning setting consists of central server and N clients. Each client & contains local unlabeled dataset Dj, and the server contains a public unlabeled dataset Dpub. The local objective at k-th client is
min F(Ox) BEL.- [Lx (Oc, (Ok,E.)) Ex))
to minimize the expected local loss of client &: on local dataset Dj, and & is the unlabeled data. In traditional FL settings, it's assumed that }0x} are identical and gain the global model using 0 = Σ === PsO2, where P& is the weight of k-th client. But in real-world cross-device scenarios, each client might have a unique model and the architecture of the model might be different, which means that traditional aggregation methods are not available. We assume that 0% is not similar to others, and use } collaborate in training the larger global model & in server. Here define the global update function is 0: G(0,-1; 01.0.........................0 ON). Our final aim is to optimize the global goal
min Fotabal (0) BggDyttoaa [C, ()),
where C(),E) is global loss function in server and is the unlabeled data sampled from global dataset.
4 Designed FedMKD Method
We propose multi-teacher knowledge distillation based federated self-supervised learning framework FedMKD, which is shown in Fig. 2. In FedMKD, besides the local self-supervised learning (Sec. 4.1),
3

---

Initialization
Local training
Multi-teacher knowledge distiilaaion
Online encoder 0
Online
encoder
Encoder
Target
Oniine enzoderdy
Attentton
61
Pres
Online
Online
On
encoderdM
Commostcc
Encoder
0g
Global model
D2
Target encoder 43
6
pr3
Global anchored alignment
Online encoder 0g
Online encoder
Unline enncodenly
Encoder
Algn
Align
Target (hy
Global
Global madel
modelf,
Clients
Server
Figure 2: The overall framework of FedMKD. Clients initialize the model architecture based on the local resource, then self-super train the local model using unlabeled local data. The server uses the multi-teache adaptive knowledge integration distillation to aggregate positive local knowledge to train the global model and then updates local models again according to the alignment module.
we design a multi-teacher adaptive knowledge integration distillation module to adaptive determine the weight of the representations from heterogencous local models with deviated representation abilities. The distilled loss combined with the global self-supervised loss, we can gain the weighted combined loss to update the global model, that the global model can encode all classes from clients in a unified space (Sec. 4.2). And the global knowledge anchored alignment could improve the representation capability of clients and further benefit the global model training (Sec. 4.3). In addition, we provide the theoretical analysis our algorithm in Appendix B.
4.1 Self-supervised model training
Each client performs self-supervised contrastive learning using asymmetric Siamese network, inspired by BYO [3]. The model M comprises an online encoder 9 and a target encoder Φ, both sharing the same architecture, with the online network incorporating an additional predictor p. That M = (p(o(-)), 0()) Given an unlabeled image x, we can obtain two augmented views, and v', serving as inputs to online and target networks, respectively. The loss function is defined as follows:
p(r) [p(r)]
Ceelf
where r 0(v) and <'v'))"))). This loss encourages the online network to produce representation p(r) that is similar to the positive sample generated by the target network r'. We then exchange the views, feeding v' to the online network and the target network, to compute L. At each
stochastic gradient descent minimize Leefs Leelf L' to update the online network alone,
training step,
0 - 10-0-1V o
(4)
The target network helps to provide regression targets train the online network. Choosing a E [o, 1] as the target decay rate, we employ the exponential moving average (EMA) of the online network to update d:
+ ad + (1 a)00.
Using this self-supervee supervupeee training method, the model learns intricate representations from unlabeled data, capturing high-level features and patterns inherent in the dataset.
4.2 Multi-teacher adaptive knowledge integration distillation.
In contrast to homogeneous federated learning, the presence of model heterogeneity poses challenge: direct aggregation of local models into a global model not feasible. overcome this, we design a multi-teacher knowledge distillation mechanism transfer local knowledge to the server.

---

Given batch of data B, the representation from the teacher model is denoted as and that from the student model as rs. the knowledge distillation loss defined as follows:
exp(sim TT,)/T)
cxp(sim((r...Ts.)) Laistill = -log (enossiiissss,aa)))) (E(B-1)
(6)
where is the temperature parameter controlling entropy and sim(.) is the similarity function between two representations.
Then we extend this knowledge distillation learning method to multi-teacher, Although the data is heterogeneous, the knowledge of each local model is valuable, each local model captures the unique characteristics of local data. Our goal is to integrate the positive knowledge of all clients to guide the global model in learning general representation unlabeled data. We design multi-teacher adaptive knowledge integration distillation that can adaptively weigh the representations from clients.
Given sample 24, representation from the n-th local model is R,, I where d is the dimension of the representation. Following [3], a fully connected layer is employed to project the representation into oower-dieensional space, enhancing the discriminate power of the learned representations. So, we map the representation from the global model Rs, into the same lower latent space, obtaining
,, g(R.,a), Fn, g(Rn..),
where an, Pa, RR, k is the dimension of the latent space and g()) is the projector.
In addition, we introduce an adapter module to learn instance-level teacher importance weights for knowledge integration. After getting Tn,i, an attention block is used to generate the weighted sum of them. In this context, representation from global model ra, is treated as the query, while those from local models = [Mi,,, .2.i..... TN,]]" is treated as the key and value. Treating representations from the global model as query ensures consistency in knowledge transfer. The attention mechanism computes attention scores to understand the relevance of each local model's representation to the global model's query. The aggregated representation is:
F; Attm(rs,as Rt) softmast Tas R RR,
where F; means the aggregated representation, and Attn(-) denotes the attention block.
Returning the knowledge distillation for unlabeled data proposed earlier, we treat the aggregate representation the positive sample, and the remaining samples in the batch as the negative sample. The adaptive weight multi-teacher knowledge distillation loss function is expressed as follows:
crp(sim(rss. F:)/T)
Ldistiil -log exp(sim((sss
(9)
F:)/T)+
exp(sim((rss Tss)/T)
Above all, the weighted combined loss for the global model is presented as:
Lserver
I.self + Y Ledistall
(10)
where y is hyper--prannnn parameter controlling the weight the distillation process.
4.3 Global knowledge anchored alignment
As we mentioned, the representations from different models are inconsistent and the representation abilities of models are also deviated. So we introduce the global knowledge anchored alignment mechanism that each local model uses the global model as an anchor. It ensures that the local representation spaces are closer to the global ones.
Unlike methods such as FedX [5] and MOON [15], which align local models to the global model locally, approach aims to train better global encoder tailored for essurce--cossraaiee federated learning scenarios. Those methods are not available when clients cannot afford to store or infer the global model locally. So transfer this alignment process the server.
After finishing the global model training, we construet local twin models in the server to realize the alignment under global view. Here, use the global online network and local online network to
5

---

(a) Standalone training (b) MOON on Partial pub-(c) FedMKD on public(d) FedMKD on Partial
ResNet18 on Partial public lic datset. dataset.
dataset
public dataset
Figure 3: T-SNE visualizations of hidden vectors from different models on CIFAR-IO, the data distribution of clients is IID.
construct new asymmetric siamese network called the twin of the original local model. The local online network 0, is the online model, and the global online network 0g is the target model, that is,
(0%, H. + {0n, 0x)
(11)
and M- {f%, 6. The training loss is updated to
erp(sim (1), EieB eap(ssm (1,(1, & (1))TT)
Latign -log
(12)
According the idea of contrastive learning, the representations learned by the online network become more consistent with the knowledge captured by the target network. This global knowledge anchored contrastive learning suggests that the global model's knowledge is used as positive example for the local model to train itself, thus making it more consistent with the target global network. Aligning local models with the global view representation helps create comprehensive understanding of the overall data distribution. The process refines the knowledge acquired locally, ensuring that contributes meaningfully the overall federated learning process. Then, we use stochastic gradient descent to minimize Lalign to update the 6',
6%, 5 6% 1/ Latign.
(13)
Once the global model anchored alignment is finished, the server will send the online network of the local twin network 0' to the corresponding client to update the local model. The local online network 8n replaced by the server-updated network 6' The target network not replaced retain more local knowledge and stabilize model training:
(0gmz 2g,n +- 11 1g-1m)
(14)
so that, the local model can benefit from the alignment process and align to the representation under global view, which can further the local data to train the model.
5 Experiments
In this section, we evaluate the representations learned from our proposed global model FedMKD on CIFAR-IO and CIFAR-100. We first describe the experimental setup and baselines, and then analyze the performance in comparison to other methods. Due to the space limitation, further hyperparameter analysis and communication cost analysis are represented in Appendix D.
5.1 Experimental setup
We use CIFAR-1O and CIFAR-100 [13] datasets to train all the models. Both of them contain 50,000 training images and 10,000 testing images. To construct the public dataset, we sample 4000 data samples from the training set, then divide the remaining data into N partitions to simulate N clients.

---

Table 2: Top-1 I accuracy comparison under linear probing on CIFAR datasets with best model performance bold and second-best results with underlines. means this method not suitable for the experiment setting.
CIFAR-I0
CIFAR-TOU (%)
Method
Pub.
Class
Dir($=0.55
ITD
Class
Dir(3=0.5)
ITD
Std. ResNet18
51.09
25.35 0.02
FedMD
45.28 0.02
45,93 1: 0.02
46.2111000
23.25 0.05
22.46 1: 0.04
23.20
0.03
FedDF
46.94 0.04
48.04
0.08
23.07 0.03
22.73 0.03
21.57
0.01
MOON-KL
44.93 = 0.0G
45.81
46.51 0..51:0000
21.34 2.02
21.82
2 0.02
MOON
IID
53.35 0.03
53.71
0.02
27.0
26.84 0.03
26.70 0.03
FedET
56.42 = 0.02
59.38 1: 0.03
1..43.10002
29.1
26.98 -:
24.48
0.02
FedU/FcdEMA
Hetero-SSFL.
59.13 0.02
64.04 + 0.04
65.61 + 0.07
30.84
29.63 - 0.06
28.89 - 0.06
FedMKD
64.81 0.02
66.98 + 0.06
69.07 + 0.04
36,33 + 0.01
35.59 + 0.07
35.94 + 0.02
Std. ResNet18
30.15
24.97 0.01
FedMD
47-16 0.03
46.39 1: 0.02
45.93 0.03
23.95 0.05
23.14 0.03
22.47 1 0.01
FedDF
52.50 0.05
53.50 0.03
54.17 0.05
27.31 1 0.04
27
0.04
MOON-KL
46.41 0.03
46.81 0.03
45.89 0.01
0.00
20.97
- 0.03
22.27
0.04
MOON
Par.
54.31 = 0.04
54.54 0.02
62.94
27.00
27.27
= 0.01
28. 26.26 0.02
FedET
57.75 - 0.01
57.08 1 0.01
1.001
29.38 0.001
28.12 0.02
29.61 1 0.01
FedU/FedEMA
Hetero-SSFL
63.20 0.08
61.93 + 0.04
61.15 0.07
30,94 0.05
29.92 - 0.03
29.56 0.03
FedMKD
66.39 0.09
67.60 + 0.04
65.88 0.03
35.82 0.02
35.55 + 0.05
34.38 0.02
To assess the validity of the public dataset, we use two sampling methods to construct it. First, we random sampling method over all classes generate public dataset 'IID'. And for the public dataset 'Partial', data is selected randomly from 40% classes two datasets.
We utilize three settings to simulate heterogeneous data distributions among all the clients. For the IID setting, each client contains the same number of samples from all classes. For the class setting, each client only has 10/N and 100/N classes two datasets and the classes between clients have no overlap. For the non-IID setting, data heterogeneity levels are described by the Dirichlet distribution Dir(š) [10], where smaller 33 represents stronger heterogeneity levels, here choose 3 = 0.5.
Regarding the self-supervised learning framework design within each client, we use ResNet18 [9] and VGG9 [22] as the encoder network and Multi-Laye Perception (MLP) as the predictor. In order construct the model heterogeneous setting, clients train the Resnet18 encoder while clients use the VGG9. And for the global representation model, Resnet18 is selected as the encoder in server.
5.2 Baselines and evaluation methods
Firstly, we select several federated knowledge distillation frameworks FedMD [14], FedDF [18], FedET [2], MOON [15], MOON- KL that use unlabeled public dataset for distillation. We then replaced the local model with a self-supervised model to evaluate the process of knowledge distillation in our method. And several federated self-supervised learning frameworks FedU [33], FedEMA [34], Hetero- SSFL [20] are also chosen baselines. To verify the client's knowledge can improve the global model, we also train the global model separately on the public dataset, denoted as Std. ResNer18. Following FedEMA [30], we evaluate the performance of learned representations using linear and semi-supervised evaluation. Due to limited space, please refer to Appendix C for more details.
5.3 Performance Evaluation
Table 2 and 3 shows the linear evaluation results and semi- cumi-supervieed evaluation results of FedMKD compared with all the baselines on CIFAR-1O and CIFAR-100. We can gain the following observation.
On the whole, different data heterogeneity level settings on both two datasets. Compared to the second- best results, FedMKD achieves significant improvement. On average, model improves CIFAR-JO by 4.22% and CIFAR- 100 by 5.31% % under linear evaluation and gain 3.66% and 2.07% improvement on two dataset under semi-supervised evaluation.
FedMKD outperforms all baselines under different public dataset settings and
The effectiveness of our multi-teacher adaptive knowledge integration distillation can be approved when compared with FedMD, FedDF, MOON-KL KL and MOON. Although these methods all designed
7

---

Table 3: Top-1 accuracy comparison on I 1 of labeled data for semi-supervised learning on CIFAR datasets with best model performance bold and second-best results with underlines. '-' means this method doesn't It apply for the experiment setting.
CIFAR-10 (%)
CIFAR- 100 (%)
Method
Pub.
Class
Dir(3=0.5)
III
Class
Dir
III
Std. ResNet18
46.81 + 0.25
15.19 = 0.20
FedMD
43.32 = 0.22
44,20 + 0.18
44.66+0.20
15.88 + 0.17
14.94 + 0.19
15.34 + 0.12
FedDF
43.60
0.44
44.13 + 0.16
0.40
14.39
+ 0.20
13.06 + 0.14
12.90
+ 0.07
MOON-KL
45.42 0.26
46.61
0.21
46.72 0.15
16.25
0.06
17.22
0.25
16.07
1 0.04
MOON
IID
49.96 0.24
50.21
0.10
51.78 0.28
0.12
17.21
0.13
17.07
0.18
FedET
0.24 52.37
56.57 + 0.17
37.44
19.70
16.82 + 0.20
15.68 + 0.18
FedU/FedEMA
Hetero-SSFL
54.30 = 0.15
58.73 1: 0.54
60.50
1.500100 0.12
0.404
19.19 0.17
18.82 1 0.17
FedMKD
59.65 0.28
61.78 +: 0.40
64.06 :: 0.32
22.57 0.12
2
22.07 - 0.15
Std. ResNet18
46.42 0.12
14.12
FedMD
44.54 0.26
13.61
0.19
0..7
0.14
16.31 + 0.07
FedDF
48.14 0.27
48.74+0.11
0.18
17.01 +.04
17.14
16.95 + 0.01
MOON-KL
46.76 = 0.05
46.92 +: 0.05
46.49 0:22
16.21 0.27
16.10 + 0.12
16.94 + 0.12
MOON
Par.
50.43 = 0.18
51.99 0.34
0919
18.64
18.92 = 0.25
19.29 0.08
FodET
52.75 = 0.07
52.61 + 0.03
0.14
18.49
18.16 + 0.13
18.01 0.22
FedU/FedRMA
1letero-SSFL
59.95 0.34
58.31 1 0.50
58.35 1 0.20
20.72 0.114
20.30 1 0.32
19.62 1 0.08
FedMKD
61.55 0.19
63.10 + 0,28
61.08 + 0.19
22.44 0.19
22,21 + 0.17
20.94 + 0.25
new federated knowledge distillation frameworks based unlabeled public dataset, since the original local model is supervised, they prefer using the class information from logits to distill. When observing the result of FedET, we find that although it also designs larger global model in server which improves the model a lot, the final result is not satisfied. This is also because it designs a distillation method based on knowledge of probability distribution over classes. Next, compared with the federated self-supervised method, our FedMKD also achieves better performance. FedU and FedEMA are not applicable in model heterogeneity setting, so we cannot evaluate their effectiveness. HSte gain the best performance among all baselines but is worse than ours. That's because it aims to train personalised client models. Only the alignment module cannot hold the inconsistent representation space perfectly. But our global model can directly generate representation from the global model, it avoids using the representation from inconsistent clients.
Apart from the client data heterogeneity, we consider the influence of the public dataset distribution. Here, we construct two public datasets, one is 'IID' to the whole data distribution and the other only has partial classes. In both two settings, our FedMKD also gets the best performance. The overall performance of 'IID' public dataset is better than 'Partial' setting. That's because the global model adapts the self-supervised learning on public dataset, and the diversity of the sample is important, which can help the model explore wide range of features and patterns present in the data. And it's observed that when the public dataset is 'IID', the performance increases with the decrease of the data heterogeneity level for CIFAR-1O. but it doesn't 't apply to the CIFAR-100. It's because there are too many classes in CIFAR-100 and the number of samples in each class is not efficient.
In order evaluate the effectiveness of FedMKD, to visualize the representation on the test dataset of CIFAR-1O from different encoders. As shown in Fig. (b))))d), the global models in FedMKD trained on both 'IID' and 'Partial' public datasets both achieve better clustering results than Standalone training and MOON. These results further verify our model can gain better generalized representations although the representation spaces of clients are inconsistent. There's also an averaged global model in MOON, but it cannot tackle the problem of inconsistent spaces well using the average method, so only gains a poor clustering performance. Additionally, the performance on 'IID' public dataset is better than 'paritial' ones from the observation of cluster performance. This suggests that the number of classes seen by the global model also affects how well the global model can encode all classes in unified space. Comparing the class distributions in Fig. 3 (c) and (d), we can find that although these two global models are trained on different public datasets, the final cluster layout is similar, which can further prove that our global model can encode all classes from clients even if it never sees some classes during the training.
use the dimensionality reduction algorithm t-sne
8

---

Table 4: Experimental results on ablation studies of FedMKD with best model performance in bold.
CIFAR-10 (%)
CIFAR-100 (%)
Method
Class
ITD
Class
IDD
Standalonc training
51.09 + 0.04
25.35 + 0.02
FedMKD KL
43.88
0.19
46.24 0.08
19.56 + 0.18
15.45 0.05
FedMKD w// adoptise
46.14
47.29 = 0.08
21.77
= 0.02
22.76
FedMKD, /u/ aligmment
34.97
29.91 0.02
FedMKD
64.81 + 0.02
69.07 0.04
36.33
0.01
35.94 0.02
5.4 Improvement of clients
In FedMKD, the global knowledge anchored alignment module is used to align the client model in the server which can further transfer the global knowledge to the client and incen- tivize the clients to participate in the federated learning. To evaluate the improvement of the clients, the local models which are standalone training locally are compared with our local models. As shown Fig. 4, the client perfor- mance in our FedMKD framework is better than local standalone training, regardless of the archi- tecture of the local model. Especially for clients with ResNet18, it improves 6.73% average. It's concluded that clients benefit from federated training by contributing to global training.
82
Wandslrne
Testing Acc
Clerc
Clent
Uhart3
Clort4
Clenk
Figure 4: Improvement of clients after involving our proposed FedMKD.
5.5 Validation of inconsistent representation spaces
As we mentioned, the representation spaces be- tween different clients are inconsistent because of data heterogeneity. To validate this opinion, use Linear Diseriminant Analysis (LDA) to reduce dimensionality in order to visualize the distribution of the representations. Here the data distribution of clients is Class. In Fig. 5 left, 'o' and xx denote representations of Client A and Client B, respectively. And different colors denote different classes. We can observe that in Fig. 5 left the classes "cat' and 'dog' almost overlap while they are from different clients, which verifies that the inconsistent representa- tion spaces did exactly exist. And Fig. right shows the visualization result of the represen- tation from the global model. It's clear that the classes 'cat' and 'dog' are embedded in different positions in global space, which proves that although the local representations are inconsistent, our global model can learn good representation.
Chr
Figure 5: LDA visualizations of hidden vectors from different models on CIFAR-10.
5.6 Ablation experiment
In order to investigate the effectiveness different parts of FedMKI), we design these comparison experiments:
Standalone training: The global model is trained alone using the public dataset, without the knowledge from client models.
FedMKD kT. The global distillation function is replaced by the KL-divergence function to measure the similarity of the aggregated representation and global representation тs-
FedMKD w : The adaptive knowledge integration module removed, each local representation has the same weight to generate the aggregated representation.

---

Table 5: Experimental results on scalability studies of FedMKD.
Client number
Method
5
10
30
Linear result
67.79
0.04
57.39 + 0.60
53.85 + 0.03
Semi result
63.37 0.51
53.30 + 0.31
48.97 + 0.86
FedMKD w//o aligmment: FedMKD.
The global knowledge anchored alignment module is removed from
Above experiment is conducted the both CIFAR-I0 and CIFAR-I00 dataset, using the IID public dataset. The results of the ablation study experiment are shown in Table 4. When comparing FedMKD with FedMKDKL. significant performance drop can be observed when the knowledge distillation function is replaced by the KL-divergence function. Thus we can conclude that for self-supervised learning, we prefer performing knowledge distillation based on representation, but the KL-divergence cannot capture the distribution characteristics from them. Therefore, the appropriate distillation method critically important in self-supervised learning due to the lack of labels. And a worse performance on both two datasets can be observed when we use the equal weight instead of the adaptive weight. Because the client models are heterogeneous, their representation capabilities are different and the representation spaces are also inconsistent, so intuitively average representations may reduce the information contained in the representation. Finally, when we remove the alignment module, the performance under each setting all decreases, which demonstrates that the alignment is not only beneficial to the local models, but also improves the whole training process.
5.7 Scalability of algorithm
In order to explore the scalability of our proposed algorithm FedMKD, we add the experiment that the number of clients is 5, 10, 30 on CIFAR- and 40% clients use the VGG model and 60% use ResNet18. And we repartition the data for each client under Dir(3 = 0.5) and set the public dataset distribution as 'IID'. The results are shown in Table increasing, the performance decreases. The reason is that the total number of data is fixed, if the number clients increases, the number of data in each client will decrease, which further affect the performance of the local model.
We can find that as the number of clients
6 Conclusion
In this work, we focused on how solve the deviated representation abilities and inconsistent representation spaces caused by the heterogeneous architectures and class skew in federated self- supervised learning. We proposed framework FedMKD to lear global model. Firstly, the adaptive knowledge integration module could learn high-quality representation knowledge from heterogeneous models. And the combination of the self- superviedd loss and the distillation loss enabled the global model to encode all classes from clients in a unified space. Then a global knowledge anchored alignment module improved the local representation models in server and fed it back to corresponding clients. The experiments conducted on two datasets demonstrated that our proposed FedMKD was state-of-the-ar and outperformed existing methods.
knowledge based federated supfvvueer learning
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grant 62202273, 62176014, 92370204, in part by National Science Fund for Excellent Young Scholars of China under Grant 62122042, in part by Shandong Provincial Natural Science Foundation of China under Grant ZR202 in part by the Fundamental Research Funds for the Central Universities, part by the Major Basic Research Program of Shandong Provincial Natural Science Foundation under Grant ZR2022ZD02, in part by the Joint Key Funds of National Natural Science Foundation of China under Grant U23A20302, in part by Cuangzhou-HKKUSTGGZ Joint Funding Program under Grant 2023 in part by the Education Bureau of Guangzhou Municipality.
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[2] Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heterogcneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv: /2703, 2022.
[3] Jean- non-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallee, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self- supervised learning. In Hugo Larochelle, Marc Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neurul Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[4] Wei Guo, Fuzhen Zhuang. Xiao Zhang. Yiqi Tong, and Jin Dong. A comprehensive survey of federated transfer learning: challenges, methods and applications. Frontiers Compus. Sci., 18(6):18636, 2024.
[5] Sungwon Han, Sungwon Park, Fangzhao Wu, Sundong Kim, Chuhan Wu, Xing Xie, and Meeyoung Cha. Fedx: Unsupervised federated learning with cross knowledge distillation. In European Conference on Computer Vision, pages 691-707. Springer, 2022.
[6] Chaoyang He, Zhengyu Yang. Erum Mushtag, Sunwoo Lee, Mahdi Soltanolkotabi, and Salman Avestimehr. Ssfl: Tackling label deficiency in federated learning via personalized self- self-supervision. arXiv preprint arXiv:2110.02470. 2021.
[7] Jingxuan He, Lechao Cheng, Chaowei Fang, Zunlei Feng, Tingting Mu, and Mingli Song. Progressive feature self-rein reinforcement weakly supervised semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2085-2093, 2024.
[8] Jingxuan Hc, Lechao Cheng, Chaowei Fang, Dingwen Zhang, Zhangye Wang, and Wei Chen. Mitigating undisciplined over-smoothing in transformer for weakly supervised semantic segmentation. arXiv preprint arXiv:2305.00 02110.11212111 2023.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2016.
[10] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribu- tion for federated visual classification. CoRR, abs/1909.063333 2019.
[11] Yilun Jin, Yang Liu, Kai Chen, and Qiang Yang. Federated learning without full labels: A survey. arXiv preprint 230: 14453, 2023.
[12] Yilun Jin, Xiguang Wei, Yang Liu, and Qiang Yang. Towards utilizing unlabeled data federated learning: A survey and prospective. arXiv preprint vrXiv:2002.11544 2020.
[13] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[14] Daliang and Junpu Wang. Fedmd: Heterogenous foderated lcarning via model distillation. arXiv preprint Ar::::/900.00000 190000381,, 2019.
[15] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10713--0722, 2021.
[16] Simou Li, Yuxing Mao, Jian Li. Yihang Xu, Jinsen Li, Xueshuo Chen, Siyang Liu, and Xianping Zhao. Fedutn: federated self-supervised learning with updating target network. Applied Intelligence, 13(9):10879 10892, 2023.
[17] Xinting Liao, Weiming Liu, Chaochao Chen, Pengyang Zhou, Fengyuan Yu, Huabin Zhu, Binhui Yao, Tao Wang, Xiaolin Zheng, and Yanchao Tan. Rethinking the representation in federated unsupervised learning with non-iid data. ArXiv, abs/2403.16333 2024.
[18] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 13:2351-2363 2020.
[19] Yi Liu, Song Guo, Jie Zhang, Qihua Zhou, Yingchun Wang, and Xiaohan Zhao. Feature correlation-guia knowledge transfer for federated self supervisd learning. arXiv preprint arx1v:221100766, 2022.
11

---

[20] Disha Makhija, Nhat Ho, and Joydeep Ghosh. Federated self-supervised learning for heterogeneous clients. arXiv preprint arXiv:2205. Stti::205..2493,, 2022.
[21] Yasar Abbas Ur Rehman, Yan Gao, Pedro Porto Buarque de Gusmao, Mina Alibeigi, Jiajun Shen, and Nicholas D Lane. L-dawa: Layer-wiss divergence aware weight aggregation in federated self-supervised visual representation learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18464-16473, 2023.
[22] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1e0:::4095550 2014.
[23] Bram van Berlo, Aaqib Saeed, and Tanir Ozcelebi. Towards federated unsupervised representation learning. In Proceedings of the third ACM international workshop on edge systems, analytics and networking, pages
31 31-36, 2020.
[24] Kun Wang. Hao Liu, Lirong Jie, Zixu Li. Yupeng Hu, and Liqiang Nie. Explicit granularity and implicit scale correspondence learning for point- superviue video moment localization. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, page 9214-9223, New York, NY, USA, 2024. Association for Computing Machinery.
[25] Lirui Wang, Kaiging Zhang, Yunzhu Li, Yonglong Tian, and Russ Tedrake. Does learning from decentral- ized non-iid unlabeled data benefit from self supervision? In The Eleventh International Conference on Learning Representations, 2022.
[26] Yangyang Wang, Xiao Zhang, Mingyi Li, Tian Lan, Huashan Chen, Hui Xiong, Xiuzhen Cheng, and Dongxiao Yu. Thcoretical convergence guaranteed resource-adaptio federated learning with mixed beterogeneity. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 2454-2455, New York, NY, USA, 2023. Association for Computing Machinery.
[27] Yuzhu Wang, Lechao Cheng, Manni Duan, Yongheng Wang, Zunlei Feng, and Shu Kong. Improving knowledge distillation via regularizing feature norm and direction. arXiv preprint ar3iv: /7007, 2023.
[28] Ziming Ye, Xiao Zhang, Xu Chen, Hui Xiong, and Dongxiao Yu. Adaptive clustering based personalized federated learning framework for next poi recommendation with location noise. IEEE Transactions on Knowledge and Data Engineering, 36(5): 1833)1854618666 2024.
[29] Chen Zhang, Yu Xie, Tingbin Chen, Wenjic Mao, and Bin Yu. Prototype similarity distillation for communication-elliienn federated unsupervised representation learning. IEEE Transactions Knowledge and Data Engineering, pages 1-13, 2024.
[30] Fengda Zhang, Kun Kuang, Long Chen, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Fei Wu, Yueting Zhuang, et al. Federated unsupervised representation learning. Frontiers of Information Technology & Electronic Engineering, 14(8):1181-11) 2023.
[31] Xiao Zhang, Ziming Ye, Jianfeng Lu, Fuzhen Zhuang, Yanwei Zheng, and Dongxiao Yu. Fine-grained preference-aware personalized federated poi recommendation with data sparsity. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 23, page 413-422 New York, NY, USA, 2023. Association for Computing Machinery.
[32] Weiming Zhuang, Xin Gan, Yonggang Wen, and Shuai Zhang, Easyfl: A low-code federated learning platform for dummies. IEEE Internet of Things Journal, 2022.
[33] Weiming Zhuang. Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12-1921 2021.
[34] Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence federated self-supervised learning. In International Conference on Learning Representations, 2022.
12

---

Appendix
We provide more details about work and experiments in the appendices:
Appendix A: the additional details the proposed algorithm FedMKD.
Appendix B: the detailed proof of the convergence analysis of our proposed algorithm FedMKD.
Appendix C: the details of experimental settings including datasets and federated simulations, reproduction details, baselines, evaluation methods and ablation experiment setting.
Appendix D: additional experimental results including Appendix D.I, the overall perfor- mance under two evaluation methods; Appendix D.2, the communication and storage efficiency analysis; Appendix D.3 discusses the impact of hyperparameters.
Appendix E: the details of limitations and broader impacts of this work.
A
Algorithm description
We outline the FedMKD algorithm in Algorithm 1. In round q, the clients and server perform the following updates:
Starting from the resource-adaptive model 0n update the local parameters for / [T]
Qqmm q, V 9,
Qq,n,t 0. + (1 a0)0 After T times local updates, upload the local parameter 9q,n,r.
Server uses all local model to process multi-teacer adaptive knowledge integration distilla- tion, update the global model for 1 E [T"],
0g.a + 04.s - VVV...
(15)
0g,s + a + (1 a))
(16)
Server aligns all client models to the global model, for n E N,
8g - Sn - 17 V 9.2
(17)
B
Convergence Analysis
In this section, we show the convergence analysis of our FedMKD. Firstly we give some commonly used assumptions in federated learning:
Assumption (Lipschitz Condition). Every function F(.) is with L-Lipschitz gradient: Vn e [N]. 0, E Rd
[VF(0) - -FF()))F(()] L110 - pll
(18)
Assumption (Bounded variance). The stochastic gradients F (0q, En,t) is an unbiased estimator of the gradient, with the variance bounded by > 0:
Be V 17FF.0y,aia (0q, En,s) (. o²,
Vg,n,t
(19)
In order to analyze the convergence rate of our proposed FedMKD, we firstly state some preliminary lemmas follows:
Lemma (Jensen's 's inequality). For any convex function h and any variable Ti. En we have
h
Maxz).
(20)
Especially, when h(x)) = ||=:||, we can gel
n
Σ
3
2
<
| 1
(21)
n1
7
i=1
13

---

Algorithm 1: Algorithm of FedMKD
Initialize: total number of clients N; Number of rounds T; Local data in clients Dı,..., DN ):
Public data in server D Local online network (f,...., ON' target network {01,..., UN J: Server online network 0g. target network os; Learning rate ry; Adapter network Attn(.). for
g 1 to Q do
Socallpdatee for ni 1 to N (all clients in parallel) do
q = then
Random initialize 4q
else
(0 (...o,0g.n.) <- 104-1,n 0g 1,
end
for
epoch = t0 T do
1 Dn)- (g.m = L 1p
10
ag,,t + 0p..1-1 - n,
11
Py,n. < g, (1 11) 9.00. 9.2.
12
end
13
Upload gg end
15
Server
# Multi-teacher adaptive knowledge integration distillation.
17
for epoch t = 1 to I" do for batch b pub do for n 1 to N do
18
19
20
rg ggnn g (0 (b))
21
end
22
R 1g.1, Ng
23
(()).
24
F Attn(rs:: R.
25
(g. = Csels (0g.s: 2i 2g.si b) + 7 (ddietalOg..s
0 q. 0g. 11 Vlg.
27
24.5 + a0g + 0x) 0x)0
28
end end
29
30
# Alignment client models in server
for n N do
32
(0' ( Hann <- T Og,s q,n Lalign (0 un ng D Dub))
33
34
0' 9.1 + 0r a. VI gm
35
end
36
Send 0' q.n
to client n.
37
end
Lemma For random variable $1, Fn we have
Bl[l21 + ....... + Fn [| < n 2 .... ...... 11].
(22)
Lemma For independent random variables #1 ......... Tn whose mean is O, we have
Elllz + .......... [| [|[[= .... 11..
(23)
Based on the above assumptions, we present the theoretical results for the non-convex problem.
Lemma 4 (Deviation bound of the optimization function) In each communication round, the function value in server reduce after T" epochs and is bounded as:
Ln
T
LT'
E[F(O,,T (B
(24)
2
14

---

Lemma 4 indicates the deviation bound of the optimization function of the server in each global round.
Theorem (Non-convex divergence for FedMKD) Let Assumplion I to 3 hold and A - F(Oo) F(Or), given any suppose that the learning rates satisfy S r S 2/1, after
2A
Q T'8(2m Loy) - L
(25)
communication round, we have
Q-IT'-1
Σ
Σ T=
(26)
QT
the convergence can be guaranteed.
Proof. Let's start the proof from L-Lipschitz condition:
(a) F(0g.41) S F(0g,c) + (VF(00.)), oy.e) 2 110g,141 0g.e1l²
= =((0,,)) -1 kn,)) + 2 (vV
where (a) from Assumption Taking expectation of both sides, we obtaín
B[F(0,,44]] s [[))) - Ln²² (O,,, En,1)
Ln² (O,, +
B[F(0,,)] nE[I Tm²
B[r(0,,)) - (1) [ + Tay² o²,
where (b) follows from Assumption 2. Let's set the learning step the start of training to T",
Ln² (F((0.)))
T
LT
E[F(O,,T)] < B[F(O,,0)] (1)
(27)
2
2
Proof of Theorem 1. According to Lemma 4,
L
Q-17'-1
Q-1
1 Q-1
+ LT'
(n 2
B( <
Σ
B[F
Σ
B[F(O..0)]
QT
QT
QT'
2
q=u Lmo
4-0
q=a
S 6(1 nLL
Therefore,
22L
LT"n
8(1
Q
2
which is equal to
2A
Q T'8(2n Lap)) (Lr
C Experiment supplements
C.1 Datasets and federated simulations
We use CIFAR-IO and CIFAR- /00 [13] datasets to train the models. Both of them contain 50,000 training images and 10,000 testing images. To construct the public dataset, sample 4000 data
15

---

Clasess
Casses
Clases
Casses
(a) Class
(b) D 0.1)
(c) B 0.5)
(d) IID
Figure 6: Illustrations of # of samples per class allocated to each client, for different distributions.
samples from the training set, then divide the remaining data into N partitions to simulate N clients. To assess the validity of the public dataset, we use two sampling methods to construct it. First, we use random sampling method over all classes generate public dataset 'IID'. And for the public dataset 'Partial', data is selected randomly from 40% classes two datasets.
We utilise three settings to simulate heterogeneous data distributions among all the clients. For the IID setting, each client contains the same number of samples from all classes. For the class setting, each client only has 10/N and 100/N classes on two datasets and the classes between clients have no overlap. For the Non-III setting, data heterogeneity levels are described by the Dirichlet distribution Dir(3) [10], where smaller 33 represents stronger heterogeneity levels. A value of 3 0.1 is chosen to simulate a high degree heterogeneity, and 3 0.5 for lower level. Fig. 6 shows the data distribution among clients on CIFAR-IO dataset. The x-axis represents 10 classes and the y-axis is total of 5 clients. The size of each circle denotes the number samples for the specific class in the respective client.
C.2 Reproduction details
Regarding the self-supervised learning framework design within each client, we use ResNet18 [9] and VGG9 [22] as the encoder network and Multi-Layer Perception (MLP) as the predictor. To compare with other baselines, we set the number of clients N 5, and conduct the experiments for R = 100 rounds. In order to construct the model heterogeneous setting, 2 clients train the Resnet18 encoder while 3 clients use the VGG9. And for the global representation model, Resnet18 selected as the encoder in server. The hyper-parameter y in the loss of global model is set to 0.9. During the training process, each client trains locally for = epochs while the server also distills for T' epochs. Finally, set the target decay rate 0.99, with batch size of B 128, and utilize SGD for optimization with a learning rate of = 0.032. We implement all the methods in Python using EasyFL[32] based on PyTorch.
C.3 Baselines
Firstly, we select several federated knowledge distillation frameworks that use unlabeled public dataset for distillation. We then replaced the local model with a self-supervised model to evaluate the process of knowledge distillation in our method.
FedMD[14]: Each client trains convergence on the public dataset and then on local data. In each round, clients upload the embedding of the public dataset to the server. The server averages the embeddings and sends averaged embedding to clients. Clients first use average embedding to update local models on public dataset, then train local data for few epochs. There's no global model, the local models are evaluated to make comparison.
FedDF[18]: Clients in FedDF train locally and upload the model parameter to server. The server uses all client models to compute the embedding of the public dataset and uses the average embedding train the global model. But it has global model, only test the performance local models.
FedET[2]]: There is global large model and several alternative small models the server. Each client selects the appropriate small model for training locally and then uploads the parameter to the server. The server uses all the small models to compute representations of
16

---

the public dataset and updates the global model with the averaged representation. Then the server utilizes the representation from the global model update all the small models.
MOON[15]: Moon introduces model-contrastive learning approach, treating the represen- tation from the global model as positive knowledge to update the local models.
MOON-KL: Instead of using the NT-Xent loss in MOON, MOON-KL utilises the KL- divergence function to measure the similarity between global representation and local representation. To make comparisons, both MOON and MOON-K KL distribute the global model to each client, and then the client uses the local model to update the global model. The server aggregates the updated global models as the final global model.
And several federated sulf-suue learning frameworks are also chosen as baselines.
FedU[33]] Clients in FedU upload the local online encoder and predictor for server aggrega- tion. Then they update the local online encoder using the aggregate ones, and dynamically update the predictor based on the divergence. But cannot applied to the model heterogene- ity setting.
[34EMA[34]: FedEMA uses the aggregated local online network as a global model and measures the divergence between global and local networks. The divergence is then applied to the local exponential moving average update. It's also not applicable when models are heterogeneous.
(SFLL2: Each client trains locally and uploads the kernel matrix over the public dataset. The server aggregates them and sends to clients. Then client uses linear-CKA normalization term to align to the averaged kernel matrix. There's no global model, so we just test local models' performance.
To verify that the client's knowledge can improve the global model, we also train the global model separately on the public dataset, denoted as Std. ResNet18.
C.4 Evaluation methods
Following [33] and [30], we evaluate the performance of learned representations using linear and semi-supervised evaluation. For linear evaluation, all the models are trained without any supervised labels. Subsequently, the model encoder is then frozen and the representations are utilized for training new classifier over 200 epochs. For semi-supervised evaluation, we consider the scenario that only a small subset of the data has label, here only 1% data are labeled. Then, different from freezing the encoder, we fine-tune the whole model with new classifier using the labeled data for 100 epochs. For FedMD, FedDF and Hetero-SSFL. SSFL, only local models can be evaluated. So we use the averaged representation from local models to do the linear evaluation. But in semi-supervised evaluation, we need to fine-tune the whole model, so we evaluate each client's performance alone and use the average result the final result.
D Additional Experiments
D.1 Numerical Results
Table 6, Table 7 and Table 8, Table 9 shows the linear evaluation results and semi-supervised evalua- tion results of FedMKD compared with all the baselines on CIFAR-10 and CIFAR- Comparing to the result in main text, we extra add the result on Dir(3 = 0.1) to prove the efficiency of our algorithm totally.
D.2 Communication and storage efficiency
Considering the experimental setting, the communication and storage capabilities are limited, so we analyze the communication and computation efficiency of FedMKD compared to Hetero-SSFL. and all the experiments maintain the original settings. Firstly, we calculate that the memory of ResNet18 and VGG9 is 42,63MB and 16.38MB separately. We use S()) denote the storage cost of model 0.
About the communication cost, the client in FedMKD only needs to upload the online encoder the server, and download the updated encoder, the cost per communication is equal to the ResNet18 or
17

---

Table 6: Top-1 accuracy comparison under linear probing on CIFAR-I0 datasets with best model performance in bold and second-best results with underlines. means this method suitable for the experiment setting.
Public dataset
CIFAR-10 (%)
Method
Class
Di.(3=0.1)
Dir(3=0.5)
Std. ResNet18
51.09 = 0.04
FedMD
45.28 + 0.02
45.48
+..88+00001 0.01
45.93 + 0.02
46.21 + 0.02
FedDF
46.94 + 0.04
0.06
48.04 0.02
48.74 0.08
MOON-KL
1
45.59.10000
45.84
46.51
MOON
IID
53.35 + 0.08
5..02
53.71
+6.14
FedET
+
5.04
59.38 +
61.43 0.02
FedU/FedEMA
Hetero-SSFL
59.13 + 0.02
62.62 + 0.14
64.04 + 0.04
65.61 + 0.07
FedMKD
64.81 + 0.02
65.96
0.03
0.06 66.98
69.07 + 0.04
Std. ResNet18
50.15 = 0.02
FedMD
47.16 + 0.03
46.87
0.04
46.39
0.02
45.93 + 0.03
FedDF
52.59 + 0.05
52.71
53.50 -: 0.03
54.17 + 0.05 0.05
MOON-KI.
46.41 + 0.03
47.03
0.02
46.81 + 0.03
45.89 + 0.01
0.01
MOON
Partial
54.31 + 0.04
54.59 +
54.54
52.94
FedET
57.75 1 0.01
56.70 0.02
57.08 1 0.01
58.59 0.01
FedU/FedEMA
Hetero-SSFL
63.20 + 0.08
63.61 + 0.02
61.93 + 0.04
61.15 - 0.07
FedMKD
66.39 + 0.09
68.35 1 0.05
67.60 + 0.04
65.88 + 0.03
Table 7: Top-1 accuracy comparison under linear probing CIFAR-100 100 dataset with best model performance bold and second-best results with underlines. means this method not suitable for the experiment setting.
Public dataset
CIFAR (%)
Method
Class
(3.(3=0.1)
Dir(3=0.5)
IID
Std. ResNet18
25.35 - 0.02
FedMD
23.25 + 0.05
23.08
0.02
22.46
23.20
FedDF
23.07 0.08
0.06
0.08
2..
MOON-KL
21.26 1 0.03
2..01
2.00
2.02
MOON
IID
27.82 + 0.01
27.63
26.70 0.08
FedET
29.11 0.03
28.55
26.98 0.01
21.480..0
FedU/FedEMA
Hetero-SSFL
+ 0.10
29.74 0.03
29.63 0.06
28.89 + 0.06
FedMKD
36.33 + 0.01
34.37
0.05
35.59
0.07
35.94 0.02
Std. ResNet18
24.97
= 0.01
FedMD
23.95 0.05
23.28
+
23.14 +.14+0003
22.47 + 0.01
FedDF
27.21
27.83
27.31 0.04
27.05 1: 0.04
MOON-KL
21.33
20.97 0.02
22.27 0.04
MOON
Partial
27.00 0.04
2.01 0.006
28.26 + 0.02 0.02
FedET
29.38 1 0.01
0.002
2.01
29.61 1.01
FedU/FodEMA
Hetero-SSFL.
30.91 + 0.05
30.78 0.07
29.92 0.03
0.03 29.56 + 0.03
FedMKD
35.82 + 0.02
34.70 0.02
35.55 0.05
34.38 + 0.02
VGG9. But for Hetero-SSFL, each client needs to upload the kernel metric K. E R R 1Dpbl the size of the public dataset. In order to make comparison, we fix the communication rounds. Compared to the Hetero-SSFL, when S(O,) 2 < ppuull our FedMKD is better in communication cost. And compared to the MOON, for larger server model, S(O,)) * < S(0.) 2, MOON will cost more when the server model is larger.
About local storage for each client, we compute the summary of the model and data. For FedMKD and Sietero-SFLL each client runs the siamese network, so the model storage is double of the model memory, that
S.,)). But for the MOON, each client processes the local contrastive learning and the model contrastive learning, so they must store two local models and one global model, that is 2 ((0,)) + SO,)). Here, we neglect the storage of the predictor which is usually 2-layer MIP. We use S(ID)) to denote the storage of the dataset. And each client in FedMKD only needs to store the
18

---

Table 8: Top-1 accuracy comparison on 1% of labeled data for semi-supervised learning on CIFAR-10 10 dataset with best model performance in bold and second-best results with underlines. '-' means this method doesn't apply for the experiment setting.
Public dataset
1IAAR-10C (%)
Method
Class
00.1
D0r(3=0.55
IID
Std. ResNet18
46.84 = 0.25
FedMD
43.32 + 0.22
43.83 + 0.25
44.20 0.18
4..66
FedDF
43.60 + 0.44
43.73 0.36
44.33
44.80
MOON-KI.
IID
45.42 + 0.26
46.18+
46.61 0.21
46.72
MOON
49.96 + 0.24
49.90 1.000:0.3.
51.78 50.21
FedET
52.37 + 0.24
54.29 +
0.17
5.14
FedU/FedEMA
Hetero-SSFL
54.30 1 0.15
56.19 1 0.21
58.73 0.54
60.50 1 0.12
FedMKD
59.65 + 0.28
60.71 + 0.05
61.78 0.40
64.06 + 0.32
Std. ResNet18
46.42
+ 0.12
FedMD
44.54
43.61
+ 0.13
42.52 + 0.19 0.19
FedDF
48.14 0.27
4..95
48.74 + 0.12
18.56 + 0.18
MOON-KI KL
46.76 + 0.05
47.52 0.20
46,92 - 0.05
46,49 + 0.22
MOON
Partial
50.43 0.18
5.20
51.99 +
49.86
= 0.19 0.19
FedET
52.75 + 0.07
52.12
52.61 0.03
54.64 0.14
FedU/FedEMA
Hctero-SSFL
59.95 + 0.34
59.34 + 0.08
58.31 + 0.50
58.35 + 0.20
FedMKD
61.55 1 0.19
62.99 1 0.17
63.10 1 0.28
61.08 1 0.19
Table 9: Top-1 accuracy comparison on 1 1% of labeled data for semi remi-supervised learning on CIFAR- 100 dataset with best model performance in bold and second-best results with underlines. '-' means this method doesn't it apply for the experiment setting.
Public dataset
CIFAR 100 (%)
Method
Class
Dir(3=0.1)
Dir(s=0.5)
IID
Std. ResNet18
15.19
1 0.20
FedMD
15.88 + 0.17
15.01 +.14
14.94 0.19
15.34 0.12
FedDF
14.39 + 0.20
0.02
0.10
02.90
MOON-KL
16.25 + 0.06
1.13
1.22
16.07 0.04
MOON
19.23 0.12
0.17
17.07 1..2
FedET
19.70
16.82 1..03
0.118
FedU/FedEMA
Hetero-SSFL
20.04 + 0.40
19.05 + 0.09
0.17 19.19 + 0.17
18.82 0.17
FedMKD
22.57 + 0.12
21,28 + 0.24
22.13 + 0.11
22.07 + 0.15
Std. ResNet18
14.12
0.09
FedMD
17.32 + 17-32+
16.91 0.11
16.47 0.14
16.31 + 0.07
FedDF
17.01 1 0.04
17.60
0.01 17.14
16.95 1 0.01
MOON-KL
16.21 0.27
16.87 0.11
16.10
16.94 + 0.12 0.12
MOON
Partial
18.24
18.92 0.229
19.29 0.08
FedET
18.49 0.12
1..11
1.11
18.01 0.22
FedU/FedEMA
Hetero-SSFL
20.72 + 0.14
20.83 1 0.23
20.30 0.32
19.62 + 0.08
FedMKD
22.44 + 0.19
21.45 0.16
22.21 0.17
20.94 + 0.25
19

---

Table 10: Communication and storage cost comparison of FedMKD and several baselines.
Communication cost
Storage cost
Methods
Resnet18
VGG9
Resnet18
VGG9
FedMKD
85.26MB
32.76MB
193.07MB
140.57MB
MOON
85,25MB
183,20MB
HStero-SSFL.
122.08MB
239.95MB
187.45MB
60
2200
2020
5000
150
202
10
Public Dataset Size
Rounds
Server Epochs
(a) The performance of FedMKD (h) The performance of FedMKD (c) The performance of FedMKD with different public dataset size with different global rounds with different server epochs Cif
on Cifar-10.
Cifar-10.
gesting (90)
Tostig
Clver
0LASS
Class
Publicc Datsset Size
Rounds
Server Epachs
(d) The performance of FedMKD(e) The performance of FedMKD(f) The performance of FedMKD with different public dataset sizewith different global rounds onwith different server epochs on on Cifar-100.
Cifar-100.
Cifar-100)
Figure 7: The t test accuracy of different hyperparameter settings CIFAR-10 and CIFAR100
local training data without the public data, that is S(ID,,)) and for Hetero-SSFL. client needs to store both two S( + S(uDpul l). MOON doesn't 't use the public dataset, so the client only needs to store the local data S([D,, Jalso. Totally, we can get
2 ((,,) + S(ID)) s 2* S(,,) + S(ID,,) + S(( 1),
2 S(O,) S(ID)) 2 S(O.) + S(e.) S(DD).
D.3 Hyperparameter analysis
To explore the influence of different hyperparameters, we conduct experiments several key parameters of FedMKD.
Impact of public dataset size. As mentioned in [20], the size and construct method of the public dataset are both critically important, especially since our global model needs training based on About the latter one we' 'e given an analysis in the last section, so we explore the influence of the size on the model performance. Considering the construct method of the public dataset, we choose part of the dataset as a public dataset and set it aside to be used. The performance change with respect to its size shown in Fig. 7 (a). We can observe that in both two datasets, when the public dataset small, the performance of FedMKD is quite worse, and as the size of the public dataset increases, the performance gets better. But when the size is larger than 4000, the gain of performance is small, and because of the design of the model, the time cost of the server computation is larger, 4000 as our experiment setting.
choose
Impact of the number of global rounds. In order to investigate the impact of the training rounds to FedMKD, we fix the other hyperparameters and train the model for 200 rounds. The results showed in Fig. 7 (b)(e) demonstrate that while the total training rounds increase, the performance of the FedMKD gets better. For both two kinds clients data distribution CIFAR-10. the global model converges at nearly 150 rounds. For CIFAR-100, the global model converges at 200 rounds. There's ' no denying that the increase of the global rounds will result in better performance, especially when the global round is small, the improvement of the performance is significant.
20

---

Impact of the number of server epochs. Since our global model is trained on the public dataset, through standalone training and knowledge distillation. The lower server epoch means the global model may under-fitiing not only the distribution of the public dataset but also the client knowledge. On the other hand, large server epoch will lead large computation cost, and over-fit the distribution of the public dataset, which is harmful to the generalization of the model, so the epoch in server is important. We choose 5 different settings 7" (1, 3, 5. 7, 10} to validate the influence of the server epochs. Based on the experiment results shown in Fig. (())(),, we
that the performance of the global model improves when the server epoch increases with the distribution the public is IID. But the change of improvement gets smaller when the server epoch is larger than 5.
E Additional discussion
In this section, we discuss the limitations and broader impacts of the work.
Limitations. Although we provide detailed explanations of the proposed algorithm and extensive experiments analysis, the theoretical proof of FedMKD not rigorous enough. The characteristic of multi-teacher distillation of the global model has not been sufficiently theoretically justified and only combined with the self-training loss as one whole loss to analyze.
Broader Impacts. FedMKD offers significant societal and technological benefits, which is crucial in kinds of domains like healthcare and finance. It promotes inclusivity by leveraging diverse data from various sources, thereby reducing biases and improving model generalization. Technologically. FedMKD lowers the dependency on labeled data, making federated self-supervised learning more efficient and scalable, and drives innovation in heterogeneous resource-limited devices. By carefully navigating these challenges: deviated representation abilities and inconsistent representation spaces, FedMKD can lead to responsible and equitable advancements in distributed Al technology.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction sections of our paper clearly articulate the main claims and contributions of our research. These include explicit statements of the proposed framework innovations, theoretical advancements, and empirical findings is summarized in lines 62-73. Furthermore, the theoretical arguments Sec.B and the experimental evidence in Scc.5 of the main body of the paper fully support the claims made in this paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: this work, we are the first to propose a multi-teacher knowledge distillation framework, namely FedMKD, to learn global representations with whole class knowledge from heterogeneous clients even under extreme class skew. However, the paper still exists some limitations about the privacy preservation. We discuss the limitations of this work in Appendix E.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
22

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In this work, we provide the full set of theoretical assumptions and the complete process process of the proposed multi-teacher knowledge distillation framework FedMKD in the Appendix.B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
We propose a multi-teacher knowledge distillation framework FedMKD and de- scribe this framework in detail in Sec.4. Furthermore, this paper disclose all the information needed to reproduce the main experimental results of it in Sec.5 and Appendix. C.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
23

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: In this work, datasets. And the code of our work is available at FedMKD, as stated in Sec.5.
the public dataset CIFAR-10 and CIFAR-100 the
com http//mmmeeeuul
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. ec/
heaeiiicggies///eee Soodeeuumissionoo aessonPolicoe oolicyyyyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips.
ssiioiolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We provide the training and test details in Sec.5 and Appendix. C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
24

---

Answer: [Yes]
Justification:
We provide the error bars of the experiments in Sec. 5 and Appendix. D.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In this work, we use the NVIDIA GeForce RTX 3090 cards with 24GB memory as the server and the clients.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://nurrsssccpppliiiiiic/c
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics. And ensure that our work does not involve any inference of personal information and that it will only be used for academic purposes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
25

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of this work in the last paragraph of Ap-
pendix.E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
26

---

Justification: In this work, we introduce the public datasets CIFAR-10 and CIFAR- 100 and the ResNet18 [9] and VGG9 [22] the backbones in the Sec.5.1. And we properly cite the related work in the main body and the appendix of this paper.
Guidelines:
existing assets. The authors should citep the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcae thcode. www.coo///aco// comddataeess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
27

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "ZBdVCmfZEl", "reviewer": "Reviewer_QsUu", "summary": "- The paper proposes a framework called FedMKD which is a multi-teacher knowledge distillation framework for federated learning. \n- To allow for different clients to have different resources, a resource adaptive approach is designed. The approach handles class skew and different architectures. \n- The knowledge distillation framework allows for learning representations from the different client representations. An SSL loss is used in addition on the common representation space. \n- Different from prior works, a global representation model is learnt despite differences in clients. \n- Approach is shown to outperform prior works on CIFAR-10 and 100 datasets.", "questions": "Please refer to the weaknesses section.", "limitations": "The paper has adequately discussed limitations and potential negative impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles an important problem in federated learning: heterogeneity in clients both from the perspective of resources and data. \n- A simple and effective solution is proposed to tackle to learn a common representation space by using techniques in self-supervised learning knowledge distillation. \n- The approach is shown to outperform prior work on CIFAR-10 and 100 benchmarks\n- The paper is easy to follow. Ablations and experiments seem thorough (see weaknesses).\n- While I have not tried it, authors have provided an implementation of their approach.", "weaknesses": "Weaknesses/Concerns & Questions:\n- How does the approach compare to the traditional FL settings when aggregation like L107 is done (assuming all client architectures are the same) ? \n- Did the authors experiment with applying the knowledge distillation to individual projected features instead of the aggregate ? \n- L167: Are the negatives also after the aggregation step ? or from individual projection layers ?   \n- Additional larger-scale datasets will be helpful to understand whether the approach will scale. \n\nPresentation and writing:\n- Table 1 needs more details to be helpful. A more detailed caption should help. More subjective columns like \"inconsistent representation space\", \"deviated representation ability\" are not apt for a table like this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer QsUu for the time and valuable feedback! We would try our best to address the comments one by one.\n\n**Response to Weakness 1:**\n\nThank you very much for the insightful comments. Our proposed FedMKD is designed for resource-aware settings, allowing each client to choose an appropriate model to train, even if the model architectures differ among clients. In this context, traditional aggregation methods are ineffective. However, if all clients use the same architecture, the model with the minimal parameters should be chosen.\n\nIn the newly designed experiments, both the clients' models and the global model are set to VGG9. The linear probing results compared to FedAVG are as follows:\n|  **Methods** \t| **Acc**|\n|:--:|:--:|\n| FedAVG| 48.32% | \n| **FedMKD**| 66.39% | \n\nIn our FedMKD approach, each client can adaptively choose the appropriate local model to fully utilize its computational resources. For example, a client with ample computational resources can select a larger model like ResNet18, while another client with limited resources can opt for a smaller model like VGG9. This flexibility enables FedMKD to outperform FedAVG on the VGG9 model.\n\n**Response to Weakness 2 & 3**\n\nFirst, we apologize for any confusion caused by our previous presentation. To clarify, in Line 167, the negative sample is derived from the individual projection layers of the global model. In knowledge distillation (KD), we have an aggregated anchor representation $\\bar{r_i}$, a positive representation $r_{s,i}$ from global model, and multiple negative representations $r_{s,j}$. The purpose of knowledge distillation is to push the anchor representation $\\bar{r_i}$ and the positive representation $r_{s,i}$ closer together in the feature space while pulling the anchor away from the negative representations. Thus, the global model could encode all classes from clients in a unified space. Formally, we define this KD loss function, which is presented in the original paper on Line 168:\n\n$L_{distill}=-log\\frac{ exp(sim(r _ {s,i}, \\bar{r} _ i)/\\tau)}{exp(sim(r _ {s,i},\\bar{r} _ i)/\\tau)+\\sum\\limits _ {j\\neq i}exp(sim(r _ {s,i},r _ {s,j})/\\tau)}.$\n\nWe also illustrate this process in the **Global Response PDF (see Figure 8)**.\n\n**Response to Weakness 4:**\n\nThank you very much for the insightful comments. We use **ImageNet-100** and **ImageNet-1k** to evaluate the effectiveness of our proposed method. The results, presented in **Table 11 of the Global Response PDF**, demonstrate that our proposed FedMKD outperforms all baseline methods.\n\n**About Presentation and writing:**\n\nThank you very much for the insightful comments. We will revise the caption of Table 1 to \"Comparison of federated self-supervised learning methods. √ indicates that the proposed method focuses on this challenge and × indicates that it does not.\". Additionally, the columns labeled \"inconsistent representation space\" and \"deviated representation ability\" will be replaced with the objective description \"data heterogeneity.\" The modified table is presented as **Table 12 in the Global Response PDF**.\n\nIf there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ymWJ8paxku", "reviewer": "Reviewer_de59", "summary": "This paper studies the problem of federated self-supervised learning. A multi-teacher knowledge distillation framework is proposed to address the two challenges: deviated representation abilities and inconsistent representation spaces. Specifically, the adaptive knowledge integration mechanism is designed to learn better representations from all heterogeneous models, and a global knowledge-anchored alignment module is used to make the local representation spaces close to the global spaces. Experiments on two datasets demonstrate the effectiveness of the proposed method.", "questions": "Weakness.", "limitations": "The authors adequately addressed the limitations in the appendix.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper is well-written and easy to follow.\n* The analysis of the challenges, i.e., deviated representation abilities and inconsistent representation spaces, is clear.\n* Introducing an attention module to integrate the knowledge is technique sound.", "weaknesses": "* Knowledge distillation is widely used in federated learning and federated self-supervised learning. The contribution of the proposed method (i.e., multi-teacher knowledge distillation framework) is incremental.\n* This paper uses small models such as resnet18 and vgg9.\n* Only CIFAR datasets are used to evaluate the proposed method. Experiments on ImageNet-1k are important to evaluate the effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer de59 for the time and valuable feedback! We would try our best to address the comments one by one.\n\n**Response to Weakness 1:**\n\nThank you very much for the insightful comments. \nAs you said, knowledge distillation (KD) has been widely used in Fed-SSL, such as FedX and other works. However, our contrastive learning-based multi-teacher KD introduces novel approaches to address the challenges of **deviated representation abilities** and **inconsistent representation spaces**:\n- First, to mitigate the impact of heterogeneous models with **deviated representation abilities**, we introduce an **adaptive knowledge integration** module to learn high-quality representations from them.\n- Second, to encode all classes from clients in a unified space, the global model updates using a **weighted combination of self-supervised loss and distillation loss** to update. \n- Finally, the global knowledge anchored alignment module is applied on the server to eliminate the **inconsistency in representation spaces** and reduce the burden on the clients.\n\nIn our experiments, FedMD and FedDF, which are based on traditional KD methods, were adopted as comparative baselines. As shown in Table 2 (Page 7) and Table 3 (Page 8), our proposed FedMKD achieves the best performance.\n\n\n**Response to Weakness 2:**\n\nThank you very much for you valuable comment. \nAdopting larger models can indeed demonstrate the scalability of our proposed FedMKD framework. \nAs suggested, we set one client's local model to **ResNet50** and re-conduct the experiment under the setting of the 'Partial' public dataset and 'Class' distribution among clients on **CIFAR-10**. All other settings remained identical to those in the original paper. The linear probing results are shown below: \n\n|  **Method** \t| **1 ResNet50**|**Original**|\n|:---:|:---:|:-:|\n| FedMD| 48.37% |47.16%|\n| FedDF| 54.46% |52.59%|\n| MOON-KL| 48.48% |46.41%|\n| MOON| 56.33% |54.31%|\n| FedET| 58.96% |57.75%|\n| Hetero-SSFL| 64.74% |63.20%|\n| FedMKD| 67.75% |66.39%|\n\nThe results indicate that the introduction of a larger model can improve overall performance, and our proposed FedMKD still outperform all baselines.\n\n**Response to Weakness 3:**\n\nThank you very much for the insightful comments. We use **Imagenet-100** and **Imagenet-1k** to evaluate the effectiveness of our proposed method. The results are presented in **Table 11 in Global Response PDF**. We can conclude that our proposed FedMKD outperforms all baselines.\n\nIf there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5eptseOyFg", "reviewer": "Reviewer_24fB", "summary": "This paper proposes a multi-teacher knowledge distillation framework named FedMKD for resource-adaptive federated self-supervised learning (Fed-SSL). The method aims to address the challenges of global representation learning in Fed-SSL caused by heterogeneous architectures and imbalanced class distributions. Experimental results demonstrate that FedMKD is significantly effective in addressing heterogeneity and class distribution imbalance issues in federated self-supervised learning.", "questions": "See weakness.", "limitations": "The characteristic of multi-teacher distillation of the global model has not been sufficiently theoretically justified and the authors only combined it with the self-training loss as one whole loss to analyze.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Innovative Framework Design: This paper proposes a new multi-teacher knowledge distillation framework, FedMKD, which combines self-supervised learning and knowledge distillation to effectively address the challenges of heterogeneity and data imbalance in federated learning.\n\nAdaptive Knowledge Integration Mechanism: Through an adaptive weighting strategy, it integrates high-quality representations from heterogeneous models, significantly enhancing the representation capabilities of the global model.\n\nGlobal Knowledge Anchored Alignment: The design of a global knowledge anchored alignment module ensures that local models' representation spaces are closer to the global representation space, thus improving the performance of local models.", "weaknesses": "1. This paper combines federated self-supervised learning and resource-aware federated learning. However, in introduction and related works, the authors lack a discussion about some works about resource-aware federated learning with heterogeneous clients.\n.\n2. The authors assume to build a global dataset to perform knowledge distillation and anchored alignment. However, random sampling makes the global dataset and client dataset have the similar data distribution, which violates the original intention of data privacy in FL. Please provide some explanation for this.\n\n3. CIFAR-10 and CIFAR-100 are both image datasets with relatively small sizes and low image resolution, which cannot fully reflect the complexity of real-world applications. It is recommended to add experiments on large-scale image datasets such as Tiny-ImageNet and ImageNet-100.\n\n4. The framework of dual encoders increases the training load in clients. It is recommended to evaluate the computation time and resources required and provide analysis for local model training.\n\n5. To evaluate the robustness and effectiveness of this framework, it is suggested to explore experiment settings with more clients, such as [1].\n\n[1] ScaleFL: Resource-Adaptive Federated Learning with Heterogeneous Clients", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer 24fB for the time and valuable feedback! We would try our best to address the comments one by one.\n\n**Response to Weakness 1:**\n\nThank you very much for the insightful comments. In the related works section (Lines 94-99), we have surveyed several existing studies that address heterogeneous client models in federated self-supervised learning. We analyzed the advantages and disadvantages of **Hetero-SSFL** and **FedFoA** in detail. Additionally, we select **Hetero-SSFL** as one of the baselines in the experiment to compare and evaluate the performance of our proposed method. And as you mentioned, we will further add the description of **Hetero-SSFL** and **FedFoA** in the introduction in the final version.\n\n**Response to Weakness 2:**\n\nThanks for your sincere comments. \nAs you are concerned, this construction method of public dataset is the experimental simulation part only does not pose a serious risk of data leakage, in our FedMKD, no data need to be transferred. Here, we just want to address one practical problem: if the distribution of public dataset is different from the data distributions of clients, can the proposed FedMKD still work?\n\nIn practice we can use appropriate \npublic dataset to distill knowledge according to the specific task instead of this random sampling method. To verify this, we conducted another experiment using **CINIC** as the public dataset on the server. **CINIC** is an extension of CIFAR-10, augmented with downsampled ImageNet images, and it shares the same size and classes as CIFAR-10.\nWe still use **CIFAR-10** as the train dataset for the clients. The public dataset is 'Partial' and the data distribution in the client is 'Class'. The other setting is identical to the draft. The linear probing results are shown as follows:\n\n| |  **Public dataset** | **Acc**|\n|:--:|:--:|:---:|\n|FedMKD| CINIC | 59.98% |\n|FedMKD| CIFAR-10 | 66.39% | \n\nHere we can see that even when intuitively choosing an appropriate public dataset, our method still works effectively. \n\n**Response to Weakness 3:**\n\nThank you very much for the insightful comments. We use **ImageNet-100** and **ImageNet-1k** to evaluate the effectiveness of our proposed method. The results are presented in **Table 11 in Global Response PDF**. We can conclude that our proposed FedMKD outperforms all baselines.\n\n**Response to Weakness 4:**\n\nThank you very much for the insightful comments. We evaluate another single-encoder framework, **SimCLR**, on the clients and assessed the computation time and resources required for local model training. Although this method is more computationally efficient, its performance is much inferior to ours. The results are shown as follows:\n\n| |  **Local method** \t| **Acc**| **VGG Memory** |**ResNet18 Memory**|\n|:--:|:--:|:---:|:---:|:---:|\n|FedMKD| SimCLR-single encoder| 48.32% | 21.066M|21.668M|\n|FedMKD| BYOL-dual encoder| 66.39% | 37.857M|49.294M|\n\n**Response to Weakness 5:**\n\nThank you very much for the insightful comments. We've explored the scalability of our proposed algorithm FedMKD in Appendix D.3. In these experiments, we test scenarios with 10 and 30 clients, where 40% of the clients use the VGG model and 60% use ResNet18. We repartition the data for each client under Dir($\\beta=0.5$) and set the public dataset distribution as IID. To compare scalability performance, we conduct the same experiments on the second-best baseline, Hetero-SSFL. The linear probing results are shown as follows:\n\n|  **Method** \t| **5-Client**| **10-Client** |**30-Client** |\n|:------------:|:-------:|:-------:|:-------:|\n| Hetero-SSFL | 59.13% | 54.51% | 49.22% |\n| Ours| 67.79% | 57.39% | 53.85%|\n\nThe results show that as the number of clients increases, our proposed method still outperforms **Hetero-SSFL**, demonstrating the scalability of our **FedMKD**.\n\nIf there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IHBKY6BClb", "reviewer": "Reviewer_sGaU", "summary": "The authors propose a multi-teacher knowledge transfer framework, FedMKD, based on two challenges in resource-adaptive federated self-supervised learning: deviated representations abilities and inconsistent representations. This framework uses an adaptive knowledge integration mechanism and a weighted combination of different loss functions to ensure that global representations with class-wide knowledge can be learned from heterogeneous clients even in the presence of extreme class skew. Extensive experiments on two datasets demonstrate the effectiveness of FedMKD, which outperforms the state-of-the-art baseline by an average of 4.78% under linear evaluation.", "questions": "1.\tAs shown in Weakness 1, it seems to that multi-instructor knowledge distillation is allowing global models to learn local models, while global knowledge anchored alignment is allowing local models to learn global models, so does the two mechanisms result in duplication of knowledge transfer?\n\n2.\tIn the problem statement, the main objective is the global loss. Since recent studies on cross-device scenarios also consider personalized FL (PFL), how about the personalization? Can your method be extended to PFL?", "limitations": "The authors pointed out the limitations of the paper that is the theoretical proof of FedMKD is not rigorous enough, the characteristic of multi-teacher distillation of the global model has not been sufficiently theoretically justified.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tThe scenarios presented and the challenges solved in this work are of importance in reality and the proposed multi-teacher knowledge distillation based federated self-supervised learning framework is able to solve the challenges effectively.\n\n2.\tBy innovatively combining self-supervised loss and distillation loss, FedMKD can encode skew classes into a unified space, which is not covered by related work.\n\n3.\tThe paper is easy to follow, including the main paper and appendices. The introduction provides a clear motivation and a brief overview of the proposed framework, and the other sections provide detailed descriptions of the method. \n\n4.\tThe experiments are well-organized and successful, making this work more convincing.", "weaknesses": "1.\tThe mechanism design seems to be confusing. As shown in the FedMKD, multi-instructor knowledge distillation is allowing global models to learn local models, while global knowledge anchored alignment is allowing local models to learn global models, knowledge transfer happens alternately.\n\n2.\tFedMKD aims to learn global class representations according to a global model. But an additional global-anchored mechanism improves the local performance. What’s the difference between personalized federated learning and the proposed FedMKD?\n\n3.\tIt’s suggested that the authors could add the instructions about the whole model training algorithm in Appendix A, which can help reviewers understand the whole process.\n\n4.\tThe presentation needs to be improved, especially for some equations. The self-supervised loss value in Eq. (10) was not discussed earlier. If it is equal to the loss value in Eq.(3), but with a different notation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer sGaU for the time and valuable feedback! We would try our best to address the comments one by one.\n\n**Response to Weakness1 & Question1:**\n\nThank you very much for your recognition. As you mentioned, we have two mechanisms to transfer knowledge, but they are not duplicated. First, the multi-teacher knowledge distillation allows the global model to encode all classes from clients in a unified space. And the global knowledge-anchored alignment module is applied to eliminate inconsistency in representation spaces of clients, which is benefit to the global model training, also. These two machenisms are critical for learning global class representation and are indispensable.\n\n**Response to Weakness2 & Question2:**\n\nThank you for your insightful comments. The additional global anchored mechanism aims to ensure that the local representation spaces are closer to the global ones so that the following knowledge distillation process could gain more from clients. This module can further eliminate the effect of inconsistent representation spaces and improve local performance. The experiment result of the improvement in client is shown in Section 5.4 which verifies the effectiveness of the module. Compared to personalized FL, the optimization objective is not identical. We aim to train a global model that can encode all classes in a unified space, and the improvement of clients is a middle process. But for PFL, it aims to learn better local model without global model, it's quite different from ours.\n\n**Response to Weakness3:**\n\nAs you suggested, we would add the instructions about the whole model training algorithm in Appendix A in our final version.\n\n**Response to Weakness4:**\n\nWe are sorry for our unclear discussion. Eq. (10) shows the weighted combined loss for the global model, it includes two parts, the self-supervised loss and knowledge distillation loss. So that the global model can encode all classes from clients in a unified space and improve its representation capability. The self-supervised one is identical to the local training loss, that is Eq. (3).\n\nIf there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FuRZg2xS3F", "reviewer": "Reviewer_hPCk", "summary": "This paper introduces a novel approach for enhancing global representation models in resource-adaptive federated self-supervised learning through a multi-teacher knowledge distillation framework, named FedMKD. The proposed method addresses the challenges posed by heterogeneous architectures and extreme class skew, demonstrating significant improvements in representation abilities across diverse clients. The authors provide detailed experimental results to support their claims.", "questions": "It would be beneficial for the authors to include a complexity analysis of the FedMKD algorithm to provide insights into its computational efficiency. If a complexity analysis is not feasible, the authors should provide a rationale for its omission.", "limitations": "The limitations of the FedMKD framework are well-addressed in the paper. These constraints appear to be inherent to the design choices made in the method and are necessary trade-offs for its functionality.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a critical issue in federated self-supervised learning, offering a unique and effective solution.\n2. FedMKD effectively leverages multi-teacher knowledge distillation to integrate knowledge from heterogeneous clients, even under extreme class skew.\n3. The adaptive knowledge integration mechanism enhances the representation abilities of heterogeneous models.\n4. The experimental section is thorough, demonstrating the efficacy of the proposed method on multiple datasets.\n5. The combination of self-supervised loss and distillation loss, along with the global knowledge anchored alignment module, significantly improves local and global representation abilities.", "weaknesses": "The Related Work section could be expanded to include a more thorough discussion of existing approaches in both transfer learning and attention mechanisms.\nThe paper may have a limited audience due to its specialized focus on image classification tasks. It could benefit from providing more context for readers unfamiliar with the specific datasets and techniques discussed.\nThe contributions of the work could be more explicitly stated in the Introduction to enhance clarity for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer hPCk for the time and valuable feedback! We would try our best to address the comments one by one.\n\n**Response to Weakness:**\n\nThank you very much for the insightful comments. \n\n- First, we survey some **federated transfer learning (FTL)** methods, and add the following discussion of them.\n\"Federated transfer learning aims to share the knowledge and insights derived from training machine learning models across various entities, all while keeping the raw data decentralized. It has been used widely in various regions, such as cross-domain recommendation [1,2], medical image classification [3,4] and financial service [5,6]\"\nAbout **attention mechanisms**, we would add the following discussion as follows.\n\"Attention mechanism is a component used in neural networks that dynamically focuses on the most relevant parts of the input data, enhancing the model's performance. By assigning varying levels of importance to different pieces of information, it allows the network to prioritize and efficiently process significant elements, has proven to be highly successful in various tasks, such as image classification [7,8], object detection [9,10] and image generation [11,12].\"\nBoth of these discussions will be added in Related Work in the final version.\n\n- By the way, we would add more detailed description for readers who are unfamiliar with the specific datasets and techniques to understand our method as follows. \"We use CIFAR-10 and CIFAR-100 to evaluate the performance. The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. CIFAR-100 dataset is just like the CIFAR-10, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class.\"\n\n\n- Finally, we present our contributions in **Lines 65-76** of the Introduction. We emphasize that in resource-adaptive Fed-SSL, we are the first to delve into global class representations learning by addressing the deviated representation abilities and inconsistent representations caused by heterogeneous architectures and class skew. We designed **FedMKD** to tackle these challenges, and the experimental results demonstrate the superiority of our proposed **FedMKD**.\n\n\n**Response to Question:**\n\nThank you very much for the insightful comments. We would like to analyze the complexity of client and server separately. First in the client, each client chooses the appropriate model according to their resource, so the computation complexity is different among them. Then in server, the training process has two phases. The complexity of knowledge distillation is $O(T' \\times |D_{pub}| \\times (N+1))$, where $T'$ is the epoch of knowledge distillation, $|D_n|$ is the size of the public dataset, and $N$ is the number of client local model. The complexity of the global knowledge anchored is $O(|D_{pub}| \\times N)$. By the way, considering that each deep network includes multiple operations and computations, we commonly use the amount of computation (FLOPS) to analyze the time complexity and the amount of memory access (Bytes) to analyze the space complexity. The results of each process are shown as follows:\n|  **Process** \t| **Memory** |**FLOPs** |\n|:------------:|:-------:|:-------:|\n| Local-VGG9 | 37.857M |121.761G|\n| Local-ResNet18| 49.294M |440.813G|\n| Distillation | 343.931M |1.701T|\n| Alignment | 60.129M |295.309G|\n\n\nIf there are any further confusions/questions, we are happy to clarify and try to address them. Thank you again and your recognition means a lot for our work.\n\n---\n[1] Ammad-Ud-Din, Muhammad, et al. \"Federated collaborative filtering for privacy-preserving personalized recommendation system.\" arXiv preprint arXiv:1901.09888 (2019).\n\n[2] Minto, Lorenzo, et al. \"Stronger privacy for federated collaborative filtering with implicit feedback.\" Proceedings of the 15th ACM Conference on Recommender Systems. 2021.\n\n[3] Gong, Xuan, et al. \"Federated learning with privacy-preserving ensemble attention distillation.\" IEEE transactions on medical imaging 42.7 (2022): 2057-2067.\n\n[4] Sui, Dianbo, et al. \"Feded: Federated learning via ensemble distillation for medical relation extraction.\" Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). 2020.\n\n[5] Shaheen, Momina, Muhammad Shoaib Farooq, and Tariq Umer. \"Reduction in data imbalance for client-side training in federated learning for the prediction of stock market prices.\" Journal of Sensor and Actuator Networks 13.1 (2023): 1.\n\n[6] Pourroostaei Ardakani, Saeid, et al. \"A federated learning-enabled predictive analysis to forecast stock market trends.\" Journal of Ambient Intelligence and Humanized Computing 14.4 (2023): 4529-4535.\n\n[7] Hu, Jie, Li Shen, and Gang Sun. \"Squeeze-and-excitation networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[8] Woo, Sanghyun, et al. \"Cbam: Convolutional block attention module.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\n[9] Dai, Jifeng, et al. \"Deformable convolutional networks.\" Proceedings of the IEEE international conference on computer vision. 2017.\n\n[10] Carion, Nicolas, et al. \"End-to-end object detection with transformers.\" European conference on computer vision. Cham: Springer International Publishing, 2020.\n\n[11] Gregor, Karol, et al. \"Draw: A recurrent neural network for image generation.\" International conference on machine learning. PMLR, 2015.\n\n[12] Zhang, Han, et al. \"Self-attention generative adversarial networks.\" International conference on machine learning. PMLR, 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers' valuable comments and feedback, which are great helpful in improving the quality of this paper. We try our best to address the concerns including making preliminary experiments as long as time allows. As the **reviewer** **QsUu**, **de59**, **24fB** concerned, we try our best to conduct **new experiments** on **large-scale image datasetes, ImageNet-100 and ImageNet-1k**. We utilize 70 RTX 3090 for more than 100 hours to conduct these experiments. Due to the time limitation, here we only present preliminary results and more results will be presented in the final version. The setting is as follows: the public dataset is 'Partial' and the data distribution in the client is 'Class'. The other setting is identical to the draft. The linear probing results are shown in **Table 11** in PDF. 

**Figure 8** in PDF illustrates the multi-teacher knowledge distillation method in response to **reviewer** **QsUu**'s Weakness 2 and Weakness 3.

**Table 12** in PDF is an updated version of the original Table 1, addressing the feedback of "Presentation and writing" from **reviewer** **QsUu**.

Thank you again for the detailed valuable comments to help improve the quality of our work.

Meta-review: The paper proposes a novel approach, which is a multi-teacher knowledge distillation framework for federated learning. In general, the reviewers are satisfied with the manuscript and the author responses. They have appreciated the proposed framework, writing and experiments. Though one reviewer response is still negative, I feel that the author's response adequately addresses the concerns regarding using other models and additional datasets. The recommendation is to accept the paper. The authors are strongly encouraged to address any remaining concerns in the camera-ready version.
Decision: Accept (poster)