Title: Resource-Aware Federated Self-Supervised Learning with Global Class Representations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Federated self-supervised learning

Heterogeneity

Distillation
Abstract: Due to the heterogeneous architectures and class skew, the global representation models training in resource-adaptive federated self-supervised learning face with tricky challenges: $\textit{deviated representation abilities}$ and $\textit{inconsistent representation spaces}$. 
In this work, we are the first to propose a multi-teacher knowledge distillation framework, namely $\textit{FedMKD}$, to learn global representations with whole class knowledge from heterogeneous clients even under extreme class skew. Firstly, the adaptive knowledge integration mechanism is designed to learn better representations from all heterogeneous models with deviated representation abilities. Then the weighted combination of the self-supervised loss and the distillation loss can support the global model to encode all classes from clients into a unified space. Besides, the global knowledge anchored alignment module can make the local representation spaces close to the global spaces, which further improves the representation abilities of local ones. Finally, extensive experiments conducted on two datasets demonstrate the effectiveness of $\textit{FedMKD}$ which outperforms state-of-the-art baselines 4.78\% under linear evaluation on average.

Resource-Aware Federated Self-Supervised Learning with Global Class Representations

Abstract

Due to the heterogeneous architectures and class skew, the global representation models training in resource-adaptive federated self-supervised learning face with tricky challenges: deviated representation abilities and inconsistent representation spaces . In this work, we are the first to propose a multi-teacher knowledge distillation framework, namely FedMKD , to learn global representations with whole class knowledge from heterogeneous clients even under extreme class skew. Firstly, the adaptive knowledge integration mechanism is designed to learn better representations from all heterogeneous models with deviated representation abilities. Then the weighted combination of the self-supervised loss and the distillation loss can support the global model to encode all classes from clients into a unified space. Besides, the global knowledge anchored alignment module can make the local representation spaces close to the global spaces, which further improves the representation abilities of local ones. Finally, extensive experiments conducted on two datasets demonstrate the effectiveness of FedMKD which outperforms state-of-the-art baselines 4.78% under linear evaluation on average.

1 Introduction

The federated self-supervised learning (Fed-SSL) has emerged as a highly promising paradigm due to the extremely limited labeled data in real-world scenarios [4, 25, 30]. The Fed-SSL mechanism can learn common representations collaboratively across all the clients without labeled data [7, 8], which could enable the aggregation of knowledge from diverse unlabeled data sources and overcome the limitations caused by the high cost and scarcity of labeled data [24, 28, 31].

Traditional Fed-SSL methods usually assume that each client should train the identical architecture model, such as FedU [33], FedEMA [34], FedCA [30]. But it would not be easy in resource-limited scenarios, especially for the existing arsing large-scale models [1, 26]. As shown in Fig.1, client A , client B and client C might train heterogeneous representation models due to the varying system resources. In addition, real-world data often exhibits skewed class distributions across clients.

Therefore, how to learn global class representations under the heterogeneous architectures and class skew in resource-aware Fed-SSL paradigm is challenging, particularly comparing with existing FedU 2 [17], FedX [5] with identical architectures.

Although both Hetero-SSFL [20] and FedFoA [19] consider the heterogeneous client models, they can not learn a global representation model. In order to aggregate the knowledge from the clients to form global class representations, some tricky challenges arise. (1) Deviated representation abilities. Even for the same data samples, the different models might encode them into different latent spaces with deviated representation abilities . For example, client A and client B all have images with dog, cat, tiger , but client model A can encode cat, tiger well into different clusters, client model B can only learn better representations of dog . So how could global representation models take advantage of the best of both client models ? (2) Inconsistent representation spaces. The skewed class distributions across clients lead to inconsistent representation spaces. For example in Fig. 1, comparing with client A , client C has different kinds of images with dog, cat, airplane . Thus how to make global representation models encode the whole classes from all the clients well in a unified space ? Therefore, different from the existing works, our goal is to break the gaps caused by the hybrid heterogeneity, which can learn the high-quality global representation model in federated self-supervised learning.

Along this line, we propose FedMKD , a multiteacher knowledge distillation based resourceadaptive Fed-SSL framework, which can learn global representations over all classes from heterogeneous clients. First, an adaptive knowledge integration module is introduced to learn high-quality representations from all the heterogeneous models with deviated representation abilities. Then in order to encode all classes from clients in a unified space, the global model uses the weighted combination of selfsupervised loss and distillation loss to update. Besides, the global knowledge anchored alignment module is applied within the server to eliminate the inconsistency in representation spaces and reduce the burden on the clients. It uses global knowledge to additionally update the local models, which can not only make the local representation spaces close to the global space but also improve the representation capability of both local models and the global ones. Code is available at https://github. com/limee-sdu/FedMKD . The main contributions of this paper can be summarized as follows.

Global

representation

space

Client A

Client B

Client C

Server

Cat

Representation Space

Representation Space

dog

plane

cat

Inconsistent

representation

dog

cat

dog

tiger

Deviated

representation

abilities

Dog

Tiger

Plane

Cat

Dog

Tiger

Plane

Cat

Dog

Tiger

Plane

In resource-aware Fed-SSL, we are the first to delve into global class representation learning through revealing the deviated representation abilities and inconsistent representation spaces caused by the heterogeneous architectures and class skew.

We design a multi-teacher knowledge distillation framework, namely FedMKD , to adaptively aggregate positive knowledge from heterogeneous models with deviated representation abilities. Through combining the self-supervised loss and the distillation loss, FedMKD can encode skewed classes into a unified space.

Extensive experiments conducted on CIFAR-10 and CIFAR-100 show the representation abilities over all classes of the FedMKD perform better than state-of-the-art baselines. Our algorithm can improve 4.22% and 5.31% separately on the two chosen datasets.

2 Related work

The federated self-supervised learning aims to learn high-quality representations from clients without large labeled datasets [11]. From the beginning, several works [12, 23] simply combine federated learning with self-supervised methods. Besides, FedU [33] designs a communication-efficient mech-

anism by only aggregating the online encoders under non-IID data. FedUTN [16] is proposed to use the aggregated online networks for the target network updating in the self-supervised framework. L-DAWA [21] proposes the layer-wise divergence aware weight aggregation to mitigate the influence of client bias. FedEMA [34] considers the divergence-aware moving average updating in clients, measuring the divergence between local models and global model. FedX [5] proposes a unsupervised federated learning framework to learn representations through a two-sided distillation method. However, all the above works intuitively gain the global model through parameters average due to the identical client models, which can not be applied in the heterogeneous clients setting directly. In addition, although FedCA [30] address the misaligned and inconsistent representation challenges by gathering features from clients, inducing potential privacy problems. FLPD [27, 29] introduces distillation method based similarity between prototypes from a labeled public dataset to update the local model. FedU 2 [17] focuses on mitigating representation collapse entanglement and obtaining unified representation spaces.

Considering heterogeneous client models in federated self-supervised learning, Hetero-SSFL [6] introduces linear-CKA to align lower-dimensional representations between the local model and global model without architectural constraints. FedFoA [19] designs a factorization-based method to extract the cross-feature relation matrix from the local representations for aggregation. However, both Hetero-SSFL and FedFoA can not learn a global representation model considering the hybrid heterogeneity, which is the main focus of our work. The comparison details are shown in Table 1.

3 Preliminaries

The goal of federated unsupervised learning is to learn the generalized representation for some downstream tasks from several distributed unlabeled data sources. A federated learning setting consists of a central server and N clients. Each client k contains a local unlabeled dataset D k , and the server contains a public unlabeled dataset D pub . The local objective at k -th client is

to minimize the expected local loss of client k on local dataset D k and ξ k is the unlabeled data. In traditional FL settings, it's assumed that { θ k } are identical and gain the global model using θ = ∑ N k =1 p k θ k , where p k is the weight of k -th client. But in real-world cross-device scenarios, each client might have a unique model and the architecture of the model might be different, which means that traditional aggregation methods are not available. We assume that θ k is not similar to others, and use { θ k } to collaborate in training the larger global model θ in server. Here we define the global update function is θ t = G ( θ t -1 ; θ 1 , · · · , θ N ) . Our final aim is to optimize the global goal

where L ( θ, ξ ) is global loss function in server and ξ is the unlabeled data sampled from global dataset.

4 Designed FedMKD Method

Wepropose a multi-teacher knowledge distillation based federated self-supervised learning framework FedMKD , which is shown in Fig. 2. In FedMKD , besides the local self-supervised learning (Sec. 4.1),

Online

encoder

Online

encoder

Encoder

Encoder

Online

encoder

Encoder

Encoder

Encoder

Encoder

Online

encoder

Target

encoder

Online

encoder

Target

encoder

Initialization

Local training

Clients

Server

Attention

Contrastive

Global anchored alignment

Align

~

~

~

Target

encoder

Online

encoder

p(·)

p(·)

p(·)

Online

encoder

p(·)

Online

encoder

Align

Multi-teacher knowledge distillation

Global

model

Global

model

Global

model

we design a multi-teacher adaptive knowledge integration distillation module to adaptive determine the weight of the representations from heterogeneous local models with deviated representation abilities. The distilled loss combined with the global self-supervised loss, we can gain the weighted combined loss to update the global model, so that the global model can encode all classes from clients in a unified space (Sec. 4.2). And the global knowledge anchored alignment could improve the representation capability of clients and further benefit the global model training (Sec. 4.3). In addition, we provide the theoretical analysis of our algorithm in Appendix B.

4.1 Self-supervised model training

Each client performs self-supervised contrastive learning using an asymmetric Siamese network, inspired by BYOL [3]. The model M comprises an online encoder θ and a target encoder ϕ , both sharing the same architecture, with the online network incorporating an additional predictor p . That is M = { p ( ϕ ( · )) , θ ( · ) } . Given an unlabeled image x , we can obtain two augmented views, v and v ′ , serving as inputs to online and target networks, respectively. The loss function is defined as follows:

where r = θ ( v ) and r ′ = ϕ ( v ′ ) . This loss encourages the online network to produce representation p ( r ) that is similar to the positive sample generated by the target network r ′ . We then exchange the views, feeding v ′ to the online network and v to the target network, to compute L ′ self . At each training step, we use stochastic gradient descent to minimize ˜ L self = L self + L ′ self to update the online network ϕ alone,

The target network helps to provide regression targets to train the online network. Choosing α ∈ [0 , 1] as the target decay rate, we employ the exponential moving average (EMA) of the online network to update ϕ :

Using this self-supervised training method, the model learns intricate representations from unlabeled data, capturing high-level features and patterns inherent in the dataset.

4.2 Multi-teacher adaptive knowledge integration distillation.

In contrast to homogeneous federated learning, the presence of model heterogeneity poses a challenge: direct aggregation of local models into a global model is not feasible. To overcome this, we design a multi-teacher knowledge distillation mechanism to transfer local knowledge to the server.

Given a batch of data B , the representation from the teacher model is denoted as r t and that from the student model as r s , the knowledge distillation loss is defined as follows:

where τ is the temperature parameter controlling entropy and sim ( · ) is the similarity function between two representations.

Then we extend this knowledge distillation learning method to multi-teacher. Although the data is heterogeneous, the knowledge of each local model is valuable, each local model captures the unique characteristics of local data. Our goal is to integrate the positive knowledge of all clients to guide the global model in learning a general representation of unlabeled data. We design a multi-teacher adaptive knowledge integration distillation that can adaptively weigh the representations from clients.

Given a sample x i , representation from the n -th local model is R n,i ∈ R d where d is the dimension of the representation. Following [3], a fully connected layer is employed to project the representation into a lower-dimensional space, enhancing the discriminate power of the learned representations. So, we map the representation from the global model R s,i into the same lower latent space, obtaining

where r n,i , r s,i ∈ R k , k is the dimension of the new latent space and g ( · ) is the projector.

In addition, we introduce an adapter module to learn instance-level teacher importance weights for knowledge integration. After getting r n,i , an attention block is used to generate the weighted sum of them. In this context, representation from global model r s,i is treated as the query , while those from local models ˜ R = [ r 1 ,i , r 2 ,i , . . . , r N,i ] T is treated as the key and value . Treating representations from the global model as query ensures consistency in knowledge transfer. The attention mechanism computes attention scores to understand the relevance of each local model's representation to the global model's query. The aggregated representation is:

where ¯ r i means the aggregated representation, and Attn ( · ) denotes the attention block.

Returning to the knowledge distillation for unlabeled data proposed earlier, we treat the aggregate representation as the positive sample, and the remaining samples in the batch as the negative sample. The adaptive weight multi-teacher knowledge distillation loss function is expressed as follows:

̸

Above all, the weighted combined loss for the global model is presented as:

where γ is a hyper-parameter controlling the weight of the distillation process.

4.3 Global knowledge anchored alignment

As we mentioned, the representations from different models are inconsistent and the representation abilities of models are also deviated. So we introduce the global knowledge anchored alignment mechanism that each local model uses the global model as an anchor. It ensures that the local representation spaces are closer to the global ones.

Unlike methods such as FedX [5] and MOON [15], which align local models to the global model locally, our approach aims to train a better global encoder tailored for resource-constrained federated learning scenarios. Those methods are not available when clients cannot afford to store or infer the global model locally. So we transfer this alignment process to the server.

After finishing the global model training, we construct local twin models in the server to realize the alignment under global view. Here, we use the global online network and local online network to

airplane

automobile

bird

cat

deer

dog

frog

horse

ship

truck

airplane

automobile

bird

cat

deer

dog

frog

horse

ship

truck

airplane

automobile

bird

cat

deer

dog

frog

horse

ship

truck

airplane

automobile

bird

cat

deer

dog

frog

horse

ship

truck

construct a new asymmetric siamese network called the twin of the original local model. The local online network θ n is the online model, and the global online network θ s is the target model, that is,

and ˜ M = { θ ′ n , ϕ ′ n } . The training loss is updated to

According to the idea of contrastive learning, the representations learned by the online network become more consistent with the knowledge captured by the target network. This global knowledge anchored contrastive learning suggests that the global model's knowledge is used as a positive example for the local model to train itself, thus making it more consistent with the target global network. Aligning local models with the global view representation helps create a comprehensive understanding of the overall data distribution. The process refines the knowledge acquired locally, ensuring that it contributes meaningfully to the overall federated learning process. Then, we use stochastic gradient descent to minimize L align to update the θ ′ n ,

Once the global model anchored alignment is finished, the server will send the online network of the local twin network θ ′ to the corresponding client to update the local model. The local online network θ n is replaced by the server-updated network θ ′ n . The target network is not replaced to retain more local knowledge and stabilize model training:

so that, the local model can benefit from the alignment process and align to the representation under global view, which can further use the local data to train the model.

5 Experiments

In this section, we evaluate the representations learned from our proposed global model FedMKD on CIFAR-10 and CIFAR-100 . We first describe the experimental setup and baselines, and then analyze the performance in comparison to other methods. Due to the space limitation, further hyperparameter analysis and communication cost analysis are represented in Appendix D.

5.1 Experimental setup

We use CIFAR-10 and CIFAR-100 [13] datasets to train all the models. Both of them contain 50,000 training images and 10,000 testing images. To construct the public dataset, we sample 4000 data samples from the training set, then divide the remaining data into N partitions to simulate N clients.

To assess the validity of the public dataset, we use two sampling methods to construct it. First, we use a random sampling method over all classes to generate public dataset 'IID'. And for the public dataset 'Partial', data is selected randomly from 40% classes in two datasets.

We utilize three settings to simulate heterogeneous data distributions among all the clients. For the IID setting, each client contains the same number of samples from all classes. For the class setting, each client only has 10 /N and 100 /N classes on two datasets and the classes between clients have no overlap. For the non-IID setting, data heterogeneity levels are described by the Dirichlet distribution Dir( β ) [10], where smaller β represents stronger heterogeneity levels, here we choose β = 0 . 5 .

Regarding the self-supervised learning framework design within each client, we use ResNet18 [9] and VGG9 [22] as the encoder network and Multi-Layer Perception (MLP) as the predictor. In order to construct the model heterogeneous setting, 2 clients train the Resnet18 encoder while 3 clients use the VGG9. And for the global representation model, Resnet18 is selected as the encoder in server.

5.2 Baselines and evaluation methods

Firstly, we select several federated knowledge distillation frameworks FedMD [14], FedDF [18], FedET [2], MOON [15], MOON-KL that use unlabeled public dataset for distillation. We then replaced the local model with a self-supervised model to evaluate the process of knowledge distillation in our method. And several federated self-supervised learning frameworks FedU [33], FedEMA [34], HeteroSSFL [20] are also chosen as baselines. To verify the client's knowledge can improve the global model, we also train the global model separately on the public dataset, denoted as Std. ResNet18 . Following FedEMA [30], we evaluate the performance of learned representations using linear and semi-supervised evaluation. Due to limited space, please refer to Appendix C for more details.

5.3 Performance Evaluation

Table 2 and 3 shows the linear evaluation results and semi-supervised evaluation results of FedMKD compared with all the baselines on CIFAR-10 and CIFAR-100 . We can gain the following observation.

On the whole, our FedMKD outperforms all baselines under different public dataset settings and different data heterogeneity level settings on both two datasets. Compared to the second-best results, FedMKD achieves significant improvement. On average, our model improves CIFAR-10 by 4.22% and CIFAR-100 by 5.31% under linear evaluation and gain 3.66% and 2.07% improvement on two dataset under semi-supervised evaluation.

The effectiveness of our multi-teacher adaptive knowledge integration distillation can be approved when compared with FedMD , FedDF , MOON-KL and MOON . Although these methods all designed

new federated knowledge distillation frameworks based on unlabeled public dataset, since the original local model is supervised, they prefer using the class information from logits to distill. When observing the result of FedET , we find that although it also designs a larger global model in server which improves the model a lot, the final result is not satisfied. This is also because it designs a distillation method based on knowledge of probability distribution over classes. Next, compared with the federated self-supervised method, our FedMKD also achieves better performance. FedU and FedEMA are not applicable in model heterogeneity setting, so we cannot evaluate their effectiveness. Hetero-SSFL gain the best performance among all baselines but is worse than ours. That's because it aims to train personalised client models. Only the alignment module cannot hold the inconsistent representation space perfectly. But our global model can directly generate representation from the global model, it avoids using the representation from inconsistent clients.

Apart from the client data heterogeneity, we consider the influence of the public dataset distribution. Here, we construct two public datasets, one is 'IID' to the whole data distribution and the other only has partial classes. In both two settings, our FedMKD also gets the best performance. The overall performance of 'IID' public dataset is better than 'Partial' setting. That's because the global model adapts the self-supervised learning on public dataset, and the diversity of the sample is important, which can help the model explore a wide range of features and patterns present in the data. And it's observed that when the public dataset is 'IID', the performance increases with the decrease of the data heterogeneity level for CIFAR-10 , but it doesn't apply to the CIFAR-100 . It's because there are too many classes in CIFAR-100 and the number of samples in each class is not efficient.

In order to evaluate the effectiveness of FedMKD , we use the dimensionality reduction algorithm t-sne to visualize the representation on the test dataset of CIFAR-10 from different encoders. As shown in Fig. 3 (b)(c)(d), the global models in FedMKD trained on both 'IID' and 'Partial' public datasets both achieve better clustering results than Standalone training and MOON . These results further verify our model can gain better generalized representations although the representation spaces of clients are inconsistent. There's also an averaged global model in MOON , but it cannot tackle the problem of inconsistent spaces well using the average method, so it only gains a poor clustering performance. Additionally, the performance on 'IID' public dataset is better than 'paritial' ones from the observation of cluster performance. This suggests that the number of classes seen by the global model also affects how well the global model can encode all classes in a unified space. Comparing the class distributions in Fig. 3 (c) and (d), we can find that although these two global models are trained on different public datasets, the final cluster layout is similar, which can further prove that our global model can encode all classes from clients even if it never sees some classes during the training.

5.4 Improvement of clients

In FedMKD , the global knowledge anchored alignment module is used to align the client model in the server which can further transfer the global knowledge to the client and incentivize the clients to participate in the federated learning. To evaluate the improvement of the clients, the local models which are standalone training locally are compared with our local models. As shown in Fig. 4, the client performance in our FedMKD framework is better than local standalone training, regardless of the archi-

Client 1

Client 2

Client 3

Client 4

Client 5

30

40

50

60

70

80

Testing Acc.(%)

standalone

FedMKD

tecture of the local model. Especially for clients with ResNet18, it improves 6.73% on average. It's concluded that clients benefit from federated training by contributing to global training.

5.5 Validation of inconsistent representation spaces

As we mentioned, the representation spaces between different clients are inconsistent because of data heterogeneity. To validate this opinion, we use Linear Discriminant Analysis (LDA) to reduce dimensionality in order to visualize the distribution of the representations. Here the data distribution of clients is Class. In Fig. 5 left, ' ◦ ' and ' × ' denote representations of Client A and Client B, respectively. And different colors denote different classes. We can observe that in Fig. 5 left the classes 'cat' and 'dog' almost overlap while they are from different clients, which verifies that the inconsistent representation spaces did exactly exist. And Fig. 5 right shows the visualization result of the represen-

ClientA

ClientB

Global

tation from the global model. It's clear that the classes 'cat' and 'dog' are embedded in different positions in global space, which proves that although the local representations are inconsistent, our global model can learn a good representation.

5.6 Ablation experiment

In order to investigate the effectiveness of different parts of FedMKD , we design these comparison experiments:

Standalone training : The global model is trained alone using the public dataset, without the knowledge from client models.

FedMKD KL : The global distillation function is replaced by the KL-divergence function to measure the similarity of the aggregated representation ¯ r and global representation r s .

FedMKD w/o adaptive : The adaptive knowledge integration module is removed, each local representation has the same weight to generate the aggregated representation.

FedMKD w/o alignment : The global knowledge anchored alignment module is removed from FedMKD .

Above experiment is conducted on the both CIFAR-10 and CIFAR-100 dataset, using the IID public dataset. The results of the ablation study experiment are shown in Table 4. When comparing FedMKD with FedMKD KL , significant performance drop can be observed when the knowledge distillation function is replaced by the KL-divergence function. Thus we can conclude that for self-supervised learning, we prefer performing knowledge distillation based on representation, but the KL-divergence cannot capture the distribution characteristics from them. Therefore, the appropriate distillation method is critically important in self-supervised learning due to the lack of labels. And a worse performance on both two datasets can be observed when we use the equal weight instead of the adaptive weight. Because the client models are heterogeneous, their representation capabilities are different and the representation spaces are also inconsistent, so intuitively average representations may reduce the information contained in the representation. Finally, when we remove the alignment module, the performance under each setting all decreases, which demonstrates that the alignment is not only beneficial to the local models, but also improves the whole training process.

5.7 Scalability of algorithm

In order to explore the scalability of our proposed algorithm FedMKD , we add the experiment that the number of clients is 5, 10, 30 on CIFAR-10 , and 40% clients use the VGG model and 60% use ResNet18. And we repartition the data for each client under Dir( β = 0 . 5 ) and set the public dataset distribution as 'IID'. The results are shown in Table 5. We can find that as the number of clients increasing, the performance decreases. The reason is that the total number of data is fixed, if the number of clients increases, the number of data in each client will decrease, which further affect the performance of the local model.

6 Conclusion

In this work, we focused on how to solve the deviated representation abilities and inconsistent representation spaces caused by the heterogeneous architectures and class skew in federated selfsupervised learning. We proposed a multi-teacher knowledge based federated self-supervised learning framework FedMKD to learn a global model. Firstly, the adaptive knowledge integration module could learn high-quality representation knowledge from heterogeneous models. And the combination of the self-supervised loss and the distillation loss enabled the global model to encode all classes from clients in a unified space. Then a global knowledge anchored alignment module improved the local representation models in server and fed it back to corresponding clients. The experiments conducted on two datasets demonstrated that our proposed FedMKD was state-of-the-art and outperformed existing methods.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heterogeneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv:2204.12703 , 2022.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent - A new approach to selfsupervised learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Wei Guo, Fuzhen Zhuang, Xiao Zhang, Yiqi Tong, and Jin Dong. A comprehensive survey of federated transfer learning: challenges, methods and applications. Frontiers Comput. Sci. , 18(6):186356, 2024.

Sungwon Han, Sungwon Park, Fangzhao Wu, Sundong Kim, Chuhan Wu, Xing Xie, and Meeyoung Cha. Fedx: Unsupervised federated learning with cross knowledge distillation. In European Conference on Computer Vision , pages 691-707. Springer, 2022.

Chaoyang He, Zhengyu Yang, Erum Mushtaq, Sunwoo Lee, Mahdi Soltanolkotabi, and Salman Avestimehr. Ssfl: Tackling label deficiency in federated learning via personalized self-supervision. arXiv preprint arXiv:2110.02470 , 2021.

Jingxuan He, Lechao Cheng, Chaowei Fang, Zunlei Feng, Tingting Mu, and Mingli Song. Progressive feature self-reinforcement for weakly supervised semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 2085-2093, 2024.

Jingxuan He, Lechao Cheng, Chaowei Fang, Dingwen Zhang, Zhangye Wang, and Wei Chen. Mitigating undisciplined over-smoothing in transformer for weakly supervised semantic segmentation. arXiv preprint arXiv:2305.03112 , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. CoRR , abs/1909.06335, 2019.

Yilun Jin, Yang Liu, Kai Chen, and Qiang Yang. Federated learning without full labels: A survey. arXiv preprint arXiv:2303.14453 , 2023.

Yilun Jin, Xiguang Wei, Yang Liu, and Qiang Yang. Towards utilizing unlabeled data in federated learning: A survey and prospective. arXiv preprint arXiv:2002.11545 , 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581 , 2019.

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10713-10722, 2021.

Simou Li, Yuxing Mao, Jian Li, Yihang Xu, Jinsen Li, Xueshuo Chen, Siyang Liu, and Xianping Zhao. Fedutn: federated self-supervised learning with updating target network. Applied Intelligence , 53(9):1087910892, 2023.

Xinting Liao, Weiming Liu, Chaochao Chen, Pengyang Zhou, Fengyuan Yu, Huabin Zhu, Binhui Yao, Tao Wang, Xiaolin Zheng, and Yanchao Tan. Rethinking the representation in federated unsupervised learning with non-iid data. ArXiv , abs/2403.16398, 2024.

Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems , 33:2351-2363, 2020.

Yi Liu, Song Guo, Jie Zhang, Qihua Zhou, Yingchun Wang, and Xiaohan Zhao. Feature correlation-guided knowledge transfer for federated self-supervised learning. arXiv preprint arXiv:2211.07364 , 2022.

Disha Makhija, Nhat Ho, and Joydeep Ghosh. Federated self-supervised learning for heterogeneous clients. arXiv preprint arXiv:2205.12493 , 2022.

Yasar Abbas Ur Rehman, Yan Gao, Pedro Porto Buarque de Gusmao, Mina Alibeigi, Jiajun Shen, and Nicholas D Lane. L-dawa: Layer-wise divergence aware weight aggregation in federated self-supervised visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16464-16473, 2023.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Bram van Berlo, Aaqib Saeed, and Tanir Ozcelebi. Towards federated unsupervised representation learning. In Proceedings of the third ACM international workshop on edge systems, analytics and networking , pages 31-36, 2020.

Kun Wang, Hao Liu, Lirong Jie, Zixu Li, Yupeng Hu, and Liqiang Nie. Explicit granularity and implicit scale correspondence learning for point-supervised video moment localization. In Proceedings of the 32nd ACM International Conference on Multimedia , MM'24, page 9214-9223, New York, NY, USA, 2024. Association for Computing Machinery.

Lirui Wang, Kaiqing Zhang, Yunzhu Li, Yonglong Tian, and Russ Tedrake. Does learning from decentralized non-iid unlabeled data benefit from self supervision? In The Eleventh International Conference on Learning Representations , 2022.

Yangyang Wang, Xiao Zhang, Mingyi Li, Tian Lan, Huashan Chen, Hui Xiong, Xiuzhen Cheng, and Dongxiao Yu. Theoretical convergence guaranteed resource-adaptive federated learning with mixed heterogeneity. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 2444-2455, New York, NY, USA, 2023. Association for Computing Machinery.

Yuzhu Wang, Lechao Cheng, Manni Duan, Yongheng Wang, Zunlei Feng, and Shu Kong. Improving knowledge distillation via regularizing feature norm and direction. arXiv preprint arXiv:2305.17007 , 2023.

Ziming Ye, Xiao Zhang, Xu Chen, Hui Xiong, and Dongxiao Yu. Adaptive clustering based personalized federated learning framework for next poi recommendation with location noise. IEEE Transactions on Knowledge and Data Engineering , 36(5):1843-1856, 2024.

Chen Zhang, Yu Xie, Tingbin Chen, Wenjie Mao, and Bin Yu. Prototype similarity distillation for communication-efficient federated unsupervised representation learning. IEEE Transactions on Knowledge and Data Engineering , pages 1-13, 2024.

Fengda Zhang, Kun Kuang, Long Chen, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Fei Wu, Yueting Zhuang, et al. Federated unsupervised representation learning. Frontiers of Information Technology & Electronic Engineering , 24(8):1181-1193, 2023.

Xiao Zhang, Ziming Ye, Jianfeng Lu, Fuzhen Zhuang, Yanwei Zheng, and Dongxiao Yu. Fine-grained preference-aware personalized federated poi recommendation with data sparsity. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '23, page 413-422, New York, NY, USA, 2023. Association for Computing Machinery.

Weiming Zhuang, Xin Gan, Yonggang Wen, and Shuai Zhang. Easyfl: A low-code federated learning platform for dummies. IEEE Internet of Things Journal , 2022.

Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4912-4921, 2021.

Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-aware federated self-supervised learning. In International Conference on Learning Representations , 2022.

Appendix

We provide more details about our work and experiments in the appendices:

Appendix A: the additional details of the proposed algorithm FedMKD.

Appendix B: the detailed proof of the convergence analysis of our proposed algorithm FedMKD.

Appendix C: the details of experimental settings including datasets and federated simulations, reproduction details, baselines, evaluation methods and ablation experiment setting.

Appendix D: additional experimental results including Appendix D.1, the overall performance under two evaluation methods; Appendix D.2, the communication and storage efficiency analysis; Appendix D.3 discusses the impact of hyperparameters.

Appendix E: the details of limitations and broader impacts of this work.

A Algorithm description

We outline the FedMKD algorithm in Algorithm 1. In round q , the clients and server perform the following updates:

Starting from the resource-adaptive model θ q,n, 0 , we update the local parameters for t ∈ [ T ]

After T times local updates, upload the local parameter θ q,n,T .

Server uses all local model to process multi-teacer adaptive knowledge integration distillation, update the global model for t ∈ [ T ′ ] ,

Server aligns all client models to the global model, for n ∈ N ,

B Convergence Analysis

In this section, we show the convergence analysis of our FedMKD . Firstly, we give some commonly used assumptions in federated learning:

Assumption 1 (Lipschitz Condition). Every function F ( · ) is with L -Lipschitz gradient: ∀ n ∈ [ N ] , θ, φ ∈ R d

Assumption 2 (Bounded variance). The stochastic gradients ∇ F n ( θ q,n,t ; ξ n,t ) is an unbiased estimator of the gradient, with the variance bounded by σ > 0 :

In order to analyze the convergence rate of our proposed FedMKD , we firstly state some preliminary lemmas as follows:

Lemma 1 (Jensen's inequality). For any convex function h and any variable x 1 , . . . , x n we have

Especially, when h ( x ) = ∥ x ∥ 2 , we can get

Algorithm 1: Algorithm of FedMKD

1 Initialize: total number of clients N ; Number of rounds T ; Local data in clients { D 1 , . . . , D N } ; Public data in server D pub ; Local online network { θ 1 , . . . , θ N } , target network { ϕ 1 , . . . , ϕ N } ; Server online network θ s , target network ϕ s ; Learning rate η ; Adapter network Attn ( · ) . for q = 1 to Q do 2 LocalUpdate: for n = 1 to N (all clients in parallel) do 3 if q = 1 then 4 Random initialize θ q,n, 0 . 5 else 6 { θ q,n, 0 , ϕ q,n, 0 } ← { θ ′ q -1 ,n , ϕ q -1 ,n,T } . 7 end 8 for epoch t = 1 to T do 9 l q,n,t = L self ( θ q,n,t -1 ; ϕ q,n,t -1 ; D n ) . 10 θ q,n,t ← θ q,n,t -1 -η ∇ l q,n,t . 11 ϕ q,n,t ← αϕ q,n,t -1 +(1 -α ) θ q,n,t . 12 end 13 Upload θ q,n,t . 14 end 15 ServerExecution: 16 # Multi-teacher adaptive knowledge integration distillation. 17 for epoch t = 1 to T ′ do 18 for batch b ∈ D pub do 19 for n = 1 to N do 20 r q,n = g ( θ q,n,t ( b )) . 21 end 22 ˜ R = [ r q, 1 , · · · , r q,N ] 23 r s = g ( θ q,s ( b )) . 24 ¯ r = Attn ( r s , ˜ R ) 25 l q,s = L self ( θ q,s ; ϕ q,s ; b ) + γ L distill ( θ q,s ; r s ; ¯ r ) . 26 θ q,s ← θ q,s -η ∇ l q,s . 27 ϕ q,s ← αϕ q -1 ,s +(1 -α ) θ q,s . 28 end 29 end 30 # Alignment client models in server. 31 for n = 1 to N do 32 { θ ′ q,n , ϕ ′ q,n } ← { θ q,n,T , θ q,s } 33 l ′ q,n = L align ( θ ′ q,n ; ϕ ′ q,n ; D pub ) . 34 θ ′ q,n ← θ ′ q,n -η ∇ l ′ q,n . 35 end 36 Send θ ′ q,n to client n . 37 end

Lemma 2 For random variable x 1 , . . . , x n we have

Lemma 3 For independent random variables x 1 , . . . , x n whose mean is 0, we have

Based on the above assumptions, we present the theoretical results for the non-convex problem.

Lemma 4 (Deviation bound of the optimization function) In each communication round, the function value in server reduce after T ′ epochs and is bounded as:

Lemma 4 indicates the deviation bound of the optimization function of the server in each global round.

Theorem 1 (Non-convex divergence for FedMKD) Let Assumption 1 to 3 hold and ∆ = F ( θ 0 ) -F ( θ T ′ ) , given any δ > 0 , suppose that the learning rates satisfy 0 ≤ η ≤ 2 /L , after

the convergence can be guaranteed.

Proof. Let's start the proof from L -Lipschitz condition:

where (a) is from Assumption 1. Taking expectation of both sides, we obtain

where (b) follows from Assumption 2. Let's set the learning step at the start of training to T ′ ,

Proof of Theorem 1. According to Lemma 4,

Therefore, which is equal to

communication round, we have

C Experiment supplements

C.1 Datasets and federated simulations

We use CIFAR-10 and CIFAR-100 [13] datasets to train all the models. Both of them contain 50,000 training images and 10,000 testing images. To construct the public dataset, we sample 4000 data

airplane

automobile

bird

cat

deer

dog

frog

horse

ship

truck

Classes

1

2

3

4

5

Client ID

Client 1

Client 2

Client 3

Client 4

Client 5

(a) Class

airplane

automobile

bird

cat

deer

dog

frog

horse

ship

truck

Classes

1

2

3

4

5

Client ID

(b) Dir(

β

= 0

.

1

)

airplane

automobile

bird

cat

deer

dog

frog

horse

ship

truck

Classes

1

2

3

4

5

Client ID

(c) Dir(

β

= 0

.

5

)

airplane

automobile

bird

cat

deer

dog

frog

horse

ship

truck

Classes

1

2

3

4

5

Client ID

(d) IID

samples from the training set, then divide the remaining data into N partitions to simulate N clients. To assess the validity of the public dataset, we use two sampling methods to construct it. First, we use a random sampling method over all classes to generate public dataset 'IID'. And for the public dataset 'Partial', data is selected randomly from 40% classes in two datasets.

We utilise three settings to simulate heterogeneous data distributions among all the clients. For the IID setting, each client contains the same number of samples from all classes. For the class setting, each client only has 10 /N and 100 /N classes on two datasets and the classes between clients have no overlap. For the Non-IID setting, data heterogeneity levels are described by the Dirichlet distribution Dir( β ) [10], where smaller β represents stronger heterogeneity levels. A value of β = 0 . 1 is chosen to simulate a high degree of heterogeneity, and β = 0 . 5 for a lower level. Fig. 6 shows the data distribution among clients on CIFAR-10 dataset. The x-axis represents 10 classes and the y-axis is the total of 5 clients. The size of each circle denotes the number of samples for the specific class in the respective client.

C.2 Reproduction details

Regarding the self-supervised learning framework design within each client, we use ResNet18 [9] and VGG9 [22] as the encoder network and Multi-Layer Perception (MLP) as the predictor. To compare with other baselines, we set the number of clients N = 5 , and conduct the experiments for R = 100 rounds. In order to construct the model heterogeneous setting, 2 clients train the Resnet18 encoder while 3 clients use the VGG9. And for the global representation model, Resnet18 is selected as the encoder in server. The hyper-parameter γ in the loss of global model is set to 0.9. During the training process, each client trains locally for T = 5 epochs while the server also distills for T ′ = 5 epochs. Finally, we set the target decay rate α = 0 . 99 , with a batch size of B = 128 , and utilize SGD for optimization with a learning rate of η = 0 . 032 . We implement all the methods in Python using EasyFL[32] based on PyTorch.

C.3 Baselines

Firstly, we select several federated knowledge distillation frameworks that use unlabeled public dataset for distillation. We then replaced the local model with a self-supervised model to evaluate the process of knowledge distillation in our method.

FedMD [14]: Each client trains convergence on the public dataset and then on local data. In each round, clients upload the embedding of the public dataset to the server. The server averages the embeddings and sends averaged embedding to clients. Clients first use average embedding to update local models on public dataset, then train on local data for a few epochs. There's no global model, the local models are evaluated to make comparison.

FedDF [18]: Clients in FedDF train locally and upload the model parameter to server. The server uses all client models to compute the embedding of the public dataset and uses the average embedding to train the global model. But it has no global model, so we only test the performance of local models.

FedET [2]: There is a global large model and several alternative small models on the server. Each client selects the appropriate small model for training locally and then uploads the parameter to the server. The server uses all the small models to compute representations of

the public dataset and updates the global model with the averaged representation. Then the server utilizes the representation from the global model to update all the small models.

MOON [15]: Moon introduces a model-contrastive learning approach, treating the representation from the global model as positive knowledge to update the local models.

MOON-KL : Instead of using the NT-Xent loss in MOON, MOON-KL utilises the KLdivergence function to measure the similarity between global representation and local representation. To make comparisons, both MOON and MOON-KL distribute the global model to each client, and then the client uses the local model to update the global model. The server aggregates the updated global models as the final global model.

And several federated self-supervised learning frameworks are also chosen as baselines.

FedU [33]: Clients in FedU upload the local online encoder and predictor for server aggregation. Then they update the local online encoder using the aggregate ones, and dynamically update the predictor based on the divergence. But it cannot applied to the model heterogeneity setting.

FedEMA [34]: FedEMA uses the aggregated local online network as a global model and measures the divergence between global and local networks. The divergence is then applied to the local exponential moving average update. It's also not applicable when models are heterogeneous.

Hetero-SSFL [20]: Each client trains locally and uploads the kernel matrix over the public dataset. The server aggregates them and sends to clients. Then client uses linear-CKA normalization term to align to the averaged kernel matrix. There's no global model, so we just test local models' performance.

To verify that the client's knowledge can improve the global model, we also train the global model separately on the public dataset, denoted as Std. ResNet18 .

C.4 Evaluation methods

Following [33] and [30], we evaluate the performance of learned representations using linear and semi-supervised evaluation. For linear evaluation, all the models are trained without any supervised labels. Subsequently, the model encoder is then frozen and the representations are utilized for training a new classifier over 200 epochs. For semi-supervised evaluation, we consider the scenario that only a small subset of the data has a label, here only 1% data are labeled. Then, different from freezing the encoder, we fine-tune the whole model with a new classifier using the labeled data for 100 epochs. For FedMD , FedDF and Hetero-SSFL , only local models can be evaluated. So we use the averaged representation from local models to do the linear evaluation. But in semi-supervised evaluation, we need to fine-tune the whole model, so we evaluate each client's performance alone and use the average result as the final result.

D Additional Experiments

D.1 Numerical Results

Table 6, Table 7 and Table 8, Table 9 shows the linear evaluation results and semi-supervised evaluation results of FedMKD compared with all the baselines on CIFAR-10 and CIFAR-100 . Comparing to the result in main text, we extra add the result on Dir( β = 0 . 1 ) to prove the efficiency of our algorithm totally.

D.2 Communication and storage efficiency

Considering the experimental setting, the communication and storage capabilities are limited, so we analyze the communication and computation efficiency of FedMKD compared to Hetero-SSFL and all the experiments maintain the original settings. Firstly, we calculate that the memory of ResNet18 and VGG9 is 42.63MB and 16.38MB separately. We use S ( θ ) to denote the storage cost of model θ .

About the communication cost, the client in FedMKD only needs to upload the online encoder to the server, and download the updated encoder, so the cost per communication is equal to the ResNet18 or

VGG9. But for Hetero-SSFL , each client needs to upload the kernel metric K ∈ R L × L , L = | D pub | is the size of the public dataset. In order to make comparison, we fix the communication rounds. Compared to the Hetero-SSFL , when S ( θ n ) ∗ 2 ≤ | D pub | 2 , our FedMKD is better in communication cost. And compared to the MOON , for larger server model, S ( θ n ) ∗ 2 ≤ S ( θ s ) ∗ 2 , MOON will cost more when the server model is larger.

About local storage for each client, we compute the summary of the model and data. For FedMKD and Hetero-SSFL , each client runs the siamese network, so the model storage is double of the model memory, that is 2 ∗ S ( θ n ) . But for the MOON , each client processes the local contrastive learning and the model contrastive learning, so they must store two local models and one global model, that is 2 ∗ S ( θ n ) + S ( θ s ) . Here, we neglect the storage of the predictor which is usually a 2-layer MLP. We use S ( | D | ) to denote the storage of the dataset. And each client in FedMKD only needs to store the

1000

2000

3000

4000

5000

Public Dataset Size

40

50

60

70

Testing Acc.(%)

IID

Class

0

50

100

150

200

Rounds

40

50

60

70

80

Testing Acc.(%)

IID

CLASS

1

3

5

7

10

Server Epochs

40

50

60

70

Testing Acc.(%)

IID

Class

1000

2000

3000

4000

5000

Public Dataset Size

20

30

40

Testing Acc.(%)

IID

Class

0

50

100

150

200

Rounds

20

30

40

50

Testing Acc.(%)

IID

CLASS

1

3

5

7

10

Server Epochs

20

30

40

Testing Acc.(%)

IID

Class

(d) The performance of FedMKD with different public dataset size on Cifar-100. (e) The performance of FedMKD with different global rounds on Cifar-100. (f) The performance of FedMKD with different server epochs on Cifar-100.

local training data without the public data, that is S ( | D n | ) and for Hetero-SSFL client needs to store both two S ( | D n | ) + S ( | D pub | ) . MOON doesn't use the public dataset, so the client only needs to store the local data S ( | D n | ) also. Totally, we can get

D.3 Hyperparameter analysis

To explore the influence of different hyperparameters, we conduct experiments on several key parameters of FedMKD .

Impact of public dataset size. As mentioned in [20], the size and construct method of the public dataset are both critically important, especially since our global model needs training based on it. About the latter one we've given an analysis in the last section, so we explore the influence of the size on the model performance. Considering the construct method of the public dataset, we choose part of the dataset as a public dataset and set it aside to be used. The performance change with respect to its size is shown in Fig. 7 (a)(d). We can observe that in both two datasets, when the public dataset is small, the performance of FedMKD is quite worse, and as the size of the public dataset increases, the performance gets better. But when the size is larger than 4000, the gain of performance is small, and because of the design of the model, the time cost of the server computation is larger, so we choose 4000 as our experiment setting.

Impact of the number of global rounds. In order to investigate the impact of the training rounds to FedMKD , we fix the other hyperparameters and train the model for 200 rounds. The results showed in Fig. 7 (b)(e) demonstrate that while the total training rounds increase, the performance of the FedMKD gets better. For both two kinds clients data distribution in CIFAR-10 , the global model converges at nearly 150 rounds. For CIFAR-100 , the global model converges at 200 rounds. There's no denying that the increase of the global rounds will result in better performance, especially when the global round is small, the improvement of the performance is significant.

Impact of the number of server epochs. Since our global model is trained on the public dataset, through standalone training and knowledge distillation. The lower server epoch means the global model may under-fitting not only the distribution of the public dataset but also the client knowledge. On the other hand, large server epoch will lead large computation cost, and over-fit the distribution of the public dataset, which is harmful to the generalization of the model, so the epoch in server is important. We choose 5 different settings T ′ = { 1 , 3 , 5 , 7 , 10 } to validate the influence of the server epochs. Based on the experiment results shown in Fig. 7(c)(f), we can see that the performance of the global model improves when the server epoch increases with the distribution of the public is IID. But the change of improvement gets smaller when the server epoch is larger than 5.

E Additional discussion

In this section, we discuss the limitations and broader impacts of the work.

Limitations. Although we provide detailed explanations of the proposed algorithm and extensive experiments analysis, the theoretical proof of FedMKD is not rigorous enough. The characteristic of multi-teacher distillation of the global model has not been sufficiently theoretically justified and we only combined it with the self-training loss as one whole loss to analyze.

Broader Impacts. FedMKD offers significant societal and technological benefits, which is crucial in kinds of domains like healthcare and finance. It promotes inclusivity by leveraging diverse data from various sources, thereby reducing biases and improving model generalization. Technologically, FedMKD lowers the dependency on labeled data, making federated self-supervised learning more efficient and scalable, and drives innovation in heterogeneous resource-limited devices. By carefully navigating these challenges: deviated representation abilities and inconsistent representation spaces, FedMKD can lead to responsible and equitable advancements in distributed AI technology.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction sections of our paper clearly articulate the main claims and contributions of our research. These include explicit statements of the proposed framework innovations, theoretical advancements, and empirical findings is summarized in lines 62-73. Furthermore, the theoretical arguments in Sec.B and the experimental evidence in Sec.5 of the main body of the paper fully support the claims made in this paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In this work, we are the first to propose a multi-teacher knowledge distillation framework, namely FedMKD, to learn global representations with whole class knowledge from heterogeneous clients even under extreme class skew. However, the paper still exists some limitations about the privacy preservation. We discuss the limitations of this work in Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In this work, we provide the full set of theoretical assumptions and the complete process process of the proposed multi-teacher knowledge distillation framework FedMKD in the Appendix.B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We propose a multi-teacher knowledge distillation framework FedMKD and describe this framework in detail in Sec.4. Furthermore, this paper disclose all the information needed to reproduce the main experimental results of it in Sec.5 and Appendix. C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: In this work, we use the public dataset CIFAR-10 and CIFAR-100 as the datasets. And the code of our work is available at https://github.com/limee-sdu/ FedMKD , as stated in Sec.5.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the training and test details in Sec.5 and Appendix. C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the error bars of the experiments in Sec. 5 and Appendix. D.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In this work, we use the NVIDIA GeForce RTX 3090 cards with 24GB memory as the server and the clients.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics. And we ensure that our work does not involve any inference of personal information and that it will only be used for academic purposes.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts of this work in the last paragraph of Appendix.E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In this work, we introduce the public datasets CIFAR-10 and CIFAR-100 and the ResNet18 [9] and VGG9 [22] as the backbones in the Sec.5.1. And we properly cite the related work in the main body and the appendix of this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should citep the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach for enhancing global representation models in resource-adaptive federated self-supervised learning through a multi-teacher knowledge distillation framework, named FedMKD. The proposed method addresses the challenges posed by heterogeneous architectures and extreme class skew, demonstrating significant improvements in representation abilities across diverse clients. The authors provide detailed experimental results to support their claims.\n\nSTRENGTHS:\n1. The paper tackles a critical issue in federated self-supervised learning, offering a unique and effective solution.\n2. FedMKD effectively leverages multi-teacher knowledge distillation to integrate knowledge from heterogeneous clients, even under extreme class skew.\n3. The adaptive knowledge integration mechanism enhances the representation abilities of heterogeneous models.\n4. The experimental section is thorough, demonstrating the efficacy of the proposed method on multiple datasets.\n5. The combination of self-supervised loss and distillation loss, along with the global knowledge anchored alignment module, significantly improves local and global representation abilities.\n\nWEAKNESSES:\nThe Related Work section could be expanded to include a more thorough discussion of existing approaches in both transfer learning and attention mechanisms.\nThe paper may have a limited audience due to its specialized focus on image classification tasks. It could benefit from providing more context for readers unfamiliar with the specific datasets and techniques discussed.\nThe contributions of the work could be more explicitly stated in the Introduction to enhance clarity for readers.\n\nQUESTIONS:\nIt would be beneficial for the authors to include a complexity analysis of the FedMKD algorithm to provide insights into its computational efficiency. If a complexity analysis is not feasible, the authors should provide a rationale for its omission.\n\nLIMITATIONS:\nThe limitations of the FedMKD framework are well-addressed in the paper. These constraints appear to be inherent to the design choices made in the method and are necessary trade-offs for its functionality.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a multi-teacher knowledge transfer framework, FedMKD, based on two challenges in resource-adaptive federated self-supervised learning: deviated representations abilities and inconsistent representations. This framework uses an adaptive knowledge integration mechanism and a weighted combination of different loss functions to ensure that global representations with class-wide knowledge can be learned from heterogeneous clients even in the presence of extreme class skew. Extensive experiments on two datasets demonstrate the effectiveness of FedMKD, which outperforms the state-of-the-art baseline by an average of 4.78% under linear evaluation.\n\nSTRENGTHS:\n1.\tThe scenarios presented and the challenges solved in this work are of importance in reality and the proposed multi-teacher knowledge distillation based federated self-supervised learning framework is able to solve the challenges effectively.\n\n2.\tBy innovatively combining self-supervised loss and distillation loss, FedMKD can encode skew classes into a unified space, which is not covered by related work.\n\n3.\tThe paper is easy to follow, including the main paper and appendices. The introduction provides a clear motivation and a brief overview of the proposed framework, and the other sections provide detailed descriptions of the method. \n\n4.\tThe experiments are well-organized and successful, making this work more convincing.\n\nWEAKNESSES:\n1.\tThe mechanism design seems to be confusing. As shown in the FedMKD, multi-instructor knowledge distillation is allowing global models to learn local models, while global knowledge anchored alignment is allowing local models to learn global models, knowledge transfer happens alternately.\n\n2.\tFedMKD aims to learn global class representations according to a global model. But an additional global-anchored mechanism improves the local performance. What’s the difference between personalized federated learning and the proposed FedMKD?\n\n3.\tIt’s suggested that the authors could add the instructions about the whole model training algorithm in Appendix A, which can help reviewers understand the whole process.\n\n4.\tThe presentation needs to be improved, especially for some equations. The self-supervised loss value in Eq. (10) was not discussed earlier. If it is equal to the loss value in Eq.(3), but with a different notation.\n\nQUESTIONS:\n1.\tAs shown in Weakness 1, it seems to that multi-instructor knowledge distillation is allowing global models to learn local models, while global knowledge anchored alignment is allowing local models to learn global models, so does the two mechanisms result in duplication of knowledge transfer?\n\n2.\tIn the problem statement, the main objective is the global loss. Since recent studies on cross-device scenarios also consider personalized FL (PFL), how about the personalization? Can your method be extended to PFL?\n\nLIMITATIONS:\nThe authors pointed out the limitations of the paper that is the theoretical proof of FedMKD is not rigorous enough, the characteristic of multi-teacher distillation of the global model has not been sufficiently theoretically justified.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multi-teacher knowledge distillation framework named FedMKD for resource-adaptive federated self-supervised learning (Fed-SSL). The method aims to address the challenges of global representation learning in Fed-SSL caused by heterogeneous architectures and imbalanced class distributions. Experimental results demonstrate that FedMKD is significantly effective in addressing heterogeneity and class distribution imbalance issues in federated self-supervised learning.\n\nSTRENGTHS:\nInnovative Framework Design: This paper proposes a new multi-teacher knowledge distillation framework, FedMKD, which combines self-supervised learning and knowledge distillation to effectively address the challenges of heterogeneity and data imbalance in federated learning.\n\nAdaptive Knowledge Integration Mechanism: Through an adaptive weighting strategy, it integrates high-quality representations from heterogeneous models, significantly enhancing the representation capabilities of the global model.\n\nGlobal Knowledge Anchored Alignment: The design of a global knowledge anchored alignment module ensures that local models' representation spaces are closer to the global representation space, thus improving the performance of local models.\n\nWEAKNESSES:\n1. This paper combines federated self-supervised learning and resource-aware federated learning. However, in introduction and related works, the authors lack a discussion about some works about resource-aware federated learning with heterogeneous clients.\n.\n2. The authors assume to build a global dataset to perform knowledge distillation and anchored alignment. However, random sampling makes the global dataset and client dataset have the similar data distribution, which violates the original intention of data privacy in FL. Please provide some explanation for this.\n\n3. CIFAR-10 and CIFAR-100 are both image datasets with relatively small sizes and low image resolution, which cannot fully reflect the complexity of real-world applications. It is recommended to add experiments on large-scale image datasets such as Tiny-ImageNet and ImageNet-100.\n\n4. The framework of dual encoders increases the training load in clients. It is recommended to evaluate the computation time and resources required and provide analysis for local model training.\n\n5. To evaluate the robustness and effectiveness of this framework, it is suggested to explore experiment settings with more clients, such as [1].\n\n[1] ScaleFL: Resource-Adaptive Federated Learning with Heterogeneous Clients\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe characteristic of multi-teacher distillation of the global model has not been sufficiently theoretically justified and the authors only combined it with the self-training loss as one whole loss to analyze.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper studies the problem of federated self-supervised learning. A multi-teacher knowledge distillation framework is proposed to address the two challenges: deviated representation abilities and inconsistent representation spaces. Specifically, the adaptive knowledge integration mechanism is designed to learn better representations from all heterogeneous models, and a global knowledge-anchored alignment module is used to make the local representation spaces close to the global spaces. Experiments on two datasets demonstrate the effectiveness of the proposed method.

STRENGTHS:
* This paper is well-written and easy to follow.
* The analysis of the challenges, i.e., deviated representation abilities and inconsistent representation spaces, is clear.
* Introducing an attention module to integrate the knowledge is technique sound.

WEAKNESSES:
* Knowledge distillation is widely used in federated learning and federated self-supervised learning. The contribution of the proposed method (i.e., multi-teacher knowledge distillation framework) is incremental.
* This paper uses small models such as resnet18 and vgg9.
* Only CIFAR datasets are used to evaluate the proposed method. Experiments on ImageNet-1k are important to evaluate the effectiveness.

QUESTIONS:
Weakness.

LIMITATIONS:
The authors adequately addressed the limitations in the appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper proposes a framework called FedMKD which is a multi-teacher knowledge distillation framework for federated learning. \n- To allow for different clients to have different resources, a resource adaptive approach is designed. The approach handles class skew and different architectures. \n- The knowledge distillation framework allows for learning representations from the different client representations. An SSL loss is used in addition on the common representation space. \n- Different from prior works, a global representation model is learnt despite differences in clients. \n- Approach is shown to outperform prior works on CIFAR-10 and 100 datasets.\n\nSTRENGTHS:\n- The paper tackles an important problem in federated learning: heterogeneity in clients both from the perspective of resources and data. \n- A simple and effective solution is proposed to tackle to learn a common representation space by using techniques in self-supervised learning knowledge distillation. \n- The approach is shown to outperform prior work on CIFAR-10 and 100 benchmarks\n- The paper is easy to follow. Ablations and experiments seem thorough (see weaknesses).\n- While I have not tried it, authors have provided an implementation of their approach.\n\nWEAKNESSES:\nWeaknesses/Concerns & Questions:\n- How does the approach compare to the traditional FL settings when aggregation like L107 is done (assuming all client architectures are the same) ? \n- Did the authors experiment with applying the knowledge distillation to individual projected features instead of the aggregate ? \n- L167: Are the negatives also after the aggregation step ? or from individual projection layers ?   \n- Additional larger-scale datasets will be helpful to understand whether the approach will scale. \n\nPresentation and writing:\n- Table 1 needs more details to be helpful. A more detailed caption should help. More subjective columns like \"inconsistent representation space\", \"deviated representation ability\" are not apt for a table like this.\n\nQUESTIONS:\nPlease refer to the weaknesses section.\n\nLIMITATIONS:\nThe paper has adequately discussed limitations and potential negative impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}