SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1 shows the experimental results with 100 clients. The clients hold the SVHN data, and we alternatively change the (labeled and unlabeled) public data and report the results for both IID and Non-IID settings. We can observe the same results as those on the 12 clients (Figure 1 in the main manuscript). All approaches can achieve the best performance when using SVHN as the public data. Although using other public data, the performance of all approaches drops, the performance change of the proposed pFedHR is lowest. Besides, even using other public data, pFedHR can achieve comparable or better performance to baselines. For example, in Figure 1(b), when pFedHR uses the MNIST as the public data, its performance is comparable to FedMD and better than FedGH using SVHN as the public data. These results confirm that the proposed pFedHR can handle the public data sensitivity issue.
2.3.2 Experiment Results of Public Data Sensitivity on CIFAR-10 with 12 Clients
We also validate the public data sensitivity on CIFAR-10, where clients hold data from CIFAR-10, and we alternatively change the public data. The results are shown in Figures 2 and 3. We have the same observations as we discussed before. Using public data with different distributions makes the performance drop, but the proposed pFedHR has the smallest drop ratio.
(a) IID with labeled public dataset
(c) IID with unlabeled public dataset
(b) Non-IID with labeled public dataset
(d) Non-IID with unlabeled public dataset
(a) IID with labeled public dataset
Figure 1: Results with 100 clients.
(b) Non-IID with labeled public dataset
Figure 2: Results with CIFAR-10 private dataset and labeled public datasets.
(a) IID with unlabeled public dataset
Figure 3: Results with CIFAR-10 private dataset and unlabeled public datasets.
(b) Non-HD with unlabeled public dataset
3 Limitation
All the experimental results demonstrate the effectiveness of the proposed pFedHR. However, pFedHR still have the following limitations:
1. In our experiments, we use four different CNN-based models and randomly send them to clients. The reason for using these simple models is that they can save computational resources. In our model design, we will calculate the CKA score between any pair of layers.
In other words, if the local model is very deep (with multiple layers), the computational complexity will be very high.
2. In real-world applications, the client models may be more complicated. To deploy the proposed pFedHR in a real environment, we need to slightly modify the layer-wise decomposition and use block-wise decomposition as [6] and redefine the rules used for generating candidates accordingly.
3. In our experiments, we only test the designed model on the image datasets. In real-world applications, multiple types of data may be stored in each client. How to handle other types of data with heterogeneous model reassembly is one of our major future works.
4 Broader Impacts
The proposed pFedHR addresses the practical challenge of model heterogeneity in federated learning. By introducing heterogeneous model reassembly and personalized federated learning, this research contributes to the advancement of federated learning techniques, potentially improving the efficiency and effectiveness of collaborative machine learning in distributed systems.
The pFedHR framework automatically generates informative and diverse personalized candidates with minimal human intervention. This has the potential to reduce the burden on human experts and practitioners, making the process of model personalization more efficient and scalable. It opens up possibilities for deploying federated learning systems in real-world scenarios where manual customization is impractical or time-consuming.
The proposed heterogeneous model reassembly technique in pFedHR tries to mitigate the adverse impact caused by using public data with different distributions from the client data. This can be beneficial in scenarios where privacy concerns limit the availability of extensive client data, such as healthcare, enabling the utilization of publicly available data while maintaining a certain level of model performance. It promotes the ethical and responsible use of data and encourages collaboration between organizations and researchers in a privacy-preserving manner.
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[2] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[4] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[5] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[6] Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. Advances in neural information processing systems, 35:25739-25753, 2022.