Towards Addressing Label Skews in One-Shot Federated Learning
Authors: Yiqun Diao
Qinbin Li
Bingsheng He
Venue: ICLR 2023 poster (2023)
Keywords: federated learning

Abstract: Federated learning (FL) has been a popular research area, where multiple clients collaboratively train a model without sharing their local raw data. Among existing FL solutions, one-shot FL is a promising and challenging direction, where the clients conduct FL training with a single communication round. However, while label skew is a common real-world scenario where some clients may have few or no data of some classes, existing one-shot FL approaches that conduct voting on the local models are not able to produce effective global models. Due to the limited number of classes in each party, the local models misclassify the data from unseen classes into seen classes, which leads to very ineffective global models from voting. To address the label skew issue in one-shot FL, we propose a novel approach named FedOV which generates diverse outliers and introduces them as an additional unknown class in local training to improve the voting performance. Specifically, based on open-set recognition, we propose novel outlier generation approaches by corrupting the original features and further develop adversarial learning to enhance the outliers. Our extensive experiments show that FedOV can significantly improve the test accuracy compared to state-of-the-art approaches in various label skew settings.

Published as a conference paper at ICLR 2023
TOWARDS ADDRESSING LABEL SKEWS IN ONE-SHOT FEDERATED LEARNING
Yiqun Diao, Qinbin Li & Bingsheng He National University of Singapore Jun, qinniiinnniiiii ginbin, hebs @comp. nus edd..gg sg
ABSTRACT
Federated learning (FL) has been a popular research area, where multiple clients collaboratively train a model without sharing their local raw data. Among existing FL solutions, one-shot FL is a promising and challenging direction, where the clients conduct FL training with single communication round. However, while label skew is a common real-world scenario where some clients may have few or no data some classes, existing one-shot Fl. approaches that conduct voting on the local models are not able to produce effective global models. Due to the limited number of classes each party, the local models misclassify the data from unseen classes into seen classes, which leads to very ineffective global models from voting. To address the label skew issue in one-shot FL, we propose a novel approach named FedOV which generates diverse outliers and introduces them an additional unknown class in local training to improve the voting performance. Specifically, based open-set recognition, we propose novel outlier generation approaches by corrupting the original features and further develop adversarial learning to enhance the outliers. Our extensive experiments show that FedOV can significantly improve the test accuracy compared to state-of-the-ar approaches in various label skew settings. Code is available https://gihuu..com//ti hot.i//oooooottta.oooooonooeeeee
INTRODUCTION
Federated learning (FL) (McMahan et al. 2016 Kairouz et al. 2019 Yang et al. 2019: Li et al. 2019)
allows multiple clients to collectively train a machine learning model while preserving individual data privacy. Most FL algorithms like FedAvg (McMahan et al. (016) require many communication rounds to train an effective global model, which cause massive communication overhead, increasing privacy concerns, and fault tolerance requirements among rounds. One-shot FL (Guha et al., 2019 Li et al. 2021c), i.e., FL with only a single communication round, has been a promising and challenging direction to address the above issues.
On the other hand, label skews are common in real-world applications, where different clients have different label distributions (e.g., hospitals on different regions can face different diseases). As parties may have few or no data of some classes, this leads even more challenges in one-shot FL. In this paper, we study whether and how we can improve the effectiveness of one-shot FL algorithm for applications with label skews.
2021c) cc is to conduct local training and collect the local models an ensemble. The ensemble is either directly used as a final model for predictions (Guha et al., 2019) or distilled However, those voting based approaches fail to produce high quality federated learning models.
simple and common one-shot FL strategy (Guaa al. 2019 Li
single model (Li et al. 2021c) with voting.
Under the label skew setting, since each client has only portion of classes, the local model predicts everything to its seen classes and the final voting results are poor. For example, in an extreme case where each client only has one label (e.g., face recognition), all clients predict the input as its own label and the voting result is meaningless. To address this issue, propose open-set voting for one- shot FL that introduces an "unknown" " class in the voting inspired by studies on open-set recognition (OSR) (Neal et al.) [2018] Zhou et al. (2021). l.. In local training, the clients train local open-set classifiers that are expected to predict its known classes correctly, while predicting "unknown" it is unsure about the input data. Then, during inference, the server conducts voting on the received open-set

---

Published as a conference paper at ICLR 2023
classifiers with the "unknown" option. In this way, open-set voting can filter the local models that do not have the knowledge of an imput to improve the voting accuracy.
Although there existing OSR studies in the centralized setting. it is challenging to apply them in the label-skewed federated setting to achieve good open-set classifiers during local training due to the limited number of local classes. For example, the state-of-the-ar OSR algorithm PROSER (Zhou et al. 202T) considers linear interpolation between different seen classes as outliers. The outliers and the original data are used train the model, where the outliers are considered as the extra class "unknown". When the number of classes is very small in a client, PROSER generates very limited types of outliers that are not sufficient for training. Moreover, the classifier has loose boundary as the distance between the training data and the generated outliers may be large. To improve the quality of open-set classifiers, we propose a new open-set approach named FedOV with two novel techniques including data destruction and adversarial outlier enhancement to generate diverse and tight outliers. In data destruction, as opposed to data augmentation, we generate rich outliers by transforming the key features from the original image using boosted data operations such random erasing and random resized crop. In adversarial outlier enhancement, we further optimize the outliers to be close to the training data in adversarial way such that the local model cannot distinguish it.
Experiments show that our proposed FedOV (Federated learning by Open-set Voting) significantly improves the accuracy compared with existing one-shot FL approaches under various label skew cases. To reduce the model size of FedOV ensemble, we also combine knowledge distillation to FedOV like previous approaches (Lin al. 2020 Li et al. (021c). Distilled FedOV also outperform state-of-the-ae one-shot FL algorithms with model distillation.
Our main contributions are summarized as follow:
To the best of our knowledge, are the first to propose open-set voting in FL by introducing the "unknown" class, which significantly improves the accuracy compared to traditional close-set voting in FL.
We propose two novel techniques, including data destruction and adversarial outlier en- hancement, to generate diverse "unknown" outliers without requirement on the number of classes of the training data.
We conduct extensive experiments to show the effectiveness of our open-set voting algo- rithm. Our algorithm consistently outperforms baselines with a significant improvement on accuracy on comprehensive label skew settings, including #C (each client has only one class) where many FL algorithms fail.
2 BACKGROUND AND RELATED WORK
2.1 NON-IID DATA IN FL
Non-III) data is prevalent among real-world applications. For example, different areas have different types of discases. Another example is that there are different species in different places. For classification tasks, suppose client i has dataset {æt, }. where I; are features and are labels. In the label skew setting, p(y.) differs across clients.
Label skew is difficult because the local optima of different clients can be much far away (Li et al. 2021b). There have been many studies (Li et al. 22200 [021d] |aan et al.) 2220a]] Luo al. 202 I Mendicta et al. (022) trying to alleviate this problem based on the model-averaging framework. For example, FedProx (Li et al. 2020) and MOON (Li et al., 2021d) ad adjusts the local training procedure to pull back local models from global model. FedNova [aan et al. 2020a) normalizes local steps of each client during aggregation. recent work (Huang et al. 2022) proposes few-shot model agnostic FL, which is able to train any models in setting where each client has very small sample size. It applies domain adaptation in the latent space with the help of large public dataset. However, these scenarios. For
algorithms need many rounds to converge, which may not be practical in example, different companies may not be willing to communicate with each other frequently due to privacy and security concerns.
2

---

Published as a conference paper at ICLR 2023
2.2 ONE-SHOT FL ALGORIHTMM
One-shot FL (i.e., FL with only one communication) is a promising research direction. It has a very low communication cost. Moreover, it enables applications such model market (Vartak] 2016). where the clients only need to sell the models to the market and users conduct learning on the models.
The original one-shot FL study (Guha al., 2019) collects local models final prediction. It further proposes to use knowledge distillation on such ensemble with public data. FedKT (Li et al. 2021c) proposes consistent voting to improve the ensemble. recent work (Zhang et al. )021) proposes a data-free knowledge distillation scheme to perform one-shot FL. It uses the same ensemble distillation method as FedDF (Lin et al., 2020). Its main contribution is the fake data generation in the server side to replace the public dataset for distillation. Such a technique is orthogonal to our FedOV, and can be combined with FedOV. All the above studies do not investigate comprchensive label skew cases. For more related studies and detailed comparison between FedOV and these studies, please refer to Appendix A.
ensemble for the
2.3 OPEN-SET RECOGNITION (OSR)
OSR is an emerging field with many important applications (Salchi et al. 2021). In OSR, an additional class "unknown" is introduced besides the original classes. A good open-set classifier is expected to predict its known classes correctly, while predicting "unknown" if is unsure about the input data. A popular approach in OSR is generate outliers and label them "unknown" in training.
One direction is to use GANS to generate outliers. For example, Neal et al. (2018) apply GANs to generate outliers from latent space that (1) is close to real samples, and (2) with high probability of outlier (low probability of any known class). However, the training of GANs is very expensive. Moreover, is challenging for GANs to generate clear nigh-dimension images.
PROSER (Zhou et al. 2021) is a state-of-the-art open-set recognition approach. It generate outliers by linear interpolation of embedding space among different classes. Moreover, introduces an additional loss to increase the possibility of predicting a sample as "unknown" when discarding its true class. Suppose the training set is D {Z,, Pi where 2; is training sample and E 10.1..... - 1} is its label. The neural network from input space to embedding space is denoted @pre(-), and (post(() is the neural network from embedding space to output space. The whole model is s())
&post ((pre(.)) The PROSER loss is shown in Equation [ where / is the cross entropy loss, e denotes class "unknown", Eyre is the outliers generated from the training data (Tpre AOpre (z)re(z;) + (1 N) (zz)), [0,1] is sampled from Beta distribution, B and two
LPROSER
Σ ((f()),)) (l(f())) - (Z.D)ED
Σ
(ppost(@pre).
(I)
(ELD))ED
Since PROSER achieves state-of-the-an results and easy to implement, consider it base method for training an open-set classifier.
3 FEDOV: ONE-SHOT FEDERATED OPEN-SET VOTING FRAMEWORK
3.1 PROBLEM STATEMENT
Suppose there are N clients Pi...., PN with local datasets D D Our goal is to train a good machine learning model over D l Use[N] with the help a server, while the raw data are not exchanged. Moreover, each client is allowed to communicate with the server only once. this paper, focus image classification task due its popularity.
3.2 MOTIVATION
Observation 1 Voting is a popular method in existing one-shot FL approaches (Guha al. 2019 Li et al. 2021c). However, these approaches suffer under extreme label skews. For example, when divide MNIST dataset into 10 clients where each client has only class, both close-set voting
3

---

Published as a conference paper at ICLR 2023
(Guha et al. 2019) and FedKT (Li al. 2021c) only have lower than 20% test accuracy. When each client has two classes, the test accuracy of both methods are lower than 50%. The problem is that the biased towards their seen classes shown in Figure ia When a test sample of class one comes, in traditional close-set voting, the first and third client make wrong predictions. Therefore, the voting result cannot predict correctly.
predictions of close-set classification models
Implication In the label skew setting of FL, close-set classifiers are weak for one-shot FL and predict every input among its known classes. For voting, would be better if models can be modest and admit unknown for its unseen classes as shown in Figure lb This motivates to apply OSR in FL to introduce an unknown class to improve the voting.
Local Data
Distribution
0
0
1
0
2
1
0
2
Input
Training
Training
Prediction
Prediction
Probability (1.0, 0.0, 0.0] + [0.0, 1.0, 0.0) [0.0, 0.0, 1.0]
10.0, 0.0, 0.0, 1.0)+(0.0, 1.0, 0.0, 0.0]+(0.0, 0.0, 0.1, 0.1, 0.9] sum
sum
output
output
[1.0, 1.0, 1.0]
[0.0, 1.0, 0.1]
1
(a) Traditional close-se set voting
(b) Open-set voting framework ('u' denotes un- known class)
Figure I:
Comparison between traditional close-set voting and our open-set voting framework. We allocate the data of classes I, and 2 into three clients Dirichlet distribution Dirg(0.02).
Observation 2 Directly applying PROSER (Zhou et al. 2021) the local training of FL cannot achieve good local open-set classifiers. We visualize the representation learned by the local model of a client using PROSER in the local training in Figure 2a The model trained on a client which only has class 0 and 6 samples of MNIST. The generated outliers quite limited and far from the training data when simply applying PROSER. The representations from the data of the seen and unseen classes are mixed and cannot be distinguished.
To better suit OSR algorithms for label skews in FI., we need new techniques to generate outliers which should 1) diverse and 2) close the seen classes. We will introduce them in Section 3.4 and Section 3.5 and explain Figure 2b and Figure 2c in Section 3.6
Implication
15
-20
418
-5
-30 -20
(a) PROSER
(b) PDD
(c) FedOV
Figure T-SNE visualisation of the features extracted by local models trained with different methods. During training, the client only has data from classes 0 and 6. In each sub-figure, we plot the representations of the data from the seen classes and 6, unseen classes, and generated outliers during training. The black lines are possible classification boundaries.
4

---

Published as a conference paper at ICLR 2023
3.3 THE OVERALL ALGORITHM
Based on the above observations and implications. we develop a new approach named FedOV to address label skews in federated learning. FedOV addresses the challenges directly applying OSR to FL in the following two aspects, First. in order to generate diverse outliers, we propose data destruction (DD) to directly generate outliers from true samples. Second, in order to generate outliers that are even closer to true samples, we propose adversarial outlier enhancement (AOE) to learn tighter boundary to surround the inliers.
The overall framework of open-set voting is described as follows. For the training stage, each client trains an open-set classifier locally and submits it to the server. For the prediction stage, the server sums up the prediction probability of all submitted models on the imput sample while discarding their "unknown" " channel. The class with maximum prediction probability is outputted as the prediction label. An example of open-set voting is shown in Figure 1b With the help of the unknown class, local models can admit its uncertainty when encountering unseen classes. The first and third model with little knowledge of class assign very high probability to "unknown", and the second model outputs class I with 100% certainty due to its expertise in class 1. In this way, the input image can be correctly classified.
The whole procedure is shown in Algorithm Suppose there are e classes and classes 0 to (c-1) 1) are the classes from the original training data. We use class c to denote the unknown class, In each client, it first initializes the local model fi (Line 2). Then, in each round, for each batch of training data, it generate outliers by data destruction and adversarial outlier enhancement (Lines 5-6, see Section (3.4] and 3.3 for more details). Next, considering the outliers as the unknown class, cross-entropy loss is computed on the outliers. By summing the PROSER loss (computed according to Equation [ and the cross-entropy loss as the total loss, the local model is updated using the Adam optimizer (Lines 7-8). The local models are sent to server after reaching the number of training rounds (Lines 9).
In the server, it aggregates all the local models new sample comes for prediction, it sums the prediction probability of each model (Lines 13-15). Then, the known class with the highest probability score is considered as the output label (Line 16).
ensemble the final model (Line 11). When
Since FedOV only requires a single communication round, its communication cost is O(N M), where M is the size of local model. The communication cost is low compared with iterative federated learning algorithms, which need many rounds to communicate the models.
Algorithm 1:
The FedOV algorithm. Lee is the cross entropy loss and a is the softmax function. Input: number of clients N, number of classes c, training rounds T
Each client executes: Initialize local model f.
Server executes:
11 Collects fI.f.. as an ensemble.
for t 1.... T do for each batch of local data (x. y) do
Prediction(z):
scores = 0 for 1, ...... N. do
x' Data (estaaestuuti))(
x" = FG
15 scores = scores + (f((z)
LPROSER + Loe Loeefftte x")),,
16 yp = arg jag. scores,
Update f; with loss L
17 return y/
Upload f to the server.
3.4 DATA DESTRUCTION
generating diverse outliers from limited training data chal- lenging. While PROSER generates outliers by mixing data from different classes, can we generate outliers from each individual sample? Inspired by data augmentation (Shorten & Khoshgoftaar 2019) which has been very popular approach to enhance the features before training, propose the novel Data Destruction (DD) methods to use data operations to transform the data to generate outliers. As opposed to applying to enhance the features, our DD applies intense data operations to corrupt the original key features, which is effective and efficient. Specifically, DD has two components: candidate data destruction operations and boosting outliers with set of such operations.
From our observation in Section 3.2
5

---

Published as a conference paper at ICLR 2023
O
J
3
/
/
2
3
42
2
5
5
(a) Outliers generated by DD.
(b) Outliers after AOE.
Generated outlier examples MNIST, MMSSST and CIFAR-I0 dataset. In Figure 3a the first column contains the real samples and the other columns are generated outliers of that sample. In Figure 3b] all figures generated outliers by adversarial outlier enhancement.
Figure 3:
Candidate Data Destruction Operations With the goal to corrupt the original features, we try comprehensive list of data operations and summarize the following useful candidate operations for randomly select rectangle region and copy it another randomly selected region in an image; (2) RandomSwap: swap two randomly selected rectangle regions; (3) RandomRotation: randomly rotate square region of image; (4) RandomErasing: randomly erasing a large rectangle region in an image; (5) GaussianBlur: blur an image by Gaussian function with a large variance; (6) RandomResizedCrop randomly crop a small portion of image and resize it to the original size. Here (1)-(3) are our proposed operations and (4)-(6) are existing data augmentation operations where we use them with abnormal hyper-parametern Examples of the generated outliers are shown in Figure 3a More details about these are elaborated in Appendix B.]
data destruction: (I) RandomCopy
Boosting Outliers with Data Destruction Set To boost diversity of outliers, we introduce randomness during the outlier generation. We do not fixed operation each image generate outliers. In each time, considering the above candidate operations as a set, we randomly sample one operation to generate diverse types of outliers generated by different operations. We show that such boosting method can significantly improve the accuracy in Appendix [B.3]
outlier each time. Then, in each batch of data during training, there exists
In summary, the intuition behind DD is to corrupt part of the key features such that 1) the generated data not similar the training data and 2) the generated data is diverse that the model does simply consider certain patterns as outliers.
3.5 ADVERSARIAL OUTLIER ENHANCEMENT
Adversarial training (Goodfellow al. 2015 Kurakin et 2016) has been a popular approach to protect machine learning models from malicious attacks. For example, Goodfellow et al. (2015) utilizes fast gradient sign method (FGSM) generate adversarial samples such that the model outputs wrong answer with high confidence. Then, the adversarial samples are used as part of the training data regularize the training.
Inspired by adversarial training, instead of using FGSM to generate adversarial samples for robust training, we apply it to optimize the generated outliers. Specifically, suppose the client is training the model If with the generated outliers x' by our data destruction method. We utilize FGSM to generate x" such that the model wrongly outputs x" as a seen sample with a high confidence. Then, the enhanced outliers x" are used together with the generated outliers z' as the unknown class to update the model. We call this method Adversarial Outlier Enhancement (AOE). Examples the enhanced outliers are shown in Figure 3bb] Compared Figure 3b with Figure 3a the outliers more normal and look like different classes from the training data after AOE (c.g., in the third row of Figure 3b the third and eight outliers look like digit "3" although they generated from digit "2".).

---

Published as a conference paper at ICLR 2023
3.6 DISCUSSION
T-SNE Visualization As shown in Figure 2b and Figure 2c DD can generate diverse outliers to help distinguish data from seen and unseen classes and AOE can further decrease the margin between the outliers and training data (i.c., class and 6) to learn a better classifier.
Extension of FedOV with Knowledge Distillation One shortage of FedOV is that the final model an ensemble of local models, therefore its prediction and storage costs may be large especially when the number of clients is large (e.g.. cross-device setting). Some existing approaches (Lin et al. 2020; Li et al. 2021c) utilize knowledge distillation to distill the knowledge from multiple local models to global model with the help of public synthetic dataset. Our approach is compatible with the above methods. With knowledge distillation, we can transform the ensemble of local models into a single global model. Then, can significantly reduce the storage and prediction costs of the final model. Moreover, considering the final model as the initialized model for iterative federated learning algorithms like FedAvg (McMahan et al., 2016). we can conduct multi-round federated learning to further improve the model. As shown in Section 4.4. FedOV can be effectively combined with the existing approaches to increase their accuracy and communication efficiency.
EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
Datasets We conduct experiments on MNIST, Mashion-MNIST, CIFAR-10 10 and SVHN datasets. We the data partitioning methods in [i al. (2021b) simulate different label skews. Specifically, we try two different kinds of partition: 1) # k: each client only has data from k classes. We first assign k random class IDs for each client. Next, randomly and equally divide samples of each class to their assigned clients; 2) P& (Br( for each class, we sample from Dirichlet distribution Pk Dir (3) and distribute Pa. portion of class k samples to client j.
Baselines We include ohe-shot FL algorithms as baselines including close-set voting (Guha et al. 2019) and FedKT (Li et al. (021c). We also compare FedOV with the iterative FL algorithms including FedAvg (McMahan et al. 2016), FedProx (Li et al. 2020). FedNova (Wang et al. 2020a and FedDF (Lin et al.j (020). We run them a single round for fair comparison. Note that FedKT and FedDF require public dataset (or synthetic dataset) for distillation. In each task, we use a half of the test dataset as the public dataset for distillation for FedKT and FedDF and the remaining for testing. Since the source code of FedSyn (Zhang al. 202I) is not publicly available and we have included FedDF (which has the same distillation approach as FedSyn) in our experiments, we do not compare it with FedOV.
By default, we follow FedAvg (McMahan et al. 2016) and other existing studies (Li et al. 2021c Wang et al. 2020b) to use simple CNN with 5 layers in experiments. There are 10 clients. For local training, we rum 200 local epochs for each client. We set batch size to 64 and learning to 0.001. For results with error bars, we three experiments with different random seeds.
Default setups
Due to the page limit, we only present some representative results in the main paper. For more experimental details and results, please refer to Appendix B
4.2 AN OVERALI- COMPARISON
We compare the accuracy between FedOV and the other baselines as shown in Table I can significantly outperform baseline algorithms with only one communication. In many settings, FedOV achieves more than 10% accuracy than close-set voting. In the extreme cases such as #C = 1, FedOV can outperform close-set voting by at lease 30%. The iterative FL algorithms cannot achieve satisfactory accuracy when running for single round.
Our algorithm
7

---

Published as a conference paper at ICLR 2023
Table I: Comparison with close-set voting and various FI. algorithms in round.
Datrort
Partiion
FodOW
Close
FalAng
FodiProx
FedNNoon
SCAFFOLD
FeanF
FedKT
40.0%
10.29-1922
10.5%6=1.0%
10.56210%
10.5
45.0% 13.4%
25.2% 2215
CIEAR 10
35.0%
13.9
2135
31.8 1.55
68.7% 10.7%
0517-1011
19.8
1805
01.7%
35
3500
10.4
189951125
74.0% 11.1%
15791705
182952905
31395288
2009 14.3
SVIN
K1.0% 11.2%
6019-1955
11.36-12
19.55
04.5
1..5
N5.71 10.3%
3555 10
31.05 9.7%
75.2%
75L770
22.39:250
73.390
10.10:103
1319
121
61.7%1100
30.99-133%
17.70
1009 1800
FMNIST
730000
570264708
40.904 11.25
3428-134
76.27
2128-1765
16.76-677
15.75
XL98-112 1209
1208--200
MNIST
13.7%
2009
2428-11555
SSIRCT
67.5%6-22.
71.075
76.3
67.752220
(0.3)
96.2% 10.4%
93.39-10.45
40.2%:5..6
36.7%
40.1%6:4.8%
35.366:7.366
82.66
68
Table 2: Experimental results of different FL voting strategies with simple CNN model.
Open-set (PROSER)
Open-set
Dataset
Partition
Close-set
FedOV
(PROSER + DD)
#C
10.2%
10.6%
33.5%
40.0% 11.7%
#C = 2
37.2% + +.55%
34.8%+44.55
41.3%
42.0% 1000%-222-. -..4%
CIFAR-10
#C=1
43.2% w2w.442.77 2.7%
50.2%
44.3%+4. 2.1%
55.6% -6.3%
Pk Dir(0.5)
65.0% +0.1%
66.6% 1%
67.6% +0.3%
65.7%-0.77%
Pk (ir(U.I)
55.9% 1..9%11.34
10.0%%
61.3%
61.7%=1.1
#C
7.3%+F..1%
6.7%++..1%
47.3% w1..111..00 1.3%
64.5% 1-55%=1-. -199
#C=2
35.7%.17.6%
42.6%1110.9%%
60.9%.11..%%
74.0% 1.1%
SVHN
#C =
60.1%
64.9%.88.44
72.7%
81.0% 1.2%
Ph (15(
0..4%
85.2%++0.35
84.9%
85.7% 1066-000 -0.3%
Pk Dir(0.1)
60.4%
65.9%
74.6%
78.7% -0.6%
#C=
00.5%
15.1%
71.0%
73.3%
1.6%
#C=2
36.3%.
33.0%
+1.7%
64.1%+6..7%
61.7%+ 11.0%
FMNIST
#C=3
57.0%
51.9%
66.1%.1..44
73.8% 1.7%
Pk ~ Dir(0.5)
10.2%
38.7%:10.
89.1%
-.33%
Ph Dir(O.1)
+0.4%
73.2%+00.9%
70.0%%+0.8%
76.2% 1-3%%=1122 -1.2%
#C=1
15.5%12.88
16.5%:10.24
76.5%.18.3%
79.3% 118%
#C=2
44.3% 6.4%
48.7% 44.2%
61.5%+
64.2%
1.6%
MNIST
#C=3
+7.6%
55.9% 10%%+11.00 1.2%
73.6%+22.6%
83.7%
1-77%=55-33 5.3%
Pk (ir(0.5)
98.3%11.11%
98.3%
98.5%.10.19%
98.6%
0.0%
Pk ~ Dir(U.1)
10.4%%
93.8%10.5%%
95.8%
96.2% 1..%%
4.3 ABLATION STUDY
We show the effect of each component of FedOV including open-set voting (PROSER), data destruc- tion (DD), and adversarial outlier enhancement (AOE). Specifically, we add one component each time and the results are shown in Table 2 From the table, we can observe that FedOV with all the three components
achieve the highest accuracy in most settings. Simply applying PROSER in FL may not increase the accuracy compared with close-set voting (e.g.. CIFAR- with #C = 2). Our proposed outlier generation techniques effectively boost the accuracy of open-set voting. Moreover, the adversarial outlier enhancement can significantly increase the accuracy in some settings (e.g., SVHN with #C 1). We compare with open-set voting (PROSER AOE), i.c., using AOE loss without DD loss in Appendix B.2
4.4 COMBINING WITH KNOWLEDGE DISTILLATION
Assuming that there exist unlabeled public data the server, FedOV can also be combined with knowledge distillation like FedDF (Lin et al. 2020) and FedKT (Li et al. 2021c). We call it Distilled FedOV. We compare Distilled FedOV with the other baselines. According to FedKT experimental
settings, we train 100 epochs for each client. For distillation, we run 100 epochs for the final student model. According to default settings, we use the simple CNN model and SVHN without
8

---

Published as a conference paper at ICLR 2023
Table 3: Comparing distilled FedOV with the other baselines. The partition is Ps Dir(0.5).
Dataset
Disiiledd
FedOv
FedKT
FedDP
SOLO
FedAvg
FedProx
FedNova
SCAFFOLD
MNIST
97.7%
50.3%
95.48
0.8%
17.3%
78.4%
47.3%
67.2%
69.0% 2.5%
67.9%142%
SVHN
81.8%
114%
75.7%
1.0%
83.64322
16.1%114.2%
13.6%
17.7%199
27.24-622%
19.26
IMNIST
85.0% -0.1%
82.5%
0.4%
80.6%
62.3% 1.5%
49,58 12.9%
48.7% 13.4%
49.48 11.3%
48.9%
CIFAR-10
2.3%
34.1%-11.9%
28,9641.55
18.79-14..%
4.1%
extended dataset. Note that the settings are different from FedKT paper (Li et al. 2021c). therefore our reported accuracy differs from FedKT paper. Further details are elaborated in Appendix [B.1
We rum distilled FedOV for three times and results are shown in Table We can observe that Distilled FedOV can achieve a higher accuracy than FedKT and the other iterative FL. baselines with the same size of final model, which further verifies the effectiveness of our open-set voting framework.
Extension to Multiple Rounds After knowledge distillation, we can further train the distilled model by FI. averaging algo- rithms (e.g., FedAvg, FedProx, etc). We conduct experiments on MNIST with 10 clients and data partitioning Pk ~ (05(.. The results shown in Figure 4 For FedOV FedProx and FedKT FedProx, FedrT_FeeProx, we run Distilled FedOV and FedKT for the first round and using the global model as the initialized model for the later rounds using FedProx. From the figure, with the help of FedOV, the accuracy after first round is high. Then, FedOV FedProx can converge much faster than the other algo- rithms.
0.9
0.8
0.7
FedOV_FedProx FedProx
0.6
FedKT FedKT_FedProx Fe_FProxx
FedAvg
0.5
FedProx
0
Communication round
Figure 4: Extension to multiple
rounds on MNIST.
4.5 SCALABILITY
We test the scalability of FedOV by varying the number of clients. Due to page limit, we only show results CIFAR-I0 in Table 4 For the results on the other datasets, please refer to Appendix BB. From the table, we can observe that FedOV still achieves the best accuracy when increasing the number of clients. Moreover, with the help of knowledge distillation, Distilled FedOV can outperform distilled close-set and other iterative FL algorithms with the same storage and inference cost.
Table 4: Experimental results of different number of clients on CIFAR- 10 with simple CNN model.
Client Number
Partition
FadOV
Distileed
Close-se
Distilled
FedAvg
FedProx
FedNova
SCAFFOLD
FedDF
FedKT
#C
41.9%
30.4%
7.9%
7.9%
10.0%
100%
10.0%
10.0%
9.9%
10.0%
45.6%
34.9%
33,0%
26.6%
10.7%
15.0%
15.4%
26.4%
20
#0 = 3
37.2%
40.2%
54.1%
37.6%
10.2%
11.5%
10.1%
16.0%
13.8%
36.8%
Ph Dar(0.5)
62.5%
46.4%
59.8%
43.5%
14.3%
23.4%
14.1%
16.9%
32.7%
39.7%
Pb
Dir[U.1]
#C =
45.3%
34.3%
10.1%
9.5%
10.0%
10.0%
10.0%
10.0%
8.3%
10.0%
#C
56.0%
40.4%
42.1%
32.3%
10.0%
10.5%
10.0%
11.1%
27 256
26. 26.39
-40
#0 =
60.8%
43.2%
52.1%
41.9%
10.0%
10.3%
10.0%
12.7%
20.8%
35.8%
Dir(0.))
59.2%
46.7%
58.25
46.7%
11.6%
18.7%
14.2%
18.4%
32.3%
37.29
Ps Dir[U.1]
55.0%
#0
43.8%
33.2%
10.0%
10.2%
9.7%
9.3%
9.8%
8.7%
9.8%
10.1%
#C
56.6%
43.1%
44.2%
34.4%
10.6%
10.1%
10.0%
10.3%
22.7%
27.5%
80
#0 =
54.1%
44.8%
53.4%
44,4%
14.7%
16.0%
17.7%
12.7%
33.4%
33.2%
(ir(0.5)
53.4%
42.6%
52.5%
44.5%
21.1%
23.9%
23.39
11.4%
31.3%
30.49
Ph. Dvr(0.1)
48.3%
44.2%
23.0%
5 CONCLUSION
In this work, we design a novel one-shot FL algorithm FedOV to address label skews in one-shot federated learning. We propose open-set voting by introducing the "unknown" class in voting. We observe that directly applying state-of-the-ant open-set recognition algorithm PROSER to one-shot learning has the problem of limited outliers due to limited number of classes. To address those issues, develop two techniques, data destruction and adversarial outlier enhancement, to improve the performance of open-set voting. Our extensive experiments show that FedOV can achieve significant accuracy improvement compared with the other baselines under diverse label skew settings.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: LISG2-RP-2020-011 Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of National Research Foundation, Singapore.
REFERENCES
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. CoRR, abs/1 412.6722,,652, 2015.
Neel Guha, Ameet S. Talwalkar, and Virginia Smith. One-shot federated learning. ArXiv, abs/1902.11175. 2019.
Hossein Hosseini, Hyumsin Park, Sungrack Yun, Christos Louizos, Joseph Soriaga, and Max Welling. Federated learning of user verification models without sharing embeddings. International Conference Machine Learning, pp. 4328 42366. PMLR, 2021.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:/909.0000 1909.0635, 2019.
Wenke Huang, Mang Ye, Bo Du, and Xiang Gao. Few-shot model agnostic federated learning. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 7309-7316, 2022.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint
2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for on-device federated learning. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020.
Anirudh Kasturi, Anish Reddy Ellore, and Chittaranjan Hota. Fusion learning: A one shot federated learning. International Conference on Computational Science, pp. 424-436. Springer, 2020.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. arXiv preprint 1rXiv:1611.0123: 2016.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9659-9669, 2021a.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. arXiv preprint arXiv:1907.09693, 2019.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. arXiv preprint arXiv:2 arXiv:2102.02079 2021
Qinbin Li, Bingsheng He, and Dawn Song. Practical one-shot federated learning for cross-silo setting. IJCAI, 2021c.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021d.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In MLSys, 2020.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33, 2020.
10

---

Published as a conference paper at ICLR 2023
Chih- Ting Liu, Chien-Y Yi Wang, Shao- Yi Chien, and Shang- Hong Lai. Fedfr: Joint optimization federated framework for generic and personalized face recognition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 1656-1664, 2022.
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-505 5984, 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication-effcciii learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629 2016.
Matias Mendieta, Taojiannan Yang. Pu Wang. Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8397-84008 2022.
Lawrence Neal, Matthew Lyle Olson, Xiaoli Z. Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In ECCV, 2018.
Pramuditha Perera, Vlad Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M Patel. cenerativediieriminaaii feature representations for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11814-11823, 2020.
Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and M. Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of-distribttion detection: Solutions and future challenges. ArXiv, 10.121100.40011. 2021.
MyungJae Shin, Chihoon Hwang. Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong-Lyum Kim. Xor mixup: Privacy-preserving data augmentation for one-shot federated learning. arXiv preprint arXiv:2006.05 051 18, 2020.
Connor Shorten and Taghi M Khoshgoftuar. A survey image data augmentation for deep learning. Journal of big data, 6 6(1):1-48, 2019.
Manasi Vartak. Modeldb: a system for machine learning model management. In HIIDA '16, 2016.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem heterogeneous federated optimization. Advances Neural Information Processing Systems, 33, 2020a.
Tianhao Wang, Johannes Rausch, Ce Zhang. R. Jia, and Dawn Xiaodong Song. A principled approach to data valuation for federated learning. In Federated Learning, 2020b.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Naemura. canssfffcatio--rcoosstuuu learning for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4016-4025, 2019.
Felix X Yu, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Federated learning with only positive labels. arXiv preprint arXiv:2004. 10342, 2020.
Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Jianghe Xu, Shouhong Ding, and Chao Wu. A practical data-free approach to one-shot federated learning with heterogeneity, arXiv preprint arXiv:2112.1233 2021.
Da-Wei Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4399 4408, 2021.
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Oliver Wu. Distilled one-shot federated learning. ArXiy, abs/2009.0700 0079999 2020.

---

Published as a conference paper at ICLR 2023
A DISCUSSIONS
A. MORE RELATED WORKS ON ONE-SHOT FL.
From the perspective of data sharing, Zhou et al. (2020) proposes perform dataset distillation and upload the distilled data to server for centralized training. Kasturi et al. (2020) proposes to fit the features in each client by some distribution. Then the server generates fake data with these distributions. Both approaches raise additional privacy concerns due to the uploaded fake data or feature distributions. Besides, we cannot find source code for both methods, so we do not compare FedOV with them.
MIr (Shin et al. 2020) proposes to use exclusive OR operation (XOR) to encode and decode samples for data sharing. However, it assumes that all clients and the server have labelled samples of a global class, which is impractical real-world applications. In our experiments, we adopt the setting like many existing studies (Li et al. 2020 Karimireddy et al. 2020 Wang et al., 2020a Hsu et al. 2019 Li et al. 20216), which cannot guarantee the above assumption. Therefore, we do not compare FedOV with XorMixFL in experiments.
We compare these one-shot FL algorithms in Table|5] None the previous one-shot FL algorithms conduct experiments both distribution-based and quantity-based label skews.
Table 5: Comparison among existing one-shot FL algorithms.
Distribution-buse bused lahel skews
Quantity-based label skews
Only model uploaded
Free of public data
Penspective
Algorithm
Close-set voting GGuha et al. 2019
Distilled Close-set (Guba
Feek (L al 2021e
x
Model ensemble
FedSya Zhann &l 2021
/
FedOv (ours)
7
Distilled FedOV (ours)
/
/
x
DOSFL (Zhou et al. 2020
x
Sharing fake data
Fusion learning (Kastor a.
2020
XorMixFLL (Shin al. 2020
A.2 IMPLICATIONS FOR THE DIFFICULTY OF GENERATING OUTLIERS BY GANS
While we propose data destruction generate outliers, another method is to generate outliers by GANs. We apply the methods in Neal et al. (2018) to generate counterfactual images in MNIST dataset. However, it is very difficult to tune the hyper-parameters to generate outliers. We encounter two problems: (1) generated images
good that the classifier goes on strike because it cannot tell the difference between original images and counterfactual images; (2) generated images are so bad that the classifier can easily tell the difference by some simple features. The loss quickly converges to zero. For example, since the generated counterfactual images are all blurry, the classifier may classify clear images as inlier and blurry images outlier. When it encounters clear images of other digits during testing, it can classify these samples into its known class.
By trials and errors, we summarize two criteria of good outliers for future researchers' reference. Good outlier needs to (1) be clearly different from inliers; and (2) not carry certain types of simple feature that can be easily told difference.
A.3 CHALLENGES OF APPLYING OSR TO FL AND POSSIBLE DIRECTIONS
One possible future direction of FedOV is to set threshold to normalize each client's outlier probability. Take a simple example, if client only sees dogs, another client only sees cats. The first model predicts 0.001 & outlier for dogs and 0.01 % outlier for cats. The second model predicts 5% outlier for cats and 50% outlier for dogs. For ROC-AUC metric, both models get 100% ROC-AUC for outlier detection among dogs and cats. However for FL open-set voting framework, the voting result always dog. This is due to the unreasonable outlier probability scale of first model.
Another promising direction is to come up with more systematic and diverse data destruction methods. Although our current approach can outperform cutpaste, the outlier types are still limited. For future research, one adapt more computer vision algorithms to further improve diversity.
12

---

Published as a conference paper at ICLR 2023
Finally, current data destruction framework only works for vision tasks. For tabular dataset language dataset, it deserves further research how to augment diverse outliers.
A.4 A BETTER BENCHMARK FOR OSR ALGORITIMS
From the perspective of open-set recognition, we find that FL voting can serve more realistic, comprehensive and challenging benchmark to test open-set recognition algorithms. Previous open- set recognition experiments mainly divide the classes into totally known and totally unknown, i.e. the training set has either full data of class no data of a class. However, in reality, grey area exists where a model sees only a few training samples of some class. Moreover, previous open-set recognition or outlier detection experiments use ROC-AUC as the metric, which avoids transforming the outlier score to some probability. Instead, ROC-AUC only cares the relative score between known and unknown class. However, it is essential output reasonable unknown probability in real-world application. Our FL open-set voting benchmark includes these challenges. Its data partitioning strategies based on et al. (2021b), which be easily adjusted various settings.
B ADDITIONAL EXPERIMENTS
In this Appendix, we first explain further experimental details Appendix Then we conduct more ablation studies about data destruction in Appendix [.2 and operations of data destruction in Appendix [B.3] Next we experiment with top-k confidence voting Appendix B.4 This is generalization of FedOV where we just count k most confident votes. For heavier models and more complicated datasets, we experiment with VGG-9 in Appendix [B.5 and ResNet-50 on CIFAR-100 -10 in Appendix B.6 In Appendix B.7 we thoroughly compare data destruction with similar method Cutpaste Li et al. (2021a). We We justify that data destruction can generate more diverse outliers and achieve higher accuracy. More experiments
scalability conducted in Appendix B.8 FedOV is further compared with FedDF under different label skews (Appendix B.9). and with FedAwS under #C=1 partition (Appendix B. We also evaluate the effectiveness of DD and AOE centralized OSR settings in Appendix B.11 Finally, we compare FedOV with baselines under multiple rounds in Appendix B.12
B. FURTHER DETAILS
We summarize the datasets in experiments in Table
Table 6: Basic information of datasets Training sample size
Datasets
Test sample size Input dimension
# of classes
CIFAR-10
50,000
10,000
3,072
10
SVHN
73,257
26,032
3,072
FMNIST
60,000
10,000
784
10
MNIST
60,000
10,000
784
CIFAR-IO 100
50,000
10,000
3,072
100
For PROSER, we choose 3 0.01, 1, according the default trade-off parameter setting in the
official code For ease of implementation, we omit the extension to multiple dummy classifiers.
For data destruction, details of our current transformations are follow. The core idea is to destroy the key features of the original image while keeping some scrappy features.
Random resized crop: with scale range (0.1, 0.33). We choose a small portion order not contain the key features and enlarge that portion original size.
Gaussian blur: with random kernel ranging from 1*3 to 5*9, and random E (10, 100). We choose a large a to blur out key features.
Random erasing: with scale range (0.33, 0.5). We choose a large portion to erase order to spoil key features.
https://github/ cow.:/oudweeeeeeeeeeoom http:/hhoooo-hh/////221111110
13

---

Published as a conference paper at ICLR 2023
Random paste: We random paste half of the image another place.
Random swap: We swap left-side and right-side, or upper-side and down-side.
Random rotation: We random rotate two square portions of the image.
For adversarial learning, set local steps and each step size 0.002.
For Distilled FedOV, the local training step is the same as FedOV. After the server collects all local models, it performs knowledge distillation based on the open-set voting results. Formally, denote the local models ..., J'N and each model outputs c + scores where the last score is for the class "unknown"" The student model Sa outputs e scores for the known classes.
For an input v, we add the output probability of each local model scores(#) =
(f)(z))
where a is the softmax function. The voting result is the first scores for the e known classes, i.e. v(a:)= scores())o,,,...co
The normalized probability Un (a) [v(z]] used as the target to distill the student model. Therefore, we have distillation loss Lais = (f (x(x)), where KL is Kullback-Leibler divergence.
=(a:)
By default, we use the first half of the test set as the public unlabelled dataset for knowledge distillation in the server and then test the distilled model the second half of the test set. We use Adam optimizer with learning rate 0.001, and train 100 epochs on the public unlabelled dataset for the distillation process.
For FedProx, we tune the hyper-paramede y. E {0.001, 0.01, 0.1, 13. For methods requiring knowl- edge distillation, we first half of test set as public unlabelled dataset. The other half used to compute accuracy.
Our simple CNN contains two 5*5 convolution layers with 2*2 max pooling layer. The first has channels and the second has 16 channels. Then it has two fully-connected layers with 120 and 84 neurons separately. We use ReLu the activation function between layers.
All experiments are conducted on single 3090 GPU. We compare computing time of different algorithms in Table 7
Table Running time different algorithms with simple CNN on CIFAR-10 dataset. There are 10 clients and each client runs 200 local epochs with only one communication. Our device is a single 3090 GPU.
Close-set voting
FedOv
FedAvg
FedProx
FedNova
h
11.5 h
1155 h
220
1.55
For the additional time cost of using data destruction (DD) and adversarial outlier enhancement (AOE), it costs about 2 times computation during local training in our experiments, depending on different datasets. The experiments shown Table are with partition Pk: Dir(0.5) into 10 clients.
Table 8: Time for the first client to finish local training of 200 epochs. Our device is a single 3090 GPU.
Dataset
FedOV
Open-set voting w/o DD & AOE min
10
17 min
FMNIST
min
min
For more implementation details, please refer to source code.
B.2 USING ADVERSARIAL LEARNING WITHOUT DATA DESTRUCTION
In this section, we test on the effect of adversarial learning. Adversarial samples are based on outliers generated by data destruction. Here we also data destruction outliers to generate adversarial samples, but we omit the loss of data destruction (DD). Results are shown in Table 9 Generally, omitting the loss of DD is not good choice.
14

---

Published as a conference paper at ICLR 2023
Table 9: Experimental results of different FL voting strategies with simple CNN model. We repeat all experiments for three times.
Open-set (PROSER + AOE)
Dataset
Partition
FedOV
#C =1
39.2%=3.1%
40.0%+1.7%%
#C =2
35.8%=3.0%
42.0%+2..4%
CIFAR-10
#C= =
0.3%%
55.6%-16.39
Pk (ir(0.5)
65.7%
0.7%
65.7 % +0.7%
Pk ~ Dir(0.1)
61.4%
61.7 #1.1%
B.3 COMPARING WITH USING SINGLE TRANSFORMATION
In this section, we verify the effectiveness of random data destruction from a pool of transformations. We compare with using single transformation. We experiment CIFAR-10 dataset, and all algorithms are open-set voting containing the same PROSER loss. The differences are their data destruction strategies. Results are shown in Table I0 None of single transformation reach the accuracy of random transformation from data destruction set.
Table 10: Experimental results of using random data destruction from a pool of transformations, compared with using single transformation.
Partition
Data
Random resized crop
Gaussian blur
Random crasing
Random paste
Random swap 32.3%
Random
destruction
rotation
#C
33.5% 2.3%
15.2%-0.8%
10.7%
11.6%
1.6%
23.9% 1.55%
0.3%
=C=2
7..3%
37.7%=0.5%
38.6%.2..0%
36.4% 160%
42.4% 55.0%
31.3%=3.06 0%
35.3%=77.15
#C' =
2.11%
50.3%%33.6%
52.3% 3%
6..7%
52.3%:24.0%
53.8%=2.88 8%
$1.2%=24.0%
DiF(0.5)
07.6%
06.0%=1.155
66.48:11229
06.6%:=0.7%
6..8%=0..8
08.0%
Dir(0.1)
6.0%%
36.9%%=..6%
56.1%%-2..5%
24.9%%
36.49::2.6%
58.2%
$7.9%
B.4 TOP-K CONFIDENCE VOTING
possible alteration to our framework is to select the top k confident models for voting. Specifically, after getting predictions of all models, the server sorts all predictions by the probability of "unknown" channel. Then we sum up predictions of the lowest k "unknown" (i.e. the top k confident) while discarding all other predictions. Hyper-parameter
be tuned for different tasks.
We visualize the voting accuracy curve versus k for experiments of VGG-9 on CIFAR-10 in Figure 5 As we can see, the best differs for different label skews. For extreme #C 2, k 1 seems the best. For slight skew pk (ir(0. k 10 is the best. Under #C = 2 cases, each model is trained only two classes, such case would better follow the most confident expert, since others more likely to make wrong prediction. Under and are generally more clever, where considering all the models for prediction becomes better.
Dir(0.5), models are trained on more classes
0.9
0.8
0.7
Test set voting
0.6
0.5
0.4
0.3
#C=2
#C=3
0.2
Dir(0.1)
0.1
Dir(0.5)
0.0
0
2
8
10
K
Figure Voting accuracy on different k of VGG-9 on CIFAR-10. 10.
15

---

Published as a conference paper at ICLR 2023
By tuning k for different settings, results of best top-k confidence voting is shown in Table I1 For close-set voting, we report the accuracy of counting votes from all clients, since all clients are equal. For open-set voting, report the best accuracy among counting votes from top 1, 10 confident clients. We can get similar conclusions as in Section 4.3.
Table 11: Experimental results of different FL voting strategies with simple CNN model. We report the best top-k confidence voting accuracy. We repeat all experiments for three times.
Open-set
Open-set
Dataset
Partition
Close-set
FedOV
(PROSER)
(PROSER + DD)
#C #C=
10.2%
10.9%10.6%%
33.5%+2..3%
40.0%
37.2%-+2.59%
43.9%
49.7% 10.5%
48.9% 1.2%
CIFAR-10 10
#C=3 =3
43.2%=22.7%
54.9%12..0%
58.9%
58.8%11.
Pk Dir(0.5)
65.0%
66.7%:10.1%
67.6%.10..2
65.7%.10.79%
Pk ~ Dir(0.1)
55.9% 1.3%
18.0%
61.3%.11.0%
61.7%
1166:.1.11 1.11%
#C=1
1.3% 0.11%
6.8%110.1%
1..3%.11.3%
64.5%
1 1.9%
#C=
35.7%=7.69
60.2%+2..%%
62.3%
74.9%
SVHN
#C=3
60.1% 1..119 1955%
73.1%-12.99
73.6%
81.4% 11.2%
Pk (0.(
85.1%=0.4%%
80.4%
84.9%
85.7%=0..3
Pk ~ (ir(
64.9%
69.2%.1..11
1..9%
78.7% 1.0.6
#C=1
10.1%:10 0.55%
18.8% 1.4%
71.0%.1..1%
73.3% 1.5%
#C=2
36.3%=3.59%
46.2%
70.5%+33.6%
74.9%+2..2%
FMNIST
#C =
$7.0%=7.00
64.4% 10.4%
76.0%.12.8%
77.4% 1166%
Pk (or(
88.5%
0..2%
68.7%10..5
89.1%
89.0%
10.3%
Pk ~ Dir(0.1)
73.6% 10.4%
73.2%.10.99%
17.9%.10.6%
78.5% 10.6%
#C=1
15.5% 2.88%
17.3%.00.6%
76.6%.18.3%%
79.4% 11.7
#C=2
44.3% +..4%
69.3%+12.49
88.6%
88. 2% 17.%%%489%%
MNIST
#C =
59,6%=77 7700%
87.3% 1.3.%%
93.7%:12.4%
96.2% 11.1%
Ph Drr(U.5)
98.3%
90.1%
98.5%%+0.119
98.6% % +0.1 98.6%=-0.15
Pk ~ Dir(0.1)
93.3%10.4%%
95.4%.11..2%
96.6%
96.6 1...%1
B.5 USING HEAVIER MODEL
In this section, use heavier VGG-9 model for CIFAR- 10, since CIFAR-10 underfits in simple CNN model. Results are shown in Table 12 and For VGG-9 experiments, our method still works.
Table 12: Experimental results of different FL voting strategies with VGG-9 model. We repeat all experiments for three times.
Open-set (PROSER)
Open-set
Dataset
Partition
Close-set
FedOV
(PROSER DD)
#C =1
10.2%.10.3%
13.2%.10.6%
32.7%14..2%
46.4%
#C
36.2%+2..49
44.8%
38.6% 10.3%
42.0% 1.4%
CIFAR-I0
#C =
50.8%+11.05
51.1%+11.89
62.4%
63.0% 4.44%
Pk Dir(0.5)
14.7%
10.0%
30.7% 10.4%
80.5%10.5%
Pk Dir(0.1)
60.9%
63.7%
68.3%
70.6%=10.4%
Table 13: Experimental results of different FL voting strategies with VGG-9 model. We report the best top-k confidence voting accuracy. We repeat all experiments for three times.
Open-set
Open-set
Dataset
Partition
Close-set
FedOV
(PROSER)
(PROSER DD)
#C =1
10.2% +0.3%
10.6%%
43.1%.44.4
17.1%+ 1.5
#C
36.2% 12.4%
42.7%.14.5%
36.0% 12. 12.1%
59.0% 10.0%.4.0%
CIFAR-10
#C = 3
50.8% +1.0%
64.2%
66.3%.44 4..1%
71.9%++ +2.9%
Ph (or(
75.8% +4.7%
79.1%+0..6%
80.8% % +0.4%
80.8% 1-%6--00 -0.4%
Pk
(ir(0.1)
60.9%.11.5%
67.8%.10.88
71.0%
72.3% 11.4%
16

---

Published as a conference paper at ICLR 2023
B.6 EXPERIMENTS ON CIFAR-IO0 100 WITH RESNET-50
Besides VGG-9, we also experiment on ResNet-50 to verify the effectiveness of FedOV. We use CIFAR-100 test on more complicated datasets. Results are shown in Table 14 For the baselines of one-round FedAvg, FedProx, SCAFFOLD and FedNova, we omit them since their one-shot accuracy are much lower (see Table
17 FedKT trains multiple models in each client and has huge computation and storage cost when using ResNet-50, so omit it. Actually in FedKT paper (Li) ct al. 202 the authors do not experiment with heavy models like ResNet-50 either. Note that ResNet-50 has batch normalization layers. Therefore in each training batch, have to mix train data and generated outliers. Previously we compute a batch of train data and another batch of generated outliers separately. For ResNet-50, it can cause problem due batch normalization. To speed up computation, we only use data destruction strategy in our experiments for FedOV. Experimental results show that FedOV is better than close-set voting and FedDF for CIFAR- 100 with ResNet-50 model.
Table 14: Experimental results on ResNet-50. run 100 local epochs each client, and train the student model for 100 epochs during distillation. Client Number
Distilled FedOV
Distilled close-set set
Dataset
Partition
FedOV
Close-set
FedDF
Pk ~ Dir(0.5)
54.0%
31.7%
51.6%
30.7%
27.5%
10
Pk Dir(0.1)
47.3%
26.6%
44.7%
25.6%
22.5%
CIFAR-100
#C= ===
3.6%
2.2%
1.0%
1.0%
0.6%
100
#C' =2
9.4%
5.9%
4.8%
4.9%
1.2%
#C
13.4%
9.9%
10.0%
8.9%
1.6%
B.7 COMPARISON WITH CUTPASTE
We find that existing study proposed CutPaste (Li et al. 202 for outlier detection task, which generates outliers by applying operations on the training data. However, CutPaste contains limited operations and these operations cannot effectively corrupt the original features.
In this section, we compare our data destruction with Cutpaste (Li et al. 2021a) and explains why we do include Cutpaste in our main experiments.
First, we show the outliers generated by Cutpaste in Figure our data destruction in Figure 3a
Cutpaste outliers are less diverse than
7
11
7
4
4
7
4
2
Figure 6: Outliers generated by Cutpaste. The first column is real samples and the other columns are Cutpaste outliers.
Next, show some experimental results. We experiment simple CNN model under CIFAR- -10 setting. Mixture means for each image, we augment using our method and Cutpaste each with 50% chance. Results shown in Table 15 Even in cases, Cutpaste mixture slightly
17

---

Published as a conference paper at ICLR 2023
outperform our method, in other method greatly outperform them. Therefore, adding Cutpaste can hardly bring sigmificant improvement.
Table 15: Experimental results of Cutpaste and augmentation. We report the accuracy of counting votes from all 10 clients.
Open-set (PROSER +
Open-set (PROSER
Open-set (PROSER DD)
Dataset
Partition
Cutpaste)
mixture)
#C==1
17.8%-4-.22%
24.7% 1.77%
33.5%
1.23%%
#C =2
36.7%
39.0%
41.3% 17. %
CIFAR-10
#C =0=3
51.5%-tt3.2%%
52.8% +-2.4%
54.3% +22.
~ (ir(0.5)
67.7%::0.7%
68.5 % 10.5%
67.6%10.3%
Pk Dir(U.1)
57.5%:1...2%
59.4%
61.3% 1.0%
We also conduct similar experiments based on the metric of outlier detection. Our dataset includes MNIST, Fashion-MNIST and CIFAR- 10. We use one class when training and all classes of the same dataset when testing. The metric is ROC-AUC of the outlier probability. We run 200 local epochs with batch size 64, learning rate 0.001. For ROC-AUC, we report the average of last 10 epochs. Results are shown in Table 16 Our augmentation method still outperforms Cutpaste and mixture.
conclusion, our approach can generate more diverse outliers and achieve better accuracy compared with CutPaste.
Table 16: Experimental results of cutpaste and augmentation under common outlier detection metric
Open-set (PROSER +
Open-set (PROSER +
Open-set
Dataset
Known class
Cutpaste)
mixture)
(PROSER DD)
84.7%
87.3%
93.7 %
96.9%
98.1 1%
98.1 %
82.9%
86. 86.9%
89.4% %
87.4%
85.7%
89.2 %
81.7%
87.7%
93.1
FMNIST
89.7%
90.1%
91.7% %
73.4%
78. 1%
80.7 %
97.4%
97.6%
95.8%
84.8%
84.4%
91.3%
97.2%
97.8%
96.8%
Average
87.6%
89.4%
92.0
97.9%
98.6%
99.1 %
96.4%
95.5%
89.0%
84.7%
83.2%
97.8 %
82.3%
83.1 1%
94.
90.6%
92.8%
94.4%
MNIST
88. 2%
86.4%
95.7
96. 1 %
98.6%
98.9 %
86.8%
92.5%
92.8%
83.1%
90.0%
96.7 %
93.2%
95.1% 1%
98.1
Average
89.9%
91.6%
95.7 %
56.9%
60.8%
68. 1
48. 7%
72. 1%
81.8 %
58.0%
58.9%
66.4%
59.2%
62.6%
61.2%
58. 58.1%
59.0%
62.2%
CIFAR-10
69.1%
74.2%
72.2%
61.0%
66.0%
71.8%
62.2%
68.6%
76.3%
62.1%
63.6%
71.5%
54.7%
61.8%
73.8 %
Average
59.0%
64.8%
70.5 %
18

---

Published as a conference paper at ICLR 2023
B.8 MORE EXPERIMENTS ON SCALABILITY
In this section, we show results of scalability experiments on SVHN, Fashion-MNIST and MNIST. Except for datasets, other experimental settings are the same as in main paper. Results are shown in Table 17 FedOV still works well for scalability on SVHN, Rashion-MNIST and MNIST. Distilled FedOV can also outperform distilled close-set voting, FedAvg, FedProx and FedNova with single communication.
Both FedDF and distilled set voting can outperform FedAvg. Note that the ensemble of FedDF is based on average logits, while distilled close-set voting uses average probability, i.e. the softmax of logits. After softmax, the probability is limited in the range of [0.1]. However in FedDF, logits can be arbitrary real number. If a stupid local model outputs some very bad logits, it can greatly influence the ensemble of FedDF. This can probability explain why FedDF has different performance compared with distilled close-set set voting, although both approaches are similar. Especially when there 80 clients where it is more likely to have some very stupid local models, FedDF has worse performance than distilled close-set voting.
Table 17: Experimental results of different number of clients with simple CNN model.
Danaset
Number
Client
Partition
PedOv
Diaillal FedOV
Clase-set
Disiile close-set
FadAng
FedProx
PedNova
SCAFFOLD
FedDF
FedKT
#0
71.0%
67.1%
16.1%
16.4%
11.1%
8,4%
6.1%
5.8%
19.7%
6.7%
50.0%
45,9%
45.9%
36.1%
23.1%
19.0%
20.1%%
P5 Dir(0.5)
861%
81.7%
85.4%
3188
39.4%
36.1%
38.2%
30.2%
76.4%
73.0%
19 Dir(0.1)
82.7%
19.2%
14.0%
71.4%
22.5%
24.8%
23.5%
2005
66.3%
65.02
#0
77.8%
0,2%
6.5%
19.0%
9.75
6,2%
10.9
6.3%
0.75%
81.6%
78.38
51.3%
49.5%
19.9%
3.7%
19.0%
53.455
76.8%
19.0%
Ph. Dir(0.5)
86.1%
83.0%
84.7%
81.2%
21.1%
21.4%
23.3%
19.6%
76.9%
67.3%
P4
(NF((11)
81.1%
79.3%
72.0%
70.055
19.6%
19.09
15.25
19.0%
65.3%
20.3%
W12
50.2%
751175
7.3%
7:2%
19.0%
13.0%
6,2%
19.39%
6.6%
0,7%
#C
84.6%
82.1%
77.8%
75.0%
22.9%
19.7%
6.4%
19.6%
69.6%
42.0%
81.1%
81.0%
19 Dir(0.5)
34.3%
$1.9%
80.5%
7%
23.6%
26.2%
25.0%
19.8
68.4%
43.3%%
10. DIP(0.11
81.4%
77.0%
68.3%
06.05
19.5%
19.0%
23.9%
19.9%
55.8%
18.09%
#50
77.2%
14.1%
10.75
10020
10.0%
12.3%
10.0%
100%
10.00%
9.35%
71.4%
09.0%
41.9%
39.6%
10.0%
10.0%
10.0%
35.9%
34.8%
58.2%
37.2%
36.0%
P4 Dir(0.5)
87.5%
$1.0%
87.2%
33.6%
61.6%
63.3%
64.8%
$8.9%
80.3%
80.5%
P4 Dir(0.1)
72.7%
70.27%
66.7%
63.6%
10.1%
36.05
35.8%
31.4%
72.3%
67.
#0
75.7%%
14.1%
5.8%
3.8%
14.1%
17.4%
13.7%
14.2%
10.45
0/15%
75.3%
3%
18.8%
FMNIST
Ph (5r(0.5)
86.9%
82.65
86.2%
83.9%
58.4%
60.4%
58.59
42.05
79.95
77.0%
P4 DNP(0.1)
81.3%
78
14.4%
14.1%
25.1%
24.1%
27.9%
15.0%
64.2%
65.45%
#1
77.8%
70.0%%
10.0%
9.9%
10.00
11.0%
10.0%
13.3%
10.2%
9.39%
78.8%
75.2%
54.4%
52.8%
35.8%
35.5%
35.7%
18.2%
47.4%
69.8%
81.0%
66.2%
47.7%
31.9%
59.0%
P5 Dir(0.5)
85.3%
$3.5%
84.5%
83.2%
57.7%
57,0%%
61.1%
44.7%
73.9%
73.6%%
16 Dir(0.1)
703%%
16.8%
71.7%
60.97
32.4%
35.27
35.9%
18.0%
63.7%
18.6%
19.3%
11.4%
11.4%
11.2%
11.4%
10.19%
10.
86.1%
34.3%
25.6%
43.0%%
21.1%
24.9%
25.3%%
20.35
59.0%
56.1%
51,4%
41.6%
78.7%
38.4%
PL ~ Dir(0.5)
96.2%
97.8%
97.3%
97.2%
77.9%
74.4%
78.4%
66.7%
97.5%
95.5%
Ph Dir(0.1)
87.5%
85.2%%
83.2%
$1.9
44.0%
32.6%
45.2%
36.3%
94.0%%
30.2%%
W12
XX1%
$4.7%
10.4%
11.3%
13.5%
15.3%
9.3%
10.3%
10.3%
10.35
60.3%
02.0%
MNIST
97.5%
69.6%
50.1%
41.5%
Ph. Dir(0.5)
97.6%
97.1%
95.4%
97.2%
72.6%
12.8%
75.2%
72.5%
87.4%
92.7%
1% DiPQ0.1)
94.3%
03..5%
16
35.0%
-0.0%
43.9%
42.6%
$4.5%
70.3%
#00
39.3%
01.4%
10.8%
10.3%
11.4%
11.3%
11.2%
11.3%
12.1%
10.35
96.0%%
95.4%
67.19
63.9%
34.0%
30.9%
31.1%
14.9
58.2%
50.8%
81.2%
Ph. (12((055)
96.5%
94.2%
69.75
20.55
71.25
81.6%
85.655
P4. (7770011
94.3%
94.4%
86.1%
3030%
20.3%%
22.0%
41.15
300%
79.0%
36.7%
B.9 COMPARING DISTILLED FEDOV WITH FEDDF
We compare Distilled FedOV with FedDF under various label skews in Table 18 We have 10 clients and each client trains 100 local epochs. The model is simple CNN and we repeat all experiments three times with random seed 0,1,2 respectively. Pk ~ Dir(2) is the most IID partition, while Pk Dir(0.1) is the most non-lID partition. As in all settings. In the more non-lID setting Ps ~ Dir(0.1), the improvement of Distilled FedOV is significantly more than the other two settings.
see, Distilled FedOV outperforms FedDF
19

---

Published as a conference paper at ICLR 2023
Table 18: Comparison between distilled FedOV and FedDF under different levels of label skews.
Partition
Dataset
Distilled FedOV
FedDF
Improvement
MNIST
98.4%
97.6%11.55%
0.8%
SVHN
84.7%
82.7%
2.0%
Dir(2)
FMNIST
85.6%+0.55%
81.8%11.99
3.8%
CIFAR-I0 10
50.2%
43.3% 1.6%
9.9%
MNIST
40.3%
97.1%
0.6%
SVHN
81.4%
80.6%13.2%
1.2%
Pk Dir(0.5)
FMNIST 85.0% +..1%
50.6%.+..3
4.4%
CIFAR-10 51.6%
34.1%.1..
17.5%
MNIST 85.4%+1..7%
6.72%
5.2%
SVHN +1.0%+1.0%
61.7%177.%%
12.1%
Pk (0r(0.1)
FMNIST 73.4%
60.3%-19.69
13.1%
CIFAR-10 45.5 10.9%
22.9% 4.33%
22.6%
B.10 COMPARING WITH FEDAWS ON #C=1
For the setting where each client only has one class, FedAwS Yu et al. (2200) proposes spreadout regularization to push the embeddings of each class apart from each other to avoid all inputs collapsing to a single point. FedUV (Hosseini al. 202T) argues that FedAwS leaks the sensitive class embedding information to the server. The authors propose to use error-correcting codes to protect the embeddings and achieve similar accuracy compared with FedAwS. In FedUV paper (Hosseini et al. 2021). both FedU'V and FedAwS are compared with FedAvg with regular softmax cross-entropy loss function on user verification tasks. The authors conclude that FedAvg with regular softmax cross-entropy loss achieves the best accuracy in most cases, however regular FedAvg leaks class embeddings to the server and all other clients. FedUV focuses more on privacy protection and has lower accuracy than FedAvg, while we focus on accuracy and our FedOV has significantly outperformed FedAvg, therefore we do not compare with FedUV.
There are also works specifically for the real- world context of #C=1, such as federated face recognition (Liu al. 2022). However, it focuses on personalized FL, and each client can access public dataset to assist training, which is different from our settings. Therefore, we do not compare with it.
Next, we compare FedOV with FedAwS on #C=1 setting. We use the default k 10 in FedAwS paper. Since we only have 10 clients, it equals calculating the distance with all other class embeddings. We tune the best (10, 100} according to the authors' suggestions. We use the same squared hinge loss with hyper-parameter 0.9 as the original paper. FedAwS paper does not state the number of local epochs. From our experiments, one local epoch enough for convergence under #C=1. For successive epochs, the loss is almost zero and the accuracy does not improve. By default, for both FedAwS and FedAvg, we train local epoch with learning rate 0.001.
Results are shown in Table 19 As we can see, FedOV significantly outperforms FedAwS and FedAvg in one round. After one round, FedAwS and FedAvg achieve similar accuracy.
Table 19: Comparison between FedOV, FedAwS, and FedAvg under #C=1 partition. The model is simple CNN. We run three times for each setting.
Dataset
FedOV
FedAwS
FedAvg 10.1% 1 1.3%
MNIST
79.3%
1.8%
11.6%.1..9%
SVHN
64.5%
1.9%
19.6%
+0.0%
18.2% + 1.5%
FMNIST
73.3% + 1.6%
10.4%=0.7%
10.2%-0.394
CIFAR-I0
40.0% 1.7%
10.0%
11.3%=1.44 1.4%
B.11 PERFORMANCE OF DD AND AOE ON CENTRALIZED OSR EXPERIMENTAL SETTINGS
Our two techniques DD and AOE can also be extended to centralized OSR training augment the outliers. this section, we explore how they perform in centralized OSR experimental settings. clarifies why DD and AOE are more suitable and more important for FL. label skew settings.
20

---

Published as a conference paper at ICLR 2023
We utilize the common setting of OSR algorithm evaluation. For MNIST, Fashion-MNISII CIFAR-10 and SVHN datasets, we regard the first 2, 4, and classes as known during training. During testing, all 10 classes in the test set appear.
We compare PROSER+DD+AOE with PROSER, and run 200 epochs each. The model simple CNN. We show the results in Table 20 As AOE techniques can also bring improvement compared to the
see, under the OSR settings, adding DD and
PROSER algorithm in cases. When there are fewer known classes, the average improvement brought by our techniques is larger. For the experimental settings of many OSR papers (Zhou et al. [022]] Neal et al. 2018
Perera et al., 2020 Yoshihashi al. 2019). the OSR model is trained on at least known classes. The scenario of a limited number of classes (such as 2 classes) is not studied.
For PROSER, we can see a clear trend that when the number known classes decreases, the accuracy decreases. Since PROSER generates outliers between classes, when the number of known classes limited, the generated outliers are not diverse enough train good OSR model. Therefore, PROSER is inapplicable to scenarios with limited known classes. Under FL label skews, some parties may have limited data some labels due to privacy regulations and data distribution heterogeneity. Therefore, some clients may have limited known classes, e.g. only 1 class or classes. And since there are multiple clients, is unlikely to ensure that all clients have at least some number of classes. Such scenarios different from typical centralized OSR experimental settings and pose great challenges for PROSER as we also show in Figure 2 Our two techniques DD and AOE can tackle these challenges under the scenario of limited number of classes. There is no clear relation between the accuracy of adding our two techniques and the number of known classes. is because the task difficulty is not linear w.r.t the number of known classes. More known classes can be both a benefit and a challenge. The benefit is that more examples of diverse classes help generate more diverse outliers in in-between for better outlier detection. The challenge is that the task is more complicated as the model has to classify more seen classes.
In conclusion, although DD and AOE can be extended centralized OSR settings, they can bring more significant improvement to FL. and are more essential for FL. with label skews, compared with PROSER when the number of classes is limited, such as in FL label skew scenarios.
Table 20: Test accuracy comparison between PROSER+DD+AOE with PROSER under centralized OSR settings. "Unknown" is regarded as a new class besides the seen classes. We run three times for each setting.
# of known classes
PROSER
Average Improvement
Dataset
PROSER
Improvement
+DD+A0E
MNIST
84.2%- 75.9% 10.9%
51.1%
33.1%
SVHN
30.7%
45.2%
2
39.7%
FMNIST
-0.5%
25.2%
44.9%
CIFAR-10 10
60.2%
1400%
24.6% 11.2%
35.6%
MNIST
+5.%=1180
52.1%+5..
13.2%
SVHN
68.4%=2.4%
56.4%+00.5%
12.0%
4
15.9%
FMNIST
7.44%
50.2%+1..55
26.1%
CIFAR-10
57.6% +2.1%
45.5%-10.6%
12.1%
MNIST
70.9% 1.11%
61.9%
5.0%
SVHN
69.6%+1..3%
70.2%+0.995
-0.6%
3.8%
FMNIST
68.4%+2..2%
58.6%
10.1%
CIFAR-10
54.4% 10.7%
53.7%.10.3%
0.7%
MNIST
85.0%=1.00%
82.3%+11
2.7%
SVHN
70.2% 11.2%
78.6% % 0.6%
-8.4%
8
-2.0%
FMNIST
77.5% +022%
6.2%
CIFAR-10
44.9%
53.3 11.1%
-8.4%
B.12 COMPARING WITH BASELINES UNDER MULTIPLE ROUNDS
In this section, FedAvg. FedProx. SCAFFOLD, and FedNova for 200 communication rounds. There are 10 clients and each client trains 10 local epochs in each round. We compare FedOV with multi-round accuracy of baseline algorithms in Figure 7 As see, FedOV ensemble can achieve
21

---

Published as a conference paper at ICLR 2023
the accuracy of baseline algorithms at 50-75 communication rounds. In extreme label skew #C=1, the four baseline FL algorithms cannot beat FedOV even after 200 communication rounds.
Feedy
Fastug
0.30
6.5
Feshog
Festooe
375
375
(a) #C=1
(b) #C=2
Fechag
Feshag
Featres
Feshras
Festoos
375
372
(c) #C=3
(d) Dir(0.1)
66
66
Fecheg
Feshag
62
Festrus
Festoos
372
372
(e) (Or(0.5)
(l) IID
Figure 7: Comparing FedOV (one-shot FL accuracy) with baseline algorithms running multiple rounds. We experiment six different partitions of CIFAR-10 dataset.
22


Reviews:
[{"review_id": "sfT5wXOyNV", "reviewer": "Reviewer_ZcP3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the label skew problem in one-shot federated learning. The author identifies the problem is attributed to the limited scale of certain class in different participants, which brings unreliable voting scores and thus results in ineffective global model. In this paper, authors propose FedOV, which leverage generated diverse outliers as additional unknown class to increase voting efficiency. Extensive experiments are conducted on MNIST, Fashion-MNIST, CIFAR-10 and SVHN datasets.", "summary_of_the_review": "This paper investigates an interesting problem, label skew in one-shot federated learning. But the proposed method is lack of novelty, seems the directly combines of existing strategies. Besides, authors do not provide in-depth rationale analysis and spend a large space of article to incorporate methodology into existing methods.\n\nPost-rebuttal:\nThe authors provide comprehensive analysis in the response. Some issues have been clarified. I would like to raise my score to 6. However, some unclear parts should be clarified in the final version (if accepted).", "strength_and_weaknesses": "Pros:\n+ It is intriguing to introduce the concept of unknown class into federated learning and investigate a novel problem open set voting federated learning\n+ The proposed method achieves new SOTA results on widely used benchmarks, which demonstrates the effectiveness of the proposed method.\n\nCons:\n- The authors propose data destruction and adversarial outlier enhancement modules to respectively generate outliers from true samples and learn a tighter boundary. However, these two components seem not specifically aims for missing class issue in label skew one-shot fl. It would be better to further consider the relation with federated learning setting.\n\n- The compared methods such as fedavg, fedprox, fednova are not designed for one-shot fl, it is nature that these methods perform worse on these experiments (Table1). So, it is not sure whether it is fair comparision.\n\n- Typing Errors\n\n1) ‘are used as part’  ‘are used as the part’\\\n2) ‘the above method’ ‘the above methods’\n3) ‘train 100 epochs each client’ ‘train 100 epochs for each client’\n4) ‘which causes massive’ ‘which cause massive’\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "nnSfbZbVn59", "reviewer": "Reviewer_hyjU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel approach named FedOV which generates diverse outliers and introduces them as an additional unknown class in local training to address the label skew issue in one-shot Federated Learning (FL). Concretely, it proposes novel outlier generation approaches by corrupting the original features and further develops adversarial learning to enhance the outliers based on open-set recognition.\n\nThe main contributions of this paper are:\n\n- This paper points out that the voting-based FL approaches are either directly used as a final model for predictions or distilled as a single model, thus failing to produce high-quality federated learning models.\n\n- This paper may be the first time to propose open-set voting in FL by introducing the “unknown” class, which significantly improves the accuracy compared to traditional close-set voting in FL.\n- This paper proposes two techniques (i.e., data destruction and adversarial outlier enhancement) for generating diverse “unknown” outliers without requirement on the number of classes of the training data.\n\n", "summary_of_the_review": "I believe this paper is innovative and puts forward some new perspectives in the field of federated learning. However, I have concerns about whether this work can be accepted in its current form. I will update my reviews based on the authors' responses.", "strength_and_weaknesses": "### Strength\n\n- The problem studied in this paper is interesting and valuable. Specifically, label skews is an essential issue that exists in one-shot federated learning and is urgent to be solved.\n- This paper provides some novel perspectives. Applying Open-Set Recognition (OSR) into FL to introduce an unknown class to improve the voting is a novel view for overcoming label skews in one-shot FL.\n- This paper conducts extensive experiments and demonstrates the advantages of the proposed method over the state-of-the-art LDL methods, which can effectively reflect the performance of the algorithm.\n- The paper is well-organized and clearly written, which is easy to follow.\n- The code is provided in the attachment for reproduction.\n\n### Weaknesses\n\n- In section 1, the author mentioned, \"For example, in an extreme case where each client only has one label (e.g., face recognition), all clients predict the input as its own label and the voting result is meaningless\". This case is a typical situation studied in this paper. However, there exist some works handling this case that are not mentioned in this paper, such as [1]. I think it is necessary to discuss and compare the proposed algorithm with these methods.\n\n- In Table 3, one can notice that the performance of Distilled FedOV is worse than FedDF. The authors should analyze the reason for this phenomenon.\n\n[1] Yu, Felix, et al. \"Federated learning with only positive labels.\" *International Conference on Machine Learning*. 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bA_1A6YLYzq", "reviewer": "Reviewer_Y1Mp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose a new approach for one-shot federated learning, by generating diverse outliers and introduces them as an additional unknown class in local training to improve the voting performance. The paper proposes novel outlier generation approaches by corrupting the original features and further develop adversarial learning to enhance the outliers. The experimental results demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "In short, the paper is novel and technically sound and easy to understand. The experimental results demonstrate the effectiveness of this paper.", "strength_and_weaknesses": "Pros:\n\n- The paper is technically sound and easy to understand.\n\n- The method of introducing unknown class is novel and reasonable in FL learning.\n\n- The experimental results show the effectiveness of the proposed method.\n\nCons:\n\n- Lack of details of the experimental setting of distilled FedOV.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster