SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
Authors: Enmao Diao
Jie Ding
Vahid Tarokh
Venue: NeurIPS 2022 Accept (2022)
Keywords: Federated Learning
Semi-Supervised Learning

Abstract: Federated Learning allows the training of machine learning models by using the computation and private data resources of many distributed clients. Most existing results on Federated Learning (FL) assume the clients have ground-truth labels. However, in many practical scenarios, clients may be unable to label task-specific data due to a lack of expertise or resource. We propose SemiFL to address the problem of combining communication-efficient FL such as FedAvg with Semi-Supervised Learning (SSL). In SemiFL, clients have completely unlabeled data and can train multiple local epochs to reduce communication costs, while the server has a small amount of labeled data. We provide a theoretical understanding of the success of data augmentation-based SSL methods to illustrate the bottleneck of a vanilla combination of communication-efficient FL with SSL. To address this issue, we propose alternate training to 'fine-tune global model with labeled data' and 'generate pseudo-labels with the global model.' We conduct extensive experiments and demonstrate that our approach significantly improves the performance of a labeled server with unlabeled clients training with multiple local epochs. Moreover, our method outperforms many existing SSFL baselines and performs competitively with the state-of-the-art FL and SSL results.

SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
Enmao Diao
Jie Ding School of Statistics University of Minnesota-Twel Cities Minneapolis, MN 55455, USA Zingj@umn. edu
Department of Electrical and Computer Engineering
Duke University
Durham, NC 27705, USA
enmao enmao.diace www.oduke.ee..oo edu
Vahid Tarokh
Department of Electrical and Computer Engineering
Duke University
Durham, NC 27705, USA
vahid.taroun twrokkrdukeeeee.oo edu
Abstract
Federated Learning allows the training of machine learning models by using the computation and private data resources of many distributed clients. Most existing results on Federated Learning (FL) assume the clients have ground-trutt labels. However, in many practical scenarios, clients may be unable to label task-specific data due lack of expertise or resource. propose SemiFL to address the problem of combining communication-efficienn FL such as FedAvg with Semi- Supervised Learning (SSL). In SemiFL,, clients have completely unlabeled data train multiple local epochs reduce communication costs, while the server has a small amount of labeled data. We provide a theoretical understanding of the success of data augmentation- based SSL methods illustrate the bottleneck of a vanilla combination of communication-effica we propose alternate training to 'fine-tune global model with labeled data' and "generate pseudo-labels with the global model." We conduct extensive experiments and demonstrate that our approach significantly improves the performance of labeled server with unlabeled clients training with multiple local epochs. Moreover, our method outperforms many existing SSFL baselines and performs competitively with the state-of-the-arr FL and SSL results. Our code is available here
FL with SSL. To address this issue,
Introduction
For billions of users around the world, mobile devices and Internet of Things (IoT) devices are becoming common computing platforms (I]. These devices produce large amount of data that be used to improve a variety of existing applications (2). Consequently, it has become increasingly appealing process data and train models locally from privacy and economic standpoints. To address this, distributed machine learning framework of Federated Learning (FL.) has been proposed 4 This method aggregates locally trained model parameters in order produce global inference model without sharing private local data.
Most existing works of FL focus supervised learning tasks assuming that clients have ground- truth labels. However, in many practical scenarios, most clients may not be experts in the task of interest to label their data. In particular, the private data of each client may be completely unlabeled. For instance, a healthcare system may involve central hub ("server") with domain experts and a
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Expert
Labeled
Data
Unlabeled
Clients
Figure 1: A resourceful server with labeled data significantly improve its learning performance by working with distributed clients with unlabeled data without data sharing.
limited number of labeled data (such medical records), together with many rural branches with non-experts and a massive number of unlabeled data. As another example, an autonomous driving startup ("server") may only afford beta-users assistance in labeling road condition but desires to improve its modeling quality with the information provided by many decentralized vehicles that are not beta-users. The above scenarios naturally lead to the following important question: How a that hosts a labeled dataset can leverage clients with unlabeled data for supervised learning task in the Federated Learning setting?
We propose new Federated Learning framework SemiFL to address the problem of Semi-Superrised Federated Learning (SSFL) illustrated in Figure I We discover that it is challenging directly combine the state-of-the-art SSL methods with the communication efficient federated learning methods such FedAvg to allow local clients train multiple epochs (4). The key ingredient that enables SemiFL to allow unlabeled clients to train multiple local epochs is that we alternate the training of a labeled server and unlabeled clients ensure that the quality of pseudo-labeling is highly maintained during training. In particular, we fine-tune the global model with labeled data and generate pseudo-labels only with the global model. We perform extensive empirical experiments to evaluate and compare our method with various basclines and ofte--of-thearr techniques. The results demonstrate that our method can outperform existing SSFL methods and perform close the state-of- -fet--rr- of FL and SSL results. In particular, we contribute the following.
We propose SemiFL in which clients have completely unlabeled data and can train multiple local epochs to reduce communication costs, while the server has small amount of labeled data. We identify the difficulty of combining communication efficient FL method FedAvg with the sfate-of-the-at SSL methods.
We develop theoretical analysis on strong data augmentation for SSL methods, the first in the literature to our best knowledge. We provide a theoretical understanding of the success of data augmentation-bree based SSL methods to illustrate the bottleneck of vanilla combination of communication eefiiiiionnnfiinnnntttei FL with SSL.
To the best our knowledge, we propose the first communication efficient SSFL method alternate training that can improve the performance of a labeled server by allowing unlabeled clients to train multiple local epochs, i.e., from 42% to 88% with 250 labeled data, and from 77% to 93% accuracy with 4000 labeled data the CIFAR1O 10 dataset. Moreover, our proposed method achieves 30% improvement over the existing SSFL methods. Furthermore, SemiFL performs competitively with the state-of-ther of-tte-art FL methods and SSL methods. i.e., only 1% and 2% away from the state-of-the-art FL and SSL results, respectively, for 4000 labeled data on the CIFARIO dataset.
The outline of the paper is given below. In Section we review the related work. In Section 3 we identify the problem combining SSL with communication-efficii theoretical analysis of how strong data augmentation can significantly improve the classification methol accuracy, and present the proposed SemiFl. method with some intuitive explanations. Section we evaluate the empirical performance of the SemiFL. We make some concluding remarks in Section
FL methods, develop

---

2 Related Work
Federated Learning The goal of Federated Learning is to scale and speed up the training of distributed models 6 FedAvg
[4] allows local clients to train multiple epochs to facilitate convergence. FedProx (Li al., 2018) performs proximal regularization against global weights. FL counterparts of Batch Normalization (79) are developed to further enhance the performance. The use of local momentum and global momentum [T0] have been shown facilitate faster convergence. FedOpt proposes federated versions of adaptive optimizers to improve performance over FedAvg.
Semi-Supervised Learning Semi Superisue Learning (SSL) refers the general problem of learning with partially labeled data, especially when the amount of unlabeled data is much larger
than that of the labeled data [12 13 The idea of self-training (namely obtain artificial labels
for unlabeled data from a pre-trained model) can be traced back to decades ago (1415). Pseudo- [ is a form of entropy minimization (18) by converting model predictions into hard labels. Consistency regularization [19) refers to training models via minimizing the distance among stochastic outputs [1 19 A theoretical analysis of
labeling [16], component of many recent SSL techniques [17).
consistency regularization was recently developed in [20]. More recently, It has been demonstrated
that the technique of strong data augmentation can lead to better outcomes [21 Strongly
augmented examples are frequently found outside of the training data distribution, which has been
shown to benefit SSL [25].
Semi-Supervised Federated Learning (SSFL) Most existing FL works focus on supervised learn- ing tasks, with clients having ground-truth labels. However, in many real-world scenarios, most clients are unlikely to be experts in the task of interest, an issue raised a recent survey paper (26]. In the research line of SSFL, the work [27] splits model parameters for labeled server and unlabeled clients separately. Another related work (28) trains and aggregates the model parameters of the labeled server and unlabeled clients in parallel with group-wise reweights. Applications of SSFL to
specific applications can be found in, e.g.,
29 30).
3 Method
3.1 Problem
In a supervised learning classification task, we are given a dataset D {x,, Di Ji where x; is a feature vector, Y/ is one-hot vector representing the class label in K-class classification problem, and N is the number of training examples. In a Semi- Supervisde Learning classification task, we have two datasets, namely a supervised dataset S and an unsupervised dataset 4. Let S = {xrt y No be a set of Ns labeled data observations, and 4 = {-r ) be set of No unlabeled observations (without the corresponding true label v()) It is often interesting to study the case where Ns < Nu.
In this work, we focus Semi-Supervised Federated Learning (SSFL) with unlabeled clients, as illustrated in Figure ! Assume M clients and let Tu.m denote the set of unsupervised data available at client 1, 2, M. Similarly, let (x., Vv) denote the set of labeled data available at the server. The server model is parameterized by model parameters W.. The client models are parameterized respectively by model parameters { (W.11..... W. ..... }. We assume that all models share the same model architecture, denoted by f (Ix, w) + f(x,, w). which maps an input x and parameters W to a vector the K -ddimensional simplex, e.g., using softmax function applied model outputs.
Communication Efficient FL with SSL In the standard communication efficient FL scenario where clients can train multiple local epochs before model aggregation (i.e., FedAvg (4), existing SSFL methods have difficulty in performing close to the sfate-of-the-ar centralized SSL methods [27 28 31). In fact, will demonstrate in Table I that existing SSFL methods cannot outperform the case of training with only labeled data. This is somewhat surprising given that their underlying methods of training unlabeled data are similar.
As shown in Figure 2 SSL methods such FixMatch can only work with FedSGD, which requires batch-wiss wise gradient aggregation and thus is not communication efficient. This is because SSL methods, such as FixMatch and MixMatch, sample from both labeled and unlabeled datasets for every batch of training data with carefully tuned ratio 32 Thus, it is not straightforward how we can combine the SSL. method in a communication-effiieee FI. scenario where we train multiple local epochs. To understand the bottleneck of this vanilla combination, need to understand better
3

---

Accuracy
60
Fully Supervised
Fully Supervised
Fartially Supervised
Partially Supervised
FedSGD RaMatch
20
FedSGD FixMatch
FedAvg RiMatch
FedAug FidMathh
600
700
Communication Rounds
Communication Rounds
(4)
Non-tiD,
2 data partition and Ns = 4000 with a vanilla combination of communication efficient FL with SSFL methods. The "Fully Supervised" and "Partially Supervised" refer training centralized model with full and 4000 labeled data respectively.
Figure 2: Results of CIFARIO dataset with (a) IID and (b) Non-IID, K
why the state-of-the-ar centralized SSL methods work. In section 3.2 we analyze the strong data augmentation for SSL and demonstrate that the success of FixMatch due to using data augmentation on pseudo-labeled data of high quality.
3.2 Theoretical Analysis of Strong Data augmentation for SSL
Pseudo-labelinn is widely used for labeling unlabeled data in SSL methods [16 22 32). However, the quality (accuracy) of those pseudo-labela can be low, especially at the beginning of the training. In this light, several papers [2 32 propose hard-threshold sharpen the pseudo-labels to improve the quantity of accurately labeled pseudo-labels. The problem with hard thresholding is that the data samples satisfying the confidence threshold have small training loss. Therefore, the model cannot be significantly improved as already performs well on the data above the threshold. To address this issue, we will use strong data augmentation [25]
generate data samples that have larger training loss. The main idea is to construct a pseudo-labeling mechanism whereby our SSL method can generate more and more high-quality pseudo- during training. Meanwhile, the augmented data for model training can produce a more considerable drop in training loss than the original data.
To provide further insights into SSL, develop theoretical analysis of the strong data augmentation, which is critical component of the state-of-the-an SSL method FixMatch (32] and SemiFL, and can be interesting in its own right. Intuitively, strong augmentation is process that maps data point (e.g., an image) from high quality to relatively low grade unilaterally. The low-quality data and their high-confidence pseudo-labels are then used for training that there sufficient "observations" in the data regime insufficiently covered by labeled data.
Our theory is based on intuitive "adequate transmission" assumption, which means that the distribution augmented data from high-confidence unlabeled data can adequately cover the data regime of interest during prediction. Consequently, reliable information exhibited from unlabeled data can be "transmitted" to data regimes that may have been insufficiently trained with labeled data, attention to
illustrated in Figure Instead of studying SSL in full generality, we restrict class of nonparametric kernel-based classification learning (33]35) and derive analytically tractable statistical risk-rate analysis. More detailed background and technical details are included in the Appendix. We provide a simplified statement as follows.
Theorem Under suitable assumptions, an SSL classifier Gual trained from ny, unlabeled data and the strong data augmentation technique has a statistical risk bound at the order of RR R(Cist) nu (attt////((t3pp--dd where d, q. a, P are constants that describe the data dimension, smoothness of the conditional distribution function (Y X)), ), class separability (or task difficulty), and inadequacy of transmission, respectively. The smaller p. the better risk bound. Moreover, suppose that O' is the classifier trained from ny labeled data, where ny ~ ni, E (0, 1). It can be verified that the bound of a(a++3))
3
R(Ō") is much smaller than that of R(C')) when Ç q(a+3+ This provides an insight into the critical region of ny where significant improvement be made from unlabeled data.

---

optimal decision boundary
2:mic)
high-confidance pseudo-fabal
P
boundary
unlabeled data
augmented
X
probability of unlabeled data
probability of labeled data
P.
Figure 3: Illustration of the strong data augmentation-basee SSL. We pick up an unlabeled point (X P.,) with high-confidence pseudo-label, obtain its tard-thresholded label (Y, which is believed to be close to the ground truth), maneuver X into X (which is believed to represent the test distribution to some extent), and then treat (Ý, X) as labeled data for training. Consequently, reliable task-specific information exhibited from unlabeled data can be transmitted to data regimes that may have been insufficiently trained with labeled data. Note that IP denotes the labeled data distribution as well as the out-sample test data distribution (used to evaluate the learning performance). The above ideas are theoretically formalized Subsection 3.2 and Appendix
3.3 Alternate Training
As depicted in Figure 4 existing SSFL works follow the state-of-the-art SSL methods to syn- chronize the training of supervised and unsupervised data [23. For example, FedMatch (27] and FedRGD (28) adopt vanilla combination of FedAvg and FixMatch. They aggregate the server model trained from labeled data and clients' models trained from unlabeled data at each communi- cation round in parallel and generate pseudo-labels for each batch of unlabeled data with the local training model. However, existing results [27 28 indicate that this vanilla combination has difficulty performing close the state-of-the-art SSL methods, even if the unlabeled clients are trained with the same SSL methods.
Aggregate server and ciients models in parallel
(k.)
Generate pseudo-labels with global model
W!
Generate pseudo-iabels with training model
Client (RL1)
Client (sa))
Fine-tune global with labeled
Client 1 (#..))
Client 1 (KL))ve)
Server (R, Be)
W!
Client M (suar)
Client (x.))
W
Client M (Rusr)
Client M
W
(a) Vanilla communication efficient FL with SSL
(b) Alternate Training (Ours)
Figure 4: An illustration of (a) vanilla combination of communication efficient FL and SSL, and (b) Alternate Training (Ours). (a) The vanilla combination trains and aggregates server and client models in parallel and generates pseudo-labels with the training models for every batch of unlabeled data. (b) Alternate Training fine-tunes the aggregated global model with labeled data and generates pseudo-labels only once upon receiving the global model from the server.
In the communication efficient FL setting, we cannot guarantee an increase in the quality of pseudo- labels during training because we allow local clients to train multiple epochs, potentially deteriorating the performance (see Figure 2). Furthermore, the aggregation of a server model trained with ground- truth labels and subset of client models trained with pseudo-labels does not constantly improve the performance of the global model over the previous communication round. A poorly aggregated global model of the last round produces worse-quality pseudo-labels. Subsequently, the performance of the aggregated model degrades in the next round.
5

---

To maintain and improve the quality of our generated pseudo-labele during training, we propose to train the labeled server and unlabeled clients in an alternate manner. as illustrated in Figure Ib). In particular, our approach consists of two important components:
Fine-tune global model with labeled data At each round, the server will retrain the global model with the labeled data. In this way, the server can provide a comparable or better model than the previous round for the active clients in the next round generate pseudo-labels. On the contrary, the vanilla method aggregates server and client models in parallel. As result, the quality of generated pseudo-labels will gradually degrade and thus deteriorate the performance.
Generate pseudo-labels with global model We will label the unlabeled data once the active clients immediately receive the global model from the server. This way, pseudo-labela quality will not degrade during the local training. On the contrary, the vanilla method labels every batch of data during the training of unlabeled clients. As result, the quality of generated pseudo-labels will gradually degrade during local training, thus deteriorating performance.
Our proposed approach ensures that the clients can continually generate better quality pseudo-labels during training. Our experimental studies show that the proposed method can significantly improve the performance of the labeled server and performs competitively even with the state-of-the-an FL and centralized SSL methods. The limitation of our approach is that we need to update the aggregated client model with labeled data from the server, which will delay the computation time. We will conduct an ablation study on each component alternative training in Table
3.4 The SemiFL Algorithm
We summarize the pseudo-code of the proposed solution in Algorithm 1 server will first update the model with the standard supervised loss L, for local epochs with data batch (I&. yb) of size B, randomly split from the supervised dataset D,, using
At each iteration t, the
La ((((s))), W.). W. W. V.,
where a(-) represents weak data augmentation, such as random horizontal flipping and random cropping, that maps one image to another. Subsequently, the server updates the static Batch Normal- ization (sBN) statistics (9) (which is discussed in Appendix B). Next, the server distributes server model parameters W, to a subset of clients. We denote the proportion of active clients at each communication round & as activity rate C, E (0. 1]. Without loss of generality, we assume that C. = a constant over time. After each active local client, say client m, receives the transmitted W., it generates pseudo-labels Yu.m as follows:
W Wn,m u W.: Fu,m f(((u,, W m.
Each local client will construct a high-confidence dataset Dfix u.m inspired by FixMatch 32 at each iteration t, defined as:
Ddx M2
((rumm Mu,m) with max (J 2 2T.
for a global confidence threshold < < pre-selected by all clients. If for some client m, we have Dlix u.m = 0 then it will stop and refrain from transmission to the server. Otherwise, we will sample with replacement to construct a dataset inspired by MixMatch [23]. In other words,
Dmix
Sample
[Dfx
with replacement (Fam Va.m)
where |Diix u.m denotes the number of elements of Díx u.m Thus Dimis DDix Subsequently.
client m trains its local model for E epoch to speed up convergence 4.. For each local training
13mix epoch of the client m, it randomly splits local data Dfx u, Dmix into batches 13fx a,m. a. of size B,.-
For each batch iteration, as in [36], client m constructs Mixup data from one particular data batch (af. vfx),, (aggix ymix) by
Amix Beta(a, a), Fmix +
Amix
Bx + (1 Amix )agix
where a is the Mixup hyperparameter. Next, client m defines the "fix" loss Lax 32 and "mix" loss
Tamix 24 by
Lix = ((f(A(=))), W W, D
Lmix Amix ((f((((m))))) W fxx + +11 Amix) -E() ((f(a(((mmii W Ww, vo ))
(5)

---

Algorithm/ Semi-Supervised Federated Learning with Alterntte Training for Unlabeled Clients
Input: Unlabeled data Fu, 1:M distributed M local clients, activity rate C, the number of com- munication rounds T, the number of local training epochs E, server and client respective batch sizes Bg and Bm local learning rate 7/. server model parameterized by W. client models parameterized by IW....... W.,Mh }, weak data augmentation function a(-), strong data augmentation function A(-), confidence threshold т, Mixup hyper-parameter a, loss hyperparameter X, common model architecture function f(-)
System executes:
for each communication round t = 1.22.... ... T'do do
W -SrreeUpdaat((r,, Us, W?)
Update the sBN statistics
St < max(IC J. active clients uniformly sampled without replacement
for each client
S; in parallel do
Distribute server model parameters to local client m, namely W 1,m +- W
Wi a,m f ClientUpdate( (Tumm, W u,m
end
Receive model parameters from M, clients, and calculate W M, Eme W Wam end
W S-r-SreeUrUrteee (2., Mas W
Update the sBN statistics
Us. ServerUpdate ys. W,):
Construct supervised dataset D, = (I., for each local epoch from
E do
Bg +- -Randomly split local data D, into batches of size B, for batch (Ir. yb) E B3 do
L. - [(f(((x)) Wz). W. - W. 1/ LL
end end
Return W.
ClientUpdate m, W w)
Generate pseudo-labels with weakly augmented data a(Fum)) namely
Yu,m
f(a(Iu,m), W.,m m.))
Construct FixMatch dataset, namely Dūx (Iuum Vu,m) with max (Yu,m) 2 r}
If I u. = then Stop. Return.
Construct an equal-size Mixup dataset, namely
Dmix
umm Sample |Dix with replacement( (Fr,m Fumm)
for each local epoch e from E
B3xx Bu,m 13mix Bunix <- Randomly split local data Dfix Dmix into batches of size Bix Bmix u,m=
for batch (a!
. v1bx), (arf : E B Buix do
Amix Beta(a,
Tmix + Amix fix + +(1- Amix )agii
LAx +- {((((((zfi)) Wum)),hh 1)
Lini t- Amix ((((xx)) W Wm,m), (xx) + Amix) {(f(a((mix) W.,m): shix)
W Wm,m W u, 1/ V w (T + X T Temix)
end
end
Return W Wamm and send the server
Here, A represents a strong data augmentation mapping, c.g., the RandAugment [37 used in our experiments, and is often the cross entropy loss for classification tasks. Finally, client performs gradient descent step with
W umm
W um
1/V w (Lus + X Lmnx);
(6)
where À > 0 a hyperparameter set to one our experiments. After training for E local epochs, client m transmits W m to the server.
7

---

90
90
80
70
Arcuracy
Accuracy
70
50
IID
Non-lID, Dir(0.3)
Non-IID, Dir(0.3)
50
Non-IID, Dir(0.
Non-IID, Dir(0.1)
Non-IID, ==
40
Non-IID, x
100 200 300 400 500 600 700 800 Communication Rounds
100 200 300 400 500 600 700 800 0
Communication Rounds
[a] Ml
(b)
Figure Results of CIFARIO dataset with (a) Ns 250 and (b) Ng 4000.
Without loss of generality, assume that clients 1,2,... My have sent their models to the server at time t. The server then aggregates client model parameters {W..1...., W..M, } by W, M," EmL Wu.m. u. (4). This process is then repeated for multiple communication rounds T. After the training is finished, the server will further fine-tune the aggregated global model by additional training with the server's supervised data using its supervised loss L.. Finally, it will update the sBN statistics one final time.
Experiments
Experimental setup To evaluate our proposed method, we conduct experiments with CIFARIO, SVHN, and CIFAR100 100 datasets [3] To compare our method with existing FL and SSFL methods, we follow the standard communication efficient FL setting, which was originally used in FedAvg (4] and widely adopted by following works, such 40 41). We have 100 clients throughout our experiments, and the activity rate per communication round is = 0.1. We uniformly assign same number of data examples for IID data partition to each client. For balanced Non-IID data partition, we ensure each client has data at most from K classes and the sample size each class is the same. We set K 2 because it is the most label-skewed case for classification, and it has been evaluated in (9 40 41] 41). For unbalanced Non-IID data partition, we sample data for each client from Dirichlet distribution Dir(a) [41 42 As 20, it reduces IID data partition. We perform experiments with a = (0.1,0.3)
To compare our method with the ofte--of-the-art SSL methods, we follow the experimental setup [2]. We use Wide ResNet28x2 (43) as our backbone model for CIFARIO and SVHN datasets and Wide ResNet28x8 for CIFAR 100 datasets throughout our experiments. The number of labeled data at the server for CIFAR 10, SVHN, and CIFAR (00 datasets Nss are (250, 4000), (100, 2500), and {2500, 10000) respectively. We conduct four random experiments for all the datasets with different seeds, and the standard errors are shown inside the parentheses for tables and by error bars in the figures. We demonstrate experimental results in Table D and the learning curves of CIFARIO 10, SVNH, and CIFAR100 datasets in Figure and Further details are included in the Appendix.
Comparison with SSL methods We demonstrate the results of Fully Supervised and Partially Supervised cases and existing SSL methods for comparison in Table The Fully Supervised case refers all data being labeled, while in the Partially Supervised case, we only train the model with the partially labeled Ns data. Our results significantly outperform the Partially Supervised case. In other words, SemiFL. can substantially improve the performance of a labeled server with unlabeled clients in cummunication-ccficciii SSL methods for IID data partition. Moreover, it is foresecable that the clients become more label-skewed for Non- IID data partition, the performance of our method degrades. However, even the most label-skewed unlabeled clients can improve the performance of the labeled server using our approach. A limitation of our work is that as the supervised data size decreases, the performance of SemiFL degrades more than the centralized SSL methods. We believe it is because cannot train labeled and unlabeled data simultaneously in one data batch.
scenario. Our method performs competitively with the state-of-the-ae art
Comparison with FL and SSFL methods We compare our results with the state-of-the-art FL and SSFL methods in Table I We demonstrate that SemiFl. can perform competitively with the ofte--ofthheart FL result trained with fully supervised data. It is worth mentioning that SSFL may
8

---

Table 1: Comparison of SemiFL with the Baselines, SSL, FL, and SSFL methods. SemiFL improves the performance of the labeled server, SemiFL significantly outperforms the existing SSFL methods, and performs close to the state-of-the-an FL and SSL methods.
Datasct
CIFARIO
SVHN
CIFARI00
Number of Supervised
250
4000
250
1000
2500
10000
Fully Supervised
95.3(0.1)
97.3(0.0)
79.3(0.1)
Baseline
Partially Supervised
42.4(1.8)
76.9(0.2)
77.1(2.9)
90.4(0.5)
27.2(0.7)
59.3(..1)
11-Model
13
45.7(4.0)
86.0(0.4)
810((9))
92.5(0.4)
42.8(0.5)
62.1(0.1)
Pseudo-Labeling
44
50.2(0.4)
83.9(0.3)
79.8(1.1)
90.1(0.6)
42.6(0.5)
63.8(0.2)
Mcan Teacher
49
67.7(2.3)
90.8(0.2)
96.((..1)
96.6(0.1)
46.1(0.6)
64.2(0.2)
SSL.
MixMatch
89.0(0.9)
93.6(0.1)
96.0(0.2)
96.5(0.3)
60.100.4)
71.7(0.3)
UDA
22
91.2(1.1)
95.1(0.2)
94.3(28)
97.5(0.2)
66.9(0.2)
75.5(0.3)
ReMix.Match
24
94.6(0.1)
95.310.1)
97.1(0.5)
97,4(0.1)
72.6(0.3)
77.0(0.)
FixMatch
94.9(0.7)
95.7(0.1)
97.5(0.4)
97.7(0.1)
71.7(0.1)
77.4(0.1)
FL
HeteroFL
9
51.5(3.6)
72.3(4.4)
3.1(0.3)
Non-IID, =
FedMatch
27
41.3(1.1)
58.3(10)
58.2(3.1)
84.3(1.0)
17.7(0.5)
30.5(0.8)
SSFL
FedRGD
28
32.7(3.6)
48.9(1.4)
21.2(2.2)
21.6(2.3
13.8(14)
26.5(3.0)
SemiFl.
60.0(0.9)
85.3(0.3)
87.5(1.1)
92.2(0.8)
35.2(0.3)
62.1(0.4)
FL
HeteroFL
85.0(0.))
95.8(0.1)
74.0(0.4)
Non-IID, Dir(0.1)
FedMatch
27
41.6(1.))
58.9(0.7)
58.4(3.4)
84.3(0.6)
17.5(0.5)
30.8(0.))
SSFL
FedRGD
31.5(2.9)
45.2(0.8)
20.0(4.0)
23.8(3.4)
13.4(1.3)
23.6(2.6)
SemiFL
63.000.6)
84.5(0.4)
91.2(0.3)
93.000.5)
49.0(1.0)
68.0(0.2)
FL
HeteroFI,
9
91.6(0.1)
96.8(0.0)
76.9(0.1)
Non-IID, Dir(0.3)
FedMatch FedRGD
41.2(111)
58.4(0.6)
59.1(.88
84.0(1.1)
17.8(0.4)
31.1(0.5)
SSFL
28
32.5(3.0)
46.9(1.6)
24.8(5.1)
22.0(3.9)
13.1(2.0)
23.8(L9)
SemiFL
71.9(1.))
88.9(0.3)
94.0(0.5)
95.2(0.2)
54.9(14)
70.0(0.3)
FT.
HeteroFI,
9
94.3(0.1)
97.5(0.0)
77.8(0.2)
FedMatch
27
41.7(1.))
58.6(0.5)
58.6(3.0)
84.3(0.9)
17.6(0.3)
31.3(1))
SSFL
FedRGD 28
33.2(1.9)
47.8(1.7)
21.3(6.5)
20.7(1.1)
13.3(1.)
23.8(2.6)
SemiF
88.2(0.3)
93.1(0.1)
96.8(0.3)
96.9(0.1)
61.3(1.2)
72.1(0.2)
outperform FL methods in the Non-IID data partition case because the server has a small set of labeled IID data. We also demonstrate that our method significantly outperforms existing SSFL methods. Existing SSFL methods fail to perform closely to the state-of-the centralized SSL methods, even their underlying SSL methods are the same. Moreover, existing SSFL methods cannot outperform the Partially Supervised case, indicating that they deteriorate the performance of the labeled server. In particular, FedMatch allocates disjoint model parameters for the server and clients, and FedRGD assigns a higher weight for the server model for aggregation. Both methods do not directly fine-tune the global model with labeled data and generate pseudo- labels with the received global model. To our best knowledge, the proposed SemiFL. is the first SSFL. method that actually improves the performance of the labeled server and performs close to the coute-of-the-e rrt and SSL. methods.
Ablation studies We conduct ablation studies on SemiFL and demonstrate the results in Table 2 Based on our extensive experiments, it is evident that "Fine-tune global model with labeled data" and "Generate pseudo-labels with global model" are the critical components of the proposed 'Alternate Training' method for the success of our method. We also conduct an ablation study on static Batch Normalization(sBN), the number of local training epochs, the Mixup data augmentation, and global SGD momentum. The detailed results can be found in the Appendix.
4.1 Quality of Pseudo Labeling
We measure the quality of Pseudo-Labeling for Semi-Supervised Learning from three aspects, including the accuracy of pseudo-labels (Pseudo Accuracy), the accuracy of thresholded pseudo- labels (Threshold Accuracy), and the ratio of pseudo-labeled data (Label Ratio) with CIFAR dataset in Figure 6 Pseudo-Labeling. The results demonstrate that our proposed alternative training, the combination of 'Fine-tune global model with labeled data" and 'Generate pseudo-labels with global model," can produce pseudo labels of much better quality when clients have completely unlabeled data and train multiple local epochs.
We perform ablation studies on our proposed method by measuring the quality of

---

Table 2: Ablation study on each component of alternative training with CIFARIO dataset. The combination of "Fine-tune global model with labeled data" and "Generate pseudo-labels with global model" significantly improves the performance.
Method
Fine-tune global model with labeled data
Generate pscudo-labels with global model
Accuracy
Non-IID, K
IID
Fully Supervised
95.33
Partially Supervised
N/A
76,92
FedAvg FixMatch
X
41.01
40.26
X
48.89
47.03
SemiFL
80.42
81.70
85.34
93.10
200
1.0
Threchaldos Accuracy
80
Pscuco Accuracy
0.8
60-
0.6
Later
40
Fine Tune, Global Average, Global Fine Tune, Training
Global Average Global
0.4
Fine Tune, Glbbal Average, Global
20-
20
Fine Tune, Training
Fine Tune, Training Average, Training
Training
Average Taining
0.0
100 0
200 300 400 500 600 700 800 Communication Rounds
0
100
200 300 400 500 600 700 800 Communication Rounds
100
200 300 400 500 600 700 800 Communication Rounds Laac Ratie ID
(0) Pscudo Accurocy ID
thrisholded Accureo 10
80
Threshoided Accuracy
80
1.0
Pocudo Accunsy
0.8
60
0.6
40-
40
0.4
Tune, Global
Tune, Global Average, Global Tune, Training Average. Training
Average Global
20-
Averoge, Global
0.2
0-
Averege Training
Tuno, Training
0-
Average Training
Tune. Treining
0.0
200
700 800
0 100 200 300 400 500 600 700 800
100
200 300
Communication Rounds (d) Preudo Accurayy (K-2)
Communication Rounds
Communication Rounds in) label Rotio, Non-ID
(4] Iirsohaldes Accunscy Nonvll (0-1)
Figure 6: Ablation studies of alternative training by measuring the quality of Pseudo Labeling with CIFARIO dataset. 'Fine Tune' and 'Global' refer to our proposed method, 'Fine-tune global model with labeled data' and 'Generate pseudo-labels with global model,' respectively. 'Average' refers to the vanilla FL method, which directly takes the average of the model parameters of the labeled server and unlabeled clients. 'Training' refers to generating pseudo-labels at each batch of local training.
5 Conclusion
In this work, we propose new communication-effin effficienteii Federated Learning (FL) framework named SemiFL to address the problem of Semi--uperre Supervised Federated Learning (SSFL) for unlabeled clients. We identify the difficulty of combining communication-efficeen Federated Learning (FI.) with state-of-the-art Semi-Super Learning (SSL). We develop theoretical analysis of strong data augmentation for SSL, which illustrates the bottleneck of vanilla combination. We propose to train the labeled server and unlabeled clients in an alternate manner by 'fine-tune global model with labeled data' and 'generate pseudo-labels with global model.' We utilize several training techniques and establish a strong benchmark for SSFL. Extensive experimental studies demonstrate that our communication-eficee
method can significantly improve the performance of labeled server with unlabeled clients. Moreover, we show that SemiFL can perform competitively with the state-of-the centralized SSL and fully supervised FL methods. Our study provides practical SSFL framework that extends the scope of FL applications.
Acknowledgments
The work of Enmao Diao and Vahid Tarokh was supported by the Office of Naval Research (ONR) under grant number N00014-18-1-2244, The work of Jie Ding was supported by the National Science Foundation (NSF) under grant number DMS-2134148.
10

---

References
[1] Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Qiang Yang, Dusit Niyato. and Chunyan Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, 2020.
Liang.
[2] Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint 1rXivv181..0300
2018.
[3] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv: 1et.:056100 2016.
[4] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficieee learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR, 2017.
[5] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H Brendan McMahan, et al. Towards federated learning scale: System design. arXiv preprint arXiv: 01060, 2019.
[6] Chaoyang He, Songze Li, Jinhyun So, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, and Salman Avestimehr, Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13:: 135/88 2020.
[7] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning, In International Conference Machine Learning, pages 4387-4398. PMLR, 2020.
[8] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623, 2021.
[9] Enmao Diao, Jie Ding, and Vahid Tarokh. HeteroFL.: Computation and communication effi- cient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2021.
[10] Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. proving communication-effciiee distributed sgd with slow momentum. arXiv preprint arXiv:1910.0 00643, 2019.
Slowmo: Im-
[11] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021.
[12] Zhi-Hua Zhou and Ming Li. Tri-training: Exploiting unlabeled data using three classifiers.
IEEE Transactions on knowledge and Data Engineering, 17(11) 11:11)11:1111111 1541,, 2005.
[13] Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko. Semi- supervised learning with ladder networks. arXiv preprint wxXiv:557.00277 2015.
[14] Henry Scudder. Probability of error of some adaptive pattern-recognitiion machines. IEEE Transactions on Information Theory, 1133:3363-37, 1965.
[15] Geoffrey J McLachlan. Iterative reclassification procedure for constructing an asymptotically op- timal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70(350):3653366 1975.
[16] Dong-Hyon Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. ICML, volume 3, 2013.
[17] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and -upervupe learning. IEEE transactions pattern analysis and machine intelligence, 41(8):1999-1993 2018
[18] Yves Grandvalet, Yoshua Bengio, et al. Semi-supervised learning by entropy minimization. In CAP, pages 281-296, 2005.

---

[19] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo ppeudo-ensemble. arXiv preprint 1412.4444,,4864, 2014.
[20] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self self-training with deep networks on unlabeled data. In International Conference on Learning Representations,
2021.
[21] Geoffrey French, Michal Mackiewicz. and Mark Fisher. Self-ensembling for visual domain adaptation. arXiv preprint arXiv:/706.05208, 2017.
[22] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc Le. Unsupervised data augmentation for consisteney training. arXiv preprint urXiv:/904. 12848, 2019.
[23] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprins arXiv:/905.0222 2019.
[24] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: supervipe learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv: 1.000978 2019.
[25] Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Ruslan Salakhutdinov. Good semi-supervised learning that requires a bad gan. arXiv preprint 1
2017.
[26]
Yilun Jin, Xiguang Wei, Yang Liu, and Qiang Yang. Towards utilizing unlabeled data in federated learning: A survey and prospective. arXiv e-prints, pages arXiv-2002, 2020.
[27] Wonyong Jeong. Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. In International Conference on Learning Representationss 2021.
[28] Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph E. Gonzalez, Kannan Ramehandran, and Michael W. Mahoney. Improving semi-supervised federated learning by reducing the gradient diversity of models. In 202/ IEEE International Conference on Big Data (Big Data), pages 1214-1225, 2021.
[29] Yuchen Zhao, Hanyang Liu, Honglin Li, Payam Barnaghi, and Hamed Haddadi. Semi- supervised federated learning for activity recognition. arXiv preprint arXiv:201 1. 1.00851, 2020.
[30] Dong Yang, Ziyue Xu, Wenqi Li. Andriy Myronenko, Holger R Roth, Stephanic Harmon, Sheng Xu, Baris Turkbey, Evrim Turkbey, Xiaosong Wang, et al. Federated semi-supervised learning for covid region segmentation in chest ct using multi-national data from china, italy, japan. Medical image analysis, 10:101992, 2021.
[31] Zewei Long, Liwei Che, Yaqing Wang, Muchao Ye, Junyu Luo, Jinze Wu, Houping Xiao, and Fenglong Ma. Fedsemi: An adaptive federated semi-supervised learning framework. arXiv preprint arXiv:2012.000 032992, 2020.
[32] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001. 20X0:20001.0665,, 2020.
[33] Jvan-Yves Audibert and Alexandre B Tsybakov. Fast learning rates for plug-in classifiers under the margin condition. arXiv preprini math/0507180, 2005.
[34] Michael Kohler and Adam Krzyzak. On the rate of convergence of local averaging plug- in classification rules under a margin condition. IEEE transactions on information theory, 1335)11735 2007.
[35] Luc Devroye, László Györfi, and Gábor Lugosi. A probabilistic theory of pattern recognition, volume 31. Springer Science & Business Media, 2013.
[36] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
[37] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[38] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
12

---

[39]
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[40] Paul Pu Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov. and Louis- Kouis-Philipppe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint w7Xiv:2001.0152, 2020.
[41] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2021.
[42] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.063 00009,0633, 2019.
[43] Sergey Zagoruyko and Nikos Komodakis. arXiv: 07,66, 2016.
Wide residual networks.
arXiv preprint
[44] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. arXiv preprint arXiv:1703.01780 2017.
[45] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 167..01607.0002, 2016.
[46] Yuxin Wu and Kaiming He. Group normalization. In Proceedings the European conference on computer vision (ECCV), pages 3-19, 2018.
[47] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXfv:1607.06450, 2016.
[48] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[49] Tehrim Yoon, Sumin Shin, Sung Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning, In International Conference on Learning Representations, 2021.
[50] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622. 2020.
[51] Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9():1441-14, 1964.
[52] Geoffrey S Watson. Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, pages 359-372, 1964.
[53] László Györfi, Michael Kohler, Adam Krzyžak, and Harro Walk. A distribution-free theory of nonparametric regression, volume 1. Springer, 2002.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See last sentences of Section 3.3
4,
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not foresee any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section D.3 in Appendix.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section D.5 in Appendix.
3. If you experiments...
13

---

(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as a URL)? [Yes] We provide source codes in the supplementary material. We use publicly available datasets.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section C.I in Appendix.
(c) Did you report error bars (c.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] The error bars are shown in figures and the numerical standard егтоr are shown in brackets of tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] One Nvidia 1080TI enough for one experiment run.
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the publicly available datasets use.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We use publicly available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We use publicly available datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "Wu9AxShUlld", "reviewer": "Reviewer_sMTb", "summary": "This paper tackles a practical problem of federated learning where labeled data is completely unavailable at local clients whereas the global server has a small set of labeled data. This paper proposes the SemiFL method, which finetunes global models with labeled data while generating pseudo-labels with the global model at local training phase. This paper demonstrates the effectiveness of their proposed model compared to the existing SSFL, FL, and SSL methods.\n", "questions": "Please cite the prior works correctly (not arXiv). i.e. FedRGD is published at IEEE International Conference on Big Data (2021), FedMatch is published at International Conference on Learning Representations (2021). \n\nPlease clarify the description of prior works in 3.3 Alternate Training. For “Vanilla communication efficient FL with SSL” methods, it is correct that FedRGD [28] performas aggregation of server and client model in parallel as you described in Fig 4 (a). For FedMatch [27], however, as far as I know, it performs disjoint learning (a bit similar to the alternate training) which trains the global model with labeled data at server “after” local training with unlabeled data is done for the labels-at-server scenario. However, in line 157 to 161, it is not correctly described and so readers may be confused while reading the part.\n", "limitations": "No negative societal impact is expected.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Clarity: (High) This paper is well-organized and easy-to-read. The authors smoothly develop their logic and key concepts throughout the paper. The figures are illustrated clearly. \n\nQuality: (High) This paper focuses on the practical problem of FL. Its theoretical analysis is convincing. The experiments are carefully designed.  \n\nOriginality: (Low) I think the critical weakness of this paper is the originality. Their proposed training algorithms seems to be 1) highly based on existing SSL methods  i.e, losses from FixMatch and Mixup, and 2) the alternate training is a bit simple to consider as a novel and significant approach.\n\nSignificance: (Moderate) The performance improvement compared to the existing SSFL methods seems significant. However, considering the originality of the proposed method, it is hard to say the overall significance of this paper is strong.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BZdn8whOo12", "reviewer": "Reviewer_Y1ff", "summary": "This paper proposes SemiFL to address the problem of Semi-Supervised Federated Learning (SSFL). SemiFL alternates training of labelled server data and unlabeled clients to maintain pseudo-labeling quality during training, allowing unlabeled clients to train multiple local epochs. This method uses labelled data to fine-tune the global model and generate pseudo-labels.", "questions": "-\tPlease explain the novelty of SemiFL. Are there any technological challenges? \n-\tThe proposed method SemiFL leverages the unlabeled data on local clients. However, in real applications, the data in local clients are private, so the server has no information about numbers and categories of data included in clients. And maybe the total amount of data is imbalanced or categories are not included in the server. How will SemiFL handle this situation? The authors should discuss these issues.\n-\tThe proposed model of this paper mainly focused on the model level. How to deal with the possible in/consistency in the feature space?\n", "limitations": "-\tThe comparison of accuracy results of SSL methods is not essential, because the proposed method only use Fixmatch and Mixmatch in this paper. Some FSSL baselines are missing, like FedBYOL, FedU, etc. Please include the results of more recent baseline methods.\n-\tPlease supplement theoretical analysis to support the model designed, e.g., convergence analysis.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n-\tThe proposed method is clear and easy to understand.\n-\tThe paper is well-organized.\n\nWeakness\n-\tThe novelty of this paper is low, just a simple application of Fixmatch/Mixmatch in federated learning.\n-\tMissing important FSSL baselines, e.g., FedBYOL and FedU.\n-\tThe theoretical proof given in this paper is only an analysis of data augmentation. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7OwY7yyhGGt", "reviewer": "Reviewer_rcxY", "summary": "-This paper studies the semi-supervised federated learning problem, where the server has access to a limited number of labeled data while the clients have access to massive unlabeled data. They claim that directly combining existing semi-supervised learning methods such as FixMatch and MixMatch with standard federated learning methods such as FedAvg cannot perform well, since the server and clients models are trained in parallel and pseudo labels for the unlabeled data are generated by the trained models for every batch, and they theoretically show that its performance depends on the quality of the augmented pseudo-labeled data: the higher quality, the better risk bound. Then, they propose to train the labeled server and the unlabeled clients in an alternate manner: they generate pseudo-labels only once upon receiving the server model and fine-tune the server model by additional training with labeled data. They also provide some favorable experimental results in the paper.\n", "questions": "-In the definition of adequate transmission (line 140-144), how to quantify the reliable information exhibited from unlabeled data, can any information-theoretic measures be used here? Otherwise, the definition looks very unclear to me.\n\n-In Figure 1, it seems that the unlabeled clients can use different models (illustrated by smartphone vs tablet vs laptop…), which may cause some misunderstandings since in line 106 it is assumed that all models share the same architecture. \n\n-In Table 2, it seems that fine-tuning the server model with labeled data helps improve performance the most (almost 40% improvement). But in practice, the number of labeled data may vary. Any results or insights on how many labeled data are sufficient for the fine-tuning?\n\n-The proposed method is expected to improve the quality of generated pseudo-labels during training in an alternating training fashion. It would be more convincing if the results of the generated pseudo-labels during training can be provided.\n\n-The proposed method introduces several hyperparameters, for example, the global confidence threshold tao which needs to be preselected by the clients. This may be difficult since the clients only receive the global model from the server and may not have other prior knowledge, how to select tao in practice?\n", "limitations": "-The paper has many unclear parts and needs to be carefully modified.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "-------------------------------\npros:\n\n-The paper considers a practical federated learning problem and is motivated by relevant real-world examples: in a healthcare system, a central hub (server) has domain experts and a limited number of labeled data, while many rural branches have non-experts and massive unlabeled data. It would be more convincing if the effectiveness of the proposed method can also be verified on those real-world datasets.\n\n-------------------------------\ncons:\n\n-The writing of the paper may not be clear, and many confusing parts need to be modified. For example, in lines 38-39, it is said that it is **infeasible** to directly combine the federated learning method (FedAvg) with semi-supervised learning methods. This statement may be too strong, as discussed in lines 158-159 some methods e.g., FedMatch can do this.\n\n-Also in lines 131-133, it is said that a successful semi-supervised learning method **must** be able to generate more and more high-quality pseudo-labels during training…This statement may also be too strong as in standard semi-supervised learning, the unlabeled data are used for regularization and the good generalization is based on the cluster assumption, the manifold assumption…\n\nsee section 1.2 in the following textbook:\n\nChapelle, Olivier, Bernhard Schölkopf, and Alexander Zien. Semi-Supervised Learning. MIT Press.\n\nSuch a statement may only hold for the pseudo-label generation-based semi-supervised learning approaches.\n\n-Some abbreviations firstly appear without the full words and citations, see lines 114-116, FixMatch, FedSGD, MixMatch…\n\n-**Lack of convergence results**. The proposed method is based on alternating training, which may be unstable in some situations. It would help make the paper stronger if some convergence results of the proposed algorithm can be given.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7E1JgMiGRqm", "reviewer": "Reviewer_zPN3", "summary": "This work studies an FL setting that clients have completely unlabeled data and can train multiple local epochs, while the server has a small amount of labeled data. For this, author(s) investigate the semi-supervised federated learning scenario and proposes SemiFL method. The focus of the semi-supervised learning (SSL) strategy is o data augmentation-based SSL on a class of nonparametric kernel-based classification learning. The author(s) theoretically methods to the illustrate critical region where significant improvement can be made from unlabeled data. Then, the author(s) propose to augment dataset with pseudo labels by adopting two SSL methods developed in centralized learning: FixMatch and MixMatch. It employs two strategies: fine-tune global model with labeled data and generating labels with global model, to improve the quality of pseudo labeling. Experiments are conducted to validate the proposed method and present clear improvement over the selected baselines.\n", "questions": "Please see the weakness. Plus: \n1. The author(s) studied the non-IID data on the client-side. Will the label distribution skewness on the server-side affect SemiFL?\n2. How will the theoretical results be generalized to a neural network classifier?\n3. Is there any warm-up training to generate the pseudo labels?\n4. The reported results on FedMatch seem much lower than those reported in the original FedMatch paper. Could the author(s) please explain the possible reasons?\n5. \"We use Wide ResNet28x2 [43] as our backbone model for CIFAR10 and SVHN datasets and Wide ResNet28x2 for CIFAR100 datasets throughout our experiments.\"  What is the reason for mentioning Wide ResNet28x2 twice if all the datasets use the same network architecture? ", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.  The paper is easy to follow and the problem is well-motivated.\n2.  The proposed method seems very effective, as indicated by the results.\n\nWeakness:\n1. It lacks a sufficient connection between the theoretical results and the contribution of the proposed method over the existing FL + SSL methods [27,28].\n2. Insufficient hyperparameter study. Although extensive ablation studies are included, it would be helpful to study the confidence threshold in Fixmatch, $a$ in Mixmatch, and $\\lambda$ in convex combination on the loss functions.\n3. Fair comparison. sBN proposed in HeteroFL [9] is not a technique specifically suitable for the proposed SemiFL. Do the baseline methods also employ sBN? \n4. The author(s) mention that the proposed training strategy \"delay the computation time.\" Understanding this limitation without offering the additional computation cost over the total training time is not straightforward.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept