Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: -This paper studies the semi-supervised federated learning problem, where the server has access to a limited number of labeled data while the clients have access to massive unlabeled data. They claim that directly combining existing semi-supervised learning methods such as FixMatch and MixMatch with standard federated learning methods such as FedAvg cannot perform well, since the server and clients models are trained in parallel and pseudo labels for the unlabeled data are generated by the trained models for every batch, and they theoretically show that its performance depends on the quality of the augmented pseudo-labeled data: the higher quality, the better risk bound. Then, they propose to train the labeled server and the unlabeled clients in an alternate manner: they generate pseudo-labels only once upon receiving the server model and fine-tune the server model by additional training with labeled data. They also provide some favorable experimental results in the paper.

strengths_and_weaknesses: -------------------------------
pros:

-The paper considers a practical federated learning problem and is motivated by relevant real-world examples: in a healthcare system, a central hub (server) has domain experts and a limited number of labeled data, while many rural branches have non-experts and massive unlabeled data. It would be more convincing if the effectiveness of the proposed method can also be verified on those real-world datasets.

-------------------------------
cons:

-The writing of the paper may not be clear, and many confusing parts need to be modified. For example, in lines 38-39, it is said that it is **infeasible** to directly combine the federated learning method (FedAvg) with semi-supervised learning methods. This statement may be too strong, as discussed in lines 158-159 some methods e.g., FedMatch can do this.

-Also in lines 131-133, it is said that a successful semi-supervised learning method **must** be able to generate more and more high-quality pseudo-labels during training…This statement may also be too strong as in standard semi-supervised learning, the unlabeled data are used for regularization and the good generalization is based on the cluster assumption, the manifold assumption…

see section 1.2 in the following textbook:

Chapelle, Olivier, Bernhard Schölkopf, and Alexander Zien. Semi-Supervised Learning. MIT Press.

Such a statement may only hold for the pseudo-label generation-based semi-supervised learning approaches.

-Some abbreviations firstly appear without the full words and citations, see lines 114-116, FixMatch, FedSGD, MixMatch…

-**Lack of convergence results**. The proposed method is based on alternating training, which may be unstable in some situations. It would help make the paper stronger if some convergence results of the proposed algorithm can be given.

questions: -In the definition of adequate transmission (line 140-144), how to quantify the reliable information exhibited from unlabeled data, can any information-theoretic measures be used here? Otherwise, the definition looks very unclear to me.

-In Figure 1, it seems that the unlabeled clients can use different models (illustrated by smartphone vs tablet vs laptop…), which may cause some misunderstandings since in line 106 it is assumed that all models share the same architecture. 

-In Table 2, it seems that fine-tuning the server model with labeled data helps improve performance the most (almost 40% improvement). But in practice, the number of labeled data may vary. Any results or insights on how many labeled data are sufficient for the fine-tuning?

-The proposed method is expected to improve the quality of generated pseudo-labels during training in an alternating training fashion. It would be more convincing if the results of the generated pseudo-labels during training can be provided.

-The proposed method introduces several hyperparameters, for example, the global confidence threshold tao which needs to be preselected by the clients. This may be difficult since the clients only receive the global model from the server and may not have other prior knowledge, how to select tao in practice?

limitations: -The paper has many unclear parts and needs to be carefully modified.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work studies an FL setting that clients have completely unlabeled data and can train multiple local epochs, while the server has a small amount of labeled data. For this, author(s) investigate the semi-supervised federated learning scenario and proposes SemiFL method. The focus of the semi-supervised learning (SSL) strategy is o data augmentation-based SSL on a class of nonparametric kernel-based classification learning. The author(s) theoretically methods to the illustrate critical region where significant improvement can be made from unlabeled data. Then, the author(s) propose to augment dataset with pseudo labels by adopting two SSL methods developed in centralized learning: FixMatch and MixMatch. It employs two strategies: fine-tune global model with labeled data and generating labels with global model, to improve the quality of pseudo labeling. Experiments are conducted to validate the proposed method and present clear improvement over the selected baselines.

strengths_and_weaknesses: Strengths:
1.  The paper is easy to follow and the problem is well-motivated.
2.  The proposed method seems very effective, as indicated by the results.

Weakness:
1. It lacks a sufficient connection between the theoretical results and the contribution of the proposed method over the existing FL + SSL methods [27,28].
2. Insufficient hyperparameter study. Although extensive ablation studies are included, it would be helpful to study the confidence threshold in Fixmatch, $a$ in Mixmatch, and $\lambda$ in convex combination on the loss functions.
3. Fair comparison. sBN proposed in HeteroFL [9] is not a technique specifically suitable for the proposed SemiFL. Do the baseline methods also employ sBN? 
4. The author(s) mention that the proposed training strategy "delay the computation time." Understanding this limitation without offering the additional computation cost over the total training time is not straightforward.
questions: Please see the weakness. Plus: 
1. The author(s) studied the non-IID data on the client-side. Will the label distribution skewness on the server-side affect SemiFL?
2. How will the theoretical results be generalized to a neural network classifier?
3. Is there any warm-up training to generate the pseudo labels?
4. The reported results on FedMatch seem much lower than those reported in the original FedMatch paper. Could the author(s) please explain the possible reasons?
5. "We use Wide ResNet28x2 [43] as our backbone model for CIFAR10 and SVHN datasets and Wide ResNet28x2 for CIFAR100 datasets throughout our experiments."  What is the reason for mentioning Wide ResNet28x2 twice if all the datasets use the same network architecture? 
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper tackles a practical problem of federated learning where labeled data is completely unavailable at local clients whereas the global server has a small set of labeled data. This paper proposes the SemiFL method, which finetunes global models with labeled data while generating pseudo-labels with the global model at local training phase. This paper demonstrates the effectiveness of their proposed model compared to the existing SSFL, FL, and SSL methods.

strengths_and_weaknesses: Clarity: (High) This paper is well-organized and easy-to-read. The authors smoothly develop their logic and key concepts throughout the paper. The figures are illustrated clearly. 

Quality: (High) This paper focuses on the practical problem of FL. Its theoretical analysis is convincing. The experiments are carefully designed.  

Originality: (Low) I think the critical weakness of this paper is the originality. Their proposed training algorithms seems to be 1) highly based on existing SSL methods  i.e, losses from FixMatch and Mixup, and 2) the alternate training is a bit simple to consider as a novel and significant approach.

Significance: (Moderate) The performance improvement compared to the existing SSFL methods seems significant. However, considering the originality of the proposed method, it is hard to say the overall significance of this paper is strong.

questions: Please cite the prior works correctly (not arXiv). i.e. FedRGD is published at IEEE International Conference on Big Data (2021), FedMatch is published at International Conference on Learning Representations (2021). 

Please clarify the description of prior works in 3.3 Alternate Training. For “Vanilla communication efficient FL with SSL” methods, it is correct that FedRGD [28] performas aggregation of server and client model in parallel as you described in Fig 4 (a). For FedMatch [27], however, as far as I know, it performs disjoint learning (a bit similar to the alternate training) which trains the global model with labeled data at server “after” local training with unlabeled data is done for the labels-at-server scenario. However, in line 157 to 161, it is not correctly described and so readers may be confused while reading the part.

limitations: No negative societal impact is expected.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes SemiFL to address the problem of Semi-Supervised Federated Learning (SSFL). SemiFL alternates training of labelled server data and unlabeled clients to maintain pseudo-labeling quality during training, allowing unlabeled clients to train multiple local epochs. This method uses labelled data to fine-tune the global model and generate pseudo-labels.
strengths_and_weaknesses: Strength
-	The proposed method is clear and easy to understand.
-	The paper is well-organized.

Weakness
-	The novelty of this paper is low, just a simple application of Fixmatch/Mixmatch in federated learning.
-	Missing important FSSL baselines, e.g., FedBYOL and FedU.
-	The theoretical proof given in this paper is only an analysis of data augmentation. 

questions: -	Please explain the novelty of SemiFL. Are there any technological challenges? 
-	The proposed method SemiFL leverages the unlabeled data on local clients. However, in real applications, the data in local clients are private, so the server has no information about numbers and categories of data included in clients. And maybe the total amount of data is imbalanced or categories are not included in the server. How will SemiFL handle this situation? The authors should discuss these issues.
-	The proposed model of this paper mainly focused on the model level. How to deal with the possible in/consistency in the feature space?

limitations: -	The comparison of accuracy results of SSL methods is not essential, because the proposed method only use Fixmatch and Mixmatch in this paper. Some FSSL baselines are missing, like FedBYOL, FedU, etc. Please include the results of more recent baseline methods.
-	Please supplement theoretical analysis to support the model designed, e.g., convergence analysis.
review_title: null

Meta-review: metareview: There was some disagreement in reviewer scores for this paper, which proposes an algorithm for semi-supervised federated learning. The recurring concerns centered on a few points: (i) the lack of theoretical analysis of the proposed algorithm, (ii) the apparent disconnect of the presented analysis from the setting under consideration, (iii) limited technical novelty, (iv) potentially missing baselines, and (v) lack of clarity in the text. Of these, the response to (i), (iii) and (v) seem reasonable: certainly a systematic analysis of a new algorithm is welcome, but hardly straightforward; simple ideas are to be commended if they definitively solve a challenging problem; and the proposed edits do go some way to addressing the reviewer concerns.

Points (ii) and (iv) remain less clear. For the former, it is a bit confusing to present a generic analysis of SSL techniques, which does not appear to directly influence the proposed algorithm design. For the latter, there are some missing references at a minimum, e.g., Zhang et al., "Benchmarking Semi-supervised Federated Learning"; Itahara et al., "Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data". Further, some of the results in Table 1 do not seem to align with what has been previously reported in the literature; e.g., Table 1 of the FedMatch paper. However, it appears that the authors have at least compared against the most recent Fed-SSL techniques, with unsupervised and self-supervised techniques being not directly in scope.

Overall the paper is a borderline case. The recurring critique of limited depth suggests that a compelling empirical analysis would be appropriate. The explicit evaluation of Fed-SSL methods against a labelled-only baseline, and showing existing methods cannot outperform this, is interesting. The paper could be strengthened by compressing Sec 3.2 and focussing instead on a better understanding of why existing Fed-SSL methods underperform. Such insights would be more broadly insightful to the community, and be a useful contribution.
confidence: Less certain
Decision: Accept