Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: -This paper studies the problem of federated learning with semi-supervised client (local) data. They propose a method that first pre-trains the client models with their labeled training data, then performs cross-client label propagation based on the embeddings of the labeled and unlabeled training data obtained from the pre-trained models, and after obtaining the pseudo labels and per-sample weights standard federated training is applied. They also provide some experimental results demonstrating the effectiveness of this method.
strength_and_weaknesses: Pros

-The writing is clear and the paper is easy-to-follow. The authors attempt to solve a practically important question and show some favorable empirical results.

Cons

-There are several issues that remain unsolved in the paper. The first concern is about novelty. To utilize the semi-supervised training data, the proposed method simply (in terms of methodology) employs the widely used label propagation in federated training. Although in practice, in order not to violate the federated training protocols (do not share data between clients) by using cross-client data information (embeddings), the authors introduce some cryptographic tools which are claimed to be fully secure, but they do not provide any privacy guarantees.

-The second concern is about feasibility. Label propagation is not free. We may not accurately estimate the data manifold with only limited non-iid data and thus label propagation may not be reliable. The authors propose to solve this problem by doing so on embeddings of the training data from all clients. However, we still cannot verify the manifold assumption in this case and can only observe from the empirical performances. This raises another concern about the convergence of the proposed method (if the label propagation is not reliable). Existing experimental results are based on benchmark datasets and the semi-supervised data are artificially designed. It would be useful to test the proposed method with real-world non-iid semi-supervised client data and see if label propagation works well.

clarity,_quality,_novelty_and_reproducibility: -The paper is clearly written and easy to follow. There are some concerns regarding the novelty and feasibility, see above.
summary_of_the_review: -From the reviewer's point of view, this paper provides some interesting empirical results but more theoretical investigations are needed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents  Federated Label Propagation (FedProp), an algorithm, to effectively learn semi-supervised models by distributing both labeled and unlabelled data across multiple clients.  Only model parameters are shared across clients through a server (following a client server architecture) but not the data, which is in fact one of the principles of federated learning. FedProp algorithm takes into account existing tools/techniques such as  Label propagation (LP) and Locality Sensitive Hashing (LSH) as part of its solution. 
 
In summary, FedProp  first initializes the global solution by learning client-wise independent solutions from corresponding labeled data (that is assigned to a client) and finally running an aggregation step for parameters shared by clients. In addition to the aggregation, the server also acts as an intermediator to share parameters across clients through a broadcasting operation whenever necessary. At each iteration clients are responsible for independently assigning pseudo labels to its portion of unlabeled data and subsequently use those as new/additional labeled data to update its parameters before broadcasting to the server. The above process (pseudo labeling, parameter update and broadcasting and aggregation) iterates for a predefined number of iterations and is expected to converge (at the end). 

The proposed technique has been tested on CIFAR-10, CIFAR-100 and Mini-Imagenet datasets and compared against existing techniques such as FedAvg, FeMatch, FedSem, and FedSiam. Reported results  are found to be comparable. 

strength_and_weaknesses: Strengths: Federated Label Propagation (FedProp) uses Label propagation (LP), Locality Sensitive Hashing (LSH), and some other established semi supervised tools/techniques and combines them in a structured fashion to guide semi supervised learning. The experiments and reported results look promising. Also, an analysis of the correctness, efficiency and robustness of their FedProp algorithm has been addressed in section 4.2. 
 
Weakness: The results reported in section 5 don’t include any statistical tests so it's difficult to verify whether the results make any significant difference to existing techniques. The proposed methodology also shows some limitations in terms of scalability (as per the results in section 5): The reported gain in performance reduces when tested for a larger (CIFAR-100 vs CIFAR-10) or a more complex datasets (Mini-Imagement -100).  

Another weakness/limitation is this work lacks major technical contributions; It uses existing tools and techniques and ties them as its solution.

clarity,_quality,_novelty_and_reproducibility: The paper is well written with minor linguistic errors. The structure is well organized and the content is easy to follow. There are some typos in the headers of Table 1 & 2. 

The work lacks major novelty as it uses some available tools and techniques and combines them  to generate its solution. As the code is not shared it is difficult to reproduce the results


summary_of_the_review: I have gone through the paper more than once including the appendices. Overall, the idea is quite simple: combine existing tools and techniques in a smart way that does the job.  Distribute labeled and unlabelled data across clients and let clients learn and share their parameters to a server; the server then performs some aggregation and shares them back to clients. Iterate these back and forth steps until the solution converges. The work lacks major technical contributions although the reported results look promising.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a solution for federated semi-supervised learning. the proposal lies averaging the manifold in the decision space over multiple clients. The sharing is done using two cryptographical protocols. The paper evaluates on three datasets
strength_and_weaknesses: Pros: 
  - The idea to use manifold for pseudo-labeling has not been used before in federated semi-supervised leaning, although it has been used in classical SSL.
- the paper has reported improvement in performance over similar federated SSL papers, but is still far from plain SSL performance

Cons:
 - the paper claims to used secure and cryptographic protocols for communication. The paper does not really explain why do we need those. I understand that there may be connection problems and communication to be disrupted and thus adding checksums and redundancy will help. I do not understand why using cryptographic protocols help ensuring intimacy of the data for each client since the framework do not assume external digital attacks. Furthermore, while the proposed method uses them, there is no evaluation of their efficiency. For instance how much damage to the communication can be absorbed without changes in performance? The fact that prior art used them, does not help claiming anything here.
- Evaluation is strong from one perspective but disappointing from another. It is carried on standard benchmarks which is very nice because it allows very good comparisons with previous similar works. It is weak because it does not propose a scenario where it really make sense to have protected data such as face related problems or where to assume that data being separated over clients, there is also a bias (either in data, either in labels) due to independent gathering and annotation. While this may not be fair to this paper, yet this is a major conference, and simply improving inside the niche, after a while becomes less interesting.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and reproducible within reasonable limits given the limited space

Novelty exists but is also limited. The method is novel for federated SSL, but is not novel in the broader and more interesting (in the sense that has a larger auditorium) of general SSL. Federated SSL is more like a niche from SSL and so is its auditorium
summary_of_the_review: The paper shows improvement over previous federated SSL methods. But I see this topic as a niche and this paper does too little to enlarge the auditorium. The innovation is limited to the topic.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studies federated semi-supervised learning and introduces the FedProp method. FedProp allows the computation of label propagation along an estimate of the data manifold to which the data of all participating clients contribute and use efficient cryptographic primitives to avoid accessing the data from different clients. Experiments on three standard benchmarks show that FedProp achieves promising semi-supervised classification performance. The main contribution lies in using LP to generate pseudo labels for unlabeled samples and applying cryptographic primitives to avoid direct data access 


strength_and_weaknesses: Strengths
+ The studied problem is important
+ Paper is easy to follow 

Weaknesses
- Novelty is limited. 
- No theoretical results.  
- Lack of  comparison with state-of-the-arts. 
- Evaluation results are insufficient 

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow. The paper novelty is limited from my perspective.  The experimental setup is clear, hence I think the results can be reproducible. 
summary_of_the_review: Detailed comments:

The main idea to leverage LP to generate pseudo labels is not new.  Also LP is shown to be not effectiveness enough to generate  pseudo labels, if the data distribution does not naturally form a manifold, e.g., when the data samples is limited.  

Why connecting data form different clients to perform label propagation? In the non-IID settings, the data distribution across different clients could be  significantly different. Is there a need to connect data from different clients to perform LP? What if each client just use its data to perform LP and obtain pseudo labels?  

Is the proposed FedRep convergent? In what convergence rate? What is the risk bound?
 
No formal privacy guarantees.   For instance, “first, the similarity is not computed between input data itself, but its feature representation according to the current model“. Using indirect feature representations cannot protect the input privacy. Particularly, many existing works has shown that using shared model gradients can reconstruct the data, so does the learnt representations.  

Zhu et al. Deep leakage from gradients. In NIPS, 2018


There exist more advance SSL methods (e.g, mixmatch, FixMatch) which can augment data with pseudo labels. Applying mixmatch to the federated SSL setting should be not difficult? What is the federated SSL performance with these data augmentation performance? 

SemiFL (NeurIPS 2021) is FedRGD are mentioned, but not compared. Actually, SemiFL has shown to significantly outperform the compared methods, e.g., FedMatch, especially in the non-IID setting. 

What is the data distributed in CIFAR100 and Mini-Imagenet? IID or non-IID? 
 
What is the impact of different number of labeled samples? 

Why connecting data form different clients to perform label propagation, especially in the non-IID setting,  is unclear to me. I suggest the authors should conduct experiments to validate this. 

There exist several federated learning methods for semi-supervised classification, where the client data itself is already a graph. Please discuss them. 


Zhang et al.,  Subgraph Federated Learning with Missing Neighbor Generation. NeurIPS 2021. 
 
Xie et al., Federated graph classifica- tion over non-iid graphs. NeurIPS 2021 
Wang et al., Graphfl: A federated learning framework for semi-supervised node classification on graphs. ICDM 2022
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The authors propose a semi-supervised federated learning method. Their method, called FedProg, adopts a manifold-based approach to semi-supervised learning and the data manifold is estimated jointly from the data of multiple clients and pseudo labels are computed through label propagation among clients. To keep the privacy of data, the authors proposed using multi-party Hamming distance computation and aggregation for secure communication. Experiments demonstrate the effectiveness of the proposed method.

Strength:
The problem setting is practical and the proposed method is demonstrated to be useful. The paper is nicely structured and written clearly.

Weakness:
Using manifold regularization for semi-supervised learning in federated learning is rather straightforward. Using secure communication is reasonable, but technical contributions are limited.

justification_for_why_not_higher_score: Thanks for the detailed feedback to the reviewers, which clarified some of the concerns raised by the reviewers in their initial reviews. Overall, the paper discusses an interesting problem and the proposed solution is reasonable and effective. However, the technical contributions are limited since the proposed method combines known methods in semi-supervised learning and secure communication. The additional reviewer ZBdp also had the same opinion. Given these facts, I cannot recommend the acceptance of this paper.
Decision: Reject