Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a new semi-supervised federated learning algorithm (FedMatch). Two scenarios are studies: 1) label-at-client (labeled and unlabeled data are at client). 2) label-at-server (labels are at server and unlabeled data are on client). The authors propose a disjoint learning which decomposes the model \theta into supervised \sigma and unsupervised $\psi$ such that $\theta = \sigma + \psi$ . the model is trained, in an alternating manner, to minimize the two loss functions for the supervised eq (4) and unsupervised model components eq (5) with inter-client consistency. The experimental results show an improvement w.r.t. to SOTA in terms of performance.

The paper is sometime difficult and its clarity could be improved. I would consider the novelty of the method, from machine learning prospective, to be somewhat borderline as it uses known elements and adapt them to the introduced learning scenarios but overall the proposed solution is interesting as it seems to work well in practice. 

some detail comments: 

- $\pi$ in one of the terms in the norm needs to be removed in inter-class consistency loss ||p θ (y|π(u)) − p θ (y|π(u))||
- The disjoint learning scenario was not well introduced in paragraph "Parameter Decomposition for Disjoint Learning"
- a missing section reference in (see section ??).
- Related to the reduction of communication costs: the bit-encoding of the model difference between the server and client is not clarified, i.e., for example if the difference is coded using 64 bits and also the same encoding for the models there is no gain in communication. 
- In Table 1, the bold should be indicative of the best method per column
review_title: Interesting new federated learning scenario

final_score: rating: 6: Marginally above acceptance threshold
review_content: In many real-world applications, local data are not always well labeled or fully labeled, and most of them are unlabeled. This paper introduced a novel learning paradigm, Federated Semi-Supervised Learning (FSSL), to handle the federated learning from distributed and partially labeled data within the clients. Under this paradigm, the paper studies two different scenarios where partly labeled data appear on client nodes and merely appear on the sever node. In the introduction, the authors clearly presented the motivations of introducing FSSL and showed the differences between the proposed method and some existing ones.

In the FSSL, the key technique is Federated Matching (FedMath), which learns inter-client consistency between clients, and decomposes parameters to reduce both interference between supervised and unsupervised tasks, and communication cost. The Inter-Client Consistency Loss was most directly taken from some existing method. Thus, the critical innovation is the Decomposition of parameters, which has several benefits to the learned models. Basically, the proposed method is intuitively and technically sound.

The authors conducted a batch of experiments to evaluate their proposed method on three different tasks under both labels-at-client and labels-at-server, compared with some baselines. The experimental results are positive. However, there are some minor issues in the experiment section. The authors mentioned that the proposed model can work under the scenario that some labels are not correct (noisy labels). However, this point was not verified in the experiment section at different error labeling rates. It is shown that FedProx-UDA/FixMatch degrade when the labeled data increases from 5 to 10 in Figure 6(d). What causes this degradation and how the proposed method avoid it?

review_title: Review comments for FSSL

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper studies the problem of learning a joint model for different local data sets. Each model is trained individually based on each individual local data set, and the inference is performed by regularizing the results of different models.  The main challenges come from the facts that each local data set belongs to different client hence the data is not shared and the labeled data in some local data set may be not sufficient. To utilize the unlabeled and labeled data, the paper proposes using unsupervised, semisupervised and supervised learning models. To incrementally training the model with different types of data, authors  propose to decompose the model parameters as supervised and unsupervised model. The description of the problem is complicated and little confusing. 

The problem seems interesting, but one of my main main concerns is that I do not think authors solve the original challenge. The original challenge is that clients do not want to share data for model training. But the method proposed utilizes all the local data sets to learn the model.

My another main concern is that the novel contribution of this work is not strong. My impression of the method proposed in this work is pretraining the model with labeled data and fine-tuning it with unlabeled data with constraints to make the fine-tuned model as close as the pretrained model. 

review_title: problem description is complicated, not solving the original problem, contribution not clear

final_score: rating: 6: Marginally above acceptance threshold
review_content: AFTER READING THE AUTHORS' REPLY, I HAVE CHANGED MY RATING TO 6.

Even though authors introduce an interesting approach for a problem that has many potential practical applications, the paper suffers from two main weaknesses:
1) it is poorly organized, which makes it very hard to follow
2) it lacks a compelling, real-world dataset

In terms of paper organization, in this reviewer's opinion, it may be beneficial to decouple and present in a serial manner the solutions to the two main scenarios (i.e., labels at client vs server). The parallel presentation makes the story harder to follow, as the reader has to keep switching context from one scenario to the other. The very long captions of Figures 2 & 3 are not helpful, and Section 4 comes a bit out of nowhere, given that the pseudo-code for the two proposed algorithms only appears in the appendix. 
A better approach would be to use the body of the paper for an illustrative running example and the pseudo-code of the algorithms, while relegating the details to the appendix.

In terms of the empirical validation, it is disappointing to see  that you are using synthetic datasets. The paper would greatly benefit from having at least one real world application domain in which the proposed approaches "move the needle." There are far too many papers in which a novel approach does great on synthetic data without impacting the state-of-the-art results on real-world domains.


Other:
- caption of Figure 1: the data is "available" rather than "given" to the local client (twice, under both "a)" and "b)")
- page 2: you make references to Figure 6 a & b, but you mean Figure 1 a & b
- page 3: in line 3 of the 1st paragraph of 3.1, you refer to "D" without defining it
- please spell-check & proof-read the paper: 
    - "Cleint" --> "Client" in the header of Table 1; 
    - "manly" --> "mainly" on page 6

review_title: Interesting paper with two main weaknesses: poor organization & lack of a motivating, real-world application dataset. 

Meta-review: This work proposes the Federated Matching algorithm as a novel method to tackle the problems in federated learning. The paper is well-written and original, and it contributes to the state-of-the-art. 
Decision: Accept (Poster)