Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose a new method for federated semi-supervised learning relying on a coarse grained regulator and fine grained regulator. This work handles scenarios where the labeled data and unlabeled data share different distribution. Empirical results show that the proposed method is able to outperform prior baselines.
strength_and_weaknesses: Strengths:
- This work proposes a novel method that handles a realistic scenario that haven't been studied before.
- This work provides formal convergence guarantee.
- The empirical results look promising.

Weaknesses:
- It is unclear what is the motivation for having both coarse-grained and fine-grained regulator. From figure 1 and Section 3, it's hard to follow what is the purpose of having each of them.
- Is there any theoretical guarantee on how the proposed method handles imbalanced data distribution between labeled and unlabeled data?
- I'm curious how the method performs in cross-device setting where the number of clients is large and each client has low probability to be selected at each round for scalability. The results would be more convincing if such results are provided.

Nit:
- In Equation (2), should it be argmin over $\phi$ rather than argmax?
clarity,_quality,_novelty_and_reproducibility: The method is novel. Details and motivations of the method needs to be clarified further.
summary_of_the_review: This work tackles an interesting case where the labeled data and unlabeled data comes from different distribution. This problem is conceptually realistic and interesting. The proposed method also shows strong empirical performance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies federated semi-supervised learning and propose a new method with a dual-regulator which can handle the heterogeneity of labeled data and unlabeled data from each client. The main contribution is the proposed dual-regulator where the first regulator updates local models  in an meta-learning based way by accessing the updating effect, the second regulator assigns weights to unlabeled instances. Experiments shows that the proposed method significantly outperforms existing methods for different data distribution scenarios. 
strength_and_weaknesses: ### Strength
1. The proposed method is technically sound. 
2. Experiments show that the proposed method significantly outperforms existing methods for different scenarios. 
3. The paper is overall well presented. 

### Weakness
1. The proposed dual-regulator is now very new; we can see similar techniques in many learning problems, e.g., semi-supervised learning, few-shot learning, learning with noisy labels. etc. With that being said, I still appreciate the value of introducing and tailoring these techniques to the federated learning scenario.
2. Many federated learning works evaluate different data types, images, texts, etc, which would definitively gives more comprehensive evaluation of the proposed method. I do not see the proposed method has an assumption for the data type and thus would be better if such experiments can be conducted.
3.  There are a few typos and inconsistency of using capitalization in the paper, which should be corrected in the future: sometimes all the words are capitalized for the first letter for the title and sometimes not. The language use is not very professional and should be improved. 
clarity,_quality,_novelty_and_reproducibility: See the above for the evaluation. 
summary_of_the_review: I am not an expert in this field,  my assessment is based on educational guess. Overall this paper looks good: it studies a problem which was overlooked by previous method, proposes a method which is technically sound, and produce significantly better performance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes FedDure, a method for semi-supervised learning (SSL) in federated learning. FedDure employs two “regulators” in order to improve performance in SSL scenarios. The first regulator is the C(oarse) reg(ulator); its job is to regularise the local model training by taking into account the pseudo-labels the local model assigns to unlabelled data. It is a network (initialised to be the network received from the server) which is trained on the pseudo-labelled dataset, where the pseudo labels are obtained from network received from the server. The authors perform a single step of training for the C-reg and then use the difference of the predictive entropies on the actual labelled data as a weight for the gradient of the local model on the cross-entropy loss of the pseudo-labelled data.  The second regulator is the F(ine) reg(ulator); its job is to learn an instance specific weight for the loss that models the impact each unlabelled datapoint should have in training. The training signal for this weight is the cross-entropy loss of the C-reg on the labelled data. The authors alternate between optimising the F-reg, optimising the C-reg for a fixed F-reg and then updating the actual local model using gradient contributions from the labelled data and gradient contributions from the unlabelled data that depend on the C-reg and F-reg. The overall architecture is evaluated on cifar10, cinic10 and fashion mnist, using various split settings that lead to data heterogeneity in both the labelled and unlabelled data. 

strength_and_weaknesses: Strengths:
- Good performance on the tasks considered 
- An ablation study was performed 

Weaknesses:
- Paper is not clearly written and can be hard to follow 
- Theory seems to be disconnected from the actual method
- FedDure has a lot of moving parts
- There are no error bars on the performance metrics, so it can be hard to determine the significance of the results
clarity,_quality,_novelty_and_reproducibility: While the idea and the concept of the C-reg and F-reg are novel, unfortunately, the paper is not clearly written and probably has issues. More specifically, the core section 3 has conflicting information; for example, in eq.3 the authors provide one update rule for the parameters of C-reg, however, in eq. 6 they provide another. Which one is used in practice and why? Furthermore, there seems to also be a disconnect between what is described in Algorithm 1, lines 16-20 and what is described in the “Meta-Process” paragraph in section 3.3; in the latter it is mentioned that C-reg is updated first, with F-reg second and then C-reg is updated once more (and I guess all of this before updating the actual local model) whereas in the algorithm the steps are a) optimize F-reg, b) optimise C-reg for fixed F-reg, c) optimise the local model. Besides that, there are critical information missing with respect to, e.g., $f_w()$; how is that defined and what is its output? From the boldface notation at eq.8 it seems that its output is the same as the dimensionality of the gradient vector? Given these concerns, I do not believe that this work is easily reproducible. 

Besides that, I also found the more detailed derivations in the appendix unclear. For example in Appendix A.4, the authors mention that they use REINFORCE and Monte-Carlo, however there is no expectation / randomness as, from what I understand, $p(\hat{y})$ is a delta peak (obtained from the argmax in eq. 1 of the main paper). Furthermore, it seems that eq.17 is missing a negative sign and from eq.19 to the next one, the difference of cross-entropies is is changed to the difference of entropies. For this to happen, you would need to replace the empirical distribution over the label (y) with the model distribution $f_s()$ and I do not follow how this happens from eq. 19. Furthermore, the proof for Theorem 1 starts by mentioning that the local model is optimised using the CE loss involving the parameters of the coarse-regulator. As the local model is updated using eq. 11 with gradients of various losses, I am not sure what the guarantee in theorem 1 is about and whether it is at all useful (as it seems to be about a different procedure than the one done at Alg. 1). I have similar concerns for Theorem 2 where the proof seems to ignore the fact that $\phi$, $w$ are affected by $\theta$. In addition, there are some peculiar statements near the end of page 19 that the $L_{CE}()$ “converges”. What do the authors mean by that? Furthermore, how is eq. 39 obtained? By taking the limit, shouldn’t you have something like $\lim_{T->\inf} \sum_{t=1}^T L_{CE}(\phi^{t+1})$, instead of some constants times the $L_{CE}$ using the $\phi$ parameters at an arbitrary time $t+1$?

summary_of_the_review: While the concept of the F-reg and C-reg are novel and the experiments seem to show improvements upon the baselines (although without error-bars so their significance is a bit unclear), I cannot recommend acceptance given the aforementioned concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new learning framework for semi-supervised data in FL environment. In particular, the authors target the situation where label data as well as unlabeled data possessed by each client are heterogeneous.

Despite the excellent performance compared to the baselines shown in the experiments, I believe that this paper is not self-contained and  is not ready for publication because the sections are not coherent and well motivated. For examples, 
- There is no mention of what kind of heterogeneity is considered in the problem setting.
- In Section 3 motivation of each component of the proposed method is not provided at all. Only methods are simply described. Therefore, it is very difficult for readers to understand why each component is proposed. In a similar context, it is difficult to understand why the performance is so much better than the existing methods even in the iid environment.
- Section 4 also simply provides the theorems (and proofs in the appendix). No discussion is provided; it is impossible to grasp the meaning of each theorem and how it can be compared with existing studies etc.

Reviewers also positively evaluated the performance improvement, but there was an opinion that it was difficult to grasp the content, and there was no reviewer who strongly supported the acceptance of this paper through discussion, so I propose a rejection.
justification_for_why_not_higher_score: there is not enough motivation and discussion about their results.
justification_for_why_not_lower_score: n/a
Decision: Reject