Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed federated learning in an unsupervised manner with some nice theoretical guarantees and nice experimental results, where the unlabeled data distributions must satisfy some non-trivial assumptions.
main_review: The proposed method allows different clients to have different unlabeled data distributions. There are two conditions required for learning from such data. First, all unlabeled data distributions must share the same set of class-conditional distributions. Second, the ratios of classes must be given, even though the data are unlabeled. Then, with the additional information about the class ratios, each client could align with the supervised learning counterpart itself, and the server would not be affected by the fact that the training data are all unlabeled.

It is worth mentioning that unsupervised federated learning is still based on empirical risk minimization (the loss function is new and the regularization is optional). This means that it goes along different lines from semi-supervised federated learning based on semi-supervised regularizations (the loss function is the same as supervised federated learning and the regularization is mandatory).

Positive points:

1. The idea is novel. The underlying task is still multi-class classification rather than clustering. As far as I know, the problem has not been studied yet, though there is already semi-supervised federated learning.

2. The idea is general. The paper proposed a new loss function, and thus it does not affect our choices of the deep network and optimizer used for training. Moreover, the new loss itself is fine with both image data and text data, which is more general than many semi-supervised techniques that can only be used with image data.

3. The idea is motivated. Since all clients have to prepare training data by themselves and no client would like to share its data with any other party, it is natural that the clients do not want to label all the unlabeled data due to the money and/or time concerns.

Negative points:

1. The assumption that "all unlabeled data distributions must share the same set of class-conditional distributions" sounds strong to me. How can the authors extend the current method so that this assumption is removed or reduced to a milder assumption?

2. The new loss function leads to some additional operations in forward and backward passes. However, in section 4 experiments, there is no experimental result about the computational efficiency. I am working on the practical side of deep learning and I think the issue of computational efficiency is very important. In tables 1 and 2, the proposed method worked much better than the baseline methods, but might the proposed method be much slower than the baseline methods too?
summary_of_the_review: This is an overall well-executed paper. Besides the above comments, the authors need clarify on a few raised questions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work presents a novel federated learning scheme to address the problem of learning from only unlabeled data. The main idea is clean and interesting, which constructs a global (server) model by aggregating the surrogate clients’ tasks from observing only unlabeled data for the classification tasks. The unlabeled data are transformed at each client to make them compatible with supervised federated learning; consequently, the learned models are transformed accordingly. Existing federated aggregation techniques, e.g. FedAvg, are applied on these transformed clients (where they call surrogate clients). Theoretical results are provided for learning the optimal model and experiments on benchmark and real data show superior performance compared to baselines.

main_review: # Pros

-The proposed method is novel and interesting. By using the surrogate loss, the method provides a framework to enable unsupervised clients (without any labels) to supervised clients (with labels in particular form); and learning is performed via the surrogate supervised learning loss. With such a framework, many existing supervised federated learning methods can also be applied in this client-without-label setting. I think this framework provides a novel perspective for solving unsupervised federated learning, instead of following the unsupervised learning approach such as clustering.

-It is also nice that the method can be easily implemented by adding a transition layer to the existing models, neither changes on the optimization process nor introduces additional hyperparameters to be tuned.

-The theoretical analysis on the optimal model recovery is also a very useful result providing both conditions for unsupervised surrogate task to match the supervised task optimal; and the interpretations for these surrogates. Its convergence directly relies on the chosen federated aggregation approach, which has quite a flexibility on advanced methods with guarantees.


# Cons

-The discussions in the related work section may not be comprehensive. The work can be quite related to personalized federated learning in the sense that each client learns a personalized model (g1,...,gc), see:
Towards Personalized Federated Learning, Tan et al., 2021.
Discussing relations and differences with this line of research seems necessary.

-Another line of research working on utilizing unlabeled data in federated learning follows the semi-supervised learning framework, see
A Survey towards Federated Semi-supervised Learning, Jin et al., 2020.
Although they assume some labeled data are available, which is not exactly the unsupervised federated learning setting, adding discussions with them in the related work may help position the paper well.

-In this paper, the authors do assume certain conditions on the unlabeled data, e.g. the assumptions on the class prior knowledge on each unlabeled dataset. Can the authors give more examples on how this information can be obtained in the real world?

# Minor comments

-In algorithm 1, it seems that only the model f is trained. In my understanding, equation 7 is the surrogate loss and model g is what is being trained. If it is correct, the current algorithm may be a bit misleading.

-In table 1, the performances of FedPL and FedUL are reasonable when the set number changes. However FedLLP and FEdLLP-VAT are not very stable. Could the authors explain more on this phenomenon?

-In the experiments, it seems that the experiments are conducted only on 5 clients, could the authors show more results with more clients?


summary_of_the_review: Overall, this paper is novel and interesting. However, some points should be improved in the revision.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors are proposing an approach for unsupervised Federated Learning. The authors propose to use Emperical risk minimizing for unsupervised learning. The authors assign labels to each of the class and uses a prior for the classes so that each of the client can learn without any labels.

The authors have shown theoretical properties for their proposed solution. Theorem 1 showing that there is a map that that exist to transform the data to the classes, Lemma 2 shows that there is a transition function for each of the clients. The authors have also shown convergences of the algorithm by deriving the upper bound.

main_review: Overall there are some good theoretical results in the paper with a thoroughly analyzed experiment section. I felt that Section 2 is not really needed in the paper as explaining the context of supervised federated learning is not relevent to the contributions in this paper. This could probably be moved to the appendix and then some of the theoretical derivations could be moved to the main body.

From the experiments, it was not exactly clear to me which class prior was used. It says "we randomly sampled class priors from range [0.1, 0.9] and then regularized them to formulate a valid \Pi_c as discussed in Section 3.1.". But it does not appears to explain what distribution the class prior is. Could this be explained.

I felt that U shouldn't be used a shorthand for unsupervised. But I guess this is up to the authors.
summary_of_the_review: The paper has provided some novel contribution in terms of the theoretical properties for unsupervised learning. I felt that the paper could have been better written and more experiments could have been conducted to make their results more convincing. But nevertheless, I do think that the results that the authors have presented are novel enough for a publication at ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper suggests a new strategy for pseudo-labeling
based on cluster structures in FedLearn. Theoretical 
derivations, abl.-studies and experiments are provided
showing the effectivness of the approach.
main_review: Strong:
- relevant topic with clear motivation and useful practical examples
- sufficient experiments
- outlined theoretical derivations + code

Weakness:
- the title is a bit misleading ... because the approach does not cover
  a general framework for unsupervised learning -> maybe the title should be changed
summary_of_the_review: Unsupervised FedLearn

comments:
- not so much to mention but only some 'concerns'
-  Unsupervised learning is really different to supervised learning.
   In your case you consider local cluster assignments as labels and use
   subsequently a supervised learning scheme to generate predictions.
   This is not really unsupervised because in general SL methods focus 
   on margin maximization of the class boundaries ... which is not the same
   as approximating a cluster distribution (so I would say your title is a bit
   wrong). Maybe better: Federated learning with surrogate labeling?
-  Further what happens if you have no uni-modal data as shown in Fig 1?
   In fact your # of labelings may change during the optimization or you
   artificially colaps different distribution into single once if the #of cluster
   is not appropriate
- 'Without any labels, FL becomes significantly harder than before, since it is unclear how to compute
   the local gradients at each client and how to aggregate them for updating the global model  '
   --> well there are unsupervised learning approaches which are gradient based
   ... you only need a differentiable cost function!
-  if the labeling in the experiment (for labeled data) follows the data distribution
   I do not expect a substantial problem ... but if the data are non-unimodal and the
   #of clusters does not fit it may become a challenge to get reliable models
- theory is likely correct and code is provided so reproducibility should be given
review_title: null

Meta-review: The paper demonstrates a case of federated learning with unlabelled but systematically partitioned data between clients. A title along terms like "FL with unlabelled data" would be much better - the considered setting here is not fully unsupervised but relies on the key assumption that while not the labels, at lease the precise label frequencies have to be known on each client, which is a strong assumption (also iid up to the class shift). Semi-supervised FL approaches should also be discusses.

Overall, reviewers all agreed that the paper is interesting, well-motivated and deserves acceptance.
We hope the authors will incorporate the open points as mentioned by the reviewers.
Decision: Accept (Poster)