Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors study a largely overlooked problem: OOD detection in FL. To turn the curse of heterogeneity in FL into a blessing that facilitates OOD detection, the authors propose a novel OOD synthesizer without relying on any real external samples, allowing a client class knowledge from other non-iid federated collaborators in a privacy-preserving manner. Empirical results showed that the proposed approach achieves SOTA performance in non-iid FL.

From the view of the problem setting, this paper studies a very important problem, which contains enough significance in the field of FL and OOD detection. From the technical part, the authors propose a novel federated OOD synthesizer to take advantage of data heterogeneity to facilitate OOD detection in FL, allowing a client to learn external class knowledge from other non-iid federated collaborators in a privacy-aware manner. This work bridges a critical research gap since OOD detection for FL is currently not yet well-studied in literature. The proposed FOSTER is the first OOD learning method for FL that does not require real OOD samples.

strength_and_weaknesses: Pros:

1.  The problem setting is very important, filling up a gap between FL and OOD detection. This study is significant in the fields of FL and OOD detection.

2. From the technical part, the authors propose a novel federated OOD synthesizer to take advantage of data heterogeneity to facilitate OOD detection in FL, allowing a client to learn external class knowledge from other non-iid federated collaborators in a privacy-aware manner. This work bridges a critical research gap since OOD detection for FL is currently not yet well-studied in literature. The proposed FOSTER is the first OOD learning method for FL that does not require real OOD samples. Note that, it is not trivial to directly use the OOD techniques in FL, which is the major technical contribution of this paper.

3. Experiments cover many aspects regarding this paper (like CIFAR10/100 and near OOD detection).


Cons:

1. The problem setting should be presented explicitly. What the data you have and what the aim this paper wants to do should be demonstrated in a separated paragraph or subsection.

2. The generated OOD data is somehow different from the true OOD data. Relevant discussions are needed.

3. What is the validation dataset used to find the best hyperparameters of your method? Validation datasets are very important to the OOD detection. We cannot select validation datasets containing OOD data. 

4. ImageNet benchmark should be used to verify the effectiveness of the proposed method.

5. To ensure the completeness of the ablation study, performance of Foster w/o pdf and soft label should be reported.

6. In section 3, it is wired to say that a sample belongs to (using \in) a distribution. For example, x \in D. It should be x ~ D. 



clarity,_quality,_novelty_and_reproducibility: This paper is clearly demonstrated. The quality and novelty are enough in terms of significance and technical contributions. This paper can be reproducible based on the algorithm provided.
summary_of_the_review: From the view of the problem setting, this paper studies a very important problem, which contains enough significance in the field of FL and OOD detection. From the technical part, the authors propose a novel federated OOD synthesizer to take advantage of data heterogeneity to facilitate OOD detection in FL, allowing a client to learn external class knowledge from other non-iid federated collaborators in a privacy-aware manner. This work bridges a critical research gap since OOD detection for FL is currently not yet well-studied in literature. The proposed FOSTER is the first OOD learning method for FL that does not require real OOD samples.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors studied the problem of OOD detection in the literature of federated learning. The authors claim that the main challenge that prevents previous state-of-the-art OOD detection methods from being incorporated to FL is that they require large amount of real OOD samples, which are closely or even infeasible to obtain in reality. Further, the authors claimed that the data heterogeneity where each client collects non-iid data can hurt the performance of the system. Accordingly, the authors suggest to so called “taking advantage of such heterogeneity and turning the curse into a blessing that facilitates OOD detection”. Specially, the authors propose a novel federated OOOD Synthesizer (FOSTER), which learns a class-condtional generator to synthesize virtual external-class OOD samples. The authors claimed their superiority over the state-of-the-art counterparts. 
strength_and_weaknesses: > Strength 

- The authors consider the problem of OOD detection in federated learning, which seems to be a realistic problem. For security sensitive applications such as autonomous driving and voice recognition authorization, I agree with the authors that FL can be useful in these scenarios and taking OOD detection into consideration can be important. 

- The authors conduct a set of experiments with various OOD detection settings, e.g., CIFAR benchmarks, STL10, and hard OOD detection. To some extent, the experimental results verify the superiority of the proposed method over a set of classical OOD detection methods. 

> Weakness

- *The literature review is not enough*. The authors attribute the existing methods in OOD detection into two classes, namely, the real-data approaches and the synthetic approaches. However, I believe there is a large group of methods that do not rely on any OOD data (either real ones or the virtual ones), which is uncovered in the authors’ discussion. These works (e.g., post-hoc approaches [1], fine-tuning approaches [2], and contrastive learning approaches [3]) are popular and effective in OOD detection (even better than OE in many cases). Since they do not rely on OOD data for training, it seems that the considered problem of OOD data scarcity is actually not a big issue. I think the authors should cover a larger group of methods, and then discuss why collecting/synthesizing additional OOD data is important in FL. 

[1] Yiyou Sun, et al. Out-of-distribution Detection with Deep Nearest Neighbors. ICML’22.

[2] Haoqi Wang, et al. ViM: Out-of-distribution with Virtual-logit Matching. CVPR’22.

[3] Vikash Sehwag, et al. SSD: A Unified Framework for Self-supervised Outlier Detection. ICLR’21. 

- The authors follow the methodology named VOS. When applying the VOS for FL, the authors claim that enormous examples should be drawn for an accurate estimation of parameters. However, *I am not sure if the statement is true*. Especially, the estimation of MoG is in low dimensional space and the number of tunable parameters are in small scale. In fact, the authors of VOS actually claimed their superiority over previous GAN-based methods [4] in easy to be optimized, which may refute this paper’s point. Therefore, I think further discussion and experimental justification may require here. 

[4] Kiin Lee, et al. Training Confidence-calibrated Classifiers for Detecting Out-of-distribution Samples. ICLR’18. 

- Another little confusion point is that *if the data are synthesized in low-dimensional space where the MoG can properly model it, is it necessary to use the generative models in data synthesis?* It may increase the number of trainable parameters, which may exaggerate the data scarcity issue in FL.






clarity,_quality,_novelty_and_reproducibility: This paper is well-structured and clearly written. However, I do not think this paper is self-contained, for examples, descriptions about the MSP and VOS are missing. The novelty of this paper, at least to me, is limited. Further, I did not check the reproducibility of the paper. 
summary_of_the_review: I think studying OOD detection in the literature of FL is an interesting problem, but I am not sure if OOD data scarcity is really an important problem that require the in-depth discussion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new algorithm named FOSTER for out-of-distribution (OoD) detection in the federated setting. Since there may be no OoD sample in the federated setting, the paper proposes to use samples from the other classes that do not exist in the current client as OoD samples for local training. Since the raw data are not allowed to transfer, FOSTER trains a conditional GAN in the server, which helps to generate the synthetic external-class samples during local training. The experiments show that FOSTER outperforms the other approaches that directly apply existing OoD studies in the federated setting.
strength_and_weaknesses: Strength: The studied problem is important and less exploited in the current FL literature.

Weaknesses:

1. The intuition of FOSTER is to utilize the external-class samples as OoD. However, in the testing, the external-class samples are not true OoD samples. Then, the models may misclassify these samples into OoD. In the experiments, do the authors consider the external-class samples when testing the ID classification accuracy? If yes, why the ID classification accuracy is high given the external-class samples?

2. FOSTER is based on personalized FL. What about the baselines? The paper only mentions that the training method of all approaches is FedAvg in Section 5. The baselines should also adopt personalized FL for fair comparison. Also, local training without FL can also be added as a baseline, e.g., the clients can simply conduct unsupervised learning locally such as one-class SVM.

3. Figure 2 is confusing. How is the training performed? Is it based on OoD training or is it based on vanilla supervised training? The green points seem to be mixed with blue and yellow points even in the left figure.

4. A generator is required in FOSTER, which introduces additional privacy concern, communication, and computation overhead. The paper should discuss these aspects.

5. Experiments to investigate the effect of client sampling and data heterogeneity can be added.

6. Typo: Line 3 of Algorithm 1: broadcast \theta and w to A; Section 6: “we studied”, “we propose”. Please keep consistency for the tense. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The motivation can be improved (i.e., Section 4.1). The paper should clearly demonstrate why external-class data can help in distinguishing the ID and OoD data. 

Quality: The paper is in a good shape but not solid enough as it lacks in-depth analyses of the effectiveness of the proposed approach.

Novelty: The idea of using GAN to help the training is not new [1]. 

[1] FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learning. 

Reproducibility: Some experimental details are missing, e.g., what is the test dataset in the experiments.

summary_of_the_review: I think the paper still needs more analysis and experiments to justify the effectiveness of the proposed approach.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a virtual OoD sample synthesizer suitable for federated learning, called FOSTER. It uses the class information collected by the client to train the generator on the server and broadcast. Under the premise of ensuring privacy, FOSTER uses the knowledge of other non-iid federated partners to generate virtual OoD samples locally on the client. Using virtual samples to join training will improve the OoD detection performance of the model. Experimental results show that FOSTER outperforms traditional OoD detection methods in federated learning.
strength_and_weaknesses: Strength.

1) This paper explores the OoD detection method for federated learning for the first time, which is indeed a problem to be solved.

2) Without the need for additional real OoD samples, FOSTER can generate valid virtual OoD samples on clients with limited class data. Experimental evaluations show that this strategy outperforms previous OoD detection methods in federated learning scenarios, which is worthy of recognition.

Weaknesses.
1) The technical novelty of this paper is not good enough. The techniques used in the paper, including virtual sample generation, tail sampling, and soft labels, have been used in previous OoD detection. The main contribution is to use these technologies in the federated learning scenario.

2) The specific training process of the sample synthesizer lacks clarity. After reading Section 4.2 I still don't understand how synthesizers learn class information and synthesize samples efficiently. Hope for a clearer explanation.
Minors.

3) Figure 2 is hard to read.

4) Typo. See the last sentence of the 'challenge' paragraph.

clarity,_quality,_novelty_and_reproducibility: 1) The paper is clearly written, but some details could be improved.
2) The theory and methods of this paper seem correct, and the evaluation is good.
3) The idea of the paper looks interesting but has somewhat limited technical novelty.
4) The author did not provide resources such as code, I am not sure if it is easy to reproduce.
summary_of_the_review: This paper explores the OoD detection technology under federated learning, proposes a virtual OoD synthesizer suitable for this scenario, and achieves performance beyond the baseline. I recognize the significance of the work and the performance gains achieved by the paper. But the paper lacks some novelty in technology, and the description of the method needs to be clearer.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studied out-of-distribution detection problem under the federated learning setting. After switching the problem of interest from classification of in-distribution data to detection of out-of-distribution data, it is interesting (if not surprising) to see that heterogeneity of data becomes a positive point rather than a negative point of training data. The philosophy is to view in-distribution data from other clients as out-of-distribution data for the current client, and the implementation is to train a class-conditional generator (called Federated Out-of-Distribution Synthesizer) in a federated manner. All reviewers agreed that the novelty and significance are above the bar and thus we should accept it for publication.
justification_for_why_not_higher_score: I am not sure how many oral presentations we will have, but perhaps this submission is not that competitive to make an oral presentation.
justification_for_why_not_lower_score: The negative side of a task can actually be the positive side of another task, if we switch the problem of interest and change our point of view accordingly. This is quite interesting and may inspire a lot of researchers/practitioners attending ICLR 2023, and hence the paper should be seen by more people as a spotlight presentation.
Decision: Accept: notable-top-25%