Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of __near__ novelty detection (near ND), with __near__ referring to
the case in which the novelty (OOD) classes derive are semantically similar to those classes
contained in the in-distribution (ID) set used for training (a novelty class under this regime, for
example, being 'fox' when the ID set contains 'dog'). The authors begin by showing that
recently-proposed ND algorithms exhibit a significant performance gap with respect to the standard (far)
ND vs. near ND setup to motivate the problem. To solve this, the authors propose to use
early-stopped diffusion models, trained on the ID data, to generate a dataset of anomalous samples, with
features for the ND scoring mechanism can be learned by framing the task as one of binary
discrimination (real data vs. diffusion-model-generated data), solved with a linear classifier
head. In order to score samples, a simple k-NN-based algorithm is employed, acting on the aforementioned
features extracted from the training set. The authors show that this simple algorithm succeeds in reducing the gap between near ND and ND performances while also improving upon previous methods in the far ND setting. It is reasoned and shown empirically GANs (as in OpenGAN) lack some the necessary properties that diffusion models that are crucial to this success (e.g. image-quality metrics do not show a monotonic improvement in generative ability as training progresses).

strength_and_weaknesses: ### Strengths

- The paper is, for the most part, well-written, easy-to-follow, and does a good job of
contextualising and motivating the work.
- The proposed method is simple and intuitive, and seems to perform well empirically
compared with the baselines in both the regular ND setting and the near ND setting, showing a
significant reduction in the performance gap of going from the former to the latter.
the NR to near-NR setting.
- The figures are well-chosen and well-put-together: they convey well the essential points of the
  paper, such as the shortcomings of existing methods and the feasibility of using GANs vs diffusion
  models for creating the synthetic data.
- Evaluation performed using a good range and selection of baseline methods and benchmark datasets.
- Results are aggregated over a good number (10) of replicates.

### Weaknesses
- Despite the authors claiming otherwise, the method is quite sensitive to the stopping criterion in
  the near ND setting with an empirical range of [68.2, 90.0]; in practice, different datasets may
  require wildly different FID thresholds and there is no reliable way of validating this given the
  nature of the problem. Also, I wonder if the number of sampling steps and the choice of sampler have a
  significant effect on the results -- there doesn't seem to be any mention of this.
  Taking this a step further, might it be feasible to train the diffusion model fully and then rely
  upon early-stopping the reverse diffusion process according to the FID score?
- The need to train, and indeed sample, from a diffusion model can be computationally burdensome for
  higher-resolution datasets.
- The description of the SDE model is a little overdone; a much briefer description with a pointing of the
  reader to the original paper would be sufficient. This is especially the case given that, as
  I understand it, the proposed method should work with any diffusion based model (Imagen, Parti,
  DALLE-2, etc.) or are there theoretically some constraints on the form said model and the
  associated sampler should take?
- While this applies to several methods of the same class, the method seems to make certain
assumptions about the distribution of the ID and OOD data that might not hold in the wild.
- While results are aggregated over a good number (10) of replicates no measure of dispersion
  (standard deviation/error) seems to be reported alongside the mean in each case.
- Some typographic and syntactical errors (e.g. 'that unlikely come' (line 1 of the introduction)).

clarity,_quality,_novelty_and_reproducibility: - Despite the simplicity of the proposed method, the proposal to use
early-stopped diffusion models to generate the OOD data is novel to my knowledge; this proposal is
justified soundly both empirically and theoretically (albeit with intuition rather than rigorous
proof).
- While there are some typographic errors and some slightly awkward phrasing, the paper is generally
strong in terms of clarity -- the explanation and rationale of the method (supported greatly by the
figures) is easy to understand as is the evaluation procedure.
- In terms of reproducibility, the paper and its appendices provide detailed descriptions of the
training/evaluation setup. Experimental (optimisation + model) settings are provided in the
appendices and code is provided as part of the supplementary material.

summary_of_the_review: The paper is well-motivated and proposes a simple but apparently strong method for tackling the
problem of near novelty-detection, a setting in which the novel (OOD) classes are semantically
close to classes in the training set. The explanation of the method is easy-to-follow and the
problem and the solution are soundly established. The results look promising and have been compiled across a good range of datasets and baselines. While I worry that the sensitivity of the
method to the stopping criterion, as well as the assumptions
that are made in the distributions of the ID/OOD datasets, might limit practical applicability, I nonetheless think the
overall the paper is strong enough to meet the threshold for acceptance based on the aforementioned merits.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is focused on a challenging task: near-distribution novelty detection (near-ND). The authors propose to exploit a diffusion generative model to produce synthetic near-distribution anomalous data(i.e., “fake"), then the  pre-trained feature extractor is  fine-tuned to distinguish generated anomalies from the normal samples.

strength_and_weaknesses: The idea of this paper is very natural: try to generate near-distribution outliers to augment the original training dataset which only contains normal data (semi-supervised learning), such that we can fine-tune the pre-trained feature extractor by training a binary classifier. 

Strengths:
(1) Using generated outliers to improve the accuracy of novelty detection is not new. There have been some similar works which use GANs instead of SDE.  The main contribution of this paper is that they replace the GANs with the diffusion model (which is increasingly popular recently).  The motivation here is that the outliers generated by SDE are “closer“ to the nominal distribution, and then using these "boarder" points  to deal with the near-ND setting.  Even though the pipeline and SDE are all from existing works, the novel combination is meaningful considering the theoretically and experimental results.  

(2) The experimental results show that the proposed method considerably improves over existing models (especially for the near-ND setting).  It is not surprising that SDE over-perform GANs for the near-ND setting, but I am surprised that the improvement is so obvious.  It will be great if the code can be provided.

Weakness:
(1) In Section 2.2 (Diffusion models vs. GANs), they try to explain, for near-ND, why choosing diffusion models instead of other generative modes (GAN and VAE), it would be better to include flow-based models in this part. Also,  VAE-related results are not included in figure3.

(2) In Figure3,  density, coverage and FID are used as the metrics to measure the fidelity and diversity.  Could you add a brief introductions/definitions for these metrics? Also, I am confused how to get the "density" from GAN-based models as they are not tractable models.

(3)About Stopping point: In section 4, "the method is robust against the stopping point with at most 2% variation for the reasonable FID scores". However,  from table4,  it is difficult to get this conclusion. what is "reasonable" FID? Actually, the stopping point decide how similar the fake points and the real points. How to decide when to stop ?

Other minor comments:
The "normal distribution" used in this paper sometimes are confusing, which means the distribution of nominal data instead of Gaussian distribution. It would be better to use other notations.

clarity,_quality,_novelty_and_reproducibility: This paper is very clearly written and easy to understand. I really enjoy reading this paper  and well-designed figures help me a lot to  understand its motivation and the proposed framework. 



summary_of_the_review: Overall, this paper sheds some insights that diffusion- based data generation considerably passes the GAN-based SOTAs in the novelty detection task, especially for the neat-ND setting.  I recommend to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novelty detection method that utilizes synthesize outliers during training. The synthetic outliers are generated from a score-based generative model trained on in-distribution data.
strength_and_weaknesses: * There is a large room for improvement in clarity.
    * The paper should be clearer on how exactly the generative model is trained and used. The most important fact, in my opinion, is that the score-based generative model is trained on in-distribution data and terminated prematurely. However, the fact is only briefly mentioned in Page 2, and then mentioned in Figure 2. Given the importance of this information, this point should be described in detail in Section 2.1. Also, other details and design choices, for example, how early the training of the score-based model should be stopped, needs to be elaborated.
    * The experimental setup described in Section 3.2 is difficult to follow. There are several comparison settings but are not well organized.
* Discussion on the differences among diffusion models, GANs, and VAEs in Section 2.2 are not very persuasive. It does not have concrete examples or quantitative arguments. Plus, the argument of Section 3.2 is mostly about generative modeling itself and not about its application to generating auxiliary data in novelty detection.
* When performing CIFAR-10 vs CIFAR-100 experiment, it is more standard and more challenging to incorporate all the classes, instead of using only one class from each dataset. Overall, I am not convinced that the experiment settings used in the paper are standard.
* Replacing a GAN with a diffusion model in the data generator for novelty detection itself is not a very novel contribution to the community.
clarity,_quality,_novelty_and_reproducibility: Please see the Strength and Weaknesses section.
summary_of_the_review: Even though the problem that the paper tackles is relevant to the community, the paper has limited novelty and clarity.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the near anomalies detection and points a significant decrease in performance of state of the art approaches on near anomalies cases. A new solution is proposed: 1) using SDE models trained to generate data from the training distribution. 2) generate a dataset from the trained SDE. 3) train a binary classifier on top of a feature extractor to distinguish between the generated data and real data. 4) use nearest neighbour for detecting anomalies.
Extensive experiments and empirical analysis are carried on. The method shows good results. 

strength_and_weaknesses: Strengths
1) Near anomalies are a very important case that should be tackled more.
2) Good insights and good results.
Weaknesses
1) I am not quite sure about the made distinction between Near ND and Near OOD. The authors stated the difference, however, without using any reference.
2) Design choices are not well explained, e.g., why a binary classifier is used, why NN is deployed for ND and not the binary classifier itself.
3) How about training complexity, hyper-parameter selection?
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. However, there seem to be many components involved and many separate training steps and hyper-parameters that would make reproducibility hard.
The method is novel in general but all components are well explored before. 
summary_of_the_review: A new method for detecting near anomalies based on SDE, binary classification training and then KNN.
Good empirical results. 
Not super clear how design choices were made.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies image-based novelty detection (ND) and proposes a challenging setting, where outliers are semantically close to the samples of the normal distribution (near-ND). The authors propose to synthesise near-distribution anomalies with diffusion-based methods and train a classifier to distinguish them from the normal samples. Extensive empirical evaluations show superiority of the proposed approach in both, ND and near-ND, settings. During rebuttal the approach has been extended to even more challenging unlabeled multi-class setting, hence departing from the trend of a single-class setting and winning over one of the reviewers. AC agrees with the reviewers. AC recommends acceptance of the paper, and urges the authors to polish their final version by addressing any remaining questions, and reporting multi-class near-ND. Congratulations to the authors!
justification_for_why_not_higher_score: All scores are weak acceptance. 
justification_for_why_not_lower_score: All reviewers and AC agree on the decision.
Decision: Accept: poster