Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles anomaly detection of multiple classes without class labels. In other words, the proposed method learns the normality of multiple classes at once without the need for class label information. The paper analyzes that the reconstruction-based anomaly detectors learn ‘identity shortcut’ and introduces techniques how to prevent this phenomenon. To this end, the paper proposes three techniques: the use of query embedding in multiple layers of a transformer, neighbor masked attention and feature jittering. The experiments are conducted on MV-Tech and CIFAR10 datasets.
strengths_and_weaknesses: ## Strength

### Problem setup
Anomaly detection on a multi-mode dataset (multi-class dataset without class information) is a relatively underexplored area. Most of the out-of-distribution detection papers assume class information given. In addition, this paper targets anomaly localization tasks.

### Analysis and extensive ablation study supports the idea
- Section 3.1 shows the performance deprecation over the training epochs showing that reconstruction-based models’ performance is unstable during training (the phenomenon of identical shortcuts).
- Section4.5 includes extensive ablation studies supporting the design of the method and sensitivity in each component and hyperparameters. Most hyperparameters are insensitive to performance showing less than ~1% gap.

### Strong performance
The proposed method shows a notable performance gap over competing methods in a unified scenario on MV-tech and CIFAR 10.


## Weakness

### Lack of analysis

- The design of the method is not targeted for a unified case but is effective. Why?
The idea of the proposed method uses general ML techniques not tailored for unified (multi-class data without label information) cases, yet effective. What would be the main reason for this? How is the problem of learning identical-shortcut relevant to the performance of unified anomaly detection scenarios?

- Why does the proposed method not gain performance when label information is added? (Separate case) In Table1 and Table2, the proposed method does not improve much by adding label information. 
questions: How did the proposed method and competing methods are trained for the unified and separate case? In the separate case, are they trained on the whole dataset and finetuned for each class-wise dataset? It is unclear how the models are trained in each scenario.
limitations: Limitations are addressed in Section5.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to learn a unified framework for detecting multi-class anomalies. Anomaly detection is only trained on normal data, so the so-called “identical shortcut” phenomenon may occur. To solve this problem, this paper proposes the following strategies: 1) layer-wise query decoder, 2) neighbor masked attention, 3) feature jittering. The experimental results on the MVTec-AD and CIFAR-10 datasets show that the proposed method can alleviate the “identical shortcut” phenomenon.

strengths_and_weaknesses: Strengths
1. This paper proposes a novel neighbor masked attention.
2. This paper is well organized, easy to understand, and clearly written.
3. Good results are achieved on the MVTec-AD and CIFAR-10 datasets.

Weaknesses
1. The introduction to evaluation metric (AUROC) is missing.
2. The ablation experiments in Table 4 are incomplete. For example, under 1 q., the results in the case of only NMA and FJ are missing; under Layer-wise q., the results in the case of only NMA are missing; under Layer-wise q., the results in the case of only FJ are missing.
3. Feature jittering is a common practice and lacks novelty.

questions: 1. On lines 125-126, the author mentions that the model may learn a trivial solution, causing anomaly detection to fail. But in MLP, this should not be the case due to the presence of nonlinear activations. Can the authors explain this further through experiments or visualizations?
2. In Table 2, for the categories Capsule and Transistor, the results of the method DRAEM (50.5 and 64.5) are significantly smaller than those of the method in this paper (98.5 and 97.9), but for the category Zipper, the results of the method DRAEM (98.3) are higher than those of the method in this paper (96.8). What are the reasons for the above phenomenon? Please analyze this.
3. Please analyze the complexity of the method in this paper compared with the one-class-one model method.

limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose the learning of multi-class decision boundaries for the task of anomaly detection (AD) over multiple object classes. For this, they employ reconstruction-based scores obtained from a transformer network, modified with a couple of simple tricks, such as masking neighboring points in the attention map, and increasing the capacity of the decoder. Results on MVTec show this is a promising direction for AD over multiple object classes.
strengths_and_weaknesses: # Strengths

To enable their transformer-based model to work with the task of adopting a complex normal distributions, the authors come up with some modifications, in particular neighbor-masked attention which they insert directly into the architecture to replace the default attention layer. While the authors employ other approaches such as "feature jittering", these correspond to a simple addition of Gaussian noise during the input stage, and given its simplicity I would suggest authors can remove this from abstract etc., as it doesn't present any significant novelty.

The experimental results are the strong point of this work, with outstanding performance on MVTec-AD and CIFAR10 (which is a much less relevant benchmark though, in particular as more challenging ones have recently been used, e.g. CIFAR100/STL10, c.f. works listed below).

Moreover, the paper is well-written and easy to follow.

# Weaknesses

The proposed modifications are relatively straightforward, in particular the ablations in Table 4 indicate that a vanilla transformer would end up outperforming the existing state of the art on MVTec-AD. Given there are recent works on using transformers for AD, e.g. AnoVIT (as pointed out by the authors on page 3 lines 99-101), this somewhat limits the novelty of the proposed method.

Surprisingly, from the ablation in Table 4 it appears as if feature jittering (FJ) boosts performance nearly as much as neighborhood-masked attention (NMA). I would suggest included pairwise coupling (e.g. NMA+FJ) to shed light on which ones of these are required in unison, or whether they obtain similar outcomes.

Moreover, there have been various works in the recent past that attempt to perform anomaly detection over classes that contain more than a single object class (with no labels assumed present to distinguish the objects). A discussion of these recent works, and how they relate/differ from the way in which multiple classes are treated here is missing, e.g.:
- "Deep Semi-Supervised Anomaly Detection", Ruff et al., ICLR 2020
- "Detection Semantic Anomalies", Ahmed & Courville, AAAI 2020
- "Transfer-Based Semantic Anomaly Detection", Deecke et al., ICML 2021
These manuscripts investigate the presence of multiple classes in the normal distribution. While the focus is different (latent classes), it should be compared against in this work.

# Update

The authors have addressed points raised in my original review by adding ablations and contrasting their proposed setting to existing ones. Some concerns remain around the novelty of the proposed approach.

The score has been increased to reflect the newly incorporated changes.
questions: This paper presents strong experimental results, however appears to target a somewhat loose definition of multi-class AD. I would suggest the authors work to improve the clarity of their manuscript.

In particular:
- clarify how the proposed multi-object AD task of "unified AD" fits in or differs from existing works that assume multiple objects in the normal distribution (e.g. "semantic AD").
- clarify the novelty of the proposed transformer components, which appear highly incremental. Improvements could consist in more direct comparisons (and ablations) against transformer-based competitor models.
limitations: Limitations are discussed in Section 5, potential societal impacts (say, anomaly detection for tasks such as video surveillance) have not been discussed.
review_title: null

Meta-review: metareview: This paper is on a highly-important topic, and makes solid contributions. Anomaly detection for multi-class datasets without class information is an underexplored area. Reviewers have appreciated the strong experimental results (especially on the important MVtech benchmark), high quality paper writing, and explainability results besides accuracy, via a novel attention mechanism. On the flip side, there were concerns on lack of deep analyses of the constituents of the method and novelty (given that there are some recent papers with similar ideas). The scores were borderline and the authors have put significant effort to address the concerns of the reviewers. Especially extra ablation studies and comparisons with other relevant papers are quite helpful in regards to convincingness of the ideas. I support the acceptance of the paper given all. Please update your paper with the additional content you have provided in the responses below.
confidence: Certain
Decision: Accept