Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: This paper presents an unsupervised learning based anomaly detection method, which masks the images at multi-levels for restoration and then refine the restored images with refinement network. To overcome hyperparameter sensitivity, this work ensembles multiple restoration results from different level of masks and propose a novel mask generation and refinement method to achieve hyperparameter robustness.  Experiments are conducted on  several  benchmarks 
strengths_and_weaknesses: Strengths 
1.This work proposed a multi-level masking scheme for image restoration and ensemble restoration results to overcome hyperparameter sensitivity. 
2.The experiments are evaluated on multiple datasets.

Weaknesses
1.The novelty of this work is very limited. The mask-based restoration has been studied by search works, and they employed the ensembles of restoration results of different scale masks or different pattern masks.
2.Some important related works are missing in the paper, and this work does not compare the AD performance with these works. E.g. “CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows”, “Learning Semantic Context from Normal Samples for Unsupervised Anomaly Detection” etc.
3.The proposed method adopts a complicated framework including multiple restoration and refinement networks, but the AD performance on MVTEC and Cifar-10 dataset is far below the state-of-the-art.

questions: What is the advantage of the proposed mask design method in comparison with the masking design in other works? Although the hypermeter issue is relieved to some degree, the restoration quality of designed mask scheme is worse than the peer works. 


limitations: The paper didn't address the limitation and potential negative societal impact of the work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This work targeted the unsupervised anomaly detection problem and proposed a reconstruction-based method. Namely, by masking out features of the input data, and adding also noise to the features, the reconstruction and refinement network are only trained on the normal data to achieve small reconstruction errors. Therefore, any anomaly would generally suffer from high reconstruction errors. The evaluation is carried out in the one-vs-rest setup of the multi-class datasets MNIST, FMNIST, CIFAR10, and the defect detection benchmark MVTecAD. These benchmarks cover both semantic and non-semantic anomaly patterns, showing the covered use scenarios of the method.

The method's gain over SoTAs is quite diverse on the benchmarks. The authors argued an important advantage of the proposed method is its less sensitivity to hyperparameters, e.g., through an ensemble of different levels of masking, and the performance generally increases with more levels and is saturated at a certain level.
strengths_and_weaknesses: Reconstruction is one type of method to address anomaly detection. It is an unsupervised approach, thus requiring little knowledge of anomalies. However, compared to other kinds of methods, it is generally underperforming. One common observation is the reconstruction network can also reconstruct the anomalies. For instance, Masked autoencoders can reconstruct the input with a masking ratio of more than 75% on the input space (surely the work is proposed for self-supervised learning, not anomaly detection). The authors' method showed reasonable performance, through feature-level masking, adding noise, and additive refinement. 

However, the achieved performance is not very strong. First, it does not provide consistent improvement over different cases. On average, it is on a par with some existing methods. It can be important and interesting to discuss why the method is effective in some cases and less effective in others. 

Second, the authors argued prior work is more sensitive to hyperparameter settings. It is not a strong argument in my opinion. All these methods are thresholding-based methods, including the proposed one. Thus, they all somewhat need some validation data to set the threshold before deployment, e.g., finding the value based on the expected false positive or true positive ratio. AUROC and AUPR are threshold-free evaluation metrics, but a threshold is still needed to really make a decision. In practice, it is probably not difficult to obtain some anomalies for validation. The important question is if the choice of hyperparameters is good for unseen anomalies at test time.

Third, at the training time, the model does not exploit any prior knowledge about anomaly patterns to train the reconstruction. In other words, the model decides on its own which cue to use for learning the best reconstruction strategy, largely depending on the given dataset. In other words, it is also hard to predict the model reacts to which type of anomalies. The evaluated benchmarks mainly considered either semantic or non-semantic anomalies. What if construct both semantic and non-semantic anomalies for the same in-distribution dataset, i.e., CIFAR10-dog as the inliers, CIFAR10-others as outliers, CIFAR10-corruption or some other non-semantic shift as another type of outliers. 

 
questions: 1. AUPR evaluation: for an imbalanced evaluation set (the ratio of normal and abnormal samples is not balanced, AUPR is a more insightful metric to evaluate than AUROC.

2. Analysis of the learned masking patterns: how they behave differently on different datasets. For instance, in the visual example Fig. 3, while the model is trained on ship of CIFAR10, it can still reconstruct the deer class. This is a known observation of reconstruction-based methods. The proposed method does not seem to provide a different observation. The reconstruction difference is mainly on the local pixels. This leads to the question if the network really exploits the semantic structure to mask and reconstruct. This may explain why the performance on CIFAR10 is quite diverse across different classes.

3. On the MVTec benchmark, there are more competitive baselines, e.g., from https://paperswithcode.com/sota/anomaly-detection-on-mvtec-ad. 

4. On the F/MNNIST and CIFAR10, the authors mainly evaluated on the one-vs-rest setting. how this approach can handle multi-class settings, namely, the inlier dataset is multi-class.

5. How well this approach can scale up to more diverse and higher-resolution datasets?

6. More discussion or empirical analysis of the effectiveness of feature masking. For instance, on which feature dimension, how does it change over different input resolutions or types of anomaly patterns, the importance of adding noise vs. just putting zeros, why the learned mask does not select a shortcut, e.g., masking out every n pixels (like a downsampling) to achieve good reconstruction. 
limitations: The limitation discussion of this work is primarily focused on cases where the method is performing on a par or worse than baselines, and also on the ablation study part. However, as provided in the previous feedback sessions (weakness and questions), I think it is still unclear on which conditions/reasons the model is particularly good and in which scenarios it is expected to be underperforming. This is very important as so far the empirical results are quite diverse in terms of gains over SoTAs.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a reconstruction-based anomaly detection algorithm. The algorithm is based on autoencoders, and noise is added in the form of masking of one of the embedding layers, which then has to be reconstructed to match the original input; input-specific masks are computed using a masking network, with a given average masking level. A further refinement network is implemented to correct the residual error from the reconstruction. At test time, predictions from various masking levels are ensembled and the reconstruction error post refinement is used as the anomaly score. Experiments performed on benchmark image anomaly datasets show competitive results with recent methods.

strengths_and_weaknesses: Overall, the paper presents a promising approach to anomaly detection but some aspects of the algorithm are not well motivated or validated, and the evaluation could be improved. Detailed comments follow.

**Originality:**

This work extends a long line of reconstruction-based anomaly detection methods using autoencoders, with recent ones including noise and even masking but at the input layers (see references below). The key difference here is in implementing the masking at the inner embedding layers, and the use of a refinement network to further improve the reconstructions. From this perspective, the proposed method seems incremental. 

- One-Class Learned Encoder-Decoder Network with Adversarial Context Masking for Novelty Detection. WACV 2022.
- Self-Supervised Masking for Unsupervised Anomaly Detection and Localization. IEEE Transactions on Multimedia 2022.


**Quality:**

Overall the approach and evaluation are reasonable, though there are several aspects that need to be addressed:

- The paper does not motivate well the need to mask embeddings versus input layers (I did not understand the justification provided on L142), nor are comparisons provided to input masking autoencoder methods provided which could help to motivate the method.
 
- The paper differentiates through the mask generation step by assuming that the average masking level condition is satisfied (Line 169). However, it is not shown whether this condition is indeed satisfied in the paper - this should be verified to ensure that the method is technically sound. 

- Experiments in the paper are limited to images, though the framework appears to be generic. It would be informative to include tabular datasets (e.g. from ODDS - http://odds.cs.stonybrook.edu/) so the generality of the masking strategy can be assessed. Moreover, "side-information" is not usually available for such data and would be better able to demonstrate the practical utility of the proposed method.

- Given that the mask is in [0,1], the ablation study (L326) should also include masks with entries uniformly generated over [0,1].


**Clarity:**
- The paper is largely clear but there are several typos scattered across the paper (e.g. L23 real-word, L154 successfullt, L158 owingto, L173 non-differential).

- It would be helpful to provide the percentage of anomalies in each dataset.

- Just a suggestion: it may be clearer to revise the definition of masking to $\tilde{e} = e \odot (1-m) + \epsilon \odot m$, such that a higher m represents a higher masking level in that more of the input is removed.


**Significance:**

Overall, while the method is interesting and has some novel elements, the significance of the results is not clear:

- In terms of the method, the refinement module is independent of the masking approach, and could also be applied to other autoencoder methods; the notion of masking is somewhat interesting and broadly applicable, but the lack of good performance makes its significance unclear.

- In terms of performance, given that other better performing methods exist on the image datasets evaluated on, it is unclear if the method will be practically useful as other training/model hyperparameters still need to be tuned in practice. As mentioned above, it would be useful to demonstrate good performance on other datasets for which side-information is not available.

- The claim that this is the first paper presenting the "hyperparameter sensitity" is not quite true, as this has been reported in the following workshop paper.

The Effect of Hyperparameter Tuning on the Comparative Evaluation of Unsupervised Anomaly Detection Methods. 6th Outlier Detection and Description Workshop @ KDD 2021.


**Other comments:**

- Given that some results are pretty close, it would be informative to provide results from multiple runs along with standard deviations.

- In addition, providing results with the more appropriate AUPRC metric for anomaly detection due to the class imbalance would be informative.

** Post rebuttal update: **
As described in the comment below, the response has addressed some of my key concerns regarding comparison to other masked-based anomaly detection methods and I am raising my score to a 5.

questions: - Does the root-finding algorithm find a solution to the equation in practice? Is the assumption used to derive the gradient in Eq (2) satisfied?

- How does the method perform on tabular or other non-image datasets (e.g. ODDS datasets)?

- How does the method perform relative to other masking-based autoencoders?

- What is the performance with a uniform mask (see above)? 

limitations: While I agree that the proposed method does eliminate the choice of the masking level hyperparameter, the autoencoder architecture to use and training hyperparameters still have to be tuned for a specific dataset, so I think it is a bit of an overreach to claim that the proposed method solves the tuning issue. To be fair this is an issue with many other methods as well.
review_title: null

Meta-review: metareview: The paper proposes a Multi-Level Masking and Restoration with Refinement  to solve the hyperparameter sensitivity problem in anomaly detection studies. Reviewers had some concerns regarding this work including limited novelty, inconsistent numerical evaluation with prior works, lack of  discussions on benefits of using prior knowledge, etc. I appreciate that the authors were active during the rebuttal period to address these concerns but I think the paper needs a bit more work before being accepted. 
confidence: Certain
Decision: Reject