Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces ADBench, a tabular anomaly detection benchmark. The problem setting is that one is given a dataset and asked to identify the outliers in the dataset. This paper compares a wide variety of methods (30 methods total, supervised, semi-supervised, and unsupervised) on a wide variety of settings (55 datasets, different kinds of anomalies, corrupted data). There is not much novelty in the datasets or methods, but there are some interesting findings afforded by the large evaluation (e.g., none of the unsupervised methods are better than the others).

---------------------
Updating my score to a 7 after author response; I'm not an expert in this area, so my confidence will remain at a 2
strengths: - Large, well-designed benchmark for tabular anomaly detection
- Lots of datasets and methods
- Explores interesting settings, e.g., supervision type, corrupted data, different types of anomalies
- Adds new evaluation settings using embeddings from DNNs on various text and image datasets
- Good GitHub repository. Everything seems easily available.
weaknesses: - At the end of the day, it doesn't feel like there is much novelty beyond the increased scale and the findings that come with it. This is OK, but still something to keep in mind.
- Some issues with related work and correctness (see below)
correctness: It's unclear whether the anomaly generation method outlined in lines 157-159 is faithful to the original dataset. It fits a GMM to the data in order to generate anomalies in a controlled manner, but this might remove interesting structure in the original dataset.
clarity: The paper is well-written, although there is superfluous self-praise in some parts of the paper, e.g., "We implement all the semi-supervised methods and release them along with ADBench—we consider them as great addition to the community." The latter part is for the community to judge and doesn't need to be said explicitly.
relation to prior work: The paper is marketed as comprehensive, but it completely ignores the large body of work on out-of-distribution detection. I understand that the OOD detection community is separate from the tabular anomaly detection community, but the two areas are related, so it would be good to at least mention OOD detection methods (e.g., MSP, Energy-based, Mahalanobis).
additional feedback: It would be good to bold the best-performing methods in the tables in the appendix (or indicate visually that there are multiple indistinguishable methods).
review_title: Official Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: * The authors propose a comprehensive Anomaly Detection benchmark, which consists of 55 datasets.

* A total of 93,654 experiments were performed on the proposed data, with both classical and deep learning models.

* The benchmark is constructed taking into account various degrees of supervision, inductive assumptions, and data modalities.

* The paper contains several insights regarding different models' performances and proposes several future directions for research in Anomaly Detection.
strengths: 
* The authors have proposed a large collection of data for benchmarking Anomaly Detection algorithms. They have performed a comprehensive analysis of existing methods and provided various insights regarding the robustness and inductive biases of these methods. The addition of synthetic data which can be used to test the inductive assumptions and types of anomalies that can be detected can be a very powerful "model debugging and development" tool.

* The paper provides several insights on existing AD methods and reports their performance when various amounts of supervision is used. The paper contains discussions about the limitations of existing AD models.

* The code and the dataset are publicly available. The paper is well written and easy to follow.

* I believe that these kinds of _unified_ benchmarks are very important for Anomaly Detection since most works are using different experimental setups, which makes the methods harder to compare and can lead to reproducibility issues. This might be a very useful resource for the AD community.

weaknesses: 
* You have not included datasets tailored for anomaly detection in your benchmark, such as MVTec-AD [[1]] or MNIST-C [[2]]. Do you have a reason for omitting them?

* I understand that the newly added datasets (CIFAR10, fMNIST, SVHN, AGNews, Amazon, IMDB, and Yelp) are present in the form of representations extracted from neural networks (BERT and ResNet18). This could limit the overall AD performance of the models, since the data is already projected to a different space by the feature extraction backbone, leading to anomalies that could be clustered together with the normal data. Moreover, less general models that use the data structural information as an inductive bias for learning will suffer from these benchmark design decisions. For example, [[3]], [[4]] and [[5]] are methods specifically designed for one-class classification in text, with [[3]] and [[5]] achieving much better performance on the AG News dataset when compared to your best results (83.1, 90.0 vs. 68.97 ROCAUC). This might be due to the different train/validation/test splits used but might also indicate that end-to-end training is desirable on these datasets. Please consider adding an option for downloading the "raw" dataset in your library.

* Some scores reported on your synthetic data are very high (for example 99.91 ROCAUC for the KNN on Global Anomalies). I still believe that the datasets are useful for testing the inductive biases when designing a model, but I don't see any value in using them when computing the overall performance metrics of new methods. It's not clear to me if you've also used them when computing the mean performance in places such as Figure 4.

[1]: https://openaccess.thecvf.com/content_CVPR_2019/papers/Bergmann_MVTec_AD_--_A_Comprehensive_Real-World_Dataset_for_Unsupervised_Anomaly_CVPR_2019_paper.pdf
[2]: https://arxiv.org/pdf/1906.02337.pdf
[3]: https://aclanthology.org/P19-1398/
[4]: https://aclanthology.org/2021.eacl-main.296/
[5]: https://aclanthology.org/2021.naacl-main.25/
correctness: 
* The datasets used are relevant to Anomaly Detection

* The experimental setup and the metrics used are sound for AD.

* The benchmarked methods are well known and the authors performed multiple experiment runs to obtain the mean performance.
clarity: 
The paper is generally well written but contains some confusing parts.

* I don't understand the claim that "none of the benchmarked unsupervised algorithms is statistically better than others" (line 35, figure 4 caption), it seems that the biggest performance gap is between DeepSVDD (51.07 ROCAUC) and CBLOF (70.20 ROCAUC), 19.13 points in ROCAUC seems like a major performance gap?

* In figure 4 you have a cross symbol for both subfigures (a) and (b), but the explanation for the symbol is just under (b). Please consider moving it in the figure caption.

* In the figure 4 captions some text seems to be missing: "(a) shows that no unsupervised algorithm can statistically outperform." - outperform what? Please expand.

* In the figure 4 captions you mention that the scores present in the figure are the "AD model performance on 55 real-world datasets.", I suppose that the reported numbers are the average score over all the datasets. Please be explicit in the captions and consider adding the standard deviations over the datasets.

* You mentioned multiple times that the collection of data represents "55 real-world datasets", please expand. What makes them "real-world"? For example, your collection of data contains MNIST and Fashion MNIST - it could be argued that they can be used in "real-world" applications and were collected from the real world, but they are curated and are usually used for benchmarking algorithms.

* In Figure 7 you show the performance change under noisy and corrupted data. I would argue that the performance difference on the OY axis is not the clearest way of expressing the degradation: for example, you could get the impression that DeepSVDD is robust to noise since its performance doesn't change a lot when using corrupted data, in reality, DeepSVDD can't lose a lot of performance since it's already close to random in the uncorrupted scenario (51.07 ROCAUC). Please consider plotting the average ROCAUC score on the OY axis (as you did in Figure 6).

* You mentioned that your work is focusing on Tabular AD (line 26), but you include datasets such as CIFAR10, SVHN, AGNews and so on, which are not tabular. Please consider rephrasing or expanding the explanation.

* Regarding Sec. 4.4, "Unsupervised methods are more susceptible to duplicated anomalies": does "anomalies are duplicated by 6 times" mean that you now have the _all_ the anomalous samples appear multiple (6) times in the test data, or that you sampled _some_ anomalies and made them appear more frequently? Please consider rephrasing and expanding the explanation. 

* Line 38: "(...) we observe that *the* best unsupervised methods (...)" 
relation to prior work: The paper cites and discusses a lot of related and prior work. 
additional feedback: It would be nice to consider expanding this work with different modalities (such as videos, for example ShanghaiTech, or graphs). I'm looking forward to seeing how this project evolves!
review_title: A nice benchmark and analysis for Anomaly Detection 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents ADBench, a comprehensive tabular anomaly detection benchmark with 30 algorithms and 55 datasets. They focus on three comparison perspectives, including the availability of supervision, the anomaly types, and the algorithm robustness under noise and data corruption. They conduct extensive experiments to show the value of algorithm selection, label supervision, and prior knowledge.

strengths: 1. ADBench has a large algorithm collection with unsupervised, semi-supervised, and full-supervised anomaly detection algorithms. It includes the latest deep learning and ensemble learning methods.
2. ADBench covers a wide range of anomaly detection application domains, including healthcare, finance, natural language processing, computer vision, etc.
3. ADBench studies multiple types of synthetic anomalies in Figure 3, which can help us better understand the impact of different anomaly types.
4. ADBench additionally considers three noisy and corruption settings to better evaluate the algorithms. It highlights the importance of developing robust and noise-resilient anomaly detection algorithms.

weaknesses: Compared to previous anomaly detection benchmarks, the proposed ADBench takes more datasets into account, but more do not necessarily mean better. One question that the authors forget to justify is that, whether these 55 datasets are really anomaly detection datasets? 

Indeed, some of these datasets are not anomaly detection datasets from my perspective. In the definition, an anomaly is a data object that deviates significantly from the majority of the objects[1], and thus usually accounts for only a small fraction of the dataset. However,  in 7 out of 55 datasets used by ADBench, anomalies make up about half of the dataset. For example, 75% of samples are labeled as anomalies in the Parkinson dataset, which is unreasonable.

[1] Han et al. Data Mining: Concepts and Techniques. 2011.

correctness: Yes. The experiments are appropriately designed and performed. The model comparison is comprehensive and fair. However, I have some concerns about the datasets. Please refer to Weaknesses for more details.

clarity: The paper is clearly written and easy to follow.
relation to prior work: This paper clearly discusses the differences with existing literature in section 2 and Table 1. In particular, it considers more ensemble learning and deep learning methods, incorporates both real-world and synthetic datasets, and provides diverse comparison angles.
Some additional References:
[1] Lavin A, Ahmad S. Evaluating real-time anomaly detection algorithms--the Numenta anomaly benchmark[C]//2015 IEEE 14th international conference on machine learning and applications (ICMLA). IEEE, 2015: 38-44.
[2] Akcay S, Ameln D, Vaidya A, et al. Anomalib: A Deep Learning Library for Anomaly Detection[J]. arXiv preprint arXiv:2202.08341, 2022.

additional feedback: 1. More justifications can be provided on whether all 55 datasets in ADBench are really designed for anomaly detection.
2. Besides AUC-ROC and AUC-PR, some other metrics can be used (e.g., various F1 scores, the top-K accuracy) for a more comprehensive comparison.
3. Figures 4 and 5 are too small to distinguish the performance of the compared algorithms.
review_title: ADBench: Anomaly Detection Benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: # Submission summary and contribution
The paper proposes a comprehensive benchmark for evaluating the performance of anomaly detection(AD) methods. Plus, it contributes 7 new datasets developed from popular datasets. 

The paper proposes to evaluate AD methods in the following 3 angles
1. the extent of supervision such as unsupervised, semi-supervised and fully supervised
2. the type of anomalies such as local, global, dependency and clustered. 
3. the performance of models with noisy data

All experiments are focused on the 3 angles with varying percentage of labeled anomaly samples. The paper presents the results of 30 methods which reveal their true performances and a few key conclusions are drawn from the results such as no unsupervised algorithms can statistically outperform. Based on the observations, the author proposes a few future directions for the research community to further explore.
strengths: 1. the ADBench proposed in the paper includes the most number of datasets(55) and algorithms(30) compared to previous benchmarks. 
2. it evaluates the algorithms with most various angles such as level of supervision, type of anomaly and dataset condition.
3. the evaluations are well performed and explained, e.g all evaluates are done under different percentage of labeled anomalies. The author explains the evaluation results and insights which also provide directions for future research directions.
weaknesses: 1. The paper depends on 1 single method to extract the features for each of the newly added datasets that may cause the feature themselves to be biased towards certain methods.  
2. The paper briefly mentions using a generative model to generate anomaly, however, no more details are given in the main text or appendix. This seems critical for readers to learn about the 7 newly introduced datasets.
3. The author provides average rank across multiple datasets in the main section and attaches the full results in the appendix, but it will be helpful if the author could give some insights into the results, e.g highlight the best performance on a certain dataset. The full table is a little bit hard to interpret now.
4. Since the paper created 7 new datasets, maybe it's better to separate the dataset from the codebase for easier access
5. There seem to be a misuse of terms, e.g "real-world", in some places, the author refers to the newly added dataset as synthetic datasets and in some other places, the author refers to them as "real-world" datasets.
correctness: # Benchmark

The benchmark evaluation in ADBench looks correct.

# New datasets

The author introduces how the new datasets are generated without giving much details. We don't know if the dataset is biased or not.
clarity: Yes. The paper is well written. It summarizes its contributions at the very beginning and then splits the main text into 3 focuses which correspond to the 3 angles the author proposes. The paper is easy to read and the tables and figures are well explained.
relation to prior work: Yes. The author adds a table to compare the work with previous works and outlines the key differences. 
additional feedback: N/A
review_title: comprehensive benchmark with new dataset contribution

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Although there are many existing AD benchmarks, this paper proposes a comprehensive AD benchmark containing 55 datasets from different domains and covers both deep and shallow algorithms. Besides the comprehensiveness, this paper studies AD from different angles like types of supervision and anomalies which are highly beneficial for AD research. Moreover, all the datasets and source code are made public for reproduction.  
strengths: 1. This paper provides a comprehensive AD benchmark containing many domains' datasets. 
2. This paper provides multiple angles on benchmark anomaly detection algorithms: for example, comparing semi-supervised and supervised anomaly detection is fascinating; studying different types of anomalies can be helpful for AD algorithm analysis and model selection. 

weaknesses: The comparison of shallow and deep anomaly detection approaches can be improved. For example, this paper assumes the training data contains a certain amount of anomalies; however, some deep anomaly detection models (e.g., DeepSVDD, AE) believe the training data is anomaly-free so that they can learn the normal representations. Thus the current comparison results may not be fair.  
correctness: The claims are correct and sound. However, the evaluation methods for benchmark results could be improved (for example, the comparison of shallow and deep approaches may not be fair, the DeepSVDD assumes the training data is anomaly-free, but that's not the case for this paper's experiments.)
clarity: The paper reads well and the text is fluent. However, the figure 3 is not clear to me. some local anomalies look similar to the global anomalies.
relation to prior work: The related work section clearly discussed the difference between the proposed work and previous works.
additional feedback: To extend the scope of current ADBench, I would suggest to add some datasets that are relevant to explainable AD [1] and Fair AD [2,3]. 

[1] Pang, Guansong, and Charu Aggarwal. "Toward explainable deep anomaly detection." Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.

[2] Zhang, Hongjing, and Ian Davidson. "Towards fair deep anomaly detection." Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 2021.

[3] Shekhar, Shubhranshu, Neil Shah, and Leman Akoglu. "Fairod: Fairness-aware outlier detection." Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 2021.
review_title: Comprehensive AD benchmark with new comparison angles

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper provides a detailed and thorough benchmarking of tabular AD. They conduct studies on 30 detection algorithms’ performance among 55 benchmark datasets. The paper pays extra attention to the type of supervision, type of anomaly, and algorithm robustness. The conclusion of each comparison is interesting and helpful for future research.
strengths: The paper is well written and easy to follow. 

The comparison is comprehensive and covers a range of datasets

The paper analyzes the performance of models from each type of supervision (such as supervised, unsupervised, and semi-supervised).

The types of the anomaly are also taken into account during evaluation (such as local, global, dependency, and clustered).

The robustness analysis is interesting and well-thought.
weaknesses: My first concern is on the additional CV and NLP datasets that are firstly adopted and included in the ADBench. However, this part is not presented in the paper and remains unclear. While authors claim that AD benchmarks for CV and NLP are naturally different from tabular AD, it would be helpful how to adapt CV/NLP data to tabular data.
 
It is also interesting to discuss the difference between the anomaly, the saliency [1], and the camouflage [2]. What makes an object anomaly, salient, or camouflaged? Do the shallow or deep models learn such desired outlier/features?
 
 
From Table 1 the reviewer would expect to see that this paper conducts an extensive benchmark (coverage), analyze the difference between real-world and synthetic datasets for anomaly detection (data source), the difference in computational cost with respect to the model type (algorithm type), and the comparison angle. The current paper addresses well the expectation of the coverage and the comparison angles. However, several comparisons are missed. It would be interesting to include:

1.	What is the domain gap between the real-world and synthetic datasets? How do AD models perform when they are trained on a real-world/synthetic dataset and tested on another synthetic/real-world dataset?

2.	What is the inference time of both Shallow and DL models? The paper presents the running and the convergence time of the models. However, the inference speed should be also taken into account.
 
From Figures 4 and 6, we can conclude that shallow methods such as XGB, LGB, and CATB perform better than deep methods such as ResNet and FTTransformer. This is a strange conclusion since firstly deep NN has already shown better performance in other domains (CV/NLP). Secondly, ResNet and FTTransformer are more recent methods compared to XGB, LGB, and CATB. Thirdly, in the original publication [3], the performances of ResNet and FTTransformer are better than XGB and CATB. The author should clarify this issue. Maybe it is due to the evaluation metric. But from this regard, this benchmark is not complete and the conclusion from this paper is biased since more meaningful metrics should be taken into account.   

 
It would be also interesting to include the recent ICLR papers such as [4] in comparison.

[1 ]Salient Objects in Clutter: Bringing Salient Object Detection to the Foreground (ECCV 18)
[2] Concealed Object Detection (TPAMI 22)
[3] Revisiting Deep Learning Models for Tabular Data (NIPS21)
[4] Anomaly Detection for Tabular Data with Internal Contrastive Learning (ICLR22)
correctness: The reviewer is not certain about the evaluation protocol which seems to result in an abnormal conclusion. In this paper, the AUCROC and AUCPR are used to analyze the performance. From this metric, early models achieve better performance than recent deep methods. However, if the metrics become ‘CA AD HE JA HI AL EP YE CO YA MI’, it can be seen from [1] that deep models perform better. The authors should address this concern.


clarity: While the paper is overall well written, there are many typos in the current version. The reviewer lists several examples as follows, while there exist A LOT of others:

majority are designed -> the majority 

we focuses -> we focus

requirements in industry -> requirements in the industry

unsupervised algorithms is statistically -> unsupervised algorithms are statistically

best unsupervised methods -> the best unsupervised methods

are therefore limited to detect - > are therefore limited to detecting

improving detection performance, and leverage - > improving detection performance and leveraging

covers both shallow and deep learning (DL) algorithms, and consider -> and considers

latest unsupervised - > the latest

in natural language processing -> in the

as great addition to the community –> as a great 

more complex representation -> more complex representations

there are a group -> there is 

varying level of -> varying levels of

anomalies in the training set is known -> are known

anomalies at interest -> of interest

generation of normal samples are -> is

probability density function of generated anomalies are -> is

observe how do AD algorithms respond to it - > observe how AD algorithms respond to it

discuss more generalized impact -> discuss the more generalized impact

make prediction on -> make a prediction on

as testing set -> as the testing set
relation to prior work: The difference compared to prior works is trivial and significant.
additional feedback: Please refer to weakness to address the concern. The authors are also encouraged to go through the whole paper to correct the typos
review_title: The conclusion of each comparison is interesting and helpful for future research.

Meta-review: metareview: All reviewers agree on accepting the paper. The authors appear to have addressed the concerns of the reviewers.
confidence: 3: The area chair is somewhat confident
Decision: Accept