Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors provide a long-time data created by using honeypots to detect anomalous traffic. The characteristics of the data have been studied by using TSNE and optimal transport. The data is divided into train, near and far sets to understand the performance of anomaly detection methods.
strengths: - This is quite a useful datasets especially because data is collected over a long time period.
- The tasks are well-defined and studies.
- Data shift is an upcoming research area in ML, and the dataset would be very useful.
weaknesses: The code repository is under developed. Tutorials are not well-structured, they do not even have comments. 
correctness: - The dataset is constructed in a sound way, and the evaluation methods (e.g., Pr-AUC) are appropriate.
clarity: Yes, the paper is well written.
relation to prior work: Yes, the prior work is discussed in the related work section.
additional feedback: - the websites must be better designed, hosting plans should be made clear.
- Github repo should have better tutorials or starting points
review_title: An unsupervised temporal anomaly dataset

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper proposes a distribution shift benchmark for unsupervised anomaly detection. The dataset is built over a network traffic data Kyoto-2006+ and the test data is split into three distributions IID, NEAR, and FAR chronologically. Some analyses based on t-SNE visualization, Jeffreys divergence, and Optimal Transport Dataset Distance are performed to show the inherent shift of Kyoto-2006+ over time. Furthermore, it conducts experiments on this dataset with some classic anomaly detection methods and a Bert-based model to check the impact of the distribution shift on IID models. A basic distillation technique is introduced to improve the anomaly detection performance on shift data.
strengths: 1.It introduces the distribution shift problem to an unsupervised learning setting: Anomaly detection. The visualization and statistical analyses indicate the inherent shift of Network traffic anomaly detection data over time and the experimental results of IID models on shift data show the accuracy decrease caused by distribution shift. The proposed problem is practical and meaningful.

2.This paper is well-written and easy to follow, the details of how to construct shift data are provided. The experimental setup is reasonable and could support the author's claim.
weaknesses: 1. As I know, there exist many time series anomaly detection datasets, such as KDD99[1], SWaT[2], and SMAP[3], but this paper includes only one dataset edited from an existing datasets, which is not enough for a solid benchmark research. 

2.More comparison baselines should be considered, including but not limited to reconstructed-based methods, GAN based methods.
3.The finetune mode and distillation mode are not detailed, their pseudocodes should be provided in the Appendix.

[1] Richard Lippmann, Joshua W Haines, David J Fried, Jonathan Korba, and Kumar Das. The 1999 DARPA off-line intrusion detection evaluation. Computer Networks, 34(4):579 – 595, 2000. 

[2] Aditya P Mathur and Nils Ole Tippenhauer. SWaT: A water treatment testbed for research and training on ICS security. In International Workshop on Cyber-physical Systems for Smart Water Networks, pages 31–36, 2016. 

[3]KyleHundman,ValentinoConstantinou,ChristopherLaporte,IanColwell,andTomSoderstrom.Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In International Conference on Knowledge Discovery & Data Mining, pages 387–395, 2018. 
correctness: The dataset is built on a previous dataset. It utilizes ROC-AUC and PR-AUC to evaluate the baselines, and the experimental design is generally OK. However, the latest methods of anomaly detection were not included, and experiments on just one data set are not convincing enough.
clarity: This article is well organized and written.
relation to prior work: Yes. It claims that this is the first work introduce distribution shift to unsupervised learning.
additional feedback: 1. In Table 1, the performance of Bert on NEAR is better than IID, What causes this phenomenon？ Is it because Bert take sequence as input?

2. Check the coordinate values of the diagram. For example, in Fig 8, ROC_AUC and PR-AUC should be in the range [0,1].
review_title: A benchmark for unsupervised anomly detection but with limited contribution

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors curated an anomaly detection dataset from a popular network intrusion detection dataset named Kyoto-2006+ and the proposed dataset focus on the distribution shift of the data over time. Extensive analysis of the dataset demonstrates its non-stationary property from multiple perspectives. Some preliminary experiments also show the performance degradation of classic anomaly detection algorithms due to the data distribution shift. Last, the authors compare different continual learning strategies for mitigating model performance degradation on the proposed benchmark and claim the necessity to address the distribution shift. 
strengths: 1. The presentation of the paper is clear and easy to follow.

2. The extensive analysis of the dataset is informative, which helps the readers and future benchmark users understand the dataset.

3. The experiments of anomaly detection baselines and the mitigation strategies are strong evidence to support the significance of the proposed benchmark. 
weaknesses: The anomaly is usually defined as data samples that differ from most data samples. However, the proposed dataset has 89.5% anomalies, and I think the task is more like a network intrusion detection task instead of a typical anomaly detection problem. Since the authors mention that Kyoto-2006+ is a classic dataset, the authors should give a comprehensive review of related work using Kyoto-2006+, and it would help me to understand how previous works use the dataset and the contribution significance of the proposed dataset in this paper. Hope the authors can provide more information regarding my concerns above.


correctness: Though the paper title includes the benchmark, I regard this paper as a dataset paper because the GitHub repo does not include all baselines evaluated on the dataset. As a dataset, it is constructed in a sound way with detailed information on how data is preprocessed.
clarity: Yes.
relation to prior work: The authors have discussed the previous distribution shift dataset and traffic anomaly detection dataset but miss the literature review of works that use the Kyoto-2006+ dataset.
additional feedback: 1. The authors propose a BERT-based anomaly detector, a novel approach to detecting tabular data anomalies. It would be better if the author could open-source the code to facilitate future research.

2. The authors claim the paper is a benchmark paper, but the GitHub repo only has the code for the IF baseline, and the paper does not have the implementation details and parameters of the baselines required by the NeurIPS benchmark track CFP. I suggest the authors either add the missing information to make the paper a benchmark one or rewrite the paper to a dataset paper.

3. Figure 4 shows the distribution shift intuitively, but it takes me some time to interpret, I suggest the authors explain the differences between rows and columns.

4. Please respond to my concerns mentioned in the Weakness section.
review_title: A good dataset focusing on data distribution shift

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this work, the authors proposed the first unsupervised anomaly detection benchmark with naturally occuring distribution shift. The authors highlighted the non-stationary nature of the data using per feature analysis, t-SNE plots and an Optimal Transport approach. A novel protocal splitting the data called AnoShift is also proposed which splits the data in IID, NEAR and FAR testing splits. Empirically, it is observed that more severe performance degradation occurs in the FAR split than NEAR split than IID split. Future methods can evaluate upon all three splits in increasing difficulty. The dataset and code are available and pro-processed version of the data is also easily accessible.  
strengths: * Understanding distribution shift (particularly in the anomaly detection literature) is relatively under-explored. The proposed Kyoto-2006+ dataset is an important step in this direction. 

* The Kyoto-2006+ dataset covers a span of 10 years with naturally occuring distribution shift. The authors also empirically validated the performance degradation of models due to distribution shift using proposed split strategies.

* Dataset statistics and visualizations are explored extensively in Section 4 to analyze the distribution shift. 

* data is open sourced already and can be easily downloaded / loaded into ML pipeline
weaknesses: * In the preprocessing stage, numerical features are converted to categorical ones to use in language modelling. Is this performed for all methods? How would the performance be without this conversion (using the numerical values directly)? 

* Is there a reason in Section 4.1 that only t-SNE visualization is chosen? How would Figure 4 look like with a PCA plot (with just 2d PCA).

* Network intrusion detection has often been studied in literature as a graph anomaly detection problem, for example in [Fast and Accurate Anomaly Detection in Dynamic Graphs
with a Two-Pronged Approach](https://arxiv.org/pdf/2011.13085.pdf) and [SEDANSPOT: Detecting Anomalies in Edge Stream](https://dhivyaeswaran.github.io/papers/icdm18-sedanspot.pdf). The DARPA dataset is a well-known network intrusion detection dataset formulated as a dynamic graph (see above two references). Have the authors considered formulating this dataset as a dynamic graph too? Are there limitations in the data which restricts it from being a graph dataset? If not, it is encouraged that the authors also considers converting the dataset into a graph dataset for use in the graph anomaly detection community. 

*In the appendix it was mentioned that "we discard the source and destination IP addresses, source and destination port numbers." Is there a specific reason for doing this? With proper anonymization, these features can also be kept and used in a graph format. 
correctness: The claims made in the submission are correct. The dataset is constructed in a sound way and experiments performed correctly. 
clarity: Yes, the paper is relatively well-written. Some clarification such as those in weakness section can help improve the writing of the paper. 
relation to prior work: Yes, most of the related work are discussed clearly. 
More discussion on dynamic graph anomaly detection methods would be helpful. See [Fast and Accurate Anomaly Detection in Dynamic Graphs
with a Two-Pronged Approach](https://arxiv.org/pdf/2011.13085.pdf) and [SEDANSPOT: Detecting Anomalies in Edge Stream](https://dhivyaeswaran.github.io/papers/icdm18-sedanspot.pdf).  
additional feedback: I ask the authors to address limitations from the weakness section and in particular, examining if a dynamic graph format of the dataset can be provided as it would be valuable to the temporal graph / dynamic graph learning community. 

[update] The authors have addressed my concerns and I believe this paper will be a valuable contribution in studying distribution shift
review_title: Interesting Dataset with Naturally Occuring Distribution Shift

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: 1)  this paper analyzes Kyoto-2006+, which is a network traffic dataset for unsupervised anomaly detection tasks, and demonstrates that it is affected by distribution shifts.

2) this paper proposes a chronology-based benchmark to enable a better estimate of the model’s performance in an environment close to the real world.  This benchmark splits the test data based on its temporal distance to the training set and introduces three testing splits IID, NEAR, and FAR.  

3) this paper proves that by acknowledging the distribution shift problem and properly addressing it, the performance can be improved compared to the classical IID training.
strengths: 1) The exploration of this benchmark in unsupervised learning is encouraging since such works can reduce the time and money burden brought by labeling data.

2)  The idea of splitting the test dataset into three splits according to their temporal distance to the training set is straightforward and reasonable. This benchmark provides a close-to-real-world way to evaluate the performance and generalization of a trained model. 

3) The distribution shift problem is analyzed via different visual experiments and the performance improvement of a model by solving this problem is verified via corresponding experiments, which gives some valuable insights into the dataset and benchmark community.
weaknesses: 1) The writing of this paper is not standardized enough. For example, the terms or abbreviations appearing for the first time in a paper should be explained, but I don’t know the meaning of ‘IID’ and other words. In addition, there are some ill sentences in this paper, such as the 56-th row. 

2) This paper lacks the comparative experiments between the unsupervised benchmark proposed in the paper and the existing supervised methods. It is really valuable to clarify the gap between unsupervised and supervised methods and encourage more people to reduce this gap. I think the authors can consider adding this experiment as the future work of this paper.

3) The explanation of some experimental results in this paper is not clear. Take Tab.1 as an example. Why the performance of the transformer model proposed in this paper (BERT for anomalies) in the IDD split is lower than the one in the NEAR split? Why the performance of this model is lower than other compared models except IsoForest in FAR split?
correctness: Yes.
clarity: The paper is overall well-motivated and easy to follow. And I think it would be better if the authors corrected this mistake in the first weakness I propose.
relation to prior work: In Section 2, the authors discuss the relation and difference between this benchmark to other benchmarks targeting distribution shift. They also discuss the relation anomalies.
additional feedback: No.
review_title: In this paper, the authors analyze the distribution shift in the Kyoto-2006+ network traffic dataset from different perspectives using the t-SNE and Optimal Transport Dataset Distance method (OTDD). They further propose an unsupervised anomaly detection benchmark, AnoShift, and validate the performance improvement by acknowledging the distribution shift problem and properly addressing it.

Meta-review: metareview: Four out of five reviewers recommend acceptance of the paper with a score of 7. These reviewers praise the quality and importance of the dataset. Moreover, all reviewers acknowledge the quality of the writing.

The authors have constructively engaged into the discussion with the reviewers, and have put a lot of effort into drastically improving their submission based on all reviewers' comments. The weaknesses raised by the reviewers have been successfully addressed.

In particular, the reviewer (dm8f) who gave a score of 4 raised the following concerns:

1. There exist other time series anomaly detection datasets.
2. More anomaly detection methods should be considered in the experiments. 
3. The finetune mode and distillation mode are not detailed, their pseudocodes should be provided.

The authors have addressed these concerns as follows:

1. In their answer, the authors went through an extensive list of existing benchmarks and justified why they are not appropriate to solve their focused problem, and how their dataset differ from those. The main added value of the proposed dataset is its timespan and the fact that its occurring distribution shifts are natural (as opposed to synthesized).
2. The authors have expanded their experiments with more anomaly detection models. The new results corroborate the previous conclusions.
3. The authors have added the pseudocode algorithm in Appendix and have added code to replicate their experiments in the git repository.

I believe that the authors have sufficiently addressed the reviewer's concerns (and more).

For these reasons, I recommend acceptance of the paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept