Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a contrastive learning method for unsupervised anomaly detection on general multivariate (tabular) data. The idea of the method is to train two encoders, one that embeds a subset of k features and one that embeds the residual d − k features, such that the resulting embeddings are closely aligned via the contrastive loss, where all other subsets of k features are taken as negative examples respectively. This is then repeated over all subsets of features of size k. Thus, the intuition of the approach is to extract common dependencies (e.g., correlations, statistical redundancy, etc.) between the features by achieving a small contrastive loss over the training data, thereby obtaining a high contrastive loss for anomalies (which presumably lack these common dependencies). An empirical evaluation on the Arrhythmia, Thyroid, KDD, and KDDRev datasets (where results from the literature exist) as well as 30 additional datasets from the ODDS library show that the proposed method performs favorably over other recent competing methods (DROCC, GOAD, COPOD) on tabular data.
main_review: *Pros*
+ Anomaly detection on general multivariate data (where few prior knowledge is available) is an important problem with many applications (fraud detection, etc.) that is relevant to the community.
+ The proposed method performs favorably over recent state-of-the-art methods (DROCC, GOAD, COPOD) on many datasets. Overall, the experimental evaluation is extensive (4 + 30 datasets), scientifically rigorous (including Wilcoxon signed rank tests for best vs. second best performing method), and includes sensitivity analyses for critical method hyperparameters.
+ The paper is overall structured well and easy to follow.
+ The work is placed well into the existing literature.

*Cons*
- The methodological novelty is rather low (contrastive learning for anomaly detection has been recently explored [1, 2], though limited to image data).
- There is not much of an explanation or intuition as to *why* the proposed method seems to work well. 


*Questions*
(1) Could you elaborate more on why simply taking consecutive features seems to work well, although tabular data is generally permutation invariant? I appreciate the empirical evidence provided in Appendix A, but I struggle to understand the underlying reason for why this simple heuristic seems to work well on many datasets.


*References*
[1] K. Sohn, C.-L. Li, J. Yoon, M. Jin, and T. Pfister. Learning and evaluating representations for deep one-class classification. ICLR, 2021. 
[2] J. Tack, S. Mo, J. Jeong, and J. Shin. CSI: Novelty detection via contrastive learning on distributionally shifted instances. NeurIPS, 2020.
summary_of_the_review: I think this work presents a simple method that shows significant improvements over previous methods in an extensive and scientifically rigorous evaluation, which makes this work a solid contribution to an important problem (anomaly detection on multivariate, tabular data). For this reason, I am leaning towards accepting this work, though I have some remaining questions and concerns (see above).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ## After rebuttal
I am impressed by the comprehensive experiments and baselines and increase my score. Authors provide very complete baselines and also compare in other datasets, which greatly reduces the concern that the proposed rule overfits to the ODDS benchmark.


## Summary
This paper learns a contrastive representation that helps differentiate between normal and abnormal data in the tabular data. Contrary to other contrastive learning methods which learns to differentiate between different examples, this paper instead differentiates between in-window vector and out-window vector for each example under a sliding k-sized window (k is a hyperparameter). Since the features are unordered, they shuffle the feature orders to get multiple score and average them. They show that it outperforms other baselines in a large suite of tabular data, ODDS benchmark, with default hyperparameter rules (e.g. the k is set based on number of samples N and d). They compare mainly with recent methods DROCC, GOAD and COPOD. 
main_review: I am very surprised this method performs so well, since this k-sliding window does not seem to make sense for tabular data since heterogeneous types of features exist (e.g. discrete, continuous or skewed distribution) and the order of features should not matter and thus the sliding-window approach should not work unlike images. But my concern is much relieved after seeing the complete set of experiments and the stability analysis, as well as some limitations the authors admit in the discussion.

My concerns are as follows (ordered by high to low concerns):
1. Since it's in the tabular data, I hope the authors can compare with long-lasted traditional baselines such as RRCF and KNN in ODDS benchmark (authors can use the default hyperparameter in the sklearn). 
2. The claimed that default set of hyperparameters achieve good performance is a bit exaggerated. A better way to phrase is a default "rule" of hyperparameter setting. Because this method still tunes hyperparameters but based on some rules-of-thumbs like the model stops when loss < 10^{-3} when d < 40 else stops when loss < 0.01, or k has to set to d - 150 when k > 150 etc. Overall I still like the stability of this method.
3. I am wondering if authors try adding the location index into the inputs of the model? This might help improve the model to learn better representation based on index j without learning independent model for each j.
4. The Dolan-More profile should be described more clearly. The text is quite confusing right now.
5. In the result of Arrhythmia, the claim is the best F1 score is 63 but all values are between 58.59% to 67.3%. Isn't the best score 67.3?


summary_of_the_review: Pros:
+ Good rule of hyperparameters which helps reproduce this method.
+ Complete set of experiments and ablation study for the architecture choices.
+ Good stability study of the sliding window k
+ Not sure how authors find out the effectiveness of tanh and two normalization schemes, but they seem to perform well.

Cons:
- No comparison to long-lasted baselines such as RRCF and KNN in the ODDS.
- I think the claim should be changed to "a default rule of hyperparameter selections performs well".

I would put the marginal acceptance for now but I will be happy to raise my score if the baselines are provided in the ODDS.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces an anomaly detection algorithm for tabular data based on contrastive learning in the semi-supervised setting (training set assumed to be only normal data). The contrastive learning task is based on masking: the features from a single training example are split into two groups (pairs), one being a subsequence of the features, and the other its complement set; the learning task is then to differentiate pairs of splits that overlap (positive) vs not (negative) (from the same split or not) for each sample. The contrastive loss defined by this task is used to train the model and as the anomaly score. The proposed method is extensively evaluated on a range of tabular datasets where it outperforms recent methods.
main_review: Strengths:
- Interesting use of contrastive learning for anomaly detection on tabular data
- Strong empirical performance outperforming recent baselines; extensive comparisons on ~30 datasets 
- Ablation studies and sensitivity analyses for hyperparameters are performed
- Paper is well written and easy to follow; limitations and assumptions are discussed
- Code is provided for reproducibility 

Weaknesses:
- My major concern is that this work is very similar to the recent ICML 2021 paper "Neural Transformation Learning for Deep Anomaly Detection Beyond Images", which aims to learn the appropriate masking strategy for tabular data; this is in some sense a more general formulation compared to the fixed masking strategy used for contrastive learning here. Comparison and discussion of this work should be included though it can be to some extent be considered contemporaneous, and does not perform as extensive comparisons on tabular data.
- It is a bit unsatisfying that the self-supervised task is solvable exactly without the need for the learned neural projections yet the success of the method relies on such neural projections; to be fair the paper does include some discussion of this point. Perhaps a more thorough exploration of the need for capacity constraints on the neural network will provide more insight into the method; one could experiment with different network architectures (e.g depth) and/or investigate how anomaly detection performance varies with overlap classification performance.  

Other comments/suggestions:
- It may be easier to compare the methods in Table 2 if the ranks and average ranks of the methods are provided.
- Typos:
  * In Table 3 caption: resutls -> results
  * In and around Eq (1) - $\Phi$ instead of $\phi$

summary_of_the_review: This is a paper presenting an interesting application of contrastive learning for anomaly detection on tabular data, with extensive experiments showing the strong empirical performance of the method. Ablation studies and sensitivity analyses are provided, as is the code for reproducibility. The main issue with this paper is the similarity to the somewhat contemporaneous ICML 2021 paper listed above, which should be discussed and compared to. On balance the relative novelty and strength of the experiments lean me towards acceptance. 

**Post Response Update:**
The authors have included a discussion of the related ICML 2021 paper and highlighted the differences, as well as added additional comparisons to baselines that support the superiority of the proposed method. I think the paper is a clear accept now.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - The authors propose a novel anomaly detection framework that can be useful for tabular data.
- The authors utilize the contrastive learning to learn the relationship between features in the tabular data. Then, using the loss as the anomaly scores for evaluation.
- The authors provide extensive experimental results which are promising.
main_review: 1. Assumption
- In the introduction, the authors assume that the feature relationship is class-dependent. 
- It would be good if the authors can provide some examples about this assumption and whether this assumption is practical.

2. Consecutive features & Same number of features
- The authors define the "consecutive features". 
- However, in tabular data, the order of the features are independent. 
- In that point of view, is there any reason that the authors are utilizing the "consecutive" features to extract the subset of the features?
- Also, can we provide some more flexibility for selecting the subsets of the features? Like a different number of features? 

3. Intuitions in Figure 1
- What is the intuition of selecting negative pairs?
- I understand that the positive pair can be a_i^j and b_i^j.
- However, it is hard to understand why the negative pair is a_i^m (where m is not j).
- For instance, in Figure 1, a_i^6 is a subset of b_i^3. I understand that we use different embedding functions. However, still the inputs are the subset.
- Maybe we can set the negative pair as a_m^j where (m is not j). 

4. Hyper-parameter optimization
- It seems like in the problem setting, we do not have a validation set. (Because we only have in-sample data).
- In that case, how did the authors optimize the hyper-parameters of the models? Like the encoder dimensions, learning rates, k, etc?
- There are some rules of thumbs in page 4. But not sure how they are determined. Are they determined by checking the testing performances? If yes, I think it is a somewhat unfair way of doing the experiments.
- Especially, it seems very heuristic of the thresholds determined by the dimensions (d). d < 40, d> 160, etc.

5. Baselines
- The authors have relevant baselines in the paper. However, some are missing.
- Maybe it would be good if the authors add the following baselines: (i) Isolated Forests, (ii) PIDForest.

6. Sensitive analyzes
- The authors provide various sensitive analyzes in the appendix and those are very helpful.
- However, in Figure 6 and 7, with very small k (like k = 1), the performances are similar to large k which is somewhat weird for me.
- For instance, if k = 1, it means that we do contrastive learning between one feature and d-1 features which makes highly imbalanced networks F and G. 
- I am not sure whether this kind of learning would achieve meaningful representations for anomaly detection.
summary_of_the_review: Strength
- The experimental results are quite promising and extensive.
- The proposed method is simple and easy to reproduce.

Weakness
- The intuitions of the proposed method is limited. It is actually my biggest concern on this paper. If this can be improved during the rebuttal period, I will increase my scores. Please see my detailed comments above.
- Some sensitive analyses are somewhat weird (e.g., k = 1).
- Some additional baselines are needed to complete the experiments.

-----------------------------------

Thank you for the detailed responses.
I carefully read the responses.
Some concerns (e.g., assumptions, consecutive features, and additional baselines) are resolved. Thank you.
However, there are still some remaining questions.

1. Intuition for negative pairing.
- It is still not clear why the authors select those pairs as the negative pairs?
- In figure 1, b_i^3 and a_i^6 are highly overlapped but still negative pair. 
- It would be good if the authors can explain about this clearer with proper intuitions.

2. k=1
- If k = 1, we do contrastive learning with the feature (with dimension = 1) and the feature (with dimension = d-1). 
- If the entire feature dimensions are large, the differences would be larger.
- I am not sure whether this is the right way to do contrastive learning between these large different features.

Although other reviewers are leaning to acceptance, I will stay with my original scores (5) until those above concerns are resolved.

Thank you.
review_title: null

Meta-review: The paper proposes contrastive learning for tabular data to improve anomaly detection. 
Strengths:
- Interesting and important problem.
- Usage of contrastive learning for anomaly detection in general multi-variate datasets is novel (as prior work mostly focuses on images)
- Extensive experiments with comparisons to multiple baselines on multiple datasets
- Well-written paper

The reviewers raised some concerns about novelty (in particular, the relationship to the closely related paper "Neural Transformation Learning for Deep Anomaly Detection Beyond Images"), hyperparameter tuning and additional baselines. The authors did a great job of addressing the concerns and multiple reviewers raised their scores. During the discussion phase, the consensus decision leaned towards accept. I recommend acceptance and encourage the authors to address any remaining concerns in the final version.

Additional AC comments:
- Please make sure that the camera ready version does not exceed page limits. https://iclr.cc/Conferences/2022/CallForPapers 
- " As far as we can ascertain, masking was not used for one-class classification before.": There's some related work on pre-training BERT for OOD detection (cf. https://arxiv.org/abs/2004.06100 or https://arxiv.org/abs/2106.03004) which might be worth discussing.
Decision: Accept (Poster)