Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an approach for tabular anomaly detection which is based on Partial Identification (PID) and on Generalized Additive Models (GAM) and extensions. The method works also in semisupervised settings and compares well with alternatives, as shown in a quite extensive experimental evaluation.
strength_and_weaknesses: Positive points
- The paper is well written and easy to read
- The topic is definitely interesting
- The structure of the manuscript is clear
- Experiments are extensive


Negative points
- Significance of the proposed method. 
My main concern is about significance of the proposed method. It seems that authors start from ideas presented in PID and PIDforest, for anomaly detection, and optimized such framework and created an effective pipeline by adding several carefully chosen ingredients. The resulting framework is definitely well performing, as shown in the experiments, but I’m wondering how large is its methodological contribution from the Anomaly Detection perspective (but of course this is my personal opinion). As for the theoretical contribution, I think it is not so relevant, being reduced to the three lines of page 5 (by the way, the proof of the proposition, found in Appendix, is not clear to me, especially the sentence which concludes the proof)

- Conclusions from the experiments.
Experiments are very extensive, based on 20 tabular datasets and involving many different competitors. One comment which applies to the whole analysis. The tables show bold values, which are meant to highlight the best results; however these do not derive from a rigorous statistical analysis, but simply by the rule “Metrics with standard error overlapped with the best number are bolded” (from the paper, caption table 2). Without a rigorous statistical evaluation conclusive observations can not be derived. I suggest authors to use the Friedman test followed by a post-hoc Nemenyi test, and to show results via critical diagram – see for example

Demšar, Janez. "Statistical comparisons of classifiers over multiple data sets." The Journal of Machine learning research 7 (2006): 1-30.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, only two comments on presentation:
- Section 6.3 of the results (part “Explaining anomalous data): this section is not completely convincing. Are you extracting something that other methods can not extract? 
Moreover, plots in Fig 4 are too small, it is very difficult to get colors, lines and in general the content

- References: in general there are many citations to arxiv papers, also for old papers. Please try to avoid this whenever it is possible. There are also some duplicated entries:
- Pang and Aggarwal 21
- Pang, Shen, van den Hengel 19
- Yoon et al 2020

For what concerns novelty: 
Estimation of the PID in section 5. Maybe authors can consider also the work by Goix and colleagues, which extends Isolation Forests by optimizing a criterion based on volume ratios (formulated however in a different shape, with respect to what authors did):

Goix, N., Drougard, N., Brault, R., Chiapino, M.: One class splitting criteria for random forests. In: ACML, pp. 1–16 (2017)


summary_of_the_review: Paper which proposes a carefully tailored pipeline for anomaly detection of tabular data. Some doubts on its significance. Large experimental parts which should be completed with a rigorous statistical analysis.

UPDATE AFTER THE REBUTTAL. 
I carefully read the responses, the clarifications, and the additional material, and I thank the authors for the significant efforts made in clarifying my doubts. Even if I consider that the paper has potential, I’ll maintain my score unchanged, since I still have few doubts, especially on the significance and on the analysis of the results (even if I acknowledge the efforts made by the authors in improving these aspects in the rebuttal)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a new anomaly detection method, namely DIAD, that uses Partial Identification (PID) as an objective to perform anomaly detection optimization with a tree structure of an existing generalized additive model. It is also flexible to use an additional loss function, such as AUC optimization or BCE loss, to utilize auxiliary labeled anomaly data in the semi-supervised setting. Due to the use of GAM model structures, it also offers a two-way feature interaction based explanation of detected anomalies. The effectiveness of the method is evaluated on 18 tabular datasets.
strength_and_weaknesses: The work aims to tackle both the interpretability and the capability of utilizing a few labeled anomaly examples in anomaly detection. It has the following strengths:
- Both of the interpretability and the sample-efficient capability are important to different ML tasks, including anomaly detection. The paper addresses both of these two important aspects.
- The presented method can work in both unsupervised and semi-supervised settings. Limited AD methods have such a property.
- Experiments on 18 datasets show the effectiveness of the method against popular unsupervised/semi supervised methods

Some issues that may require further investigation include:
- The technical novelty of the method is weak. The methods like PID were used in anomaly detection, and the AUC loss is a widely used loss function and it does not bring much improvement over BCE. The full method seems to be a combination of these widely used components.
- The method is weak in the unsupervised anomaly detection setting, which does not show clear improvement over the closely related competing methods like PIDForest and IF. There are also other more effective improved IF methods that tackle the studied issues in the paper, such as [a-c] and references therein. It would be more convincing to review these studies and possibly include them in the comparison.
- In terms of the semi-supervised setting, the proposed method involves two stages of training, unsupervised training and fine-tuning. I wonder whether the competing methods like CST and DevNet are trained from scratch, or only fine-tuned in a similar way as the proposed method. Questions here include how much the contribution is it from the unsupervised training in the semi-supervised setting? Does such a training strategy also help improve other existing semi-supervised methods?
- The work argues a two-way feature interaction-based anomaly explanation, but the presented result in Figure 4 is only the explanation of individual features. Figure 4(d) is a two-way interaction example, but the results indicate only individual features, e.g., the Area feature, are sufficient for the detection and interpretation.
- The interpretation results are limited to qualitative ones, and do not involve comparison to other methods, and thus, they are not convincing enough.
- There exist a large number of gradient backpropagation-based methods for anomaly explanation, especially on image data. This type of methods is applicable to tabular data, too. The authors are suggested to elaborate more based on more related work in the discussion in Related Work. 

**References**
- [a] "Sparse modeling-based sequential ensemble learning for effective outlier detection in high-dimensional numeric data." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1. 2018.
- [b] "Extended isolation forest." IEEE Transactions on Knowledge and Data Engineering 33, no. 4 (2019): 1479-1489.
- [c] "Generalized isolation forest for anomaly detection." Pattern Recognition Letters 149 (2021): 109-119.
clarity,_quality,_novelty_and_reproducibility: The paper clarity and technical novelty is weak. The reproducibility is fair.
summary_of_the_review: Considering both the pros and cons above, the paper is at the borderline and towards the reject side.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new interpretable method for unsupervised or semi-supervised anomaly detection on tabular data, in the presence of noisy or unlabeled data. 
strength_and_weaknesses: Strengths:
- Innovative approach based on generative additive models.
- The model is explainable.
- There is a little technical contribution (Prop 1) creating some insight into the method's inner workings.

Weaknesses:
- Section 6.1. on unsupervised AD experiments has some problems in terms of the chosen datasets and baselines: [1] determined [2] as the overall best performing deep AD method on tabular data. However, the method [2] is missing in the comparison here. On the 14 datasets from Gopalan et al., the proposed approach shows inferior performance, except for SatImage, where a tiny improvement of 0.4% is achieved. Out of the 9 datasets from Pang et al., only 6 have been selected. To avoid dataset selection bias, all 9 should be analyzed. Besides NeuTraL, also DSVDD - a common baseline - is missing in the comparison. 
- In the noise experiments, only two datasets have been analyzed, which is insufficient to draw conclusions.
- Section 6.2 on semi-supervised AD: These experiments look better. The approach achieves higher AUROC than the Devnet baseline. However, the original Devnet paper reported on AUPRC, which is why I cannot confirm that the present results are consistent with their analysis. The semi-supervised DSVDD baseline is missing.



[1] Maxime Alvarez, Jean-Charles Verdier, D'Jeff K. Nkashama, Marc Frappier, Pierre-Martin Tardif, Froduald Kabanza: A Revealing Large-Scale Evaluation of Unsupervised Anomaly Detection Algorithms. https://arxiv.org/abs/2204.09825
[2] Chen Qiu, Timo Pfrommer, Marius Kloft, Stephan Mandt, Maja Rudolph Proceedings of the 38th International Conference on Machine Learning, PMLR 139:8703-8714, 2021.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to read and follow. The approach is novel. The reproducibility is rather high, but could be improved at places (see above).
summary_of_the_review: In summary, the authors propose an innovative approach to AD on tabular data. The experimental evaluation shows good promise, but is not entirely convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper develops an interpretable anomaly detection framework, DIAD,  by combining the NodeGAM structure with the PID objective. The NodeGAM provides an interpretable mechanism, and PID can be trained for anomaly detection. Overall, the framework makes sense to me. The paper includes detailed experimental evaluation by comparing with several state-of-the-art baselines on multiple datasets.
strength_and_weaknesses: Strength
1. The paper targets an important research problem, interpretable anomaly detection.
2. The proposed framework leverages the existing techniques and makes sense.
3. Detailed evaluation is conducted to show the performance of anomaly detection.


Weakness:
1. In the unsupervised setting, the proposed DIAD can achieve better or comparable performance on large datasets, but may not be as good as baselines when the datasets only have small amounts of samples. Do the authors have any insight into such observations? 

2. My major concern is in the interpretation part. How to achieve interpretable anomaly detection based on DIAD is not very clear to me. For example, given an anomaly, do we explain the result based on the ranking of features evaluated by the sparsity values or based on the feature-wise function? Meanwhile, as Mammo. only consists of 6 features, I am also curious about the interpretability of the proposed model on high-dimensional data. I believe for low-dimensional data, it is not very challenging for domain experts to exam the anomalies. 
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is not very hard to follow as long as the readers have knowledge about NodeGAM and PID. By combining these two approaches, DIAD can achieve interpretable anomaly detection on tabular data, which is somewhat novel. The authors promise to provide code upon paper acceptance.
summary_of_the_review: I feel that the proposed approach has some merits, but I am also not very sure whether the combination of NodeGAM and PID is trivial or challenging. Meanwhile, the term "interpretable anomaly detection" in the title really draws my eye, but the evaluation in this part is weak in my view. I understand currently there is no golden metric to evaluate the interpretability, but maybe showing more cases could be more convincing, just like the papers in the CV area. 

I also have one more question, which may be outside the scope of this paper: If the purpose is to achieve sample-wise interpretation (explaining the anomalies), could we apply other post-hoc interpretation techniques to interpret PIDForest?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a tabular anomaly detection method, referred to as DIAD, which tackles both interpretability and leveraging small amount of labeled examples. It bases its development on partial identification (PID) and generalized additive model (GAM). Certainly, some critical changes, instead of just combining existing methods, are proposed to make the method effective. However, most of reviewers feel that such changes made in combining PID and GAM are not substantial. In addition, the evaluations in the aspect of interpretable anomaly detection are weak. While the authors made efforts in responding to reviewers’ concerns, the main concern on the limited novelty kept them to stood by their earlier decisions. Therefore, the paper is not recommended for acceptance in its current form. I hope authors found the review comments informative and can improve their paper by addressing these carefully in future submissions.

justification_for_why_not_higher_score: Some changes made in combing two existing methods (PID and GAM) are marginal, so I suggest it not to be accepted in its current form.
Decision: Reject