Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for using any anomaly detection method in an Active Anomaly Detection (AAD) setting. The proposed method consists of three phases: 1) training of a proxy model to emulate the results of the base anomaly detection model(s), 2) training of a data transformation module using human feedback on the anomalous samples, and 3) applying the transformation layer in front of the base anomaly detectors in the data pipeline. The proposed method shows superior results in AAD across several different benchmark datasets. The paper also uses some ablation and comparison results to support the underlying insight beyond the method, namely that there needs to be a diversity-aware sampling strategy for presenting anomalous points to a human.
strength_and_weaknesses: The paper has strong empirical results, is attacking a significant problem, and has a novel approach to making its method agnostic to the base AD algorithm(s). The performance increases seen by using the proposed method, especially with different types of base AD models, are very convincing of the soundness of the presented method and its underlying insight. I also especially appreciate the way the paper attacks the problem of making an agnostic method by attacking the data featurization as a way of improving the result; it very much embodies the fundamental tenets of data-centric data science.

There are a few weaknesses in (or questions about) the correctness of the proposed method and the clarity of the paper.

•	Why does the method need a proxy model? If the proxy model is frozen at Phase 2 of training when the representation adjuster is trained, why not just take the user feedback and base AD model results and train the neural network adjuster directly? It's not clear to me why there needs to be a proxy model for the method when it seems like one could take the feedback loss and the consolidation loss directly from the base AD outputs to train the representation adjuster.

•	Why is the proxy model a single-layer neural network with no bias term? Roughly speaking, adding depth to a neural network improves its ability to do non-linear transformations, so why not opt for a deeper network for the representation adjuster?

•	How did you decide what the proxy model should be? Why is it a neural network and what is the architecture of that network? Were other models considered?

•	Figure 6 does not really show the comparison of using a base model to using a base model + LMADA, which is the claim in section 4.4. It shows how different base model + LMADA models perform, which is also good, but there needs to be something more to show how adding LMADA for any base model improves that model.

clarity,_quality,_novelty_and_reproducibility: The novelty and quality of the paper are high. While the problem they are attacking is not novel, the creation of a model-agnostic method by transforming the data rather than the model is novel. The breadth and depth of the empirical results greatly benefit the quality of the proposed method. There are some questions that need to be addressed (see the previous section) to improve the clarity and, possibly, some aspects of the quality of the paper.
summary_of_the_review: Given the good insight in the paper (diversity-based sampling) and the solid empirical validation done to support that insight, the paper probably merits publication. However, I do have some concerns about the proposed method that prevents me from being sure that it should be published.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a diversity-aware query strategy for active anomaly detection. The proposed query strategy is based on DPP. A positive semi-definite pairwise similarity matrix between highest ranked anomalous instances is first constructed, and then a subset of instances is computed by maximizing its principal minor. The subset of instances selected in this manner has been shown to maximize diversity in previous literature.
strength_and_weaknesses: 1. The paper targets an important problem in anomaly detection.

2. Section 4.1 Datasets and Settings: The dataset information is not complete. For example, it is not clear which categories in each dataset were treated as anomaly and normal. The Table 1 in appendix has different number of instances from cited previous works (Das et al., Siddiqui et al.).


3. Section 4.1: "We run 5 feedback iterations and query 20 samples in each iteration. Same" -- 20 samples per iteration is an arbitrary number -- specifically for smaller datasets this is pretty large. (Das et al., Siddiqui et al.) used 1 query per iteration. For fair comparison, need to show results with this setting and another set of ablation experiments where the number of queries per iteration is varied.


4. The paper claims that one of the contributions is the sampling strategy. However, the experiments are not correctly designed to demonstrate its effectiveness. Typically, active learning algorithms have two aspects which are/should be independently replaceable: (a) the query strategy to select samples for user feedback (such as query most anomalous) and, (b) update the algorithm/model parameters with the new labeled data from user. The correct way would be to use the existing benchmark active learning algorithms and replace just their query strategy with the new strategy; then check whether the performance improves/degrades. For the query strategy to be useful and generic, it should work well with other algorithms instead of being tied to a specific one.


5. Section 4.2: "Specifically, we calculate F1-Score on the entire dataset after finishing an iteration of feedback." -- This is inappropriate for active learning. The F1-score should be computed on an independent dataset, or a different metric should be used suitable for active learning.


6. Figure 5: The plot of F1-score along the y-axis is improper. Consider for example if there are exactly 20 true anomalies in the dataset and all get detected in the first iteration. Then F1 will be 1.0 in the first round and then decrease monotonously over successive iterations simply because the precision will decrease. This gives a wrong impression about the algorithm behavior -- that the performance degrades with successive iterations, whereas, that is not true. For an active anomaly detection algorithm, it would be more appropriate to measure the % of true anomalies detected with each feedback iteration (or the AUC).


7. Section 3.2: "In this way, LMADA achieves feedback incorporation in a lightweight manner, only with a non-linear representation transformation." -- It is misleading to call this approach 'lighweight' just because one (last) stage appears to be simple.
clarity,_quality,_novelty_and_reproducibility: The paper is mostly easy to understand, however, some things need better clarification e.g., it is not obvious which base detector has been used for the plots in Figure 5. The paper presents a mildly novel work, but the overall architecture is quite convoluted. It is possible that the presence of the proxy neural network hinders explainability of the anomalies.
summary_of_the_review: The paper leaves out crucial experiments that should compare just the selection strategy across active learning algorithms (i.e., replace the selection strategy in existing benchmark active learning algorithms with the proposed DPP technique) with existing standard strategies. Due to this, the paper falls short of being technically sound.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper focuses on the two critical issues in ADD problem and proposes a lightweight, model-agnostic, and diversity-aware AAD method. The first issue is the negligence of lower-ranked samples in active learning, and the second issue is the generalization of the specified unsupervised detector to other models given different datasets or tasks. The new models manage to address both issues and demonstrate on several benchmarks. 
strength_and_weaknesses: Strength:
* An interesting diversity-aware sample selector that not only considers the base detector scores but also the diversity of the samples was proposed and developed.
* A novel model-agnostic tuner is developed to integrate both the base model and user feedback. 
* More specifically, first, a proxy model is developed to mimic the base detector. Second, the proxy network is frozen, and a new representation adjustor is added to learn new feature space, which seems a concise and effective design.

Weakness:
* There is no significant weakness identified in this paper. While the sample selector might be time-consuming and the time complexity of Determinantal Point Process (DPP) is O(n^2), the run time, in reality, will most likely not be overwhelming due to the limited amount of anomaly samples. Another minor to consider is adding additional experiments, although the current meta-ADD dataset has 24 tasks.

clarity,_quality,_novelty_and_reproducibility: The paper is organized and written well. The paper is novel in the sense that the authors identified two issues in AAD, and manage to solve them through an interesting framework that makes itself distinct from conventional solutions. The details are well elaborated, based on which the paper can be reproduced.
summary_of_the_review: In brief, the paper solves critical problems in AAD through an interesting lightweight, model-agnostic and diversity-aware AAD (LMADA). The nature of model-agnostic allows it to extend to other learning models or tasks with ease. The model has been evaluated extensively on meta-AAD datasets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a lightweight, model-agnostic and diversity-aware active
anomaly detection method. It takes the diversity of samples into account and designs a diversity-aware sample selector powered by determinantal point process. It proposes a model-agnostic strategy to incorporate feedback information, which approximates diverse unsupervised detectors with a unified proxy model.

strength_and_weaknesses: Strength:
1.	It considers the diversity of samples instead of anomaly score top-selection strategy for the feedback querying, preventing the over-fitting situation to the top-ranked samples
2.	The proposed method shields the details of different unsupervised detectors and achieves lightweight feedback incorporation, only via a non-linear representation transformation.
3.	It conducts extensive experiments on eight public datasets and outperforms other comparative AAD approaches

Weaknesses:
1.	The proposed sampling method is somewhat incremental as it simply uses an existing method, DPP, to select samples. The authors may need to re-summarize the main contributions of the paper. The contributions are not clear.
2.	The proposed sampling method making the pre-truncated ratio alpha too important. It takes efforts to select such alpha technically by extensive experiments. And such alpha may differ in various dataset. For example, as shown in Appendix. A.8, when the ratio changed from 10% to 8%, the performance will decrease by 5%, which is too sensitive.
3.	It lacks the comparison of different sampling methods from selecting samples from the top alpha samples, like simple random sampling. I mean, the proposed method uses DPP to select diverse samples from top 10% anomaly data, but simply using random sampling to select samples from the same top 10% anomaly data can also guarantee the data diversity.
4.	The experiments of LAMDA under different base unsupervised detectors only consider classical unsupervised detectors like PCA, OCSVM. When using deep model, I am not sure whether the proxy model could mimic the detector or the cost in time and memory will be too large.
5.	This paper uses a proxy model to imitate different anomaly detectors. Authors claim that it is model agnostic. However, when using different anomaly detectors, the learned parameters in the proxy models are different. Thus, I think it is not model agnostic.
clarity,_quality,_novelty_and_reproducibility: The writing is clear. The idea of considering diversity samples is good but it simply uses the existing DPP to solve the problem. It is hard to reproduce the results because there is no source codes or network details provided in this paper.  
summary_of_the_review: As listed in the strength and weaknesses, the experiments are not convincing. It would be better if authors can add supplementary experiments on the model-agnostic evaluation and sample selector validation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new method for active anomaly detection (AAD), where users can give feedback on the results of anomaly detection to tune the anomaly detection model.  The proposed approach overcomes two limitations of existing approaches.  First, while the existing approaches select most anomalous samples to get users' feedback, the proposed approach induces diversity in the samples via determinantal point processes (DPP).  Second, while the base anomaly detection method and the active sampling methods are tightly coupled in existing approaches, the proposed framework allows the decomposition, so that the proposed approach can work with any base anomaly detection method.

A major strength of the paper is in its strong empirical support, which demonstrates that the proposed approach significantly outperform existing AAD methods in a wide range of tasks.  This strength clearly outweigh the paper's weakness that the technical novelty of the proposed approach is rather limited, as the use of DPP to induce diversity in this context is rather standard.
justification_for_why_not_higher_score: Since the novelty of the proposed method is not particularly high, the paper is likely to have impact within the area of active anomaly detection.
justification_for_why_not_lower_score: The paper proposes a novel approach with strong empirical support for the important subject of active anomaly detection.
Decision: Accept: poster