Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper is a combination of Latent Outlier Exposure (Qiu 2022) and an active learning strategy from the same group. The idea is to exploit knowledge about the outlier rate to control active learning sample selection during training. The assumption is that finding anomalies in the (unlabeled) training set will improve the quality of anomaly detection in the test phase. A lot of experiments on different ML tasks (images, medical images, videos, etc.) demonstrate improvement over existing techniques. The authors also support their claims with some theorems. 
strength_and_weaknesses: I like the rigorous analysis from a theoretical perspective and the extraordinary evaluation of their method on real data. The paper is well-written and builds upon existing publications. I do not agree with the assumption or application case. In my opinion, anomaly detection is no longer such a task if you are able to identify such anomalies from the training data. It then becomes a supervised task with the anomalous events being an additional class. This statement might sound philosophical. However, if you do active learning in an supervised setting, the authors ignore methods from active learning. One example is EMOC, see Freytag et al.: Selecting Influential Examples: Active Learning with Expected Model Output Changes. European Conference on Computer Vision (ECCV). If we know that there are novel classes in our training set (the known one is the inlier data, the novel one the outlier), we shall also study methods that combine active learning in a continuous learning scenario, for example, Kaeding et al.: Active and Continuous Exploration with Deep Neural Networks and Expected Model Output Changes. NIPS Workshop on Continual Learning and Deep Networks. 2016. I find that the authors should consider such work as well and include it in the comparison. 
clarity,_quality,_novelty_and_reproducibility: This is well-done work. Novelty is a bit limited since existing methods are combined. However, I am honouring the theoretical study found in the paper as a novelty. The paper is very well written and easy to follow. I see good reproducibility. 
summary_of_the_review: The paper tackles a (from my perspective) very special (and not such common) case in anomaly detection. For me, it is not clear whether the detection of real anomalies and outliers can be improved. Real anomalies are those who have never been seen before, for example, novel classes in a continual learning scenario. It is quite natural that one can improve on anomaly detection if one can optimize the model during training on that already. Then, anomaly detection (even with only 10% outlier rate) boils down to a supervised setting (of course, an unbalanced one). Since the author missed to take this perspective as well and as a consequence to study and compare with literature in areas like normal active learning and continual learning, I am not that happy about the paper. Finally, I am missing experiments on image data beyond X-MNIST, like one can find results for normal active learning.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a diversified query strategy for an anomaly detector that can be applied in an active learning setup. The querying strategy is based on querying diverse cluster centers seeded by k-means++.
strength_and_weaknesses: 1. Abstract: "However, correctly identifying anomalies requires an estimate of the fraction of anomalies in the data." -- This has never been proven (to my knowledge). While specific algorithms might make assumptions, this is not true in general. Anomalies are by nature unknown unknowns. Even their fraction is not knowable. The paper probably makes this statement w.r.t the specific LOE algorithm discussed and should clarify that or remove this statement.


2. Section 2: "... not comparable to Pelleg and Moore (2004), Siddiqui et al. (2018), and Ghasemi et al. (2011), who fit a density model on the raw input data ..." -- Siddiqui et al. is tree-based. They do not fit a density model.


3. Section 3.2: "... determined by the smallest radius \delta such that ..." -- Is the radius \delta here an absolute distance measure, or in normalized units? It does not make sense if the distance is unnormalized and if it (distance between data points) could vary from zero to infinity.


4. Section 3.2: "... all unlabeled points are within distance \delta of one queried sample of the same type." -- This makes strong and oversimplifying assumption about the data. It basically means assuming that data from various classes are tightly clustered together.


5. Section 3.2: "... we propose a querying strategy that encourages diversity: k-means++ ..." -- Querying cluster centers is a classic active query strategy and not novel. For example, [r1] proposes an approach to query the centroids of clusters of instances that lie closest to the decision boundary.


6. Assumption 2: I suspect Assumption 2 has fallen for a statistical fallacy.

As an analogy to the Assumption 2 made here, suppose we consider undergrad students at a top ranked US university. The average (say) SAT scores will be very high and the distribution of the scores around that (high) average will be statistical noise -- just like the off-diagonal correlation mentioned here. However, that does not mean that the distribution of SAT scores among the general population will be the same as at the university.


7. Section 4: "... and all |Q| queries are collected at once." -- This is not really an active learning setup. For comparing with active learning algorithms on a fair basis, the algorithm should iterate over multiple label feedbacks from a (simulated) human-in-the-loop.


8. Table 2: "For all experiments, we set the contamination ratio as 10%." -- Experiments should be run with actual contamination factor in the datasets, not the artificial 10%.


9. Section 4.2: "... which have an outlier ratio of at least 30%." -- This amount of anomalies does not really make it an anomaly detection problem. The set of experiments are missing some harder real-world datasets where the anomaly fraction is really small ~1-2%.


10. Appendix C: "Hybr2. This hybrid strategy by Das et al. (2019) makes positive diverse queries." -- The main query strategy for diversity in Das et al. (2019) is based on compact description. The strategy explained here is likely one of the alternatives proposed on the related github repo (https://github.com/shubhomoydas/ad_examples). This needs to be clarified. It should be noted that the compact description based approach in Das et al. is basically a clustering-based approach.


References:

[r1] Xu, Z., Akella, R., & Zhang, Y. (2007). Incorporating Diversity and Density in Active Learning for Relevance Feedback. ECIR.


>>>>>>>>>>> Update after author response

I thank the authors for responding to my comments. At this time I do not have sufficient reason to change my scores. Mainly, I do not see this as a true 'active learning' anomaly detection algorithm and it comes across as mostly a class-imbalanced classification problem. The paper started off with an unproven and unjustified premise that correctly identifying anomalies requires estimating the anomaly fraction. It appears that the premise might have some bearing under a few strong assumptions for a specific anomaly detection algorithm, but that is not generalizable. Some more comments:


1. Clarification on Assumption 2 (and also Assumption 1) comment: The paper assumes that if off-diagonal elements of the score matrix constructed from a subset of data are zero, then you can deduce some properties of the general population from that. But, that is wrong as I pointed out with an example. You can have query schemes where the off diagonal correlations can be zero, but the general population's score distribution cannot be deduced from that. That is why the theory is not well-grounded.


2. "Different application areas require a different active learning setup.... The experimental comparison is fair because all methods are evaluated in the same setting..." -- This is where we have a contradiction. Most (all?) of the competitor/benchmark active learning algorithms used in this paper were designed for continuous active learning. So then why compare a one-shot feedback algorithm with the continuous feedback algorithms?


3. "This video dataset is a benchmark dataset for anomaly detection and has the original outlier ratio of around 30%." -- In that case let's use more / other datasets where the anomaly ratios are very small.


4. "We refer to their strategy based on the description in the paragraph 'Diversity-based Query Selection Strategy' on page 12" -- The method described in the current work is not the same as Das et al. (2019) and should be made clear.


5. "..reviewer i6D3 found our theoretical study novel and reviewer 7TvR said.." -- My review not being in agreement with other reviewers is (among other things) a sign of having an independent opinion -- which I last checked is highly valued among reviewers at ICLR.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written, but lacks novelty. Classic technique from classification based active learning has been adopted.
summary_of_the_review: The theory and assumptions are not well grounded. The claim that this is in an active learning setting does not match the experiment setup where instead of multiple rounds of feedback, all query instances are selected at one shot.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an active learning approach for deep anomaly detection where a k-means++ based diversified querying strategy is adopted to ensure that the queries cover both the normal data and the anomalies well and two losses for queried and unqueried samples are designed to prevent neither the queried nor the unqueried data points from dominating the learning task. This paper also provides a method to estimate the unknown contamination rate from queried samples thus avoiding the assignment of the most important hyperparameters. This paper also provides a detailed theoretical derivation and experimental verification of the assumptions and theories used in the method. The experimental results show that the proposed method outperforms the compared methods significantly. The ablation study also shows the effectiveness of the proposed method.

strength_and_weaknesses: Strength:
1.	The proposed method can fully exploits the supervised and unsupervised information in anomaly detection.
2.	The important hyperparameters in the proposed method are designed adaptively. 
3.	The theorems and assumptions used in the paper are theoretically derived and experimentally verified in detail and the experimental results of the proposed method are promising.

Weakness:
1)	The time complexity of the proposed method is not clearly. In the experiment, why the proposed method is not verified on large-scale dataset.
2)	In the proposed method, a querying strategy that encourages diversity with k-means++ is proposed method. However, the main contributions of this strategy are not clearly presented.
3)	In the proposed method, the equation(3) mainly deals with binary classification. How does the proposed method deal with the multiple classification setting.


clarity,_quality,_novelty_and_reproducibility: This paper is well written and it is easy to follow. Meanwhile, a novel active learning strategy is proposed and termed as active latent outlier exposure (ALOE). The parameters in ALOS are adaptively designed.
summary_of_the_review: This paper proposed an active learning approach for deep anomaly detection which is composed of unsupervised anomaly detection with latent outlier exposure (LOE) and supervised active anomaly detection with k-means++ querying strategy. It fully exploits the supervised and unsupervised information into anomaly detection, which ensures the diversity of the selected samples and the generalization of the scoring function to unlabeled and labeled sets. The theoretical derivations and experiments in the article are very detailed. Overall the article is novel enough and well-written.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper considers the problem of anamoly detection in an active learning setting where a batch of diverse queries are selected. The key idea is to select diverse cluster centers using k-means++ algorithm.

Clustering based active learning is a well-known idea in the classification setup. Some prototypical example references are:
https://www.cs.columbia.edu/~djhsu/papers/hier.pdf 
https://icml.cc/Conferences/2004/proceedings/papers/94.pdf
and followup references.
A similar idea has also been explored by Das et al. as pointed out by one reviwer in the context of active anomaly detection using tree-based ensembles (e.g., Isolation Forest).
So the main idea lacks some novelty.

The proposed method relies on some strong and may not be practical assumptions. Similarly, experimental evaluation is lacking in many ways.
- Contamination ratio should be known/estimated. The authors' argue that their method is robust in estimating this quantity, but all experiments are run with a (artificial) fixed 10% contamination factor. This claim requires more evidence with more comprehensive experiments.
- Empirical evaluation is done using a batch of queries selected in a single round. The paper needs to argue the motivation and real-world applications behind this problem setting. Otherwise, there is a lot of recent work on human-in-the-loop learning for anomaly detection which queries in several rounds. Even if the setting considered is practical, it would have been interesting to see how the proposed method performs in several rounds of batch queries.
- Some of the assumptions need more clarification in the main paper and possibly a good intuitive illustration. Since the method relies on these assumptions, it is important to convince the reader about them.
- More experiments on realistic anomaly detection tasks where the fraction of anomalies are less than two percent will help understand the overall effectiveness of the proposed method.

The paper is promising, but it falls short of acceptance for the above reasons. Hence, I recommend rejecting the paper. I strongly encourage the authors' to improve the paper based on the review feedback for resubmission.
Decision: Reject