Towards Lightweight, Model-Agnostic and Diversity-Aware Active Anomaly Detection
Authors: Xu Zhang
Yuan Zhao
Ziang Cui
Liqun Li
Shilin He
Qingwei Lin
Yingnong Dang
Saravan Rajmohan
Dongmei Zhang
Venue: ICLR 2023 poster (2023)
Keywords: Active Anomaly Discovery
Diversity Sampling
Deep Learning

Abstract: Active Anomaly Discovery (AAD) is flourishing in the anomaly detection research area, which aims to incorporate analysts’ feedback into unsupervised anomaly detectors. However, existing AAD approaches usually prioritize the samples with the highest anomaly scores for user labeling, which hinders the exploration of anomalies that were initially ranked lower. Besides, most existing AAD approaches are specially tailored for a certain unsupervised detector, making it difficult to extend to other detection models. To tackle these problems, we propose a lightweight, model-agnostic and diversity-aware AAD method, named LMADA. In LMADA, we design a diversity-aware sample selector powered by Determinantal Point Process (DPP). It considers the diversity of samples in addition to their anomaly scores for feedback querying. Furthermore, we propose a model-agnostic tuner. It approximates diverse unsupervised detectors with a unified proxy model, based on which the feedback information is incorporated by a lightweight non-linear representation adjuster. Through extensive experiments on 8 public datasets, LMADA achieved 74% F1-Score improvement on average, outperforming other comparative AAD approaches. Besides, LMADA can also achieve significant performance boosting under any unsupervised detectors.

Published as a conference paper at ICLR 2023
TOWARDS LIGHTWEIGHT, MODEL-AGNOSTIC AND DIVERSITY-AWARE ACTIVE ANOMALY DETECTION
Xu Zhang', Yuan Zhao?, Ziang Cui Liqun Shilin He', Qingwei Lin'; Yingnong Dang', Saravan Rajmohan', Dongmei Zhang'
Microsoff Research, Peking University, SSouhheat University, *Microsoft Azure, 'Microsoft 365
ABSTRACT
Active Anomaly Discovery (AAD) is flourishing in the anomaly detection re- search area, which aims to incorporate analysts' feedback into unsupervised anomaly detectors. However, existing AAD approaches usually prioritize the samples with the highest anomaly scores for user labeling, which hinders the ex- ploration of anomalies that were initially ranked lower. Besides, most existing AAD approaches are specially tailored for a certain unsupervised detector, mak- ing it difficult to extend to other detection models. To tackle these problems, we propose a lightweight, model-agnostic and diversity-aware AAD method, named LMADA. In LMADA, we design a diversity-aware sample selector powered by Determinantal Point Process (DPP). It considers the diversity of samples in addi- tion to their anomaly scores for feedback querying. Furthermore, we propose a model-agnostic tuner. It approximates diverse unsupervised detectors with a uni- fied proxy model, based on which the feedback information is incorporated by a lightweight non-linear representation adjuster. Through extensive experiments on 8 public datasets, LMADA achieved 74% F1-Score improvement on average, outperforming other comparative AAD approaches. Besides, LMADA can also achieve significant performance boosting under any unsupervised detectors.
INTRODUCTION
Anomaly detection aims detect the data samples that exhibit significantly different behaviors compared with the majority. It has been applied in various domains, such as fraud detection (John & Naaz, 2019), cyber intrusion detection (Sadaf & Sultana, 2020), medical diagnosis (Fernando et al., 2021), and incident detection (Wang et al., 2020). Numerous unsupervised anomaly detectors have been proposed (Zhao al., 2019; Boukerche et al., 2020; Wang et al., 2019). However, practitioners are usually unsatisfied with their detection accuracy (Das et al., 2016), because there is usually a discrepancy between the detected outliers and the actual anomalies of interest to users (Das et al., 2017; Zha et al., 2020; Siddiqui et al., 2018). To mitigate this problem, Active Anomaly Discovery (AAD) (Das et al., 2016), proposed to incorporate analyst's feedback into unsupervised detectors that the detection output better matches the actual anomalies.
The general workflow of Active Anomaly Discovery is shown in Fig. In the beginning, a base unsupervised anomaly detector is initially trained. After that, small number of samples are selected to present to analysts for querying feedback. The labeled samples are then utilized to update the detector for feedback information incorporation. Based on the updated detection model, a new set of samples are recommended for the next feedback iteration. Finally, the tuned detection model is ready to be applied after multiple feedback iterations, until the labeling budget is exhausted.
Despite the progress of existing AAD methods (Das et al., 2017: Zha et al., 2020; Siddiqui et al., 2018; Keller et al., 2012: Zhang al., 2019; Li et al., 2019; Das al., 2016), some intrinsic lim- itations of these approaches still pose great barriers to their real-world applications. Firstly, most AAD methods adopt the top-selection strategy for the feedback querying (Das al., 2017: Zha et al., 2020; Siddiqui et al., 2018; Li et al., 2019). i.c., the samples with the highest anomaly scores are always prioritized for user labeling. However, hinders exploring the actual anoma- lies that are not initially scored highly by the base detector. As such, these AAD approaches are
Qingwei Lin is the corresponding author.

---

Published as a conference paper at ICLR 2023
highly susceptible over-fitting the top-ranked samples, resulting in suboptimal recall with respect to all anomalies. We shall demonstrate this with a real example in Sec. 2.1. Secondly, most existing AAD approaches (Das et al., 2017: 2016; Siddiqui et al., 2018) are tightly tailored for a certain kind of detection model, making it difficult to extend to other unsupervised detectors.
They need to modify the internal structure of a particular type of unsupervised detector, en- dowing them with the ability of feedback inte- gration. Therefore, it is impractical and ad-hoc re-design them each time facing such vari- ety of unsupervised detection models. Recent AAD methods (Zha et al., 2020; Li et al., 2019) attempted to generalize to arbitrary detectors. However, they can barely scale because their mode size grows with the number of samples.
Base Detector
Sample Selection
Feedback Incorpanation
Dona
Feedback
Figure 1: The general workflow of AAD.
To tackle these problems in AAD, propose Lightweight, Model-Agnostic and Diversity-Aware active anomaly detection approach, named LMADA. It consists of two components, i.e, sample se- lector (for sample selection) and model tuner (for feedback incorporation). In the sample selector, we take the anomaly scores as well as the diversity of samples into account, instead of solely picking up the most anomalous ones for feedback querying. Specifically, fuse anomaly scores and the feedback repulsion scores into a diversity-aware sampling technology powered by Determinantal Point Processes (DPP) (Chen et al., 2018; Kulesza al., 2012). In the model tuner, we first leverage a neural network as the proxy model to approximate an arbitrary unsupervised detector. After that, fix the weights of the proxy model and learn representation adjuster on top of it. The repre- sentation adjuster is responsible for transforming the input feature vector to fit the beedback-labele samples. Finally, each sample to be detected is transformed by the representation adjuster and then fed back to the base detector to estimate its anomaly score. In this way, the model tuner shields the details of different unsupervised detectors and achieves lightweight feedback incorporation, only via non-linear representation transformation.
We conducted extensive experiments on 8 public AD datasets to evaluate the effectiveness of our proposed method. The experimental results show that LMADA can achieve 74% F1-Score im- provement on average, outperforming other comparative AAD approaches under the same feedback sample budget. In addition, also validated that LMADA works well under various unsupervised anomaly detectors.
2
RELATED WORK AND MOTIVATION
In this section, we will give brief introduction to the existing AAD work and analyze their limita- tions from two aspects: (1) sample selection and (2) feedback incorporation.
2.1 SAMPLE SELECTION
Most AAD approaches (Siddiqui et al., 2018; Das et al., 2017; Zha et al., 2020; Li et al., 2019; Das al., 2016) adopt the top-selection strategy. The anomalous samples, that are not ranked on the top initially by the base detector, would have little chance to be selected for feedback, and therefore can hardly be recalled subsequently. We show a real example using KDD-99 SA', which a famous intrusion detection dataset. The dataset contains (3.3%) of various intrusion types. We applied the Isolation Forest (Liu et al., 2012) detector (a widely accepted one) to this dataset and found that the recall was around 0.28. We show the anomaly score distribution for the normal samples and three major intrusion types, respectively, in Fig. 2. Only the samples of two intrusion types, i.e., "neptune" and "satan", assigned high anomaly scores (0.60 ~ 0.70). However, the samples of another major intrusion type "smurf" (accounts for 71.27% of all anomalous samples) are assigned relatively low anomaly scores (0.50 0.55), which even below the anomaly scores of many normal samples (4168 normal samples vs. 15 "smurf" anomalies were assigned anomaly scores over 0.55). Under this circumstance, selecting the top samples only for feedback can hardly improve the recall for the "smurf" type. In LMADA, we consider both anomaly scores as well as the diversity of samples during the sample selection. In this way, samples
normal class (96. (9) and 11 anomalous classes
2

---

Published as a conference paper at ICLR 2023
normal
neptune
1600
200
@Samples
15000
Ssampu
1200
Samples
150
@Samples
800
100
5000
400
0.2
0.4
anomaly score
anomaly score
IF anomaly score
anoomaay score
Figure 2: The anomaly score distribution of different classes in the KDD99-SA dataset.
not initially ranked the top, like the "smurf" anomalies in example, have an opportunity to present to analysts.
2.2 FEEDBACK INCORPORATION
How to incorporate feedback information is another focus of AAD. Das ala et al., 2017) added set of adjustable weights to the random projections generated by LODA detector (Pevný, 2016), by which the feedback can be incorporated. They also modified Isolation Forest (Liu et al., 2012) by adding weights to the tree paths, re- weightttnn the isolation score based on the feedback (Das et al., 2016). Siddiqui et al.(Siddiqui et al., 2018) extended the re-weighting strategy to the Gen- eralized Linear Anomaly Detectors (GLAD) with the help of online convex optimization (Hazan et al., 2016). iRRCF-Active (Wang et al., 2020) also borrowed the above similar idea into iRRCF detector (Guha et al., 2016). In summary, the above methods require tailoring the weights specific to the certain model structure of different unsupervised detectors and then adjusting the weights with feedback-labeled samples by gradient descent. However, it is impractical for such diverse range of unsupervised detectors as the modification is sophisticated and case-by-case, In LMADA, propose model-agnostic method incorporate feedback information, regardless of the type of unsupervised detectors.
We also note that some AAD approaches have been proposed and attempted to support arbitrary base detectors. Meta-AAD (Zha et al., 2020) first extracts a set of transferable features based on k-neighbors to labeled instances and feeds them into a pre-trained meta-policy model for detection. GAOD (Li et al., 2019) leverages label spreading (Zhou et al., 2003), graph- based semi-supervised model, to iteratively spread label information to neighbors. In summary, both AAD methods lever- age neighborhoods of labeled instances to exploit feedback information but require persisting the entire dataset for neighboring sample retrieval. Therefore, the final tuned detection model would become increasingly heavier and heavier. In this paper, the feedback incorporation of LMADA is achieved by only a non-lines transformation, which is lightweight enough for real-world applica- tion.
APPROACH
In this section, will elaborate the details about LMADA. Following the general AAD work- flow shown in Fig.1, LMADA consists of two components, i.e., sample selector and model tuner. In the sample selector, consider the diversity in addition the anomaly scores when recommending valuable samples for labeling. In the model tuner, we proposed a model-agnostic strategy to incor- porate feedback information for arbitrary unsupervised detectors. is achieved in lightweight manner, only relying on a simple non-linear transformation.
3.1 SAMPLE SELECTOR
As discussed in Sec. 2.1, sample selection of AAD should consider the diversity the selected samples in addition to the anomaly scores. The diversity here is not in terms of anomaly scores but in the distribution of the samples. In summary, our attempt to select subset of samples with high anomaly scores, and meanwhile, are dissimilar from each other. We use the example shown in Fig. 3 to illustrate this idea. There are two types of anomalies A and B that stray from the majority of samples. The anomaly scores (based the Isolation Forest) are indicated by the colors. The
3

---

Published as a conference paper at ICLR 2023
deeper the color, the higher the anomaly score. The selected samples indicated by the blue cross markers. The number of selected samples is fixed as 20. Type-B anomalies are assigned relatively lower anomaly scores compared with type-A because they are more adjacent to the normal samples. If we use the top-selection strategy, the selected samples would mostly come from type-A (as shown in the left subfigure of Fig.3), which may cover the other types of anomalies. There- fore, the feedback would not help the AAD to recall more anomalies, e.g., type-B in this ex- ample. The desired sample selection is shown in the right subfigure of Fig.3, where the se- lector achieves good coverage for all sam- ples with relatively high anomaly scores. this way, we can enhance the anomaly scores of all anomaly types, instead of only those originally ranked high by the base detector.
Top-sel selection
Desired Results
Anomaly Type
Anomaly Type
Anumall Type
Anumaly BI
Figure 3: An illustration example of the top- selection and the desired sampling results.
Inspired by (Chen et al., 2018), we leverage a andely-adopted diversity sampling method, i.e., De- trrminantal Point Processes (DPP) (Kulesza al., 2012), achieve the above sampling target. We first introduce DPP in Sec. 3.1.1, and then describe how we balance the dual objectives, i.e., anomaly score and diversity, in Sec. 3.1.2.
3.1.1 DETERMINANTAL POINT PROCESSES (DPP)
The Determinantal Point Process (DPP) was originally introduced from fermion systems in ther- mal equilibrium (Macchi, 1975; Chen et al., 2018). Recently, it has been successfully applied to various machine learning tasks, e.g., image search (Kulesza & Taskar, 201 document summa- rization (Kulesza & Taskar, 201 1b) and recommendation systems (Gillenwater et al., 2014). Given dataset D {81, 82 ..., 8. DPP aims to select subset from D. Specifically, DPP constructs a real positive semidefinite (PSD) kernel matrix L E R derived from D. For each subset C C 'D, the probability of selecting C from D, denoted as P(C), is proportional to det LL)). where det Lc) is the determinantal value of the principal minor Lc. The objective of DPP is to derive C which maximizes the value of de)( shown in Eq. As an example, to achieve maximum diversity, the kernel matrix could be constructed as the pairwise similarity matrix (Kulesza et al., 2012).
C argmaxccpdet(Le.
How to approximately solve this NP-hard problem (Ko et al., 1995) has been well studied in (Gillen- water al., 2012; Han al., 2017: Li et al., 2016; Chen et al., 2018) and we adopt the greedy algo- rithm proposed in (Chen et al., 2018) in our paper. We will introduce how to construct specially tailored kernel matrix for AAD in the next section.
3.1.2 KERNEL MATRIX CONSTRUCTION
In LMADA, we construct a kernel matrix L, whose entries can be formally written Eq.2.
Li ( = a,ayrery (s,,
(2)
where a; denotes the anomaly score uniformly re-scaled in the range of [0, 1]. It used motivate DPP to select samples with high anomaly scores. Meanwhile, we need to select diverse samples within and across feedback iterations. In each feedback iteration, the inner product (s) mea- sures the pairwise similarity of all candidate samples, based on which DPP prefers dissimilar sam- ples (Kulesza et al., 2012). As there are multiple feedback iterations, we expect the samples selected in the current iteration are also different from those sampled in previous iterations. To achieve so, we maintain a data pool P preserving the selected samples from the previous feedback iterations. The minimum distance between candidate sample 8; and the selected samples cached in P, is defined as the feedback repulsion score r;, as shown in Eq.3.
r; min (11 (1- (s S&) VSk E P))
From Eq.2, we can conclude that det Le) is proportional to o,a,r,r and is inversely proportional to (8;, 85) among the selected samples in C. In this way, it induces DPP select more anomalous
4

---

Published as a conference paper at ICLR 2023
Proxy Model Approximation
Rupresentation Adjustar Tuning
Representation Aduuttr Applying
Denntey
((S
Lyresy
Desstar
Foome
From Modkl
Denection
User Feedback
Figure 4: The overview of the model tuner. The blue boxes and red boxes denote the trainable/frozee components, respectively.
(i.e., higher a,a.) data points that are not adjacent to the previously selected examples (i.e., higher r.r;). Mcanwhile, the data points are also distinguish enough from each other (i.e., lower (8,, 8y)). The qualitative analysis can be referred to Appendix Sec.A.l.
Theoretically, the complexity of constructing L is O (n²), which is expensive for large dataset. However, anomalous samples generally account for small percentage of the whole dataset com- pared with the normal class (Zhao et al., 2019; Boukerche et al., 2020). For the instance in KDD99- SA dataset introduced Sec.2.1, only 3.3% of samples belong to anomalies. It is unnecessary to regard all samples as candidates for the sample selector. Consequently, we construct the kernel ma- trix with only the pre-truncated top a samples ranked by their anomaly scores. In general, a is small enough (c.g.. < 3%), the selected samples would be those with the highest anomaly scores, ie., similar to the top-selection. On the other hand, if a is large (e.g.. > 30%). the selected samples would become too diverse to retrieve samples worthwhile for feedback. We will evaluate different a settings in Appendix Sec.A.8.
3.2 MODEL TUNER
After labeling the examples recommended by the sample selector, the model tuner focuses on how incorporate newly labeled data points. The model tuner should be agnostic the base unsuper- vised detectors. In other words, any unsupervised detection model can be easily integrated into our framework. To achieve this goal, we propose a three-phases model tuner in LMADA, as shown in Fig. 4. Firstly, we set up a neural network as the proxy model (Coleman et al., 2019) to mimic the behaviors of diverse base detectors. After that, a representation adjuster added in front of the frozen proxy model to get trained based on the labeled samples. Finally, the tuned representation adjuster used to transform the original samples into new representation vectors, which will be fed back to the base detector for re-scoring. The feedback continues for multiple iterations until the sampling budget is exhausted. The tuned representation adjuster can be applied as illustrated the Phase-3 of Fig.4. Given testing sample 8t, first transform it into h; via the representation adjuster 12 (s;). Then we directly feed h; into the base anomaly detector and get the final detection results (h:). In this way, LMADA achieves feedback incorporation in lightweight manner, only with a non-linear representation transformation.
representation vector
3.2.1 PROXY MODEL APPROXIMATION
As introduced in Sec. 2.2, unsupervised detectors of various types pose a great challenge to model- agnostic AAD. There are significant differences between the model structures of different unsuper- vised detectors. Most existing AAD work (Siddiqui et al., 2018; Das et al., 2017; 2016; Wang et al., 2020) needs to specifically modify the internal structure of unsupervised detectors.
To tackle this problem, we utilize a deep neural network as the proxy model to approximate the behaviors of diverse unsupervised detectors. In this way, we can turn unsupervised detectors into gradient-optimizable neural networks, which facilitate the subsequent representation adjuster tun- ing (more details presented in Sec.3.2.2). As shown in Phase-1 of Fig. 4, we use the normal- ized anomaly scores f(x.) generated by the base detector as the pseudo-labels and set up ral network & in parallel to fit them. The proxy model is composed of one input layer and mul- tiple hidden layers followed by output layer activated by the sigmoid function. The Mean-
5

---

Published as a conference paper at ICLR 2023
Squared- Error (MSE) is adopted the loss function during proxy model training, shown in Eproay Ei= ( (84) (s+))² where b denotes the batch size.
After the proxy model training, the anomalous patterns that are captured by the base detectors have been learned by the proxy model, i.e., the proxy anomaly scores (s:) R I (#:). The key point
here is that the internal structures of different unsupervised detectors do not need to be considered in this training process.
3.2.2 REPRESENTATION ADIUSTER TUNING
In Phase-2, devise representation adjuster Ω in front of the proxy model incorporate the feedback information. The representation adjuster is a simple non-linear transformation layer, which takes the original sample vector 9; as the input and transforms into a new feature space but with the same dimensions, i.c., h; = S2 (8:) sigmoid (W whhee h; E E and 8, E g"
As shown in the middle of Fig.4, the transformed h2 will be fed into the trained proxy model +b and generate the proxy anomaly score 4 (h4). Based on that, W will be updated under the loss function in Eq. The representation adjuster can be trained by a gradient descent optimizer because the subsequent proxy model (as shown in Fig. 4) is also a neural network. The parameters of the proxy model are frozen during the representation adjuster tuning phase.
Ladjuster Ljeedback cconsoliaation
Cudjuster is composed of three components, i.e., feedback loss, consolidation loss and a regulariza- tion item 17. Ljeedback is used to fit the labeled samples in the data pool P. as shown in Eq.5, where represents the feedback label (+1 for the anomalous class and -1 for the normal class) for the sample 8i.
feeddback = =-
Σ ys log ( (h+)) Vs, E P
(S)
Training with only a few labeled samples would make the representation adjuster biased toward the feedback labels but ignore the patterns already learned from the base detector. So we design another component inspired by (Li & Hoiem, 2017), i.e., cconsolidation: that serves for consolidating the knowledge of the base unsupervised detector, as shown Eq.6. h4 denotes the transformed sample representation in the last feedback iteration (h; 8; the first feedback iteration). It forces the proxy anomaly scores O (h+) of the remaining unlabeled samples to be stabilized around the original anomlay scores f (mi:) in the newly transformed feature space. We note that coonsoidation not conducive to fitting Cjeedbeck achieve a trade-off between them, we assign weight for the consolidation loss of each sample. Intuitively, if an unlabeled sample 8; is similar to the labeled samples in the feedback data pool P, its consolidation loss should have lower weight, reducing the constraints for fitting Leeddbock- On the contrary, those unlabeled samples, which are unlike the data points P, should be assigned a higher weight to enhance the influence of the consolidation loss. This intuition is fully aligned with the feedback repulsion score r; introduced in S.2.3.1.2 and we thus use it as the weight of consolidation loss.
the former tends to remain the original representation. To
b Σ rs (0+) - ((i)) Vss $ P
C Cconoolidation
(6)
The last component is the penalty for feature space transformation because the extremely dramatic change to the original sample vectors undesired. To achieve so, we set y as Σ || More training details for the representation adjuster can be found in Appendix Aec.A.2.
8: 1i
4 EXPERIMENT
4.1 DATASETS AND SETTINGS
We evaluated our proposed method on 8 public datasets, including PageBlocks, Annthyroid, Cardio, Cover, KDD99-Hup, Mammography, KDD99-SA, Shuttle, which widely used by existing AAD

---

Published as a conference paper at ICLR 2023
approaches (Siddiqui al., 2018; Zha al., 2020; Li al., 2019; Das al., 2017; 2019). The details of these datasets can be found in Appendix Sec. A.3. We run 5 feedback iterations and query 20 samples in each iteration. Same the existing work, used simulation instead of real user feedback since the ground truth is known for these public datasets. The experimental environment and the parameters setting can be found in Appendix Sec. A.4 and Sec. A.5, respectively.
4.2 COMPARISON METHODS AND METRICS
We compared LMADA with three state-of-the-an AAD methods, i.e., FIF (Siddiqui et al., 2018), Meta-AAD (Zha et al., 2020), and GAOD (Li et al.. 2019). FIF adds a set of weights to the tree branches of the Isolation Forest detector and tunes them via online convex optimization with feed- back information. GAOD utilizes the remi-supervise method (label spreading Zhou et al. (2003)) to consume user feedback. Both of the above approaches adopt the top-selection strategy. Meta-AAD extracts set of transferable features for pre-trained meta-policy detection model, considering both long-term and short-term benefits in querying feedback.
We F1-Score Curve to evaluate the effectiveness of different AAD methods. Specifically, calculate F1-Score on the entire dataset after finishing an iteration of feedback. Besides, we also calculate the Area-Under-Curve (AUC) (Ling et al., 2003) of the F1-Score Curve.
4.3 COMPARISON EXPERIMENT RESULTS
We compared our proposed method with three state-of-the-ee AAD approaches and the results are illustrated in Fig. 5. For fairness, we used Isolation Forest the base detector because it was adopted by all the comparison methods (Zha et al., 2020; Siddiqui et al., 2018; Li et al., 2019). To ensure reproducibility,
repeated our experiments 10 times each dataset and plotted the average FI- Score and the standard error bar (Aluman Bland, 2005). The AUC value of each FI-Score Curve shown in the legend.
PageBlocks
Annthyroid
Cardio
1.0
0.6
AJ0 sa
AcciLi AcCASS
1.0
score
0.6
score
0.8
0.6
0.6
0.44
0.2
0.4
0.2
0.2
0.2
0.0
0.0
2
0.0
0.0
Iterations
Iterations
iterations
Iterations
KDD99 Http
Mammography
KDD99-SA
Shuttle
0.8
437 Ea4
art n54
score
0.8
1.0
score
0.6
Eai
0.8
0.8
1.0
0.6
0.4
0.6
0.4
0.2
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
iterations
iterations
iterations
LMADA
GAOD
FIF
Meta-AAD
Figure The experiment results comparing LMADA with the state-of-the-an art AAD methods.
From the results, we can confirm that L.ADA performs better than other AAD methods. With 20 feedback samples per iteration, LMADA achieved consistently higher F1-Score most datasets. Especially on KDD99-SA, Cover, and Cardio datasets, LMADA boosted the FI-Score the base detector by average of 144% to 0.80+ after 5 feedback iterations. For PageBlocks, Annthyroid, and Mammography datasets, LMADA also increased the F1-Score by 60% on average, significantly outperforming other AAD models. As for the KDD99-Http and Shuttle dataset, can that the initial performance of the base detector has reached a relatively high level. Under this circumstance, LMADA also can hold high detection accuracy, exhibiting its robustness.
7

---

Published as a conference paper at ICLR 2023
Among the comparison methods, Meta-AAD performed much better than the other two because utilizes reinforcement learning to lear a meta-policy for feedback querying, rather than simply picking up the samples with the highest anomaly scores. However, the diversity of samples is not taken into account explicitly, resulting in relatively lower performance compared with LMADA 0.87 AUC of LMADA in KDD99-SA dataset). FIF and GAOD
(e.g. 0.29 AUC of Meta-AAD even had difficulty preserving the upward trend of their Fl-Score curves, although more feedback samples were added. As we discussed in Sec.2.1, the top-selection strategy of both methods hinders the exploration of the lower-ranked anomalous samples. Moreover, their detectors were tuned to over-fit the scarce feedback-labes samples, leading a decreasing recall. We have verified this in Appendix Sec. A.9.
4.4 MODEL-AGNOSTIC EVALUATION
We target to propose mgdel-agnostic AAD approach, which can be easily extended to arbitrary unsupervised detectors. such, we evaluated the effectiveness LMADA under five different but commonly-used unsupervised detectors, including AutoEncoder (Vincent et al., 2010), PCA (Shyu et al., 2003). OCSVM (Schölkopf et al., 2001). LODA (Pevný, 2016; Das et al., 2016), and IF. The experimental settings are the same that in Sec.4.3 3 and the results are shown in Fig. 6.
From these figures, we can conclude that LMADA works well on different unsupervised detectors. It can consistently improve the F1-Score on all eight datasets whatever the base detector is adopted. More than that, we also found that the performance gains achieved by LMADA vary with different unsupervised detectors. Taking the KDD99-Http dataset as example, can that LODA performs much worse than the other base detectors at the beginning (F1-Score 0.02 compared to 00.82 of the other detectors). Even so, LMADA from 0.02 to 0.96 after 5 iterations. We also noted that the variance of its results is significantly larger than the others. The reason that LODA is inaccurate and unstable on KDD99-Http dataset, making it difficult to provide effective information for the sample selector and the model tuner. These experiment results confirm that the initial performance of base detectors has a great influence to AAD approaches.
also able improve the performance of LODA
PageBlocks
Annthyroid
Cardio
Cover
0.8
1.0
1.0
0.6
0.8
0.8
score
0.4
0.6
00.4
0.2
0.4
0.2
0.0
0.0
iterations
aerations
iterations
KDD99-Http
Mammography
KDD99-SA
Shuttle
1.0
0.8
1.0
0.8
0.8
0.8
score
0.6
0.6
0.66
0.4
A.C.0.95
AUG
0..4
AUC.8.9
A.C.0.05
AUC053
AUC.9.11
AUC.0.5
0.2
A.C.0.71
AUC.041
0.2
AUC-0.1
0.2
AUC.0.00
AUC.0.9
AUC.043
AUC-081
AUC.0.9
0.0
0
3
0.0
2
0.0
1
2
4
0.0
0
2
3
iterations
iterations
iterations
iterationn:
PCA
OCSVM
t
LODA
AutoEncoder
Figure 6: The results of LMADA under different base unsupervised detectors.
4.5 SAMPLE SELECTOR VALIDATION
In this section, validated the effectiveness of our proposed sample selector in LMADA. As discussed in Sec. 2.1, diversity plays a critical role in AAD. order to verify this point, we con- ducted ablation study the KDD99-SA dataset. In this dataset, 11 anomalous classes and the normal class are well annotated separately so that we can study how samples would selected by different sampling strategies. We compared our proposed sampling method with the coseonly-used
8

---

Published as a conference paper at ICLR 2023
8
1.0
1.0
0.8
Categoriess
6-
F1 score
0.8
score
0.6
0.6
4-
0.4
0..4
2
0.2
LIARA C.S.
0.2
Base Detertor AUC:0.87
top-emsten AUCOUTT
Proxc Model
0.0
0.0
iterations
iterations
iterations
(a)
(b)
(c)
Figure 7: (a) The selected anomaly classes under different sampling strategies. (b) The Fl-Score comparison results under different sampling strategies. (c) The Fl-Score comparison results be- tween the proxy model and the base detector.
top-selection strategy (Das et al., 2017; 2016; Siddiqui et al., 2018). and the stratified sampling de- scribed in (Guha et al., 2016) (i.e., divide samples into g groups based on their anomaly scores and then select examples randomly from each group). The model tuner fixed. The selected anomalous classes under these settings and their corresponding improved F1-Scores are shown in Fig. 7(a) and Fig. 7(b), respectively.
From Fig. 7(a), can see that the sample selector of LMADA is able to cover more anomaly classes, compared with the other two sampling strategies. Furthermore, we also confirm the necessity of the diversity-aware selection from 7(b) since our sample selector achieved much higher FI-Scores than those under the top-selection or the stratified sampling methods. For example, in the first feedback iteration, proposed sample selector chose "smurf" samples (shown in blue color) for feedback, which were missed by the other two. As we stated in Sec.2.1, "smurf" samples were not assigned high anomaly scores by the base detector (IF) but they actually account for 71.27% of all anomalies. Therefore, we can see that FI-Score can be significantly improved from 0.28 to 0.94 with labeled "smurf" anomalies, while the other two strategies failed to achieve this high F1-Score. The complete results on all datasets can be found in Appendix Sec. A.6.
4.6 MODEL TUNER VALIDATION
In this section, we will present the effectiveness of our proposed model tuner. As introduced in Sec.3.2, the transformed representations h; trained based the proxy model but will be fed back to the base unsupervised detector to get the final anomaly scores. We aim to study how large the difference between the anomaly scores generated by the base detector (h,) and the proxy model (h:), respectively. We also conducted this ablation experiment on the KDD99-SA dataset and the results are exhibited Fig. 7(c).
This figure shows that there is only a narrow gap in Fl-Scores between the proxy model (green line) and the base unsupervised detector (red line). It manifests that the proxy model has captured the knowledge learned by the base detection method as they produced similar anomaly scores. As such, the transformed representations h4 trained via the proxy model can be smoothly transferred the base unsupervised detector. The complete experimental results on all datasets can be referred to Appendix Sec. A.7.
CONCLUSION
In this paper, we propose LMADA, lightweight, model-agnostic and diversity-aware active anomaly detection method. In the sample selector of LMADA, we take the anomaly scores as well as the diversity of samples into account, unlike most existing AAD work that solely picks the most anomalous ones for feedback querying. In the model tuner of LMADA, model-agnostic strategy to incorporate feedback information, regardless of the type of unsupervised detector. It can be achieved by a lightweight non-linear transfomnation. Through the extensive eval- uation on 8 public AD datasets, we show that LMADA can achieve 74% F1-Score improvement on average, significantly outperforming other comparative AAD approaches.
propose

---

Published as a conference paper at ICLR 2023
REFERENCES
Douglas G Altman and J Martin Bland. Standard deviations and standard errors. Bmj, 331(7521): 903, 2005.
Azzedine Boukerche, Lining Zheng, and Omar Alfandi. Outlier detection: Methods, models, and classification. ACM Computing Surveys (CSUR), 53(3):1-37, 2020.
John Browne. Grassmann Algebra Volume I: Foundations: Exploring Extended Vector Algebra with Mathematica, volume 1. John M Browne, 2012.
Miguel A Parreira-Perpinán. A review of dimension reduction techniques. Department of Computer Science. University of Sheffield. Tech. Rep. CS-96-09, 9:1-69, 1997.
Laming Chen, Guoxin Zhang, and Eric Zhou. Fast greedy map inference for determinantal point process improve recommendation diversity. Advances in Neural Information Processing Sys- lems, 31, 2018.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang. Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint PrXivv9906. /829, 2019.
Shubhomoy Das, Weng-Keon Wong, Thomas Dietterich, Alan Fern, and Andrew Emmott. Incorpo- rating expert feedback into active anomaly discovery. In 2016 IEEE 16th International Confer- on Data Mining (ICDM), pp. 853 858. IEEE, 2016.
Shubhomoy Das, Weng-Keen Wong, Alan Fern, Thomas G Dietterich, and Md Amran Siddiqui. In- corporating feedback into tree-based anomaly detection. arXiv preprint arXiv: / 770..09444111 2017.
Shubhomoy Das, Md Rakibul Islam, Nitthilan Kannappan Jayakodi, and Janardhan Rao Doppa. Active anomaly detection via ensembles: Insights, algorithms, and interpretability. arXiv preprint arXiv:1901.089300 2019.
Tharindu Fernando. Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes. Deep learning for medical anomaly detection-a survey, ACM Computing Surveys (CSUR), 54(7): 1-37, 2021.
Jennifer Gillenwater, Alex Kulesza, and Ben Taskar. Near-optimal map inference for determinantal point processes. Advances in Neural Information Processing Systems, 25, 2012.
Jennifer A Gillenwater, Alex Kulesza, Emily Fox, and Ben Taskar. iopectation-maxiiizatioo for learning determinantal point processes. Advances in Neural Information Processing Systems, 27,
2014.
Sudipto Guha, Nina Mishra, Gourav Roy, and Okke Schrijvers. Robust random cut forest based anomaly detection on streams. In International conference machine learning, pp. 2712-2721. PMLR, 2016.
Insu Han, Prabhanjan Kambadur, Kyoungsoo Park, and Jinwoo Shin. Faster greedy map inference for determinantal point processes. In International Conference on Machine Learning, pp. 1384- 1393. PMLR, 2017.
Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends@ in Opti- mization, 2(3-4):157-325, 2016.
Hyder John and Sameena Naaz. Credit card fraud detection using local outlier factor and isolation forest. Int. J. Comput. Sci. Eng, 7(4):1660-104,, 2019.
Fabian Keller, Emmanuel Muller, and Klemens Bohm. Hies: High contrast subspaces for density- based outlier ranking. In 2012 IEFE 28th international conference on data engineering, pp- 1037-1048. IEEE, 2012.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 2014.
10

---

Published as a conference paper at ICLR 2023
Chun-Wake Ko, Jon Lee, and Maurice Queyranne, An exact algorithm for maximum entropy sampling. Operations Research, 43(4):684-60 691, 1995.
Alex Kulesza and Ben Taskar. k-dpps: Fixed-size determinantal point processes. In JCML, 201 la.
Alex Kulesza and Ben Taskar. Learning determinantal point processes. 2011b.
Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning, Foundations and Trends@ in Machine Learning, 5(2-3):123-286, 2012.
Chengtao Li, Suvrit Sra, and Stefanie Jegelka. Gaussian quadrature for matrix inverse forms with applications. In International Conference on Machine Learning, pp. 1 1766-1773 PMLR, 2016.
Yongmou Li, Yijie Wang, Xingkong Ma, Cheng Qian, and Xiaoyong Li. graph-based method for active outlier detection with limited expert feedback. IEEE Access, 7:152267-152277 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947 2017.
Charles X Ling, Jin Huang, Harry Zhang, et al. Auc: a statistically consistent and more discriminat- ing measure than accuracy. In Ijcai, volume 3, pp. 519-524, 2003.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation based anomaly detection. ACM Trans- actions Knowledge Discovery from Data (TKDD), 6(1):1-39, 2012.
Odile Macchi. The coincidence approach to stochastic point processes. Advances in Applied Prob- ability, 7(1):33-22,, 1975.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Tomáš Pevný. Loda: Lightweight on-line detector of anomalies. Machine Learning, 102(2):275- 304, 2016.
Kishwar Sadaf and Jabeen Sultana. Intrusion detection based on autoencoder and isolation forest in fog computing. IEEE Access, 8:167059-167068, 2020.
Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert Williamson. Estimating the support of a high-dimensional distribution. Neural computation, 13(7)114431471, 2001.
Mci-Ling Shyu, Shu-Ching Chen, Kanoksri Sarinnapakom, and LiWu Chang. A novel anomaly detection scheme based on principal component classifier. Technical report, Miami Univ Coral Gables FI Dept of Electrical and Computer Engineering. 2003.
Md Amran Siddiqui, Alan Fern, Thomas Dietterich, Ryan Wright, Alec Theriault, and David W Archer. Feedback-guided anomaly discovery via online optimization. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2200- 2209, 2018
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning reseurch, 11(12), 2010.
Hongzhi Wang, Mohamed Jaward Bah, and Mohamed Hammad. Progress in outlier detection tech- niques: A survey. leee Access, 7:107964-108000, 2019.
Yao Wang, Zhaowei Wang, Zejun Xie, Nengwen Zhao, Junjie Chen, Wenchi Zhang, Kaixin Sui, and Dan Pei. Practical and white- box anomaly detection through unsupervised and active learning. In 2020 29th international conference on computer communications and networks (ICCCN), pp- 1-9. IEFF, 2020.
11

---

Published as a conference paper at ICLR 2023
Daochen Zha, Kwei- Herng Lai, Mingyang Wan, and Xia Hu. Meta-aad: Active anomaly detec- tion with deep reinforcement learning. 2020 JEEE International Conference on Data Mining (ICDM), pp- 771-780. IEEE, 2020.
Xu Zhang. Junghyun Kim, Qingwei Lin, Keunhak Lim, Shobhit O Kanaujia, Yong Xu, Kyle Jamieson, Aws Albarghouthi, Si Qin, Michael J Freedman, et al. Cross-dataset time series anomaly detection for cloud systems. In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pp. 1063-1076. 2019.
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection. arXiv preprint arXiv:1901.013: 2019.
Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in neural information processing systems, 16, 2003.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 THE QUALITATIVE ANALYSIS OF EXTENDED DPP IN SAMPLE SELECTOR
The kernel matrix L is shown as 2q.2. As introduced in Sec.3.1.1, we aim select subset C with highest det (Le). The principal minor Le as follows.
atrी (81,
angriy (81,
anarin (81;8 aaagarn 8(2)
020172r1 81)
020grer (82,
a:d1T (8i,
aagrary (81,
aaaaaa (1))
Ieqaren (8e, 81)
gegasmer33 (810))
ca (* (11
The det (Lc) can be calculated in Eq 8. det (Lc) (-1)
Pa...PP....Pill Lip, L2p Licpp
(8)
(PI.P2... P2.... Pic)) represents the where P1, P2, PIc] denote all permutations {1,2.... [C],, and r (pr,
reverse order number of PI P2, -PicI- According 2. det (Le) can be further expanded as Eq.
[c|
det (Lc)
a
2
(1)) (81, 8p,) (82, 8p2) (s]c), Spici)
(9)
[C]
a?r??
det ([$1,
[s 82, s2, sg....)..))
(10)
1=1
[C]
a?r? (s)
82 00......00 )
agr? 12
(11)
1=1
CCI
air? is the common factor extracted from det (Lc). As such, we can conclude that det (Lc) is proportional to a, and ri. inducing DPP to select samples that have high anomaly scores and are different from those have already been selected in the data pool P.
The second term, Σ (-1))(p--P2...--P (#1, 8p1) (82, 8p2) (8|c)- Spel)> can further rewrote
as the exterior product form (8) @ 82 6 ... 2 $1c1)² shown Eq. According to the definition of exterior product (Browne, 2012), it geometrically represents the volume V of the parallel polyhe- dron spanned by vectors {81, #2, :.. Consequently, the more dissimilar they are, the larger the volume V of the spanned polyhedron is, the larger det (Le) is.
A.2 LABELED SAMPLES OVERSAMPLING
In the model tuner, we use the labeled samples to train the representation adjuster. Nevertheless, compared the unlabeled samples, the feedback-labeled samples only account for tiny percent- age of the overall dataset (e.g., 20 samples per iteration vs. 286048 samples in total of the Cover dataset). Therefore, we need to over-sample the labeled samples each training batch to improve the utilization of such few feedback samples, so that we can fully exploit the feedback information and accelerate the loss convergence. Half each training batch are labeled samples, which are re- peatedly drawn from the data pool P, and the other half are unlabeled samples, which are randomly sampled from the all unlabeled samples.
A.3 DATASETS INFORMATION
We used eight public datasets for the evaluation. PageBlocks, Annthyroid, Cardio, Cover, Mammog- raphy, Shuttle are available in ODDS 2 KDD99-Http and KDD99- are available in UCI Machine
13

---

Published as a conference paper at ICLR 2023
Table 1: The Detailed Information of Experimental Datasets
Datasets
Samples
Dimension
Anomaly Number
Anomaly Rate
PageBlocks
5393
10
510
9,46%
KDD99-SA
100655
95
3377
3.36%
Annthyroid
7200
6
534
7.42%
Cardio
1831
21
176
9.61% %
Cover
286048
10
2747
0.96%
KDD99-Huup
58725
3
2209
3.76%
Mammography
11183
260
2.32%
Shuttle
49097
9
3511
7.15%
Learning Repository'. PageBlocks can referred to ADBench The detailed information of these datasets is shown in 1. The number of samples ranges from 1.8K to 286K and the anomaly rate is spanning from 0.96% to 9.61%.
A.4 EXPERIMENT ENVIRONMENT
We built LMADA based on PyTorch 1.12.0 (Paszke et al., 2019) and used base unsupervised anomaly detectors implemented in PyOD 1.0.3 (Zhao al., 2019). In our experiments, we set up Virtual Machine (VM) with 64 Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz processors and 256GB RAM. The operating system is Ubuntu-20.04, In the VM, had NVIDIA Tesla M40 GPU with CUDA 11.4 for deep learning model training.
A.5 EXPERIMENT SETTING DETAILS
For the sample selector of LMADA, we set the pre-truncation rate a 10%. We introduce two hyper-parameters À and 7 to adjust the preference of anomaly score and diversity (Le = (a,a,)* (rery (8;,);)))) In the experiments, we set À 1 and y 1. In the model tuner, we utilized the Adam optimizer (Kingma & Ba, 2014) and set the epoch number to 10, the learning rate 0.01, and the batch size to 512, for both the proxy model approximation phase and the representation adjuster tuning phase. The size of the proxy model hidden layer is set to 64. Specifically for SA dataset, we performed dimension reduction Carreira-Perpinaae 1997) on it because it is characterized by its high feature dimensions and sparsity.
LMADA:
Meta-AAD: We used the source code available in the link provided by the original paper'. We uti- lized 12 datasets (including toy, yeast, glass, ionosphere, lympho, pima, thyroid, vertebral, vowels, wbc, wine, yeast) for meta-policy training in experiment. All the datasets available in the released code repository After that, we directly applied the trained meta-policy to the targeted 8 public datasets. We borrowed the the default settings from the original paper in our experiments: rollout steps T = 128, entropy coefficient = 0.01, learning rate Ir 2.5 x 10 10-4, value function coefficient G1 0.5,
0.95, clip range 0.2, balance parameter ? 0.6.
We used the source code released in the link provided by the original paper 7. We chose the Log-Likelihood loss function for FIF in the experiment. We the type of regularizer w 2 and the learning rate a = 1.
FIF:
et al. (2019) by ourselves because lacking the released source code. We set the number of nearest neighbors k = 30 and the learning rate of label spreading a 0.995. The standard deviation Gaussian function is set to half of 95-percentile of k-th nearest neighbor distances.
GAOD: We implemented GAOD according
14

---

Published as a conference paper at ICLR 2023
PageBlocks
Annthyroid
Cardio
Cover
0.8
1.0
0.8
0.4
0.8
Score
0.6
score
0.6
0.4
0.2
0.4
0.2
WICEST Acceat
0.2
0.0
0.2
0.0
Iterations KDD99-Http
iterations
iterations KDD99-SA
Iterations
Mammography
Shuttle
0.8
1.0
1.0
1.0
0.8
0.6
0.8
0.8
0.6
0.66
0..6
0.4
0.44
0.44
0.2
0.2
0.2
0.2
0.0
0.0
0.0
iterations
iterations
iterations
iterations
LMADA
top-selection
stratified sampling
Figure 8: The complete results of sample selector validation.
We note that the pairwise distance matrix is required for Meta-AAD and GAOD (for neighborhood retrieval). As such, both approaches would fail work under large data volume due the high space complexity (O(n))). Taking the largest dataset Cover as an example (shown in Table. 1), the pairwise distance matrix would consume 610 GB memory in theory, which would trigger the Out- Of-Memory (OOM) problem in our experiment environment. Therefore, we only keep the top 50% and 20% samples for KDD99-SA and Cover, respectively, based on the anomaly scores produced by the base detector. Only these samples are involved in the feedback incorporation of Meta-AAD and GAOD.
A.6 THE COMPLETE RESULTS OF SAMPLE SELECTOR VALIDATION
We illustrated the sample selector validation results on all 8 datasets in Fig.8. Our sampling strategy outperforms other sampling methods on most datasets. Compared with the results of FIF and GAOD shown in Fig.5, we also found that our proposed method still achieved much better F1 F1-Scores even using the top-selection strategy the same manner. It confirms the effectiveness of our proposed model tuner on the other side.
A.7 THE COMPLETE RESULTS OF MODEL TUNER VALIDATION
We show the model tuner validation results on all eight datasets in Fig.9. From these figures, confirm the conclusion in Sec. 4.6. The proxy model has captured the knowledge learned by the base detection method as they produced similar anomaly scores. As such, the transformed representation h; can be directly fed into the base detector.
A.8 EFFECTIVENESS OF PRE- TRUNCATION IN SAMPLE SELECTOR
In Sec. 3.1.2, we introduced the pre-truncation improve the sampling efficiency. this section, we aim to validate its effectiveness in the sample selector. Specifically, we adjusted from 1% to 60%. We recorded the running time and its corresponding AUC of F1-Score Curve under different values, which are shown in Fig. 10. From the left figure of Fig. 10, we can draw conclusion that the running time can be significantly reduced by more pre-truncation. For example, the running time can be saved in half if we adjust from 50% to ~6%. Moreover, from the right figure of Fig. 10, we can see that the AUC of F1-Score arises when & < 10% and then gradually drops when we keep increasing a. As we have discussed in Sec. 3.1.2, it manifests that either too broad or too narrow set of candidate samples leads to suboptimal feedback querying. Generally speaking, we set around the estimated contamination ratio, such 10%.
15

---

Published as a conference paper at ICLR 2023
PageBlocks
Annthyroid
Cardio
Cover
0.8
1.0
1.0
0.4
0.8
F1 Score
0.6
score
score
0.08
0.66
0.4
0.2
0.4
0.2
0.2
0.4
Iterations
iterations
iterations KDD99-SA
Iterations
KDD99-Http
Mammography
Shuttle
1.0
1.0
1.0
0.8
0.6
0.8
0.8
Scor
0.6
0.6
0.66
0.4
0.44
0.44
0.2
0.2
0.2
0.2
0.0
0
0.0
4
0.0
iterations
iterations
iterations
iterations
Base Detector
Proxy Model
Figure 9: The complete results of model tuner validation.
500
400
Runtime (5)
0.9
300
0.8
200
100
0.7
40 50
(1%)
((%)
Figure 10: F1-Score Curve AUC and running time comparison under different
A.9 EXPLOARATION OF -VER-FITTING PROBLEM
PageBlocks (Noutliers: 510)
KDD99-SA [Nouttiers: 33771
Shuttle (#outliers: 867)
0.6
0.99
0.70
0.5
0.98
Scooee
0.97
0.3
0.50
0.45
0.2
0.95
0.40
Rerations
Rerations
Iterations
Figure 11: GAOD evaluation results under the increased number of queried samples.
In Sec.4.3, we found that the comparison methods performed much worse than LMADA. From the feedback incorporation perspective, caused by the overfitting the few top-ranked samples (see 1ec. To verify this point, we take GAOD as an example and gradually increase the number of querying samples in each feedback iteration to mitigate the overfitting problem. We rerun GAOD on three datasets (PageBlocks, Shuttle and KDD99-SA), where it did not perform well. According to the settings described the original GAOD paper, the size the data pool should be set #outliers (Li et al., 2019). Therefore, we enlarged the data pool size spanning from 0.5 to 2 #outliers by a stride of 0.5. From the results shown Fig. 11, we see that GAOD can only achieve improvements in F1-Score with least 0.5x #outliers (e.g., the number of queried samples reaches 168 per iteration in KDD99-SA dataset. which far beyond our proposed approach with 20 per iteration). Therefore, it requires significantly larger labeling effort.
16

---

Published as a conference paper at ICLR 2023
A.10 QUERY NUMBER EXPLORATION
We conducted the comparison experiment under different query numbers per feedback iteration (1, 10, 20) KDD-SA dataset, which be found in Fig. 12. From the figure, we can that LMADA can achieve a consistent performance improvement, even with only sample per iteration. On the contrary, the FI-Scores of FIFIGAOD/Meta-AAA fail to increase because they only select the top-ranked samples for updating the model, ignoring the low-ranked anomaly samples, such as the "smurf" type (as we presented in Sec.2.1).
samples per
samples
samples per
samples per
1.0
1.0
1.0
1.0
8 A6
@ 0.8
0.8
0.8
0.8
0.4
[ 0.4
[ 0.4
2 0.4
0.2
0.2
0.2
0.2
0.0
Iterations
Iterations
iteration
Iterations
LMADA
GAOD
FIF
Meta-AAD
Figure 12: Comparison results under different query number per feedback iterations
A.11 ADDITIONAL EXPERIMENT
We add the experimental results of the top-random query strategy in Fig. 13, which represents dom selection from samples with high anomaly scores. From the results, we can conclude that our sampling method significantly outperforms the top-random PageBlocks, Cardio. Cover, Mam- mography, KDD99-SA datasets and achieve similar performance on Annthyroid, KDD99-Http, and Shuttle datasets. Moreover, it is worth noting that the variance of the top-random strategy is much larger than that of ours.
PageBlocks
Annthyroid
Cover
0.8
1.0
0.8
0.8
0.6
scooee
0.6
score
0.6
0.44
0.2
0.2
0.0
AICO-A
0.0
2
0.0
4
iterations
iterations Mammography
iterations
iterations
KDD99-Http
KDD99-SA
Shuttle
0.8
1.0
1.0
0.8
score
0.6
0.4
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.0
AUCASt
0.0
2
2
5
0
4
0.0
2
3
iterations
Iterations
Iterations stratified sampling
Iteraaions
LMADA
top-selection
top random
Figure 13: The F1-Score comparison results under different sampling strategies (including top random strategy).
17


Reviews:
[{"review_id": "dBNl0xRF-1W", "reviewer": "Reviewer_6kFK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the two critical issues in ADD problem and proposes a lightweight, model-agnostic, and diversity-aware AAD method. The first issue is the negligence of lower-ranked samples in active learning, and the second issue is the generalization of the specified unsupervised detector to other models given different datasets or tasks. The new models manage to address both issues and demonstrate on several benchmarks. ", "summary_of_the_review": "In brief, the paper solves critical problems in AAD through an interesting lightweight, model-agnostic and diversity-aware AAD (LMADA). The nature of model-agnostic allows it to extend to other learning models or tasks with ease. The model has been evaluated extensively on meta-AAD datasets.", "strength_and_weaknesses": "Strength:\n* An interesting diversity-aware sample selector that not only considers the base detector scores but also the diversity of the samples was proposed and developed.\n* A novel model-agnostic tuner is developed to integrate both the base model and user feedback. \n* More specifically, first, a proxy model is developed to mimic the base detector. Second, the proxy network is frozen, and a new representation adjustor is added to learn new feature space, which seems a concise and effective design.\n\nWeakness:\n* There is no significant weakness identified in this paper. While the sample selector might be time-consuming and the time complexity of Determinantal Point Process (DPP) is O(n^2), the run time, in reality, will most likely not be overwhelming due to the limited amount of anomaly samples. Another minor to consider is adding additional experiments, although the current meta-ADD dataset has 24 tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_aLDdVyXs5t", "reviewer": "Reviewer_NYZD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a diversity-aware query strategy for active anomaly detection. The proposed query strategy is based on DPP. A positive semi-definite pairwise similarity matrix between highest ranked anomalous instances is first constructed, and then a subset of instances is computed by maximizing its principal minor. The subset of instances selected in this manner has been shown to maximize diversity in previous literature.", "summary_of_the_review": "The paper leaves out crucial experiments that should compare just the selection strategy across active learning algorithms (i.e., replace the selection strategy in existing benchmark active learning algorithms with the proposed DPP technique) with existing standard strategies. Due to this, the paper falls short of being technically sound.", "strength_and_weaknesses": "1. The paper targets an important problem in anomaly detection.\n\n2. Section 4.1 Datasets and Settings: The dataset information is not complete. For example, it is not clear which categories in each dataset were treated as anomaly and normal. The Table 1 in appendix has different number of instances from cited previous works (Das et al., Siddiqui et al.).\n\n\n3. Section 4.1: \"We run 5 feedback iterations and query 20 samples in each iteration. Same\" -- 20 samples per iteration is an arbitrary number -- specifically for smaller datasets this is pretty large. (Das et al., Siddiqui et al.) used 1 query per iteration. For fair comparison, need to show results with this setting and another set of ablation experiments where the number of queries per iteration is varied.\n\n\n4. The paper claims that one of the contributions is the sampling strategy. However, the experiments are not correctly designed to demonstrate its effectiveness. Typically, active learning algorithms have two aspects which are/should be independently replaceable: (a) the query strategy to select samples for user feedback (such as query most anomalous) and, (b) update the algorithm/model parameters with the new labeled data from user. The correct way would be to use the existing benchmark active learning algorithms and replace just their query strategy with the new strategy; then check whether the performance improves/degrades. For the query strategy to be useful and generic, it should work well with other algorithms instead of being tied to a specific one.\n\n\n5. Section 4.2: \"Specifically, we calculate F1-Score on the entire dataset after finishing an iteration of feedback.\" -- This is inappropriate for active learning. The F1-score should be computed on an independent dataset, or a different metric should be used suitable for active learning.\n\n\n6. Figure 5: The plot of F1-score along the y-axis is improper. Consider for example if there are exactly 20 true anomalies in the dataset and all get detected in the first iteration. Then F1 will be 1.0 in the first round and then decrease monotonously over successive iterations simply because the precision will decrease. This gives a wrong impression about the algorithm behavior -- that the performance degrades with successive iterations, whereas, that is not true. For an active anomaly detection algorithm, it would be more appropriate to measure the % of true anomalies detected with each feedback iteration (or the AUC).\n\n\n7. Section 3.2: \"In this way, LMADA achieves feedback incorporation in a lightweight manner, only with a non-linear representation transformation.\" -- It is misleading to call this approach 'lighweight' just because one (last) stage appears to be simple.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Hqu3ZZxBgG", "reviewer": "Reviewer_NuRo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for using any anomaly detection method in an Active Anomaly Detection (AAD) setting. The proposed method consists of three phases: 1) training of a proxy model to emulate the results of the base anomaly detection model(s), 2) training of a data transformation module using human feedback on the anomalous samples, and 3) applying the transformation layer in front of the base anomaly detectors in the data pipeline. The proposed method shows superior results in AAD across several different benchmark datasets. The paper also uses some ablation and comparison results to support the underlying insight beyond the method, namely that there needs to be a diversity-aware sampling strategy for presenting anomalous points to a human.", "summary_of_the_review": "Given the good insight in the paper (diversity-based sampling) and the solid empirical validation done to support that insight, the paper probably merits publication. However, I do have some concerns about the proposed method that prevents me from being sure that it should be published.", "strength_and_weaknesses": "The paper has strong empirical results, is attacking a significant problem, and has a novel approach to making its method agnostic to the base AD algorithm(s). The performance increases seen by using the proposed method, especially with different types of base AD models, are very convincing of the soundness of the presented method and its underlying insight. I also especially appreciate the way the paper attacks the problem of making an agnostic method by attacking the data featurization as a way of improving the result; it very much embodies the fundamental tenets of data-centric data science.\n\nThere are a few weaknesses in (or questions about) the correctness of the proposed method and the clarity of the paper.\n\n•\tWhy does the method need a proxy model? If the proxy model is frozen at Phase 2 of training when the representation adjuster is trained, why not just take the user feedback and base AD model results and train the neural network adjuster directly? It's not clear to me why there needs to be a proxy model for the method when it seems like one could take the feedback loss and the consolidation loss directly from the base AD outputs to train the representation adjuster.\n\n•\tWhy is the proxy model a single-layer neural network with no bias term? Roughly speaking, adding depth to a neural network improves its ability to do non-linear transformations, so why not opt for a deeper network for the representation adjuster?\n\n•\tHow did you decide what the proxy model should be? Why is it a neural network and what is the architecture of that network? Were other models considered?\n\n•\tFigure 6 does not really show the comparison of using a base model to using a base model + LMADA, which is the claim in section 4.4. It shows how different base model + LMADA models perform, which is also good, but there needs to be something more to show how adding LMADA for any base model improves that model.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "37P7Xt3Ps9N", "reviewer": "Reviewer_SZBd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a lightweight, model-agnostic and diversity-aware active\nanomaly detection method. It takes the diversity of samples into account and designs a diversity-aware sample selector powered by determinantal point process. It proposes a model-agnostic strategy to incorporate feedback information, which approximates diverse unsupervised detectors with a unified proxy model.\n", "summary_of_the_review": "As listed in the strength and weaknesses, the experiments are not convincing. It would be better if authors can add supplementary experiments on the model-agnostic evaluation and sample selector validation.", "strength_and_weaknesses": "Strength:\n1.\tIt considers the diversity of samples instead of anomaly score top-selection strategy for the feedback querying, preventing the over-fitting situation to the top-ranked samples\n2.\tThe proposed method shields the details of different unsupervised detectors and achieves lightweight feedback incorporation, only via a non-linear representation transformation.\n3.\tIt conducts extensive experiments on eight public datasets and outperforms other comparative AAD approaches\n\nWeaknesses:\n1.\tThe proposed sampling method is somewhat incremental as it simply uses an existing method, DPP, to select samples. The authors may need to re-summarize the main contributions of the paper. The contributions are not clear.\n2.\tThe proposed sampling method making the pre-truncated ratio alpha too important. It takes efforts to select such alpha technically by extensive experiments. And such alpha may differ in various dataset. For example, as shown in Appendix. A.8, when the ratio changed from 10% to 8%, the performance will decrease by 5%, which is too sensitive.\n3.\tIt lacks the comparison of different sampling methods from selecting samples from the top alpha samples, like simple random sampling. I mean, the proposed method uses DPP to select diverse samples from top 10% anomaly data, but simply using random sampling to select samples from the same top 10% anomaly data can also guarantee the data diversity.\n4.\tThe experiments of LAMDA under different base unsupervised detectors only consider classical unsupervised detectors like PCA, OCSVM. When using deep model, I am not sure whether the proxy model could mimic the detector or the cost in time and memory will be too large.\n5.\tThis paper uses a proxy model to imitate different anomaly detectors. Authors claim that it is model agnostic. However, when using different anomaly detectors, the learned parameters in the proxy models are different. Thus, I think it is not model agnostic.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster