Supervised score aggregation for active anomaly detection
Authors: Kevin Bleakley
Martin Royer
Benjamin Auder
Venue: Accepted by TMLR ()

Abstract: Detecting rare anomalies in batches of multidimensional data is challenging.
We propose an original supervised active-learning framework that sends a small number of data points from each batch to an expert for labeling as `anomaly' or `nominal' via two mechanisms: (i) points most likely to be anomalies in the eyes of a supervised classifier trained on previously-labeled data; and (ii) points suggested by an active learner. Instead of training the supervised classifier directly on currently-labeled raw data, we treat the scores calculated by an ensemble of $M$ user-defined unsupervised anomaly detectors as if they were the learner's input features. Our approach generalizes earlier attempts to linearly aggregate unsupervised anomaly detector scores, and broadens the scope of these methods from unordered bags of data to ordered data such as time series. Simulated and real data trials suggest that this method usually outperforms---often significantly---linear strategies.   
The Python library acanag implements our proposed method.

Published in Transactions on Machine Learning Research (1/2026)
Supervised score aggregation for active anomaly detection
Kevin Bleakley
kevin, kevin, www..beeaaklly@iiri.......
Inria and Laboratoire de Mathématiques d' d'Orsoy, Prance
Mouhcine Mendil
meulciiemmndidrrtssiineeeeyyy
IRT Saint Exupéry, Pronce
Martin Royer IRT SystemX, France
martin..oger@irt-spttemm..
Benjamin Auder
Laboraloire de Mathématiques d' d'Orsay, France
Reviewed on OpenReview: https: // opwpreveeeer........mmme net/ ftrmm? id= Rrmjj3IMMA3
Abstract
Detecting rare anomalies in batches of multidimensional data is challenging. We propose an original supervised active-learning framework that sends small number of data points from each batch to an expert for labeling as 'anomaly' or 'nominal' via two mechanisms: (i) points most likely to be anomalies in the eyes supervised classifier trained on previously-labele data; and (ii) points suggested by an active learner. Instead of training the supervised clas- sifier directly on currently-labeled raw data, we treat the scores calculated by ensemble of M user-defined unsupervised anomaly detectors as if they were the learner's input features. Our approach generalizes carlier attempts to lincarly aggregate unsupervised anomaly de- lector scores, and broadens the scope of these methods from unordered bags of data to ordered data such as time series. Simulated and real data trials show that this method outperforms-ofter significantly-linar strategics. The Python library acanag implements our proposed method.
1 Introduction
1.1 General setting
Anomaly detection in large multidimensional data sets is difficult when anomalies are rare, and few (or zero) true anomalies have been previously labeled. In order to deal with situations like this, anomaly detection methods have evolved over time from an original focus on unsupervised strategies more sophisticated techniques today.
Many previous studies have considered anomaly detection in unordered bags of data [Islam et al. 2018 Pevný 2016). A common statistical assumption in such settings is that individual data i.id. (independent and identically distributed) samples from mixture distribution of "normal" (nominal) and "abnormal" (anomaly) data with density = Tfanom + (1 - 7 Here, T is the probability of drawing an anomaly (from fanom) and (1-r) the probability of drawing a nominal (from fnom) (Pimentel et al. 2020). However, this i.i.d. mixture model does not capture more general scenarios. For instance, an anomaly may not solely be a function of its raw data value; its status may instead depend on its value in the contert of other raw data values surrounding it in unordered bag ordered time series. For example, in non-stationary time

---

Published in Transactions on Machine Learning Research (1/2026)
series, anomalies might be related to abrupt changes between successive time points rather than directly to raw data values.
In this paper we propose a general method to detect anomalies in batches of ordered or unordered data using unsupervised anomaly detectors, supervised learning and active learning. Our method builds on previous
developments in unsupervised anomaly detection (Pevný] 2016 Das et al. 2016 Islam et al. 2018 Williams
et al. 2002 Schölkopf et al. 1999; He et al. 2003 Liu et 2012);
aggregation of anomaly detectors
[Benferhat & Tabia 2008 Gao & Tan 2006; Gao et al. 2012 Kriegel al.)
2011 Kruegel & Vigna 2003
Kruegel al. 2003). supervised learning [Almgren & Jonsson
2004
Chapelle et al.) 2006), and active
learning [Balcan ct al.
2007 Dasgupta et al.
2005; Freund ct al.
1997
Roy & McCallum, 2001 Das et al.
2016 Danka & Horvath, 2018 Bodor et al.
2022;
Das et al.)
2017
Islam et al.] [0018] Pimentel et al.) 2020
Siddiqui et al. 2018 Tang et al. 2020
Görnitz et al.
2013). Please refer to the Appendix for a more detailed
introduction to these methods.
1.2 Why the problem matters
Rare anomaly detection in high-dimensional datasets is a very hard problem yet also important modern world incrcasingly overwhelmed by data. Critical applications inchude detecting network intrusion (Yaman-) ishi et al. 2000), analyzing satellite images for unexpected features (Asuncion et al. 2007), and looking for anomalies in mammography images (Rocke & Woodruff 1996).
1.3 Main contribution and intuition
Individual unsupervised anomaly detectors are typically designed in the "hope" that true anomalies get the highest scores and are thus detectable. Some previous methods aggregate such scores from ensembles of unsupervised anomaly detectors using linear combinations (Pevný 2016 Das et al. 2016: Islam el al. 2018), since aggregation can potentially improve overall anomaly detection. Aggregation can be done in an unsupervised way (c.g., mean score) or supervised way (c.g., learn best lincar weights) if some labeled data is available. However, if true anomalies actually have "med-range" scores while nominals have lower and higher scores, linear methods simply cannot map anomalies to the highest values.
Our main contribution from distribution. We then use labeled scores (obtained via the data) to train a supervised classifier to predict new anomalies in new data batches. This is direct generalization of linear methods. Since in real-world settings it can be expensive to check large numbers of predictions, we also use active learning to provide a small number of candidates from each new batch to an expert for labeling. We prove that under certain hypotheses, pooled anomaly scores and pooled nominal scores have probability distributions. Hence, supervised learning on scores and labols is well-defined.
directly treat ensemble anomaly scores for data batches as if they were data
1.4 Paper structure
In Scction we present our method in detail and describe its similaritics and differences previous mothods. In Section we test it on simulated and real data sets and compare it to three other methods. In Section we conclude and provide perspectives for future work.
2 Methods
2.1 Description of the method
A simplified flowchart describing the AAA Active Anomaly Aggregation method is presented in Fig. 1 while Algorithm gives the full pseudo-code.
The key idea is that unsupervised anomaly detectors whose score distributions (see Section 2.4 for true anomalics turns out be "different enough" from those of nominals can potentially provide discriminative information supervised classifier. The AAA method requires three main a priori choices: (1) an ensemble of M unsupervised anomaly detectors, (2) supervised classifier C, and (3) active learning stretggy A.
2

---

Published in Transactions on Machine Learning Research (1/2026)
Fig. presents the basic process: receive a new data batch, calculate its anomaly scores, predict the most likely anomaly candidates in that batch using the current state of the supervised classifier (trained on all scores associated with previously-labeled data), add small number of extra candidates via active learning, give all candidates to an expert for labeling, and then retrain the classifier on all currently-labelod data's 's scores.
2.2 Original features of the approach
To the best of our knowledge, our method is the first to combine anomaly scores from ensembles of unsu- pervised anomaly detectors, supervised learning on these anomaly scores, and active learning. In particular, using score vectors as if they were training features seems conceptually new in the general sctting, though hints of this appear in specific settings for other methods (see Section 2.3
An implicit hypothosis underlies most anomaly detection methods in the unsupervised scores sctting: you need true anomalies to gel the highest scores in order to be delected. It is thus important to emphasize that, unlike previous methods (e.g., Das et al.] (2016) or Pevny (2016)), the proposed method does not require anomalics to obtaín the highest (or lowest) scores from at least one anomaly detector in an ensemble. Instead, any consistent difference in score distributions between anomalies and nominals can be leveraged by the classifier, regardless of whether the scores are high, low, or in-between.
2.3 Comparisons with previous work
Comparisons here are for the most part chronological. In Kruegel et al. (2003): Kruegel & Vigna (003
Gao & Tan (2006); Benferhat & Tabia (2008): Kriegel ct al. (2011). various unsupervised schemes were presented aggregate anomaly scores from an ensemble using Bayesian techniques to hopefully push linear combinations of scores for true anomalies higher than those for nominals, but with all of the usual defaults
of purely unsupervised anomaly detection.
The ALADIN algorithm (Stokes et al. 2008) involved active learning on the original data itself rather than on anomaly scores, training a supervised classifier to learn a decision boundary that separated anomalies and nominals based on labeled data. The SSAD algorithm in Görnitz et al. (2013) performed active learning with supervised or semi-supervised learning for anomaly detection on the original data using support vector data descriptions (Tax & Duin 2004) and kernel functions, but without an ensemble of anomaly detectors. In Rabenoro et al. (2014) in original twist-the authors generated a large number of anomaly detectors, not only by having an ensemble of different models, but also by including the same model, with different parameter values, as separate models. They did not use active learning to improve aggregation over time.
A random projection stratogy called LODA was used in Pevny (2016) to create an ensemble of detoctors. In it, each anomaly detector correspouded a randomly generated sparse d-dimensional vector of real numbers, mapping the d-dimensional data to IR by taking the scalar product. An empirical distribution R, represented as an equal-bin-widtt histogram, could then be formed for the random projector by applying it to the full data set. The motivation was that projected data points in low density areas of such histograms should be more likely, average across the set of random projections, to correspond to true anomalies. They therefore defined the score as the negative log of each projected point's empirical bin density in order to hopefully push anomalies' scores to the top. Some geometric intuition on LODA provided in the Appendix- partlypatlllyyl explains why LODA can fail completely in some settings (see Section
Since LODA unsupervised and constrained to use the arithmetic mean score across projectors as the final score, the idea was extended to active-LODA (Das al.) 2016):: greedy active learning with a linear classifier, optimizing ensemble weights in such way that labeled true anomalies are pushed toward higher real numbers than labeled true nominals. In contrast, our AAA method is not constrained to inflexible hyperplanes decision boundaries induced by linear combinations when aggregating scores. Note that due to its pairwise optimization constraints between all currently labeled-data, active-T.ODA struggles to converge once there are hundreds labeled data points.
Some issues with active-LODA were taken into account the GLAD algorithm (Islam al. 2018 using
LODA projections anomaly detectors in parallel with neural network. Instead of the weights being
3

---

Published in Transactions on Machine Learning Research (1/2026)
Choose anoomaly detectors Choose classifier C train on old labeled data Choose active learning strategy A
Receive new batch of B data points Calculate Ad scores for each point
Apply current C the M scores (for each new point) to predict most likely anomalies to label in batch
Apply A to choose > further points to label
Retrain C on all currently labeled data (scores)
Figure 1: Simplified schematic showing how Active Anomaly Aggregation works.
Algorithm AAA: Supervised score aggregation for active anomaly detection
Initialization:
Start with either (i) no old data or (ii) T' (possibly time-ordered) d-dimensional real-valued data points XI...., Xr stored as a T'x d matrix Xats. Assune either (a) none the data labeled, (b) some data labeled, or (c) data is labeled. Choose supervised classifier C. Choose an active learning strategy A. Choose unsupervised
anomaly detectors {UI,..., Und Choose an integer > O. Choose an integer D > 0.
if (i) or (ii & a) then initialize Scurr
empty matrix and Yeurr empty vector.
else
Calculate anomaly scores for Xata each the anomaly detectors at each the T data points, giving a T x M score matrix Sasa Extract rows of Satd corresponding to rows of X.ta which had labels. Call this submatrix Scoer and put the corresponding labels in vector Yeurr. if there are at least labeled data points from each class then train C Scurr and Yeure-
end if
end if
Iteration:
For iteration 12 1, receive batch of B; S D new data points X Concatenate X..w onto the bottom of Xours. Extract the last D rows and replace X by them.
if there are not at least labeled data points from each class then Randomly select qi queries < q. S B.) from the new batch only. Send them to the expert for labeling, thus obtaining 9, labels as a vector Ynew- Update Yourr by concatenating Ynew on to the end of it. Calculate the x M matrix of scores for Xeurr and extract from it the rows corresponding to the q queries; call this submatrix Snew- Update Scurr by concatenating Snew on to the end of it. if there are now least labeled data points from each class then train C on Scurr and Yeur.. end if
else
Calculate the score matrix for Xeuer and retain the last B, rows; call this Snezt. Apply C to to predict labels. Choose r, queries (0 < r, S B.) based on the predicted labels and send them the expert. Apply A to Smezt with respect to and, based on this, choose & queries (0 < < B,) for the expert. Remove any duplicate queries and send the remaining ones to the expert. Update Sourr by concatenating all queried rows of Snert on to it. Update Yourr by concatenating all newly obtained labels to it. Retrain C on Sour and Yourr. end if
4

---

Published in Transactions on Machine Learning Research (1/2026)
determined via a loss function for weighted linear combinations and requiring identical weights to be ap- plied to all input data, they learned "local" weights by-in in parallel to performing projections- passing each original data point through a neural network trained to output "good" weights, on average, for each currently-labeled data point. Since these output weights were now not the same for different data points, this algorithm corresponded to kind of 'local correction' to underlying weighted linear combination learning. It nevertheless required learning thousands of parameters of neural network simply to output linear combi- nation weights, often with tiny amounts of labeled data to begin with not necessarily ideal when anomalies are rare. Furthermore, if anomalies are highly contextual (e.g., function of their relationship to other data points rather than of their individual values), GLAD can fail completely (see Section g In contrast to this unwieldy idea, the AAA method is simple: Apply a user-selected trained supervised learner directly to the anomaly score veclor output by an ensemble of anomaly detectors in order to predict anomalies, with the additional assistance of an active learner.
Since these three originally LODA projection- based strategies described above Das ct al. (2016) Islam et al. (2018); ; Pevny (2016) span the range from unsupervised aggregation up to the use of neural networks, and furthermore can be directly applied to any ensemble of anomaly detectors (i.e., not just LODA projections).
wC chose to compare their performance with AAA on a range of simulated and real data scttings in Section 3
For completeness, we briefly mention a few other related methods. In (Das et al. 2017 2018 Siddiqui et al. 2018), the detector ensemble was made up specifically of different isolation forest trees but still involved learning linear combinations of scores, while in [ang et al.] (2020) the authors used arbitrary anomaly detectors in the ensemble, but still in parallel trained a neural network on the original data to learn weights in order to perform "local" linear combinations of the scores like in Tslam et al. (2018) In Veeramachaneni ett al. 2016) the authors used an ensemble of anomaly detectors to output scores, then used non-adaptive rules to provide queries to an expert; they then trained a classifier on the subset of the original data which was now labeled (i.e., not on the scores). Lastly, in Xin et al. (2023) the authors specifically trained neural network directly on the scores output by an ensemble. However, they pre-processed the data using PCA and did not use active learning, instead supposing a fully-labeled data set- io-pplusliillee in real-world settings.
2.4 The elephant in the room: Score distributions
So far we have implicitly assumed that the "score distributions" we treat as input features are either math- ematically well-defined, or at least useful in practice. We now take a closer look this issue. Here, each of the M anomaly score functions maps bag batch of n d-dimensional data points (X1,..., to n scores in R. A successful supervised classifier acting on these scores would require (1) some kind of stability in the scores and (2) "separation" between typical scores assigned to true anomalies and true nominals, in order to learn well. A potential spanner in the works is that, for example, even in the simple case of one fixed LODA projection as the score function (sce Section 2.3 the obtained score for any data point depends on the values of the other n dola poinls in the bog//bates bbathh, due to LODA's deterministic histogram binning process; see Pevný (2016) and Birgé & Rozenholc (2006). The same issue affects other contextual anomaly detectors like isolation forest and local outlier factor ŞLiu et al.) 2012 He et al. 2003).
More precisely, it is not intuitively clear that by repeatedly drawing new bags batches of size n, pooled scores for true nominals and true anomalies will act as if generated by probability distributions ("score distributions"). The natural question is therefore: Do there exist conditions under which "score distributions" correspond to actual probability distributions? We do not have a general answer to this question, but can prove some preliminary results the LODA projection setting.
Consider the LODA projection setting (Section 2.3 and i.i.d. d-dimensional data arriving in batches X == 1,,,,.... of fixed size 1:
x6 (x(1) x(0) E (R4)"
generated from a mixture distribution
/ = (1 - nom + Taanom:
E|0, 1[
where Puom and aanom are probability laws on I for nominal and anomalous data, respectively, and between- batch data is also i.i.d. Following the LODA method in the one-dimensional setting, we draw random
5

---

Published in Transactions on Machine Learning Research (1/2026)
projection vector L E Rd (independent. of all x(O)) by generating i.i.d. (0, 1) entries and randomly setting all but [vā] coordinates to zero, where [-] denotes the ceiling function; L is then fixed that value. Next, we define measurable transform T, corresponding to calculating LODA scores the jth batch, as follows:
Map each x! to a real number y = L
2. Within the batch, rum a deterministic algorithm D (see (Birgé & Rozenholc 2006)) that chooses an optimal number of equal-length bins bbest {l....,m} based on the values {r,²1, then assigns each y1 value
R log empirical frequency of the bin containing y
Then, let Ký denote the number of true anomalies in the batch; in the current setting, Kj ~ Bin(n, r). If K₃- = O, then the batch contributes nothing to the pooled anomaly scores from the previous j - batches. Otherwise, let
(d) anom, anom, K;
(
be the corresponding anomaly scores, Define the empirical cumulative distribution function (cdf) of pooled anomaly scores over the first m batches:
(m(r): E k=1 1 1 & S 5r} Σラ K2
with the convention that sum over empty index set is O. The following theorem deals with the edf of anomaly scores in this setting; it immediately applics to the edf of nominal scores too.
Theorem (Convergence of Pooled Anomalies' Scores). As m - 00, the empirical edf F. converges almost surely to well-defined cumulative distribulion function Fpoo((r)
E
1( S {r]
Fin + Foool())
1NT where the erpectation is over both the random batch composition and the stochasticity of the data.
The proof of Theorem is deferred to the Appendix, along with a corollary extending this result to multiple LODA projections.
Remark. For cach finite the function F. is the empirical edf of the anomaly score of a uniformly randomly chosen anomaly among all anomalies observed in the first m batches. The theorem says that as m > 00, this empirical edf converges almost surely to Pooot- Thus, Fpont can be interpreted as the limiting edf anomaly scores drawn in this way. We conjecture that results a similar vein can be obtained with batch data for other unsupervised anomaly detectors such as local outlier factor and one class SVM, and likely even for ber-batch isolation forest-desppa its additional internal randomization process.
3 Results
3.1 Results roadmap
We briefly describe the acanag Python library and practical issues in Section 3.2 In Sections 3.3 and 3.4
we probe the strengths and weaknesses of the AAA method in scries of simulations. In Section 3.5 we test the method on batch data from eight well-known benchmark datasets (five real, one synthetic, two semi-synthetic). We also look at the effect of the choice of, and number of, ensemble anomaly detectors. Finally Soction 3.6 we test the method on real time scries data containing real anomalics along with additional synthetic ones from the GECCO challenge Moritz al. (2018). showing the method's potential for anomaly detection in time series.

---

Published in Transactions on Machine Learning Research (1/2026)
3.2 The acanag library
The acanag Python library contains functions to run the AAA algorithm on data batches from ordered or unordered datasets. Users can choose the number and types of anomaly detectors to include, the supervised classifier, and the active learning strategy. All other parameters have default values that can be modified by the user. A full example notebook is included in the library, as well as reproducible code for all trials found in this paper.
AAA is fast algorithm. Its run-time basically depends on how quick the classifier and active learner are. As an example, with a random forest classifier and uncertainty sampling providing 5 examples to label per batch, it takes less than second per batch to run AAA on laptop PC for 100 batches of 10-dimensional data of size 500.
3.3 Mixture distributions with varying separability
We tested AAA against three originally LODA-inspired methods: LODA, active-LODA, and GLAD (Pevný 2016 Das et al. 2016 Islam et al. 2018) which range from aggregated unsupervised anomaly detection with no learning (evvy 2016) to active anomaly detection with a proxy neural network (Islam et al. 2018). We implemented these methods in the batch sctting in order to compare them directly with AAA, and allowed five data points to be sent to the expert from each batch.
In all simulations, unless otherwise stated we set the number of initial data points to 1000, the proportion of them with already known labels to 0.1 (i.e., 100 pre-labeled of which we enforeed 98 to be nominals and two anomalies), the batch size to B 500, the number of batches to 200 (except for active-LODA: 50 100 batches due to optimization issues); results were averaged over five runs.
We first considered the setting in which i.i.d. data points X; were generated from a mixture distribution of nominals and anomalies fx ~ (1 - F)mm + Tfaoom where fnom was the distribution of nominals, fanom the distribution of anomalics, and r = 0.01 the mixture parameter. Thus, for each data point we performed Bernoulli trial B(0.01) and if we obtained we generated one value from Snomi otherwise we generated one value from fanom- Figure 2 (column 1) shows the component densities of four 2-dimensional data sets simulated in this way with nominals generated from (N((,, 0), 12) and anomalies from (V(c,,) 12/10), where f0.5, 1, 1.5,2) and In is the 2-dimensional identity matrix. We set logistic regression to act as AAA's supervised classifier in these trials, with class class_weight = 'balanced', and sent 5 greedy active learning- selected candidates to the expert from each batch. The ensemble models were LODA projections obtained following Pevný (2016) with the constraint of 15 projections at most.
For the three hardest cases (c E {0.5, 1, 1.53), AAA significantly outperformed the other methods both in terms of AUC and cumulative anomalics detected, even though those methods were created specifically for LODA projections. GLAD performed slightly better than AAA in the simplest setting (c = 2). Supple- mentary Figs 2 and 3 in the Appendix provide visual interpretation of how well each of the four methods separates anomaly scores from nominal scores after aggregation.
We lested several extensions to these results.
10-dimensional data. Plots for the same trials but with 10-dimensional data, with nominals generated from NN((0, o,.., 0), 410) and anomalies from N((c.c.... c), /10/ 10) for o (0.5, 1, 1.5), can be found in Supplementary Fig. in the Appendix; conclusions are similar to those above.
Other 2-dimensional distributions. Three further difficult 2-dimensional mixture densities are shown in Supplementary Fig. 5 in the Appendix. In all of them, AAA succeeded in detecting some anomalies, while all three LODA-inspired methods failed.
Active learning tradeoff. Under the original 2-dimensional setting above, we tested the tradeoff of a e (0, 1, 4, 5, active uncertainty sampled points versus 5 a greedily sampled points (also an active learning strategy) sent to the expert in each batch. Increasing did not systematically

---

Published in Transactions on Machine Learning Research (1/2026)
Density plot (c 0.5)
AUC over time
Cumul. anomalies detected
Nominals Anomalies
LODA
Active-LODA
Averaag AUC
GLAD
LODA
AAA
0.5
GLAD
50
0.4
50
100
203
150
220
X-axis
Batch
Batch
Density plot (c 1
AUC time
Cumul. anomalies detected
Nominals l Anomalies
0.80
LDDA
LODA
Active-LODa
detected
Active LODA
Average AUC
0.75
AM
AAA
0.70
-2
Batch
Density plot (c 1.5)
AUC over time
Cumul. anomalies detected
Nominals Anomallc alles
Anamalies delerted
1.0
Active LODA
Average AUC
0.52
AAA
0.50
0.28
LODA
Active-LODA
GLAD
0.80
160
203
50
250
220
X-axis
Batch
Batch
Density plot (c 2)
AUC over time
Cumul, anomalies detected
Nominals Anoma alies
600
0.50
Anomaallss detected
Active-LODA
200
Average AUC
0.50
LODA
GLAD
0.57
Active-LOOA LOAA
GLAD
0.30
200
0.55
200
0.64
50
100
150
703
60
160
150
200
X-axis
Batch
Batch
Figure 2: Two-dimensional i.i.d. Gaussian mixtures of nominals and anomalies. AUC and cumulative anomalies detected over time for four methods as new batches arrive. Anomalies occur with probability 0.01. Nominals follow N((O, I2) and anomalies ((c, I2/10) with results averaged over trials.
affect the average number of anomalies detected over time (See Supplementary Fig. 6 in the Ap- pendix). The only statistically significant effect was in the easy e = 2 discrimination sctting where unsurprisingly there was little value in adding an uncertainty sampler to the greedy selector. See the Appendix for more details on the Wilcoxon paired sign rank tests performed.
Effect of the choice of supervised classifier. We ran repeats of the same 2-dimensional trials for each of logistic regression, random forest, and a multilayer perceptron as the classifier. All three classifiers performed well and there was no clear winner; see Supplementary Fig. in the Appendix along with technical details.
Effect of the batch size. It not immodiately clear what influence the batch size B might have on overall performance, over time, of AAA-or or the other three methods. It would be of interest to try and understand the influence of B further, but given the number of variables in play over
8

---

Published in Transactions on Machine Learning Research (1/2026)
and above the batch size: data distribution over time, choice of score functions, choice of supervised classifier and its parameters, number of candidates to send to the expert from each batch, choice of active learner, true anomaly frequency), we consider general answer to this question out of the scope of the current paper.
3.4 Score distribution simulations
In this second series of simulations, we lake the score distribulion rather than dala distribution point of view to show that for AAA, true anomalies' scores need not be high to be informative; they just need to have sufficiently different distributions to nominals.
Score distributions. Four score distributions are presented in columm 1 of Fig. 3 It is important to understand that these plots are not data density plots like in Fig. 2 they correspond distributions of downstream anomaly and nominal scores. In all four trials, two-dimensional i.i.d. Gaussian data were generated. Each data point was then assigned vector of ten random uniform scores between O and 1, corresponding to the scores output from ten random uniform "anomaly detectors". We then artificially labeled each data point as an anomaly or nominal via some function of the random uniform scores output by the first (of the ten) random score anomaly detectors. Full details of the simulation setting for each of the four trials are provided the A Appendix. Fig. showss that AAA significantly outperformed the three LODA-based methods on all four score distributions.
(simulated)
Time series anomalies. In time series data, anomalies can be contextual: they may not directly depend on the individual data values at each time point but instead relationships between data values at different time points. We set the first point œ1 = (O.....0) 0) and then randomly generated 10-dimensional Gaussian mixture jumps for 2 2, leading to the following (random walk) data:
(1-²) 2
+ + (5(.5, ..5),,,,...,,........ 1001000 + (-=²7 N((G....,6)) 6), (o).
X; ~ + +
2
True anomalies were considered the points generated with "medium-sized jumps" from N(((5...... 5.5), /o0/100) with probability - = 0.01. We constructed a diverse ensemble of ten anomaly detectors made up of five LODA projections, isolation forest, one class SVM, local outlier factor, random score function, and Fuclidean distance score function (which could partially recognize jump magnitudes). All other parameter details can be found in the Appendix. Figure 4 shows what one of the data dimensions looks like (top left), what the score distributions look like for the Euclidean distance score function (bottom left), and the performance of the four methods (top and bottom right). AAA successfully detected anomalies and got better at doing so over time, while the other three methods failed.
3.5 Benchmark batch datasets
We tested the iterative batch performance of AAA on the eight benchmark datasets with more than 1000 points from the original isolation forest paper (Luu et al.) 2012). of which five were real data, one was simulated, and two were real data with injected synthetic anomalies. We used the same diverse set of ten anomaly detectors from Scction 3.4 We treated the eight datasets as unordered data sets and ran consecutive batch trials on five random permutations of each. We then averaged the cunulative anomalies detected over time new batches arrived. All technical details can be found in the Appendix. Fig. 3 shows that AAA outperformed the other three methods on all eight datasots, sometimes by a large amount.
Importance of the choice and number of ensemble detectors. In all of the trials after the initial simulations, we used the same diverse ensemble of ten anomaly detectors. To analyze the effect of the selection and number of detectors, we took the Satellite dataset (sce Fig. and the Appendix) and our usual diverse ensemble of ten anomaly detectors and (i) randomly shuffled the data 5 times; then for each shuffle (ii) randomly selected 100 subsets of size for each E (1,..., . 10) and ran AAA on batches of size B 100,
9

---

Published in Transactions on Machine Learning Research (1/2026)
Score densities
Average AUC over time
Cumul. anomalies detected
Nominals Arumalies
400
LODA
Active-LODA
2.
300
GLAD
30-
LODA
800
AAA
Aveorage
Active-LODA
Anomalies
GLAD
2.7
ANA
200
16
3.6
100
35
Score
Score densities
Average AUC over time
Cumul. anomalies detected
Nomirals
Anomalles detected
Active-LODA
220
GLAD
Density
2.8
LODA GLAD
200
Averrage
0/7
350-
200
25
130
150
Score
Batch
Batch
Score densities
Average AUC over time
Cumul. anomalles detected
200
1.25
Anomalless detectod
500-
GLAD
Active-LODA
Average AUC
Density
LODA
400
Nominals
Active-LODA
Aromalies
GLAD
300-
2:30
-2
120
150
200
50
100
250
200
Score
Batch
Batch
Score densities
Average AUC over time
Cumul. anomalies detected
Nominals
1.0
Anomaless class
Jnomaaies detected
200
Active-LODA
AAA
Density
Average
Active-LODA
GLAD
0.7
1:00
150
700
45
160 Batch
150
200
Score
Batch
Figure 3: AUC and cumulative anomaly detection performance of LODA, active-LODA, GLAD, and AAA for an ensemble of ten score models. The first score model has the score distribution shown in column 1. The other nine models were uninformative random uniform scores (not shown). The score distributions shown have not been scaled by their relative proportions the mixture in order to see the (rare) anomaly scores' distributions better. Results were averaged over five trials.
with a random forest classifier and greedy active learning. All subplots in Fig, 6 show the same average cumulative anomalies detected for the ten possible subset sizes solid lines. For each subset size's subplot, colored interval shows the 95% range of actual cumulative anomalies detected after certain number of batches across the 500 values. This shows that certain subsets of s anomaly detectors can perform better over time that certain subsets of s' delectors for s' < 8. However, for instance, an average subset of size 4 performs better than 95% subsets of size 1, while 95% the time, the set of 10 detectors performs better than the average subset size We also see that the average improvement increases over time but loss and less so over time. This suggests that adding even more detectors the setup would be possible in this example.
10

---

Published in Transactions on Machine Learning Research (1/2026)
First dimension series
Average AUC time
5-
M Mnn
1//0
6:00
Mal
LO04
Active LODA
GLAD
-100
-193
Time
Batch
Weighted score distribution for the 1st anomaly detector
Cumulative anomalies detected
Nominal (T-0)
140
LODA
Anommlly (1-1)
Active-LODA
GLAG
Anomalleo alees desecitd
AMA
720
175 305
anomaly score
Figure 4: Time series simulations for anomaly detection. Top-left: the data for the first (of ten) dimensions along with anomaly status. Bottom-left: the joint distribution of anomaly and nominal scores for the absolute Euclidean distance score only. Top-right: AUC over time as more batches are added, for four anomaly detection methods. Bottom-right: cumulative anomalies detected.
3.6 The GECCO water quality time series
Given we would also like to apply the AAA method to ordered time series data batches, we further tested it on the GECCO challenge dataset (Moritz et al. 2018), which involved 139 566 consecutive data points made up of water-quality related variables associated with anomaly or nominal labels. Real anomaly events were supplemented by artificial anomalies by the challenge organizers. We used the same diverse set of ten anomaly detectors from Section 3.4 Each sequence of contiguous anomalies was replaced by one randomly selected anomaly before running AAA; this random choice was repeated 20 times and results were averaged. Further technical details be found in the Appendix. Fig. shows the average cumulative mumber of anomalies detected. On average AAA detected 39 of the 50 remaining anomalies, significantly higher than the number detected by the other three methods. Since the GECCO data was highly non-stationary, it was not a surprise that active-LODA appeared to perform better than GLAD, given that GLAD tried (and therefore failed) use the raw data to optimize score weights via its neural network. Indeed, GLAD which is initialized equal neights-performet similarly to unsupervised LODA.
4 Discussion
AAA, the supervised score aggregation method for active anomaly detection presented here, extends active lcarning-bart basod anomaly detection methods like active-LOOA and GLAD general supervised framework and the batch ordered data setting for arbitrary ensembles of unsupervised anomaly detectors. The main feature differentiating AAA from previous score-based active anomaly detection methods is in how it drops the hypothesis permeating the literature that only individual anomaly score models that output "mostly higher scores for anomalies than for nominals" are informative. Here we show that even "mid-range" scores for true anomalies provide useful discriminatory information as long as there are differences bet ween the distributions of anomaly and nominal scores.
Despite no systematic performance difference between the different classifiers in Supplementary Fig. in the Appendix, we recommend using in the absence of prior knowledge random forest when running AAA
11

---

Published in Transactions on Machine Learning Research (1/2026)
Camul arerna lts deneddoo
200
Cumdl arame lex derected
200
1.00
100
1.00
40
60
100
Batth
Batch
Ounul antmale 5400040
500
Cumuu anomal d.0000d
200
20
1.00
60
100
Batch
Batch
Shuttle
Nammegraphyy
Canu artomaees detertedd
Camdd aroma tes detected
220
100
Batch
Batch
Annthyrald
Satellite
Cumuu unsenatos Sioedee
120
Oumul snunsiees 30000dd
80-
160
60
40
20
18
Figure 5: Average cummlative anomalies detected (over 5 trials) in the eight datasets with than 1000 points in Table III of [Liu et al. 2012).
from the acanag library. This is simply because random forest. is robust against correlated score functions (unlike logistic regression) and robust-uu unlike simple neural networks-and against ensemble functions which may output wildly different ranges of scores.
As for choosing an active learning stralegy when rumning AAA, it remained inconclusive whether was worth supplementing greedy active learning (sending the most likely-to-be anomalies the expert) with another active learner such uncertainty sampling. When the associated classification task was simple yet unbalanced (Supplementary Fig. 6, top-left), fully greedy active learning statistically significantly better than full uncertainty sampling. However, in the other haieehhader--ae cases, the trade-off between the two did not systematically suggest concentrating on labeling candidates from one or the other.
As it currently stands, AAA could under peeffor early labeled data suggested that all the true anomalies' scores were one zone of the M-dimensional score space, whereas in fact there were significant (undetected) concentrations of true anomalies elsewhere in the space. associated supervised classifier the set of future predicted anomalies and future active learner-obtained candidates sent to the expert for labeling in subsequent batches. One way to get around this would be to also provide a randomly chosen candidate to the expert from time to time in order to better explore the score space, though if anomalies were extremely rare, it might take a long time for this mechanism to stumble upon anomalies elsewhere in the score space. In any case, anomalics hidden elsewhere in the score space were associated with the highest scores from least one of the score detectors in the ensemble, the three KODA-inspired methods would not find them either.
decision boundary was learned by AAA''s 's unrepresentative set of anomalies, this could overtly inftuence both
12

---

Published in Transactions on Machine Learning Research (1/2026)
Cumnul detec. anom.
100
50
Subset size
Subset size
Cumul. detec. anom.
125
25
50
25
Subset size s
Subset sze 6
Cumul. detec. anom.
125
75
25
Subset size
Subset size
Curnul. detec. anoom.
125
75
50
25
Subset size
Subset size 10
Cumul. detec. anom..
100
75
30
25
20
Figure 6: Influence of the size 8 E (I,.... 10} of the ensemble of anomaly detectors on cumulative anomaly detection performance, for the Satellite dataset (see Appendix for details). Solid lines indicate the average over 500 random subsets of size from the original ten anomaly detectors. Colored zones indicate the 95% range of values for each choice of over the 500 trials and across batches.
GLAD
AAA
Gunulative anomalles detected
30
0
100
250
Batch
Figure 7: Average cumulative anomalies detected (over 20 trials) in the GECCO time series dataset, with randomly selected anomaly representative from each of the 51 anomaly intervals included in each trial.
13

---

Published in Transactions on Machine Learning Research (1/2026)
supposed a relatively high anomaly rate t (typically 1 in 100 in our trials), even though this could be much lower in real-world settings. Nevertheless, up to a point, AAA is agnostic with respect to this value, as long as it is still shown like in the simulations-a small number useful labeled anomalies and nominals to begin with. Indeed, on the real data from the GECCO industrial challenge (the real anomalies were also supplemented by synthetic anomalics by the organizers), despite an anomaly rate of 0.0004, AAA discovered on average of the anomalies despite only seeing one labeled anomaly begin with.
In simulations, in previous work (Das et al. 2016 Pevný 2016)
5
Acknowledgements
This work was supported by the French government under the France 2030 program, as part of the SystemX Technological Research Institute within the Confiance.at ai project.
References
Magnus Almgren and Erland Jonsson. Using active learning in intrusion detection. In Proceedings. 1 17th IEEE Computer Security Foundations Workshop, 2004., pp. 88-98. IEEE, 2004.
Arthur Asuncion, David Newman, et al. Uci machine learning repository, 2007.
Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin basod active learning. In International Conference on Compulational Learning Theory, pp. 35 50. Springer, 2007.
Salem Benferhat and Karim Tabia. New schemes for anomaly score aggregation and thresholding. In International Conference on Security and Cryplography, volume 2, pp. 21-21 28. SCITEPRESS, 2008.
Lucien Birgé and Yves Rozenhole. How many bins should be put in regular histogram. ESAIM: Probubilily and Statistics, 10:24-45 45, 2006.
Hamza Bodor, Thai V. Hoangl, and Zonghua Zhang. Little help makes a big difference: Leveraging active
learning to improve unsupervised time series anomaly detection. In Oriented Compufing-ICSOC 2021 Workshops: AIOps, STRAPS, AI-PA and Satellite Events, Duboi, United Arab Emirates, November 22 25, 2021, Proceedings, volume 13236, pp. 165. Springer Nature, 2022.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-supervised learning (adaptive computation and machine learning); 2006.
Tivadar Danka and Peter Horvath. modal: A modular active learning framework for python. arXie preprint arXiv:1805.00999 00979, 2018.
Shubhomoy Das, Weng- Wong, Thomas Dietterich, Alan Fern, and Andrew Emmott. Incorporating expert foodback into active anomaly discovery. 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 853 858. IEEE, 2016.
Shubhomny Das, Weng-Keen Wong, Alan Fern, Thomas G Dietterich, and Md Amran Siddiqui. Incorporat- ing feedback into brse-based anomaly detection. arXie preprint arXiv: 1et...70...44444 2017.
Shubhomoy Das, Md Rakibul Islam, Nitthilan Kannappan Jayakodi, and Janardhan Rao Doppa. Active anomaly detection via ensembles. arXiv preprint arXiv:1809.06/ 06477,,777 2018.
Sanjoy Dasgupta, Adam Tauman Kalai, and Claire Monteleoni. Analysis of perceptron-based active learning. In International conference on computational learning theory, pp. 463. Springer, 2005.
Yoav Freund, H Sebastian Scung, Eli Shamir, and Naftali Tishby. Selective sampling using the query by committee algorithm. Machine learning, 28:133-168, 1997.
Jing Gao and Pang-Ning Tan. Converting output scores from outlier detection algorithms into probability estimates. In Sirth International Conference Data Mining (ICDM 06), pp. 212 221. IEFF, 2006.
14

---

Published in Transactions on Machine Learning Research (1/2026)
Jun Gao, Weiming Hu, Zhongfei Zhang, and Ou Wu. Unsupervised ensemble learning for mining top-n outliers. In Pacific-Asia Conference on Knowledge Discovery and Dala Mining, pp. 418-430 Springer, 2012.
Nico Görnitz, Marius Kloft, Konrad Rieck, and Ulf Brefeld. Toward supervised anomaly detection. Journal of Artificial Intelligence Research, 16:235-262, 2013.
Zengyou Hc, Xiaofei Xu, and Shengehun Deng. Discovering cluster-based local outliers. Pattern recognition letters, 24(9-10):1611-1650 2003.
Md Rakibul Islam, Shubhomoy Das, Janardhan Rao Doppa, and Sriraam Natarajan. Glad: Glocalized anomaly detection via human-in-Bee in-the--oooppp learning. arXiv preprint (xXie:1810001403 2018.
Hans-Peter Kriegel, Peer Kroger, Erich Schubert, and Arthur Zimek. Interpreting and unifying outlier scores. In Proceedings of the 2011 SIAM International Conference Data Mining, pp. 13-24. SIAM, 2011.
Christopher Kruegel and Giovanni Vigna. Anomaly detection of web-based attacks. In Proceedings of the 10th ACM conference on Computer and communications securily, pp. 251 2511-661, 2003.
Christopher Kruegel, Darren Mutz, William Robertson, and Fredrik Valeur. Bayesian event classification for intrusion detection. In 19th Annual Computer Security Applications Conference, 2003. Proceedings., pp. 14 23. IEEE, 2003.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation-based anomaly detection. ACM Transactions Knowledge Discovery from Data (TKDD), 6(1):1-39, 2012.
Steffen Moritz, Frederik Rehbach, Sowmya Chandrasekaran, Margarita Rebolledo, and Thomas Bartz- Beielstein. Gecco industrial challenge 2018 dataset: A water quality datasct for the 'internet of things: Online anomaly detection for drinking water coliietioootttioonnn at the genetic and evolutionary compu- tation conference 2018, kyoto, japan. Kyoto, Japan, 2018.
Tomáš Pevný. Loda: Lightweight on-line detector of anomalies. Machine Learning, 102:275-31 304, 2016.
Tiago Pimentel, Marianne Monteiro, Adriano Veloso, and Nivio Ziviani. Deep active learning for anomaly detection. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
Tsirizo Rabenoro, Jérôme Lacaille, Maric Cottrell, and Fabrice Rossi. Anomaly detection based aggrega- tion of indicators. arXi preprint or oxXiv:1407.0880, 2014.
David M Rocke and David L Woodruff. Identification of outliers in multivariate data. Journal of the American Statistical Association, (1(435):1047-1055 1996.
Nicholas Roy and Andrew McCallum. Toward optimal active learning through monte carlo estimation of error reduction. Teml, williamstowm, 2(441-448):4, 2001.
Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawo-Taylor and John Platt. Support vector method for novelty detection. Advances in neural information processing systems, 12, 1999.
Md Amran Siddiqui, Alan Fern, Thomas Dietterich, Ryan Wright, Alec Theriault, and David W Archer. Feedback-guided anomaly discovery via online optimization. Tn Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2200 2209, 2018
Jack W Stokes, John Platt, Joseph Kravis, and Michael Shilman. Aladin: Active learning of anomalies to detect intrusions. Technical Report, 2008.
Xuning Tang, Yihua Shi Astle, and Craig Freeman. Deep anomaly detection with ensemble- based active learning. In 2020 IEEE International Conference on Big Data (Big Dala), pp. 1663-1670 TEFF, 2020.
David MJ Tax and Robert PW Duin. Support vector data description. Machine learning, 54:45 66, 2004.
15

---

Published in Transactions on Machine Learning Research (1/2026)
Kalyan Veeramachaneni, Ignacio Arnaldo, Vamsi Korrapati, Constantinos Bassias, and Ke Li. Ai 2: training a big data machine to defend. In 2016 IEEE 2nd international conference on big dala securily on cloud (BigDataSecurity), IEEE international conference on high performance and smart computing (HPSC), and IEEE international conference on intelligent data and security (IDS), pp. 49-54. IEEE, 2016.
Graham Williams, Rohan Baxter, Hongxing Hc, Simon Hawkins, and Lifang Gu. A comparative study of rmn for outlier detection in data mining. In 2002 IEEE International Conference Dala Mining, 2002. Proceedings. pp.
IEEE, 2002.
Ruyue Xin, Hongyun Liu, Peng Chen, and Zhiming Zhao. Robust and accurate performance anomaly detection and prediction for cloud applications: novel ensemble learning-based framework. Journal of Cloud Computing, 12(1):7, 2023.
Kenji Yamanishi, Jun-Ichi Takcuchi, Graham Williams, and Peter Milne. On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms. In Proceedings of the sirth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 220-324, 2000.
16


Reviews:
[{"review_id": "IblBi1Ckf1", "reviewer": "Reviewer_qvMC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Active Anomaly Aggregation (AAA), a framework that combines unsupervised anomaly detectors, supervised learning, and active learning to detect rare anomalies in complex or time-series data. Instead of working directly with raw data, it uses the detectors’ output scores as features, allowing the model to learn more flexible decision boundaries and make efficient queries to experts. The experiments show that AAA performs better than previous methods such as LODA, active-LODA, and GLAD, particularly when anomalies are subtle or depend on context.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s claims are well supported through theoretical analysis and comprehensive experiments. Results across simulated and real-world datasets clearly demonstrate AAA’s advantages over existing methods. The evidence is consistent, detailed, and aligns with the authors’ stated goals.", "audience_yn": "Yes", "audience_explanation": "I believe the findings will be in the interest of TMLR’s audience, scpeially those working on anomaly detection, active learning, and ensemble methods.", "requested_changes": "The paper would benefit from clearly stating its main research questions in the introduction, followed by an explanation of the hypotheses and how the experimental design is built around them. It would also help to add a dedicated subsection discussing the importance and real-world applications of the proposed method, as this aspect is currently underdeveloped. The writers mention of the Python library acanag in the abstract should be expanded with more details about its implementation, computational complexity, and usability. Finally, improving the overall writing flow and structure would make the paper clearer and more engaging to read.", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jgf7ffEaK8", "reviewer": "Reviewer_TKX2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel approach to anomaly detection that combines a traditional active learner with a classifier trained on the outputs of an ensemble of unsupervised anomaly detectors. While the main idea is an original extension of previous work on linear combinations of unsupervised anomaly detectors, the paper suffers from two main weaknesses: structure/organization and empirical evaluation.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper's claims are credible. The evidence on the synthetic and \"semi-synthetic\" datasets is promising, but the paper lacks a motivating, real-world dataset (given the modifications to GECCO, this domains only counts as \"semi-synthetic\"). Ideally, the paper should show empirical evidence on several real-world datasets for which THE SAME ENSEMBLE of unsupervised anomaly detectors leads to compelling results. Alternatively, the paper should make it crystal clear if the practitioners are expected to come up with customized ensembles for each application (definitely a less compelling alternative).", "audience_yn": "Yes", "audience_explanation": "Anomaly detection is an important practical application.", "requested_changes": "The paper would greatly benefit from:\n1) a motivating real-world domain that is clearly well-suited for the proposed approach. The edits that have to be made to GECCO make it obvious that GECCO is NOT such a domain\n2) several real-world datasets in which the same ensemble of unsupervised anomaly detectors leads to compelling results; alternatively, a comprehensive discussion on how one has to build the ensemble for a particular dataset\n3) a massive re-organization:\n- the current INTRO takes 2 full pages to get to the proposed approach; move its related work paragraphs to the appropriate section, and go for a traditional 4-paragraph structure: general setting, why the problem matters, intuition behind the proposed approach, and a brief summary of the main contributions & results\n- Fig 1 looks more complicated than it should be because it seem to \"translate\" traditional pseudo-code into \"pseudo-code within a flow chart\"; you can either turn it in proper pseudo-code, or simplify the flow chart\n- the paper has many very long sentences that are hard to parse; for example, see the 4-line first sentence in second paragraph on page 6; similarly see the 3rd sentence in 3.1\n- most of the section 2.4 should go in the APPENDICES, where it will not impact the flow of the paper\n- please avoid weak statements such as \"usually outperforms\" (in Abstract; try to quantify it instead ) or \"we did not feel it necessary\" (page 8)\n- reorganize Section 3, which consists of 10 pages,  so that it does NOT feel like a long list of \"and we did this, and we did that\";  you should provide a crisp roadmap to it early in 3.1, and then make sure that you make each point as crisply as possible. In particular, you are using 9 of 10 pages to discuss the results on synthetic data, which is an over-kill. Given the limitations of synthetic data sets, one should focus there on the few main points, rather than the fine nuances, which should be backed by real-world experiments", "broader_impact_concerns": "n/a", "additional_comments": "n/a", "extra_scores": {}, "extra_text": {}}, {"review_id": "qu6ZcTp9hm", "reviewer": "Reviewer_oy49", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the **Active Anomaly Aggregation (AAA)** framework, a supervised active-learning system designed to efficiently identify rare anomalies in incoming data batches. The core contribution is developing a superior method for combining the output scores from multiple anomaly detectors.\n\nInstead of training the supervised classifier directly on the raw data, the paper trains it **only on the scores** generated by an ensemble of simple, unsupervised anomaly detectors. This allows the system to learn a **non-linear, optimal combination** of those scores.\n\nThe AAA framework successfully integrates three key concepts: **unsupervised ensemble detection, supervised learning, and active learning.**\n\n### Key Contributions of the AAA Framework\n\n1.  **Improved Score Aggregation:** By leveraging a supervised classifier to analyze the detector scores, the method learns a non-linear relationship, which is a significant advancement over simple, fixed-weight linear aggregation techniques.\n2.  **Hybrid Label Sampling:** The active learning component employs a robust, two-step strategy for selecting data points for expert labeling:\n    * **(i) Exploitation:** Prioritizing points that the current supervised model is most certain are anomalies.\n    * **(ii) Exploration:** Querying other points suggested by a standard active learning policy (e.g., uncertainty sampling) to ensure the model learns about diverse or new types of anomalies.\n3.  **Wider Applicability:** Since the method operates exclusively on the score vectors rather than the raw feature space, it is effective for both standard, independent data and **ordered data like time series**, thereby significantly broadening its practical utility.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors show the system's effectiveness through solid tests on various datasets.\n\nThe tests against the baselines seem fair. The main comparison is against linear aggregation methods, which are the simplest way to combine scores. The AAA framework consistently performs better - often by a large margin - across all trials. \n\nRealistic Scope: By including simulated data, real-world benchmark data, and time series data, the authors prove the system is robust and not just working on selected datasets.\n\nI checked out the python library and everything is reproducible.\n\nThe results strongly suggest that teaching a model how to read the detectors' scores works much better than just averaging them, which is the core idea of the paper.", "audience_yn": "Yes", "audience_explanation": "The paper sits at the intersection of several active research and application areas:\n\n1. Anomaly Detection -> real-world applications like fraud detection or industrial failure monitoring. This paper delivers a practical, high-performing technique for efficiently using scarce expert labels, making it immediately useful for practitioners.\n\n2. Active Learning ->The design of the hybrid query strategy is a key topic in active learning. This framework provides a concrete, successful example of this strategy being used in a difficult, imbalanced data setting.\n\n3. Ensemble Methods & Meta-Learning -> The idea of using a primary model's output (the scores) as the feature set for a secondary learning model is a great example of meta-learning.", "requested_changes": "The paper is good enough to be accepted, but I suggest addressing these points to make the contribution even stronger and clearer for the community:\n\n1. Prove the Query Strategy: Please add a simple experiment to show what happens if you only query the most likely anomalies (mechanism (i)) and compare that performance to the full AAA strategy. We need to see if the extra effort of the \"exploration\" part (mechanism (ii)) actually improves the final result.\n\n2. Ensemble Choice Sensitivity: Discuss what happens if the original group of unsupervised detectors (M) are not very good or if they all give very similar scores. Does the AAA still work well? Adding a quick check on the system's robustness to a poorly chosen initial ensemble would be helpful guidance for users.\n\n3. Runtime Check: Briefly mention the practical cost of the approach. Since the system retrains the supervised classifier on every data batch, a small note on the training overhead compared to simpler, fixed-weight methods would help people decide if they can deploy it in a fast-paced environment.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)