SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Authors: Saumya Gaurang Shah
Abishek Sankararaman
Balakrishnan Murali Narayanaswamy
Vikramank Singh
Venue: ICML 2025 poster (2025)
Keywords: anomaly detection
online learning
streaming
continual learning
Primary area: general_machine_learning->online_learning_active_learning_and_bandits
TL;DR: This paper introduces SEAD: The first technique for ensembling unsupervised anomaly detectors in the streaming setting, adapting to the distribution of scores generated by base anomaly detectors on each dataset.

Abstract: Can we efficiently choose the best Anomaly Detection (AD) algorithm for a data-stream without requiring anomaly labels? Streaming anomaly detection is hard. SOTA AD algorithms are sensitive to their hyperparameters and no single method works well on all datasets. The best algorithm/hyper-parameter combination for a given data-stream can change over time with data drift. 
'What is an anomaly?' is often application, context and dataset dependent. 
We propose SEAD (Streaming Ensemble of Anomaly Detectors), the first model selection algorithm for streaming, unsupervised AD. All prior AD model selection algorithms are either supervised, or only work in the offline setting when all data from the test set  is available upfront. We show that SEAD is {\em(i)}  unsupervised, i.e., requires no true anomaly labels, {\em(ii)}  efficiently implementable in a streaming setting,  {\em (iii)}  agnostic to the choice of the base algorithms among which it chooses from, and {\em (iv)}  adaptive to non-stationarity in the data-stream. Experiments on 14 non-trivial public datasets and an internal dataset corroborate our claims.

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Saumya Gaurang Shah
Abishek Sankararaman' Vikramank Singh"
Balakrishnan (Murali) Narayanaswamy
Abstract
We study the streaming unsupervised setting, where AD algorithm makes decision as each data point streams in. Further, the algorithm is never given any feedback as to whether any of the inputs seen thus far were truly anoma- lous. This setting is increasingly common in modern cloud- computing and software monitoring systems where the vol- ume and velocity of data is high and labeling any given event as anomalous not is expensive in terms of human time and effort (Xu et al., 201 8a;b; Sankararaman al., 2022). Current unsupervised AD algorithms are effective at identifying anomalous events when the data satisfies their modeling assumptions such as distance preservation in low dimensional random projections (Manzoor al., 2018), our- liers increasing model complexity (Guha et al., 2016) or anomalies isolated from the center of benign data points (Ding & Fei, 2013).
Can we efficiently choose the best Anomaly De- tection (AD) algorithm for data-stream without requiring anomaly labels? Streaming anomaly detection is hard. SOTA AD algorithms are sensitive to their hyperparameters and no sin- gle method works well on all datasets. The best algorithm/hhper--arrane
combination for given data-stream can change over time with data drift. 'What is an anomaly?' often application. context and dataset dependent. We propose SEAD (Streaming Ensemble of Anomaly Detectors), the first model selection algorithm for streaming, un- supervised AD. All prior AD model selection al- gorithms are either supervised, or only work in the offline setting when data from the test set available upfront. We show that SEAD (i) unsu- pervised, i... requires no true anomaly labels, (ii) efficiently implementable in streaming setting, (iii) agnostic to the choice of the base algorithms among which it chooses from, and (iv) adaptive to non-stationarity in the data-stream. Experiments on 14 non-trivial public datasets and an internal dataset corroborate our claims.
However, no single model performs well on all datasets (Schmidl et al., 2022; Paparrizos et al., 2022; Han et al., 2022), and existing methods are sensitive to their hyperpa- rameters (Zhao et al., 2021; Schmidl et al., 2022; Paparrizos et al., 2022; Han al., 2022). For example, on the Pendigits dataset (Keller et al., 2012; Sathe & Aggarwal, 2016a), the IForestASD algorithm (Ding & Fei, 2013) obtains superior performance when compared to simple rule based Statisti- cal Process Control (SPC) (Shewhart, 1931) model, but the ordering is reversed on an internal Telemetry dataset (Table in Sec 4). Similarly, on the Letter Recognition dataset (Slate, 1991; Micenková al., 2014), changing the sliding window hyperparameter from 64 128 for the RRCF al- gorithm (Guha et al., 2016) increases the average precision score (APS) by ~ 42%, but setting it to the default value of 256 brings down the APS by ~ 13%. (cf. Table 7)
1. Introduction
Detecting anomalies on noisy, high-dimensional and non- stationary data-streams becoming an important sub- routine in several applications such monitoring (AWS, 2025; Hagemann & Katsarou, 2020), fault-detection (Singh et al., 2024; Islam et al., 2021), intrusion detection (Lazare- vic et al., 2003) and more. See the surveys of (Chandola et al., 2009; Iglesias Vázquez et al., 2023). Given this ubiq- uity, several anomaly detection (AD) settings and algorithms have been designed in the literature with various modeling assumptions (Ruff et al., 2021).
Table 1. Accuracy numbers on the 3 datasets shown in Figure 1. Each dataset has a perfect detector, but none of the perfect detectors perform well on all 3 datascts. SEAD performs model selection online to yield perfect accuracy on datasets without access to any labels.
'Amazon Web Services, Santa Clara, CA. USA. Correspon- dence to: Saumya Gaurang Shah <saumyags@ wwwannyag@aaazzo..com amazon.com. com">.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
DETECTORI
DETECTOR2
DETECTOR3
SEAD
RIIS, ITPIIIITTIII AFATION_CCTEESS
100%
0%
100%
ELB. REQUEST.COUNS COUT.
0%
100%
0%
100%
100%

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
rds uus_cou__ttilliatioo_cco5
1.2. Design principle behind SEAD
25
Detector 1
20
The key insight that SEAD leverages is that anomalies by definition are 'rare', which SEAD uses to work in fully unsupervised fashion. Anomaly detection algorithms output an anomaly score for each input, with the convention that larger the score, more likely the input is anomalous (Schmidl et al., 2022). In SEAD we exploit the observation that a candidate AD algorithm is 'good' on a data-stream if it has consistently output small anomaly scores in the past, and similarly is 'poor' if it has oulput larger scores in the past. This observation builds the assumption that anomalies on any data- stream, by definition are rare. SEAD maintains a weight for each candidate AD algorithm with algorithms that have consistently output lower scores having higher weight and vice-versa. When a new data point comes in, SEAD outputs as its anomaly score the weighted combination of anomaly scores from the individual models.
15
10
500
1500
2000
2500
3000
3500
4000
elb _e__reqestt count_ 8c0756
600
Detector 2
200
500
1000
1500
2000
2500
3000
3500
4000
1e8
ec2 [c2_network_BB in 257a54
Detector 3
The main technical contribution in SEAD is method- ology to set the weights for the individual models. SEAD chooses the weight using the classical multiplicative weights update (MWU) (Cesa-Bianch et al., 1997) to predict with expert advice. We treat each candidate AD algorithm as an expert and treat both the advice and the loss incurred by an expert as its output anomaly score. This formulation ensures that candidate AD algorithm that consistently out- puts smaller anomaly scores, will have a higher weight. The complete pseudo code is in Algorithm 1.
500
1000
1500
2000
2500
3000
3500
4000
Figure 1. Three AD datasets from the Numenta Anomaly Detec- tion Benchmark (Ahmad al., 2017) (NAB), each of which has a perfect but diffcrent detector shown by the orange horizontal dotted line. However, each detector performs poorly on the other 2 datasets. SEAD is the perfect detector for all datasets, as shown in Table
We motivate our intuition of down weighing detectors with high scores in Figure 1, using results from 3 simple univari- ale datasets in the Numenta Anomaly Detection Benchmark (NAB) (Ahmad al., 2017). Most NAB datasets are eas- ily solvable by rules (Wu & Kcogh, 2020), including the three datasets that we consider. Each dataset has a perfect anomaly detector of the form y >> a with a different value of the parameter a for each dataset, as shown in Table I and Figure 1. We show that SEAD is able to select the best detector in each case. Later, we compare with state-of-the the art streaming AD methods on more complex datasets, and show that SEAD obtains the best performance using both quantitative metrics and qualitative examples. SEAD is only as good as its individual components, so it needs at least one model to perform well at detecting anomalies on the given dataset i.e. the underlying data must satisfy assumptions of at least one base model.
SEAD provides an algorithmic solution to a key challenge encountered in practice- - the choice of which model to use when, especially when labels are rare or unavailable, and decision needs to be made in a streaming fashion. The vast literature on AD suggests that there are good models for most situations one encounters in practice. However, the assumption that one knows the distribution of data in ad- vance is becoming an unrealistic one, especially in modern dlouuccomputing systems that serve a variety of workloads (Zhang et al., 2021; Singh et al., 2024).
1.1. Summarizing our contributions
We propose SEAD, the first online model selection algo- rithm for AD, that is both completely unsupervised, and works online as the data streams in. In particular, SEAD is (i) unsupervised and does not require anomaly labels, (ii) efficiently implementable online, i.c., the computational complexity process each data-point on the stream does not depend on the stream length, (iii) agnostic to the choices of the base algorithms among which it chooses from, and (iv) adaptive to non-statinn non-stationarity in the data-stream, i.e., the optimal choice of the algorithm changes as the data-stream's distribution changes.
An advantage of SEAD is its computational efficiency on data-stream. Existing outlier ensembles cannot be used in the streaming setting because. they use computationally ex- pensive (n=²) operations like ranking the anomaly scores across time (Rayana & Akoglu, 2014; 2016) or using all data points to compute agreements between base detectors
(Rayana al., 2016), where n is the number of data points seen so far. Similarly, meta learning based model selection
2

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
methods (Ding et al., 2024; Zhao et al., 2021; 2022) also use expensive (n²2) operations, looking each historical data point. On the Ozone dataset (Zhang et al., 2008), MetaOD (Zhao et al., 2021) takes roughly 2 days to make predictions on a data stream of about 2,534 data points, often taking > 1 minute for a single data point. Ozone sensors should have low detection times of <! 1 min because ozone expo- sure can have serious health consequences (Barreto al., 2022). Our method SEAD takes only ~ 1746 seconds the Ozone dataset, taking less than half minute on average and processing each data point in (11) time. SEAD ++ further improves runtime, taking only ~ 945 seconds with comparable detection performance.
Performance Metric. The metric of interest for the algo- rithm A is the Averaged-Preecison score (APS), which is informative of AD performance given the imbalanced nature of anomalies being only small fraction of all datapoints (Ruff et al., 2021). We use this metric as it has been well documented that for most applications of AD, the APS is a meaningful metric of performance in practice compared to other binary classification metrics such as AUROC. How- ever, note we cannot directly optimize the APS since the true anomaly labels are never revealed.
3. The SEAD Algorithm
The challenge in designing an ensembling algorithm the fact that the true anomaly labels Yi, ...,., Yr are never re- vealed the algorithm. This might lead one conclude that the problem of online unsupervised aggregation can- not be solved. We show that if the data-stream is such that anomalies are very rare for at least one of the base detectors, then SEAD given in Algorithm leverages this property to get an unsupervised anomaly aggregation algorithm with good performance.
Organization Section describes the problem statement and the formal description of SEAD is in Section 3. Experi- mental results are in Section 4 followed by related work in Section 5 and conclusions in Section 6.
2. Problem Setting
We consider the problem of online anomaly detection, where at each time = 1, .... a data- point X, E X is presented to an algorithm which returns non-negative score S. R+ with the interpretation that larger the score, more anomalous the input. Throughout, we will assume that each data-point is an element of the set X, which could be numerical, or a mix of numerical and categorical or even a mix of numerical and free-form text. The exact description of X is not crucial for our exposition. Additionally, associated with each data point X₁, is binary label Y {0, 1} with the interpretation that Y 1 implies that X, is anomalous. The AD algorithm does not have access to this label, nor can it get access to this over time.
Online data-stream.
Recall that the goal of an anomaly detection algorithm is to output smaller scores for non-anomalous inputs and larger scores for anomalous inputs. On a data-stream that has a very small number of anomalies, a good anomaly detection algorithm is one that usually outputs smaller scores. We leverage this observation and use the unsupervised loss of directly minimizing the anomaly score as proxy for the supervised loss of APS in SEAD.
3.1. Unsupervised ensembling
Our proposed methodology is to output as anomaly score at time t, S, = Σ w. we 2 0 for all i. Ideally, we want the weight w to be large if algorithm is 'good' and small when algorithm i is 'poor'. Since in our setting, the true binary labels are never revealed, we rely on the following unsupervised proxy loss to set the weights. An algorithm that has consistently output smaller anomaly scores in the past is likely to be superior and thus should be assigned a higher weight.
Base AD algorithms. We assume that we have access to N (.), where
(4) 5(1) where Σ w/" = 1 and
base algorithms, denoted by (.), ()), s(A)) = -1('( (X+) is the anomaly score output by algorithm i on imput X,. Each base algorithm is also indexed by time allowing it to change over time, e.g. through incremental or online learning.
Online ensembling. The goal of our system is to predict an anomaly score S, E IR, at each time for the input X,, based only on the historical information seen so far. At each time I, the input X, is scored by each of the N base algorithms to give the N individual scores { Te-1- The ensembling algorithm then outputs a single anomaly score S. E K4 by taking these N scores as input. The ensembling algorithm can be 'stateful', i.e., can depend on the past and present inputs (X.)sst, past and present base anomaly scores {s!²) (NEEN and past outputs {Ss)set- The algorithm however must be (i) online, i.e., produce the output S, before observing the input X and (ii) be unsupervised, i.e., Y, never revealed.
3.2. Normalizing the anomaly scores
A roadblock in this proxy is that the anomaly scores across different base algorithms may not be comparable. For exam- ple consider two different algorithms - the first one outputs the scores in the range [1, 2]. while the second one only outputs in the range [0, 1]. If we directly use the anomaly scores as the loss for each algorithm, the first algorithm will always get a lower weight compared to the second one, no matter the relative performance as measured by APS.
3

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
In SEAD, we circumvent this issue by normalizing each AD algorithm's 's output be the quantile of the anomaly score according to the empirical distribution of the past scores, i.e., use Quantile (5(0) : - 8s33) (Line 8 in Algorithm 1).
Algorithm 1 SEAD Algorithm
1: Input Streaming data {x1, { J. learning rate 17 >
0, regularization À 2 0. prior distribution π E A N-1
Initialize weights W1 < [1,,... 1] ERR,
3: Initialize base AD detectors A ()), for all SiSN
Henceforth, we refer to St E [o, 1] to be the normalized anomaly score output at time t. by base algorithm i. The streaming quantiles can be efficiently implemented for ex- ample using t-digest
4: for Time t = 2, ... do
5: Receive input x,
6:
for each base detector E (I,,,,,, ,NH do A
(Dunning & Ertl, 2019). For larger data-streams, a windowed quantile that transform s(6) to be the quantile of the scores only in the past W time- steps, where W is hyper-parameter can be efficient. The implementation in Algorithm 1 corresponds to W- 00.
7:
8:
S(9) + Quantile (3)
1 i7 Ex= )5:0)
end for
10: Output Score S. + E
explw s(O))
N 2 ex
11: Update weights according to Equations (1) and (2).
3.3. Prediction with expert advice
12: A < Update-Algo-i(A) (A(0)
()), xx), Vi [N]
In order to set the weight (6) to be large, if the scores s("1) are small and vice-versa, we use the clas- (FTRL) algorithm to
13:
end for
...., (S(").... sical follow--he--rgullaiie----- predict with expert advice Ressa-Biacchi et al., 1997; Cesa- Bianchi & Lugosi, 2006). In our setting, the anomaly scores by each algorithm is both the prediction of that expert, as well as the loss incurred by following that expert. The ensembling algorithm observes the N expert predictions (s(") , s{(")) and outputs score St as convex combi- nation of the N expert's predictions.
vious weights wi-1, and (ii) re-parametrize w to be un- constrained using the softmax transform. particular, we set the weights W, E R-N and output the score (Line of Algorithm
S. : Σ
N
exp(w,
( m
S(O)
j==
Formally, the FTRL primitive is to choose the weights (1) Wr : [w) , ..... " at time to be the minimizer of the sum of historical performance and regularizer as w == arg min N-1 E.=1(w (w.S.) + RwW,,, where A N- is the simplex on N dimensions and (-(-) is a non- negative regularization function.
and update the weights using a single step of gradient de- scent (Line 13 of Algorithm 1)
W++1 + We V7 (.
(I)
where the loss function (Line 12 of Algorithm 1)
[.(w)= +-
N Σ
exp(ws)
s(1)
3.4. Putting it together
== exp (xp((W))
1=1
SEAD employs the KL divergence as the regularizer, which penalizes deviation of the weights w from a fixed distri- bution v, which is hyperparamettr of SEAD This dis- tribution ı encodes prior information over the base AD algorithms as to which ones are better than others. For ex- ample, if there is some offline data from the stream available, one can use the offline model-selection algorithm of (Zhao et al., 2021), to inform a prior distribution for the weights on the online stream. On the other hand, if there is no prior information, we set as the uniform distribution, which reduces the KL divergence regularizer to the negative tropy of the weights w, which corresponds to the classical multiplicative-weigh update (MWU) algorithm. This is beneficial since it is well known in online learning literature that MWU U is optimal (upto constant factors) in the absence of any prior information (Cesa-Biae -isachi et al., 1997).
cxp(w)
AKL
(2)
2j=1 exp(wy)
Observe that this loss function ensures that the final anomaly score output is minimized while ensuring that the weight distribution does not deviate too much from the prior infor- mation (if any) that is available.
3.5. Online update to the base algorithms
In SEAD we allow for the N base-algorithms to also be incremented with each incoming sample x; : for example. a standard auto-encoder could be updated with single gradi- ent descent step as in (Mirsky et al., 2018), or an isolation forest can be incremented using custom tree-update algo- rithm as in (Leveni et al., 2024). SEAD itself is agnostic to the update mechanism, except for the fact that the up- dates need to be unsupervised, i.c., the true anomaly label information is not known.
Further to reduce complexity, (i) we set the weights time t denoted by w, by taking one gradient step from the pre-
4

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Algorithm 2 SEAD ++: Optimizing runtime by sampling
detectors. We propose a modification SEAD in Algorithm 2, which only considers the anomaly scores and incremental updates to K out of N algorithms at each time. The design of Algorithm 2 is motivated by the classical Thompson sampling a algorithm (Russo et al., 2018), originally proposed to solve the special case when exactly one out of the N base detectors can be chosen at each time. Formally, Algorithm takes K S N as input and at each time t, samples a subset E, consisting of K out of the N detectors. This subset is sampled without replacement from the set of N detectors with each detector i sampled with probability proportional to exp(w(() (Line 6 of Algorithm 2). The output S, is given by weighted combination of the scores by the detectors in Ee (line 11 of Algorithm 2). The weight vector Wz+1 is obtained by gradient step
1: Input: Streaming data {x1, X2,-. J. learning rate v) >> 0, regularization strength À 2 0. prior distribution π E
A AN-1, K E (1,..., ,
2: Initialize weights Wi _1,.............. ER.
R*
3: Initialize base AD detectors (-), for all SISN
4: for Time 1 = 1,2, 1,2,
do
5: Receive input x
6: E, < set of K detectors sampled without replace- ment with detector E [N] sampled with probability proportional to exp(w, ())
7:
for each base detector i E Eg do
5(t0) A (x:)
8:
9:
A = (1 S Stst. Es}
10:
5(00 + Quantile 5(0 |
Eueapo Ogen
W+11 f We V [w(, E:)lwew.
(3)
11:
end for
modified loss function L. (w, Ex) that accounts for the sub-sampling of detectors
12:
Output Score S. EiEE,
explu(())
S((i)
Ejee, exp(uol1))
13:
Update weights according to Equations (3) and (4).
14:
A.21()) + Update-Algo-i(4" ()). vi
Es
[. E.)
Σ iEE.
exp(w:)
s(()
)+
15:
end for
ΣJEB, exp(w,)
exp(w)
AKL
(T) (4)
3.6. Theoretical Guarantees
ΣE= exp(w)))
Since SEAD is built on the FTRL abstraction, it also inherits several desirable properties of FTRL. In particular, SEAD enjoys small 'regret', namely the sum all anomaly scores output by SEAD, no more than O( [TI((n))))) In())) compared to the sum of anomaly scores output by any base-algorithm. When the parameters of learning rate and regularization strength are appropriately chosen, the MWU algorithm, and as a consequence SEAD, is adaptive to distribution shifts
Observe that when K = N, Algorithm 2 is identical to Algorithm 1. The main savings in computation in Algorithm 2 comes in lines 8 and 14, where only K detectors are queried for an anomaly score and incrementally updated at each time. In contrast, Algorithm queries anomaly scores and incremenaally updates all N algorithms at each time.
The main reduction of runtime in SEAD i+is in not needing to do forward pass/inference and backward pass/gradient calculation on K N AD models. For many big models, these forward and backward passes are the main computa- tional bottlenecks. While SEAD ++ does require maintain- ing weights for each base detector, the updates and storage for the weights are trivial and independent of the size of the base AD models.
Theorem 3.1 (Single best in hindsight, (Cesa- Besa-Bianchi & Lugosi, 2006)). The difference between the expected cumu- lative sum of anomaly scores output by our algorithm and any base AD algorithm is no more than O (VTIn(N))
However, this result does not say that the performance of SEAD as measured by APS, which requires labels matches that of the best base-algorithm. Nevertheless, this theorem indicates why is possible for SEAD be as competitive as the best possible AD algorithm in hindsight, which our experiments corroborate.
4. Experiments
We empirically evaluate the following research questions.
1. Is SEAD effective, i.e., can match the performance of the best base AD Algorithm on any data-stream?
B.7.SEAD ++: Runtime optimization by sampling base AD Algorithms
2. SEAD efficient, i.c., has better detection perfor- mance and runtime in the streaming setting compared to offline methods?
A drawback in SEAD as given in Algorithm 1 is that at every time t, all N base algorithms provide an anomaly-score and are incrementally updated. The computational cost can be prohibitive for large N, which can occur when different hyper-paramest configurations correspond to different base-
3. Does SEAD ++ improve over the runtime of SEAD and, how does it affect detection performance?
In addition, we also show an anecdotal case study to further give intuition on how SEAD works.
5

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
4.1. Experimental Setup
IForestASD is the best method on the Pendigits, Cardio and Ionosphere datasets. Surprisingly, the rule based method per- forms the best on multiple datasets: Pima, Mammography, HTTP and Telemetry. All base methods show sensitivity to their hyperparameters. Examples include RRCF on WBC, xStream on Pendigits and IForestASD on Optdigits datasets. The sensitivity to model selection and hyperparameter con- figurations underscores the need for an ensemble model that can appropriately aggregate base models depending on the dataset.
We use 4 streaming anomaly detectors in our base mod- els: IForestASD (IF) (Ding & Fei, 2013), xStream (XS) (Manzoor et al., 2018), Robust Random Cut Forest (RRCF) (Guha et al., 2016) and a simple rule based model based on Statistical Process Control (SPC) (Shewhart, 1931). For IForestASD, xStream and RRCF, we use open source im- plementations from PySAD (Yilmaz& & Kozat, 2020). For the rule based model, we compute anomaly score for a D dimensional incoming data point = [x],-.. [ as de where μ4 and o; are the mean and stan-
EΣP li-/s, dard deviation for the jth dimension. We initialize each of IForestASD, xStream and RRCF with 4 different hyperpa- rameter configurations including the default configuration (please refer to Appendix C for more details). Since we focus on the streaming setting this work, we only perturb parameters related to number of past data points stored in memory. We choose other parameters as recommended in their open source implementations. The rule based model does not have any parameters. We get a total of N = 13 base models. For our method SEAD, we choose hyperpa- rameters 77 = 1, = 10 6 and π = Uniffrr form distribution across all experiments. We use tdigest (Dunning & Ertl, 2019) for streaming quantile computation for normalizing anomaly scores. We set the first 100 data points for warm starting the base models and SEAD, but not for evaluation, i.e., is 'cold start', We performed all experiments on a single c5. 2xlage AWS EC2 instance.
From Table 2, we see that SEAD obtains the best rank on average, outperforming about 60% of the base models and having the lowest variance. In the worst case, ranks 9th among all models. For all other base models, the rank in the worst case is >= 10, with most base models achieving the worst rank at least one dataset. This demonstrates that even on the worst dataset, the performance of SEAD is good, while all other algorithms have at-least one dataset in which their performance is bad.
We also compare with simple ensemble baselines that ag- gregate base detectors using (a) mean, (b) max or (c) min of the anomaly scores in Tables 3 and in Appendix A. We only compare against simple baselines because SEAD is the first unsupervised online ensembling technique. SEAD outperforms simple ensemble baselines, achieving a higher rank on average with the lowest variance. We later show that SEAD ++ achieves detection performance comparable to these simple ensemble baselines, while reducing the runtime by half.
4.2. Datasets considered
We evaluate the statistical significance of our results using Wilcoxon signed-rank test, concluding that our method is statistically different from all base models and ensemble baselines (in Appendix H). Using the same methodology for computing statistical significance, we show critical dif- ference diagram for SEAD, the best performing variants of each base method and simple ensemble baselines in Figure 2. We report hyperparameter ablations in Appendix D.
We perform experiments on 15 datasets, of which 11 are from the Outlier Detection DataSets (ODDS) (Rayana, 2016), 3 are from the USP Data Stream Repository (Souza et al., 2020) and one an internal telemetry dataset from a multiserver database cloud service. Our choice of datasets cover diverse applications and use-cases, including the INSECTS dataset (Souza et al., 2020) which has non- stationarity, a private telemetry dataset that has only 0.03% of its stream as anomalies and the IONOSPHERE dataset (Rayana, 2016; Keller et al., 2012) which has 36% of its stream as anomalies. Dataset characteristics are Table 10 in Appendix B. The number of data points range from 214 to 567, 000 and the number of features vary from 1 to 71. We report the rankings of the base models and SEAD in Table 2 based on the AP metric.
4.4. Comparison with offfine model selection method
Offline model selection methods cannot be used in the streaming setting, which requires processing each incoming data point in constant time. Nevertheless, for comparison, we relax the streaming requirement. We allow the offline meta learning method to look back at all of the data points seen so far, allowing (nn) computation instead of the O(1) computation time required in the streaming setting.
4.3. Effectiveness of SEAD
We compare with the offline model selection algorithms MetaOD (Zhao et al., 2021) and the method deseribed in (Goswami et al., 2023). We evaluate on an open source dataset on which neither of the offline algorithms has been trained on: Ozone dataset (Zhang et al., 2008) from the USP
We first observe that our datasets are non-trivial, i.e., sin- gle algorithm performs well across all of these datasets (see Table 2 and Appendix A). RRCF outperforms other meth- ods on the Letter, Glass and Telemetry datasets. xStream outperforms on the Ozone, Optdigits and INSECTS datasets.
6

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Table Comparison with hase models; Ranks real world datasets
INDEX
SPC
KRCE.O
RRCT_1 1
RECE
RRCF.3
XS.0
X5.1
XS
2
XS.3
IF.0
IF.1
IP-2
IP.3
SEAD
PENDIGITS
13
10
11
11
12
4
LETTER
6
3
11
12
10
JONOSPHERE
14
9
7
13
10
11
12
2
WBC
14
12
13
GLASS
11
13
12
10
VERTERRAL
13
12
1I
HETP
II
11
11
12
10
9
INSECTS ORADUAL IMBALANCED
INSECTS INCREMENTALIMMALAKCC
11
12
10
OZONE
10
8
AVERAGE RANK
7.53
8.27
7.67
6.87
7.60
9.60
5.87
5.40
5.53
6.20
6.27
6.27
5.47
5.07
VARIANCE
25.41
18.78
17.38
13.84
13.11
11.54
11:12
10.83
12.27
6.89
13.07
9.07
7.12
6.64
Table 3. Comparison with ensemble baselines: Ranks real world datasets
INDEX
MEAN
MAX
MIN
SEAD (OuRS)
PIMA
2
2
2
PENDIGITS
12
OPTDIGITS
13
IONOSPHERE
4
9
7
WBC
9
9
11
MAMMOGRAPHY
GLASS
VERTEBRAL
10
5
CARDIO
5
7
HTTP
4
12
INSECTS ANADD
INSECTS JACREMENTAL,,MBAJ...A
OZONE
7
TELEMETRY
AVERAGE RANK
5.73
5.80
6.60
5.07
VARIANCE
12.35
6.74
20.69
6.64
dataset repository (Souza et al., 2020). We compare APS and runtime vs SEAD Table SEAD slightly improves over MetaOD and (Goswami al., 2023) on detection per- formance, while providing 35x speedup in runtime. We describe the experimental details for adapting (Goswami et al., 2023)'s method to the online setting in Appendix G
Critical Difference Diagram
SEAD
XS.
Meen
Max
Min
RREF-2
SPC
4.5. Comparison with SEAD
5.07
3:40
3.00
6.60
6.87
7.53
We choose the number of sampled detectors K [N/2] 6. We report ranks based on AP scores and runtime in seconds for SEAD ++ in Table 4. Note that SEAD and the ensemble baselines have approximately the same runtime because they differ in computationally trivial operations like taking mean, min and max or a linear sum across all N base anomaly detector scores. SEAD ++ obtains AP score comparable to the best ensemble baselines computed using
Figure Critical difference diagram for SEAD, best perform- ing variants of each base method and simple ensemble baselines based on rank comparisons in Table and Table The values on the axis represent the average ranks. All shown methods are statistically different from each other. Appendix describes hyperparameter configurations for each base model.

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Table 4. Comparison with SEAD ++ (K = =6
RUNTIME (S)
AP RANK
DATASET
SEAD
SEAD++
SEAD
SEAD++
PIMA
817.58
318.45
2
PENDIGITS
3680.16
1924.70
7
6
LETTER
1209.85
570.29
4
5
OPTDIGITS
3140.09
1554.65
5
6
IONOSPHERE
332.99
164.82
6
6
WBC
353.20
170.91
9
MAMMOGRAPHY
5567.67
2877.58
GLASS
199.60
91.06
5
VERTEBRAT.
224.47
98.82
8
8
CARDIO
1334.02
648.94
5
10
HTTP
274102.52
116252.01
5
10
INSECTS GRADUAA_JMBALLNCEE)
24000.80
12536.13
3
INSECTS INCREMENTAL IMBALANCED
77740.12
38245.36
1
OZONE
1746.19
1002.05
TELEMETRY
2235.19
969.20
AVERAGE RANK
5.07
5.8
VARIANCE
6.64
8.74
Table 5. Comparison with offline model selection methads on the Ozone dataset
AVERAGE PRECISION
RUNTIME (S)
SEAD
(ZHAO ET AL., 2021)
(GosWAMI ET AL., 2023)
SEAD
(ZHAO ET AL., 2021)
(GosWAMI ET AL., 2023)
0.059
0.052
0.070
1746
173684
61543
the mean and max of the anomaly scores, while providing 2x speedup in runtime. Compared to SEAD it trades off detection performance with reduced runtime. We report APS scores for SEAD ++ Appendix A and comparisons with = N/44 and [3//4] in Appendix D.
IForestASD. This behavior results in fewer false positives.
4.7. Robustness to uninformative base detectors
An advantage of SEAD is that its comparison against base- lines like the mean aggregator improves with the addition of many random detectors. We perform an experiment where we double the number of base detectors to N = 26, adding 13 random detectors that each gives a random number as anomaly score to each input. We evaluate on the ODDS datasets, excluding the larger HTTP dataset because com- putational reasons. As mance of SEAD does not diminish lot while that of base- lines like mean, max and min deteriorates significantly. The reason being that SEAD quickly identifies the bad detectors and down-weighs them, while baselines like mean cannot do this. When we add random detectors, the performance of SEAD is much more distinguishable than that of baselines. We report APS metries in Appendix F.
4.6. Case study illustrating the evolution of weights over time the base-algorithms SEAD
Figure 3 shows an example where SEAD reduces the weight for misfiring detector on the Ionosphere dataset (Sig- illito et al., 1989; Keller al., 2012). The red lines mark the ground truth labeled anomalies. The top plot shows the anomaly scores for the two models having the high- est weight during this duration IForestASD and xStream, and the bottom plot shows the weights assigned by SEAD to these two models. During the anomalous period till in- dex 255, both IForestASD and xStream have similar scores. SEAD assigns higher weight xStream during this period. xStream outputs higher scores riod. The xStream scores on the benign data points are even higher than the xStream scores on the anomalous data points. Due to xStream misfiring on benign data points, SEAD downweights xStream and increases the weight for
see in Table 6 the perfor-
after the anomalous pe-
8

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
FooresAAD vs xStream Anomaly scores
it has been observed that selecting based on these unsu- pervised measures does not yield useful results (Ma et al., 2023).
1.0
0.8
5.2. Offline Model selection algorithms
0.4
The offline case when all data is available up-front has been studied in recent past. HYPER (Ding et al., 2024) uses hypernetwork to predict model parameters for deep models, however it is limited only to deep models. MetaOD (Zhao ct al., 2021) uses meta features to identify similar histori- cal labeled datasets, while ELECT (Zhao et al., 2022) uses performance-drivee similarity. All these algorithms need input a collection of labeled datasets similar to the test dataset, while SEAD also works in the absence of offline labeled datasets. (Goswami et al., 2023) while proposing state-of-the-ar offline model selection method that does not require labeled offline data, nevertheless needs to rank the anomaly scores for all the test data across all models making inefficient to implement in the streaming setting, Ensembling algorithms (Gao al., 2012) combine top-n outlier detection scores using outlier score normalization. (Rayana & Akoglu, 2014; 2016) use rank aggregation tech- niques for combining anomaly scores from base models. Other outlier ensembling methods use techniques like fea- ture bagging (Lazarevic & Kumar, 2005), subsampling data (Zimek et al., 2013) and improved normalization (Gao Tan, 2006; Kriegel et al., 2011).
0.2
0.0
260
300
Detector weights across time
4.0
3.5
3.0
2.5
2.0
1.5
240 260
280
300 320
Index
Figure 3. Evolution weights across time for a subset the Iono- sphere (Sigillito al., 1989; Keller al., 2012) dataset. SEAD re- duces false positives by down weighing xStream when it scores be- nign data points higher, and reassigning the weight to IForestASD.
Table Average performance across ODDS datasets for SEAD and ensemble baselines after adding random detectors
MEAN
MAX
MIN
SEAD (OuRS)
However, all of these offline methods use the entire testing data to choose the best models, i.e., take O(t) time to pro- duce an anomaly score for the /th data point. Thus these are not computationally feasible in the streaming setting (see also Section 4).
9.75%
22.3 1 %
22.04%
0.88%
5. Related Work
5.1. Anomaly detection algorithms
6. Conclusions
AD is widely studied both in the machine learning literature as well as in applied data-science literature (see surveys of (Schmidl et al., 2022; Paparrizos et al., 2022; Iglesias Vázquez et al., 2023; Ruff al., 2021)). Some AD algo- rithms inherently ensemble based in order to be robust to dataset properties. These methods however only build ensemble of
We propose SEAD the first unsupervised online model selection algorithm for anomaly detection. The main idea behind SEAD is to observe that the anomaly scores output by the candidate algorithms, can be used as an unsupervised proxy for its performance. SEAD ensembles anomaly scores using varying weights set by multiplicative weights update algorithm. We further optimized SEAD 's runtime by Thompson sampling that reduces run-time without signif- icant accuracy degradation. We see through comprehensive experiments the value and applicability of SEAD.
ensembles of specific class of models histogram density estimators (Pevný. 2016), autoencoders (Mirsky et al., 2018), half-space chains (Manzoor et al., 2018), decision trees (Liu et al., 2008; Guha et al., 2016) and subspace outlier detectors (Sathe Aggarwal, 2016b). On the contrary, SEAD is agnostic to the base-algorithms.
Our work leaves open regret guarantees on SEAD that holds even under non-stationarity. Further optimizing SEAD to leverage offline labeled datasets to improve online selection under non-stationarity is interesting future work.
Selecting models based on new unsupervised metries has been proposed in the literature- (Goix, 2016) proposes extension of Excess-Mast Masss and Mass- Vols-Volume based metrics to higher dimensions, while IREOS (Marques et al., 2020) measures how casily each outlier can be separated from other objects using maximum margin classifiers. However,
9

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Impact Statement
Ding, Z. and Fei, M. An anomaly detection approach based isolation forest algorithm for streaming data using sliding window. IFAC Proceedings Vol- umes, 46(20):12-17, 2013. ISSN 1474-6670. doi:
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
URL httpp://www.sceeneddrree.. com/ scdemce//articcleppp/////// /
References
3rd IFAC Conference on Intelligent Control and Automation Science ICONS 2013.
Aggarwal, C. C. and Sathe, S. Theoretical foundations and algorithms for outlier ensembles. SIGKDD Explor. Newsl., 17(1):24-47, September 2015, ISSN 1931-0145. URL https: : //doi.
Dunning, T. and Ertl, O. Computing extremely ac- curate quantiles using I-digests. arXiv preprint arXiv:1902.0402: 2019.
doi: 10. I (0.114//8833444288880000 org/ h0g/110.0 wtt...115/2288000 2830549.
Gao, J. and Tan, P.-n. Converting output scores from outlier detection algorithms into probability estimates. In Sixth International Conference on Data Mining (ICDM'06 pp. 212-221, 2006. doi: 10. 10910000....0006.44
Ahmad, S., Lavin, A., Purdy, S., and Agha, Z. Unsuper- vised real-time anomaly detection for streaming data.
Neurocompuling, 262:134-147, 2017. ISSN 0925-2312.
doi: http://bbboogg/...ooo....ummmmmp com/ URL wttpa////r..cee///////2222111111
Gao, J., Hu, W., Zhang, Z., and Wu, O. Unsupervised ensemble learning for mining top-n outliers. In Tan, P.- N., Chawla, S., Ho, C. K., and Bailey, J. (cds.), Advances Knowledge Discovery and Data Mining, pp. 418 418-330, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 978- 778-3-642-30217-6
Online Real-Time Learning Strategies for Data Streams.
AWS.
Using cloudwatch anomaly detection ama- zon cloudwatch, 2025. URI. https://docs.
aws amazon. com/ htt.///aoooooolaaacccco/ooooo///
hatest/nooittorin///looooooo html.
Anomaly_
Goix, N. How to evaluate the quality of unsupervised anomaly detection algorithms?, 2016. URL https: / //laxiiv.rrg/attee...... www./a///////ooooo//o./o///aaaee 111..01152... 01152.
Barreto, D. N., Silva, R., Mizaikoff, B., and da Sil- veira Petruci, J. F. Monitoring ozone using portable substrate-inten interrate hollow waveguide-buie based absorbance sensors in the ultraviolet range. ACS Measure- ment Science Au, 2(1):39-45, 2022. doi: 10.1021/
Goswami, M., Challu, C. I., Callot, L.., Minorics, L.,, and Kan, A. Unsupervised model selection for time series anomaly detection. The Eleventh International Confer- ence on Learning Representations, 2023. URL https: / //openreview.o net/ forum?id=gO0 _0A0ooPP_AANPP....
catnecasuuesciaalcc000022 httpstss: /tw.///o.oooo/ .org//or///ooo 10.1021 httpe//accmmssussssaaaaaaoaaaaae
Guha, S., Mishra, N., Roy, G., and Schrijvers, O. Robust random cut forest based anomaly detection on streams. In Balcan, M. F. and Weinberger, K. (eds.), Proceedings of The 33rd International Conference on Machine Learn- ing, volume 48 of Proceedings of Machine Learning Re- search, pp. 2712-2721 New York, New York, USA, 20- 22 Jun 2016. PMLR. URL ht pottpps://t httss////ccceiig..... mir.presse v48
Cesa-Bianchi, N. and Lugosi, G. Prediction, learning, and games. Cambridge university press, 2006.
Cesa-Bianchi, N., Freund, Y., Haussler, D., Helmbold, D. P., Schapire, R. E., and Warmuth, M. K. How to use expert advice. Journal of the ACM (JACM), 44(3):427-485, 1997.
/ guhal 1 6 .tmll.
Chandola, V., Banerjee, A., and Kumar, V. Anomaly detection: A survey, ACM Comput. Surv, 41(3), jul 2009. ISSN 0360-0300. doi: 10.1145/1541880. 1541882. URL attps://dol wtw.i.oogg1..1100.
Hagemann, T. and Katsarou, K. A systematic review on anomaly detection for cloud computing environments. In Proceedings of the 2020 3rd Artificial Intelligence and Cloud Computing Conference, pp. 83-96, 2020.
1541880. 1541882.
Ding, X., Zhao, Y., and Akoglu, L. Fast unsupervised deep outlier model selection with hypernetworks. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp.
Han, S., Hu, X., Huang, H., Jiang, M., and Zhao, Y. AD- Bench: Anomaly detection benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https: / //openrevien ew, net/ forum? f SFQ92000
585-596, New York. NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704901.. doi: 10. 1145/6637528.6670003 URL attpss:/doi.... 10. 10.1145/3637528.1 3672003.
10

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Iglesias Vázquez, F,, Hartl, A., Zseby, T., and Zimek, A. Anomaly detection in streaming data: A com- parison and evaluation study. Expert Systems with Applications, 233:120994, 2023. ISSN 0957-4174. doi: https//doioon//000000f/eeee22222 URL
Explor Newsl., 25()):19-35, July 2023. ISSN 1931- 0145. doi: 10. 10.1145/36067443366777, URL https: 3606277.
Manzoor, E., Lamba, H., and Akoglu, L. xstream: Outlier detection in feature-evolving data streams. In Proceed- ings of the 24th ACM SIGKDD International Confer- ence on Knowledge Discovery DDta Mining, KDD '18, pp. 1963-1972, New York, NY, USA, 2018. Associa- tion for Computing Machinery. ISBN 9781450355520. doi: 10.1 1444532211819922200011 URL https: /doi. org / 1tt.:/...111/11111 0111/////2211111.22200022 3220107.
Islam, M. S., Pourmajidi, w., Zhang, L., Steinbacher, J.,
Erwin, T., and Miranskyy, A. Anomaly detection in a large-scale cloud platform. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Soft- ware Engineering Practice (ICSE-SEIP), pp. 150-159. IEEE, 2021.
Marques, H. O., Campello, R. G. B., Sander, J., and Zimek, A. Internal evaluation of unsupervised outlier detection. ACM Trans. Knowl. Discov. Data, 14(4), June 2020. ISSN 1556-4681. doi: 10. (0.1145/3394053. URL https: / /woi.dii..///// orrg 11 00..0..111/333945500
Keller, F., Muller, E., and Bohm, K. Hics: High contrast subspaces for density-basee -based outlier ranking. In Proceed- ings of the 2012 IEEE 28th International Conference on Data Engineering, ICDE 12, pp. 1037-1048, USA, 2012. IEEE Computer Society, ISBN 9780769547473. doi: 10.1 10.1109/CDE.22012.88 URL https: /ttt::://doi.rgg//// 10.1109/ICDE.20
Micenková, B., McWilliams, B., and Assent, I. Learning outlier ensembles: The best of both worlds-supervised and unsupervised. 2014. ACM SIGKDD 2014 Workshop ODD2: Conference date: 24-08-2014 Through 24-08- 2014.
Kriegel, H.PP., Kröger, P., Schubert, E., and Zimek, A. Interpreting and unifying outlier scores. In SDM, 2011. URL http:::/apt..emmttticcchh semanticschooll.r otr/Co::psID:11222722
Mirsky, Y., Doitshman, T., Elovici, Y., and Shabtai, A. Kitsune: An ensemble of autoencoders for on- line network intrusion detection. In 25th Annual Network and Distributed System Security Sympo- sium, NDSS 2018, San Diego California, USA, February 18-21, 2018. The Internet Society, 2018. URL https: http:://wwww... nass symposiupmisssss org/ WP content/uppooad//2001 /ttp//20001///////1111..1111 03A-3 1ar333MMisyy____eee pdf.
Lazarevic A. and Kumar, V. Feature bagging for outlier detection. In Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, KDD '05, pp. 157-166, New York, NY, USA, 2005. Association for Comput- ing Machinery. ISBN 159593135X. doi: 10.1145/ 1081870. 1081811111111 URL https://do..org/// 1145/1081870 1081891.
Paparrizos, J., Kang, Y., Boniol, P., Tsay, R. S., Palpanas, T., and Franklin, M. J. Tsb-uad: an end-to-end bench- mark suite for univariate time-series anomaly detection. Proceedings of the VLDB Endowment, 15(8):1697-1711, 2022.
Lazarevic, A., Ertoz, L., Kumar, V., Ozgur, A., and Sri- vastava, J. A comparative study of anomaly detection schemes in network intrusion detection. In Proceedings of the 2003 SIAM international conference on data mining, pp. 25-36. SIAM, 2003.
Pevný, T. Loda: Lightweight on-line detector of anomalies. Machine Learning, 102:275-304, 2016. URL atttps://ags https://ap..eeenaiiicchoo. semanticscholar. org/CorpusID:174555
Leveni, F., Cassales, G. W., Pfahringer, B., Bifet, A., and Boracchi, G. Online isolation forest. In Proceedings of the 41st International Conference on Machine Learning, ICML'24, 24. JMLR.org. 2024.
Rayana, S. ODDS library, 2016. URL https:t/////eett cs. .s.stonybrook. edu.
Liu, F. T., Ting, K. M., and Zhou, Z.H. Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining, pp. 413 422, 2008. doi: N0.110//CDM..008..1
Rayana, S. and Akoglu, L. An ensemble approach for event detection and characterization in dynamic graphs. 2014. ACM SIGKDD ODD Workshop. 2014.
Ma, M. Q., Zhao. Y., Zhang, X., and Akoglu, L. The need for unsupervised outlier model selection: A review and evaluation of internal evaluation strategies. SIGKDD
Rayana, S. and Akoglu, L. Less is more: Building selective anomaly ensembles. ACM Trans. Knowl. Discov: Data 10 (4), May 2016. ISSN 1556-4681. doi: 10. 155/2890008 URL htw.thtts///ete //op:::://d//i/o wrt.o/...1////.......0

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Rayana, S., Zhong, W. and Akoglu, L. Sequen- tial ensemble learning for outlier detection: A bias- variance perspective. 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 1167-1172 2016. URL https: http:/api...o/ Ptg/CozpusID:100720 set..mimcmtiicccolee....
Souza, V. M. A., Reis, D. M., Maletzke, A. G., and Batista, G. E. A. P. A. Challenges in benchmarking stream learning algorithms with real-world data. Data Mining and Knowledge Discovery, 34:1805-1858, 2020. doi:
Wu, R. and Keogh, E. J. Current time series anomaly detection benchmarks are flawed and are cre- ating the illusion of progress. IEEE Transactions on Knowledge and Data Engineering, 35:2421-2429, 2020. URL https: / /ppi. semanticschooeeet. org/ hegg/CorpusI:::0 :2:
Ruff, L.,, Kauffmann, J. R., Vandermeulen, R. A., Montavon, G. Samek, W. Kloft, M., Dietterich, T. G., and Müller, K.-R. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 10965):56-775, 2021.
Russo, D. J., Van Roy, B., Kazerouni, A., Oshand, I., Wen, Z., et al. A tutorial on thompson sampling. Foundations and Trends Machine Learning, 111):119,, 2018.
Xu, H., Chen, W., Zhao, N., Li, Z., Bu, J., Li, Z., Liu, Y., Zhao, Y., Pei, D., Feng, Y., Chen, J., Wang, Z., and Qiao, H. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In Proceedings of the 2018 World Wide Web Conference, WWW '1, pp. 187-196, Republic and Canton of Geneva, CHE. 2018a. International World Wide Web Conferences Steering Committee. ISBN 9781450356398, doi: 10. 1145/3178876.11899999 URL https://doi.....
Sankararaman, A., Narayanaswamy, B., Singh, V. Y., and Song, Z. Fitness:(fine tune on new and similar samples) to detect anomalies in streams with drift and outliers. In International Conference on Machine Learning, pp- 19153-19177. PMLR, 2022.
Sathe, S. and Aggarwal, C. LODES: Local Den- sity Meets Spectral Outlier Detection, pp. 171- 179. 2016a. doi: 10.1137/197816611944488888 URL https: siam. orgp////////////.ooooooo 1137/1.9781611972
1 10.
Xu, H., Chen, W., Zhao, N., Li, Z., Bu, J., Li, Z., Liu, Y., Zhao, Y., Pei, D., Feng, Y., Chen, J., Wang. Z., and Qiao, H. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In Proceedings of the 2018 World Wide Web Conference, WWW '18. pp. 187-196, Republic and Canton of Geneva, CHE, 2018b. International World Wide Web Conferences Steering Committee. ISBN 9781450356398. doi: 10. 1145/317887633189999 URL httpps://oo.... 10. 0..114//317876 3185995.
Sathe, S. and Aggarwal, C. Subspace outlier detection in linear time with randomized hashing. In 2016 IEFE 16th International Conference Data Mining (ICDM), pp. 459-468, 2016b, doi: 10.1109/CDM.2001..005
Schmidl, S., Wenig, P., and Papenbrock, T. Anomaly de- tection in time series: a comprehensive evaluation. Proc. VLDBB Endow. 1509):179-17797 May 2022. ISSN 2150- 8097. doi: 10. 10.147788333385988333888888 3538602. //doi wwtp:///........0e0.1211
Yilmaz, S. F. and Kozat, S. S. Pysad: A streaming anomaly detection framework in python. arXiv preprint MiXiv:209.02257. 2020.
URL https:
Shewhart, W. Economic Control of Quality of Manufac- lured Product. Books from Bell Telephone Laboratories. D. Van Nostrand Company, Incorporated, 1931. ISBN 9780598557667. URL https: httts:////ockks com/ bwt.booksssi----ttaAAaaaa /wt.osooooo/ooooo.....o......... google.
Zhang, Kun, Fan, Wei, Yuan, and Xiaoling. Ozone Level Detection. UCI Machine Learning Repository, 2008.
DOI: https://bb..rr/10.2445///5
Zhang, Y., Guan, Z., Qian, H., Xu, L.., Liu, H., Wen, Q., Sun, L., Jiang, J., Fan, L.,, and Ke, M. Cloudrca: root cause analysis framework for cloud computing platforms. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 4373- 4382, 2021.
Sigillito, V., Wing, S., Hutton, L., and Baker, K. Iono- sphere. UCI Machine Learning Repository, 1989. DOI: https:/do..og/10/2444////3
Singh, V., Song, Z., Narayanaswamy, B. M., Vaidya, K. E.,, and Kraska, T. Vista: Machine learning based database performance troubleshooting framework in amazon rds. In Proceedings of the 2024 ACM Symposium on Cloud Computing. pp. 83-98, 2024.
Zhao, Y., Rossi, R., and Akoglu, Automatic unsupervised outlier model selection. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4489-4502. Curran Associates, Inc.,
Slate, D. Letter Recognition. UCI Machine Learning Repos- itory, 1991. DOL: http:://bb.rrrgl1.33333//2222
2021. URL https: /ttp///peoo.ss..../////// neurips.
12

---

SEAD Unsupervised Ensemble of Streaming Anomaly Detectors
cc/pppee_iiileeepppp///0/////// 2288 pdf.
Zimek, A., Gaudet, M., Campello, R. J., and Sander, J. Subsampling for efficient and effective unsuper- vised outlier detection ensembles. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '13, pp- 428-436, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450321747. doi: 10. 1145/448755524876666 URL https: /dg...rg// 10. 10. 10.1445/244575.2487777
Zhao, Y., Zhang, S., and Akoglu, L. Toward Unsupervised Outlier Model Selection In 2022 JEEE International Conference on Data Mining (ICDM), pp. 773-782, Los
Alamitos, CA. USA, December 2022. IEEE Computer Society. doi: 10. 10.1100/CCDM544442200220000000 https: http::///ooo...eeccoomorrrrreees 10. 00.110//rE00546442202..0000
URL
13

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
A. Average Precision Metrics
report Average Precision scores for the base models and SEAD in Table 7, ensemble baselines Table and SEAD ++ with 6 in Table 9.
Table 7. Comparison with base models: Average precision scores on real world datasets
DATANRT
k41+.J1
(OURE))
0.430
0:-51
0.695
0.501
0.083
RESDAGES
B111
114771
0.280
0.275
0.089
[.433
11,91
1.113
1.083
0.264
0.202
0.254
0.252
[002]
0.275
0.078
0.343
0:457
0.455
6.585
0.530
0.305
0.58
0.576
0.313
0.605
0.670
0.678
0.667
0.535
0.136
0.500
0.788
0716
0.691
CLASS
B.02
0.180
0.135
0.173
0.145
0403
0.115
0.083
0.313
0.61
0.104
0.00
8133
1.60
0.131
0.251
0.353
0153
BITT
B:423
0.054
0.204
0.004
0.003
0.003
0.019
0.073
0,913
0007
0,024
0.611
INSECTS GRADUAL DIBALANCED
0.11
0.152
0.:41
0.155
0.151
0.140
0.185
0.165
0.178
0.352
0.55
0.154
0.154
0.183
0.81
0.081
0.075
8800
0.087
0.093
TILENETRY
1090
1,000
1.000
8.353
1.000
0.025
1,000
0.200
0.300
0,005
0929
0,004
0,067
1,000
Table 8. Comparison with ensemble baselines: Average precision scores
DATASET
MEAN
MAX
MIN
SEAD (OURS)
PIMA
0.526
0.509
0.522
0.519
PENDIGITS
0.133
0.124
0.037
0.091
LETTER
0.055
0.060
0.058
0,084
OPTDIGITS
0.041
0.075
0.022
0.084
IONOSPHERF
0.624
0.423
0.497
0.555
WBC
0.464
0.362
0.319
0.486
MAMMOGRAPHY
0.117
0.108
0.076
0.118
GLASS
0.090
0.088
0.096
0.115
VERTEBRAL
0.149
0.174
0.186
0.154
CARDIO
0.237
0.219
0.180
0.505
HTTP
0.005
0.059
0.002
0.035
INSECTS LMBLLACD
0.230
0.162
0.219
0.161
INSECTS INCREMENTAL AMBALANNEE
0.293
0.188
0.268
0.189
OZONE
0.086
0.067
0.112
0.059
TELEMETRY
1.000
0.022
1,000
1.000
B. Dataset descriptions
We report the number of data points, number of dimensions and the anomaly percentage for each dataset in Table 10. For the two INSECTS datasets, we use the most populous class 'Aedes aegypti (female)' as the benign class and the least populous class 'Culex quinquefasciatus (male)' as the anomalous class.
C. Hyperparameter configurations for base models
We report hyperparameter configurations for all base models in Table 11.
D. A Alations
We compare against the choices of 1 = [1, 01,0 and [10 20- 10-4, 4, 10 10100-9 1100-)(e for SEAD Table 13. We report ablations for SEAD ++ in Table 12, varying the number of sampled detectors K.
14

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Table 9, Average Precision scores for SEAD+ (K'
DATASET
SEAD++
PIMA
0.537
PENDIGITS
0.092
LETTER
0.078
OPTDIGITS
0.076
IONOSPHERE
0.535
WBC
0.536
MAMMOGRAPHY
0.091
GLASS
0.108
VERTEBRAL
0.155
CARDIO
0.131
HTTP
0.004
INSECTS GRADUNEL
0.168
INSECTS INCREMENTAL. IMBALANCED
0.183
OZONE
0.076
TELEMETRY
1.000
Table 10. Dataset descriptions for the ODDS (Rayana, 2016), USP (Souza al., 2020) and internal datasets
DATASET
#POINTS
#DIM.
#OUTLIERS (%)
PIMA (ODDS)
768
8
268 (35%)
PENDIGITS (ODDS)
6870
16
156 (2.27%)
LETTER (ODDS)
1600
32
100 (6.25%)
OPTDIGITS (ODDS)
5216
64
150 (3%)
IONOSPHERE (ODDS)
351
33
126 (36%)
WBC (ODDS)
278
30
(5.6%)
MAMMOGRAPHY (ODDS)
11183
260 (2.32%)
GLASS (ODDS)
214
9
(4.2%)
VERTEBRAL (ODDS)
240
6
30 (12.5%)
CARDIO (ODDS)
1831
21
176 (9.6%)
HTTP (ODDS)
567479
3
2211 (0.4%)
INSECTS IRADUALLIMBALANCH (USP)
47251
33
4867 (10.3%)
INSECTS INCREMENTALIMBBLLNCEE (USP)
148048
33
13331 (9%)
OZONE (USP)
2534
71
160 (12.5%)
TELEMETRY (PRIVATE)
3726
1 (0.03%)
Table 11. Hyperparameter configurations for the base models. Since we primarily focus on the streaming setting in this work, we only vary hyperparameters related to number of past data points stored in memory.
MODEL
HYPERPARAMETERS
RRCF
NUM TREES 4, SHINGLE SIZR=4, TREE SIZF=|32,, 64, 128, 256]
IFORESTASD
X NONE, WINDOW.SIZE=1255. 512, 1024, 2048]
XSTREAM
NUM COMPONENTS = 100, 100. DEPTH=25, WINDOW SIZE=[8, 16, 25, 32]
15

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Table 12. Ablations for SEAD with different values for the number of sampled detectors K. The total number of base detectors
N 13.
RUNTIME (S)
AP ScORE
DATASET
K=3
K=6
K=9
K=3
K=6
K=9
PIMA
143.347
318.449
589.054
0.455
0.537
0.526
PENDIGITS
1456.997
1924.700
2772.344
0.189
0.092
0.117
LETTER
440.250
570.286
916.786
0.073
0.078
0.133
OPTDIGITS
1552.878
1554.646
2299.575
0.043
0.076
0.142
IONOSPHERE
72.610
164.815
228.618
0.588
0.535
0.535
WBC
82.772
170.914
246.968
0.480
0.536
0.465
MAMMOGRAPHY
1955.149
2877.584
4006.977
0.086
0.091
0.098
GLASS
51.274
91.061
141.936
0.100
0.108
0.110
VERTEBRAL
65.503
98.824
151.341
0.195
0.155
0.182
CARDIO
351.500
648.944
941.799
0.252
0.131
0.190
Table 13. Performance across different hyperparameter combinations
compraaaraatterrocmmbiinniooo (17, A)
0.1
1) = 0.01
Dataset
A 10-6
10
10-2 2
10-6
10
10
10
10-4
10-2
pima
0.519
0.519
0.520
0.531
0.531
0.531
0.527
0.527
0.527
pendigits
0.091
0.255
0.188
0.164
0.164
0.159
0.147
0.147
0.147
letter
0.084
0.083
0.076
0.066
0.066
0.064
0.056
0.056
0.056
optdigits
0.084
0.031
0.030
0.041
0.041
0.039
0.042
0.042
0.042
ionosphere
0.555
0.544
0.557
0.568
0.568
0.568
0.570
0.570
0.570
wbe
0.486
0.483
0.471
0.487
0.487
0.487
0.488
0.488
0.488
mammography
0.118
0.128
0.125
0.118
0.118
0.120
0.123
0.123
0.123
glass
0.115
0.129
0.124
0.093
0.093
0.093
0.094
0.094
0.094
vertebral
0.154
0.158
0.158
0.152
0.152
0.152
0.154
0.154
0.154
cardio 0.505
0.508
0.173
0.240
0,240
0.231
0.209
0.209
0.209
E. Example of evolution of weights
We show two examples where the best performing methods are different, and SEAD is able to select the best performing methods in each case. Figures and show the evolution of weights for two datasets. In Figure 4, on a variant of the INSECTS (Souza et al., 2020) dataset, SEAD gives highest weights to xStream models which obtain the best average precision. On the Cardio dataset (Aggarwal & Sathe, 2015; Sathe & Aggarwal, 2016a) in Figure 5, SEAD has the highest weights for the IForestASD models which obtain the best detection performance.
F. APS metrics after adding random detectors
We report Average Precision scores after adding random detectors for SEAD and ensemble baselines in Table 14. Adding random detectors demonstrates SEAD's robustness to uninformative base detectors SEAD's performance drop is lower than the baselines shown in Table 6.
G. Adapting (Goswami et al., 2023)'s method to the online setting for comparison in Table 5
The main issue with (Goswami et al., 2023)'s algorithm is that it is designed in the offline case where all the train data is available up-front, a single model is selected and applied as batch to the entire inference dataset. In order to adapt it to the online setting, we ran inferences in batches of 50 data points, where we retrained and applied it to a contiguous batch of 50 data points. For each re-training, we use the entire data-stream seen thus far and re-train from scratch each time.
16

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Foretard forettnd
foreteed
x2rean_3
8
vdream]] 3
Weights
met
2
20000
40000
60000
80000
100000
120000
140000
Index
Figure 4. Evolution of weights across time for variant of the INSECTS dataset, which is known to exhibit strong non-stationarity (Souza al., 2020). SEAD performance as indicated Table weight to the second best performing base model.
evolution of weights above shows SEAD having the highest
Porestand Hanestiand Forestadd rstreamm
Weights
250
500
750
1000
1250
1500
1750
Index
Figure 5. Evolution weights across time for the Cardio dataset (Aggarwal & Sathe, 2015: Sathe & Aggarwal, 2016a). SEAD selects the top two best performing models.
17

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Table 14. Comparison with ensemble baselines after adding random detectors; the last shows average performance drop across
datasets after adding random detectors
DATASET
MEAN
MAX
MIN
SEAD (OuRS)
PIMA
0.513
0.399
0.358
0.520
PENDIGITS
0.099
0.051
0.034
0.246
LETTER
0,056
0.055
0.073
0.088
OPTDIGITS
0.045
0.052
0.024
0.048
IONOSPHERE
0,493
0.323
0.385
0,491
WBC
0.436
0.176
0.121
0.292
MAMMOGRAPHY
0.081
0.053
0.029
0.060
GLASS
0.082
0.127
0.070
0.078
VERTEBRAL
0.154
0.193
0.192
0.137
CARDIO
0.195
0.147
0.098
0.518
AVG PERFORMANCE DROP
9.75%
22.31%
22.04%
0.88% %
Operating this way significantly increased run-time due to repeated training on the stream. Repeated re-training is needed since we are operating on data-streams with non-stationaritics. Adapting the algorithm presented in (Goswami et al., 2023) to be online where some computations can re-used (for example the re-trainings need not be from scratch) is non-trivial and requires further research that is beyond the scope of this paper. The 35r increase in run-time was the fundamental reason we did not evaluate the algorithm from (Goswami al., 2023) (and even MetaOD (Zhao et al., 2021)) on all the datasets.
H. Statistical significance tests to analyze performance difference between SEAD and base models
We perform Wilcoxon signed-rank test to evaluate the statistical significance our results in Table 15. The datasets that we test on are diverse and no single method performs well on all datasets. Hence, each method has a high variance in Average Precision Score (APS) across the 15 datasets that have tested on. The high variance makes statistical significance tests unreliable
To overcome this issue, we split each dataset into chunks of 50 contiguous data points. This is also relevant for the online learning paradigm where we want to evaluate the model continously over time. Since APS is not defined for chunks having all data points labeled as non anomalous, we only consider chunks which have at least one anomalous label. Using this splitting mechanism gives us 3,282 chunks across all datasets. We report p values for the Wilcoxon signed-rank test using the APS scores on the 3,282 chunks. Using a threshold 0.01, we conclude that our method is statistically different from all base models and ensemble baselines.
18

---

SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
Table 15. Statistical significance of SEAD performance compared base detectors and baseline methods
MODEL
WILCOXON P- P-VALUE
SPC
6.93 X -58
258
RRCF_O
4.26 10 10-68
RRCF_1
1.47 10-43
RRCF 2
3.16 x 10
RRCF_3
7.43 x 10 10-29
XSTREAM_0
2.09 x 10 10-95
XSTREAM_I
3.53 x 10-12
XSTREAM 2
1.80 x 10-4
XSTREAM 3
1.60 x 10-3
IFORESTASD
1.88 10 10
IFORESTASD 1
8.08 10-63
IFORESTASD_2
3.48 X 10 10-57
IFORESTASD 3
2.94 X 10
MEAN
6.41 X 10-171
MAX
6.48 X 10-5
MIN
6.17 10-11
19


Reviews:
[{"review_id": "IdoBXjRrRG", "reviewer": "Reviewer_xWBi", "summary": "The paper introduces SEAD (Streaming Ensemble of Anomaly Detectors), an unsupervised, online model selection algorithm for anomaly detection in streaming data, where labels are unavailable, and data distributions change over time. SEAD dynamically assigns weights to multiple anomaly detection models using Multiplicative Weights Update (MWU), ensuring that the best-performing model is prioritized without needing ground truth labels. The method operates in constant time per data point (O(1)), is model-agnostic, and adapts to concept drift by adjusting model weights as the data evolves. SEAD++ further optimizes runtime by sampling a subset of models per step, making SEAD a practical and scalable solution for real-time anomaly detection in streaming environments.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable questions and comments. Our responses to the raised points are outlined below.\n\n\n> Limitations of SEAD: The performance can only be as good as the best detector\n\n\nSEAD is the first online, unsupervised model-selection algorithm for anomaly detection (AD). The most pressing issues left open in the literature is the choice of which model to use when. The vast literature on AD suggests that there are good models for most situations one encounters in practice. Nevertheless, in practical settings, the choice of which model to choose when is hard to do. This challenge is exacerbated in the online unsupervised setting where there are no ground-truth labels and the best model can change over time. \n\n\n> On comparison with simple baselines like mean aggregator\n\n\nWe thank the reviewer for this point. One advantage of SEAD is that its comparison against baselines like the mean aggregator improves with the addition of many random detectors. To complement the results in the paper, we performed one more experiment where we added 13 random detectors that give a random number as anomaly score to each input. As we can see the performance of SEAD does not diminish a lot while that of baselines like mean, max and min deterioate significantly. The reason being that SEAD quickly identifies the bad detectors and down-weighs them, while baselines like mean cannot do this. \n\nIn particular, we will add this table showing that as more random detectors are added, the performance of SEAD is much more distinguishable than that of baselines. \n\n\n**dataset**\t| sead\t| mean\t| max\t| min\n---- | --- | --- | ---- | ----\npima |\t**0.52**\t| 0.513| 0.399\t| 0.358\npendigits\t| **0.246**\t| 0.099\t| 0.051\t| 0.034\nletter\t| **0.088**\t| 0.056\t| 0.055\t| 0.073\noptdigits\t| **0.048**\t| 0.045\t| 0.052\t| 0.024\nionosphere\t| 0.491\t| **0.493**\t| 0.323 | \t0.385\nwbc\t|0.292\t| **0.436**\t| 0.176\t| 0.121 \nmammography\t|0.06\t|**0.081**\t|0.053\t|0.029\nglass\t|0.078\t|0.082\t|**0.127**\t|0.07\nvertebral\t|0.137\t|0.154\t|**0.193**\t|0.192\ncardio\t|**0.518**\t|0.195\t|0.147\t|0.098", "claims_and_evidence": "The claims in the paper are largely well-supported by both theoretical reasoning and experimental results. SEAD’s ability to perform unsupervised model selection without labeled data is convincingly demonstrated through its weighting mechanism, which dynamically adjusts model importance based on anomaly scores. Its constant time complexity per data point (O(1)) is justified by its efficient online update strategy, contrasting with slower offline approaches like meta-learning.\nHowever, some claims could be refined. While SEAD performs consistently well, it does not always outperform all methods across every dataset, making it more accurate to say it ranks among the top-performing models rather than claiming universal superiority. Additionally, SEAD assumes that at least one base model is effective, meaning it may struggle if all base models are weak, a limitation that is not fully addressed.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem of unsupervised anomaly detection in streaming environments, as they align with the challenges of real-time detection, model selection without labels, and handling non-stationary data. The use of a diverse set of anomaly detection algorithms as base models ensures that SEAD’s model selection approach is tested across different detection paradigms. The reliance on anomaly score weighting instead of labeled data makes sense in real-world applications where obtaining labeled anomalies is infeasible.\nThe choice of evaluation datasets is also relevant, as the authors include 15 publicly available datasets that cover different types of anomalies and distributions, ensuring that SEAD is tested in diverse scenarios. The Averaged Precision Score (APS) metric is an appropriate choice for evaluating anomaly detection performance, as it accounts for the highly imbalanced nature of anomaly detection tasks. However, the evaluation could be expanded by testing SEAD’s performance in high-dimensional datasets or comparing it against deep learning-based anomaly detectors, which are increasingly used in large-scale anomaly detection problems.", "theoretical_claims": "The paper presents theoretical guarantees for SEAD, particularly in the form of a regret bound. This theoretical result aims to justify that SEAD's selection strategy will be competitive as the performance of the best anomaly detector in hindsight, even without labeled data.", "experimental_designs_or_analyses": "The experimental design in the paper appears well-structured, with evaluations conducted on 15 public datasets, covering a diverse range of anomaly detection scenarios. One aspect that could be further scrutinized is the robustness of the experimental setup across datasets with significantly different anomaly distributions. The paper does not provide a detailed breakdown of how SEAD performs on datasets with varying anomaly rates, which could impact its effectiveness.", "relation_to_broader_scientific_literature": "The key contributions of the paper are well-situated within the broader anomaly detection and online model selection literature, but their novelty and impact appear somewhat incremental rather than groundbreaking. The idea of leveraging multiple anomaly detection models in a streaming setting is an interesting direction, aligning with prior work on ensemble-based anomaly detection and meta-learning for model selection. However, the main theoretical foundation of the method—Multiplicative Weights Update (MWU)—is not novel and has been extensively studied in online learning literature. While the paper applies MWU in a new context by using it for online model selection in anomaly detection, the underlying theory itself is borrowed from well-established frameworks.\nFrom an empirical standpoint, SEAD demonstrates competitive but not significantly superior performance compared to existing baselines. The results indicate that while SEAD adapts dynamically to different data distributions, its overall detection accuracy does not show a substantial improvement over simpler ensemble baselines such as mean and max aggregation of anomaly scores. This suggests that while the method is effective, its contribution in terms of practical anomaly detection performance is not dramatically beyond existing approaches.\nFurthermore, while the experimental validation is comprehensive in terms of the number of datasets (15 public datasets), the selection is somewhat limited in diversity, focusing primarily on tabular datasets. The inclusion of vision-based anomaly detection benchmarks or datasets with high-dimensional, structured data (e.g., image or sensor data) could have strengthened the paper’s practical contributions by demonstrating the generalizability of SEAD beyond classical tabular anomaly detection tasks. Expanding the evaluation scope would provide stronger evidence of SEAD’s applicability in real-world, high-dimensional anomaly detection problems.\nIn summary, while SEAD presents an interesting adaptation of MWU to streaming anomaly detection, its contribution is somewhat incremental in both theoretical and empirical dimensions. The method does not introduce a fundamentally new learning principle, and its empirical improvements over baselines are relatively modest. A broader experimental validation across diverse domains, such as vision-based anomaly detection, would have further enhanced its practical significance.", "essential_references_not_discussed": "While there may be newer or alternative approaches to anomaly detection ensembles that are not explicitly cited, none appear to be critical omissions that would significantly alter the context or positioning of SEAD’s contributions.", "other_strengths_and_weaknesses": "While the empirical results are comprehensive, a more detailed discussion of failure cases or performance variability across datasets would strengthen the argument for SEAD’s robustness.", "other_comments_or_suggestions": "The overall presentation and completeness of the paper could be improved. In particular, Table 2 has text that is too small, making it difficult to read, while Tables 4 and 5 have comparatively larger text, creating an inconsistency in formatting. Additionally, on page 8, there are noticeable empty spaces, which could have been utilized more effectively to improve readability and layout balance. A more efficient arrangement of tables and text could enhance the paper’s clarity and visual coherence, making it easier for readers to follow the results and comparisons.", "questions_for_authors": "There are no critical questions for the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZIwmpKhaBA", "reviewer": "Reviewer_dMSV", "summary": "This paper proposes streaming ensemble of anomaly detectors, a model selection algorithm for streaming, unsupervised AD. The key insight that SEAD leverages is that anomalies by definition are ‘rare’, which SEAD uses to work in a fully unsupervised fashion. SEAD sets the weights for the individual models and chooses the weight using the classical multiplicative weights update (MWU) to predict with expert advice. Experiments verify the idea.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> How is SEAD adaptive to distribution changes\n\n\nSEAD updates the weigths of the base detectors using the multiplicative weight updates (MWU). In the learning theory literature, it has been established that when the parameters of learning rate and regularization strength are appropriately chosen, the MWU algorithm is adaptive to distribution shifts. See the Cesa-Bianchi et al., 1997 in the submission. \n\nEmpirically, we do show that with the proposed hyper-parameters, SEAD works in the presence of different types of concept drifts. We do this by evaluating SEAD on two types of the large-scale INSECTS dataset —  one having gradual concept drift and the other sudden shifts. \n\n\n> SEAD++ still needs to maintain and adjust weights between all detectors.”\n\n\nThe main reduction in SEAD++ is in not needing to do forward-pass or inference and back-ward pass or gradient calculation on half of the AD models. For many big models, these forward and backward passes are the main computational bottlenecks. While SEAD++ does require maintaining weights for each base detector, the updates and storage for the weights are trivial and independent of the size of the AD model. In this sense, we do not view maintaining separate weights for each base detector as computationally demanding. \n\n\n> Why use Average-Prceision as the metric and not other metrics like PR-AUC\n\n\nBoth the AP score used in the paper and the PR-AUC are two different methods to compute is a the area under the precision-recall curve. This metric of area under the precision-recall curve is a widely accepted metric for performance in imbalanced binary classification such as Anomaly detection. We use the particular implementation APS reported in the paper instead of the PR-AUC since the trapezoidal approximation to compute the integral for the area under the curve is more faithful for downstream performance compared to the PR-AUC implementation in sklearn (\n\nhttps://towardsdatascience.com/the-wrong-and-right-way-to-approximate-area-under-precision-recall-curve-auprc-8fd9ca409064/ \n\nWhile there are other metrics such as best F1, we did not use them as the best F1 only shows the performance at the optimally chosen threshold value and lacks interpretability, and sensitive to the threshold tuning method used to choose the threshold.  \n\n\n> On comparison against Goswami et.al., ICLR 2023 paper \n\n\nWe thank the reviewers for this oversight. We will add this to the related work section as the state-of-art in offline unsupervised model selection and add the following empirical result to Table 5 as well. \n\nBy using implementation from https://github.com/mononitogoswami/tsad-model-selection, we will add two extra columns for the method of Goswami et.al. in Table 5 in the main paper with APS of 0.07, and run-time of 61543 seconds. Thus, the updated Table 5 will read \n\nDataset| SEAD Average Precision | SEAD Runtime | MetaOD Average Precision | MetaOD Runtime | Goswami et.al. Average Precision | Goswami et.al. runtime\n--- | ----- | ---- | ----- | ----- | ------ | ----- \nOzone | 0.059 | 1746 | 0.052 | 173684 | 0.07 | 61543\n\n\n\nIn summary, our result here shows that while the method of Goswami et.al. results in similar accuracy as SEAD, its run-time is 40x slower! \n\n\n> Conceptual differences with Goswami et.al.‘s ICLR 2023 paper \n\n\nThe main issue with Goswami et.al’s algorithm is that it is designed in the offline case where all the train data is available up-front, a single model is selected and applied as a batch to the entire inference dataset. In order to adapt it to the online setting, we ran inferences in batches of 50 data-points, where we retrained and applied it to a contiguous batch of 50 datapoints. For each re-training, we use the entire data-stream seen thus far and re-train from scratch each time. \n\nOperating this way significantly increased run-time due to repeated training on the stream. Repeated re-training is needed since we are operating on data-streams with non-stationarities. \nAdapting the algorithm presented in Goswami et.al. to be online where some computations can be re-used (for example the re-trainings need not be from scratch) is non-trivial and requires further research that is beyond the scope of this paper. This 40x increased in run-time was the fundamental reason we did not evaluate the algorithm from Goswami et.al. (and even MetaOD) on all the datasets.", "claims_and_evidence": "SEAD relies on the assumption that there are fewer anomalies in the data stream, that is, it adjusts the weights based on the assumption that algorithms with lower anomaly scores are more reliable. However, if the proportion of anomalies is high, some detection algorithms may systematically misjudge, but SEAD will still give them higher weights. This assumption  seems to be not applicable to anomaly detection of complex patterns (such as concept drift causing some anomaly categories to become frequent).", "methods_and_evaluation_criteria": "SEAD needs to calculate anomaly scores for all base detection algorithms and perform MWU weight updates, which may be computationally expensive. In large-scale streaming data, calculating anomaly scores for all detection algorithms and normalizing them may be too expensive. SEAD++ attempts to reduce computation by subsampling detectors, but still needs to maintain and adjust weights between all detectors.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "1. Only Averaged-Precision score (APS) is used for evaluation , which is insufficient . I suggest the authors to include more comparisons.\n2. The experiments in the paper are mainly based on standard datasets (ODDS, USP database, etc.), but lack real industrial data verification. These datasets may not fully simulate the abnormal distribution of the real world (such as network security, industrial fault detection, etc.). The paper only contains an internal telemetry dataset, but does not provide detailed analysis or reproducible experimental details.\n3. The paper does not compare with the latest unsupervised anomaly detection model selection methods (such as the time series anomaly detection model selection method proposed by Goswami et al., 2023). The paper is only compared with MetaOD (offline method) and not with other online unsupervised methods. I suggest the authors to choose the latest unsupervised methods for a fairer experimental comparison.", "relation_to_broader_scientific_literature": "[1] SEAD relies on the accumulated information of historical anomaly scores for detector weighting, but does not explicitly handle concept drift. If the data stream distribution changes significantly (such as anomaly pattern changes), SEAD may maintain outdated detector weights, resulting in performance degradation. \nSome concept drift literature should be discussed.\n\n[2] Not compared with \"Unsupervised Model Selection for Time-series Anomaly Detection\". ICLR 2023 spotlight.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "other weakness:\n\nSEAD cannot outperform the base detector in all cases, and can only work when at least one detector performs well. However, if all detectors perform poorly on a certain data stream, SEAD cannot achieve good results. SEAD cannot actively improve the base detector, but only does a weighted combination, which cannot correct the inherent defects of the base detector.", "other_comments_or_suggestions": "the presentation should be large improved.", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aZFiAY55P6", "reviewer": "Reviewer_tquN", "summary": "The authors study unsupervised anomaly detection on data streams, where data distribution can change over time, affecting single model performance. The authors introduce a weighted ensemble that combine individual anomaly detectors based on how low their normalized scores are. The method is tested on 15 datasets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and address the main concern raised, namely that of the variance.\n\n> The reported variances seem very large. Not clear why this happens for SEAD, but this raises questions about the stability of the method.\n\nAs mentioned in the paper, SEAD has the *lowest* variance among all the methods across the diverse datasets we test against. This actually suggests that the datasets are diverse and not necessarily that the method is unstable. Among the competing methods, we observe that SEAD has the lowest variance in the observed ranks across the datasets - making it more performant than baselines.", "claims_and_evidence": "The authors claim that the proposed method, SEAD, is an effective and efficient ensemble method. However, a comparison with other ensemble methods is not conducted. The comparison is carried out only w.r.t. individual models.", "methods_and_evaluation_criteria": "The method is evaluated on sufficient datasets.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The reported variances seem very large. Not clear why this happens for SEAD, but this raises questions about the stability of the method.", "relation_to_broader_scientific_literature": "The paper studies anomaly detection in streaming data, where the assumption is that data distribution can change over time. This is an interesting topic which is understudied in literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "- Typos:\n   - \"we propose SEAD , the first\" => \"we propose SEAD, the first\";\n   - \"with state of the art streaming\" => \"with state-of-the-art streaming\";", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sUCGKfAdZu", "reviewer": "Reviewer_rbaj", "summary": "The paper proposes SEAD, an unsupervised ensemble method for streaming anomaly detection (AD) that dynamically selects the best base detectors without labeled data. It leverages multiplicative weights updates to adjust model weights based on normalized anomaly scores and introduces SEAD++ for runtime optimization via sampling. Experiments on 15 datasets demonstrate SEAD’s effectiveness and efficiency compared to base models and offline methods like MetaOD.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for valuable comments and suggestions. We propose to add these to the final camera ready version. \n\n> On hyper-parameter ablations\n\nWe add the following experiment to compare against the choice of $\\eta = [1, 0.1, 0.01]$ and $\\lambda = [10^{-2}, 10^{-4}, 10^{-6}]$. This is in the table below where $(1,10^{-6})$ implies $\\eta = 1$ and $\\lambda = 10^{-6}$. We will add this to the camera ready version of the paper.\n\n\n| dataset     | (1, 10^-6) | (1, 10^-4) | (1, 10^-2) | (0.1, 10^-6) | (0.1, 10^-4) | (0.1, 10^-2) | (0.01, 10^-6) | (0.01, 10^-4) | (0.01, 10^-2) |\n|-------------|-------------|-------------|-------------|---------------|---------------|---------------|----------------|----------------|----------------|\n| pima        | 0.519       | 0.519       | 0.52        | 0.531         | 0.531         | **0.531**         | 0.527          | 0.527          | 0.527          |\n| pendigits   | 0.091       | **0.255**       | 0.188       | 0.164         | 0.164         | 0.159         | 0.147          | 0.147          | 0.147          |\n| letter      | **0.084**       | 0.083       | 0.076       | 0.066         | 0.066         | 0.064         | 0.056          | 0.056          | 0.056          |\n| optdigits   | **0.084**       | 0.031       | 0.03        | 0.041         | 0.041         | 0.039         | 0.042          | 0.042          | 0.042          |\n| ionosphere  | 0.555       | 0.544       | 0.557       | 0.568         | 0.568         | 0.568         | **0.57** | **0.57** | **0.57** |\n| wbc         | 0.486       | 0.483       | 0.471       | 0.487         | 0.487         | 0.487         | **0.488** | **0.488** | **0.488** |\n| mammography | 0.118       | **0.128**       | 0.125       | 0.118         | 0.118         | 0.12          | 0.123          | 0.123          | 0.123          |\n| glass       | 0.115       | **0.129**       | 0.124       | 0.093         | 0.093         | 0.093         | 0.094          | 0.094          | 0.094          |\n| vertebral   | 0.154       | **0.158**       | **0.158**       | 0.152         | 0.152         | 0.152         | 0.154          | 0.154          | 0.154          |\n| cardio      | 0.505       | **0.508**       | 0.173       | 0.24          | 0.24          | 0.231         | 0.209          | 0.209          | 0.209          |\n\n> On statistical tests comparing the performance of SEAD against other forms of aggregation such as mean, max and min.\n\nWe compare SEAD against the following competitor methods using a few different statistical tests. In each case, we are computing the p-values for the null-hypothesis that the Average Precision score by SEAD is statistically identical to that by the competitor methods across the 15 datasets we test on. From the table, we can see that with the 15 datasets, we can only reject the null hypothesis for the RRCF methods. We believe this to be an artifact of the fact that we only test on 15 datasets and not more. \n\n| Competitor_Model | T_Test_PValue_OneSided | Wilcoxon_PValue | MannWhitney_PValue | Sign_Test_PValue | Cohens_d |\n|---|---|---|---|---|---|\n| rule_based_models_0 | 0.48809 | 0.34162 | 0.3937 | 0.21198 | 0.00812 |\n| rrcf_0 | 0.02438 | 0.01516 | 0.14504 | 0.02869 | 0.57685 |\n| rrcf_1 | 0.04179 | 0.02399 | 0.22136 | 0.02869 | 0.4979 |\n| rrcf_2 | 0.04579 | 0.0473 | 0.2807 | 0.15088 | 0.48413 |\n| rrcf_3 | 0.03284 | 0.01292 | 0.33156 | 0.02869 | 0.53363 |\n| xstream_0 | 0.02094 | 0.00076 | 0.04073 | 0.00049 | 0.59848 |\n| xstream_1 | 0.20831 | 0.30767 | 0.48345 | 0.60474 | 0.22371 |\n| xstream_2 | 0.13756 | 0.31934 | 0.39371 | 0.5 | 0.30355 |\n| xstream_3 | 0.14407 | 0.27546 | 0.50827 | 0.39526 | 0.2951 |\n| iforestasd_0 | 0.32432 | 0.48898 | 0.33913 | 0.15088 | 0.12444 |\n| iforestasd_1 | 0.37355 | 0.55481 | 0.35445 | 0.30362 | 0.0879 |\n| iforestasd_2 | 0.3849 | 0.56235 | 0.36997 | 0.21198 | 0.07974 |\n| iforestasd_3 | 0.38341 | 0.73776 | 0.47519 | 0.69638 | 0.08081 |\n| mean | 0.37577 | 0.53751 | 0.49173 | 0.39526 | 0.0863 |\n| max | 0.07483 | 0.0535 | 0.20926 | 0.15088 | 0.40743 |\n| min | 0.08475 | 0.11045 | 0.31664 | 0.21198 | 0.38714 |", "claims_and_evidence": "Supported Claims: SEAD’s unsupervised nature and adaptability to non-stationarity are supported by experiments showing it outperforms base models and adapts weights over time (e.g., Figure 2). Efficiency claims are validated by runtime comparisons (e.g., SEAD++ reduces runtime by ~50% vs. SEAD in Table 4).\nProblematic Claims: The claim that SEAD \"matches the performance of the best base algorithm\" (Sec. 4.3) is not fully supported; Table 2 shows SEAD occasionally ranks lower (e.g., 9th on WBC). A statistical significance test is missing.", "methods_and_evaluation_criteria": "Normalizing anomaly scores via quantiles (Sec. 3.2) addresses score incomparability. The use of APS (Averaged Precision Score) is appropriate for imbalanced anomaly detection.", "theoretical_claims": "Theorem 3.1 cites regret bounds from FTRL literature but does not provide a self-contained proof. While plausible, the paper assumes anomalies are \"rare\" for at least one detector; this assumption’s impact on regret guarantees is not analyzed.", "experimental_designs_or_analyses": "Comprehensive evaluation across 15 datasets with varying anomaly rates and dimensions. Runtime comparisons with MetaOD (Table 5) highlight streaming efficiency.\nNo ablation study on SEAD’s hyperparameters (e.g., learning rate η, regularization λ). Missing comparison to state-of-the-art streaming ensembles (e.g., LODA, xStream variants).", "relation_to_broader_scientific_literature": "SEAD addresses a gap in unsupervised online model selection for AD", "essential_references_not_discussed": "Most of the references were provided and due to my lack of knowledge in the field, it is difficult for me to provide an accurate judgement.", "other_strengths_and_weaknesses": "Combines multiplicative weights with quantile normalization for unsupervised AD, a novel approach. Provides a practical solution for real-time monitoring systems where labeling is infeasible.\nThe pseudo-code (Algorithm 1) is clear, but the loss function (Eq. 2) could be better motivated.", "other_comments_or_suggestions": "Typos: \"t-digest\" (Sec. 3.2), \"detector\" misspelled in Table 1 header.\nFigure 1 is referenced but not included in the provided content.", "questions_for_authors": "1. Are the performance differences between SEAD and base models statistically significant (e.g., via paired t-tests)?\n2. How do choices of η and λ impact SEAD’s performance?\nI lack sufficient knowledge in this area, but I will refer to other reviewers' comments to revise the final rating.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces SEAD, an unsupervised model selection procedure for streaming anomaly detection. This is the first work of this kind, and provides both theoretical guarantees and robustness to uninformative AD models compared to baselines. However, the empirical evaluation is not entirely convincing in the showing advantages over the average baseline. Considering critical difference diagrams for adjusted hypothesis testing might be beneficial to provide a clearer quantitative comparison.
Decision: Accept (poster)