mTSBench: Benchmarking Multivariate Time Series Anomaly Detection and Model Selection at Scale
Authors: Xiaona Zhou
Constantin Brif
Ismini Lourentzou
Venue: Accepted by TMLR ()

Abstract: Anomaly detection in multivariate time series is essential across domains such as healthcare, cybersecurity, and industrial monitoring, yet remains fundamentally challenging due to high-dimensional dependencies, the presence of cross-correlations between time-dependent variables, and the scarcity of labeled anomalies. We introduce mTSBench, the largest benchmark to date for multivariate time series anomaly detection and model selection, consisting of 344 labeled time series across 19 datasets from a wide range of application domains. We comprehensively evaluate 24 anomaly detectors, including the only two publicly available large language model-based methods for multivariate time series. Consistent with prior findings, we observe that no single detector dominates across datasets, motivating the need for effective model selection. We benchmark three recent model selection methods and find that even the strongest of them remain far from optimal. Our results highlight the outstanding need for robust, generalizable selection strategies. We open-source the benchmark at \url{https://plan-lab.github.io/mtsbench} to encourage future research.

Published in Transactions on Machine Learning Research (02/2026)
Benchmarking Multivariate Time Series Anomaly Detection and Model Selection at Scale
M
MTSBench:
Xiaona Zhou
ziaona:20iliioois edu
Siebel School of Computing and Data Science University of Ilinois Urbana- Catona-Champpiige
Constantin Brif
enbrif@sandia.pot
Center of Computation and Analysis for National Security Sandia National Laboratories
Ismini Lourentzou
lourent2 edu
School of Information Sciences
University of Blinois Urbana-Chin Channppg
Reviewed on OpenReview: // net/ forum? id= FL
Abstract
Anomaly detection in multivariate time series is essential across domains such as healthcare, cybersccurity, and industrial monitoring, yct remains fundamentally challenging due to high-dimonn nigh-dimensional dependencies, the presence of cross-correlations between time-dependent variables, and the scarcity of labeled anomalies. We introduce mTSBeench, the largest benchmark to date for multivariate time series anomaly detection and model selection, consisting of 344 labeled time series across 19 datasets from wide range of application domains. We comprehensively evaluate 24 anomaly detectors, including the only two publicly available large language model-based methods for multivariate time scries. Consistent with prior findings, we observe that no single detector dominates across datasets, molivating the need for effective model selection. We benchmark three recent model selection methods and find that even the strongest of them remains far from optimal. Our results highlight the outstanding need for robust, generalizable selection strategics. We open-source the benchmark at htttss:/plaan
ttithub.io/mtbennn. to encourage future research.
1 Introduction
Multivariate Time Series Anomaly Detection (MTS-AD) is critical for identifying unexpected patterns in multi-signal temporal data across domains such as healthcare, cybersecurity, industrial monitoring, and finance SBlaziue--Garcia et al. 2021 Garg et al. (021).. The rapid digital transformation of these sectors has led to a surge in high-dimensional time series data, where timely and accurate anomaly detection is essential to prevent system failures, mitigate security threats, and optimize operational efficiency (Basu Meckesheimer 2007; (Sucgliia et al. 2022: Yang al. 2023). However, identifying anomalies in multivariate time series remains challenging due to their inherent complexity and helerogeneity, compounded by factors such non-linear temporal relationships, intir-variable correlations, and the sparsity of anomalous events.
A diverse set of approaches to anomaly detection has been developed, including the use of foundation
models [Bian el al.) 2024 Zhou et al. 2023), deep learning (Xu et al. 2022
Sakurada & Yairi 2014
Munir
et al.
2018 Xu et al. 2018). ; classic machine learning (Hariri et al. 2019 Goldstein Dengel 2012 Yairi
et al.
2001 and statistical models (Roussccuw & Van Dricssen, 1999 Hochenbaum et al. 2017)
However,
the performance of anomaly detection methods varies widely across datasets, with recent works (Braii &
Wagner 2020
Ho et al. 2025 Zamanzadeh Darban et al. 2024 Schmidl et. al. 2022
Paparrizos al.
2022b

---

Published in Transactions on Machine Learning Research (02/2020)
AutoEncoder
O OmniAnomaly
ALLM4TS
CBLOF
CNN
COPOD
Donut
EIF
FITS
HBOS
FForest
KM RMeansAD
KNN
LOF
LSTMAD
MCD
OCSVM
OFA
PCA
RobustPCA
TimesNet
TranAD
USAD
Callt2
49
78
82
39
78
74
54
89
50
68
82
72
81
Daphnet
57
49
43
63 56
6
53
59
66
59
60
71
68
59
53
42
41
66
58
58
37
64
55
56
Exathlon
55
50
66
53 71
56 75
57
64
54
62 .0
52
54
70
65
64
71
76
56 66
.72
71
GECCO
65
51
.33
72 .90
74
66
75
70
74 68
83. 71
88
63
69
.97
86 86
32
.1
.89
.22
CHL
85
12.22.
73
78. 84
76
.53
85
.2
76.77
76
77 74
02 43
.4.67
83
46
70 70
.0 .31
.6.6688
Genesis
26
60
53
.1
.10
53
.39
59
73
75
.6
.2
.9.92
0.8
GuGenTAG
75
51
70
66 76
39
64
60
70
49
70
51
46
60
76
48
53
.1
.84 .84.44
84
57
166
.7.755
MITDB
64
11..0..5
53
50.
68
53
.58
53
59
.53
64
556
50
50
59
53
55
.61
.62 62
51
662
.59.60
Dataset
MSL 53
15.60.60
.5
.59 .5.88666680 57
.55
71
62.663//
.70.70
.1
.6.1666
0.6
AUC-ROC
OPPORTUNITY
42
53
85
.93 .74 .8.66.7 79. 78
32
71
87 16.46
.35
12.52.8000
$2.80.95.
78 .30
.7.77 .7
.17 .31
.79.99
PSM
42
53
.8.88.
.8.66..1
.7..7..8
73
.6
.66677.77
.7
.9..59.5533
40
66
75.
59 58
.63.66
0.4
SMAP
49
64
.6.59
.5
.75
55.55.00555
.1.61.5555
.5
44
.55
SMD
71
46
71
78 81
.7.77
72
81
74
74
71
81
.4.59
59
79
70
86
.8. 88
57
82 .77
.80
SVDB
60
50
51
49
63
51
53
50
63
51
68
50
.60
51
51
.70.70
52
60
,56
0.2
CIC-IDS-2011
48
49
56
46
51
52
32
42
42
54
44 44
36
47
55
75
43
.43
40 40
61
.40
.4
44
CreditCard
63
49.9999
69
68
95
72
6166164
67
55
68
55 51
69
52
.95
81
65
70
01.71.7 .71
.35
57 .59
.90
Metro
82
52
46
81
81
79
81
36
.7
0.0
Occupancy
89
51
89
73
86
95
79
90
76
92
91
78
86
45
85
96
96
82
82
83
88
77
.69
.55
SWAN-SF 50
15.50.1
52
48.8
.6.1
.50
.63
46 1.42
.44
65
.26
55
53
.58
.77
1..49.9
.6.1
.65
.65
.87
.50
.57.55
.54
Figure 1: Average AUC-ROC (f) Performance of 24 Anomaly Detection Methods (I-axis) Evaluated Across 19 mTSBench Datasets (y-axis). The substantial performance variability across datasets highlights the need for robust model selection strategics. mTSBench benchmarks the capability of model selection techniques to systematically identify the optimal anomaly detection method among 24 state-of-the-art detectors evaluated on comprehensive collection of 344 multivariate time series.
emphasizing repeatedly that existing algorithus do not consistently excel in all anomaly detection scenarios. This inconsistency is further compounded by the unsupervised nature of anomaly detection tasks (Mejri ct al. 2024 Belay et al. 2023), where ground truth labels are often unavailable, making model selection an open challenge. These challenges in MTS-AD highlight the need for adaptive model selection stralegies that can identify the optimal detector for a given multivariate time series dataset.
Existing model selection methods be grouped into meta-learning, unsupervised, internal evaluation-bass and casssifeerbbseeee with many of them leveraging historical performance metrics, meta-features. internal
performance measures to identify the best models for new datasets (Zhao et al. 2021 2022 Navarro et al. 2023 Zhang et al. 2022b; Goswari et al. 2021). However, their evaluation is often conducted on disparate datasets and tasks, leading to inconsistent and incomparable results across studies. This lack of standardization not only hampers progress developing effective model selection techniques but also obscures the real-world applicability of proposed methods. Consoquently, there need for a unified benchmark that systematically evaluates model selection approaches under consistent settings and can facilitate the development of robust. selection methods, specifically tailored for the challenging task of MTS-AD.
To address these challenges, we introduce mTSBench, the largest and most diverse benchmark for MTS-AD and model selection to date. mSBBench consists of 344 multivariate lime series from 19 publicly available datasets, covering 12 application domains. These datasets include both point-based and range- based anomalies, reflecting real-world temporal dependencics and cross-signal interactions. Morcover, the mTSBench evaluation suite spans 24 anomaly detection methods based on various approaches, including reconstruction, prediction, isatis and large language model (LLM). Our empirical analysis highlights substantial variability in
2

---

Published in Transactions on Machine Learning Research (02/2026)
Table I: Comparison Between TSBBench and Existing Anomaly Detection Benchmarks that Contain Multivariate Time Series. Algorithm categories comprise foundation models (FM), deep learning (DL), classic machine learning (ML) and Other (e.g., statistical, data mining, etc.).
MTS Data
Model Selection
Anomaly Detection
# Datasets
#
# Selectors
# Metries
# FM
# DL
# ML
# Other
Metries
TODS Lai e5 al.
25
TIMESEAD (Wageer
2025
21
26
0
EEAD Zhhan et al 2023
8
8
X
X
0
6
0
4
TMMEVAL (Scaminll
22/22
238
15
7
11
TSB-AD
Liu Papariizos 20/21
200
mTBBencc (Ours)
19
344
10
7
5
13
anomaly detection performance across datasets, For example, Figure showcases the mean AUC-ROC scores of the 24 detectors across the 19 datasets, illustrating how certain detectors achieve near-perfect accuracy on particular datasets while modest results on others. This inconsistency detection performance emphasizes the critical importance of model sclection strategies that account for datasct characteristics and lemporal dependencies. To address this gap, unlike existing benchmarks summarized in Table 1 m TSBnch uniquely integrates model selection methods and corresponding evaluation metrics, offering insights into their robustness and adaptability - a crucial aspect of MTS-AD that remains underexplored.
The contributions of our work can be sunmarized as follows:
(1) We introduce mISBench, the largest and most comprehensive MTS-AD and model selection benchmark to date, featuring 344 labeled multivariate time series from 19 datasets across 12 application domains. mTSBench systematically evaluates 24 anomaly detection methods, including the only LLM-based methods for MTS-AD, reflecting their performance in multivariate settings with real-world temporal dependencies and cross-signal interactions.
(2) Our empirical analysis reveals that, among the evaluated methods, no single anomaly detection method performs consistently well across datasets in m'SBencch underscoring the need for adaptive selection strategics. To this end, TSBencc is the first to integrate unsupervised model selection methods and benchmark their effectiveness across diverse time series contexts and under consistent settings.
(3) To drive reproducible comparisons, mTSBeench introduces unified evaluation suite with point-based and ranking- based metrics for anomaly detection and model selection. Using this standardized setup, we observe substantial gaps between the evaluated unsupervised model selection methods and both optimal and trivial basclincs. These results highlight limitations of current unsupervised selection strategics and underscore the need for more adaptive model selection mechanisms.
2 Related Work
Time Series Anomaly Detection.
Time series data, represented as ordered sequences of real-valued observations, can be categorized into univari- ate (e.g., single sensor readings) and multivariate (e.g., multiple sensors capturing joint phenomena). Time series anomaly detection is inherently challenging due to the diverse manifestations of anomalies, typically categorized into point anomalies, representing isolated deviations; contextual anomalies, which are abnormal only within a specific temporal context; and collective anomalies, emerging from atypical patterns spanning Blazque al. 2021 Shaukat al. 2021 Chandola al.,
multiple time steps (Boniol al.) 2024
2009). Existing surveys have extensively evaluated anomaly detection algorithms across range of paradigms, including unsupervised (Mejri et al.
2024
explainable (Li et al. 2023), model-based (Correia et al. 2024), and
transformer-based approaches (Wen et al. 2023). Dedicated reviews have also covered univariate [Braci & Wag.
ner 2020: Paparrizos el al. 2022b
T0e2maar Freeman el al.) 2021). graph-based (Jin et al. 2024a Ho el al. 2025), and
deep learning-basee based methods (Zamanzadeh Darban et al.
2024 Yan et al. 2024 Chalapathy & Chawla 2019). with recent efforts turning to foundation models (Ye ct al.
20024 Su al. 2024 Jin al., 2024b). Recent work
(Schmidl et al.) (022)) performed the most comprehensive evaluation date, analyzing 71 anomaly detection
methods across both univariate and multivariate time series.
All these studies consistently affirm that single
3

---

Published in Transactions on Machine Learning Research (02/2026)
anomaly detection method excels universally across domains anomaly types, motivating the need for dynamic model selection strategies that can adaptively identify the best-performing anomaly detector for a given time series instance. However, while these works provide valuable insights, they largely focus on univariate settings, leaving critical gaps in understanding how detection methods generalize to complex multivariate dependencics.
Model Selection for Time Series. Recent advances have led to the development of diverse model selection frameworks that leverage intrinsic dataset properties, meta-learning signals, or evaluation heuristics to find the most suitable anomaly detector for a given time scries (Sylligardos et al., 2023 Valenzuela et al., 2024 Trirat
et al. 2024 Eldele et al. 2021). selection strategies can be broadly categorized into four main approaches:
(1) Classificr-baseee methods, which train classifiers to rank detectors based on extracted meta-features (Ying 2020: Chatterjee et al. 2022;; ; (2) Meta-learning approaches, which predict model performance on unseen
et al.
2022 Zhao et al, 2021 Navarro et al.] 2023 Zhang et al., 2022b); (3) Reinforcement learning- based methods, which optimize
datasets by generalizing from historical observations (Singh & Vanschoren 2022 Yu et al.)
2022a); and (4) Human-in--he--log mechanisms, which involve expert-driven relinement to narrow down potential detectors based on observed time series characteristies (Freeman & Beaver 2019). Ensemble learning strategies that
detector choice by itcratively adjusting based on feodback signals (Wang et al.) 2022 Zhang ct al.
aggregate detector outputs using model variance as a sclection signal (Jung ct al. 2021 and methods that basc
selection on internal measures (Ma et al. 2021 Zhao et al. 2022) further extend these capabilities. Despite these advances, current evaluations are fragmented and often restricted to limited datasets. Among existing unsupervised selection methods, many have not been evaluated with time series anomaly detectors (Zhang
ct al. 2022b; Singhh & Vanschoren 2022:
Zhao et al., 2021] Yu al. 2022 Zhao et al.) 2022, Ma ct al.
2021 while of hers are tested on limited set. of detectors with few time series datasets (Navarro al. 2023
Jung et al. 2021 Chatterjoc et al. 2022 Goswami et al., 2021 Ying et al.) 2020), leaving significant gaps
in understanding model selection performance on multivariate time series data. A recent study offers useful insights into re-purposing time series classifiers for model selection in univariate anomaly detection (Sylligardos) ct al. 2023 However, benchmarking established model selection methods for multivariate time-series anomaly detection has remained largely underexplored. Our work aims to fill this gap by providing the first systematic benchmark that enables consistent comparison of selectors and analysis of their failure modes.
Anomaly Detection Benchmarks. Existing benchmarks for MTS-AD often fall short in terms of scale, diversity, and real-world representativeness. While several benchmarks have been proposed [Braci & Wagner
2020 Dau et al.
2018
Paparrizos et al. 2022b Laptev et al. 2015 Lavin & Ahmad 2015 Lai et al. 2021
Schmidl et al. 2022 Wagner et al.
2023
Zhang et al. 2023
Liu & Paparrizos 2024). only a handful include
multivariate time series (Lai et al. 2021
Schmidl ct al. 2022
Wagner et al., 2023 Zhang ct al. 2023
Liu &
Paparrizos 2024 and even those predominantly focus on univariale time series or have very limited mu mullival livari- ate representation. Notably, none of these benchmarks support model selection evaluation, which is critical for deploying robust anomaly detectors in practice (Table []. To address these limitations, m'TSeench introduces the largest and most diverse multivariate time series collection from 19 publicly available sources, spanning 12 application domains. Most importantly, mTSBencc is the first benchmark to comprehensively evaluate model selection strategies for MTS-AD, an important aspect that is underexplored in existing benchmarks.
3 mTSBench Benchmark
Problem Definition, A multivariate time series is a sequence of observations recorded over time, represented as [=(((1, (x2, 12),.... (Xp; 4n),, where x, E Rd is a d-dimensional feature vector observed at timestamp t, and n is the length of the time scries (in practice, timestamps are often omitted). In supervised scttings, each observation x; in the time series can be associated with a binary anomaly label yi E (0,1), where yi indicates an anomalous observation, and =0= indicates normal behavior. However, in practice, labels y= = [in, U2. J are often unavailable, necessitating the development of unsupervised methods for anomaly detection. Anomaly detection models my .dd + R assign an anomaly score 8; to each observation x,, í.e., 8; mx))),, where a greater 8; indicates a higher likelihood of x; being anomalous. These scores are typically converted into binary predictions di E 10 using a threshold Tj, Le., = > T,).
There exists a large number of various anomaly detection algorithms for time series, yet no single one consistently outperforms others across all data distributions. This variability has driven the need for
4

---

Published in Transactions on Machine Learning Research (02/2026)
Daphnet
Datasets
MSL
GHL
F Metrics AUC-P-Rr
Callt2
SMAP
GECCO
PSM Exathlon
Recall AUC-ROC
VUS-PR VUS-ROC R-based-F,
SMD
Metro
SWAN-SF
Genesis
Precision
CreditCard
AUC-PR Max-Fi
SVDB
MITDB
CIC-IDS-2017
mTS Bench
Affiliation-Fr PA-F,
Occupancy
GutenTAG
Event-based-F,
Orthus
FMMS
LSTMAD TransAD
ALLM4TS OFA USAD
MetaOD
Oracle
Random
OmniAnomaly
Transformer
CNN
TimesNet
IFOREST CBLOF
Near-Optimal
AutoEncoder KMeansAD
COPOD
RobustPCA
PCA
LOF
EIF
Model Selection
KNN
FITS
Anomaly Detection
OCSVM Donut
Figure 2: TSBeenc Overview. mTSBencc is the largest and most diverse benchmark for multivariate
time series anomaly detection and model selection, spanning 19 multivariate time series datasets across various application domains and establishing platform for robust anomaly detection and adaptive model selection in real-world multivariate contexts. mTTSBench's 's comprehensive evaluation suite and diverse collection of state-of-the-art anomaly detectors, including statistical, deep learning, and LLM-basod approaches, facilitates standardized comparison of model selection strategies.
unsupervised model selection methods to identify the most suitable anomaly detector for a specific dataset. Formally, let M {m,, m,.,,,,...., denote a set of k anomaly detection models. The objective of unsupervised model selection for anomaly detection is to identify the best model m E M for given unlabeled test. time series Trest- In the absence of labels, unsupervised model selection relies on estimating the relative performance models through proxy performance function P(m,, Trest), which approximates the performance of each model based on intrinsic characteristies of Trest and properties of the models in M. The best model m² is selected as m² arg maxm, EM P(m,, Tust).
Model selection for MTS-AD has many applications across diverse industry sectors, including healtheare, cybersecurity, industrial monitoring, and finance, but remains underexplored due to the scarcity of compre- hensive benchmarks that enable robust evaluation across multiple domains and realistic scenarios. mSBeench aims fill this gap with comprehensive suite of datascts, methods, and evaluation metrics.
3.1 m mTSBench Overview
We introduce mTSBench, the largest and most diverse benchmark for MTS-AD and model selection date. mTSBench consists of 344 multivariate time scries drawn from 19 publicly available sources and spans 12 application domains. Unlike existing benchmarks that primarily focus on univariate time series, m TSBench provides a rich representation of real-world scenarios by including both point-based and range- anomalies, capturing complex temporal dependencies and cross-signal interactions. To enable robust evaluation, mTSS Bench integrates 24 anomaly detectors spanning approaches based on reconstruction, prediction, statistics, and LLM, as well as 3 model selection methods that leverage surrogate metries, factorization machine, and meta-learning identify the best anomaly detection method for given time series. Finally, the evaluation suite in mTSeench includes 13 anomaly detection metrics and 3 model selection metrics, offering comprehensive framework to benchmark efficacy and robustness of anomaly detection and model selection.
Anomaly Detection Methods. m'SBeenc has a diverse pool of open-source anomaly detectors, com- prising 10 unsupervised and 14 semi-supervised methods, including currently existing LLMs designed for MTS-AD. The selected anomaly detection methods are summarized in Appendix A
Unsupervised anomaly detection methods operate without the need for labeled training data and can be directly applied at test time to identify anomalies. In contrast, semi-supervised methods require training on
5

---

Published in Transactions on Machine Learning Research (02/2020)
anomaly-frec time scries. TSBennc encompasses wide variety of unsupervised methods: CBLOF (He et al.
2003),
COPOD (Li et al.
2020). EIF (Hariri et al., 2019), HBOS (Goldstein & Dengel 2012).
IForest (Liu
iet al.
)008), AD Yairi et al. 2001), KNN (Ramaswamy et al. 2000), LOF (Breunig et al. 2000
PCA
[Aggarwal 2017). and CobustPCA (Paffenroth et al. 2018), and semi-supervised methods: LSTM-
AD
Malhotra et al.] 2015), AutoEncoder (Sakurada & Yairi 2014 CNN (Munir et al.
2018), Donut (Xu
ct al.
2018); FITS (Xu et al.
2023),
MCD (Roussceuw & Van Driessen 1999), OCSVM (Schölkopf ct al.
1999
TemesNet (Wu et al.
2023); TranAD (Tuli et al. 2002), Anomaly Transformer (Xu et al.
2022
OmniAnomaly (Su et al.
2019], and USAD (Audibert et al., p.20 mSSBench also includes recent
foundation models designed for MTS-AD: ALLM4TS (Bian et al. 2024), and OFA (Zhou et al.
2023
which leverage large-scale pretraining to enable cross-domain generalization.
Unsupervised Model Selection Methods. To enable robust model selection across diverse multivari- ate time series, mTSBench includes three methods applicable to multivariate data. These include MctaOD (Navarro ct al.) 2023), a meta-learning method that employs Principal Component Analysis (PCA) to extract latent represenlations of a dataset and trains a random forest to map these representations to a precomputed performance matrix; Orthus (Zhao et al.) 2021), a method that extracts univariate time-scries meta-fcatures and employs regression or random forcsts, depending on whether the test cases are unseen or similar to the training set; and FMMS (Zhang et al. 2022b). a factorization machine-based method that employs a regression model to directly relate meta-features to model performance.
In addition, we benchmark these model selection methods against three trivial baselines that establish performance bounds: Oracle, a baseline that consistently selects the best anomaly detection method for each time series (based on ground truth labels), serving as an upper bound; Near-optimal, baseline that chooses the second-best anomaly detector for each time series, providing a practical reference that is more attainable than the theoretical maximum of the Oracle baseline; and Random, a baseline that selects an anomaly detector for each time series at random, offering a lower-bound estimate of model selection performance by disregarding dataset-specific characteristics. Additionally, we include PCA, a well-cstabliseed classic machine learning method (Aggarwal) 2017); as a fixed-choice baseline; and an Ensemble baseline, where we compute the mean anomaly score across all 24 detectors at each time step to produce a single combined prediction (Sylligardos et al. 2023).
Datasets. Multivariale time series data in m TSBech span wide range of application domains, including in-
dustrial process (GECCO (Rehbach et al.) 2018). GHL (Filonov et al. 2016). GENESIS fvon Birgelen & Nigge-
mann 2018) ), healthcare (DAPINET Bachlin et al. 2009), MITDB (Goldberger et al. 2000). SVDB (Green
wald et al. 1990)), cybersecurity CIC-IDS-2011 (Canadian Institute for Cybersccurity 2017)). ) finance
(CReDeFCARD (Dal Pozzolo et al.] 2018)), ). IT infrastructure (EXAthI.ON (Jacob et al. 2202)] p. PSM (Abdulaal
et al. 2021). , SMD (Su et a 2019)), smart building (CALIT2 (Hutchins) 20066), OCCUPANCY (Candanedo
& Feldheim 2016) spacecraft telemetry (MSL, SMAP (Hundman et al. 2018)), ) etc. These time series vary
significantly in scale, with lengths ranging from thousands to over half a million points, and dimensionalities ranging from d = 3 to d = 73, presenting a challenging and realistic testbed for evaluating MTS-AD and model selection strategies. The datasets also differ in their anomaly structures: some contain sparse point anomalies (e.g., CREDITCARD and OCCUpAnCY), while others include complex or long-range anomalous sequences (e.g.. MITDB and CIC-IDS-20117 Each time series in mSBBench includes a clean training/tes test partition. To ensure consistency across datascts, we apply a unified data-splitting protocol: (1) if a datasct provides official train/test splits, we use them directly; (2) if anomaly labels are available but no official split. is provided, we extract long contiguous segment without labeled anomalies the training set and use the remaining portion of the time scries as the test set. Appendix B provides additional details on the datasct characteristics.
Evaluation Metrics. To provide a comprehensive evaluation, m TSBench reports 13 MTS-AD evaluation metrics that span both pose and range- based performance. Point-wise metrics include Precision, Recall, and Fi, which evaluate the ability of the selected model to accurately identify individual anomaly points while balancing false positives and false negatives. under the Receiver Operating Characteristics Curve (AUC-ROC), Area Under the Precision-Recall Curve (AUC-PR), and Area Under the range-based Precision, range-based Recall Curve (AUC-PTRT)) )are employed, assessing performance across varying thresholds (Tatbul et al. 2018). AUC-PR, in particular, is critical for detecting anomalies in imbalanced datasets where positive samples are sparse. Furthermore,
To capture broader detection quality, Area
6

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Model Selection Performance Comparison. Entries are meantssd over 344 time series in mTSBench for the top-1 detector chosen by each method. Bold and denote best and second-bes methods per metric. A (%) is the relative difference between mean values for the best method and Near-optimal.
Trivial Model Selection Baselines
Unsupervised Model Selection Methods
Metric
Oracle
Near-optinal
Random
PCA
MetaOD
FMMS
Octhus
Eusemble
A(%)
Fi
0.5:6 1. 0,263
0.514 0.256
0.299 = 0.252
0.252
0.195
0.135
0.135
0.222 1 0.185
0.222 0.200
0,189 1 0.187
-56.80
Precision
0.114 1 0.328
0.393 1: 0.319
0.191 0.236
0,295
0.305
0.189
0.236
0,2 +: 0.299
0.297 1 0.311
0.237 a: 0.202
-24.43
Rocall
0.483 + 0.238
0.438
0,287
0.275 0.313
0.319
0.310
0.205
0.210
0.307 + 0.276
0.283 0.271
0.300 + 0.901
-32.97
Affliiaton-Fi
0.879 0.107
0.868
0.101
0.785
0.124
0.818
0.777
0.100
0.834
0.112
0.814
0.111
0.792 0.107
-3.92
Event-based-
0.287
0.667 0.263
0.427 = 0.338
0.506
0.329
0.305
0.548
0.333
0.340
0.433 0.530
-17.84
Max-F,
0.548 4: 0.263
0.514 0.256
0,299 = 0.252
0,408 = 0.268
0.262
0.248
0.381 =: 0.263
0.386
0.270
0.379 4: 0.270
-24.90
PA-F1
0.825 +: 0.235
0,803 :: 0.226
0.705 0.316
0.627
0.320
0.658
0.324
0.738 : 0,285
0.647 + 0.332
0.984 +: 0.332
-8.09
R-Iaoed-F'
0,450 +: 0.237
0,430
0.224
0,252 0.190
0,410
0.257
0,242
0.104
0.366 +: 0.224
0.370
0.236
0.326 :: 0.246
-13.95
AUC-PyRT
0.417 0.276
0.400 0.260
0.234 0.228
0.322
0.278
0.217
0.218
0.318
0.273
0.309
0.272
0.278 +: 0.260
-0.500
AUC-PR
0.492 0.296
0.455
0.212 = 0.222
0.327
0.280
0.210
0.231
0.304 + 0.266
0.315
0.279
0.287 + 0.271
-30.77
AUC-ROC
0.811 1 0.130
0.796 0.148
0.613 = 0.178
0.697 = 0.219
0.605
0.189
0.659 1: 0.207
0.672
0.219
0.683 1: 0.214
-15.58
VUS-PR
0.521 1: 0,296
0.485 0.290
0.236
0.256
0,348
0.290
0.229
0.281
0.526 :: 0.277
0.333 1: 0.288
0.312 1: 0.279
-31.34
VUS-ROC
0.847 + 0.134
0.832 0.134
0.640 0.188
0.721
0.211
0.655 0.179
0.008 0.204
0.707
0.212
0.714 + 0.203
-1.002
lange-based metrics Volume Under the Receiver Operating Characteristies Surface (VUS-ROC) and Volume Under the Precision-Recal Surface (VUS-PR) (Paparrizos et al.) 2022a) generalize AUC by applying a tolerance buffer and contimuous scoring over anomaly boundaries. Additional metrics include PA-F, (Xul et al.] 2018) that applies heuristic point adjustment, Event-based-F, (Garg et al.) 2021) that treats each anomaly segment as a single event, Max-F, that reports the maximum F1 score achievable across all decision thresholds, R-based-Fr (Tatbul et al.) 2018)) that captures structural properties such as existence, overlap, and cardinality, and Affiliation-Fi (Huet et al., 2022) that measures proximity between predicted and ground-truth intervals. Collectively, these metrics ensure m TSBecch holistically evaluates anomaly detection methods across isolated, sequential, and sparse anomaly manifestations.
For model selection evaluation, m'SSeench includes ranking metrics Precision@k, Recall@k, and Normalized Discounted Cumulative Gain (NDCG). Precision@kk evaluates the model selection mothod's ability to prioritize high-performing detectors within its top-k recommendations, and refers to the proportion of selected detectors that are among the fighest-perfomming ones for a given dataset. In contrast, Recall@k quantifies the proportion of all sigh-performing detectors that are successfully retrieved within the top-k recommendations. Finally, NDCG (or NDCG@k for top-k recommendations) evaluates how well model selection preserves the correct ranking order, taking into account the relevance and relative order of the selected anomaly detectors.
4 Experimental Results
4.1 Anomaly Detection Performance Across Evaluation Metrics
The results, visualized in Figure 3 reveal distinct performance clusters among detectors. PCA, despite its simplicity, demonstrates strong and stable performance, particularly in VUS-PR and AUC-ROC, highlighting its generalizability across diverse time series. OmniAnomaly, a deep generative model, performs comparably well across all metrics, including those sensitive to temporal structure such as AUC-P,RT, indicating its ability to model temporal dependencies. The higher median scores across VUS-PR and VUS-ROC suggest that these models excel not only in isolating anomalous regions but also in maintaining high relative rankings over various sensitivity levels, critical for long-tat tail anomaly distributions in multivariate time series. In contrast, methods such as KNN, Transformer, LOF, and HBOS occupy the lower end of the performance spectrum across all five metrics, with particularly poor AUC-PR and AUC-PTRT scores. The low AUC-PR scores indicate struggling to detect anomalies in highly imbalanced datasets where true anomalies are sparse, while the poor AUC-P7RT performance suggests an inability to effectively manage the precision-recall trado-off across thresholds. In contrast, methods like COPOD and RobustPCA, while similarly underpetforming. display narrower distributions, suggesting stable but limited detection capacity.
7

---

Published in Transactions on Machine Learning Research (02/2026)
1.0
VUS-PR
0.5
0.0
1.0
VUS-ROC
0.5
1.0
AUC-PR
0.5
0.0
1.0
AUC-ROC
0.5
0.0
1.0
AUC-PPRT
0.5
0.0
PCA
OmniAnomaly
OFA
CNN
IForest
ALLM4TS
LSTMAD
KMeansAD
TranAD
USAD
MCD
AutoEncoder
TimesNet
Donut
FITS
OCSVM
CBLOF
EIF
RobustPCA
COPOD
HB0S
KNN
LOF
Transformer
Figure 3: Comparison of Anomaly Detection Methods Across Five Evaluation Metrics. The boxplots illustrate the distribution of performance scores for each method evaluated over all MTSeench datasels, measured using VUS-PR, VUS-ROC, AUC-PR, AUC-ROC, and AUC-PrRT. Delectors are ordered by their average VUS-PR score. Boxes represent interquartile ranges, with solid lines indicating the median and dashed lines indicating the mean. In-depth analysis in 44.1
In terms of LLM--bsed detectors, OFA consistently achieves higher median scores across VUS-PR, VUS- AUC-PR, AUC-ROC, and AUC-PrRT, with relatively compact interquartile ranges for AUC-ROC and AUC-PTRT suggesting that it is less sensitive to distributional shifts. In contrast, ALLM4TS exhibits broader performance variability, particularly in VUS-PR and VUS-ROC, indicative of its sensitivity dataset-specific noise. These observations suggest that both methods benefit from large-scale pretraining, where pretrained multimodal embeddings can capture richer temporal and semantic representations compared to traditional baselines. However, the observed variability in ALLM4TS and the performance gaps relative to more traditional baselines such as PCA suggest that significant improvements are still possible. For example, future work can explore modality-specific adapters, allowing L.LM-basod detectors finc-tune their representations based on the temporal, spatial, and contextual properties of anomalies. The inherent instruction dastructionffollovingg capability and generalization across diverse tasks of LLM-based methods enable human-in-the-loop
8

---

Published in Transactions on Machine Learning Research (02/2026)
Me
FMMS
Orthus
OD
FMMS
Orthus
0.43
0.8
0.30
AUC-ROC
VUS-PR
0.7
n.66
0.65
0.65
0.32
0363
0.31
0.60
0.29
0.57
0.24
0.22
0.2
<10
10-30
>30
<10
10-30
>330
Figure 4: Model Selection Performance Grouped by Time Series Dimensionality. VUS-PR (left) and AUC-ROC (right) for three dimensionality groups (<10, 10-30, >30). Discussion is in $42
anomaly detection and data analysis applications. To fully harness their potential, however, LLM-based detectors must demonstrate high adaptability to varying anomaly patterns and heterogeneous application domains. This necessitates robust model selection mechanisms that dynamically configure detection models to match the specific context and anomaly profile of incoming data.
4.2 Analysis of Model Selection Performance Across Evaluation Metrics
Table 2 presents the performance of the top-1 recommended anomaly detector for each model selection method across 13 evaluation metrics. FMMS and Orthus consistently outperform MetaOD across nearly all metrics, with Orthus achieving the best performance in 8 out of 13 metrics and FMMS leading in 5 metrics. MetaOD often performs on par with or worse than the Random baseline, suggesting limited transferability of its non-temporal fetarfeature design to time-series settings. Orthus performs well in AUC-ROC and R-based-F,, which indicates its strong capacity minimize false positives and maintain high anomaly-ranking accuracy. Meanwhile, FMMS performs well in Recall and Svent-based-- suggesting it more effective capturing distributed and event-based anomalies. However, both Orthus and FMMS demonstrate significant gaps relative to the Near-optimal baseline, with mean deficit of about 15% to 30% across key metrics. Their considerable underperformance in AUC-PR and VUS-PR, as compared to the Near-optimal baseline, means they select detectors that miss many true anomalies that best detectors would have caught. We also observe that PCA surpasses all three model selectors (MetaOD, FMMS, Orthus) on 9 out of 13 metrics, although its overall performance remains well below the Near-optimal and Oracle references. This further reinforces that existing model-selection approaches are far from reliable and that principled selection remains open challenge in MTS-AD. The ensemble bascline performs even worse, trailing FMMS and MctaOD AUC-ROC and VUS-ROC. These results
across all metrics, and achieving only comparable performance highlight the limitations of the existing unsupervised model selection mechanisms, which currently lack the ability to adapt to varying characteristics of diverse time series data. Promising directions include meta-learning-bate based adaptation, continual learning, and self-supervised representations that could enable finer control and improved performance in non-stationary environments.
Additionally, Figure shows performance grouped by time series dimensionality. FMMS and Orthus outperform MetaOD in both VUS-PR and AUC-ROC across all dimensionality groups. FMMS achieves its best performance for high dimensionalities (d 30), while Orthus performs consistently well across all dimensionalitics. highlighting its robustness across wider range of settings. Beyond dimensionality, we also analyze selector performance with respect to anomaly ratio and anomaly sequence length. As shown in Table (left), selector performance generally declines as the anomaly ratio increases. Both MetaOD and FMMS: 0.758 0.668), while Orthus peaks at moderate ratios (5-10%, 0.730) before dropping at > 10% (0.640). Although standard deviations are sizable and adjacent bins may overlap, the overall trend indicates that higher anomaly density
FMMS exhibit monotonic decrease AUC-ROC (MetaOD: 0.640+00554
9

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Model Selection Performance by Anomaly Characteristics. Entries are meantist of AUC-ROC across datasets grouped by anomaly ratio (left) and anomaly sequence length (right). and denote the best and second-best methods for the group; respectively. The best group is bolded for each selector. Discussion is in $4.2
Anomaly Ratio
Anomaly Sequence Length
Group
MetaOD
FMMS
Orthus
Group
MetaOD
FMMS
Orthus
<3%
0.640
0.206
0.758
0.189
0.653 0.269
0.671
0.200
0.833
0.148
0.848
0.136
4-5%
0.615
0.168
0.676
0.166
0.691 0.187
50-100
0.542 1 0.197
0.671 - 0.156
0.705 1 0.201
5-10%
0.592 0.148
0.641
0.202
0.730 = 0.185
100-201
0.619 + 0.160
0.694 = 0.181
0.641 = 0.252
> -10%
0.545 0.151
0.608 0.175
0.640 0.184
>200
0.576 + 0.181
0.638 = 0.203
0.666 + 0.184
MetaOD
FMMS
Orthus
40
35.71
Percentage (%)
44
135.
@ecall@k
30
25,87
320
0.5
20
13 00.08
MetaOD
11.19
8.62
FMMS
10
Orthus
0
4
12
16
20
24
Precision@3
NDCG03
k
Figure 5: Ranking Comparison of Model Selection Methods. (Left) Precision@3, Recall@3, and NDCG@3 for each method. (Right) Recall@k as a function of k. Discussion is in $4.3
tends to make model selection more challenging across methods. In contrast, Table (right) reveals that selector performance is more strongly affected by anomaly sequence length. All three methods perform better on datasets with shorter anomaly sequences, whereas performance degrades for longer segments. This reflects the difficulty of ranking detectors when anomalies span extended durations, where evaluation metrics (e.g., AUC-ROC, Precision@k) are less sensitive to subtle localization errors. Shorter anomalies tend to produce more distinct and localized deviations from normal behavior, making it easier for detectors to identify them and for selectors to distinguish between well-performing and poorly performing detectors.
4.3 Comparison of Ranking Capability of Model Selection Methods
Figure presents comparative analysis of unsupervised model selection methods in terms of their ability to rank top-performing anomaly detectors. Top-3 results (Figure 5 left) show that Orthus achieves the highest scores for all three ranking metrics, indicating its superior ability to identify relevant anomaly detectors early in the ranked list. FMMS ranks second, while MetaOD is a distant third, performing only marginally better than random selection. However, even Orthus's NDCG@3 score about 33% implies that its top-selected anomaly detector often positioned far from the true top ranks, which highlights the need for continued progress toward more reliable adaptive model selection strategies across diverse time series contexts. terms of overall ranking quality (Figure right), Orthus and FMMS alternate in achieving better Recall@k rates across broad ranges of k values, while MetaOD remains consistently lower. We also evaluate selector performance across ranking depths by reporting Precision@k and NDCG@k as function of & (Figure 0. Orthus achieves the highest Precision@k at small k; indicating a stronger ability to prioritize high-perforning detectors in the very top positions. FMMS also performs competitively in this regime but exhibits a more gradual decline, reflecting stable ranking quality across intermodiate depths. In contrast, MetaOD consistently lags behind other mothods across all underscoring its limited effectiveness in the anomaly detection setting. For NDCG@k (right panel), Orthus maintains advantage at small to mid-k, while FMMS eventually matches surpasses it. for
10

---

Published in Transactions on Machine Learning Research (02/2026)
0.15
MctaOD
FMMS
Precision Ok
0.1
Orthus
NDCG@R
0.5
0.05
MetaOD
FMMS
Orthus
8
12
16
20
24
4
8
12
16
20
24
k
Figure 6: Ranking Comparison of Model Selection Methods, (Left) Procisionãk as function of k. (Right) NDCG@K as function of k.
Table 4: NDCG@5 by Anomaly Sequence Length. Entries are meantstd of NDCG@5 grouped by anomaly sequence length. and denote best and second-best methods within each group, respectively. The best NDCG@5 is bolded for each selector. Discussion is in 44.3
Anomaly Seq. Length
MetaOD
FMMS
Orthus
<50
0.093 0.097
0.364 0.257
0.391 + 0.286
50-100
0.000 + 0.000
0.378 1 0.248
0.519 + 0.286
100-200
0.150 + 0.019
0.448 + 0.264
0.519 1: 0.197
>00
0.214 0.000
0.357 + 0.227
0.384 + 0.218
larger values (k: 6). These findings suggest that current selectors are more effective at producing a shortlist. of strong detectors than at reliably identifying a single best one, aligning with practical deployment needs.
To further examine robustness, report NDCG@5 stratified by anomaly sequence length. As shown in Table 4 Orthus consistently outperforms FMMS and MetaOD across all groups. The gap is most significant for anomalies of length 50- 10-1 where MetaOD fails to retrieve any of the top-5 detectors. Both FMMS and Orthus achieve their strongest performance on moderate-length anomalics (100-200), with NDCG@5 scores of 0.448 and 0.519, respectively. In contrast, MetaOD performs poorly on short anomalies (<50) and only marginally improves on long ones (>200), underscoring its limited robustness. These results reinforce the need for more adaptive selectors that can maintain ranking quality across varying anomaly characteristics.
To assess generalizability, also compute domain-specific AUC-ROC scores for each model selector across four representative application domains (Table 5). Results indicate notable variation across domains. FMMS achieves the strongest performance in IT infrastructuur (where it outperforms the other two selectors), but underperforms healthcare. Orthus also achieves the strongest performance in IT infrastructure and shows good results in spacecraft telemetry and healthcare (outperforming the other two selectors both these domains), while its weakest performance by a large margin is on industrial process data. MctaOD excels in industrial process (where it oulperforms the other two selectors), but underperforms in spacecraft telemetry and IT infrastructure. These findings highlight that selector effectiveness is domain-dependent, reinforcing the need for domain-aware model selection strategics.
4.4 Inference Time Analysis of Model Selection Methods
Inference time is a critical considcration for practical deployment of anomaly detection systems, as the ability rapidly detect and respond anomalies can significantly impact the effectiveness of downstream decision-making processes in safety-critical applications. To assess the efficiency of each model selection
11

---

Published in Transactions on Machine Learning Research (02/2026)
Table 5: Domain-level Model Selection Performance. Entries are meantstd AUC-ROC across four representative application domains. and denote the best and second-best methods for the domain, respectively. The best group is bolded for each selector. Results show clear variability across domains, highlighting the importance of domain-aware selection strategies.
Domain
MetaOD
FMMS
Orthus
Healthcare
0.654 + 0.122
0.617 + 0.161
0.661 + 0.168
IT Infrastructure
0.636 + 0.221
0.790 + 0.152
0.770 + 0.176
Industrial Process
0.792 0.198
0.774 + 0.160
0.344 + 0.343
Spacecraft Telemetry
0.514 0.247
0.564 0.223
0.703 + 0.224
MetaOD
FMMS
Orthus
Runtime (soconds)
40
30
20
9.91
210
0.00
.10
10
Gu
Callt2
Daphnne
Exathlon
GECCO
GHL
Genesis
Guten TAG
MITDB
OPPORTUNITY
MSL
PSM
SMAP
SMD
SVDB
CIC-IDS-200 2017
CreditCard
Metro
Occupancy
SWAN-SP
Figure 7: Average Inference Time of Model Selection Methods. Discussion in
4.4
method, we analyze their average inference times on TSBennch. Figure 7 shows that FMMS is generally the fastest selector across most datasets, with runtimes typically under except for larger datasets like CIC-IDS- 2017 (47.05 s), MITDB (19.51 s), and CREDITCARD (17.75 s), where other selectors also experience increased latency. Orthus consistently achieves a strong balance between inference efficiency and detection accuracy, with typically moderate runtimes (e.g., 9.94 s on SWAN-SF and 7.22 s on PSM) and even best efficiency on larger datasets (e.g., 1.09 s on MITDB), while maintaining strong performance. MetaOD cxhibits faster inference in certain cases (e.g., GHL, GENESIS), but shows higher variability and scales poorly on larger datasets, e.g., MITDB (21.66 s) and CIC-IDS- 2017 (49.40 s). These findings show the challenge of achieving both fast and high-quality recommendations time series grow in length and complexity.
5 Practical Takeaways
Across datasots in mTSBennc, the evaluated unsupervised selectors consistently underperform optimal or oracle baselines, indicating that effective model selection remains challenging in MTS-AD. This gap reflects several recurring limitations observed in evaluation. For example, selectors rely primarily coarse, dataset-level meta-fest meta-features, which provide limited information about temporal structure, long-range depen- dencies, or cross-signal interactions that are critical for distinguishing detector bohavior across heterogencous time series. Moreover, detector performance varies substantially with both time-series properties (e.g., dimensionality, noise characteristics, and cross-channel coupling) and anomaly characteristics (c.g., density, duration, and temporal dispersion), yet these factors are not explicitly represented in existing selector features. As result, selectors trained on aggregate statistics as meta-features exhibit limited ability to adapt dataset-specific characteristics, contributing to the observed gap relative to oracle performance in BTBBench.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Our results also suggest several considerations for interpreting unsupervised model selection outcomes. The persistent gap between selector performance and oracle baselines indicates that identifying single best detector is difficult under dataset heterogeneity and limited prior information. Accordingly, top-1 selector outputs should be interpreted with caution and may be better viewed as coarse rankings of candidate detectors rather than definitive choices. In such set tings, relying on well-established detectors or considering a small set of high-ranked candidates can provide more stable behavior when domain knowlodge limited.
6 Conclusion
We introduce m'TSBench, the largest, most. comprehensive benchmark for MTS-AD and model selection. Through systematic evaluation across 344 labeled time series from diverse application domains, mTTSeench highlights the substantial variability in anomaly detection performance and the critical need for robust model selection stralegies. Empirical results demonstrate that existing unsupervised model selection methods, while promising, fall significantly short of optimal performance within m mSBeench, exposing critical gaps in handling complex temporal dependencies and cross-signal interactions. To address these limitations, mTSBench provides a unified evaluation suite to facilitate reproducible research and accelerate progress in robust anomaly detection and adaptíve model selection, enabling more resilient multivariate time series analysis across domains like healthcare, industrial monitoring, and cybersecurity. These findings motivate new research directions in conlext-aware model selection, adaptive selectors that dynamically respond to lemporal shifts, and integration with foundation models to enhance cross-domain generalization and robustness.
Broader Impact Statement
This work introduces a standardized benchmark for model selection in MTS-AD, encouraging the development of more robust, adaptive, and generalizable anomaly detection systems. The socictal impact spans various high-stakes domains, including healthcare, industrial monitoring, cybersecurity, and financial systems, where reliable anomaly detection can enhance safety, efficiency, and prcision-maaing Model selection in MTS-AD has the potential to improve early warning mechanisms, reduce downtime in safety-critical systems, and support human docision-making in complex temporal environments. However, recognize that deploying model selection methods without robust validation could lead to unintended consequences, such as false alarms or missed critical events. While our benchmark highlights generalization gaps in existing methods, it is important that future rescarch also considers fairness, transparency, and robustness to data distribution shifts, particularly in sensitive domains. is also important that future research considers computational efficiency and energy requirements, especially as L.M.-based and deep learning selectors become more prevalent. Balancing accuracy, efficiency, and resource usage will be essential for responsible and scalable deployment of model selection methods. By open-sourcing mTSBennch, we aim to facilitate access to rigorous evaluation tools, foster reproducibility, and accelerate research in time series analysis. We anticipate this work will support practitioners in deploying more effective and trustworthy anomaly detection pipelines, while also inspiring new methods that adapt to complex, real-world data.
Acknowledgments
This work was supported by the U.S. Department of Energy, National Nuclear Security Administration, Office of Defense Nuclear Nonproliferation Research and Development, and by the Laboratory Directed Research and Development program at Sandia National Laboratorics. Sandia National Laboratories a multi-mission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC (NTESS), wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy's 's National Nuclear Security Administration (DOE/NNSA) under contract DE-NA0003525. This written work is authored by an employee of NTESS. The employec, not NTESS, owns the right, title, and interest in and to the written work and is responsible for its contents. Any subjective views or opinions that might be expressed in the written work do not necessarily represent the views of the U.S. Government, The publisher acknowledges that the U.S. Government retains non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this written work or allow others to do so, for U.S. Government purposes. The DOE will provide public access to results of federally sponsored research in accordance with the DOF Public Access Plan.
13

---

Published in Transactions on Machine Learning Research (02/2026)
References
Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In ACM SIGKDD Conference on Knowledge Discovery and 2021.
Data Mining, pp.
Charu C. Aggarwal. Outlier Analysis. Springer International Publishing, 2nd edition, 2017. ISBN 978-3-319- 47577-6.
Rafal Angryk, Petrus Martens, Berkay Aydin, Dustin Kempton, Sushant Mahajan, Sunitha Basodi, Azim Ahmadzadeh, Xumin Cai, Soukaina Filali Boubrahimi, Shah Muhammad Hamdi, Micheal Schuh, and Manolis Georgoulis. Swan-sf: Space weather analytics dataset-solar flares. Harvard Dataverse dataset, pp- 102, 2020.
Julien Audibert, Pietro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A Zuluaga. USAD: unsupervised anomaly detection on multivariate time scries. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2020.
Mare Bachlin, Meir Plotnik, Daniel Roggen, Inbal Maidan, Jeffrey M Hausdorff, Nir Giladi, and Gerhard Troster. Wearable assistant for Parkinson's 's disease patients with the freezing of gait symptom. IEEE Transactions on Information Technology in Biomedicine, 14(2):436 446, 2009.
Sabyasachi Basu and Martin Meckesheimer. Automatic outlier detection for time series: an application to sensor data. Knowledge and Information Systems, 11:137-154, 2007.
Mohammed Ayalew Belay, Sindre Stenen Blakseth, Adil Rasheed, and Pierluigi Salvo Rossi. Unsupervised anomaly detection for loT-based multivariate time scries: existing solutions, performance analysis and future directions. Sensors, (3(5):2844, 2023.
Yuxuan Bian, Xuan Ju, Jiangtong Li, Zhijian Xu, Dawei Cheng, and Qiang Xu. Multi-patch prediction: adapting language models for time serics representation learning. In International Conference on Machine Learning, pp. 3889 118993912, 2024.
Ane Garcia,, Angel Conde, Usue Mori, and Jose A Lozano. A review on outlier/anomal, detection in time series data. ACM Computing Surveys (CSUR), 54(3): 33, 2021.
Paul Boniol, Qinghua Liu, Mingyi Huang, Themis Palpanas, and John Paparrizos. Dive into time-series anomaly detection: a decade review. arXiv preprint ar 22. 2024.
Mohammad Braei and Sebastian Wagner. Anomaly detection in univariate lime-series: survey on the state-of-the-art, arXiv preprint arXie:2004.00430 2020.
Markus M Breunig, Hans-Pete Kriegel, Raymond T Ng, and Jörg Sander. LOF: identifying density-based local outliers. ACM SIGMOD International Conference on Management of Data, 1922 2000.
Canadian Institute for Cybersecurity. CIC-IDS-2017 Dataset. 174 htt.:/..1.44....../011aaaatttt 2017. Accessed: 2025-05-10.
Luis M Candanedo and Véronique Feldheim. Accurate occupancy detection of an office room from light, temperature, humidity and CO2 measurements using statistical learning models. Energy and Buildings, 112:28 39, 2016.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: survey, arXiv preprint arXiv:1901.08407, 2019.
Varun Chandola, Arindam Banerjec, and Vipin Kumar. Anomaly detection: a survey. ACM Computing Surveys (CSUR), 41(:):1 58, 2009.
Sourav Chatterjee, Rohan Bopardikar, Marius Guerard, Uttam Thakore, and Xiaodong Jiang. Mospat: automl based model selection and parameter tuning for time series anomaly detection. ar preprint ar 211:2205.11755, 2022.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Lucas Correia, Jan-Christoph Goos, Philipp Klein, Thomas Bäck, and Anna V Kononova. Online model-based anomaly detection in multivariate time series: taxonomy, survey, research challenges and future directions. Engineering Applications of Artificial Intelligence, 138:109323, 2024.
Andrea Dal Pozzolo, Giacomo Boracchi, Olivier Caelen, Cesare Alippi, and Gianluca Bontempi. Credit card fraud detection: a realistic modeling and a novel learning strategy. IEEE Transactions on Newral Networks and Learning Systems, 29(8):3784 3797, 2018.
Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Hu Yanping, Hu Bing, Nurjahan Begum, Authony Bagnall, Abdullah Mueen, Gustavo Batista, and Hexagon-ML. The UCR time series classification archive. https://www ucr. www.eedd/eamonnttteee time/ teaeameeettmeeeeeeeeeeeeeeeeeeee series data _a18/ 2018.
Emadeldeen Fldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, and Xiaoli Li. Time-series representation learning via temporal and contextual contrasting, In International Joint Conference on Artificial Intelligence (IJCAI), pp. 2352-2359, 2021.
Pavel Filonov, Andrey Lavrentyev, and Artem Vorontsov. Multivariate industrial time series with cyber-attack simulation: fault detection using an lstm-based predictive data model. Xiv preprint orXi::1622.060 06676, 2016.
Cynthia Freeman and Tan Beaver. Human-in--th-loop selection of optimal time series anomaly detection methods. In AAAI Conference on Human Computation and Crowdsoureing (HCOMP), 2019.
Cynthia Freeman, Jonathan Merriman, Ian Beaver, and Abdullah Mueen. Experimental comparison and survey of twelve time series anomaly detection algorithms. Journal of Artificial Intelligence Research, 72: 849 899, 2021.
Nicolo Fusi, Rishit Sheth, and Melih Elibol. Probabilistic matrix factorization for automated machine learning. In Advances in Neural Information Processing Systems (NewrIPS), 2018.
Astha Garg, Wenyn Zhang, Jules Samaran, Ramasamy Savitha, and Chuan-Sheng Foo. An evaluation of anomaly detection and diagnosis in multivariate time series. IEEE Transactions on Neural Networks and Learning Systems, 33(6):25082255
2021.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Kang Peng, and H Eugene Stanley. PhysioBank, PhysioToolkit, and PhysioNet: components of a new rescarch resource for complex physiologic signals. circulation, 101 :23): e220, 2000.
Markus Goldstein and Andreas Dengel. Hlstoggaam-based outlier score (HBOS): fast unsupervised anomaly detection algorithm. In German Conference on Artificial Intelligence, pp. 59. Citeseer, 2012.
Mononito Goswami, Cristian Ignacio Challu, Laurent Callot, Lenon Minories, and Andrey Kan. Unsupervised model selection for time series anomaly detection. In International Conference Learning Representations, 2021.
Scott David Greenwald, Ramesh S Patil, and Roger G Mark. Improved detection and classification of arrhythmias in noise-corrupted electrocardiograms using contextual information. In Computers in Cardiology, pp. 461-464, 1990.
Sahand Hariri, Matias Carrasco Kind, and Robert J Brunner. Extended isolation forest. IEFE Transactions on Knowledge and Data Engineering, 33(4):1479 1489, 2019.
Zengyou He, Xiaofei Xu, and Shengchun Deng. Discovering based local outliers. Pattern Recognition Letters, 24(9-10):1641 1650, 2003.
Nikolai Helwig, Elisco Pignanelli, and Andreas Schütze. Condition monitoring of a complex hydraulic system using multivariate statistics. In IEEE International Instrumentation and Measurement Technology Conference (12MTC), pp. 210-211 215, 2015.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Thi Kieu Khanh Ho, Ali Karami, and Narges Armanfard. Graph anomaly detection in time series: survey. IEEF Transactions on Pattern Analysis and Machine Intelligence, 2025.
Jordan Hochenbaum, Owen S Vallis, and Arun Kejariwal. Automatic anomaly detection in the cloud via statistical learning. arXiv preprint arXio: 0ao:10040077000 2017.
Alexis Huet, Jose Manuel Navarro, and Dario Rossi. Local evaluation of time series anomaly detection algorithms. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.
Kyle Hundman, Valentino Constantinou, Christopher Laporte, lan Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using istms and nonparametric dynamic thresholding. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2018.
Jon Hutchins. Callt2 building people counts. UCI Machine Learning Repository, 2006.
Alexander Thler, Jon Hutchins, and Padhraic Smyth. Adaptive event detection with time-varying poisson processes. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2006.
Vincent Jacob, Fei Song, Arnaud Stiegler, Bijan Rad, Yanlei Diao, and Nesime Tatbul. Exathlom: benchmark for explainable anomaly detection over time series. VLDB Endoument, 14(11):261322226 2021.
Ming Jin, Huan Yee Koh, Qingsong Wen, Daniele Zambon, Cesare Alippi, Geoffrey I Webb, Irwin King, and Shirui Pan. A survey graph neural networks for time series: forecasting, classification, imputation, and anomaly detection. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2024a.
Ming Jin, Yifan Zhang, Wei Chen, Kexin Zhang, Yuxuan Liang, Bin Yang, Jindong Wang, Shirui Pan, and Qingsong Wen. Position: what can large language models tell us about time series analysis. In International Conference on Machine Learning, 2024b.
Deokwoo Jung, Nandini Ramanan, Mehrnaz Amjadi, Sankeerth Rao Karingula, Jake Taylor, and Clau- dionor Nuncs Coelho Jr. Time series anomaly detection with label-free model selection. arXiv preprint arXiv:2106.0777 07447773,,,, 2021.
Kwei-Herng Lai, Daochen Zha, Junjie Xu, Yue Zhao, Guanchu Wang, and Xia Hu. Revisiting time series outlier detection: definitions and benchmarks. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Nikolay Laptev, Saced Amizadeh, and Ian Flint. Generic and scalable framework for automated time-series anomaly detection. In ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2015.
Alexander Lavin and Subutai Ahmad. Evaluating real-time anomaly detection algorithms the numenta anomaly benchmark. In IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 38 44, 2015.
Zheng Li, Yue Zhao, Nicola Botta, Cezar lonescu, and Xiyang Hu. COPOD: copula- based outlier detection. In IEEE International Conference on Data Mining (ICDM), 2020.
Zhong Li, Yuxuan Zhu, and Matthijs Van Leouwen. A survey on explainable anomaly detection. ACM Trunsactions on Knowledge Discovery from Data, 18(1):1 54, 2023.
Fci Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In IEEE International Conference on Dala Mining (ICDM), 2008.
Qinghua Liu and John Paparrizos. The elephant in the room: towards reliable time-series anomaly detection benchmark. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Martin Q Ma, Yue Zhao, Xiaorong Zhang, and Leman Akoglu. A large-scale study unsupervised outlier model selection: do internal strategies suffice? arXiv preprint DxXt::2004.01222 2021.
Pankaj Malhotra, Lovekesh Vig, Gautam Shroff, Puncet Agarwal, et al. Long short term memory networks for anomaly detection in time series. In Esann, volume 2015, pp. 89, 2015.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Nesryne Mejri, Laura Lopez-Fuenteds Kankana Roy, Pavel Chernakov, Enjie Ghorbel, and Djamila Aouada. Unsupervised anomaly detection in time-series: an extensive evaluation and analysis of state-of-hhe-ar. methods. Expert Systems with Applications, pp. 124922, 2024
Mohsin Munir, Shoaib Ahmed Siddiqui, Andreas Dengel, and Sheraz Ahmed. DeepAnT: a deep learning approach for unsupervised anomaly detection in time series. IEEE Access, 2018.
Jose Manuel Navarro, Alexis Huet, and Dario Rossi. Meta-learning for fast model recommendation unsupervised multivariate time series anomaly detection. In International Conference on Automated Machine Learning. PMLR, 2023.
Randy Paffenroth, Kathleen Kay, and Les Servi. Robust pea for anomaly detection in cyber networks. arXiv preprint orXte:1801.01522 2018
John Paparrizos, Paul Boniol, Themis Palpanas, Ruey S Tsay, Aaron Elmore, and Michael J Franklin. Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection. VLDB Endowment, 15(11):277--2777 2022a.
John Paparrizos, Yuhao Kang, Paul Boniol, Ruey S Tsay, Themis Palpanas, and Michael J Franklin. TSB- UAD: an end-to-end benchmark suite for univariate time-serics anomaly detection. VLDB Endoument, 15 (8):1697-171) 2022b.
Sridhar Ramaswany, Rajeev Rastogi, and Kyuscok Shim. Efficient algorithms for mining outliers from large data sets. In ACM SIGMOD International Conference on Management of Dala, 2000.
Frederik Rehbach, Steffen Moritz, Sowmya Chandrasekaran, Margarita Rebolledo, Martina Friese, and Thomas Bartz-Beielstein. GECCO 2018 industrial challenge: monitoring of drinking-water quality, 2018.
Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Tukowiez, David Bannach, Gerald Pirkl, Aloís Ferscha, et al. Collecting complex activity datasets in highly rich networked sensor environments. In International Conference on Networked Sensing Systems (INSS),
pp. 233 240, 2010.
Peter Rousseeuw and Katrien Van Driessen. A fast algorithm for the minimum covariance determinant estimator. Technometrics, 41(3):212-22 1999.
Mayu Sakurada and Takehisa Yairi. Anomaly delection using auloencoders with nonlinear dimensionality reduction. In Machine Learning for Sensory Data Analysis, pp. 1-11, 2014.
Sebastian Schmidl, Phillip Wenig, and Thorsten Papenbrock. Anomaly detection in time series: a comprehen- sive evaluation. VLDB Endowment, 15(9):1779-117)) 2022.
Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe-Taylor, and John Platt. Support vector method for novelty detection. In Advances in Newral Information Processing Systems (NewrIPS), 1999.
Arnaldo Sgueglia, Andrea Di Sorbo, Corrado Aaron Visaggio, and Gerardo Canfora. A systematic literature review of iot time series anomaly detection solutions. Future Generation Computer Systems, 134:170 186, 2022.
Kamran Shaukat, Talha Mahboob Alam, Suhuai Luo, Shakir Shabbir, Ibrahim A Hameed, Jiaming Li, Syed Konain Abbas, and Umair Javed. A review of time-series anomaly detection techniques: a step to future perspectives. In Future of Information and Communication Conference (FICC), pp. 865-66 665-877. Springer, 2021.
Prabhant Singh and Joaquin Vanschoren. Meta- leannie for unsupervised outlier detection with optimal transport. arXiv preprint arXio:2211.00372 2022.
Jing Su, Chufeng Jiang. Xin Jin, Yuxin Qiao, Tingsong Xiao, Hongda Ma, Rong Wei, Zhi Jing, Jiajun Xu, and Junhong Lin. Large language models for forecasting and anomaly detection: a systematic literature review. Xve preprint or ar 003500, 2024.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2019.
Emmanouil Sylligardos, Paul Boniol, John Paparrizos, Panos Trahanias, and Themis Palpanas. Choose wisely: an extensive evaluation of model selection for anomaly detection in time series. VLDB Endowment, 16(11):3418-344) 3432, 2023.
Nesime Tatbul, Tae Jun Lee, Stan Zdonik, Mejbah Alam, and Justin Gottschlich. Precision and recall for time series. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Patara Trirat, Yooju Shin, Junhyeok Kang, Youngeun Nam, Jihye Na, Minyoung Bae, Joeun Kim, Byunghyun Kim, and Jae-Gil Lee. Universal time-series representation learning: a survey, arXiv preprint 2010::240.003777, 2024.
Shreshth Tuli, Giuliano Casale, and Nicholas Jennings. TranAD: deep transformer networks for anomaly detection in multivariate time scries data. VLDB Endoument, pp. 1201, 2022.
Hector Valenzuela, Madalina Fiterau, and Ananth Srivastava. Evaluation- time-series forecasting model selection via meta-learning. ACM Transactions on Knowledge Discovery from Data, 18(1):1-36, 2024.
Alexander Birgelen and Oliver Niggemann. Anomaly detection and localization for cyber-physical Modelling Approoches for Production
production systems with self-organizing maps. Systems to Raise Validatable Efficiency: Intelligent Melhods for the Factory of the Future, pp. 55-55--77 35571, 2018.
Dennis Wagner, Tobias Michels, Florian CF Schulz, Arjun Nair, Maja Rudolph, and Marius Kloft. TimeSeAD: benchmarking deep multivariate time-series anomaly detection. Transactions on Machine Learning Research, 2023.
Yujing Wang, Luoxin Xiong, Mingliang Zhang, Hui Xue, Qi Chen, Yaming Yang, Yunhai Tong, Congrui Huang, and Bixiong Xu. Heat-RL: online model selection for streaming time-series anomaly detection. In Conference on Lifelong Learning Agents. PMLR, 2022.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: survey. In International Joint Conference Artificial Intelligence (IJCAI), 2023.
Phillip Wenig, Sebastian Schmidl, and Thorsten Papeubrock. TimeEval: a benchmarking toolkit for time series anomaly detection algorithms. VLDB Endowment, 15(12):3678 3681, 2022.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. TimesNet: temporal 2d- variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
Renjie Wu and Eamonn Keogh. Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress. IEEE Transactions on Knowledge and Data Engineering, 35(3):2421 2429, 2023.
Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications. In The World Wide Web Conference, 2018.
Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long, Anomaly transformer: time series anomaly detection with association discrepancy. In International Conference on Learning Representations, 2022.
Zhijian Xu, Ailing Zeng, and Qiang Xu. FITS: modeling time series with 10k parameters. In International
Conference on Learning Representations, 2023.
Takchisa Yairi, Yoshikiyo Kato, and Koichi Hori. Fault detection by mining association rules from house- keeping data. In International Symposium on Artificial Intelligence, Robotics and Automation in Space (i-SAIRAS), 2001.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Peng Yan, Ahmed Abdulkadir, Paul-Philipp Luley, Matthias Rosenthal, Gerrit A Schatte, Benjamin F Grewe, and Thilo Stadelmann. A comprehensive survey of deep transfer learning for anomaly detection in industrial time series: methods, applications, and directions. IEEE Access, 2024.
Xue Yang, Xuejun Qi, and Xiaobo Zhou. Deep learning technologies for time series anomaly detection in hcalthcarc: A review. IEEE Access, 2023.
Jiexia Ye, Weiqi Zhang, Ke Yi, Yongzi Yu, Ziyue Li, Jia Li, and Fugee Tsung. survey of time series foundation models: generalizing time series representation with large language model. arXiv preprint arXiu:2405.02358, 2024.
Yuanxiang Ying, Juanyong Duan, Chunlei Wang, Yujing Wang, Congrui Huang, and Bixiong Xu. Automated model selection for time-series anomaly detection. arXiv preprint 2Xiv:2009.04390 2020.
Guangrong Yu, Qinsheng Yang, Yongjin Zhu, Shiwei Zhang, Baotai Wu, Shangdong Liu, and Yimu Ji. Unsupervised optimal anomaly detection model selection in power data. In IEEE China Automation Congress (CAC), pp. 5661 5666, 2022.
Zahra Zamanzadch Darban, Geoffrey I Webb, Shirui Pan, Charu Aggarwal, and Mahsa Salchi. Deep lcarning for time series anomaly detection: a survey. ACM Compuling Surveys (CSUR), 57(1):1 42, 2024.
Aoqian Zhang, Shuqing Deng, Dougping Cui, Ye Yuan, and Guoren Wang. An experimental evaluation of anomaly detection in time series. VLDB Endowment, (7(3):8834999 2023.
Jiuqi Elise Zhang, Di Wu, and Benoit Boulet. Time series anomaly delection via reinforcement learning-basee model selection. In IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), 2022a.
Ruyi Zhang, Yijie Wang, Hongzuo Xu, and Haifang Zhou. Factorization machine-based unsupervised model selection method. In IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2022b.
Yue Zhao, Ryan Rossi, and Leman Akoglu. Automatic unsupervised outlier model selection. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Yue Zhao, Sean Zhang, and Leman Akoglu. Toward unsupervised outlier model selection. In IEEE International Conference on Data Mining (ICDM), 2022.
Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. One fits all: power general time scries analysis by pretrained LM. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
19

---

Published in Transactions on Machine Learning Research (02/2026)
m mTSBench: Appendices
Table Contents:
Appendix
A
Categorization of anomaly detection methods.
Appendix
B
Dataset statistics, domains, and anomaly types, example MTS visualizations.
Appendix
Time series quality case studies.
Appendix
Description of model selection methods.
Appendix
Additional results the impact of anomaly length.
Appendix
Implementation ion details and anomaly detector runtimes.
Appendix
Discussion of limitations.
A
Anomaly Detectors
mTSBench includes anomaly detectors applicable to multivariate time series (MTS), as shown in Table A.1 spanning four major areas: outlier detection, classic machine learning (ML), deep learning, and large language models (LLMs). These can be grouped into six method families based their underlying detection strategies. Distance-based methods identify anomalies based on their proximity to cluster centers nearest neighbors. Distribution-basen methods model statistical propertios and detect deviations from expected distributions. Reconstruction-bass flag large reconstruction errors as anomalies. Tree-based methods rely on recursive partitioning of the feature space to isolate anomalies, leveraging the principle that anomalies are easier to separate due to their sparsity. Forccasting-bass basod methods predict future time-sorics values from prior data and detoct anomalics significant deviations from the forecast. Foundation model-based methods utilize pre-trained LLMs to perform anomaly detection.
methods learn compact representations of the input and
Table A.1: Overview of Anomaly Detection Methods in mTSBench, encompassing diverse range of techniques, including foundation models, distance-based lorecasting- and reconstrruction-base approaches, ensuring comprehensive representation of different methodological paradigms.
Learning
Anomaly Detection Method
Area
Method Family
Outlier
Distance
COPOD
Li ct al
2020
Outlier Detection
Distribution
Classscc
Tree
HBOS
Goldstein & Dengel 2012
Classic ML
Distance
Outlier Detection
Tree
Unsupervised
KMeansAD
Yuiri et al
2001
Classie ML
Distance
KNN Ranmswaryy
Classic ML
Distance
LOF
Brennig et al
2010
Outlier Detection
Distance
PCA Aggarwal
Classic ML
Reconstruction
RobustPCA
Patleuroth et al.
2018
Classie ML
Reconstruction
ALLM4TS Bian
LLM
Foundation Model
Transformer
Nu al
2022
Deep Learning
Reconstruction
AutoFacoder
Sakurada Yairi
2014
Deep Learning
Reconstruction
CNN Munir es ml.
2018
Deep Learning
Pororasting
Deep Learning
Reconstruction
FITS
Xu ot ml.
2023
Deep Learning
Porerasting
LSTMAD
Malhotza
Deep Learning
Forerasting
Semi-supervised
MCD
Rouseeure & Van Driessen
1989
Classic ML.
Reconstruction
OCSVM Schölkopf
Outlice Detection
Distribution
OFA
Zhon ot ul
2023
LLM
Foundation Model
OumiAnomuay
Deep Leuruing
Reconstruction
TimesNet
Wia et al 2020
Deep Learuing
Forecasting
TranAD
Deep Learning
Forecasting
USAD
Audibert et al
2020
Deep Learuing
Reconstruction
20

---

Published in Transactions on Machine Learning Research (02/2026)
Table B.1:
Overview of the 19 MTS datasets used in mTSBench. For each dataset, we record its domain, number of time series (#TS), dimensionality (#Di)), average time series length, number of anomalous points (# Anommtts) and anomalous sequences (# AnomSeqs) per time series, and license.
Dataset
Domaia
#18
#Din
Length
#AnomPts
#nnomSoqs
License
CIC-IDS-2017
Canadian Institate for Cybemocarity
2017
Cybersecudity
73
>00K
D-S056
0-2646
Citation Requized
Calle2 Hutchius
2006
Smact Buikding
1
2
>.k
21
CC BY 4.0 Citation Requized
CreditCard
Unl Poczolo et
2018
Finanes Fouud Detection
1
29
DIIK
219
10
Daphnet
Bachia et al
2009
Heahthaaee
26
>-50K
1-16
CC DY 4.0
Exathlon
Jarob 05 31
2011
IT Infrastructuree
30
20
>-0K
0-4
0-6
Apache 2.0 Citation Required
GECCO
Rebhoch et d 2016
Irchusrria Procse
>50K
37
GHI, F Toooo
ml
X010
Indusorial Prucess
14
16
> LOOK
1-4
Contaer Authons
Genesis 10p Bargelica & Nageman
2018
Indusorial Prucess
1
18
>
2
NC 4.0
GulenTAG (Shhmiil et ml.
2022
Synthetic Benewark
30
20
> SOX
1
MET OnC-Ry v1.0 BSD 3-Clome CD BY 4.0
MITDB Goldiiiryur en al.
3000
Hoalthram
47
2
>00KK
1-720
MSL (Fundman et al
2016
Spacecraft Telemetry
26
55
>SK
1-3
Metro Heiwig et al
2016
Trazsportation
>0K
20
5
OPPORTUNITY Roggen of
2010
Humon Activity Recogaition
13
32
>26K
OD BY 4.0
Oompancc (Candordd & Roklhnim 2016
Smart Buikling
2
6
>SK
1.3
9 13
OD BY 4.0
PSM
Abdulnal et al.
20111
Infrastructure
26
>-0K
39
CC BY 4.0
SMAP Hundmon et
2018
Spacecraf: Telemstey
4H
25
1-3
BSD & C-Case
SMD Su mll
2019
Infrastructure
18
38
>10K
4-24
MIT
SVDB (Greuvald a)
1980
Healthrare
78
2
>00K
2-678
ODC By v1.0
SWAN-SP
Angryk ct a.
2020
Astropigsiss
38
>-0K
5233
1382
MIT
B Time Series Datasets
As summarized in Table B.1 m'SSeench contains 344 multivariate time series from 19 datasets, covering range of domains, anomaly types, and time series characteristics to ensure diverse and representative benchmark. In Figure B.1 we visualize four example time series to illustrate the variability in sequence length and anomaly patterns. For example, some time series (e.g-, METRO) contain short and rare anomalies, while others (e.g., CALIT2 and OCCUpANCY) exhibit. longer or more frequent anomalous segments. Brief descriptions of the 19 datasets are provided below.
CIC-IDS-2017 (Canadian Institute for Cybersecurity 2017) contains network traffic data, including benign behavior and wide range of attack scenarios such as DDoS, bruto-force, and infiltration. This dataset provides labeled flow-based features extracted from raw PCAP files, enabling evaluation of intrusion detection systems.
CalIt2 (Thler et al. 2006) consists of data recording the number of people entering and exiting the main door the Call12 building at UCI over 15 weeks, 48 time slices per day. The goal is to detect events, such as conferences, indicated by unusually high people counts during specific days or time periods.
CreditCard (Dal Pozzolo et al. 2018) contains anonymized variables derived from subset of 284,807 online credit card transactions (including 492 frandulent transactions) that occurred during two days in September 2013. Due to extreme class imbalance, this dataset is well-suited for evaluating anomaly detection methods in high-stakes, imbalanced classification scenarios.
Daphnet (Bachlin et al., 2009) consists of annotated readings from three acceleration sensors placed on the hip and leg of Parkinson's discase patients who experience freezing of gait (FoG) during walking tasks, such as straight line walking and walking with numerous turns.
Exathion (Jacob et al.) 2021) is benchmark for explainable anomaly detection in dighesimensional time series, based on real Apache Spark stream processing traces. Executions were intentionally disturbed by introducing six types of anomalous events (bursty input, bursty input until crash, stalled input, CPU contention, driver failure, and executor failure).
GECCO (Rchbach et al. 2018) was originally introducod in the GECCO Challenge 2017. This datasct contains sensor readings from water treatment plants and distribution systems, capturing parameters such as water quality, flow rates, and pressure.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Mirtt
Al
CaLlT2
GECCO
w
OCCUPANCY
METRO
Figure B.1: Examples of MTS Segments in mTSSench spanning diverse domains and temporal patterns. From top left to bottom right: CALLIT2 (smart building sensors), GECCO (water quality monitoring), OCCUPANCY (smart building sensors), and METRO (transportation systems). Shaded red regions indicate ground-truth anomalies, highlighting the variability in anomaly characteristics across datasets.
GHL (Filonov et al. 2016 records the operational status of three reservoirs, including variables such temperature and water level. Anomalies correspond to shifts in maximum temperature or pump frequency.
Genesis (yon Birgelen & Niggemann 2018 contains sensor readings collected from a portable pick-and- -alaccc demonstrator, a cyber-physical system that sorts two different materials (conductive and non-conductive) from magazine into their corresponding target. locations. The dataset contains five primary continuous signals, thirteen discrete signals, and single Unix timestamp. Two out of 42 production cycles contain anomalous behavior.
GutenTAG is synthetic anomaly detection benchmark dataset. We adapted the implementation from (Wenig et al., 2022) to create a customized version of the original Guten dataset, consisting of 30 multivariate time series, each comprising 20 dimensions. The dataset is partitioned into training, validation, and testing sets with shapes (1000, 20), (4000, 20), and (10000, 20), respectively. The underlying signals are derived from six base oscillation types: CBF, ECG, Random Mode Jump, Sawtooth, Dirichlet, and MLS. Each base type is used in 100 dimensions, resulting total of 600 unique dimensions across the dataset. Anomalies are drawn from 10 types: amplitude, extremum, frequency, mean, pattern, pattern_shit platform, trend, variance, and mode-correlation and injected into randomly chosen subset of 1, 2, 5, 10, all 20 dimensions for each time series. Each anomaly-affected dimension contains to 3 anomalies of length 50 to 200 points, located randomly between 10% and 90% of the time series. The type of anomaly is selected randomly from the set of types compatible with the base oscillation in that dimension.
MITDB (Goldberger et al.) 2000) consists of 48 half-hour excerpts of two-channel ambulatory ECG recordings collected from 47 individuals by the BIH Arrhythmia Laboratory during the years 1975 to 1979.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Note that the time series in this dataset contain trivial anomalies that are similar to one another, and the performance of detectors on MITDB should be interpreted with caution.
MSL (Hundman et al.
2018) originates from NASA's Mars Science Laboratory mission, and includes telemetry data from the Curiosity rover.
Metro (Helwig et al. 2015) contains hourly traffic volume data from a highway in Minncapolis-S9 St Paul, collected by single loop detector over several years, and includes features such weather conditions, time-based attributes, and traffic volume counts.
OPPORTUNITY (Roggen et al. 2010) is a benchmark datasct for human activity recognition, encompassing tasks such as classification,, segmentation, sensor fusion, and feature extraction. The dataset consists of motion sensor recordings collected as users performed routine daily activities.
Occupancy (Candanodo & Feldheim 2016) consists of indoor environmental sensor readings, including lemperature, humidity, light, and CO2 levels, collected from monitored room to predict occupancy stalus. The data is labeled and time-stamped, supporting tasks such as binary classification, energy efficiency modeling, and real-time occupancy prediction.
PSM (Abdulaal et al.
2021 contains data collected over 21 weeks from multiple application server nodes at eBay. It includes 26 variables that describe server machine metrics such as CPU utilization and memory.
SMAP (Hundman et al. 2018) originates from NASA's Soil Moisture Active Passive satellite mission, and con- tains telemetry data collected from the satellite's ' sensors, capturing various operational aspects of the satellite.
SMD (Su et al. 2019) is composed of time scries data collected over fivo-weck period from server machines in data center. This dataset captures metrics such as CPU usage, memory consumption, disk I/O, and network traffic, making it representative dataset for monitoring IT infrastructure systems.
SVDB (Greenwald et al. 1990) contains 78 half-hour ECG recordings featuring a combination of supraven- tricular and vicull ectopic beats within a normal sinus rhythm. Similar to MITDB, detector performance on this dataset should be interpreted with caution.
SWAN-SF (Angryk et al. 2020 comprises multivariate time series data derived from solar photospheric vector magnelograms in the SHARP data series, designed for space weather analylics and solar flare prediction.
C Time Series Data Quality Case Studies
Dataset quality plays a critical role in the reliable evaluation of anomaly detection methods. Inaccurate or inconsistent labels, such as normal points mislabeled as anomalies or vice versa, can systematically distort evaluation results, penalizing detectors that correctly identify true anomalics while favoring methods that produce random or misaligned predictions. Consequently, detector performance measured on low-quality datasets may not reflect the true capabilities of the underlying models. To mitigate the influence of such noise, m TSBenc includes a large and diverse collection of 344 labeled time series, reducing the impact of individual low-quality instances and enabling more robust, aggregate performance analysis across datasets.
In addition, to better understand how time-series characteristics and labeling quality affect anomaly detection, Figure C.I contrasts datasets with ambiguous versus well-justified anomaly annotations. In EXATHILON and GHL, the labeled anomalous segments exhibit limited visual deviation from surrounding behavior, making it difficult to identify clear temporal or cross-dimensional patterns that justify their annotation. Such ambiguity suggests the presence of label noise or dataset-specffic annotation criteria, which may penalize detectors that capture meaningful but unlabeled deviations. In contrast, GUTENTAG and DAPHNET exhibit anomalies that align with clear and interpretable signal changes visible in the plots. In GUTENTAG: the anomalous interval in the ninth dimension (counting from top) shows a marked incrcase in oscillation frequency relative to both preceding and subsequent. segments. Similarly, in DAPHNET, the labeled anomalies correspond to pronounced changes in frequency and fluctuation patterns compared to normal behavior in multiple dimensions. These examples illustrate how differences in label quality and anomaly salience can substantially influence detector performance and underscore the importance of careful dataset curation when benchmarking anomaly detection and model selection methods.
23

---

Published in Transactions on Machine Learning Research (02/2026)
EXATHLON
GHL
GUTENTAG
DAPHNET
Figure C.1: Representative time series from mTSBench highlighting differences in label quality (red shaded regions denote ground-truth anomalics). In EXATHLON and GHL, anomalous segments lack clear visual justification. In contrast, anomalies in GUTENTAG (ninth dimension) and DAPHNET correspond to pronounced changes in frequency and fluctuation patterns.
Derritromaly
PCA
Ground Louch
DAPHNET
ioomst
CRN
071
Ground
EXATHLON
Figure C.2: Ground Truth versus Predicted Anomalies. Tn DAPHNET (top), predictions from all five detectors overlap with the ground-truth anomaly interval. In EXATHLON (bottom), only the CNN detector exhibits partial overlap with the ground-truth anomaly.
We further illustrate the impact of labeling quality by comparing ground-truth anomalies with predictions from five representative detectors, PCA, OmniAnomaly, OFA, CNN, and IForest, Figure C.2 For DAPHNET, where the labeled anomalous interval aligns with clear signal changes in the plot, predictions from all five detectors overlap with the ground-truth anomaly. In addition, PCA, OmniAnomaly, and OFA produce predictions that closely match the annotated interval, whereas CNN and IForest generate anomalous regions that only partially overlap with the labeled boundaries and extend well outside the ground-truth anomaly. In contrast, for EXATHLON, where anomaly annotations are visually ambiguous, only CNN shows partial overlap with the labeled anomaly. However, this overlap is difficult to interpret, as CNN also predicts anomalies across much of the window, making it challenging to distinguish meaningful detections from spurious ones.
These examples illustrate how poor or ambiguous labeling can obscure differences between stronger and weaker detectors, limiting the interpretability of anomaly detection results. Improving label quality is itself
24

---

Published in Transactions on Machine Learning Research (02/2026)
challenging, the datasets span diverse domains and often require expert knowledge for reliable reamnotation. An alternative approach is to examine agreement across multiple detectors: when several met identify similar anomalous regions, as in DAPHNET, annotations are more likely to reflect true underlying irregularities. However, such an agreement is neither guaranteed nor sufficient, as it may reflect shared inductive biases among detectors or lavor specific anomaly characteristics. This motivates our use of mTSSeench which aggregates results over 344 time series, reducing the influence of a small number of unknown low-quality datasets and enabling more robust, large-scale evaluation.
D Model Selection Methods for Time Series Anomaly Detection
mTSBench contains three unsupervised model selection methods for MTS-AD, described below.
MetaOD (Zhao et al. 2021] is meta-learning framework that estimates model's performance on new dataset by leveraging knowledge from its prior performance on training datasets. MetaOD operates in two phases: offline training and online model selection. During offline training, it evaluates the models on all datasets, constructing a performance matrix that records each model's performance on each dataset. MetaOD extracts meta-fcatures from the datascts, capturing intrinsic properties such as statistical summarics and landmark features. To reduce dimenniinnllity, the extracted meta-features are processed through Principal Component Analysis (PCA), producing latent representations that initialize a dataset matrix. Simultaneously, model matrix is initialized with values from normal distribution, and matrix factorization is applied to approximate the relationship between datasets and models. This factorization is optimized using rank-discounted cumulative gain (DCG) objective to uncover latent interactions. After matrix factorization, regression model is trained to map the meta-features to the optimized latent representations of datasets. In the online phase, when a new dataset is introduced, its meta-features are extracted and reduced through PCA to generate latent representations. These representations are then passed through the trained regression model to compute the datasot's position in the latent space. By combining this latent representation with the precomputed model matrix, MetaOD estimates the performance of each model on the new dataset, and selects the model with the highest estimated performance score the most suitable for the task.
FMMS (Zhang et al. 2022b) (Factorization Machine-based Unsupervised Model Selection) employs fac- torization to transfer model performance on prior known datasets into a second-order regression function that describes the relationship between the dataset's meta-features and its performance matrix. FMMS incorporates the meta-feature matrix, the interactions between fcatures, and regression parameters to predict model performance. The regression function is optimized using cosine distance loss to ensure accurate predictions. During inference, mela-features of the test dataset are passed through the trained regression function to estimate model performance. The model with the highest predicted performance is selected.
Orthus Navarro et al.) 2023) extracts set of 22 univariate time series meta-features, summarized using five statistics: {min, first quartile, mean, third quartile, max). This results in a total of 110 meta- feature values. Orthus splits the model selection problem into two scenarios: when the test data is novel and when the lest data is similar to the training data. To determine the scenario, test data is projected in meta-feature space using Uniform Manifold Approximation and Projection (UMAP). If the test data belongs to cluster that does not contain any points from training data, it is considered novel, and recommendations are estimated via Regression, which applies Singular Value Decomposition to the performance matrix and fits a Random Forest regressor to predict the decomposed components from meta-features. During inference, the test data's meta-features are used to estimate performance across models, selecting the one with the highest predicted performance. Otherwise, if the test data is similar to the training data, UMAP is applicd to the performance matrix to form clusters, each with its own regressor. The test meta-features are evaluated across all cluster-specific regressors, and the top-performing model is selected.
E Impact of MTS Anomaly Length on Detection Performance
To investigate the impact of anomaly length on detection performance, we evaluate the models across time series with extreme anomaly durations. Specifically, we focus on two cases: (1) long anomalies, where the average anomaly length exceeds 2500 data points, and (2) short anomalies, where the average anomaly
25

---

Published in Transactions on Machine Learning Research (02/2026)
OmniA OmalAnomaly
ALLM4TS
Transformer
AutoBacoder
CBLOF
CNN
COPOD
Donut
EIF
FIT3
HBO5
Farast
KMeansAD
KNN
LOF
LSTMAD
MCD
OCSVM
OFA
PCA
RobustPCA
TimesNet
TranAD
USAD
Avg
44
66
.36
54
.60
39
42
53
.72
.37
43
47
CIC-IDS-201.3
55
51
86
.16
33
34
.20
.05
499
39
31
.17
36
45
45
79
52
22
.09
.8
63
26
17
.12
Exattslon, 1.,00000000000 67
16
1.0
Exathion 9 1000000, 66
87
85
.32
.35
88
66
.66
.75
.19
52
83
87
29
MITDB_20 207
.60
49
57
58
76
5.0
.65
56
.54
.7
72
.62
.00
.50
.35
56
.57
58
.8
.7
45
56
86
68
0.8
Time Series
MIIDB 234
83
J6
35
63
48
SMAP D-7
.08
.79
466
.80
.66
S1
.69
.67
66
84
.76
34
.7
.08
43
0.6
AUC-ROC
SMAP E-4
.20
50
440
.45
41
74
.14
.38
80
.35
42
.1
64
42
53
.07
.08
.R8
49
A6
60
0.4
SMAP E-3
.08
.31
.28
80
.65
.70
.11
.65
.71
.30
A6
49
38
84
.71
.12
.12
.22
21
.11
.72
.88
SMAP D-4
.13
80
.69
46
.7
.59
77
.47
A66
:00
64
56
J56
54
80
776
48
.52
.33
.00
:39
38
SMAP
775
0.2
SMAP A-9
.91
49
.11
.34
.72
.31
.76
.37
.30
.39
.03
47
.79
.61
.61
.11
.83
.64
97
30
.87
.86
SMAP A-8
.93
44
.27
400
.80
36
.26
44
.15
67
.88
.08
40
.14
77
770
93
82
83
0.00
SMAP D-2
.69
.56
.19
/2
.81
.73
800
.48
.80
.3
.47
800
.78
95
190
44
.30
.30
.6
.47
.79
.25
Figure E.1: AUC-ROC (t) of Anomaly Detectors (x-axis) Evaluated on MTS with Extremely Long Anomalies mTSBench (y-axis). Top row presents the average AUC-ROC over these MTS.
ALLM4TS
Transformer
AutoEacoden
CBLOF
CNN
COPOD
Donut
ETF
FITS
HBOS
IForsst
KMsansAD
KNN
LOF
LSTMAD
MCD
OCSVM
OFA
Omni OmnlAnomaly
Anomaly
PCA
RobustPCA
TimesNet
TranAD
USAD
Avg
51
.00
65
.63
61
.67
52
64
.68
69
.70
.78
Calite, Calltt
67
49
32
78
82
.77
39
.78
74
54
500
50
.89
550
75
.68
.82
82
87
72
61
.81
1.0
92
.91
92
Time Series
CIC-IDS-201722
56
499
.18
65
70
68
30
59
54
66
46
71
16
43
72
89
20
55
.2
52
77
52
60
60
0.8
CIC-IDS-2017.5
.57
35
.64
.64
CarditCard
63
.69
95
772
67
.55
69
.95
.81
770
.71
35
57
500
0.6
AUC-ROC
Emtthlon 5
1000000000000 91
.99
499
.48
-9
88
47
40
JS5
.77
29
.55
84
29
46
.07
48
56
.75
.73
84
29
83
.00
.98
Exathlea 2
.24
Gepesis
26
60
.66
53
91
61
400
.53
43
.39
89
96
59
71
.93
.73
23
.75
.64
64
42
64
.89
.92
0.2
Metro trautte-oolume
.79
.70
.81
MSL_5-2
.2
.99
.69
83
.70
.67
46
.60
.56
.67
.99
1.00
62
.02
.71
.47
.73
.96
.96
96
.44
90
.90
.89
SWAN SF
50
50
52
46
61
50
466
42
44
65
26
55
58
.77
49
61
65
87
50
.77
Figure F.2: AUC-ROC (t) of Anomaly Detectors (z-axis) Evaluated on MTS with Extremely Short Anomalies in mTSBench (y-axis). Top row presents the average AUC-ROC over these MTS.
length is loss than 20 data points. As shown in Figure E.1 scmi-supervised methods MCD and OCSVM achieve the highest average AUC-ROC on long anomalies, outperforming deep learning and LLM-based approaches, despite their simplicity. For short anomalies (Figure E.2 USAD performs best, likely due to architocture's scnsitivity to localized, transient deviations. These findings highlight the robustness of classic semi-supervised methods across anomaly duration extremes. Notably, LLs-b detectors do not exhibit clear advantages more traditional techniques in either regime.
26

---

Published in Transactions on Machine Learning Research (02/2026)
F Implementation Details
Anomaly Detection Methods. In real-world deployments, users often rely on off-the-shelf anomaly detectors without the time, supervision, or expertise required for data-specific tuning. Effective model selection is therefore critical: given a pool of detectors, the goal is to automatically choose the one likely to perform best for a new time scries. To reflect this practical constraint, we use default settings for all anomaly detectors, details of which can be found in the mTSSeench repository. Model selectors operate solely by ranking detector performance, ensuring that their contribution is isolated from detector optimization. This setup allows us to evaluate selectors under realistic, low-supervision conditions where anomaly characteristics may vary widely across domains. In practice, anomaly detection is often used in real-world unsupervised settings, where true labels are not available. In such cases, threshold tuning is impractical. To reflect. practical settings, we avoid method-specific threshold tuning and adopt a fixed quantile- based strategy: the top 7.5% of anomaly scores are labelod as anomalics, regardless of the detector or datasct. While this may not yicld the best absolute performance per method, it ensures consistent comparisons across detectors and datasets and reflects a realistic and fair evaluation scenario, especially for benchmarking unsupervised methods.
Model Selection Methods. MetaOD employs PCA to compute meta-features, with the number of components originally set to 3. However, for datasets with fewer than three dimensions (excluding label and timestamp); the number of components is adjusted accordingly. This method produces a ranked list of detectors. Similarly, FMMS generates a ranking of anomaly detection methods. FMMS leverages data and features from matrix factorization methods (Fusi et al., 2018); however, the code for computing these meta-features is not publicly available, nor there exist sufficient explanations of their computation process. To ensure consistency, we utilize the same meta-features MctaOD, with no other modifications to the FMMS method. Meta-features for MetaOD and FMMS are derived by extracting statistical properties of the datasets and characteristics from various outlier detection models, including structural features and outlier scores. They capture critical dataset characteristics to identify similar datasets, thereby enhancing model selection for anomaly and outlier detection tasks. In contrast, Orthus evaluates six detectors with various configurations and employs 22 univariate time series meta-features, which are summarized into five statistical values {min, Q1, mean, Q3, max} for each feature, resulting in a total of 110 meta-features. To accommodate smaller anomaly detection algorithm sets, we adjust the number of neighbors and components to lower values. Moreover, all model selection methods in mTBBench require performance matrix. Rather than constructing this matrix at the individual time series level, which assumes access to labeled data for every time series, we adopt a more practical approach by building the matrix at the dataset level. We argue that in realistic settings, obtaining labeled instances for every time series solely to build a performance matrix is impractical. Moreover, if such labels were available, more direct strategies for choosing detectors could be employed. Accordingly, we construct performance matrix of shape 19 24, representing 19 datascts and 24 detectors, for training the model selectors. Detector rankings used to evaluate model selection performance are derived from their VUS-PR. scores on the test set.
Computational Resources. All experiments were conducted on a high-performanee computing cluster using a single Intel CPU node with 36 cores at 2.3 GHz. Figure F.1 (vop) shows the average runtime of each anomaly detector over 344 multivariate time series in mSBchhch (for semi-supervised methods, the runtime includes both training and inference times). Among the 24 methods, IForest is the most efficient, followed by ALLM4TS and MCD, while EnomalyTransformer exhibits the highest computational cost, and OFA is the second most expensive. We further include a heatmap reporting the mean runtime of all detectors across the 19 datasets (Figure F.1 bottom). The results show no consistent correlation between dataset group and runtime efficiency. Instead, detector behavior is highly variable: method that is efficient on one dataset may incur substantially higher cost on another. For example, PCA records its longest. mean runtime on CIC-IDS-2017, where several other methods, e.g., IForest, remain comparatively efficient. Conversely, IForest's 's largest runtimes are concentrated on SWAN-SF, OCCUPANCY, and MITDB
G Limitations
While mTSeench represents the most comprehensive benchmark for MTS-AD and model selection to date, several limitations remain. The current benchmark focuses on selecting single best-performing detector
27

---

Published in Transactions on Machine Learning Research (02/2026)
Average
107
10a
10
Callt2
Daphnet
Exathlon
GECDO
GHL
10a
Genesis
GutenTAG
MITDB
Dataset
10a
Runtime
OPPORTUNITY
SMAP
SMD
SVDB
10a
IDS 2017
CrediCCard
Metro
Occupancy
SWAN-SF
104
IForest
ALLMATS
MCD
LOF
COPOD
Dozut
OCSVM
AutnEncador
HBOS
RobustPCA
CNV
USAD
ETF
TranAD
KNN
KMeansAD
FITS
OmnlAnomaly LSTMAD
CBLOF
PCA
TimestVet
Tranaformer
OPA
Figure F.l: Average Runtime (t) Anomaly Detectors (r-axis) over mTSBench Datasets (y-axis).
Top bars indicate per-detector average runtime over all 344 MTS in MTSBench.
per time series. Future work may explore ensemble-based selectors context-aware mixtures of experts. In addition, although the of default hyperparameter values mirrors practical, low-supervision deployments, it may underrepresent the potential of methods that benefit from tuning. Nevertheless, m m'TSBench provides an interface to modify detector configurations, enabling future work to explore the impact of tuning under more controlled settings. We also note that the datasets used in the experiments, like many publicly available time-series datasets used in anomaly detection studies, may be affected by issues documented in prior work (Wu Koogh, 2023), , such as mislabeled ground truth, unrcalistic anomaly density, trivial anomaly patterns, and run-to-failure bias. For example, some ECG datasets (e.g., MITDB and SVDB) contain multiple occurrences of the same arrhythmia class; however, individual events in typical ECG data. differ in morphology, amplitude, duration, and multivariate expression across loads. Other datasct problems include potential unlabeled anomalous segments (e.g., in MSL and Genesis), high- high-magnitude anomalies in certain channels (e-g., in PSM, SMD, and SMAP), and simplified patterns in some synthetic datasets. These properties reflect common challenges in anomaly annotation and are not unique to mTSBench. Users may select or exclude datasots based on their requirements and tolerance for labeling noise. We point out these potential issues to support the responsible use of mTSBeench and emphasize the need for future work on diverse, reliably annotated, and reprosentative datasets for MTS-AD. Lastly, while TSBench includes two LLM-based detectors, which, to the best of our knowledge, are the only publicly available such methods for MTS-AD, the space of foundation model-based approaches is rapidly evolving. Extending the benchmark to include fine-tuned LLMs or multimodal foundation models remains a promising direction for future research.
28


Reviews:
[{"review_id": "9TxuZZiVCN", "reviewer": "Reviewer_z1GP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces mTSBench, the largest benchmark to date for multivariate time series anomaly detection (MTS-AD) with explicit support for unsupervised model selection. The benchmark aggregates 344 labeled multivariate time series from 19 datasets across 12 domains and evaluates 24 anomaly detectors, spanning statistical, classical ML, deep learning, and two publicly available LLM-based methods.\n\nA key contribution is the first systematic benchmarking of unsupervised model selection methods for MTS-AD, comparing MetaOD, FMMS, and Orthus under a unified protocol against strong baselines (Oracle, Near-optimal, Random). The paper further introduces a comprehensive evaluation suite with 13 anomaly detection metrics and ranking-based selection metrics.\n\nEmpirically, the results show substantial performance variability across datasets and confirm that no detector or selector consistently dominates, highlighting a clear and well-quantified gap between current methods and optimal selection. Overall, the work provides a valuable, reproducible benchmark and identifies an important open problem in multivariate time series analysis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are supported by extensive experiments across a diverse benchmark, with clearly defined protocols, strong baselines, and multiple complementary metrics. Importantly, conclusions are framed relative to Oracle and Near-optimal references, avoiding overclaiming and making the evidence convincing and transparent.", "audience_yn": "Yes", "audience_explanation": "The paper is directly relevant to researchers in time series analysis, anomaly detection, model selection, and benchmarking. Its scale, methodological rigor, and inclusion of LLM-based detectors make it particularly timely and of broad interest to the TMLR community.", "requested_changes": "**Critical**\n1. **Clarify scope of conclusions about model selection**.\n\n    The paper should more explicitly state that the reported limitations apply to currently evaluated unsupervised selectors, not to supervised, domain-informed, or future adaptive approaches.\n\n**Non-Critical**\n1. **Strengthen failure-mode discussion**.\n\n    A brief synthesis explaining why selectors struggle (e.g., limited temporal meta-features, sensitivity to anomaly density or length) would improve interpretability.\n2. **Add short practical guidance**.\n\n    A concise paragraph summarizing implications for practitioners (e.g., when simple baselines outperform selectors, or when ranking is preferable to top-1 choice) would strengthen applied relevance.", "broader_impact_concerns": "The paper raises no significant ethical concerns. It focuses on benchmarking and evaluation rather than deployment. The Broader Impact Statement appropriately notes risks of false alarms or missed anomalies in high-stakes domains and emphasizes the need for robust validation, transparency, and domain awareness. No additional impact discussion is required.", "additional_comments": "This is a careful and well-executed benchmark paper that fills a clear gap in MTS-AD research. The experimental design is rigorous, the analysis is thorough, and the conclusions are appropriately scoped. With minor clarifications, the work meets TMLR’s standards for soundness, relevance, and reproducibility.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jF8JJ0zpWX", "reviewer": "Reviewer_gyyr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a new benchmark for anomaly detection of time series (i.e. detecting which points in a time series are potential anomalies). The benchmark is orignal for two reasons:\n- its scale (in terms of numbers of datasets, competitors, etc.): it is slightly larger than existing benchmarks,\n- they include a benchmark of model selection techniques, which are often overlooked in this context.\n\nThe authors also draw insights on anomaly detection for time series, and in particular highlight the importance of model selection and conclude that there is no absolute best method. An interesting conclusion of their benchmark is also that a simple PCA-based baseline is amongst the strongest competitors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims are that\n- the benchmark is the largest existing one, which appears to be true, although the scale is not dramatically larger than TSB-AD-M\n- the benchmark is the first one to systemically look at model selection. While this appears to be true for multivariate time series, I think that the authors should acknowledge more clearly that such bechmarks exist for univariate time series.", "audience_yn": "Yes", "audience_explanation": "I think model selection in this context is a somewhat overlooked and important problem, thus this benchmark could be quite useful to the community. The benchmark contains nicely diverse datasets, methods, and evaluations.\n\nThe main two limitations are the use of default hyperparameters (model selection should precisely be useful to choose hyperparameters, so looking into this would seem very relevant) and the lack of a basic ensemble baseline.", "requested_changes": "These changes are ordered by importance. The first two are critical, and the last one would significantly improve the insights provided by the benchmark, but is less critical.\n\n1. In the \"related works\" section, the authors devote one subsection to \"Model Selection for Time Series\" and one to \"Anomaly Detection Benchmarks\". It would be important to indicate that the intersection between these two is non-empty for univariate time series, and includes at least a benchmark cited by the authors (Sylligardos et al., 2023). Discussing the relationship between the findings and design choices of the authors and this benchmark would improve the paper. I am not an expert of time series, and I am not aware of other benchmarks than Sylligardos et al. (2023) but if other benchmarks for model selection exist in the univariate case, they should be discussed as well.\n\n2. I think one of the main findings of the paper, that resonates with other works (in particular TSB-AD) is that simple methods like PCA are very strong baselines. This should be emphasized more. In particular, it would be interesting to discuss how PCA fares agains the model selection baselines.\n\n3. I think adding a basic ensembling baseline against model selection methods would be insightful. This is a very natural alternative to model selection, and it was done for instance in the univariate benchmark of Sylligardos et al. (2023).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RzljIECZ4w", "reviewer": "Reviewer_D2mp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper's primary contribution is the introduction of mTSBench, a benchmark designed to tackle the critical challenge of model selection in multivariate time series anomaly detection (MTS-AD). By systematically evaluating a large collection of 344 time series, the benchmark not only underscores the significant performance variability among existing detectors but also delivers the key finding that current model selection strategies are substantially suboptimal, revealing critical gaps in their ability to handle complex temporal dynamics. To catalyze future research, mTSBench provides a unified evaluation suite that establishes a foundation for reproducible work and motivates new directions in adaptive, context-aware model selection and the integration of foundation models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1.\tThe introduction of mTSBench addresses a critical and underexplored gap in the field: standardized benchmarking for model selection in multivariate time series anomaly detection (MTS-AD). Its scale (344 time series, 19 datasets, 24 detectors) and focus on model selection make it a comprehensive resource of its kind.\n2.\tThe empirical evaluation is exceptionally thorough. The use of 13 diverse evaluation metrics for anomaly detection and 3 for model selection, combined with comparisons against strong baselines (Oracle, Near-optimal, Random), provides a robust and multi-faceted assessment of performance.\n3.\tThe key results are clearly demonstrated: no single detector is universally superior, and more importantly, current model selection methods remain significantly sub-optimal. These findings provide a powerful motivation for future research, while the performance analysis across dimensions, anomaly lengths, and domains offers nuanced insights.\n4.\tThe paper is well-structured and clearly defines the problem, methodology, and evaluation criteria. The extensive appendices detailing datasets, detector implementations, and meta-feature extraction greatly enhance reproducibility and trust in the findings.", "audience_yn": "Yes", "audience_explanation": "The findings presented are highly relevant to the TMLR audience as they address the critical, real-world problem of model selection for multivariate time series anomaly detection, which is a major deployment bottleneck. The paper provides a timely shift in focus from proposing new detectors to solving the model selection challenge itself. It offers valuable early insights into emerging LLM-based methods and, by open-sourcing a comprehensive benchmark, establishes an essential foundation for future research in automated and adaptive machine learning for time series.", "requested_changes": "1.\tThe benchmark is currently restricted to three specific model selection methods (MetaOD, FMMS, Orthus). Please clarify why choose these three methods. Other promising paradigms, such as reinforcement learning-based selection or more sophisticated ensemble strategies (beyond selecting a single best model), are not explored. \n2.\tPlease describe the relationship and distinction between the three trivial baselines and the three model selection methods presented in Table 2. Additionally, the data for \"Oracle\" and \"Near-optimal\" are highlighted with a gray background. What does this formatting signify?", "broader_impact_concerns": "The paper's broader impact statement is well-considered, highlighting positive societal benefits through improved robustness in safety-critical domains and acknowledging risks like missed detections or false alarms from poorly-validated selectors. The benchmark itself mitigates these risks by exposing current limitations. A valuable addition would be to explicitly address the computational and environmental costs of model selection, particularly for LLM-based methods, and to encourage future work on efficient as well as accurate selectors.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9yUyIpcnWL", "reviewer": "Reviewer_Ma2w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors assemble some datasets and do a \"bake off\"", "claims_and_evidence_yn": "No", "claims_explanation": "The datasets used are deeply flawed, which \"poisons\" any conclusions.", "audience_yn": "Yes", "audience_explanation": "See main review.", "requested_changes": "I don't think this paper can be saved.\n\nProducing a single good multiD TSAD datasets would be worthy of a paper. But it seems like no one is able or willing to do it.", "broader_impact_concerns": "none", "additional_comments": "I appreciate that the paper required lots of effort.\n\nHowever, at the end of the day, it has assembled a bunch of datasets of dubious provenance, that are known to contain lots of false positive and false negatives, suffer from triviality, and some of the datasets have a huge problem with overcounting success (Not K anomalies, but one anomies repeated K times [a])  [b]'\n\nThe authors need to have a lot more introspection about the data, and edit out all the dubious ones (or at least warn the user, something like “this dataset may have up to 50% false negatives in the ground truth”.\n\n\n1)\tFor almost all of SVDB and MITDB, there is a huge overcounting problem. The data is labeled as having upto 720 anomalies. But it does not, it has a single anomaly, 720 times. For almost all these datasets, in each given dataset the anomaly is a arrhythmia, but it is the SAME arrhythmia again and again. If you can detect one, you can detect them all! There is a short video that explains this [a].\n\n2)\tFor SVDB and MITDB, where do the labels come from? The answer is that over 25 years ago, someone ran a simple beat labeling tool to label the anomalous beats.  But think about that, that means there is a simple, fast bit of software out there that can get 100% accuracy on these datasets! Does that say that these are too simple to be interesting?\n\n3)\tFor almost all of SVDB and MITDB, they yield to Wu’s one-liner argument [b]. Does that say that these are too simple to be interesting?\n\n4)\tMetro. This  is really a classification dataset. The labels are “US National holidays plus regional holiday, Minnesota State Fair” which are not really anomalous. Yet, an entire weekend with no traffic (presumably construction or a broken senseo) is not anomalous?!? There are no annotations for major car crashes, whiteout blizzards etc. This dataset is plagued with dozens of false positives and false negatives.\n\n5)\tGHL. This datasets is too trivial to be interesting. Plot LevcorrTempfaultseed199vars23(:,6) with the ground truth labels, now plot plot(zscore((LevcorrTempfaultseed199vars23(:,6)))>1.1)  , it produced the exact same labels.\n\n6)\tGutenTAG. These datasets is too trivial to be interesting. Take a look at  GutenTAG ecg-diff-count-2  , now plot( movstd(test(:,2),25)<1.5);,   It is an embarrassingly simple pair of anomalies.  This entire dataset is embarrassing, it should never have been published.\n\n7)\tMSL. This datasets is plagued by false negatives. Let's look at G-1. The only anomaly labeled in 4770 to 4890. However, 4270 to 4285 and 6880 to 6894 are anomalies too, they were missed by the human labeler.\n\n8)\tSDM is a joke, plot it with the labels a just look at it.\n\n9)\tFig B.1. Please us x-axis labels in such plots.\n\n10)\tIn this paper you compute “average AUC-ROC”, is that meaningful when the datasets have different sizes, default rates, difficulties etc?  No, it is clearly meaningless. I know other papers do it, but it is still meaningless.\n\n[a] https://www.youtube.com/watch?v=fR4vqwmALZM\n\n[b] Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress. IEEE Trans. Knowl. Data Eng. 35(3): 2421-2429 (2023)", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)