# Anticipation, earliness, alarm cardinality: A new metric for industrial time-series anomaly detection

Raphaël Dion, Mazen Alamir, Thibaut Le Magueresse

► To cite this version:

Raphaël Dion, Mazen Alamir, Thibaut Le Magueresse. Anticipation, earliness, alarm cardinality: A new metric for industrial time-series anomaly detection. SAFEPROCESS 2024 - 12th IFAC Symposium on Fault Detection, Supervision and Safety for Technical Processes, IFAC, Jun 2024, Ferrara, Italy. hal-04577634

HAL Id: hal-04577634

https://hal.science/hal-04577634

Submitted on 16 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Anticipation, earliness, alarm cardinality: A new metric for industrial time-series anomaly detection

Raphaël Dion*,** Mazen Alamir* Thibaut Le Magueresse**

* Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-Lab, F-38400 Saint Martin D'Hères, France, (e-mail: {raphael.dion,mazen.alamir}@grenoble-inp.fr).

** Amiral Technologies, F-38000 Grenoble, France, (e-mail: thibaut.lemagueresse@amiraltechnologies.com)

**Abstract:**

Among the increasingly adopted data-driven solutions in various industrial processes, Time-Series-based Anomaly Detection (TSAD) became a hot topic in the past few years. As the performance, complexity and quantity of related emerging algorithms have quickly risen, the need for an appropriate evaluation and ranking of their performance in the specific underlying industrial context becomes a crucial task. Unfortunately, the specific area of the TSAD metrics still lacks the appropriate handling of industrial specific issues, such as the temporally ordered nature of the data, the importance of early detection that might contradicts the available labels since viewed as false alarms rather than the welcomed and even searched for anticipation, and the a-synchrony between the operator's labeling granularity and the effective possible detectability of the faults over the operational context. In this paper, a new TSAD metric is proposed which has been built based on a list of industrial specifications that is comprehensively discussed in the paper. Moreover, an extensive benchmark of the common metrics is proposed over tailored use-cases where the industrial-oriented ranking is rather obvious for a standard operator while the existing metrics lead to totally different ranking. The results highlight the relevance of the proposed metric.

**Keywords:** fault detection, time-series, industry, performance metrics, ranking.

## 1. INTRODUCTION

With its recent Fourth Revolution [Schwab (2017)], the industry has seen the blossoming of the connected equipment and thus of the subsequent data generated [Raheem and Iqbal (2022)]. Useful for monitoring machines in real time, one of the most common forms of these data is the time-series [Schmidl et al. (2022), Kim et al. (2023)].

Likewise, there is a growing interest in Anomaly Detection [Chandola et al. (2009)], culminating in the joint Time-Series Anomaly Detection (TSAD)[Sørbø and Ruocco (2023), Le Magueresse and Le Gall (2023)]. Typical use cases include predictive maintenance, in various industrial devices such as wind turbines, satellites [Chen et al. (2021)] or furnaces [Pota et al. (2023)], fraud prevention and fault monitoring in broad domains such as finance, IT, medical or even civil engineering construction such as road structure [Chabi et al. (2023)].

Since Anomaly Detection is essential for the proper functioning of the monitored system for obvious reasons related to safety, availability and production quality to cite but few items, it becomes almost mandatory to have tools that establish ranking of the different possible dedicated algorithms [Han et al. (2022)] that pop-up here and there on an almost weekly basis. This task is obviously challenging.

The difficulty in designing a benchmark for algorithm dedicated to anomaly detection in industrial time-series stems various sources, namely

* The difficulties that are inherited from the ones that come with benchmarks' design in general [Tavallaee et al. (2010), Wu and Keogh (2020)] because of the famous No Free-Lunch theorem [Wolpert and Macready (1997)] which requires the benchmark to be very rich and representative of possibilities.

* The very nature of time-series that show specific properties compared to static ones [Lavin and Ahmad (2015), Park (2017)]

* The peculiarities of the industrial context in terms of requirements and labelling processes. These are discussed in details below.

To introduce their NAB (Numenta Anomaly Benchmark) metric, Lavin and Ahmad (2015) highlight the importance of defining metric requirements before establishing it. Furthermore, as this paper focuses on the industrial context and strives to take in account the inherent flaws of the labeling [Wu and Keogh (2020)], the need to establish the precise specifications is more important than anywhere else. Accordingly, a list of desired properties was coined from the NAB one, and the selected specifications are
# Anticipation, earliness, alarm cardinality: A new metric for industrial time-series anomaly detection

Raphaël Dion*,** Mazen Alamir* Thibaut Le Magueresse**

* Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-Lab, F-38400 Saint Martin D'Hères, France, (e-mail: {raphael.dion,mazen.alamir}@grenoble-inp.fr).

** Amiral Technologies, F-38000 Grenoble, France, (e-mail: thibaut.lemagueresse@amiraltechnologies.com)

**Abstract:**

Among the increasingly adopted data-driven solutions in various industrial processes, Time-Series-based Anomaly Detection (TSAD) became a hot topic in the past few years. As the performance, complexity and quantity of related emerging algorithms have quickly risen, the need for an appropriate evaluation and ranking of their performance in the specific underlying industrial context becomes a crucial task. Unfortunately, the specific area of the TSAD metrics still lacks the appropriate handling of industrial specific issues, such as the temporally ordered nature of the data, the importance of early detection that might contradicts the available labels since viewed as false alarms rather than the welcomed and even searched for anticipation, and the a-synchrony between the operator's labeling granularity and the effective possible detectability of the faults over the operational context. In this paper, a new TSAD metric is proposed which has been built based on a list of industrial specifications that is comprehensively discussed in the paper. Moreover, an extensive benchmark of the common metrics is proposed over tailored use-cases where the industrial-oriented ranking is rather obvious for a standard operator while the existing metrics lead to totally different ranking. The results highlight the relevance of the proposed metric.

**Keywords:** fault detection, time-series, industry, performance metrics, ranking.

## 1. INTRODUCTION

With its recent Fourth Revolution [Schwab (2017)], the industry has seen the blossoming of the connected equipment and thus of the subsequent data generated [Raheem and Iqbal (2022)]. Useful for monitoring machines in real time, one of the most common forms of these data is the time-series [Schmidl et al. (2022), Kim et al. (2023)].

Likewise, there is a growing interest in Anomaly Detection [Chandola et al. (2009)], culminating in the joint Time-Series Anomaly Detection (TSAD)[Sørbø and Ruocco (2023), Le Magueresse and Le Gall (2023)]. Typical use cases include predictive maintenance, in various industrial devices such as wind turbines, satellites [Chen et al. (2021)] or furnaces [Pota et al. (2023)], fraud prevention and fault monitoring in broad domains such as finance, IT, medical or even civil engineering construction such as road structure [Chabi et al. (2023)].

Since Anomaly Detection is essential for the proper functioning of the monitored system for obvious reasons related to safety, availability and production quality to cite but few items, it becomes almost mandatory to have tools that establish ranking of the different possible dedicated algorithms [Han et al. (2022)] that pop-up here and there on an almost weekly basis. This task is obviously challenging.

The difficulty in designing a benchmark for algorithm dedicated to anomaly detection in industrial time-series stems various sources, namely

* The difficulties that are inherited from the ones that come with benchmarks' design in general [Tavallaee et al. (2010), Wu and Keogh (2020)] because of the famous No Free-Lunch theorem [Wolpert and Macready (1997)] which requires the benchmark to be very rich and representative of possibilities.

* The very nature of time-series that show specific properties compared to static ones [Lavin and Ahmad (2015), Park (2017)]

* The peculiarities of the industrial context in terms of requirements and labelling processes. These are discussed in details below.

To introduce their NAB (Numenta Anomaly Benchmark) metric, Lavin and Ahmad (2015) highlight the importance of defining metric requirements before establishing it. Furthermore, as this paper focuses on the industrial context and strives to take in account the inherent flaws of the labeling [Wu and Keogh (2020)], the need to establish the precise specifications is more important than anywhere else. Accordingly, a list of desired properties was coined from the NAB one, and the selected specifications are
indicated in italics, namely, the numbering in italics refers to the NAB version (detailed in Lavin and Ahmad (2015)):

(1) Rewarding correct detection of anomalous events (i)

(2) Rewarding anticipation, i.e. consistent detection of anomalies before the labeling. (ii)

(3) Rewarding early detection, i.e. detecting at the start of the labeling. (ii)

(4) Penalizing false alarms. (iii)

(5) Penalizing the number of exceeding alarms, i.e. raising multiple single anomalies instead of ranges.

Some differences are noticeable. First, the specification (ii) has been divided in two sub-properties: *anticipation* and *early detection*, since one could value one without the other. The properties (iv) and (v) present in the NAB version, namely, the need to work with streaming data and the one of no parameter tuning, have been dropped, as the goal is to evaluate algorithms on a whole time-series dataset and not on streaming data. As for parameter tuning, our experience is that industrial practitioners almost uniformly admit the no free-lunch theorem and might accept few tuning to match the specificity of their problem.

The contribution of this work is two-fold, and can be summarized as follows:

* ✓ A new TSAD metric, the MTI (Merged Time-series Indicators) is proposed based on a list of specifications that are inspired from the required properties in an industrial context (as witnessed by the authors through the now rich AMIRAL TECHNOLOGIES industrial experience and use-case studies). This list browses through the concepts of anticipation, earliness, false positive and alarm cardinality.

* ✓ A list of tailored time-series is designed including specific typologies of faulty sequences such that a common sense understanding of the best anomaly detector's behaviour is indisputable. This induces an ideal ranking that a perfect metrics should lead to. Then the major TSAD metrics as well as the one proposed in this contribution are applied on the benchmark time-series in order to examine the corresponding ranking.

The remainder of this paper is organized as follows: section 2 presents an overview of the related works regarding evaluation metrics, section III proposes a novel metric for Time-Series Anomaly Detection, section IV provides a comparison between existing metrics and ours. Finally, section V concludes our work and gives some hints for further investigation.

## 2. RELATED WORKS

As the need to evaluate TSAD algorithms is not novel [Chandola et al. (2009)], there already are related works in this field [Kim et al. (2023), Sørbø and Ruocco (2023)], which will be presented through a simple taxonomy (see Fig. 1).

**Thresholding:** The classic output of a TSAD algorithm is a continuous score, which can be an estimation of the outlier probability or an unbound inlier space deviation. It is common to determine a threshold, i.e. a value acting

as a separation between predicted inliers and outliers: in this case, the output become binary. Some metrics, such as the AUC or the VUS [Paparrizos et al. (2022)] do not require such threshold, whereas others, such as F-Score or SPD [Doshi et al. (2022)] need one.

**Conventionality:** Most common TSAD evaluation metrics are based *exclusively* on the Precision and/or on the Recall. These two concepts are ranged between 0 and 1, with the Precision representing the portion of true anomalies among all raised novelties; and the Recall the portion of raised anomalies among all labeled anomalies. These two values can be computed thanks to the confusion matrix or using range-tailored estimations.

**Confusion Matrix:** This matrix is built point-by-point and typically does not convey temporal performances. It displays four values: the number of True Positive (TP), False Positive (FP), True Negative (TN) and False Negative (FN). Using this matrix, $Precision = \frac{TP}{TP+FP}$ and $Recall = \frac{TP}{TP+FN}$. It is worth highlighting that common metric components such as False Positive Rate (FPR) and Specificity can be obtained using these point-by-point definitions:

$$ 
\begin{aligned}
FPR &= \frac{FP}{FP+TN} = \frac{FP+TN}{FP+TN} - \frac{TN}{FP+TN} = 1 - \text{Recall}_{\text{inliers}}, \\
Specificity &= \frac{TN}{FP+TN} = \text{Recall}_{\text{inliers}},
\end{aligned}
 $$

with $Recall_{inliers}$ the Recall computed using the transposed confusion matrix with switched diagonal coefficients.

Finally, some works on a temporal computation of the Confusion Matrix is proposed with SoftED [Salles et al. (2023)], resulting in a continuous Confusion Matrix using the delay between ground truth and predictions.

**Range-based Precision and Recall:** To fully capture the temporal nature of TSAD predictions, some studies [Tatbul et al. (2018), Hwang et al. (2019), Huet et al. (2022), Garg et al. (2022)] mention the need to move away from the classic point-by-point definition of the Precision and Recall. While expressing the same concepts, these metrics view the predictions as ranges and not points. For example, the TaPR [Hwang et al. (2019)] focuses on the range overlap between ground truth and predictions, and the Affiliation [Huet et al. (2022)] considers each anomalous range independently.

**Non-conventionality:** In opposition to conventional metrics, non-conventional ones can be computed using *not exclusively* Precision or Recall (NAB [Lavin and Ahmad (2015)]), or *not at all* (WDD [Kovács et al. (2019)]). The metric proposed in this paper is non-conventional due to the fact that it combines Precision and Recall with other measures.

## 3. PROPOSED METRIC

The MTI, for Merged Time-series Indicators, is the weighted mean of four weakly-dependent component metrics: the Anticipation / Earliness, the Recall, the Masked Specificity, and the Alarms Cardinality. Its output value is ranged between 0 and 1, with 1 being the best. The computation of these four values will be detailed, with the use of the simple time-series example of Fig. 2 as a support for illustration.
indicated in italics, namely, the numbering in italics refers to the NAB version (detailed in Lavin and Ahmad (2015)):

(1) Rewarding correct detection of anomalous events (i)

(2) Rewarding anticipation, i.e. consistent detection of anomalies before the labeling. (ii)

(3) Rewarding early detection, i.e. detecting at the start of the labeling. (ii)

(4) Penalizing false alarms. (iii)

(5) Penalizing the number of exceeding alarms, i.e. raising multiple single anomalies instead of ranges.

Some differences are noticeable. First, the specification (ii) has been divided in two sub-properties: *anticipation* and *early detection*, since one could value one without the other. The properties (iv) and (v) present in the NAB version, namely, the need to work with streaming data and the one of no parameter tuning, have been dropped, as the goal is to evaluate algorithms on a whole time-series dataset and not on streaming data. As for parameter tuning, our experience is that industrial practitioners almost uniformly admit the no free-lunch theorem and might accept few tuning to match the specificity of their problem.

The contribution of this work is two-fold, and can be summarized as follows:

* ✓ A new TSAD metric, the *MTI* (Merged Time-series Indicators) is proposed based on a list of specifications that are inspired from the required properties in an industrial context (as witnessed by the authors through the now rich AMIRAL TECHNOLOGIES industrial experience and use-case studies). This list browses through the concepts of anticipation, earliness, false positive and alarm cardinality.

* ✓ A list of tailored time-series is designed including specific typologies of faulty sequences such that a common sense understanding of the best anomaly detector's behaviour is indisputable. This induces an ideal ranking that a perfect metrics should lead to. Then the major TSAD metrics as well as the one proposed in this contribution are applied on the benchmark time-series in order to examine the corresponding ranking.

The remainder of this paper is organized as follows: section 2 presents an overview of the related works regarding evaluation metrics, section III proposes a novel metric for Time-Series Anomaly Detection, section IV provides a comparison between existing metrics and ours. Finally, section V concludes our work and gives some hints for further investigation.

## 2. RELATED WORKS

As the need to evaluate TSAD algorithms is not novel [Chandola et al. (2009)], there already are related works in this field [Kim et al. (2023), Sørbø and Ruocco (2023)], which will be presented through a simple taxonomy (see Fig. 1).

**Thresholding:** The classic output of a TSAD algorithm is a continuous score, which can be an estimation of the outlier probability or an unbound inlier space deviation. It is common to determine a threshold, i.e. a value acting

as a separation between predicted inliers and outliers: in this case, the output become binary. Some metrics, such as the AUC or the VUS [Paparrizos et al. (2022)] do not require such threshold, whereas others, such as F-Score or SPD [Doshi et al. (2022)] need one.

**Conventionality:** Most common TSAD evaluation metrics are based *exclusively* on the Precision and/or on the Recall. These two concepts are ranged between 0 and 1, with the Precision representing the portion of true anomalies among all raised novelties; and the Recall the portion of raised anomalies among all labeled anomalies. These two values can be computed thanks to the confusion matrix or using range-tailored estimations.

**Confusion Matrix:** This matrix is built point-by-point and typically does not convey temporal performances. It displays four values: the number of True Positive (TP), False Positive (FP), True Negative (TN) and False Negative (FN). Using this matrix, $Precision = \frac{TP}{TP+FP}$ and $Recall = \frac{TP}{TP+FN}$. It is worth highlighting that common metric components such as False Positive Rate (FPR) and Specificity can be obtained using these point-by-point definitions:

$$ 
\begin{aligned}
FPR &= \frac{FP}{FP+TN} = \frac{FP+TN}{FP+TN} - \frac{TN}{FP+TN} = 1 - \text{Recall}_{\text{inliers}}, \\
Specificity &= \frac{TN}{FP+TN} = \text{Recall}_{\text{inliers}},
\end{aligned}
$$

with $Recall_{inliers}$ the Recall computed using the transposed confusion matrix with switched diagonal coefficients.

Finally, some works on a temporal computation of the Confusion Matrix is proposed with SoftED [Salles et al. (2023)], resulting in a continuous Confusion Matrix using the delay between ground truth and predictions.

**Range-based Precision and Recall:** To fully capture the temporal nature of TSAD predictions, some studies [Tatbul et al. (2018), Hwang et al. (2019), Huet et al. (2022), Garg et al. (2022)] mention the need to move away from the classic point-by-point definition of the Precision and Recall. While expressing the same concepts, these metrics view the predictions as ranges and not points. For example, the TaPR [Hwang et al. (2019)] focuses on the range overlap between ground truth and predictions, and the Affiliation [Huet et al. (2022)] considers each anomalous range independently.

**Non-conventionality:** In opposition to conventional metrics, non-conventional ones can be computed using *not exclusively* Precision or Recall (NAB [Lavin and Ahmad (2015)]), or *not at all* (WDD [Kovács et al. (2019)]). The metric proposed in this paper is non-conventional due to the fact that it combines Precision and Recall with other measures.

## 3. PROPOSED METRIC

The MTI, for *Merged Time-series Indicators*, is the *weighted mean* of four weakly-dependent component metrics: the *Anticipation / Earliness*, the *Recall*, the *Masked Specificity*, and the *Alarms Cardinality*. Its output value is ranged between 0 and 1, with 1 being the best. The computation of these four values will be detailed, with the use of the simple time-series example of Fig. 2 as a support for illustration.
<figure><img src="image_1.png" /><figcaption>Fig. 1. List of state-of-the-art metrics dedicated to TSAD and divided in two categories based on their conventionality. Other properties are highlighted, such as the reliance on the confusion matrix, the requirement of a threshold or the time awareness.</figcaption></figure>

<figure><img src="image_2.png" /><figcaption>Fig. 2. Example for the computation of the MTI. The top plot is the ground truth labeling, whereas the bottom one is a model output score with the associated predicted labels.</figcaption></figure>

## 3.1 Anticipation / Earliness

The Anticipation / Earliness component metric is computed on a restricted range of the labels, which is around the start of the anomaly. The first part of this range is the anticipation area computed using the anticipation time value provided by the user. It can be a fixed time, or a percentage of the anomaly duration. The second part is the earliness area, computed in the same fashion. In the Fig. 2, the anticipation time is of two time-units (ticks 3 and 4) and the earliness time is of one (tick 5).

Once the range is estimated, a weight function is mapped over the points as explained in the Table 1. The sigmoid function uses the range of [-6, 6] for computation, following the same decision as the inverse sigmoid of the TaPR [Hwang et al. (2019)].

As an example, on the time-series displayed on the Fig. 2, as both the anticipation area and the earliness area are predicted as outliers, the Anticipation / Earliness component value is 1.

<figure><img src="image_3.png" /><figcaption>Table 1. Summary of the different Anticipation / Earliness importance and the affiliated weight functions. The x-axis represents an example binary prediction, with the vertical dotted line marking the separation between the anticipation area (left) and the earliness area (right). The y-axis represents the weight mapped to each prediction.</figcaption></figure>

On a dataset with multiple anomalous ranges, this component metric is computed for each area and then averaged.
<figure><img src="image_2.png" /><figcaption>Fig. 1. List of state-of-the-art metrics dedicated to TSAD and divided in two categories based on their conventionality. Other properties are highlighted, such as the reliance on the confusion matrix, the requirement of a threshold or the time awareness.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2. Example for the computation of the MTI. The top plot is the ground truth labeling, whereas the bottom one is a model output score with the associated predicted labels.</figcaption></figure>

## 3.1 Anticipation / Earliness

The Anticipation / Earliness component metric is computed on a restricted range of the labels, which is around the start of the anomaly. The first part of this range is the anticipation area computed using the anticipation time value provided by the user. It can be a fixed time, or a percentage of the anomaly duration. The second part is the earliness area, computed in the same fashion. In the Fig. 2, the anticipation time is of two time-units (ticks 3 and 4) and the earliness time is of one (tick 5).

Once the range is estimated, a weight function is mapped over the points as explained in the Table 1. The sigmoid function uses the range of [-6, 6] for computation, following the same decision as the inverse sigmoid of the TaPR [Hwang et al. (2019)].

As an example, on the time-series displayed on the Fig. 2, as both the anticipation area and the earliness area are predicted as outliers, the Anticipation / Earliness component value is 1.

<figure><img src="image_4.png" /><figcaption>Table 1. Summary of the different Anticipation / Earliness importance and the affiliated weight functions. The x-axis represents an example binary prediction, with the vertical dotted line marking the separation between the anticipation area (left) and the earliness area (right). The y-axis represents the weight mapped to each prediction.</figcaption></figure>

On a dataset with multiple anomalous ranges, this component metric is computed for each area and then averaged.
Input parameters: anticipation / earliness importance, anticipation time, earliness time.

## 3.2 Recall

The Recall is a measure dedicated to evaluating an algorithm ability to correctly predict outliers. The MTI uses the traditional one by default but another Recall calculation can be selected, like a time specific one such as the TaPR [Hwang et al. (2019)] or the range Recall [Tatbul et al. (2018)].

As an example, the traditional Recall can be computed on the time-series displayed on the Fig. 2. Since the ground truth presents 4 outliers ($N_{outliers}$), and that 3 of them are correctly predicted ($N_{TP}$), the Recall value is of $\frac{N_{TP}}{N_{outliers}} = \frac{3}{4} = 0.75$.

Input parameter: Recall metric.

## 3.3 Masked Specificity

The Specificity is a measure dedicated to evaluating an algorithm ability to correctly predict inliers. To compute the Masked Specificity (or Masked True Negative Rate), it is necessary to discard the anticipation areas from the computation of specificity in order to avoid penalizing those models that are good enough to provide anticipation. Likewise, dynamical models such as industrial machinery can display an inertia delay, i.e. a time delay that occurs after the source of anomaly has been fixed by the operator and the associated label has been immediately switched to healthy value. This delay represents the necessary time to return to normality.

Once the anticipation-earliness-inertia-delay mask is applied on the ground truth and predictions, the Specificity is computed.

As an example, the traditional Specificity can be computed on the time-series displayed on the Fig. 2. Since the masked ground truth, i.e. avoiding the labels of the anticipation, earliness and inertia areas, presents 10 inliers ($N_{inliers}$), and that 8 of them are correctly predicted ($N_{TN}$), the Specificity value is of $\frac{N_{TN}}{N_{inliers}} = \frac{8}{10} = 0.8$.

Input parameter: inertia delay, Specificity metric.

## 3.4 Alarm Cardinality

To compute the Alarm Cardinality, two values are necessary: the number of labelled anomalous ranges $N_{gt}$ and the number of predicted anomalous ranges $N_{pred}$. In this count, a range is seen as strictly continuous timestamps. The value of this component is

$$ \text{Alarm Cardinality} := \min \left\{ \frac{N_{gt}}{N_{pred}}, \frac{N_{pred}}{N_{gt}} \right\} $$

with the idea to penalize the case of a higher number of alarms and the case of lesser number of alarms. As an example, the Alarm Cardinality can be computed on the time-series displayed on the Fig. 2. Since the ground truth presents 1 anomalous range ($N_{gt}$) and that the predictions present 4 ($N_{pred}$), this component value is $\min\{\frac{1}{4}, \frac{4}{1}\} = 0.25$.

Input parameter: None.

## 3.5 Summary of MTI computation

The output value of the MTI is the weighted average of its four components. As an example, on the time-series displayed on the Fig. 2, for uniform weights, it is $\frac{1+0.75+0.8+0.25}{4} = 0.7$.

Input parameters: anticipation/earliness weight, Recall weight, masked Specificity weight, alarm cardinality weight.

## 3.6 Application-specific setting

Following a concept introduced by the NAB metric [Lavin and Ahmad (2015)], the MTI proposes many different application profiles. Thanks to the self-explainable parameters, the MTI can be tuned to fit different specifications, from high level ones with components weights to low level ones such as the importance shift between anticipation and earliness.

As an example, using the time-series displayed on the Fig. 2, different profiles can be constructed:

* For an industrial case where only anticipation, earliness and limited false alarms are of importance, the Recall and Alarms Cardinal weights can be set to zero and thus the output score would be $\frac{1+0.8}{2} = 0.9$.

* Conversely, it might happen that anticipation is not desired, nor that too much alarms is a problem. In such case, the anticipation delay can be set to 0 and the Alarm Cardinality weight set to zero, which will output a score of $\frac{1+0.75+0.8}{3} = 0.85$.

# 4. RANKING EVALUATION

## 4.1 Experimental methodology

One of the main applications of an TSAD metric is to systematically rank, various algorithms accordingly to their quality expressed as the degree of conformity with the user's expectation. To evaluate a metric capacity to fulfill this task, this paper introduces the use of similarity score. These metrics take as input a ground truth ranking and a predicted ranking, to output a score between them indicating their likeness. Several similarity scores exist, such as the Jaccard Similarity or the Kendall Tau, and while each of these has many qualities, this study will focus on the Rank-Biased Overlap (RBO) [Webber et al. (2010)]. The main motive to use the RBO in this analysis is its ability to focus either on the head of the ranking list or on the whole ranking list, depending on an input parameter. This allows to evaluate a metric ability to globally rank the algorithms while assessing its capacity to precisely order the best ones.

As a support for this study, 26 synthetic scores have been generated, spread in four datasets (see Fig. 3, 4). Each dataset presents different kind of generic outlier behavior, and the linked scores have been ranked following an industrial-inspired specifications list. As a reflection of the list in section 1, the desired properties of an ideal TSAD algorithm in this evaluation follow the same order.

As several TSAD metrics do need binary predictions to be computed, the methodology in this paper will be the one known as "best metric", i.e. each thresholding-locked
Input parameters: anticipation / earliness importance, anticipation time, earliness time.

## 3.2 Recall

The Recall is a measure dedicated to evaluating an algorithm ability to correctly predict outliers. The MTI uses the traditional one by default but another Recall calculation can be selected, like a time specific one such as the TaPR [Hwang et al. (2019)] or the range Recall [Tatbul et al. (2018)].

As an example, the traditional Recall can be computed on the time-series displayed on the Fig. 2. Since the ground truth presents 4 outliers ($N_{outliers}$), and that 3 of them are correctly predicted ($N_{TP}$), the Recall value is of $\frac{N_{TP}}{N_{outliers}} = \frac{3}{4} = 0.75$.

Input parameter: Recall metric.

## 3.3 Masked Specificity

The Specificity is a measure dedicated to evaluating an algorithm ability to correctly predict inliers. To compute the Masked Specificity (or Masked True Negative Rate), it is necessary to discard the anticipation areas from the computation of specificity in order to avoid penalizing those models that are good enough to provide anticipation. Likewise, dynamical models such as industrial machinery can display an inertia delay, i.e. a time delay that occurs after the source of anomaly has been fixed by the operator and the associated label has been immediately switched to healthy value. This delay represents the necessary time to return to normality.

Once the anticipation-earliness-inertia-delay mask is applied on the ground truth and predictions, the Specificity is computed.

As an example, the traditional Specificity can be computed on the time-series displayed on the Fig. 2. Since the masked ground truth, i.e. avoiding the labels of the anticipation, earliness and inertia areas, presents 10 inliers ($N_{inliers}$), and that 8 of them are correctly predicted ($N_{TN}$), the Specificity value is of $\frac{N_{TN}}{N_{inliers}} = \frac{8}{10} = 0.8$.

Input parameter: inertia delay, Specificity metric.

## 3.4 Alarm Cardinality

To compute the Alarm Cardinality, two values are necessary: the number of labelled anomalous ranges $N_{gt}$ and the number of predicted anomalous ranges $N_{pred}$. In this count, a range is seen as strictly continuous timestamps. The value of this component is

$$ \text{Alarm Cardinality} := \min \left\{ \frac{N_{gt}}{N_{pred}}, \frac{N_{pred}}{N_{gt}} \right\} $$

with the idea to penalize the case of a higher number of alarms and the case of lesser number of alarms. As an example, the Alarm Cardinality can be computed on the time-series displayed on the Fig. 2. Since the ground truth presents 1 anomalous range ($N_{gt}$) and that the predictions present 4 ($N_{pred}$), this component value is $\min\{\frac{1}{4}, \frac{4}{1}\} = 0.25$.

Input parameter: None.

## 3.5 Summary of MTI computation

The output value of the MTI is the weighted average of its four components. As an example, on the time-series displayed on the Fig. 2, for uniform weights, it is $\frac{1+0.75+0.8+0.25}{4} = 0.7$.

Input parameters: anticipation/earliness weight, Recall weight, masked Specificity weight, alarm cardinality weight.

## 3.6 Application-specific setting

Following a concept introduced by the NAB metric [Lavin and Ahmad (2015)], the MTI proposes many different application profiles. Thanks to the self-explainable parameters, the MTI can be tuned to fit different specifications, from high level ones with components weights to low level ones such as the importance shift between anticipation and earliness.

As an example, using the time-series displayed on the Fig. 2, different profiles can be constructed:

* For an industrial case where only anticipation, earliness and limited false alarms are of importance, the Recall and Alarms Cardinal weights can be set to zero and thus the output score would be $\frac{1+0.8}{2} = 0.9$.

* Conversely, it might happen that anticipation is not desired, nor that too much alarms is a problem. In such case, the anticipation delay can be set to 0 and the Alarm Cardinality weight set to zero, which will output a score of $\frac{1+0.75+0.8}{3} = 0.85$.

# 4. RANKING EVALUATION

## 4.1 Experimental methodology

One of the main applications of an TSAD metric is to systematically rank, various algorithms accordingly to their quality expressed as the degree of conformity with the user's expectation. To evaluate a metric capacity to fulfill this task, this paper introduces the use of similarity score. These metrics take as input a ground truth ranking and a predicted ranking, to output a score between them indicating their likeness. Several similarity scores exist, such as the Jaccard Similarity or the Kendall Tau, and while each of these has many qualities, this study will focus on the Rank-Biased Overlap (RBO) [Webber et al. (2010)]. The main motive to use the RBO in this analysis is its ability to focus either on the head of the ranking list or on the whole ranking list, depending on an input parameter. This allows to evaluate a metric ability to globally rank the algorithms while assessing its capacity to precisely order the best ones.

As a support for this study, 26 synthetic scores have been generated, spread in four datasets (see Fig. 3, 4). Each dataset presents different kind of generic outlier behavior, and the linked scores have been ranked following an industrial-inspired specifications list. As a reflection of the list in section 1, the desired properties of an ideal TSAD algorithm in this evaluation follow the same order.

As several TSAD metrics do need binary predictions to be computed, the methodology in this paper will be the one known as “best metric”, i.e. each thresholding-locked
<table><caption>Table 2. Example of RBO scores, using the Dataset A. The RBO is computed using the output ranking of the algorithms, which are compared to the ground truth ranking.</caption><thead><tr><th>Metrics</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>RBO@0.2 (@1)</th></tr></thead><tbody><tr><td>MTI</td><td>Anticip. + Early</td><td>Earliness</td><td>Too many alarms</td><td>Many alarms+FA</td><td>Random</td><td>1 (1)</td></tr><tr><td>F1</td><td>Earliness</td><td>Anticip. + Early</td><td>Too many alarms</td><td>Many alarms+FA</td><td>Random</td><td>0.200 (0.800)</td></tr><tr><td>NAB</td><td>Earliness</td><td>Many alarms</td><td>Anticip. + Early</td><td>Many alarms+FA</td><td>Random</td><td>0.120 (0.700)</td></tr><tr><td>Ground truth</td><td>Anticip. + Early</td><td>Earliness</td><td>Too many alarms</td><td>Many alarms+FA</td><td>Random</td><td></td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>Fig. 3. Synthetic datasets generated for the analysis. Plain green areas represent inliers, dashed red areas represent outliers. **For each dataset the ground truth ranking is top-down.** The Dataset A has been constructed as an obvious case for TSAD algorithms ranking, while the Dataset B ranking is subject to discussion. In this case, an exaggerated importance has been put on earliness and the absence of false alarms.</figcaption></figure>

metric will be computed through numerous threshold values and subsequent binary predictions, and the best value will be output.

In this study<fnref n="1" />, 100 thresholds have been used ranging from the 0.01 quantile of the evaluated score to its 0.99 quantiles. Furthermore, to replicate industrial concerns, only the thresholds where the *False Positive Rate (FPR)* does not exceed 0.1 have been kept.

The TSAD metrics chosen for this evaluation are the MTI, the traditional F1-Score, the AUC, the Affiliation F1-Score [Huet et al. (2022)], the eTaPR F1-Score, which is the most recent iteration of the TaPR [Hwang et al. (2019)], the Matthews Correlation Coefficient (MCC), the VUS [Paparrizos et al. (2022)] and the NAB Lavin and Ahmad (2015)]. These metrics were computed using the default parameters.

## 4.2 Results summary

To evaluate the TSAD metrics ranking qualities, two RBO parametrizations have been chosen: the RBO@0.2, to focus on the head, and the RBO@1, for a uniform attention.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Synthetic datasets generated for the analysis. For each dataset the ground truth ranking is top-down. The Dataset C focuses on anomalous areas vs anomalous single point, with its ranking emphasising on areas. The Dataset D follows the same ranking specifications of the B, but with numerous short anomalies.</figcaption></figure>

An example of a RBO computation is displayed on Table 2. A perfect ranking has a score of 1, while imperfect ones can be scored as low as 0. On this example, the F1 and NAB [Lavin and Ahmad (2015)] metrics display a better RBO@1 than at RBO@0.2 because, while the ranking list head is wrong, there is in the end only one replacement. The results can be found on the Table 3 and show that the MTI and the VUS [Paparrizos2022] appear to be the two most industry adapted TSAD metrics, with the MTI displaying a higher average value.

## 5. CONCLUSION

In this study, a crucial deficiency has been identified in the TSAD metrics: the lack of focus on industrial needs. To mend it, this paper proposed a new TSAD metric, the MTI, built from an industrial specifications list. Using similarity score and synthetic cases, this metrics have shown interesting performance when compared to its most known challengers.

While the MTI and the NAB share the ability to offer application profiles, this has not been deeply studied in

<footnote n="1">¹ Script and more detailed results will be made available on the paper GitHub at https://github.com/r-dion/MTI.</footnote>
<table><caption>Table 2. Example of RBO scores, using the Dataset A. The RBO is computed using the output ranking of the algorithms, which are compared to the ground truth ranking.</caption><thead><tr><th>Metrics</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>RBO@0.2 (@1)</th></tr></thead><tbody><tr><td>MTI</td><td>Anticip. + Early</td><td>Earliness</td><td>Too many alarms</td><td>Many alarms+FA</td><td>Random</td><td>1 (1)</td></tr><tr><td>F1</td><td>Earliness</td><td>Anticip. + Early</td><td>Too many alarms</td><td>Many alarms+FA</td><td>Random</td><td>0.200 (0.800)</td></tr><tr><td>NAB</td><td>Earliness</td><td>Many alarms</td><td>Anticip. + Early</td><td>Many alarms+FA</td><td>Random</td><td>0.120 (0.700)</td></tr><tr><td>Ground truth</td><td>Anticip. + Early</td><td>Earliness</td><td>Too many alarms</td><td>Many alarms+FA</td><td>Random</td><td></td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Fig. 3. Synthetic datasets generated for the analysis. Plain green areas represent inliers, dashed red areas represent outliers. **For each dataset the ground truth ranking is top-down.** The Dataset A has been constructed as an obvious case for TSAD algorithms ranking, while the Dataset B ranking is subject to discussion. In this case, an exaggerated importance has been put on earliness and the absence of false alarms.</figcaption></figure>

metric will be computed through numerous threshold values and subsequent binary predictions, and the best value will be output.

In this study<fnref n="1" />, 100 thresholds have been used ranging from the 0.01 quantile of the evaluated score to its 0.99 quantiles. Furthermore, to replicate industrial concerns, only the thresholds where the *False Positive Rate (FPR)* does not exceed 0.1 have been kept.

The TSAD metrics chosen for this evaluation are the MTI, the traditional F1-Score, the AUC, the Affiliation F1-Score [Huet et al. (2022)], the eTaPR F1-Score, which is the most recent iteration of the TaPR [Hwang et al. (2019)], the Matthews Correlation Coefficient (MCC), the VUS [Paparrizos et al. (2022)] and the NAB Lavin and Ahmad (2015)]. These metrics were computed using the default parameters.

## 4.2 Results summary

To evaluate the TSAD metrics ranking qualities, two RBO parametrizations have been chosen: the RBO@0.2, to focus on the head, and the RBO@1, for a uniform attention.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Synthetic datasets generated for the analysis. For each dataset the ground truth ranking is top-down. The Dataset C focuses on anomalous areas vs anomalous single point, with its ranking emphasising on areas. The Dataset D follows the same ranking specifications of the B, but with numerous short anomalies.</figcaption></figure>

An example of a RBO computation is displayed on Table 2. A perfect ranking has a score of 1, while imperfect ones can be scored as low as 0. On this example, the F1 and NAB [Lavin and Ahmad (2015)] metrics display a better RBO@1 than at RBO@0.2 because, while the ranking list head is wrong, there is in the end only one replacement. The results can be found on the Table 3 and show that the MTI and the VUS [Paparrizos2022] appear to be the two most industry adapted TSAD metrics, with the MTI displaying a higher average value.

## 5. CONCLUSION

In this study, a crucial deficiency has been identified in the TSAD metrics: the lack of focus on industrial needs. To mend it, this paper proposed a new TSAD metric, the MTI, built from an industrial specifications list. Using similarity score and synthetic cases, this metrics have shown interesting performance when compared to its most known challengers.

While the MTI and the NAB share the ability to offer application profiles, this has not been deeply studied in

<footnote n="1">¹ Script and more detailed results will be made available on the paper GitHub at https://github.com/r-dion/MTI.</footnote>
<table><caption>Table 3. RBO@0.2 similarity scores of different candidate metrics. The RBO@0.2 focuses on the ranking head. In parentheses are the result with RBO@1, which focuses on the whole ranking. A score close to 1 indicates a ranking highly similar to the ground truth, whereas one of 0 indicates a ranking nearly opposite to the ground truth.</caption><thead><tr><th>Datasets</th><th>MTI</th><th>F1</th><th>AUC</th><th>Affiliation</th><th>eTaPR</th><th>MCC</th><th>VUS</th><th>NAB</th></tr></thead><tbody><tr><td>Dataset A</td><td><b>1 (1)</b></td><td>0.20 (0.80)</td><td>0.20 (0.80)</td><td>0.20 (0.80)</td><td>0.20 (0.80)</td><td>0.20 (0.80)</td><td><b>1 (1)</b></td><td>0.12 (0.70)</td></tr><tr><td>Dataset B</td><td><b>1 (0.94)</b></td><td>0.19 (0.77)</td><td>0.19 (0.77)</td><td>0.19 (0.77)</td><td>0.200 (0.79)</td><td>0.19 (0.77)</td><td><b>0.99 (0.89)</b></td><td>0.11 (0.47)</td></tr><tr><td>Dataset C</td><td>0.99 (0.99)</td><td>0.20 (0.83)</td><td>0.20 (0.83)</td><td>0.19 (0.70)</td><td>0.19 (0.70)</td><td>0.20 (0.83)</td><td><b>1 (1)</b></td><td><b>1 (0.92)</b></td></tr><tr><td>Dataset D</td><td><b>1 (0.87)</b></td><td>0.11 (0.66)</td><td>0.11 (0.60)</td><td>0.11 (0.65)</td><td><b>0.19 (0.69)</b></td><td>0.11 (0.63)</td><td>0.11 (0.60)</td><td>0 (0.40)</td></tr></tbody></table>

this paper. Future works could be designed around the learning of custom application profiles, to fit the various shades of the industrial requirements.

## ACKNOWLEDGEMENTS

The main author of this paper would like to thank Vincent Heurtin for his contribution to this paper and Amiral Technologies for their support.

## REFERENCES

* Chabi, F., Schmidt, F., and Bercher, J.F. (2023). Analyse et prédiction de séries temporelles multivariées pour la surveillance de la santé d'ouvrages d'art. 1-4. GRETSI.

* Chandola, V., Benerjee, A., and Kumar, V. (2009). Anomaly detection: A survey. *ACM Computing Surveys*, 41, 1-58. doi:10.1145/1541880.1541882.

* Chen, S., Jin, G., and Ma, X. (2021). Satellite on-orbit anomaly detection method based on a dynamic threshold and causality pruning. *IEEE Access*, 9, 86751-86758. doi:10.1109/ACCESS.2021.3088439.

* Doshi, K., Abudalou, S., and Yilmaz, Y. (2022). Tisat: Time series anomaly transformer. URL http://arxiv.org/abs/2203.05167.

* Garg, A., Zhang, W., Samaran, J., Savitha, R., and Foo, C.S. (2022). An evaluation of anomaly detection and diagnosis in multivariate time series. *IEEE Transactions on Neural Networks and Learning Systems*, 33(6), 2508-2517. doi:10.1109/TNNLS.2021.3105827.

* Han, S., Hu, X., Huang, H., Jiang, M., and Zhao, Y. (2022). Adbench: Anomaly detection benchmark.

* Huet, A., Navarro, J.M., and Rossi, D. (2022). Local evaluation of time series anomaly detection algorithms. 635-645. Association for Computing Machinery. doi: 10.1145/3534678.3539339.

* Hwang, W.S., Yun, J.H., Kim, J., and Kim, H.C. (2019). Time-series aware precision and recall for anomaly detection considering variety of detection result and addressing ambiguous labeling. 2241-2244. Association for Computing Machinery. doi:10.1145/3357384.3358118.

* Kim, B., Alawami, M.A., Kim, E., Oh, S., Park, J., and Kim, H. (2023). A comparative study of time series anomaly detection models for industrial control systems. *Sensors*, 23. doi:10.3390/s23031310.

* Kovács, G., Sebestyen, G., and Hangan, A. (2019). Evaluation metrics for anomaly detection algorithms in time-series. *Acta Universitatis Sapientiae, Informatica*, 11, 113-130. doi:10.2478/ausi-2019-0008.

* Lavin, A. and Ahmad, S. (2015). Evaluating real-time anomaly detection algorithms-the numenta anomaly benchmark.

* Le Magueresse, T. and Le Gall, S. (2023). Comparison of anomaly detection models in an industrial context: application to non-cyclic datasets.

* Paparrizos, J., Boniol, P., Palpanas, T., Tsay, R.S., Elmore, A., and Franklin, M.J. (2022). Volume under the surface: A new accuracy evaluation measure for time-series anomaly detection. volume 15, 2774-2787. VLDB Endowment. doi:10.14778/3551793.3551830.

* Park, K.I. (2017). *Fundamentals of Probability and Stochastic Processes with Applications to Communications*. Springer Cham. doi:10.1007/978-3-319-68075-0.

* Pota, M., Pietro, G.D., and Esposito, M. (2023). Real-time anomaly detection on time series of industrial furnaces: A comparison of autoencoder architectures. *Engineering Applications of Artificial Intelligence*, 124. doi:10.1016/j.engappai.2023.106597.

* Raheem, F. and Iqbal, N. (2022). Artificial intelligence and machine learning for the industrial internet of things (IIOT). *Industrial Internet of Things*, 1-20. doi:10.1201/9781003145004-1.

* Salles, R., Lima, J., Coutinho, R., Pacitti, E., Masseglia, F., Akbarinia, R., Chen, C., Garibaldi, J., Porto, F., and Ogasawara, E. (2023). Softed: Metrics for soft evaluation of time series event detection. URL http://arxiv.org/abs/2304.00439.

* Schmidl, S., Wenig, P., and Papenbrock, T. (2022). Anomaly detection in time series: A comprehensive evaluation. volume 15, 1779-1797. VLDB Endowment. doi: 10.14778/3538598.3538602.

* Schwab, K. (2017). *The Fourth Industrial Revolution*.

* Sørbø, S. and Ruocco, M. (2023). Navigating the metric maze: A taxonomy of evaluation metrics for anomaly detection in time series. URL http://arxiv.org/abs/2303.01272.

* Tatbul, N., Lee, T.J., Zdonik, S., Alam, M., and Gottschlich, J. (2018). Precision and recall for time series. URL http://arxiv.org/abs/1803.03639.

* Tavallaee, M., Stakhanova, N., and Ghorbani, A.A. (2010). Toward credible evaluation of anomaly-based intrusion-detection methods. *IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews*, 40, 516-524. doi:10.1109/TSMCC.2010.2048428.

* Webber, W., Moffat, A., and Zobel, J. (2010). A similarity measure for indefinite rankings. *ACM Transactions on Information Systems*, 28. doi:10.1145/1852102.1852106.

* Wolpert, D.H. and Macready, W.G. (1997). No free lunch theorems for optimization.

* Wu, R. and Keogh, E.J. (2020). Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress. doi:https://doi.org/10.48550/arXiv.2009.13807.
<table><caption>Table 3. RBO@0.2 similarity scores of different candidate metrics. The RBO@0.2 focuses on the ranking head. In parentheses are the result with RBO@1, which focuses on the whole ranking. A score close to 1 indicates a ranking highly similar to the ground truth, whereas one of 0 indicates a ranking nearly opposite to the ground truth.</caption><thead><tr><th>Datasets</th><th>MTI</th><th>F1</th><th>AUC</th><th>Affiliation</th><th>eTaPR</th><th>MCC</th><th>VUS</th><th>NAB</th></tr></thead><tbody><tr><td>Dataset A</td><td><b>1 (1)</b></td><td>0.20 (0.80)</td><td>0.20 (0.80)</td><td>0.20 (0.80)</td><td>0.20 (0.80)</td><td>0.20 (0.80)</td><td><b>1 (1)</b></td><td>0.12 (0.70)</td></tr><tr><td>Dataset B</td><td><b>1 (0.94)</b></td><td>0.19 (0.77)</td><td>0.19 (0.77)</td><td>0.19 (0.77)</td><td>0.200 (0.79)</td><td>0.19 (0.77)</td><td><b>0.99 (0.89)</b></td><td>0.11 (0.47)</td></tr><tr><td>Dataset C</td><td>0.99 (0.99)</td><td>0.20 (0.83)</td><td>0.20 (0.83)</td><td>0.19 (0.70)</td><td>0.19 (0.70)</td><td>0.20 (0.83)</td><td><b>1 (1)</b></td><td><b>1 (0.92)</b></td></tr><tr><td>Dataset D</td><td><b>1 (0.87)</b></td><td>0.11 (0.66)</td><td>0.11 (0.60)</td><td>0.11 (0.65)</td><td><b>0.19 (0.69)</b></td><td>0.11 (0.63)</td><td>0.11 (0.60)</td><td>0 (0.40)</td></tr></tbody></table>

this paper. Future works could be designed around the learning of custom application profiles, to fit the various shades of the industrial requirements.

## ACKNOWLEDGEMENTS

The main author of this paper would like to thank Vincent Heurtin for his contribution to this paper and Amira Technologies for their support.

## REFERENCES

* Chabi, F., Schmidt, F., and Bercher, J.F. (2023). Analyse et prédiction de séries temporelles multivariées pour la surveillance de la santé d'ouvrages d'art. 1-4. GRETSI.

* Chandola, V., Benerjee, A., and Kumar, V. (2009). Anomaly detection: A survey. *ACM Computing Surveys*, 41, 1-58. doi:10.1145/1541880.1541882.

* Chen, S., Jin, G., and Ma, X. (2021). Satellite on-orbit anomaly detection method based on a dynamic threshold and causality pruning. *IEEE Access*, 9, 86751-86758. doi:10.1109/ACCESS.2021.3088439.

* Doshi, K., Abudalou, S., and Yilmaz, Y. (2022). Tisat: Time series anomaly transformer. URL http://arxiv.org/abs/2203.05167.

* Garg, A., Zhang, W., Samaran, J., Savitha, R., and Foo, C.S. (2022). An evaluation of anomaly detection and diagnosis in multivariate time series. *IEEE Transactions on Neural Networks and Learning Systems*, 33(6), 2508-2517. doi:10.1109/TNNLS.2021.3105827.

* Han, S., Hu, X., Huang, H., Jiang, M., and Zhao, Y. (2022). Adbench: Anomaly detection benchmark.

* Huet, A., Navarro, J.M., and Rossi, D. (2022). Local evaluation of time series anomaly detection algorithms. 635-645. Association for Computing Machinery. doi: 10.1145/3534678.3539339.

* Hwang, W.S., Yun, J.H., Kim, J., and Kim, H.C. (2019). Time-series aware precision and recall for anomaly detection considering variety of detection result and addressing ambiguous labeling. 2241-2244. Association for Computing Machinery. doi:10.1145/3357384.3358118.

* Kim, B., Alawami, M.A., Kim, E., Oh, S., Park, J., and Kim, H. (2023). A comparative study of time series anomaly detection models for industrial control systems. *Sensors*, 23. doi:10.3390/s23031310.

* Kovács, G., Sebestyen, G., and Hangan, A. (2019). Evaluation metrics for anomaly detection algorithms in time-series. *Acta Universitatis Sapientiae, Informatica*, 11, 113-130. doi:10.2478/ausi-2019-0008.

* Lavin, A. and Ahmad, S. (2015). Evaluating real-time anomaly detection algorithms-the numenta anomaly benchmark.

* Le Magueresse, T. and Le Gall, S. (2023). Comparison of anomaly detection models in an industrial context: application to non-cyclic datasets.

* Paparrizos, J., Boniol, P., Palpanas, T., Tsay, R.S., Elmore, A., and Franklin, M.J. (2022). Volume under the surface: A new accuracy evaluation measure for time-series anomaly detection. volume 15, 2774-2787. VLDB Endowment. doi:10.14778/3551793.3551830.

* Park, K.I. (2017). *Fundamentals of Probability and Stochastic Processes with Applications to Communications*. Springer Cham. doi:10.1007/978-3-319-68075-0.

* Pota, M., Pietro, G.D., and Esposito, M. (2023). Real-time anomaly detection on time series of industrial furnaces: A comparison of autoencoder architectures. *Engineering Applications of Artificial Intelligence*, 124. doi:10.1016/j.engappai.2023.106597.

* Raheem, F. and Iqbal, N. (2022). Artificial intelligence and machine learning for the industrial internet of things (IIOT). *Industrial Internet of Things*, 1-20. doi:10.1201/9781003145004-1.

* Salles, R., Lima, J., Coutinho, R., Pacitti, E., Masseglia, F., Akbarinia, R., Chen, C., Garibaldi, J., Porto, F., and Ogasawara, E. (2023). Softed: Metrics for soft evaluation of time series event detection. URL http://arxiv.org/abs/2304.00439.

* Schmidl, S., Wenig, P., and Papenbrock, T. (2022). Anomaly detection in time series: A comprehensive evaluation. volume 15, 1779-1797. VLDB Endowment. doi: 10.14778/3538598.3538602.

* Schwab, K. (2017). *The Fourth Industrial Revolution*. Sørbø, S. and Ruocco, M. (2023). Navigating the metric maze: A taxonomy of evaluation metrics for anomaly detection in time series. URL http://arxiv.org/abs/2303.01272.

* Tatbul, N., Lee, T.J., Zdonik, S., Alam, M., and Gottschlich, J. (2018). Precision and recall for time series. URL http://arxiv.org/abs/1803.03639.

* Tavallaee, M., Stakhanova, N., and Ghorbani, A.A. (2010). Toward credible evaluation of anomaly-based intrusion-detection methods. *IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews*, 40, 516-524. doi:10.1109/TSMCC.2010.2048428.

* Webber, W., Moffat, A., and Zobel, J. (2010). A similarity measure for indefinite rankings. *ACM Transactions on Information Systems*, 28. doi:10.1145/1852102.1852106.

* Wolpert, D.H. and Macready, W.G. (1997). No free lunch theorems for optimization.

* Wu, R. and Keogh, E.J. (2020). Current time series anomaly detection benchmarks are flawed and are creating the illusion of progress. doi:https://doi.org/10.48550/arXiv.2009.13807.