SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chih-Yu Lai*
chihyul@mit.edu
Fan-Keng Sun
fankeng@mit.edu
Zhengqi Gao
zhengqi@mit.edu
Jeffrey H. Lang
lang@mit.edu
Duane S. Boning
boning@mtl.mit.edu
Time series anomaly detection is challenging due to the complexity and variety of patterns that can occur. One major difficulty arises from modeling time-dependent relationships to find contextual anomalies while maintaining detection accuracy for point anomalies. In this paper, we propose a framework for unsupervised time series anomaly detection that utilizes point-based and sequence-based reconstruction models. The point-based model attempts to quantify point anomalies, and the sequence-based model attempts to quantify both point and contextual anomalies. Under the formulation that the observed time point is a two-stage deviated value from a nominal time point, we introduce a nominality score calculated from the ratio of a combined value of the reconstruction errors. We derive an induced anomaly score by further integrating the nominality score and anomaly score, then theoretically prove the superiority of the induced anomaly score over the original anomaly score under certain conditions. Extensive studies conducted on several public datasets show that the proposed framework outperforms most state-of-the-art baselines for time series anomaly detection.
Time series anomaly detection involves identifying unusual patterns or events in a sequence of data collected over time [1]. This technique is crucial in fields such as finance [2], healthcare [3], manufacturing [4], transportation [5], and more [6]. A comprehensive evaluation of different techniques can be found in [7]. Time series anomaly detection using unsupervised learning approaches is favored by recent studies due to the fact that they don't require labeled data and have better detection of unseen anomalies [8], which is also the approach used in this work. Most unsupervised time series anomaly detection methods involve calculating an anomaly score at each time point and then comparing this score to some threshold. For calculating this score, we can categorize different methods into three main groups: Reconstruction-based methods involve reconstructing the original time series data and comparing the reconstructed data with the actual data [9, 10, 11]. Prediction-based methods involve predicting the next value in the time series and comparing it with the actual value [12, 13, 14]. Dissimilarity-based methods measure the distance between the value obtained from the model and the distribution or cluster of the aggregated data [15, 16, 17, 18, 19, 20]. There can also be hybrid techniques where multiple methods are applied.
*Code available at https://github.com/andrewlai61616/NPSR
While the classification approaches for the types of anomalies differ in the literature [6, 7], we will focus on two classes: point anomalies and contextual anomalies. Point anomalies refer to individual data points that significantly deviate from the expected behavior of the time series and can be detected by observing the data at a single time point. Contextual anomalies, on the other hand, refer to data points that deviate from the expected behavior of the time series in a specific context or condition. These anomalies cannot be detected by observing the data at a single time point and can only be detected by observing the contextual information. As a result, point anomalies can be detected using any general anomaly detection technique that does not require temporal information. Such models are called point-based models in this work. However, detecting contextual anomalies requires a model that can learn temporal information. Such models require a sequence of input, hence they are termed sequence-based models in this work. A time point may contain both point and contextual anomalies. Obviously, contextual anomalies are harder to detect. An important trade-off arises when a model tries to learn time-dependent relationships for detecting contextual anomalies but loses precision or accuracy in finding point anomalies. This trade-off becomes more significant in high-dimensional data, where modeling the temporal relationship is difficult [21].
Our intuition comes from the observation that state-of-the-art methods using sequence-based reconstruction models encounter the point-contextual detection trade-off, resulting in noisy reconstruction results and suboptimal performance. As an alternative, we start by experimenting with point-based reconstruction methods, which exhibit a lower variance as they do not require the modeling of time-dependent relationships. Despite the absence of temporal information, we find that the corresponding anomaly score for a point-based reconstruction model can already yield a competitive performance. To further bridge the gap between point-based and sequence-based models, we introduce a nominality score that can be calculated from their outputs and derive an induced anomaly score based on the nominality score and the original anomaly score. We find that the induced anomaly score can be superior to the original anomaly score. To provide theoretical proof of our findings, we frame the reconstruction process as a means to fix the anomalies and identify underlying nominal time points and prove that the induced anomaly score can always perform better or just as well as the original anomaly score under certain conditions. We also conduct experiments on several single and multi-entity datasets and demonstrate that our proposed method surpasses the performance of recent state-of-the-art techniques. We coin our method Nominality score conditioned time series anomaly detection by Point/Sequential Reconstruction (NPSR).
Reconstruction-based techniques have seen a diversity of approaches over the years. The simplest reconstruction technique involves training a separate model sequentially for each channel using UAE [11]. One type of improvement focuses on network architecture, with notable examples like LSTM-VAE [22], MAD-GAN [23], MSCRED [24], OmniAnomaly [9], and TranAD [25]. Additionally, hybrid architectures like DAGMM [26] and MTAD-GAT [27] have been proposed. Another type aims to improve the anomaly score instead of using the original reconstruction error. Designing this anomaly score involves a considerable amount of art, given the high diversity of methods for calculating it across studies. For instance, USAD uses two weighted reconstruction errors [28], OmniAnomaly employs the "reconstruction probability" as an alternative anomaly score [9], MTAD-GAT combines forecasting error and reconstruction probability [27], and TranAD uses an integrated reconstruction error and discriminator loss as the anomaly score [25]. In the context of network architecture, our method utilizes a straightforward performer-based structure without incorporating any specialized components. Based on our insights into the point-sequential detection tradeoff, our approach stands out by integration of point-based and sequence-based reconstruction errors for competitive performance.
Let  $\mathbf{X} = \{\mathbf{x}_1, \dots, \mathbf{x}_T\}$  denote a multivariate time series with  $\mathbf{x}_t \in \mathbb{R}^D$ , where  $T$  is the time length and  $D$  is the dimensionality or number of channels. There exists a corresponding set of labels  $\mathbf{y} = \{y_1, \dots, y_T\}, y_t \in \{0, 1\}$  indicating whether the time point is normal ( $y_t = 0$ ) or anomalous ( $y_t = 1$ ). For a given  $\mathbf{X}$ , the goal is to yield anomaly scores for all time points  $\mathbf{a} = \{a_1, \dots, a_T\}$ ,  $a_t \in \mathbb{R}$  and a corresponding threshold  $\theta_a$  such that the predicted labels  $\hat{\mathbf{y}} = \{\hat{y}_1, \dots, \hat{y}_T\}$ , where  $\hat{y}_t \triangleq \mathbb{1}_{a_t \geq \theta_a}$ , match
$\mathbf{y}$  as much as possible. To quantify how matched  $\hat{\mathbf{y}}$  and  $\mathbf{y}$  is, or how good  $\mathbf{a}$  is for potentially yielding a good  $\hat{\mathbf{y}}$ , there are several performance metrics that take either  $\mathbf{a}$  or  $\hat{\mathbf{y}}$  into account [11, 12, 29, 30, 31]. This work mainly focuses on the best F1 score ( $F1^*$ ) without point-adjust, also known as the pointwise F1 score, which is defined as the maximum possible F1 score considering all thresholds. (A complete derivation for  $F1^*$  is covered in Appendix A.)
3.2 Nominal Time Series and Two-stage Deviation
We denote the observed data as  $\mathbf{X}^0 = \{\mathbf{x}_1^0, \dots, \mathbf{x}_T^0\}$ , where  $\mathbf{x}_t^0 \in \mathbb{R}^D$ . Assume that for each  $\mathbf{X}^0$ , there exists a corresponding underlying nominal time series data  $\mathbf{X}^* = \{\mathbf{x}_1^*, \dots, \mathbf{x}_T^*\}$  that comes from a nominal time-dependent process  $\mathbf{x}_t^* = \mathbf{f}^*(t): \mathbb{N} \to \mathbb{R}^D$ . The corresponding total deviation at  $t$  ( $\Delta \mathbf{x}_t^0$ ) is defined as  $\Delta \mathbf{x}_t^0 \triangleq \mathbf{x}_t^0 - \mathbf{x}_t^*$ . We denote  $\mathcal{X}^*$  as the set of all possible  $\mathbf{x}_t^*$  for all  $t \in \{1, \dots, T\}$ .  $\Delta \mathbf{x}_t^0$  can be separated into two additive factors  $\Delta \mathbf{x}_t^c$  and  $\Delta \mathbf{x}_t^p$ , such that, by definition,  $\mathbf{x}_t^c = \mathbf{x}_t^* + \Delta \mathbf{x}_t^c$  and  $\mathbf{x}_t^0 = \mathbf{x}_t^c + \Delta \mathbf{x}_t^p$ . We define  $\Delta \mathbf{x}_t^c$  as the in-distribution deviation, where  $\mathbf{x}_t^c \in \mathcal{X}^*$ ; and  $\Delta \mathbf{x}_t^p$  as the out-of-distribution deviation, which is non-zero if and only if  $\mathbf{x}_t^0 \notin \mathcal{X}^*$ .  $\Delta \mathbf{x}_t^p$  can be a means for quantifying the point anomaly, and  $\Delta \mathbf{x}_t^c$  can be a means for quantifying the contextual anomaly. This is reasonable, since no matter how large  $\Delta \mathbf{x}_t^c$  is, we still have  $\mathbf{x}_t^c \in \mathcal{X}^*$ , i.e., the in-distribution deviated value  $\mathbf{x}_t^c$  is still in the set of all possible nominal time point data, and cannot be detected using a point-based model. On the other hand, it is possible that having learned  $\mathcal{X}^*$ , a point-based model can negate the deviated value caused by  $\Delta \mathbf{x}_t^p$ . Fig. 1(a) gives an illustration of the relationships between the variables at time  $t$ . We clarify this using the example below.
Assume we obtain a dataset from the streaming data of a 2D position sensor, where  $\mathbf{x}_t^*$ ,  $\mathbf{x}_t^c$ ,  $\mathbf{x}_t^0 \in \mathbb{R}^2$ , and we have learned that the nominal time series is the circular movement of a point around the origin with some angular velocity  $\omega$  and radius  $r$ , where  $R_{min} \leq r \leq R_{max}$ . Accordingly, we can deduce that  $\mathcal{X}^* = \{[x y]^T | R_{min}^2 \leq x^2 + y^2 \leq R_{max}^2\}$  and  $\mathbf{x}_t^* = [r \cos \omega t r \sin \omega t]^T$ . One possible cause (among many others) of contextual anomalies might be due to an unexpected change in angular velocity. For instance, failures in the system might lead to a slowdown of the circular movement between  $t_1$  and  $t_2$ , i.e.,  $\Delta \mathbf{x}_t^c = [r (\cos \omega' t - \cos \omega t) r (\sin \omega' t - \sin \omega t)]^T$  for  $t \in \{t_1, \dots, t_2\}$  and  $\Delta \mathbf{x}_t^c = \mathbf{0}$  elsewhere. Moreover, noisy measurements of individual time points may induce point anomalies in the observed time series, i.e.,  $\Delta \mathbf{x}_t^p = [w_{x,t} w_{y,t}]^T$  where  $w_{x,t}$  or  $w_{y,t}$  is nonzero such that  $\mathbf{x}_t^0 \notin \mathcal{X}^*$  for some  $t$ . Fig. 1(b)(c) gives an illustration of the above example. The black dots are time points where  $\mathbf{x}_t^0 = \mathbf{x}_t^*$  (no anomalies). The blue dots are time points exhibiting a slowdown (contextual anomalies). The red dots are time points with noisy measurements (point anomalies). The purple dots are time points with both slowdown and noisy measurements (point and contextual anomalies). The green dots are time points with noisy measurements but  $\mathbf{x}_t^0 \in \mathcal{X}^*$ , so are still contextual anomalies since their deviations cannot be detected by observing a single time point.
3.3 The Nominality Score
Now we conceptualize the Nominality Score  $N(\cdot)$ . Analogous to the anomaly score,  $N(\cdot)$  indicates how normal a time point is. A nominality score  $N(\cdot)$  is appropriate if for every possible  $\theta_N > 0$ ,  $\mathbb{P}(N(t) > \theta_N | y_t = 0) > \mathbb{P}(N(t) > \theta_N | y_t = 1)$  for all  $t \in \{1, \dots, T\}$ , i.e., the portion of normal points that has a nominality score larger than  $\theta_N$  is strictly larger than the portion of anomaly points
Figure 1: (a) Relationships between variables, (b) observed time series on 2D plane, and (c) radial and angular displacement vs time from nominal time series  $(\mathbf{x}_t^*)$  for the 2D position sensor example.
that has a nominality score larger than  $\theta_N$ . There are many ways to define  $N(\cdot)$ . In this study, we define  $N(t)$  as the ratio of the squared L2-norm between  $\Delta \mathbf{x}_t^c$  and  $\Delta \mathbf{x}_t^0$ .
$$
N (t) \triangleq \frac {\left\| \Delta \mathbf {x} _ {t} ^ {c} \right\| _ {2} ^ {2}}{\left\| \Delta \mathbf {x} _ {t} ^ {0} \right\| _ {2} ^ {2}} = \frac {\left\| \Delta \mathbf {x} _ {t} ^ {c} \right\| _ {2} ^ {2}}{\left\| \Delta \mathbf {x} _ {t} ^ {c} + \Delta \mathbf {x} _ {t} ^ {p} \right\| _ {2} ^ {2}} = \frac {\left\| \mathbf {x} _ {t} ^ {c} - \mathbf {x} _ {t} ^ {*} \right\| _ {2} ^ {2}}{\left\| \mathbf {x} _ {t} ^ {0} - \mathbf {x} _ {t} ^ {*} \right\| _ {2} ^ {2}} \tag {1}
$$
We provide an example as to when  $N(\cdot)$  will be appropriate. In this derivation, we add an  $n$  and  $a$  in the subscript to denote variables that are only associated with normal and anomaly points, respectively. Consider a toy dataset, where  $\Delta \mathbf{x}_{t,n}^{c}, \Delta \mathbf{x}_{t,n}^{p}, \Delta \mathbf{x}_{t,a}^{c},$  and  $\Delta \mathbf{x}_{t,a}^{p}$  have been defined:
$$
\Delta \mathbf {x} _ {t, n} ^ {c} \sim \mathcal {N} (0, I _ {D}), \quad \Delta \mathbf {x} _ {t, n} ^ {p} \sim \mathcal {N} (0, I _ {D}), \quad \Delta \mathbf {x} _ {t, a} ^ {c} \sim \mathcal {N} (0, I _ {D}), \quad \Delta \mathbf {x} _ {t, a} ^ {p} \sim \mathcal {N} (0, \alpha^ {2} I _ {D}) \tag {2}
$$
According to (1), we have
$$
2 N _ {n} (t) = 2 \frac {\| \Delta \mathbf {x} _ {t , n} ^ {c} \| _ {2} ^ {2}}{\| \Delta \mathbf {x} _ {t , n} ^ {c} + \Delta \mathbf {x} _ {t , n} ^ {p} \| _ {2} ^ {2}} \sim \mathrm {F} (D, D) \tag {3}
$$
$$
(1 + \alpha^ {2}) N _ {a} (t) = (1 + \alpha^ {2}) \frac {\| \Delta \mathbf {x} _ {t , a} ^ {c} \| _ {2} ^ {2}}{\| \Delta \mathbf {x} _ {t , a} ^ {c} + \Delta \mathbf {x} _ {t , a} ^ {p} \| _ {2} ^ {2}} \sim \mathrm {F} (D, D) \tag {4}
$$
where  $\mathrm{F}$  is the F-distribution with  $D$  and  $D$  degrees of freedom. Fig. 2 illustrates the probability density function of  $N_{n}(\cdot)$  and  $N_{a}(\cdot)$  for different  $D$  and  $\alpha$ . If  $\alpha > 1$ , then  $N(\cdot)$  becomes an appropriate nominality score, since
$$
\mathbb {P} (N (t) > \theta_ {N} | y _ {t} = 0) = \int_ {2 \theta_ {N}} ^ {\infty} f (x; D, D) d x > \int_ {(1 + \alpha^ {2}) \theta_ {N}} ^ {\infty} f (x; D, D) d x = \mathbb {P} (N (t) > \theta_ {N} | y _ {t} = 1) \tag {5}
$$
where  $f(\cdot; D, D)$  is the probability density function of the F-distribution with degrees of freedom  $D$  and  $D$ . Indeed, it is reasonable to assume that  $\Delta \mathbf{x}_{t,a}^p$  has a larger variance than  $\Delta \mathbf{x}_{t,n}^p$ .
3.4 The Induced Anomaly Score
Having  $N(\cdot)$  defined, we now propose a method for integrating any given  $N(\cdot)$  and anomaly score  $A(\cdot)$  to yield an induced anomaly score  $\hat{A}(\cdot)$ , and show some instances where the performance will improve over using  $A(\cdot)$  or a smoothed  $A(\cdot)$ . Consider a dataset that contains subsequence anomalies. For two near time points  $t$  and  $\tau$ , it is natural to assume that the possibility of  $t$  being anomalous is affected by  $\tau$ . We quantify this effect as  $A(t; \tau)$ , which is the induced anomaly score at  $t$  due to  $\tau$ . By summing over a range of  $\tau$  around  $t$ , we get the induced anomaly score at  $t$ :
$$
\hat {A} (t) \triangleq \sum_ {\tau = \max  (1, t - d)} ^ {\min  (T, t + d)} A (t; \tau) \tag {6}
$$
where  $d$  is the induction length. Furthermore, we define  $A(t;\tau)$  as a gated value of  $A(\tau)$ , which is controlled by the nominality score from  $\tau$  to  $t$ :
$$
A (t; \tau) \triangleq A (\tau) \prod_ {k = \min  (\tau + 1, t)} ^ {\max  (t - \mathbb {1} _ {t = \tau}, \tau - 1)} g _ {\theta_ {N}} (N (k)) = \left\{ \begin{array}{l l} A (\tau) g _ {\theta_ {N}} (N (\tau + 1)) \dots g _ {\theta_ {N}} (N (t)) & t > \tau \\ A (\tau) & t = \tau \\ A (\tau) g _ {\theta_ {N}} (N (\tau - 1)) \dots g _ {\theta_ {N}} (N (t)) & t <   \tau \end{array} \right. \tag {7}
$$
Figure 2: The probability density function for  $N_{n}$  and  $N_{a}$  of the toy dataset at (a)  $D = 2$  and (b)  $D = 100$ .
where the gate function  $g_{\theta_N}(N)$  is some transformation function of  $N$  conditioned on a threshold  $\theta_N$ . A reasonable assumption is that  $g_{\theta_N}(N)$  is a non-increasing function of  $N$ , i.e.,  $N > N'$  implies  $g_{\theta_N}(N) \leq g_{\theta_N}(N')$ . Indeed, if  $N(k)$  is large, then time point  $k$  is likely a normal point, and any two points  $t_1, t_2$ , where  $t_1 < k < t_2$ , are unlikely to be in the same anomaly subsequence, hence  $A(t_1; t_2)$  and  $A(t_2; t_1)$  should be small. Explicitly, we can use  $\hat{A}(t) = \hat{A}(t; g_{\theta_N})$  and  $A(t; \tau) = A(t; \tau, g_{\theta_N})$  to denote that these values are conditioned on  $g_{\theta_N}$ . Overall,  $\hat{A}(\cdot)$  can be thought of as some (unnormized) weighted smoothed value of  $A(\cdot)$ , where the weights are the product of  $g_{\theta_N}(N(\cdot))$  across some range. We consider the following two cases:
Claim 1 Using a soft gate function,
$$
g _ {\theta_ {N}} (N) \triangleq \max  \left(0, 1 - \frac {N}{\theta_ {N}}\right) \tag {8}
$$
If there exists  $\theta_{1}$  such that  $N(t)\geq \theta_{1}$  for all normal points  $(y_{t} = 0)$ , then  $\mathrm{F1}^{*}(\hat{A} (\cdot ;g_{\theta_1});\mathbf{y})\geq$ $\mathrm{F1}^{*}(A(\cdot);\mathbf{y})$ , i.e., the best F1 score using the induced anomaly score with  $g_{\theta_1}$  as the gate function is greater or equal to the best F1 score using the original anomaly score.
Proof 1 For any normal time point  $t_n$ , we have
$$
\hat {A} \left(t _ {n}; g _ {\theta_ {1}}\right) = \sum_ {\tau = \max  \left(1, t _ {n} - d\right)} ^ {\min  \left(T, t _ {n} + d\right)} A \left(t _ {n}; \tau , g _ {\theta_ {1}}\right) = \sum_ {\tau = \max  \left(1, t _ {n} - d\right)} ^ {\min  \left(T, t _ {n} + d\right)} A (\tau) \mathbb {1} _ {t _ {n} = \tau} = A \left(t _ {n}\right) \tag {9}
$$
The equality in the middle arises from the fact that  $g_{\theta_1}(N(t_n)) = 0$ , according to (8) and the assumptions. However, for any anomaly point  $t_a$ , we have
$$
\hat {A} \left(t _ {a}; g _ {\theta_ {1}}\right) = \sum_ {\tau = \max  \left(1, t _ {a} - d\right)} ^ {\min  \left(T, t _ {a} + d\right)} A \left(t _ {a}; \tau , g _ {\theta_ {1}}\right) \geq A \left(t _ {a}\right) \tag {10}
$$
This is because  $N(t_{a})$  might be lower than  $\theta_{1}$ , and hence  $g_{\theta_1}(N(t_a))\geq 0$ . By potentially having a higher  $\hat{A} (t)$  than  $A(t)$  for anomaly points, we get a potentially higher  $\mathrm{F1^{*}}$ .
Such a  $\theta_{1}$  indeed exists in real applications (e.g. the minimum nominality score among all normal points  $t_n$ ). However, targeting this value barely leads to any improvement for  $\mathrm{F}1^{*}$  in practice. This is because anomaly points are generally fewer than normal points, resulting in barely any anomaly points  $t_a$  having  $N(t_a) < \theta_1$ . Nevertheless, we have shown that the soft gate function along with the induced anomaly score can provably yield equal or better  $\mathrm{F}1^{*}$  under some threshold.
Claim 2 Using a hard gate function,
$$
g _ {\theta_ {N}} (N) \triangleq \mathbb {1} _ {N <   \theta_ {N}} \tag {11}
$$
If  $d = 1$ , and there exist two thresholds: (i)  $\theta_{2}$  such that  $N(t) < \theta_{2}$  for all anomaly time points  $(y_{t} = 1)$  (ii)  $\theta_{\infty} = \infty$ ; then  $\mathrm{F1}^{*}(\hat{A} (\cdot ;g_{\theta_{2}});\mathbf{y})\geq \mathrm{F1}^{*}(\hat{A} (\cdot ;g_{\theta_{\infty}});\mathbf{y})$ , i.e., the best F1 score using the induced anomaly score with  $g_{\theta_2}$  as the gate function is greater or equal to the best F1 score using the induced anomaly score with  $g_{\theta_{\infty}}$  as the gate function.
Proof 2 For any anomaly time point  $t_a$ , we have
$$
\hat {A} \left(t _ {a}; g _ {\theta_ {2}}\right) = \sum_ {\tau = \max  \left(1, t _ {a} - 1\right)} ^ {\min  \left(T, t _ {a} + 1\right)} A (\tau) = A \left(t _ {a} - 1\right) \mathbb {1} _ {t _ {a} > 1} + A \left(t _ {a}\right) + A \left(t _ {a} + 1\right) \mathbb {1} _ {t _ {a} <   T} \tag {12}
$$
where the first equality arises from the fact that  $g_{\theta_2}(N(t_a)) = 1$ , according to (11) and the assumptions. For any normal time point  $t_n$ , we have
$$
\hat {A} \left(t _ {n}; g _ {\theta_ {2}}\right) = \sum_ {\tau = \max  \left(1, t _ {n} - 1\right)} ^ {\min  \left(T, t _ {n} + 1\right)} A \left(t _ {n}; \tau , g _ {\theta_ {2}}\right) \leq A \left(t _ {n} - 1\right) \mathbb {1} _ {t _ {n} > 1} + A \left(t _ {n}\right) + A \left(t _ {n} + 1\right) \mathbb {1} _ {t _ {n} <   T} \tag {13}
$$
since  $N(t_{n})$  might be greater than  $\theta_{2}$  and hence  $g_{\theta_2}(N(t_n))\leq 1$ . However, we have
$$
\hat {A} (t; g _ {\theta_ {\infty}}) = \sum_ {\tau = \max  (1, t - 1)} ^ {\min  (T, t + 1)} A (\tau) = A (t - 1) \mathbb {1} _ {t > 1} + A (t) + A (t + 1) \mathbb {1} _ {t <   T} \tag {14}
$$
regardless of normal or anomaly points since  $N(t) < \theta_{\infty}$  for any  $t$ . Therefore, since  $\hat{A}(t_a; g_{\theta_2}) = \hat{A}(t_a; g_{\theta_{\infty}})$  and  $\hat{A}(t_n; g_{\theta_2}) \leq \hat{A}(t_n; g_{\theta_{\infty}})$ , we get a potentially higher F1* when using  $\theta_2$  compared to using  $\theta_{\infty}$ .
$\hat{A} (\cdot ;g_{\theta_{\infty}})$  can be viewed as the smoothed value (or shifted simple moving average) over  $A(\cdot)$  with a period of  $2d + 1$ . This averaging method is common among other studies [11, 12, 32, 33]. Claim 2 implies that by conditioning on  $N(\cdot)$  and calculating  $\hat{A} (t)$ , the performance can be improved over using a simple smoothing value of  $A(\cdot)$ . In practice, we can relax the constraint of  $d$ , and use other gated functions to yield a more flexible architecture. Note that the appropriateness of a nominality score is critical for this to work.
3.5 Point-based Reconstruction Models
Consider some model  $\mathcal{M}_{pt}$  that reconstructs each time point  $t$  point-wise:  $\hat{\mathbf{x}}_{pt,t} \triangleq \mathcal{M}_{pt}(\mathbf{x}_t^0)$ . Using  $\mathcal{M}_{pt}$ , a method for yielding the anomaly score is by using the point-based reconstruction mean-squared error, defined as  $\mathbf{a}_{pt} = \{a_{pt,1},\dots,a_{pt,T}\}$ , where  $a_{pt,t} \triangleq \| \hat{\mathbf{x}}_{pt,t} - \mathbf{x}_t^0\|_2^2$ . One concern for using this kind of anomaly score is that we are not taking into account any time-dependent relationships for deriving  $\mathbf{a}_{pt}$ , so simply using  $\mathcal{M}_{pt}$  and  $\mathbf{a}_{pt}$  can barely be classified as a time series anomaly detection approach. Surprisingly, however, we find that a simple realization of  $\mathcal{M}_{pt}$  can already achieve impressive results for F1* (section 4.4).
Since  $\mathcal{M}_{pt}$  learns to capture the distribution of all normal point data, we assume that  $\hat{\mathbf{x}}_{pt,t} \in \mathcal{X}^*$  or is very close. Moreover, since  $\hat{\mathbf{x}}_{pt,t}$  can offset point-anomalies, we assume  $\mathbf{x}_t^c \approx \hat{\mathbf{x}}_{pt,t}$ , and implement a point-based reconstruction model to calculate  $\hat{\mathbf{x}}_{pt,t}$  in practice.