Towards a General Time Series Anomaly Detector with Adaptive Bottlenecks and Dual Adversarial Decoders

## Abstract

Abstract Time series anomaly detection plays a vital role in a wide range of applications. Existing methods require training one specific model for each dataset, which exhibits limited generalization capability across different target datasets, hindering anomaly detection performance in various scenarios with scarce training data. Aiming at this problem, we propose constructing a general time series anomaly detection model, which is pre-trained on extensive multi-domain datasets and can subsequently apply to a multitude of downstream scenarios. The significant divergence of time series data across different domains presents two primary challenges in building such a general model: (1) meeting the diverse requirements of appropriate information bottlenecks tailored to different datasets in one unified model, and (2) enabling distinguishment between multiple normal and abnormal patterns, both are crucial for effective anomaly detection in various target scenarios. To tackle these two challenges, we propose a General time series anomaly D etector with A daptive Bottlenecks and D ual A dversarial Decoders (DADA) , which enables flexible selection of bottlenecks based on different data and explicitly enhances clear differentiation between normal and abnormal series. We conduct extensive experiments on nine target datasets from different domains. After pre-training on multi-domain data, DADA, serving as a zero-shot anomaly detector for these datasets, still achieves competitive or even superior results compared to those models tailored to each specific dataset.

## 1 Introduction

With the continuous advancement of technology and the widespread application of various sensors, time series data is ubiquitous in many real-world scenarios (Anandakrishnan et al., ; Cook et al., ; Kieu et al., ). Effectively detecting anomalies in time series data helps to identify potential issues in time and takes necessary measures to ensure the normal operation of systems, thereby avoiding possible economic losses and security threats (Yang et al., ). For example, in the cyber-security field, timely detection of abnormal network traffic can prevent service interruptions (Dong et al., ; Wang & Zhu, ; Li et al., ); in the healthcare sector, detecting anomalies in user physiological indicators is crucial for preventing diseases (Wang et al., ).

Anomaly detection methods based on deep learning (Xu et al., ; Su et al., ; donghao & wang xue, ) have recently achieved significant success due to their powerful representation capabilities for data. However, existing methods for detecting anomalies in time series data typically require constructing and training specific models for different datasets (Cook et al., ; Wang et al., ; Campos et al., ; Yang et al., ). Although these methods show good performance on each specific dataset, their generalization ability across different target scenarios is limited (Zhang et al., ). In some scenarios, there are not enough data or resources for specific model training, such as those where data collection is impeded by data scarcity, time and labor costs, or user privacy concerns. Therefore, the feasibility of existing methods in practice may be greatly restricted. Aiming at this problem, we propose constructing a general time series anomaly detection (GTSAD) model. As shown

Figure: Figure 1: General time series anomaly detection (GTSAD) and standard methods training specific detectors for each scenario.
Refer to caption: x1.png

in Figure 1, by pre-training the model on large time series data from multiple sources and domains, it is encouraged to learn anomaly detection capabilities from richer temporal information, which gives the potential to mitigate the overfitting of domain-specific patterns and learn patterns and models that are more generalizable. Then, the model can be efficiently applied to a wide range of downstream scenarios. However, constructing a general anomaly detection model still faces the following two major challenges.

First, the large-scale time series data are from multiple sources and domains with different information densities, which raises the challenge of meeting the diverse requirements of appropriate information bottlenecks tailored to different datasets in one unified model. Existing anomaly detection methods emphasize accurately capturing the characteristics of normal data with autoencoder, due to the inherently scarce nature of anomalies (Chalapathy & Chawla, ). Information bottleneck is considered as a trade-off between compactly compressing the intrinsic information of the original data and high-fidelity reconstruction (Kawaguchi et al., ). A large bottleneck can reconstruct the original data accurately but may cause the model to fit unnecessary noise unrelated to normal patterns. On the contrary, a small bottleneck can compactly compress the intrinsic information but may result in the loss of diverse normal patterns and poor reconstruction (Wang et al., ). Existing anomaly detection methods typically tune a fixed internal information bottleneck for each scenario, which is limited in inflexible representation capability and insufficient generalization ability when facing multi-domain time series data with significantly different data distributions, periodicity, and noise levels (Liu et al., ), making it difficult to adapt well to downstream scenarios.

Second, the diverse manifestations of anomalies across multiple domains pose another significant challenge of robust distinguishment between normal patterns and diverse anomaly patterns. A model capable of general anomaly detection not only needs to have a clear understanding of specific normal patterns in time series data from different domains but also requires a clear delineation of decision boundaries between diverse normal and abnormal patterns.
Existing anomaly detection methods are often based on the one-class classification assumption and only learn normal patterns in time series data for each specific domain, lacking explicit differentiation of the anomalies. Meanwhile, simply learning to discriminate normal and abnormal patterns specific to one domain is not generalizable enough, as the distributions of normal and abnormal patterns change across domains. As a result, they may struggle to distinguish between normal and abnormal patterns for general time series, where multi-domain data exist with diverse anomaly manifestations and the decision boundaries between normal and abnormal patterns are more complex.

In this paper, we propose a novel general time series anomaly Detector with Adaptive bottlenecks and Dual Adversarial decoders (DADA). For the first challenge, we consider the model’s generalization ability from the perspective of a dynamic bottleneck and introduce the Adaptive Bottlenecks module to enhance the learning of normal time series patterns from multi-domain data. We employ a bottleneck to compress features into the latent space, the size of which can manifest different information densities for multi-domain data (Wang et al., ). To meet the requirements of different information bottlenecks for divergent data, we establish a bottleneck pool containing various bottlenecks with different latent space sizes. Then, a data-adaptive mechanism is further proposed to enable the flexible selection of proper internal sizes based on the unique reconstruction requirements of the input data. To address the second challenge, we propose the Dual Adversarial Decoders module, which works with the encoder to enlarge the robust distinguishment between normal and anomaly patterns, where the normal decoder learns normal patterns for accurate reconstruction of normal sequences, and the anomaly decoder learns different
anomaly patterns from anomaly time series.
By injecting noise that represents more common anomaly patterns, we prevent the model from overfitting domain-specific anomaly patterns that may vary across domains.
We design an adversarial training mechanism for the encoder and the anomaly decoder on the reconstruction of abnormal series, which learns explicit decision boundaries between normal time series and common anomalies during the multi-domain training, thereby improving anomaly detection capability across different scenarios. Our main contributions are summarized as follows:

- •
We propose DADA, a novel General Time Series Anomaly Detector with Adaptive Bottlenecks and Dual Adversarial Decoders. By pre-training on multi-domain time series data, we achieve the goal of “one-model-for-many”, meaning a single model can perform anomaly detection on various target scenarios efficiently without domain-specific training.
- •
The proposed Adaptive Bottlenecks is the first to consider model’s generalization ability from the perspective of the dynamic bottleneck in time series anomaly detection, which adaptively addresses the flexible reconstruction requirements of multi-domain data.
- •
The proposed Dual Adversarial Decoders explicitly amplify the decision boundaries between normal time series and common anomalies in an adversarial training way, which improves the general anomaly detection capability across different scenarios.
- •
Our model acts as a zero-shot time series anomaly detector that achieves competitive or superior performance on various downstream datasets, compared with state-of-the-art models trained specifically for each dataset.

## 2 Related Work

#### Time series anomaly detection.

Time series anomaly detection methods can mainly be categorized into none-learning, classical learning, and deep learning ones Zhao et al. (). Non-learning methods include density-based methods (Breunig et al., ; Kriegel et al., ; Tang et al., ; ) that determine which data points are abnormal by analyzing the density of the distribution of data points in the cluster, and similarity-based methods (Yeh et al., ) that series significantly dissimilar from most series are marked as abnormal. Classical learning methods (Liu et al., ) partition the time series into windows and use a training dataset consisting of only normal data to classify whether the test data is similar to the normal data or not (Schölkopf et al., ). Deep learning methods mainly contain reconstruction-based and prediction-based methods. The reconstruction-based methods compress the raw input data and then attempt to reconstruct this representation, the abnormal data is often difficult to reconstruct correctly (Zhao et al., ). This class of methods uses the difference between the reconstructed result and the original input as a criterion to determine the anomaly. The classic approaches include the use of AEs in  (Campos et al., ; Krizhevsky et al., ), VAEs in  (Su et al., ; Park et al., ; Li et al., ) or GANs in  (Zhou et al., ; Li et al., ; Schlegl et al., ), and more recently, the highly successful utilization of transformer architectures in (Xu et al., ; Yang et al., ; Chen et al., ). The prediction-based methods (Pang et al., ) use past observations to predict the current value and use the difference between the predicted result and the real value as an anomaly criterion. Unlike these methods that require building a model for each dataset, our method mainly focuses on constructing a general time series anomaly detection model by pre-training on multi-domain time series data, which can be efficiently applied to various target datasets.

#### Time series pre-training models.

Time series pre-training has received extensive attention. Some works develop general training strategies for time series. PatchTST (Nie et al., ) and SimMTM (Dong et al., ) employ the BERT-style masked pre-training. InfoTS (Luo et al., ) and TS2Vec (Yue et al., ) utilize contrastive learning between augmented views of data. Additionally, Some works repurpose large language models to time series tasks (Gruver et al., ), such as GPT4TS (Zhou et al., ). Recently, novel time series pre-training models with zero-shot forecasting capabilities have emerged (Garza & Canseco, ; Woo et al., ; Liu et al., ; Dooley et al., ; Liu et al., ; Gruver et al., ), which demonstrate remarkable performance even without training on the target datasets. Nonetheless, there remains a dearth of time series anomaly detection models that support zero-shot. Existing pre-training methods for anomaly detection necessitate training on the target dataset. Different from previous methods, DADA pre-trained on multi-domain datasets is uniquely designed for time series anomaly detection, excelling in zero-shot anomaly detection across various target scenarios, thus filling this gap in the field.

## 3 Methodology

Given a time series $\mathcal{D}_{\text{test}}=\left(\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x
}_{T}\right)\in\mathbb{R}^{d\times T}$ containing $T$ successive observations, where $d$ is the data dimensions. Time series anomaly detection outputs $\mathcal{\hat{Y}}_{\text{text}}=\left(y_{1},y_{2},\cdots,y_{T}\right)$, where $y_{t}\in\{0,1\}$ denoting whether the observation $\mathbf{x}_{t}\in\mathbb{R}^{d}$ at a certain time $t$ is anomalous or not. In this paper, we focus on building a general time series anomaly detection model that is pre-trained on $M$ multi-domain time series datasets $\mathcal{D}=\{\mathcal{D}^{(i)}\}_{i=1}^{M}$, where each dataset $\mathcal{D}^{(i)}=\left(\mathbf{x}^{(i)}_{1},\mathbf{x}^{(i)}_{2},...,\mathbf{
x}^{(i)}_{T^{(i)}}\right)\in\mathbb{R}^{d^{(i)}\times T^{(i)}}$ has $d^{(i)}\text{-variates}$ and $T^{(i)}$ time points. Then, the pre-trained model can be applied for anomaly detection on a target downstream dataset $\mathcal{D}_{\text{test}}\notin\mathcal{D}$ without any fine-tuning. Time series from different domains have varying numbers of variates.
Thus, we use channel independence to extend our model across different domains (Nie et al., ; Yang et al., ; Goswami et al., ; Liu et al., ). Specifically, a $d$-variate time series is processed into $d$ separate univariate time series.

### 3.1 Overall architecture

We propose DADA, a novel general time series anomaly Detector with Adaptive bottlenecks and Dual Adversarial decoders. As shown in Figure 2, DADA employs a mask-based reconstruction architecture to model time series. During the pre-training stage, we utilize normal series, where no time points are labeled as anomalies, and abnormal series, which contains noise perturbation. Both normal and abnormal time series are concurrently input to DADA. The patch and complementary mask module maps each series to masked patch embeddings, which are fed into the encoder to extract temporal features. DADA employs adaptive bottlenecks for multi-domain time series data by dynamically selecting suitable bottleneck sizes. Finally, the dual adversarial decoders are employed to reconstruct the normal and abnormal series. During the inference stage, the anomaly decoder is removed, and DADA employs multiple complementary masks to produce multiple pairs of masked patch embeddings. The variance of the reconstructed values serves as the anomaly score, where the normal data can be stably reconstructed.

Figure: Figure 2: (a) The workflow during the pre-training stage. DADA mainly consists of Patch and Complementary Mask, Encoder, Adaptive Bottlenecks, and Dual Adversarial Decoders. (b) The structure of Adaptive Bottlenecks. (c) The workflow during the inference stage.
Refer to caption: x2.png

### 3.2 Complementary mask modeling

Time series mask modeling reconstructs masked parts using unmasked parts, to capture normal temporal dependencies from unmasked parts to masked parts. High reconstruction errors indicate anomalies that deviate from normal behavior. In this paper, we employ a complementary mask strategy by generating a pair of masked series with mutually complementary mask positions, which reconstruct each other to further capture comprehensive bi-directional temporal dependencies.

The complementary mask modeling consists of three steps. First, we split a univariate input time series into $P$ patches with patch size $S$, denoted as $\mathbf{X}\in\mathbb{R}^{P\times S}$. The division of time series data with patches has proven to be very helpful for capturing local information within each patch and learning global information among different patches, enabling the capture of complex temporal patterns (Nie et al., ; Yang et al., ). Then, we generate a masked series by randomly masking a portion of patches along the temporal dimension, formalized by $\mathbf{X}_{\text{m}}=\mathbf{M}\odot\mathbf{X}$, where $\mathbf{M}\in\{0,1\}^{P\times S}$ is the mask and $\odot$ is the element-wise multiplication. At the same time, we generate another masked time series that complements $\mathbf{X}_{\text{m}}$ according to the mask $\mathbf{M}$, formalized by $\mathbf{\bar{X}}_{\text{m}}=(1-\mathbf{M})\odot\mathbf{X}$. Then, we reconstruct the masked parts, $\mathbf{\bar{X}}_{\text{m}}$, based on $\mathbf{X}_{\text{m}}$, and vice versa. This provides the model fully utilizing all data points to capture comprehensive bi-directional temporal dependencies. Finally, we combine the complementary reconstructed results. Denoting the process of reconstruction as $\mathrm{Recon}(\cdot)$, the final reconstructed results $\mathbf{\hat{X}}$ is calculated by:

$$ $\mathbf{\hat{X}}=(1-\mathbf{M})\odot\mathrm{Recon}(\mathbf{X}_{\text{m}})+ \mathbf{M}\odot\mathrm{Recon}(\mathbf{\bar{X}}_{\text{m}}).$ (1) $$

In this paper, we employ this complementary mask reconstruction architecture to reconstruct normal and abnormal series respectively, leveraging the reconstruction outputs for model training and anomaly detection as detailed in Section 3.4.

### 3.3 Adaptive bottlenecks

As aforementioned, DADA is pre-trained on multi-domain time series datasets and applied to a wide range of downstream scenarios. This requires DADA to have the ability to learn generalizable representations from multi-domain time series datasets that show significant differences in data distribution, noise levels, etc. (Woo et al., ), and exhibit distinct preferences for the bottleneck. The size of the latent space can be considered as the internal information bottleneck of the model (Wang et al., ). A large bottleneck may cause the model to fit unnecessary noise, while a small bottleneck may result in the loss of diverse normal patterns. The fixed bottleneck approaches used by existing methods (Campos et al., ; Su et al., ; Sakurada & Yairi, ) are poor in generalization ability and fail to detect anomalies across domains. Thus, we innovatively consider the model’s generalization ability from the perspective of a dynamic bottleneck and propose an Adaptive Bottlenecks module (AdaBN) integrated with an adaptive router and a bottleneck pool, which dynamically allocates appropriate bottlenecks for multi-domain time series, enhancing the generalization ability of the model and enabling it to directly apply in various target scenarios.

#### Bottleneck pool.

We compress the features into different latent spaces to achieve different information bottlenecks with different information densities. To accommodate the varying requirements of the multi-domain time series data, we have configured a pool of $B$ different sizes of bottlenecks $\mathrm{BN}_{i}(\cdot),\ i=1,2,...,B$, and each of them can compress the features into a different-sized latent space with dimension $d_{i}$. The masked time series input encoder can generate the corresponding representation $\mathbf{z}\in\mathbb{R}^{d_{\text{r}}}$, where $d_{\text{r}}$ denotes the dimension of representation. The process of each bottleneck can be formalized as:

$$ $\mathrm{BN}_{i}(\mathbf{z})=\mathrm{UpNet}_{i}\left(\mathrm{DownNet}_{i}\left( \mathbf{z}\right)\right).$ (2) $$

$\mathrm{DownNet}_{i}(\cdot)$ represents the network which compresses the representation $\mathbf{z}$ into a latent space, $\mathbb{R}^{d_{\text{r}}}\rightarrow\mathbb{R}^{d_{\text{i}}}$, where $d_{\text{i}}$ is the latent space dimension for the i-th bottleneck $\mathrm{BN}_{i}(\cdot)$ and $d_{\text{i}}<d_{\text{r}}$. $\mathrm{UpNet}_{i}(\cdot)$ restores the representation $\mathbf{z}$ from the latent space, $\mathbb{R}^{d_{\text{i}}}\rightarrow\mathbb{R}^{d_{\text{r}}}$.

#### Adaptive router.

Employing all bottlenecks within the bottleneck pool indiscriminately will diminish model performance due to the mismatching between the information capacity of the latent space and the information densities of the data. An ideal model would dynamically allocate appropriate bottlenecks based on the intrinsic properties of the time series data. Therefore, we introduce the Adaptive Router, a dynamic allocation strategy that can flexibly select bottleneck size for each time series to address the flexible reconstruction requirements. As shown in Figure 2(b), the adaptive router employs a routing function to generate the selecting weights of each bottleneck from the bottleneck pool based on the representation. To avoid repeatedly selecting certain bottlenecks, causing the corresponding bottlenecks to be repeatedly updated while neglecting other potentially suitable bottlenecks, we add noise terms to increase randomness. We obtain the overall formula for the routing function $\mathrm{R}(\mathbf{z})$ as:

$$ $\mathrm{R}(\mathbf{z})=\mathbf{z}\mathbf{W}_{\text{router}}+\epsilon\cdot \mathrm{Softplus}\left(\mathbf{z}\mathbf{W}_{\text{noise}}\right),\epsilon\sim \mathcal{N}\left(0,1\right),$ (3) $$

where $\mathbf{W}_{\text{router}}$ and $\mathbf{W}_{\text{noise}}\in\mathbb{R}^{{d_{\text{r}}}\times B}$ are learnable matrices for weights generation, $B$ is the number of bottlenecks and $\mathrm{Softplus}$ is the activation function. $\mathrm{R}(\mathbf{z})$ maps representation $\mathbf{z}$ to selection weights for $B$ bottlenecks. To encourage the model to update key bottlenecks, we select $k$ bottlenecks with the highest weights, denoting the set of their indexes as $\mathcal{K}$. Last, we assign higher attention weights to the output of more important bottlenecks and fuse their final output as:

$$ $\mathrm{AdaBN}(\mathbf{z})=\sum_{i\in\mathcal{K}}\frac{\exp\left(\mathrm{R} \left(\mathbf{z}\right)_{i}\right)}{\sum_{j\in\mathcal{K}}\exp\left(\mathrm{R} \left(\mathbf{z}\right)_{j}\right)}\mathrm{BN}_{i}\left(\mathbf{z}\right).$ (4) $$

### 3.4 Dual adversarial decoders

#### Reconstruction of normal time series.

As shown in Eq. (1), each input series $\mathbf{X}$ is reconstructed into result $\mathbf{\hat{X}}$. To achieve the goal of anomaly detection, we learn normal patterns from the reconstruction of normal time series. As shown in Figure 3, we use a feature extractor $G(\cdot;\theta_{g})$, and a normal decoder $D_{\text{n}}(\cdot;\theta_{n})$ with parameter $\theta_{g}$ and $\theta_{n}$ to denote the parts of the model used for reconstructing normal series. The $G$ includes the patch and complementary mask module, the encoder, and the adaptive bottlenecks. The decoder $D_{\text{n}}$ uses the output from the adaptive bottlenecks to reconstruct the series. We learn normal time series patterns by minimizing the reconstruction error of normal series:

$$ $\mathcal{L}_{\text{n}}(\theta_{g},\theta_{n})=\sum_{i=1}^{N_{\text{n}}}\| \mathbf{X}_{\text{n}}^{(i)}-\mathbf{\hat{X}}_{\text{n}}^{(i)}\|_{2}^{2}=\sum_{ i=1}^{N_{\text{n}}}\|\mathbf{X}_{\text{n}}^{(i)}-D_{n}(G(\mathbf{X}_{\text{n}} ^{(i)};\theta_{g});\theta_{n})\|_{2}^{2},$ (5) $$

where $N_{\text{n}}$ is the number of normal training series. $\mathbf{X}_{\text{n}}^{(i)}$ is the $i\text{-th}$ normal series and $\mathbf{\hat{X}}_{\text{n}}^{(i)}$ is the corresponding reconstruction result.

Figure: Figure 3: The forward and backward propagation of normal data and abnormal data.
Refer to caption: x3.png

#### Reconstruction of abnormal time series.

However, merely learning to capture normal patterns is insufficient to detect anomalies in new scenarios. As a general time series anomaly detection model, the model will confront multi-domain data with diverse anomaly manifestations, and some patterns may only be specific to a certain domain and fail to generalize across domains, making the decision boundaries more complex. Thus, we need to explicitly enhance the model’s ability to discriminate between normal patterns and some common abnormal patterns. We incorporate abnormal series with noise perturbations representing common anomaly patterns into the pre-training stage and propose an adversarial training stage that minimizes reconstruction errors for normal series while maximizing them for abnormal ones. Simply amplifying the reconstruction error for anomalies can confuse the feature extractor $G$ and fail to learn normal patterns. To address this, we introduce an anomaly decoder $D_{\text{a}}(\cdot;\theta_{a})$ with parameter $\theta_{a}$ for adversarial training to constrain the model and encourage the feature extractor $G(\cdot;\theta_{g})$ to learn normal pattern features. We seek the parameters $\theta_{g}$ of the feature extractor that maximizes the reconstruction loss of the abnormal data to make the representations contain abnormal information as little as possible, while simultaneously seeking the parameters $\theta_{a}$ of the anomaly decoder that minimizes the reconstruction loss of the abnormal data to learn anomaly patterns. The reconstruction loss for abnormal series can be formalized as:

$$ $\mathcal{L}_{\text{a}}(\theta_{g},\theta_{a})=\sum_{i=1}^{N_{\text{a}}}\|\left (\mathbf{X}_{\text{a}}^{(i)}-\mathbf{\hat{X}}_{\text{a}}^{(i)}\right)\odot \mathbf{Y}_{\text{a}}^{(i)}\|_{2}^{2}=\sum_{i=1}^{N_{\text{a}}}\|\left(\mathbf {X}_{\text{a}}^{(i)}-D_{\text{a}}(G(\mathbf{X}_{\text{a}}^{(i)};\theta_{g}); \theta_{a})\right)\odot\mathbf{Y}_{\text{a}}^{(i)}\|_{2}^{2},$ (6) $$

where $N_{\text{a}}$ is the number of abnormal training series, $\mathbf{X}_{\text{a}}^{(i)}$ and $\mathbf{Y}_{\text{a}}^{(i)}$ denote the $i\text{-th}$ abnormal series and its anomaly label. Our method does not strictly require labeled data.
To reduce dependence on human-labeled data and to avoid overfitting to specific anomaly patterns, we generate anomalous data with more common anomaly patterns through anomaly injection. More details about anomaly injection are shown in Appendix A.3.

#### Model training.

As aforementioned, the overall optimization goal is to seek the parameters $\theta_{g}$ of feature extractor that maximize Eq. (6), while simultaneously seeking the parameters $\theta_{a}$ of anomaly decoder that minimize Eq. (6). In addition, we seek to minimize Eq. (5). As shown in Figure 3, we employ a Gradient Reversal Layer (GRL) (Ganin & Lempitsky, ) between $G$ and ${D}_{\text{a}}$. GRL alters the gradient from $D_{\text{a}}$, multiplies it by $-\lambda$, and passes it to $G$. That is to say, the partial derivatives of the loss for the encoder $\frac{\partial\mathcal{L}_{a}}{\partial\theta_{g}}$ is replaced with $-\lambda\frac{\partial\mathcal{L}_{a}}{\partial\theta_{g}}$, guiding parameters of different optimization objectives toward the desired direction for gradient descent, thereby avoiding the need for two separate optimizations. Finally, our objective can be formalized by:

$$ $\displaystyle(\theta_{g}^{*},\theta_{n}^{*})$ $\displaystyle=\arg\min_{\theta_{g},\theta_{n}}\mathcal{L}_{\text{n}}(\theta_{g },\theta_{n})-\mathcal{L}_{\text{a}}(\theta_{g},\theta_{a}),$ (7) $\displaystyle\theta_{a}^{*}$ $\displaystyle=\arg\min_{\theta_{a}}\mathcal{L}_{\text{a}}(\theta_{g},\theta_{a }).$ $$

#### Anomaly criterion.

During the inference stage, the model generates multiple pairs of complementary masked series for a single series and outputs multiple reconstructed series. Normal data points can be stably reconstructed, with the higher proximity of the reconstruction values. Conversely, reconstructions of abnormal data points are difficult and tend to be more unstable. Therefore, we utilize the variance of reconstructed values at the same time point as the anomaly score.
Following existing works (Wang et al., ; Su et al., ), we run the SPOT (Siffer et al., ) to get the threshold $\delta$, and a time point is marked as an anomaly if its anomaly score is larger than $\delta$.

## 4 Experiments

### 4.1 Experimental settings

#### Datasets.

We employ a subset of the Monash (Godahewa et al., ) data hub and 12 datasets (Li et al., ; Jacob et al., ; Ren et al., ; Roggen et al., ; Cui et al., ; Thill et al., ; Moody & Mark, ; Greenwald et al., ; Laptev et al., ) from anomaly detection task for pre-training. The pre-training datasets consist of approximately 400 million time points and intricately represent a wide range of domains. To demonstrate the effectiveness of our method, we evaluate five widely used benchmark datasets: SMD (Su et al., ), MSL (Hundman et al., ), SMAP (Hundman et al., ), SWaT (Mathur & Tippenhauer, ), PSM (Abdulaal et al., ), and NeurIPS-TS (including CICIDS, Creditcard, GECCO and SWAN) (Lai et al., ). We also evaluate UCR (Wu & Keogh, ) in the Appendix C.2. None of the downstream datasets for evaluation are included in the pre-training datasets. More details about the pre-training and evaluation datasets are shown in Appendix A.

#### Baselines.

We compare our model with 19 baselines for comprehensive evaluations, including the linear transformation-based models: OCSVM (Schölkopf et al., ), PCA (Shyu et al., ); the density estimation-based methods: HBOS (Goldstein & Dengel, ), LOF (Breunig et al., ); the outlier-based methods: IForest (Liu et al., ), LODA (Pevný, ); the neural network-based models: AutoEncoder (AE) (Sakurada & Yairi, ), DAGMM (Zong et al., ), LSTM (Hundman et al., ), CAE-Ensemble (Campos et al., ), BeatGAN (Zhou et al., ), OmniAnomaly (Omni) (Su et al., ), Anomaly Transformer (A.T.) (Xu et al., ), MEMTO (Song et al., ), DCdetector (Yang et al., ), D3R (Wang et al., ), GPT4TS (Zhou et al., ), ModernTCN (donghao & wang xue, ), SensitiveHUE (Feng et al., ). To ensure a fair comparison, we adopt the same settings across all baselines, including the metrics and thresholding protocol, following common practices in anomaly detection.

#### Metrics.

Many existing methods use point adjustments (PA) to adjust the detection result. However, recent works have demonstrated that PA can lead to faulty performance evaluations (Huet et al., ). Even if only one point in an anomaly segment is correctly detected, PA will assume that the model has detected the entire segment correctly, which is unreasonable (Wang et al., ). To overcome this problem, we use the affiliation-based F1-score (F1) (Huet et al., ), which has been widely used recently (Wang et al., ; Yang et al., ). This score takes into account the average directed distance between predicted anomalies and ground truth anomaly events to calculate affiliated precision (P) and recall (R). Since the precision and recall are significantly influenced by thresholds, focusing solely on one of them fails to provide a comprehensive evaluation of the model, and thus anomaly detection requires balancing these two metrics. Therefore, we pay more attention to the F1 score like widely used in anomaly detection Yang et al. (); Xu et al. (). We also employed the AUC_ROC (AUC) metric (Wang et al., ; Campos et al., ).
More implement details are shown in the Appendix B.
All results are in %. The best ones are in bold and the second ones are underlined.

**Table 1: Results for five real-world datasets.**
| Dataset | SMD | MSL | SMAP | SWaT | PSM |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 |
| OCSVM | 66.98 | 82.03 | 73.75 | 50.26 | 99.86 | 66.87 | 41.05 | 69.37 | 51.58 | 56.80 | 98.72 | 72.11 | 57.51 | 58.11 | 57.81 |
| PCA | 64.92 | 86.06 | 74.01 | 52.69 | 98.33 | 68.61 | 50.62 | 98.48 | 66.87 | 62.32 | 82.96 | 71.18 | 77.44 | 63.68 | 69.89 |
| HBOS | 60.34 | 64.11 | 62.17 | 59.25 | 83.32 | 69.25 | 41.54 | 66.17 | 51.04 | 54.49 | 91.35 | 68.26 | 78.45 | 29.82 | 43.21 |
| LOF | 57.69 | 99.10 | 72.92 | 49.89 | 72.18 | 59.00 | 47.92 | 82.86 | 60.72 | 53.20 | 96.73 | 68.65 | 53.90 | 99.91 | 70.02 |
| IForest | 71.94 | 94.27 | 81.61 | 53.87 | 94.58 | 68.65 | 41.12 | 68.91 | 51.51 | 53.03 | 99.95 | 69.30 | 69.66 | 88.79 | 78.07 |
| LODA | 66.09 | 84.37 | 74.12 | 57.79 | 95.65 | 72.05 | 51.51 | 100.00 | 68.00 | 56.30 | 70.34 | 62.54 | 62.22 | 87.38 | 72.69 |
| AE | 69.22 | 98.48 | 81.30 | 55.75 | 96.66 | 70.72 | 39.42 | 70.31 | 50.52 | 54.92 | 98.20 | 70.45 | 60.67 | 98.24 | 75.01 |
| DAGMM | 63.57 | 70.83 | 67.00 | 54.07 | 92.11 | 68.14 | 50.75 | 96.38 | 66.49 | 59.42 | 92.36 | 72.32 | 68.22 | 70.50 | 69.34 |
| LSTM | 60.12 | 84.77 | 70.35 | 58.82 | 14.68 | 23.49 | 55.25 | 27.70 | 36.90 | 49.99 | 82.11 | 62.15 | 57.06 | 95.92 | 71.55 |
| BeatGAN | 74.11 | 81.64 | 77.69 | 55.74 | 98.94 | 71.30 | 54.04 | 98.30 | 69.74 | 61.89 | 83.46 | 71.08 | 58.81 | 99.08 | 73.81 |
| Omni | 79.09 | 75.77 | 77.40 | 51.23 | 99.40 | 67.61 | 52.74 | 98.51 | 68.70 | 62.76 | 82.82 | 71.41 | 69.20 | 80.79 | 74.55 |
| CAE-Ensemble | 73.05 | 83.61 | 77.97 | 54.99 | 93.93 | 69.37 | 62.32 | 64.72 | 63.50 | 62.10 | 82.90 | 71.01 | 73.17 | 73.66 | 73.42 |
| MEMTO | 49.69 | 98.05 | 65.96 | 52.73 | 97.34 | 68.40 | 50.12 | 99.10 | 66.57 | 56.47 | 98.02 | 71.66 | 52.69 | 83.94 | 64.74 |
| A.T. | 54.08 | 97.07 | 66.42 | 51.04 | 95.36 | 66.49 | 56.91 | 96.69 | 71.65 | 53.63 | 98.27 | 69.39 | 54.26 | 82.18 | 65.37 |
| DCdetector | 50.93 | 95.57 | 66.45 | 55.94 | 95.53 | 70.56 | 53.12 | 98.37 | 68.99 | 53.25 | 98.12 | 69.03 | 54.72 | 86.36 | 66.99 |
| SensitiveHUE | 60.34 | 90.13 | 72.29 | 55.92 | 98.95 | 71.46 | 53.63 | 98.37 | 69.42 | 58.91 | 91.71 | 71.74 | 56.15 | 98.75 | 71.59 |
| D3R | 64.87 | 97.93 | 78.02 | 66.85 | 90.83 | 77.02 | 61.76 | 92.55 | 74.09 | 60.14 | 97.57 | 74.39 | 73.32 | 88.71 | 80.29 |
| ModernTCN | 74.07 | 94.79 | 83.16 | 65.94 | 93.00 | 77.17 | 69.50 | 65.45 | 67.41 | 59.14 | 89.22 | 71.13 | 73.47 | 86.83 | 79.59 |
| GPT4TS | 73.33 | 95.97 | 83.13 | 64.86 | 95.43 | 77.23 | 63.52 | 90.56 | 74.67 | 56.84 | 91.46 | 70.11 | 73.61 | 91.13 | 81.44 |
| DADA (zero shot) | 76.50 | 94.54 | 84.57 | 68.70 | 91.51 | 78.48 | 65.85 | 88.25 | 75.42 | 61.59 | 94.59 | 74.60 | 74.31 | 92.11 | 82.26 |

### 4.2 Main results

Each evaluation dataset has a training set and a test set. The standard baseline methods are trained on the training set and tested on the corresponding test set. Differently, our model DADA follows the zero-shot protocol. After the model is pre-trained on multi-domain datasets, it does not train on the downstream training set and is directly tested on the downstream test set. Note that all downstream evaluation datasets are not included in our pre-training datasets.
As shown in Table 1, compared to the baselines directly trained with full data in each specific dataset, DADA, as a zero-shot detector, achieves state-of-the-art results in all five datasets, which demonstrates that DADA learned a general detection ability from a wide range of pre-training data, with clear distinguishment between multiple normal and anomalous patterns.
We also evaluate our DADA on the NeurIPS-TS benchmark in Table 2, compared with the five recent methods that perform well as shown in Table 1. It can observed that these baselines fail to achieve consistent good results across different datasets. However, benefiting from pre-training on large-scale data, DADA achieves the best or most competitive results on all datasets, including the NeurIPS-TS datasets which are rich in anomaly types (Xu et al., ; Yang et al., ).
It demonstrates the superior adaptability of DADA for different anomaly detection scenarios. More other metrics evaluation details can be found in Appendix C.

**Table 2: Overall results for the NeurIPS-TS datasets.**
| Dataset | CICIDS | Creditcard | GECCO | SWAN |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | F1 | AUC | F1 | AUC | F1 | AUC | F1 | AUC |
| A.T. | 34.71 | 49.00 | 65.14 | 52.55 | 64.27 | 51.60 | 33.67 | 44.74 |
| DCdetector | 40.02 | 53.95 | 58.28 | 42.36 | 66.18 | 45.38 | 14.42 | 43.48 |
| D3R | 67.79 | 41.99 | 72.03 | 93.59 | 91.83 | 80.72 | 43.19 | 37.31 |
| ModernTCN | 51.74 | 65.33 | 73.80 | 95.55 | 90.18 | 95.95 | 46.45 | 52.63 |
| GPT4TS | 54.00 | 67.91 | 72.88 | 95.58 | 88.11 | 90.21 | 47.27 | 51.93 |
| DADA (zero shot) | 73.49 | 69.33 | 75.12 | 95.73 | 90.20 | 93.44 | 71.93 | 53.29 |

#### Ablation study.

We further delve into the impact of individual components on the performance of DADA in zero-shot setting, as shown in Table 3. Firstly, we explore the impact of the AdaBN module. The first line results use a single fixed bottleneck for multi-domain datasets, eliminating the adaptive selection for various bottlenecks. When we removed the AdaBN module, the outcomes exhibited a decline of 5.04%.
This emphasizes the criticality of dynamic bottlenecks for multi-domains pre-training.
Then, we also examine the influence of the dual adversarial decoders and its adversarial mechanism. The second line results directly maximize the reconstruction error of abnormal data, removing the adversarial mechanism. The third line results use a single decoder to reconstruct normal time series and abnormal time series, and the fourth line results simultaneously remove the adversarial mechanism and use a single decoder. We observe degradation in performance of w/o Adversarial by 16.26% (79.49%$\to$63.23%) compared with DADA, and 7.89% (71.17%$\to$63.23%) compared with the fifth line results, suggesting the needs for confrontation with the feature extractor, and directly maximizing abnormal time series will increase confusion. Additionally, employing a single decoder to handle both normal and abnormal time series causes zero-shot performance to degrade, instructing the necessity of dual decoders.

**Table 3: Ablation studies. We study the influence of adaptive bottlenecks, adversarial training, and dual decoders. All results are in %, and the best ones are in bold.**
| Dataset | SMD | MSL | SMAP | Avg |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AdaBN | Adversarial | Dual Decoders | P | R | F1 | P | R | F1 | P | R | F1 | F1 |
| ✗ | ✓ | ✓ | 71.13 | 93.61 | 80.84 | 62.79 | 91.74 | 74.55 | 58.91 | 84.10 | 69.29 | 74.45 |
| ✓ | ✗ | ✓ | 69.46 | 55.32 | 61.59 | 49.89 | 88.57 | 63.83 | 52.62 | 82.59 | 64.28 | 63.23 |
| ✓ | ✓ | ✗ | 75.77 | 87.82 | 81.35 | 64.82 | 84.32 | 73.29 | 61.50 | 80.27 | 69.64 | 74.76 |
| ✓ | ✗ | ✗ | 77.82 | 89.95 | 83.44 | 67.54 | 88.56 | 76.63 | 65.65 | 83.09 | 73.34 | 77.81 |
| ✗ | ✗ | ✗ | 78.60 | 74.96 | 76.74 | 65.20 | 72.69 | 68.74 | 55.22 | 88.57 | 68.03 | 71.17 |
| ✓ | ✓ | ✓ | 76.50 | 94.54 | 84.57 | 68.7 | 91.51 | 78.48 | 65.85 | 88.25 | 75.42 | 79.49 |

### 4.3 Model analysis

We analyze the effectiveness of adaptive bottlenecks and multi-domain pre-training and visualize the anomaly scores. We conducted additional analytical experiments, and the results are presented in Appendix D and Appendix E.

#### Finetune with downstream data.

Figure: Figure 4: Results of fine-tuning DADA under different data percentages.
Refer to caption: x4.png

As depicted in Figure 4, we further demonstrate the results of fine-tuning DADA on downstream datasets under different data scarcities. After pre-training, the model develops strong generalization capabilities and exhibits promising zero-shot anomaly detection performance.
By applying the model to a downstream dataset for further learning of domain-specific patterns, the model’s detection ability can be further enhanced. After fine-tuning, the model’s performance increases from 78.48% to 79.90% on MSL, and from 84.57% to 85.50% on SMD.

#### Analysis on adaptive bottlenecks.

To validate the effectiveness of AdaBN, we remove the AdaBN module and instead employ a fixed bottleneck size. As shown in Figure 5(left), we observe that different bottleneck sizes exhibit different preferences for various downstream scenarios, while also introducing notable drawbacks in certain scenarios, e.g. the model with a bottleneck size of 32 achieves 74.81% on Creditcard, whereas only 67.56% on MSL. However, DADA, with adaptive bottlenecks dynamically allocating appropriate bottlenecks, consistently outperforms models utilizing a single bottleneck, which further substantiates the efficacy of our method. We also visualize the proportion of different adaptive bottleneck sizes that each dataset selects in Appendix E.5

Figure: Figure 5: Model Analysis. (left) Comparison between DADA using AdaBN and using a single bottleneck. (right) Pre-training baseline model on multi-domain datasets and then evaluate them as a zero-shot detector on the target dataset.
Refer to caption: x5.png

#### Analysis on multi-domain pre-training.

We further evaluate the baseline’s zero-shot performance after multi-domain pre-training. Adopting the same setting as DADA, these models are first pre-trained extensively on multi-domain datasets, followed by zero-shot evaluation on the target datasets. From the presented Figure 5(right), we can observe that these baselines are incapable of effectively extracting generalization capability from multi-domain datasets, thus failing to deliver satisfactory results on the target dataset. This indicates that although multi-domain pre-training is important, simply using multi-domain datasets without proper methods still cannot obtain desired generalization ability. In contrast, DADA, equipped with its unique adaptive bottlenecks and dual adversarial decoders module tailored for the task of general anomaly detection, demonstrates great multi-domain pre-training and general detection ability.

#### Visual analysis.

To further explain the efficacy of each module in DADA, we randomly sample time segments of length 100 for visualization. Figure 6 illustrates the anomaly scores at every timestamp within a given period and ground truth labels. After DADA removes AdaBN (W/O AdaBN), the model cannot adapt well to the normal pattern of multi-domain data and gets a higher false positive. After removing the dual decoders (W/O Dual Decoders), the model’s decision boundary between normal and abnormal data becomes blurred and the false negative becomes higher. DADA (ours) shows that DADA not only achieves superior detection accuracy but also maintains a lower false alarm rate, a testament to its refined performance in time series anomaly detection.

Figure: Figure 6: Visualization of the anomaly scores. The pink intervals denote the ground truth anomalies, while the blue lines represent the anomaly scores produced by corresponding models. A red horizontal dashed line signifies the threshold. Any timestamp with an anomaly score exceeding this threshold is identified as an anomaly.
Refer to caption: x6.png

## 5 Conclusion

This paper proposes a novel general time series anomaly detection model DADA. By pre-training on extensive multi-domain datasets, DADA can subsequently apply to a multitude of downstream scenarios without fine-tuning. We propose adaptive bottlenecks to meet the diverse requirements of information bottlenecks tailored to different datasets in one unified model. We also propose dual adversarial decoders to explicitly enhance clear differentiation between normal and abnormal patterns. Extensive experiments prove that as a zero-shot detector, DADA still achieves competitive or even superior results compared to those models tailored to each specific dataset.

## Appendix A Datasets

### A.1 Datasets for evaluation

We evaluate DADA and baselines on various datasets come from different domains, which can be broadly divided into spacecraft, servers, water treatment, finance, networking, and space weather: (1) SMD (Server Machine Dataset) collects resource utilization of computer clusters from an Internet company (Su et al., ). (2) MSL (Mars Science Laboratory dataset), collected by NASA, encompasses telemetry data reflecting the operational status of both sensors and actuators aboard the Martian rover (Hundman et al., ). (3) SMAP (Soil Moisture Active Passive dataset) is also collected by NASA and reflects soil moisture data from spacecraft monitoring systems (Hundman et al., ). (4) SWaT (Secure Water Treatment) collects sensor data from a continuously operating infrastructure (Mathur & Tippenhauer, ). (5) PSM (Pooled Server Metrics dataset) is collected from eBay Server Machines (Abdulaal et al., ). (6) NeurIPS-TS (NeurIPS 2021 Time Series Benchmark) is a dataset proposed by  (Lai et al., ) and we use four sub-datasets: CICIDS, Creditcard, GECCO, SWAN with a variety of anomaly scenarios. (7) UCR Dataset is collected by Wu & Keogh (), which contains 250 sub-datasets. Each sub-datasets contains one dimension and one anomaly segment. For MSL and SMAP datasets, we only retain the first continuous dimension (Wang et al., ). We summarize the datasets in Table 4.

**Table 4: Details of benchmark datasets for evaluation. AR (anomaly ratio) represents the abnormal proportion of the whole dataset.**
| Dataset | Domain | Dimension | Training | Validation | Test(labeled) | AR(%) |
| --- | --- | --- | --- | --- | --- | --- |
| MSL | Spacecraft | 55 | 46,653 | 11,664 | 73,729 | 10.5 |
| PSM | Server Machine | 25 | 105,984 | 26,497 | 87,841 | 27.8 |
| SMAP | Spacecraft | 25 | 108,146 | 27,037 | 427,617 | 12.8 |
| SMD | Server Machine | 38 | 566,724 | 141,681 | 708,420 | 4.2 |
| SWaT | Water treatment | 51 | 396,000 | 99,000 | 449,919 | 12.1 |
| Creditcard | Finance | 29 | 113,922 | 28,481 | 142,404 | 0.17 |
| GECCO | Water treatment | 9 | 55,408 | 13,852 | 69,261 | 1.25 |
| CICIDS | Web | 72 | 68,092 | 17,023 | 85,116 | 1.28 |
| SWAN | Space Weather | 38 | 48,000 | 12,000 | 60,000 | 23.8 |
| UCR | Natural | 1 | 1,790,680 | 447,670 | 6,143,541 | 0.6 |

### A.2 Datasets for pre-training

We collect two datasets for pre-training, namely Anomaly Detection Data and $\text{Monash}^{+}$, each of them contains normal time series and abnormal time series. None of the downstream datasets for evaluation are included in the pre-training datasets. Firstly, we collect the Anomaly Detection Data from time series anomaly detection task. Table 5 lists subsets in Anomaly Detection Data, including ASD (Li et al., ), Exathlon (Jacob et al., ), ECG (Moody & Mark, ), MITDB (Moody & Mark, ), OPP (Roggen et al., ), SVDB (Greenwald et al., ), GAIA(^1^11https://github.com/CloudWise-OpenSource/GAIA-DataSet), IOPS (Ren et al., ), MGAB (Thill et al., ), NYC (Cui et al., ), SKAB(^2^22https://www.kaggle.com/dsv/1693952), YAHOO (Laptev et al., ). In the field of time series anomaly detection, datasets usually contain default normal training data and labeled test data with anomalies. We take the training data of this part as normal data, and the data with anomaly labels in test data as abnormal data.

**Table 5: Anomaly Detection Data selected from multi-domain time series anomaly detection task. AR (anomaly ratio) represents the abnormal proportion of the whole dataset.**
| Dataset | Domain | Dimension | Length | AR(%) |
| --- | --- | --- | --- | --- |
| ASD | Application Server | 19 | 384,493 | 1.55 |
| Exathlon | Application Server | multi | 131,610 | 8.71 |
| ECG | Health | 1 | 9,681,300 | 4.7 |
| MITDB | Health | 1 | 13,650,000 | 3.44 |
| OPP | Health | 1 | 14,567,752 | 4.11 |
| SVDB | Health | 1 | 17,049,600 | 4.68 |
| GAIA | AIOps | 1 | 1,868,506 | 1.21 |
| IOPS | Web | 1 | 5,840,488 | 2.15 |
| MGAB | Mackey-Glass | 1 | 1,000,000 | 0.2 |
| NYC | Transport | 3 | 17,520 | 0.57 |
| SKAB | Machinery | 8 | 46,707 | 3.65 |
| YAHOO | Multiple | 1 | 565,827 | 0.62 |

To further enlarge our pre-training data, we build the $\text{Monash}^{+}$ dataset. The normal time series in $\text{Monash}^{+}$ comes from the Monash Prediction Library (Godahewa et al., ) as shown in Table 6, and the abnormal time series in $\text{Monash}^{+}$ are generated from Monash by anomaly injection. The anomaly injection is shown in Figure 7.
These time series with different variate numbers and series lengths from multi-domains reflect the diverse application of the real world and the overall pre-training datasets are summarized in Table 7.

**Table 6: Multi-domain time series datasets from the Monash Prediction Library.**
| Dataset | Domain | Frequency | Length |
| --- | --- | --- | --- |
| Aus. Electricity Demand | Energy | Half Hourly | 1,155,264 |
| Wind | Energy | 4 Seconds | 7,397,147 |
| Wind Farms | Energy | Minutely | 172,178,060 |
| Solar Power | Energy | 4 Seconds | 7,397,223 |
| Solar | Energy | 10 Minutes | 7,200,857 |
| London Smart Meters | Energy | Half Hourly | 166,527,216 |
| Saugeen River Flow | Nature | Daily | 23,741 |
| Sunspot | Nature | Daily | 73,924 |
| Weather | Nature | Daily | 43,032,000 |
| KDD Cup 2018 | Nature | Daily | 2,942,364 |
| US Births | Nature | Daily | 7,305 |
| FRED_MD | Economic | Monthly | 77,896 |
| Bitcoin | Economic | Daily | 75,364 |
| NN5 | Economic | Daily | 87,801 |

Figure: Figure 7: Anomaly injection. The blue lines represent the time series, while the pink areas intervals the anomalies we generated. We visualize the original time series and eight types of anomaly, including hmirror, vmirror, scale, outlier, noise, compress, stretch, and pattern.
Refer to caption: x7.png

### A.3 Anomaly injection

We use abnormal time series in the pre-training to explicitly enable robust distinguishment between normal and abnormal patterns with our dual adversarial decoders. Due to the scarcity of labeled data in the field of time series anomaly detection, we enrich abnormal samples by anomaly injection on the Monash dataset. These injected anomalies can be considered more common anomaly patterns. By conducting adversarial training on such data, we can prevent the model from overfitting to specific anomaly patterns that may change between domains. Utilizing methods such as vmirror, hmirror, compression, stretching, scaling, and others, as shown in Figure 7, we generate an expanded collection of novel time series exhibiting diverse anomaly types and patterns, thereby enhancing our model’s ability to distinguish between normal and abnormal time series.

**Table 7: Description of pre-training datasets with normal data and abnormal data.**
| Dataset | Normal Time Series | Abnormal Time Series |
| --- | --- | --- |
| Anomaly Detection Data | Training set | Test set |
| $\text{Monash}^{+}$ | All of Moansh data | Generated from Monash by anomaly injection |

## Appendix B Implementation details

We summarize all the default hyper-parameters as follows. We use a dilated CNN as an encoder with ten layers Yue et al. (). In the implementation phase, we run a sliding window with a window size of 100 and conduct anomaly detection using non-overlapping windows the same as existing works (Xu et al., ; Su et al., ). Our patch size is 5. The bottleneck pool contains the sizes [16, 32, 64, 128, 192, 256] and the adaptive router selects $k=3$ bottlenecks with the highest weights. The AdamW optimizer with an initial learning rate of $10^{-4}$ is used in pre-training. We set the batch size to 2048 with 5 epochs in pre-training. In the inference stage, we use 5 pairs of complementary masked series. After obtaining the anomaly score, we adopt commonly used SPOT (Siffer et al., ) following existing works (Wang et al., ; Su et al., ) to get the threshold $\delta$ to determine whether each point is an outlier (Wang et al., ). We conduct all experiments using Pytorch with NVIDIA Tesla-A800-80GB GPU.

## Appendix C Additional experimental results

### C.1 More evaluation metrics

In the realm of time series anomaly detection, there is a persistent debate regarding the most appropriate evaluation metrics to employ. This is due to the fact that different metrics can offer varying insights into the performance of a model, and the choice of metric can significantly influence the perceived effectiveness of an anomaly detection model. In our main text, we include the more reasonable Affiliated F1 and AUC_ROC scores. To further enhance the evaluation of our model, we also include the Volume Under the Surface (VUS) metric (Paparrizos et al., ), which takes anomaly events into consideration based on the receiver operator characteristic curve. As shown in Table 8, DADA achieves good results on most metrics. GPT4TS, as a good baseline, demonstrates outstanding performance on the SMD dataset. However, its performance across different datasets is not consistent. For instance, there remains a noticeable gap compared to DADA on the PSM dataset. In contrast, DADA exhibits robust detection capabilities across all three datasets.

**Table 8: Multi-metrics results.**
| Dataset | Method | F1 | AUC_ROC | R_AUC_ROC | R_AUC_PR | VUS_ROC | VUS_PR |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SMD | A.T. | 66.42 | 50.02 | 51.22 | 7.09 | 51.17 | 7.96 |
| D3R | 78.02 | 53.34 | 62.89 | 11.20 | 61.69 | 11.01 |  |
| GPT4TS | 83.13 | 71.15 | 77.27 | 17.68 | 76.79 | 17.45 |  |
| ModernTCN | 83.16 | 70.21 | 77.54 | 16.28 | 77.07 | 15.96 |  |
| DADA | 84.57 | 71.98 | 78.14 | 17.39 | 77.50 | 17.13 |  |
| MSL | A.T. | 66.49 | 34.05 | 38.65 | 10.25 | 38.90 | 10.41 |
| D3R | 77.02 | 45.77 | 56.35 | 20.05 | 55.69 | 19.76 |  |
| GPT4TS | 77.23 | 72.63 | 77.65 | 28.05 | 76.97 | 27.69 |  |
| ModernTCN | 77.17 | 74.96 | 77.88 | 30.91 | 77.47 | 30.10 |  |
| DADA | 78.48 | 75.15 | 77.99 | 31.05 | 77.48 | 30.50 |  |
| PSM | A.T. | 65.37 | 50.11 | 52.70 | 33.12 | 51.86 | 33.09 |
| D3R | 80.29 | 50.22 | 52.93 | 39.62 | 52.18 | 39.31 |  |
| GPT4TS | 81.44 | 58.94 | 65.16 | 46.54 | 64.66 | 45.99 |  |
| ModernTCN | 79.59 | 58.46 | 65.50 | 47.48 | 64.80 | 46.68 |  |
| DADA | 82.26 | 61.08 | 67.08 | 47.40 | 66.52 | 46.89 |  |

### C.2 Evaluation on the UCR benchmark

We evaluate DADA on UCR Anomaly Archive (Wu & Keogh, ), where the task is to find the position of an anomaly within the test set. To achieve this, we compute the anomaly scores for each time step in the test set and subsequently rank them in descending order. Following the approach of Timer (Liu et al., ), if the time step with the $\alpha$ quantile hits the labeled anomaly interval in the test set, the anomaly detection is accomplished. Figure 8 presents the results of our evaluation. The left figure displays the number of datasets in which the model successfully detect anomalies at quantile levels of $3\%$ and $10\%$; a higher count indicates a stronger detection capability of the model. The right figure shows the quantile distribution and the average quantile across all UCR datasets. DADA has a lower average quantile compared to Timer, and which indicates better performance of anomaly detection.

Figure: Figure 8: Downstream anomaly detection results for UCR benchmark
Refer to caption: x8.png

## Appendix D Parameter sensitivity

We study the parameter sensitivity of the DADA. Figure 9 shows the model performance under different window sizes, $k$ bottlenecks with the highest weights that the adaptive router can select, and different patch sizes. Window size is a very important parameter for time series analysis. In a time series, a single point does not contain any information, so the window size determines the length of the sample context. Our experiments show that DADA is insensitive to window size and performs well across different window sizes. We then explore the influence of parameter $k$ in the adaptive bottlenecks. From the performance point of view, our model shows a relatively stable effect under different $k$, where we can achieve good performance even when selecting 1 bottleneck for each sample. Finally, we study the performance under different patch sizes. Patch splitting has proven to be very helpful for capturing local information within each patch and learning global information among different patches. The results in Figure 9 show that DADA is robust under different patch sizes.

Figure: Figure 9: (a) Parameter sensitivity for sliding window size, (b) $k$ bottlenecks with the highest weights that the adaptive router can select, and (c) patch sizes (c).
Refer to caption: x9.png

## Appendix E Additional model analysis

### E.1 Show case

To provide a more intuitive illustration of how our model works, we employ anomaly injection to generate univariate time series featuring various anomalies, including outliers, horizontal mirroring, pattern shifts, and scale changes. As shown in Figure 10, our findings indicate that DADA can robustly and timely distinguish these anomalies from normal data points, which is crucial for real-world applications as it allows for the prompt prevention of potential losses. Moreover, leveraging the Dual Adversarial Decoders, if the model exhibits limitations in detecting some certain anomalies, its detection capabilities can be enhanced by incorporating the corresponding types of anomalies into the pre-training dataset.

Figure: Figure 10: Visualization of learned criterion. The blue line in the first row represents the time series, and the blue line in the second row represents the anomaly scores obtained by the model, while the pink areas intervals the anomalies.
Refer to caption: x10.png

### E.2 Without human-labeled data

Our model does not strictly rely on human-labeled data. As mentioned in Appendix A, we can generate a variety of anomalies through anomaly injections. We conduct an experiment to validate this. We do not use any labeled data and only consider the anomaly injection method to pre-train a model (Without Label), and the zero-shot results under F1 are as shown in Table 9. It verifies that training our model without human-labeled supervised labels does not significantly impact the performance.

**Table 9: Detection results without human-labeled data.**
| Dataset | SMD | MSL | SMAP | SWAT | PSM |
| --- | --- | --- | --- | --- | --- |
| Without Label | 84.69 | 78.05 | 75.35 | 74.07 | 82.39 |
| DADA | 84.57 | 78.48 | 75.42 | 74.60 | 82.26 |

### E.3 Comparison with MoE

Mixture-of-Experts (MoE) (Jacobs et al., ) includes the gating network and multiple experts, each with the same structure. The gating network assigns different experts according to the characteristics of the data. In our AdaBN, each bottleneck in the bottleneck pool has a different hidden dimension. Each bottleneck compresses time series representation to a different hidden space to accommodate the varying requirements of the multi-domain time series data. As shown in table 10, to compare DADA with MoE, we replace all bottlenecks in the bottleneck pool with a consistent size of 256, which performs better compared to other hidden dimensions as shown in Figure 5. The results indicate that the improvement in our model’s performance is not just due to a larger number of parameters and more experts, but more crucially, to the special design of the bottleneck, adapted to the unique demands of general anomaly detection modeling.

**Table 10: Comparison of DADA and MoE. Single 256 replaces the AdaBN with a single bottleneck with size 256. The MoE 256 changes all bottlenecks in the bottleneck pool to the same size 256. The result is reported with Affiliated F1, and the best ones are in bold.**
| Dataset | SMD | SMAP | MSL | SWaT | PSM |
| --- | --- | --- | --- | --- | --- |
| Metric | F1 | F1 | F1 | F1 | F1 |
| single 256 | 81.69 | 74.37 | 77.01 | 69.93 | 80.15 |
| MoE 256 | 84.25 | 74.87 | 78.41 | 71.68 | 81.69 |
| DADA | 84.57 | 75.42 | 78.48 | 74.60 | 82.26 |

### E.4 Time overhead for multiple AE structures

Our AdaBN introduces multiple AutoEncoder structures, but the additional time overhead they incur is acceptable. We adopt a sparse design where data goes through only top-k bottlenecks rather than all. Each bottleneck in our AdaBN is designed to be lightweight, implemented using a simple MLP. Thus, the cost of this part is tiny compared with the overall model, and the overall increase after using AdaBN is much less than k times. Taking the MSL dataset as an example, when we increase the number of bottlenecks from one to six, the time required to complete the testing of the entire dataset increases by 6% (from 4.96 seconds to 5.29 seconds). As shown in Figure 5(left), the F1 score of the model with adaptive bottlenecks increases from 74.65% to 78.48%. Compared with the improvement in model performance, this minor overhead is acceptable.

### E.5 Visualization of adaptive bottlenecks

As shown in Figure 11(left), our visualization illustrates the proportion of different adaptive bottleneck sizes that each dataset selects and prefers. The figure highlights distinct preferences among datasets, indicating variability in their selection criteria. Notably, the SMAP and MSL datasets, sourced from NASA spacecraft monitoring systems, display a similarity in their data characteristics. This observation is further supported by our experimental results in Figure 11(left), showcasing consistent preferences for bottleneck sizes within these datasets.

Figure: Figure 11: (left) Visualization of the proportion of different bottleneck sizes that each dataset prefers. (right) Pre-training with multi-domain datasets.
Refer to caption: x11.png

### E.6 Pre-training with multi-domain datasets

We investigated the impact of the pre-training datasets on our model’s performance. As depicted in Figure 11(right), we employ the $\text{Monash}^{+}$ in Table 6 and Anomaly Detection Data in Table 5 individually for pre-training, followed by zero-shot evaluation on the SMD, MSL, SMAP, SWaT, and PSM datasets. Our model consistently exhibits great performance with different pre-training datasets.

Furthermore, using both datasets together to expand the domain scope of pre-training datasets enhances the breadth of information assimilated by our adaptive bottlenecks and dual adversarial decoders. Consequently, this enrichment contributes to further improvements in model performance.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: I hope the author can explain the difference between the routing function proposed in "PATHFORMER: MULTI-SCALE TRANSFORMERS WITH ADAPTIVE PATHWAYS FOR TIME SERIES FORECASTING". At the same time, give a detailed explanation of the reconstruction process Recon(·).
soundness: 3
strengths: The overall writing of this article is good and the logic is clear。
This paper first proposes an adaptive bottleneck module to compress features into the latent space, reflecting the different information densities of multi-domain data. It solves the challenge of different requirements for appropriate information bottlenecks for different datasets.
This paper uses dual adversarial decoders to expand the robustness distinction between normal and abnormal modes, improving the general anomaly detection ability in different scenarios.
summary: This paper proposes a new general time series anomaly detector with adaptive bottleneck and dual adversarial decoder. By pre-training on multi-domain time series data, anomaly detection is performed efficiently in different target scenarios without domain-specific training.
The overall writing of this article is good and the logic is clear。
This paper first proposes an adaptive bottleneck module to compress features into the latent space, reflecting the different information densities of multi-domain data. It solves the challenge of different requirements for appropriate information bottlenecks for different datasets.
This paper uses dual adversarial decoders to expand the robustness distinction between normal and abnormal modes, improving the general anomaly detection ability in different scenarios.
weaknesses: The adaptive router formula of the adaptive bottleneck module proposed in this paper lacks certain innovation, and its formula is similar to the routing function proposed in “PATHFORMER: MULTI-SCALE TRANSFORMERS WITH ADAPTIVE PATHWAYS FOR TIME SERIES FORECASTING”.
Some formulas in this paper lack clear explanations, which may cause some confusion when reading. For example, the reconstruction process proposed in the article is denoted as Recon(·), but there is a lack of detailed description of this process.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In the ablation experiment, the result of the fourth row is the second highest. In my opinion, AdaBN should be basically equivalent to moe, except for the different dimensions. However, at this time, the model effect is basically close to the final model, which means that the main source of performance is not the specially designed optimization method. This makes me doubt the component innovation of the paper. At the same time, Table 10 in the appendix also supports this conclusion. The result of moe 256 is completely better than the result of the fourth row. This seriously indicates that AdaBN is likely to have a much smaller actual effect than Moe 256. Further, the innovation of AdaBN should be greatly discounted.
2. The experimental analysis lacks the analysis of COMPLEMENTARY MASK MODELING. The author takes this item as the key point of the method but lacks further research.
3. The visualization analysis does not state what dataset is used, nor is there visualization of the original data. Furthermore, there is a lack of visualization comparison with other algorithms, lacking the credibility of the experiment.
4. I hope the author can open source the generation method and generation script of Monash+ and provide a directly usable data set. At present, the explanations in the paper are completely insufficient.
soundness: 2
strengths: The writing is clear and easy to understand.
The pictures and tables in the paper are beautiful and clear.
The proposed method, DADA, can achieve zero-shot.
summary: Authors propose a General time series anomaly Detector with Adaptive Bottlenecks and Dual Adversarial Decoders (DADA), which enables flexible selection of bottlenecks based on different data and explicitly enhances clear differentiation between normal and abnormal series. The zero-shot capability is indeed impressive.
weaknesses: 1. AdaBN has almost no difference from the previous moe, yet the author exaggerates their design. I think there is a question of contribution. Moreover, the author does not explain the difference from Moe in the main text nor add relevant references [a]. I think this is somewhat deceptive. 
2. The role of AdaBN is not as great as the author claims. See the analysis in question 1. There is excessive exaggeration of contribution.
[a] Shazeer, Noam, et al. "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer." arXiv preprint arXiv:1701.06538 (2017).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Does "anomaly injection" refer to incorporating multiple types of anomaly patterns into each dataset? More details about the implementation of anomaly injection should be included in the paper since it is crucial for conducting the supervised setting.
2. How do you adapt various baseline methods pre-trained on multi-domain datasets for zero-shot evaluation if they are typically utilized in the standard form?
3. How can DADA be applied during the inference step to further fine-tune the target datasets for downstream tasks in the experiments?
soundness: 3
strengths: 1. The general time series anomaly detection plays a key role in the real application.
2. DADA employs the adaptive bottlenecks for multi-domain time series data by dynamically selecting suitable bottleneck sizes. 
3. The effectiveness of both AdaBN and Dual Decoders has been demonstrated in the experiments.
summary: This paper develops a novel general time series anomaly detection by pre-training on multi-domain time series.  The proposed adaptive bottleneck enables the general model to be applied to various target scenarios without requiring domain-specific training. To improve the general anomaly detection capability, the dual adversarial decoder is further employed to amplify the decision boundary.
weaknesses: 1. If anomaly injection is used to establish a supervised setting, this implies that all original samples(some anomalies are included) are treated as normal samples. I suggest the author add an experiment to examine performance under varying levels of anomaly contamination in the training samples.
2. During model training, two optimization losses are utilized, along with the optimization approach and any challenges encountered in the optimization process. The visualizations of loss curves for the two losses are essential for further demonstration.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The paper does not compare the inference time of DADA with the baselines. Compared to the baselines, does DADA have a shorter inference time?  In particular, when compared to other pre-trained models, does DADA have an advantage?

2. In the DADA model, the original data first undergoes complementary masking operations, and then features are extracted by the encoder. In subsection 3.2, it states that the masked portions of the original data input to the encoder have a value of 0, but there may also be 0 values in the non-masked portions of the original data. How does the encoder differentiate between the masked and non-masked portions?
soundness: 3
strengths: The paper addresses the issue that each dataset in the field of time series anomaly detection requires a specifically trained model, which holds practical significance. The paper proposes a novel method DADA for zero-shot inference on the target domain after pre-training on multiple datasets from different domains. DADA proposes the Adaptive Bottlenecks to addresses the flexible reconstruction requirements of multi-domain data, along with dual adversarial decoders to enhance the clear differentiation between normal and abnormal patterns, showcasing a degree of originality. The paper has a clear structure and expression. In the experimental section of the paper, DADA demonstrates superiority over multiple state-of-the-art methods across nine datasets and provides extensive analysis to prove the necessity of each module, showcasing high quality.
summary: The paper proposes DADA method for general time series anomaly detection, which is pre-trained on extensive multi-domain datasets and perform anomaly detection on various target scenarios without domain-specific training. DADA proposes Adaptive Bottlenecks to addresses the flexible reconstruction requirements of multi-domain data. DADA also employs Dual Adversarial Decoders amplify the decision boundaries between normal time series and common anomalies in an adversarial training way, which improves the general anomaly detection capability across different scenarios. The paper conducts extensive experiments on nine target datasets from different domains, and demonstrate that DADA achieves superior results compared with state-of-the-art models trained specifically for each dataset.
weaknesses: 1. The paper lacks an analysis of the complementary mask rate's effect on anomaly detection performance. For example, the author could add an experiment to demonstrate the changes in anomaly detection F1-score under various complementary mask rates and explain why this mask performs the best.

2. In the experimental section, Figure 4 shows that after fine-tuning, the model's performance improved from 78.48% to 79.90% on MSL and from 84.57% to 85.50% on SMD, an increase of about 1%. The data shows that the improvement of the model after fine-tuning is small.

3. In the ablation study, Table 3 shows that the model performs better when removing the AdaBN, Adversarial, and Dual Decoders modules compared to removing only the Adversarial module. The paper explains that directly maximizing the anomalous time series will increases confusion. Further explanation is needed to clarify why maximizing the anomalous time series leads to increased confusion.

4. There are labeling mistakes in the experimental results. For example, in Table 8, the R_AUC_PR value of DADA on the PSM dataset is 47.40, which is less than ModernTCN's 47.48, but 47.40 is bolded. In Table 2, the F1 metric column for the CICIDS dataset is missing a underline marking.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - Can you list the data difference between the pretrain dataset and the evaluation dataset? How you select the pretrain dataset?

- Are there any data preprocessing on the input data?

- What's the computation cost(overhead) of importing Adaptive Bottlenecks and Dual Adversarial Decoders? Please specify.
soundness: 3
strengths: + The task on zero-shot time series anomaly detection is quite challenging, the author build a unified end-to-end solution and demonstrate great results. 

+ The paper is well-organized, and it introduces the background and motivation of the proposed methods clearly. The figures also explain their method pretty well.

+ The paper demonstrate its experiment results on enough datasets.  The ablation studies on Sec 4.3 also makes the paper convincible.
summary: The author propose a solution for general time series anomaly detection, which is call time series anomaly Detector with Adaptive bottlenecks and Dual Adversarial decoders (DADA). This model is  pre-trained on multi-domain time series data. Two components called Adaptive Bottlenecks and Dual Adversarial Decoders is leveraged to enhance the model performance. The model achieves competitive performance on various datasets even compared with SOTA models trained specifically for each dataset.
weaknesses: - The difference between the pretrain dataset and the evaluation dataset is not listed. How you select those dataset?

- It will be great to specify some more experiment details. Are there any data preprocessing on the input data?

- What's the computation cost(overhead) of importing Adaptive Bottlenecks and Dual Adversarial Decoders? Please specify.

Meta-review: The work aims to learn a generalist model for zero-shot time series anomaly detection. The model is built with Adaptive Bottlenecks and Dual Adversarial Decoders. It is trained on multi-domain datasets and then tested directly on different datasets. 

**Strengths**
- Paper is well written [WTbT, hvUu, e5Ze, EBSp]
- The studied setting is practically sound [e12v, e5Ze]
- The model combining adaptive information bottleneck and dual adversarial decoders is novel [WTbT, e5Ze, EBSp]
- The model shows effective zero-shot TSAD performance across various downstream datasets [WTbT, hvUu, e12v, e5Ze, EBSp]


**Weaknesses**
- The adaptive bottleneck module itself lacks novelty [WTbT, hvUu]
- Lack of analysis on part of equations, modules or the empirical results [WTbT, hvUu, e5Ze]
- Lack of investigation into the performance under varying level of anomaly contamination [e12v]
- Some key implementation details of the proposed method, the evaluation protocol, the competing methods are missing [e12v, EBSp]
- The model has limited improvement if it is fine-tuned on the target dataset [e5Ze]
- Lack of computational time analysis [e5Ze, EBSp]

Decision: Accept (Poster)