# Unsupervised Deep Anomaly Detection for Industrial Multivariate Time Series Data
## 3 Materials and Methods
### Characterization Network  
To better learn the spatial features of multivariate time-series data, we adopt a convolutional autoencoder structure suitable for handling anomaly detection in multivariate time-series data [30]. This structure integrates CNN into the AutoEncode framework, where the convolutional kernel slides in the time dimension, extracting features at different time steps. By stacking multiple convolutional layers, higher-level time-series features are gradually extracted, learning the features and patterns of multivariate time-series data. The AutoEncode structure, through backpropagation and optimization methods like gradient descent, uses the input data itself as supervision to guide the neural network in attempting to learn a mapping relationship, resulting in a reconstructed output, achieving unsupervised training. The characterization network consists of three main parts: encoder, low-dimensional feature extraction, and decoder. The structure of the characterization network is illustrated in Figure 2.  
The encoding process from the input layer to the hidden layer for the raw data \(S\):  
\[D=Encoder(S). \tag{2}\]  
Figure 1: The architecture of the AT-DCAEP model. Figure 2: Characterization network structure diagram. The process of extracting low-dimensional features:  
\[D^{\prime}=MultiHeadAtt(D,D,D). \tag{3}\]  
The decoding process from the hidden layer to the output layer:  
\[S^{\prime}=Decoder\big{(}D^{\prime}\big{)}. \tag{4}\]  
The role of the encoder is to encode the high-dimensional input \(S\) into a low-dimensional latent variable \(D\), enabling the neural network to learn the most informative features. The encoder achieves this by mapping the input \(S\) to the hidden representation \(D\) through two one-dimensional convolutional layers and one dropout layer. The addition of the dropout layer aims to enhance the network's robustness, contributing to improved performance on noisy or highly variable data. The stride of each convolutional layer is set to 2, reducing the dimension of features and generating smaller-sized output feature maps based on the size of the convolutional kernel. The intermediate hidden layer features encoded by the encoder, which represent the low-dimensional spatial feature information \(D\), play a crucial role in identifying anomalies. Methods like CAE-M and DAGMM emphasize the significance of these features. To focus on the crucial information in the low-dimensional spatial features \(D\), we introduce multi-head attention [30] into the low-dimensional space. Multi-head attention is capable of learning relationships between different low-dimensional features, combining various features to enhance the representation of the low-dimensional space, and capturing diverse dependencies within the low-dimensional space. The calculation for each attention head \(d_{i}(i=1,\cdots,h)\) in multi-head attention, given query \(q\in\mathbb{R}^{d_{q}}\), key \(k\in\mathbb{R}^{d_{k}}\), and value \(v\in\mathbb{R}^{d_{v}}\), is as follows:  
\[d_{i}=f\Big{(}W_{i}^{(q)}q,W_{i}^{(k)}k,W_{i}^{(v)}v\Big{)}\in\mathbb{R}^{p_{v }}, \tag{5}\]  
\(W_{i}^{(q)}\in\mathbb{R}^{p_{q}\times d_{q}}\), \(W_{i}^{(k)}k\in\mathbb{R}^{p_{k}\times d_{k}}\), and \(W_{i}^{(v)}\in\mathbb{R}^{p_{v}\times d_{v}}\) are learnable parameters. \(f\) represents the attention pooling function. The output of multi-head attention undergoes a linear transformation by concatenating multiple heads together and can be expressed as  
\[W_{o}\cdot d_{i}\in\mathbb{R}^{p_{v}}, \tag{6}\]  
where \(W_{o}\in\mathbb{R}^{p_{v}\times hp_{pv}}\) is a learnable parameter. Based on this design, each head can focus on different parts of the low-dimensional features. Through this multi-head attention mechanism, the model can capture the intrinsic relationships within the low-dimensional spatial features. We have demonstrated the effectiveness of this approach through ablation experiments.