# Unsupervised Deep Anomaly Detection for Industrial Multivariate Time Series Data
## 3 Materials and Methods
### Prediction Network
The computation process for the hidden state \(h_{t}\) is as follows:  
\[f_{t}=\sigma\left(W_{f}\cdot[h_{t-1},e_{t}]+b_{f}\right), \tag{10}\]  
\[i_{t}=\sigma(W_{i}\cdot[h_{t-1},e_{t}]+b_{i}), \tag{11}\]\[o_{t}=\sigma(W_{o}\cdot[h_{t-1},e_{t}]+b_{o}), \tag{12}\]  
\[\widetilde{C}_{t}=tanh\cdot(W_{c}\cdot[h_{t-1},e_{t}]+b_{c}), \tag{13}\]  
\[C_{t}=f_{t}\times C_{t-1}+i_{t}\times\widetilde{C}_{t}, \tag{14}\]  
\[h_{t}=o_{t}\times tanh(C_{t}), \tag{15}\]  
Here, \([h_{t-1},e_{t}]\) represents the horizontal concatenation of the hidden state \(h_{t-1}\); the input \(e_{t}\sigma(\cdot)\) and \(tanh(\cdot)\), respectively, represent the sigmoid and tanh activation functions; \(W_{g}\) denotes weights; \(g\) stands for the gate neuron; \(b_{g}\) is the bias of the gate; and \(h\) is the output of the previous LSTM cell. The LSTM forget gate, memory gate, and output gate control the information preservation and transmission in the LSTM, ultimately reflecting in the cell state \(C_{t}\) and output signal \(h_{t}\). The key to LSTM is the cell state, which is like a conveyor belt running through the entire chain. The LSTM cell state allows the unit state to forget and replace values; then, it decides which values to output and sends them to the next unit. It controls the information passed to the next time step during the computation of the hidden state. It can consider global and local time information when calculating the hidden state. To better handle time series data, we also introduce an attention mechanism, allowing the model to focus more on important information. Self-attention mechanisms [33] are widely used in computer vision [34], natural language processing [35], and other tasks due to their ability to capture long-term dependencies. However, self-attention mechanisms have a significant drawback: they require a considerable amount of computation, leading to some computational redundancy. Moreover, self-attention mechanisms only utilize information within their samples, neglecting potential connections between different samples. To extract long-term dependencies in time series data and the potential connections between samples while reducing computational costs, we employ an external attention mechanism [36] using a small, learnable, and shared memory. External attention only uses two linear layers and normalization layers, possessing linear complexity and implicitly considering relationships between different feature maps. External attention is achieved by introducing two external memory units, implicitly learning features of the entire dataset. It computes attention between the input and external memory units. The memory unit is represented as \(M\in\mathbb{R}^{F\times d}\), where \(F\) and \(d\) are hyperparameters. It can be expressed as  
\[A=(\alpha)_{i,j}=Norm\,(OM^{T}), \tag{16}\]  
\[P_{out}=AM, \tag{17}\]  
where \(O\) represents the output of the LSTM at each time step in the input sequence and \((\alpha)_{i,j}\) represents the similarity between the \(i\)-th element and the \(j\)-th row of \(M\). The memory unit is a parameter independent of the input, serving as the memory for the entire training dataset. In the output layer, the prediction error \(P\) is obtained by calculating the difference between the output \(P_{out}\) of the prediction network and the true value \(R\). The loss function of the prediction network can be expressed as  
\[E=Upsample(D^{\prime}). \tag{9}\]0