# Anomaly Detection through Grouping of SMD Machine Sounds Using Hierarchical Clustering
## 3 Proposed Method
### Pre-Training the Autoencoder Model  
Before explaining the pre-training of the autoencoder model, we introduce the autoencoder. The autoencoder [3] is a neural network that uses unsupervised learning algorithms to train so that the output is equal to the input. The autoencoder consists of an encoder that encodes a high-dimensional input \(\mathbf{x}=[x_{0},x_{1},\ldots,x_{n-1}]\) into a low-dimensional hidden representation \(\mathbf{h}=[h_{0},h_{1},\ldots,h_{d-1}]\) and a decoder that reconstructs the low-dimensional hidden representation \(\mathbf{h}\) to a high-dimensional \(\mathbf{\hat{x}}=[\hat{x}_{0},\hat{x}_{1},\ldots,\hat{x}_{n-1}]\). The equations for the encoder and decoder networks are as follows:  
\[\mathbf{h}=f(\mathbf{W}_{\mathbf{h}}\mathbf{x}+\mathbf{b}_{\mathbf{h}}), \tag{15}\]  
\[\mathbf{\hat{x}}=g(\mathbf{W}_{\mathbf{g}}\mathbf{h}+\mathbf{b}_{\mathbf{g}}), \tag{16}\]  
where \(\mathbf{x},\mathbf{\hat{x}}\in R^{n}\), \(n\) is the dimension of the input \(\mathbf{x}\) and output \(\mathbf{\hat{x}}\), and \(\mathbf{h}\in\mathbf{R}^{d}\), \(\mathbf{W}_{\mathbf{h}}\in\mathbf{R}^{d\times n}\), \(\mathbf{W}_{\mathbf{g}}\in\mathbf{R}^{n\times d}\), \(d\) is the dimension of the hidden representation \(\mathbf{h}\); \(\mathbf{W}\) and \(\mathbf{b}\) denote the weights and bias vectors, and \(f\), \(g\) are non-linear activation functions [22]. The autoencoder is trained by reducing the reconstruction error by Equations (17) and (18) between the input \(\mathbf{x}\) and the output \(\mathbf{\hat{x}}\). If the input is continuous, we calculate the reconstruction error with the mean squared error in the following equation:  
\[\mathcal{L}(\mathbf{x},\mathbf{\hat{x}})=\|\mathbf{x}-\mathbf{\hat{x}}\|^{2}, \tag{17}\]  
If the input is categorical, we calculate the cross-entropy loss using the following equation:  
\[\mathcal{L}(\mathbf{x},\mathbf{\hat{x}})=-\frac{1}{n}\sum_{i=1}^{n}(\mathbf{x}_{i}\log\bm {\hat{x}}_{i}+(1-\mathbf{x}_{i})\log(1-\mathbf{\hat{x}}_{i})), \tag{18}\]  
where \(n\) is the number of categories of categorical data. In the process of compressing and reconstructing the input data, the encoder of the autoencoder is trained to extract important  
Figure 1: The flowchart of the proposed method.  
features from the input data so that the decoder can accurately reconstruct the hidden representation.  
We pre-train the model SSS-AE [9] to utilize the characteristics of the autoencoder. The SSS-AE is one of the autoencoder-based SMD anomaly detection models and has the best performance among existing SMD anomaly detection models, meaning that the model represents the input well in the latent space. The SSS-AE is a Seq2 Seq-based autoencoder model, and RNN layers of the SSS-AE are LSTM. The encoder and decoder of the SSS-AE consist of \(N\) blocks, consisting of LSTM, multi-head self-attention of the transformer [23], and layer normalization [24] being sequentially connected. The residual connection [25] is used to add the output of each sub-layer of the block before layer normalization. The last block of the decoder consists of one LSTM layer for reconstructing the output of the previous block as input.  
We pre-train SSS-AE using normal data from products that contain abnormal data according to Algorithm 1. This is because abnormal data will be a threshold for finding untrained data with characteristics similar to those trained. Then, we obtain latent vectors for abnormal data of the trained product and normal data of untrained products using the pre-trained SSS-AE model.  
```
0: Normal dataset \([x^{0},x^{1},\ldots,x^{n-1}]\) of the product,  where \(n\) is the number of product data samples Output: Encoder of the autoencoder repeat  calculate \(L(X,\hat{X})\) by Equations (9), (10), and (17),  where \(X\) is a set of data samples \(x\) and \(\hat{X}\) denotes the output sequence data.  update parameter using gradients of \(L\). until epochs given in the experiment.
```  
**Algorithm 1** Pre-trained autoencoder model with normal data