## 3 Void Detection Based on Deep SVDD
### Autoencoder Based on ELMo  
An autoencoder [36] is a deep learning approach that copies an input to an output, and consists of an encoder and a decoder. The encoder uses the input to train a latent vector, which is an encoded representation. Using a latent vector that the encoder represents, the decoder is trained to generate an output that is equivalent to the input. As compressed information about the input, the latent vector is the core of the autoencoder and must be designed in a way that can properly represent it. ELMo (embeddings from language models) [30], a word-embedding method based on pretrained language models, was introduced in 2018. ELMo is a contextualized word  
Figure 7: Basic process of Deep SVDD. embedding that reflects the fact that even the same vocabulary can have different meanings in different contexts. For this, ELMo used the LSTM, which achieved good performance in time series data. The LSTM comprises LSTM cells that are trained in a time order and have three gates (input gate, forget gate, output gate) equal to the length of the time series data. Information from the previous time step can be transferred to the current time step as the cell state (\(C_{t}\)) in the LSTM cell calculated by the out puts of the input gate (\(i_{t}\)) and forget gate (\(f_{t}\)). Equations (6)-(11) describe how the LSTM cell is calculated. \[i_{t}=\sigma\big{(}W_{x,h_{t}}x_{t}+W_{h,h_{t}}h_{t-1}+b_{h_{t}}\big{)}, \tag{6}\]  
\[f_{t}=\sigma\big{(}W_{x,h_{f}}x_{t}+W_{h,h_{f}}h_{t-1}+b_{h_{f}}\big{)} \tag{7}\]  
\[o_{t}=\sigma(W_{x,h_{o}}x_{t}+W_{h,h_{o}}h_{t-1}+b_{h_{o}}) \tag{8}\]  
\[\widetilde{C}_{t}=tanh\big{(}W_{x,h_{C}}x_{t}+W_{h,h_{C}}h_{t-1}+b_{h_{C}} \big{)} \tag{9}\]  
\[C_{t}=f_{t}\ \odot\ C_{t-1}+i_{t}\ \odot\ \widetilde{C}_{t}, \tag{10}\]  
\[h_{t}=o_{t}\ \odot\ tanh(C_{t}), \tag{11}\]  
While \(\widetilde{C}_{t}\) uses the hyperbolic tangent function to generate new candidate values from the input, \(i_{t}\) uses the sigmoid function to determine which values in the input to update. The two outputs are used to produce a value that will update the cell state. \(f_{t}\) determines how much information from \(h_{t-1}\) is eliminated. To determine what information in \(C_{t}\) should transfer, \(o_{t}\) calculates the input as a sigmoid function. These outputs enable the LSTM cell to calculate \(h_{t}\), which represents the current information that will be transferred to the following time step. However, it does not include contextual information, so Bi-LSTMs that train forward and backward information are mainly used. Instead of a typical Bi-LSTM, ELMo uses a structure that trains word vectors represented in a pre-trained language model in the forward and backward directions, respectively. Figure 8 shows the difference between Bi-LSTM and ELMo. Signals arise in the sequence of time. When performing an anomaly detection task, signals are evaluated in the forward direction to identify when a defect occurs. Environmental factors such as the characteristics of concrete structures and the geometry of irregular defects along with the utilization of impact-generated stress waves due to impact make IE different from other domain signals. For this process, we developed an autoencoder that, like ELMo, trains the forward and backward directions separately. LSTM layers in the encoder train in the forward and backward directions, respectively. The output of each LSTM layer is transferred to the LSTM layer that trains in the same direction. The second LSTM layer's outputs are all concatenated and used as latent vectors. Figure 8: (**a**) Example of Bi-LSTM’s structure; (**b**) example of ELMo’s structure. By reversing the encoder, the decoder trains to provide an output that is equal to the input. In Deep SVDD, the feature space was represented using only the encoder structure of the autoencoder. The feature space represented by the encoder of the autoencoder is as shown in Equation (12).