# Image Anomaly Detection Using Normal Data Only by Latent Space Resampling
## 3 Method
### Structuring Latent Space  
VQ-VAE is originally proposed as a compression model with good results, and we use it as a reconstruction model to construct the latent space. It is different from VAE; VAE assumes that the latent space satisfies a Gaussian prior and reconstructs using the re-parameterization trick, which implies that the process of getting the latent representation is indeterministic. However, VQ-VAE encodes the input to the latent representation with a deterministic mapping and reconstructs data from quantized vectors. VQ-VAE provides a sufficiently large latent space dimension, resulting in far better reconstruction performance than VAE and AE. Since the latent space is discrete, it can be modeled by deep autoregressive models, and the estimated probability distribution is the basis for the proposed resampling operation. VQ-VAE consists of an encoder, a codebook with \(K\) embedding vectors \(\mathbf{e}\in\mathbb{R}^{K\times D}\), and a decoder. The method trains discrete latent variables \(\mathbf{z}\in\mathbb{R}^{M\times N}\) using the codebook combined with nearest neighbor search, which is performed on the encoder output \(\mathbf{z_{e}}\in\mathbb{R}^{M\times N\times D}\) using the \(\ell^{2}\)-distance. Then, the vectors in the codebook replace \(\mathbf{z_{e}}\) to yield \(\mathbf{z_{q}}\in\mathbb{R}^{M\times N\times D}\) as input to the decoder, where \(\mathbf{z}\) is used as the index table for \(\mathbf{z_{e}}\) transformation to \(\mathbf{z_{q}}\). The model optimizes the parameters of the encoder, decoder, and codebook, with the goal of minimum reconstruction errors. Specifically, as shown in step 1 in Figure 1, given an anomaly-free input image \(x\), the encoder first encodes \(x\) into \(M\times N\) D-dimensional vectors \(z_{e}\) that maintain the two-dimensional spatial structure of the  
Figure 1: Pipelines of the proposed framework. The top is the training stage, and the bottom is the prediction stage. The framework for VQ-VAE comes from [PERSON] et al. (2013). image. These vectors are then quantized based on their nearest distance to the embedding vectors \(e_{1,2,\ldots,k}\) in the codebook; the process can be defined by Equation (1):  
\[z_{i}=\operatorname*{arg\,min}_{j\in\{1,\ldots,K\}}\left\|(z_{e})_{i}-e_{j} \right\|_{2} \tag{1}\]  
This quantization process is essentially a lookup process, resulting in an index table \(z\). The index table maintains the same spatial structure as \(z_{e}\), whose value for each component is the sequence number of embedding vectors in the codebook. Each vector in \(z_{e}\) will be replaced by the \(e\) nearest to it before decoding, where \(e\) is selected directly in the codebook based on the value of the corresponding position in \(z\), as shown in Equation (2):  
\[(z_{q})_{i}=e_{z_{i}} \tag{2}\]  
Finally, the decoder maps this replaced vectors \(z_{q}\) back to the image \(x_{d}\). With the quantization operation, the discrete \(\mathbf{z}\) can represent the whole latent space, and thus changing the value of \(\mathbf{z}\) at prediction stage can generate the desired reconstructed images. To learn these mappings, we randomly initialize \(e_{1,2,\ldots,k}\) by sampling in the Gaussian distribution \(N(0,1)\) before training and through the loss function described in Equation (3) to train the model, where \(sg[\cdot]\) operation represents equality in forward propagation and zero derivative in backward propagation:  
\[\mathcal{L}=\|\mathbf{x}-\mathbf{x_{d}}\|_{2}+\|sg[\mathbf{z_{e}}]-\mathbf{z_ {q}}\|_{2}+\beta\|\mathbf{z_{e}}-sg[\mathbf{z_{q}}]\|_{2} \tag{3}\]  
The entire loss function has three components. The first term is reconstruction loss, which encourages the output to be as close to the input as possible. The second term is used to constrain the embedding vectors in the codebook, minimizing the loss of information caused by replacing \(\mathbf{z_{e}}\) with \(\mathbf{z_{q}}\).