## 3 Data Pre-Processing
### Variational Auto-Encoder Based on Gaussian Mixture Model
#### 3.2.1 Variational Auto-Encoder  
Variational Auto-Encoder (VAE) [32; 33] is a deep generation model, which aims to build a distribution model \(p_{model}\) to approximate unknown data distribution \(p_{data}\). The data set \(X=\{x_{i}\}_{i=1}^{N}\) contains \(N\) independent and identically distributed discrete variables. Assuming that the variable \(x\) is generated by the latent variable \(z\), the generating process is as follows: (1) The variable \(z\) is generated, which is denoted as \(p_{\theta}\left(z\right)\); (2) The variable \(x\) is generated, which is denoted as \(p_{\theta}\left(x\mid z\right)\). In addition, \(q_{\phi}\left(z\mid x\right)\) is used to approximate a posteriori probability \(p_{\theta}\left(z\mid x\right)\) which is difficult to calculate. The whole process is shown in Figure 11.  
The edge likelihood of \(X\) is obtained by summing the edge likelihood of each independent data. Namely:  
\[logp_{\theta}\left(x^{\left(1\right)},\cdots,x^{\left(N\right)}\right)=\sum_{ i=1}^{N}logp_{\theta}\left(x^{\left(i\right)}\right) \tag{5}\]  
Since \(q_{\phi}\left(z\mid x\right)\) approximates \(p_{\theta}\left(z\mid x\right)\), then:  
\[logp_{\theta}\left(x^{\left(i\right)}\right)=D_{KL}\left(q_{\phi}\left(z\mid x ^{\left(i\right)}\right)\parallel p_{\theta}\left(z\mid x^{\left(i\right)} \right)\right)+L\left(\theta,\phi;x^{\left(i\right)}\right) \tag{6}\]  
Figure 10: The comparison of true and estimated measurement error data.  
Figure 9: The estimated measurement error data according to method 2.  
The first term on the right side of the equation is the Kullback-Leibler (KL) divergence of approximate from the true posterior. Since the KL divergence is non-negative, the second item on the right side of the equation is called the (variational) lower bound on the marginal likelihood of data-point \(i\), and can be written as:  
\[L\left(\theta,\phi;x^{(i)}\right)=-D_{KL}\left(q_{\phi}\left(z\mid x^{(i)} \right)\parallel p_{\theta}\left(z\right)\right)+E_{q_{\phi}\left(z\mid x^{(i) }\right)}\left[logp_{\theta}\left(x^{(i)}\mid z\right)\right] \tag{7}\]  
Let the latent variable \(z\) be the normal Gaussian distribution and \(p_{\theta}\left(x\mid z\right)\) be the Gaussian distribution. So formula (3) can be written as follows:  
\[L\left(\theta,\phi;x^{(i)}\right)\cong\frac{1}{2}\sum_{j=1}^{J}\left(1+log \left(\left(\sigma_{j}^{(i)}\right)^{2}\right)-\left(\mu_{j}^{(i)}\right)^{2} -\left(\sigma_{j}^{(i)}\right)^{2}\right)+\frac{1}{L}\sum_{l=1}^{L}logp_{ \theta}\left(x^{(i)}\mid z^{(i,l)}\right) \tag{8}\]  
The reparameterization is \(z^{(i,l)}=\mu^{(i)}+\sigma^{(i)}\odot\epsilon^{(l)}\). Where \(\epsilon^{(l)}\) is an auxiliary noise variable, and \(\epsilon^{(l)}\sim N(0,I))\). The objective of optimization is to make \(logp_{\theta}\left(x^{(i)}\right)\) as large as possible. We use a neural network for the probabilistic encoder \(q_{\phi}\left(z\mid x\right)\) and where the parameters \(\phi\) and \(\theta\) are optimized jointly with the auto-encoding variational Bayes algorithm. The variational approximation distribution can be regarded as an encoder, which maps observable variables to latent variables. The generated model can be regarded as a decoder, which maps latent variables into observable variables. The structure of VAE is shown in Figure 12.  
The traditional VAE algorithm assumes that the posterior distribution \(p_{\theta}\left(z\mid x\right)\) of the latent variable \(z\) satisfies a single Gaussian distribution, which is easy to cause the low-dimensional representation is too simple and cannot fit the space of the latent variable well, leading to the low  
Figure 11: The generation process of variable x.  
Figure 12: The structure of VAE.  
accuracy of the generated model. Based on this, the Gaussian mixture model is used to fit the latent variable space.