# A Study on the Channel Expansion VAE for Content-Based Image Retrieval
## 3 Method
### Variational Auto-Encoder  
The auto-encoder was composed of an encoder extracting features from the query image and a decoder reconstructing the image. All images can be represented by a compressed vector in latent space. The VAE had the same structure, but it used variational inference for data distribution. Its structure is shown in Figure 2.  
The VAE included a layer to approximate the output of an encoder inducing normal distribution. The objective can be formulized as follows:  
\[p(x)=\int p_{\theta}(z|x)p_{\varphi}(x|z)dz \tag{1}\]  
\(p_{\theta}(z|x)\) and \(p_{\varphi}(x|z)\) correspond to the encoder and decoder, respectively. Data distribution can be estimated by optimization of \(\theta\) and \(\varphi\), but this is convoluted. Thus, the model adopted variational inference to approximate general distribution. Consequently, a loss occurred between original data distribution \(p_{\theta}(z|x)\) and approximation distribution \(q_{\theta}(z|x)\). The loss can be described as following Kullback-Leibler divergence.  
\[D_{\text{KL}}(q_{\theta}(z|x)\parallel p_{\theta}(z|x))=D_{\text{KL}}(q_{ \theta}(z|x)\parallel p(z))+\log p(x)+E_{z\sim q(z)}\big{[}\log p_{\varphi}(x |z)\big{]}\geq 0 \tag{2}\]  
Figure 2: Variational auto-encoder (VAE) structure.  
This \(D\) is bigger than zero, and this formulation can be deduced to evidence low boundary (ELBO).  
\[\log p(x)\geq-D_{KL}(q_{\theta}(z|x)\parallel p(z))+E_{z\sim q(z)}\big{[}\log p_{ \varphi}(x|z)\big{]} \tag{3}\]  
Equation (3) means that it is able to maximize the objective by the right-hand side. Just as in the ELBO equation, the first term is latent distribution regularization loss, and the second term is reconstruction loss. Consequently, the final objective loss is written as follows:  
\[Loss(\theta,\varphi)=D_{KL}(q_{\theta}(z|x)\parallel p(z))+E_{z\sim q_{ \theta}(z)}\big{[}\log p_{\varphi}(x|z)\big{]} \tag{4}\]  
Usually, the approximation distribution for VAE is Gaussian distribution, and the VAE model needs the reparameterization trick [16] for a learning network, because the reconstruction loss term, \(E_{z\sim q(z)}\big{[}\log p_{\varphi}(x|z)\big{]}\), is derived from sampling on \(q\). Nonetheless, it is not able to differentiate between \(\mu\) and \(\sigma\). So, sampling for normal distribution was separated from the encoder, and it made back-propagation possible. Latent distribution \(z\) can be shown as follows:  
\[z=\mu+\sigma^{2}\odot\epsilon,\;\epsilon\sim N(0,I) \tag{5}\]  
The VAE learns data distribution and embeds image features into latent space. This continuity of multi-dimensional latent space defines data similarity and generates images. That is, the VAE can obtain compressed multi-dimensional vector representation from hundreds of thousands image pixel data, which enables faster retrieval from enormous databases. In particular, variances of certain dimensions are prominent in the distribution of latent space, which implies that the latent vector can be compressed into a less dimensional vector with a small performance loss. Therefore, VAE has a benefit in terms of compression.