## 2 State-Of-Art  
### Generative Models and Variational Auto-Encoders  
_Generative models_ aim to understand a given set of input examples \(\mathbf{x}\in\mathbb{R}^{d_{x}}\) by modeling the underlying probability distribution of the data \(p(\mathbf{x})\). To do so, we introduce _latent variables_ defined in a lower-dimensional space \(\mathbf{z}\in\mathbb{R}^{d_{z}}\) (\(d_{z}\ll d_{x}\)). These variables can be seen as a higher-level representation that could have led to generate a given example. The complete model is then defined by the joint distribution \(p(\mathbf{x},\mathbf{z})=p(\mathbf{x}|\mathbf{z})p(\mathbf{z})\). In order to obtain \(p(\mathbf{x})\), we would need to marginalize \(\mathbf{z}\) from the joint probability as follows  
\[p(\mathbf{x})=\int p(\mathbf{x}\mid\mathbf{z})p(\mathbf{z})d\mathbf{z}. \tag{1}\]  
Unfortunately, as real-world data follow complex distributions, this formulation usually cannot be solved analytically. The idea of _variational inference_ (VI) is to solve this problem through _optimization_ by assuming a simpler approximate distribution \(q_{\phi}(\mathbf{z}|\mathbf{x})\in\mathcal{Q}\) from a family of parametric densities [10], where \(\phi\) denotes the variational parameters that we can optimize. The goal of VI is to minimize the difference between this approximation and the real distribution, by minimizing the Kullback-Leibler (KL) divergence between these densities  
\[q_{\phi}^{*}(\mathbf{z}|\mathbf{x})=\text{argmin}_{q_{\phi}(\mathbf{z}| \mathbf{x})\in\mathcal{Q}}\mathcal{D}_{KL}\big{[}q_{\phi}\left(\mathbf{z}| \mathbf{x}\right)\parallel p\left(\mathbf{z}|\mathbf{x}\right)\big{]}. \tag{2}\]  
However, our original problem is that we did not have a closed-form solution to the posterior \(p(\mathbf{z}\mid\mathbf{x})\). By developing this KL divergence, re-arranging terms (the detailed development can be found in [5]) and introducing parametric distributions for the likelihood \(p_{\theta}(\mathbf{x}\mid\mathbf{z})\) and prior \(p_{\theta}(\mathbf{z})\), we obtain  
\[\log p(\mathbf{x})-\mathcal{D}_{KL}\big{[}q_{\phi}(\mathbf{z}|\mathbf{x}) \parallel p(\mathbf{z}|\mathbf{x})\big{]}=\mathbb{E}_{\mathbf{z}}\big{[}\log p (\mathbf{x}|\mathbf{z})\big{]}-\mathcal{D}_{KL}\big{[}q_{\phi}(\mathbf{z}| \mathbf{x})\parallel p(\mathbf{z})\big{]}. \tag{3}\]  
This formulation describes the quantity we want to model \(\log p(\mathbf{x})\) minus the error we make by using an approximate \(q\) instead of the true \(p\). Therefore, we can optimize this alternative objective, called the _evidence lower bound_ (ELBO), by optimizing the parameters \(\phi\) and \(\theta\) of the distributions  
\[\mathcal{L}_{\theta,\phi}=\mathbb{E}\big{[}\log p_{\theta}(\mathbf{x}| \mathbf{z})\big{]}-\beta\cdot\mathcal{D}_{KL}\big{[}q_{\phi}(\mathbf{z}| \mathbf{x})\parallel p_{\theta}(\mathbf{z})\big{]}. \tag{4}\]  
Intuitively, the ELBO minimizes the reconstruction error through the likelihood of the data given a latent \(\log p_{\theta}(\mathbf{x}|\mathbf{z})\), while regularizing the distribution \(q_{\phi}(\mathbf{z}|\mathbf{x})\) to follow a given prior distribution \(p_{\theta}(\mathbf{z})\). We can see that this equation involves \(q_{\phi}(\mathbf{z}|\mathbf{x})\) which _encodes_ the data \(\mathbf{x}\) into the latent representation \(\mathbf{z}\) and a _decoder_\(p_{\theta}(\mathbf{x}|\mathbf{z})\), which generates \(\mathbf{x}\) given a \(\mathbf{z}\). This structure defines the _Variational Auto-Encoder_ (VAE), where we can use parametric neural networks to model the _encoding_ (\(q_{\phi}\)) and _decoding_ (\(p_{\theta}\)) distributions. VAEs are powerful representation learning frameworks, while remaining simple and fast to learn without requiring large sets of examples [11]. However, the original formulation of the VAE entails several limitations. First, it has been shown that the KL divergence regularization can lead both to uninformative latent codes (also called _posterior collapse_) and variance over-estimation [12].