# Classical Music Prediction and Composition by Means of Variational Autoencoders
## 3 Variational Autoencoders  
Variational Autoencoders arose from the evolution of autoencoders [6; 36; 37]. Both techniques aim to codify a set of data into a smaller vector, and then reconstructing the original data from this vector. In AEs, the set of data points in the input space is mapped to a smaller vector, which is a point in a new space with fewer dimensions called latent space. Usually, Multilayer Perceptrons (MLPs) are used for the codification and decodification tasks. This approach allows for interesting tasks such as information compression. However, in this latent space, other points different than those obtained from an input do not usually correspond to output with significance in the original space. The latent space built by VAEs is usually more compact by the other latent spaces built by other techniques such as autoencoders. Thus, in this latent space, those similar inputs usually have latent representations close to each other, and the points in the space between them correspond to outputs that have a significance in the original space. VAEs allow powerful representations while being a simple and fast learning framework. The objective of Variational Autoencoders is to find the underlying probability distribution of the data \(p(x)\), where \(x\) is a vector in a high dimensional space. In a lower-dimensional space \(z\), a set of latent variables are considered. It should be noticed that, on the following equations, the bold font variables always represent vectors while regular ones represent single values. The model is defined by the probability distribution described in the Equation (1). \[p(\mathbf{x},\mathbf{z})=p(\mathbf{x}|\mathbf{z})p(\mathbf{z}). \tag{1}\]  
The function \(p(\mathbf{x}|\mathbf{z})\) represents a probabilistic decoder that models how the generation of observed data \(\mathbf{x}\) is conditioned on the latent vector \(\mathbf{z}\sim p(\mathbf{z})\), i.e., describes the distribution of the encoded variable given the original one. The function \(p(\mathbf{z})\) represents the probability distribution of the latent space and is usually modeled by a standard Gaussian distribution. For a specific observation \(\mathbf{x}\), the latent vector can be calculated as \(\mathbf{z}\sim p(\mathbf{z})\), and from this vector an approximation of the original input vector can be calculated by \(\mathbf{x}\sim p(\mathbf{x}|\mathbf{z})\). The probabilistic encoder is given by the function \(p(\mathbf{z}|\mathbf{x})\), which describes the distribution of the encoded variable given the original one. Given the Bayesian rule (Equation (2))  
\[p(\mathbf{z}|\mathbf{x})=\frac{p(\mathbf{x}|\mathbf{z})p(\mathbf{z})}{p( \mathbf{x})}=\frac{p(\mathbf{x}|\mathbf{z})p(\mathbf{z})}{\int p(\mathbf{x}| \mathbf{z})p(\mathbf{z})dz}. \tag{2}\]  
The integral in the denominator can be approximated by means of variational inference with distribution \(q(\mathbf{z}|\mathbf{x})\). This model works as an encoder, and, for a specific \(\mathbf{x}\), emits two latent vectors, \(\mu\) and \(\sigma\), that represent the mean and standard deviation of the Gaussian probability for that data:  
\[\boldsymbol{q(\mathbf{z}|\mathbf{x})}=\boldsymbol{\mathcal{N}(\mathbf{z};\mu( \mathbf{x}),\sigma^{2}(\mathbf{x}))}. \tag{3}\]  
The optimization problem consists of minimizing the Kullback-Leibler (KL) divergence between the approximation and the original density. This value measures the information lost when using \(q(\mathbf{z}|\mathbf{x})\) to approximate \(p(\mathbf{z}|\mathbf{x})\), and is given by:  
\[D_{KL}(q(\mathbf{z}|\mathbf{x})||p(\mathbf{z}|\mathbf{x}))=\mathbb{E}_{q( \mathbf{z})}[log\,q(\mathbf{z}|\mathbf{x})]-\mathbb{E}_{q(\mathbf{z})}[log\,p (\mathbf{x},\mathbf{z})]+log\,p(\mathbf{x}). \tag{4}\]  
This equation can be rewritten as  
\[log\,p(\mathbf{x})=D_{KL}(q(\mathbf{z}|\mathbf{x})||p(\mathbf{z}|\mathbf{x})) -\mathbb{E}_{q(\mathbf{z})}[log\,q(\mathbf{z}|\mathbf{x})]+\mathbb{E}_{q( \mathbf{z})}[log\,p(\mathbf{x},\mathbf{z})]. \tag{5}\]  
Here, a function called evidence lower bound (ELBO) can be defined.