## 2 State-Of-Art
### Normalizing Flows  
In order to transform a probability distribution, we can rely on the _change of variable_ theorem. As we deal with probability distributions, we need to _scale_ the transformed density so that it still sums to one, which is measured by the Jacobian of the transform. Formally, let \(\mathbf{z}\in\mathbb{R}^{d}\) be a random variable with distribution \(q(\mathbf{z})\) and \(f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}\) an invertible smooth mapping. We can use \(f\) to transform \(\mathbf{z}\sim q(\mathbf{z})\), so that the resulting random variable \(\mathbf{z}^{\prime}=f(\mathbf{z})\) has the following probability distribution  
\[q(\mathbf{z}^{\prime})=q(\mathbf{z})\left|\det\frac{\partial f^{-1}}{\partial \mathbf{z}^{\prime}}\right|=q(\mathbf{z})\left|\det\frac{\partial f}{\partial \mathbf{z}}\right|^{-1} \tag{5}\]  
where the last equality is obtained through the inverse function theorem [7]. As we can see, this allows us to perform inference by relying on a more complicated (transformed) distribution \(q(\mathbf{z}^{\prime})\), while still being able to keep the simplicity of a mathematical development based on \(q(\mathbf{z})\). Now, we can iteratively apply this reasoning and perform an arbitrary number of transforms to our original variable such that \(\mathbf{z}_{k}=f_{1}\circ...\circ f_{k}(\mathbf{z}_{0})\), in order to obtain a final distribution \(\mathbf{z}_{k}\sim q_{k}(\mathbf{z}_{k})\) given by  
\[\begin{split} q_{k}(\mathbf{z}_{k})&=q_{0}(f_{1}^{- 1}\circ...\circ f_{k}^{-1}(\mathbf{z}_{k}))\prod_{i=1}^{k}\left|\det\frac{ \partial f_{i}^{-1}}{\partial\mathbf{z}_{i}}\right|\\ &=q_{0}(\mathbf{z_{0}})\prod_{i=1}^{k}\left|\det\frac{\partial f _{i}}{\partial\mathbf{z}_{i-1}}\right|^{-1}\end{split} \tag{6}\]  
This series of transformations, called a _normalizing flow_[7], can turn a simple distribution into a complicated multimodal density. For practical use of these flows in inference, we need to define transforms whose Jacobian determinants are easy to compute. Interestingly, _Auto-Regressive_ (AR) transforms fit this requirement as they lead to a triangular Jacobian matrix. Different types of AR flows were proposed such as _Inverse AR Flows_ (IAF) [14] and _Masked AR Flows_ (MAF) [15]. These flows allow to introduce dependencies between different dimensions of the original random variables.  
Normalizing Flows in VAEs  
Normalizing flows allow to address the simplicity of variational approximations by complexifying their posterior distribution [7]. In the case of VAEs, we parameterize the approximate posterior distribution with a flow of length \(K\), \(q_{\phi}(\mathbf{z}|\mathbf{x})=q_{K}(\mathbf{z}_{K})\), and the new optimization loss can be simply written as an expectation over the initial distribution \(q_{0}(\mathbf{z})\)  
\[\begin{split}\mathcal{L}&=\mathbb{E}_{q_{\phi}( \mathbf{z}|\mathbf{x})}\left[\log q_{\phi}(\mathbf{z}|\mathbf{x})-\log p( \mathbf{x},\mathbf{z})\right]\\ &=\mathbb{E}_{q_{0}(\mathbf{z_{0}})}\left[\ln q_{0}(\mathbf{z_{0} })\right]-\mathbb{E}_{q_{0}(\mathbf{z_{0}})}\left[\log p(\mathbf{x},\mathbf{z }_{K})\right]-\mathbb{E}_{q_{0}(\mathbf{z_{0}})}\left[\sum_{i=1}^{k}\log\left| \det\frac{\partial f_{i}}{\partial\mathbf{z}_{i-1}}\right|\right]\end{split} \tag{7}\]  
The resulting objective can be easily optimized since \(q_{0}(\mathbf{z})\) is still a Gaussian distribution from which we can easily sample. However, the final samples \(\mathbf{z}_{k}\) used by the decoder are drawn from the much more complex transformed distribution.