<figure><img src="image_87.png" /></figure>

Dynamical
compression

<figure><img src="image_88.png" /></figure>

State-space
model

**Figure 4.13:** The process that we call dynamical compression rather extracts a single latent vector for a given sequence, using a RNN-like feature extractor, while the state-space approach used previously models one latent vector by incoming observation.

using RNNs.

**Latent auto-regressive modeling.** Considering AEVB as an observation model $p(x_t|z_t)$ with a corresponding approximated posterior $q(z_t|x_t)$, we thus only miss to model the corresponding transition operator $p(z|z_{t-1})$ to turn the system into a full state-space system. In a recent paper Razavi & al. proposed to model a prior $p(z_t)p(z_{t-1}) = N(\alpha z_{t-1}, \sigma_\epsilon)$, such that the correlation between successive latent steps can be controlled with the parameter $\alpha$. Similarly, Rochemore & al. propose to model the latent dynamics using Dynamical Movement Primitives, that is a point attractor system described by second-order dynamics $\tau\ddot{z} = \alpha(\beta(g-y) - \dot{z}) + f$, $g$ being the position goal of the movement and $f$ enforcing the trajectory dynamics. The work proposed by Archer & al. is the first one to explicitly bridge AEVB and state-space models, modeling the posterior distribution $q(z|x) = q(z_1, ..., z_T|x_1)$ as a product of normal distributions, such that $q(z|x) = N(z|0, C)N(z|\mu(x), \Sigma(x))$, hence separating the covariance matrix obtained from amortization and the desired temporal deviation of the sequence [363]. A variational derivation of Kalman processes 4.7 was also proposed by Krishnan & al, relaxing the linear assumptions of transition and observation matrices using variational inference. This model enabled to use variational learning for *counter-factual* inference, allowing to predict the reaction of the system's behavior over a different set of external actions [364].

**RNN-based approaches.** Another way to introduce recurrence in AEVB models is to define encoding and decoding functions as RNNs modules, generating a vector of hidden variables $[h_1...h_T]$ for each step $x_1...x_T$. If we aim to model the full sequence using a single latent vector $z$, we can then use the last hidden state $h_T$ as a *context vector*, that is fed to an additional layer to obtain the parameters of the distribution $p(z|x_1...x_T)$. In that case, we can generate a full sequence $x_1...x_T$ from a single vector $z$ by giving a replicated sequence $[z...z]$ as input for the decoding RNN. This method differs from auto-regressive decoders by using $z$ as an input for each step of decoding, preventing over-pruning of the latent space. This model, called *recurrent variational auto-encoder*, does not change the expression of ELBO 2.18 and then can be trained as a regular auto-encoder [365].

Alternatively, we can also choose a state-space approach to variational auto-encoding, using the vector $[h_1...h_T]$ given by the encoding RNN to compute a sequence $z = [z_1, ..., z_T]$. Hence, the difference with standard AEVB is that the encoding module is using both the past state $h_{t-1}$ in addition to $x_t$, then inferring the latent state $z_t$ using the module memory. Assuming an auto-regressive distribution for the model evidence $p(x) = \prod_{t=1}^{T} p(x_t|\mathbf{x}_{<t})$, we then derive a sequential lower-bound such that [366, 367]

$$ \log p(\mathbf{x}_{0...T}) = \mathbb{E}_{z_1...z_T \sim q(z|x)} \left[ \sum_{t=0}^{T} \log p(\mathbf{x}_t|z_t) \right] - D_{KL}[q(z)||p(z)] \quad (4.18) $$
<figure><img src="image_89.png" /><figcaption>Figure 4.14: Different propositions of variational recurrent networks: RVAE [365], STORN [372], VRNN [373], SRNN [369], Z-forcing [370]</figcaption></figure>

using the Markov hypothesis of the decoding function $\log \prod_{t=1}^{T} p(x_t|x_{<t}) = \log \prod_{t=1}^{T} p(x_t|\mathbf{h}_t)$. Bayer & al. propose to define the prior $p(\mathbf{z})$ as Gaussian process $GP(0, k(\min r, t))$, regularizing the latent sequence $[\mathbf{z}_0, \mathbf{z}_1 - \mathbf{z}_0, ..., \mathbf{z}_T - \mathbf{z}_{T-1}]$ on $p(\mathbf{z}) \sim N(0, I)$ [366]. However, in this case the transition model $p(\mathbf{z}_t|\mathbf{z}_{t-1})$ does not explicitly appear in the variational process. Chung & al. then proposed Variational Recurrent Neural Network (VRNN), defining inference and generation processes as

$$p(\mathbf{z}_t) = N(\mu_p(\mathbf{h}_{t-1}), \sigma_p^2(\mathbf{h}_{t-1}))$$

$$p(\mathbf{x}_t|\mathbf{z}_t) = N(\mu_p(\mathbf{h}_{t-1}, \mathbf{z}_t), \sigma_p^2(\mathbf{h}_{t-1}, \mathbf{z}_t))$$

$$q(\mathbf{z}_t|\mathbf{x}_t) = N(\mu_q(\mathbf{h}_{t-1}, \mathbf{x}_t), \sigma_q^2(\mathbf{h}_{t-1}, \mathbf{x}_t))$$

where the transition model is modeled in the prior, using an additional MLP module. A similar model was recently proposed by Gregor & al., performing random time jumps between states to reinforce the temporal robustness of the learned transition function [368].

Another way to reinforce the temporal features learned by sequential AEVB is to train the encoder using the future steps of the sequence, splitting the information used by the encoding and decoding modules to prevent latent over-pruning. Fraccaro & al. propose to directly mode the transition $q(\mathbf{z}_t|\mathbf{z}_{t-1}, \mathbf{h}_{t>T}, \mathbf{x})_t$ performing variational approximation of the transition model using an anti-causal RNN for inference [369]. This system can then be understood as a stochastic bi-directional RNN, with an additional latent space that joins the information between backward and inference processes. This proximity to LSTMs brought some authors to directly use these models as LSTM units in more complex architectures, where additional variational models can infer anti-causal features from causal features [370] and vice-versa [371].

A summarizing table showing probabilistic graphs for inference and generation models for all these systems can be seen fig. 4.14.
**Disentangling temporal and static features.** The methods introduced in previous paragraphs are then split in two different approaches, modeling a full sequence $x_{0:T}$ whether by a single latent state $z$, or by a sequence of states $z_{0:T}$. However, performing both could provide a way to disentangle dynamical from static features, splitting the latent variables in two kinds : dynamical latent variables $z' = [z'_1 .. z'_N]$, that encode dynamics of the input series, and static features $z^d$ that are invariant across the whole sequence (see fig. 4.15). Hsu & al. recently proposed to disentangle these latent variables using a structured prior $p(z'_t, z^d_t, \mu) = p(\mu)p(z^d_t|\mu)p(z^d_t)$ where the variables $z^d_t$ is conditioned on a feature vector $\mu$ that is extracted from the whole sequence. A version of this model was consequently proposed by Li & al., where an explicit transition model between $z_t$ and $z_{t-1}$ in both variational and generation models [374]. In a similar manner, Grathwohl & al. disentangled the two latent variables $z^d$ and $z'$ between space and time by using different generation priors, the first modeling independent variables across time and the second with an explicit transition model [375]. These experiments show that AEVB is able to shape its variational distribution on sequential data depending on its architecture, motivating our temporal hierarchical approach developed in section 4.4.

## 4.3 Variational methods for latent space prediction

In the last section we described two different approaches for time series analysis, that are somehow reflecting to the two first sections of chapter 2. The first rather model a given time series {$x_t$}$_{t \in [0...T]}$ as a sample of a stochastic process, using model identification or dynamical Bayesian inference to infer higher-order parameters of the underlying model, while the second rather rely on high-capacity series models such as RNNs to whether extract high-level features using a suitable loss function, or to model an auto-regressive process that learns by predicting the next inputs over a short time range. While the first are still commonly used in the domain of DSP, they do not allow to model long-term temporal scales, and the tractability assumption of Bayesian models often prevent to model complex time series, and enforce the definition of discrete latent spaces. Reversely, the second approach provided powerful modeling tools for time series, the recent models being able to generate convincing raw audio waveform with a extended temporal field compared to former auto-regressive techniques. However, the prohibitive computational cost of these models, as well as their lack of interpretability and control make these models very difficult to involve in creative processes.

We saw that AEVB could be beneficially used to mix both approaches, whether by performing dynamical compression (identifying a full sequence with a single vector $z$) or by defining a state-space model (identifying the series with a latent sequence $z_1, z_T$). Sequential approaches

<figure><img src="image_90.png" /><figcaption>**Figure 4.15:** Separating the latent space between static (red) and dynamical (green) dimensions can enforce the disentangling between global and local features of the sequence.</figcaption></figure>
with AEVB, contrary to most neural methods to time series modeling, allow to perform both analysis and generation sequences, thus gaining in terms of control and interpretability. Furthermore, as they allow expressive relations between the data and latent spaces, they also provide an alternative to most Bayesian dynamical approaches, whose tractability requirement often reduced the scope of possible models. We summarized sec. 4.2.3 the models proposed so far to perform time series modeling with the AEVB framework, that whether rely on the explicit definition of a transition model, or resorting to RNNs as encoding and decoding functions. A full definition of recurrent AEVB objectives can be then described

* analysis, extracting useful local features from a time series for a supervised objective

* generation, using the extracted features to generate the corresponding data

* temporal modeling, retrieving the latent dynamics

then adding an additional task to standard AEVB methods. In this section, we propose to add a fourth objective to recurrent AEVB, that is the prediction of the future latent states of the series. Indeed, most of the described models are not able to predict future values of the time series, except the ones from Chung & al. and Bayer & al. [372, 373]. However, even these two systems are modeling recurrence directly in the encoding and decoding modules, preventing the dynamical information to be entirely represented in the latent space. In this work we rather propose to model the series dynamics entirely in the latent space, relying on the idea that performing prediction only using then latent space will enforce it to get a temporal consistency. Moreover, performing dynamics modeling in the latent space is much cheaper computationally than in the data or hidden layers because of its much lower dimensionality, allowing to predict much longer temporal scopes than previous models, only predicting one step at a time (except for SampleRNN). Hence, in this section we do not model one-step transition models $p(z_t|z_{t-1})$ but extended predictive distributions $p(z_{>t}|z_{\le t})$, that are jointly trained with the VAE. As the joint training of the model and of the predictor will apply an implicit regularization of the latent space, its topology will be strongly influenced by the chosen prediction method.

We then propose three different prediction methods, each of them relying on diverse theoretical backgrounds. The first proposed method is based on *Contrastive Predicting Coding*, a prediction technique that is based on mutual information maximization between extracted *contexts* and the predicted latent features. The second method uses *Normalizing Flows* (see sec. 2.3.2.1), that uses a sequence of invertible local transformations to model the temporal evolution of the predicted latent sequences. Finally, we propose a *Gaussian Process* prediction method, that performs a continuous regression on the encoded sequences to predict the next latent states.

<figure><img src="image_91.png" /><figcaption>**Figure 4.16:** We add a *prediction* task in addition to analysis and synthesis process, modeling the dynamics of the data.</figcaption></figure>
<figure><img src="image_92.png" /><figcaption>**Figure 4.17:** In the CPC prediction method, we extract higher-level variables, named *contexts*, that are enforced to model *slow features* that are used to predict the next step using linear predictors.</figcaption></figure>

### 4.3.1 Contrastive Predictive Coding

Contrastive Predicting Coding (CPC) is a framework proposed by van den Oord to extract *context variables* from a given set of latent features, based on maximizing the mutual information between the extracted contexts and the feature vector [376]. CPC is based on Noise-Contrastive Information (NCI), a framework proposed by Gutmann & Hyvärinen used for density estimation, allowing to train a parametric model $p(x; \theta)$ on log-likelihood using a contrastive data distribution $\mathbf{n}$. Density estimation formally aims to find the optimal parameters $\theta$ of a generative distribution $p(x; \theta)$ that maximizes the overall likelihood of the data. As the generative $p(x; \theta)$ is a probability density function it is constrained by the equality $\int p(x; \theta)d\mathbf{x} = 1$, that is often reformulated

$$p(\mathbf{x}) = \frac{\hat{p}(\mathbf{x}; \theta)}{Z(\theta)}$$

where $Z(\theta) = \text{intp}(\mathbf{x}; \theta)d\mathbf{x}$ is the *partition function* of $\hat{p}(\mathbf{x}; \theta)$, that is the *unnormalized density function*, or *free energy*, of $p(\mathbf{x}; \theta)$. This decomposition, that is often used in undirected probabilistic graphs such as Boltzmann Machines [377, 378] (see sec. 2.1.2), allows to model probabilities with arbitrary score functions without caring about the normalization. However, modeling this partition function is often problematic, requiring estimation methods that are often tedious. NCI rather propose to train the parameters of $p(\mathbf{x}; \theta)$ by using the density ratio trick (see sec. 2.3.2.3) transforming a density estimation to a classification task

$$\mathcal{L}_{\text{NCE}} = \mathbf{E}_{\mathbf{x}} \log r(\mathbf{x}; \theta) + \log 1 - r(\mathbf{n}; \theta) \quad (4.19)$$

where $r(\mathbf{x}; \theta) = \sigma(\frac{q(\mathbf{x})}{p(\mathbf{x};\theta)})$ is the density ratio between $p(\mathbf{x}, \theta)$ and a contrastive distribution $p(\mathbf{x})$. This framework then allows to optimize
$\hat{p}(x; \theta)$ without $Z(\theta)$, optimizing the parameters $\theta$ to maximize the density $p(x, \theta)$ on data $x$ while minimizing it when drawn from the contrastive distribution $q(x)$.

The prediction method proposed by CPC is to extract context variables $c_t$ from a sequence of latent variables $[z_0...z_t]$ that is then used to predict latent variables $z_{t+1}...z_{t+K}$ with $K$ simple linear predictors (see fig 4.17). The context variables and the predictors are based on InfoNCE, that derives the NCE criterion to maximize the mutual information between latent vectors $[z_t...z_{t+k}]$ and contexts $c_t$

$$ \mathbb{L}_{InfoNCE} = -\mathbb{E}_{x} \frac{f_k(\mathbf{z}_{t+k}, \mathbf{c}_t)}{\sum_{\mathbf{z}_j} f_k(\mathbf{z}_j, \mathbf{c}_t)} \quad (4.20) $$

where $f_k$ models the density ratio between variables $z_{t+k}$ and $c_t$, and $z_j$ are randomly taken contrastive sequences. InfoNCE intrisically optimizes the mutual information between predictions $z_{t+k}$ and the context $c_t$, as it lower-bounds $\mathbb{I}(z_{t+k}; c_t) \ge \log N - \mathcal{L}_{InfoNCE}$. Maximizing this mutual information then enforces the context variables $c_t$ to represent slow features of the evolution of $z$, such that $c_t$ represents robust local states that are not sensitive about the latent dynamics. Contexts $c_t$ are obtained using an auto-regressive model $c_t = g(z_{\le t})$, generally modeled with GRUs (see sec. 4.2.1).

While CPC is a light method to extract slow-feature context vectors from latent variables, the original framework can only be used to extract the context variables $c_t$ from the hidden vectors $h$, that can be used to obtain non-invertible representations of opaque models (such as WaveNet). Here, we involve the latent states predicted by the CPC module in the generation process, and evaluate how the predicted states can be used to generate the future steps $x_{>t}$. This prediction method can thus be understood as modeling an implicit distribution $q(z_{t>\tau}|z_{<\tau}))$ (see sec. 2.1.3.2). The overall model can then be described with

$$ q(\mathbf{z}_{t \le \tau} | \mathbf{x}_{t \le \tau}) = \prod_{i=0}^{\tau} q(\mathbf{z}_i | \mathbf{x}_i) \quad (4.21) $$

$$ c_t = g(\mathbf{z}_{<t}); \mathbf{z}_{t+k} = \mathbf{W}_k c_t \quad (4.22) $$

$$ p(\mathbf{x}|\mathbf{z}) = \prod_{i=0}^{\tau} p(\mathbf{x}_i|\mathbf{z}_i) \quad (4.23) $$

where the full latent sequence is obtained by concatenating the encoded and predicted vectors $\mathbf{z} = [\mathbf{z}_{t \le \tau}; \mathbf{z}_{t+1}...\mathbf{z}_T]$. We then derive the following lower-bound to train the overall model

$$ \mathcal{L}_{CPC} = \mathbb{E}_{\mathbf{z}_0...\mathbf{z}_{\tau} \sim q(\mathbf{z}_{\le \tau} | \mathbf{x}_{\le \tau})} [p(\mathbf{x}|\mathbf{z})] + D_{KL}[q(\mathbf{z}_{t \le \tau} | \mathbf{x}_{t \le \tau}) \| p(\mathbf{z})] \quad (4.24) $$

$$ + \mathcal{L}_{InfoNCE}(c_t, \mathbf{z}_{t+1...t+k}) + \mathcal{R}[\mathbf{z}_{t>\tau}, p(\mathbf{z})] \quad (4.25) $$

where $\mathcal{R}[\mathbf{z}_{t>\tau}]$ is an optional regularization term enforcing the implicit distribution of $\mathbf{z}_{t>\tau}$ to match the prior $p(\mathbf{z})$, such as MMD or an
<figure><img src="image_93.png" /><figcaption>**Figure 4.18:** With this prediction methods we use normalizing flows to model filtrations of the latent space, modeling complex multi-modal distributions to encode the evolution through time.</figcaption></figure>

adversarial criterion.

### 4.3.2 Filtration learning with temporal normalizing flows

In this section we rather rely on another framework, normalizing flows (see sec. 2.3.2.1), to derive a flexible yet powerful method for latent state prediction. This approach is inspired by the concept of filtrations of the stochastic process literature, that can be intuited by the idea that the more a stochastic process is observed, the more we can accurately describe its underlying properties. More formally, we consider stochastic process {$X(t)}_{t \in \mathcal{T}}$ (T having a total order $\le$) where every $X(t)$ is a random variable from a probability space $(\Omega, \mathcal{A}, \mathcal{P})$ with sample space $\Omega$, $\sigma$-algebra $\mathcal{A}$ and probability measure $\mathcal{P}$. A stochastic is said filtrated if the $\sigma$-algebra $\mathcal{F}_t$ induced by the successive $X(t)$ are ordered non-increasingly, i.e. $\mathcal{F}_k \subseteq \mathcal{F}_l \subseteq \mathcal{A}$ for $k \le l$. The notion of filtration is central in modern probabilities, expressing that the more information we have on a process, the finer we can identify a partition of its sample space $\Omega$.

With this prediction method we aim to represent this concept of filtration in the latent space with normalizing flows, a sequence of invertible transformations $x_K = f_K \circ f_{K-1} \circ \dots \circ f_1(z)$ that can be used to model tractable expressive posteriors $q_K(z_k|x)$ from simple distributions $q_0(z|x)$. Here, we rather propose using normalizing flows to model the local temporal deviation of a starting latent point $z_\tau$ that evolves through time. Therefore, we use all the successive $z_{\tau+k} = f_{k-1}(z_k)$ to predict the missing time steps, that we then give to the generative model to recover the data $x_{\tau>\tau}$.

This method is related to the idea of filtration as the entrance point $z_\tau$,
which has an uninformative prior $p(\mathbf{z} \sim \mathcal{N}(0, 1))$ will be progressively refined by the flow across time steps, shaping the uniform Gaussian to a complex distribution modelled by the latent dynamics. The locality of the deformations $f_k$ also regularize the temporal consistency of the latent space, such that wide moves in the latent space will consequently model important deformations. Furthermore, normalizing flows are able to transform uni-modal variational distributions $q(\mathbf{z}_\tau|\mathbf{x}_\tau)$ to multi-modal ones, such that flow is able to model the uncertainty of latent dynamics over multiple possible solutions. Normalizing flows can then be a powerful framework for latent prediction, that is able to model complex chains $\mathbf{z}_\tau \to \mathbf{z}_{\tau+1} \to \dots \to \mathbf{z}_{\tau+K}$, while preserving the local consistency of the latent space. In this prediction method, we use a simple planar flow for each time step, that is expressed [172]

$$f(\mathbf{z}) = \mathbf{z} + \mathbf{u}h(\mathbf{w}^{\top}\mathbf{z} + \mathbf{b})$$

that has a tractable log-jacobian, and is very cheap to compute. However, applying the flow to a single $\mathbf{z}_\tau$ to obtain the following latent vectors $\mathbf{z}_t \in [\tau; T]$ would only give a one-step back prediction of the input sequence, and is thus unlikely to accurately model the information given by the entire sequence. We thus propose to amortize the flow by a feature extractor such as RNNs, such that flow is conditioned by a memory vector that embeds the past sequence

$$\begin{aligned}
\mathbf{z}_k &= f_k(\mathbf{h}_\tau) \odot f_{k-1}(\mathbf{h}_\tau) \odot \dots \odot f_1(\mathbf{h}_\tau)[\mathbf{z}_\tau] \\
\mathbf{h}_t &= \text{RNN}[\mathbf{z}_t, \mathbf{z}_{t-1}]
\end{aligned}$$

Each flow $f_k$ is amortized, such that its parameters are obtained from the last state of the RNN with a single-layer network with a ReLU non-linearity. Flow amortization then allows to condition the flow on past steps $\mathbf{z}_1 \dots \mathbf{z}_\tau$, and is not restricted to a given context size because of the RNN underlying process. Amortized temporal flows can then be naturally integrated in the ELBO, as the distributions $q(\mathbf{z}_t|\mathbf{z}_{t-1})$ are tractable. The complete ELBO is then expressed

$$\mathcal{L}_{flow} = \mathbb{E}_{\mathbf{z}_0, \dots, \mathbf{z}_\tau \sim q(\mathbf{z}_{t \le \tau} | \mathbf{x}_{t \le \tau})}[p(\mathbf{x}|\mathbf{z})] + D_{KL}[q(\mathbf{z}_{t \le \tau} | \mathbf{x}_{t \le \tau}) \| p(\mathbf{z})] \quad (4.26)$$

$$+ \sum_{k=1}^{K} \log |\det \frac{\partial f_k(\mathbf{h}_\tau)}{\partial \mathbf{z}_{\tau+k-1}}| \quad (4.27)$$

that is entirely tractable.

### 4.3.3 Gaussian processes regularization

Finally, the third prediction method is to use Gaussian processes as a prior for $\mathbf{z}_{t>\tau}$, using a continuous time regression to model the trajectory $[\mathbf{z}_0 \dots \mathbf{z}_{\tau-1}]$ and then predict the consecutive steps $\mathbf{z}_{t>\tau}$. Gaussian processes are used to get a probability distribution over functions $f$ verifying $\mathbf{z}(t) = f(t)$, where the $t$ are fake inputs representing the index of sequence $\mathbf{z}_{t<\tau}$, normalized between 0 and 1. Remembering section 4.9, we then use a Gaussian prior $\text{GP}(0, K_{tt})$ to model the regression
<figure><img src="image_94.png" /><figcaption>**Figure 4.19:** We can use the obtained the regression performed by the Gaussian process to infer the following states of the trajectory, using the predictive posterior distribution with non-observed values of t.</figcaption></figure>

function f conditioned on pairs {($z_t, t$)}, and obtain the regression over inputs $t^*$ by using the posterior distribution

$$q(\mathbf{z}|\mathbf{f}, \mathbf{Z}, t^*) = \mathcal{N}(k(t^*, t)(\mathbf{K}_{tt} + \sigma_n^2\mathbf{I})^{-1}\mathbf{z}, \quad (4.28)$$

$$k(t^*, t^*) - k(t^*, t)^\top(\mathbf{K}_{tt} + \sigma_n^2\mathbf{I})^{-1}k(t^*, t)^\top \quad (4.29)$$

to obtain the predictions $\mathbf{z}_{>\tau}$ on prediction times $t^* = \tau...\tau + k$.

This Gaussian Processes regression does not suffer from the scalability issues of standard Gaussian processes, at is only involves the inversion of a matrix $\tau \times \tau$, and performs fast as it does not involve any additional module as previous prediction methods. Furthermore, it is also the only method that gives an explicit distribution over $q(\mathbf{z}_{>\tau}|\mathbf{z}_{\le \tau})$, allowing to directly model an overall normal distribution over the full vector $\mathbf{z}$. Introducing the posterior distribution $q(\mathbf{z}|\mathbf{f}, \mathbf{Z}, t^*)$, we obtain the ELBO

$$\mathcal{L}_{GP} = \mathbb{E}_{q(z)}[\log p(\mathbf{x}|\mathbf{z})] - D_{KL}[q(\mathbf{z}|\mathbf{x})||p(\mathbf{z})] \quad (4.30)$$

$$= \mathbb{E}_{q(z)}[\log p(\mathbf{x}|\mathbf{z})] - D_{KL}[q(\mathbf{z}_{<\tau}|\mathbf{x}_{<\tau})||p(\mathbf{z})] \quad (4.31)$$

$$- D_{KL}[q(\mathbf{z}_{>\tau}|\mathbf{f}, \mathbf{Z}, t^*)||p(\mathbf{z})] \quad (4.32)$$

that is entirely tractable, as the distribution $q(\mathbf{z}_{>\tau}|\mathbf{f}, \mathbf{Z}, t^*)$ is a multivariate normal distribution. In this study, we use a simple radial basis function kernel with time length $\ell_{RDF} = \sqrt{3}$ and an input noise variance $\sigma_n^2 = 1e^{-2}$. Note that we could have used the noise variance given from the variational distribution $q(\mathbf{z}|\mathbf{x})$, obtaining a precise estimation of the variances $\sigma_q^2$. Moreover, another interesting feature of the proposed GP prediction is to be defined on real time indices $t \in \mathbb{R}$, allowing a sample random latent positions on the predicted curve for continuous time prediction, or to upsample the trajectory to perform time-stretching operations and enforce the temporal consistency of the latent space. Unfortunately, due to a lack of time, we leave this
improvement to a future work.

### 4.3.4 Learning and evaluation framework.

**Database** As we want to evaluate the performances of the proposed prediction methods on modeling temporal evolutions, the datasets previously used are not relevant, being mostly composed by static sounds. Hence, we rather rely on the *diva_dataset*, described section 3.2.1, that contains more than 11 thousands of synthesizer sounds with many different dynamical shapes (see fig. 4.20). In this experiment, the models were trained on STFT representations of 60 steps on spectral frames of 2048 bins, pre-processed with a log1p non-linearity.

**Models** Similarly to previous experiments, we split the dataset between train and tests sets with a balance of 0.8. The architecture of both encoding and decoding functions are defined using convolutional networks of [64, 32, 16, 8] channels and [3, 5, 7, 9] kernels, using ELU non-linearities and batch-norm normalization. For the CPC auto-regressive modules, we chose a GRU with one layer of 200 units, and the amortized flows are obtained from simple linear layers with a ReLU non-linear units. The VAEs and prediction modules were trained jointly on 100 epochs with an ADAM optimizer a learning rate of $1e-4$, with a $D_{KL}$ regularization with $\beta = 1$ and a warm-up of 10 epochs. We also found beneficial to add a *teacher warm-up* procedure during the first 5 epochs, initially decoding the latent positions coming from the encoder and progressively replaced by the ones provided by the prediction module. This procedure allows to train the latent space directly on the signal during the first epochs, increasing the robustness of the latent process. The number of predicted steps is 30, for 30 input steps, amounting to a total sequence length of 60.

<figure><img src="image_95.png" /><figcaption>**Figure 4.20:** Examples taken from the *diva_dataset* database</figcaption></figure>

<table><caption>**Table 4.1:** Table of the results obtained with the proposed prediction methods on the overall sequence. The error is normalized stepwise, to be compared with the results obtained in section 3. The results out of parenthesis are the obtain with train partition of the dataset, while numbers within are the errors obtained with the test partition.</caption><thead><tr><th></th><th>- log p(x|z)</th><th>SC</th><th>ISD</th><th>D<sub>KL</sub>[q||p]</th><th>MMD[p, q]</th><th>TC[p, q]</th></tr></thead><tbody><tr><th>CPC prediction</th><td>693.61 (705.45)</td><td>0.671 (0.682)</td><td>4.22e-5 (1.58e-4)</td><td>21.49 (21.87)</td><td>4675 1075.73)</td><td>347.04 (360.74)</td></tr><tr><th>Flow prediction</th><td>680.84 (677.58)</td><td>2.00e-2 (2.04e-2)</td><td>4.32e-5 (1.63e-4)</td><td>24.45 (23.72)</td><td>5570 1198.32)</td><td>393.52 (361.23)</td></tr><tr><th>GP prediction</th><td>677.00 (692.59)</td><td>3.16e-2 (3.31e-2)</td><td>4.29e-5 (1.63e-4)</td><td>24.58 (24.04)</td><td>3476 (772.40)</td><td>231.94 (232.67)</td></tr></tbody></table>
<table><caption>**Table 4.2:** Table of the results obtained with the proposed prediction methods, only on the predicted part.</caption><thead><tr><th></th><th>-log p(x|z)</th><th>ℓ<sup>1</sup></th><th>ℓ<sup>2</sup></th></tr></thead><tbody><tr><td>CPC</td><td>673.63 (685.73)</td><td>105.5 (102.03)</td><td>98.40 (111.53)</td></tr><tr><td>Flow</td><td>661.06(658.36)</td><td>94.67 (91.90)</td><td>94.60 (97.87)</td></tr><tr><td>GP</td><td>674.53 (702.00)</td><td>89.83 (105.9)</td><td>106.93 (106.76)</td></tr></tbody></table>

<figure><img src="image_96.png" /><figcaption>**Figure 4.21:** reconstruction examples (top) and original spectra (bottom) for (a) CPC (b) Flow (c) GP prediction methods.</figcaption></figure>

**Reconstruction evaluation.** We show the results obtained with the overall sequence in table 4.1, providing reconstruction and regularization estimators, and the reconstruction results obtained only on the prediction part 4.2.

We observe that the reconstruction results obtained by the three methods are close, flow and GP methods performing slightly better results than the CPC. If we only observe the prediction part, the best performance is achieved by the flow prediction method, even if all the results are also very close. To study qualitatively the reconstruction probabilities, pairs of obtained reconstructions are depicted fig. 4.21. We can see that the predictions manage to catch inner variations of the spectra, showing that the modules efficiently model inner the inner dynamics of each data examples. Though, we can see that CPC has a certain tendency to produce undesired artifacts, as can be seen on the second example that are absent of other prediction methods. Conversely, GP predictions seem to smooth the generated sound, that is coherent with the inner smoothing effect of Gaussian process (see next paragraph). Finally, flow predictions are almost perfect, managing to catch even subtle variations of the data. The qualitative properties of these predictions methods are thus a little different, and can then be optimally chosen depending on the nature of the dataset.

**Latent evaluation.** Regarding latent spaces, we can see from the results table 4.1 that the GP prediction method provides the best regularization result. However, the Kullback-Leibler divergence are low and for the three methods, showing that all of them provide accurately regularized latent spaces. Differences between the methods can be still observed with the MMD and to Total Covariance, the best scores being obtained by the GP while the worst are obtained with the temporal flows. This points that the GP prediction method, the most restrictive in terms of geometrical constraints, manage to successfully regularize the latent space without (or a little) altering the reconstruction abilities of the model.

To get more qualitative insights of the temporal organization of the space, we can observe the paths obtained by forwarding two different sounds from the dataset, and to generate from several interpolation steps between the two obtained trajectories. Examples of trajectories for the three methods are depicted fig. 4.23, providing interesting insights on their respective behavior (projected on a PCA extracted from the latent projects of the full dataset). First, we can see that the trajectories obtained with the CPC method are favoring stable latent "nodes", the distribution of projection points corresponding to the stationary part of the signal lying on a narrow latent region. This is enforced by the slow-feature encourages by the CPC training (see fig. 4.22), and then show that CPC influences the topology of the latent space around attractors representing specific spectral distributions. Regarding temporal flows, we can see that the high capacity modeling abilities of NFs allow to model arbitrary trajectories of the latent space, showing that an expressive prediction method does not regularize the representation and rather fits the latent topology; however, this flexibility allow to provide the best reconstruction results. Finally, we can see that
GP prediction is the one that conditions the most the latent topology,
providing very smooth trajectories that correspond to curves sam-
pled from a GP process. This method is then the one that perform the
strongest regularization of the latent space, hence explaining its high
generalization scores but also its lowest (albeit satisfactory) reconstruc-
tion results.

Hence, we can see that these three prediction methods, while providing
satisfying results, are qualitatively different, and apply diverse implicit
regularization about the temporal behavior of the latent space. These
methods can be then wisely chosen depending on the desired structure
of the representation, from the less restraining and the more efficient
in terms of reconstruction (flows) to the most regularizing (GP), or
providing compact latent evolutions (CPC). However, while allowing
to regularize the latent space on temporal criteria, prediction methods
do not yet allow to *model* the trajectories of the dataset. Hence, an
additional modeling is required, that would allow us to extract features
from the obtained trajectories.

<figure><img src="image_97.png" /><figcaption>**Figure 4.22:** CPC curves obtained for 5 random examples, each dimension plotted apart. We can see that the CPC are evolving slowly, contrary to the corresponding latent curves.</figcaption></figure>
<figure><img src="image_98.png" /><figcaption>**Figure 4.23:** Examples of morphing between an original (blue) and target (yellow) trajectories (through PCA) obtained from dataset examples for (top) CPC (middle) Flow (bottom) GP prediction modules.</figcaption></figure>
## 4.4 Hierarchical latent spaces and raw waveform learning

In the previous section, we investigated three different methods to perform prediction directly on the latent space, modeling dynamics $q(z_{t>\tau}|p(z_{t\le\tau}))$ with a broader prediction scope than the one provided by state-space models approaches. The inner idea of these methods was that training jointly the model on inference, generation and prediction objective was constraining the latent space to reflect temporal features of the data. We saw that these three prediction methods was providing substantially different behaviors, that could thus be chosen depending of the target application or dataset. However, these prediction methods do not help us to really extract temporal features of the data, as each state vector $z_t$ represents a single data step $x_t$.

Reminding section 4.2.3, there are two different manners to encode temporal features in the latent representation: whether formulating the VAE as a state-space model, then having one latent vector $z_t$ per data input $x_t$ and a suitable transition model $p(z_t|z_{t-1})$, or using recurrent encoders / decoders to obtain a global latent vector $z$ for the whole sequence. We investigated the first approach during previous section, showing the ability of latent spaces to incorporate temporal features with a given prediction model. However, the second is also interesting, as it extracts global features from a target sequence and then also implicitly models its dynamics.

We then propose to combine both approaches with hierarchical latent spaces, modeling the trajectories observed in the obtained representation. This approach, called *ShrubVAE*, allows us to represent different temporal scales of the input, encoding the overall structure of the sequence at top-level, and progressively model the local temporal structure by upsampling across layers. This method then allows to considerably extend the scope of prediction process, as a single vector corresponds to several steps of the sequence. Furthermore, this architecture allows us to access the retrieved features corresponding to different time scales, then performing multi-scale analysis. We can then analyze and generate with different temporal scopes by choosing the interaction layer, from a low-level state-space model to a full sequence at the top.

### 4.4.1 ShrubVAE: Multi-layered models for multi-scale learning

We then propose a hierarchical latent model performing multi-scale analysis and generation of data signals, that is based on a progressive down-sampling of the incoming time series using variational learning. This approach models long-term dependencies with a few latent vectors in the top layer of the model, while intermediate layers extract local dynamical information. We also perform the prediction methods
<figure><img src="image_99.png" /><figcaption>**Figure 4.24:** Prediction is performed at the higher level of temporality, allowing to significantly reduce the number of prediction states needed to predict the signal.</figcaption></figure>

proposed on the top layer of this model, and requiring less prediction steps as each top latent feature corresponds to several samples of the sequence. These prediction methods should perform better, as it requires a fewer number of steps and are supposedly performed on features containing high-level temporal information. We then evaluate the proposed model by comparing this model to the previous prediction performances obtained with single-layered VAEs, evaluating it the proposed hierarchical architecture is easing the prediction.

**Hierarchical latent spaces.** While AEVB convincingly extracts compressed representations from high-dimensional data, the number of latent dimensions is often constrained by the reconstruction capacity of the model. Some approaches then proposed to alleviate this issue by adding stochastic layers on top of the variational models, extracting higher-level latent features directly from $z_0$. Such approaches where popularized by ladder networks, a deterministic auto-encoder composed from multiple stacks of latent codes, that were trained on a reconstruction error between each top-down and down-top processes [379]. In the domain of variational approaches, this was developed by Rezende & al. with *Deep Latent Gaussian Models*, proposing a multi-layer approach to AEVB using a deterministic encoder across successive layers. A full variational approach to multi-layer models were consequently proposed by Kaae & al., based on the conditional
inference & generation processes with L layers [184]

$$q(\mathbf{z}_L, \dots, \mathbf{z}^1|x) = q(\mathbf{z}^1|x) \prod_{l=2}^{L} q(\mathbf{z}_l|\mathbf{z}_{l-1}) \qquad (4.33)$$

$$p(\mathbf{z}_L, \dots, \mathbf{z}^1, \mathbf{x}) = p(\mathbf{z}^L)p(\mathbf{x}|\mathbf{z}^1) \prod_{l=1}^{L-1} p(\mathbf{z}^{l-1}|\mathbf{z}^l) \qquad (4.34)$$

where each conditioned model is parametrized as a diagonal normal distribution, similarly to the VAE. The obtained ELBO is then

$$p(\mathbf{x}) \geq \mathbb{E}_{\mathbf{z}^1, \dots, \mathbf{z}^L \sim q(\mathbf{z})}[p(\mathbf{x}|\mathbf{z}^1)] + D_{KL}[q(\mathbf{z})\|p(\mathbf{z}_L)] \qquad (4.35)$$

$$+ \sum_{l=2}^{L-1} D_{KL}[q(\mathbf{z}^l|\mathbf{z}^{l-1})\|p(\mathbf{z}^l|\mathbf{z}^{l+1})] \qquad (4.36)$$

The L regularization terms $D_{KL}[q(\mathbf{z}^l|\mathbf{z}^{l-1})\|p(\mathbf{z}^l|\mathbf{z}^{l+1})]$, that are all tractable, can then be understood as a criterion that matches the layer-wise distributions provided by encoders and decoders, ensuring the consistency of the representation. Unfortunately, further investigations of these hierarchical models showed that these higher-level latent variables were rather inexpressive, and barely used by the low-level decoding model $p(\mathbf{x}|\mathbf{z}^1)$. Information sharing between encoding and decoding processes was consequently proposed by Kaae with Ladder Variational Networks, improving the inference of higher-level latent variables $\mathbf{z}^l$ [380] by explicit information sharing between encoders and decoders. However, the gradient dissipation across higher levels often weakens their representativeness, requiring an additional criterion as semi-supervised task to be non-degenerated.

**Multi-scale inference with hierarchical spaces.** ShrubVAE rather proposes to use hierarchical latent layers to model different time scales, amounting to successive down-/up-sampling (resp. during inference / generation) steps of the latent sequences. This progressive dilation task prevents the higher-level representations to be data-insensitive, as each feature vector has to represent the temporal information of the incoming sequence. We use simple generative and inference models as above, excepting a window $\mathbf{z}^l_{t_l, \dots, t_l+v_l}$ of size $v_l$ is encoded to one single latent vector $\mathbf{z}^{l+1}_{t_l+1}$ using a RNN encoder. The inference distribution can then be formulated as

$$q(\mathbf{z}_t^0|\mathbf{x}_t) = \mathcal{N}(\mu_{q,0}(\mathbf{x}_{t_0}), \sigma_{q,0}(\mathbf{x}_{t_0})) \qquad (4.37)$$

$$q(\mathbf{z}_{t_l}^l|\mathbf{z}_{t_l v_l:(t_l+1)v_l}^{l-1}) = \mathcal{N}(\mu_{q,l}(\mathbf{z}_{t_l v_l:(t_l+1)v_l}^{l-1}), \sigma_{q,l}(\mathbf{z}_{t_l v_l:(t_l+1)v_l}^{l-1})) \qquad (4.38)$$

$$q(\mathbf{z}^L|\mathbf{z}^{L-1}) = \mathcal{N}(\mu_{q,L}(\mathbf{z}^{L-1}), \sigma_{q,L}(\mathbf{z}^{L-1})) \qquad (4.39)$$

such that a downsampling of factor $v_l$ is applied at each layer $l$. We choose to not perform any downsampling on the first layer $\mathbf{z}^0$, modeling it as a state-space representation of the incoming data, that can be accessed directly to generate data elements separately. The parameters
<figure><img src="image_100.png" /><figcaption>**Figure 4.25:** Probabilistic graph explaining the hierarchical multi-scale architecture of ShrubVAE. This architecture is based on multi-layer dynamical compression, thus performing Bayesian down/up-sampling that allows to efficiently model the temporal structure of the incoming signal.</figcaption></figure>

$\mu_{q,l}(\cdot), \sigma_{q,l}(\cdot)$ are obtained in a similar manner to the RVAE encoding model (see sec. 4.2.3), such that

$$h_t = \mathrm{RNN}(z_t^{l-1}, h_{t-1}) \quad t \in [t_l v_l \dots (t_l + 1) v_l] \quad (4.40)$$

$$\mu_{q,l}(z_t^{l-1}) = A_{\mu}^{\top} h_{(t_l+1)v_l} \quad (4.41)$$

$$\sigma_{q,l}(z_t^{l-1}) = A_{\sigma}^{\top} h_{(t_l+1)v_l} \quad (4.42)$$

so we take the final hidden vector of a RNN to encode $z^l$, trained on a $v_l$-sized window of $z^{l-1}$. The temporal scope taken by the RNN is quite restricted, such that we do not suffer any of the gradient degeneracy problems evoked in sec. 4.2.1. Intuitively, the vector $z^l$ can thus bee understood as encoding the local dynamics of the sequence fragment $z_{t:l+v_l}^{l-1}$, projected on a higher-level latent space. Note that, with this system, a window $v_l$ is held fixed during the training process, but can be arbitrarily set when performing inference or generation.

The generation model is mirroring the encoding process, and *upsamples* an incoming vector $z_{t:l+1}^{l}$ of a factor $\tau_l$, obtaining a vector $z_{t:l+1}^{l} \dots z_{t:l+\tau_l}^{l}$
through the following generative process

$$p(\mathbf{z}^L) = \mathcal{N}(0, I) \qquad (4.43)$$

$$p(\mathbf{z}_{t_l}^{l-1} | \mathbf{z}_{t_l}) = \prod_{\tau=1}^{v_l} \mathcal{N}(\mu_{p,l}(\mathbf{z}_{t_l}^l, \tau), \sigma_{p,l}(\mathbf{z}_{t_l}^l, \tau)) \qquad (4.44)$$

$$p(\mathbf{x}_{t_0} | \mathbf{z}_{t_0}) = \mathcal{N}(\mu_{p,0}(\mathbf{z}_{t_0}), \sigma_{p,0}(\mathbf{z}_{t_0})) \qquad (4.45)$$

such that the joint distribution of the upsampled vector $\mathbf{z}^{l-1}$ is factorized over time steps $t_{l-1}$. Similarly, the parameters of successive distributions $\mu_{p,l}(\cdot, \tau)$, $\sigma_{p,l}(\cdot, \tau)$ is obtained with a RNN, such that

$$\mathbf{h}_t = \text{RNN}(\mathbf{z}_{t+1}^{l}, \mathbf{h}_{t-1}) \quad t \in [t_{l+1}v_{l+1}...(t_{l+1} + 1)v_{l+1}] \qquad (4.47)$$

$$\mu_{p,l}(\mathbf{z}_t^{l+1}) = \mathbf{B}_{\mu}^{\top} \mathbf{h}_t \qquad (4.48)$$

$$\sigma_{p,l}(\mathbf{z}_t^{l+1}) = \mathbf{B}_{\sigma}^{\top} \mathbf{h}_t \qquad (4.49)$$

where the successive distributions for the steps $[t..t + \tau]$ are obtained by iterating a RNN $\tau$ times using the replicated input $\mathbf{z}_{t+1}^{l+1}$. This system then allows to generate several latent variables from a single one, each step being conditioned on the previous with a recurrent mechanism, then generating local dynamics of the curve. Despite the down-/up-sampling procedures performed by stacked latent layers, the ELBO (4.35) is not modified, as the shape of latent trajectories are the same during encoding and decoding processes, and the distributions are factorized across time steps.

### 4.4.2 Learning procedure and progressive hierarchical warm-up

All the parameters of the ShrubVAE are trained end-to-end using backpropagation algorithms, similarly to regular variational auto-encoders. As both generative and inference models are defined as normal distributions with normal covariances, the reparemetrization trick applied by Kingma & al. is still applicable, such that the convergence of the whole training is fast and guaranteed [22]. In this article, we resort to the ADAM optimization scheme [123], using a learning rate of $10e^{-4}$. However, despite the use a warm-up procedure for the top layer and the use of the MMD at the lower level, the training process of the ShrubVAE can still encounter difficulties, due to the network depth. During experiments, we observed that this was due to the balance between inference and generation process, where the effect of the regularization where canceling the information coming from the data during early-training. While similar to the latent pruning shown in standard and multi-layered VAEs, this effect is emphasized by the down-/up-sampling process of the model.

To counter this unwanted tendency, we add to the top *warm-up* layer an additional procedure, called *progressive hierarchical warm-up*, that
schedules the information fed into the different decoders. This scheme aims at introducing the information provided by higher latent layers progressively during early training, such that each encoder / decoder pairs are suitably trained before being given to the lower levels. This is done by regularly increasing the parameter $\alpha$ of a Bernoulli distribution, that is sampled to select whether the input of the decoder $p(\mathbf{z}^l|\mathbf{z}^{l+1})$ comes from the higher decoding layer or from the corresponding encoding distribution as follows:

$$m_l \sim \text{Bernoulli}(\alpha) \qquad (4.50)$$

$$\mathbf{z}_l \sim \begin{cases} q(\mathbf{z}_l|\mathbf{z}_{l-1}) & \text{if } m_l = 0 \\ p(\mathbf{z}_l|\mathbf{z}_{l+1}) & \text{if } m_l = 1 \end{cases} \qquad (4.51)$$

This procedure enforces each decoder to use the information coming from the encoder during early training, and are progressively led to use the information from above layers. In this article, we linearly increment the $\alpha$ parameter from 0 to 1 during the first 20 epochs, that corresponds to approximately the mid-term of the training process with the used datasets.

**Experimental procedure.** To evaluate the ShrubVAE performances, we chose a similar experimental framework than the one used in the previous section. Hence, we first evaluate the generation abilities of the model, comparing it with the results obtained on single-layered VAE (described in the previous section), to ensure that the reconstruction loss does suffer from the down-/up-sampling processes. We then observe the results obtained by each of the three proposed prediction methods above, analyzing how it gains from the temporal hierarchy provided by the model. To compare accurately on these criteria, we then take sequences of the same length than in the previous experiment, such that the total sequence length is 60 with a predictive scope of 30, performed on magnitude spectral bins $\mathbb{R}^{2048}$. All the recurrent modules used to perform dynamical compression are 1-layer units of 100 hidden units, chosen to be very light. We thus trained a total of 6 models, corresponding to the three prediction methods with two different hierarchical architectures:

* a two-layered ShrubVAE-21 with 10x down-sampling, such that the top latent vector sequence $\mathbf{z}^1$ has a total step amount of 6 elements, reducing the amount of predicted states to 3

* a three-layered ShrubVAE-31 with respective 5x and 3x down-sampling, the first latent sequence $\mathbf{z}^1$ then having a total step amount of 12 and the second latent sequence $\mathbf{z}^2$ a total amount of 4 elements, reducing the amount of predicted states to 2.

we used the same prediction settings than in the previous sections, using a 10 epochs prediction warm-up over a total amount of 80 epochs. A $D_{KL}$ divergence were used in every layer $l > 1$, while a MMD loss was used in the first latent layer to provide the best reconstruction results.
<table><caption>**Table 4.3:** Full table of results obtained with 1-layer (previous section), 2-layer and 3-layer ShrubVAEs, for all models. For regularization losses, the layer-wise divergences are sorted in increasing order.</caption><thead><tr><th></th><th>-log p(x|z)</th><th>SC</th><th>ISD</th><th>D<sub>KL</sub>[q||p]</th><th>MMD[p, q]</th><th>TC[p, q]</th></tr></thead><tbody><tr><td><strong>CPC prediction</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>1-layer</td><td>693.61 (705.45)</td><td>0.671 (0.682)</td><td>4.22e-5 (1.58e-4)</td><td>21.49 (21.87)</td><td>4675 (1075.73)</td><td>347.04 (360.74)</td></tr><tr><td>ShrubVAE-2l</td><td>722.76 (730.17)</td><td>0.394 (0.391)</td><td>4.28e-5 (1.61e-4)</td><td>2.81e25 (2.8e25)<br>3.91 (3.934)</td><td>5158 (1253.5)<br>913.30 (222.24)</td><td>775.12 (865.78)<br>52.83 (58.05)</td></tr><tr><td>ShrubVAE-3l</td><td>783.58 (771.37)</td><td>0.165 (0.168)</td><td>4.47e-5 (1.64e-4)</td><td>5.59e25 (5.51e25)<br>1.01e+09 (80.87)<br>1.78 (1.78)</td><td>12539 (2775.7)<br>98.86 (21.47)<br>303.15 (68.723)</td><td>1308.3 (1168.79)<br>82.31 (85.39)<br>12.64 (13.04)</td></tr><tr><td><strong>Flow prediction</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>1-layer</td><td>680.84 (677.58)</td><td>2.00e-2 (0.204)</td><td>4.32e-5 (1.63e-4)</td><td>24.45 (23.72)</td><td>5570 (1198.32)</td><td>393.52 (361.23)</td></tr><tr><td>ShrubVAE-2l</td><td>720.37 (730.66)</td><td>2.60e-2 (2.61e-2)</td><td>4.46e-5 (1.67e-4)</td><td>nan (nan)<br>4.08 (3.49)</td><td>4289 (908.56)<br>976.48 (205.68)</td><td>572.23 (537.48)<br>46.87 (58.05)</td></tr><tr><td>ShrubVAE-3l</td><td>716.14 (732.47)</td><td>0.120 (0.117)</td><td>4.22e-5 (1.58e-4)</td><td>5.59e25 (5.51e25)<br>1.01e+09 (80.87)<br>1.78 (1.78)</td><td>12539 (2775.7)<br>98.86 (21.47)<br>303.15 (68.723)</td><td>1308.3 (1168.79)<br>82.31 (85.39)<br>12.64 (13.04)</td></tr><tr><td><strong>GP prediction</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>1-layer</td><td>677.00 (692.59)</td><td>3.16e-2 (3.31e-2)</td><td>4.29e-5 (1.63e-4)</td><td>24.58 (24.04)</td><td>3476 (772.40)</td><td>231.94 (232.67)</td></tr><tr><td>ShrubVAE-2l</td><td>666.09 (667.4)</td><td>4.03e-2 (4.30e-2)</td><td>4.14 e-5 (1.56e-4)</td><td>nan (nan)<br>4.12 (4.06)</td><td>4437.7 (975.94)<br>780.61 (174.0)</td><td>568.3 (491.68)<br>39.11 (35.72)</td></tr><tr><td>ShrubVAE-3l</td><td>695.00 (707.97)</td><td>6.44e-2 (6.26e-2)</td><td>4.34e-5 (1.62e-4)</td><td>2.32+25(2.30e25)<br>3.69 (80.87)<br>2.86 (2.88)</td><td>7437 (2775.7)<br>39.8 (9.01)<br>607.3 (68.723)</td><td>546.79 (550.02)<br>162.07 (158.92)<br>20.63 (18.49)</td></tr></tbody></table>

**Reconstruction results.** The results obtained for the full sequence are listed table 4.3, and the reconstruction of only the predicted vectors are displayed fig. 4.4. We can see that the ShrubVAE the down-sampling process across layers does not impact significantly the reconstruction error. The overall uncertainty – log p(x|z) of the model globally increase, but the deterministic spectral losses slightly change, and are even decreasing in the case of CPC. A interesting fact is that, while the best reconstruction performance of all the sequence is given by using flow prediction method, similarly to single-layered VAEs, it is the GP prediction method that has the best performances on the predicted sequence, and is barely influenced by the number of added layers. These results then confirm the intuition given previously about the generalization abilities of this method. Nevertheless, for every method the ShrubVAE accurately performs dynamical compression, identifying the overall sequences with only one latent vector without influencing too much the reconstruction loss. However, we do not especially gain in terms of reconstruction from predicting a lower amount of steps, we ensure that the system preserves the overall consistency of the input sequence.

**Regularization results.** Latent losses are also depicted table 4.3, allowing us to describe the overall structure of the latent space of each layer. As we are using multi-layer latent architectures, only the last layer is regularized with an isotropic normal distribution $N(0, I)$, such that lower level latent losses rather reflect the divergence between encoding and decoding distributions $q(z^l|z^{l-1})$ and $p(z^l|z^{l+1})$. We can see that, as we are using MMD to regularize the first latent layer, the $D_{KL}$ is very high. By analysing the decoding projections, we found that the decoding distributions $p(z^1|z^2)$ had very low variance, while inference distributions $q(z^1|x)$ where much broader, explaining the very high values taken by the divergence. However, we can see that higher-level latent spaces are well regularized, especially in ShrubVAE with three

<table><caption>**Table 4.4:** Results obtained with the ShrubVAE with the three prediction methods, only on the predicted part of the sequence.</caption><thead><tr><th></th><th>-log p(x|z)</th><th>ℓ<sup>1</sup></th><th>ℓ<sup>2</sup></th></tr></thead><tbody><tr><td><strong>CPC</strong></td><td></td><td></td><td></td></tr><tr><td>1l</td><td>673.63 (685.73)</td><td>105.5 (102.03)</td><td>98.40 (111.53)</td></tr><tr><td>2l</td><td>676.36 (680.16)</td><td>106.8 (117.16)</td><td>66.40 (117.23)</td></tr><tr><td>3l</td><td>673.63 (734.67)</td><td>105.5 (136.03)</td><td>98.40 (129.5)</td></tr><tr><td><strong>Flow</strong></td><td></td><td></td><td></td></tr><tr><td>1l</td><td>661.06 (658.36)</td><td>94.67 (91.90)</td><td>94.60 (97.87)</td></tr><tr><td>2l</td><td>666.94 (660.6)</td><td>106.8 (98.5)</td><td>99.13 (99.16)</td></tr><tr><td>3l</td><td>697.10 (713.7)</td><td>112.6 (120.67)</td><td>117.2 (125.5)</td></tr><tr><td><strong>GP</strong></td><td></td><td></td><td></td></tr><tr><td>1l</td><td>674.53 (702.00)</td><td>89.83 (105.9)</td><td>106.93 (106.76)</td></tr><tr><td>2l</td><td>642.23 (646.6)</td><td>93.40 (93.4)</td><td>89.80 (92.10)</td></tr><tr><td>3l</td><td>707.97 (662.32)</td><td>99.16 (105.37)</td><td>99.87 (107.07)</td></tr></tbody></table>
layers where the top latent space seems to be very close to an isotropic
normal distribution, without providing degenerated representations.
This can be seen by observing the latent interpolations depicted figure
4.26, where we can see that the trajectories are significantly smoother
for each model. This seems to point out that, in addition to down-
sampling, hierarchical latent spaces allow to regularize the temporal
behavior of the sequences, thus accurately providing a regularized
representation of its inner dynamics. This results are very encouraging,
and are worth to be investigated on higher temporal scopes.
<figure><img src="image_101.png" /><figcaption>**Figure 4.26:** Examples of trajectory morphing between two examples of the dataset for (top) CPC (middle) Flow (bottom) GP prediction modules.</figcaption></figure>
# Composing and performing with generative spaces

In this chapter, we tackle a different aspect of our work that fosters the creative use of the methods developed previously for musical applications. Indeed, while machine-learning techniques are now widely investigated in numerous scientific domains, our main motivation was to develop a framework *design* a generation tool for sound synthesis. Our objective in then two-sided: from one hand, designing machine-learning techniques that are able to opens new technical possibilities for sound synthesis, and from the other hand starting from musical practices to motivate the development of new models. This joint scientific and musical approach is to us seminal in computer music, such that we also aim to develop an usable toolbox, that could be used by both expert and non-export users, and the creation of a musical piece, to explore the creative abilities of the proposed framework.

Thus, in this section we will invest another approach, and adopt a *research & creation* method to the framework development. Indeed, the proposed analysis/synthesis technique differ from synthesis methods developed so far by three aspects: the control space it automatically extract from the data, providing an alternative representation even in the case of sounds synthesized with simple techniques (such as additive synthesis), the diversity sounds it can produce from a limited amount of examples, and the data-centered approach of these methods, integrating the choice of the dataset and of the model architecture in the creative workflow. We will first introduce the developed toolbox, vsacids, and the various features it implements. We will then experiment this workflow with the composition of a performance, ægo, exploring the latent space with a reinforcement-learning agent interacting with a human performer. This work, done jointly with Hugo Scurto, was accepted at the CMMR2019 conference as both as a musical performance and technical paper.

## 5.1 vsacids: a toolbox for variational generation of musical signals

As we aim to provide an analysis-synthesis framework that can be used by both expert and non-expert users, we have to develop several layers of interaction to allow both customization of the algorithm and high-level usage. We then developed a library, called vsacids, that allows easily to train models and to use it real-time or non real-time use cases. This library was conceived in a modular way, allowing deeper customization of models, easily implementation of model improvements. The library also provide methods audio data import, model definitions, and training routines, such that the implemented models can be easily trained by just defining a signature. It also implements high-level methods for non-expert usage, automatically proposing an
architecture for a given dataset with high-level descriptors. This library is using the Python language, and is based on the open-source framework PyTorch* [381], an elegant machine learning framework that allows dynamical definitions of machine-learning models. This library is distributed as a python package, that is open-source be available here: https://github.com/domkirke/vschaos_package.

### 5.1.1 Architecture and design

The vschaos library is based on three seminal classes, *Dataset*, *Model*, and *Loss*, that can be sub-classed to implement new models or specific behaviors for particular applications. The architecture of vsacids is built around three main classes:

**Data format.** The *Dataset* class is an high-level class for collecting data that allow useful routines for data import, transformation, metadata processing, and asynchronous data loading for big datasets. This object assumes a formatted dataset format, that allows it to build a data structure that holds various formats of metadata, so it naturally integrates with the other modules. This dataset structure is defined as shown in figure 5.1, sorted in three folders : a data folder, that contains the audio files, the analysis folder, that contains a list of audio transforms used for training, and a *metadata* folder, that contains a list of tasks. This simple structure allows the *Dataset* object to organize the dataset, and to automatically retrieve an efficient data structure for model training. We also provide a toy dataset generator, based on the grid sampling of parameters of a digital synthesizer, to provide a fast method to evaluate the emerging properties of a model configuration.

**Model definition.** The *Model* class is the base of all the models trainable by the library, and implements the three methods *encode*, *decode* and *forward* that are used to train and use the model. All the models of the library can be defined by a *signature* [*input_params*, *latent_params*], plus model dependent arguments, that defines the variational and generative distributions of the model. These two specifications are defined by an attribute *dim*, that specifies their dimensions, and *dist*, that specifies the distribution of the layers. The model can be easily multi-layered by defining a list of latent parameters, and every input or layer can be split in an arbitrary number of parts. Each encoding and decoding functions are defined as the conjugation of an hidden module and of a distribution module, such that each transformation is defined on the encoded distribution (see figure 5.2).

The model is also attached two additional objects : a list of optimizers, that are used to optimize the model during training, and a list of *manifolds*, that are low-dimensional transformations saved during interaction. Model also provides specific load and save functions, such that the model can be saved with the performance parameters and other diverse useful information.

<figure><img src="image_102.png" /><figcaption>**Figure 5.1:** The *Dataset* class, organising data and metadata from audio transforms</figcaption></figure>

<figure><img src="image_103.png" /><figcaption>**Figure 5.2:** The *Model* class, encoding and decoding the input data</figcaption></figure>

<footnote>* https://pytorch.org/</footnote>
<figure><img src="image_104.png" /><figcaption>**Figure 5.3:** The Loss class implements different loss functions, providing additional features such as loss history tracking, and simple algebra.</figcaption></figure>

**Algebraic losses.** The Loss object is used for defining the objective criteria of the training. These objects are also defined as models, allowing to embed discriminative criteria such that *DRE* or adversarial regularization directly in the loss object, then lightening the model. In the case of trainable losses, the optimizer is then embedded in the Loss object. Also, as the training criterion is a determinant parameter for the obtained model, we wanted to make manipulation of losses convenient and intuitive. We therefore implemented basic arithmetic operators between loss objects (sum, subtraction, multiplication, division, power), such that an additional criterion to the ELBO object can be easily formulated by ELBO + Discriminator(task), and can then be experimented by the user without to much effort 5.3. The Loss object also provides automatic loss tracking, such that each individual loss can write the values into its memory.

**Overall training process.** The overall process given a *Dataset*, a *Model* and a *Loss* objects is performed by a *Trainer*, that also implements common machine-learning routines to train the given model. The *Trainer* object takes care of processing the usual steps of training such as data loading, loss optimization, model saving, loss tracking, and test / train procedures. It can also regularly plot the state of the model using an intern *Monitor* object, that synthesizes audio files and plots specific to variational learning regularly during training to provides an accurate monitoring of the process. The plot routines are also conceived hierarchically from low-level to high-level functions, to ease the definition of new monitoring methods for specific models.

**Max interface for real-time generation** To experiment the capabilities of the proposed framework, we implemented a real-time interface with the *vsacids* library using the *Max* software*, a audio programming widely used in the computer music community. Max is then used as a sound engine, inverting in real-time the magnitude spectrograms sent by the generative model, and as a user interface, sending to the model the generative coordinates sent by the user (see fig. 5.4). The communication between Max & vsacids is done with the OSC protocol using a dedicated Python server, that interfaces the *Model* instance with Max. As we argued, the lightness of obtained models allows us to interact in real time with the latent space, and can thus be efficiently used as a generative synthesis algorithm. Real-time generation also provides a way to evaluate the properties of extracted spaces directly through *interaction*. Furthermore, the recent coupling between Max and the digital audio workstation Ableton Live † allows to use vsacids directly into a Live work session, and then enter the creative workflow of most musicians. Navigation in the latent space can thus be processed with automation and MIDI controlling, transforming the vsacids models in real synthesizers.

<footnote>* cycling74.com</footnote>

<footnote>† https://www.ableton.com/</footnote>
<figure><img src="image_105.png" /><figcaption>**Figure 5.4:** Schematics of the vsacids real-time environment.</figcaption></figure>

### 5.1.2 Workflow

The creative flow of the vsacids framework then decomposes in three steps: training, exploration, and exploitation. Indeed, as the various parameters and models of the framework are of prime importance for the obtained properties of the representation, we involve the training of the algorithm as part of the framework's practical use. Some machine learning algorithms focused on interaction and usability, such as the Wekinator of Fiebrink & al., opened a new way to investigate the development of this technique under a different perspective [382]. Similarly, the use of variational methods could allow such uses, as it scales well to little datasets and models that train quite fast, even on CPU. We then separate the exploration of the representation, using the real-time interface developed with Max, and the exploitation of the model, based on several techniques that exploits the representation offline.

**Training options.** As the training step is involved in the creative process, the choice of the model in terms of architecture, losses, and of some training criterion parameters are determinant for the interaction with the developed synthesis method. While our library is opened to developers and researchers that may want to investigate the proposed synthesis method, we also have to model a user-compliant procedure for non-expert users. We define these conception steps as follows:

* fix a dataset $\mathcal{D}$, that can be taken in a sample bank or generated using the toy dataset generator

* fix the desired dimensionality of the latent space, and the resolution of the STFT (trading efficiency and sound quality)

* define the desired regularization, and some additional criteria to shape the latent space (semi-supervised learning, conditioning, geometric constraints)

* train the model on a defined number of epochs

This process allows for each user to develop heuristics on the training methods, that he will then validate through experimentation. The input parameters are the most relevant for the final shape of the representation, filtrating the parameters of less importance to avoid the over-parametrization of the tool. Thus, grounded by the theoretical analysis of section 1 and by the benchmark performed section 2, we
can design a set of heuristics to set the training parameters according
to high-level features

► *narrowness*, that defines the capacity of the involved neural networks, trading between *bottleneck* and *sparse* modelling of the encoding / decoding functions,

► *coarseness*, that can trade between explicit (Kullback-Leibler, Renyi or Jensen-Shannon divergences) and implicit regularizations (MMD or adversarial), and tuning the *β* parameter, to implicitly define on the amount of mutual information between examples in the representation

► *influence*, setting the relative weight of the external criteria to the unsupervised loss.

These compliant high-level features can then allow non-expert users
to intuition the underlying of these parameters, and still experiment
with the system without being enforced to enter the details of the
methods.

**Interaction methods.** The interface with Max software then opens the framework to online interaction, such that user can experiment in real-time with the previously trained model. The most straightforward method for latent navigation is to access separately every dimensions of the input space, with a set of sliders that it can perform during play-ing. However, the number of latent dimensions can be high, and then provide an over-parametrized interface that can make its use tedious, not helped by the non-linear behavior of the latent space. Hence, the user can rather navigate on sub-sets of the latent space, whether by dimension selection or with dimensionality reduction techniques such as PCA or ICA. The creative process is then leaded by the choice of these representations, that the user can save with the model.

As blind navigation in this space can be an unsatisfactory, additional visualization of the space can provide a useful feedback for bettering the performance. We can then provide a global representation of the currently explored subspace, annotated with training points to inform the user if he is exploring trained or untrained zone of the representation, and possibly with labelling information, provided its availability. The descriptor plots used previously can also provide precious information, allowing the user to target some zones with desired perceptual properties. If such visual feedback is provided, we can thus provide additional controllers to allow the user to navigate more freely into the space, as for example with a rotation knob that allows to rotate the explored representation. Alternatively, we can also rely on collaborative human-machine interaction techniques to explore the space over its full span, using reinforcement learning methods to reduce the number of parameters to a single reward. This was explored with *ægo*, that is presented below.

**Offline methods.** Finally, another way to use the framework is to
consider it as a offline synthesizer, that can generate some sounds
based on some requests. The user then explores the possibility of the
algorithm by experimenting the several generation methods provided
by the flexibility of the framework. We summarize these possible
offline interactions below :

* **trajectory generation**: using a *trajectory generator* to obtain a multi-dimensional curve in the latent space, and sample the corresponding audio distribution

* **sound morphing**: taking two or more target points, performing an interpolation in the latent space, and generate the corresponding audio distribution.

* **full sound morphing**: is the same than below, but with hierarchi-cal temporal models, then performing "full sound" interpolation

* **cross-synthesis**: encoding and decoding an out-of-domain sound, and listen how it is "filtered" by the model 5.5

* **cross-modal translation**: if the model has a symbolic counterpart (see sec. 3.3), performed transfer between different domains

* **context translation**: if the model is conditioned by external in-formation, regenerate a sound with an different context

* **auto-run**: if a model has a prediction model, let the prediction run over several time spans

* **prediction translation**: performs prediction of a model with a different temporal context (as a different context vector in CPC)

we see that the possibilities of generation with the proposed method
are numerous, and are quite different from other synthesis techniques.
We then hope that this framework could be found inspiring in the com-
puter music community, and maybe can motivate other applications
we did not suppose or being deviated from its original purpose.

<figure><img src="image_106.png" /><figcaption>**Figure 5.5:** An example of cross-synthesis, with a sample of voice passed through a model trained on *toy_additive* dataset. We can see that the voice is somehow "recomposed" by the features learned by the model.</figcaption></figure>

**5.2 A step towards research and creation**
**process : aego**

To experiment the creative framework proposed above we presented *aego*, an improvisational piece with interactive sound and image for one performer. One of our intentions was to explore the representations extracted with AEVB by focusing on the emerging properties of its internal organization, without focusing on the technical aspects of the generating model but rather on the feedback loop created by the performer, the interactive agent, and the synthesis engine.

To represent this idea of exploration, the dimensionality of the navigated projection is increasing through time, making the growing complexity of the space perceptible, such that the audience is directly witnessing the progressive discovery of the space. In addition, we projected the generated sound spectra on stage to provide the audience with a visual representation that accentuate, not disrupt, the sonic perception of the piece, and emphasize the impression of movement.

Another intention was to display the *exploration behavior* of the reinforcement learning agent in front of the audience. To do this, we
wanted to challenge the skills and abilities usually at stake in perfor-
mance, by summoning an ecological approach and evoking a sense
of reciprocal interaction between the human and the machine. In this
sense, rather than using it for control purposes, we used the body of
the performer to convey kinesthetic information about how machine
exploration may be internally experienced by a human. In parallel, we
added raw textual information about the exploration agent's internal
state at top left of the image projection to emphasize the machine's
encoded perception of the performer.

Finally, this piece was created to make a first step towards a research
& creation approach to the development of the proposed framework.
Hence, we aimed to use it in a real-time performance, identifying
first the technical & compositional challenges of using such mod-
els in creative applications, then to propose a musical realization of
this method to an audience, allowing us to get some feedback about
the global æsthetics emerging from the proposed system. While this
"creativity-oriented" part of the evaluation would clearly demand a
more extensive methodology, involving external artists and perform-
ers, and the development of well-defined usability studies, the creation
of this piece was very beneficial to us, and helped us to focus more on
these aspects in our respective works.

5.2.1 Reinforcement Learning for Sonic Exploration

Sonic exploration is a central task in music creation [383]. Specifi-
cally, exploration of digital sound synthesis consists in taking multiple
steps and iterative actions through a large number of technical pa-
rameters to move from an initial idea to a final outcome. Yet, the
mutually-dependent technical functions of parameters, as well as the
exponential number of combinations, often hinder interaction with the
underlying sound space. A way to support exploration of large sound
synthesis spaces can be found in *reinforcement learning*, that defines a
statistical framework for the interaction between a learning agent and
its environment [27]. The agent can learn how to act in its environment
by iteratively receiving some representation of the environment's state
S, taking an action A on it, and receiving a numerical reward R. The
agent's goal, roughly speaking, is to maximize the cumulative amount
of reward that it will receive from its environment.

For our case of sonic exploration, we propose that the musician would
listen to the agent exploring the space, and teach it how to explore
by giving reward data (Fig. 5.6). Formally, the environment's state
is constituted by the numerical values of all synthesis parameters.
The agent's actions are to move one of the parameters up or down
at constant frequency. Finally, the musician communicates *positive or
negative reward* to the agent as a subjective feedback to agent actions.
We implemented a deep reinforcement learning model to support
learning from human reward signal in high-dimensional parametric
spaces [384]. A crucial requirement for reinforcement learning agents
is to *autonomously explore their environment*, to keep on discovering
which actions would yield the most reward. We developed a statistical

<figure><img src="image_107.png" /><figcaption>**Figure 5.6:** Reinforcement learning for sonic exploration. The agent learns which actions to take on a sound synthesis environment based on reward given by the musician. The agent implements an exploration method to foster discovery along interaction.</figcaption></figure>
method, based on intrinsic motivation, which pushes the agent to "explore what surprises it". The resulting interactive learning workflow was found to be useful to relax musicians' control over all synthesis parameters, while also provoking discoveries by exploring uncharted parts of the sound space.

## 5.2.2 Instrument Design

For this piece, we conceived a dedicated musical instrument combining latent-based generative models and reinforcement-based parameter exploration, and leverages their learning capabilities from a design perspective. This instrument is based both on software, interfacing the system presented in the first section and the parameter exploration framework proposed by Scurto & al. [385, 386], and on hardware, using a movement recognition device. The exploration agent then acts as an expressive partner [387], that adapts to the feedback given by the user and navigates in the high-dimensional variational space. Hence, the latent space can be seen as a creative interface, that allows the user to experiment with high the non-linearity of the representation topology.

**Workflow.** The interactive workflow of the instrument is then twofold, as shown in Fig. 5.7. The *setup* phase allows musicians to configure the instrument, create a customized sound dataset for the unsupervised learning model, experiment with various training parameters, or also load a previously-built latent sound space. They can also change dimensionality of the reinforcement learning agent to explore specific dimensions of the latent sound space, as well as the frequency at which it would take actions inside the latent space. During this setup phase, we followed the creative workflow developed in the first section. We thus trained several models on different datasets, varying the architecture, and tested them in our real-time implementation to select which one was the most satisfactory in terms of interaction. Then, the *playing* phase allows musicians to improvise with the agent by means of feedback. The agent produces a continuous layer of sound from the spectrum output of the VAE. Musicians can either cooperate with its learning by giving consistent feedback data to attain a sonic goal. Or, they can obstruct its learning by giving inconsistent feedback data to improvise through sonic exploration.

<figure><img src="image_108.png" /><figcaption>**Figure 5.7:** Interaction diagram of the *aego* instrument. The reinforcement module monitors the reward given by the user and the current latent state to decide a future action *A*, that is used to control the navigation.</figcaption></figure>

## 5.2.3 Engineering

Technically (see Fig. 5.7), the reinforcement learning agent receives a representation of the environment's state *S* as a position in the latent space *z*. Then, it takes an action *A* corresponding to a displacement along some dimension of the latent space. The resulting position has the unsupervised learning model generate a sound spectrum *x*. Based on the sound, the musician would communicate reward *R* to the agent. The latter would progressively learn to explore the latent space in
relation to the musician's feedback data.

**Hardware.** To support embodied musical interaction, we designed a hardware prototype that consists in two velcro rings, each of them equipped with a wireless inertial measurement unit*. We took each unit angular rotation about each forearm axis and summed them to compute a single, normalized numerical reward signal. This, combined with the lightweight, nonintrusive velcro rings, lets musicians experiment with a wide range of gesture vocabulary [388] to communicate positive or negative feedback to the agent.

**Software.** Both machine learning models are implemented in Python, based on *vsacids* and the *coexplorer*† library provided by Scurto & al. We developed a Max/MSP patch to implement a user interface for the setup phase, as well as a hardware data converter for the playing phase. The communication between the synthesis engine, the exploration agent and the audio renderer is leveraged with the OSC protocol.

### 5.2.4 Aesthetics and writing

Our artistic motivation for *ægo* was to open a sensitive reflection on what may actually be learned on a musical level through interaction with machine learning, both by the human and its artificial alter ego—the machine. To share this reflection with members of an audience, we opted for a performance format that displays a human and a machine mutually learning to interact with each other—on an embodied level for the human, and on a computational level for the machine—through live improvisation.

The learning machine possesses a distinctive musical behavior, as well as two latent sound spaces, that are all originally unknown to the human performer. The latter will expressively negotiate control of these spaces with the machine, communicating positive or negative feedback using our instrument and its motion sensors placed in both hands. The slowly-evolving spectromorphologies, synthesized and projected in real-time on stage, create a contemplative, minimalist atmosphere intended to let members of the audience freely consider potential learnings of musical qualities by the human and the machine.

**Composition.** The piece is based on the exploration of two different latent spaces, the first obtained with the *toy_additive* and *toy_-fm datasets, built from synthetic sounds obtained with additive and FM synthesis, and the second with the *acidsInstrument-ordinario* dataset obtained with the sounds of classical instruments [281]. The representations were obtained using simple VAEs, using 3-layered convolutional networks, and have a total amount of 8 dimensions. The

<footnote>* http://ismm.ircam.fr/riot/
† https://github.com/Ircam-RnD/coexplorer</footnote>
<figure><img src="image_109.png" /><figcaption>**Figure 5.8:** Temporal structure composed for the piece. The dimensions are progressively increased and then decreased when the dataset is changed.</figcaption></figure>

performers explore a sub-plane to this latent space whose dimensions are increased through time (1, 2, 4 and 8), whether by the performer or by an external operator. We empirically found that ICA was more satisfactory than PCA (see section 2.2.1), as it progressively opens axis that are orthogonal to the previously explored ones (see fig. 5.8).

Once the full dimensionality of the additive latent space is reached, we change the representation, and reversely progressively shrink the number of projected dimensions to 1. This enables to write form within different soundscapes, allowing the building of a narrative from elementary sinusoidal spectra to richer instrumental timbres. The narration of the piece then distinguish two different phases. An *exploration* phase, that makes perceptible the exploration of an unknown space of increasing complexity, and a *decaying* phase, where the performer is suddenly projected in a richer space, whose complexity is progressively reduced to be fixed in one dimension. The performance is stopped when the performer manages to stay in a fixed point, concluding the performance. The exploration then consists in the improvisational paths taken by the reinforcement learning agent following the performer's feedback data. We set the frequency of agent actions between 30 and 100 milliseconds. This choice allowed for slow, continuous evolution of spectromorphologies, which enables to grasp the behaviour of the agent inside the latent spaces.

**Performance.** While the piece is intended to be improvised, our sole instruction toward the stage performer is that he or she globally performs with the machine with an overall sense of attentiveness*. We propose that the performer would start the piece facing the audience, relaxed, using the instrument with small forearm rotations only. As the piece would unfold over time, the performer would be free to adapt its gestures in response to the slowly evolving complexity of the explored spaces, focusing on embodied interaction with the machine. A second contributor is required to manage the two remaining temporal scales of the piece—i.e., changing dimensionalities, and switching latent spaces.

<footnote>* See the following video recording of a rehearsal: https://youtu.be/gCzOoNChlJQ</footnote>
<figure><img src="image_110.png" /><figcaption>**Figure 5.9:** Pictures taken from the live setup of the piece, showing different moments of the performance.</figcaption></figure>
Conclusion

The current work investigated the use of unsupervised / semi-supervised
methods for audio synthesis and generation using *Auto-Encoding Vari-
ative Bayes*, a hybrid approach based on deep representation learning
and Bayesian inference. The development of these methods thus al-
lowed to jointly address the problem of *high-level feature extraction*,
focusing on the automatic discovery of the underlying structure of
a given dataset, and the problem of *generation*, inverting this repre-
sentation to generate back the data in the audio domain. To this end,
we leveraged expressive probability distribution modelers, based on
neural networks, to model complex dependencies between the audio
domain and the obtained representation, that we trained using black-
box variational inference methods.

Starting from this ground, the work achieved in this PhD first tar-
geted the development of latent regularization strategies specifically
addressed to the audio domain, providing first steps of using meth-
ods derived from AEVB in the audio signal processing domain. First,
we proposed a method to use this latent space as a *translation* space
between audio and symbols, allowing to associate sounds and custom
vocabularies in the latent representation. This regularization method
then allowed us to perform in parallel symbol extraction and con-
strained audio generation, that we experienced by matching sounds
from orchestral instruments with their corresponding pitches and dy-
namics. Then, we proposed a method to topologically enforce the
latent space to reflect high-level audio properties, using dissimilarity
distances obtained from perceptual measurements. This regulariza-
tion method allowed us to propose a new approach of *timbre spaces*,
a fundamental notion of audio perception, allowing the perceptual
inference of sounds coming from instruments unknown to the system,
and audio generation based on high-level properties, opening the way
to *descriptor-based synthesis*.

Consequently, we addressed the temporality of audio signals, propos-
ing to use this latent representation as expressive *state-space models*. We
first modeled the dynamics of these audio signals by performing a
prediction task in the latent space, that is trained jointly with overall
model to intrinsically regularize its organization on temporal con-
straints. We provided three different prediction methods : a method
based on *contrastive predictive coding*, that allows the extraction of *slow
features* from the latent space, a method based on *normalizing flows*,
allowing to model complex latent dynamics grounded on the idea of
stochastic *filtrations*, and finally a method based on *Gaussian processes*,
performing a Bayesian regression task that allows the continuous
modeling of the latent trajectories. After showing the efficiency and
respective properties of these prediction methods, we also proposed
a Bayesian multi-scale architecture, *ShrubVAE*, that performs progres-
sive downsampling of incoming data, then allowing to hierarchically
model the temporal structure of the learned sequence and to generate
a signal up to custom temporal scales. The hierarchical refinement of
the temporal information represented by the latent layers were shown
to not reduce the reconstruction of the signals, opening the way to
fast long-term analysis and generation of audio time series, that is still
missing in the current state of the art.

Finally, we addressed the creative use of the developed methods by
proposing a full audio synthesis framework, allowing off-line and on-
line exploration of the extracted latent spaces, that could be hopefully
used in music production or in real-time performances. Therefore, we
provided first steps for a *research and creation* approach, by develop-
ing this new instrument and by the composition of a new musical
piece, *ægo*, based on multi-dimensional latent space navigation using
a reinforcement-learning exploratory agent. The creation of this piece
allowed us to experience the musical use of these methods and to de-
velop a specific creative workflow, that we implement in an framework
usable for experts and non-expert users. Doing that, we addressed
the *intrinsic* exploration of machine-learning generation methods, fo-
fusing on the structural behavior of the models rather than on their
performance towards an explicit task, and allowing an experimental
approach similar to digital audio synthesis practices.

**Future works and perspectives.** We understand this thesis as a preliminary work of using deep generative models both for high-level feature analysis, extracting representations reflecting underlying properties of the data that could be used for visualization or explicit tasks, and for an intrinsic creative use of these models, discovering their underlying properties to allow a fruitful creative interaction with a human user. We now that most of the works initiated in this PhD could be beneficially detailed and precised, and that the number of possible ameliorations and further investigations are numerous. We will here shortly describe the points that we find the most important, and that we target to develop in the future.

**Phase and raw-waveform learning.** The main issue that prevent the generation of involved models to be entirely satisfactory from a audio quality perspective is the use of magnitude spectra, that require phase reconstruction algorithms during inversion. This problem could be alleviated in two ways : whether also learning the phase of the spectrogram, resorting to complex neural networks or complex distributions, to prevent the use of extensive Griffin-Lim iterations, or directly model the raw waveform of the signal. While both solutions are non-trivial, the second seems to us be the best option, and is one of our main motivation for addressing the temporal aspect of AEVB methods.

**Regularization strategies.** The *regularization* aspect of these models is seminal, as it allows to enforce the latent properties of the model to be organized for a specific analytic or creative purpose. The two
regularization methods we proposed verified the flexibility of Auto-
Encoding Variational Bayes to shape custom constraints, showing the
flexibility of these methods. Hence, the development of additional reg-
ularization constraints addressing different motivations could allow
further creative uses of this model, as for example to model high-level
synthesis macro-parameters of existing synths (a parallel work was
recently proposed by Esling & al., that also involves this work [389]),
custom vocabularies for specific creative uses, explicit axis disentangle-
ment of desired audio properties, and of course detailing the temporal
regularizations that we initiated in this work.

**Temporality.** Due to time constraints, we could not address the mod-
eling of long-time dependencies, that could allow to extend the ap-
proaches described above to significant temporal scopes. Furthermore,
the efficient downsampling performed by the proposed ShrubVAE
could be investigated more intensively, as the obtained generation
results are surprisingly good regarding the temporal scope of the top
latent representation (approximatively between one and two seconds
per latent vector). Even the simple use of ShrubVAE developed in
this thesis multiplies by three times the temporal scope of algorithms
such as WaveNet, while using only two layers and involving neural
networks of reasonable shapes. Also, the developed multi-scale ap-
proach to time series modeling could be used to perform constrained
generation on multiple temporal scales, and thus be able to efficiently
perform analysis / generation based on high-level musical features
(several seconds). With such systems, we can then hope to be able
to generate at the raw waveform level, hopefully in real time, then
preventing to use spectral transforms and allowing us to extract dy-
namical features from audio signals.

**Usability.** Despite being only approached in the last section of this
manuscript, the development of a *research & creation* method, that is
a flourishing dialog between scientific topics and musical creation,
seems to us mandatory as we aim to develop creative tools from the
studied models. Therefore, further developments should also investi-
gate more deeply the usability of the proposed framework, involving
the collaboration with external artists, composers, and performers, as
well as well-defined usability studies that could opening this work
to design and human-computer interaction. Also, the integration of
alternative media (image, video) and *movement* in the process was also
very beneficial, allowing to provide a physicality to the system that
is also mandatory for live performance purposes. Finally, the parallel
development of a framework usable in dedicated software for music
production could also spread this new method onto musical practices,
and could in turn point out specific scientific, design and musical ques-
tions on the use of machine-learning based generation processes for
creativity.
List of publications.

► [1] Jérôme Nika, Ken Déguernel, Axel Chemla–Romeu-Santos, Emmanuel Vincent, Gérard Assayag. *DYCI2 agents: merging the "free", "reactive", and" scenario-based" music generation paradigms*. hal.archives-ouvertes.fr, 2017.

► [2] Philippe Esling, Axel Chemla–Romeu-Santos, Adrien Bitton. *Bridging Audio Analysis, Perception and Synthesis with Perceptually-regularized Variational Timbre Spaces*. ISMIR, 2018.

► [3] Philippe Esling, Axel Chemla–Romeu-Santos, Adrien Bitton. *Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics*. DaFX, 2018.
https://arxiv.org/abs/1805.08501

► [4] Adrien Bitton, Philippe Esling, Axel Chemla–Romeu-Santos. *Modulated Variational auto-Encoders for many-to-many musical timbre transfer*. ISMIR, 2018. https://arxiv.org/abs/1810.00222

► [5] Philippe Esling, Naotaké Masuda, Adrien Bardet, Romeéo Despres, Axel Chemla–Romeu-Santos. *Universal audio synthesizer control with normalizing flows*. ISMIR, 2019. https://arxiv.org/abs/1907.00971

► [6] Axel Chemla–Romeu-Santos, Stavros Ntalampiras, Philippe Esling, Goffredo Haus, Gérard Assayag. *Cross-Modal Variational Inference for Bijective Signal-Symbol Translation*. DAFX, 2019.

► [7] Axel Chemla–Romeu-Santos, Hugo Scurto. *Machine Learning for Computer Music Multidisciplinary Research: A Practical Case Study*. CMMR, 2019.

► [8] Axel Chemla–Romeu-Santos, Hugo Scurto. ægo. CMMR, 2019.
List of acronyms

AE Auto-Encoder 38

AEVB Auto-Encoding Variational Bayes

ALI Adversarially Learned Inference

ANN Artificial Neural Network

AR Auto-Regressive

ARMA Auto-Regressive Moving Average

CAE Contractive Auto-Encoder

CPC Contrastive Predictive Coding

DAE Denoising Auto-Ecoder

DFT Discrete Fourier Transform

DNN Deep Neural Networks

DRE Density Ratio Estimation

DSP Digital Signal Processing

ELBO Evidence Lower-Bound

EM Expectation-Maximization

GAN Generative Adversarial Networks

GP Gaussian Processes

HMM Hidden Markov Models

MA Moving Average

MIR Music Information Retrieval

MLP Multi-Layer Perceptron

MMD Maximum Mean Discrepancy

NCI Noise Constrastive Estimation

NF Normalizing Flows

NN Neural Networks (same as ANN)

OT Optimal Transport

PCA Principal Component Analysis

RNN Recurrent Neural Network

SGD Stochastic Gradient Descent

SSM State-Space Models

STFT Short-Term Fourier Transform

SVD Singluar Value Decomposition

SVI Stochastic Variational Inference

VAE Variational Auto-Encoder

VI Variational Inference
# Figures and tables for baseline results

A

In this appendix we show all the complete results tables for the preliminary results of section 2. The number in parenthesis are obtained with the test set, while the results in regular characters are obtained with train tests.

## A.1 Benchmark reconstruction results.

### A.1.1 Dimensions benchmark

#### A.1.1.1 toy_additive table

<table><thead><tr><th></th><th>- log p(x|z)</th><th>ℓ²</th><th>SC</th><th>ISD</th><th>log-ℓ¹</th></tr></thead><tbody><tr><td><strong>2d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>910.64 (910.10)</td><td>340.71 (340.40)</td><td>92.75 (90.57)</td><td>3.435e-4 (1.27e-3)</td><td>217.21 (216.93)</td></tr><tr><td>PCA</td><td>-</td><td>368.17</td><td>111.68</td><td>nan</td><td>253.27</td></tr><tr><td>ICA</td><td>-</td><td>368.17</td><td>111.68</td><td>nan</td><td>253.27</td></tr><tr><td><strong>4d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>550.13 (551.74)</td><td>49.25 (50.14)</td><td>31.99 (32.02)</td><td>3.78e-4 (1.38e-4)</td><td>85.25 (86.61)</td></tr><tr><td>PCA</td><td>-</td><td>338.72</td><td>117.94</td><td>nan</td><td>251.66</td></tr><tr><td>ICA</td><td>-</td><td>338.72</td><td>117.83</td><td>nan</td><td>251.66</td></tr><tr><td><strong>8d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>510.78 (510.56)</td><td>25.52 (25.45)</td><td>13.60 (14.03)</td><td>3.16e-4 (1.18e-3)</td><td>64.82 (64.46)</td></tr><tr><td>PCA</td><td>-</td><td>289.12</td><td>127.40</td><td>nan</td><td>245.70</td></tr><tr><td>ICA</td><td>-</td><td>289.12</td><td>127.34</td><td>nan</td><td>245.71</td></tr><tr><td><strong>16d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>488.37 (488.37)</td><td>13.33 (13.21)</td><td>11.26 (11.07)</td><td>1.08e-3 (2.92e-4)</td><td>47.79 (48.20)</td></tr><tr><td>PCA</td><td>-</td><td>137.29</td><td>160.42</td><td>nan</td><td>139.35</td></tr><tr><td>ICA</td><td>-</td><td>137.29</td><td>160.42</td><td>nan</td><td>139.35</td></tr></tbody></table>
A.1.1.2 toy_fm table

<table><thead><tr><th></th><th>- log p(x|z)</th><th>ℓ<sup>2</sup></th><th>SC</th><th>ISD</th><th>log-ℓ<sup>1</sup></th></tr></thead><tbody><tr><td><strong>2d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>1816.64 (1820.59)</td><td>1070.21 (1073.57)</td><td>347.40 (349.29)</td><td>3.72e-4 (1.40e-3)</td><td>617.31 (619.38)</td></tr><tr><td>PCA</td><td>-</td><td>1087.64</td><td>469.06955</td><td>nan</td><td>650.80</td></tr><tr><td>ICA</td><td>-</td><td>1087.64</td><td>469.1</td><td>nan</td><td>650.80</td></tr><tr><td><strong>4d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>1996.2 (1989.7)</td><td>1187.8 (1179.9)</td><td>2003.321 (2036.61)</td><td>4.18e-4 (1.57e-3)</td><td>837.52 (833.86)</td></tr><tr><td>PCA</td><td>-</td><td>1039.62</td><td>469.07</td><td>nan</td><td>641.61</td></tr><tr><td>ICA</td><td>-</td><td>1087.64</td><td>469.1</td><td>nan</td><td>650.8</td></tr><tr><td><strong>8d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>1003.5 (994.03)</td><td>427.81 (419.91)</td><td>54.24 (54.40)</td><td>4.32e-4 (1.62e-3)</td><td>289.95 (285.81)</td></tr><tr><td>PCA</td><td>-</td><td>1039.62</td><td>453.68</td><td>nan</td><td>641.61</td></tr><tr><td>ICA</td><td>-</td><td>950.35</td><td>453.7558</td><td>nan</td><td>650.8</td></tr><tr><td><strong>16d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>820.670 (825.75)</td><td>258.13 (262.16)</td><td>15.59 (15.817)</td><td>3.70e-4 (1.37e-3)</td><td>190.71 (193.48)</td></tr><tr><td>PCA</td><td>-</td><td>859.12</td><td>527.28</td><td>nan</td><td>608.13</td></tr><tr><td>ICA</td><td>-</td><td>859.00</td><td>527.74</td><td>nan</td><td>608.13</td></tr></tbody></table>

A.1.1.3 acidsInstruments-ordinario table

<table><thead><tr><th></th><th>- log p(x|z)</th><th>ℓ<sup>2</sup></th><th>SC</th><th>ISD</th><th>log-ℓ<sup>1</sup></th></tr></thead><tbody><tr><td><strong>2d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>334.41 (334.34)</td><td>68.12 (68.12)</td><td>21.72 (21.5)</td><td>8.53e-5 (3.15e-4)</td><td>84.14 (84.03)</td></tr><tr><td>PCA</td><td>-</td><td>104.76</td><td>16.47</td><td>nan</td><td>107.22</td></tr><tr><td>ICA</td><td>-</td><td>104</td><td>16.48</td><td>nan</td><td>107.22</td></tr><tr><td><strong>4d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>286.55 (287.0)</td><td>31.90 (31.93)</td><td>17.45 (18.21)</td><td>8.53e-5 (3.15e-4)</td><td>66.27 (66.17)</td></tr><tr><td>PCA</td><td>-</td><td>92.7</td><td>17.02</td><td>nan</td><td>104.82</td></tr><tr><td>ICA</td><td>-</td><td>92.7</td><td>17.03</td><td>nan</td><td>104.82</td></tr><tr><td><strong>8d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>270.1 (269.9)</td><td>20.23 (20.11)</td><td>12.10 (12.28)</td><td>8.30e-5 (3.06e-4)</td><td>51.56 (51.28)</td></tr><tr><td>PCA</td><td>-</td><td>71.54</td><td>16.79</td><td>nan</td><td>98.40</td></tr><tr><td>ICA</td><td>-</td><td>71.54</td><td>16.79</td><td>nan</td><td>98.40</td></tr><tr><td><strong>16d</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>276.56 (276.69)</td><td>23.49 (23.67)</td><td>22.96 (20.13)</td><td>1.31e-3 (4.79e-3)</td><td>56.74 (56.72)</td></tr><tr><td>PCA</td><td>-</td><td>48.65</td><td>15.16</td><td>nan</td><td>219.70</td></tr><tr><td>ICA</td><td>-</td><td>48.65</td><td>15.16</td><td>nan</td><td>85.34</td></tr></tbody></table>
A.1.2 Divergences benchmark

<table><thead><tr><th></th><th>-log p(x|z)</th><th>ℓ²</th><th>SC</th><th>ISD</th><th>log-ℓ¹</th></tr></thead><tbody><tr><td>toy_additive_mini</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>D<sub>KL</sub></td><td>488.37 (488.37)</td><td>13.33 (13.21)</td><td>11.26 (11.07)</td><td>1.08e-3 (2.92e-4)</td><td>47.79 (48.20)</td></tr><tr><td>MMD</td><td><b>474.46 (474.75)</b></td><td><b>5.90 (6.06)</b></td><td><b>6.78 (6.67)</b></td><td>3.12e-4 (1.16e-3)</td><td><b>33.66 (34.39)</b></td></tr><tr><td>D<sub>α=2</sub></td><td>487.89 (487.84)</td><td>13.035 (13.01)</td><td>9.03 (9.03)</td><td>2.95e-4 (1.09e-3)</td><td>46.75 (46.72)</td></tr><tr><td>JSD</td><td>519.41 (519.1379)</td><td>28.07 (24.36)</td><td>24.34 (7.60)</td><td>3.22e-4 (1.20e-3)</td><td>70.70 (70.36)</td></tr><tr><td>Adv</td><td>485.78 (485.88)</td><td>11.69(10.18)</td><td>10.25(10.0)</td><td><b>2.91e-4 (1.075e-3)</b></td><td>46.44 (46.49)</td></tr><tr><td>PCA</td><td>-</td><td>137.29</td><td>160.42</td><td>nan</td><td>139.35</td></tr><tr><td>ICA</td><td>-</td><td>137.29</td><td>160.42</td><td>nan</td><td>139.35</td></tr><tr><td>toy_fm</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>D<sub>KL</sub></td><td>910.64 (910.10)</td><td>340.71 (340.40)</td><td>15.59 (15.81)</td><td>3.435e-4 (1.27e-3)</td><td><b>190.71 (193.48)</b></td></tr><tr><td>MMD</td><td><b>771.6 (766.8)</b></td><td><b>225.08 (221.78)</b></td><td>62.14 (62.19)</td><td>4.00e-4 (1.50e-3)</td><td>205.46 (202.95)</td></tr><tr><td>D<sub>α=2</sub></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>JSD</td><td>792.52 (259.07)</td><td>244.99 (242.90)</td><td>27.83 (27.60)</td><td><b>8.28e-05 (3.06e-4)</b></td><td>199.34 (0.63)</td></tr><tr><td>Adv</td><td>1273.90 (789.56)</td><td>524.03 (525.16)</td><td><b>6.28 (6.46)</b></td><td>4.24e-4 (1.59e-3)</td><td>253.13 (197.16)</td></tr><tr><td>PCA</td><td>-</td><td>859.12</td><td>527.28</td><td>nan</td><td>608.13</td></tr><tr><td>ICA</td><td>-</td><td>859.95</td><td>527.28</td><td>nan</td><td>608.13</td></tr><tr><td>aIns-o</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>D<sub>KL</sub></td><td>276.56 (276.69)</td><td>23.49 (23.67)</td><td>22.96 (20.13)</td><td>1.31e-3 (4.79e-3)</td><td>56.74 (56.72)</td></tr><tr><td>MMD</td><td>248.19 (248.19)</td><td>8.43 (8.43)</td><td>5.25 (5.31)</td><td>8.25e-05 (3.04e-4)</td><td>35.37 (35.26)</td></tr><tr><td>D<sub>α=2</sub></td><td>255.66 (255.74)</td><td>12.33 (12.38)</td><td>10.23 (10.53)</td><td>8.27e-05 (3.05e-4)</td><td>43.39 (43.40)</td></tr><tr><td>JSD</td><td>259.36 (259.07)</td><td>7.75 (14.22)</td><td>7.74 (7.60)</td><td>8.28e-05 (3.06e-4)</td><td>43.19 (43.07)</td></tr><tr><td>Adv</td><td><b>246.8 (246.7)</b></td><td>7.76 (7.71)</td><td>9.97 (10.0)</td><td>8.26e-5 (000)</td><td>35.83 (35.83)</td></tr><tr><td>PCA</td><td>-</td><td>48.65</td><td>15.16</td><td>nan</td><td>219.70</td></tr><tr><td>ICA</td><td>-</td><td>48.65</td><td>15.16</td><td>nan</td><td>85.34</td></tr></tbody></table>
### A.1.3 β benchmarks

<table><thead><tr><th></th><th>-log p(x|z)</th><th>ℓ<sup>2</sup></th><th>SC</th><th>ISD</th><th>log-ℓ<sup>1</sup></th></tr></thead><tbody><tr><td><strong>toy_additive</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>β = 1</td><td><strong>276.56 (276.69)</strong></td><td><strong>23.49 (23.67)</strong></td><td>22.96 (20.13)</td><td>1.31e-3 (4.79e-3)</td><td>56.74 (56.72)</td></tr><tr><td>β = 4</td><td>518.45 (518.12)</td><td>31.57 (31.43)</td><td><strong>19.33 (19.46)</strong></td><td>3.11e-4 (1.15e-3)</td><td>68.58 (68.03)</td></tr><tr><td>β = 10</td><td>554.01 (553.48)</td><td>58.18 (57.91)</td><td>37.34 (37.85)</td><td>3.09e-4 (1.14e-3)</td><td>89.51 (88.79)</td></tr><tr><td>PCA</td><td>-</td><td>137.29</td><td>160.42</td><td>nan</td><td>139.35</td></tr><tr><td>ICA</td><td>-</td><td>137.29</td><td>160.42</td><td>nan</td><td>139.35</td></tr><tr><td><strong>toy_fm</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>β = 1</td><td><strong>820.670 (825.75)</strong></td><td><strong>258.13 (262.16)</strong></td><td><strong>15.59 (15.817)</strong></td><td><strong>3.70e-4 (1.37e-3)</strong></td><td><strong>190.71 (193.48)</strong></td></tr><tr><td>β = 4</td><td>901.00 (897.3193)</td><td>319.03 (316.24)</td><td>17.094276 (15.61)</td><td>3.96e-4 (1.47e-3)</td><td>211.48 (208.22)</td></tr><tr><td>β = 10</td><td>1098.0 (1099.0)</td><td>451.97 (453.92)</td><td>16.15 (15.97)</td><td>4.24e-4 (1.59e-3)</td><td>248.34 (250.49)</td></tr><tr><td>PCA</td><td>-</td><td>859.12</td><td>527.28</td><td>nan</td><td>608.13</td></tr><tr><td>ICA</td><td>-</td><td>859.95</td><td>527.28</td><td>nan</td><td>608.13</td></tr><tr><td><strong>aIns-0</strong></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>β = 1</td><td>261.37 (261.37)</td><td>15.24 (15.34)</td><td>14.384189 (13.74)</td><td>8.294699e-5 (3.06e-4)</td><td>56.74 (47.81)</td></tr><tr><td>β = 4</td><td>279.98 (279.83688)</td><td>27.82 (27.77)</td><td>17.26 (16.56)</td><td>8.30e-05 (3.07e-4)</td><td>56.96 (56.77)</td></tr><tr><td>β = 10</td><td>303.88 (304.09)</td><td>45.06 (45.21)</td><td>41.46 (44.54)</td><td>8.3674e-05 (3.09e-04)</td><td>72.98 (73.11)</td></tr><tr><td>PCA</td><td>-</td><td>859.12</td><td>527.28</td><td>nan</td><td>608.13</td></tr><tr><td>ICA</td><td>-</td><td>859.12</td><td>527.28</td><td>nan</td><td>608.13</td></tr></tbody></table>

## A.2 Regularization benchmarks.

### A.2.1 Dimensions benchmark

#### A.2.1.1 toy_additive

<table><thead><tr><th></th><th>D<sub>KL</sub>[q||p]</th><th>D<sub>KL</sub>[p||q]</th><th>D<sub>2</sub><sup>α</sup>[q||p]</th><th>JSD<sub>0.5</sub>[p||q]</th><th>MMD[p, q]</th><th>TC[p, q]</th></tr></thead><tbody><tr><td><strong>2d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>6.78 (6.78)</td><td>745.24 (744.37)</td><td>7.80 (7.80)</td><td>184.83 (184.61)</td><td>5287.61 (1338.22)</td><td>8.29 (7.56)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>48961</td><td>32.85</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>20020</td><td>1.65e-4</td></tr><tr><td><strong>4d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>13.87 (13.89)</td><td>44620.4 (4449.24)</td><td>15.87 (15.89)</td><td>179.54 (1109.28)</td><td>21510 (125.44)</td><td>11.72 (10.78)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>22944</td><td>56.711</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>4587.23</td><td>1.0826e-4</td></tr><tr><td><strong>8d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>20.64 (20.64)</td><td>1658.67 (1647.13)</td><td>24.63 (24.62)</td><td>410.29 (407.40)</td><td>840.89 (212.56)</td><td>19.00 (18.83)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>141890</td><td>118.63</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>13725</td><td>2.66e-4</td></tr><tr><td><strong>16d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>35.15 (35.20)</td><td>438.08 (440.41)</td><td>49.45 (49.50)</td><td>103.48 (104.05)</td><td>673.49 (159.52)</td><td>22.75 (22.89)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>759.39</td><td>400.13</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>34.193</td><td>4e-2</td></tr></tbody></table>
A.2.1.2 toy_fm

<table><thead><tr><th></th><th>D<sub>KL</sub>[q||p]</th><th>D<sub>KL</sub>[p||q]</th><th>D<sub>2</sub><sup>a</sup>[q||p]</th><th>JSD<sub>0.5</sub>[p||q]</th><th>MMD[p, q]</th><th>TC[p, q]</th></tr></thead><tbody><tr><td><strong>2d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>6.31 (6.31)</td><td>612.88 (629.59)</td><td>7.32 (7.32)</td><td>151.85 (156.02)</td><td>3648.16 (869.68)</td><td>29.82 (23.33)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>12620</td><td>19.62</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>4209.7</td><td>1.04e-3</td></tr><tr><td><strong>4d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>15.47 (15.46)</td><td>11962 (11883)</td><td>17.47 (17.46)</td><td>2987 (2967)</td><td>2987 (1470)</td><td>30.53 (31.69)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>43748</td><td>158.63</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>2871.0</td><td>1.65e-3</td></tr><tr><td><strong>8d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>28.44 (28.46)</td><td>9854.3 (9735.94)</td><td>32.44 (32.46)</td><td>2457.2 (2427.7)</td><td>5010.1 (1252.14)</td><td>40.31 (43.71)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>102061</td><td>86.44</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>15727</td><td>3.22e-3</td></tr><tr><td><strong>16d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>48.98 (49.20)</td><td>8632 (8750.4)</td><td>56.98 (57.20)</td><td>2147.54 (2176)</td><td>10081 (2406)</td><td>71.43 (65.32)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>207162</td><td>406.33</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>7628</td><td>1.76e-3</td></tr></tbody></table>

A.2.1.3 acidsInstruments-ordinario

<table><thead><tr><th></th><th>D<sub>KL</sub>[q||p]</th><th>D<sub>KL</sub>[p||q]</th><th>D<sub>2</sub><sup>a</sup>[q||p]</th><th>JSD<sub>0.5</sub>[p||q]</th><th>MMD[p, q]</th><th>TC[p, q]</th></tr></thead><tbody><tr><td><strong>2d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>5.97 (5.97)</td><td>1042 (1042)</td><td>6.97 (6.98)</td><td>259.2 (259.2)</td><td>16054 (4025)</td><td>25.68 (24.46)</td></tr><tr><td>PCA</td><td>-</td><td>104.76</td><td>16.47</td><td>nan</td><td>nan</td><td>107.42</td></tr><tr><td>ICA</td><td>-</td><td>104</td><td>16.48</td><td>nan</td><td>nan</td><td>107.42</td></tr><tr><td><strong>4d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>-694.1 (9.84)</td><td>726.39 (000)</td><td>11.83 (11.81)</td><td>179.54 (180.04)</td><td>110.60 (5549)</td><td>57.23 (56.79)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>nan</td><td>161016</td><td>148.2</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>65115</td><td>6.12e-5</td></tr><tr><td><strong>8d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>13.88 (13.9)</td><td>353.27 (351.9)</td><td>17.734 (17.72)</td><td>85.62 (85.29)</td><td>26389 (6529)</td><td>81.51 (84.46)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>nan</td><td>188622</td><td>148.2</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>48852</td><td>1e-4</td></tr><tr><td><strong>16d</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>VAE</td><td>-694.1 (000)</td><td>0.093 (000)</td><td>-416.6 (000)</td><td>0.177 (000)</td><td>0.177 (000)</td><td>0.177 (000)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>39409</td><td>204.82</td></tr><tr><td>ICA</td><td>-</td><td>48.68</td><td>15.19</td><td>-</td><td>7881</td><td>3.9e-4</td></tr></tbody></table>
## A.2.2 Divergence regularization results

<table><thead><tr><th></th><th>D<sub>KL</sub>[q||p]</th><th>D<sub>KL</sub>[p||q]</th><th>D<sub>2</sub><sup>2</sup>[q||p]</th><th>JSD<sub>0.5</sub>[p||q]</th><th>MMD[p, q]</th><th>TC[p, q]</th></tr></thead><tbody><tr><td>toy_additive</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>D<sub>KL</sub></td><td>35.15 (35.20)</td><td>438.08 (440.41)</td><td>49.45 (49.50)</td><td>103.48 (104.05)</td><td>673.49 (159.52)</td><td>22.75 (22.89)</td></tr><tr><td>MMD</td><td>279.43 (279.86)</td><td>22.49e11 (86.10e9)</td><td>295.43 (295.86)</td><td>16.46e9 (521.5e9)</td><td>729.60 (588.83)</td><td><b>20.13 (22.67)</b></td></tr><tr><td>D<sub>α=2</sub></td><td>35.40(35.42)</td><td>456.00 (458.25)</td><td>49.77 (49.78)</td><td>107.87 (108.42)</td><td>689.97 (174.62)</td><td>21.82 (22.25)</td></tr><tr><td>JSD</td><td>341.51 (340.71)</td><td>13.67e6 (13.22e6)</td><td>357.52 (356.72)</td><td>3418281 (3.30e6)</td><td>43200 (3.30e6)</td><td>206.66 (205.71)</td></tr><tr><td>Adv</td><td>947.45 (948.73)</td><td>3.70e32 (3.53e32)</td><td>955.44 (956.72)</td><td>9.24e31 (8.81e31)</td><td>470097 (117231)</td><td>213.88 (211.75)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>759.39</td><td>400.13</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>34.193359375</td><td>4e-2</td></tr><tr><td>toy FM</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>D<sub>KL</sub></td><td>48.98 (49.20)</td><td>8632 (8750.4)</td><td>56.98 (57.20)</td><td>2147.54 (2176)</td><td>10081 (2406)</td><td>71.43 (65.32)</td></tr><tr><td>MMD</td><td>359.16 (359.16)</td><td>3.91e28 (3.87e28)</td><td>367.16 (295.86)</td><td>9.77e27 (9.67e27)</td><td><b>1659.7 (405.44)</b></td><td><b>36.33 (35.98)</b></td></tr><tr><td>D<sub>α=2</sub></td><td>210.4 (210.86)</td><td>3.31e16 (3.40e16)</td><td>218.40 (218.86)</td><td>8.26e15 (8.51e15)</td><td>63233 (15523)</td><td>157.91 (155.37)</td></tr><tr><td>JSD</td><td>153.88 (153.00)</td><td><b>277.95 (276.26)</b></td><td>243.02 (241.64)</td><td>61.25 (60.88)</td><td>178043 (43708)</td><td>200.24 (197.73)</td></tr><tr><td>Adv</td><td>2178.50 (2179.06)</td><td>4.55e29 (4.51e29)</td><td>2186.50 (2187.07)</td><td>1.14e29 (1.13e+29)</td><td>585633 (143165)</td><td>411.74 (409.4)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>207162</td><td>406.33</td></tr><tr><td>ICA</td><td>-</td><td>48.68</td><td>15.19</td><td>-</td><td>7628</td><td>1.76e-3</td></tr><tr><td>acidsInstruments-o</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>D<sub>KL</sub></td><td>18.69 (18.82)</td><td>247.97(252.12)</td><td>25.65 (25.80)</td><td>58.63 (59.63)</td><td>155.60 (35.72)</td><td>25.35 (23.36)</td></tr><tr><td>MMD</td><td>141.91 (141.70)</td><td>22.49e11 (26.39e12)</td><td>149.92 (149.70)</td><td>5.622e11 (65.98e12)</td><td>2399.50 (588.83)</td><td>20.13 (19.35)</td></tr><tr><td>D<sub>α=2</sub></td><td>18.12 (18.08)</td><td>398.89 (399.56)</td><td>24.41 (24.35)</td><td>96.35 (96.52)</td><td>4124.59 (1012.24)</td><td>27.83 (28.28)</td></tr><tr><td>JSD</td><td>32.92 (32.84)</td><td>49.64 (49.53)</td><td>57.69 (57.55)</td><td>11.144 (11.12)</td><td>105911 (26344)</td><td>130.77 (130.11)</td></tr><tr><td>Adv</td><td>1302.11 (1303.22)</td><td>1e+31 (1.75+31)</td><td>1310.11 (1311.22)</td><td>4.7e30 (4.37e30)</td><td>1.88e6 (000)</td><td>542.7 (535)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>39409</td><td>204.82</td></tr><tr><td>ICA</td><td>-</td><td>48.68</td><td>15.19</td><td>-</td><td>7881</td><td>3.9e-4</td></tr></tbody></table>

## A.2.3 β-benchmarks.

<table><thead><tr><th></th><th>D<sub>KL</sub>[q||p]</th><th>D<sub>KL</sub>[p||q]</th><th>D<sub>2</sub><sup>2</sup>[q||p]</th><th>JSD<sub>0.5</sub>[p||q]</th><th>MMD[p, q]</th><th>TC[p, q]</th></tr></thead><tbody><tr><td>toy_additive</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>β = 1</td><td>35.15 (35.20)</td><td>438.08 (440.41)</td><td>49.45 (49.50)</td><td>103.48 (104.05)</td><td>673.49 (159.52)</td><td>22.75 (22.89)</td></tr><tr><td>β = 4</td><td>18.30 (18.28)</td><td>155.44 (155.22)</td><td>25.72 (25.71)</td><td>35.75 (35.70)</td><td>605.64(143.10)</td><td>20.34 (19.83)</td></tr><tr><td>β = 10</td><td>12.48 (12.47)</td><td>60.37 (60.04)</td><td>19.30 (19.29)</td><td>13.30 (13.22)</td><td>550.72 (140.6)</td><td>19.93 (20.10)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>759.39</td><td>202.13</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>8875.41</td><td>6.77e-4</td></tr><tr><td>toy FM</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>β = 1</td><td>34.94 (49.20)</td><td>8632 (8750)</td><td>56.97 (57.20)</td><td>2147 (2176)</td><td>10081 (2406)</td><td><b>71.43 (65.33)</b></td></tr><tr><td>β = 4</td><td>34.94 (34.77)</td><td>1791.2 (1736.4)</td><td>43.06 (42.9)</td><td>440.84 (427.29)</td><td>10163 (2534)</td><td>85.73 (89.76)</td></tr><tr><td>β = 10</td><td><b>24.86 (24.87)</b></td><td><b>500.77 (504.15)</b></td><td><b>33.09 (33.09)</b></td><td><b>121.5 (121.8)</b></td><td><b>10053 (2419.8)</b></td><td>108.52 (113.10)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>205598</td><td>405.70</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>7594.93</td><td>6.10e-4</td></tr><tr><td>acidsInstruments-o</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>D<sub>KL</sub></td><td>19.19 (19.28)</td><td>282.62 (284.09)</td><td>26.29 (26.39)</td><td>67.20 (67.54)</td><td>11557 (2840.25)</td><td>51.95 (50.27)</td></tr><tr><td>β = 4</td><td>9.36 (9.36)</td><td>57.24 (57.50)</td><td>14.67 (14.66)</td><td>13.00 (12.98)</td><td>10473 (2527.10)</td><td>51.63 (50.80)</td></tr><tr><td>β = 10</td><td>5.424(5.423)</td><td>26.63 (26.67)</td><td>8.78 (8.78)</td><td>5.90 (5.912)</td><td>7539.1 (1807.7)</td><td>45.781 (45.52)</td></tr><tr><td>PCA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>39409</td><td>204.82</td></tr><tr><td>ICA</td><td>-</td><td>-</td><td>-</td><td>-</td><td>7881.27</td><td>3.92e-4</td></tr></tbody></table>
## A.3 Disentangling.

### A.3.1 toy_additive

<table><thead><tr><th></th><th>f_0</th><th>random</th><th>n_partials</th><th>random</th><th>decay</th><th>random</th></tr></thead><tbody><tr><td>2d</td><td>4.90</td><td>4.97</td><td>2.29</td><td>2.26</td><td>1.75</td><td>1.75</td></tr><tr><td>4d</td><td>4.79</td><td>4.87</td><td>2.10</td><td>2.15</td><td><strong>1.69</strong></td><td>1.68</td></tr><tr><td>8d</td><td><strong>4.75</strong></td><td>4.81</td><td><strong>2.011</strong></td><td>2.09</td><td>1.70</td><td>1.71</td></tr><tr><td>D<sub>KL</sub></td><td>4.97</td><td>4.79</td><td>2.22</td><td>2.12</td><td>1.73</td><td><strong>1.72</strong></td></tr><tr><td>MMD</td><td><strong>4.77</strong></td><td>4.82</td><td>2.14</td><td>2.13</td><td>2.50</td><td>2.63</td></tr><tr><td>D<sub>α=2</sub></td><td>4.79</td><td>4.78</td><td>2.17</td><td><strong>2.12</strong></td><td><strong>1.72</strong></td><td><strong>1.72</strong></td></tr><tr><td>JSD</td><td>6.99</td><td>7.13</td><td>3.63</td><td>3.49</td><td>2.91</td><td>2.91</td></tr><tr><td>Adv</td><td>12.02</td><td>7.52</td><td>6.97</td><td>6.38</td><td>5.31</td><td>5.32</td></tr><tr><td>β = 4</td><td>4.86</td><td>4.78</td><td>2.10</td><td>2.09</td><td>1.72</td><td>1.70</td></tr><tr><td>β = 10</td><td>4.73</td><td>4.78</td><td>2.06</td><td>2.09</td><td>1.69</td><td>1.70</td></tr><tr><td>PCA</td><td>5.55</td><td>7.13</td><td>3.94</td><td>4.02</td><td>2.51</td><td>2.51</td></tr><tr><td>ICA</td><td>4.61</td><td>4.61</td><td>1.95</td><td>1.95</td><td>1.61</td><td>1.61</td></tr></tbody></table>

### A.3.2 toy_fm

<table><thead><tr><th></th><th>f_carrier</th><th>random</th><th>f_multiplier</th><th>random</th><th>fm_ratio</th><th>random</th></tr></thead><tbody><tr><td>2d</td><td>3.54</td><td>3.54</td><td>3.93</td><td>3.90</td><td>3.44</td><td>3.48</td></tr><tr><td>4d</td><td>3.31</td><td>3.31</td><td>3.60</td><td>3.63</td><td>3.23</td><td>3.25</td></tr><tr><td>8d</td><td>3.49</td><td>3.49</td><td>3.77</td><td>3.76</td><td>3.42</td><td>3.46</td></tr><tr><td>D<sub>KL</sub></td><td>3.58</td><td>3.55</td><td>3.82</td><td>3.85</td><td>3.54</td><td>3.58</td></tr><tr><td>MMD</td><td>3.46</td><td>3.43</td><td>3.78</td><td>3.71</td><td>3.29</td><td>3.37</td></tr><tr><td>D<sub>α=2</sub></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>JSD</td><td>5.68</td><td>5.60</td><td>5.68</td><td>6.00</td><td>5.97</td><td>5.50</td></tr><tr><td>Adv</td><td>14.32</td><td>14.34</td><td>16.12</td><td>16.40</td><td>14.44</td><td>15.49</td></tr><tr><td>β = 4</td><td>3.62</td><td>3.52</td><td>3.82</td><td>3.82</td><td>3.33</td><td>3.46</td></tr><tr><td>β = 10</td><td>3.43</td><td>3.48</td><td>3.90</td><td>3.81</td><td>3.61</td><td>3.48</td></tr><tr><td>PCA</td><td>6.40</td><td>6.04</td><td>5.81</td><td>7.02</td><td>4.84</td><td>6.34</td></tr><tr><td>ICA</td><td><strong>3.22</strong></td><td>3.22</td><td><strong>3.54</strong></td><td>3.53</td><td><strong>3.17</strong></td><td>3.18</td></tr></tbody></table>
A.3.3 acidsInstruments-ordinario

<table><thead><tr><th></th><th>octave</th><th>random</th><th>pitch</th><th>random</th><th>dynamics</th><th>random</th><th>instrument</th><th>random</th></tr></thead><tbody><tr><td>2d</td><td>2.52</td><td>2.46</td><td>2.72</td><td>2.74</td><td>1.73</td><td>1.69</td><td>2.74</td><td>2.72</td></tr><tr><td>4d</td><td>2.55</td><td>2.42</td><td>2.41</td><td>2.67</td><td>1.77</td><td>1.98</td><td>2.79</td><td>2.76</td></tr><tr><td>8d</td><td>2.51</td><td>2.47</td><td>2.60</td><td>2.65</td><td>1.70</td><td>1.66</td><td>2.75</td><td>2.65</td></tr><tr><td>D<sub>KL</sub></td><td>2.77</td><td>2.47</td><td>2.76</td><td>2.66</td><td>1.70</td><td>1.72</td><td>2.46</td><td>2.69</td></tr><tr><td>MMD</td><td>2.4</td><td>2.35</td><td>2.50</td><td>2.64</td><td>2.50</td><td>2.63</td><td>1.68</td><td>1.75</td></tr><tr><td>D<sub>α=2</sub></td><td>2.27</td><td>2.33</td><td>2.61</td><td>2.60</td><td>1.74</td><td>1.72</td><td>2.73</td><td>2.61</td></tr><tr><td>JSD</td><td>2.733</td><td>2.6653</td><td>3.02</td><td>2.97</td><td>1.97</td><td>2.06</td><td>3.12</td><td>3.07</td></tr><tr><td>Adv</td><td>6.94</td><td>7.52</td><td>10.112</td><td>10.03</td><td>6.08</td><td>7.42</td><td>8.74</td><td>8.25</td></tr><tr><td>β = 4</td><td>2.64</td><td>2.45</td><td>2.75</td><td>2.62</td><td>1.80</td><td>1.73</td><td>2.38</td><td>2.61</td></tr><tr><td>β = 10</td><td>2.33</td><td>2.33</td><td>2.65</td><td>2.58</td><td>1.65</td><td>1.68</td><td>2.78</td><td>2.57</td></tr><tr><td>PCA</td><td>3.00</td><td>3.04</td><td>3.35</td><td>3.39</td><td>2.04</td><td>2.04</td><td>3.27</td><td>3.19</td></tr><tr><td>ICA</td><td>2.41</td><td>2.265</td><td>2.53</td><td>2.53</td><td>2.53</td><td>1.65</td><td>1.61</td><td>2.53</td></tr></tbody></table>
<figure><img src="image_111.png" /></figure>
*What doth life ?*

- Xavier Renegade Angel
<figure><img src="image_112.png" /></figure>
Bibliography

[1] Max V Mathews. 'The digital computer as a musical instrument'. In: *Science* 142.3592 (1963), pp. 553–557 (cited on page 2).

[2] Jean-Claude Risset and David L Wessel. 'Exploration of timbre by analysis and synthesis'. In: *The psychology of music* 2 (1982), p. 151 (cited on pages 2, 88).

[3] John M Chowning. 'The synthesis of complex audio spectra by means of frequency modulation'. In: *Journal of the audio engineering society* 21.7 (1973), pp. 526–534 (cited on pages 2, 58).

[4] Noam Chomsky. *Aspects of the Theory of Syntax*. Tech. rep. MASSACHUSETTS INST OF TECH CAMBRIDGE RESEARCH LAB OF ELECTRONICS, 1964 (cited on page 2).

[5] Richard Ohmann. 'Generative grammars and the concept of literary style'. In: *Word* 20.3 (1964), pp. 423–439 (cited on page 2).

[6] Grzegorz Rozenberg and Arto Salomaa. *The mathematical theory of L systems*. Vol. 90. Academic press, 1980 (cited on page 2).

[7] Robert M Haralick, Stanley R Sternberg, and Xinhua Zhuang. 'Image analysis using mathematical morphology'. In: *IEEE transactions on pattern analysis and machine intelligence* 4 (1987), pp. 532–550 (cited on page 2).

[8] Martin Norgaard. 'How Jazz Musicians Improvise'. In: *Music Perception: An Interdisciplinary Journal* 31.3 (2014), pp. 271–287. DOI: 10.1525/mp.2014.31.3.271 (cited on page 2).

[9] Mark J Steedman. 'A generative grammar for jazz chord sequences'. In: *Music Perception: An Interdisciplinary Journal* 2.1 (1984), pp. 52–77 (cited on page 2).

[10] E.R. Miranda. 'Evolving cellular automata music: From sound synthesis to composition'. In: *Proceedings of the Workshop on Artificial Life Models for Musical Applications* (2001), p. 12 (cited on page 2).

[11] Dave Burraston et al. 'Cellular Automata in MIDI based Computer Music'. In: *Proceedings of the International Computer Music Conference* 4 (2004), pp. 71–78 (cited on page 2).

[12] Fleischer Manufacturing. '( 12 ) United States Patent'. In: 1.12 (2002) (cited on page 2).
[13] Adriano Baratè, Goffredo Haus, and Luca A Ludovico. 'Music analysis and modeling through Petri nets'. In: *International Symposium on Computer Music Modeling and Retrieval*. Springer. 2005, pp. 201–218 (cited on page 2).

[14] C. E. Shannon and W. Weaver. 'The Mathematical Theory of Communication'. In: *The mathematical theory of communication* 27.4 (1949), p. 117. DOI: 10.2307/3611062 (cited on pages 2, 23).

[15] Luciano Floridi. 'What is the Philosophy of Information ?' In: (1997) (cited on page 3).

[16] Antti Honkela and Harri Valpola. 'Variational learning and bits-back coding: an information-theoretic view to Bayesian learning'. In: *IEEE Transactions on Neural Networks* 15.4 (2004), pp. 800–810 (cited on page 3).

[17] Jie Cheng et al. 'Learning Bayesian networks from data: an information-theory based approach'. In: *Artificial intelligence* 137.1-2 (2002), pp. 43–90 (cited on page 3).

[18] Jürgen Schmidhuber. 'Formal theory of creativity, fun, and intrinsic motivation (1990–2010)'. In: *IEEE Transactions on Autonomous Mental Development* 2.3 (2010), pp. 230–247 (cited on pages 3, 4).

[19] Peter D. Grünwald. *The Minimum Description Length Principle*. 2007, p. 736 (cited on pages 3, 27).

[20] Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. 'Testing the manifold hypothesis'. In: *Journal of the American Mathematical Society* 29.4 (2016), pp. 983–1049. DOI: 10.1090/jams/852 (cited on pages 3, 29).

[21] Yoshua Bengio, Aaron Courville, and Pascal Vincent. 'Representation Learning: A Review and New Perspectives'. In: 1993 (2012), pp. 1–30. DOI: 10.1145/1756006.1756025 (cited on page 4).

[22] Diederik P Kingma and Max Welling. 'Auto-encoding variational bayes'. In: *arXiv preprint arXiv:1312.6114* (2013) (cited on pages 4, 42, 43, 47, 127).

[23] Ian Goodfellow et al. 'Generative adversarial nets'. In: *Advances in neural information processing systems*. 2014, pp. 2672–2680 (cited on pages 4, 59).

[24] Diederik P. Kingma and Prafulla Dhariwal. 'Glow: Generative Flow with Invertible 1x1 Convolutions'. In: 2 (2018), pp. 1–10 (cited on page 4).

[25] Margaret A Boden. 'of Creativity'. In: (2009), pp. 23–34 (cited on page 4).

[26] Philippe Pasquier et al. 'An introduction to musical metacreation'. In: *Computers in Entertainment (CIE)* 14.2 (2016), p. 2 (cited on page 4).

[27] Richard S Sutton and Andrew G Barto. *Reinforcement learning: An introduction*. MIT press, 2018 (cited on pages 4, 138).
[28] Simon Colton, John Charnley, and Alison Pease. 'Computational Creativity Theory: The FACE and IDEA Descriptive Models'. In: (1992), pp. 90–95 (cited on page 4).

[29] Jesse Engel et al. 'Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders'. In: (2017) (cited on page 5).

[30] Adam Roberts et al. 'A hierarchical latent vector model for learning long-term structure in music'. In: *arXiv preprint arXiv:1803.05428* (2018) (cited on page 5).

[31] Diederik P. Kingma et al. 'Semi-Supervised Learning with Deep Generative Models'. In: (2014), pp. 1–9 (cited on page 9).

[32] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 'Stochastic Backpropagation and Approximate Inference in Deep Generative Models'. In: (2014). DOI: 10.1051/0004-6361/201527329 (cited on pages 9, 44, 51).

[33] Ronald Aylmer Fisher. *Statistical methods for research workers*. Genesis Publishing Pvt Ltd, 2006 (cited on page 11).

[34] Christopher M Bishop. *Pattern Recognition and Machine Learning*. Springer. 2006 (cited on page 13).

[35] G Cooper. 'Computational Complexity of probabilistic inference using Bayesian belief networks (research note)'. In: *Machine Learning* 42.1990 (1990), pp. 393–405 (cited on page 14).

[36] Judea Pearl. 'Fusion, propagation, and structuring in belief networks'. In: *Artificial Intelligence* 29.3 (1986), pp. 241–288. DOI: 10.1016/0004-3702(86)90072-X (cited on page 14).

[37] Andriy Mnih and Karol Gregor. 'Neural Variational Inference and Learning in Belief Networks'. In: 32 (2014). DOI: 10.1117/12.526813 (cited on page 14).

[38] Geoffrey E Hinton and Ruslan R Salakhutdinov. 'Reducing the dimensionality of data with neural networks'. In: *science* 313.5786 (2006), pp. 504–507 (cited on page 15).

[39] Martin J Wainwright, Tommi S Jaakkola, and Alan S Willsky. 'A new class of upper bounds on the log partition function'. In: *IEEE Transactions on Information Theory* 51.7 (2005), pp. 2313–2335 (cited on page 15).

[40] Gerasimos G Potamianos and John K Goutsias. 'Partition function estimation of Gibbs random field images using Monte Carlo simulations'. In: *IEEE Transactions on Information Theory* 39.4 (1993), pp. 1322–1332 (cited on page 15).

[41] Frank R Kschischang, Brendan J Frey, Hans-Andrea Loeliger, et al. 'Factor graphs and the sum-product algorithm'. In: *IEEE Transactions on information theory* 47.2 (2001), pp. 498–519 (cited on page 15).
[42] Jingwei Zhang. 'An Optimal Transport View on Generalization'. In: () (cited on page 15).

[43] Raddford Neal and Geoffrey E. Hinton. 'A view of the EM algorithm that justifies incremental, sparse, and other variants'. In: *Learning in graphical models*. Ed. by Springer. 1998, pp. 355–368 (cited on page 16).

[44] Matthew J Beal. 'Variational algorithms for approximate bayesian inference'. In: *PhD Thesis* May (2003), pp. 1–281 (cited on page 17).

[45] Tommi S Jaakkola and Michael I Jordan. 'Bayesian parameter estimation via variational methods'. In: *Statistics and Computing* 10.1 (2000), pp. 25–37 (cited on page 17).

[46] Tommi S Jaakkola. 'Tutorial on Variational Approximation Methods'. In: *In Advanced Mean Field Methods: Theory and Practice* (2000), 129{\textendash}159. DOI: 10.1.1.31.8989 (cited on page 17).

[47] Hagai Attias. 'Inferring Parameters and Structure of Latent Variable Models by Variational Bayes'. In: *Journal of Chemical Information and Modeling* 53.9 (2013), pp. 1689–1699. DOI: 10.1017/CB09781107415324.004 (cited on page 17).

[48] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. 'Variational inference: A review for statisticians'. In: *Journal of the American Statistical Association* just-accepted (2017) (cited on page 17).

[49] Carsten Peterson. 'A mean field theory learning algorithm for neural networks'. In: *Complex systems* 1 (1987), pp. 995–1019 (cited on page 17).

[50] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. 'Variational Inference: A Review for Statisticians'. In: *Journal of the American Statistical Association* 112.518 (2017), pp. 859–877. DOI: 10.1080/01621459.2017.1285773 (cited on page 17).

[51] Michael I. Jordan et al. 'Introduction to variational methods for graphical models'. In: *Machine Learning* 37.2 (1999), pp. 183–233. DOI: 10.1023/A:1007665907178 (cited on page 17).

[52] Christopher M Bishop. *Pattern recognition and machine learning*. springer, 2006 (cited on page 17).

[53] Martin J Wainwright, Michael I Jordan, et al. 'Graphical models, exponential families, and variational inference'. In: *Foundations and Trends® in Machine Learning* 1.1–2 (2008), pp. 1–305 (cited on page 17).

[54] Iñigo Urteaga and Chris H Wiggins. 'Variational inference for the multi-armed contextual bandit arXiv: 1709.03163v1 [stat.ML] 10 Sep 2017'. In: 1933 (2017), pp. 1–13 (cited on page 17).

[55] Dustin Tran, Rajesh Ranganath, and David M. Blei. 'The Variational Gaussian Process'. In: (2015), pp. 1–14 (cited on pages 17, 19).
[56] Roger Frigola, Yutian Chen, and Carl E. Rasmussen. 'Variational Gaussian process state-space models'. In: *Advances in Neural Information Processing Systems* 4.January (2014), pp. 3680–3688 (cited on pages 17, 105).

[57] Lawrence Saul and Michael Jordan. 'Exploiting tractable substructures in in-tractable networks'. In: (1996), pp. 486–492 (cited on page 18).

[58] Xing E T Al, Eric P Xing, and Michael I Jordan. 'A generalized mean field algorithm for variational inference m exponential families'. In: (1999), pp. 583–591 (cited on page 19).

[59] Rajesh Ranganath et al. 'Deep exponential families'. In: *Artificial Intelligence and Statistics*. 2015, pp. 762–771 (cited on pages 19, 44).

[60] David M Blei, Michael I Jordan, et al. 'Variational inference for Dirichlet process mixtures'. In: *Bayesian analysis* 1.1 (2006), pp. 121–143 (cited on page 19).

[61] Anirban Roychowdhury and Brian Kulis. 'Gamma Processes, Stick-Breaking, and Variational Inference'. In: *arXiv preprint* 1994 (2014), pp. 1–19 (cited on page 19).

[62] Chong Wang and David M Blei. 'Variational inference for the nested Chinese restaurant process'. In: *Advances in Neural Information Processing Systems*. 2009, pp. 1990–1998 (cited on page 19).

[63] Samuel Gershman, Matt Hoffman, and David Blei. 'Nonparametric variational inference'. In: *arXiv preprint arXiv:1206.4665* (2012) (cited on page 19).

[64] Rajesh Ranganath, Dustin Tran, and David Blei. 'Hierarchical variational models'. In: *International Conference on Machine Learning*. 2016, pp. 324–333 (cited on page 19).

[65] Felix Agakov and David Barber. 'An Auxiliary Variational Method'. In: April (2004) (cited on page 19).

[66] Shakir Mohamed and Balaji Lakshminarayanan. 'Learning in implicit generative models'. In: *arXiv preprint arXiv:1610.03483* (2016) (cited on page 19).

[67] Steffen Bickel, Michael Brückner, and Tobias Scheffer. 'Discriminative learning for differing training and test distributions'. In: *ACM International Conference Proceeding Series* 227 (2007), pp. 81–88. DOI: 10.1145/1273496.1273507 (cited on page 19).

[68] Vincent Dumoulin et al. 'Adversarially Learned Inference'. In: (2017), pp. 1–18 (cited on page 19).

[69] Alireza Makhzani et al. 'Adversarial Autoencoders'. In: (2015) (cited on pages 19, 73).

[70] Lars Mescheder et al. 'Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks'. In: () (cited on page 19).
[71] Nikolaos Gianniotis. 'Mixed Variational Inference'. In: (2019), pp. 1–15 (cited on page 19).

[72] Mingzhang Yin and Mingyuan Zhou. 'Semi-Implicit Variational Inference'. In: (2018) (cited on page 19).

[73] Dmitry Molchanov et al. 'Doubly Semi-Implicit Variational Inference'. In: Vi (2018) (cited on page 19).

[74] Jiaxin Shi, Shengyang Sun, and Jun Zhu. 'Implicit Variational Inference with Kernel Density Ratio Fitting'. In: 1 (2017) (cited on page 20).

[75] Arthur Gretton et al. 'A kernel method for the two-sample-problem'. In: *Advances in neural information processing systems*. 2007, pp. 513–520 (cited on pages 20, 21, 50).

[76] Tim Van Erven and Peter Harremo. 'Renyi Divergence and Kullback-Leibler Divergence'. In: 6.1 (2007), pp. 1–24 (cited on pages 20, 25).

[77] Yingzhen Li and Richard E Turner. 'Renyi Divergence Variational Inference'. In: (), pp. 1–20 (cited on page 20).

[78] Ayananendran Bassu, I A N R Harris, and M C Jones. 'Robust and Efficient Estimation by Minimising a Density Power Divergence'. In: (), pp. 1–26 (cited on page 20).

[79] Jean-Baptiste Regli and Ricardo Silva. 'Alpha-Beta Divergence For Variational Inference'. In: (2018) (cited on page 20).

[80] Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. 'Generalized Variational Inference'. In: 1 (2019), pp. 1–61 (cited on page 21).

[81] Arshia Cont, Shlomo Dubnov, and Gérard Assayag. 'On the information geometry of audio streams with applications to similarity computing'. In: *IEEE Transactions on Audio, Speech, and Language Processing* 19.4 (2010), pp. 837–846 (cited on page 21).

[82] Yujia Li, Kevin Swersky, and Richard Zemel. 'Generative Moment Matching Networks'. In: (2014) (cited on page 21).

[83] Gintare Karolina Dziugaite and Daniel M Roy. 'Training generative neural networks via Maximum Mean Discrepancy optimization'. In: () (cited on page 21).

[84] Ilya Tolstikhin et al. 'Wasserstein Auto-Encoders'. In: (2017) (cited on pages 21, 50, 72).

[85] Luca Ambrogioni et al. 'Wasserstein Variational Inference'. In: () (cited on page 21).

[86] C Villani. 'Optimal transport: old and new'. In: (2008) (cited on page 21).
[87] Trevor Henderson and Justin Solomon. 'Audio Transport: A Generalized Porta-
mento via Optimal Transport'. In: *arXiv preprint arXiv:1906.06763* (2019) (cited
on page 21).

[88] Espen Bernton et al. 'On parameter estimation with the Wasserstein distance'. In: (2017), pp. 1-47 (cited on page 22).

[89] Antoine Rolet, Marco Cuturi, and Gabriel Peyré. 'Fast dictionary learning with a smoothed Wasserstein loss'. In: *Proceedings of the 19th International Conference on Artificial Intelligence and Statistics*, AISTATS 2016 51 (2016), pp. 630-638 (cited on page 22).

[90] Olivier Bousquet et al. 'From optimal transport to generative modeling: the VEGAN cookbook'. In: *arXiv preprint arXiv:1705.07642* (2017) (cited on page 22).

[91] Alireza Makhzani, Brendan Frey, and Ian Goodfellow. 'Adversarial Autoencoders'. In: (2014) (cited on page 22).

[92] Alfréd Rényi. 'On Measures of Entropy and Information'. In: *Journal of Physics 114.49* (1961), pp. 16184-16188. DOI: 10.1021/jp106846b (cited on page 25).

[93] Eshed Ram. 'On Rényi Entropy Power Inequalities'. In: *IEEE Transactions on Information Theory* (2016) (cited on page 25).

[94] Bruce G. Lindsay. 'The Geometry of Mixture Likelihoods: A General Theory'. In: *The Annals of Statistics* 11.1 (1983), pp. 86-94. DOI: 10.1214/aos/1176348654 (cited on page 25).

[95] Naftali Tishby and Noga Zaslavsky. 'Deep learning and the information bottleneck principle'. In: *2015 IEEE Information Theory Workshop, ITW 2015* (2015). DOI: 10.1109/ITW.2015.7133169 (cited on page 26).

[96] Alexander A Alemi et al. 'An Information-Theoretic Analysis of Deep Latent-Variable Models'. In: *arXiv preprint arXiv:1711.00464* (2017) (cited on page 26).

[97] Jorma Rissanen. 'A Universal Prior for Integers and Estimation by Minimum Description Length'. In: *Annals of Statistics* 14.2 (1986), pp. 590-606 (cited on page 27).

[98] Geoffrey E Hinton and Richard S Zemel. 'Autoencoders, minimum description length and Helmholtz free energy'. In: *Advances in neural information processing systems*. 1994, pp. 3-10 (cited on pages 27, 28).

[99] Karol Gregor et al. 'Deep AutoRegressive Networks'. In: *32* (2013) (cited on page 28).

[100] Hariharan Narayanan and Sanjoy Mitter. 'Sample complexity of testing the manifold hypothesis'. In: *Nips* (2010), pp. 1-9. DOI: 10.1590/0074-02760150454 (cited on page 29).
[101] Shuyin Xia et al. ‘Effectiveness of the Euclidean distance in high dimensional spaces’. In: *Optik* 126.24 (2015), pp. 5614–5619. DOI: 10.1016/j.ijleo.2015.09.093 (cited on page 29).

[102] Svante Wold, Kim Esbensen, and Paul Geladi. ‘Principal component analysis’. In: *Chemometrics and intelligent laboratory systems* 2.1-3 (1987), pp. 37–52 (cited on page 30).

[103] Jonathon Shlens. ‘A tutorial on principal component analysis’. In: *arXiv preprint arXiv:1404.1100* (2014) (cited on page 30).

[104] Emmanuel J. Candes et al. ‘Robust Principal Component Analysis?’ In: (2009). DOI: 10.1145/1970392.1970395 (cited on page 30).

[105] Hui Zou, Trevor Hastie, and Robert Tibshirani. ‘Sparse principal component analysis’. In: *Journal of computational and graphical statistics* 15.2 (2006), pp. 265–286 (cited on page 30).

[106] M. E. Tipping and C. M. Bishop. *Probabilistic Principal Component Analysis*. 1991. DOI: 10.1111/1467-9868.00196 (cited on page 30).

[107] Aapo Hyvarinen and Erkki Oja. ‘Independent component analysis by general nonlinear Hebbian-like learning rules’. In: *Signal Processing* 64 (1998), pp. 301–313. DOI: 10.1016/s0165-1684(97)00197-7 (cited on page 30).

[108] Harri Valpola et al. ‘Nonlinear Independent Component Analysis Using Ensemble Learning: Experiments and Discussion’. In: *Proc. 2^{\{\;nd} Int. Workshop on Independent Component Analysis and Blind Signal Separation – (ICA 2000)* (2000), pp. 351–356 (cited on page 30).

[109] Aapo Hyvärinen and Petteri Pajunen. ‘Nonlinear independent component analysis: Existence and uniqueness results’. In: *Neural Networks* 12.3 (1999), pp. 429–439 (cited on page 31).

[110] Joseph B Kruskal and Myron Wish. *Multidimensional scaling*. Vol. 11. Sage, 1978 (cited on pages 31, 88).

[111] Geoffrey E Hinton and Sam T Roweis. ‘Stochastic neighbor embedding’. In: *Advances in neural information processing systems*. 2003, pp. 857–864 (cited on page 31).

[112] Laurens van der Maaten and Geoffrey Hinton. ‘Visualizing data using t-SNE’. In: *Journal of machine learning research* 9.Nov (2008), pp. 2579–2605 (cited on page 32).

[113] S. Mika et al. ‘Kernel PCA and de-noising in feature spaces’. In: *Advances in Neural Information Processing Systems i* (1999) (cited on page 32).

[114] S. T. Roweis. ‘Nonlinear Dimensionality Reduction by Locally Linear Embedding’. In: *Science* 290.5500 (2000), pp. 2323–2326. DOI: 10.1126/science.290.5500.2323 (cited on page 33).
[115] Mikhail Belkin and Partha Niyogi. 'Laplacian Eigen Map'. In: *Neural Computation* 15 (2003), pp. 1373-1396 (cited on page 33).

[116] David L Donoho and Carrie Grimes. 'PNAS_Hessen eigenmap.pdf'. In: (2003). DOI: http://www.pnas.orgcgidoi10.1073pnas.1031596100/ (cited on page 33).

[117] Z Zhang and H Zha. *Principal Manifolds and Nonlinear Dimension Reduction via Local Tangent Space Alignment* (Dept of Computer Science, Pennsylvania State Univ, University Park, PA). Tech. rep. Tech Rep CSE-02-019, 2002 (cited on page 33).

[118] Joshua B Tenenbaum, Vin De Silva, and John C Langford. 'A global geometric framework for nonlinear dimensionality reduction'. In: *science* 290.5500 (2000), pp. 2319-2323 (cited on page 33).

[119] Leland McInnes, John Healy, and James Melville. 'UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction'. In: (2018) (cited on page 33).

[120] Herbert Robbins and Sutton Monro. 'A stochastic approximation method'. In: *The annals of mathematical statistics* (1951), pp. 400-407 (cited on page 35).

[121] John Duchi, Elad Hazan, and Yoram Singer. 'Adaptive subgradient methods for online learning and stochastic optimization'. In: *Journal of Machine Learning Research* 12.Jul (2011), pp. 2121-2159 (cited on page 35).

[122] Tijmen Tieleman and Geoffrey Hinton. 'Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude'. In: *COURSERA: Neural networks for machine learning* 4.2 (2012), pp. 26-31 (cited on page 35).

[123] Diederik P Kingma and Jimmy Ba. 'Adam: A method for stochastic optimization'. In: *arXiv preprint arXiv:1412.6980* (2014) (cited on pages 35, 70, 127).

[124] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 'Multilayer feedforward networks are universal approximators'. In: *Neural networks* 2.5 (1989), pp. 359-366 (cited on page 36).

[125] Kurt Hornik. 'Approximation Capabilities of Multilayer Neural Network'. In: *Neural Networks* 4.1991 (1991), pp. 251-257 (cited on page 36).

[126] Ken-Ichi Funahashi. 'On the approximate realization of continuous mappings by neural networks'. In: *Neural networks* 2.3 (1989), pp. 183-192 (cited on page 36).

[127] Zhou Lu et al. 'The expressive power of neural networks: A view from the width'. In: *Advances in Neural Information Processing Systems* 2017-Decem.Nips (2017), pp. 6232-6240 (cited on page 36).

[128] Boris Hanin. 'Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations'. In: *arXiv preprint arXiv:1708.02691* (2017) (cited on page 36).
[129] Ronen Eldan and Ohad Shamir. 'The power of depth for feedforward neural networks'. In: *Conference on learning theory*. 2016, pp. 907-940 (cited on page 36).

[130] Monica Bianchini and Franco Scarselli. 'On the complexity of shallow and deep neural network classifiers'. In: *22nd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2014 - Proceedings* April (2014), pp. 371-376 (cited on page 36).

[131] Guido Montúfar et al. 'On the number of linear regions of deep neural networks'. In: *Advances in Neural Information Processing Systems* 4.January (2014), pp. 2924-2932 (cited on page 36).

[132] Samuel S Schoenholz et al. 'Deep Information Propagation'. In: *arXiv preprint arXiv:1611.01232* (2016) (cited on page 36).

[133] Kaiming He et al. 'Identity mappings in deep residual networks'. In: *European conference on computer vision*. Springer. 2016, pp. 630-645 (cited on page 36).

[134] Emmanuel Abbe and Colin Sandon. 'Provable limitations of deep learning'. In: (2018) (cited on page 37).

[135] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 'Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps'. In: (2013), pp. 1-8 (cited on page 37).

[136] Sebastian Bach et al. 'On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation'. In: *PLoS ONE* 10.7 (2015), pp. 1-46. DOI: 10.1371/journal.pone.0130140 (cited on page 37).

[137] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 'Learning important features through propagating activation differences'. In: *34th International Conference on Machine Learning, ICML 2017* 7 (2017), pp. 4844-4866 (cited on page 37).

[138] Marylou Gabrié et al. 'Entropy and mutual information in models of deep neural networks'. In: (2018). DOI: *arXiv:1805.09785v1* (cited on page 37).

[139] Ziv Goldfeld et al. 'Estimating Information Flow in Neural Networks arXiv: 1810.05728v1 [cs.LG] 12 Oct 2018'. In: () (cited on page 37).

[140] Andrew Saxe et al. 'On The Information Bottleneck of Deep Learning'. In: *Iclr 2018* 24.3 (2018), pp. 3-6. DOI: 10.1108/eb040537 (cited on page 37).

[141] Henry W Lin, Max Tegmark, and David Rolnick. 'Why does deep and cheap learning work so well?' In: *Journal of Statistical Physics* 168.6 (2017), pp. 1223-1247 (cited on page 37).

[142] Chiyuan Zhang et al. *Theory of deep learning iii: Generalization properties of sgd*. Tech. rep. Discussion paper, Center for Brains, Minds and Machines (CBMM). Preprint, 2017 (cited on page 37).
[143] Nitish Srivastava et al. 'Dropout: a simple way to prevent neural networks from overfitting'. In: *The journal of machine learning research* 15.1 (2014), pp. 1929–1958 (cited on page 37).

[144] Alex Hernández-García and Peter König. 'Do deep nets really need weight decay and dropout?' In: (2018), pp. 1–5 (cited on page 37).

[145] Radford M. Neal. 'Bayesian Learning for Neural Networks'. In: 118 (1996). DOI: 10.1007/978-1-4612-0745-0 (cited on page 37).

[146] Nir Friedman and Iftach Nachman. 'Gaussian process networks'. In: *Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence*. Morgan Kaufmann Publishers Inc. 2000, pp. 211–219 (cited on page 37).

[147] David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. 'Learning Representations by Back Propagation Errors'. In: *Letters to Nature* 5.3 (323), pp. 533–536 (cited on page 38).

[148] Pierre Baldi and Kurt Hornik. 'Neural networks and principal component analysis: Learning from examples without local minima'. In: *Neural Networks* 2.1 (1989), pp. 53–58. DOI: 10.1016/0893-6080(89)90014-2 (cited on page 38).

[149] Marc'Aurelio Ranzato et al. 'Efficient Learning of Sparse Representations with an Energy-Based Model Marc'Aurelio'. In: *Advances in neural information processing systems* (2007), pp. 1137–1144 (cited on page 38).

[150] Alessandro Achille and Stefano Soatto. 'Emergence of Invariance and Disentanglement in Deep Representations'. In: 18 (2018), pp. 1–34 (cited on pages 39, 52, 73).

[151] Pascal Vincent et al. *Extracting and Composing Robust Features with Denoising Autoencoders*. Tech. rep. 2008 (cited on page 39).

[152] Pascal Vincent et al. 'Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion'. In: *Journal of Machine Learning Research* 11.Dec (2010), pp. 3371–3408 (cited on page 39).

[153] Guillaume Alain and Yoshua Bengio. 'What Regularized Auto-Encoders Learn from the Data Generating Distribution'. In: (2011), pp. 1–31 (cited on page 39).

[154] Salah Rifai et al. 'Contractive auto-encoders: Explicit invariance during feature extraction'. In: *Proceedings of the 28th international conference on machine learning (ICML-11)*. 2011, pp. 833–840 (cited on page 39).

[155] Matthew D Hoffman et al. 'Stochastic variational inference'. In: *The Journal of Machine Learning Research* 14.1 (2013), pp. 1303–1347 (cited on page 41).

[156] Rajesh Ranganath, Sean Gerrish, and David M Blei. 'Black Box Variational Inference'. In: () (cited on page 41).
[157] George Casella and Christian P Robert. 'Rao-Blackwellization of Sampling Schemes Rao-Blackwellization of Sampling Schemes'. In: July (1994) (cited on page 41).

[158] Sheldon M Ross and Kyle Y Lin. 'Applying variance reduction ideas in queuing simulations'. In: *Probability in the Engineering and Informational Sciences* 15.4 (2001), pp. 481–494 (cited on page 41).

[159] Yann LeCun et al. 'Backpropagation applied to handwritten zip code recognition'. In: *Neural computation* 1.4 (1989), pp. 541–551 (cited on pages 42, 66).

[160] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 'Stochastic backpropagation and approximate inference in deep generative models'. In: *arXiv preprint arXiv:1401.4082* (2014) (cited on pages 43, 51).

[161] Samuel R Bowman et al. 'Generating Sentences from a Continuous Space'. In: (2015) (cited on pages 43, 44, 48).

[162] Chris Cremer, Xuechen Li, and David Duvenaud. 'Inference Suboptimality in Variational Autoencoders'. In: (2018) (cited on page 43).

[163] Xi Chen et al. 'Variational Lossy Autoencoder'. In: (2016), pp. 1–17 (cited on page 44).

[164] Serena Yeung et al. 'Tackling Over-pruning in Variational Autoencoders'. In: *arXiv preprint arXiv:1706.03643* (2017) (cited on pages 44, 48, 49).

[165] Jakub M. Tomczak and Max Welling. 'Improving Variational Auto-Encoders using Householder Flow'. In: *2* (2016) (cited on page 44).

[166] Ke Sun and Xiangliang Zhang. 'Coarse Grained Exponential Variational Autoencoders'. In: *arXiv preprint arXiv:1702.07904* (2017) (cited on page 44).

[167] Shun-Ichi Amari. *Information Geometry and Its Applications (front matter)*. Vol. 194. 2000, p. 736 (cited on page 44).

[168] Prasoon Goyal et al. 'Nonparametric Variational Auto-encoders for Hierarchical Representation Learning'. In: *arXiv preprint arXiv:1703.07027* (2017) (cited on page 45).

[169] E G Tabak and Cristina V Turner. 'A family of nonparametric density estimation algorithms'. In: *Communications on Pure and Applied Mathematics* 66.2 (2000), pp. 145–164 (cited on page 45).

[170] Esteban G. Tabak and Eric Vanden-Eijnden. 'Density estimation by dual ascent of the log-likelihood'. In: *Communications in Mathematical Sciences* 8.1 (2009), pp. 217–233 (cited on page 45).

[171] Diederik P. Kingma and Prafulla Dhariwal. 'Glow: Generative Flow with Invertible 1x1 Convolutions'. In: *2* (2018), pp. 1–10 (cited on pages 45, 59).
[172] Danilo Jimenez Rezende and Shakir Mohamed. 'Variational Inference with Normalizing Flows'. In: 37 (2015) (cited on pages 45, 117).

[173] Diederik P Kingma et al. 'Improved variational inference with inverse autoregressive flow'. In: *Advances in Neural Information Processing Systems*. 2016, pp. 4743-4751 (cited on pages 46, 49).

[174] George Papamakarios, Theo Pavlakou, and Iain Murray. 'Masked Autoregressive Flow for Density Estimation'. In: *arXiv preprint arXiv:1705.07057* (2017) (cited on page 46).

[175] Micha Livne and David Fleet. 'TzK: Flow-Based Conditional Generative Model'. In: NeurIPS (2019), pp. 1-7 (cited on page 46).

[176] Prince Zizhuang Wang. 'Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling'. In: (2018) (cited on page 46).

[177] Chris Cremer, Quaid Morris, and David Duvenaud. 'Reinterpreting Importance-Weighted Autoencoders'. In: *arXiv preprint arXiv:1704.02916* (2017) (cited on page 47).

[178] Andriy Mnih and Danilo Rezende. 'Variational inference for monte carlo objectives'. In: *International Conference on Machine Learning*. 2016, pp. 2188-2196 (cited on page 47).

[179] Chris J Maddison et al. 'Filtering Variational Objectives'. In: *arXiv preprint arXiv:1705.09279* (2017) (cited on page 47).

[180] Bin Dai et al. 'Veiled Attributes of the Variational Autoencoder'. In: *arXiv preprint arXiv:1706.05148* (2017) (cited on pages 48, 51).

[181] Andrea Asperti. 'Sparsity in Variational Autoencoders'. In: 2 () (cited on page 48).

[182] Michal Rolinek, Dominik Zietlow, and Georg Martius. 'Variational Autoencoders Pursue PCA Directions (by Accident)'. In: (2018), pp. 1-22 (cited on page 48).

[183] Samuel R. Bowman et al. 'Generating Sentences from a Continuous Space'. In: (2015). DOI: 10.18653/v1/K16-1002 (cited on pages 49, 109).

[184] Casper Kaae Sønderby et al. 'How to train deep variational autoencoders and probabilistic ladder networks'. In: *arXiv preprint arXiv:1602.02282* (2016) (cited on pages 49, 125).

[185] Xi Chen et al. 'Variational lossy autoencoder'. In: *arXiv preprint arXiv:1611.02731* (2016) (cited on page 49).

[186] Ali Razavi et al. 'Preventing Posterior Collapse with delta-VAEs'. In: (2019) (cited on page 49).
[187] Matthew D Hoffman and Matthew J Johnson. 'Elbo surgery: yet another way to carve up the variational evidence lower bound'. In: Workshop in Advances in Approximate Bayesian Inference, NIPS. 2016 (cited on page 49).

[188] Giorgio Patrini et al. 'Sinkhorn AutoEncoders'. In: (2018) (cited on page 50).

[189] Shengjia Zhao, Jiaming Song, and Stefano Ermon. 'InfoVAE: Information Maximizing Variational Autoencoders'. In: (2017) (cited on page 50).

[190] Hiroshi Takahashi et al. 'Variational Autoencoder with Implicit Optimal Priors'. In: (2018) (cited on page 50).

[191] Ehsan Abbasnejad, Anthony Dick, and Anton van den Hengel. 'Infinite Variational Autoencoder for Semi-Supervised Learning'. In: arXiv preprint arXiv:1611.07800 (2016) (cited on page 51).

[192] Finale Doshi-Velez and Been Kim. 'Towards A Rigorous Science of Interpretable Machine Learning'. In: Ml (2017), pp. 1-13 (cited on page 51).

[193] Ilyes Khemakhem, Diedrik Kingma, and Aapo Hyvärinen. 'Variational Autoencoders and Nonlinear ICA: A Unifying Framework'. In: () (cited on page 51).

[194] Irina Higgins et al. 'beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework'. In: ICLR July (2017), pp. 1-13 (cited on pages 51, 73).

[195] Yan Zhang et al. 'Information Potential Auto-Encoders'. In: (2017), pp. 1-15 (cited on page 52).

[196] Sherjil Ozair, Corey Lynch, and Yoshua Bengio. 'Wasserstein Dependency Measure for Representation Learning'. In: (2015) (cited on page 52).

[197] Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. 'Latent Space Oddity: on the Curvature of Deep Generative Models'. In: 2 (2017), pp. 1-15 (cited on pages 52, 77).

[198] P. Thomas Fletcher Hang Shao, Abhishek Kumar. 'The Riemannian Geometry of Deep Generative Models'. In: (2017) (cited on page 53).

[199] Ricky T Q Chen et al. 'Neural Ordinary Differential Equations'. In: NeurIPS (2018), pp. 1-19 (cited on page 53).

[200] Vlado Menkovski and Jacobus W Portegies. 'Diffusion Variational Autoencoders'. In: (2019) (cited on page 53).

[201] Ivan Ovinnikov. 'Poincaré Wasserstein Autoencoder'. In: NeurIPS (2018) (cited on page 53).

[202] Octavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann. 'Hyperbolic Neural Networks'. In: (2018) (cited on page 53).
[203] Gaëtan Hadjeres, Frank Nielsen, and François Pachet. 'GLSR-VAE: Geodesic Latent Space Regularization for Variational AutoEncoder Architectures'. In: (2017), pp. 1-11 (cited on page 53).

[204] Joseph Redmon and Ali Farhadi. 'YOLO9000: Better, Faster, Stronger'. In: (2016). DOI: 10.1142/9789812771728_0012 (cited on page 53).

[205] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 'ImageNet Classification with Deep Convolutional Neural Networks Alex'. In: *Handbook of Approximation Algorithms and Metaheuristics* (2007), pp. 60-1-60-16. DOI: 10.1201/9781420010749 (cited on page 53).

[206] Shlomo E Chazan, Sharon Gannot, and Jacob Goldberger. 'Deep clustering based on a mixture of autoencoders'. In: () (cited on page 53).

[207] Klaus Greff et al. 'Multi-Object Representation Learning with Iterative Variational Inference'. In: (2019) (cited on page 53).

[208] Karol Gregor et al. 'DRAW: A Recurrent Neural Network For Image Generation'. In: (2015) (cited on page 53).

[209] S M Ali Eslami et al. 'Attend, infer, repeat: Fast scene understanding with generative models'. In: *Advances in Neural Information Processing Systems*. 2016, pp. 3225-3233 (cited on page 53).

[210] Jason Tyler Rolfe. 'Discrete Variational Autoencoders'. In: (2016), pp. 1-33 (cited on page 54).

[211] Akash Srivastava and Charles Sutton. 'Autoencoding variational inference for topic models'. In: *arXiv preprint arXiv:1703.01488* (2017) (cited on page 54).

[212] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 'Neural Discrete Representation Learning'. In: Nips (2017) (cited on page 54).

[213] Abubakar Abid, Muhammad Fatih, and Balin James. 'Concrete Autoencoders for Differentiable Feature Selection and Reconstruction'. In: (1997) (cited on page 54).

[214] Eric Jang, Shixiang Gu, and Ben Poole. 'Categorical Reparameterization with Gumbel-Softmax'. In: (2016), pp. 1-13 (cited on page 54).

[215] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 'The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables'. In: (2016), pp. 1-20 (cited on page 54).

[216] Jiacheng Xu and Greg Durrett. 'Spherical Latent Spaces for Stable Variational Autoencoders'. In: (2018) (cited on page 55).

[217] Nat Dilokthanakul et al. 'Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders'. In: 2016 (2016), pp. 1-12 (cited on page 55).
[218] Lars Maaløe et al. 'Auxiliary deep generative models'. In: *arXiv preprint arXiv:1602.05473*
(2016) (cited on page 55).

[219] Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. 'Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations'. In: *arXiv preprint arXiv:1705.08841* (2017) (cited on page 56).

[220] Philippe Esling and Carlos Agon. 'Multiobjective time series matching for audio classification and retrieval'. In: *IEEE Transactions on Audio, Speech, and Language Processing* 21.10 (2013), pp. 2057–2072 (cited on page 57).

[221] J Markel and A Gray. 'On autocorrelation equations as applied to speech analysis'. In: *IEEE Transactions on audio and Electroacoustics* 21.2 (1973), pp. 69–79 (cited on page 57).

[222] Olivier Lartillot, Petri Toiviainen, and Tuomas Eerola. 'A matlab toolbox for music information retrieval'. In: *Data analysis, machine learning and applications*. Springer, 2008, pp. 261–268 (cited on page 58).

[223] Jaap Haitsma and Ton Kalker. 'Speed-change resistant audio fingerprinting using auto-correlation'. In: *2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP'03)*. Vol. 4. IEEE. 2003, pp. IV–728 (cited on page 58).

[224] MIReS Consortium. *Roadmap for Music Information Research*. Tech. rep. 2013 (cited on page 58).

[225] Rainer Typke, Frans Wiering, and Remco C Veltkamp. 'A survey of music information retrieval systems'. In: *Proc. 6th International Conference on Music Information Retrieval*. Queen Mary, University of London. 2005, pp. 153–160 (cited on page 58).

[226] Meinard Muller et al. 'Signal processing for music analysis'. In: *IEEE Journal of Selected Topics in Signal Processing* 5.6 (2011), pp. 1088–1110 (cited on page 58).

[227] Alain De Cheveigné and Hideki Kawahara. 'YIN, a fundamental frequency estimator for speech and music'. In: *The Journal of the Acoustical Society of America* 111.4 (2002), pp. 1917–1930 (cited on page 58).

[228] Gaël Richard and Roland Badeau. 'Détection de fréquences fondamentales multiples'. In: () (cited on page 58).

[229] Geoffroy Peeters. 'Music pitch representation by periodicity measures based on combined temporal and spectral representations'. In: *2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings*. Vol. 5. IEEE. 2006, pp. V–V (cited on page 58).

[230] Miguel A Alonso Arevalo. 'Extraction d'information rythmique à partir d'enregistrements musicaux'. PhD thesis. Télécom ParisTech, 2006 (cited on page 58).
[231] Gilbert Nouno. 'Suivi de tempo appliqué aux musiques improvisées, à la recherche du temps perdu...' PhD thesis. Paris 6, 2008 (cited on page 58).

[232] Peter Grosche, Meinard Müller, and Frank Kurth. 'Cyclic tempogram—A mid-level tempo representation for musicsignals'. In: *2010 IEEE International Conference on Acoustics, Speech and Signal Processing*. IEEE. 2010, pp. 5522–5525 (cited on page 58).

[233] Juan P Bello. 'Measuring structural similarity in music'. In: *IEEE Transactions on Audio, Speech, and Language Processing* 19.7 (2011), pp. 2013–2025 (cited on page 58).

[234] Mark Levy and Mark Sandler. 'Structural segmentation of musical audio by constrained clustering'. In: *IEEE transactions on audio, speech, and language processing* 16.2 (2008), pp. 318–326 (cited on page 58).

[235] Goffredo Haus, Luca A Ludovico, and Giorgio Presti. 'Automatic Annotation of Timbre Variation for Musical Instruments'. In: *Computer Music Multidisciplinary Research*. Les éditions de PRISM. 2017, pp. 493–504 (cited on page 58).

[236] Thomas Kemp et al. 'Strategies for automatic segmentation of audio data'. In: *2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 00CH37100)*. Vol. 3. IEEE. 2000, pp. 1423–1426 (cited on page 58).

[237] Yi-Hsuan Yang and Homer H Chen. 'Machine recognition of music emotion: A review'. In: *ACM Transactions on Intelligent Systems and Technology (TIST)* 3.3 (2012), p. 40 (cited on page 58).

[238] Jean-Julien Aucouturier et al. 'Covert digital manipulation of vocal emotion alter speakers' emotional states in a congruent direction'. In: *Proceedings of the National Academy of Sciences* 113.4 (2016), pp. 948–953 (cited on page 58).

[239] Bob L Sturm. 'A survey of evaluation in music genre recognition'. In: *International Workshop on Adaptive Multimedia Retrieval*. Springer. 2012, pp. 29–66 (cited on page 58).

[240] Nicolas Scaringella, Giorgio Zoia, and Daniel Mlynek. 'Automatic genre classification of music content: a survey'. In: *IEEE Signal Processing Magazine* 23.2 (2006), pp. 133–141 (cited on page 58).

[241] Marius Kaminskas and Francesco Ricci. 'Contextual music information retrieval and recommendation: State of the art and challenges'. In: *Computer Science Review* 6.2-3 (2012), pp. 89–119 (cited on page 58).

[242] Peter Knees and Markus Schedl. 'A survey of music similarity and recommendation from music context data'. In: *ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)* 10.1 (2013), p. 2 (cited on page 58).
[243] Geoffray Bonnin and Dietmar Jannach. 'Automated generation of music playlists: Survey and experiments'. In: *ACM Computing Surveys (CSUR)* 47.2 (2015), p. 26 (cited on page 58).

[244] Jan Schlüter and Sebastian Böck. 'Improved musical onset detection with convolutional neural networks'. In: *2014 iee international conference on acoustics, speech and signal processing (icassp)*. IEEE. 2014, pp. 6979–6983 (cited on page 58).

[245] Hong Su et al. 'Convolutional neural network for robust pitch determination'. In: *2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE. 2016, pp. 579–583 (cited on page 58).

[246] Eric J Humphrey and Juan P Bello. 'Rethinking automatic chord recognition with convolutional neural networks'. In: *2012 11th International Conference on Machine Learning and Applications*. Vol. 2. IEEE. 2012, pp. 357–362 (cited on page 58).

[247] Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon. 'An end-to-end neural network for polyphonic piano music transcription'. In: *IEEE/ACM Transactions on Audio, Speech, and Language Processing* 24.5 (2016), pp. 927–939 (cited on page 58).

[248] Antoine Liutkus, Roland Badeau, and Gäel Richard. 'Gaussian processes for underdetermined source separation'. In: *IEEE Transactions on Signal Processing* 59.7 (2011), pp. 3155–3167 (cited on page 58).

[249] Benoit Fuentes et al. 'Probabilistic model for main melody extraction using constant-Q transform'. In: *2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE. 2012, pp. 5357–5360 (cited on page 58).

[250] Aditya Arie Nugraha, Antoine Liutkus, and Emmanuel Vincent. 'Multichannel audio source separation with deep neural networks'. In: *IEEE/ACM Transactions on Audio* 24.10 (2016), pp. 1652–1664. DOI: 10.1109/TASLP.2016.2580946> (cited on pages 58, 64).

[251] Andreas Jansson et al. 'Singing voice separation with deep U-Net convolutional networks'. In: (2017) (cited on page 58).

[252] Curtis Hawthorne et al. 'Onsets and frames: Dual-objective piano transcription'. In: *arXiv preprint arXiv:1710.11153* (2017) (cited on page 58).

[253] Jan Schlüter and Thomas Grill. 'Exploring Data Augmentation for Improved Singing Voice Detection with Neural Networks.' In: *ISMIR*. 2015, pp. 121–126 (cited on page 58).

[254] Jean-Claude Risset and David L Wessel. 'Exploration of timbre by analysis and synthesis'. In: *The psychology of music*. Elsevier, 1999, pp. 113–169 (cited on page 58).
[255] Curtis Roads, John Strawn, et al. *The computer music tutorial*. MIT press, 1996 (cited on page 58).

[256] Philippe Depalle, Guillermo Garcia, and Xavier Rodet. 'Tracking of partials for additive sound synthesis using hidden Markov models'. In: *1993 IEEE International Conference on Acoustics, Speech, and Signal Processing*. Vol. 1. IEEE. 1993, pp. 225-228 (cited on page 58).

[257] Piotr Kleczkowski. 'Group additive synthesis'. In: *Computer Music Journal* 13.1 (1989), pp. 12-20 (cited on page 58).

[258] Curtis Roads. 'Introduction to granular synthesis'. In: *Computer Music Journal* 12.2 (1988), pp. 11-13 (cited on page 58).

[259] Julius O Smith. 'Physical modeling using digital waveguides'. In: *Computer music journal* 16.4 (1992), pp. 74-91 (cited on page 58).

[260] Davide Rocchesso, Roberto Bresin, and Mikael Fernstrom. 'Sounding objects'. In: *IEEE MultiMedia* 10.2 (2003), pp. 42-52 (cited on page 58).

[261] Stefan D Bilbao. *Numerical sound synthesis*. Wiley Online Library, 2009 (cited on page 58).

[262] Aaron van den Oord et al. 'Wavenet: A generative model for raw audio'. In: *arXiv preprint arXiv:1609.03499* (2016) (cited on pages 59, 62).

[263] Mathieu Germain et al. 'MADE: masked autoencoder for distribution estimation'. In: *Proceedings of the 32nd International Conference on Machine Learning (ICML-15)*. 2015, pp. 881-889 (cited on page 59).

[264] Mehdi Mirza and Simon Osindero. 'Conditional generative adversarial nets'. In: *arXiv preprint arXiv:1411.1784* (2014) (cited on page 59).

[265] Xi Chen et al. 'Infogan: Interpretable representation learning by information maximizing generative adversarial nets'. In: *Advances in Neural Information Processing Systems*. 2016, pp. 2172-2180 (cited on page 59).

[266] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. 'WaveGlow: A Flow-based Generative Network for Speech Synthesis'. In: (2018) (cited on page 59).

[267] Huadong Liao, Jiawei He, and Kunxian Shu. 'Generative Model with Dynamic Linear Flow'. In: (2019) (cited on page 59).

[268] Roland Badeau, Gaël Richard, and Bertrand David. 'Fast adaptive esprit algorithm'. In: *IEEE/SP 13th Workshop on Statistical Signal Processing, 2005*. IEEE. 2005, pp. 289-294 (cited on page 59).

[269] Sadaoki Furui. 'Cepstral analysis technique for automatic speaker verification'. In: *IEEE Transactions on Acoustics, Speech, and Signal Processing* 29.2 (1981), pp. 254-272 (cited on page 59).
[270] Xavier Rodet and Philippe Depalle. 'Spectral envelopes and inverse FFT synthesis'. In: *Audio Engineering Society Convention* 93. Audio Engineering Society. 1992 (cited on page 64).

[271] Jonathan Allen. 'Short term spectral analysis, synthesis, and modification by discrete Fourier transform'. In: *IEEE Transactions on Acoustics, Speech, and Signal Processing* 25.3 (1977), pp. 235-238 (cited on page 64).

[272] Judith C Brown. 'Calculation of a constant Q spectral transform'. In: *The Journal of the Acoustical Society of America* 89.1 (1991), pp. 425-434 (cited on page 64).

[273] Gino Angelo Velasco et al. 'Constructing an invertible constant-Q transform with non-stationary Gabor frames'. In: *Proceedings of DAFX11, Paris* (2011), pp. 93-99 (cited on page 64).

[274] Hans Georg Zimmermann, Alexey Minin, and Victoria Kusherbaeva. 'Comparison of the Complex Valued and Real Valued Neural Networks Trained with Gradient Descent and Random Search Algorithms'. In: *April* (2011), pp. 27-29 (cited on page 64).

[275] Christian Berg. 'Complex Analysis'. In: (2012), pp. 1-101 (cited on page 64).

[276] Chiheb Trabelsi et al. 'Deep Complex Networks'. In: 2016 (2017), pp. 1-19. DOI: 10.1159/000448939 (cited on page 64).

[277] Bernard Picinbono, Second-order Complex Random, and Normal Distributions Ieee Trans-. 'Second-Order Complex Random Vectors and Normal Distributions To cite this version : HAL Id : hal-01736682 Second-Order Complex Random Vectors and Normal Distributions'. In: (2018) (cited on page 64).

[278] Daniel Griffin and Jae Lim. 'Signal estimation from modified short-time Fourier transform'. In: *IEEE Transactions on Acoustics, Speech, and Signal Processing* 32.2 (1984), pp. 236-243 (cited on page 65).

[279] Nathanaël Perraudin, Peter Balazs, and Peter L Søndergaard. 'A fast Griffin-Lim algorithm'. In: *2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics*. IEEE. 2013, pp. 1-4 (cited on page 65).

[280] Zdenek Pruša and Peter L Søndergaard. 'Real-time spectrogram inversion using phase gradient heap integration'. In: *Proc. Int. Conf. Digital Audio Effects (DAFx-16)*. 2016, pp. 17-21 (cited on page 65).

[281] Guillaume Ballet et al. 'Studio online 3.0: An internet' killer application' for remote access to IRCam sounds and processing tools'. In: *Journée d'Informatique Musicale (JIM)* (1999) (cited on pages 69, 140).

[282] John S Gero and Udo Kannengiesser. 'The situated function-behaviour-structure framework'. In: *Design studies* 25.4 (2004), pp. 373-391 (cited on page 69).
[283] Li-chia Yang Alexander Lerch. 'On the evaluation of generative models in music'. In: (2018) (cited on page 69).

[284] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 'Fast and accurate deep network learning by exponential linear units (elus)'. In: *arXiv preprint arXiv:1511.07289* (2015) (cited on page 70).

[285] Sergey Ioffe and Christian Szegedy. 'Batch normalization: Accelerating deep network training by reducing internal covariate shift'. In: *International Conference on Machine Learning*. 2015, pp. 448–456 (cited on page 70).

[286] Sercan O. Arik, Heewoo Jun, and Gregory Diamos. 'Fast spectrogram inversion using multi-head convolutional neural networks'. In: *IEEE Signal Processing Letters*. 26.1 (2019), pp. 94–98. DOI: 10.1109/LSP.2018.2880284 (cited on page 71).

[287] Fumitada Itakura. 'Analysis synthesis telephony based on the maximum likelihood method'. In: *The 6th international congress on acoustics, 1968*. 1968, pp. 280–292 (cited on page 71).

[288] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. 'The Numerics of GANs'. In: *Nips* (2017) (cited on page 73).

[289] Hyunjik Kim and Andriy Mnih. 'Disentangling by Factorising'. In: (2018) (cited on page 73).

[290] Sangchul Hahn and Heeyoul Choi. 'Disentangling Latent Factors with Whitening'. In: (2018) (cited on page 73).

[291] Cian Eastwood and Christopher K.L. Williams. 'A Framework for the Quantitative Evaluation of Disentangled Representations'. In: *ICLR* (2018) (cited on page 74).

[292] Jeremy Marozeau and Alain de Cheveigné. 'The effect of fundamental frequency on the brightness dimension of timbre'. In: *The Journal of the Acoustical Society of America* 121.1 (2007), pp. 383–387. DOI: 10.1121/1.2384910 (cited on pages 76, 88).

[293] Anssi Klapuri and Manuel Davy. *Signal processing methods for music transcription*. Springer Science & Business Media, 2007 (cited on page 76).

[294] Shlomo Dubnov. 'Generalization of spectral flatness measure for non-Gaussian linear processes'. In: *IEEE Signal Processing Letters* 11.8 (2004), pp. 698–701. DOI: 10.1109/LSP.2004.831663 (cited on page 76).

[295] Diederik P Kingma et al. 'Semi-supervised learning with deep generative models'. In: *Advances in Neural Information Processing Systems*. 2014, pp. 3581–3589 (cited on pages 79, 80).
[296] Xinchen Yan et al. 'Attribute2image: Conditional image generation from visual attributes'. In: *European Conference on Computer Vision*. Springer. 2016, pp. 776–791 (cited on page 79).

[297] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. 'A Learned Representation For Artistic Style'. In: 2016 (2016) (cited on page 79).

[298] Ethan Perez et al. 'FiLM: Visual Reasoning with a General Conditioning Layer'. In: (2017) (cited on page 79).

[299] Adrien Bitton, Philippe Esling, and Axel Chemla-Romeu-Santos. 'Modulated Variational auto-Encoders for many-to-many musical timbre transfer'. In: 2015 (2018), pp. 1–13 (cited on page 79).

[300] Yang Li et al. 'Disentangled Variational Auto-Encoder for Semi-supervised Learning'. In: (2017) (cited on page 79).

[301] Ramakrishna Vedantam et al. 'Generative Models of Visually Grounded Imagination'. In: *arXiv preprint arXiv:1705.10762* (2017) (cited on page 80).

[302] Irina Higgins et al. 'SCAN: Learning Abstract Hierarchical Compositional Visual Concepts'. In: *arXiv preprint arXiv:1707.03389* (2017) (cited on pages 80, 82).

[303] Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. 'Joint Multimodal Learning with Deep Generative Models'. In: *arXiv preprint arXiv:1611.01891* (2016) (cited on page 81).

[304] Ming-Yu Liu, Thomas Breuel, and Jan Kautz. 'Unsupervised Image-to-Image Translation Networks'. In: 2 (2017) (cited on page 81).

[305] Elena Agullò Cantos. 'Flute audio labelled database for Automatic Music Transcription'. In: () (cited on pages 83, 85).

[306] John M Grey and John W Gordon. 'Perceptual effects of spectral modifications on musical timbres'. In: *The Journal of the Acoustical Society of America* 63.5 (1978), pp. 1493–1500 (cited on page 88).

[307] J. Krimphoff, S. McAdams, and S. Winsberg. 'Caracterisation du timbre des sons complexes. II. analyses acoustiques et quantification psychophysique'. In: *Journal De Physique* 4.5 pt 1 (1994), pp. 625–628 (cited on page 88).

[308] A. M. Noll and M. R. Schroeder. 'Short-Time "Cepstrum" Pitch Detection'. In: *The Journal of the Acoustical Society of America* 36.5 (1964), pp. 1030–1030. DOI: 10.1121/1.2143271 (cited on page 88).

[309] William Brent. 'Cepstral analysis tools for percussive timbre identification'. In: *Proceedings of the 3rd International Pure Data Convention, Sao Paulo, Brazil*. sn. 2009 (cited on page 88).
[310] Antti Eronen and Anssi Klapuri. 'Musical instrument recognition using cepstral coefficients and temporal features'. In: *2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 00CH37100)*. Vol. 2. IEEE. 2000, pp. II753-II756 (cited on page 88).

[311] Olivier Lartillot and Petri Toiviainen. 'A Matlab toolbox for musical feature extraction from audio'. In: *International conference on digital audio effects*. Bordeaux. 2007, pp. 237-244 (cited on page 88).

[312] Stephen McAdams and Bruno L Giordano. 'The perception of musical timbre'. In: *The Oxford handbook of music psychology* (2009), pp. 72-80 (cited on page 88).

[313] Kai Siedenburg, Ichiro Fujinaga, and Stephen McAdams. 'A Comparison of Approaches to Timbre Descriptors in Music Information Retrieval and Music Psychology'. In: *Journal of New Music Research* 45.1 (2016), pp. 27-41. DOI: 10.1080/09298215.2015.1132737 (cited on page 88).

[314] Anne Caclin et al. 'Acoustic correlates of timbre space dimensions: A confirmatory study using synthetic tones'. In: *The Journal of the Acoustical Society of America* 118.1 (2005), pp. 471-482. DOI: 10.1121/1.1929229 (cited on page 88).

[315] Stephen McAdams and Jean-Christophe Cunible. 'Perception of Timbral Analogies'. In: *The Royal Society* 367.1587 (1993), pp. 430-438 (cited on page 88).

[316] Reinier Plomp. 'Auditory analysis and timbre perception'. In: *Auditory analysis and perception of speech* (1975), pp. 7-22 (cited on page 89).

[317] J Douglas Carroll and Jih-Jie Chang. 'Analysis of individual differences in multidimensional scaling via an N-way generalization of "Eckart-Young" decomposition'. In: *Psychometrika* 35.3 (1970), pp. 283-319 (cited on page 89).

[318] Arnab Ghosh et al. 'Multi-Agent Diverse Generative Adversarial Networks'. In: *arXiv preprint arXiv:1704.02906* (2017) (cited on page 89).

[319] Paul Iverson and Carol L Krumhansl. 'Isolating the dynamic attributes of musical timbrea'. In: *The Journal of the Acoustical Society of America* 94.5 (1993), pp. 2595-2603 (cited on page 89).

[320] Stephen McAdams et al. 'Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes'. In: *Psychological Research* 58.3 (1995), pp. 177-192. DOI: 10.1007/BF00419633 (cited on page 89).

[321] Stephen Lakatos. 'A common perceptual space for harmonic and percussive timbres'. In: *Perception & psychophysics* 62.7 (2000), pp. 1426-1439 (cited on page 89).

[322] Philippe Esling, Axel Chemla-Romeu-Santos, and Adrien Bitton. 'Bridging Audio Analysis, Perception and Synthesis with Perceptually-regularized Variational Timbre Spaces.' In: *ISMIR*. 2018, pp. 175-181 (cited on page 91).
[323] Douglas O'Shaughnessy. 'Linear predictive coding'. In: *IEEE potentials* 7.1 (1988), pp. 29–32 (cited on page 102).

[324] W Bastiaan Kleijn. 'Continuous representations in linear predictive coding'. In: [*Proceedings ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing*. IEEE. 1991, pp. 201–204 (cited on page 102).

[325] HS Peter Yue and Rafi Rabipour. *Methods and apparatus for noise conditioning in digital speech compression systems using linear predictive coding*. US Patent 5,642,464. June 1997 (cited on page 102).

[326] Masayuki Nishiguchi and Jun Matsumoto. 'Harmonic and noise coding of LPC residuals with classified vector quantization'. In: *1995 International Conference on Acoustics, Speech, and Signal Processing*. Vol. 1. IEEE. 1995, pp. 484–487 (cited on page 102).

[327] Kevin Patrick Murphy. 'Dynamic Bayesian Networks: Representation, Inference and Learning'. In: (1994) (cited on page 103).

[328] Clare A McGrory and DM Titterington. 'Variational Bayesian analysis for hidden Markov models'. In: *Australian & New Zealand Journal of Statistics* 51.2 (2009), pp. 227–244 (cited on page 103).

[329] Nick Foti et al. 'Stochastic variational inference for hidden Markov models'. In: *Advances in neural information processing systems*. 2014, pp. 3599–3607 (cited on page 103).

[330] Carl Edward Rasmussen and Christopher K I Williams. *Gaussian processes for machine learning*. Vol. 1. MIT press Cambridge, 2006 (cited on page 104).

[331] Matthias Seeger. 'Bayesian Gaussian Process Models: PAC-Bayesian Generalisation Error Bounds and Sparse Approximations'. In: (2004). DOI: 10.1162/153244303765208377 (cited on page 104).

[332] Michalis K. Tsitsias. 'Variational Learning of Inducing Variables in Sparse Gaussian Processes'. In: *Artificial Intelligence and Statistics* 5 (2009), pp. 567–574. DOI: 10.1117/12.591679 (cited on page 104).

[333] Andrew Gordon Wilson, David A. Knowles, and Zoubin Ghahramani. 'Gaussian process regression networks'. In: *Proceedings of the 29th International Conference on Machine Learning*, ICML 2012 1 (2012), pp. 599–606 (cited on page 104).

[334] Neil Lawrence. 'Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models'. In: *Journal of Machine Learning Research* 6 (2005), pp. 1783–1816 (cited on page 105).

[335] Jack M. Wang, David J. Fleet, and Aaron Hertzmann. 'Gaussian process dynamical models'. In: *Advances in Neural Information Processing Systems* (2005), pp. 1441–1448 (cited on page 105).
[336] Stefanos Eleftheriadis et al. 'Identification of Gaussian process state space models'. In: *Advances in Neural Information Processing Systems* 2017-Decem.2014 (2017), pp. 5310–5320 (cited on page 105).

[337] Andreas Doerr et al. 'Probabilistic Recurrent State-Space Models'. In: *35th International Conference on Machine Learning*, ICML 2018 3 (2018), pp. 2060–2075 (cited on page 105).

[338] Lincoln C Mattos et al. 'Recurrent Gaussian Processes'. In: *3* (2016), pp. 1–12 (cited on page 105).

[339] David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. 'Learning representations by back-propagating errors'. In: *Cognitive modeling* 5.3 (1988), p. 1 (cited on page 106).

[340] Jeffrey L. Elman. 'Finding structure in time'. In: *Cognitive Science* 14.2 (1990), pp. 179–211. DOI: 10.1016/0364-0213(90)90002-E (cited on page 106).

[341] Randall D. Beer. 'On the Dynamics of Small Continuous-Time Recurrent Neural Networks'. In: *Adaptive Behavior* 3.4 (1995), pp. 469–509. DOI: 10.1177/105971239500300405 (cited on page 106).

[342] Kenji Doya. 'Bifurcations of recurrent neural networks in gradient descent learning'. In: *IEEE Transactions on neural networks* 1.75 (1993), p. 164 (cited on page 106).

[343] Herbert Jaeger and Harald Haas. 'Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication'. In: *Science* 304.5667 (2004), pp. 78–80. DOI: 10.1126/science.1091277 (cited on page 106).

[344] Alex M Lamb et al. 'Professor forcing: A new algorithm for training recurrent networks'. In: *Advances In Neural Information Processing Systems*. 2016, pp. 4601–4609 (cited on page 106).

[345] Salah El Hihi and Yoshua Bengio. 'Hierarchical Recurrent Neural Networks for Long-Term Dependencies.' In: *Nips* (1995), pp. 493–499 (cited on page 106).

[346] Sepp Hochreiter and Jürgen Schmidhuber. 'Long short-term memory'. In: *Neural computation* 9.8 (1997), pp. 1735–1780 (cited on page 107).

[347] Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins. 'Learning to forget: Continual prediction with LSTM'. In: *Neural Computation* 12.10 (2000), pp. 2451–2471. DOI: 10.1162/089976600300015015 (cited on page 107).

[348] Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. 'Bidirectional LSTM networks for improved phoneme classification and recognition'. In: *Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)* 3697 LNCS (2005), pp. 799–804 (cited on page 107).
[349] Ashish Vaswani et al. 'Attention Is All You Need'. In: Nips (2017). DOI: 10.1017/S0952523813000308 (cited on page 107).

[350] Sneha Chaudhari et al. 'An Attentive Survey of Attention Models'. In: (2019) (cited on page 107).

[351] Kazuki Irie et al. 'Language Modeling with Deep Transformers'. In: () (cited on page 107).

[352] Mostafa Dehghani et al. 'Universal Transformers'. In: (2018), pp. 1-23 (cited on page 107).

[353] Kyunghyun Cho et al. 'On the properties of neural machine translation: Encoder-decoder approaches'. In: arXiv preprint arXiv:1409.1259 (2014) (cited on page 107).

[354] Moritz Wolter and Angela Yao. 'FOURIER RNNS FOR SEQUENCE ANALYSIS AND PREDICTION'. In: (2018) (cited on page 107).

[355] Marta Garnelo et al. 'Neural Processes'. In: (2018) (cited on page 107).

[356] Timon Willi et al. 'Recurrent Neural Processes'. In: (2019) (cited on page 107).

[357] Chao Ma, Yingzhen Li, and José Miguel Hernández-Lobato. 'Variational Implicit Processes'. In: (2018) (cited on page 107).

[358] Hugo Larochelle and Iain Murray. 'The neural autoregressive distribution estimator'. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. 2011, pp. 29-37 (cited on page 108).

[359] Tapani Raiko et al. 'Iterative neural autoregressive distribution estimator nadek'. In: Advances in neural information processing systems. 2014, pp. 325-333 (cited on page 108).

[360] Soroush Mehri et al. 'SampleRNN: An Unconditional End-to-End Neural Audio Generation Model'. In: (2016), pp. 1-11 (cited on page 108).

[361] Albert Haque, Michelle Guo, and Prateek Verma. 'Conditional End-to-End Audio Transforms'. In: i (2018), pp. 1-5 (cited on page 109).

[362] Jesse Engel et al. 'Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders'. In: arXiv preprint arXiv:1704.01279 (2017) (cited on page 109).

[363] Evan Archer et al. 'Black box variational inference for state space models'. In: (2015), pp. 1-11. DOI: 10.1038/nature08121 (cited on page 110).

[364] Rahul G. Krishnan, Uri Shalit, and David Sontag. 'Deep Kalman Filters'. In: 2000 (2015), pp. 1-17 (cited on page 110).

[365] Otto Fabius and Joost R. van Amersfoort. 'Variational Recurrent Auto-Encoders'. In: (2014), pp. 1-5 (cited on pages 110, 111).
[366] Justin Bayer and Christian Osendorfer. 'Variational inference of latent state sequences using recurrent networks'. In: *arXiv preprint arXiv:1406.1655* (2014), pp. 1-12 (cited on pages 110, 111).

[367] Iulian Vlad Serban et al. 'A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues.' In: *AAAI*. 2017, pp. 3295-3301 (cited on page 110).

[368] Karol Gregor et al. 'Temporal Difference Variational Auto-Encoder'. In: (2018), pp. 1-17 (cited on page 111).

[369] Marco Fraccaro et al. 'Sequential neural models with stochastic layers'. In: *Advances in Neural Information Processing Systems* (2016), pp. 2199-2207 (cited on page 111).

[370] Anirudh Goyal, Alessandro Sordoni, and Nan Rosemary Ke. 'Z-Forcing: Training Stochastic Recurrent Networks'. In: Nips (2017) (cited on page 111).

[371] Samira Shabanian et al. 'Variational Bi-LSTMs'. In: (2017), pp. 1-12 (cited on page 111).

[372] Justin Bayer and Christian Osendorfer. 'Variational inference of latent state sequences using recurrent networks'. In: *stat* 1050 (2014), p. 6 (cited on pages 111, 113).

[373] Junyoung Chung et al. 'A recurrent latent variable model for sequential data'. In: *Advances in neural information processing systems*. 2015, pp. 2980-2988 (cited on pages 111, 113).

[374] Yingzhen Li and Stephan Mandt. 'Disentangled Sequential Autoencoder'. In: (2018) (cited on page 112).

[375] Will Grathwohl and Aaron Wilson. 'Disentangling Space and Time in Video with Hierarchical Variational Auto-encoders'. In: *arXiv preprint arXiv:1612.04440* (2016) (cited on page 112).

[376] Aaron Van Den Oord. 'Representation Learning with Contrastive Predictive Coding'. In: () (cited on page 114).

[377] Geoffrey E Hinton, Terrance Sejnowski, and David Ackley. *Boltzman Machines*. 1984 (cited on page 114).

[378] Ruslan Salakhutdinov and Geoffrey Hinton. 'Deep Boltzmann machines'. In: *Journal of Machine Learning Research* 5.3 (2009), pp. 448-455 (cited on page 114).

[379] Antti Rasmus et al. 'Semi-supervised learning with ladder networks'. In: *Advances in Neural Information Processing Systems*. 2015, pp. 3546-3554 (cited on page 124).

[380] Casper Kaae Sønderby et al. 'Ladder variational autoencoders'. In: *Advances in Neural Information Processing Systems*. 2016, pp. 3738-3746 (cited on page 125).
[381] Adam Paszke et al. 'PyTorch: An imperative style, high-performance deep learning library'. In: *Advances in Neural Information Processing Systems*. 2019, pp. 8024–8035 (cited on page 133).

[382] Rebecca Fiebrink and Perry R Cook. 'The Wekinator: a system for real-time, interactive machine learning in music'. In: *Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010)(Utrecht)*. 2010 (cited on page 135).

[383] Sølvi Ystad, Mitsuko Aramaki, and Richard Kronland-Martinet. *Timbre from Sound Synthesis and High-Level Control Perspectives*. 2019, pp. 361–389 (cited on page 138).

[384] Garrett Warnell et al. 'Deep TAMER: Interactive agent shaping in high-dimensional state spaces'. In: *32nd AAAI Conference on Artificial Intelligence, AAAI 2018* (2018), pp. 1545–1553 (cited on page 138).

[385] Hugo Scurto, Frédéric Bevilacqua, and Baptiste Caramiaux. 'Perceiving Agent Collaborative Sonic Exploration In Interactive Reinforcement Learning'. In: 2018 (cited on page 139).

[386] Hugo Scurto et al. 'Designing Deep Reinforcement Learning for Human Parameter Exploration'. In: *arXiv preprint arXiv:1907.00824* (2019) (cited on page 139).

[387] Gérard Assayag et al. 'Omax brothers: a dynamic yopology of agents for im-provization learning'. In: *Proceedings of the 1st ACM workshop on Audio and music computing multimedia*. ACM. 2006, pp. 125–132 (cited on page 139).

[388] Atau Tanaka and Marco Donnarumma. 'The body as musical instrument'. In: *The Oxford Handbook of Music and the Body* (2019), p. 79 (cited on page 140).

[389] Philippe Esling et al. 'Universal audio synthesizer control with normalizing flows'. In: *CoRR abs/1907.00971* (2019) (cited on page 145).