<figure><img src="image_1.png" /></figure>

HAL
open science

# Manifold representations of musical signals and generative spaces

Axel Chemla-Romeu-Santos

► To cite this version:

Axel Chemla-Romeu-Santos. Manifold representations of musical signals and generative spaces. Sound [cs.SD]. Università Degli Studi di Milano; Sorbonne Université, 2020. English. NNT: tel-03543235

HAL Id: tel-03543235

https://hal.science/tel-03543235

Submitted on 4 Feb 2022

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
PhD thesis in partnership

Manifold representations of musical signals
and generative spaces

Axel Chemla-Romeu-Santos

Matricolo : R11707
Academic year 2018-2019
XXXIIth cycle
PhD defense : Jan 30, 2020

Director :
Goffredo HAUS

Director:
Gérard ASSAYAG

Co-director :
Philippe ESLING

Coordinator :
Paolo BOLDI

Coordinator :

Jury committee :

Pr. Antonio CAMURRI (president)

Pr. Federico FONTANA

Pr. Antonio RODÁ

Pr. Goffredo HAUS

Pr. Nicolas BODINI

Pr. Nicolas BREDECHE

Pr. Geoffroy PEETERS

Pr. Philippe ESLING
<figure><img src="image_2.png" /></figure>
# Details

This PhD, funded by the Computer Science Department of Università Degli Studi di Milano in the Laboratorio d'Informatica Musicale (LIM), has been accomplished in partnership with the doctoral school EDITE of Sorbonne Universités, in the IRCAM STMS lab.

## Referees

### Advisors

* Pr. Goffredo HAUS

* Pr. Gérard ASSAYAG

### Co-advisor

* Pr. Philippe ESLING

### Reviewing Committee

* Pr. Shlomo DUBNOV

* Pr. Davide ROCCHESSO

* Pr. Michèle SEBAG

### Defense Committee

* Pr. Antonio CAMURRI (president)

* Pr. Federico FONTANA

* Dr. Antonio RODÁ

* Pr. Goffredo HAUS

* Pr. Geoffroy PEETERS

* Pr. Nicolas BREDECHE

* Pr. Olivier BODINI

* Pr. Philippe ESLING
Abstract (english)

Among the diverse research fields within computer music, synthesis and generation of audio signals epitomize the cross-disciplinarity of this domain, jointly nourishing both scientific and artistic practices since its creation. Inherent in computer music since its genesis, audio generation has inspired numerous approaches, evolving both with musical practices and scientific/technical advances. Moreover, some synthesis processes also naturally handle the reverse process, named *analysis*, such that synthesis parameters can also be partially or totally extracted from actual sounds, and providing an alternative *representation* of the analyzed audio signals.

On top of that, the recent rise of machine learning algorithms earnestly questioned the field of scientific research, bringing powerful data-centred methods that raised several epistemological questions amongst researchers, in spite of their efficiency. Especially, a family of machine learning methods, called *generative models*, are focused on the generation of original content using features extracted from an existing dataset. In that case, such methods not only questioned previous approaches in generation, but also the way of integrating this methods into existing creative processes. While these new generative frameworks are progressively introduced in the domain of image generation, the application of such generative techniques in audio synthesis is still marginal.

In this work, we aim to propose a new audio analysis-synthesis framework based on these modern generative models, enhanced by recent advances in machine learning. We first review existing approaches, both in sound synthesis and in generative machine learning, and focus on how our work inserts itself in both practices and what can be expected from their collation. Subsequently, we focus a little more on generative models, and how modern advances in the domain can be exploited to allow us learning complex sound distributions, while being sufficiently flexible to be integrated in the creative flow of the user. We then propose an *inference / generation* process, *mirroring* analysis/synthesis paradigms that are natural in the audio processing domain, using *latent models* that are based on a continuous higher-level space, that we use to *control* the generation. We first provide preliminary results of our method applied on spectral information, extracted from several datasets, and evaluate both qualitatively and quantitatively the obtained results. Subsequently, we study how to make these methods more suitable for learning audio data, tackling successively three different aspects. First, we propose two different latent regularization strategies specifically designed for audio, based on and signal / symbol translation and perceptual constraints. Then, we propose different methods to address the inner *temporality* of musical signals, based on the extraction of multi-scale representations and on prediction, that allow the obtained generative spaces that also model the dynamics of the signal.

As a last chapter, we swap our scientific approach to a more *research & creation*-oriented point of view: first, we describe the architecture and the design of our open-source library, vsacids, aiming to be used by expert and non-expert music makers as an integrated creation tool. Then, we propose an first musical use of our system by the creation of a real-time performance, called *aego*, based jointly on our framework vsacids and an explorative agent using reinforcement learning to be trained during the performance. Finally, we draw some conclusions on the different manners to improve and reinforce the proposed generation method, as well as possible further creative applications.
Abstract (italiano)

Tra i diversi campi di ricerca nell'ambito dell'informatica musicale, la sintesi e la generazione di segnali audio incarna la pluridisciplinalità di questo settore, nutrendo insieme le pratiche scientifiche e musicale dalla sua creazione. Inerente all'informatica dalla sua creazione, la generazione audio ha ispirato numerosi approcci, evolvendo colle pratiche musicale e gli progressi tecnologici e scientifici. Inoltre, alcuni processi di sintesi permettono anche il processo inverso, denominato *analisi*, in modo che i parametri di sintesi possono anche essere parzialmente o totalmente estratti dai suoni, dando una rappresentazione alternativa ai segnali analizzati.

Per di più, la recente ascesa dei algoritmi di l'apprendimento automatico ha vivamente interrogato il settore della ricerca scientifica, fornendo potenti *data-centered* metodi che sollevavano diversi epistemologici interrogativi, nonostante i sui efficacia. Particolarmente, un tipo di metodi di apprendimento automatico, denominati *modelli generativi*, si concentrano sulla generazione di contenuto originale usando le caratteristiche che hanno estratti dei dati analizzati. In tal caso, questi modelli non hanno soltanto interrogato i precedenti metodi di generazione, ma anche sul modo di integrare questi algoritmi nelle pratiche artistiche. Mentre questi metodi sono progressivamente introdotti nel settore del trattamento delle immagini, la loro applicazione per la sintesi di segnali audio e ancora molto marginale.

In questo lavoro, il nostro obiettivo e di proporre un nuovo metodo di audio sintesi basato su questi nuovi tipi di generativi modelli, rafforazzi dalle nuove avanzati dell'apprendimento automatico. Al primo posto, facciamo una revisione dei approcci esistenti nei settori dei sistemi generativi e di sintesi sonore, focalizzando sul posto di nostro lavoro rispetto a questi disciplini e che cosa possiamo aspettare di questa collazione. In seguito, studiamo in maniera più precisa i modelli generativi, e come possiamo utilizzare questi recenti avanzati per l'apprendimento di complesse distribuzione di suoni, in un modo che sia flessibile e nel flusso creativo del utente. Quindi proponiamo un processo di inferenza / generazione, il quale rifletta i processi di analisi/sintesi che sono molto usati nel settore del trattamento del segnale audio, usando modelli latenti, che sono basati sull'utilizzazione di un spazio continuato di alto livello, che usiamo per *controllare* la generazione. Studiamo dapprima i risultati preliminari ottenuti con informazione spettrale estratte da diversi tipi di dati, che valutiamo qualitativamente e quantitativamente. Successivamente, studiamo come fare per rendere questi metodi più adattati ai segnali audio, fronteggiando tre diversi aspetti. Primo, proponiamo due diversi metodi di regolarizzazione di questo generativo spazio che sono specificamente sviluppati per l'audio: una strategia basata sulla traduzione segnali / simboli, e una basata su vincoli percettivi. Poi, proponiamo diversi metodi per fronteggiare il aspetto *temporale* dei segnali audio, basati sull'estrazione di rappresentazioni multiscala e sulla predizione, che permettono ai generativi spazi ottenuti di anche modellare l'aspetto dinamico di questi segnali.

Per finire, cambiamo il nostro approccio scientifico per un punto di visto più ispirato dall'idea di ricerca e creazione. Primo, descriviamo l'architettura e il design della nostra libreria open-source, vsacids, sviluppata per permettere a esperti o non-esperti musicisti di provare questi nuovi metodi di sintesi. Poi, proponiamo una prima utilizzazione del nostro modello con la creazione di una performance in real-time, chiamata *ægo*, basata insieme sulla nostra libreria vsacids e sull'uso di une agente di esplorazione, imparando con rinforzo nel corso della composizione. Finalmente, tramo dal lavoro presentato alcuni conclusioni sui diversi modi di migliorare e rinforzare il metodo di sintesi proposto, nonché eventuale applicazione artistiche.
Abstract (français)

À travers les différents domaines de recherche de la musique computationnelle, l'analysie et la génération de
signaux audio sont l'exemple parfait de la trans-disciplinarité de ce domaine, nourrissant simultanément
les pratiques scientifiques et artistiques depuis leur création. Intégrée à la musique computationnelle
depuis sa création, la synthèse sonore a inspiré de nombreuses approches musicales et scientifiques,
évoluant de pair avec les pratiques musicales et les avancées technologiques et scientifiques de son temps.
De plus, certaines méthodes de synthèse sonore permettent aussi le processus inverse, appelé analyse,
de sorte que les paramètres de synthèse d'un certain générateur peuvent être en partie ou entièrement
obtenus à partir de sons donnés, pouvant ainsi être considérés comme une représentation alternative
des signaux analysés. Parallèlement, l'intérêt croissant soulevé par les algorithmes d'apprentissage au-
tomatique a vivement questionné le monde scientifique, apportant de puissantes méthodes d'analyse
de données suscitant de nombreux questionnements épistémologiques chez les chercheurs, en dépit de
leur effectivité pratique. En particulier, une famille de méthodes d'apprentissage automatique, nommée
modèles génératifs, s'intéressent à la génération de contenus originaux à partir de caractéristiques extraites
directement des données analysées. Ces méthodes n'interrogent pas seulement les approches précédentes,
mais aussi sur l'intégration de ces nouvelles méthodes dans les processus créatifs existants. Pourtant,
alors que ces nouveaux processus génératifs sont progressivement intégrés dans le domaine la génération
d'image, l'application de ces techniques en synthèse audio reste marginale.

Dans cette thèse, nous proposons une nouvelle méthode d'analyse-synthèse basés sur ces derniers
modèles génératifs, depuis renforcés par les avancées modernes dans le domaine de l'apprentissage
automatique. Dans un premier temps, nous examinerons les approches existantes dans le domaine des
systèmes génératifs, sur comment notre travail peut s'insérer dans les pratiques de synthèse sonore
existantes, et que peut-on espérer de l'hybridation de ces deux approches. Ensuite, nous nous focaliserons
plus précisément sur comment les récentes avancées accomplies dans ce domaine dans ce domaine
peuvent être exploitées pour l'apprentissage de distributions sonores complexes, tout en étant suffisam-
ment flexibles pour être intégrées dans le processus créatif de l'utilisateur. Nous proposons donc un
processus d'inférence / generation, reflétant les paradigmes d'analyse-synthèse existant dans le domaine
de génération audio, basé sur l'usage de modèles latents continus que l'on peut utiliser pour contrôler la
génération. Pour ce faire, nous étudierons déjà les résultats préliminaires obtenus par cette méthode
sur l'apprentissage de distributions spectrales, prises d'ensembles de données diversifiés, en adoptant
une approche à la fois quantitative et qualitative. Ensuite, nous proposerons d'améliorer ces méthodes
de manière spécifique à l'audio sur trois aspects distincts. D'abord, nous proposons deux stratégies de
régularisation différentes pour l'analyse de signaux audio : une basée sur la traduction signal/symbole,
ainsi qu'une autre basée sur des contraintes perceptuelles. Nous passerons par la suite à la dimension
temporelle de ces signaux audio, proposant de nouvelles méthodes basées sur l'extraction de représenta-
tions temporelles multi-échelle et sur une tâche supplémentaire de prédiction, permettant la modélisation
de caractéristiques dynamiques par les espaces génératifs obtenus.

En dernier lieu, nous passerons d'une approche scientifique à une approche plus orientée vers un
point de vue *recherche & création*. Premièrement, nous présenterons notre librairie open-source, vsacids,
visant à être employée par des créateurs experts et non-experts comme un outil intégré. Ensuite, nous
proposons une première utilisation musicale de notre système par la création d'une performance temps
réel, nommée ægo, basée à la fois sur notre librarie et sur un agent d'exploration appris dynamiquement
par renforcement au cours de la performance. Enfin, nous tirons les conclusions du travail accompli
jusqu'à maintenant, concernant les possibles améliorations et développements de la méthode de synthèse proposée, ainsi que sur de possibles applications créatives.
Acknowledgements

First, I would like to warmly thank Pr. Goffredo Haus for his trust during these three years, and for having me offered the chance of working on this fascinating topic. Then, I would also like to thanks and their valuable support during these three years : Giorgio, Luca, Adriano, Stavros, and Federico for their great welcome in the LIM, and Lorena, that helped me a lot through all the difficulties of a PhD in partnership. These years at the LIM in Milano have been very interesting, and I will always be thankful for the opportunity it offered to me.

On the French side, I would like to thank Gérard, for his unwavering support since the internship he offered me 5 years ago, and of course Philippe, who I had the luck to meet during my early years at IRCAM and who, in addition to be a precious friend, also revealed to be a precious mentor (and still did not demand the divorce after 5 years, thank you for that). I would also like to thank the marvelous ACIDS team, from the old members (as me, I'm afraid) to the newest : Léopold (we are the elders my friend), Tristan, Mathieu, Adrien, Constance, and Mikhaël, the *Representations Musicales*\* team Carlos, Jean-Louis, Karim, Jean. A special thought to my "lab-mate" Hugo Scurto, whose desk contiguity was a substantial contribution in this kind of post-semantic concept that is real life. To finish, I would also like to thank Sylvie, Benjamin, Jérôme, and to apologize in advance for the people I forgot.

Finally, I would also like to sincerely thank my parents, who in addition to their substantial contribution to my simple existence have brought me a critical support during all these years, my family, present and missing ones, and of course all of my friends and music/theatre fellows, that I cannot mention here, but who definitely contributed and will contribute again to the extension of my understanding of the world.

<footnote>\* en francais dans le texte.</footnote>
# Detailed Contents

<table>
  <tr>
    <td>Details</td>
    <td>iii</td>
  </tr>
  <tr>
    <td>Abstract</td>
    <td>iv</td>
  </tr>
  <tr>
    <td>Detailed Contents</td>
    <td>ix</td>
  </tr>
  <tr>
    <td>1 Introduction and background</td>
    <td>1</td>
  </tr>
  <tr>
    <td>2 Generative models, representation learning and variational inference</td>
    <td>7</td>
  </tr>
  <tr>
    <td>2.1 Bayesian probabilistic generative models</td>
    <td>9</td>
  </tr>
  <tr>
    <td>2.1.1 Bayesian learning for statistical inference</td>
    <td>10</td>
  </tr>
  <tr>
    <td>2.1.2 Latent models and posterior approximation methods.</td>
    <td>13</td>
  </tr>
  <tr>
    <td>2.1.3 Variational methods for posterior approximation</td>
    <td>16</td>
  </tr>
  <tr>
    <td>2.1.4 Information theory and latent models</td>
    <td>22</td>
  </tr>
  <tr>
    <td>2.2 From dimensionality reduction to deep unsupervised learning</td>
    <td>28</td>
  </tr>
  <tr>
    <td>2.2.1 Dimensionality reduction</td>
    <td>30</td>
  </tr>
  <tr>
    <td>2.2.2 Neural networks and back-propagation</td>
    <td>33</td>
  </tr>
  <tr>
    <td>2.2.3 Auto-encoders as invertible representation extractors</td>
    <td>38</td>
  </tr>
  <tr>
    <td>2.3 Bridging representation learning and variational inference</td>
    <td>40</td>
  </tr>
  <tr>
    <td>2.3.1 Auto-Encoding Variational Bayes.</td>
    <td>40</td>
  </tr>
  <tr>
    <td>2.3.2 Emerging properties of variational auto-encoders</td>
    <td>43</td>
  </tr>
  <tr>
    <td>2.3.3 Discrete latent spaces.</td>
    <td>54</td>
  </tr>
  <tr>
    <td>3 Learning spectral representations and audio regularization strategies</td>
    <td>57</td>
  </tr>
  <tr>
    <td>3.1 Spectral representations of audio signals</td>
    <td>60</td>
  </tr>
  <tr>
    <td>3.1.1 Invertible spectral representations of audio signals</td>
    <td>60</td>
  </tr>
  <tr>
    <td>3.2 Audio analysis-synthesis with variational auto-encoders</td>
    <td>65</td>
  </tr>
  <tr>
    <td>3.2.1 Datasets.</td>
    <td>66</td>
  </tr>
  <tr>
    <td>3.2.2 Evaluation of generative and representational properties.</td>
    <td>69</td>
  </tr>
  <tr>
    <td>3.3 Bijective signal-symbol regularization</td>
    <td>78</td>
  </tr>
  <tr>
    <td>3.3.1 (Semi-)supervised methods for latent regularization</td>
    <td>78</td>
  </tr>
  <tr>
    <td>3.3.2 Signal-symbol translation with latent space matching</td>
    <td>81</td>
  </tr>
  <tr>
    <td>3.3.3 Evaluation of domain transfer and further applications.</td>
    <td>83</td>
  </tr>
  <tr>
    <td>3.4 Perceptual regularization of orchestral instruments</td>
    <td>87</td>
  </tr>
  <tr>
    <td>3.4.1 Audio perception and timbre spaces</td>
    <td>87</td>
  </tr>
  <tr>
    <td>3.4.2 Defining perceptual regularization and similarity space matching</td>
    <td>88</td>
  </tr>
  <tr>
    <td>3.4.3 Evaluation, perceptual inference and descriptor-based generation</td>
    <td>91</td>
  </tr>
  <tr>
    <td>3.4.4 Conclusion</td>
    <td>96</td>
  </tr>
  <tr>
    <td>4 Time and prediction in generative models</td>
    <td>98</td>
  </tr>
  <tr>
    <td>4.1 Time series analysis and prediction</td>
    <td>99</td>
  </tr>
  <tr>
    <td>4.1.1 Stochastic processes and model identification.</td>
    <td>99</td>
  </tr>
  <tr>
    <td>4.1.2 State-space models.</td>
    <td>102</td>
  </tr>
  <tr>
    <td>4.1.3 Gaussian processes.</td>
    <td>103</td>
  </tr>
  <tr>
    <td>4.2 Neural methods for time series modeling</td>
    <td>106</td>
  </tr>
  <tr>
    <td>4.2.1 Recurrent Neural Networks</td>
    <td>106</td>
  </tr>
  <tr>
    <td>4.2.2 Dynamical Bayes and AR-models</td>
    <td>107</td>
  </tr>
  <tr>
    <td>4.2.3 Variational auto-encoding for time-series</td>
    <td>109</td>
  </tr>
</table>
<table>
  <tbody>
    <tr>
      <td>4.3</td>
      <td>Variational methods for latent space prediction</td>
      <td>112</td>
    </tr>
    <tr>
      <td>4.3.1</td>
      <td>Contrastive Predictive Coding</td>
      <td>114</td>
    </tr>
    <tr>
      <td>4.3.2</td>
      <td>Filtration learning with temporal normalizing flows</td>
      <td>116</td>
    </tr>
    <tr>
      <td>4.3.3</td>
      <td>Gaussian processes regularization</td>
      <td>117</td>
    </tr>
    <tr>
      <td>4.3.4</td>
      <td>Learning and evaluation framework.</td>
      <td>119</td>
    </tr>
    <tr>
      <td>4.4</td>
      <td>Hierarchical latent spaces and raw waveform learning.</td>
      <td>123</td>
    </tr>
    <tr>
      <td>4.4.1</td>
      <td>ShrubVAE: Multi-layered models for multi-scale learning</td>
      <td>123</td>
    </tr>
    <tr>
      <td>4.4.2</td>
      <td>Learning procedure and progressive hierarchical warm-up</td>
      <td>127</td>
    </tr>
    <tr>
      <td>5</td>
      <td>Composing and performing with generative spaces</td>
      <td>132</td>
    </tr>
    <tr>
      <td>5.1</td>
      <td>vsacids: a toolbox for variational generation of musical signals</td>
      <td>132</td>
    </tr>
    <tr>
      <td>5.1.1</td>
      <td>Architecture and design</td>
      <td>133</td>
    </tr>
    <tr>
      <td>5.1.2</td>
      <td>Workflow</td>
      <td>135</td>
    </tr>
    <tr>
      <td>5.2</td>
      <td>A step towards research and creation process: aego.</td>
      <td>137</td>
    </tr>
    <tr>
      <td>5.2.1</td>
      <td>Reinforcement Learning for Sonic Exploration</td>
      <td>138</td>
    </tr>
    <tr>
      <td>5.2.2</td>
      <td>Instrument Design</td>
      <td>139</td>
    </tr>
    <tr>
      <td>5.2.3</td>
      <td>Engineering</td>
      <td>139</td>
    </tr>
    <tr>
      <td>5.2.4</td>
      <td>Aesthetics and writing</td>
      <td>140</td>
    </tr>
    <tr>
      <td colspan="2">Conclusion</td>
      <td>143</td>
    </tr>
    <tr>
      <td colspan="2">List of publications</td>
      <td>146</td>
    </tr>
    <tr>
      <td colspan="2">Conclusion</td>
      <td>146</td>
    </tr>
    <tr>
      <td colspan="2">List of used acronyms</td>
      <td>147</td>
    </tr>
    <tr>
      <td colspan="2">APPENDICES</td>
      <td>148</td>
    </tr>
    <tr>
      <td>A</td>
      <td>Figures and tables for baseline results</td>
      <td>148</td>
    </tr>
    <tr>
      <td>A.1</td>
      <td>Benchmark reconstruction results.</td>
      <td>148</td>
    </tr>
    <tr>
      <td>A.1.1</td>
      <td>Dimensions benchmark</td>
      <td>148</td>
    </tr>
    <tr>
      <td>A.1.2</td>
      <td>Divergences benchmark</td>
      <td>150</td>
    </tr>
    <tr>
      <td>A.1.3</td>
      <td>β benchmarks.</td>
      <td>151</td>
    </tr>
    <tr>
      <td>A.2</td>
      <td>Regularization benchmarks.</td>
      <td>151</td>
    </tr>
    <tr>
      <td>A.2.1</td>
      <td>Dimensions benchmark</td>
      <td>151</td>
    </tr>
    <tr>
      <td>A.2.2</td>
      <td>Divergence regularization results.</td>
      <td>153</td>
    </tr>
    <tr>
      <td>A.2.3</td>
      <td>β-benchmarks.</td>
      <td>153</td>
    </tr>
    <tr>
      <td>A.3</td>
      <td>Disentangling.</td>
      <td>154</td>
    </tr>
    <tr>
      <td>A.3.1</td>
      <td>toy_additive</td>
      <td>154</td>
    </tr>
    <tr>
      <td>A.3.2</td>
      <td>toy_fm</td>
      <td>154</td>
    </tr>
    <tr>
      <td>A.3.3</td>
      <td>acidsInstruments-ordinario</td>
      <td>155</td>
    </tr>
  </tbody>
</table>
Figures

<table>
  <tbody>
    <tr>
      <td>2.1</td>
      <td>Supervised approaches vs. unsupervised approaches.</td>
      <td>8</td>
    </tr>
    <tr>
      <td>2.2</td>
      <td>Descriptive and inferential statistics</td>
      <td>10</td>
    </tr>
    <tr>
      <td>2.3</td>
      <td>Difference between frequentist and Bayesian observation processes.</td>
      <td>10</td>
    </tr>
    <tr>
      <td>2.4</td>
      <td>Maximum Likelihood estimation (blue) and Maximum A Posteriori estimation (red). ML gives a point-estimate of the mean, while MAP finds the maximum over a distribution of possible means.</td>
      <td>12</td>
    </tr>
    <tr>
      <td>2.5</td>
      <td>Example : a Beta(α, β) prior equally distributed between 0 and 1 shots (blue). Conjugate posterior after one more observation Beta(α, β + 1). As beta is a natural mean conjugate of Bernoulli distribution, after one new "1" observation the distribution is a new Beta, with β &lt; β + 1</td>
      <td>13</td>
    </tr>
    <tr>
      <td>2.6</td>
      <td>Conditional graph between observed variables (x, in gray) and unobserved variables (z, in gray)</td>
      <td>14</td>
    </tr>
    <tr>
      <td>2.7</td>
      <td>Generation (top) and inference (down) probabilistic graphs of a joint distribution p(x<sub>1</sub>, z<sub>1</sub>, z<sub>2</sub>, z<sub>3</sub>)</td>
      <td>14</td>
    </tr>
    <tr>
      <td>2.8</td>
      <td>Example of Markov random field, where each node is linked by a weight w<sub>ij</sub>. No direction is assumed, and the probability of the all system is maximized in turn.</td>
      <td>15</td>
    </tr>
    <tr>
      <td>2.9</td>
      <td>Decomposition of log-likelihood p(x|θ) with any distribution q(z). Increasing S(q, θ) reduces the divergence D<sub>KL</sub>[q(z) || p(z|x, θ)], such that q(z) = p(z|x, θ).</td>
      <td>16</td>
    </tr>
    <tr>
      <td>2.10</td>
      <td>Assuming the Independence between z<sub>1</sub> and z<sub>2</sub> allows to optimize separately the components of the distribution, optimizing in turn until convergence.</td>
      <td>17</td>
    </tr>
    <tr>
      <td>2.11</td>
      <td>Example of hierarchical stick-breaking process, allowing to define arbitrary partitions over a probability space</td>
      <td>19</td>
    </tr>
    <tr>
      <td>2.12</td>
      <td>Auxiliary variables allow to increase the expressiveness of the inference model, but does not modify yet the generative distribution.</td>
      <td>19</td>
    </tr>
    <tr>
      <td>2.13</td>
      <td>Density ratio Estimation uses a classifier to separate one distribution from the other, and replaces the original probabilities by classification rates</td>
      <td>20</td>
    </tr>
    <tr>
      <td>2.14</td>
      <td>Effect of D<sub>KL</sub> asymmetry on with variational distributions. As q(z) cannot model p(z), the effect of D<sub>KL</sub> asymmetry entails a sub-optimal distribution matching, whether mode-focusing (above) or mass-covering in the other (below).</td>
      <td>20</td>
    </tr>
    <tr>
      <td>2.15</td>
      <td>Optimal transport studies the distribution over mappings Γ(x, z) to transform p(z) in q(z) that minimizes the cost c(x, G(x))</td>
      <td>22</td>
    </tr>
    <tr>
      <td>2.16</td>
      <td>Rate-distortion trade-off : given a distribution p(x, z), there is an intrinsic trade-off between optimal rate R and optimal distortion D, threshold by the entropy H.</td>
      <td>25</td>
    </tr>
    <tr>
      <td>2.17</td>
      <td>Bits-back in MDL formulation of variational inference</td>
      <td>27</td>
    </tr>
    <tr>
      <td>2.18</td>
      <td>PCA will capture the axis that maximizes the variance of the original data</td>
      <td>30</td>
    </tr>
    <tr>
      <td>2.19</td>
      <td>MDS tries to matches the distances of the original space in the embedded space</td>
      <td>31</td>
    </tr>
    <tr>
      <td>2.20</td>
      <td>A single neuron, by drawing a straight line in a multi-dimensional space, is able to split it two parts. Adding a non-linearity allows, a hard tanh in the above example, to define a function that equals -1 if far above the line, and +1 far below.</td>
      <td>34</td>
    </tr>
    <tr>
      <td>2.21</td>
      <td>Several neurons with classification units can perform complex segmentation of the input space, and can then perform multi-class classification with sigmoid units.</td>
      <td>34</td>
    </tr>
    <tr>
      <td>2.22</td>
      <td>The well-known XOR example : a problem that is non-separable in an input space can become separable with additional dimensions.</td>
      <td>35</td>
    </tr>
    <tr>
      <td>2.23</td>
      <td>Schema of gradient back-propagation through two neuron layers, using the chain rule to obtain the derivative of the loss function with respect to a single neuron weight.</td>
      <td>35</td>
    </tr>
    <tr>
      <td>2.24</td>
      <td>Example of residual network, adding skip connections to inject gradient information in deeper layers</td>
      <td>36</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>2.25</td>
      <td>Graph of an auto-encoder, that is built from two parts: an encoding function, generally a neural network followed by a sigmoid non-linearity, and a decoding function, that inverts the representation to recover the corresponding data. The system is then trained using an ℓ² loss between the original example and its reconstruction.</td>
      <td>38</td>
    </tr>
    <tr>
      <td>2.26</td>
      <td>The denoising auto-encoder, by adding a noise with fixed variance to the input, enforces the auto-encoder to learn how to project a neighboring zone of the data manifold to its correct reconstruction.</td>
      <td>39</td>
    </tr>
    <tr>
      <td>2.27</td>
      <td>The reparametrization of samples allows to separate the gradient coming from the stochastic and deterministic components.</td>
      <td>41</td>
    </tr>
    <tr>
      <td>2.28</td>
      <td>The variational auto-encoder, instead of defining a point-estimate of data x and latent code z, rather define the parameter of the generative and variational distributions.</td>
      <td>42</td>
    </tr>
    <tr>
      <td>2.29</td>
      <td>The KL-divergence between true and approximated posteriors can be divided in two parts: an amortization gap, that represent the gap between the current approximated posterior and the optimal one, and the gap between the optimal approximation and the true posterior.</td>
      <td>43</td>
    </tr>
    <tr>
      <td>2.30</td>
      <td>Normalizing flows allow to mode complex posterior distributions with a transformation chain from z₀</td>
      <td>45</td>
    </tr>
    <tr>
      <td>2.31</td>
      <td>Importance weighting samples several samples from the distribution q(z|x), that are then weighted by their probability in the final loss.</td>
      <td>47</td>
    </tr>
    <tr>
      <td>2.32</td>
      <td>(a) an auto-encoder associates to a code z to a given example x, but is not trained between the examples. (b) a well-balances VAE enforces the distributions to overlap, being able to find mixtures between the two distributions (c) if the recognition model q(z|x) matches the prior, the variables z and x are independent.</td>
      <td>48</td>
    </tr>
    <tr>
      <td>2.33</td>
      <td>Matching the aggregated prior q(z) instead of q(z|x) allows to prevent over-pruning, as its matching the overall distribution rather than the posterior.</td>
      <td>50</td>
    </tr>
    <tr>
      <td>2.34</td>
      <td>Disentangling: given two factors of variation (running and color), here neither axis are covariant with axis of latent space, and are not orthogonal between them.</td>
      <td>51</td>
    </tr>
    <tr>
      <td>2.35</td>
      <td>Evaluating the true underlying factors of a given dataset implies not only to extract the exact sub-manifold from the original space, but also to infer its underlying metrics.</td>
      <td>52</td>
    </tr>
    <tr>
      <td>2.36</td>
      <td>Relaxation of discrete probability function: (a) discrete probability function p(x) (b) obtain cumulative distribution function ∫ p(x)dx (c) relaxation by approximating the CDF with a continuous distribution. Provided its tractable reciprocal, this distribution can be sampled.</td>
      <td>54</td>
    </tr>
    <tr>
      <td>2.37</td>
      <td>Semi-supervised learning: a supervised task is performed in the latent space, and the extracted label is given to the decoder for class-dependent generation.</td>
      <td>55</td>
    </tr>
    <tr>
      <td>3.1</td>
      <td>The inference/generation processes of AEVB can be thought as mirroring a DSP analysis/synthesis system, extracting invertible features from a given corpus of audio signals.</td>
      <td>57</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>Most MIR approaches extract acoustical features from the signal, and/or from signal representations to extract higher-level features, as could be used here from chord detection.</td>
      <td>58</td>
    </tr>
    <tr>
      <td>3.3</td>
      <td>Classical audio synthesis methods are based on a physical (analog) or algorithmic (digital) structure, some parameters being offered to the user to shape the sound.</td>
      <td>58</td>
    </tr>
    <tr>
      <td>3.4</td>
      <td>Even if a distribution p(x) perfectly represents the modelled data, the lack of parameters makes it unusable.</td>
      <td>58</td>
    </tr>
    <tr>
      <td>3.5</td>
      <td>Diagrams showing the three models used in this section. (left) simple AEVB system, trained on audio reconstruction (middle) AEVB with translating latent space, using audio x and symbol data y 3.3 (right) perceptual regularization of the latent space, using dissimilarity experiments of instruments sounds from audio perception studies 3.4.</td>
      <td>61</td>
    </tr>
    <tr>
      <td>3.6</td>
      <td>(a) analog signal x(t) (b) sampled series x[n] (c) aliasing: the discretization of the waveform introduces an infinite amount of possible frequencies falias = f<sub>s</sub> - nf - π</td>
      <td>61</td>
    </tr>
    <tr>
      <td>3.7</td>
      <td>Decomposition of a square signal over the 15 first spectral components.</td>
      <td>62</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>3.8</td>
      <td>A complex number can be whether expressed in Cartesian coordinates (a, b) or polar coordinates (r, θ).</td>
      <td>62</td>
    </tr>
    <tr>
      <td>3.9</td>
      <td>(top) signal x[n] (middle) magnitude spectrum |x[f]|, with the aliasing k &gt; f<sub>s</sub>/2 and (down) the phase &ang;X[f].</td>
      <td>63</td>
    </tr>
    <tr>
      <td>3.10</td>
      <td>Windowing the signal and performing FFT on each segment allow to represent the frequency evolving through time.</td>
      <td>64</td>
    </tr>
    <tr>
      <td>3.11</td>
      <td>Examples of wavelets. The input wave is decomposed in wavelets, allowing to perform multi-resolution analysis.</td>
      <td>64</td>
    </tr>
    <tr>
      <td>3.12</td>
      <td>Discrete Cosine Transform (DCT) encodes the phase in the negative components of the spectrum. It can be understood as a DFT in a different base.</td>
      <td>64</td>
    </tr>
    <tr>
      <td>3.13</td>
      <td>Examples of spectra generated by the toy dataset. (top) : examples from toy_additive, generated from a fundamental frequency and a given number of partials, whose decay are parametrized with the exponential decay parameter. (down) : examples from toy_fm, that are typically much more complex than the ones from toy_additive, where the partials are not spread according to harmonic relationships.</td>
      <td>68</td>
    </tr>
    <tr>
      <td>3.14</td>
      <td>Examples of spectra from the acidsInstruments-ordinario dataset.</td>
      <td>69</td>
    </tr>
    <tr>
      <td>3.15</td>
      <td>Examples or reconstructions obtained from a latent space with (a) 2 dimensions (b) 8 dimensions. While the system performs honourably with two dimensions, it cannot generate some examples.</td>
      <td>71</td>
    </tr>
    <tr>
      <td>3.16</td>
      <td>PCA representation of a VAE trained in acidsInstruments-ordinario, each point being a data point projection coloured by its pitch class. We can see that the extracted representation has been successfully shaped as an isotropic normal distribution, but that this visualization does not provide any information on the classwise repartition.</td>
      <td>73</td>
    </tr>
    <tr>
      <td>3.17</td>
      <td>Statistics of latent projections obtained with two VAEs, one trained with D<sub>KL</sub> and one trained with adversarial criterion. (a) variances of latent means, obtained with D<sub>KL</sub> regularization (b) means of variances obtained, with D<sub>KL</sub> regularization (c) variance of means, obtained with adversarial regularization (d) means of variances, obtained with adversarial regularization</td>
      <td>74</td>
    </tr>
    <tr>
      <td>3.18</td>
      <td>Statistics obtained from a VAE trained with β = 10. (a) variances of means obtained with D<sub>KL</sub> (b) means of variances obtained with D<sub>KL</sub> (c) variances of means obtained with adversarial (d) means of variances with adversarial.</td>
      <td>75</td>
    </tr>
    <tr>
      <td>3.19</td>
      <td>Descriptor plots obtained from (left) toy_additive with D<sub>KL</sub>, with PCA (above) and ICA (below), and from (right) acidsInstruments-ordinario with adversarial, with PCA.</td>
      <td>77</td>
    </tr>
    <tr>
      <td>3.20</td>
      <td>Four different approaches for conditioned data generation. (a) un-conditioned system (b) direct conditioning : the label information is directly given to the decoder, making the latent representation independent to the corresponding task. (c) semi-supervised learning : an additional classification system is added on the top of latent space, predicting the label it is missing (d) translation : the latent space is used as a translation space, that is shared between labels and data</td>
      <td>79</td>
    </tr>
    <tr>
      <td>3.21</td>
      <td>Two VAEs, one in the signal domain x, one in the signal domain y, that share the same representation.</td>
      <td>81</td>
    </tr>
    <tr>
      <td>3.22</td>
      <td>Reconstruction and transfer performed by the system. We can see that both transfer and reconstructions are perfect with one instrument. While the performances are still satisfactory with two instruments, they decrease with three instruments, showing that the system struggles to disentangle correctly the sources. This can be observed by observing the reconstructions and transfers in the symbolic domain : while the most symbols are accurately found, they are affected to the wrong instrument, showing the limitations of the followed approach.</td>
      <td>84</td>
    </tr>
    <tr>
      <td>3.23</td>
      <td>Reconstruction obtained from a MIDI file converted in audio with our model, trained on flute sounds.</td>
      <td>86</td>
    </tr>
    <tr>
      <td>3.24</td>
      <td>Functional graph of the symbol-signal translation system.</td>
      <td>86</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>3.25 Example of random trajectory navigating the latent space obtained from a model trained on violin.</td>
      <td>87</td>
    </tr>
    <tr>
      <td>3.26 MDS can be leveraged to obtain an Euclidean space, whose distances are enforced to matching a given set of dissimilarity measures.</td>
      <td>88</td>
    </tr>
    <tr>
      <td>3.27 Obtained perceptual spaces from 11845 dissimilarity measures, representing averages human perceptive distances between orchestral instruments.</td>
      <td>92</td>
    </tr>
    <tr>
      <td>3.28 PCAs of perceptually regularized latent spaces for the different regularization techniques (a) un-regularized (b) prior (b) ℓ² (d) Student-t</td>
      <td>93</td>
    </tr>
    <tr>
      <td>3.29 Perceptive features inferred by the model from unknown instruments.</td>
      <td>94</td>
    </tr>
    <tr>
      <td>3.30 Perceptual maps obtained from PCA planes of the model.</td>
      <td>94</td>
    </tr>
    <tr>
      <td>3.31 Example of descriptor synthesis</td>
      <td>95</td>
    </tr>
    <tr>
      <td>4.1 Example of a continuous Weiner process, that can be obtained from a cumulative sum of centered normal distributions.</td>
      <td>100</td>
    </tr>
    <tr>
      <td>4.2 Graph of a probabilistic auto-regressive process</td>
      <td>100</td>
    </tr>
    <tr>
      <td>4.3 Auto-correlation allow to give estimates of the intern regularities of the signal (here, original frequencies in red)</td>
      <td>101</td>
    </tr>
    <tr>
      <td>4.4 Digital filter equivalent to ARMA model 4.3, where z<sup>-1</sup> is a unit delay. Poles and zeros can be extracted from coefficients a<sub>0</sub>...a<sub>N</sub> and b<sub>1</sub>...b<sub>M</sub>, representing the frequency response on the unit circle.</td>
      <td>101</td>
    </tr>
    <tr>
      <td>4.5 General graph of a state-space model (SSM), with observed variables x, hidden variables z and actions u.</td>
      <td>103</td>
    </tr>
    <tr>
      <td>4.6 Example of a Hidden Markov Model, a generalized acyclic graph where temporality is driven in hidden discrete latent variables.</td>
      <td>103</td>
    </tr>
    <tr>
      <td>4.7 Example of Gaussian Process regression performed on a noisy curve. The input signal is the noisy black line, the bold red line is the MAP posterior of the GP, and all the other little lines are functions drawn from the GP.</td>
      <td>104</td>
    </tr>
    <tr>
      <td>4.8 Example of an Recurrent Neural Network, that uses an hidden representation updated at each step with a specific operation.</td>
      <td>106</td>
    </tr>
    <tr>
      <td>4.9 Graph of LSTM unit. Layer here is a sum of each input multiplied by a specific parameter matrix, and the gate is just a sigmoid unit allowing to mask given dimensions with Hadamard product ⊙.</td>
      <td>107</td>
    </tr>
    <tr>
      <td>4.10 The Neural Autoregressive Distribution Estimator [358]</td>
      <td>108</td>
    </tr>
    <tr>
      <td>4.11 The WaveNet auto-regression module, that transform auto-regressive neural modules (top) using dilated convolutions (down) to extend the temporal scope using less connections.</td>
      <td>108</td>
    </tr>
    <tr>
      <td>4.12 SampleRNN hierarchical modeling used to learn the underlying auto-regressive distribution p(x)</td>
      <td>108</td>
    </tr>
    <tr>
      <td>4.13 The process that we call dynamical compression rather extracts a single latent vector for a given sequence, using a RNN-like feature extractor, while the state-space approach used previously models one latent vector by incoming observation.</td>
      <td>110</td>
    </tr>
    <tr>
      <td>4.14 Different propositions of variational recurrent networks: RVAE [365], STORN [372], VRNN [373], SRNN [369], Z-forcing [370]</td>
      <td>111</td>
    </tr>
    <tr>
      <td>4.15 Separating the latent space between static (red) and dynamical (green) dimensions can enforce the disentangling between global and local features of the sequence.</td>
      <td>112</td>
    </tr>
    <tr>
      <td>4.16 We add a prediction task in addition to analysis and synthesis process, modeling the dynamics of the data.</td>
      <td>113</td>
    </tr>
    <tr>
      <td>4.17 In the CPC prediction method, we extract higher-level variables, named contexts, that are enforced to model slow features that are used to predict the next step using linear predictors.</td>
      <td>114</td>
    </tr>
    <tr>
      <td>4.18 With this prediction methods we use normalizing flows to model filtrations of the latent space, modeling complex multi-modal distributions to encode the evolution through time.</td>
      <td>116</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>4.19</td>
      <td>We can use the obtained the regression performed by the Gaussian process to infer the following states of the trajectory, using the predictive posterior distribution with non-observed values of t.</td>
      <td>118</td>
    </tr>
    <tr>
      <td>4.20</td>
      <td>Examples taken from the diva_dataset database</td>
      <td>119</td>
    </tr>
    <tr>
      <td>4.21</td>
      <td>reconstruction examples (top) and original spectra (bottom) for (a) CPC (b) Flow (c) GP prediction methods.</td>
      <td>120</td>
    </tr>
    <tr>
      <td>4.22</td>
      <td>CPC curves obtained for 5 random examples, each dimension plotted apart. We can see that the CPC are evolving slowly, contrary to the corresponding latent curves.</td>
      <td>121</td>
    </tr>
    <tr>
      <td>4.23</td>
      <td>Examples of morphing between an original (blue) and target (yellow) trajectories (through PCA) obtained from dataset examples for (top) CPC (middle) Flow (bottom) GP prediction modules.</td>
      <td>122</td>
    </tr>
    <tr>
      <td>4.24</td>
      <td>Prediction is performed at the higher level of temporality, allowing to significantly reduces the number of prediction states needed to predict the signal.</td>
      <td>124</td>
    </tr>
    <tr>
      <td>4.25</td>
      <td>Probabilistic graph explaining the hierarchical multi-scale architecture of ShrubVAE. This architecture is based on multi-layer dynamical compression, thus performing Bayesian down/up-sampling that allows to efficiently model the temporal structure of the incoming signal.</td>
      <td>126</td>
    </tr>
    <tr>
      <td>4.26</td>
      <td>Examples of trajectory morphing between two examples of the dataset for (top) CPC (middle) Flow (bottom) GP prediction modules.</td>
      <td>131</td>
    </tr>
    <tr>
      <td>5.1</td>
      <td>The Dataset class, organising data and metadata from audio transforms</td>
      <td>133</td>
    </tr>
    <tr>
      <td>5.2</td>
      <td>The Model class, encoding and decoding the input data.</td>
      <td>133</td>
    </tr>
    <tr>
      <td>5.3</td>
      <td>The Loss class implements different loss functions, providing additional features such as loss history tracking, and simple algebra.</td>
      <td>134</td>
    </tr>
    <tr>
      <td>5.4</td>
      <td>Schematics of the vsacids real-time environment.</td>
      <td>135</td>
    </tr>
    <tr>
      <td>5.5</td>
      <td>An example of cross-synthesis, with a sample of voice passed through a model trained on toy_additive dataset. We can see that the voice is somehow "recomposed" by the features learned by the model.</td>
      <td>137</td>
    </tr>
    <tr>
      <td>5.6</td>
      <td>Reinforcement learning for sonic exploration. The agent learns which actions to take on a sound synthesis environment based on reward given by the musician. The agent implements an exploration method to foster discovery along interaction.</td>
      <td>138</td>
    </tr>
    <tr>
      <td>5.7</td>
      <td>Interaction diagram of the ægo instrument. The reinforcement module monitors the reward given by the user and the current latent state to decide a future action A, that is used to control the navigation.</td>
      <td>139</td>
    </tr>
    <tr>
      <td>5.8</td>
      <td>Temporal structure composed for the piece. The dimensions are progressively increased and then decreased when the dataset is changed.</td>
      <td>141</td>
    </tr>
    <tr>
      <td>5.9</td>
      <td>Pictures taken from the live setup of the piece, showing different moments of the performance.</td>
      <td>142</td>
    </tr>
  </tbody>
</table>

Tables

<table>
  <tbody>
    <tr>
      <td>3.1</td>
      <td>Results obtained on test data with the acidsInstruments-ordinario dataset, for different latent space dimensionalities.</td>
      <td>71</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>Reconstruction scores obtained for different regularization divergences, using test partition of the datasets.</td>
      <td>72</td>
    </tr>
    <tr>
      <td>3.3</td>
      <td>Reconstruction scores obtained for different β, using test partition of the datasets.</td>
      <td>72</td>
    </tr>
    <tr>
      <td>3.4</td>
      <td>Latent organization descriptors obtained from the three datasets.</td>
      <td>74</td>
    </tr>
    <tr>
      <td>3.5</td>
      <td>Table of results for latent descriptors obtained with models with various β. As we can see, increasing β does not lead to reduce the total covariance TC.</td>
      <td>75</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>3.6</td>
      <td>Disentanglement scores for tasks <i>octave</i>, <i>pitch</i>, <i>instrument</i> obtained with the dataset <i>acidsInstruments-ordinario</i>.</td>
      <td>75</td>
    </tr>
    <tr>
      <td>3.7</td>
      <td>Signal reconstruction and transfer performances</td>
      <td>84</td>
    </tr>
    <tr>
      <td>3.8</td>
      <td>Symbolic inference reconstruction and classification results (successively <i>pitch</i>, <i>octave</i> and <i>dynamics</i>). Scores without parenthesis are reconstruction scores obtained within the symbolic domain, while scores in parenthesis are obtained when performing transfer from the signal domain</td>
      <td>85</td>
    </tr>
    <tr>
      <td>3.9</td>
      <td>Results obtained with the validation <i>flute</i> dataset.</td>
      <td>85</td>
    </tr>
    <tr>
      <td>3.10</td>
      <td>Generative capabilities evaluated on the log likelihood and reconstruction error over the test set.</td>
      <td>93</td>
    </tr>
    <tr>
      <td>3.11</td>
      <td>Discriminative capabilities in classifying <i>family</i>, <i>instrument</i>, <i>pitch</i> and <i>dynamics</i> of the test set.</td>
      <td>94</td>
    </tr>
    <tr>
      <td>4.1</td>
      <td>Table of the results obtained with the proposed prediction methods on the overall sequence. The error is normalized stepwise, to be compared with the results obtained in section 3. The results out of parenthesis are the obtain with train partition of the dataset, while numbers within are the errors obtained with the test partition.</td>
      <td>119</td>
    </tr>
    <tr>
      <td>4.2</td>
      <td>Table of the results obtained with the proposed prediction methods, only on the predicted part.</td>
      <td>120</td>
    </tr>
    <tr>
      <td>4.3</td>
      <td>Full table of results obtained with 1-layer (previous section), 2-layer and 3-layer <i>ShrubVAEs</i>, for all models. For regularization losses, the layer-wise divergences are sorted in increasing order.</td>
      <td>129</td>
    </tr>
    <tr>
      <td>4.4</td>
      <td>Results obtained with the <i>ShrubVAE</i> with the three prediction methods, only on the predicted part of the sequence.</td>
      <td>129</td>
    </tr>
  </tbody>
</table>
# Introduction and background

In this PhD thesis, we study the utilization of recent unsupervised learning techniques, that aroused in the machine-learning domain, to provide a novel framework for digital audio synthesis. The proposed techniques, based on the extraction of invertible features with deep representation learning, can thus be used at the same time for an analysis purpose, providing a *compressed* representation of the analyzed data, and for generation, using the extracted features as defining a *generative space* that can be explored and controlled by a human user. Contrary to most machine-learning applications developed so far in the domain of digital audio synthesis, we aim to foster an *experimental* approach (that we call *implicit*) to the use of these frameworks, rather motivated by the emerging behavior of these models than by their ability to perform external tasks. In this introduction, we will glance over the related domains broached in this work, from sound synthesis, generative systems, computational creativity, and summarizing the motivations underlying this work.

**Analysis and synthesis.** Sound synthesis is a seminal field of digital audio processing, that motivated the development of diverse systems focused on both human creativity and the understanding of acoustical and perceptual phenomena. While one could anachronistically describe lute-making as a first incarnation of sound synthesis, involving to understand the physical properties of the instruments to target a given sound and a specific behavior, the technical possibility to *experiment* acoustical phenomenons greatly enhanced the understanding of human perception and of audio signals, both from a musical and a scientific point of view. However, dissecting the inner mechanisms of sound needs to divide the integrity of natural phenomena in interpretable and controllable parts. This process, called *analysis* in philosophy, thus aims to extract individual concepts from an complex observation, that can be then described and experimented separately. Then, these elements of knowledge have to be gathered together whether to re-create the original phenomenon, evaluating the consistency of the analysis method, or to allow new combinations or transformations of the extracted concepts, that can be respectively be called *reasoning* or *imagination*.

In audio processing, this dichotomy between *analysis* and *synthesis* is seminal, describing accurately the reflexive process between experimentation and abstraction that is specific to natural sciences. Indeed, while the experimentation of acoustical and perpetual phenomena was already done with musical instruments through numerous practices (orchestration, acoustics, lute-making...), the possibility brought by analog instruments to manipulate even smaller components opened
significantly the field of experimentation : first, single-frequency oscil-
lators (from analogical theremin to complex additive synthesis), noise
filtering (substractive synthesis), and afterwards micro-sounds (gran-
ular synthesis), recordings (sampling), physical modeling, and many
others. Hence, this analytic process of decomposing complex acoustical
phenomena in smaller components could be whether used to propose
scientific descriptions of audio signals (resp. time-frequency represen-
tations, linear predictive coding, wavelets, acoustics), or to be trans-
formed, altered, composed, and re-synthesized, to allow the generation
of sounds that do not exist in nature. This experimental approach to
sound synthesis was seminal in the work of electro-acoustical pioneers
such as Max Mathews [1], Jean-Claude Risset [2], John Chowning [3]
and many others, that somehow decomposed the real to create the un-
real by extending or transforming its generating rules. And, reversely,
synthesis could also be used to evaluate a given analysis under a set
of criteria method through simulation, hence closing the loop of a
reflexive and iterative discovery process.

**Generative systems and artificial intelligence.** The cognitive pro-
cess of detecting the existence of structuring regularity patterns of a
given set of observations is fundamental in sciences, and bases the
extraction of *rules* underlying the corresponding phenomenon. Among
the wide variety of methodologies that exist to describe and under-
stand reality, the difference between the *description* of a *phenomenon*,
and the act of *modeling* it, is seminally different, although comple-
mentary. Indeed, while the first rather targets to find an appropriate
semantic grid or a relevant description to describe a phenomenon, the
second will *infer* its underlying structure using an external definition.
While numerous modeling methodologies have since been developed
to this end, the *generative* approach consists in developing a set of
generation rules that describe the construction process of a given set
of data. This approach can consist in defining a set of transformation
rules, such as *generative grammars* [4–6], mathematical morphology
[7], or generative approaches to music theory [8–13]. In that case, the
likelihood of inferred generative rules is obtained by comparing the
generated outputs to the targeted observations ; then, the model can
be updated whether by changing the underlying model behind rules
(*top-down approaches*) or, inversely, to extract the generative rules from
the data itself (*down-top approaches*).

However, the extraction of generative patterns from the data itself is
highly non-trivial. Especially, designing domain-independent algo-
rithms for generative modeling has to rely on objective descriptions of
the observations. A first approach to this purpose, called *information
theory*, is a seminal signal processing framework brought by Shannon
during the 50's that quantifies the amount of *information* transmitted
by a given signal by analyzing its recurring patterns [14]. Fundamental
notions coming from information theory, such as the concept of *infor-
mation*, quantifying the amount of surprisal of a signal element from
a sequence, or the concept of *entropy*, quantifying the total disorder
amount of a signal. Such notions porvided the basis to an analytic
methodology to communication, and have since been extended to other domains, even in philosophy with the *philosophy of information*, that proposes to use this concept as a "first-order phenomena represented by the world of information, computation, and the information society" [15]. Alternatively, another generic method for structure extraction rather grounds on *statistical inference*, aiming to describe a given set of observations using density estimation techniques based on probabilities. This methods can be based on an explicit probabilistic parametric model, whose parameters can be optimized using criteria (such as data likelihood), and then describe the observed data with the corresponding parameters, at the cost of dropping the exact reversibility of the model. Advanced statistical inference methods, such as Bayesian or latent models, allow more the definition of more complex inference systems that can be used to extract *latent representations* of a given dataset, brding with representation learning and information theory [16, 17].

Despite thier differences, these two approaches are related by their common based of *compression*, as they are both based on the extraction of regularity patterns underlying a target process [18]. While these patterns can have different names or formulations depending on the domain (*syntax* in generative grammar, *code* in information theory, *model* in inferential statistics), they all aim to reduce the complexity of a target data by extracting invariant structures, based on non domain-specific properties of the data. This underlying relation between compression and complexity motivated criteria for model selection : based on the idea of *Kolmogorov complexity*, that defines the complexity of a signal by the bit length of the smaller code that can generate it, methods such as *Minimum Description Length* use this generative criterion to select the one that achieves the best compression, as an information-theoretic version of Occam's razor [19].

This intrinsic correspondence between compressibility and explainabil-
ity is also underlying *representation learning* methods, that are based on
the representation of a set of high-dimensional data in a smaller space.
While such methods can be beneficially leveraged for visualization,
they also have a seminal interest in *machine-learning*, a domain based
on the automatic execution of a given set of tasks. This data-centered
approach to automatic feature extraction recently raised a significant
interest in the scientific community, was caused not only by the re-
cent increase of computation power allowed by parallel computing,
enabling the use of connectionist approaches (grounded by cybernet-
ics) such as neural networks that enhanced the modeling abilities of
such methods, but also by the ever growing amount of available data.
While the domain of machine-learning is rather focused on the ex-
traction of optimal representations for a given set of *external* tasks,
the recent merge of *unsupervised* approaches with machine-learning
methods allowed to leverage these modern techniques for learning
compressed representations of a given set of data. Such approaches,
mostly grounded on manifold hypothesis [20], assert a given set of
high-dimensional data to lie on a sub-manifold of the embedding
space. These methods are then aiming to the extract this manifold to
recover the underlying generative process, and are then related to the
compression-based systems explained above, and bridging representa-
tion learning and generative models [21]. Moreover, recent hybridiza-
tions between generative and machine-learning models gave birth to
a flourishing class of expressive density estimation methods, enabling
the development of powerful data-centered approaches to generative
models (variational auto-encoders [22], generative adversarial nets
[23], and recently generative normalizing flows [24]).

**Computational creativity, autonomy and empowerment.** While these recent advances in the domain of data-centered approaches for data modeling engendered powerful systems for numerous applications, can generative models be called *creative*? Indeed, the recent raise of interest in machine-learning was intrinsically driven by the growing interest in *autonomous* systems, that in this case are able to learn a given set of tasks. However, as generative models are based on the reconstruction of the original data, their *creative* aspect may be tempered. The seminal works of Margaret Boden [25] are enlightening on this point, grounding an accurate thinking about creativity and machines. Indeed, the distinction between *P-creativity*, defining objects that are only unknown to the generating agent, and *H-creativity*, defining objects things are *historically new*, emphasizes the fact that the *creative* dimension of an artifact is in part (if not totally) driven by the context. Furthermore, Boden also underlines the co-existence of several *modalities* of creation, distinguishing *exploratory* creativity, that is the discovery of new objects by navigating an existing *generative space*, or the *transformative* creativity, that is the creation of new objects by the transformation of existing ones. *Combinatorial* creativity, added afterwards by Pasquier [26], also underlines the distinction between transforming and composing objects.

Modern machine-learning models that have been developed to directly
address creativity are rather focused on *autonomous* systems, whose
emerging behavior is generally driven or learned with a set of *reward
signals*. These models, based on reinforcement or active learning, up-
date their internal state by observing a suitable reward signal, whether
*internal* and based on an the autonomous analysis on a given creativity
measure, or *external*, provided by the external world (interaction with
human, active observation with receptors) [18, 27, 28].

However, how to evaluate the creative aspects of non-autonomous
generative models, such as the ones investigated in this thesis? Indeed,
the creative constraints of machine-learning based methods for gen-
eration seems rather driven by another paradox unveiled by Boden
(also addressed by Boulez in his fundamental essay *A la limite du pays
fertile*) : "*The more stylistically fundamental the altered constraint, the more
surprising—even shocking— the new ideas will be.*". Indeed, the direct
generation of data judged "realistic" or at least "understandable" for a
human is a very complex task, that is not eased with the computational
power of these methods. Indeed, modern generative models are rather
focused on their ability to reconstruct the original data, in other terms
the performance obtained on their training task. However, at which
extent can they be used for creation? On this point, the role of *representation* seems mandatory : allowing external *interaction* with the model (by the human or another model) implies to get a understandable description of the system's behavior. Hence, using the Boden modalities summarized above, what are the possibilities offered to *explore* the generative abilities of the model, and how can they *transform* or *compose* inner properties of the data?

**Towards an experimental approach to machine-learning**

Audio synthesis methods, that on these creative aspects are not different from generative models, alleviated this *creative* evaluation with the close relationship it maintained with creation and production, being directly embedded in musical practices since its creation. However, the utilization made so far of machine-learning techniques in audio processing domain are still what we could call *extrinsic*, i.e. constrained to the correct execution of a given set of external tasks.

While the integration of machine-learning based generation models starts to establish in the domain of image creation with well-known systems like DeepDream, GANs, or the recently released Runway platform *, there is no similar trend in audio synthesis (apart from works of Magenta†, interestingly focusing on musical creative flows [29, 30]). Indeed, machine-learning generative models for music are so far constrained in the symbolic domain, targeting to model musical styles such as AIVA ‡. In the signal domain, recent commercial software propose to introduce machine learning in mixing or mastering to target specific aesthetics, such that Lindr § or iZotope Neutron ¶. However, these algorithms are still leveraging machine-learning to re-produce existing structures in the musical domain, and do not propose an explicit relation to the *experimental* approach that is yet seminal in musical creation. However, these models can also be used from an *instrinsic* manner, aiming to really explore the inner behaviour of this new methods and proposing a new framework for digital synthesis, involving their specificities (complexity, learning, modularity, stochasticity) in a real creative process. Such approaches could then address

► *transformational* creativity, allowing to transform external data with the system or transforming the system itself

► *exploratory* creativity, providing an browsable representation whose design is involved the creative process

► *crafting*, involving the definition of model and of the desired properties of the representation into the creative process

aiming to find this combination of mastering and surprise that make
audio synthesis methods really *creative* and *enjoyable*. However, to
challenge this objective, we had to tackle several scientific questions
such that the *representativeness* of these extracted synthesis space, that
led us to the development of regularization strategies for focused on

<footnote>* https://runwayml.com/</footnote>

<footnote>† https://magenta.tensorflow.org/</footnote>

<footnote>‡ aiva.ai</footnote>

<footnote>§ https://www.landr.com/fr/</footnote>

<footnote>¶ https://www.izotope.com/en/products/neutron.html</footnote>
audio signals, as well as the introduction of their temporal dimension
in the used frameworks. Then, as this approach also aims to integrate
the developed framework into the creative flow of a human user,
expert or non-expert, we experimented our framework by proposing a
novel musical piece based on this system, ægo, that provided a first step
towards a *research and creation* approach. We know that this work is
still preparatory, but we hope to propose an accurate way of involving
these systems into music creation with an experimental perspective,
rather than just as way of mocking existing content. Good reading!
Alternatively, some methods drop the probabilistic assumption of the data distribution and rather analyze geometric or topological properties from the dataset, providing low-dimensional representations (see sec. 2.2.1) that aim to reflect information about the data structure. Differently, some methods leverage the use of high-capacity function approximators such as neural or convolutional networks (see sec. 2.2.2) to extract representation from the data, whether based on an external task that enforce a given structure to the representation (such as classification or regression), hence called *supervised learning*, or trained to act as a compressed space representing the data, hence called *unsupervised learning* (see sec. 2.2.3). While the first can be very efficient to obtain representations that can be used for feature extraction, the high-capacity of the involved feature extractors generally lead to degenerated representations, that overemphasize the target task at the cost of not extracting any structural information about the data. Furthermore, supervised methods assume to have the desired outputs for every sample of the dataset, that can be an important limitation for the creative applicability of our method. Conversely, unsupervised learning is appealing because of its independence to externally defined tasks; however, the capacity of the involved function often prevent the extracted representations to represent interpretable generative factors of the input data.

Going back to our case, what would be the requirements for such extracted representations in order to be used as a control space for sound synthesis? ALI the different representations properties described above could find its place in a framework, especially if the definition of the model should make part of the creative process. Indeed, it seems mandatory that the extracted control space should be low-dimensional and represent structural properties of the data, otherwise the exploration of the provided representation would be insensitive and discouraging for the human user. As we described previously, Bayesian latent models allow to obtain sensitive and interpretable control parameters, but unfortunately their limited modeling capacity often yield to a weak expressiveness, that could provoke a diminished interactive interest. Reversely, representations obtained with high-capacity function approximators such that neural networks got the target expressiveness between the control and data space, but are often degenerated and high-dimensional. Hence, the ideal model for our purpose would be an hybridization of both, allowing the ensure the consistency of the extracted representation but that would be also based on expressive inference and generation processes. Furthemore, the introduction of explicitly defined tasks could be also interesting, allowing the representation to represent user-defined factor of variations, but still reflecting structural properties of the data. The desired framework should then be flexible enough to allow all of these criteria, but also be light enough to be easily trained by the human user.

In the last section of this chapter, will described the chosen framework, called *Auto-Encoding Variational Bayes* (AEVB), that bridges Bayesian inference processes and deep representation learning. This framework, that can be understood as an hybridation between *auto-encoders* (see sec. 2.2.3) and Bayesian latent models, advantageously leverages the

<figure><img src="image_3.png" /><figcaption>**Figure 2.1: Supervised approaches vs. unsupervised approaches.**</figcaption></figure>
use of neural networks to model expressive relationships between
the data space and the higher-order features. This flexible framework,
proposed simultaneously by Kingma & Welling [31] and Rezende & al.
[32], raised a significant interest in the machine-learning community,
and since gave birth to many developments and further analysis, no-
tably on theoretical and optimization perspectives.

Hence, in this chapter, we will first provide a state of the art of latent
Bayesian models focusing on an approximation scheme, called *varia-
tional learning*, that allows to drop the strong tractability assumptions
of direct Bayesian inference. In the next section, we will rather focus
on non-probabilistic systems based on the extraction of representa-
tions from the data, then called *representation learning*, by describing
dimensionality reduction methods, neural networks properties and
optimization, and neural-network based generative models. Finally,
we will present an extensive state of the art of the used framework, that
leverage techniques of these two domains, summarizing the emerg-
ing properties of these systems and various developments that can
be advantageously used for the extraction of interactive generative
spaces.

## 2.1 Bayesian probabilistic generative models

As we approached in the introduction, a *generative model* is a system
that can output a finite or infinite set of realizations, whose generation
process is defined by a set of rules (physical, mathematical, logical)
that define the *identity* of the corresponding model. Hence, extracting a
generative model from a finite set of samples comes back to recovering
the underlying generation rules of the corresponding system, resorting
to various identification methods. While some identification methods
assume strong properties about the structure of the generative model,
developing general methods for model identification resorting the
minimum set of assumptions is a challenging task. A method to ad-
dress this challenge are *statistics*, resorting to probabilistic models to
describe, or model, a restricted set of data.

Though, there exists different trends in statistics, differing in how they
extract these information, leading to different philosophies (see fig.
2.2)). *Descriptive statistics* focuses on how to accurately describe a sam-
ple of data, and aims to formulate *statistical descriptors* of the properties
of the sample (distribution, covariance, dispersion...), without adding
any further assumption on the data structure. While this framework
provides efficient methods for data analysis and representation, the
absence of an underlying probabilistic model prevents to generate new
content from the observed data, and does not infer any explaining fac-
tor that would describe the generative process. Conversely, *inferential
statistics* rather consider the observed data **x** as a set of *realizations*, that
are drawn from a probability distribution *p*(**x**). Hence, inferential statis-
tics aims to recover this underlying distribution, then recovering the
inner structure behind the observed data. Furthermore, if we are able
to sample the inferred distribution *p*(**x*), we can then generative new
data that matches the extracted structure, then proposing a probabilis-
<figure><img src="image_4.png" /><figcaption>**Figure 2.2:** Descriptive and inferential statistics</figcaption></figure>

tic method for *generative modeling*. However, extracting the underlying distribution from a limited amount of data is not a trivial task, as the exact extraction of the underlying probabilistic model would require an infinite amount of observations.

Therefore, several strategies can be used to address this challenge, that are quite philosophically different even if based on the same mathematical framework. The first strategy is to choose a *frequentist* approach, that defines the probability of an event by the number of its occurrences within the observations. The underlying concept is that the most probable events of a random process are the ones that happens the most, then assuming that the process is composed by a lot of reproducible and independent trials. The frequentist approach then models the probability distribution underlying a set of observations by accurately modeling its events' frequency. This idea, based on the seminal *Law of Large Numbers*, ensures the convergence of the obtained estimation as the number of observed events goes to +∞. However, this approach raises several difficulties: first, the number of samples needed to accurately model the target distribution can be very high, and is not able to model probabilities of rare events. Another problem is that this approach does not model the uncertainty of the model, such that it may be *over-fitting* the distribution, and performs badly in case of noisy distributions.

### 2.1.1 Bayesian learning for statistical inference

**Bayesian inference.** The Bayesian philosophy of probabilities is then rather different from its related frequentist, detaching probability from the idea of repetition to translate it in the domain of **believes**, modeling the certainty of a given assumption after some observations. While the first were targeting to uncover the underlying distribution $p(x)$ without any further assumption on the observed data, Bayesian learning models a *prior* knowledge on the process, that is progressively modified as the number of observed data items increase.

<figure><img src="image_5.png" /><figcaption>**Figure 2.3:** Difference between frequentist and Bayesian observation processes.</figcaption></figure>

Let be an hypothesis *H* on an experiment realizing observations *O*, that can be observed by the emitter. Given an *a-priori* probability distribution $p(H)$ defined on the space of hypothesis, the observer will then expect a distribution $p(O|H)$ over the observations *O*, given the prior hypothesis *H*. However, after an observation *O*, the emitter will refine its hypothesis accordingly to $p(H|O)$, then modifying its prior assumptions. We can see that this formulation differs from the previous one by the introduction of a *prior* knowledge on the observation, that will be progressively biased by the observed realizations obtained from the experiment *O*. Furthermore, the hypothesis *H* will not be modified to much if the observation *O* was already considered as probable, while an unexpected observation will incite the *posterior* $p(H|O)$ to deviate the original hypothesis $p(H)$ more substantially. This trade-off can be expressed with the Bayes' rule, that can be easily obtained from the conditional decomposition of the joint distribution $p(O, H)$
$$p(O, H) = p(O|H)p(H) = p(H|O)p(O) \underbrace{p(H|O)}_{\text{posterior}} = \frac{p(H, O)}{p(O)} = \frac{\underbrace{p(O|H)}_{\text{likelihood prior}} \widehat{p(H)}}{\underbrace{p(O)}_{\text{marginal likelihood}}}$$

The Bayes' equation thus identify four terms: the *posterior distribution* $p(H|O)$ of the hypothesis *H* after some observations *O*, the prior knowledge $p(H)$, the *likelihood* $p(O|H)$ of the observation *O* under the hypothesis *H*, and the probability of observation $p(O)$. Bayesian statistics are then more robust under small variance noise over *O*, as the posterior distribution $p(H|O)$ won't be modified if the probability of the observation $p(O)$ corresponds to the likelihood $p(O|H)$. Furthermore, this Bayesian formulation allows us to define a probability distribution over *H* rather than a point-estimate, such that the posterior process also gives information about the *uncertainty* of the prediction.

**Likelihood estimation.** In statistical inference, we target to recover the generative distribution $p^*(x)$ underlying some finite observations $\{x_i\}_{i=1...N}$. As the true distribution is unknown to us, statistical inference relies on modeling this distribution with a defined model $p(x)$ that we will force to match with the observations. These methods can be whether *non-parametric*, using descriptive statistics estimators to model the distribution, or *parametric*, such that the model $p(x; \theta)$ depends on a set of parameters $\theta$. While the first can perform well on simple data, the second allows us to define a precise structure for the distribution $p(x)$, then enforcing some properties of the distribution. However, estimating the parameters $\theta$ of a model $p(x; \theta)$ is not straightforward, and requires an explicit criterion to optimize. The most common approach to maximize the likelihood of the data given the model $p_{\theta}$, optimizing the parameters $\theta$ of the model [33]

$$\theta^* = \max_{\theta} p(x; \theta) \qquad (2.1)$$

In some cases this optimization is tractable, such that the optimal parameters $\theta^*$ can be directly obtained from the data. Furthermore, maximum likelihood has been shown to be a *consistent* (converging in probability to the real density $p(x)$), and *efficient* (having the smallest variance) estimator when the sample size tends to infinity. However, tractable models for log-likelihood maximization are often not expressive enough to shape the underlying complexity of the data, such that more elaborated schemes have to be used such as gradient descent algorithms (see sec. 2.3.1), normalizing flows (see sec. 2.3.2.1) or contrastive estimation (see sec. 4.3.1) to give an estimate of the parameters $\theta$. Alternatively, we can turn statistical inference into Bayesian problem, considering parameters $\theta$ themselves as random variables. In that case, the hypothesis *H* are represented by the parameters $\theta$ with a corresponding prior distribution $p(\theta)$, such that Bayes' equation can
<figure><img src="image_6.png" /><figcaption>Figure 2.4: Maximum Likelihood estimation (blue) and Maximum A Posteriori estimation (red). ML gives a point-estimate of the mean, while MAP finds the maximum over a distribution of possible means.</figcaption></figure>

be written

$$p(\theta|\mathbf{z}) = \frac{p(\mathbf{x}|\theta)p(\theta)}{p(\mathbf{x})} \qquad (2.2)$$

$$= \frac{p(\mathbf{x}|\theta)p(\theta)}{\int p(\mathbf{x}|\theta)d\theta} \qquad (2.3)$$

providing a posterior distribution $p(\theta|\mathbf{x})$ over the parameters. Modeling the estimation of $\theta$ with a probability distribution rather than a point-estimate allows us to access the model uncertainty, from which we can get an estimate by whether selecting the $\theta$ with highest probability, then called the maximum a-posteriori estimation (MAP), or to sample $\theta|\mathbf{z}$ over a given confidence range of $p(\theta|\mathbf{z})$. Setting a uniform prior for $\theta$ recovers the ML estimator of 2.1 up to the $p(\mathbf{x})$ term, that can be excluded from the optimization as it is independent from $\theta$.

We can now see that Bayesian learning gains from frequentist approaches by formulating an a-priori knowledge on the parameters' distribution, that can be thought as a soft regularizer over the parameter estimation. The definition of prior distribution $p(\theta)$ can whether be chosen non-informative (uniform or isotropic normal) or informative, in the case of available external knowledge or iterative model updating. Furthermore, this method also allows us two different methods to predict new values $\hat{\mathbf{x}}$ from the inferred model

$$p_{\text{post}}(\hat{\mathbf{x}}|\mathbf{x}) = \int p(\hat{\mathbf{x}}|\theta)p(\theta|\mathbf{X})d\theta \qquad (2.4)$$

$$p_{\text{prior}}(\hat{\mathbf{x}}) = \int p(\hat{\mathbf{x}}|\theta)p(\theta)d\theta \qquad (2.5)$$

The first prediction method, called posterior predictive distribution, predict new values for $\mathbf{x}$ marginalizing out the parameters $\theta$ using the posterior distribution while the second, called the prior predictive distribution, rater marginalizes the parameters using the prior information. The regularization effect brought by the probabilistic distributions $p(\theta|\mathbf{X})$ and $p(\theta)$ can thus be seen by their marginalization over $\theta$, meaning that all the possible parameters values modeled by these distributions are involved in the prediction of $\hat{\mathbf{x}}$. Bayesian learning is then more robust than its point-estimate counterpart, that would only take one possible $\theta$ to perform the prediction.

**Conjugate priors.** Modeling the posterior distribution $p(\theta|\mathbf{x})$ rather than obtaining point-estimate values $\theta$, has then several advantages: also modeling the uncertainty about the parameters estimation, modeling both parameters' inference and data generation, and data prediction. However, computing these distributions assumes that all of the involved terms from 2.2 are tractable, even the marginal likelihood $p(\mathbf{x})$ that involves the marginalization of the likelihood $p(\hat{\mathbf{x}}|\theta)$ over $\theta$. The full tractability of this expression can be ensured by formulating
a generative distribution and its corresponding prior with similar algebraic forms, i.e. taking a conjugate prior as initial distribution $p(z)$. If the two distributions are conjugate, the posterior distribution will then have a similar algebraic form, coming back to an update of the prior parameters after the observation of data samples [34]. The simplest example is the multivariate normal distribution, that is a self-conjugate distribution; indeed, taking a likelihood model $N(\mu, \Sigma)$ with a fixed covariance matrix $\Sigma$, we can also model the prior of the parameter $\mu$ with a normal distribution, such the corresponding prior and the posterior distributions can be written

$$p(\mu|\mu_0, \Sigma_0) = N(p(\mu|\mu_0, \Sigma_0)) \qquad (2.7)$$

$$p(\mu_0|\mu) = N((\Sigma_0^{-1} + n\Sigma^{-1})^{-1}(\Sigma_0^{-1}\mu_0 + n\Sigma\bar{x}), \qquad (2.8)$$

$$(\Sigma_0^{-1} + n\Sigma^{-1})^{-1} \qquad (2.9)$$

where $\bar{x}$ is the sample mean of the $n$ data $x_1...x_n$. Intuitively, the posterior of the mean can be understood as the precision weighting of the prior mean $\mu_0$ and the empirical mean $\bar{x}$, whose precision is the sum of both prior and observations distributions. This simple example shows how the use of conjugate priors can provide tractable posteriors with interpretable parameters. Prior distributions are available for many widely used distributions such as Bernoulli, Dirichlet, or Gamma functions, and then provide an efficient way to perform Bayesian inference of model parameters. Furthermore, using conjugate distributions allows us to perform whether offline, having all the observed data in a dataset $x$, or online, where we successively update the prior information based on the previously observed data (see fig 2.5).

<figure><img src="image_7.png" /><figcaption>Figure 2.5: Example: a Beta($\alpha, \beta$) prior equally distributed between 0 and 1 shots (blue). Conjugate posterior after one more observation Beta($\alpha, \beta + 1$). As beta is a natural mean conjugate of Bernoulli distribution, after one new "1" observation the distribution is a new Beta, with $\beta \leftarrow \beta + 1$</figcaption></figure>

### 2.1.2 Latent models and posterior approximation methods.

**Latent variables and graphs.** Bayesian learning hence allows a robust and elegant way to learn generative models, gaining on likelihood-based methods by inferring posterior distributions over the model parameters. Despite the elegance of the conjugate prior approach, it significantly constrains the expressiveness of the generative model $p_\theta(x)$, enforcing the involved distributions to have a similar shape. Furthermore, conjugate priors can be only used to get posterior distributions for the parameters of the generative model, and can not be used for additional random variables that would increase the model expressiveness.

An alternative approach for statistical inference is to condition the observations $x$ on auxiliary variables $z$, then called latent, that are controlling the generation process of the underlying data distribution (see fig. 2.6). In this case we maximize the joint distribution $p_\theta(x, z; \theta)$ instead of $p_\theta(x; \theta)$. Introducing latent variables to parameters model
<figure><img src="image_8.png" /><figcaption>**Figure 2.6:** Conditional graph between observed variables (x, in gray) and unobserved variables (z, in gray)</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Figure 2.7:** Generation (top) and inference (down) probabilistic graphs of a joint distribution p(x₁, z₁, z₂, z₃)</figcaption></figure>

can whether be used to ease the probabilistic formulation of a problem, such that the overall model p(x, z) is more easy to sample, or to model the influence of an external factor, observable or hidden, on the data x. It also provides a way to control the generation of p(x, by choosing a latent variable z and retrieving corresponding x samples from p(x|z). Latent variables is then a very convenient tool in statistical inference, and is now present in most machine learning methods. Similarly, latent variables can be inferred from a given generative distribution p_θ(x) with Bayes' rule

$$p(\theta|\mathbf{z}) = \frac{p_{\theta}(\mathbf{x}|\mathbf{z})p(\mathbf{z})}{\int p_{\theta}(\mathbf{x}, \mathbf{z})d\mathbf{z}} \quad (2.10)$$

where we explicitly condition the generative process on z, θ being the parameters of the conditional distribution. Bayesian learning is though still probable, providing the tractability of the conditional generation p_θ(x|z). While this formulation increases the model expressiveness model, it also makes the integral ∫ p_θ(x, z)dz intractable, enforcing whether to have discrete latent variable (so the integral becomes a summation Σ_k p_θ(x, z_k)), or to leverage optimization schemes in order to maximize the overall probability p_θ(x, z). While the first can be very useful for some problems, such as mixture of distributions, having continuous latent variables z ∈ ℝ could allow us to extract hidden factors from the data, that could be used to retrieve generative factors from the data. Inference of continuous latent variables can be achieved with optimization methods, such as expectation-maximization (EM) or variational inference (VI), that we will discuss in the next section.

Extending this idea to multiple variables [x₁, ..., xₙ, z₁, ..., zₘ], composed by n observed variables and m latent variables, we can define complex probabilistic structures by formulating group dependencies between observed and latent variables, as can be seen fig. 2.7 with the generative model p(x₂|z₁, z₂)p(x₁|x₁, x₂). Such conditioned structures define an underlying directed graph, such that these methods are called probabilistic directed graphical models. Such structure are widely used in the machine-learning field, as it allows to define intuitively model relationships between groups of observed variables that can be explained by one or several common latent factors. A first occurrence of these models can be found Bayesian belief networks, that are directed acyclic graphs modeling conditional dependencies between adjacent nodes. The overall probability of the observed nodes xₙ conditioned on its parent nodes (pa(xₙ), the set of nodes conditioning the node xₙ) is then [35–37]

$$p(\mathbf{x}) = \prod_{i=1}^{N} p(\mathbf{x}_i | \text{pa}[\mathbf{x}_i])$$

such that the system verifies the Markov Blanket Property, assuming that a node is only conditioned by their direct neighbors. Bayesian belief networks can then be easily trained with the Bayes' rule 2.2, as the marginal likelihood of discrete latent variables can be easily computed with conditional probability tables. More generally, infer-
ence in complex graphs can be possible by modeling a chain over the conditional probabilities using the graph structure, and then to train with maximum likelihood or Bayesian inference on the observed nodes. However, defining directed probabilistic graphs can be a too strong assumption on the model, as it explicitly defines the conditioning structure between the variables. Alternative approaches are rather based on *undirected graphical methods*, where the total probability of a set of connected nodes $x_k$ is expressed as

$$p(\mathbf{x}) = \frac{1}{Z} \exp \left( \sum_{k} \sum_{l \in \mathcal{N}(k)} w_{k,l} \cdot f_{k,l}(x_l) \right) \underbrace{- \mathcal{E}(\mathbf{x})}_{\text{...}}$$

with feature functions $f_k$, $\mathcal{N}_k$ denotes the neighborhood of a node $k$, and $w_{k,l}$ is the weight of the relation between the nodes $x_k$ and $x_l$. These models, also called Markov Random Fields, are then based on the minimization of the energy $\mathcal{E}$, amounting to maximizing the overall probability of the model (see fig. 2.8). The undirectivity of the graph requires to estimate the partition function $Z$, such that the overall distribution $p(\mathbf{x})$ sums to 1. However, computing the partition function $Z$ is a tedious task, forcing whether to constrain the graph architecture (such as Restricted Boltzmann Machines[38]) or to approximate the partition function using approximation schemes[39, 40]. However, the duality between directed and undirected graph motivated *message passing* approaches to Bayesian graphical models such as the *sum-product* algorithm, that uses the factor decomposition of a graph to efficiently model the free energy of adjacent nodes [41].

**Expectation-Maximization** Unfortunately, the graph-based methods quickly presented above also assume the overall tractability of the model, therefore constraining the possible dependencies between involved random variables. Maximizing the model likelihood $p_\theta(\mathbf{x})$ without computing the integral term in 2.10 then involve optimization schemes, that replaces the direct computation of the posterior by iterative model updates that are perfomed until convergence of the model. A common optimization scheme, called *Expectation-Maximization* (EM), is a framework that proposes an iterative scheme both parameters and latent variables of a model $p(\mathbf{x}, \mathbf{z}; \theta)$ provided the tractability of $p(\mathbf{z}|\mathbf{x}, \theta)$ [42]. Expectation-Maximization is based on a two-step procedure

* an *expectation step*, estimating the expectation $E_{p(\mathbf{z}|\mathbf{x}, \theta)}[p(\mathbf{z}, \mathbf{x}|\theta)]$ over the latent variables $\mathbf{z}$

* an *maximization step*, that optimizes the parameters $\theta$ to maximize the first expectation.

The in-turn application of these two steps models an implicit lower-bound of the data likelihood $p_\theta(\mathbf{x})$, that converges to a local maximum of $p_\theta(\mathbf{x}, \mathbf{z})$. The intuition behind EM optimization if that maximizing the joint probability $p(\mathbf{z}|\mathbf{x}, \theta)$ over the distribution $p(\mathbf{z}, \mathbf{x}|\theta)$ amounts to optimize the data log-likelihood, without having to calculate the marginal in 2.10. An alternate view, proposed by Neal &

<figure><img src="image_10.png" /><figcaption>**Figure 2.8:** Example of Markov random field, where each node is linked by a weight $w_{ij}$. No direction is assumed, and the probability of the all system is maximized in turn.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>**Figure 2.9:** Decomposition of log-likelihood $p(x|\theta)$ with any distribution $q(z)$. Increasing $\mathcal{L}(q, \theta)$ reduces the divergence $D_{KL}[q(z) \parallel p(z|x, \theta)]$, such that $q(z) = p(z|x, \theta)$</figcaption></figure>

Hinton [43], formulates E-M optimization as an example of coordinate ascent of two estimators. Taking an arbitrary function $q(z)$, we can derive

$$ 
\begin{aligned}
\log p(x|\theta) &= \log p(x|\theta) \left[ \int q(z) dz \right] \\
&= \int q(z) \log p(x|\theta) dz \\
&= \int q(z) \log \frac{p(x, z|\theta)}{p(z|\theta)} dz \\
&= \int q(z) \log \left\{ \frac{p(x, z|\theta)}{q(z)} \cdot \frac{q(z)}{p(z|\theta)} \right\} dz \\
&= \int q(z) \log \frac{p(x, z|\theta)}{q(z)} dz - \int q(z) \log \frac{p(z|\theta)}{q(z)} dz \\
&= \mathcal{L}(q, \theta) + D_{KL}[q(z) \parallel p(z|x, \theta)] \quad (2.11)
\end{aligned}
$$

showing the intrinsic decomposition provided by E-M between a lower-bound $\mathcal{L}(q)$, quantifying the expected density ratio between the joint distribution $p(x, z|\theta)$ and the arbitrary $q(z)$, and a KL-divergence between the distribution $q(z)$ and the posterior distribution $p(z|x, \theta)$ (see fig. 2.9). Thus, during the expectation step, $\mathcal{L}(q, \theta)$ is optimized with respect to $q$, updating the current model distribution modeled on $z$. Conversely, the maximization step updates the bound $\mathcal{L}(q, \theta)$ respectively to $\theta$, also increasing $\mathcal{L}(q, \theta)$ as both of these terms are positive. Hence, optimizing $\mathcal{L}(q, \theta)$ reduces the divergence $D_{KL}[q(z) \parallel p(z|x, \theta)]$, then matching $q(z)$ and $p(z|x, \theta)$ at convergence. EM demonstrates how we can avoid to compute the marginal log-likelihood of 2.10, while still optimizing an implicit distribution to match the posterior. This intuition provides the basis of variational inference, which is the core of this PhD.

### 2.1.3 Variational methods for posterior approximation

#### 2.1.3.1 Mean-field variational inference

The investigation of Expectation-Maximization thus showed that the full tractability of Bayes' rule (2.10) was not necessary to perform inference on latent variables, replacing the exact computation of the posterior by a maximization problem. The relaxation of this hypothesis thus enables the formulation of a wider class of generative distributions, at the cost of maybe not retrieving the exact posterior. While the performances achieved by EM are satisfactory, this method requires the tractability of the posterior $p(z|x, \theta)$, preventing to use more complex generative models.

Variational methods alleviate this problem by rather modeling approximate posteriors $q(z)$, relaxing the need of a tractable posterior $p(z|x)$. The tractability of variational methods is achieved by decomposing the approximate distribution among latent variables, allowing us to
optimize each latent variables independently [44–48]. Variational inference is based on the same decomposition than E-M (2.11), but rather explicitly defines the latent distribution $q(\mathbf{z}) \in \mathbb{Q}$ in a family of model $\mathbb{Q}$. A common choice for variational posteriors is the mean-field decomposition, that enforces the independence between each latent variables and every latent dimension [49]

$$q(\mathbf{z}) = \prod_{i=1}^{d} q(z_i) \qquad (2.12)$$

in that case, the intractable calculation of the true posterior (2.10) is replaced by the in-turn optimization of each posterior component $q(z_i)$

$$q_j(z_j) = \frac{\exp^{\mathbb{E}_{ij}[\log p(\mathbf{z}, \mathbf{x})]}}{\int \exp^{\mathbb{E}_{ij}[\log p(\mathbf{z}, \mathbf{x})]} dz_j} \qquad (2.13)$$

that is tractable, provided the conjugacy between $q_j(z_j)$ and its respective prior parameters. The approximated posterior can then be retrieved by introducing $q(\mathbf{z})$ directly in the eq. 2.11, such that we can optimize the term $\mathcal{L}(q, \theta)$ to reduce the divergence $D_{KL}[q(z)\|p(z|\mathbf{x}, \theta)]$ between approximation and true posterior. Using (2.12), we then obtain

$$\begin{aligned} \mathcal{L}(q, \theta) &= \int q(z) \log \frac{p(\mathbf{x}, z | \theta)}{q(z)} dz \\ &= \mathbb{E}_{q(z)}[\log p(\mathbf{x}, z | \theta)] - \mathbb{E}_{q(z)}\left[\prod_{i} \log q_{i}(z_{i})\right] \\ &= \mathbb{E}_{q(z)}[\log p(\mathbf{x}, z | \theta)] - \sum_{i} \mathbb{E}_{q(z)}[\log q_{i}(z_{i})] \end{aligned} \qquad (2.14)$$

the mean-field assumption allowing to factorize the distribution $q(\mathbf{z})$. As each latent dimension are optimized in turn, we can isolate the dependencies of each latent dimension $z_j$

$$\mathcal{L}(q, \theta) = \mathbb{E}_{q(z_j)}[\mathbb{E}_{q(z_{i \neq j})} \log p(\mathbf{x}, z | \theta)] - \mathbb{E}_{q(z_j)}[\log q_j(z_j)] + C \qquad (2.15)$$

$$= \mathbb{E}_{q(z_j)}[\log \tilde{p}_j(\mathbf{x}, x_j | \theta)] - \mathbb{E}_{q(z_j)}[\log q_j(z_j)] + C \qquad (2.16)$$

where $C = \sum_{i \neq j} \mathbb{H}[q_i(z_j)]$ is a constant with respect to $z_j$. Optimization can thus been made in turn for each $\log \tilde{p}_j(z_j, z)$, whose expression is now tractable as the co-dependencies with other $z_i$ have been excluded. For more examples see [50, 51].

Variational learning is thus very useful to approximate a posterior distribution over parameters or latent variables, using the mean-field property of the variational distribution to obtain tractable expressions for the different components. It also allows to define custom dependencies across parameters, and is compatible with a lot of distributions, being generalizable to exponential families [52, 53]. This versatility then motivated the use of variational inference for numerous applications such as graphs [51], bandits [54], Gaussian processes [55], and state-space models [56] (see chapter 4). Furthermore, variational methods can also be used for black-box inference, allowing us to use neural networks to model generative and inference distributions. This method, that is mandatory in our work, will be explained section

<figure><img src="image_12.png" /><figcaption>Figure 2.10: Assuming the Independence between $z_1$ and $z_2$ allows to optimize separately the components of the distribution, optimizing in turn until convergence.</figcaption></figure>
2.3.1.

### 2.1.3.2 Variational sub-optimality.

Despite the versatility and the efficiency of variational learning, it can still face some sub-optimality issues, that can be unveiled by derivating further decompositions of the bound (2.14)

$$ \mathcal{L}(q, \theta) = -\mathbb{E}_{q(z)}[\log \frac{q(z)}{p(\mathbf{x}, z|\theta)}] + \mathbb{H}[p(\mathbf{x})] \quad (2.17) $$

this decomposition, called joint-contrastive, shows that maximizing the lower-bound $\mathcal{L}(q)$ maximizes the joint model $p(\mathbf{x}, z)$, up to some constant that is the entropy of the data. Alternatively, we can also derive

$$ \begin{aligned} \mathcal{L}(q, \theta) &= \mathbb{E}_{q(z)}[\log p(\mathbf{x}|z) + \log p(z)] - \mathbb{E}_{q(z)}[\log q(z)] \\ &= \mathbb{E}_{q(z)}[\log p(\mathbf{x}|z, \theta)] - \mathbb{E}_{q(z)}[\log q(z) - \log p(z)] \\ &= \mathbb{E}_{q(z)}[\log p(\mathbf{x}|z, \theta)] - D_{KL}[q(z) \| p(z)] \end{aligned} \quad (2.18) $$

this decomposition, that we can call prior-contrastive, clearly shows the trade-off between the divergence term $\mathbb{E}_{q(z)}[\log p(\mathbf{x}|z)]$, that evaluates the data likelihood under the model $p_{\theta}(\mathbf{x}|z)$, and the divergence between approximated posterior $q(z)$ and the real prior $p(z)$. In the optimal case, the second term equals 0, such that the likelihood $p(\mathbf{x}|z)$ is maximal. This decomposition then points out two possible sources of sub-optimality, that can prevent the divergence to cancel and then lowering the bound $\mathcal{L}(q, \theta)$: formulating a distribution $q(z)$ is unable to match the prior $p(z)$, or choosing a prior distribution $p(z)$ that is not optimal for the overall model $p_{\theta}(\mathbf{x}, z)$. Additionally, sub-optimality can also emerge from the Kullback-Leibler divergence $D_{KL}$ itself, leading the variational distribution $q(z)$ to degenerated solutions. In the following paragraphs, we will shortly detail the origin of these issues, as well as various solutions that can be taken to alleviate this degeneracy.

**Mean-field approximation.** We saw from eq. 2.18 that the lower-bound $\mathcal{L}(q, \theta)$ was penalized by the divergence between the approximated posterior $q(z)$ and the prior $p(z)$. Hence, if the chosen family variational $\mathcal{Q}$ for the distribution $q$ is not able to match the prior $p(z)$, an unavoidable cost will be added to the lower-bound (2.14), and providing poorly approximated models. Mean-field distributions, by example, cannot express any dependency between latent variables and latent dimensions, can then provide poor approximations of the distribution $p(z|x)$.

Researchers then investigated solutions to make these distributions more expressive, reducing the gap to provide a tighter lower bound. Tractable variational strategies to express latent covariance was proposed by Saul & al. [57], allowing partial grouping of dimensions $q_j$. Some authors rather propose to replace the common Gaussian
assumption for both prior and posterior with more complex distributions, such as mean-field exponential families [58, 59], increasing the expressive power of the involved models. Dirichlet processes[60], such as Stick-Breaking Processes [61] or nested Chinese Restaurant Processes (nSCRP) [62] can be beneficially used in variational inference (see fig 2.11), empowering the flexibility of the variational approximation. Alternatively, mixture of gaussians [63], Gaussian processes [55] or graphical models with hierarchical conditioning [64] or auxiliary variables [65] (see fig. 2.12) can be leveraged to increase the expressiveness of the variational family.

While showing the flexibility of variational approaches, the explicit definition of a given family for the variational distribution can also be seen as a limitation. Implicit methods rather propose to infer variational distributions without explicitly modeling their parameters, using distribution matching methods to regularize the intractable posterior with the prior. A way to perform implicit distribution matching is to use implicit distributions, modelled by a function $z = f(\epsilon)$ over some source of noise $\epsilon$, that is learned during training to shape the posterior [66]. As the expression of its density function is intractable, comparing the two distributions using respective samples can be achieved with Density Ratio Estimation (DRE) [67], estimating the ratio between both densities $p(z)/q(z)$ to evaluate their similarity (see fig. 2.13)

<figure><img src="image_13.png" /><figcaption>Figure 2.11: Example of hierarchical stick-breaking process, allowing to define arbitrary partitions over a probability space</figcaption></figure>

$$r(x) = \frac{p(x))}{q(x)} = \frac{\hat{p}(x|y = 1)}{\hat{q}(x|y = 0)} = \frac{\mathcal{D}(x)}{1 - \mathcal{D}(x)} \quad (2.19)$$

Hence, the Kullback-Leibler divergence in (2.18) can be replaced by a DRE estimator, allowing us to estimate the divergence between the two implicit distributions. The application of implicit regularization for latent inference have been proposed by Dumoulin & al. with the Adversarially Learned Inference framework [68] (ALI), and was then proposed in variational learning with by Makhzani & al. [69] and Mescheder & al. [70]. Alternatively, Laplace approximation of implicit distributions for can be used to compute their $D_{KL}$ [71]; however, this method do not alleviate the weaknesses brought by mean field asymmetries.

While these implicit variational methods are very efficient to generate expressive distributions, they suffer from two main drawbacks. First, AVB and ALI are both based on a discriminator, that can be trained to accurately separate the two distributions, then introducing additional parameters and replacing the choice of variational distribution by the choice of a discriminator. Secondly, implicit methods remove the interpretability of the variational distributions, as the underlying distribution between the model $q(z)$ is not explicitly defined. Some methods, such as Semi-Implicit Variational Inference, addresses the second issue by formulating a two-stage variational distribution $q(z) = \int q(z|\psi)q(\psi)$, where the distribution $q(\psi)$ is allowed to be implicit while $q(z|\psi)$ is held explicit [72]. The same principle can also be applied to the prior $p(z)$ [73], allowing to define both prior and variational distribution as semi-implicit. Alternatives to DRE can also be chosen, matching the

<figure><img src="image_14.png" /><figcaption>Figure 2.12: Auxiliary variables allow to increase the expressiveness of the inference model, but does not modify yet the generative distribution.</figcaption></figure>
<figure><img src="image_15.png" /><figcaption>**Figure 2.13:** Density ratio Estimation uses a classifier to separate one distribution from the other, and replaces the original probabilities by classification rates</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>**Figure 2.14:** Effect of $D_{KL}$ asymmetry on with variational distributions. As $q(z)$ cannot model $p(z)$, the effect of $D_{KL}$ asymmetry entails a sub-optimal distribution matching, whether *mode-focusing* (above) or *mass-covering* in the other (below)</figcaption></figure>

moments of the two distributions with kernel density ratio fitting [74] or maximum mean discrepancy [75] (see below).

**$D_{KL}$ sub-optimality** Kullback-Leibler divergence is the most used way of computing the divergence between two distributions, as it naturally occurs in information theory (see sec. 2.1.4) and has a tractable expression for most distributions. However, some properties of the $D_{KL}$ can also prevent the optimal maximization of the variational distribution. $D_{KL}$ can be expressed

$$D_{KL}[p(x) \| q(x)] = \int p(x) \log \frac{p(x)}{q(x)} dx \quad (2.20)$$

$D_{KL}$ is asymmetric, and then has two different behaviors depending on the order of the involved distributions. Optimizing $D_{KL}[q(z) \| p(z)]$, will enforce $q(z)$ to have a *mode-seeking* behavior, heavily penalizing zones of $q(z)$ where $p(z)$ is low (hence called *zero-forcing*). This property then rather under-evaluate the variance of the original $p(z)$, focusing on the mode but maybe missing most of the target support. Reversely, optimizing $D_{KL}[p(z) \| q(z)]$ will push $q(z)$ to cover all the support of $p(z)$ to have a *mass-covering* behavior, then over-estimating the target variance (hence *zero-focusing*). While these properties are inactive if the chosen variational family $\mathcal{Q}$ is able to match the real posterior perfectly, this behavior will be decisive in case of sub-optimal variational approximations (see fig. 2.14).

Hence, while altering the original formulation of variational learning, alternative divergences have been proposed to replace the original $D_{KL}$ of (2.18). In addition to the DRE estimation presented above for implicit distribution matching an important alternative divergence is the *Rényi Divergence* (RD), that is derived from an extension of the original notion of Shannon entropy (see section 2.1.4). This divergence depends on a parameter $\alpha$, that balances between under- and over-estimation of the divergence: $\alpha \leftarrow \infty$ enforces a zero-forcing behavior, and reversely $\alpha \leftarrow 0$, enforces a zero-avoiding optimum [76]. Note that, in that case, $\mathcal{L}_{\alpha}$ will be a lower bound to $p(\mathbf{x})$ if $\alpha > 0$, and an upper-bound if $\alpha < 0$. Li & al. then propose to sandwich $p(\mathbf{x})$ with several $\alpha$, ensuring the optimality of the solution [77]. Another divergence, $\chi^n$-divergence, is based on the same principle, increasing the its mode-seeking behavior when increasing the power index $n$. The $\beta$-divergence and $\gamma$-divergence [78] are rather adjusting the sensitivity of the divergence to low-probability zones of $p(\mathbf{x})$, allowing to adjust its sensitivity to outliers. The proposed $\alpha\beta$-divergence propose to blend Renyi and $\beta$-divergences, allowing to trade between both robustness/sensitivity and mode-seeking/mass-covering behavior [79].

Alternatively, the generalization of divergences using external defined convex functions $f$ were proposed in the literature, providing a theoretical basis for probability metrics. $f$-divergences, deriving a divergence for any convex function $f$ verifying $f(0) = 1$, such that
$D_f[p||q] = \int q(\mathbf{x}))f(\frac{p(\mathbf{x})}{q(\mathbf{x})})d\mathbf{x}$. By example, choosing $f(x) = x \log x$ covers the $D_{KL}$ 2.20 [80]. This divergence can be approximated for any valid $f(\cdot)$ with a Taylor expansion of the function, or resorting to DRE. Furthermore, such functions $f$ can be understood as a restricted class of cost functions used for the optimal transport formulation of probabilistic inference (see sec 2.1.3.3). Alternatively, Bregman divergences rather defines the divergence $D_f = f(p) - f(q) - \langle \nabla f(q), p - q \rangle$, where $\nabla f(q)$ is the first-order Taylor expansion of $f(q)$ and $\langle \cdot, \cdot \rangle$ is the scalar product. A noticeable property of Bregman divergences is that they can be used as a substitute for affine connections in the field of information geometry, as they define a dual potential through Legendre transform. This property of Bregman divergences allow to generalize many techniques of optimization theory to be performed in the space of probability distributions, that has been used for example in audio analysis with Music Information Geometry by Cont & al. [81].

Finally, alternative divergence methods leverage statistical testing methods to evaluate the divergence directly using samples for the distributions, allowing an implicit approach to distribution matching. Maximum Mean Discrepancy (MMD) is a two-sample test that computes distance between the means of the distributions by projecting the samples into a kernel Hilbert space (RKHS) [75]. MMD can be formulated

$$ \text{MMD}_f(p(\mathbf{z}), q(\mathbf{z})) = \sup_{f \in \mathcal{F}/\mathcal{M} \mathcal{H} \mathcal{C} \mathcal{A} \mathcal{H} \mathcal{C} \mathcal{A} \mathcal{L} \mathcal{F}} \left( \mathbb{E}_{p(\mathbf{x})}[f(\mathbf{x})] - \mathbb{E}_{q(\mathbf{x})}[f(\mathbf{y})] \right) \quad (2.21) $$

$$ = \| \int_{\mathcal{E}} k(z, \cdot) dp(z) - \int_{\mathcal{E}} k(z, \cdot) dq(z) \| \quad (2.22) $$

where an usual choice for $k$ is the radial-basis function kernel $k(\cdot, \cdot) = \exp(\mathbf{x} - \mathbf{x}')^2$. MMD have been successfully applied to generative model as a training criterion in the data domain [82, 83], but can also be used as a dissimilarity measure between the two probabilities $p(\mathbf{z})$ and $q(\mathbf{z})$ [84, 85] (see sec. 2.1.3.3).

### 2.1.3.3 Optimal transport.

Some recent approaches propose to address statistical inference under an optimal transport perspective. Optimal Transport (OT) is a mathematical problem that consists of finding the optimal coupling between two probability spaces under a given cost function [86]. This general problem has been extensively studied in various fields such as biology, fluid dynamics, or physics, even recently in audio processing [87]) but also in the domain of probabilities, then recently introduced in the machine learning field. Indeed, statistical inference of latent from observed variables can be understood as finding a transport map between $\mathbf{x}$ and $\mathbf{z}$, optimized using a given cost function $c$. Defining $\Lambda(p, q)$ the space of joint probabilities whose respective marginals are
<figure><img src="image_17.png" /><figcaption>**Figure 2.15:** Optimal transport studies the distribution over mappings $\Gamma(\mathbf{x}, \mathbf{z})$ to transform $p(\mathbf{z})$ in $q(\mathbf{z})$ that minimizes the cost $c(\mathbf{x}, \mathbf{G}(\mathbf{x}))$</figcaption></figure>

$p(\mathbf{x}, \mathbf{z})$ and $q(\mathbf{x}, \mathbf{z})$, optimal transport can be formulated

$$W_c(p, q) : \inf \gamma \in \Lambda(p, q) \int c(z, \mathbf{x}) d\gamma(z, \mathbf{x})$$

that is a divergence between the two probabilities $p$ and $q$. Generative models can then be formulated from an optimal transport point of view, finding the transport map from the empirical distribution $p*(\mathbf{x})$ to a target distribution $p(\mathbf{x})$ and minimizing the above Wasserstein divergence. While some methods are based the direct optimization of this problem, using Minimum Kantorovitch Estimators or Sinkhorn's algorithm [88, 89], we can also use OT formulate a latent inference problem by taking a distribution $p(\mathbf{x})$ that is itself obtained from the transformation of a variable $z$, assuming $\Lambda(p, q) = \int_{\mathcal{X}} p_G(\hat{\mathbf{x}}|z)q(z|x)p_X(x)dz$, such that $q(\mathbf{z}) = \int q(\mathbf{z}|\mathbf{x})p_X(\mathbf{x})dx = p_Z(z)$ for all $z \in \mathcal{X}$ [90], such that the ideal OT optimization problem can be taken as (see fig. 2.15)

$$W_c(P_X, P_G) \le W_c^\dagger(P_X, P_G) = \inf Q: Q_Z = P_Z \mathbb{E}_{P_X} \mathbb{E}_{Q(Z|X)}[c(X, G(Z))] \quad (2.23)$$

where $W_c^\dagger$ is optimal if the generative function is atomic, such that $p_G(\hat{\mathbf{y}}|z = z)$. A relaxed formulation can be written

$$W^\beta(P_X, P_G) = \inf_{q(z|x)} \mathbb{E}_{P_X} \mathbb{E}_{Q(Z|X)}[c(X, G(z))] + \beta F(Q)$$

where $F$ is a regularizer such that $F(Q) = 0$ when $p(\mathbf{z}) = q(\mathbf{z})$. The comparison with the ELBO 2.18 is straightforward, taking the cost $c(\mathbf{x}, G(\mathbf{z}))$ as the likelihood function. However, the main difference with the variational formulation is the regularization term, applied on the *aggregated posterior* $q(\mathbf{z})$ instead of $p(\mathbf{z})$. Furthermore, Bousquet & al. argue that the noise added to the decoder, such as in the case of VAE where the output is a factorized normal distribution, add a penalty of $\sum_{i=1}^D \sigma^2$ to the objective 2.23 and indubitably yields a non-optimal solution. This conclusion reinforces the propositions of implicit regularization methods, such as Adversarial Bayes or. Based on this conclusion, Wasserstein Auto-Encoders (WAE) then propose to use Maximum-Mean Discrepancy as a latent regularizer (see divergences), and propose an OT formulation of the Adversarial Auto-Encoder [91].

Alternatively, the Wassertein distance can also be used in variational inference to optimize the divergence $\mathbb{L}(\cdot, \cdot) = W_c(p(\mathbf{z}, \mathbf{x}), q(\mathbf{z}, \mathbf{x}))$. This divergence can be approximated with a Monte-Carlo estimator $\mathcal{L}_C(p_n, q_N) = \inf_{\lambda} \sum_{j,k} C^{p,q}(x_1^{(j)}, z_1^{(j)}, x_2^{(k)}, z_2^{(k)}),$ that generalizes $f$-divergences taking $C^{p,q}(x_1, x_2) = f(p(x_2)/q(x_2))$.

### 2.1.4 Information theory and latent models

Before moving on to dimensionality reduction and representation learning methods, we first introduce some important notions of information theory, that will help us to bridge the concept of latent variables
with the one of *representation*. Information theory, known to be initiated by Shannon in telecommunications [14], is a seminal sub-field of signal processing that leverages probabilities to quantify the amount of *information* brought by a given signal, centered on the notion of *entropy*. Information-theory based signal analysis can be seen as a probabilistic framework that aims to describe global features of the signal, estimating its probability law and quantifying the amount of *order* of the underlying generative process. Information theory then allows to formulate the notion of *latent code*, that describes the underlying structure of the signal by extracting the minimum description of the system that maximizes the information. The notion of code can thus be related to the idea of *latent variables* presented above, and will provide the essential interface between *compression* algorithms and Bayesian learning that grounds the results of this thesis. Furthermore, the *code* of a given signal can be understood as a way to describe the inner structure of a signal, discarding its non-informative content. Hence, extracting a code from a given signal can be thought as extracting its structuring laws, bridging with generative models.

### 2.1.4.1 Information measures

**Shannon Entropy.** The central notion of information theory is then the notion of *information*, that can be defined as the amount of knowledge brought by an observation to the analysed system. This quantity can be intuited by the notion of *surprise* brought by an element of the signal, considering that an unlikely observation will bring a more substantial amount of information than an observation that would be already considered probable. Hence, it also relates the amount of *redundancy* of the signal, that is a fundamental notion of compression algorithms and signal analysis. Information theory then uses a probabilistic framework to quantify the order amount of a signal, taking inspiration from thermodynamics with the notion of *entropy*. The entropy of a signal x with possible values {$x_1...x_N$} can be given by the following expression

$$ \mathbb{H}[X] = - \sum_{i=0}^{N} p(x_i) \log p(x_i) \qquad (2.24) $$

that can be seen as the expectation of the *information content* – $\log p(x_i)$ of each symbol. As $p(x_i) \in [0; 1]$, we can see that a very low probability event will bring an important amount of information, while an almost certain element will bring very low information. The entropy then quantifies the *amount of disorder* of a signal, and then also describes its *uncertainty*. By example, the uniform distribution $U[a, b]$ is the maximal entropy distribution that can be defined on a support $]a, b]$, while the entropy of the empirical distribution of a time series $\frac{1}{N} \sum_{i=1}^{N} \delta_{x_i}$ has a minimum entropy – $\log N$, that tends to 0 when number of elements tend to 1. A similar notion can be derived in the continuous domain, called *differential entropy*)

$$ \mathbb{H}[X] = \int p(x) \log p(x) dx \qquad (2.25) $$
that defines a continuous entropy estimator using the probability density function $p(x)$, replacing the discrete values taken by the previous $p(x_i)$ by infinitesimal interval $dx$. This notion is thus fundamental, describing the amount of uncertainty of the continuous signal and then quantifying is compressibility. The definition of entropy can be extended to describe the uncertainty of mutual incoming signals, then quantifying their mutual amount of redundancy, even if they are defined on separate domains. The joint entropy can be then defined

$$ \mathbb{H}[X, Y] = \int p(x, y) \log p(x, y)dxdy \qquad (2.26) $$

describing the entropy of the joint probability of the two signals. If $X$ and $Y$ are independent, such that $p(x, y) = p(x)p(y)$, the entropy of both variables are summed: $\mathbb{H}[X, Y] = \mathbb{H}[X] + \mathbb{H}[Y]$. If the signals are not independent, meaning that they share some information, the joint entropy will be lower than their individual sum. Joint entropy then also quantifies the correlation between two signals, cancelling if they share the same probability density. Another interesting value is the conditional entropy, that computes the additional amount of information brought by a random variable $X$ given a another random variable $Y$

$$ \mathbb{H}[X|Y] = \int p(x, y) \log p(x|y)dxdy \qquad (2.27) $$

such that $\mathbb{H}[X|Y] = \mathbb{H}[X, Y] - \mathbb{H}[Y]$, meaning that the joint entropy $\mathbb{H}[X, Y]$ is given by the individual entropy $\mathbb{H}[Y]$ plus the additional amount of information $\mathbb{H}[X|Y]$. A related quantity, that will be important in the frame of variational methods, is the mutual information

$$ \mathbb{I}[X, Y] = \int p(x, y) \log \frac{p(x, y)}{p(x)p(y)} dxdy \qquad (2.28) $$

that quantifies how many information can be gained by observing a variable when the other one is observed, or equivalently how the amount of independence between $X$ and $Y$. Unlike conditional entropy, mutual information is a symmetric estimator, and can be understood as being the $D_{KL}$ (see 2.20) divergence between the joint probability $p(x, y)$ and the product $p(x)p(y)$

$$ \mathbb{I}[X, Y] = D_{KL}[p(x, y)\|p(x)p(y)] \qquad (2.29) $$

such that mutual information cancels if the two variables are independent, as $p(x, y) = p(x)p(y)$. Information theory provides us estimators for quantifying the uncertainty amount of signals, and provides a framework to evaluate the amount of information shared between signals. Note that while this theory was initially developed for time series analysis, there is no explicit assumption relation to time, such that these estimators can then be also used on other types of data. Information theory can then be used to describe global properties of the involved distributions, involving all examples of the target data. Connecting with the previous section, information-theoretic estimators can then also be used in Bayesian inference, especially with latent models where it can derive interesting estimators between $x$ and $z$.
**Rényi entropy** The original formulation of entropy was originally designed to ensure information extensibility, i.e. the summability of information given two independent events. $H[x, y] = H[x] + H[y]$. Shannon's entropy is only the first solution of this constraint, the second one being the *Rényi Entropy* [92]

$$H_{\alpha}[p(\mathbf{x})] = \frac{1}{1-\alpha} \log \int_{\mathbb{R}^d} p(\mathbf{x})^{\alpha} d\mathbf{x} \quad (2.30)$$

that is then dependent of a parameter $\alpha$, tuning tune to importance of low-probability events in the divergence. Renyi entropy recovers the *collision entropy* for $\alpha = 2$, that is a well-known entropy estimator reducing the bias for short series. Rényi entropy also generalizes other estimators, such as the *Hartley entropy* $H_0[p(\mathbf{x})] = \log|\text{supp}(\mathbf{x})|$ or *min-entropy* $H_{+\infty}[p(\mathbf{x})] = -\log \max p(\mathbf{x})$ [93]. Similarly to Shannon's entropy, we can derive a corresponding divergence, thus called *Rényi Divergence* [76]

$$D_r^{\alpha}[p(\mathbf{x})] = \frac{1}{\alpha - 1} \int_{-\infty}^{\infty} p(\mathbf{x})^{\alpha} q(\mathbf{x})^{1-\alpha} d\mathbf{x} \quad (2.31)$$

that is also dependent of a parameter $\alpha$. Similarly to Rényi entropy, Rényi divergence is also a generalization of KL divergence (that it equals at $\alpha = 1$), such that $D_0^r[p\|q] = -\log q(p > 0)$, and $D_{\infty}^r[p\|q] \log \text{esssup}_q^p$ is the worst-case regret in the minimum description length principle. Some noticeable values for $\alpha$ are also $\alpha = 0.5$, where in this case $D_{0.5}^r[p\|q] = -2\log 1 - \text{Hel}^2[p\|q]$ and $\alpha = 2$, where $D_2^r[p\|q] = \log 1 + \chi^2[p\|q]$ Another interesting property is that, R'enyi divergence is also continuous on $\alpha$, with a discontinuity at $\alpha = 1$.

### 2.1.4.2 Latent rate-distortion trade-off

Information theory then provides global estimators describing the amount of uncertainty of given distributions. These estimators can then be used to provide an interesting understanding of Bayesian models, analyzing the dependence between latent variables $\mathbf{x}$ and $\mathbf{z}$ in terms of information. More precisely, it can quantify how much the provided latent variables act as a *code* of the data, then bridging the Bayesian concept of *latent factors* and the idea of *compressibility* of the observed data. This possible equivalence between latent factors and data compression will allow us to bridge it to the idea of *representation*, that will be seminal in this thesis. Here, we will focus on *rate-distortion* theory, that studies the possible amounts of information shared by a set of data samples and its corresponding latent code.

*Rate-distortion* theory [94] studies the relation between the length of a compression code, called the *rate*, and *distortion* by retrieving a given signal from its corresponding code. Given a distribution $q(\mathbf{z})$ for the code and a generative distribution $p(\mathbf{x}|\mathbf{z})$ (then similar to decompres- sion), the *distortion* can be expressed as

$$\mathbb{D}[p, q] = \mathbb{E}_{\mathbf{x} \sim p(\mathbf{x}), \mathbf{z} \sim q(\mathbf{z}|\mathbf{x})}[d(\mathbf{x})] \quad (2.32)$$

<figure><img src="image_18.png" /><figcaption>**Figure 2.16:** Rate-distortion trade-off : given a distribution $p(\mathbf{x}, \mathbf{z})$, there is an intrinsic trade-off between optimal rate $R$ and optimal distortion $D$, threshold by the entropy $H$</figcaption></figure>
given a distortion measure $d(\cdot)$. In Bayesian optimization, this measure can be seen as the likelihood of the data given $\mathbf{z}$, such that $d(\mathbf{x}) = p_{\theta}(\mathbf{x}|\mathbf{z})$. Estimating the *rate* of the representation is more difficult with codes defined in $\mathbb{R}$, as the notion of *code length* does not really exist in the continuous domain. TIshby & Zaslavsky then propose to define the rate as the mutual information between the data and the code obtained from $q(\mathbf{z}|\mathbf{x})$ [95]

$$ \mathbb{R}[p, q] = \mathbb{E}_{\mathbf{x} \sim p(\mathbf{x}) \mathbf{z} \sim p(\mathbf{z}|\mathbf{x})} \left[ q(\mathbf{z}|\mathbf{x}) \log \frac{q(\mathbf{z}|\mathbf{x})}{p(\mathbf{z})} \right] \quad (2.33) $$

Distortion and rate can be used to evaluate the efficiency of the compression, as minimizing the first will reduce the error of the reconstruction, and minimizing the second will reducte the code length. We can define the *representation mutual information* by quantifying the mutual information using the implicit *aggregated posterior* $q(\mathbf{z})$, that is the representation obtained by marginalizing the posterior $q(\mathbf{z}|\mathbf{x})$ over $\mathbf{x}$ [96]

$$ \mathbb{I}_{repr}[p, q] = \mathbb{E}_{\mathbf{x}, \mathbf{z} \sim p(\mathbf{x}, \mathbf{z})} \left[ q(\mathbf{z}|\mathbf{x}) \log \frac{p(\mathbf{x}, \mathbf{z})}{p(\mathbf{x})q(\mathbf{z})} \right] \quad (2.34) $$

where $p(\mathbf{x}, \mathbf{z})$ is defined as $p(\mathbf{x})q(\mathbf{z}|\mathbf{x})$. We can show that this value is lower-bounded by the distortion and upper-bounded by the rate

$$ \mathbb{H}[p(\mathbf{x})] - \mathbb{D}[p, q] \leq \mathbb{I}_{repr}[p, q] \leq \mathbb{R}[p, q] \quad (2.35) $$

such that, if both $\mathbf{x}$ and $\mathbf{z}$ are independent, $\mathbb{I}_q = 0$ and then $\mathbb{R} = \mathbb{H}$ while $\mathbb{D} = 0$, meaning that the code is optimum (*auto-encoding limit*). Reversely, $\mathbb{R} = 0$ implies $q(\mathbf{z}|\mathbf{x}) = p(\mathbf{z})$, showing that the rate is quantifying the extra-cost we pay by not fitting the prior distribution. In that case, $\mathbb{D} = \mathbb{H}$, that is called the *auto-decoding limit*. Rate-distortion decomposition can be derived to obtain as minimization criterion to obtain an optimal representation $\mathbf{z}$

$$ \begin{aligned} & \min \mathbb{D} + \beta \mathbb{R} \\ & \min \mathbb{E}_{\mathbf{x} \sim p(\mathbf{x}), \mathbf{z} \sim q(\mathbf{z}|\mathbf{x})} \left[ -p(\mathbf{x}|\mathbf{z}) + \beta \frac{q(\mathbf{z}|\mathbf{x})}{p(\mathbf{z})} \right] \end{aligned} \quad (2.36) $$

that is closely relating to the ELBO (2.18), plus a factor $\beta$ controlling the rate/distortion rate. Here, we show that the variational objective inherently achieves a trade-off between minimum distortion (how can we achieve maximum likelihood with a model $p(\mathbf{x}, \mathbf{z})$) and minimum rate (that is the compression of the representation). The *rate* can then be defined as the distribution matching between the posterior $q(\mathbf{z}|\mathbf{x})$ and $p(\mathbf{z})$, that can be intuited as the average additional number of bits we have to add to $p(\mathbf{z})$ to identify $p(\mathbf{z}|\mathbf{x})$. A high rate thus means of lower sensitivity of the latent space to the data, and if the prior $p(\mathbf{z})$ is an isotropic normal distribution, an higher independence of the dimensions. This last observation is very important if we want the extracted latent factors $\mathbf{z}$ to be disentangled (see section 2.3.2.4).
### 2.1.4.3 Minimum description length and bits-back coding.

In information theory, modeling the signal is then related to compression, leveraging the redundancy of the signal to reduce its code length whether without altering the contained information (called lossless compression), or targeting an subset of information. The obtained compression code can be considered as a representation of the signal of lower dimensionality. Hence, we can presume that, given a subset of signals, the compression with the best rate is the one that manages to retrieve the underlying structure of the signal. If we consider the target data as generated by a set of factors, finding an optimal compression amounts to recovering these generative factors, retrieving the optimal representation from the signal. An interesting bridge then happens between statistical inference and variation theory : optimizing a model $p_{\theta}(x)$ with K parameters on a dataset with N data can then be considered equivalent as finding an optimal code of length K given a fixed model, defined by the probability distribution. Defining a latent model $z$ can then be defined in a similar way, where $z$ acts as the compressed information about $x$.

A theoretic framework based on this philosophy is the Minimum Description Length principle (MDL), proposed by Grünwald & al. [19, 97], for a given dataset, the best model is the one that requires the less amount of bits to be communicated. MDL were developed to use the model size as a minimization criterion, imposing a code length constraint to prevent over-fitting of models. An example of such criteria is the Akaike Information Criterion (AIC), that reformulates the likelihood objective as

$$AIC = 2K - 2 \log p(x|z, \theta)$$

where K would be the total number of parameters (latent variables included). AIC then explicitly adds the code length of the model to the evaluation, and can be used as a criterion for model selection. An MDL approach then consists in defining a cost that the receiver of the message will pay to recover the original data, including the cost of the model itself. Applying MDL to Bayesian learning, we can then define three different costs : a reconstruction cost, that quantifies the errors made by the model, the model cost, that describes the number of parameters of the model, and the code cost, that quantifies the quality of the code [98]. An interpretation of the code cost can enlighten a common point between Bayesian learning and MDL : the more accurately a given hypothesis explains a data, the lower is the cost paid to face new incomers. This idea also provides the background of the free energy of Helmholtz systems, as Markov fields.

An information theoretic formulation of latent models could thus be described as follows : an emitter sends the code $z$ to a receiver, who uses the model $p(x|z)$ to recover the data $x$ (here $z$ represents both latent variables and model parameters). The total code length of this configuration is then the addition of the model code length, the reconstruction code $-p(x|z)$, plus the code cost $-p(z)$, such that the

<figure><img src="image_19.png" /><figcaption>Figure 2.17: Bits-back in MDL formulation of variational inference</figcaption></figure>
cost paid for probable event is little. However, this code definition
does not take into account the stochasticity of the code, as z also
contains random bits added during sampling. Then, if the receiver
has the same encoding algorithm $q(z|x)$ than the emitter, it can also
recover these random bits to reduce the total cost, bring the *bits-back*
(see fig. 2.17). The total cost length of the system is then added a term
$q(z|x)$, the additional cost of random bits that can be recovered. This
learning scheme, called *bits-back coding*, then amounts to the variational
formulation developed previously. The relation between bits-back
coding and variational learning can helps to understand the behavior
of the system from an information-theoretic point-of-view, such that
the *information preference* of variational auto-encoders (see section 2.3.2)
and can motivated the development of specific MDL training criteria
[99] [98].

## 2.2 From dimensionality reduction to deep unsupervised learning

In the previous section, we investigated statistical inference methods
under a Bayesian perspective, modeling a given set of observations
using a parametric model. We also introduce the idea of latent models,
using external variables can be used to *condition* the genera-
tive distribution. The concept of latent variables is seminal in Bayesian
methods, as it allows to explain the data with higher-order generative
parameters. However, Bayesian inference requires some tractability
assumptions over its terms, and thus limit the possible choices gen-
erative models. Fortunately, we saw that the tractability of all the
involved terms could be bypassed by transforming the closed-form
evaluation of the Bayes' rule into an optimization problem, allowing to
model more complex generative distributions. More particularly, we
presented *variational inference*, that allows to maximize the likelihood
by replacing the true posterior $p(z|x)$ by an *approximated* posterior $q(z)$.
This method relaxes the tractability of the posterior, and then allows
to train more complex generative models. However, the variational
methods presented still relied on the tractability of the ELBO 2.18 to
get closed-forms for optimal variational distributions.

In this section, we will address the extraction of high-level features
from the perspective of *representation learning*, a domain based on
dimensionality reduction techniques and motivated by *automatic dis-
covery* of features, visualization, and compression. This approach is
different from statistical inference as it does not assume the invertibil-
ity of the obtained representation, and does not necesseraly resort to
probabilities techniques. Formally, a *representation* can be simply de-
scribed as a function $z = f(x)$ that embed an input space into another
one with additional constraints, by example on its dimensionality,
organization, or enforced to reflect given properties of the data. The
motivations for representation learning algorithms are then quite di-
verse, and highly depending on the target application. *Dimensionality*
*reduction* techniques are focused on the extraction of representations that aims to preserve most of the data diversity, extracting intrinsic sources of variance inside the dataset. This approach can be motivated by *manifold hypothesis*, stating that high dimensional data lies in the neighborhood of a low-dimensional sub-manifold of the input space. Recovering the sub-manifold with dimensionality reduction techniques then targets to reocver the inner structure of the data, and hopefully its natural factors of variation [20, 100].

Representation learning is also widely investigated in the field of *ma-
chine learning*, a family of statistical models based on the optimization
of given set of tasks or criteria. Dimensionality reduction is indeed an
efficient remedy to *curse of dimensionality*, that can make the application
of machine learning methods impossible in high-dimensional input
space. Indeed, the volume unit of a space $\mathbb{R}^D$ grows exponentially
with $D$, such that Euclidean distances between two points tends to be
less significant, converging to some constant [101]. As almost every
machine-learning methods are based on the optimization of Euclidean
metrics of some values of interest, *curse of dimensionality* can be criti-
cal and then prevent to directly use these techniques in the data space.
Therefore, resorting to dimensionality reduction techniques can allow
to alleviate this problem, first projecting the data on a space of lower
dimensionality and then applying the target machine learning task.
Conversely, dimension *augmentation* can also help to sparsify the data,
and then help to separate problems that are not separable in the input
domain.

The recent gain of interest in machine learning has then caused the
development of powerful representation extraction methods, based on
high-capacity function approximators (such as neural networks) that
can be trained to represent a given task. However, defining specific
criteria for the extraction of robust representations, i.e. reflecting under-
lying properties of the data, is a non-trivial task. Indeed, obtaining a
representation trained on explicitly defined tasks such as classification
(hence in a *supervised* setup) can provide degenerated representations,
focusing too much on the target task at the cost of losing the inner
data structure. An alternative framework, called *unsupervised* learning,
rather focuses on automatic feature extraction without resorting to an
external task. However, it still need a suitable criteria to be trained,
giving rise to substantially different approaches. In our case, we are
focusing on both inference and generation of audio signals, such that
the invertibility of the obtained representation is mandatory. Hence,
we resort to *auto-encoding*, an unsupervised framework extracting a
low-dimensional representation based on *reconstruction* criteria.

In this section, we will first present dimensionality reduction methods, that are based on the extraction of statistical, geometrical or topological descriptors of the data to provide low-dimensional representations of the data. Then, we will shortly introduce *neural networks*, widely used high-capacity function approximators that are now grounding
advanced machine learning techniques. Then, we will present *auto-encoders*, and discuss how they are related to *manifold hypothesis* and how they can be used to extract meaningful representations of the data.

### 2.2.1 Dimensionality reduction

In this section, we summarize methods that intend to reflect structural properties of the data based on the extraction of statistical, geometrical or topological descriptors. These dimensionality reduction methods aim to describe the inner variance of a given dataset based on three different criteria: preserving the *variance* of the samples (principal component analysis), preserving the *distance* between the samples (multi-dimensional scaling), or retrieving the underlying sub-manifold of the data (*manifold learning*). In this section, we will shortly summarize these three different methods.

**Principal Component Analysis** *Principal component analysis* is a seminal dimensionality reduction method, first proposed by Wold & al. [102, 103]. The idea underlying PCA is to identify the projection from the data space $\mathbb{R}^D$ to a linear subplane $\mathbb{R}^d$ that retains the maximum variance of the dataset (see fig 2.18). Mathematically, PCA can be achieved using Singular Value Decomposition (SVD), that decomposes the full data matrix $X \in N \times D$ in three matrices

$$X = U\Sigma W^T$$

where the orthogonal vectors U and W are the right- and left- eigenvectors along dimensions N and D, and the matrix Σ is the diagonal projection matrix with singular values $\sigma_1...\sigma_p$. The transformation P: $\mathbb{R}^D \rightarrow \mathbb{R}^d$ that maximizes the covariance $XX^T$ can be shown to be the truncated $W_d$, such that PCA can be obtained with $X_{PCA} = XW_d$. PCA can thus be understood as the projection of X on d-principal components, then maximizing the retained variance. Despite its simplicity, PCA is a cheap and invertible transformation that is still widely used for multivariate data analysis. PCA then motivated numerous improvements such as *Robust PCA* [104], helping the PCA resisting to outliers, *Sparse PCA* that encourages the sparsity of the projecting axes [105], or *probabilistic PCA*, that encodes the uncertainty of the projection using a linear Gaussian model [106].

The main drawback of PCA is that its projection axis can be highly covariant, thus harnessing the dimensions of the projection to represent independent factors. Independent Component Analysis (ICA) reformulates the problem [107, 108] by extracting an orthogonal base from the data space by first whitening the data, and then perform PCA analysis. This method provides an interesting alternative as it enforces the independence of the dimensions, and then to recover independent variations of the data. The extension of ICA to non-linear

<figure><img src="image_20.png" /><figcaption>**Figure 2.18:** PCA will capture the axis that maximizes the variance of the original data</figcaption></figure>
function spaces have been proposed by Hyvärinen, providing theoretical foundations for the uniqueness of solutions [109]. Hence, ICA is typically used in orthogonal component extraction, such as blind source separation methods.

**Distance regularization.** Instead of preserving the variance of the dataset, some methods rather propose to extract low-dimensional representations by preserving the pairwise distances of the original space (see fig. 2.19). These methods then model the distances between points rather than their absolute position, aiming to approximate the implicit metrics of a compact manifold. Multi-dimensional scaling (MDS) is a method based on optimizing the distances on a projected space $\mathcal{D}(z_i, z_j)$ using a mean-squared error on the original distances $\mathcal{D}(x_i, x_j)$. Given $N$ points and the corresponding pairwise dissimilarity measures $\mathcal{D}(x_i, x_j)$ between points $x_i$ and $x_j$, the objective MDS is formulated [110]

$$ \min_{z_1, \dots, z_N} \sum_{i \le j} (\|x_i - x_j\|^2 - d_{i,j})^2 \qquad (2.37) $$

that minimizes the $\ell^2$ norm between original and projected distances. MDS then targets to find isometric projections of the input space, that can be obtained by whitening the distance matrix $\mathcal{D}(x_i, x_j)$ and taking the $m$ highest values of its SVD. However, MDS assumes an Euclidean distances, and does not model the uncertainty of the targeted distances. A probabilistic approach to MDS is provided by Stochastic Neighbour Embedding (SNE), that expresses the probability of the point $x_j$ to belong to a given neighbourhood using the conditional distribution [111]

$$ p(\mathcal{D}_{ij}^x) = \frac{\exp(-\|x_i - x_j\|^2 / 2\sigma^2)}{\sum_{k \neq i} \exp(-\|x_i - x_j\|^2 / 2\sigma^2)} \qquad (2.38) $$

where $\sigma^2$ is the Gaussian variance of the corresponding kernel, and $\mathcal{D}_{ij}^x$ represent the distance between the points $x_i$ and $x_j$. The probability $p(x_j|x_i)$ can then be understood as a multinomial distribution over the most probable neighbor $x_j$. SNE preserve this conditional distribution in the target representation $z$, where a similar density can be defined

<figure><img src="image_21.png" /><figcaption>Figure 2.19: MDS tries to matches the distances of the original space in the embedded space</figcaption></figure>

$$ q(\mathcal{D}_{ij}^z) = \frac{\exp(-\|z_i - z_j\|^2 / 2\sigma^2)}{\sum_{k \neq i} \exp(-\|z_i - z_j\|^2 / 2\sigma^2)} \qquad (2.39) $$

We can then match these two tractable distributions by minimizing the $D_{KL}$ between $p(\mathcal{D}_{ij}^x)$ and $q(\mathcal{D}_{ij}^z)$

$$ \mathbf{L}_{SNE} = \sum_{i} D_{KL}[p_i \| q_i \|] = \sum_{i,j} p(\mathcal{D}_{ij}^x) \frac{p(\mathcal{D}_{ij}^x)}{p(\mathcal{D}_{ij}^z)} \qquad (2.40) $$

whose optimization is tractable using gradient descent. SNE then performs smooth dimensionality reduction method using a probabilistic model of the input pairwise distances, then providing more robust representations than MDS. Moreover, MSE intrinsically performs unsupervised clustering by minimizing the distances between close points
while pushing the others away, sparsifying the obtained representa-
tion. However, SNE is particularly subject to curse of dimensionality
because of the *crowding effect*, as the Euclidean distance between origi-
nal points, that exponentially grow with the number of dimensions.
This undesirable effect thus tends to cancel out the saliency of the
inter-point distances, and provide degenerated projected representa-
tions where the distances are preserved by projecting every points on a
small support around 0. This undesirable effect can be alleviated using
$t$-SNE, that rather models the distances using a Student-$t$ distribution

$$
p(x) = \frac{\Gamma(\frac{\nu+1}{2})}{\sqrt{\nu\pi}\Gamma(\frac{\nu}{2})} \left(1 + \frac{x^2}{\nu}\right)^{-\frac{\nu+1}{2}} \quad (2.41)
$$

where $\Gamma(x)$ is the Gamma function, and $\nu$ is the degree of liberty of the distribution. Student-t distribution is a common conjugate distribution of the normal distribution (see sec 2.3), modeling the uncertainty of its mean vector once the variance has been marginalized out

$$
p(\mu|\mathcal{D}) = \int p(\mu|\mathcal{D}, \sigma^2)p(\sigma^2|\mathcal{D})d\sigma^2 \quad (2.42)
$$

where $p(\mu|\mathcal{D}, \sigma^2)$ is a normal distribution centred on the approxi-
mated mean of the observations $\mathcal{D}$, and $p(\sigma^2|\mathcal{D})$ is a scaled inverse
chi-squared distribution. The marginalization of the variance provides
an heavier tail than a standard normal distribution, hence giving a
higher probability to distant data points. Taking $\nu = 1$, we can thus
express the probability density of the distances in 2.39 with [112]

$$
q(\mathcal{D}_{ij}^z) = \frac{(1 + \|\mathbf{z}_i - \mathbf{z}_j\|^2)^{-1}}{\sum_{k \neq i} (1 + \|\mathbf{z}_i - \mathbf{z}_j\|^2)^{-1}} \quad (2.43)
$$

This formulation of SNE, then called *t*-SNE, is much more robust to
high-dimension input spaces, and is still provides one of the most
used techniques for dimensionality visualization. *t*-SNE will provide
the inspiration of the Student-*t* regularization method, that we will
develop section 3.4.

**Manifold Learning methods** The main flaw of PCA is that it projects the data onto a *linear* sub-space, hence unable to accurately represent dataset that lie on non-linear sub-manifold of the original space. For example, if the data is organized around a curvated surface (such as a sphere), the representation provided by the PCA will project points placed on opposite locations on the same position. Non-linear approaches to PCAs have also been developed, such as Kernel PCA [113], but non-linearities often makes the projection basis non-unique and then intractable with formal calculus.

Some methods propose instead to extract non-linear subspaces using
the concept of a *manifold*, that is an embedded surface that is *locally dif-
feomorphic to an Euclidean plane of dimensions *d* < *D*. This manifold
can then be retrieved by taking local linear maps taken from the neigh-
bourhood of each point, and then gluing the maps to infer a system
of coordinates. This idea was first investigated by Roweis al. &. [114] using local PCAs, and then improved with Laplacian Eigenmaps [115] and Hessian eigenmaps [116], rather using local Eigenmaps to perform linear approximation. Alternatively, Local Tangent Space Alignment aims to find projections that aligns the approximated tangent spaces of each data point, hence flattening the manifold [117].

Another approach rather consists in estimating the intrinsic metrics of the underlying manifold by computing pairwise distances along the manifold's surface. This can been done by constructing a graph representation of the manifold by linking neighbouring data points, such that each vertex is a data point and each edge is the distance between the points. The *geodesics* between two points can then be estimated finding the shortest path linking them, summing the lengths of the crossed edges. This construction is used by the ISOMAP method [118], that proposes to perform a MDS with the estimated geodesics rather than the Euclidean distances, hence preserving the local geometry of the underlying manifold. A similar method is used by Uniform Manifold Approximation and Projection (UMAP) [119], that uses fuzzy topological representations to merge the local projections extracted from the data.

Manifold learning then provides very interesting methods to extract representations from an underlying manifold, helping to capture non-linear axis of variations that would not be represented in linear dimensionality reduction techniques. Furthermore, these methods can provide interesting information about the local structure of the data, finding more natural interpolation between points. However, these methods are often heavy to compute, making them hardly applicable to large and high-dimensional datasets. Additionally, these methods are non-invertible, and are then unsuitable for generative purposes. However, the underlying intuitions behind these methods still nourishes modern advances of machine learning, as Riemannian approaches to auto-encoding (see section 2.3.2.4).

### 2.2.2 Neural networks and back-propagation

While performing well with small datasets, the methods described above suffers from two major drawbacks. First, most of them are hardly scalable to large high-dimensional datasets, as they operate directly on the entire data matrix $x$. Secondly, they are based on simple geometric constraints that may fail to capture the complexity of the data, limiting the expressiveness of provided representations. These challenges motivated the development of alternative approaches based on *stochastic optimization*, allowing to iteratively train systems on random data subsets. A seminal optimization method, called *stochastic gradient descent* (SGD), allows to train arbitrarily complex models by only requiring their derivability, provided a well-defined training criterion. The development of SGD, simultaneously with the ever-growing computational power of modern computers, enabled the efficient integration of *neural*
networks in machine-learning methods, increasing significantly their performance. In this subsection, we will shortly describe neural networks and how to train them with SGD.

**Neural networks.** Neural networks (NN), also called *Artificial Neural Networks* (ANN), can be described as trainable computation blocks composed by small units, called *neurons*, that can be linked to model complex functions of arbitrary complexity. Formerly inspired from the cybernetics and cognitive science fields, neural networks now provides the basis for black-box optimization methods used in modern machine learning techniques, from the simplest (simple classification or regression systems) to most complex applications (such as deep learning). As these systems are now very well documented, we will try to give a brief but intuitive explanation of these systems.

A *neuron* is a unit composed by a linear or affine function $y = f(x) : \mathbb{R}^n \to \mathbb{R}$, then corresponding to a linear subplane of the input space. A neuron is also generally followed by a non-linearity, that will help him to use this segmentation to model non-linear functions of the input space. For example, taking a sigmoid activation $\sigma(x) = 1/(1+e^x)$ (also called logistic as it equals 0 when $x \to -\infty$ and 1 when $x \to \infty$), we are able to split the space in two, where the function equals 1 on the left of the line and -1 on right of the line, such that the line drawn act as a *decision boundary* (fig 2.20).

To give an intuition about the inner working of neural networks, we will present a modern formulation of the *perceptron*, one of the oldest supervised machine learning system. Taking *d* neurons, we are then able to simulate functions $f(\mathbf{x}) : \mathbb{R}^n \to \mathbb{R}^d$. Using sigmoid non-linearities, the system is then able to split the input space in several regions, and then to discriminate up to *d* classes (see fig. 2.21). However, such classification can only work if the target classes are linearly separable. We can alleviate this problem by adding a second layer of neurons, connected to the first, that can increase the modeling capacity of the network. Indeed, if the number of neurons of the hidden layer *H* is lower than the input dimension, the neural network is called a *bottleneck*, as it reduces the dimensionality of the input. Reversely, if the number of hidden neurons is higher, the network is able to discriminate non-linearly separable regions of the input space, as it is embedded in a space of higher dimension (see fig 2.22) for an example with the well-known XOR example). Such networks are then called *fully-connected networks*, or *multi-layer perceptron (MLP)*. However, the choice of a sigmoid activation is not optimal for the hidden neurons, as it bounds the output between $\sigma(x) \in [0; 1]$, and then can weaken the expressiveness of the hidden representation. A common alternative choice is then the *Rectifier Linear Unit* (ReLU), that can be understood as a way to bypass the left part of the neuron's subplane while not affecting the other. Hence, multi-layer perceptrons can be understood as systems that embedding the input space to a hidden representations, that can be trained as an intermediary space used for binary-,

<figure><img src="image_22.png" /><figcaption>**Figure 2.20:** A single neuron, by drawing a straight line in a multi-dimensional space, is able to split it two parts. Adding a non-linearity allows, a hard tanh in the above example, to define a function that equals -1 if far above the line, and +1 far below.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>**Figure 2.21:** Several neurons with classification units can perform complex segmentation of the input space, and can then perform multi-class classification with sigmoid units.</figcaption></figure>
multi-classification, or regression.

**Backpropagation and gradient descent** A neural network can thus be understood as a parametric non-linear function $y = NN_{\theta}(x)$ with an arbitrary number of inputs and outputs, that has to be trained on given pairs $(x, y)$ to model the targeted function. Neural networks then requires the definition of a *loss function*, that has to be accurately designed to fulfill a given task. By example, in a supervised classification setting, the training dataset is composed by pairs ${x, y}_{i=1}^N$, where $x$ is the input of the network and $y$ is the corresponding label information. The network is then typically trained on a classification loss function $\mathcal{L}(\hat{y}, y)$ between the network prediction $\hat{y}_i$ and the true label $y_i$, as a cross-entropy loss.

However, given a neural network with parameters $w_{ij}$ and a suitable loss function $\mathcal{L}(y)$, we still need an optimization method to train the network towards the desired output. Back-propagation algorithms allow us to derivate the gradient $\frac{\partial \mathcal{L}}{\partial w_{ij}}$ of each weight of the network with respect to the loss function $\mathcal{L}(\hat{y})$, thus assuming the full derivability of the system *. Back-propagation is based on a recursive *chain rule* that is applied on the full network, as can be shown fig. 2.23 where a few neurons are shown for simplicity. Once obtained the derivatives $\frac{\partial \mathcal{L}}{\partial w_{ij}}$ for each weight $w_{ij}$, we can optimize the network using *gradient descent*, an optimization scheme that performs iterative updates of the weights using the corresponding derivative

$$w_{ij} \leftarrow w_{ij} - \nu \frac{\partial \mathcal{L}}{\partial w_{ij}} \quad (2.44)$$

where the *step size* $\nu$ is generally shrunk during training. Indeed, if $\frac{\partial \mathcal{L}}{\partial w_{ij}} > 0$, an increase of $w_{ij}$ will lower the loss $\mathcal{L}$, and reversely, if $\frac{\partial \mathcal{L}}{\partial w_{ij}} < 0$, decreasing $w_{ij}$ will decrease $\mathcal{L}$. This procedure is then terminated until convergence $\mathcal{L} \to 0$, or according to a diverse stopping policies. While the optimization was initially performed on the overall dataset, this procedure can be very time- and memory-consuming for large amounts of examples, then limiting the scalability of the method. An alternative optimization scheme, called *stochastic gradient descent* (SGD), rather performs iterations over a small batch of examples chosen randomly in the dataset [120]. This optimization scheme has the same convergence guarantees as the original gradient descent but is much more scalable, as it does not imply to perform back-propagation over the full dataset. Furthermore, it has been shown that the stochastic nature of SGD increased the robustness of the trained model, preventing the system to suffer severe over-fitting. Since the creation of SGD, gradient descent has been a main topic in the optimization field, and still provides the basis of advanced methods such as Adagrad [121], RMSProp[122], or ADAM [123], just to cite the most recent ones.

<figure><img src="image_24.png" /><figcaption>**Figure 2.22:** The well-known XOR example: a problem that is non-separable in an input space can become separable with additional dimensions.</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>**Figure 2.23:** Schema of gradient back-propagation through two neuron layers, using the chain rule to obtain the derivative of the loss function with respect to a single neuron weight.</figcaption></figure>

<footnote>* this hypothesis is not always respected, as for example with the systems using ReLU activation. To this end, the recently proposed ELU activation is often used instead of ReLU, because of its derivability.</footnote>
**Theoretic guarantees of neural networks.** Despite of the concep-
tual simplicity and the efficiency of such *connectionist* approaches, the
lack of interpretability of the network's parameters and their high
over-fitting potential, due to their capacity, often provoked the mis-
trust of statisticians. Indeed, neural networks can be dauntingly hard
to approach with analytical methods, especially with the latter raise of
*deep learning* that could use up to hundreds of stacked hidden layers
(hence called *Deep Neural Networks*, or DNN). Furthermore, some ex-
periments showed that such models can also trivially record the data,
rather than extracting robust features from it. The over-fitting tendency
of deep neural networks is generally alleviated by learning on very
large datasets, providing efficient yet greedy models. Therefore, the
expressiveness provided by these methods can also be part of their
weaknesses, and then requires a particular attention on the obtained
results.

Studying the intrinsic behavior of neural networks is thus very chal-
lenging. First theoretical guarantees investigated by the researchers
focused on the approximation capacity of these systems, and have
been widely studied since the 90s. A seminal universal approximation
theorem, proposed by Hornik, proved the existence of a single hidden
layer MLP that can approximate arbitrary well any function defined
on some compact set [124–126], independently to the chosen activation
and input dimension. However, this theorem does not state about the
number of hidden units required, or about the training procedure. Re-
cently, Lu & al. proved that any function could be approximated with
a width *d* ≠ *n* + 4, regardless of the depth [127]. Hanin showed that it
rather required *d* ≤ *n* + 1, and provided a bound on the approximation
error depending on the network depth’s [128]. While these theorems
evaluate the approximation abilities based on an infinite amount of
input examples, Zhang & al. rather study the approximation capacity
of neural networks on finite samples, showing the existence of a two-
layer neural network with ReLU activations and 2*n* + *d* weights that
can represent any function on a sample of size *n* in *d* dimensions.
Other methods rather foster the expressiveness, i.e. is the diversity of
functions a network can model, and how this diversity evolves with
the network’s depth. By example, Eldan & al. showed that 3-layers
neural networks were able to express functions that a 2-layer network
could not model [129]. Topological analysis of NNs can also provide
interesting results, as shown by Bianchini & al. who proved that topo-
logical capacity of a network increased with its depth [130]., or by
Montufar & al. expressing the number of maximum linear boundaries
of a MLP with arbitrary depth and ReLU non-linearities [131].

While approximation theorems aim to provide theoretical basis on
the modeling capacity of neural networks, some study rather focus
on their learning dynamics. By example, the analytical study of in-
formation propagation in NN underlines a duality between forward
and backward passes [132], preventing to perform SGD in very deep
fully-connected networks (then justifying *residual networks*, that add
skip-connections between layers [133] as shown fig. 2.24). DNNs have

<figure><img src="image_26.png" /><figcaption>**Figure 2.24:** Example of *residual* network, adding skip connections to inject gradient information in deeper layers</figcaption></figure>
also been demonstrated to fail learning simple low cross-predictability
data-label pairs with SGD, underlining the impact of the chosen opti-
mization method [134].

Due to the intractability of neural networks, some authors proposed to develop black-box analysis methods to gain some insights on the neural networks’ internal behavior. *Saliency maps*, for example, estimate the sensibility of each network’s weight with respect to a given input dimension based on its derivative [135–137]. Alternatively, some information-theory grounded methods were proposed to study information propagation in DNNs across layers. However, as neural networks are deterministic, mutual information between layers can be whether infinite or constant. Some estimators can ye bet derived by injecting little amount of noise in the layers’ output, and allows to demonstrate interesting compression abilities as the number of layers increase [138]. The information bottleneck framework can also be leveraged to study neural networks in the case of supervised learning, showing that during the train process the system first reduces the empirical error of the task (empirical error minimization), then compresses its representation [139]. However, such effect has been mitigated by Saxe & al., arguing that this effect is dependent of the used non-linearity [140].

Regarding the generalization abilities of DNNs, most validation proce-
dures are based the evaluation of the network's performance on test
data. However, while over-fitting is an admitted tendency of DNNs,
some authors investigate why DNNs are not over-fitting much more,
unveiling an implicit regularization property pushing them to non-
degenerated solutions [141]. While the compression effect demon-
strated by Tishby & al. could be held responsible, compression can not
be generally held as being related to generalization [140]. This regu-
larization is actually performed by the SGD algorithm, as averaging
gradient among batches of data naturally flattens the error landscape
[142]. Furthermore, it can be shown that with SGD the number of train-
ing examples has much more impact on the generalization than the
model size, explaining why taking very large architectures does not au-
tomatically yield to pathological over-fitting. Despite these theoretical
guarantees generalization of DNNs may be enforced by external meth-
ods, preventing the networks' to converge on degenerated solution.
Similarly to classical regression approaches, weight-normalization of
the weights can be used to prevent over-fitting. Alternatively, Dropout
is a widely used method for improving generalization, consisting in
randomly zeroing hidden dimensions during training to push covari-
ances between networks' weight [143]. However, recent analysis show
that such techniques decrease the capacity of the model, while gaining
too little when compared to data augmentation [144]. Finally, weights
of neural networks can also be considered as random variables, and
then be trained with Bayesian inference [145], or even be trained as
Gaussian Processes [146]. Despite the real generalization gain brought
by Bayesian techniques for neural networks, they are barely used in
the actual machine-learning field because of their low scalability and
their high computational cost.
### 2.2.3 Auto-encoders as invertible representation extractors

Most of machine learning techniques using neural networks are trained on explicit tasks, providing efficient methods for regression, classification, or reinforcement learning. As we saw in the previous sections, deep neural networks can be understood as feature extractors whose layers are progressively transforming the input into a suitable space, whose emerging topology is trained to represent the targeted task. However, the retrieved representations are generally task-dependent, and are not enforced to retrieve structural information about the data itself. Auto-encoders provide an alternative to such supervised methods, extracting a representation from the data without the definition of any external task, but rather trained with a reconstruction objective. Auto-encoders can then be understood as invertible representation extractors, based on the joint learning of two processes: an encoding process, projecting the data in the representation, and the decoding process, that projects back the representation in the data space. As we will see, representations provided by standard auto-encoders are often quite close to the ones obtained with PCAs; however, the development probabilistic formulations of auto-encoding allowed the formulation perform black-box manifold learning techniques, and opens the way to the Auto-Encoding Variational Bayes that grounds this PhD.

**Auto-encoders.** The original auto-encoders, proposed by Rumelhart & al., are built from two distinct 1-layer sigmoid networks: an encoder that transforms the input $x \in \mathbb{R}^D$ into a code $z \in \mathbb{R}^d$, and a decoder, that transforms the output $z$ back to the data $x$ [147] (see fig. 2.25)

$$z = \sigma(\mathbf{A}x + \mathbf{c}) \qquad (2.45)$$

$$x = \sigma(\mathbf{B}x + \mathbf{d}) \qquad (2.46)$$

where the matrices $\mathbf{A}$ and $\mathbf{B}$ are trained with gradient descent on the reconstruction error $\|x, x'\|^2$. Without the sigmoid non-linearity, and provided that $d < D$, auto-encoders converge to PCA, as shown by Baldi & al. [148]. Auto-encoders can then be assumed to perform somehow non-linear PCA, yet being trained by gradient descent. Moreover, the system can be trained using alternative objective functions such as cross-entropy or $\| \cdot \|_1$ to provide different solutions, showing the flexibility of the method.

Auto-encoders thus allow *invertible* representation leaning, both for encoding and decoding functions $f(x)$ and $g(x)$ are trained by gradient descent. As the choice of $d$ is arbitrary, we can distinguish two separate cases. Taking $d \gg D$ provides an *over-complete* representation, embedding the data in a larger space where projections can be sparse. Sparsity can be used for example to linearly separate data points that would not be separable in the input space [149]. Reversely, when $d \ll D$ the representation is said *under-complete*, and the latent code $z$ can then be seen as a *compressed* representation of $x$. In this

<figure><img src="image_27.png" /><figcaption>**Figure 2.25:** Graph of an auto-encoder, that is built from two parts: an *encoding* function, generally a neural network followed by a sigmoid non-linearity, and a *decoding* function, that inverts the representation to recover the corresponding data. The system is then trained using an $\ell^2$ loss between the original example and its reconstruction.</figcaption></figure>
case, the auto-encoders weights are forced to share information among several inputs, entangling the representation. Though, as we are here interested by low-dimensional representations, we will only consider the second case.

**Manifold learning with auto-encoders** While auto-encoding is an efficient method to learn representations from an unsupervised manner, the extracted features are not very robust and can be very sensitive to small changes of data, providing degenerated reconstructions [150]. Furthermore, as the representation is only trained on the projection points of the input data, a direct exploration of the space often produces unrealistic samples, such that the obtained representation is unusable for generation purposes. Hence, Vincent & al. proposed to reinforce the extracted features by adding noise to the input data, learning the auto-encoder to denoise the input data and then to be less sensitive to small perturbations [151]. *Denoising auto-encoders* (DAE) may then be understood as a stochastic operator $p(x|\hat{x})$, that learns to project low-probability noisy observations $x_{\epsilon} = x + \epsilon$ to the closer high-probability point (see figure 2.26). Therefore, DAEs is inherently extracting the true underlying manifold $p(x)$ by learning the transition operator $x + \epsilon \rightarrow x$ [152]. Alain & al. showed that if $\epsilon \sim N(0, \sigma^2I)$, the total loss amounts [153]

$$ \mathcal{L}_{DAE}(r_{\sigma}(\mathbf{x})) = \mathbb{E}\left[\|r_{\sigma}(\mathbf{x}) - \mathbf{x}\|_{2}^{2} + \sigma^{2}\left\|\frac{r_{\sigma}(\mathbf{x})}{\partial \mathbf{x}}\right\|_{F}^{2}\right] + o(\sigma^{2}) $$

showing that adding a noise of variance $\sigma^2$ penalizes the first-order derivative of the reconstruction function $r_{\sigma}(\mathbf{x})$ with respect to $\mathbf{x}$, then also matching also the tangent space of the manifold $p(\mathbf{x})$. This result caused a surge of interest in auto-encoding systems, as the DAEs demonstrated some abilities to catch the inner data manifold $p(\mathbf{x})$. However, DAEs were shown to be robust facing only one type of noise, then limiting the efficiency of the approach. Similarly, *contractive auto-encoders* propose to penalize the input sensitivity of the representation by penalizing the Jacobian of the encoder's output [154]

$$ \mathcal{L}_{CAE} = \|r(\mathbf{x}) - \mathbf{x}\|_{2}^{2} + \sum_{i=0,j=0}^{D,d} \frac{\partial z_{j}(\mathbf{x})}{\partial \mathbf{x}_{i}} $$

where the regularization of the Jacobian enforces the representation to be contracted, and to "whiten" its axis of variations to provide parsimonious projections. The emerging properties of DAEs and CAEs provoked a regain of popularity in auto-encoders, showing that these methods could be used for black-box manifold learning. While auto-encoders were initially deterministic problems, probabilistic formulations first raised by Renzato & al. and then by Vincent & al. allowed to turn latent *codes* to latent *spaces*, where the full span of the latent space $\mathbf{z}$ is used. Furthermore, as the extracted representation were shown being closer to the real underlying manifold of $p(\mathbf{x})$, and then significantly increasing its consistency, the invertibility of the representation

<figure><img src="image_28.png" /><figcaption>**Figure 2.26:** The denoising auto-encoder, by adding a noise with fixed variance to the input, enforces the auto-encoder to learn how to project a neighboring zone of the data manifold to its correct reconstruction.</figcaption></figure>
allowed to directly explore the extracted manifold and then turn the DAE into a proper *generative model*.

## 2.3 Bridging representation learning and variational inference

In the previous sections, we summarized two different trends for extracting useful representations from signals. The first, grounded on Bayesian learning, formulates a generative distribution $p_{\theta}(x|z)$ and, based on some prior information $p(z)$, is able to extract a posterior distribution $p(z|x)$ in order to infer the latent variables $z$ corresponding to a given data $x$. The second, grounded on representation learning, proposes to model functions $z = f(x)$ whether by extracting some structural information from the data, or based on an explicitly defined loss. While the first provide robust and interpretable representations, it requires the tractability of involved terms and then strongly limits the expressiveness of modeled distributions. Conversely, machine-learning methods using neural networks are able to model functions or arbitrary complexity, and can be trained on any derivable loss functions; however, they do not ensure any representational consistency except the one indicated by the task, and can suffer from strong over-fitting. Furthermore, the obtained representations are generally not invertible, preventing to use the obtained systems as generative models.

Auto-encoders, presented above, seemed to hint at an interesting way to combine both of best worlds. Indeed, this system jointly trains encoding and decoding functions, allowing to train invertible representations using neural networks, and then enabling their use as generative models. We could then take inspiration from auto-encoding systems but rather use neural networks to model a generative distribution $p(z|x)$ as a decoding process, and another network to model the variational distribution $q(z|x)$ as an encoding process. This would allow us to learn expressive relations between the data $x$ and the latent variables $z$, while preserving the robustness and the invertibility of Bayesian inference. This framework, called Auto-Encoding Variational Bayes (AEVB), will then be introduced in the next sections.

### 2.3.1 Auto-Encoding Variational Bayes

**Stochastic optimization of ELBO** Reminding the ELBO (2.18), the variational lower-bound for the data evidence $p(x)$ can be written

$$ \mathcal{L}(q) = \int q(z) \log \frac{p(x, z)}{q(z)} dz \qquad (2.47) $$

$$ = \mathbb{E}_q[\log p(x, z) - \log q(z|\lambda)] \qquad (2.48) $$

where each latent parameters $z_j$ is optimized in turn to maximize the term $\mathcal{L}(q)$. However, this approach is still assuming the tractability of (2.18) for each $q(z_i)$, still preventing to optimize the ELBO with
intractable variational distributions. In the previous section, we introduced SGD, that allows to train a parametric model with respect to a defined criterion using iterative gradient descent optimization. As the ELBO (2.18) provides a lower-bound of the model evidence $p(x)$, it could then be used as a training criterion for SGD optimization. However, contrary to the non-stochastic cases introduced above, here the involved quantities are distributions, such that the derivation must be obtained respectively to the distribution parameters. This framework, called *Stochastic Variational Inference* (SVI), were first proposed by Hoffman & al. for exponential conjugate priors, successively updating the ELBO gradient with respect to the *natural parameters* of exponential families [155]. Another approach is black-box variational inference [156], that allows to train a variational distribution $q(z|\lambda)$ given an arbitrary generative model $p(x|z)$

$$ \nabla_{\theta}\mathcal{L}(q) = \nabla_{\theta}\mathbb{E}_{q}[(\log p(x, z) - \log q(z|\lambda))] \quad (2.49) $$

where $\lambda$ are the parameters of the distribution $q(z; \lambda)$. As the involved expectation is generally intractable, we can estimate it with Monte-Carlo approximation, a simple sampling method drawing $L$ samples from a distribution and then averaging the outputs obtained with a function $f(\cdot)$

$$ \nabla_{\theta}\mathbb{E}_{q_{\theta}(z)}[f(z)] = \frac{1}{L}\sum_{l=1}^{L}f(z)\nabla_{\theta}q_{\theta}^{l}(z^{l}) \quad (2.50) $$

this estimator, called REINFORCE, is unbiased as $l \to +\infty$. However, while its generality makes it applicable to a wide variety of models, its efficiency is awfully reduced by its high variance, dauntingly slowing down the training convergence. Several methods can then be used to reduce this slowdown, such as *Rao-Blackwellization* [157] and *control-variates* [158]. However, these methods are not sufficient to perform SVI with high-capacity neural networks, and then limits the efficiency of the variational approximation.

### Reparametrization trick & Variational Auto-Encoder
Despite the advances brought by SVI, that allows to perform variational inference with black-box approximation, the variance of the ELBO estimator (2.49) makes it inapplicable for complex cases. This problem can be alleviated under some mild assumptions for the chosen variational family @ thanks to the *reparametrization* of the network's gradient. Indeed, provided that sampling from $q_{\phi}(z|x)$ can be split between a deterministic part and stochastic component $\epsilon \sim p(\epsilon)$, such that $g(x, \epsilon)$, the expectation in (2.49) can performed with respect to $p(\epsilon)$ instead of $\mathbb{E}[q_{\phi}(z|x)]$ thanks to the *Law Of Unconscious Statistician* (LOTUS)

$$ \mathbb{E}_{q_{\phi}(z|x)}[f(z)] = \mathbb{E}_{p(\epsilon)}[f(g_{\phi}(\epsilon, x))] $$

so the gradient of the deterministic part is separated from the gradient of the stochastic part (see fig. 2.27). This method significantly reduces the variance of the ELBO, and then fastens its convergence rate. This

<figure><img src="image_29.png" /><figcaption>**Figure 2.27:** The reparametrization of samples allows to separate the gradient coming from the stochastic and deterministic components.</figcaption></figure>
<figure><img src="image_30.png" /><figcaption>Figure 2.28: The variational auto-encoder, instead of defining a point-estimate of data x and latent code z, rather define the parameter of the generative and variational distributions.</figcaption></figure>

reparametrization trick available for many distributions such as Normal, Logistic, Uniform, and Laplace distributions, but also to composition of these, such as Log-Normal, Dirichlet, Beta, and many others. More generally, it can also be applied to distributions with tractable inverse Cumulative Distribution Function, extending this trick to Exponential, Cauchy, and Gumbel distributions. This application of this trick to ELBO estimation was first proposed by Kingma & al., allowing a scalable and fast method for black-box amortized Variational Bayes with arbitrary encoding / decoding functions [22]. A seminal example of this method, called the Variational Auto-Encoder (VAE), is a particular parametrization of this system where both generative distributions p(x|z) and q(z|x) are defined as mean-field normal distributions (see fig. 2.28)

$$p_{\theta}(\mathbf{x}|\mathbf{z}) = \mathcal{N}(\mathbf{x}|\mu_{\theta}(\mathbf{z}), \sigma_{\theta}^{2}(\mathbf{z})) \qquad (2.51)$$

$$q_{\phi}(z|x) = \mathcal{N}(z|\mu_{\phi}(x), \sigma_{\phi}^{2}(x)) \qquad (2.52)$$

where $\theta$ and $\phi$ are respectively the parameters of generative and variational models, both defined with neural networks. As outlined by the name, VAE then bridges Bayesian inference with auto-encoders, and are optimized with SGD on the ELBO (2.18), that can be re-written from an auto-encoder perspective

$$\mathcal{L}[q] = \underbrace{\mathbb{E}_{q(z)}[\log p(\mathbf{x}|z)]}_{\text{reconstruction error}} - \underbrace{D_{KL}[q(z|\lambda)\|p(z)]}_{\text{regularization error}}$$

The first term can be understood as a reconstruction error, where the log-likelihood replaces the $\ell^2$ loss, while the second is a regularization term, preventing the variational model to move away from the prior $p(z)$. VAEs can then be formulated as a stochastic version of the auto-encoder, with an additional regularization term that enforces the approximated posterior distribution to match the prior. Variational auto-encoders raised a significant enthusiasm in the machine learning community, because of its efficient trade between simplicity, scalability, and flexibility. Indeed, the possibility of modeling the dependencies between data x and latent variables z with neural networks greatly enhance the expressiveness of Bayesian latent models, relaxing almost all of the previous tractability assumptions. Also, VAEs gain from auto-encoders by inferring its latent representation with a Bayesian learning process, increasing the generalization of the model and the robustness of the representation. Furthermore, representations obtained with variational auto-encoders on simple datasets were surprisingly coherent, pointing out that the VAE successfully retrieved the data underlying manifold (see fig. for an example with the MNIST dataset [159]). These appealing properties turned VAE as one of the most popular recent machine-learning generative model, and were then consequently widely investigated by the community.
### 2.3.2 Emerging properties of variational auto-encoders

Since the seminal works of Kingma & Welling [22] and Rezende & al.[160], emergent properties of AEVB have then been extensively studied in the domain of probabilistic generative models field. Despite the surprising consistency of the extracted representations, researchers unveiled two main drawbacks of VAE, preventing these models to be extended to datasets of higher complexity. The first issue was the blurriness of generated samples, pointing out the sub-optimality of the generation process. A second issue was that, in some cases, the latent spaces extracted by VAEs were degenerated, indicating that the latent space was bypassed by the model [161]. In this section we will review the main sources of degeneracy that occur when training variational auto-encoders, and provide a state of the art of current solutions and improvements brought so far by the research community.

**Approximation vs amortization gaps.** Despite the gain of expressiveness brought by parametrizing $q(x|z)$ with neural networks, the variational model distribution is still defined as normal distribution with diagonal covariance, then still belonging to the mean-field family. Choosing a given family of distributions $\mathcal{G}$ for our variational model, we can derivate the ELBO 2.18 to outline two possible sources of sub-optimality [162] (see fig.2.29)

$$ 
\begin{aligned}
\mathcal{L} &= \log p(\mathbf{x}) - \mathcal{L}[q] \\
&= \log p(\mathbf{x}) - \mathcal{L}[q^*] + \mathcal{L}[q^*] - \mathcal{L}[q] \\
&= \underbrace{D_{KL}[q^*(z|\mathbf{x}) \| [p(z|\mathbf{x})]]}_{\text{approximation gap}} + \underbrace{D_{KL}[q(z|\mathbf{x}) \| [p^*(z|\mathbf{x})]] - D_{KL}[q^*(z|\mathbf{x}) \| p(z|\mathbf{x})]}_{\text{amortization gap}}
\end{aligned} 
\quad (2.53) $$

where $q^*(z|x)\mathcal{G}$ is the optimal distribution minimizing the ELBO $\mathcal{L}$. This reformulation allows to separate the inference gap in two different parts: an *approximation gap*, that quantifies the loss obtained from choosing a given a non-optimal family of distributions $\mathcal{G}$, and the *amortization gap*, that quantifies the loss obtained between the actual inference and optimal distributions (respectively $q$ and $q^*$).

The first term quantifies the loss that occurs when the chosen variational family $\mathcal{G}$ is unable to approximate the real posterior $p(z|x)$ distribution. As mean-field normal distributions are unable to match complex posteriors, the system will not be able to learn the corresponding generative distribution, explaining the blurriness of generated samples. Approximation gap can be reduced by choosing more expressive variational families, or normalizing flows (see sec 2.3.2.1), or rather using implicit variational distribution to match the aggregated posterior $q(\mathbf{z}) = \int q(\mathbf{z}|\mathbf{x})d\mathbf{x}$ (see sec. 2.3.2.3). The second term rather quantifies the optimization error, that can be caused by the Monte-Carlo approximation used to estimate the expected likelihood in (2.18), or imperfect convergence. This gap can be reduced by using better ELBO estimators, such as importance-weighting approximations that provide tighter bounds than AEVB (2.3.2.2).

<figure><img src="image_31.png" /><figcaption>**Figure 2.29:** The KL-divergence between true and approximated posteriors can divided in two parts: an amortization gap, that represent the gap between the current approximated posterior and the optimal one, and the gap between the optimal approximation and the true posterior.</figcaption></figure>
**Latent over-pruning.** Another surprising property of AEVB is latent over-pruning, a phenomenon observed quite early in the AEVB literature [161, 163, 164]. Over-pruning describes the tendency of VAEs to bypass some latent dimensions during training, over-regularizing the representation by automatically shrinking the number dimensions used by the decoder. While this effect could be thought desirable, pointing out that VAE naturally retrieves the true manifold underlying the data and getting rid of unnecessary dimensions, it was also noticed in degenerated cases, such that the system was not able to generate convincing samples. While the exact origins of latent over-pruning are still unclear, several solutions have been developed to control this effect, allowing diagnosis and efficient training of VAEs.

### 2.3.2.1 Approximation gap: towards more expressive inference

**Alternative inference distributions.** A way to reduce the approximation gap is then to choose more flexible variational families $\mathcal{G}$, that would be able to match the true posteriors more accurately. Even if, theoretically, any inference distribution can be chosen, these distributions must be reparametrizable, samplable, and preferably have a derivable $D_{KL}$, to be correctly trained. Normal distributions with full-rank covariance matrices $\mathcal{N}(z|\mu, \Sigma)$ can provide an alternative, allowing to model dependencies between latent dimensions. However, low-rank approximation methods have to be used, as sampling from full-rank covariance normal distributions has $\mathcal{O}(d^3)$ complexity. Rezende & al. propose a 1-rank matrix with diagonal correction $\Sigma^{-1} = \sigma^2\mathbf{1} + \mathbf{u}\mathbf{u}^T$, allowing arbitrary rotations along one principal direction $\mathbf{u}$ [32]. Alternatively, the Householder flow was proposed by Tomczak & al. to encode full-rank covariance matrices with a lower computational cost [165]. However, normal distributions with full-rank covariance could not be sufficient, as it only allows to fit the first two moment of the true posterior. Alternatively, the exponential family is a generalization of many commonly-used distributions that is specified by its base measure $h(\mathbf{x})$, sufficient statistics $T_i(\mathbf{x})$ and natural parameters $\mathbf{v}$ [59, 166],

$$p(\mathbf{x}|\theta) = h(\mathbf{x}) \exp \sum_{i=1}^{s} v_i(\theta) T_i(\mathbf{x}) - Z(\theta) \quad (2.54)$$

where $Z(\theta)$ is the partition function of $p(\mathbf{x}|\theta)$ that normalizes the distribution. The exponential family generalizes a lot of well-known distributions: by example taking the first two moments as sufficient statistics, $T(\mathbf{x}) = [\mathbf{x}, \mathbf{x}\mathbf{x}^T]$, $(2\pi)^{-k/2}$ as base measure, and $[\Sigma^{-1}\mu, -\frac{1}{2}\Sigma^{-1}]$ as natural parameters, provide a reparamterization of the multivariate normal distribution. The exponential family also extend such as Bernoulli, Poisson, Multinomial, Gamma, and all conjugate priors and combinations of these. Exponential families can then provide an interesting framework for generalized inference families, as proposed by Ranganath & al. [59]. Furthermore, exponential families also have a natural connection to information geometry [167], with a direct equivalence between its natural parameters and its Fisher information matrix [166].
Another weakness of normal distributions is their unimodality, that is hence unable to fit multi-modal posteriors. In this case, modeling multi-modal posteriors with diagonal normals can provide faulty inference results, worsened by the $D_{KL}$ properties(see sec. 2.20). Some authors thus investigate Bayesian non-parametric methods, that can allow arbitrary information capacity for flexible posterior matching. Serban & al. proposes to define a piecewise parametrization of the latent space, that is divided in equalparts and modeled using a Piece-wise Constant Distribution, for both inference and prior distributions. This approach, inspired by non-parametric Bayesian inference, is then able to shape very generic posteriors, but unfortunately grows exponentially with the number of dimensions. Another way to use non-parametric distributions is to model a *Dirichlet process*, that is sampled to obtain the posterior distribution. Dirichlet processes can be built with stick-breaking processes, a sequence of Beta distributions whose parameters can be inferred by the variational model from the data [168], or defined hierarchically using nested Chinese Restaurant Processes, that can allow interesting unsupervised extraction of hierarchical features as shown in some studies in video classification [168].

**Normalizing Flows.** Choosing an approximated posterior among an extended variational family $\mathcal{Q}$ can indeed provide more expressive inference, but also add a significant computational cost to the training. Alternatively, *Normalizing Flows* (NF) allow to obtain expressive implicit distributions by transforming samples from a simple distribution, enriching the expressiveness of posterior with a little computational cost. Proposed by Tabak-Turner & al., a normalizing flow is a sequence of local invertible transformations $x_K = F(x_0) = (f_K \circ \dots \circ f_2 \circ f_1)(x)$ that is trained to transform a set of input $x$ into an output $x_K$, that we assume drawn from a simple distribution (such as uniform or isotropic normal). The key point of NFs is that, provided the invertibility of maps $f_i$, the log-likelihood still tractable, so we can train the flow parameters directly on the output space [169, 170]. As normalizing flows do not perform any dimensionality reduction they can be very tedious with high-dimensional data, then outshined by deep learning techniques. However, a recent surge of interest in normalizing flows has been allowed by the increasing capacity of parallel computing, providing very convincing generation results [171].

Turning the original idea upside-down, Rezende & al. propose to take samples drawn from simple distributions, and to transform them into complex ones with normalizing flows [172]. Hence, taking the output of a variational model $z_0 \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\sigma}^2)$, we can then use a NF to transform the input latent vector to obtained an implicit posterior distribution, such that $z_K = F(z_0)$. As each component of the flow is invertible, the expression of the ELBO 2.18 is still tractable (see fig. 2.30)

<figure><img src="image_32.png" /><figcaption>**Figure 2.30:** Normalizing flows allow to model complex posterior distributions with a transformation chain from $z_0$</figcaption></figure>

$$ \log q(\mathbf{z}_K | \mathbf{x}) = \log q(\mathbf{z}_0 | \mathbf{x}) - \sum_{k=1}^{K} \log |\det \frac{\partial f_k}{\partial \mathbf{z}_{k-1}}| \quad (2.55) $$
such that NF can be naturally integrated in the ELBO formulation. In addition to its tractability, the significant strength of this approach resides in its simplicity. Indeed, taking a simple planar flow and a isotropic normal prior, we are able to obtain a bi-modal distribution that can be trained to approximate any mixture of two Gaussian. Then, even with little chains of flows $f(\mathbf{z}) = \mathbf{z} + \mathbf{u}h(\mathbf{w}^\top\mathbf{z} + \mathbf{b})$ and simple flow blocks, we are able to approximate complex posterior distributions with a very low computation cost. Since the seminal paper of Rezende, numerous alternative flow blocks have been proposed :Inverse Auto-Regressive Flows, proposed by Kingma & al., modeling auto-regressive distributions [173], masked auto-regressive flows [174], conditional flows [175], or riemannian normalizing flows, using planar flows to regularize the curvature of the latent space with respect to the input [176].

### 2.3.2.2 Amortization gap: improving variational approximations

As shown by the decomposition (2.53), another source of sub-optimality in AEVB is the amortization gap, the loss between the optimal distribution for a given variational family $q^*(\mathbf{z}|\mathbf{x}) \in \mathcal{Q}$ and the distribution $q(\mathbf{z}|\mathbf{x})$ fixed after training. While traditional variational methods were based on tractable solutions, black-box VI is based on gradient-descent based optimization, such that the obtained solution at convergence is not necessarily optimal. We leave optimization issues due to gradient descent out of this work, as such problems are not specific to AEVB. However, some contributions to the amortization gap can be found in the ELBO itself. In the source of sub-optimality in AEVB is the amortization gap, the loss between the optimal distribution for a given variational family $q^*(\mathbf{z}|\mathbf{x}) \in \mathcal{Q}$ and the distribution $q(\mathbf{z}|\mathbf{x})$ fixed after training. While traditional variational methods were based on tractable solutions, black-box VI is based on gradient-descent based optimization, such that the obtained solution at convergence is not necessarily optimal. We leave optimization issues due to gradient descent out of this work, as such problems are not specific to AEVB. However, some contributions to the amortization gap can be found in the ELBO itself. One of the main source of amortization error in AEVB is due to the Monte-Carlo approximation of the expectations over $q(\mathbf{z}|\mathbf{x})$ involved in the ELBO. Remembering the expression of black-box VI gradient estimator (ee sec. 2.49)

$$ 
\begin{aligned}
\nabla_{\theta}\mathcal{L}(q) &= \nabla_{\theta}\mathbb{E}_{q}[(\log p(\mathbf{x}, \mathbf{z}) - \log q(\mathbf{z}|\mathbf{x}))] \\
&= \nabla_{\theta}\mathbb{E}_{q}[\log p(\mathbf{z}|\mathbf{x})] - D_{KL}[q(\mathbf{z}|\mathbf{x})\|p(\mathbf{z})]
\end{aligned}
$$

the exact formulation of the gradient estimator is using expectations over $q(\mathbf{z}|\mathbf{x})$, that are generally intractable in the black-box VI framework. We then estimated these expectations with Monte-Carlo approximation

$$ \nabla_{\theta}\mathbb{E}_{q_{\theta}(z)}[f(z)] = \frac{1}{L}\sum_{l=1}^{L} f(z)\nabla_{\theta}q_{\theta}^{l}(z^{l}) \quad (2.56) $$

In practice $L$ is chosen very low, generally taking $L = 1$, such that this approximation is computed using only one sample from the recogni-