RAVE: A variational autoencoder for fast and high-quality neural audio synthesis

Deep generative models applied to audio have improved by a large margin the state-of-the-art in many speech and music related tasks . However , as raw waveform modelling remains an inherently difficult task , audio generative models are either computationally intensive , rely on low sampling rates , are complicated to control or restrict the nature of possible signals . Among those models , Variational AutoEncoders ( VAE ) give control over the generation by exposing latent variables , although they usually suffer from low synthesis quality . In this paper , we introduce a Realtime Audio Variational autoEncoder ( RAVE ) allowing both fast and high-quality audio waveform synthesis . We introduce a novel two-stage training procedure , namely representation learning and adversarial fine-tuning . We show that using a post-training analysis of the latent space allows a direct control between the reconstruction fidelity and the representation compactness . By leveraging a multi-band decomposition of the raw waveform , we show that our model is the first able to generate 48kHz audio signals , while simultaneously running 20 times faster than real-time on a standard laptop CPU . We evaluate synthesis quality using both quantitative and qualitative subjective experiments and show the superiority of our approach compared to existing models . Finally , we present applications of our model for timbre transfer and signal compression . All of our source code and audio examples are publicly available . 1 Introduction . Deep learning applied to audio signals proposes exciting new ways to perform speech generation , musical composition and sound design . Recent works in deep audio modelling have allowed novel types of interaction such as unconditional generation ( Chung et al. , 2015 ; Fraccaro et al. , 2016 ; Oord et al. , 2016 ; Vasquez & Lewis , 2019 ; Dhariwal et al. , 2020 ) or timbre transfer between instruments ( Mor et al. , 2018 ) . However , these approaches remain computationally intensive , as modeling audio raw waveforms requires dealing with extremely large temporal dimensionality . To cope with this issue , previous approaches usually rely on very low sampling rates ( 16 to 24kHz ) , which can be sufficient for speech signals , but is considered as low-quality for musical applications . Furthermore , the auto-regressive sampling procedure used by most models ( Engel et al. , 2017 ) is prohibitively long . This precludes real-time application which are pervasive in musical creation , while parallel models ( Défossez et al. , 2018 ) can only allow fast generation at the cost of a lower sound quality . More recently , Engel et al . ( 2019 ) and Wang et al . ( 2019 ) proposed to leverage classical synthesis techniques to address these limitations , by relying on pre-computed audio descriptors as an extraneous information to condition the models . While these approaches achieve state-of-the-art results in terms of audio quality , naturalness and computational efficiency , the extensive use of audio descriptors highly restricts the type of signals that can be generated . A possible solution to alleviate this issue would be to rely on Variational Autoencoders ( Kingma & Welling , 2014 ) , as they provide a form of trainable analysis-synthesis framework ( Esling et al. , 2018 ) , without explicit restrictions on the type of features learned . However , estimating the dimensionality of the latent representation associated with a given dataset prior to model training is far from trivial . Indeed , a wrong estimation of the latent dimensionality may result in either poor reconstruction or uninformative latent dimensions , which makes latent exploration and manipulation difficult . In this paper , we overcome the limitations outlined above by proposing a VAE model built specifically for fast and high-quality audio synthesis . To do so , we introduce a specific twostage training procedure where the model is first trained as a regular VAE for representation learning , then fine-tuned with an adversarial generation objective in order to achieve high quality audio synthesis . We combine a multi-band decomposition of the raw waveform alongside classical synthesis blocks inspired by Engel et al . ( 2019 ) , allowing to achieve highquality audio synthesis with sampling rates going up to 48kHz without a major increase in computational complexity . We show that our model is able to converge on complex datasets using a low number of parameters , achieving state-of-the-art results in terms of naturalness and audio quality , while being usable in real-time on a standard laptop CPU . We compare our model with several state-of-the-art models and show its superiority in unsupervised audio modeling . In order to address the dimensionality of the learned representation , we introduce a novel method to split the latent space between informative and uninformative parts using a singular value decomposition , and show that replacing the latter part with random noise does not affect the reconstruction quality . This procedure allows easier exploration and manipulation of latent trajectories , since we only need to operate on a subset of informative latent dimensions . Finally , we discuss the application of our model in signal compression and timbre style transfer . Our key contributions are : • A two-stage training procedure where the model is first trained as a regular VAE , then fine-tuned with an adversarial generation objective , as depicted in figure 1 • A post-training analysis of the latent space providing a way to balance between reconstruction fidelity and representation compactness • High-quality audio synthesis models with sampling rates going up to 48kHz • 20 times faster than realtime synthesis on a standard laptop CPU Audio samples and supplementary figures are provided in the accompanying website1 . We highly encourage readers to listen to accompanying samples while reading the paper . 2 State-of-art . 2.1 Variational autoencoders . Generative models aim to understand a given dataset x ∈ Rdx by modelling its underlying distribution p ( x ) . To simplify this problem , we can consider that the generation of x is conditioned by latent variables z ∈ Rdz , responsible for most of the variations present in x . Therefore , the complete model is defined by the joint distribution p ( x , z ) = p ( x|z ) p ( z ) , which is usually not analytically solvable given the complexity of real-world data . Variational 1https : //anonymous84654.github.io/ICLR_anonymous/ autoencoders address this problem by introducing an inference model qφ ( z|x ) , optimized to minimize its Kullback-Leibler ( KL ) divergence with the true posterior distribution p ( z|x ) φ∗ = argmin φ DKL [ qφ ( z|x ) ‖p ( z|x ) ] , ( 1 ) which can be rearranged to obtain the final objective used to train a VAE , called the Evidence Lower BOund ( ELBO ) , as shown by Kingma & Welling ( 2014 ) Lφ , θ ( x ) = −Eqφ ( z|x ) [ log pθ ( x|z ) ] +DKL [ qφ ( z|x ) ‖p ( z ) ] . ( 2 ) The ELBO minimizes the reconstruction error of the model through the likelihood of the data given a latent log pθ ( x|z ) , while regularizing the posterior distribution qφ ( z|x ) to match a predefined prior p ( z ) . Both posterior distributions qφ and pθ are parametrized by neural networks respectively called encoder and decoder . Higgins et al . ( 2016 ) proposes to weight the KL divergence in equation ( 2 ) with a parameter β to control the trade-off between accurate reconstruction and strong latent regularization . They show that increasing β > 1 leads to less entangled latent dimensions , at the detriment of the reconstruction quality . 2.2 Autoencoding raw waveform . One of the first approaches addressing the raw waveform modelling task are WaveNet ( Oord et al. , 2016 ) and SampleRNN ( Mehri et al. , 2017 ) , where the probability of a waveform x is factorized as a product of conditional probabilities p ( x ) = ∏ t > 1 p ( xt|x1 , . . . , xt−1 ) . ( 3 ) Those models require a large amount of data and parameters to properly converge . Furthermore , the autoregressive nature of the synthesis process makes it prohibitively slow , and prone to accumulate errors . WaveNet has also been adapted by Engel et al . ( 2017 ) for their NSynth model , addressing the representation learning task . Unlike equation ( 2 ) , they do not regularize the learned representation , and rather encode the raw waveform determinisitically to its latent counterpart . It implies the absence of a prior distribution p ( z ) and , therefore , prevents sampling from the latent space . This restricts the applications of the model to simple reconstructions and interpolations . As a way to speed-up the synthesis process , Défossez et al . ( 2018 ) proposed an autoencoder with feed-forward convolutional networks parametrizing both the encoder and the decoder . They use a perceptually-motivated distance between waveforms called spectral distance as the reconstruction objective l ( x , y ) = ∥∥log ( STFT ( x ) 2 + ) − log ( STFT ( y ) 2 + ) ∥∥ 1 , ( 4 ) where STFT is the Short-Term Fourier Transform . Since they use a squared STFT , the phase component is discarded making the loss permissive to inaudible phase variations . They show that their model is 2500 times faster that NSynth during synthesis , at the expense of a degraded sound quality . Following the recent advances in generative adversarial modelling ( Goodfellow et al. , 2014 ) , Kumar et al . ( 2019 ) proposed to use an adversarial objective to address the parallel audio modelling task . The discriminator is trained to differentiate true samples from generated ones , while the generator is optimized to produce samples that are classified as true by the discriminator . A feature matching loss is added to the adversarial loss , minimizing the L1 distance between the discriminator feature maps of real and synthesized audio . This feature matching mechanism can be seen as a learned metric to evaluate the distance between two samples , and has been successfully applied to the conditional waveform modelling task ( e.g spectrogram to waveform or replacement of the decoder in a pretrained autoencoder model ) . 3 Method . 3.1 Two-stage training procedure . Ideally , the representation learned by a variational autoencoder should contain high-level attributes of the dataset . However , two perceptually similar audio signals may contain subtle phase variations that produce dramatically different waveforms . Hence , estimating the reconstruction term in equation ( 2 ) using the raw waveform penalizes the model if those subtle variations are not included in the learned representation . This might both hamper the learning process and include in the latent space those low-level variations about audio signal that are not relevant perceptually . To address this problem , we split the training process in two stages , namely representation learning and adversarial fine-tuning . 3.1.1 Stage 1 : Representation learning . The first stage of our procedure aims to perform representation learning . We leverage the multiscale spectral distance S ( · , · ) proposed by Engel et al . ( 2019 ) in order to estimate the distance between real and synthesized waveforms , defined as S ( x , y ) = ∑ n∈N [ ‖STFTn ( x ) − STFTn ( y ) ‖F ‖STFTn ( x ) ‖F + log ( ‖STFTn ( x ) − STFTn ( y ) ‖1 ) ] , ( 5 ) where N is a set of scales , STFTn is the amplitude of the Short-Term Fourier Transform with window size n and hop size n/4 , and ‖ · ‖F , ‖ · ‖1 are respectively the Frobenius norm and L1 norm . Using an amplitude spectrum-based distance does not penalize the model for inaccurately reconstructed phase , but encompasses important perceptual features about the signal . We train the encoder and decoder with the following loss derived from the ELBO Lvae ( x ) = Ex̂∼p ( x|z ) [ S ( x , x̂ ) ] + β ×DKL [ qφ ( z|x ) ‖p ( z ) ] , ( 6 ) We start by training the model solely with Lvae , and once this loss converges , we switch to the next training phase .

Summary: This paper proposes an autoencoder for high-quality waveform synthesis. The approach is based on multi-band decomposition, VAE and adversarial fine-tuning. The synthesis speed is substantially faster than existing approaches on both CPU and GPU.