RAVE: A variational autoencoder for fast and high-quality neural audio synthesis

Deep generative models applied to audio have improved by a large margin the state-of-the-art in many speech and music related tasks . However , as raw waveform modelling remains an inherently difficult task , audio generative models are either computationally intensive , rely on low sampling rates , are complicated to control or restrict the nature of possible signals . Among those models , Variational AutoEncoders ( VAE ) give control over the generation by exposing latent variables , although they usually suffer from low synthesis quality . In this paper , we introduce a Realtime Audio Variational autoEncoder ( RAVE ) allowing both fast and high-quality audio waveform synthesis . We introduce a novel two-stage training procedure , namely representation learning and adversarial fine-tuning . We show that using a post-training analysis of the latent space allows a direct control between the reconstruction fidelity and the representation compactness . By leveraging a multi-band decomposition of the raw waveform , we show that our model is the first able to generate 48kHz audio signals , while simultaneously running 20 times faster than real-time on a standard laptop CPU . We evaluate synthesis quality using both quantitative and qualitative subjective experiments and show the superiority of our approach compared to existing models . Finally , we present applications of our model for timbre transfer and signal compression . All of our source code and audio examples are publicly available . 1 Introduction . Deep learning applied to audio signals proposes exciting new ways to perform speech generation , musical composition and sound design . Recent works in deep audio modelling have allowed novel types of interaction such as unconditional generation ( Chung et al. , 2015 ; Fraccaro et al. , 2016 ; Oord et al. , 2016 ; Vasquez & Lewis , 2019 ; Dhariwal et al. , 2020 ) or timbre transfer between instruments ( Mor et al. , 2018 ) . However , these approaches remain computationally intensive , as modeling audio raw waveforms requires dealing with extremely large temporal dimensionality . To cope with this issue , previous approaches usually rely on very low sampling rates ( 16 to 24kHz ) , which can be sufficient for speech signals , but is considered as low-quality for musical applications . Furthermore , the auto-regressive sampling procedure used by most models ( Engel et al. , 2017 ) is prohibitively long . This precludes real-time application which are pervasive in musical creation , while parallel models ( Défossez et al. , 2018 ) can only allow fast generation at the cost of a lower sound quality . More recently , Engel et al . ( 2019 ) and Wang et al . ( 2019 ) proposed to leverage classical synthesis techniques to address these limitations , by relying on pre-computed audio descriptors as an extraneous information to condition the models . While these approaches achieve state-of-the-art results in terms of audio quality , naturalness and computational efficiency , the extensive use of audio descriptors highly restricts the type of signals that can be generated . A possible solution to alleviate this issue would be to rely on Variational Autoencoders ( Kingma & Welling , 2014 ) , as they provide a form of trainable analysis-synthesis framework ( Esling et al. , 2018 ) , without explicit restrictions on the type of features learned . However , estimating the dimensionality of the latent representation associated with a given dataset prior to model training is far from trivial . Indeed , a wrong estimation of the latent dimensionality may result in either poor reconstruction or uninformative latent dimensions , which makes latent exploration and manipulation difficult . In this paper , we overcome the limitations outlined above by proposing a VAE model built specifically for fast and high-quality audio synthesis . To do so , we introduce a specific twostage training procedure where the model is first trained as a regular VAE for representation learning , then fine-tuned with an adversarial generation objective in order to achieve high quality audio synthesis . We combine a multi-band decomposition of the raw waveform alongside classical synthesis blocks inspired by Engel et al . ( 2019 ) , allowing to achieve highquality audio synthesis with sampling rates going up to 48kHz without a major increase in computational complexity . We show that our model is able to converge on complex datasets using a low number of parameters , achieving state-of-the-art results in terms of naturalness and audio quality , while being usable in real-time on a standard laptop CPU . We compare our model with several state-of-the-art models and show its superiority in unsupervised audio modeling . In order to address the dimensionality of the learned representation , we introduce a novel method to split the latent space between informative and uninformative parts using a singular value decomposition , and show that replacing the latter part with random noise does not affect the reconstruction quality . This procedure allows easier exploration and manipulation of latent trajectories , since we only need to operate on a subset of informative latent dimensions . Finally , we discuss the application of our model in signal compression and timbre style transfer . Our key contributions are : • A two-stage training procedure where the model is first trained as a regular VAE , then fine-tuned with an adversarial generation objective , as depicted in figure 1 • A post-training analysis of the latent space providing a way to balance between reconstruction fidelity and representation compactness • High-quality audio synthesis models with sampling rates going up to 48kHz • 20 times faster than realtime synthesis on a standard laptop CPU Audio samples and supplementary figures are provided in the accompanying website1 . We highly encourage readers to listen to accompanying samples while reading the paper . 2 State-of-art . 2.1 Variational autoencoders . Generative models aim to understand a given dataset x ∈ Rdx by modelling its underlying distribution p ( x ) . To simplify this problem , we can consider that the generation of x is conditioned by latent variables z ∈ Rdz , responsible for most of the variations present in x . Therefore , the complete model is defined by the joint distribution p ( x , z ) = p ( x|z ) p ( z ) , which is usually not analytically solvable given the complexity of real-world data . Variational 1https : //anonymous84654.github.io/ICLR_anonymous/ autoencoders address this problem by introducing an inference model qφ ( z|x ) , optimized to minimize its Kullback-Leibler ( KL ) divergence with the true posterior distribution p ( z|x ) φ∗ = argmin φ DKL [ qφ ( z|x ) ‖p ( z|x ) ] , ( 1 ) which can be rearranged to obtain the final objective used to train a VAE , called the Evidence Lower BOund ( ELBO ) , as shown by Kingma & Welling ( 2014 ) Lφ , θ ( x ) = −Eqφ ( z|x ) [ log pθ ( x|z ) ] +DKL [ qφ ( z|x ) ‖p ( z ) ] . ( 2 ) The ELBO minimizes the reconstruction error of the model through the likelihood of the data given a latent log pθ ( x|z ) , while regularizing the posterior distribution qφ ( z|x ) to match a predefined prior p ( z ) . Both posterior distributions qφ and pθ are parametrized by neural networks respectively called encoder and decoder . Higgins et al . ( 2016 ) proposes to weight the KL divergence in equation ( 2 ) with a parameter β to control the trade-off between accurate reconstruction and strong latent regularization . They show that increasing β > 1 leads to less entangled latent dimensions , at the detriment of the reconstruction quality . 2.2 Autoencoding raw waveform . One of the first approaches addressing the raw waveform modelling task are WaveNet ( Oord et al. , 2016 ) and SampleRNN ( Mehri et al. , 2017 ) , where the probability of a waveform x is factorized as a product of conditional probabilities p ( x ) = ∏ t > 1 p ( xt|x1 , . . . , xt−1 ) . ( 3 ) Those models require a large amount of data and parameters to properly converge . Furthermore , the autoregressive nature of the synthesis process makes it prohibitively slow , and prone to accumulate errors . WaveNet has also been adapted by Engel et al . ( 2017 ) for their NSynth model , addressing the representation learning task . Unlike equation ( 2 ) , they do not regularize the learned representation , and rather encode the raw waveform determinisitically to its latent counterpart . It implies the absence of a prior distribution p ( z ) and , therefore , prevents sampling from the latent space . This restricts the applications of the model to simple reconstructions and interpolations . As a way to speed-up the synthesis process , Défossez et al . ( 2018 ) proposed an autoencoder with feed-forward convolutional networks parametrizing both the encoder and the decoder . They use a perceptually-motivated distance between waveforms called spectral distance as the reconstruction objective l ( x , y ) = ∥∥log ( STFT ( x ) 2 + ) − log ( STFT ( y ) 2 + ) ∥∥ 1 , ( 4 ) where STFT is the Short-Term Fourier Transform . Since they use a squared STFT , the phase component is discarded making the loss permissive to inaudible phase variations . They show that their model is 2500 times faster that NSynth during synthesis , at the expense of a degraded sound quality . Following the recent advances in generative adversarial modelling ( Goodfellow et al. , 2014 ) , Kumar et al . ( 2019 ) proposed to use an adversarial objective to address the parallel audio modelling task . The discriminator is trained to differentiate true samples from generated ones , while the generator is optimized to produce samples that are classified as true by the discriminator . A feature matching loss is added to the adversarial loss , minimizing the L1 distance between the discriminator feature maps of real and synthesized audio . This feature matching mechanism can be seen as a learned metric to evaluate the distance between two samples , and has been successfully applied to the conditional waveform modelling task ( e.g spectrogram to waveform or replacement of the decoder in a pretrained autoencoder model ) . 3 Method . 3.1 Two-stage training procedure . Ideally , the representation learned by a variational autoencoder should contain high-level attributes of the dataset . However , two perceptually similar audio signals may contain subtle phase variations that produce dramatically different waveforms . Hence , estimating the reconstruction term in equation ( 2 ) using the raw waveform penalizes the model if those subtle variations are not included in the learned representation . This might both hamper the learning process and include in the latent space those low-level variations about audio signal that are not relevant perceptually . To address this problem , we split the training process in two stages , namely representation learning and adversarial fine-tuning . 3.1.1 Stage 1 : Representation learning . The first stage of our procedure aims to perform representation learning . We leverage the multiscale spectral distance S ( · , · ) proposed by Engel et al . ( 2019 ) in order to estimate the distance between real and synthesized waveforms , defined as S ( x , y ) = ∑ n∈N [ ‖STFTn ( x ) − STFTn ( y ) ‖F ‖STFTn ( x ) ‖F + log ( ‖STFTn ( x ) − STFTn ( y ) ‖1 ) ] , ( 5 ) where N is a set of scales , STFTn is the amplitude of the Short-Term Fourier Transform with window size n and hop size n/4 , and ‖ · ‖F , ‖ · ‖1 are respectively the Frobenius norm and L1 norm . Using an amplitude spectrum-based distance does not penalize the model for inaccurately reconstructed phase , but encompasses important perceptual features about the signal . We train the encoder and decoder with the following loss derived from the ELBO Lvae ( x ) = Ex̂∼p ( x|z ) [ S ( x , x̂ ) ] + β ×DKL [ qφ ( z|x ) ‖p ( z ) ] , ( 6 ) We start by training the model solely with Lvae , and once this loss converges , we switch to the next training phase .

Summary: This paper aims to achieve fast and high-quality audio synthesis. The authors propose a variational autoencoder-based model called real-time audio variational autoencoder (RAVE) to achieve this aim. In particular, the authors introduce a two-stage training scheme to alleviate the difficulty in joint learning of the adequate representation and high-fidelity audio synthesis. More concretely, in the first stage, representation learning is conducted in a VAE framework. In the second stage, audio fidelity is improved using adversarial fine-tuning. The authors also introduce a post-analysis method to manipulate the balance between fidelity and compactness. The effectiveness of the proposed method was evaluated on the Strings and VCTK datasets. The applications to timbre transfer and signal compression were also demonstrated.