RAVE: A variational autoencoder for fast and high-quality neural audio synthesis
Authors: Antoine Caillon
Philippe Esling
Venue: ICLR 2022 Submitted (2022)
Keywords: Variational Autoencoder
generative models
audio
music
deep learning
representation learning
latent space

Abstract: Deep generative models applied to audio have improved by a large margin the state-of-the-art in many speech and music related tasks. However, as raw waveform modelling remains an inherently difficult task, audio generative models are either computationally intensive, rely on low sampling rates, are complicated to control or restrict the nature of possible signals. Among those models, Variational AutoEncoders (VAE) give control over the generation by exposing latent variables, although they usually suffer from low synthesis quality. In this paper, we introduce a Realtime Audio Variational autoEncoder (RAVE) allowing both fast and high-quality audio waveform synthesis. We introduce a novel two-stage training procedure, namely representation learning and adversarial fine-tuning. We show that using a post-training analysis of the latent space allows a direct control between the reconstruction fidelity and the representation compactness. By leveraging a multi-band decomposition of the raw waveform, we show that our model is the first able to generate 48kHz audio signals, while simultaneously running 20 times faster than real-time on a standard laptop CPU. We evaluate synthesis quality using both quantitative and qualitative subjective experiments and show the superiority of our approach compared to existing models. Finally, we present applications of our model for timbre transfer and signal compression. All of our source code and audio examples are publicly available.

Under review as a conference paper at ICLR 2022
RAVE: A VARIATIONAL AUTOENCODER FOR FAST AND HIGH-QUALITY NEURAL AUDIO SYNTHESIS
A Anonymous authors Paper under double-blind review
ABSTRACT
Deep generative models applied to audio have improved by a large margin the state-of-tho-art in many speech and music related tasks. However, as raw waveform modelling remains inherently difficult task, audio gen- erative models are either computationalyy intensive, rely on low sampling rates, are complicated to control or restrict the nature of possible signals. Among those models, Variational AutoEncoders (VAE) give control over the generation by exposing latent variables, although they usually suffer from low synthesis quality. In this paper, we introduce a Realtime Audio Variational autoEncoder (RAVE) allowing both fast and high-quality au- dio waveform synthesis. We introduce a novel two-stage training procedure, namely representation learning and adversarial fine-tuning. We show that using a post-training analysis of the latent space allows a direct control be- tween the reconstruction fidelity and the representation compactness. By leveraging a multi-band decomposition of the raw waveform, we show that our model the first able to generate 48kHz audio signals, while simulta- neously running 20 times faster than real-time We evaluate synthesis quality using both quantitative and qualitative sub- jective experiments and show the superiority of our approach compared to cxisting models. Finally, we present applications of our model for timbre transfer and signal compression. All of our source code and audio examples are publicly available.
standard laptop CPU,
INTRODUCTION
Deep learning appliod to audio signals proposes exciting now ways to perform speoch gen- eration, musical composition and sound design. Recent works in deep audio modelling have allowed novel types of interaction such as unconditional generation (Chung et al. 2015
Fraccaro et al., 2016 Oord et al. 2016 Vasquez & Lewis 2019 Dhariwal al. 2020) or
timbre transter between instruments (Mor et al.) 2018). However, these approaches remain
computationally intensive, modeling audio raw waveforms requires dealing with extremely large temporal dimensionality. To cope with this issue, previous approaches usually rely on very low sampling rates (16 to 24kHzz), which can be sufficient for speech signals, but con- sidered as low-quality for musical applications. Furthermore, the auto-regressive sampling procedure used by most models (Engel et al. 20177) is prohibitively long, This precludes real-time application which are pervasive in musical creation, while parallel models (Défossez et al. 2018 can only allow fast generation at the cost of a lower sound quality.
More recently. Engel et al. (019)) and Wang et al. 2019 proposed to leverage classical synthesis techniques to address these limitations, by relying on pre-computed audio descrip- tors as extrancous information to condition the models. While these approaches achieve state-of-the-art results in terms audio quality, naturalness and computational efficiency, the extensive use of audio descriptors highly restricts the type of signals that can be gener- ated. A possible solution to alleviate this issue would be to rely on Variational Autoencoders
(Kingma & Welling 2014), they provide a form of trainable amalysis-synthest framework Esling et a 2018). without explicit restrictions on the type of features learned. However, estimating the dimensionality of the latent representation associated with given dataset.

---

Under review as a conference paper at ICLR 2022
prior to model training is far from trivial. Indeed, a wrong estimation of the latent dimen- sionality may result in either poor reconstruction or uninformative latent dimensions, which makes latent exploration and manipulation difficult.
In this paper, we overcome the limitations outlined above by proposing a VAE model built specifically for fast and high-quality audio synthesis. To do so, introduce specific two- stage training procedure where the model is first trained as a regular VAE for representation learning, then fine-tuned with an adversarial generation objective in order achieve high quality audio synthesis. We combine a multi-band decomposition of the raw waveform alongside classical synthesis blocks inspired by Engel al. (2019). allowing to achieve high- quality audio synthesis with sampling rates going up to 48kHz without a major increase im computational complexity. We show that our model is able to converge on complex datasets using low number of parameters, achieving state-of-the-ane results in terms of naturalness and audio quality, while being usable in real-time on a standard laptop CPU. We compare our model with several state-of-the-ard models and show its superiority in unsupervised audio modeling. In order to address the dimensionality novel method to split the latent space between informative and uninformative parts using a singular value decomposition, and show that replacing the latter part with random noise does not affect the reconstruction quality. This procedure allows easier exploration and manipulation of latent. trajectories, since we only need to operate on a subset of informative latent dimensions. Finally, we discuss the application of our model in signal compression and timbre style transfer. Our key contributions are:
the learned representation, we introduce
A two-stage training procedure where the model is first trained a regular VAE, then fine-tuned with an adversarial generation objective, as depicted in figure
A post-training analysis of the latent space providing way to balance between reconstruction fidelity and representation compactness
High-quality audio synthesis models with sampling rates going up to 48kHz
20 times faster than realtime synthesis on a standard laptop CPU
Audio samples and supplementary figures are provided the accompanying website We highly encourage readers to listen to accompanying samples while reading the paper.
muttbann spectral
Stage 1:
distance
Representation Coarning
multiaan
enooder
decoder
mulliband
multhand
decodier
muttionnd
Stage Adversadisl fia-ttninn
sg
dsotminator
Figure 1: Overall architecture of the proposed approach. Blocks in blue are the only ones optimized, while blocks in grey are fixed or frozen operations.
STATE-OF-ART
VARIATIONAL, AUTORNCODERS
Generative models aim to understand given dataset. x E Rds by modelling its underlying distribution p(x). To simplify this problem, we can consider that the generation of x is conditioned by latent variables x. Therefore, the complete model is defined by the joint distribution p(x, z) = p(x|2)p(z),
Id, responsible for most of the variations present in
which is usually not analytically solvable given the complexity of real-world data. Variational
https::/aannyymous84454 githhuuuuuee..uuuub.o... github. io/ICIRRIIAIAA.ai ant.yy/ounnommuss/////
2

---

Under review as a conference paper at ICLR 2022
autoencoders address this problem by introducing an inference model (s(z/x), optimized to minimize its Kullback-Leibler (KL) divergence with the true posterior distribution p(z|x)
Φ = argmin PKL (xLlls/z/>))lp(xxx)/
(1)
which be rearranged to obtain the final objective used to train VAE, called the Evidence Lower BOund (ELBO), as shown by Kingma & Welling (0014)
Cx.0(x) = - (x(x)
(x plg +
(2)
The ELBO minimizes the reconstruction error of the model through the likelihood the data given latent log (x( while regularizing the posterior distribution (z(x)) to match a predefined prior p(z). Both posterior distributions qs and po are parametrized by noural networks respectively called encoder and decoder. Higgins et al. (2016) proposes weight the KL divergence in equation 22 with a parameter 3 to control the trade-off between accurate reconstruction and strong latent regularization. They show that increasing leads to less entangled latent dimensions, at the detriment of the reconstruction quality.
2.2 AUTOENCODING RAW WAVEFORM
One of the first approaches addressing the raw waveform modelling task are WaveNet (Oord el. al.) 2016) and SampleRNN (Mehri et al. 2017), where the probabilily of a waveform x is lactorized a product of conditional probabilities
p(x) Il p( 11..,,
(3)
Those models require a large amount of data and parameters properly converge. Fur- thermore, the autoregressive nature of the synthesis process makes it prohibitively slow, and prone to accumulate errors.
WaveNet has also been adapted by Engel et al. 2017 for their NSynth model, addressing the representation learning task. Unlike equation representation, and rather encode the waveform determinisitically to its latent counter- part. It implics the absence of a prior distribution p(z) and, therefore, prevents sampling from the latent space. This restricts the applications of the model to simple reconstructions and interpolations.
2 they do not regularize the learned
As a way to speed-up the synthesis process, Défossez al. 2018) proposed autoencoder with feed-forward convolutional networks parametrizing both the encoder and the decoder. They use perceptually-mnt motivvtedd distance between waveforms called spectral distance as the reconstruction objective
(4) where STFT is the Short-Term Fourier Transform. Since they use a squared STFT, the phase component is discarded making the loss permissive to inaudible phase variations. They show that their model is 2500 times faster that NSynth during synthesis, at the expense of a degraded sound quality.
l(x,y) = [log(STFT(x) +e) Log(STF((y) + +c
Following the recent advances in generative adversarial modelling (Goodfellow et al. 2014 Kumar et al. (2019) proposed to use an adversarial objective address the parallel audio modelling task. The discriminator is trained to differentiate true samples from generated ones, while the generator is optimized to produce samples that are classified as true by the discriminator. A feature matching loss is added to the adversarial loss, minimizing the LI distance between the discriminator feature maps of real and syuthesized audio. This feature matching mechanism samples, and has been successfully applied to the conditional waveform modelling task (e.g spectrogram to waveform replacement. of the decoder in pretrained autoencoder model).
be seen learned metric to evaluate the distance between two
3

---

Under review as a conference paper at ICLR 2022
3 METHOD
3.1 TWO-STAGE TRAINING PROCEDURE
Idcally, the representation learned by a variational autocncoder should contain high-level attributes of the datasct. However, two perceptually similar audio signals may contain subtle phase variations that produce dramatically different waveforms. Hence, estimating the reconstruction term in equation (2) using the raw waveform penalizes the model if those subtle variations are not included in the learned representation. This might both hamper the learning process and include in the latent space those low-level variations about audio signal that are not relevant perceptually. To address this problem, we split the training process in two stages, namely representation learning and adversarial fine-tuning.
3.1.1 STAGE REPRESENTATION LEARNING
The first stage of our procedure aims to perform represenlation learning. We leverage the multiscale spectral distance S(.,-) proposed by Engel et al. (2019) in order estimate the distance between real and synthesized waveforms, defined as
|STFT - STFT,(y)) + log ((STFFT.)))STFTT..G)))) STFT, ITTFT,, (x)lle
s(x,y) = Σ nEN
(5)
where N set of scales, STFT, the amplitude of the Short-Term Fourier Transform with window size and hop size n/4, and || IIr, || ||/h are respectively the Frobenius norm and L1 norm. Using an amplitude spectrun- based distance does not penalize the model for inaccurately reconstructed phase, but encompasses important perceptual features about the signal. We train the encoder and decoder with the following loss derived from the ELBO
Lvae (x) Exxpp(xe) &)) + B DKL (x
We start by training the modol solely with Lvee, and once this loss converges, we switch to the next training phase.
3.1.2 STAGE 2: ADVERSARIAL FINE-TUNING
The second training stage aims at improving the synthesized audio quality and naturalness. As we consider that the learned representation has reached satisfactory state at this point, we freeze the encoder and only train the decoder using an adversarial objective.
GANs are implicit generative models allowing to sample from complex distribution by transforming a simpler one, called the base distribution. Here, we use the learned latent space in the first stage the base distribution, and train the decoder to produce syuthesized signals similar to the real ones by relying on a discriminator D. We use the hinge loss version of the GAN objective, defined
Laix 2.) = max(0. Dxx)) + F Ex (ma 1 D())),
Cgea (z) = - Ex-ppxx)
(7)
In order to ensure that the synt hesized signal & does not diverge too much from the ground
truth x, we keep minimizing the spectral distance defined in equation (5). but also add the
feature matching loss LFM proposed by Kumar et al. (0019) Altogether, this yields the following objective for the decoder
Liotal (x, 2) Egen(2) Re)p(())) [S(x, 8) LFM x)).
(8)

---

Under review as a conference paper at ICLR 2022
3.2 LATENT REPRESENTATION COMPACTNESS
The loss proposed in equation contains two terms, a reconstruction and regularisation lerm. Those Iwo terms somewhat conflicting, since the reconstruction term maximises the mutual information between the latent, representation and the data distribution, while the regularisation term guides the posterior distribution towards independence with the data (potentially causing posterior collapse). In practice, the pressure applied by the regularisa- tion term the encoder during training encourages it to learn compact representation, where informative latents have the highest KL divergence from the prior, while uninforma- tive latents have KL divergence close to 0 (Higgins et al. 2016).
Here, we address the task of identifying the most informative parts of the latent space in order to restrict the dimensionality of the learned representation to the strict minimum required to reconstruct. a signal. To do so, we adapt the method for range and null space estimation (see appendix AA to this problem. Let Z E 1xxd be a matrix composed samples E Rd, where on to solve the problem of finding informative parts of the latent space would not be relevant given the high variance present in the collapsed parts of Z. In order to adapt this to our problem, we first remove the variance from Z, by considering the matrix E Raxd that verifies
4o (z)x). Using a Singular Value Decomposition (SVD) directly
Z/ argmax (o(z)x);
Hence, dimensions of the posterior distribution go (z(x) that have collapsed to the prior p(z) will result in a constant value in Z', that we set to by removing the average of Z' across the first dimension. The only dimensions of Z' with non-zero values are therefore correlated with the input, which constitute the informative part of the latent space. Applying a SVD on this centered matrix, we can obtain the matrix Σ containing the singular values of Z', by computing
= -EVT,
(10)
As detailed in appendix A the rank r of Z' is equal the number of non-zero singular values in Σ. Given the high variation that cxists in real-world data, it is unlikely that the
vanishing singular values of Z' are equal to O. Therefore, instead of tracking the exact rank of Z', we define fidelity parameter f [0- - 1J, with the associated rank ry defined as the smallest integer verilying
Eisry Eii Σ, En
2
(11)
Given the fidelity value f. and a latent representation z ~ qo (z|x), we reduce the dimension- ality of z by projecting it on the basis defined by V and keep only the ry first dimensions. We obtain low-rank representation zy, whose dimensionality depends and f. Before providing 2 to the decoder, we concatenate it with noise sampled from the prior distribution, and project it back its original basis using V. We demonstrate in section 5.3 the influence of f on the reconstruction.
both the dataset
4 EXPERIMENTS
4.1 RAVE
Here, we introduce our Realtime Audio Variational autoEncoder (RAVE) built for high- quality audio representation learning and faster than realtime synthesis. Since we target the modelling of 48kHz audio signals, we leverage multiband decomposition of the raw waveform (see appendix B) as a way to decrease the temporal dimensionality of the data Yang al. 2020 This allows us to expand the temporal receptive field of our model
5

---

Under review as a conference paper at ICLR 2022
without major increase in computational complexity. We demonstrate the influence of the multiband decomposition on the synthesis speed in section 5.2
Using 16-band decomposition, we successfully model 48kHz audio signals while producing a compact latent representation using the post-training analysis presented in section 3.2
Encoder. We define our encoder as the combination of a multiband decomposition fol- lowed by simple convolutional neural network, transforming the raw waveform into 128-dimensional latent representation. A detailed description of the architecture of the encoder given in annex C.1
Decoder. Our decoder is a modified version of the generator proposed by Kumar et al. (2019). We use the same alternation of upsampling layers and residual networks, but instead of directly outputting the raw waveform we feed the last hidden layer to three sub-net works. The first aub-network (waveform) synthesizes multiband audio signal (with tanh activa- tion), which is multiplied by the output the second nebonetwork (loudress), generating an amplitude envelope (with sigmoid activation). The last sub-network is noise synthesizer as proposed in |Engel et al. (2019). and produces a multiband filtered noise added to the previous signal.
We found that using an explicit amplitude envelope helps reducing artifacts in the silent parts of the signal, while the noise synthesizer slightly increases the reconstruction natural- ness of noisy signals. See annex C.2 for more details about the architecture of the decoder.
Discriminator. We use the exact same discriminator as in Kumar et al. (011)) which strided convolutional network applied on different scales of the audio signal prevent artifacts. We also use the same feature matching loss as in the original paper.
Training. We follow the training procedure proposed in 3.1 and train RAVE for 3M steps, specifically 1.5M steps for each stage, summing to a total of approximately 6 days on single TITAN V GPU, We use the Adam optimizer (Kingma & Ba 2015) with learning rate of 10-1, 3 = (0.5, 0.9), and batch size 8. We use dequantization, random crop and allpass filters with random coefficients
data augmentation strategy.
Baselines. We evaluate our model in the context of unsupervised representation learn- ing. We compare it against two state-of-the-art models: an unsupervised NSynth (Engel et al.) 2017) and the autoencoder from SING (Défossez al., 2018). We use the official implementations for both baselines with default parameters. We do not to evaluate the DDSP (Engel al., 2019 and NSF (Wang al.) 2019) approaches since we target the unsupervised modelling of any type of audio signals. Notably, our proposal is able to model both monophonic and polyphonic signals, while the aforementioned methods are restricted to monophonic signals.
4.2 DATASETS
Strings. Since our main target is the modelling of musical audio signals, we use an in- ternal dataset composed of approximately 30 hours of raw recordings of strings in various configurations (monophonic solos and polyphonic group performances, with different styles and recording configurations) sampled at 48kHz. We employ a 90/10 train/test split, We downsample this dataset. to 16kHz when using NSynth and SING.
The Voice Conversion ToolKit Yamagishi et al. 2019 is speech dataset com- posed of approximately 44 hours raw audio sampled at 48kHz, produced by 110 different speakers with various accents. We use it to evaluate the performances of RAVE when ad- dressing the speech modelling task in an unsupervised fashion. We also employ a 90/10 train/tess /est split and downsample to 16 kHz for other methods.
VCTK.

---

Under review as a conference paper at ICLR 2022
5 RESULTS
To encourage reproducibility, we release the source code of our model alongside pretrained weights used produce the presented results. Audio samples corresponding to the different parts of this section are available our accompanying website.
5.1 SYNTHESIS QUALITY
First, performed qualitative experiment where participants were asked to rate audio samples on a scale of 1 to 5. The test consisted of 15 trials, with each presenting an audio sample of the strings dataset never seen during training, alongside its reconstruction by the three models. The order in which the trials and samples are presented was randomized. A total of 33 participants took the test, with most being audio professionals. We report the results of this study in table
Table Reconstruction quality evaluation (Mean Opinion Score)
Model
MOS
95%
CI Training time
Parameter count
Ground truth
4.21
-0.04
NSynth
2.68
10.04
13 days
64.7M
SING
1.15
#0.02
~ 5 days
80.8M
RAVE (Ours)
3.01
#0.05
~ 7 days
17.6M
As we can see, RAVE outperforms both NSynth and SLNG in terms of audio quality without relying on autoregressive generation. Furthermore, it achieves high-quality audio synthesis with least 3.5 times less paramcters. Thore still a gap the evaluation betwoen the ground truth and the other models. This might come from the difficulty of modeling the variely of room acoustics conditions present the original datasel (as discussed in Engel ee al. (2019)), sometimes making it obvious that the evaluated samples are synthesized.
5.2 SYNTHESIS SPEED
adversarial models (Kmar et al. 2019) can achieve faster than realtime synthesis, but only by relying on GPUs. Other approaches make strong assumptions about the signal in order to simplify the generation process (Engel et al. 2019 Wang et al.) [2019), allowing real-time synthesis on CPU, while restricting the range of audio signals that can be modeled. In table 2 we evaluate the synthesis speed of all models on both CPU and GPU. The synthesis speed is calculated as the average number of audio samples generated per second for 100 trials.
Several approaches using normalizing flows (Prenger al. 2019)
Table 2: Comparison of the synthesis speed for several models Model
CPU synthesis
GPU synthesis
NSynth SING
18 Hz
57 Hz
304 kHz
9.8 MHz
RAVE (Ours) w/o multiband
38 kliz
3.7 MHz
RAVE (Ours)
985 kHz
11.7 MHz
Being the only model relying on autoregressive synthesis, NSynth is also the slowest, peak- ing at 57Hz during generation. As expected, the parallel nature of SING and RAVE makes them several orders of magnitude faster than NSynth. The addition of the multiband de- composition speeds up RAVE by factor of 25, allowing model to outperform SING on both CPU and GPU. Overall, we obtain audio synthesis at 48kHz with a 20 faster than realtime factor CPU, and up to 240x GPU.
7

---

Under review as a conference paper at ICLR 2022
5.3 BALANCING COMPACTNESS AND FIDELITY
Having a compact learned representation has several benefits, since it is easier to analyse, manipulate and understand. However, it usually comes at the expense of trading off the reconstruction quality, Instead of determining the latent space dimensionality prior to the training, we rely on the fidelity parameter to estimate it post-training and accordingly crop the learned representation, as explained in section 3.2
In figure 2 we compute the relationship between f and the estimated number of dimensions ry for both the strings and vetk datascts. We depict the influence of on the reconstruc- tion quality by measuring the spectral distance (see equation 5) between the original and reconstructed samples.
strings
strings
dimension number
vctk
spectral distance
1.1
10a
1.0
0.9
0.8
109
0.2
0.8
0.0
0.2
fidelity
fidelity
Figure 2: Estimated latent space dimensionality according to the fidelity parameter and its corresponding influence on the reconstruction quality.
By setting f = 0.99, the dimensionality of the learned representation drops from 128 to just 24 on the strings dataset and 16 the vetk dataset. The downsampling factor of the encoder is 2048, resulting in a latent representation sampled at ~ 23Hz. Further decreasing results in a higher spectral distance the learned representation size. To exemplify this, we reconstruct sample from the velk dataset using different fidelity values, and display the resulting melspectrograms in figure
shown in figure (right), but significantly reduces
(a) Original sample
(b) 0.99
(c) f = 0.90
(d) f f 0.80
Figure 3: Reconstruction of an input sample with several fidelity parameters f.
As the fidelity paramcter decreases, reconstructed samples get less accurate, loosing parts of their attributes such phonemes speaker identity.
5.4 TIMBRE TRANSFER
We demonstrate that RAVE can be used to perform domain transfer even if it has not been specifically trained to address this particular task. We perform domain transfer by simply providing to pretrained RAVE model audio samples coming from outside of its original training distribution (e.g violin samples are reconstructed with a model trained on speech, section F for more details).
8

---

Under review as a conference paper at ICLR 2022
5.5 SIGNAL COMPRESSION
Since the representation learned by RAVE significantly more compact than the raw audio waveform, it can be used a data-driven compression system for transmission and storage purposes, or to produce the base signal for a higher-level model to work on. Applying our model provides a compression ratio of 2048, producing latent. signals sampled at 23Hz, which can be used as a simpler representation for further learning tasks.
To illustrate this, we train WaveNet-inspired model generating latent signals in an autoregressive fashion for performing audio synthesis with RAVE. Combining this autore- gressive model with the decoder from RAVE, we still obtain synthesis at a times faster than realtime factor. Furthermore, since latent signals are sampled at much slower rate than the raw waveform, we obtain a 3-second-long receptive field with as few as 9M param- eters, corresponding to only 10 layers of the original WaveNet architecture. Examples of unconditional generation are available on our accompanying website.
RELATED WORK
The combination of VAE and GAN has already been studied as way to build a represen- tation learning model with high quality generation abilities. Larsen et al. (2015) propose to replace the reconstruction loss with a learned metric similar to the feature matching loss we use in equation g This learned metric can be seen a perceptual loss (Kumar et al. 2019), but we demonstrate in annex E that using it to train our encoder results in larger estimated latent space dinensionalitty, making the learned representation harder to analyse and interact with.
Previous approaches combine perceptual losses and auxiliary losses to achieve high quality audio synthesis. Yamamoto et al. (2020) leverage a multiscale spectral loss together with an adversarial objective to address the mel-scale spectrogram inversion task, while Ping et al.) (2018) use spectral loss as a guide to stabilize their distillation procedure. Contrary to our two-stage procedure, they mainly use this combination
warmup mechanism to help their model converge, whereas we use both losses with two separate tasks in mind: representation learning encompassing spectral altribules of the signal, and high-qualily andio synlhesis based on an adversarial objective, using a fixed learned latent space.
The use of multiband decomposition of the raw waveform has been successfully applied by Yu et al. (2019) and Yang et al. (2020) to the audio modelling task, and they both show that it helps producing higher quality results while redueing the training and synthesis time. Modelling 48kHz audio is challenging in term of temporal complexity and memory requirements, and building up from the aforementioned work we show that using 16 band decomposition allows RAVE to address this task.
7 CONCLUSION
In this paper, we introduced RAVE: a Realtime Audio Variational autoEncoder for fast and high-quality neural audio synthesis. First, we proposed two-stage training procedure, which ensures that the latent representation is adequate, prior to performing adversarial fine- tuning for generating high-quality audio signals. We showed that our method outperforms the performances of previous approaches in both quantitative and qualitative analyses. By leveraging multiband docomposition of the raw waveform, we are able to achieve 20 times faster than realtime synthesis on a standard laptop CPU. Finally, we proposed a method to control the trade-off between reconstruction quality and representation compactness, easing the analysis and control of the learned representation. We provide the open-source code of RAVE and hope that this will spark contributions and creative uses of our model.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A Recurrent Latent Variable Model for Sequential Data. Advances in Newral Information Processing Systems, J01urrr00 2988, 6 2015. ISSN 10495258. URL https://arxiv.. wttp://.rraabb//////oo/ 02216v6
Alexandre Défossez, Neil Zeghidour, Nicolas Usunier, Léon Bottou, and Francis Bach. SING: tymmoll-o-Instrument
Symbol-to- Instrument Neural Generator. Advances in Neurol Information Processing Sys- tems, 2018- 900--eeemmer:99011-220000
10 2018, URL httph:tt//rrrtttae wwwpziiiv.rrgaeeo aww/aaaaa.1...1 o 09785v1
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A Generative Model for Music, orXiv, 4 2020. URL http://arxiv.
org/abs/2005.0035 00341
Josse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Dou- glas Eck, and Karen Simonyan. Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders, 7 2017. ISSN 2640-3498. URL ltttps:/ppooceedingg: mlr. press//7ss///// engel17a.html
Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts. DDSP: Differentiable Digital Signal Processing. Technical report, 9 2019. URL https://goo. g1 //mmagett//
ddsp-examples
Philippe Esling, Axel Chemla- Romou-Santos, and Adrien Bitton. Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics. DA 2018 - Proceedings:
21st International Conference on Digital Audio Effects, pp. 369 376, 2018. URL http:
//arxiv org/ 1aobs//880....0 08501
Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, and Ole Winther. Sequential Neural Models with Stochastic Layers. Advances in Newral Information Processing Systems, pp. 5 2016. ISSN 10495258. URL https: hoapsi///rxivv..// arxiv. org/ abss/6605.077111
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. Commu- nications of the ACM, 63(11):139-144, 6 2014. URL https:///axxi: www.xii.org/aaaaaa org/abs//406.
2661v1
Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAFF Learning Basic Visual Concepts with a Constrained Variational Framework, 2016.
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings. International Conference on Learning Representations, ICLR, 2015.
Dioderik Kingma and Max Welling, Auto-encoding variational bayes. In 2nd International Conference on Leurning Representations, ICLR 2014 Conference Track Proceedings. International Conference on Learning Representations, ICLR, 2014. URL https:
//arxiv. /arxiv.org/abbel arggabs//3//3 0tt///112...1111 131.11111116661
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, and Aaron Courville. MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis. arXiv, 10 2019.
URL
19t1/19111.00011111
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Murela Ranzato. Fader Networks: Manipulating Images by Sliding Attributes. Advonces in Neural Information Processing Systems, D017-December:5968 5977, 6 2017.
ISSN 10495258. URL [ https:tt//// attts://aaxivv http:/oow///bbb///00000000.
10

---

Under review as a conference paper at ICLR 2022
Anders Boesen Lindbo Larsen, Søren Kaae Sonderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. 33rd International Con- ference Machine Learning, ICML 2016, 12241-23453 12 2015. URL https://arxiv.
org/abs/ 2
Yuan Pei Lin and P. P. Vaidyanathan. A kaiser window approach for the design of protolype filters of cosine modulated filterbanks. IEEE Signal Processing Letters, 5(6))132-1344 1344 1998. ISSN 10709908. doi: 10.1109/97.666 97.11177777
M. Rossi, Jin- Yun Zhang, and W. Steenaart. A new algorithm for designing prototype filters for M-band Pseudo QMF banks, 1996. URL https:///eeexpplort ieee. wrgg/dcuuenn.....ooooooom
7083013
Soroush Mchri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Josc Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn: An unconditional end-to-end neural audio generation model. In 5th International Conference on Learning Representa- tions, ICLR 2017 Conference Track Proceedings. International Conference on Learning Representations, ICLR, 2017.
Noam Mor, Lior Wolf, Adam Polyak, and Yaniv Taigman. A Universal Music Translation Network. Xrviv, 5 2018. CRL http:://arriv.org/bbb//// 078488
Truong Q. Nguyen. on Signal Processing, 42(1):65-76, 1994. ISSN 19410476. doi: 10.1109/78.25811
Pseudo-QMP Banks. IEEE Transactions
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A Gen-
crative Model for Raw Audio. 9 2016. URL htttp://axxiv.rrrt httg://bs/160...000
Wei Ping, Kainan Peng, and Jitong Chen. ClariNet: Parallel Wave Generation in End-to- End Text-to- Speeec. Technical report, 9 2018. URL
gitthb..
10/
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A Flow-based Generative Network for Speech Synthesis. In TCASSP, IEFE International Conference on Acoustics, Speech and Signal Processing - Proceedings, volume 2019-May pp. 3617-3621. Institute of Electrical and Electronics Engineers Inc., 5 2019. ISBN 9781479981311. doi: 10.1109/ 8683143.
ICASSP 201.:201112000 231.8883.1...
Sean Vasquez and Mike Lewis. MelNet: Generative Model for Audio in the Frequency Domain. arviv, 6 2019. URL http:://axiiii http:///xxxvvvoro////e/ wttp://///90./0000000000000
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural source-filter waveform models for statistical parametric speech synthesis. TEEE/ACM, Transactions on Audio Speech and Language Processing, 28:402-415, 415, 4 2019. URL https:://axxi..rrrtt wwtp://///////////t////o.ossssa
Jumichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. CSTR VCTK Corpus:
English Multi-speaker Corpus for CSTR Voice Cloning Toolkit, 2019.
Ryuichi Yamamoto, Eunwoo Song, and Jae Min Kim. Parallel Wavegan: A Fast Wave- form Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram. In ICASSP, IEEE International Conference on Acoustics, Speech and Sig- nal Processing- Proceedings, volume 2020-May, pp. 6199-6203, Institute of Electrical and Electronics Engineers Inc., 5 2020. ISBN 9781509066315. doi: 10.1109/ICASSSP40076. 2020.9053795.
Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie. Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to- -epeech orXiv, 5 2020. URL http:
//arxiv. org/ abs 2005. 05106
Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng Liu, Deyi Tuo, Shiyin Kang, Guangzhi Lei, Dan Su, and Dong Yu. DurIAN: Duration Informed Atlention Network For Multimodal Synthesis. 9 2019. URL http:://axx:t wttp:///www//bb/....oooo.o 01700
11

---

Under review as a conference paper at ICLR 2022
A RANGE AND NULL SPACE ESTIMATION
Let Z be a b x d real-valued matrix with Σ, Zig 0. decomposition of Z
We consider the singular value
Z UEVI
(12)
where U and V are orthogonal real-valued matrices of respective dimensions bx and d x d, and Σ is a rectangular diagonal matrix with non-negative values on the diagonal called singular values, sorted by decreasing value. The number of non-zero singular values gives the rank r of the matrix, i.c the dimension of the vector space spanned by its columns.
The range and mull space associated with Z are respectively spanned by the first and last r r columns of VT. A low rank version of can be obtained by keeping only the r first columns of Σ and V, such that
Z= = UÊVT,
(13)
where E and VT are respectively of dimension b x and r x d. Note that the equality in equation
13 only holds if the last d - singular values are equal to 0. Rearranging
equation 13 we get the formula to perform a Principal Component Analysis (PCA) on Z
with components
Ż - ZV = UÊ
(14)
Since V is orthogonal, we get Z back by multiplying equation
14 by VT.
B MULTIBAND DECOMPOSITION
The main use of a multiband docomposition is to represent an audio signal sampled at a given sampling rate (c.g 48kHz) combination of several downsampled sub-signals (c.g 3kHz), where each sub-signal covers a particular range of frequencies. This decomposition useful for compression purposes such as mp3 of different bit rates on specific areas of the spectrum.
sreedinn/deodiing since it allows the use
Multiband decompositions of the raw waveform have already been repeatedly applied in the context of audio generative modelling. (Yu et al.] 2019) specifically use Pseudo Quadra- ture Mirror filters (PQMF), definod by Nguyen (1994). PQMF are M-ndnd filter-banks that split a signal into M sub-signals decimated by a factor M. The M filters are cosinc modulations of prototype low-pass filter with cutoff frequency
fc 2M
(15)
carefully designed to avoid aliasing in the reconstructed signal. In practice, it is impossible to create perfect prototype filter (i.e perfectly rejected band), but we can instead design filter that allows the cancellation of aliasing between neighbouring sub-bande using methods such as the optimisation of a non-lineas objective function as described by M Rossi et al. (1996), or using simple Kaiser window whose parameters are optimised in analysis- synthesis pipeline, as proposed by Lin & Vaidyanathan (1998).
Since the filter-bank is orthogonal basis of the signal due to the cosine modulations implied during its creation, the re-synthesis process can be performed by reapplying a tem- porally flipped version of the filter-bank to the sub-hands.
12

---

Under review as a conference paper at ICLR 2022
C MODEL ARCHITECTURE
C.1 ENCODER
The encoder of RAVE is convolutional neural network with leaky ReLU activation and batch normalization (see figure 4). For all of our experiments we use N = 4, with hidden sizes [64, 128, 256, 512] and strides [4, 4, 2]. The full latent space has 128 dimensions.
audio
multiband decomposition
Convid
BetchNormtd
Leaky ReLU
Convid
softplus
variance
Figure 4: Architecture of the encoder used in the RAVE model.
C.2 DECODER
We detail the different blocks composing the decoder.
xN
corvv
latent representation
upsampling layer
residual
loudness corvv
audio
noise synthesizer
Figure 5: Overview of the proposed decoder. The latent representation is upsampled using alternating upsampling layers and residual stack. The result is processed by three sub- networks, respectively producing waveform, loudness envelope and filtered noise signals.
x
3
leaky ReLU
Convid
leaky ReLU
Conv Transpose
(a) Upsampling layer
(b) Residual stack
N
leaky ReLU
Convid
filter
filtered noise
white noise
(c) Noise synthesizer
Figure 6: Detailed architecture of the decoder blocks used in the RAVE model.
13

---

Under review as a conference paper at ICLR 2022
D LATENT COMPONENT COLLAPSE
The regularisation term in equation @ applies a pressure on the encoder to produce a poste- rior distribution that is close to the prior, and therefore not correlated with the input data. Since this objoctive is contrary to the reconstruction objective, it has been empirically shown by Higgins al. (2016) that some latents are more informative than others, as can be seen in ligure in the case of RAVE.
divergence from prior
divergence from prior
10
10a
10-1
10
10
10
10
25
50
75
100
125
25
50
75
100
125
latent component number
latent component number
(a) Strings dataset
(b) VCTK dataset
Figure Mean Kullback Leibler divergence for cach latent component between the posterior distribution (z(x) estimated over the test set of the strings and VCTK datascts and the prior distribution.
For both datasets, most of the latents have a low KT. divergence (lower than 0.01), and only a few (respectively 16 and for the strings and VCTK datasets) have a KT. divergence higher than 0.1, which is consistent with the dimensionality estimated by the method proposed in section 3.2 for a fidelity parameter f = 0.95.
E
TWO STAGE TRAINING AND LATENT SPACE COMPACTNESS
Compared to previous approaches combining VAE and GANs, there no adversarial losses involved when training the encoder. We demonstrate in figure S how training the encoder
to minimize the feature matching loss as proposed by Larsen et al. 22055) during the second
stage results in dramatically increased estimated latent space dimensionality, which goes against our objective of building a compact representation.
Latent dimensionaliyy
f-0.80
Latent dimensionality
f-0.80
f-0.90
60
f=0.90
f-0.95
f-0.95
f-0.99
f-0.99
40-
ww Vmmn
20
20
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Iteration number
Iteration number
Frozen encoder
(b) Trained encoder
Comparison of the estimated latent space dimensionality for two trainings of RAVE on the Strings dataset with and without freezing the encoder during the adversarial fine tuning stage (starting at 1.10 iterations).
Figure 8:
14

---

Under review as a conference paper at ICLR 2022
F OUT OF DOMAIN LATENT REPRESENTATION
We demonstrate figure that RAVE can be used to address the timbre transfer task (see section 5.4
(a) Transfer from strings to velk
(b) Transfer from welk strings
Figure 9: Example timbre transfer using RAVE.
High-level attributes such as the overall loudness and the fundamental frequeney of the harmonic components are kept after performing domain transfer. Other audio attributes such as formants are absent in the examples coming from the strings dataset, and are added by the model in the speech-transfers version.
However, there are no guarantees that the encoder will encode signals from an out-of-domain distribution into a latent representation that matches the prior. We empirically evaluate the KL divergence of two trained RAVE models for in and out-of-domain data distributions, and report the results in table
Table KL divergence from the prior for in and out-of-domain data.
RAVE Strings RAVE VCTK
data from Strings
0.11 = 0.09
0.16 0.11
data from VCTK
0.24 + 0.37
0.09 + 0.02
Unsurprisingly, the smallest KL divergences are observed when encoding data sampled from the training distribution, while encoded out-of-domain signals roughly double the KL di- vergence. Stronger transfer abilities may be achieved by integrating a domain adaptation technique such the one proposed by Lample et al. (0017) or Mor al. (0188)
15


Reviews:
[{"review_id": "kNX0upFQBt7", "reviewer": "Reviewer_4x1t", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "– Missing references to Fraccaro et al, 2016, and Chung et al, 2016 with the SRNN and VRNN respectively.\n– Missing references to other literature using VAE and GAN in combination for training. \n– Eq. 2 is only the real ELBO for beta=1. That should be stated.\n– Section 3.2 paragraph 1: You would need some references to underline these claims. The reviewer would argue that these are to a certain extent not always true when ref to literature.\n– It seems to me that the main evaluation of the model is performed in Table 1. However, these samples are reconstructions. How can you make sure that after fine-tuning the model, your KL-term remains low so that your sample quality can be high?\n– In the article, you claim quantitative evaluation of the proposed model. Since you have an ELBO in the model, it would be very nice to see an evaluation of the approximation to the likelihood and compare this to other models, e.g., a purely autoregressive model with its exact likelihood.", "summary_of_the_paper": "The authors propose a model for audio generation, trained in two stages: 1) as a changed VAE with a likelihood evaluated by comparing the STFT representations and a temperature (beta) on the KL term in order to avoid posterior collapse, 2) as a GAN-optimization scheme to fine-tune the model.", "summary_of_the_review": "Thank you for a very nice read. While finding the results interesting, I have a hard time arguing whether the proposed innovations are in fact impactful. I'm lacking quantitative experiments, e.g., ablations studies, on the components and the optimization details introduced. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bQQEm5FDDMl", "reviewer": "Reviewer_Pcii", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n1. Not only the audio synthesis but also two interesting applications (timbre transfer and signal compression) are demonstrated.\n\n2. Audio samples are provided on the web page. They help understand the effectiveness of the proposed method.\n\n[Weaknesses]\n1. My primary concern is that the technical novelty is limited or not well described. Although the objectives are not entirely the same, a similar two-stage training scheme (in the first stage, only the reconstruction loss is used, and in the second stage, the adversarial loss is introduced) has already been used in previous studies (e.g., [A]). In Sections 3.1.1 and 3.1.2, I cannot find new losses or new techniques, which are unique to the presented task. At least, it would be better to clarify the technical difference from previous two-stage training schemes.\n\n- [A] R. Yamamoto et al., “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” ICASSP 2020.\n\n2. The combination of VAE and GAN has been proposed in previous studies, e.g., [B]. Also, in this respect, I feel that the technical novelty is limited. At least, it would be better to explain the relationships with such related work more politely.\n\n- [B] A.B.L., Larsen et al., “Autoencoding beyond pixels using a learned similarity metric,” ICML 2016.\n\n3. The improvement of synthesis speed is included in the main contribution. However, it mainly relies on incorporating multi-band decomposition, which was proposed in the previous study [Yu et al., 2019]. The combination of GAN-based waveform synthesis and multi-band decomposition was also already introduced in [C]. Therefore, I think that the incorporation of multi-band decomposition into the current framework is not so novel.\n\n- [C] G. Yang et al., “Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech,” SLT 2021.\n\n4. In Section 2, some related work is discussed. However, the discussion is minimal. For example, in Section 2.1, VAE is addressed under the title of “state-of-the-art”; however, only the original one [Kingma & Welling, 2014] and one variant [Higgings et al., 2016] are presented despite extensive studies in this field. In addition, there are many relevant work (e.g., [A][B][C]), which should be discussed more comprehensively in this paper.\n\n5. In the experiments, 48 kHz audio waveforms are used for the proposed model, while 16 kHz audio waveforms are used for the baseline models. This experimental setting is unfair. I have a question (1) how performance is changed when the proposed model is applied to 16 kHz audio waveforms and (2) how performance is changed when the baseline models are applied to 48 kHz audio waveforms.\n\n6. Although timbre transfer is an interesting application, it is not easy to judge whether the proposed model performs well. In general, the VAE encoder does not ensure that out-of-distribution data is encoded to the valid latent space. I cannot understand whether such an encoding error is alleviated in the proposed model.", "summary_of_the_paper": "This paper aims to achieve fast and high-quality audio synthesis. The authors propose a variational autoencoder-based model called real-time audio variational autoencoder (RAVE) to achieve this aim. In particular, the authors introduce a two-stage training scheme to alleviate the difficulty in joint learning of the adequate representation and high-fidelity audio synthesis. More concretely, in the first stage, representation learning is conducted in a VAE framework. In the second stage, audio fidelity is improved using adversarial fine-tuning. The authors also introduce a post-analysis method to manipulate the balance between fidelity and compactness. The effectiveness of the proposed method was evaluated on the Strings and VCTK datasets. The applications to timbre transfer and signal compression were also demonstrated.", "summary_of_the_review": "My primary concern is that the technical novelty is limited or not well described. The two-stage training scheme, the combination of VAE and GAN, and the incorporation of multi-band decomposition have already been proposed in the previous studies. I expect that the authors clarify this point in the rebuttal. In addition, I hope that the authors explain the validity of the experiments regarding the issues raised above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_F7q7yGZ9N5", "reviewer": "Reviewer_kQnp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The synthesis speed is much faster than baselines due to non-autoregressive inference and the compact architecture.\n2. The dimension of the latent representation can be significantly reduced without much degradation in audio quality. SVD is applied to omit less informative information in the latent representation.\n3. The autoencoder is able to perform timbre transfer while largely preserving loudness and fundamental frequency.\n\nWeakness:\nIt would be good to apply the proposed model as a vocoder in text-to-speech synthesis. For example, train a speech synthesis model to predict latent representation, which is then converted to waveform.", "summary_of_the_paper": "This paper proposes an autoencoder for high-quality waveform synthesis. The approach is based on multi-band decomposition, VAE and adversarial fine-tuning. The synthesis speed is substantially faster than existing approaches on both CPU and GPU.", "summary_of_the_review": "This paper proposes an audio synthesis approach that runs fast on CPU and yields high-fidelity audio. It combines VAE and GAN loss for non-autoregressive waveform synthesis. Although several use cases are discussed, no solid application (such as TTS) has been demonstrated.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to high quality waveform synthesis using multi-band decomposition. The resulting synthesis speed is substantially faster the past work on both CPU and GPU -- a feature that all reviewers viewed as a significant strength. However, the majority of reviewers raised concerns about discussion of and contextualization within past work, as well the novelty of the proposed approach. Finally, one reviewer pointed out a potential concern with experimental evaluation (sample rate of proposed system outputs vs baseline's). Author response clarified the relationship with some past work but did provide evidence to mitigate the concerns about experimental evaluation. Overall, this paper could still benefit from another round of review.
Decision: Reject