Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an autoencoder for high-quality waveform synthesis. The approach is based on multi-band decomposition, VAE and adversarial fine-tuning. The synthesis speed is substantially faster than existing approaches on both CPU and GPU.
main_review: Strength:
1. The synthesis speed is much faster than baselines due to non-autoregressive inference and the compact architecture.
2. The dimension of the latent representation can be significantly reduced without much degradation in audio quality. SVD is applied to omit less informative information in the latent representation.
3. The autoencoder is able to perform timbre transfer while largely preserving loudness and fundamental frequency.

Weakness:
It would be good to apply the proposed model as a vocoder in text-to-speech synthesis. For example, train a speech synthesis model to predict latent representation, which is then converted to waveform.
summary_of_the_review: This paper proposes an audio synthesis approach that runs fast on CPU and yields high-fidelity audio. It combines VAE and GAN loss for non-autoregressive waveform synthesis. Although several use cases are discussed, no solid application (such as TTS) has been demonstrated.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to achieve fast and high-quality audio synthesis. The authors propose a variational autoencoder-based model called real-time audio variational autoencoder (RAVE) to achieve this aim. In particular, the authors introduce a two-stage training scheme to alleviate the difficulty in joint learning of the adequate representation and high-fidelity audio synthesis. More concretely, in the first stage, representation learning is conducted in a VAE framework. In the second stage, audio fidelity is improved using adversarial fine-tuning. The authors also introduce a post-analysis method to manipulate the balance between fidelity and compactness. The effectiveness of the proposed method was evaluated on the Strings and VCTK datasets. The applications to timbre transfer and signal compression were also demonstrated.
main_review: [Strengths]
1. Not only the audio synthesis but also two interesting applications (timbre transfer and signal compression) are demonstrated.

2. Audio samples are provided on the web page. They help understand the effectiveness of the proposed method.

[Weaknesses]
1. My primary concern is that the technical novelty is limited or not well described. Although the objectives are not entirely the same, a similar two-stage training scheme (in the first stage, only the reconstruction loss is used, and in the second stage, the adversarial loss is introduced) has already been used in previous studies (e.g., [A]). In Sections 3.1.1 and 3.1.2, I cannot find new losses or new techniques, which are unique to the presented task. At least, it would be better to clarify the technical difference from previous two-stage training schemes.

- [A] R. Yamamoto et al., “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” ICASSP 2020.

2. The combination of VAE and GAN has been proposed in previous studies, e.g., [B]. Also, in this respect, I feel that the technical novelty is limited. At least, it would be better to explain the relationships with such related work more politely.

- [B] A.B.L., Larsen et al., “Autoencoding beyond pixels using a learned similarity metric,” ICML 2016.

3. The improvement of synthesis speed is included in the main contribution. However, it mainly relies on incorporating multi-band decomposition, which was proposed in the previous study [Yu et al., 2019]. The combination of GAN-based waveform synthesis and multi-band decomposition was also already introduced in [C]. Therefore, I think that the incorporation of multi-band decomposition into the current framework is not so novel.

- [C] G. Yang et al., “Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech,” SLT 2021.

4. In Section 2, some related work is discussed. However, the discussion is minimal. For example, in Section 2.1, VAE is addressed under the title of “state-of-the-art”; however, only the original one [Kingma & Welling, 2014] and one variant [Higgings et al., 2016] are presented despite extensive studies in this field. In addition, there are many relevant work (e.g., [A][B][C]), which should be discussed more comprehensively in this paper.

5. In the experiments, 48 kHz audio waveforms are used for the proposed model, while 16 kHz audio waveforms are used for the baseline models. This experimental setting is unfair. I have a question (1) how performance is changed when the proposed model is applied to 16 kHz audio waveforms and (2) how performance is changed when the baseline models are applied to 48 kHz audio waveforms.

6. Although timbre transfer is an interesting application, it is not easy to judge whether the proposed model performs well. In general, the VAE encoder does not ensure that out-of-distribution data is encoded to the valid latent space. I cannot understand whether such an encoding error is alleviated in the proposed model.
summary_of_the_review: My primary concern is that the technical novelty is limited or not well described. The two-stage training scheme, the combination of VAE and GAN, and the incorporation of multi-band decomposition have already been proposed in the previous studies. I expect that the authors clarify this point in the rebuttal. In addition, I hope that the authors explain the validity of the experiments regarding the issues raised above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a model for audio generation, trained in two stages: 1) as a changed VAE with a likelihood evaluated by comparing the STFT representations and a temperature (beta) on the KL term in order to avoid posterior collapse, 2) as a GAN-optimization scheme to fine-tune the model.
main_review: – Missing references to Fraccaro et al, 2016, and Chung et al, 2016 with the SRNN and VRNN respectively.
– Missing references to other literature using VAE and GAN in combination for training. 
– Eq. 2 is only the real ELBO for beta=1. That should be stated.
– Section 3.2 paragraph 1: You would need some references to underline these claims. The reviewer would argue that these are to a certain extent not always true when ref to literature.
– It seems to me that the main evaluation of the model is performed in Table 1. However, these samples are reconstructions. How can you make sure that after fine-tuning the model, your KL-term remains low so that your sample quality can be high?
– In the article, you claim quantitative evaluation of the proposed model. Since you have an ELBO in the model, it would be very nice to see an evaluation of the approximation to the likelihood and compare this to other models, e.g., a purely autoregressive model with its exact likelihood.
summary_of_the_review: Thank you for a very nice read. While finding the results interesting, I have a hard time arguing whether the proposed innovations are in fact impactful. I'm lacking quantitative experiments, e.g., ablations studies, on the components and the optimization details introduced.
review_title: null

Meta-review: This paper presents an approach to high quality waveform synthesis using multi-band decomposition. The resulting synthesis speed is substantially faster the past work on both CPU and GPU -- a feature that all reviewers viewed as a significant strength. However, the majority of reviewers raised concerns about discussion of and contextualization within past work, as well the novelty of the proposed approach. Finally, one reviewer pointed out a potential concern with experimental evaluation (sample rate of proposed system outputs vs baseline's). Author response clarified the relationship with some past work but did provide evidence to mitigate the concerns about experimental evaluation. Overall, this paper could still benefit from another round of review.
Decision: Reject