Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper presents an approach based on generative adversarial models for the unconditional  generation of audio. The authors take inspiration from WaveGAN, to which they add more sophisticated upsampling blocks (called the bandwidth extension module) instead of transposed convolutions. They also propose to add a sinc convolution layer to the discriminators to improve training. Finally, they propose a progressive training scheme similar in spirit to the progressive training of GANs in images. Experiments are performed on generating audio pronunciation of digits, and the authors compare their work in terms of inception score, human evaluation and computation cost to WaveGAN.

As stated by the authors, one of the main motivation oof their approach is the reduction of computation cost. The motivation, expressed in terms of the 10ms "interactive rate" follows a good story. The measurements performed at the end of the paper show a ~ x2 performance gains compared to WaveGAN on CPU and about same running times on GPU, which is significant but not compelling.

Overall, I liked the story of the paper, but the paper lacks clarity and details. An important aspect of the paper is the progressive training, which is detailed nowhere (e.g., what is the stopping criterion to get to the next stage), should there be a special initialization of the last block of a new stage, etc.). The "Bandwidth extension module", which from my understanding is one of thee main contribution of the paper, is detailed in the appendix and comes essentially without justification. One of the main motivations of the paper is to be able to generate 44kHz audio, but the only results available at this resolution are inception scores that are below those of the 16kHz generation, which leaves open the question of whether the goal is effectively achieved.

I found the different versions of PUGAN difficult to read. The picture uses 2 blocks of bandwidth extension to generate 16kHz, whereas the evaluations are done with PUGAN-1 (1 block), which if I understand correctly is based on the lightweight WaveGAN that generates at 8kHz (whereas Section 4.1 suggests that 16kHz is generated from 4kHz generations by WaveGAN and two blocks). Also, the fact that evaluations are carried out with PUGAN-1 suggests that the progressive training does not really works well past a single block.

In the text it is suggested that spectral normalization and sinc conv on the discriminator is an "improvement" of WaveGAN, and an independent contribution of the article. While Table 2 clearly shows an improvement in terms of inception score, the human evaluation is not that clear: the accuracy of human labelers drops to 0.52 from 0.63 and the quality seems totally within the variance (2.7 +/- 1.5 vs 2.6 +/- 1.3). While Table 3 also shows clear improvement over the basic WaveGAN with the changes made by the authors (in particular in terms of win ratio vs PUGAN-1), the loss of accuracy should be discussed.

The performance obtained by PUGAN-1 is nonetheless noticeable -- +1 quality score over WaveGAN, and much better pairwise win ratios. Nonetheless, the paper lacks clarity and motivation for the exact form of the bandwidth extraction module, and does not fulfill its promises (importance of progressive training, high quality generation at 44kHz), so I am leaning towards rejection.


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors detail PUGAN, architectural changes to models for raw waveform generation with GANs. They do a good job of motivating the challenge of raw audio generation with GANs and of methods for progressive training. PUGAN incorporates U-Net modules in the generator ("Bandwidth expansion"), sinc convolution as bandlimiting inputs to the generators, and the "style gan" type method of adding the noise at each level of the generator. Using listener studies and inception score, they show modest improvements over the state of the art (at time of submission), WaveGAN. Notably, their architecture is also more computation and parameter efficient. 

The paper is well motivated and experiments are correct, but the quality improvements overall are a little underwhelming. To some extent, that can't be helped, and the authors wisely focus on the improvements in inference time as one of their central claims, and indeed produce evidence to support this claim. 

More problematic, however, the paper motivates the problem of multi-scale generation of waveforms, but does not clearly show that the proposed architectures address those issues. The motivation in terms of interpolation artifacts and band-limited upsampling in Figure 1 give a misleading sense that Kaiser resampling is explicitly incorporated into the model. The authors argue that the U-Net layers implicitly learn an upsampling method, but the lack of model comparison / ablation makes it difficult to see if that really is the case. It would help support the claim to show samples from the model at different resolutions and demonstrate the lack of artifacts at each level. In the appendix, the authors mention that any alterations to the architecture resulted in failed training, but the lack of an ablation study makes it hard to know the relative value of each component. For example, it is unclear how important the sinc layers and bandlimiting are for the discriminators, the intermediate noise, and the use of the BWE architectures.

Despite these shortcomings I still recommend a weak accept, as the problem is difficult and the paper documents a well-motivated avenue for approaching it.

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Summary:

This work is a follow-up of WaveGAN. It uses the first few layers of the original WaveGAN to synthesize low resolution waveform (4kHz), and applies several bandwidth extension modules to progressively output the higher resolution raw audios.

pros:
- The proposed PUGAN has significantly smaller number of parameters than WaveGAN (e.g., 20x smaller).

cons:
- WaveGAN was a preliminary and encouraging trial for raw audio synthesis with GAN. Note that, its audio fidelity is far away from the state-of-the-art results and it was only tested on simple dataset (sounds of ten-digit commands). In contrast, the state-of-the-art autoregressive models (e.g., WaveNet) and parallel flow-based models (e.g., Parallel WaveNet) have been tested on challenging high-fidelity speech synthesis. As a result, one may focus on improving the audio fidelity of GAN on more challenging tasks. However, the proposed PUGAN was still tested on very simple dataset (sounds of ten-digit commands), and the quality of generated samples are only comparable to WaveGAN. 

Detailed comment:

-- The attached samples are pretty noisy (e.g., noticeable artifacts on posted spectrograms). One may introduce the feature matching (e.g., STFT loss in ClariNet) as an auxiliary loss to improve the audio fidelity. 

-- Did the authors try conditional generation, e.g., conditioned on the digit label? The posted failure cases and some samples tend to have overlapped sounds from different digits. 
review_title: Official Blind Review #2

Meta-review: Inspired by WaveGAN, this paper proposes a PUGAN to synthesizes high-quality audio in a raw waveform. The paper is well motivated. But all the reviewers find that the paper is lack of clarity and details, and there are some problems in the experiments.
Decision: Reject