Progressive Upsampling Audio Synthesis via Effective Adversarial Training
Authors: Youngwoo Cho
Minwook Chang
Gerard Jounghyun Kim
Jaegul Choo
Venue:  (2020)
Keywords: audio synthesis
sound effect generation
generative adversarial network
progressive training
raw-waveform

Abstract: This paper proposes a novel generative model called PUGAN, which progressively synthesizes high-quality audio in a raw waveform. PUGAN leverages on the recently proposed idea of progressive generation of higher-resolution images by stacking multiple encode-decoder architectures. To effectively apply it to raw audio generation, we propose two novel modules: (1) a neural upsampling layer and (2) a sinc convolutional layer. Compared to the existing state-of-the-art model called WaveGAN, which uses a single decoder architecture, our model generates audio signals and converts them in a higher resolution in a progressive manner, while using a significantly smaller number of parameters, e.g., 20x smaller for 44.1kHz output, than an existing technique called WaveGAN. Our experiments show that the audio signals can be generated in real-time with the comparable quality to that of WaveGAN with respect to the inception scores and the human evaluation.

Under review as a conference paper at ICLR 2020
PROGRESSIVE UPSAMPLING AUDIO SYNTHESIS VIA EFFICIENT ADVERSARIAL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes novel generative model called PUGAN, which progres- sively synthesizes high-quality audio in a raw waveform. PUGAN leverages on the recently proposed idea of progressive generation of higher-resolution images by stacking multiple encode-decoder architectures. To effectively apply it to raw audio generation, propose two novel modules: (1) neural upsampling layer and (2) sinc convolutional layer. Compared to the existing state-of-the-ar model called WaveGAN, which uses single decoder architecture, our model generates audio signals and converts them in higher resolution in a progressive manner, while using a significantly smaller number of parameters, e.g.. 20x smaller for 44.1 kHz output, than an existing technique called WaveGAN. Our experiments show that the audio signals can be generated real-time with the comparable quality to that of WaveGAN with respect to the inception scores and the human evaluation.
INTRODUCTION
Synthesis of realistic sound is long-studied research topic, with various real-world applications such as text-to-speech (TTS) (Wang et al. [0177] Ping et al. (2018), sound effect (Raghuvanshi et al. 2016), and music generation (Briot al. 2017 Dong et al. 2018 Huang al., (019)). Various techniques have been developed ranging from the sample-based to more computational ones such as the additive/subbractive synthesis, frequency modulation granular synthesis, and even a full physics- based simulation (Cook) 2002). Human's i audible frequency range is up to 20 kHz, so the standard sampling rate for music and sound is 44.1 kHz. Thus, for interactive applications and live perfor- mances, the generation of the high cemporal-resolution audio (i.e., 44.1 kHz) in real-time has to meet the standard of human perceptual sensitivity sound. However, the aforementioned methods often fail to do so, due to their heavy computational complexity with respect to the data size. Because of this, professional sound synthesizers usually have no choice but rely on hardware implementa- Kion..Wessee & Wright 2002)
Generative adversarial networks (GANs) (Goodfellow et al. 2014) have emerged as promising approach to the versatile (e.g., conditional generation from low-dimensional latent vector (Mirza Osindero 2014)) and high-quality (e.g., super-resolution GAN (Ledig et al. 2017) image. One of the first GAN models for sound synthesis have been designed first produce the spectrogram (or some other similar intermediate representations) (Donahue et al. 2019 Engel et al. 2019 Marafioti A spectrogram is compact 2D representation of audio signals in terms of its frequency
et al. 2019). | spectrum over time. The spectrogram can then be converted into the estimated time-domain wave- form using the Griffin & Lim algorithm (Griffin & Lim| 1984). However, such conversion process does not only introduces nontrivial errors but also runs slowly, preventing the approach from being applied at an interactive rate] WaveGAN (Donahue et al. (2019) was the first and state-of-the-art GAN model that can generate raw waveform audio from scratch.
The first generations of sound -onn--genrrating GANs, like the WaveGAN and its followers, have been influenced much by the enormously successful generative models for image synthesis. They can
'The interactive rate refers to the the maximum temporal threshold of around 10msec Wessel & Wright 2002) over which humans would not he able to recognize the sound making event and the resultant sound as accuring the same time.

---

Under review as a conference paper at ICLR 2020
be divided into those that employ the single decoder architecture (e.g.. DCGAN and StyleGAN (Radford et al. 2016; Karras et al.) 2019) and those that encode and decode the intermediate rep- resentations in several and progressive stages (e.g.. StackGAN and progressive GAN (Zhang et al. 2017 Karras et al. (. WaveGAN is the direct descendant of DCGAN with modification for the 11) audio data, while GANSynth applied the concept of progressive generation of audio, but using the 2D spectrogram, treating the audio as 2D image. No previous work in GAN based audio gener- ation has attempted the direct and fast synthesis of ID raw audio waveform employing the multiple and progressive mecoer-ddeoder architecture.
Therefore, in this paper, we propose PUGAN, modification and extension of WaveGAN architec- ture for efficiently synthesizing www-waeeform audio through progressive training. PUGAN gener- ates low sampling rate audio using the first few layers of the original WaveGAN (referred to as the lightweight WaveGAN module). The latter layers of WaveGAN are replaced with the bandwidth extension modules, each of which is composed of the neural upsampling layer and encoder/decoder. They progressively output (progressively trained too) the higher sampling rate audio. For the effec- tive progressive training and generation, instead of the usual upsampling method such as the nearest neighbor used in image generation, PUGAN uses new upsampling methods often employed in the digital signal processing (DSP) field in an attempt to preserve the frequency information of the original data (Oppenheim] 1999). This upsample process consists of the zero insertion and 1D convolution to function as an interpolation infinite impulse response (IIR) filter. On the discrimi- nator side, we add the Sine convolution (Ravanelli & Bengio, 2018) before the first layer to repli- cate the function of the parameterized low pass Sinc filter, also a popular technique in the DSP area. We have also evaluated PUGAN in terms of both quantitative computational performance and qualitative metrics including the human perceptual evaluation. (demo and code: https://pugan-iclr demo. www.oheookkpppcoo..
Overall, our contributions include the following:
propose PUGAN, with novel neural modules (upsampling and bandwidth extension) for the efficient generation of raw waveform audio,
apply the concept of resampling (in the generator) and sine convolution layers (in the dis- criminator) suitable for handling sound generation instead of the conventional upsampling convolution methods, and
demonstrate the effectiveness of the proposed approach by generating raw waveform audio with significantly less number of parameters real-time with equivalent output quality as WaveGAN.
2
RELATED WORK
We first review related research in two areas, namely, the GAN-based sound generation and audio- to-audio conversion.
2.1 GAN BASED AUDIO GENERATION
WaveGAN (Donahue al. 2019 and GANSynth (Engel al. 2019) are the two recent notable work that have applied the GAN technique to sound effects generation for the first time. WaveGAN modified the DCGAN and took the approach operate for and generate one dimensional sound data (raw-orveform) fast and directly (and distinguishing itself from the work like the SpecGAN (Donahue al., 2019) which used the usual 2D/image-base processing and spectrogram output representation). WaveGAN also added the phase shuffle module to prevent the discriminator from learning the checkerboard artifact, and post-processing convolution layer with a relatively wide ker- nel size for noise reduction.
GANSynth generated sound effects through the spectrogram-like representation, but its output qual- ity was satisfactory for only pure tone instrumental sounds. TiF-GAN (Marafioti et al. 2019) made a marginal improvement by adding a provisional step for the phase information reconstruction. Note that in the generative setting, the 2D based approach (using representations like spectrograms) is considered problematic as spectrograms are not fully invertible to sound without a loss (thus inex- act) and the inversion by, say, the most popular Griffin & Lim algorithm is time-consuming.
2

---

Under review as a conference paper at ICLR 2020
1.5
Nearest neighbor (8kHz)
1.0
Original
Amplitude
0.75
Linear interpolation (8kHz)
olltude
0.8
Kaiser resampling (8kHz)
0.6
Original (4kHz)
0.4
-0.75
0.2
-1.5
0
0.0005
0.001 Time (sec)
0.0015
0.002
0.5
1.0
1.5
2.0
Frequency (kHz)
1.0
Nearest neighbor
Linear interpolation
Ideal
Amplitude
0.8
0.6
0.4
0.2
0
1
2
3
0
2
3
0
1
Frequency (kHz)
Frequency (kHz)
Frequency (kHz)
Figure 1: An original signal composed of three frequency components at 600, 750 and 900Hz - time domain plot (top left) and frequency domain (top right) plots. The plots below show the upsam- pling results using the nearest neighbor, linear interpolation and Kaiser resampling methods. The examples illustrate the occurrences of high frequency sidebands (noise) when the first simpler upsampling methods are used. This may be problematic the high resolution output is required. Kaiser resampling is regarded ideal result (bottom right) and we use neural upsampling layer to minimize noise.
fast version of Griffin Lim algorithm (Perraudin al. 2013) and Deep Griffin & Lim iteration approaches (Masuyama et al. 2019) have been proposed to improve the inversion performance; they have not reached the aforementioned level required for interactive applications.
addition, all previous GAN-based sound synthesizers were configured and experimented to output up only 16 kHz audio. Such a sampling rate is sufficient for general TTS systems, but, as noted earlier, not for general sound effects music. WaveNet (Oord et al.) 2016) can generate 44.1 kHz sampling rate audio, but aside from the questionable quality the output, its autoregressive nature make it difficult to achieve real-time performance through the GPU parallelization. In this regard, once trained, the generator part of GAN generally can operate faster. But for WaveGAN to generate 44.1 kHz audio, additional transposed convolutional layers (to the current 16 kHz generator) would make the architectural parameters prohibitively large (two times higher) and likewise the generation time.
2.2 AUDIO-TO-AUDE CONVERSION
Audio-to-audio conversion refers to the task of taking an input audio sample and converting into another with different characteristics. Most deep learning based audio conversion models have been
influenced by similar senage-to-imase translation research. For instance, CycleGAN- VC (Kaneko &
Kameoka 2018). StarGAN- VC (Kameoka et al. 2018). and WaveCycleGAN (Tanaka et al. 2018) all looked into the problem of voice conversion, and were based on the previous works of CycleGAN (Zhu et al. 2017) and StarGAN (Choi et al.) (018). The task of denoising (Pascual al.. 2017)
generation of super-resolution signal (Eskimez & Koishida 2019) can also be regarded as a form of signal (or audio) conversion. Recently, few attempts have made apply GAN the task of bandwidth extensions such as the SSRGAN (Eskimez & Koishida 2019: [i et al., 2019). Note that our objective is the generation of nrigh-resolution audio and sound effects rather than just conversion.
DATA CHARACTERISTICS, AUDIO VERSUS IMAGE
In this section, we discuss potential reasons of why conventional GAN architectures have been successful in generating 2D images (Zhu al. 2017 Karras et al. 2018 2019). but less for ID sound waves with respect to their data characteristics. This analysis can give us hints on how to newly configure the GAN architecture generate the sound signal faster with higher quality.
3

---

Under review as a conference paper at ICLR 2020
Image and sound, both signals, contain information across the frequency domain. Sound has the added dimension of time. Humans are highly sensitive to variation over time of the sound con- tent over all the frequency range, which makes its quality depend on reproduction of all frequency components. In other words, in sound, the different frequency range may represent a particular char- acteristic (e.g., low bass male sound vs. high pitch female sound) (Klevans & Rodman T997). In contrast, in images, high resolution components often correspond to details or even noises, and as such static image recognition and understanding may depend less on them(Heittola et al. 2009).
Upsampling of the data are important parts of the GAN architecture (especially with respect to the conversion process). In image generation, for the reason mentioned above, the upsampling by stan- dard interpolation, such as the nearest neighbor or linear interpolation, may suffice. Fig. Fig.Looompan compares the application of the simple nearest neighbor based upsampling and linear interpolation to the sine function based upsampling (or resampling as better known in the DSP area). Fig. [ shows the up- sampling results using the nearest neighbor, linear interpolation and Kaiser resampling methods. The examples illustrate the occurrences of high frequency sidebands (noise) when the first two simpler upsampling methods are used. This may be particularly problematic if the high nigh-resolution output is required.
Another possibly effective method for dealing with signals of multi multi-frequency components is the essolutionwwise progressive generation (and training) technique, as was demonstrated by the work of Progressive GAN (Karras et al.] 2018). While the original Progressive GAN was applied for 2D images, and similarly to spectrogram generation, we have applied the same idea to the ID audio signal. However, the preliminary pilot result was not satisfactory; the reconstructed results were unnaturally smooth the high frequency range. This is attributed to the similar reason, the stride-1 transposed convolution layer effectively acting as simple moving averaging method.
On the other hand, for an audio generation as WaveGAN has implemented, the upsampling based on the transposed convolution is more proper than the others such as nearest neighbor. It was deemed more accurate "capturing" (filtering out) the frequency-wise characteristics in the generation pro- cess in comparison to using the nearest neighbor. The only problem may be the fact that the number of the relevant architectural parameters grows excessively according to the output size, which ulti- mately would render the generation process non real-time.
To summarize, based on these observations, the newly proposed PUGAN architecture would pro- ceed to first train to learn the gross structure of the aural information distribution and fast produce the low resolution audio, then incrementally convert and enrich the output to a higher resolution effi- ciently instead of having to deal with the entire scale space with computationally heavy architecture simultaneously.
4 PUGAN: PROGRESSIVE UPSAMPLING GAN
In this section, we explain the details of the PUGAN, as also shown in Fig. 2 Note that the objective of the proposed design is to produce 44.1 kHz raw audio waveform with reasonable quality in real- time deployable for interactive and live applications.
The Generative Adversarial Network (GAN) can be applied to generate probablistic solutions to domain problem by framing it as a supervised learning problem with two sub-models: the generator model that one trains to generate new examples, and the discriminator model that tries to classify examples as either real (from the domain) or fake (generated). The two models are trained together in zero-sum game, adversarial, until the discriminator model is fooled about half the time, meaning the generator model is generating plausible examples. The details of the GAN architecture is omitted and referred to (Goodfellow al. 2014).
4.1 PROBABILISTIC MODEL
The probablistic model of our raw waveform generation can be stated as below. We denote the set of audio data downsampled from the maximum sampling rate of Bg, Bo, successively times as X = (xBo, ,,,,, "B, ..., B". and consider the joint probability of X. That is, the audio resolution i is dependent on all its lower resolution data, p(X) II Bi 1xx IxB.11 xB.), or simplified as being dependent only its immediate predecessor, p(X)= = II B B..1).
4

---

Under review as a conference paper at ICLR 2020
Original WaveGAN
Noise
Transposed Convolutions
Trans Conv
Trans Conv
Real (16kHz)
Disc (16kHz)
Gen (16kHz)
Generated (16kHz)
Noise
PUGAN
FC
FC
0
Transposed
Convolutions
Lightweight WaveGAN (4kHz)
BWE Module (4KHz BkHz)
BWE Module (8kHz 16kHz)
Sinc Conv
Sinc Conv
Sinc Conv
Disc (4kHz)
Disc (BKHz)
Disc (16KH2)
Real (4kHz)
Real (8kHz)
Real (16kHz)
Figure 2: Overview of PUGAN. The generator is composed of the lightweight Wavegan Module for first generating the low sampling rate audio and several encoder-decodee architectures called "band- width extension module (BWE)", which upsample the input to high sampling rate and enriches the content. We eliminate the number of layers from WaveGAN and instead attach series of B BWEs. BWE, the neural upsampling layer (indicated with with up and down bidirectional arrow) trained to preserve the frequency information of the original data.
4.2 GENERATOR
The generator part of PUGAN is composed of the lightweight WaveGAN module for first generating the low resolution waveform, and series of U-net based "bandwidth extension modules (BWE)" each of which upsample the input to higher resolution (up to 44.1 kHz) and enriches the content. In other words, the transposed convolutional layers the original WaveGAN generator are replaced with the BWE modules.
4.2.1 LIGHTWEIGHT WAVEGAN MODULE
The original WaveGAN generator synthesizes one second 16 kHz sampling rate audio. We reduce the number of transposed convolutional layers and decrease the sampling rate of the output audio by a factor of four (4 kHz). For the later experimental purpose, we have implemented other versions that would produce kHz and 8 kHz well by adjusting the noise input dimension accordingly. The output from the lightweight WaveGAN module becomes the input for the next step, the BWE. Therefore, depending on the output size of this WaveGAN module, the number of subsequent BWE's would differ (e.g., with kHz lightweight WaveGAN module, two BWEs for 16 kHz or 4 BWEs for 44.1 kHz output).
4.2.2 BANDWIDTHI EXTENSION MODULE (BWE)
The bandwidth extension module plays the role of inserting and adding high frequency information into the input audio. The module has a neural upsampling layer and an encoder-decoder architecture for conversion. In the default configuration, the upsampling unit doubles the sampling rate of the input audio. Moreover, no information loss from the lower resolution data will occur and the upsam- pling unit is emulate the ideal windowed sine filter with its width parameter tuned to the nature of the audio signal. Note that the convolution computation used in the deep learning architectures is mathematically equivalent y-axis symmetric cross-correlation and also the sinc filter.
5

---

Under review as a conference paper at ICLR 2020
Ratio
Sampling rate
No. of Params. (in mill.
Architecture
Lightweight WaveGAN
Number of BWE module
2
WaveGAN
19.69 (100%)
100%
16 kHz
PUGAN with one BWE
2.33 (11.8%)
92.4%
7.6%
PUGAN with two BWEs
1.96 (9.9%)
82.8%
8.2%
9.0%
WaveGAN PUGAN with four BWEs
46.00 (100%)
100%
44.1 kHz
2.44 (5.3%)
66.4%
6.5%
7.2%
8.6%
11.3%
Table 1: The number of parameters among the compared architectures, different configurations and by output resolution (WaveGAN, PUGAN with one BWE, PUGAN with two BWEs and PUGAN with four BWES for two sampling rates, 16 kHz and 44.1 kHz). Note that all varied configurations have significantly less number of parameters than the comparable WaveGAN. Within the PUGAN, also note that most of the parameters are subsumed in the lightweight WaveGAN module. WaveGAN was made to generate the 1.48 second long 44.1 kHz audio 4-second long 16 kHz audio.
The number of architectural parameters in WaveGAN is approximately geometrically proportional the product of the input and output channels and the needed transposed convolutional layers. Compared to the 16 kHz WaveGAN, its 44.1 kHz version would increase this complexity 2-fold, while PUGAN with equivalent output would possess only about 5% of this figure respectively. See Table. for more detailed description and comparison.
The number of parameters of bandwidth extension module does not change except for the fully connected layer. Thus, the number of parameters in the entire model increases in proportion to the number modules. Also, the bandwidth extension module itself has fewer parameters compared to equivalent WaveGAN based generator layer. When we measure the ratio of the number of param- eters of each module in PUGAN, bandwidth extension module is highly efficient because the lightweight WaveGAN module accounts for up to 90% of the total.
4.3 DISCRIMINATOR
As we focus improving the generator performance for real-time application, opt to use the same discriminator architecture of WaveGAN with the aforementioned phase shuffle module. As in- dicated in our data characteristics observation in Section 3 and demonstrated in the work of SincNet (Ravanelli & Bengio 2018),, we added a sinc convolutional layer in the discriminator to help the module learn and discover more meaningful and effective features. On every layer in the discrimina- tor, we also added the spectral normalization (Miyato et al., (018)) which is a well-known technique stabilize the discriminator by restricting the Lipschitz constant. To demonstrate the significance of both modules, we created Improved WaveGAN that uses those modules in the discriminator and compared the performance with other models.
There are separate discriminators for each generator module (outputting intermediate and final audio different sample rates), namely the lightweight WaveGAN module and BWE's.
EXPERIMENT
We introduce dataset and training process, and demonstrate how to evaluate PUGAN in terms of both quantitative computational performance and qualitative metries including the human perceptual evaluation.
5.1 DATASET
The Speech Commands (Warden 2018) dataset a collection of voice command recordings from various speakers. The length of each audio sample is under one second, containing only one word. Similarly to WaveGAN and TiFGAN, we used a subset of the dataset, sounds of ten-digit commands (i.e., "zero", "one". etc.). The training does not involve any phonology information as evaluation only concerns the generation as sound effects. The sampling rate the data 16 kHz, and the number of total training samples is about 18,000. We reproduced the under-sampled data from the

---

Under review as a conference paper at ICLR 2020
original needed by the progressive training in PUGAN. As the undersampling method can affect the structure of waveform, we used the LibROSA library (McFee et al. 2013) to resample the audio and applied the kaiser-best method.
5.2 TRAINING
The WGAN-GP and Adam optimizer (Kingma & Ba 2015)
used the loss function and optimization algorithm in our model training. As our model is progressive, and likewise the training process; the output resolution doubles and generators previous modules continues to update their parameters. However, the discriminators corresponding each generator stop learning once the resolution level is increased. We will cover the training details in the Appendix.
learned. Also, the learning of the
5.3 INCEPTION SCORE
The Inception Score (IS) (Salimans et al. 2016) is a well-known metric for assessing the quality of the data generated from GAN. It utilizes the classification model and computes the KL divergence of classification probabilities between ground truth and generated data. We evaluated PUGAN com- paratively, based on IS, to the baseline model (WaveGAN) by employing the pretrained network from the official WaveGAN repository?
5.4 HUMAN EVALUATION
We created 18 18,000 audio samples (16 kHz) of real data, PUGAN (with BWE), WaveGAN and improved (by the authors) WaveGAN for comparative human evaluation. The data were labeled (i.c., which digit) by using the pretrained WaveGAN. Three hundred samples per class in the order of the prediction probabilities from the pretrained classifier were chosen to be presented to the 14 human subjects. In the first session, pair-wise comparison tests conducted for the six combinations; the subject was to choose the one perceived as having better subjective quality. In the second session, 80 audio samples selected in a balanced fashion from the four data categories (ground truth, PUGAN, WaveGAN, improved WaveGAN) were presented to the subjects who were asked to identify the class labels. The task accuracy was recorded along with subjective quality ratings.
RESULTS AND DISCUSSION
In this section, we discuss the quantitative and qualitative evaluation and the results show the supe- riority of our proposed model compared to the existing models.
6.1 IS AND HUMAN EVALUATION
Table. 2 shows the results of the qualitative and quantitative evaluation. We compared the IS and other subjective quality metrics among the original WaveGAN, an improved WaveGAN (which contains spectral normalization and sine convolution in the discriminator) and the varied configura- tions PUGAN. IS was measured using the pretrained WaveGAN model from the official repository. Human evaluation includes the accuracy of identifying the correct label and subjective rating of the sound quality in the scale of 5.
Compared to the original WaveGAN, the improved one with spectral normalization and sine con- volution exhibited an increase in the IS, and likewise for the accuracy and subjective quality. This implies for the positive effect of the use of sine convolution, which is also used equally in the lightweight WaveGAN module of PUGAN. We varied the PUGAN in terms of the sampling rate of the intermediate low-resolution data produced by the lightweight WaveGAN and subsequently the number of required BWE modules to finally produce 16 kHz 44.1 kHz output, namely, BWE- and BWE-2. Regardless of the variation, the table shows that PUGAN resulted in the higher IS than WaveGAN, A trend of increasing IS for less number of BWE modules observed, and this seems to attributed to the trade-off between the number of architectural parameters and output

---

Under review as a conference paper at ICLR 2020
Architecture
Inception score
Accuracy
Quality
Real (Train)
9.18 #0.04
0.98
4.8 #0.7
Real (Test)
7.98 #0.20
WaveGAN
2.65 +0.03
0.63
2.6
+ specnorm
3.11 +:003
sine conv
3,41 #0.04
0.52
2.7 +1.5
PUGAN with one BWE
4.63 +0.05
0.76
3.4 +1.2
PUGAN with two BWEs
3,94 10.04
PUGAN with three BWEs
3,80
+0.08
Table 2: Result of qualitative and quantitative evaluation. We compared the IS and other subjective quality metrics among the original WaveGAN, an improved WaveGAN (which contains spectral normalization and sinc convolution in the discriminator) and the varied configurations of PUGAN. IS was measured using the pretrained WaveGAN model from the official repository. Human evaluation includes the accuracy of identifying the correct label and subjective rating of the sound quality in the scale of 5.
vs. Improved WaveGAN
PUGAN with one BWE
Architecture
wins
vs. Real
vs. WaveGAN
VS.
Real
793
95%
93%
91%
WaveGAN
210
5%
44%
24%
Improved WaveGAN
274
7%
56%
40%
PUGAN with one BWE
403
9%
76%
60%
Table Number of wins on the pair-wise comparison among real data, original WaveGAN, the im- proved WaveGAN (which contains spectral normalization and sinc convolution in the discriminator) and PUGAN resulting highest inception score.
quality. Further research will be needed to find the right balance the division of labor between the lightweight WaveGAN and the BWE modules in the regard.
The subjective human evaluation highest IS), and all aspects, PUGAN showed better subjective ratings (accuracy and subjective quality). Table. [3] shows the pair-wise comparison. PUGAN also greatly improved the number of wins against the WaveGANs. Against the ground truth data, PUGAN improved the number of wins up to 9% compared to the 5% mark by WaveGAN. However, in absolute scale, it also implies that there remains much room (and future research) for further improvement in the output quality. In the same vein, several participants reported in the post- best-briefing that certain sounds (e.g., "six") were heard clearer that others, and sounds seemingly juxtaposed with the few (e.g., "four" and "one" mixed up together) were sometimes perceived. These could be artifacts from generating the samples without any conditioning (vs. using the phonetics in the context of usage for TTS as done in the WaveGAN evaluation).
carried out using PUGAN with BWE (who scored the
6.2 COMPUTATION COST
One of the primary concerns in our work is whether the proposed model can properly synthesize the sound samples in real-time. The maximum temporal threshold around 7- 7-10msec upon the sound making the event; under this threshold, it is known humans would recognize the sound-making event and the resultant sound as separate events. We compared the computational cost (time) among the various configurations of our model and WaveGAN. The comparison was made for the synthesis of (1) one second long kHz sampling rate audio samples and (2) 1.48 seconds long 44.1 kHz sampling rate audio samples. The execution time measurements were repeated 100 times and the average was taken. The computing hardware used for the comparison was a workstation with the Intel Core 7920X CPU (with 32GB RAM), and using the NVidia TITAN RTX GPU.
For the 16 kHz audio, PUGAN could generate 100 samples (in a batch) in 0.56 sec. using just the CPU and 0.005 sec. when the GPU was used. The WaveGAN took 1.19 sec. and 0.06 sec. respec- tively. For the 44.1 kHz audio samples, PUGAN took 2.47 sec. and 0.006 sec., while WaveGAN required 4.4 sec. and 0.18 sec. For 44.1 kHz audio, 100 samples in 2.47 sec. 0.006 sec. translates
8

---

Under review as a conference paper at ICLR 2020
to 24.7 msec 0.06 msec of lateney for short sound effects (e.g. "bang", "clunk"), barely suffi- cient for real time usage. On the other hand, WaveGAN, twice as slower, will not meet the real time requirement for interactive high resolution (44.1 kHz) audio generation.
6.3 CONCLUSIONS AND FUTURE WORK
In this paper, we proposed a novel GAN model based on a stacked encoder-decoder architecture for high-quality real-time audio generation from scratch. Inspired by signal processing literature, the key to success lies in our neural upsampling layer and sinc convolution, allowing, for the first time, the progressive growth of high frequency audio signals with a significantly lightweight architecture, compared to existing sate-of-the-ar methods such as WaveGAN.
As future work, we plan to improve a discriminator architecture in progressive manner so that we can properly generate realistic audio signals in longer time span. On the other hand, we will also explore the applicability of our proposed neural upsampling layer to image generation models in computer vision domains.
REFERENCES
Marcelo Araya-Salas and Grace Smith-Vidaurre. warbler: An package to streamline analysis of animal acoustic signals. Methods Ecology and Evolution, 8(2):184-19)) 2017.
Jean-Pierre Briot, Gaëtan Hadjeres, and François Pachet. Deep learning techniques for music generation-a survey, arXiv preprint arXiv:1709.01620 2017.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to tosse-ttoiie to-image translation. In Proc. the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 9789-88777, 2018.
Perry R Cook. Real sound synthesis for interactive applications. AK Peters/CRC Press, 2002.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In Proc. the International Conference on Learning Representations (ICLR), 2019.
Hao-Wer Dong, Wen-Yr Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Associa- tion of the Advanced of Artificial Intelligence (AAAI), 2018.
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. In Proc. the International Conference on Learning Representations (ICLR), 2019.
Sefik Emre Eskimez and Kazuhito Koishida. Speech super resolution generative adversarial net- work. In Proc. the IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 3717-3721, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Proc. the Advances in Neural Information Processing Systems (NeurIPS), pp. 2672-2680, 2014.
Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing, 32(2):236-243, 1984.
Toni Heittola, Anssi Klapuri, and Tuomas Virtanen. Musical instrument recognition in polyphonic audio using fourte-fllter model for sound separation. In ISMIR, pp. 327-332, 2009.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew Dai, Matthew Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. 2019.
Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Stargan-ve: Non-parallel many-to-mane voice conversion using star generative adversarial networks. In Proc. the IEEE Spoken Language Technology Workshop (SI.T), pp. 266-273, 2018.

---

Under review as a conference paper at ICLR 2020
Takuhiro Kaneko and Hirokazu Kameoka. Cyclegan-vee Non-parallel voice conversion using cycle- consistent adversarial networks. In Proc. the European Signal Processing Conference (EU- SIPCO), pp. 2100-2104, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In Proc. the International Conference on Learning Repre- sentations (ICLR), 2018
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4401 -410, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.
Richard L Klevans and Robert D Rodman. Voice recognition. Artech House, Inc., 1997.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image resolttcso using generative adversarial network. In Proc. the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4681-4690, 2017.
Xinyu Li, Venkata Chebiyyam, and Katrin Kirchhoff. Speech audio super-resolution for speech recognition. 2019.
Andrés Marafioti, Nathanaël Perraudin, Nicki Holighaus, and Piotr Majdak. Adversarial genera- tion of time-frequency features with application in audio synthesis. In Proc. the International Conference on Machine Learning (ICML), pp. 4352-4362, 2019.
Yoshiki Masuyama, Kohei Yatabe, Yuma Koizumi, Yasuhiro Oikawa, and Noboru Harada. Deep griffin-lim iteration. In Proc. the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 61-65, 2019.
Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis python. 2015.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv:1411.1788 2014.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proc. the International Conference on Learning Repre- sentations (ICLR), 2018.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukeuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 2016.
Alan V Oppenheim. Discrete-time signal processing. Pearson Education India, 1999.
Santiago Pascual, Antonio Bonafonte, and Joan Serrà. Segan: Speech enhancement generative ad- versarial network. Proc. the Interspeech, pp. 3642-3646, 2017.
Nathanaël Perraudin, Peter Balazs, and Peter L Søndergaard. fast griffin-lim algorithm. In JEEE Workshop Applications of Signal Processing to Audio and Acoustics, pp. 1-4, 2013.
Wei Ping, Kainan Peng. Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In Proc. the Inter- national Conference on Learning Representations (ICLR), pp. 214-217, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proc. the International Conference on Learning Representations (ICLR), 2016.
Nikunj Raghuvanshi, Christian Lauterbach, Anish Chandak, Dinesh Manocha, and Ming C Lin. Real-time sound synthesis and propagation for games. Communications of the ACM, 50, 2016.
10

---

Under review as a conference paper at ICLR 2020
Mirco Ravanelli and Yoshua Bengio. Interpretable convolutional filters with sinenet. arXiv preprint arXiv:1811.09725, 2018.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Proc. the Advances in Neural Information Processing Systems (NeurlPS), pp. 2234-2242, 2016.
Kou Tanaka, Takuhiro Kaneko, Nobukatsu Hojo, and Hirokazu Kameoka. Wavecyclegan: Synthetic- to-natural speech waveform conversion using cycle-consistent adversarial networks. arXiv preprint arXiv:1809. 10008888891088, 2018.
Willem-Pier Vellinga and Robert Planqué. The xeno-canto collection and its relation to sound recog- nition and classification. CLEF (Working Notes), 2015.
Yuxuan Wang. RJ Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. Proc. the Interspeech, pp. 4006-4010, 2017.
Pete Warden. Speech commands: A dataset for limited-vocabulan speech recognition. arXiv preprint arXiv:1804.032000 2018.
David Wessel and Matthew Wright. Problems and prospects for intimate musical control of com- puters. Computer music journal, 26(3):11-22 2002.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative ad- versarial networks. In Proc. the IEEE International Conference on Computer Vision (ICCV), pp. 5907-5915, 2017.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In Proc. the IEEE International Conference on Computer Vision (ICCV), pp. 2223-2232, 2017.
A APPENDIX
A.1 ARCHITECTURE DETAILS
We implemented WaveGAN using Py Poroh and modifed to lightweight wavegan generator, In Ta- ble ??, is the batch size, and increase in proportion sampling of output audio and d is the the number of channel. We attached sine convolution to WaveGAN discriminator with no changes.
Operation
Kernel Size
Output Shape
Input ~ U(-1,1)
(n, 128)
Dense
(128, 256d)
(n, 25,d)
Reshape
(n, 8d)
ReLU
(n, 1n,32,84) 8d)
Trans Conv ID (Strde=4)
(25, 8d, 4d)
(n, 128, 4d)
ReLU
(n, 128, 4d)
Trans Conv ID (Strde=4)
(25, 4d, 2d)
(n,512, 2d)
ReLU
(n,512,
Trans Conv ID (Strde=4)
(25, 2d, d)
(n,2048, d)
ReLU
(n, (n,2048,44
Trans Conv D (Strde=4)
(25, d,
(n,
Tanh
(n, 8192,12,, d)
Conv LD (Stride=1)
(256, 1,
(n, (n,8192,
Table 4: Lightweight WaveGAN outputting kHz sampling rate audio architecture, Scale d = 32.
11

---

Under review as a conference paper at ICLR 2020
Operation
Kernel Size
Output Shape
Input audio
(n,1,1)
Zero Insertion
(n, 20, 1)
Conv (Stride 1)
(15, 1, (25, 1,
(n,
Conv (Stirde = 4)
(n,12.d)
LReLU (a 0.2)
(n,1//2,d)
Cony ID (Stirde = 4)
(25, d, 2d)
(n,1//8,2d)
LReLU (a 0.2)
(n,1//8,2d)
Conv ID (Stirde = 4)
(25, 2d, 4d)
(n.1//32,4d)
LReLU (a 0.2)
(n.1//32,4d)
Noise Injection
(4d, 4d + 1)
(n.1//32,4d+
Trans Conv ID (Stride = 4)
(25, 4d 1, 1,2d)
(n,1//8,2d)
LReLU (a 0.2)
(n,1//8,2d)
Concat
(n,1//8,4d)
Trans Conv (Stride 4)
(25,
(n,1//2,d)
LReLU (a 0.2)
(n,1//2,d)
Concat
(n,1//2,2d)
Trans Conv1D (Stride = 4)
(25, 2d, 1)
(n, 11,
Tanh
(n, 21,
Conv ID (Stride=1)
(m, 1, 1)
(n, 8192, d)
Table 5: Bandwidth extension module architecture.
A.2 TRAINING DETAILS AND HYPERPARAMETERS
We considered other alternatives in the module architectures. If there were no skip-connections in the bandwidth extension module, the model could not generate valid waveform. Instance Normalization interrupted the training of the model. When we did not inject random noise intermediate of bandwidth extension module, the output quality was worse than the default architecture. Those models, that do not contain neural upsampling layer or were not trained progressively, fail to generate recognizable sounds.
Name
Value
Channel size (d) (lightweight WaveGAN)
Channel size (d) (bandwidth extension module)
16
Batch size
Optimizer
Adam (a lc-4, 31 31 0.0, 81 0.9) WGAN-GP
Loss
WGAN-GP À D updates per module
10
Table 6: Hyperparameters of PUGAN.
A.3 ADDITIONAL RESULTS
To show superior, trained PUGAN bird dataset, which has 44.1 kHz sampling rate audios
(Vellinga & Planqué 2015). We attached more BWEs to our model to generate 65,536 length of audio that means 1.48 sec in 44.1 kHz sampling rate. We downloaded by using the library warbleR AAayy--Sals & Smith- Vidaurre (017). and we searched singing sounds of robin genus from we picked the instances which have 44.1 kHz sampling rate and quality A. The number of sounds is 23,920. See the demo site (https//pugan-icl--ee) haw.aaaaidddddmmmmeeeuuuu...pooo herokuapp.com/)
A.4 ADDITIONAL FIGURES
12

---

Under review as a conference paper at ICLR 2020
1.5
Nearest neighbor (8kHz)
1.0
Original
Amplitude
0.75
Linear interpolation (8kHz)
Amplitude
0.8
Kaiser resampling (8kHz)
0.6
Original (4kHz)
0.4
-0.75
0.2
-1.5
0.08
0.081
0.082
0.083
0.084
0.085
0.5
1.0
1.5
2.0
Time (sec)
Frequency (kHz)
1.0
Nearest neighbor
Linear interpolation
Ideal
Amp Ampituude itude
0.8
0.6
0.4
0.2
1
3 Frequency (kHz)
0
Frequency (kHz)
Frequency (kHz)
Figure 3: Original signal composed three frequency components at 60, 75 and 90Hz - time domain plot (top left) and frequency domain (top right) plots. The plots below show the upsampling results using the nearest neighbor, linear interpolation and Kaiser resampling methods. The examples illus- trate the occurrences of high frequency side bands (noise) similar Fig. however, the magnitude of noise decrease. It demonstrates that upsampling method is more important in high frequency than low one.
Real
Ours
One
Two
Three
Four
Five
Real
Ours
Six
Seven
Eight
Nine
Ten
Figure 4: Samples from speech commands dataset and generated by PUGAN. Our model generate the same shape of formant for each voice data.
13

---

Under review as a conference paper at ICLR 2020
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
1st
2nd
3rd
Figure 5; Each sample set shows the results generated by the generator for each step.
14


Reviews:
[{"review_id": "rygWVYtxcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach based on generative adversarial models for the unconditional  generation of audio. The authors take inspiration from WaveGAN, to which they add more sophisticated upsampling blocks (called the bandwidth extension module) instead of transposed convolutions. They also propose to add a sinc convolution layer to the discriminators to improve training. Finally, they propose a progressive training scheme similar in spirit to the progressive training of GANs in images. Experiments are performed on generating audio pronunciation of digits, and the authors compare their work in terms of inception score, human evaluation and computation cost to WaveGAN.\n\nAs stated by the authors, one of the main motivation oof their approach is the reduction of computation cost. The motivation, expressed in terms of the 10ms \"interactive rate\" follows a good story. The measurements performed at the end of the paper show a ~ x2 performance gains compared to WaveGAN on CPU and about same running times on GPU, which is significant but not compelling.\n\nOverall, I liked the story of the paper, but the paper lacks clarity and details. An important aspect of the paper is the progressive training, which is detailed nowhere (e.g., what is the stopping criterion to get to the next stage), should there be a special initialization of the last block of a new stage, etc.). The \"Bandwidth extension module\", which from my understanding is one of thee main contribution of the paper, is detailed in the appendix and comes essentially without justification. One of the main motivations of the paper is to be able to generate 44kHz audio, but the only results available at this resolution are inception scores that are below those of the 16kHz generation, which leaves open the question of whether the goal is effectively achieved.\n\nI found the different versions of PUGAN difficult to read. The picture uses 2 blocks of bandwidth extension to generate 16kHz, whereas the evaluations are done with PUGAN-1 (1 block), which if I understand correctly is based on the lightweight WaveGAN that generates at 8kHz (whereas Section 4.1 suggests that 16kHz is generated from 4kHz generations by WaveGAN and two blocks). Also, the fact that evaluations are carried out with PUGAN-1 suggests that the progressive training does not really works well past a single block.\n\nIn the text it is suggested that spectral normalization and sinc conv on the discriminator is an \"improvement\" of WaveGAN, and an independent contribution of the article. While Table 2 clearly shows an improvement in terms of inception score, the human evaluation is not that clear: the accuracy of human labelers drops to 0.52 from 0.63 and the quality seems totally within the variance (2.7 +/- 1.5 vs 2.6 +/- 1.3). While Table 3 also shows clear improvement over the basic WaveGAN with the changes made by the authors (in particular in terms of win ratio vs PUGAN-1), the loss of accuracy should be discussed.\n\nThe performance obtained by PUGAN-1 is nonetheless noticeable -- +1 quality score over WaveGAN, and much better pairwise win ratios. Nonetheless, the paper lacks clarity and motivation for the exact form of the bandwidth extraction module, and does not fulfill its promises (importance of progressive training, high quality generation at 44kHz), so I am leaning towards rejection.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lWhT9AtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors detail PUGAN, architectural changes to models for raw waveform generation with GANs. They do a good job of motivating the challenge of raw audio generation with GANs and of methods for progressive training. PUGAN incorporates U-Net modules in the generator (\"Bandwidth expansion\"), sinc convolution as bandlimiting inputs to the generators, and the \"style gan\" type method of adding the noise at each level of the generator. Using listener studies and inception score, they show modest improvements over the state of the art (at time of submission), WaveGAN. Notably, their architecture is also more computation and parameter efficient. \n\nThe paper is well motivated and experiments are correct, but the quality improvements overall are a little underwhelming. To some extent, that can't be helped, and the authors wisely focus on the improvements in inference time as one of their central claims, and indeed produce evidence to support this claim. \n\nMore problematic, however, the paper motivates the problem of multi-scale generation of waveforms, but does not clearly show that the proposed architectures address those issues. The motivation in terms of interpolation artifacts and band-limited upsampling in Figure 1 give a misleading sense that Kaiser resampling is explicitly incorporated into the model. The authors argue that the U-Net layers implicitly learn an upsampling method, but the lack of model comparison / ablation makes it difficult to see if that really is the case. It would help support the claim to show samples from the model at different resolutions and demonstrate the lack of artifacts at each level. In the appendix, the authors mention that any alterations to the architecture resulted in failed training, but the lack of an ablation study makes it hard to know the relative value of each component. For example, it is unclear how important the sinc layers and bandlimiting are for the discriminators, the intermediate noise, and the use of the BWE architectures.\n\nDespite these shortcomings I still recommend a weak accept, as the problem is difficult and the paper documents a well-motivated avenue for approaching it.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BygVwssiYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work is a follow-up of WaveGAN. It uses the first few layers of the original WaveGAN to synthesize low resolution waveform (4kHz), and applies several bandwidth extension modules to progressively output the higher resolution raw audios.\n\npros:\n- The proposed PUGAN has significantly smaller number of parameters than WaveGAN (e.g., 20x smaller).\n\ncons:\n- WaveGAN was a preliminary and encouraging trial for raw audio synthesis with GAN. Note that, its audio fidelity is far away from the state-of-the-art results and it was only tested on simple dataset (sounds of ten-digit commands). In contrast, the state-of-the-art autoregressive models (e.g., WaveNet) and parallel flow-based models (e.g., Parallel WaveNet) have been tested on challenging high-fidelity speech synthesis. As a result, one may focus on improving the audio fidelity of GAN on more challenging tasks. However, the proposed PUGAN was still tested on very simple dataset (sounds of ten-digit commands), and the quality of generated samples are only comparable to WaveGAN. \n\nDetailed comment:\n\n-- The attached samples are pretty noisy (e.g., noticeable artifacts on posted spectrograms). One may introduce the feature matching (e.g., STFT loss in ClariNet) as an auxiliary loss to improve the audio fidelity. \n\n-- Did the authors try conditional generation, e.g., conditioned on the digit label? The posted failure cases and some samples tend to have overlapped sounds from different digits. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Inspired by WaveGAN, this paper proposes a PUGAN to synthesizes high-quality audio in a raw waveform. The paper is well motivated. But all the reviewers find that the paper is lack of clarity and details, and there are some problems in the experiments.
Decision: Reject