Chunked Autoregressive GAN for Conditional Waveform Synthesis
Authors: Max Morrison
Rithesh Kumar
Kundan Kumar
Prem Seetharaman
Aaron Courville
Yoshua Bengio
Venue: ICLR 2022 Poster (2022)
Keywords: audio generation
speech synthesis
deep learning
generative models
autoregression
generative adversarial networks

Abstract: Conditional waveform synthesis models learn a distribution of audio waveforms given conditioning such as text, mel-spectrograms, or MIDI. These systems employ deep generative models that model the waveform via either sequential (autoregressive) or parallel (non-autoregressive) sampling. Generative adversarial networks (GANs) have become a common choice for non-autoregressive waveform synthesis. However, state-of-the-art GAN-based models produce artifacts when performing mel-spectrogram inversion. In this paper, we demonstrate that these artifacts correspond with an inability for the generator to learn accurate pitch and periodicity. We show that simple pitch and periodicity conditioning is insufficient for reducing this error relative to using autoregression. We discuss the inductive bias that autoregression provides for learning the relationship between instantaneous frequency and phase, and show that this inductive bias holds even when autoregressively sampling large chunks of the waveform during each forward pass. Relative to prior state-of-the-art GAN-based models, our proposed model, Chunked Autoregressive GAN (CARGAN) reduces pitch error by 40-60%, reduces training time by 58%, maintains a fast inference speed suitable for real-time or interactive applications, and maintains or improves subjective quality.

Published as a conference paper at ICLR 2022
CHUNKED AUTOREGRESSIVE GAN FOR CONDITIONAL WAVEFORM SYNTHESIS
Max Morrison" Northwestern University
morrimax@u. northwestern.edi
Rithesh Kumar, Kundan Kumar & Prem Seetharaman Descript, Inc.
frithesh, kundan, pzem)edescript.com
Aaron Courville Yoshua Bengio!.3 1.3
Mila, Québec Artificial Intelligence Institute, Université de Montréal
2 CIFAR Fellow
3 CIFAR Program Co-director
ABSTRACT
Conditional waveform synthesis models learn a distribution of audio waveforms given conditioning such text, mel-spectrograms. MIDI. These systems ploy deep generative models that model the waveform via either sequential (au- loregressive) parallel (non-auttoregresivve) sampling. Generative adversarial networks (GANs) have become a common choice for non-autoregressive wave- form synthesis. However, state-of-the-art GAN-based models produce artifacts when performing mel-spectrogram inversion. In this paper, we demonstrate that these artifacts correspond with an inability for the generator to learn accurate pitch and periodicity. We show that simple pitch and periodicity conditioning is insuf- ficient for reducing this error relative to using autoregression. We discuss the inductive bias that autoregression provides for learning the relationship between instantaneous frequency and phase, and show that this inductive bias holds even when autoregressively sampling large chunks of the waveform during each for- ward pass. Relative to prior state-of-th ofte-of-thheart GAN-based models, our proposed model, Chunked Autoregressive GAN (CARGAN) reduces pitch error by 40-60% reduces training time by 58%, maintains a fast generation speed suitable for real- time or interactive applications, and maintains improves subjective quality.
INTRODUCTION
Conditional audio waveform generation has seen remarkable improvements in fidelity and com- putational cost in recent years, benefiting applications such as text-to-speech (Kim et al. 202T). voice editing QQan et al., 2020, Morrison al. 2020), and controllable music generation (Dhari- wal et al.) 2020). These advancements are driven by concurrent improvements in deep generative models (Bond-Tayles Taylor et al.) 2021), which model the distribution of training data with or without aux- iliary conditioning information. Conditional audio generation models distributions of audio wave- forms with conditioning such as text (Kim et al. 2021), mel-spectrograms (Kong et al.) 2020), or MIDI (Hawthorne et al. 2018). In this paper, focus tasks such spectrogram- tootttttwwwww---ttt tovwaveform inversion, where the temporal alignment between the conditioning and the waveform is known and the alignment does not have be learned by the model (Donahue et al. 2020 Kim al. 2021).
Generative neural networks are the dominant method for audio generation tasks, strongly outper- forming comparable methods that use only digital signal processing (DSP) techniques (Griffin & Lim 1984] Morise et al. 2016). Neural methods learn the waveform distribution directly from
"This work carried out during an internship Descript Inc.

---

Published as a conference paper at ICLR 2022
data. Let {I11.... .r) be audio waveform and fci...., cu' be aligned but not necessarily same length-ccnditionee conditioning signal (e.g., a mel-sppetroggrmm. Conditional neural audio generation aims to learn the conditional distribution p(x]cc. Such models are commonly classified as either generating samples sequentially (autoregressive) or in parallel (non-autoregressivv)) We next describe autoregressive and von-autoregressive methods for audio generation, well as hybrid methods that aim to combine the benefits of both methods.
Autoregressive methods Autoregressive models parameterize p(æ]ee via the following factoriza- tion.
T p(x/c) II p( El. -1-1,, t=0
These models produce subjectively high-quality audio. However, autoregressive generation involves sequentially sampling each audio sample, where each sample requires forward pass through neural network. Thus, one major disadvantage of autoregressive models is that generation is slow.
Examples of autoregressive models include WaveNet (Oord et al. 2016)) SampleRNN (Mehri al.) 2016). WaveRNN (Kalchbrenner et al. 2018), and Jukebox (Dhariwal et al. 2020). WaveNet in- troduced the idea of autoregressive waveform generation, demonstrating high quaa-iuaitty generation of speech from aligned linguistic features (e.g., pitch and phonemes) as well as unconditioned genera- tion of speech and music. SampleRNN uses hierarchy of gated recurrent units (Cho et al.] 2014) that operate at increasing temporal resolution to reduce the computation needed to generate each sample. WaveRNN uses a smaller architecture and methods such as weight sparsification (Narang) eet al., 2017: Zhu & Gupta 2017) to improve generation speed without sacrificing quality. The sulting model can perform real-time generation, but requires a complex engineering pipeline that includes sparse matrix operations and custom GPU kernels. Jukebox uses sparse autoregressive transformer (Child et al. (2019) to generate discrete codes of a hierarchical VQVAE (Razavi et al.) 2019). Jukebox can generate coherent musical excerpts from variety of genres with rich condition- ing, but requires over eight hours on a V100 GPU to produce one minute of audio.
Non-autoregressive methods Non-autoreerressiv models parameterize p p(z/ee) directly, using meth- ods such as normalizing flows (Dinh et al. 2014), denoising diffusion probabilistic models (DDPMs) (Ho al., 2020), source-filter models (Fant 1970). and generative adversarial networks (GANs) (Goodfellow et al.] 2014). Exemplar models for conditional audio generation include the flow-based WaveGlow (Prenger et al.) 2019). the DDPM-based WaveGrad (Chen et al. 2020). the Neural Source-Filter (NSF) model (Wang et al. 2019), and the GAN- HiFi-GAN (Kong et al.)
2020 WaveGlow composed of invertible operations with simple Jacobians such as 1x1 convo- lutions and affine transformations (Kingma & Dhariwal] 2018) to model a sequence of functions of random variables drawn from increasingly complex distributions. While generation with WaveGlow is fast, it requires substantial training resources (c.g.. week of training on 8 V V100s) well as mem- ory resources due to the high parameter count. WaveGrad models the gradient of the log probability density (i.e., the score function) via score matching (Hyvärinen & Dayan
2005) and draws samples
from the score-basee based model using Langevin dynamics (Song & Ermon 2019). Samples provided
by the authors of WaveGrad indicate that the model does not accurately reproduce high-frequency content. NSF creates a deterministic waveform with the correct frequency and uses a neural network to transform this waveform into speech. Source-filter models require accurate pitch estimation and are restricted to monophonic speech signals. HiFi-GAN uses a convolutional generator to generate audio and multiple discriminators that evaluate the generated waveform at various resolutions and strides. HiFi-GAN permits fast generation on CPU and GPU, but requires two weeks of training on two V100 GPUs and regularly produces artifacts, which we discuss in Section 2
Hybrid methods Hybrid methods combine the strengths of both autoregressive and non- autoregressive models by generating more than one sample on each forward pass. Existing hybrid methods for waveform synthesis parameterize p(z(c) by factoring into chunks of adjacent audio samples that are autoregressively generated.
p(zlc)
II
p( ......,.................... 1t-k-1,))
(=(A:2R....??)
Subscale WaveRNN (Kalchbrenner et al., 2018 is one such model, generating 16 samples at a time. WaveFlow (Ping et al.] 2020) and Wave-Tacotron (Weiss al. 2021) are both flow-based
2

---

Published as a conference paper at ICLR 2022
hybrid models. WaveFlow uses autoregression to handle short-range dependencies and autoregressive architecture to model long-range correlations, while Wave-Tacotron uses autoregres- sion for long-range dependencies and nun-auuoregrresivve model for short-range dependencies. These hybrid models demonstrate a tradeoff between the high model capacity of autoregressive models and the low latency of non-autoregressiiee models.
In this work, we present Chunked Autoregressive GAN (CARGAN), hybrid GAN- model for conditional waveform synthesis. CARGAN features fast training, reduced pitch error, and equivalent or improved subjectíve quality relative to previous GAN-based models. Specifically, we make the following contributions.
We show that GAN-based models such as HiFi-GAN do not accurately preserve the pitch and periodicity of the audio signal, causing audible artifacts in the generated audio that persist across many recent works.
We demonstrate a close relationship between pitch, phase, and autoregression, and show that autoregressive models possess an inductive bias towards learning pitch and phase that is related to their ability to learn the cumulative sum operator.
We show that performing autoregression large chunks allows us to maintain the inductive bias of autoregression towards modeling pitch and phase while maintaining a fast gener- ation speed, and that the optimal chunk size is related to the causal receptive field of the generator.
We identify and document common artifacts in GAN-based waveform synthesis, providing examples and offering possible causes. We also make our code available under open- source l
PROBLEMS WITH NON-AUTOREGRESSIVE GANS
State-of-the-art GAN-based waveform synthesis models, such as HiFi-GAN (ong et al., 2020 demonstrate impressive subjective quality and generation speed, but exhibit pitch inaccuracy, audible artifacts caused by periodicity inaccuracy, and limitations induced by the objective function.
140
1.0
Original
130
0.8
HiFi-GAN
Pitch (H2)
Periodicity
CARGAN
120
0.6
110
0.4
100
0.2
90
0.0
0.5
1.0
0.0
2
Time (seconds)
Time (seconds)
Figure 1: Left An example of pitch inaceuracy where HiFi-GAN has an average error of 65.8 cents and CARGAN obtains an етror of 15.1 cents. Right An example of periodicity inaccuracy where HiFi-GAN has an RMSE of 0.19 and CARGAN obtains an RMSE of 0.04.
Pitch error artifacts We trained HiFi-GAN (V1) on the VCTK dataset (Yamagishi et al. 2019) and evaluated the pitch accuracy on 256 randomly selected sentences from a validation set containing speakers seen during training. We use the pitch representation described in Appendix A to extract pitch as well as a binary voiced/unvoiced classification for each frame, which indicates whether the frame exhibits the periodic structure of pitched sound. We measure the pitch error in regions where both the original and generated speech are voiced. We measure the root-mean-square of the pitch error in cents, defined 1200 log, (v/ / for pitch values y.ú in Hz. We find an average error of than half a semi-tone A side-by-side listening comparison of examples with high pitch error and the ground truth audio confirms that current GAN models make audible intonation emrrs (see Figure l}. Listening examples available on the companion website
51.2 cents-moot
'Code is available at htt.hi/wwttttape..t.t.....//pppa
Audio examples are available tttppa.//mmmoorrisonooooo

---

Published as a conference paper at ICLR 2022
Periodicity artifacts High pitch error indicates inaccurate reconstruction, but is often not percepti- ble in isolation as speech with slightly different pitch may still sound natural. We manually collect examples from HiFi-GAN that contain audible artifacts. Using our periodicity representation de- scribed in Appendix A which uses the confidence of a pretrained neural pitch estimator to measure whether frame of audio exhibits periodie structure between zero (completely aperiodic) and one (completely periodic), we find that these artifacts are commonly regions of high periodicity error (see Figure [). This indicates that existing GAN-based models make audible voiced/unvoiced er- rors that perceptually degrade the audio. On the companion website, we provide listening examples as well as examples of this artifact from multiple recent works to demonstrate its prevalence. We strongly encourage readers to listen to these examples, as addressing this artifact is central to our work.
Inherent limitations Current GAN-based models for waveform synthesis depend on feature matching loss (Larsen et al. 2016 Kumar al. 2019). This loss is computed by passing the gener- ated and corresponding real audio through the discriminators, and taking the L1 distances between all activations. Penalizing the LI distance between activations of early layers requires the generator to produce waveform that is close to ground truth after only one convolution and non-linearity. This requires the generator to know the initial phase. Otherwise, the generator can incur a large loss due phase rotation. As will discuss in Section 3 the initial phase is not known in the non-autoregressive setting.
3 RELATING AUTOREGRESSION, PITCH, AND PHASE
Our work uses autoregression address the issues presented in Section 2 We autoregression to improve the pitch accuracy of HiFi-GAN after many less successful experiments using more intuitive methods such as conditioning the generator on pitch, conditioning the discriminator on pitch, and more, as described in Appendix B
Autoregression sensible choice for addressing the issues presented in Section 2 because of the relationship between pitch and phase in deterministic, periodic signal. Consider a perfectly periodic signal with instantaneous frequency f (fi.....fr in Hz and sampling rate r. The unwrapped instantaneous phase Φ {oo, Φ1, ΦT) can be autoregressively computed as follows.
2T
de = 0
The relationship between f and Φi is therefore cumulative sum operation. Autoregression provides an inductive bias for learning an arbitrary-length cumulative sum operation. which relates the in- stantaneous frequency and instantancous phase in deterministic signals. Specifically, incorporating autoregression allows the network to learn Φ and ft from previous waveform samples, which informs the generator of the ground truth phase during training. Prior autoregressive vocoders have shown that the entropy of the distribution over audio samples learned by the model within voiced regions is low-especially relative to unvoiced regions (Jin et al. 2018). Thus, our assumption of a deterministie, periodic signal is a reasonable approximation in voiced regions.
Equation S assumes that a single sample is generated during each forward pass. However, the generator G' is capable of learning fixed-length cumulative sum of length ks we reduce the number of forward passes by a factor of k. As well, the instantaneous frequency can be estimated from the last n samples long as is sufficiently large represent one period of the frequency. This allows us to replace explicit pitch conditioning with conditioning on n previous samples.
Ø .... Øc+k G(o--n-i.... .I-1)
Given generator G, how can we determine the maximum length of cumulative sum that the gen- crator can learn? First, consider a fully-connected layer with é imput and output channels. This fully- connected layer can learn cumulative sum of length É when the weights form upper-triangular matrix of all ones. Next, consider a non-causal convolutional layer with kernel size m. Equation J strictly causal, so we use only the causal receptive field of the convolutional layer. This indicates that the maximum length learnable cumulative sum is [m + 1) By induction, a convolutional generator G is capable of learning a cumulative sum with length equal to its causal receptive field.
4

---

Published as a conference paper at ICLR 2022
Previous samples
Real speech
concat lime
E
concat features
G
concat
D
time
dell-spctrooraan am
Generated speech
Figure 2: Overview of our proposed Chunked Autoregressive GAN (CARGAN), Blue trapezoids are the learned encoder (E), generator (G), and discriminators (D). Dashed lines represent operations only performed during training. The red line and red highlighted region are the autoregressive loop, where previously generated samples are passed as autoregressive conditioning when generating the next chunk. The previous samples are always prepended when performing concatenation along the time axis.
4 CHUNKED AUTOREGRESSIVE GAN
We describe Chunked Autoregressive GAN (CARGAN), our proposed model for conditional wave- form synthesis that uses autoregression to address the issues presented in Section 2 CARGAN (Figure 2 consists of three components: (1) an autoregressive conditioning stack, which summa- rizes the previous & samples into a fixed-length vector, (2) a generator network that converts input conditioning (e.g., a mel-spectrogram) and autoregressive conditioning into a waveform, and (3) series of discriminators that provide adversarial feedback. The generator and autoregressive con- ditioning stack are jointly trained to minimize a weighted sum of a differentiable mel-spectrogram loss, the discriminator losses, and the feature matching loss described in Section 2 The discrimina- tors are trained to discern between real and generated audio in a binary classification task.
The previous l samples used for autoregressive conditioning are prepended the real and gener- ated audio evaluated by the discriminators. This allows the discriminators to evaluate the boundary between the autoregressive and generated audio. Without this step, the generator produces boundary artifacts that sound like periodic clicks. Listening examples available on companion website.
For generator and discriminator architectures, et al. 2020) as well as a modified version the generator from GAN-TTS (Binkowski et al. 2019), both of which are described in Appendix C We find that this modified GAN-TTS generator provides superior performance over the generator of HiFi-GAN regardless of whether autoregression is used.
the discriminators from HiFi-GAN (Kong
4.1 AUTOREGRESSIVE CONDITIONING STACK
The autoregressive conditioning stack converts the previous samples of the waveform into fixed- length embedding. We use simple architecture, consisting of five linear layers. We use leaky ReLUU with slope of -0.1 between layers. The hidden size of all layers is 256 and the output size is 128. The output embedding is concatenated to each frame of the spectrogram that is passed as input to the generator (i.c., the embedding is repeated over the time dimension).
We also experimented with multiple fully convolutional architectures, including convolutional ar- chitectures with more layers, a larger capacity, and residual connections (He et al. 2016). We found that the simple fulle-connected model outperforms convolutional models. We hypothesize that fully-connected layers are advantageous because they provide an inductive bias towards learn- ing the autocorrelation function, which is common operation used in DSP-based pitch estima-

---

Published as a conference paper at ICLR 2022
tion (De Cheveigné & Kawahara 2002 Rabiner & Schafer 1978) While stack of convolutional layers can learn an autocorrelation, the autocorrelation function cannot be modeled by a single con- volution with kernel size smaller than the signal length. However, autocorrelation can be learned with a single fully-connected layer with equal-sized input and output.
4.2 Loss COMPENSATION
The generator is trained using both feature matching and mel-spectrogram losses. is important that the ratio of these loss terms remains balanced. Otherwise, artifacts occur. When training with autoregressive conditioning, the feature matching loss decreases significantly relative to the non- autoregressive setting. We find that autoregression also improves other phase-dependent metrics not used as training losses, such as the L2 loss between the generated and ground-truth waveform and the squared error between the phase components of the signal, where each bin of the phase spectrum is weighted by the corresponding magnitude. This indicates that the autoregressive conditioning stack successfully providing phase information to the generator. However, with the feature matching loss reduced, the mel-spectrogram error dominates the training criteria. This produces a metallic sound in the audio, especially during breath sounds and unvoiced fricatives of speech data. For example, /f/ (e.g., "she") ends up sounding closer to the corresponding voiced fricative /3/ (e.g., "Asia"). Listening examples available on the companion website. We fix these metallic artifacts by rebalancing the losses: while HiFi-GAN originally uses weights of 2 and 45 for the feature matching and mel- specttoremo losses, respectively, we use 7 and 15.
EXPERIMENTS
The following sections describe experiments that evaluate our proposed claims. First, we will show that autoregressive models are more capable than non-autoregressive models at learning an arbitrary length cumulative sum (Section 5.I). Next, we perform spectrogram-to-wavee speech (Section 5.2). and show that CARGAN produces better pitch accuracy. better subjective quality, faster traming time, and reduced memory consumption during training. In Appendix [ we also perform spectrogram-to- tererogranttowwwveerrr inversion music data. We show that CARGAN is capable of modeling reverberant signals, exhibits greater pitch accuracy on musical data, and more capable of modeling accurate low-frequency information relative to HiFi-GAN.
inversion on
All models are trained with a batch size of 64. We use the AdamW optimizer (Loshchilov Hutter) 2017) with learning rate of 2 x 10-4 and 8 (.8, .99). We use an exponential learning rate schedule that multiplies the learning rate by .999 after each epoch. For all tables, 1 means higher is better and! means lower is better.
5.1 SYNTHETIC CUMULATIVE SUM EXPERIMENT
Here we show that CARGAN is more capable of modeling an arbitrary-length cumulative sum than HiFi-GAN. We generate synthetic dataset by replacing each audio sample in the VCTK dataset YYamagishi et al., 2019) with random sample taken from a uniform distribution between zero and one. We take the cumulative sums of these random vectors, and normalize both the random vector and cumulative sum by the sum of the random vector. The random vectors are the inputs to the model, and their cumulative sums are the ground truth training targets.
To train on this synthetic dataset, we must modify the generator architecture and training criteria. Given that the input and target sequences have the same sampling rate, we remove all upsampling operations from the generator. We reduce the number of channels in each layer by factor of four to reduce training time. The target sequences share a deterministic relationship with the input. As such, we replace the mel error, adversarial, and feature matching losses with simple LI loss.
The autoregressive conditioning of CARGAN informs the model of the current total of the cumu- lative sum. Because HiFi-GAN does not have access to this information, train HiFi-GAN using only training data where the current total is zero. This can be viewed as subtracting the current total from the training target prior non-autoregressive training.
We train each model for 100,000 steps and evaluate the LI distance on 256 held-out examples. We train CARGAN with chunk size of 2048 and use 512 previous samples for autoregressive

---

Published as a conference paper at ICLR 2022
conditioning, We perform evaluation for signal lengths of 1024, 2048, 4096, 8192, and "Full", where "Full" indicates that the entire available signal length is used. We include two additional conditions. The first is HiFi-GAN with modified GAN-TTS generator (+ GAN-TTS) demonstrate that our improvements on this synthetic task are primarily due to autoregression, and not the larger causal receptive field of the generator. The second is CARGAN with larger kernel size of 15 in all convolution layers (+ Large kernel) to show the impact of a larger causal receptive field in the autoregressive case.
Results are presented in Table T We find that CARGAN provides substantial improvement in the LI error for all lengths relative sequence length of 8192 and overfit to that sequence length, leading to reduced accuracy even for small sequence lengths. Specifically, Figure shows that the non-autoregressive model converges to the mean of the training data, whereas the CARGAN converges to the piece-wise mean for each chunk. When a larger kernel used, CARGAN has causal receptive field greater than the chunk size. This allows CARGAN to learn smooth interpolation of each chunk and model very long cumulative sums-consistent with our analysis Section 3 Note that this analysis does not include upsampling operations in the generator, which also impacts the size of the causal receptive field in tasks such as spectrogram inversion.
The non-autoregressive models were trained with
Method
Causal Receptive Field
1024
2048
4096
8192
Full
HiFi-GAN
245
.050
.042
.028
.031
.447
+ GAN-TTS
402
.052
.042
.028
.029
.447
CARGAN
402
.009
.015
.021
.025
.359
+ Large kernel
2802
.009
.013
.019
.024
.132
Table I: Results of our synthetic cumulative sum experiment. All values are L1 distances between the predicted and ground truth cumulative sums.
1024
2048
4096
8192
Full
Cumulative sum
1.00
0.10
0.10
0.10
0.15
0.75
0.10
0.05
0.05
0.05
0.50
0.25
0.00
0.00
0.00
0
500
1000
1000
2000
2000
4000
0
5000
50000
100000
Ground truth
HiFi-GAN
CARGAN
CARGAN-LK
Figure 3: Example output of our synthetic cumulative sum experiment at various lengths. CARGAN-LK is our proposed model with a larger kernel size.
5.2 SPECTROGRAM-TS WWW-CRRAAM-TTOWWEEORRRR
We perform spectrogram to- waveforrm inversion on speech. We compute the mel-spectrogram from the ground truth audio, and train models recover the audio waveform. The exact steps for comput- ing our mel-spectrogram representation are detailed in Appendix D All audio is sampled at sam- pling rate of 22050 Hz. Audio with a maximum absolute value less than .35 is peak- normalized to .35. All models are trained for 500,000 steps. HiFi-GAN was originally trained for 2.5 million steps. However, this takes about one month on one RTX A6000, prohibiting fast experimentation. In Appendix I we justify using 500,000 steps by training CARGAN and HiFi-GAN to 2.5 million steps and showing a comparable gap in pitch and periodicity errors.
We train the VCTK dataset (Yamagishi et al. 2019) and evaluate both VCTK and DAPS (Mysore 2014). For training on VCTK, we randomly select 100 speakers. We train on a random 95% of the data from these 100 speakers, using data from both microphones. For evalu- ation on DAPS, we use the segmented dataset of the first script of the clean partition available on Zenodo (Morrison et al. 2021)
We use a chunk size of 2048 samples and an autoregressive conditioning input size of 512 samples in proposed CARGAN model. In other words, the model generates 2048 samples on each forward
7

---

Published as a conference paper at ICLR 2022
VCTK Periodicity.
DAPS Periodicity
Method
Pitch..
F11
Pitch.
F1
HiFi-GAN
51.2
.113
.941
54.7
.142
.942
CARGAN
29.4
.086
.96
21.6
.107
.999
GAN--TTS
37.9
.099
.949
27.0
.117
.953
-ooss balance
33.7
.104
.943
34.1
.119
.922
Prepend
24.6
.088
.955
24.4
.108
.958
Table 2: Objective evaluation results spectrogram-to-vaveeoor inversion on VCTK and DAPS.
CARGAN
32.0%
096)
42.0%
1.302
26.0%
(.333 094)
GGA--TTS (VCTK)
CARGAN
36.8%
$20 2. 098)
(.190 #. .777)
38.4%
22.3%
( ..900.009)) 089)
Loss balance (VCTK)
CARGAN
077)
+ 070)
(.041 040)
Prepend (VCTK)
CARGAN
32.0%
4.24 0971
37.7%
(.122
30.3%
(.404 097) HiFi-GAN (VCTK)
CARGAN
44.0%
1.449-4 0 098)
24.7%
(245 # 088))
31.3%
(.306 # .091) HiFi-GAN (DAPS)
Figure 4: Subjective pairwise test results for audio quality and naturalness on VCTK and DAPS. Each row contains the percent preference between two methods (including ties) well as Bernoulli confidence intervals (CIs) for the probability of our method being better, equal to, or worse than the competing method at p- p-value of 0.05. Bolded CIs indicate statistically significant winning method. Ablations are defined in Section
pass, and passes the last 512 generated samples as autoregressive conditioning for the next forward pass. In contrast, HiFi-GAN produces 8192 samples each forward pass during training, but many more samples during generation. The optimal chunk size is selected based on the hyperparame- ter search described in Appendix F As the number of previous samples passed autoregressive conditioning increases, we notice two behaviors: (1) it lowers the minimal representable frequency that can be modeled and (2) significantly increases training time, the discriminators which dominate training time-must also process the full autoregressive conditioning. We approximate the minimum pitch of typical human speech as 50 Hz, which corresponds wavelength of 441 samples given sampling rate of 22,050 Hz. Thus, 512 is the smallest power of two capable of representing at least one cycle a 50 Hz waveform.
We perform both objective and subjective evaluation. For the objective evaluation, measure pitch error (in cents), periodicity RMSE, and voiced/unvoiced classification Fl score. We use pitch and periodicity representations described in Appendix[A] Objective evaluation is performed on 256 randomly selected held-out examples from VCTK and DAPS. For VCTK, we select examples from seen speakers not used during training. For subjective evaluation, we Amazon Mechanical Turk to perform pairwise tests between our proposed CARGAN and the baseline HiFi-GAN on both VCTK and DAPS, In each pairwise test, 15 participants listen to 20 randomly selected pairs and select which example sounds best based on speech naturalness and audio quality. Participants also given the option of indicating that both examples exhibit equal quality and naturalness. We construct pairs from 100 randomly selected examples from each dataset. For VCTK, we additionally perform three pairwise ablation tests measure the importance of our proposed methods: (1) GAN-TTS we use the generator from HiFi-GAN instead of GAN-TTS, (2) Loss balance we omit the loss compensation of Section 4.2 using the original weights of 2 and 45 for the mel-spectrogram and feature matching losses, respectively, and (3). Prepend omit prepending the autoregressive conditioning to the signals passed into the discriminator, so that the discriminator is unable to see the boundary between the autoregressive conditioning and the generated real continuation.
Results for objective evaluation of our spectrogram waveform experiments on VCTK and DAPS data presented in Table 2 Figure shows the results of the subjective listening tests on both datasets. For the objective evaluation, we find that CARGAN substantially reduces the pitch and periodicity error and increases the F1 score of voiced/unvoiced classification. This corroborates
8

---

Published as a conference paper at ICLR 2022
hypothesis presented in Section thtat autoregression provides inductive bias for learning accurate pitch and phase information. While our - Prepend ablation exhibits slightly better pitch accuracy on VCTK, subjective evaluation demonstrates that the boundary artifacts overwhelmingly degrade the audio signal. For subjective evaluation, we find that CARGAN exhibits equal quality to HiFi- GAN on VCTK, and superior quality DAPS. CARGAN also ties with GAN-TTS ablation; however, the objective evaluation indicates that CARGAN exhibits superior pitch and periodicity error, well FI, relative this ablation. We also explored increasing the causal receptive field of the generator by increasing the kernel size of all convolutional layers to 15, as in Section 5.1 We obtain a small improvement in pitch error (24.3 and 20.3 cents on VCTK and DAPS, respectively), but no improvement in periodicity error or Fl and a significant reduction in generation speed.
In Appendix G we report the DeepSpeech distances proposed in Binkowski et al. (2019). We do not find that these experimental metrics correlate well with our objective or subjective metrics. In Appendix[H] we show that there a weak correlation between subjective preference and periodicity RMSE, and a smaller correlation between subjective preference and pitch RMSE, This corroborates our hypothesis in Section 2
that intonation errors are not as perceptible as periodicity errors.
5.3 TRAINING AND GENERATION SPEED
Training
Generation
Method HiFi-GAN CARGAN
Speed (ms/step)) Memory use (GB).,
GPU (RTF)
CPU (RTF)
1186
40.5
180.8
1.21
502
12.4
10.7
0.45
Table 3: Results for time and memory benchmarking of HiFi-GAN and CARGAN, Real-time factor (RTF) is the number of seconds of audio that can be generated per second.
We benchmark HiFi-GAN and CARGAN to determine the relative speed and memory consumption when performing spectrogram-tor toveeeforrr inversion on speech data (Section 5.2p. We use single RTX A6000 for training and generation on a GPU, and two cores of an AMD EPYC 7742 with one thread per core and a 2.25 GHz maximum clock speed for CPU benchmarking.
Results of our benchmarking are presented in Table 3 We find that CARGAN reduces training time by 58% and reduces memory consumption during training by 69%. These improvements are due to the large reduction in training sequence length from 8192 in HiFi-GAN to 2048 in CARGAN. While generation is slower with CARGAN, we can easily improve generation speed the cost of reduced training speed, increased memory usage, and slightly increased pitch error by changing the chunk size (see Appendix [F). As well, the autoregressive nature of CARGAN makes it suitable for treaming-bassed applications running over a low-bandwidth network, as not all features have to be available to begin generation (i.c., the conditioning in Equation 2 is also factorized to be causal).
6 CONCLUSION
In this paper, we proposed Chunked Autoregressive GAN (CARGAN), a GAN-hased model for conditional waveform synthesis. Relative to existing methods, CARGAN demonstrates improved subjective quality and pitch accuracy, while significantly reducing training time and memory con- sumption. We show that autoregressive models permit learning an arbitrary length cumulative sum operation, which relates the instantaneous frequency and phase of a periodic waveform. We demon- strate that the benefits of autoregressive modeling can be realized while maintaining a fast generation speed by generating large chunks of audio during each forward pass, where the optimal chunk size related to the causal receptive field of the generator architecture.
Our work reveals multiple directions for future improvement. The generation speed of CARGAN largely determined by the chunk size. Designing generators with large causal receptive field may further improve generation speed by permitting generation in larger chunks. As well, while CARGAN addresses artifacts caused by pitch and periodicity errors, it induces occasional bound- ary artifacts in the signal-eve even when the discriminator is able to evaluate the boundary between autoregressive conditioning and the generated or real continuation. Future work that addresses this artifact will further improve subjective quality.
9

---

Published as a conference paper at ICLR 2022
Acknowledgments The authors would like to thank Jose Sotelo, Lucas Gestin, Vicki Anand, and Christian Schilter for valuable discussions and inputs.
Ethics statement Our work involves a human subject study, utilizes standard speech datasets, and produces a system that can be incorporated into a voice cloning system. Our human subject study conducted with care. We use standard speech datasets that are free of profane objectionable content. We also include instructions that encourage participants to set their volume levels at a reasonable level prevent hearing loss. The datasets we use are common the speech research community, but are biased toward American and British English. We note the need for clean speech datasets that contain proportional or equal representations of accents and dialects across the globe. Finally, our system could be used as one component of voice cloning system. While such systems have profound utility in podcast and film dialogue editing, they can also be used to mimic one's voice without consent. We do not condone the use of our work for non-consensual voice cloning.
Reproducibility In order facilitate reproduction of our research, we provide documented, open- source code that permits reproducing and evaluating all experiments in our paper, except for pairwise subjective evaluation on Amazon Mechanical Turk, which uses proprietary code. We also provide pretrained model, script and library for performing generation, and packaging via PyPi and PyTorch Hub.
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, al. Deep speech End-to- end speech recognition in english and mandarin. In International conference on machine learning.
pp. 173-182. PMLR, 2016.
Mikotaj Binkowskii, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. arXiv preprint arXiv: 1646, 2019.
Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G Willcocks. Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models. arXiv preprint arXiv:2103.cm 04920,9922222 2021.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009. 00713, 2020.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcchre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078 2014.
Alain De Cheveigné and Hideki Kawahara. Yin, fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America, 111(4):117-11990, 2002.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Wook Kim, Alec Radford, and llya Sutskever. Jukebox: generative model for music. arXiv preprint vrXiv:200S.00341 2020.
Laurent Dinh, David Krueger, and Yoshua Bengi.. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv: 1xXXx::440.85// 2014.
Jeff Donahue, Sander Dieleman, Mikolaj Binkowski,, Erich Elsen, and Karen Simonyan. adversarial text-to-speech. arXiv preprint arXiv:2006.035/1 2020.
Gunnar Fant. Acoustic theory of speech production. Number 2. Walter de Gruyter, 1970.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
10

---

Published as a conference paper at ICLR 2022
Daniel Griffin and Jac Lim. Signal estimation from modified short-time fourier transform. 1EEE Transactions on acoustics, speech, and signal processing, 32(2):236 243, 1984.
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng -hin Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel. and Douglas Eck. Enabling factorized piano music modeling and generation with the maestro dataset. arXiv preprini arXiv: /8/0. 10.f0.22244 /2247, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho. Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Zeyu Jin, Adam Finkelstein, Gautham J Mysore, and Jingwan Lu. Fftnet: A real-time speaker- dependent neural vocoder. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2251-2255. IEEE, 2018.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pp. 2410-2419, PMLR, 2018.
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. arXiv preprint arXiv:2106.06/03, 2021.
Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: A convolutional repre- sentation for pitch estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 161-165. IEEE, 2018
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. arXiv preprint arXiv: 1807. 000099,.99 2018
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. arXiv preprint arXiv:2010.05646, 2020.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. arXiv preprint arXiv: 2019.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. In International conference on machine learning, pp. 1558-1566. PMLR, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05100 2017.
RG McCurdy, Tentative standards for sound level meters. Electrical Engineering, 55(3):260-263, 1936.
Soroush Mehri, Kundan Kumar, Ishaan Guirajani. Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplerm:: An unconditional end-to-end neural audio generation model. arXiv preprint 161iv:1612.07837, 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 2018.

---

Published as a conference paper at ICLR 2022
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: vocoder-based high-quality speech synthesis system for real-time applications. IEICE TRANSACTIONS on Information and Systems, 99(7):1877-1884, 2016.
Max Morrison. torchcrepe
torchcrepe, 2020.
URL https::/ggthab..cc///aaaaaa/cce
Max Morrison, Zeyu Jin, Justin Salamon, Nicholas Bryan, and Gautham Mysore. Controllable neural prosody synthesis. arXiv preprint arXiv:2008. 03388, 2020.
Max Morrison, Zeyu Jin, Nicholas J. Bryan, Juan-Pablo Caceres, and Bryan Pardo.
Segmented
DAPS (Device and Produced Speech) Dataset, May 2021. URL ftttps://doi.cret/
5281/zenodb.4788
Gautham Mysore. Can we automatically transform speech recorded on common consumer devices in real-world environments into professional production quality speech?- dataset, insights, and challenges. IEEE Signal Processing Letters, 22(8): 1006-1010,, 2014.
Sharan Narang, Eric Undersander, and Gregory Diamos. Block-sparse recurrent neural networks. arXiv preprint arXiv: 2017.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 2016.
Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. Waveflow: A compact flow-based model for raw audio. In International Conference Machine Learning, pp. 7706-7716. PMLR, 2020.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621. IEEE, 2019.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David Cox. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pp. 7836-7846. PMLR, 2020.
Lawrence R Rabiner and Ronald W Schafer. Digital processing of speech signals. New Jersey, pp. 121-123, 1978.
Zafar Rafii, Antoine Liutkus, Fabian-Rober Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. mussdb18-he an uncompressed version of musdb18, August 2019. URL htwp::ttt///// //ooi. org/ 10. 5281 zenodo. 3338373
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with -.. In Advances in neural information processing systems, pp. 14866- 14876, 2019.
Tim Salimans and Durk Kingma. Weight normalization: simple reparameterization accelerate training of deep neural networks. Advances in neural information processing systems, 29:901- 909, 2016.
Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. International Conference on Learning Representations, 2020.
Mike Senior. Mixing secrets for the small studio. Taylor & Francis, 2011.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint
2019.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural fatre--flter-bbaeed waveform model for statistical parametric speech synthesis. In /CASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5916-5920. IEFE, 2019.
12

---

Published as a conference paper at ICLR 2022
Ron J Weiss, RJ Skenry- Ryan, Eric Battenberg, Soroosh Mariooryad, and Diederik P Kingma. Wave- tacotron: Spectrogram-frt free end-to-end text-to-speech synthesis. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5679-5683. IEEE, 2021.
Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. Cstr vetk corpus: English multi- speaker corpus for estr voice cloning toolkit (version 0.92). 2019.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv: 1rXiv:1710.01878, 2017.
A PITCH REPRESENTATION
We use torchcrepe (Morrison 2020 to extract pitch and periodicity features from an audio waveform. torcherepe first predicts a categorical distribution over possible pitch values using CREPE (Kim et al. 2018), a neural pitch estimator trained on music data. We set the probability of any pitch values outside the human speaking range (approximately 50-550 Hz) to zero and normalize the resulting distribution to sum to one. This produces a sequence of categorical distributions over each frame of audio. We perform Viterbi decoding on this sequence of distributions to determine the maximum likelihood path. For Viterbi transition probabilities, we use a triangular distribution that assigns zero probability to pitch jumps greater than one octave between adjacent frames. Each step along the maximum likelihood path has an index associated with pitch value as well as a probability value predicted by the model. We convert the path indices to frequencies in Hz to produce a pitch contour, and use the sequence of probability values the periodicity contour. Because CREPE is invariant to amplitude, low-bit noise during nearly silent regions can induce high periodicity. We compute the A- A-weighted loudness (McCurdy) (936) at each frame of audio, and set the periodicity to zero in frames where the loudness is less than -60 dB relative to a reference level of 20 dB.
CREPE independently predicts pitch distribution for 1024 samples of audio at sampling rate of 16000 Hz. Our audio data is sampled at 22050 kHz, and our mel spectrogram features are com- puted with hop size of 256 samples. To align pitch features with the mel spectrogram, the default padding of CREPE with the same reflection padding used for mel spectrogram com- putation (Section 5.2) and run CREPE with hop size of 256 16000 22000/22050
replace
185.8 samples. torchcrepe does not currently support a fractional hop size. We instead use hop size of 185 samples and linearly resample the target length if needed. Resampling of pitch values is performed in base-2 log-space.
Given the pitch and periodicity contour of speech signal, we can extract a binary value for each frame that indicates whether or not the frame contains a voiced phoneme. Intuitively, when the entropy of the distribution produced by CREPE is low, the signal is likely to be periodic with a frequency given by the peak of the distribution (i.e., the periodicity will be high). While one could perform this voiced/unvoiced classification by simply thresholding the periodicity at a given value, find that this induces chatter, which occurs when a signal oscillates above and below a threshold. We propose to use hysteresis thresholding, which requires the signal to remain above the threshold for certain number of frames in order be considered voiced. This hysteresis thresholding is also how chatter is reduced in audio compressors (Senior 2011).
B PITCH CONDITIONING EXPERIMENTS
Our work proposes to use autoregression to improve the pitch and periodicity error of GAN-based speech synthesis. However, autoregression is not an intuitive solution for improving pitch and pe- riodicity error compared to, e.g., conditioning on pitch features. During development, we also ex- plored number of these more intuitive methods. Here describe subset of the experiments tried. All experiments are performed using the baseline HiFi-GAN model.
Generator conditioning We condition the generator the pitch and periodicity features described in Appendix A These are concatenated to the input mel-spectrogram features as two additional channels. Pitch features are converted to base-2 log-space prior to concatenation.
13

---

Published as a conference paper at ICLR 2022
Diseriminator conditioning We condition the discriminator pitch and periodicity features. We linearly upsample both the pitch (in base-2 log-space) and periodicity features to the same temporal resolution the generated waveform, and concatenate the upsampled pitch and periodicity two additional channels. Intuitively, this allows the discriminator to determine if the pitch and periodicity of the generated speech is consistent with the ground truth pitch and periodicity features.
CREPE perceptual loss We pass the real and generated audio into the CREPE pitch estimator (Kim al. 2018) and perform feature matching between all activations, including the unnormalized log- its. The weights of CREPE are frozen during training. We compute the L1 loss between all activa- tions and add the result the generator loss of HiFi-GAN.
Pitch discriminator Inspired by the OASIS model for semantic image segmentation (Schönfeld al. 2020), design discriminator that jointly performs pitch estimation and real/fake classifi- cation. We use the architecture of CREPE for the discriminator. Our pitch-estimating discriminator predicts 256-dimensional categorical distribution, where the first 255 categories represent evenly quantized pitch values between 50 and 550 Hz in base-2 log-space, and the final category is used by the discriminator to indicate that the input audio is fake. The discriminator is trained to catego- rize all real data as its ground truth pitch and all generated data as fake. The generator is trained to produce audio that is categorized as the ground truth pitch by the pitch-estimating discriminator. These generator and discriminator losses are added to the generator and discriminator losses of the baseline HiFi-GAN.
VCTK
DAPS
Method
Pitch..
Periodicity
F1f
Pitch
Periodicity.
F11
HiFi-GAN
51.2
.113
.941
54.7
.142
.942
Generator conditioning
39.1
.108
.943
49.7
.135
.945
Diseriminator conditioning
33.9
.096
.951
43.0
.121
.952
CREPE perceptual loss
53.6
,101
.948
54.6
.137
.948
Pitch discriminator
48.9
.109
.943
53.4
.139
.943
CARGAN
29.4
.086
.956
21.6
.107
.959
Table 4: Results for objective evaluation of non-autoregressive pitch conditioning relative to the baseline HiFi-GAN and our proposed, autoregressive CARGAN.
We provide the pitch and periodicity error for these four experiments, well as the baseline HiFi- GAN, on the same 256 examples from the DAPS dataset as used for evaluation in Section 5.2 Results presented in Table These results indicate that autoregression plays important role in reducing pitch and periodicity error that cannot be fully compensated for by more intuitive methods in the non-autoregressive case.
C NETWORK ARCHITECTURE
Layer
Input channels
Output channels
Upsampling ratio
768
768
768
768
768
384
4
384
384
4
384
384
4
384
384
384
192
2
192
192
1
192
96
2
96
96
Table 5: Layer configuration for all GBlocks in modified GAN-TTS generator.
Our modified GAN-TTS generator consists of 1x convolution followed by sequence of GBlocks and an output convolution with tanh activation. All convolutional layers the generator have a ker- nel size of three and are padded maintain the same temporal dimension unless otherwise speci-
14

---

Published as a conference paper at ICLR 2022
fied. Each GBlock contains two convolutional blocks. The first block consists of a ReLU activation, optional nearest neighbors upsampling, a convolutional layer, ReLU activation, and a dilated convo- lutional layer with dilation rate of three samples. The second block consists of ReLU activation, dilated convolution with a dilation rate of nine, a second ReI.U, and a second dilated convolution with dilation rate of 27. The input the generator is passed through the first convolutional block as well as a residual path that consists only of optional upsampling and a Ixl convolution. The output of the first block and the residual connection are added and passed through the second convolutional block. A residual connection adds the input and output of the second convolutional block. Table provides the number of channels and ratio of nearest neighbors upsampling for each GBlock in our generator.
We use the same discriminators as HiFi-GAN (Kong et al. 2020). HiFi-GAN utilizes eight discrimi- nators to provide adversarial feedback to the generator. Thrre of these discriminators are multi-scale discriminators (MSDs), each of which evaluates waveforms at a different resolution: raw audio, 2x average-pooled audio, and 4x average pooled-audio. Each MSD consists of eight grouped and strided ID convolutional layers with Leaky ReLU activations. The MSD that operates on raw audio spectral normalization (Miyato et al.) 2018), while the other mans & Kingmal 2016). The other five discriminators are multi caulti-prrio discriminators (MPDs), each of which evaluates the input audio of length 7 by first reshaping into 2D matrices of dimension T/p x p for a different prime number p E [2, 3, 5, 7, 11]. Each MPD consists of six strided 2D) convolutions with weight normalization and Leaky ReLU activations.
weight normalization (Sali-
D MEL-SPECTROGRAM REPRESENTATION
To compute the mnelsppectrogram, first compute the magnitude of the STFT of the waveform. We map each frame of the magnitude spectrogram onto the mel scale using a triangular filterbank. We clamp the resulting mel energies have minimum of x 10-5 and take the base- 10 log. We use 1024 frequency channels for the STFT, with window size of 1024 samples and a hop size of 256 samples. We use 80 frequency channels for our mel spectrogram representation. Prior computing the STFT, we pad the audio using reflection padding so that the real and generated waveforms are equal length when the real audio divisible by the hop size.
E PITCH AND PERIODICITY ERROR AT 2.5 MILLION STEPS
VCTK Periodicity
DAPS Periodicity.
Method
Pitch
F11
Pitch.
F1
HiFi-GAN (0.5M)
51.2
.113
.911
54.7
.142
.942
HiFi-GAN (2.5M)
61.2
.094
.50
43.5
.124
.952
CARGAN (0.5M)
29.4
.086
.956
21.6
.107
.959
CARGAN (2.5M)
31.7
.077
.961
22.6
.090
.966
Table 6: Pitch and periodicity error for a varying number of steps
We compute the pitch and periodicity error of HiFi-GAN and CARGAN at 0.5 million and 2.5 million steps. For HiFi-GAN, training to 2.5 million steps takes about one month GPU. We use the same 256 examples from VCTK and DAPS as used for objective evaluation in Section 5.2 Results
RTX A6000
presented in Table 6 We that CARGAN outperforms HiFi-GAN at both 0.5 million and 2.5 million steps. Given the shorter training time and proportional pitch and periodicity errors, use 0.5 million steps for all other experiments.
F CHUNK SIZE ABLATION
Our proposed autoregressive model, CARGAN, generates 2048 samples during each forward pass. Here we justify using 2048 samples on each forward pass by comparing the pitch and periodicity error well training and generation speeds various chunk sizes. We train our best model using chunk sizes of 512, 1024, 2048, 4096, and 8192 and evaluate using the same 256 samples from
DAPS used for evaluation in Section 5.2 Results presented in Table 7 We find that chunk
15

---

Published as a conference paper at ICLR 2022
Objective metrics
Speed
Chunk size
Pitch..
P
F11
Training (ms/step))
GPU Generation (RTF)
512
22.9
.109
.96
269
2.43
1024
22.1
.112
.955
354
4.77
2048
21.6
.107
.999
502
10.7
4096
24.8
.111
.957
821
20.5
8192
24.1
.108
.959
1411
39.4
Table Results for objective evaluation and benchmarking of CARGAN using various chunk sizes. Real-time factor (RTF) is the number of seconds of audio that can be generated per second.
size of 2048 is optimal for pitch, periodicity and FI. As the chunk size increases, training speed decreases and generation speed increases. In informal subjective evaluation, we find that smaller chunk sizes are more likely to introduce boundary artifacts (see Section 4). while larger chunk sizes more likely to introduce periodicity artifacts discussed in Section
G DEEPSPEECH OBJECTIVE METRICS
Method
FDSD
cFDSD
KDSD
cKDSD
HiFi-GAN
4.03
.520
5 10-9
14 10
CARGAN
4.05
.778
7 10
-11 10
-GAN-TTS
4.04
.776
7 10
-11 10
-Loss balance
4.01
.554
X 10
-13 10
Prepend
4.07
.875
13 x 10
-6 10
Table 8: DeepSpeech distances on the VCTK dataset
Method
FDSD
cFDSD
KDSD
CKDSD
HiFi-GAN
3.65
.432
2 x 10
-10 x 10
CARGAN
3.69
.644
7 10-5
-6 x 10-5
Table 9: DeepSpeech distances on the DAPS dataset
We report the Fréchet DeepSpeech distance (FDSD), the Kernel DeepSpeech distance (KDSD) and their conditional variants (cFDSD and cKDSD) (Binkkowsii al. 2019) on HiFi-GAN, CARGAN, and all ablations described in Section 5.2 These are objective metrics inspired by the Fréchet Inception Distance (Heusel et al., 201 7) and are meant approximate subjective preference by comparing distances between embeddings of real and generated audio produced by the DeepSpeech speech recognition system (Amodei et al., 2016). We use the public implementation of these distances published by the original authors. This implementation requires at least 10,000 samples per condition. obtain these samples, we sample 40 random one-second chunks from each of the 256 examples from the VCTK and DAPS datasets used for objective evaluation. Results VCTK are presented in Table 3 and results on DAPS are presented in Table g Note that the public implementation reports limited precision for KDSD and cKDSD. Comparing these results to subjective evaluation in Table we find that the only statistically significant comparison that also reflected in the DeepSpeech distances is between CARGAN and the Prepend ablation. Therefore, the DeepSpeech distances do not seem to be an adequate indicator of subjective quality when the conditions are as close subjective quality and naturalness as the ones considered in this paper.
H CORRELATION BETWEEN SUBJECTIVE PREFERENCE AND PITCH AND PERIODICITY
We analyze the subjective pairwise results between HiFi-GAN and CARGAN described in Sec- tion [.2] We consider a loss (i.c., the participant thinks HiFi-GAN sounds better) to have value of tie 0.5, and win (i.e., the participant thinks CARGAN sounds better) 1. We compute
16

---

Published as a conference paper at ICLR 2022
score for each example by averaging over all losses, wins, and ties, and compute the Pearson relation between this score and the RMSE gap between HiFi-GAN and CARGAN. On VCTK, this provides an insignificant correlation of 0.065 with two-tailed p-value of 0.531. The correlation is more pronounced on DAPS, with a value of 0.257 and p-value of 0.011. The correlations for pitch RMSE are significantly less (-0.187 with p-value of 0.068 VCTK and -0.005 with p-value of 0.962 on DAPS), which corroborates our hypothesis in Section[that intonation errors are not as perceptible as periodicity errors. These correlations also indicate that the artifacts associated with periodicity errors described in Section
2 are not the only source of variability in the reconstructed signal that affect periodicity RMSE.
I APPROXIMATE SPEED COMPARISON WITH OTHER MODELS
We provide an approximate comparison of the inference speed of CARGAN relative to WaveNet, WaveGilo, MelGAN, and HiFi-GAN (see Table 1O). To compute speed values for CARGAN, we assume linear relationship in both CPU and GPU computation speed between the benchmarking conditions used in HiFi-GAN (Kong et al. 2020) and the conditions described in Section[5.3 where the slope is computed via the relative speeds of HiFi-GAN between benchmarks. This means that the values for CARGAN are not the result of properly controlled benchmarking, and should only be used as an illustrative guide.
Method
GPU (RTF)
CPU (RTF)f
HiFi-GAN
167.900
1.43
CARGAN
9.937
0.53
WaveNet (MoL)
0.003
WaveGlow
22.800
0.21
MelGAN
645.730
6.59
Table 10: Approximate time benchmarking of some recent waveform synthesizers. Real-time fac- tor (RTF) is the number of seconds of audio that can be generated per second. Values other than CARGAN are from Table in the original HiFi-GAN paper.
J MUSIC EXPERIMENTS
To show how our model can be adapted to domains other than speech, we apply our proposed model to spectrogram inversion of music. We train on random 80% of the stems of the MUSDB-HO dataset (Rafii et al. 2019). we use the same hyperparameters as we used for speech data for CAR- GAN (a chunk size of 2048 with 512 previous samples used autoregressive conditioning) we find that the resulting audio contains significant degradations. We hypothesize that this is due to the music dataset containing low frequencies, long reverberation, and polyphony. To handle lower fre- quencies and long reverberation, we increase the number of previous samples used as conditioning to 16384 samples, which greatly extends the receptive field. Increasing the size of the autoregressive conditioning decreases the feature matching loss. We increase the weight on the feature matching loss from 7 to 21 compensate. We call this model CARGAN-16K.
We provide listening examples on held-out data from MUSDB-HQ on the companion website. We also design three simple experiments evaluate the ability of HiFi-GAN, CARGAN, and CARGAN-16K to model low frequencies, reverb, and polyphony. To probe the ability of the models learn accurate now-frequency information, we pass as imput a repeated kick drum sample while gradually increasing the center frequency of high-pass filter. We repeat the kick drum sample four times. We turn off the high-pass filter on the first repetition, and thereafter use center frequencies of 30, 60, and 90 Hz. The Q-factor of the filter is fixed at 1. In Figure 5 we see that HiFi-GAN ig- nores the high-pass filter and overemphasizes low frequencies, while CARGAN and CARGAN-I6k exhibit frequency energy closer to ground truth.
To test the ability of the models on reverberant audio, we use a repeated snare drum sample with gradually increasing decay time. We repeat the snare sample five times. We turn off the reverb on the first repetition, and thereafter use decay times of 250 ms, 500 ms, second, and seconds. In Figure 6, we see that all models are capable of modeling the reverb in this simple example. How-
17

---

Published as a conference paper at ICLR 2022
HiFi-GAN and CARGAN both overemphasize the high frequencies of the transient relative to CARGAN- 16k and the ground truth audio.
To test the ability of the model to generate polyphonic audio, ing a C chord with gradually increasing number notes. In Figure 7 we see that all models are capable of generating polyphonic audio. However, they exhibit different artifacts: HiFi-GAN ex- hibits a wide vibrato that indicates low pitch accuracy, while CARGAN and CARGAN- 16k exhibit boundary artifacts that appear as repeated clicks. Audio for all experiments in this section can be found on the companion website.
MIDI piano instrument play-
Original
100
50
HIR-GAN
100
50
CARGAN
100
50
CARGAN-166
Time (seconds)
Figure 5: Spectrogram visualizations for low frequency modeling experiment repeated kick drum sample with a gradually increasing center frequency on a high-pass filter. Comparing the lowest frequencies of each transient shows that CARGAN-16K is most capable of modeling the effect of this high-pass filter.
18

---

Published as a conference paper at ICLR 2022
6192
4096
Original
2048
1024
HIR-GAN
4096
2048
1024
512
1.5
8192
CARGAN
4096
1024
512
0
1.5
4.5
7.5
6192
CARGAN-161
4096
2048
1024
0-
Time (seconds)
Figure 6: Spectrogram visualizations for our reverb modeling experiment on a repeated snare sample with gradually increasing reverb decay time. All models are capable of reproducing reverb this simple case. Comparing the high-frequencies of each transient shows that CARGAN-16K is most capable of accurately modeling the rrquuereqesssss.
8192
Orgiinal
4096
2048
512
0-
4.5
7.5
HIR-GAU
4096
2048
1024
512
1.5
8192
CARGAN
4096
2048
512
0
4.5
7.5
10
8192
CARGAN-161
4096
2048
1024
512
7.5
Time (seconds)
Figure 7: Spectrogram visualizations for our polyphonic modeling experiment on MIDI piano instrument with gradually increasing number of notes. HiFi-GAN exhibits strong vibrato that is easier to hear than see on a spectrogram. CARGAN and CARGAN- exhibit periodic boundary artifacts which are clearly visible in the spectrogram.
19


Reviews:
[{"review_id": "tBYcqV6tqtN", "reviewer": "Reviewer_agkP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- the pitch motivation is well presented and natural.\n- the solution proposed is a good trade-off between accuracy and speed, and is straightforward to put in place.\n- the authors provide extensive evaluations, including MOS.\n- the authors provide extensive examples to highlight different types of artifacts\n\nWeaknesses:\n- Some remarks (see after) should be addressed.\n- a single baseline (HiFi GAN), while their exist a considerable number of vocoders (auto-regressive, flow etc).\n- the phase prediction issue is intuitive and could be more compactly treated.\n\n\nQuestions and Remarks:\n- In [1], the authors condition on a melody synthetize from the pitch (i.e. a cosine waveform with consistent phase and the right pitch). This means the generator can use the phase from the waveform as a base for its generation. This is much more efficient than giving the pitch directly for the reasons the authors state. This is related to your work, and it could be interesting to see how HiFi-GAN performs with that simple fix. Of course the advantage of the method proposed here is that there is no need to extract pitch information for generation.\n- for the toy task, the results for HiFi GAN should be the same pattern repeated as the input to the model is constant and so each output frame should be the same no?\n- can you comment and add speed comparison with other methods, in particular waveglow, wavernn etc.\n- do the authors know if Parallel WaveNet would suffer from pitch artifacts ? because it is no longer autoregressive and only conditioned on random noise. intuitively one might think that this would be problematic. Same question for flow based methods like waveglow.\n\nThere a number of typos:\n- p2. 'deniosing'\n- p2 'hybird'\n- p 4: 'Autoregression is an sensible' -> 'a sensible'\n\n\n[1]: Unsupervised Cross-Domain Singing Voice Conversion, Polyak et al 2020.", "summary_of_the_paper": "The authors present a chunk-wise auto-regressive generative model for audio with adversarial loss.\nIn particular, the authors note the limitations of purely convolutional adversarial audio generation model for text to speech as those fail to provide a consistent pitch for an extended duration. The authors provide a number of audio samples to illustrate the issue.\nThey evaluate the proposed method on standard datasets (VCTK) and perform subjective evaluation. they show that their method achieve better pitch correctness, and can improve perceptual evaluations, in some cases.\nThey also evaluate the speed of evaluation and training of their method. Unsurprisingly, the method proposed is slower than non auto-regressive generation but the chunking makes it theoretically faster than purely auto-regressive methods.\n", "summary_of_the_review": "Simple method suggested by the author to fix some known artifacts from GAN audio vocoders.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Z1rdDLpXQOu", "reviewer": "Reviewer_Zbrk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a novel conditional waveform generative model, CARGAN, combining the advantages of AR and non-AR models. \nFurthermore, it not only improves the performance of the CWS, but it also conducts the side-by-side listening comparison and pitch analysis, which makes it easier for readers to understand what is the problem of the previous non-AR CWS model. Also, I think that the consideration about choosing the optimum chunk size depending on the size of the receptive field also can give good insights to future works.\n\nHowever, I have several concerns about this work.  \n\nFirst, I am not sure that the comparison of the ability to learn the cumulative sum operator is enough to explain the performance gap. This is because, unlike the cumulative sum operator, there is a range for the phase values ([0~2pi]), so it might be possible for HIFI-GAN to cover the phase if it has enough receptive field to cover the phase. If it is not, I think it would be better if there is an experiment showing the change of the performance when the length of the input gets longer, because the loss for learning cumsum gets larger when generating full sequence.\n\nSecond, although it conducts diverse analyses and can give many insights about designing the architecture of the model, I think there is a slight lack of novelty. As mentioned in Section 1, there have been several models that generate audio autoregressively in the chunk level [1]. Also, in text-to-speech, there is already a technique of using a reduction factor that controls the trade-off between the sampling speed and the speech quality [2].\n\nThird, there are several questions about the paper that I am confused about.\n* In Section 2, it is a little difficult for me to understand how penalizing the L1 distance between activations leads to requiring a model to produce a waveform that is close to the ground truth waveform.\n* According to Figure 2, it seems that a part of the generated speech is fed back to the model. Is it also true during training or the ground-truth segment is used? (i.e. teacher-forcing)\n* If the CARGAN and HIFI-GAN have the same generator and discriminator architecture, how the training speed and memory usage can be improved? I think CARGAN would rather be slightly inferior because of the additional previous waveform information. If this is because of the difference in the sequence length used in training (8192 vs. 2048), what happens when training the HIFI-GAN with 2048 segment size?\n\nLastly, there are some personal opinions about this paper.\n* As far as I know, there is a method to implement the causal convolution with the non-causal convolution layer based on padding operation [3]. Therefore, I think the maximum length of a learnable cumulative sum is not [(m+1)/2].\n* I think it would be better if the loss is described more in detail to understand the loss compensation technique.\n* I wondered about which chunk size 'k' is used in the experiment of training the cumulative sum operation, in order to understand the relationship between receptive field, chunk size, and the sequence length.\n* There is no subsubsection 5.2.2, so I think the subsubsection 5.2.1 can be combined with subsection 5.2\n* I think that it would be better if the transcripts for the audio samples in the demo page are written with the words to focus emphasized.\n\n\n[1] Weiss, R. J., Skerry-Ryan, R., Battenberg, E., Mariooryad, S., and Kingma, D. P. Wave-Tacotron: SpectrogramFree End-to-End Text-to-Speech Synthesis. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.\n\n[2] Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., and Saurous, R. A. Tacotron: Towards end-to-end speech synthesis. In Interspeech, 2017.\n\n[3] https://theblog.github.io/post/convolution-in-autoregressive-neural-networks/", "summary_of_the_paper": "This paper proposes a conditional waveform synthesis (CWS) model called Chunked Autoregressive GAN (CARGAN). By combining the advantages of the AR and non-AR generative models, CARGAN achieves better pitch accuracy with faster training speed and less memory usage without much decline in generation speed.\n\nFirst, it shows that the previous GAN-based non-AR CWS models do not accurately preserve the pitch and periodicity of the audio signal. Then, this paper demonstrates that there is a close relationship between pitch and phase, and the inductive bias of AR model is good at learning the relationship. Especially, this paper supports the claim by conducting an experiment that compares the ability of  AR and non-AR models to learn the cumulative sum operation, because the instantaneous frequency and instantaneous phase have the cumulative sum relationship. Lastly, by comparing HIFI-GAN and CARGAN based on spectrogram-to-waveform conversion task, this paper shows that CARGAN reconstructs the original waveform more precisely. Furthermore, CARGAN shows faster training speed and less memory usage than those of the HIFI-GAN.", "summary_of_the_review": "Overall, I think this paper gives several insights to design a conditional waveform synthesis model by combining AR and non-AR TTS model.\nHowever, I think it slightly lacks novelty in that there have been several models designing an autoregressive model with chunk-level output. Also, I think it is a little weak in supporting its claim and there is room for improving the writing. As a result, I will give a score of 5 for this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KdOQayGbmGg", "reviewer": "Reviewer_dhJa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well presented and the reasoning is well discussed.\n2. Literature review is satisfactory, but some more references can be added.\n3. The artifacts caused by parallel generation are highlighted in the web page with extensive audio examples.\n4. The method is evaluated by looking at pitch, periodicity errors and F1 score of voiced/unvoiced classification as well as A/B comparison subjective listening tests with other conditional synthesizer methods.\n\nWeaknesses:\n1. An example of hybrid methods would be: \"Wave-tacotron\" which is very similar to this paper but uses flow instead of GANs. I think it needs to be cited.\n2. The boundary errors are quite audible in music examples even with context inclusion. Maybe the hyper-parameters such as chunk size and context size need to be tuned for music.", "summary_of_the_paper": "The paper introduces a Chunked Autoregressive GAN (CARGAN) method for conditional synthesis which is autoregressive over chunks of audio but uses Hifi-GAN like parallel generation within a chunk. The method is motivated by the periodicity and pitch errors shown by existing parallel (non-autoregressive) GAN generators. The authors argue that the periodicity and pitch errors are caused by the parallel GAN generators which may disregard continuity of the periodicity and pitch when generating audio. The auto-regression makes sure that the generated audio is less error-prone in regards to continuity and accuracy of the periodicity and the pitch. The discriminator takes in some contextual samples as well as the chunk that is generated to make sure there is less boundary artifacts.", "summary_of_the_review": "The paper reads well and has a very extensive demo page. I believe it furthers the state of the art in this area.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a hybrid autoregressive and adversarial model for sound synthesis (including but not limited to speech), conditioned on various types of control signals. Although recent adversarial approaches have gained favor over previously popular autoregressive approaches in this domain, because of their ability to produce audio signals much more quickly, the authors argue that these models tend to introduce certain types of artifacts which stem from an inability to learn accurate pitch and periodicity. They propose to address this by reintroducing some degree of autoregression, without compromising too much on inference speed.

Reviewers praised the presentation of this work, the thoroughness of the experimental evaluation, and the audio examples provided. A few concerns were also raised regarding related work and the clarity of some parts of the paper, which the authors have taken the time to address. After the discussion phase, all reviewers chose to recommend acceptance, and I will follow their recommendation.
Decision: Accept (Poster)