Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a Chunked Autoregressive GAN (CARGAN) method for conditional synthesis which is autoregressive over chunks of audio but uses Hifi-GAN like parallel generation within a chunk. The method is motivated by the periodicity and pitch errors shown by existing parallel (non-autoregressive) GAN generators. The authors argue that the periodicity and pitch errors are caused by the parallel GAN generators which may disregard continuity of the periodicity and pitch when generating audio. The auto-regression makes sure that the generated audio is less error-prone in regards to continuity and accuracy of the periodicity and the pitch. The discriminator takes in some contextual samples as well as the chunk that is generated to make sure there is less boundary artifacts.
main_review: Strengths:
1. The paper is well presented and the reasoning is well discussed.
2. Literature review is satisfactory, but some more references can be added.
3. The artifacts caused by parallel generation are highlighted in the web page with extensive audio examples.
4. The method is evaluated by looking at pitch, periodicity errors and F1 score of voiced/unvoiced classification as well as A/B comparison subjective listening tests with other conditional synthesizer methods.

Weaknesses:
1. An example of hybrid methods would be: "Wave-tacotron" which is very similar to this paper but uses flow instead of GANs. I think it needs to be cited.
2. The boundary errors are quite audible in music examples even with context inclusion. Maybe the hyper-parameters such as chunk size and context size need to be tuned for music.
summary_of_the_review: The paper reads well and has a very extensive demo page. I believe it furthers the state of the art in this area.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a chunk-wise auto-regressive generative model for audio with adversarial loss.
In particular, the authors note the limitations of purely convolutional adversarial audio generation model for text to speech as those fail to provide a consistent pitch for an extended duration. The authors provide a number of audio samples to illustrate the issue.
They evaluate the proposed method on standard datasets (VCTK) and perform subjective evaluation. they show that their method achieve better pitch correctness, and can improve perceptual evaluations, in some cases.
They also evaluate the speed of evaluation and training of their method. Unsurprisingly, the method proposed is slower than non auto-regressive generation but the chunking makes it theoretically faster than purely auto-regressive methods.

main_review: Strengths:
- the pitch motivation is well presented and natural.
- the solution proposed is a good trade-off between accuracy and speed, and is straightforward to put in place.
- the authors provide extensive evaluations, including MOS.
- the authors provide extensive examples to highlight different types of artifacts

Weaknesses:
- Some remarks (see after) should be addressed.
- a single baseline (HiFi GAN), while their exist a considerable number of vocoders (auto-regressive, flow etc).
- the phase prediction issue is intuitive and could be more compactly treated.


Questions and Remarks:
- In [1], the authors condition on a melody synthetize from the pitch (i.e. a cosine waveform with consistent phase and the right pitch). This means the generator can use the phase from the waveform as a base for its generation. This is much more efficient than giving the pitch directly for the reasons the authors state. This is related to your work, and it could be interesting to see how HiFi-GAN performs with that simple fix. Of course the advantage of the method proposed here is that there is no need to extract pitch information for generation.
- for the toy task, the results for HiFi GAN should be the same pattern repeated as the input to the model is constant and so each output frame should be the same no?
- can you comment and add speed comparison with other methods, in particular waveglow, wavernn etc.
- do the authors know if Parallel WaveNet would suffer from pitch artifacts ? because it is no longer autoregressive and only conditioned on random noise. intuitively one might think that this would be problematic. Same question for flow based methods like waveglow.

There a number of typos:
- p2. 'deniosing'
- p2 'hybird'
- p 4: 'Autoregression is an sensible' -> 'a sensible'


[1]: Unsupervised Cross-Domain Singing Voice Conversion, Polyak et al 2020.
summary_of_the_review: Simple method suggested by the author to fix some known artifacts from GAN audio vocoders.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a conditional waveform synthesis (CWS) model called Chunked Autoregressive GAN (CARGAN). By combining the advantages of the AR and non-AR generative models, CARGAN achieves better pitch accuracy with faster training speed and less memory usage without much decline in generation speed.

First, it shows that the previous GAN-based non-AR CWS models do not accurately preserve the pitch and periodicity of the audio signal. Then, this paper demonstrates that there is a close relationship between pitch and phase, and the inductive bias of AR model is good at learning the relationship. Especially, this paper supports the claim by conducting an experiment that compares the ability of  AR and non-AR models to learn the cumulative sum operation, because the instantaneous frequency and instantaneous phase have the cumulative sum relationship. Lastly, by comparing HIFI-GAN and CARGAN based on spectrogram-to-waveform conversion task, this paper shows that CARGAN reconstructs the original waveform more precisely. Furthermore, CARGAN shows faster training speed and less memory usage than those of the HIFI-GAN.
main_review: This paper proposes a novel conditional waveform generative model, CARGAN, combining the advantages of AR and non-AR models. 
Furthermore, it not only improves the performance of the CWS, but it also conducts the side-by-side listening comparison and pitch analysis, which makes it easier for readers to understand what is the problem of the previous non-AR CWS model. Also, I think that the consideration about choosing the optimum chunk size depending on the size of the receptive field also can give good insights to future works.

However, I have several concerns about this work.  

First, I am not sure that the comparison of the ability to learn the cumulative sum operator is enough to explain the performance gap. This is because, unlike the cumulative sum operator, there is a range for the phase values ([0~2pi]), so it might be possible for HIFI-GAN to cover the phase if it has enough receptive field to cover the phase. If it is not, I think it would be better if there is an experiment showing the change of the performance when the length of the input gets longer, because the loss for learning cumsum gets larger when generating full sequence.

Second, although it conducts diverse analyses and can give many insights about designing the architecture of the model, I think there is a slight lack of novelty. As mentioned in Section 1, there have been several models that generate audio autoregressively in the chunk level [1]. Also, in text-to-speech, there is already a technique of using a reduction factor that controls the trade-off between the sampling speed and the speech quality [2].

Third, there are several questions about the paper that I am confused about.
* In Section 2, it is a little difficult for me to understand how penalizing the L1 distance between activations leads to requiring a model to produce a waveform that is close to the ground truth waveform.
* According to Figure 2, it seems that a part of the generated speech is fed back to the model. Is it also true during training or the ground-truth segment is used? (i.e. teacher-forcing)
* If the CARGAN and HIFI-GAN have the same generator and discriminator architecture, how the training speed and memory usage can be improved? I think CARGAN would rather be slightly inferior because of the additional previous waveform information. If this is because of the difference in the sequence length used in training (8192 vs. 2048), what happens when training the HIFI-GAN with 2048 segment size?

Lastly, there are some personal opinions about this paper.
* As far as I know, there is a method to implement the causal convolution with the non-causal convolution layer based on padding operation [3]. Therefore, I think the maximum length of a learnable cumulative sum is not [(m+1)/2].
* I think it would be better if the loss is described more in detail to understand the loss compensation technique.
* I wondered about which chunk size 'k' is used in the experiment of training the cumulative sum operation, in order to understand the relationship between receptive field, chunk size, and the sequence length.
* There is no subsubsection 5.2.2, so I think the subsubsection 5.2.1 can be combined with subsection 5.2
* I think that it would be better if the transcripts for the audio samples in the demo page are written with the words to focus emphasized.


[1] Weiss, R. J., Skerry-Ryan, R., Battenberg, E., Mariooryad, S., and Kingma, D. P. Wave-Tacotron: SpectrogramFree End-to-End Text-to-Speech Synthesis. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.

[2] Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., Le, Q., Agiomyrgiannakis, Y., Clark, R., and Saurous, R. A. Tacotron: Towards end-to-end speech synthesis. In Interspeech, 2017.

[3] https://theblog.github.io/post/convolution-in-autoregressive-neural-networks/
summary_of_the_review: Overall, I think this paper gives several insights to design a conditional waveform synthesis model by combining AR and non-AR TTS model.
However, I think it slightly lacks novelty in that there have been several models designing an autoregressive model with chunk-level output. Also, I think it is a little weak in supporting its claim and there is room for improving the writing. As a result, I will give a score of 5 for this work.
review_title: null

Meta-review: This work proposes a hybrid autoregressive and adversarial model for sound synthesis (including but not limited to speech), conditioned on various types of control signals. Although recent adversarial approaches have gained favor over previously popular autoregressive approaches in this domain, because of their ability to produce audio signals much more quickly, the authors argue that these models tend to introduce certain types of artifacts which stem from an inability to learn accurate pitch and periodicity. They propose to address this by reintroducing some degree of autoregression, without compromising too much on inference speed.

Reviewers praised the presentation of this work, the thoroughness of the experimental evaluation, and the audio examples provided. A few concerns were also raised regarding related work and the clarity of some parts of the paper, which the authors have taken the time to address. After the discussion phase, all reviewers chose to recommend acceptance, and I will follow their recommendation.
Decision: Accept (Poster)