Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The authors introduce MelNet, an autoregressive model of Mel-frequency scaled spectrograms. They convert audio into high resolution spectrograms to reduce the audio artifacts introduced by inverting spectrograms (here they use gradient-based inversion over Griffin-Lim). To improve modeling of long term dependencies, they perform multi-scale splitting of the spectrograms and maximize the likelihood at each scale (avoiding dominance of noise at higher resolutions). They condition generation at finer scales from coarser scales, enabling sampling through an ancestral process. The authors also highlight the difference between temporal and frequency dimensions, creating different conditioning stacks for the past in time vs. the "past" in frequency (lower frequencies), and mixing conditioning between the two stacks through layers of the network. Multilayer RNNs are used throughout the network and external conditioning is incorporated at the input. 

The challenge the authors are attempting to address is modeling of audio structure on both long and short timescales. As the authors demonstrate with strong baselines, WaveNet models, while superior on fine-scale fidelity, fail to capture dynamics more than a couple hundred milliseconds. The experiments demonstrate improvements on state-of-the-art for unconditional generation on text-to-speech datasets (generating coherent words and phrases) and the MAESTRO piano dataset (generating sections with consistent dynamics/timing/motifs). The continuations of primed examples in both domains are particularly impressive qualitatively, as they maintain much of the character of the priming sample. Ablation experiments qualitatively demonstrate the importance of multi-scale modeling for unconditional generation. Human listener studies support the claims made from qualitative evaluation of long term structure.

This paper should be accepted because it represents a non-trivial adaptation of autoregressive modeling to handle multi-scale structure in audio. The baselines comparisons and strong, and experiments validate the claims of the paper. 

That said, several things could be done to improve the clarity and significance of the paper.

* While the network architecture is described in detail and some figures, the full network structure itself is non-trivial and still somewhat opaque from the plain text description. A schematic diagram of the full network architecture, even in the appendix, could help clarify how many layers are present connecting each component of the model, which would improve reproducibility.

* The paper is a bit thin on metrics. Human listening studies compare long-term structure, but not short-scale fidelity. For TTS, there are clear artifacts from the spectrogram inversion process. Mean opinion scores on conditional samples could help to quantify the importance of each element of the network for audio quality. For instance, how does MOS compare between Griffin-Lim MelNet, Gradient Inversion MelNet, and WaveNet? How does MelNet compare to Linear scaled spectrograms?

* Generating MelSpectrograms to model long-term structure is a fairly established technique, most notably employed by all of the Tacotron variants (https://google.github.io/tacotron/). These models are perhaps a more appropriate comparison for MelNet in many ways, and opt for spectrogram inversion by smaller WaveRNN models. One of the claims of the paper is that it is important to model the fine-scale structure of spectrograms, but it is not clear if that really is the case. A proper comparison to Tacotron models (where spectrograms are generated at the same resolution / the same inversion methods are used) would help clarify the importance of end-2-end training, vs. the learned inversion approach. 


review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: In this paper the authors present a new generative model for audio in the frequency domain to capture better the global structure of the signal. For this, they use an autoregressive  procedure combined with a multiscale generative model for two-dimensional time-frequency visual representation (STFT spectrogram). The proposed method is tested across a diverse set of audio generation tasks

Overall, The idea of generating audio from 2D spectrogram is original but in my point of view the use of STFT is not appropriate in this context, especially with its lossy criterion. 

Given the clarifications and the author’s responses below, I increased the score from 3 to 6. 



Detailed comments:

Pro: 

Mitigate the problem of generating signal using only local dependencies (on a narrow time scale) and this by capturing high level dependency that emerges on larger timescale (several seconds) using spectrogram.



Cons: 

(1)The use of  STFT is not justified why not wavelet spectrogram to capture both scale and time?
(2)It is still confusing how the use of high resolution spectrogram improve the lossy representation?
(3)If you increase the STFT hope size you come back to the main problem that you are trying to resolve (i.e,  the bias towards capturing local dependencies) 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This work treats 2-D spectrogram as image and uses an autoregressive models which factorizes over both time and frequency dimensions. 

Detailed comments:

- MelNet is not a "fully end-to-end generative model of audio". It generates the spectrogram and relies on other algorithmic component (Griffin-Lim or gradient-based inversion) to generate raw audio.

- MelNet can model the long range structure for unconditional generation of speech, but its audio fidelity is not as good as autoregressive or non-autoregressive models on raw waveforms. The major reason is that MelNet discards the phase information which is useful for high-fidelity speech synthesis. It would be more interesting if MelNet jointly models the magnitude and phase information. 

- The mixture density networks are well known. One may omit the details (or put them in Appendix) in Section 3 for space reason. Overall, the paper is clearly written, but it can be shortened in several ways.

- "making the use of 2D convolution undesirable."
It's still unclear the conv2d is useful or undesirable for modeling spectrograms in generative tasks. Even in recognition tasks, I have seen different results from different papers for different settings,  e.g., In Deep Speech 2, conv2d is useful to reduce WER in ASR. 

- Missing connection in related work: previous conditional generation methods (e.g., Tacotron, Deep Voice 3) are autoregressive over time, but assume conditional independence over frequency bins. 

- There is TTS experiments on the demo website, but I didn't find any details. For example, where does the conditional information (**aligned** linguistic feature) come from?

My major concern is about the usefulness of the model:

1) The unconditional speech generation is an uncommon & less useful task in general. If the task is purposely constructed, the learned representation is more useful than the generation itself (e.g., van den Oord et al. 2017). However, the authors have not demonstrated the usefulness of the learned representation for any downstream task. 

2) The MelNet is autoregressive over both time and frequency. Thus, it is as slow as autoregressive waveform models at synthesis with worse audio fidelity, which make it less preferred in potential TTS applications.
review_title: Official Blind Review #1

Meta-review: The paper proposed an autoregressive model with a multiscale generative representation of the spectrograms to better modeling the long term dependencies in audio signals. The techniques developed in the paper are novel and interesting. The main concern is the validation of the method. The paper presented some human listening studies to compare long-term structure on unconditional samples, which as also mentioned by reviewers are not particularly useful. Including justifications on the usefulness of the learned representation for any downstream task would make the work much more solid. 
Decision: Reject