MelNet: A Generative Model for Audio in the Frequency Domain
Authors: Sean Vasquez
Mike Lewis
Venue:  (2020)

Abstract: Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps.  While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms.  By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales which time-domain models have yet to achieve.  We demonstrate that our model captures longer-range dependencies than time-domain models such as WaveNet across a diverse set of unconditional generation tasks, including single-speaker speech generation, multi-speaker speech generation, and music generation.

Under review as a conference paper at ICLR 2020
MELNET: A GENERATIVE MODEL FOR AUDIO IN THE FREQUENCY DOMAIN
Anonymous authors Paper under double-blind review
ABSTRACT
Capturing high-level structure audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependen- cies are difficult to model directly the time domain, we show that they can be more tractably modelled in two-dimensional tine-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and multiscale generation procedure, we design model capable of generating high-fidelity audio samples which cap- ture structure timescales which time-domain models have yet achieve. We demonstrate that our model captures longer-range dependencies than time-domain models such as WaveNet across a diverse set of unconditional generation tasks, including single-speaker speech generation, multi-speaker speech generation, and music generation.
INTRODUCTION
Audio waveforms have complex structure at drastically varying timescales, which presents a challenge for generative models. Local structure must be captured to produce high-fidelity audio, while long- range dependencies spanning tens of thousands of timesteps must be captured to generate audio which globally consistent. Existing generative models of waveforms such as WaveNet (van den Oord et al., 201 6a) and SampleRNN (Mehri et al., 2016) are well-adapted to model local dependencies, but as these models typically only backpropagate through fraction a second, they are unable to capture high-level structure that emerges the scale of several seconds.
We introduce a generative model for audio which captures longer-range dependencies than existing end-to-end models. We primarily achieve this by modelling 2D time-frequency representations such as spectrograms rather than ID time-domain waveforms (Figure 1). The temporal axis a spectrogram is orders of magnitude more compact than that of waveform, meaning dependencies that span tens of thousands of timesteps in waveforms only span hundreds of timesteps in spectrograms. In practice, this enables spectrogram models to generate unconditional speech and music samples with consistency over multiple seconds whereas time-domain models must be conditioned on intermediate features capture structure at similar timescales.
Modelling spectrograms can simplify the task of capturing global structure, but can weaken a model's ability capture local characteristies that correlate with audio fidelity. Producing high-fidelity audio has been challenging for existing spectrogram models, which we attribute to the lossy nature of spectrograms and oversmoothing artifacts which result from insufficiently expressive models. To reduce information loss, we model high-resolution spectrograms which have the same dimensionality as their corresponding time-domain signals. limit oversmoothing, we use highly expressive autoregressive model which factorizes the distribution over both the time and frequency dimensions.
Modelling both fine-grained details and high-level structure high-dimensional distributions is known be challenging for autoregressive models. To capture both local and global structure in spectrograms with hundreds of thousands of dimensions, we employ a multiscale approach which generates spectrograms in coarse-to- to-rse manner. A low-resolution. subsampled spectrogram that captures high-level structure is generated initially, followed by an iterative upsampling procedure that adds high-resolution details.

---

Under review as a conference paper at ICLR 2020
Fraquency
www.
Time
Time
Spectrogram representation
Waveform representation (Ix, 5x, 25x, 125x)
Figure 1: Spectrogram and waveform representations of the same 4 second audio signal. The waveform spans nearly 100,000 timesteps whereas the temporal axis of the spectrogram spans roughly 400. Complex structure is nested within the temporal axis of the waveform at various timescales, whereas the spectrogram has structure which is smoothly spread across the time-frequency plane.
Combining these representational and modelling techniques yields highly expressive and broadly applicable generative model of audio. Our contributions are are as follows:
We introduce MelNet, generative model for spectrograms which couples a fine- fine-grained autoregressive model and multiscale generation procedure to jointly capture local and global structure.
We show that MelNet is able to model longer-range dependencies than existing time-domain models. Additionally, we include an ablation to demonstrate that multiscale modelling is essential for modelling long-range dependencies.
We demonstrate that MelNet is broadly applicable to variety of audio generation tasks, including unconditional speech and music generation. Furthermore, MelNet able model highly multimodal data such as multi-speaker and multilingual speech.
2 PRELIMINARIES
We briefly present background regarding spectral representations of audio. Audio is represented digitally as a one-dimensional, discrete-time signal y = (yI..... Vn). Existing generative models for audio have predominantly focused on modelling these time-domain signals directly. We instead model spectrograms, which are two-dimensional time-frequency representations which contain information about how the frequency content of an audio signal varies through time. Spectrograms by taking the squared magnitude of the short-time Fourier transform (STFT) of a time-domain signal, iee. [|STFT(y)] The value of Tij (referred as amplitude energy) corresponds to the squared magnitude of the jth element of the frequency response at timestep i. Each slice a,, is referred to a frame. We assume time-major ordering. but following convention, all figures are displayed transposed and with the frequency axis inverted.
computed
Time-frequue frequeccy representations such as spectrograms highlight how the tones and pitches within an audio signal vary through time. Such representations are closely aligned with how humans perceive audio. To further align these representations with human perception, we convert the frequency axis to the Mel scale and apply an elementwise logarithmic rescaling of the amplitudes. Roughly speaking, the Mel transformation aligns the frequency axis with human perception pitch and the logarithmic rescaling aligns the amplitude axis with human perception of loudness.
Spectrograms are lossy representations of their corresponding time-domain signals. The Mel trans- formation discards frequency information and the removal of the STFT phase discards temporal information. When recovering a time-domain signal from a spectrogram, this information loss man- ifests distortion in the recovered signal. To minimize these artifacts and improve the fidelity of generated audio, we model high-resolution spectrograms. The temporal resolution of a spectrogram can be increased by decreasing the STFT hop size, and the frequency resolution be increased by
2

---

Under review as a conference paper at ICLR 2020
Time-delayed stack
(h) Frequency-delayee stack
Figure 2: The context æcij (grey) for the element 2ij (black) is encoded using 4 RNNs. Three of these are used in the time-delayed stack to extract features from preceding frames. The fourth is used in the requeeccyddelayee stack to extract features from all preceding elements within the current frame. Each arrow denotes an individual RNN cell and arrows the same color use shared parameters.
increasing the number of Mel channels. Generated spectrograms are converted back to time-domain signals using classical spectrogram inversion algorithms. We experiment with both Griffin- (Grif- fin & Lim, 1984) and gradient-base base inversion algorithm (Decorsière et al., 2015), and ultimately use the latter as it generally produced audio with fewer artifacts.
PROBABILISTIC MODEL
We use an autoregressive model which factorizes the joint distribution over spectrogram as product of conditional distributions. Given an ordering of the dimensions of x, we define the context Teij as the elements x that precede Zij. We default to a row-major ordering which proceeds through each frame Fg,, from low to high frequency, before progressing the next frame. The joint density factorized as
p(x)
IIIII p(zy azzji 0:j),
where 0g parameterizes univariate density Fig- We model each factor distribution Gaussian mixture model with K components. Thus, 0i consists of 3K parameters corresponding to K and mixture coefficients {TiHe The resulting
means {Piga K 1 standard deviations {aigk] factor distribution then be expressed
K
p(ay Kziji 0ij)
Σ k=1
Tisk NZzzi Pijks Tijk).
Following the work on Mixture Density Networks (Bishop, 1994) and their application to autoregressive models (Graves, 2013), 0i is modelled as the output of a neural network and computed as function of the context reij. Precisely, for some network f with parameters s/, we maximum--lkklihood
have 6,y f 1)
maximum likelihood estimate for the network parameters computed by minimizing the negative log-likelihood via gradient descent.
To ensure that the network output parameterizes valid Gaussian mixture model, the network first computes unconstrained parameters {Rigk, Oijk, Rigk H as a vector d,z R3K and enforces constraints on Oi by applying the following transformations:
Pisk Pisk
(3)
Tyyk: exp(ā
(4)
exp(frisk)
Tijk
Ek k= (xp(f
These transformations ensure the standard deviations Fijk positive and the mixture coefficients Tiyk sum to one.
4 NETWORK ARCHITECTURE
To model the distribution in an autoregressive manner, we design a network which computes the distribution over Vij function of the context &zgig- The network architecture draws inspiration
3

---

Under review as a conference paper at ICLR 2020
from existing autoregressive models for images (Theis & Bethge, 2015; van den Oord et al., 2016c;b; Chen et al., 2017; Salimans et al., 2017; Parmar et al., 2018; Child et al., 2019). In the same way that these models estimate distribution pixel-by-pixel over the spatial dimensions of image, model estimates a distribution Wementtby-eeemeee over the time and frequency dimensions of a spectrogram. A noteworthy distinction is that spectrograms are not invariant to translation along the frequency axis, making 2D convolution less desirable than other 2D network primitives which do not assume invariance. Utilizing multidimensional recurrence instead of 2D convolution has been shown to be beneficial when modelling spectrograms in discriminative settings (Li et al., 2016; Sainath Li, 2016), which motivates our use of an entirely recurrent architecture.
Similar to Gated PixelCNN (van den Oord et al., 2016b), the network has multiple stacks of computa- tion. These stacks extract features from different segments of the input to collectively summarize the full context Ezji
The time-delayed stack computes features which aggregate information from all previous frames T<i,a-
The frequency-delayed stack utilizes all preceding elements within a frame, Ti,sj. as well as the outputs of the time-delayed stack, to summarize the full context Zz(j
The stacks are connected at each layer of the network, meaning that the features generated by layer I of the time-delayed stack are used as input to layer of the delluencydellyed stack. To facilitate the training of deeper networks, both stacks use residual connections (He et al., 2016). The outputs of the final layer of the reequencydelaayee stack are used to compute the unconstrained parameters Å.
4.1 TIME-DELAYE STACK
The time-delayed stack utilizes multiple layers of multidimensional RNNs to extract features from Tei,,, the two-dimensional region consisting of all frames preceding Ziy. Each multidimensional which runs forwards along the frequency
RNN is composed of three one-dimensional RNNs: axis, one which runs backwards along the frequency axis, and one which runs forwards along the time axis. Each RNN runs along each slice of given axis, shown in Figure 2. The output of each layer of the time-delayed stack is the concatenation of the three RNN hidden states.
We denote the function computed at layer of the time-delayed stack (three RNNs followed by con- catenation) as Ft. At each layer, the time-delayed stack uses the feature map from the previous layer, [[[-1]. to compute the subsequent feature map Fi (h-(1-1)) which consists of the three concatenated RNN hidden states. When using residual connections, the computation of h from h²(Z 1] becomes
h [1] = W - 11),, + hi - 1].
(6)
To ensure the output h [1] is only function of frames which lie in the context Kejij the inputs to the time-delayed stack are shifted backwards one step in time: hly [0] = Wf
4.2 FREQUENCY-DEIN DELAVED STACK
The frequency-delaye stack is one-dimensional RNN which runs forward along the frequency axis. Much like existing one-dimensional autoregressive models (language models, waveform models, etc.), the eequency-delayee stack operates one-dimensional sequence (a single frame) and estimates the distribution for each element conditioned on all preceding elements. The primary difference is that it is also conditioned upon the outputs of the time-delayed stack, allowing use the full two-dimensional context Veejj
We denote the function computed by the frequency-delaye stack as F/. At each layer, the frequency- delayed stack takes two inputs: the the previous-laye outputs of the frequency-delaye stack, N, [t_(1])] and the current- layer outputs of the dime-delayed stack ht,11 [1]. These inputs are summed and used as input to dimensinnii RNN to produce the output feature map Fi (h [ 1), (2(2)) which consists of the RNN hidden state:
= [00=://F/ W/F
+ +NG M.
1).
4

---

Under review as a conference paper at ICLR 2020
h1
h 11]
W
Figure 3: Computation graph for single layer of the network. -ii and Ff the functions computed by the time-delayed stack and frequency-delayed stack, respec- tively. The outputs of these functions are projected (by the matrices W and Wi/ and summed with the layer inputs to form residual blocks.
F
W
F
ht
11 1]
To ensure that h computed using only elements in the context Tecyy the inputs the frequency- delayed stack are shifted backwards one step along the frequency axis: N w W 1. At the final layer, layer L, a linear map is applied to the output of the frequency-delayer stack to produce the unconstrained Gaussian mixture model parameters, i.e. 0.j Woh
4.3 CONDITIONING
To incorporate conditioning information into the model, conditioning features 2 are simply projected onto the input layer along with the inputs
h [0] = W -0 + W
(8)
h [0] = W v. + W! zg.
(9)
Reshaping, upsampling, and broadcasting can be used as necessary to ensure the conditioning features have the same time and frequency shape as the input spectrogram, c.g. one-hot vector representation for speaker ID would first be broadcast along both the time and frequency axes.
5 MULTISCALE MODELLING
To improve audio fidelity, we generate high-resolution spectrograms which have the same dimensionality their corresponding time-domain representations. Under this regime, single training example has several hundreds of thousands dimensions. Capturing global structure such high-dimensional distributions is challenging for autoregressive models, which are biased towards capturing local dependencies. To counteract this, we utilize a multiscale approach which effectively permutes the autoregressive ordering so that spectrogram is generated in coarse-to-fine order.
The elements of a spectrogram & are partitioned into G tiers x1 _t..... IG such that each successive tier contains higher-resolution information. We define x as the union of all tiers which precede 2² i.e. x
(x "......""""....... The distribution is factorized over tiers:
p(a; v) = II p(29 x89 19),
(10)
and the distribution of each tier is further factorized element- by-entt----eeeettt- as described in Section 3. We explicitly include the parameterization by vé (i) ,..., G) to indicate that each tier is modelled by separate network.
5

---

Under review as a conference paper at ICLR 2020
(a) Tier (32 x 50)
(b) Tiers 1-3 (64 x 100)
(c) Tiers 1-6 (256 x 200)
Figure 4: A sampled spectrogram viewed at different stages of the multiscale generation procedure. The initial tier dictates high-level structure and subsequent tiers add fine-gratd fine-grained details. Each upsampling tier doubles the resolution of the spectrogram.
2
2
2
2 2
(a) p(a"; )
(b) p(a" x²; (b))
(c) p(a" 1x,0,00
16
(d) p(x
Figure 5: Schematic showing how tiers of the multiscale model interleaved and used condition the distribution for the subsequent tier. a) The initial tier is generated unconditionally. b) The second tier generated conditionally given the the initial tier. c) The outputs of tiers and 2 are interleaved along the frequency axis and used to condition the generation of tier 3. d) Tier 3 is interleaved along time axis with all preceding tiers and used condition the generation of tier
5.1 TRAINING
During training, the tiers generated by recursively partitioning spectrogram into alternating along either the time or frequency axis. We define a function split which partitions an input into and odd rows along given axis. The initial step of the recursion applies the sp1 it function to a spectrogram x, or equivalently x <C++1 so that the even-numbered rows are assigned to z and the odd-numbered rows are assigned
Subssequen tiers defined similarly in a recursive manner:
x9 <x spl it (x
(11)
At each step of the recursion, model the distribution p(29 ~99 19)). The final step of the recursion models the unconditional distribution over the initial tier p(z"; 6²))
To model the conditional distribution p(x9 x 169), the network each tier needs mechanism to incorporate information from the preceding tiers x ~9 To this end, we add feature extraction network which computes features from x S0 which used condition the generation of 2 We use multidimensional RNN consisting of four one-dimensional RNNs which run bidirectionally along slices of both axes of the context 59 layer of the feature extraction network is similar of the time-delayed stack, but since the feature extraction network is not causal, we include an RNN which runs backwards along the time axis and do not shift the inputs. The hidden states of the RNNs in the feature extraction network are used to condition the generation of r As each tier doubles the resolution, the features extracted from x: < have the same time and frequency shape as x9, allowing the conditioning mechanism described in section 4.3 to be used straightforwardly.
layer
5.2 SAMPLING
To sample from the multiscale model we iteratively sample value for 2: conditioned on L9 using
the learned distributions defined by the estimated network parameters = (v1) 3 ). The initial tier, x², is generated unconditionally by sampling from p(z"; e²)) and subsequent tiers are sampled from p(29 S9; (.)) At each tier, the sampled x is interleaved with the context ~9
1g+1 inntenneeaeeeeee <9).
(12)
The interleave function is simply the inverse of the split function. Sampling terminates once a full spectrogram, x: <G+1 has been generated. A spectrogram generated by multiscale model is shown in Figure 4 and the sampling procedure is visualized schematically in Figure

---

Under review as a conference paper at ICLR 2020
6 EXPERIMENTS
To demonstrate the MelNet is broadly applicable as a generative model for audio, we train the model on diverse set of audio generation tasks (single-speaker speech generation, multi-speaker speech generation, and music generation) using three publicly available datasets. Generated audio samples for each task are available on the accompanying web page https://audio-saan satiiiissmmlss...... siimplesgitthbbio.. We include samples generated using the priming and biasing procedures described by Graves (2013). Biasing lowers the temperature of the predictive distribution and priming seeds the model state with a given sequence of audio prior to sampling. Hyperparameters for all experiments are available in Appendix A.
Speech and music have rich hierarchies of latent structure. Speech has complex linguistic structure (phonemes, words, syntax, semantics, etc.) and music has highly compositional musical structure (notes, chords, melody and rhythm, etc.). The presence of these latent structures in generated samples can be used proxy for how well generative model has learned dependencies timescales. As such, a qualitative analysis of unconditional samples is an insightful method of evaluating generative models of audio. To facilitate such a qualitative evaluation, we train MelNet on each of the three unconditional generation tasks and include samples on the accompanying web page. For completeness, briefly provide some of our own qualitative observations regarding the generated samples (Sections 6.1. 6.2, and 6.3). In addition to qualitative analysis, we conduct a human evaluation experiment to quantitatively compare how well WaveNet and MelNet capture high-level structure (Section 6.4). Lastly, we ablate the impact of the multiscale generation procedure on MelNet's ' ability model long-range dependencies (Section 6.5).
various
6.1 SINGKERSPEAKER SPEECH
To test MelNet's ability to model single speaker in controlled environment, we utilize the Blizzard 2013 dataset (King, 201 which consists of audiobook narration performed in a highly animated manner by a professional speaker. We find that MelNet frequently generates samples that contain coherent words and phrases. Even when the model generates incoherent speech, the intonation, prosody, and speaking style remain consistent throughout the duration of the sample. Furthermore, the model learns to produce speech using a variety of character voices and learns to generate samples which contain elements of narration and dialogue. Biased samples tend to contain longer strings of comprehensible words but are read in less expressive fashion. When primed with real sequence of audio, MelNet able to continue sampling speech which has consistent speaking style and intonation.
6.2 MULTI-SPEAKER SPEECH
Audiobook data is recorded in highly controlled environment. To demonstrate MelNet's capacity to model distributions with significantly more variation, we utilize the VoxCeleb2 dataset (Chung al., 2018). The VoxCeleb2 dataset consists of over 2,000 hours of speech data captured with real world noise including laughter, cross-talk, channel effects, music and other sounds. The dataset is also multilingual, with speech from speakers of 145 different nationalities, covering wide range of accents, ages, ethnicities and languages. When trained on the VoxCeleb2 dataset, we find that MelNet able to generate unconditional samples with significant variation in both speaker characteristics (accent, language, prosody, speaking style) well as acoustic conditions (background noise and recording quality). While the generated speech is often not comprehensible, samples can often be identified belonging to specific language, indicating that the model has learned distinct modalities for different languages. Furthermore, is difficult to distinguish real and fake samples which are spoken in foreign languages. For foreign languages, semantic structures are not understood by the listener and cannot be used to discriminate between real and fake. Consequently, the listener must rely largely on phonetic structure, which MelNet is able realistically model.
6.3 MUSIC
To show that MelNet can model audio modalities other than speech, apply the model to the task of unconditional music generation. We utilize the MAESTRO dataset (Hawthorne et al., 2018), which consists of over 172 hours of solo piano performances. The samples demonstrate that MelNet lears musical structures such as melody and harmony. Furthermore, generated samples often maintain consistent tempo and contain interesting variation in volume, timbre, and rhythm.
7

---

Under review as a conference paper at ICLR 2020
WaveNet
MelNet
Wave2Midi2Wave
MelNet
Blizzard 0.0%
100.0%
MAESTRO
37.7% %
62.3 %
VoxCeleb2
0.0%
100.0% %
MAESTRO 4.2%
95.8 %
(b) Comparison between MeINet and Wave2Midi2Was Wave. Wave2Midi2Wave a two-stage model consisting of a Music Transformer trained on lahelled MIDI followed by conditional WaveNet model. The MelNet model, on the other hand, is trained without any intermediate supervision.
(a) Comparison between MelNet and WaveNet. Both models are trained in an en- tirely unsupervised manner.
Table I: Selection rates of human evaluators when asked to identify which model generates samples with longer-term structure. Results show that MelNet captures long-range structure better than WaveNet. Furthermore, MelNet outperforms two-stage model which conditions WaveNet generated MIDI
6.4 HUMAN EVALUATION
Making quantitative comparisons with existing generative models such as WaveNet difficult for various reasons and previous works have ultimately relied on largely empirical evaluations by the reader (Dieleman et al., 2018). To allow the reader to make these judgements for themselves, we provide samples from both WaveNet and MelNet for each of the tasks described in the previous sections. Furthermore, in an effort to provide quantitative metrics to support the claim that MelNet generates samples with improved long-range structure in comparison WaveNet, we conduct human experiment whereby participants are presented anonymized samples from both models and asked to select which sample exhibits longer-term structure. We resort such evaluations since standard metrics for evaluation of generative models such as density estimates cannot be used to compare WaveNet and MclNet as that these models operate on different representations.
The methodology for this experiment is as follows. For each of the three unconditional audio generation tasks, we generated 50 samples from WaveNet and 50 samples from MelNet. Participants were shown an anonymized, randomly-drawe sample from each model and instructed to "select the sample which has more coherent long-term structure." We collected 50 evaluations for each task. Results, shown in Table Ia, show that evaluators overwhelmingly agreed that samples generated by MelNet had more coherent long-range structure than samples from WaveNet across all tasks.
In addition to comparing MelNet to unconditional WaveNet model for music generation, also compare to a two-stage Wive2Midi2Wave model (Hawthorne et al., 2018) which conditions WaveNet on MIDI generated by separately-traine Wave2Midi2Wave model has the advantage of directly modelling labelled musical notes which distill much of the salient, high-level structure in music into a compact symbolic representation. Despite this, as shown by the results in Table 1b, the two-stage model does not capture long-range structure well MelNet model that is trained without access to any intermediate representations.
Music Transformer (Huang et al., 2018). The two-stage
6.5 ABLATION: MULTISCALE MODELLING
To isolate the impact of multiscale modelling procedure described in Section 5, we train models with varying numbers of tiers and evaluate the long-term coherence of their respective samples. As noted before. long-term coherence is difficult to quantify and we provide samples on the accompanying web page so that the reader can make their own judgements. We believe the samples clearly demonstrate that increasing the number of tiers results in samples with more coherent high-level structure. We note that our experiment varies the number of tiers from two to five. Training a single-tier model on full-resolution spectrograms was prohibitively expensive in terms of memory consumption. This highlights another benefit of multiscale modelling- larelennnnnrneeeee.eeeeee deep networks can be allocated to learning complex distributional structure in the initial tiers while shallower networks can be used for modelling the relatively simple, lotreny distributions in the upsampling tiers. This allows multiscale models to effectively allocate network capacity in proportion to the complexity of the modelling task.
8

---

Under review as a conference paper at ICLR 2020
7 RELATED WORK
The predominant line of research regarding generative models for audio has been directed towards modelling time-domain waveforms with autoregressive models (van den Oord et al., 2016a; Mchri et al., 2016; Kalchbrenner et al., 2018). WaveNet is a competitive baseline for audio generation, and as such, is used for comparison in many of our experiments. However, we note that the contribution of our work is in many ways complementary to that of WaveNet. MelNet is more proficient at capturing high-level structure, whereas WaveNet is capable of producing have demonstrated that time-domain models can be used to invert spectral representations to high- fidelity audio (Shen et al., 2018; Prenger al., 2019; Arkk et al., 2019), suggesting that MelNet could be used in concert with time-domain models such as WaveNet.
gigher-fidelity audio. Several works
Dieleman et al. (2018) and van den Oord et al. (2017) capture long-range dependencies in waveforms by utiliziig a hierarchy of autoencoders. This approach requires multiple stages of models which must be trained sequentially, whereas the multiscale approach in this work can be parallelized over tiers. Additionally, these approaches do not directly optimize the data likelihood, nor do they admit tractable marginalization over the latent codes. We also note that the modelling techniques devised in these works can be broadly applied to autoregressive models such as ours, making their contributions largely complementary to ours.
Recent works have used generative adversarial networks (GANs) (Goodfellow et al., 2014) model both waveforms and spectral representations (Donahue et al., 2018; Engel et al., 20.8). As with image generation, it remains unclear whether GANs capture modes of the data distribution. Furthermore, these approaches are restricted to generating fxxedduration segments of audio, which precludes their usage in many audio generation tasks.
Generating spectral representations is common practice for end-to-end text-to-speech models (Ping et al., 2017: Sotelo al., 2017; Wang al., 2017; Taigman et al., 2018). However, these models use probabilistic models which are much less expressive than the fineigraieed autoregressive model used by MelNet. Consequently, these models are unsuitable for modelling high-entropy, multimodal distributions such as those involved in tasks like unconditional music generation.
The network architecture used for MelNet is heavily influenced by recent advancements in deep autoregressive models for images. Theis & Bethge (2015) introduced an LSTM architecture for autoregressive modelling of 2D images and van den Oord al. (2016c) introduced PixeIRNN and PixelCNN and scaled up the models to handle the modelling of natural images. Subsequent works in autoregressive image modelling have steadily improved state-of-the-ar for image density estimation (van den Oord et al., 2016b; Salimans et al., 2017; Parmar et al., 2018; Chen et al., 2017; Child al., 2019). We draw inspiration from many of these models, and ultimately design recurrent architecture of our own which is suitable for modelling spectrograms rather than images. We note that our choice of architecture not a fundamental contribution of this work. While we have designed the architecture particularly for modelling spectrograms, did not experimentally validate whether it outperforms cxisting architectures and make no such claims to this effect.
We use a multidimensional recurrence in both the time-delayed stack and the upsampling tiers to extract features from wio-dimensional inputs. Our multidimensional recurrence is effectively 'factorized' as it independently applies one-dimensional RNNs across each dimension. This approach differs from the tightly coupled multidimensional recurrences used by MDRNNs (Graves al., 2007; Graves & Schmidhuber, 2009) and GridLSTMs (Kalchbrenner et al.. 2015) and more closely resembles the approach taken by ReNet (Visin et al., 2015). Our approach allows for efficient training as we can extract features from an M x N grid in max (M, N) sequential recurrent steps rather than the MM + N sequential steps required for tightly coupled recurrences. Additionally, our approach enables the use of highly optimized one-iimesiional RNN implementations.
Various approaches to image generation have succeeded in generating high-resolution, globally coherent images with hundreds thousands of dimensions (Karras et al., 2017: Reed et al., 2017: Kingma & Dhariwal, 2018). 8). The methods introduced in these works are not directly transferable to waveform generation, as they exploit spatial properties of images which are absent in mieedimnnioona audio signals. However, these methods are more straightforwardly applicable to two-dimensional representations such as spectrograms. Of particular relevance to our work are approaches which combine autoregressive models with multiscale modelling (van den Oord et al., 201 6e; Dahl et al.,

---

Under review as a conference paper at ICLR 2020
2017: Reed et al., 2017: Menick & Kalchbrenner, 2018). Our work demonstrates that the benefits of a multiscale autoregressive model extend beyond the task of image generation, and can be used to generate high-resolution, globally coherent spectrograms.
CONCLUSION & FUTURE WORK
We have introduced MelNet, generative model for spectral representations of audio. MelNet combines a highly expressive autoregressive model with a multiscale modelling scheme to generate high-resolution spectrograms with realistic structure both local and global scales. In comparison to previous works which model time-domain signals directly, MelNet is particularly well-suited to model long-range temporal dependencies. Experiments show promising results across diverse of audio generation tasks.
Furthermore, believe MclNet provides a foundation for various directions of future work. Two particularly promising directions are text-to-speech synthesis and representation learning:
Text-to-Speech Synthesis: McINet utilizes more flexible probabilistic model than existing end-to-end text-to-speech models, making it well-suited model expressive, multi-modal speech data.
Representation Learning: MelNet able to uncover salient structure from large quantities of unlabelled audio. Large-scale, pre-trained autoregressive models for language modelling have demonstrated significant benefits when fine-tuned for downstream tasks. Likewise, representations learned by MelNet could potentially aid downstream tasks such speech recognition.
REFERENCES
Sercan Arik, Heewoo Jun, and Gregory Diamos. Fast spectrogram inversion using multi-head convolutional neural networks. IEEE Signal Processing Letters, 26(1):94- 98, 2019.
Christopher M Bishop. Mixture density networks. Technical report, Citeseer, 1994.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Picter Abbeel. Pixelsnail: An improved autore- gressive generative model. arXiv preprint arXiv: 171.0:771, 2017.
Rewon Child, Scott Gray, Alee Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. arXiv preprint arXiv:
2018
Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. Pixel recursive super resolution. In Proceed- 2017.
ings of the IEEE International Conference Computer Vision, pp.
Rémi Decorsière, Peter Søndergaard, Ewen MacDonald, and Torsten Dau. Inversion of au- ditory spectrograms, traditional spectrograms, and other envelope representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 23(1)::6-555 56, 2015.
Sander Dicleman, Aaron van den Oord, and Karen Simonyan. The challenge of realistic music generation: modelling raw audio at scale. In Advances in Neural Information Processing Systems, pp.
2018.
Chris Donahue, Julian McAuley, and Miller Puckette. Synthesizing audio with generative adversarial networks. arXiv preprint arXiv: 04208, 2018.
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014.
10

---

Under review as a conference paper at ICLR 2020
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv: 2013.
Alex Graves and Jürgen Schmidhuber. Offline handwriting recognition with multidimensional recurrent neural networks. Advances neural information processing systems, pp. 545-552, 2009.
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Multi-dimensional recurrent neural networks. In International conference on artificial neural networks, pp. 549-558. Springer, 2007.
Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing, 32(2):236-243, 1984.
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the maestro dataset. arXiv preprint 18X0v 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M Dai, Matthew Hoffman, and Douglas Eck. Music transformer: Generating music with long-term structure. arXiv preprint (eXii:/809.0440 2018.
Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. Grid long short-term memory. arXiv preprint arXiv:150.001526, 2015.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. arXiv preprint arXiv:1802.08433 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 11X1v:1710.10196, 2017.
Simon King. The blizzard challenge 2011, 2011.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69888 6980, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Information Processing Systems, pp. 10236- 10245, 2018.
Jinyu Li, Abdelrahman Mohamed, Geoffrey Zweig, and Yifan Gong. Exploring multidimensional Istms for large vocabulary ast. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pp. 4940-4944. IEEE, 2016.
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.07837 2016.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv: 1812. 2018.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, and Alexander Ku. Image transformer. arXiv preprint arXiv:1802.05575: 2018.
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice Scaling text-to-speech with convolutional sequence learning. arXiv preprint 17 2017.

---

Under review as a conference paper at ICLR 2020
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621. IEEE, 2019.
Scott Reed, Aäron den Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density estimation. arXiv preprint 17X1::7703.3364, 2017.
Tara N Sainath and Bo Li. Modeling time- -requenny patterns with Istm vs. convolutional architectures for Ivesr tasks. In INTERSPEECH, pp. 813-817, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint 1rX1v:1701.05511/ 2017.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
IEEE, 2018.
Jose Sotelo, Soroush Mchri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron Courville, and Yoshua Bengio. Char2wav: End-to-end speech synthesis. 2017.
Yaniv Taigman, Lior Wolf, Adam Polyak, and Eliya Nachmani. Voiceloop: Voice fitting and synthesis via a phonological loop. 2018
Lucas Theis and Matthias Bethge. Generative image modeling using spatial Istms. In Advances in Neural Information Processing Systems, pp. 1927-1935, 2015.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.034999 2016a.
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in Neural Information Processing Systems, pp. 4790-4798, 2016b.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv: 1/601106777 2016c.
Aaron van den Oord, Oriol Vinyals, al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Francesco Visin, Kyle Kastner, Kyunghyun Cho, Matteo Matteucci. Aaron Courville, and Yoshua Bengio. Renet: A recurrent neural network based alternative to convolutional networks. arXiv preprint arXiv: arXiv:/505.000 550.00035.0 2015.
Yuxuan Wang, RJ Skeny-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv: 1703. 10135, 2017.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 HYPERPARAMETERS & TRAINING DETAILS
All RNNs use LSTM cells (Hochreiter & Schmidhuber, 1997). All models are trained with RMSProp (Tieleman & Hinton, 2012) with a learning rate of 10-4 and momentum of 0.9. The initial values for all recurrent states are trainable parameters. single hyperparameter controls the width of the network-all hidden sizes (RNN state size, residual connections, etc.) are defined by single value, denoted hidden size in table 2.
Table 2: MelNet hyperparameters.
Blizzard
MAESTRO
VoxCeleb2
Tiers
6
5
Layers (Initial Tier)
12
16
16
Layers (Upsampling Tiers)
5-4-3-2-2
6-5-4
6-5-4-3
Hidden Size
512
512
512
GMM Mixture Components
10
10
Batch Size
32
16
128
Sample Rate (Hz)
22,050
22,050
16,000
Max Sample Duration (s)
10
6
Mel Channels
256
256
180
STFT Hop Size
256
256
180
STET Window Size
6 252
6 256
680
A.2 WAVENET BASELINE
The human evaluation experiments require samples from baseline WaveNet model. For the Blizzard and VoxCeleb2 datasets, encoding and models each sample with a discrete distribution. Each model is trained for 150,000 steps. We the Adam optimizer (Kingma & Ba, 2014) with learning rate of 0.001 and batch size of 32. Additional hyperparameters are reported in Table 3.
reimplementation. Our WaveNet model uses 8-bit μ-llaw
Table 3: WaveNet hyperparameters.
Blizzard
VoxCeleb2
Sample Rate (Hz)
22,050
16,000
Layers
50
60
Kernel Size
3
3
Dilation (at layer i)
mod
2 mod 10
Residual Channels
512
512
Skip Channels
512
512
Receptive Field (samples)
10,240
12,288
Receptive Field (ms)
464
768
Max Sample Duration (s)
2
2
We do not use our WaveNet implementation for human evaluation on the MAESTRO dataset. The authors that introduce this dataset provide roughly minutes of audio samples their website for both unconditional WaveNet and Wave2Midi2Wav Wave models. We generate 50 random 10 second slices from these 2 minutes and directly them for the human evaluations.
13


Reviews:
[{"review_id": "rylkOqJptr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work treats 2-D spectrogram as image and uses an autoregressive models which factorizes over both time and frequency dimensions. \n\nDetailed comments:\n\n- MelNet is not a \"fully end-to-end generative model of audio\". It generates the spectrogram and relies on other algorithmic component (Griffin-Lim or gradient-based inversion) to generate raw audio.\n\n- MelNet can model the long range structure for unconditional generation of speech, but its audio fidelity is not as good as autoregressive or non-autoregressive models on raw waveforms. The major reason is that MelNet discards the phase information which is useful for high-fidelity speech synthesis. It would be more interesting if MelNet jointly models the magnitude and phase information. \n\n- The mixture density networks are well known. One may omit the details (or put them in Appendix) in Section 3 for space reason. Overall, the paper is clearly written, but it can be shortened in several ways.\n\n- \"making the use of 2D convolution undesirable.\"\nIt's still unclear the conv2d is useful or undesirable for modeling spectrograms in generative tasks. Even in recognition tasks, I have seen different results from different papers for different settings,  e.g., In Deep Speech 2, conv2d is useful to reduce WER in ASR. \n\n- Missing connection in related work: previous conditional generation methods (e.g., Tacotron, Deep Voice 3) are autoregressive over time, but assume conditional independence over frequency bins. \n\n- There is TTS experiments on the demo website, but I didn't find any details. For example, where does the conditional information (**aligned** linguistic feature) come from?\n\nMy major concern is about the usefulness of the model:\n\n1) The unconditional speech generation is an uncommon & less useful task in general. If the task is purposely constructed, the learned representation is more useful than the generation itself (e.g., van den Oord et al. 2017). However, the authors have not demonstrated the usefulness of the learned representation for any downstream task. \n\n2) The MelNet is autoregressive over both time and frequency. Thus, it is as slow as autoregressive waveform models at synthesis with worse audio fidelity, which make it less preferred in potential TTS applications.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Skxk8C7CFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce MelNet, an autoregressive model of Mel-frequency scaled spectrograms. They convert audio into high resolution spectrograms to reduce the audio artifacts introduced by inverting spectrograms (here they use gradient-based inversion over Griffin-Lim). To improve modeling of long term dependencies, they perform multi-scale splitting of the spectrograms and maximize the likelihood at each scale (avoiding dominance of noise at higher resolutions). They condition generation at finer scales from coarser scales, enabling sampling through an ancestral process. The authors also highlight the difference between temporal and frequency dimensions, creating different conditioning stacks for the past in time vs. the \"past\" in frequency (lower frequencies), and mixing conditioning between the two stacks through layers of the network. Multilayer RNNs are used throughout the network and external conditioning is incorporated at the input. \n\nThe challenge the authors are attempting to address is modeling of audio structure on both long and short timescales. As the authors demonstrate with strong baselines, WaveNet models, while superior on fine-scale fidelity, fail to capture dynamics more than a couple hundred milliseconds. The experiments demonstrate improvements on state-of-the-art for unconditional generation on text-to-speech datasets (generating coherent words and phrases) and the MAESTRO piano dataset (generating sections with consistent dynamics/timing/motifs). The continuations of primed examples in both domains are particularly impressive qualitatively, as they maintain much of the character of the priming sample. Ablation experiments qualitatively demonstrate the importance of multi-scale modeling for unconditional generation. Human listener studies support the claims made from qualitative evaluation of long term structure.\n\nThis paper should be accepted because it represents a non-trivial adaptation of autoregressive modeling to handle multi-scale structure in audio. The baselines comparisons and strong, and experiments validate the claims of the paper. \n\nThat said, several things could be done to improve the clarity and significance of the paper.\n\n* While the network architecture is described in detail and some figures, the full network structure itself is non-trivial and still somewhat opaque from the plain text description. A schematic diagram of the full network architecture, even in the appendix, could help clarify how many layers are present connecting each component of the model, which would improve reproducibility.\n\n* The paper is a bit thin on metrics. Human listening studies compare long-term structure, but not short-scale fidelity. For TTS, there are clear artifacts from the spectrogram inversion process. Mean opinion scores on conditional samples could help to quantify the importance of each element of the network for audio quality. For instance, how does MOS compare between Griffin-Lim MelNet, Gradient Inversion MelNet, and WaveNet? How does MelNet compare to Linear scaled spectrograms?\n\n* Generating MelSpectrograms to model long-term structure is a fairly established technique, most notably employed by all of the Tacotron variants (https://google.github.io/tacotron/). These models are perhaps a more appropriate comparison for MelNet in many ways, and opt for spectrogram inversion by smaller WaveRNN models. One of the claims of the paper is that it is important to model the fine-scale structure of spectrograms, but it is not clear if that really is the case. A proper comparison to Tacotron models (where spectrograms are generated at the same resolution / the same inversion methods are used) would help clarify the importance of end-2-end training, vs. the learned inversion approach. \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1gKu8yCFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors present a new generative model for audio in the frequency domain to capture better the global structure of the signal. For this, they use an autoregressive  procedure combined with a multiscale generative model for two-dimensional time-frequency visual representation (STFT spectrogram). The proposed method is tested across a diverse set of audio generation tasks\n\nOverall, The idea of generating audio from 2D spectrogram is original but in my point of view the use of STFT is not appropriate in this context, especially with its lossy criterion. \n\nGiven the clarifications and the author’s responses below, I increased the score from 3 to 6. \n\n\n\nDetailed comments:\n\nPro: \n\nMitigate the problem of generating signal using only local dependencies (on a narrow time scale) and this by capturing high level dependency that emerges on larger timescale (several seconds) using spectrogram.\n\n\n\nCons: \n\n(1)The use of  STFT is not justified why not wavelet spectrogram to capture both scale and time?\n(2)It is still confusing how the use of high resolution spectrogram improve the lossy representation?\n(3)If you increase the STFT hope size you come back to the main problem that you are trying to resolve (i.e,  the bias towards capturing local dependencies) \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed an autoregressive model with a multiscale generative representation of the spectrograms to better modeling the long term dependencies in audio signals. The techniques developed in the paper are novel and interesting. The main concern is the validation of the method. The paper presented some human listening studies to compare long-term structure on unconditional samples, which as also mentioned by reviewers are not particularly useful. Including justifications on the usefulness of the learned representation for any downstream task would make the work much more solid. 
Decision: Reject