MelNet: A Generative Model for Audio in the Frequency Domain

Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps . While long-range dependencies are difficult to model directly in the time domain , we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms . By leveraging this representational advantage , in conjunction with a highly expressive probabilistic model and a multiscale generation procedure , we design a model capable of generating high-fidelity audio samples which capture structure at timescales which time-domain models have yet to achieve . We demonstrate that our model captures longer-range dependencies than time-domain models such as WaveNet across a diverse set of unconditional generation tasks , including single-speaker speech generation , multi-speaker speech generation , and music generation . 1 INTRODUCTION . Audio waveforms have complex structure at drastically varying timescales , which presents a challenge for generative models . Local structure must be captured to produce high-fidelity audio , while longrange dependencies spanning tens of thousands of timesteps must be captured to generate audio which is globally consistent . Existing generative models of waveforms such as WaveNet ( van den Oord et al. , 2016a ) and SampleRNN ( Mehri et al. , 2016 ) are well-adapted to model local dependencies , but as these models typically only backpropagate through a fraction of a second , they are unable to capture high-level structure that emerges on the scale of several seconds . We introduce a generative model for audio which captures longer-range dependencies than existing end-to-end models . We primarily achieve this by modelling 2D time-frequency representations such as spectrograms rather than 1D time-domain waveforms ( Figure 1 ) . The temporal axis of a spectrogram is orders of magnitude more compact than that of a waveform , meaning dependencies that span tens of thousands of timesteps in waveforms only span hundreds of timesteps in spectrograms . In practice , this enables our spectrogram models to generate unconditional speech and music samples with consistency over multiple seconds whereas time-domain models must be conditioned on intermediate features to capture structure at similar timescales . Modelling spectrograms can simplify the task of capturing global structure , but can weaken a model ’ s ability to capture local characteristics that correlate with audio fidelity . Producing high-fidelity audio has been challenging for existing spectrogram models , which we attribute to the lossy nature of spectrograms and oversmoothing artifacts which result from insufficiently expressive models . To reduce information loss , we model high-resolution spectrograms which have the same dimensionality as their corresponding time-domain signals . To limit oversmoothing , we use a highly expressive autoregressive model which factorizes the distribution over both the time and frequency dimensions . Modelling both fine-grained details and high-level structure in high-dimensional distributions is known to be challenging for autoregressive models . To capture both local and global structure in spectrograms with hundreds of thousands of dimensions , we employ a multiscale approach which generates spectrograms in a coarse-to-fine manner . A low-resolution , subsampled spectrogram that captures high-level structure is generated initially , followed by an iterative upsampling procedure that adds high-resolution details . Combining these representational and modelling techniques yields a highly expressive and broadly applicable generative model of audio . Our contributions are are as follows : • We introduce MelNet , a generative model for spectrograms which couples a fine-grained autoregressive model and a multiscale generation procedure to jointly capture local and global structure . • We show that MelNet is able to model longer-range dependencies than existing time-domain models . Additionally , we include an ablation to demonstrate that multiscale modelling is essential for modelling long-range dependencies . • We demonstrate that MelNet is broadly applicable to a variety of audio generation tasks , including unconditional speech and music generation . Furthermore , MelNet is able to model highly multimodal data such as multi-speaker and multilingual speech . 2 PRELIMINARIES . We briefly present background regarding spectral representations of audio . Audio is represented digitally as a one-dimensional , discrete-time signal y = ( y1 , . . . , yn ) . Existing generative models for audio have predominantly focused on modelling these time-domain signals directly . We instead model spectrograms , which are two-dimensional time-frequency representations which contain information about how the frequency content of an audio signal varies through time . Spectrograms are computed by taking the squared magnitude of the short-time Fourier transform ( STFT ) of a time-domain signal , i.e . x = ‖STFT ( y ) ‖2 . The value of xij ( referred to as amplitude or energy ) corresponds to the squared magnitude of the jth element of the frequency response at timestep i . Each slice xi , ∗ is referred to as a frame . We assume a time-major ordering , but following convention , all figures are displayed transposed and with the frequency axis inverted . Time-frequency representations such as spectrograms highlight how the tones and pitches within an audio signal vary through time . Such representations are closely aligned with how humans perceive audio . To further align these representations with human perception , we convert the frequency axis to the Mel scale and apply an elementwise logarithmic rescaling of the amplitudes . Roughly speaking , the Mel transformation aligns the frequency axis with human perception of pitch and the logarithmic rescaling aligns the amplitude axis with human perception of loudness . Spectrograms are lossy representations of their corresponding time-domain signals . The Mel transformation discards frequency information and the removal of the STFT phase discards temporal information . When recovering a time-domain signal from a spectrogram , this information loss manifests as distortion in the recovered signal . To minimize these artifacts and improve the fidelity of generated audio , we model high-resolution spectrograms . The temporal resolution of a spectrogram can be increased by decreasing the STFT hop size , and the frequency resolution can be increased by increasing the number of Mel channels . Generated spectrograms are converted back to time-domain signals using classical spectrogram inversion algorithms . We experiment with both Griffin-Lim ( Griffin & Lim , 1984 ) and a gradient-based inversion algorithm ( Decorsière et al. , 2015 ) , and ultimately use the latter as it generally produced audio with fewer artifacts . 3 PROBABILISTIC MODEL . We use an autoregressive model which factorizes the joint distribution over a spectrogram x as a product of conditional distributions . Given an ordering of the dimensions of x , we define the context x < ij as the elements of x that precede xij . We default to a row-major ordering which proceeds through each frame xi , ∗ from low to high frequency , before progressing to the next frame . The joint density is factorized as p ( x ) = ∏ i ∏ j p ( xij | x < ij ; θij ) , ( 1 ) where θij parameterizes a univariate density over xij . We model each factor distribution as a Gaussian mixture model with K components . Thus , θij consists of 3K parameters corresponding to means { µijk } Kk=1 , standard deviations { σijk } Kk=1 , and mixture coefficients { πijk } Kk=1 . The resulting factor distribution can then be expressed as p ( xij | x < ij ; θij ) = K∑ k=1 πijk N ( xij ; µijk , σijk ) . ( 2 ) Following the work on Mixture Density Networks ( Bishop , 1994 ) and their application to autoregressive models ( Graves , 2013 ) , θij is modelled as the output of a neural network and computed as a function of the context x < ij . Precisely , for some network f with parameters ψ , we have θij = f ( x < ij ; ψ ) . A maximum-likelihood estimate for the network parameters is computed by minimizing the negative log-likelihood via gradient descent . To ensure that the network output parameterizes a valid Gaussian mixture model , the network first computes unconstrained parameters { µ̂ijk , σ̂ijk , π̂ijk } Kk=1 as a vector θ̂ij ∈ R3K , and enforces constraints on θij by applying the following transformations : µijk = µ̂ijk ( 3 ) σijk = exp ( σ̂ijk ) ( 4 ) πijk = exp ( π̂ijk ) ∑K k=1 exp ( π̂ijk ) . ( 5 ) These transformations ensure the standard deviations σijk are positive and the mixture coefficients πijk sum to one . 4 NETWORK ARCHITECTURE . To model the distribution in an autoregressive manner , we design a network which computes the distribution over xij as a function of the context x < ij . The network architecture draws inspiration from existing autoregressive models for images ( Theis & Bethge , 2015 ; van den Oord et al. , 2016c ; b ; Chen et al. , 2017 ; Salimans et al. , 2017 ; Parmar et al. , 2018 ; Child et al. , 2019 ) . In the same way that these models estimate a distribution pixel-by-pixel over the spatial dimensions of an image , our model estimates a distribution element-by-element over the time and frequency dimensions of a spectrogram . A noteworthy distinction is that spectrograms are not invariant to translation along the frequency axis , making 2D convolution less desirable than other 2D network primitives which do not assume invariance . Utilizing multidimensional recurrence instead of 2D convolution has been shown to be beneficial when modelling spectrograms in discriminative settings ( Li et al. , 2016 ; Sainath & Li , 2016 ) , which motivates our use of an entirely recurrent architecture . Similar to Gated PixelCNN ( van den Oord et al. , 2016b ) , the network has multiple stacks of computation . These stacks extract features from different segments of the input to collectively summarize the full context x < ij : • The time-delayed stack computes features which aggregate information from all previous frames x < i , ∗ . • The frequency-delayed stack utilizes all preceding elements within a frame , xi , < j , as well as the outputs of the time-delayed stack , to summarize the full context x < ij . The stacks are connected at each layer of the network , meaning that the features generated by layer l of the time-delayed stack are used as input to layer l of the frequency-delayed stack . To facilitate the training of deeper networks , both stacks use residual connections ( He et al. , 2016 ) . The outputs of the final layer of the frequency-delayed stack are used to compute the unconstrained parameters θ̂ . 4.1 TIME-DELAYED STACK . The time-delayed stack utilizes multiple layers of multidimensional RNNs to extract features from x < i , ∗ , the two-dimensional region consisting of all frames preceding xij . Each multidimensional RNN is composed of three one-dimensional RNNs : one which runs forwards along the frequency axis , one which runs backwards along the frequency axis , and one which runs forwards along the time axis . Each RNN runs along each slice of a given axis , as shown in Figure 2 . The output of each layer of the time-delayed stack is the concatenation of the three RNN hidden states . We denote the function computed at layer l of the time-delayed stack ( three RNNs followed by concatenation ) as F tl . At each layer , the time-delayed stack uses the feature map from the previous layer , ht [ l−1 ] , to compute the subsequent feature mapF tl ( ht [ l−1 ] ) which consists of the three concatenated RNN hidden states . When using residual connections , the computation of ht [ l ] from ht [ l−1 ] becomes htij [ l ] =W t l F tl ( ht [ l − 1 ] ) ij + htij [ l − 1 ] . ( 6 ) To ensure the output htij [ l ] is only a function of frames which lie in the context x < ij , the inputs to the time-delayed stack are shifted backwards one step in time : htij [ 0 ] =W t 0xi−1 , j .

Summary: The authors introduce MelNet, an autoregressive model of Mel-frequency scaled spectrograms. They convert audio into high resolution spectrograms to reduce the audio artifacts introduced by inverting spectrograms (here they use gradient-based inversion over Griffin-Lim). To improve modeling of long term dependencies, they perform multi-scale splitting of the spectrograms and maximize the likelihood at each scale (avoiding dominance of noise at higher resolutions). They condition generation at finer scales from coarser scales, enabling sampling through an ancestral process. The authors also highlight the difference between temporal and frequency dimensions, creating different conditioning stacks for the past in time vs. the "past" in frequency (lower frequencies), and mixing conditioning between the two stacks through layers of the network. Multilayer RNNs are used throughout the network and external conditioning is incorporated at the input.