MelNet: A Generative Model for Audio in the Frequency Domain

Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps . While long-range dependencies are difficult to model directly in the time domain , we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms . By leveraging this representational advantage , in conjunction with a highly expressive probabilistic model and a multiscale generation procedure , we design a model capable of generating high-fidelity audio samples which capture structure at timescales which time-domain models have yet to achieve . We demonstrate that our model captures longer-range dependencies than time-domain models such as WaveNet across a diverse set of unconditional generation tasks , including single-speaker speech generation , multi-speaker speech generation , and music generation . 1 INTRODUCTION . Audio waveforms have complex structure at drastically varying timescales , which presents a challenge for generative models . Local structure must be captured to produce high-fidelity audio , while longrange dependencies spanning tens of thousands of timesteps must be captured to generate audio which is globally consistent . Existing generative models of waveforms such as WaveNet ( van den Oord et al. , 2016a ) and SampleRNN ( Mehri et al. , 2016 ) are well-adapted to model local dependencies , but as these models typically only backpropagate through a fraction of a second , they are unable to capture high-level structure that emerges on the scale of several seconds . We introduce a generative model for audio which captures longer-range dependencies than existing end-to-end models . We primarily achieve this by modelling 2D time-frequency representations such as spectrograms rather than 1D time-domain waveforms ( Figure 1 ) . The temporal axis of a spectrogram is orders of magnitude more compact than that of a waveform , meaning dependencies that span tens of thousands of timesteps in waveforms only span hundreds of timesteps in spectrograms . In practice , this enables our spectrogram models to generate unconditional speech and music samples with consistency over multiple seconds whereas time-domain models must be conditioned on intermediate features to capture structure at similar timescales . Modelling spectrograms can simplify the task of capturing global structure , but can weaken a model ’ s ability to capture local characteristics that correlate with audio fidelity . Producing high-fidelity audio has been challenging for existing spectrogram models , which we attribute to the lossy nature of spectrograms and oversmoothing artifacts which result from insufficiently expressive models . To reduce information loss , we model high-resolution spectrograms which have the same dimensionality as their corresponding time-domain signals . To limit oversmoothing , we use a highly expressive autoregressive model which factorizes the distribution over both the time and frequency dimensions . Modelling both fine-grained details and high-level structure in high-dimensional distributions is known to be challenging for autoregressive models . To capture both local and global structure in spectrograms with hundreds of thousands of dimensions , we employ a multiscale approach which generates spectrograms in a coarse-to-fine manner . A low-resolution , subsampled spectrogram that captures high-level structure is generated initially , followed by an iterative upsampling procedure that adds high-resolution details . Combining these representational and modelling techniques yields a highly expressive and broadly applicable generative model of audio . Our contributions are are as follows : • We introduce MelNet , a generative model for spectrograms which couples a fine-grained autoregressive model and a multiscale generation procedure to jointly capture local and global structure . • We show that MelNet is able to model longer-range dependencies than existing time-domain models . Additionally , we include an ablation to demonstrate that multiscale modelling is essential for modelling long-range dependencies . • We demonstrate that MelNet is broadly applicable to a variety of audio generation tasks , including unconditional speech and music generation . Furthermore , MelNet is able to model highly multimodal data such as multi-speaker and multilingual speech . 2 PRELIMINARIES . We briefly present background regarding spectral representations of audio . Audio is represented digitally as a one-dimensional , discrete-time signal y = ( y1 , . . . , yn ) . Existing generative models for audio have predominantly focused on modelling these time-domain signals directly . We instead model spectrograms , which are two-dimensional time-frequency representations which contain information about how the frequency content of an audio signal varies through time . Spectrograms are computed by taking the squared magnitude of the short-time Fourier transform ( STFT ) of a time-domain signal , i.e . x = ‖STFT ( y ) ‖2 . The value of xij ( referred to as amplitude or energy ) corresponds to the squared magnitude of the jth element of the frequency response at timestep i . Each slice xi , ∗ is referred to as a frame . We assume a time-major ordering , but following convention , all figures are displayed transposed and with the frequency axis inverted . Time-frequency representations such as spectrograms highlight how the tones and pitches within an audio signal vary through time . Such representations are closely aligned with how humans perceive audio . To further align these representations with human perception , we convert the frequency axis to the Mel scale and apply an elementwise logarithmic rescaling of the amplitudes . Roughly speaking , the Mel transformation aligns the frequency axis with human perception of pitch and the logarithmic rescaling aligns the amplitude axis with human perception of loudness . Spectrograms are lossy representations of their corresponding time-domain signals . The Mel transformation discards frequency information and the removal of the STFT phase discards temporal information . When recovering a time-domain signal from a spectrogram , this information loss manifests as distortion in the recovered signal . To minimize these artifacts and improve the fidelity of generated audio , we model high-resolution spectrograms . The temporal resolution of a spectrogram can be increased by decreasing the STFT hop size , and the frequency resolution can be increased by increasing the number of Mel channels . Generated spectrograms are converted back to time-domain signals using classical spectrogram inversion algorithms . We experiment with both Griffin-Lim ( Griffin & Lim , 1984 ) and a gradient-based inversion algorithm ( Decorsière et al. , 2015 ) , and ultimately use the latter as it generally produced audio with fewer artifacts . 3 PROBABILISTIC MODEL . We use an autoregressive model which factorizes the joint distribution over a spectrogram x as a product of conditional distributions . Given an ordering of the dimensions of x , we define the context x < ij as the elements of x that precede xij . We default to a row-major ordering which proceeds through each frame xi , ∗ from low to high frequency , before progressing to the next frame . The joint density is factorized as p ( x ) = ∏ i ∏ j p ( xij | x < ij ; θij ) , ( 1 ) where θij parameterizes a univariate density over xij . We model each factor distribution as a Gaussian mixture model with K components . Thus , θij consists of 3K parameters corresponding to means { µijk } Kk=1 , standard deviations { σijk } Kk=1 , and mixture coefficients { πijk } Kk=1 . The resulting factor distribution can then be expressed as p ( xij | x < ij ; θij ) = K∑ k=1 πijk N ( xij ; µijk , σijk ) . ( 2 ) Following the work on Mixture Density Networks ( Bishop , 1994 ) and their application to autoregressive models ( Graves , 2013 ) , θij is modelled as the output of a neural network and computed as a function of the context x < ij . Precisely , for some network f with parameters ψ , we have θij = f ( x < ij ; ψ ) . A maximum-likelihood estimate for the network parameters is computed by minimizing the negative log-likelihood via gradient descent . To ensure that the network output parameterizes a valid Gaussian mixture model , the network first computes unconstrained parameters { µ̂ijk , σ̂ijk , π̂ijk } Kk=1 as a vector θ̂ij ∈ R3K , and enforces constraints on θij by applying the following transformations : µijk = µ̂ijk ( 3 ) σijk = exp ( σ̂ijk ) ( 4 ) πijk = exp ( π̂ijk ) ∑K k=1 exp ( π̂ijk ) . ( 5 ) These transformations ensure the standard deviations σijk are positive and the mixture coefficients πijk sum to one . 4 NETWORK ARCHITECTURE . To model the distribution in an autoregressive manner , we design a network which computes the distribution over xij as a function of the context x < ij . The network architecture draws inspiration from existing autoregressive models for images ( Theis & Bethge , 2015 ; van den Oord et al. , 2016c ; b ; Chen et al. , 2017 ; Salimans et al. , 2017 ; Parmar et al. , 2018 ; Child et al. , 2019 ) . In the same way that these models estimate a distribution pixel-by-pixel over the spatial dimensions of an image , our model estimates a distribution element-by-element over the time and frequency dimensions of a spectrogram . A noteworthy distinction is that spectrograms are not invariant to translation along the frequency axis , making 2D convolution less desirable than other 2D network primitives which do not assume invariance . Utilizing multidimensional recurrence instead of 2D convolution has been shown to be beneficial when modelling spectrograms in discriminative settings ( Li et al. , 2016 ; Sainath & Li , 2016 ) , which motivates our use of an entirely recurrent architecture . Similar to Gated PixelCNN ( van den Oord et al. , 2016b ) , the network has multiple stacks of computation . These stacks extract features from different segments of the input to collectively summarize the full context x < ij : • The time-delayed stack computes features which aggregate information from all previous frames x < i , ∗ . • The frequency-delayed stack utilizes all preceding elements within a frame , xi , < j , as well as the outputs of the time-delayed stack , to summarize the full context x < ij . The stacks are connected at each layer of the network , meaning that the features generated by layer l of the time-delayed stack are used as input to layer l of the frequency-delayed stack . To facilitate the training of deeper networks , both stacks use residual connections ( He et al. , 2016 ) . The outputs of the final layer of the frequency-delayed stack are used to compute the unconstrained parameters θ̂ . 4.1 TIME-DELAYED STACK . The time-delayed stack utilizes multiple layers of multidimensional RNNs to extract features from x < i , ∗ , the two-dimensional region consisting of all frames preceding xij . Each multidimensional RNN is composed of three one-dimensional RNNs : one which runs forwards along the frequency axis , one which runs backwards along the frequency axis , and one which runs forwards along the time axis . Each RNN runs along each slice of a given axis , as shown in Figure 2 . The output of each layer of the time-delayed stack is the concatenation of the three RNN hidden states . We denote the function computed at layer l of the time-delayed stack ( three RNNs followed by concatenation ) as F tl . At each layer , the time-delayed stack uses the feature map from the previous layer , ht [ l−1 ] , to compute the subsequent feature mapF tl ( ht [ l−1 ] ) which consists of the three concatenated RNN hidden states . When using residual connections , the computation of ht [ l ] from ht [ l−1 ] becomes htij [ l ] =W t l F tl ( ht [ l − 1 ] ) ij + htij [ l − 1 ] . ( 6 ) To ensure the output htij [ l ] is only a function of frames which lie in the context x < ij , the inputs to the time-delayed stack are shifted backwards one step in time : htij [ 0 ] =W t 0xi−1 , j .

Summary: In this paper the authors present a new generative model for audio in the frequency domain to capture better the global structure of the signal. For this, they use an autoregressive  procedure combined with a multiscale generative model for two-dimensional time-frequency visual representation (STFT spectrogram). The proposed method is tested across a diverse set of audio generation tasks