Title: SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
Authors: Soroush Mehri
Kundan Kumar
Ishaan Gulrajani
Rithesh Kumar
Shubham Jain
Jose Sotelo
Aaron Courville
Yoshua Bengio
Venue: ICLR 2017
Keywords: Speech
Deep learning
Unsupervised Learning
Applications

Abstract: In this paper we propose a novel model for unconditional audio generation task that generates one audio sample at a time. We show that our model which profits from combining memory-less modules, namely autoregressive multilayer perceptron, and stateful recurrent neural networks in a hierarchical structure is de facto powerful to capture the underlying sources of variations in temporal domain for very long time on three datasets of different nature. Human evaluation on the generated samples indicate that our model is preferred over competing models. We also show how each component of the model contributes to the exhibited performance.

TL;DR: Novel model for unconditional audio generation task using hierarchical multi-scale RNNs and autoregressive MLP.

Review discussion:
Following is a link to the generated samples from our model and baseline models:
http://soundcloud.com/samplernn/sets
Code is also available on Github:
https://github.com/soroushmehr/sampleRNN_ICLR2017
1) What's the exact RNN used in the result comparison? In section 3, GRU and LSTM are mentioned as two variants of RNNs used. Which one produces the numbers presented in the results section?
2) The results show RNN is consistently better than Wavenet, and some cases by a large margin, which is contrary to what the Wavenet paper reports. The paper did mention in 3.1 the implementation is probably different for the original paper.
But if wavenet numbers are not better than simple RNN, the experimental comparisons between SampleRNN and wavenet are hence untrustworthy. It leads to questions whether the theoretical benefits "In contrast to WaveNets, we have the ability to use information from arbitrary past by using stateful RNNs" is useful or not.
3) Reproducibility. The paper complained in section 3.1 "but owing to missing details of architecture and hyperparameters" that they cannot reproduce the wavenet model. In the paper, what recurrent unit (GRU or LSTM) and what's the configuration of those units (cell size etc) are used in the final SampleRNN that generated results? More details would be great.
Thanks for your questions.
Re. 1) From hyper-parameter search, GRU showed the best performance (just slightly better than LSTM).
Re. 2) I would like to answer to this question in two parts:
A. The results in WaveNet paper are only subjective comparisons with models that have been trained with conditioning information while we are showing quantitative (NLL) and qualitative (AB test) results for unconditional models.
Most importantly, the baseline RNN that we are presenting is trained on quantized raw audio samples which is the same for all other compared models. The LSTM-RNN-based statistical parametric model [1, 2] cited in WaveNet paper is outputting acoustic features (conditioned on linguistic features), spectral enhancement was applied and afterward waveform was generated using Vocaine Vocoder.
So they are different despite their similarity in the names. The main purpose of reporting RNNs in our work as a baseline was to show that it can be improved upon by a hierarchical structure such as one presented in SampleRNN model.
B. We are aware of the fact that the quality of our WaveNet samples is not close to the ones from original paper.
However, with no code, details, nor even any performance measure reported, we spend more than enough time and resources on re-implementation that could have been easily spent on increasing the quality of presented work.
Hence, penalizing us solely because of that would be unfair and we hope that the publication of our paper might prompt them to share more details with the community.
Re. 3)  You can find the code and best found hyper-parameters in the following Github repo:
https://github.com/soroushmehr/sampleRNN_ICLR2017
[1] Zen, Heiga, and Haşim Sak. "Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis." In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4470-4474. IEEE, 2015.
[2] Zen, Heiga, Yannis Agiomyrgiannakis, Niels Egberts, Fergus Henderson, and Przemysław Szczepaniak. "Fast, compact, and high quality LSTM-RNN based statistical parametric speech synthesizers for mobile devices." arXiv preprint arXiv:1606.06061 (2016).
1) For the onomatopoeia dataset, it is mentioned that the data is extremely unbalanced. What is the relevance of this given that the model is not trained to do classification? Did you train conditional models?
2) In the last part of Section 4 it is pointed out that the proposed model can use information from arbitrarily far back, but this conflicts with this statement from the introduction: "in practice it is a known problem that [RNNs do] not scale well at such a high temporal resolution as is found when generating acoustic signals one sample at a time." The multiscale structure mitigates this problem to an extent, but does not solve it completely: in practice the models will still have a finite memory horizon. Could the authors clarify these statements and resolve this apparent conflict?
Thanks for reading our paper and your questions.
Re. 1.1) This is mostly a part of the description of the dataset. It affects the generalization and diversity of generated samples in a sense that model will favor the dominant category.
Re. 1.2) We do not have any conditional models.
Re. 2) Thanks for noting that. You are correct that it will mitigate this problem and not solve it. We will clarify this point in the next update along with the following experiment on a related note.
We trained our best model --SampleRNN (3-tier)-- on a dataset of 2 speakers, one male and one female, each with almost 10 hours of audio. We observed that it learned to stay consistent generating samples from the same speaker without having any knowledge about the the speaker ID or any other conditioning information. This effect is more apparent here in comparison to the unbalanced Onomatopoeia that sometimes mixes two different categories of sounds.
Another experiment was conducted to test the effect of memory and study the effective memory horizon. We inject 1 second of silence in the middle of sampling procedure in order to see if it will remember to generate from the same speaker or not. Initially when sampling we let the model generate 2 seconds of audio as it normally do. From 2 to 3 seconds instead of feeding back the generated sample at that timestep a silent token (zero amplitude) would be fed. From 3 to 5 seconds again we sample normally;
feeding back the generated token. We did classification based on mean fundamental frequency of speakers for the first and last 2 seconds. In 83% of samples SampleRNN will generate from the same person in two separate segments. This is in contrast to a model with fixed past window like WaveNet where injecting 16000 silent tokens (3. 3 times the receptive field size) is equivalent to generating from scratch which has 50% chance (assuming each 2-second segment is coherent and not a mixed sound of two speakers).
Overall, this is an interesting alternative the wavenet.
In addition to other comments made so far -
It's not clear to me why up-sampling is needed: Why can't the high-frequency module just read the same (repeated) state of low frequency module?
And if you want to have a transformation there, why make it r linear projections? This seems both a redundant and weak modeling choice.
Thanks.
Pros:
The authors are presenting an RNN-based alternative to wavenet, for generating audio a sample at a time.
RNNs are a natural candidate for this task so this is an interesting alternative.
Furthermore the authors claim to make significant improvement in the quality of the produces samples.
Another novelty here is that they use a quantitative likelihood-based measure to assess them model, in addition to the AB human comparisons used in the wavenet work.
Cons:
The paper is lacking equations that detail the model. This can be remedied in the camera-ready version.
The paper is lacking detailed explanations of the modeling choices:
- It's not clear why an MLP is used in the bottom layer instead of (another) RNN.
- It's not clear why r linear projections are used for up-sampling, instead of feeding the same state to all r samples, or use a more powerful type of transformation.
As the authors admit, their wavenet implementation is probably not as good as the original one, which makes the comparisons questionable.
Despite the cons and given that more modeling details are provided, I think this paper will be a valuable contribution.
Dear AnonReviewer2,
Thanks for reading the paper.
I would like to respond to the cons section of the review.
- Re. lacking equations: we are in the process of updating the paper.
- Re. MLP vs RNN: Modelling nearby samples is relatively easy and adding another RNN is making the model more complex and slower to train; there is no need to run an RNN if it is getting input from couple of past samples, there is no long-term dependency there.
Besides that, if a memoryless module can cope with that, there is no reason theoretically to think that a RNN cannot accomplish the same.
- Re. upsampling:
> Why can't the high-frequency module just read the same (repeated) state of low frequency module?
The high-frequency module can certainly do that; we would consider this still a sort of upsampling (nearest-neighbor).
> And if you want to have a transformation there, why make it r linear projections? This seems both a redundant and weak modeling choice.
One argument for r separate projections (c_1=W_1*h_t,...,c_r=W_r*h_t) is that the RNN, roughly, should be modeling p(x_{t+1}...x_{t+r}|x0...xt). Considering the case where the lower module is an MLP which is run r times with the same state vector as input, there's no easy way for it to know which part of that state vector to pay attention to (i.e. what the current timestep is) otherwise.
Also, our approach of upsampling with r linear projections is exactly equivalent to upsampling by adding zeros and then applying a linear convolution. This is sometimes called "perforated" upsampling in the context of CNNs. It was first demonstrated to work well in [1], and is a fairly common upsampling technique.
[1] Dosovitskiy, Alexey, Jost Springenberg, Maxim Tatarchenko, and Thomas Brox. "Learning to Generate Chairs, Tables and Cars with Convolutional Networks." (2016).
The paper introduces SampleRNN, a hierarchical recurrent neural network model of raw audio. The model is trained end-to-end and evaluated using log-likelihood and by human judgement of unconditional samples, on three different datasets covering speech and music. This evaluation shows the proposed model to compare favourably to the baselines.
It is shown that the subsequence length used for truncated BPTT affects performance significantly, but interestingly, a subsequence length of 512 samples (~32 ms) is sufficient to get good results, even though the features of the data that are modelled span much longer timescales. This is an interesting and somewhat unintuitive result that I think warrants a bit more discussion.
The authors have attempted to reimplement WaveNet, an alternative model of raw audio that is fully convolutional. They were unable to reproduce the exact model architecture from the original paper, but have attempted to build an instance of the model with a receptive field of about 250ms that could be trained in a reasonable time using their computational resources, which is commendable.
The architecture of the Wavenet model is described in detail, but it found it challenging to find the same details for the proposed SampleRNN architecture (e.g. which value of "r" is used for the different tiers, how many units per layer, ...). I think a comparison in terms of computational cost, training time and number of parameters would also be very informative.
Surprisingly, Table 1 shows a vanilla RNN (LSTM) substantially outperforming this model in terms of likelihood, which is quite suspicious as LSTMs tend to have effective receptive fields of a few hundred timesteps at best. One would expect the much larger receptive field of the Wavenet model to be reflected in the likelihood scores to some extent.
Similarly, Figure 3 shows the vanilla RNN outperforming the Wavenet reimplementation in human evaluation on the Blizzard dataset. This raises questions about the implementation of the latter. Some discussion about this result and whether the authors expected it or not would be very welcome.
Table 1 and Figure 4 also show the 2-tier SampleRNN outperforming the 3-tier model in terms of likelihood and human rating respectively, which is very counterintuitive as one would expect longer-range temporal correlations to be even more relevant for music than for speech. This is not discussed at all, I think it would be useful to comment on why this could be happening.
Overall, this an interesting attempt to tackle modelling very long sequences with long-range temporal correlations and the results are quite convincing, even if the same can't always be said of the comparison with the baselines. It would be interesting to see how the model performs for conditional generation, seeing as it can be more easily be objectively compared to models like Wavenet in that domain.
Other remarks:
- upsampling the output of the models is done with r separate linear projections. This choice of upsampling method is not motivated. Why not just use linear interpolation or nearest neighbour upsampling? What is the advantage of learning this operation? Don't the r linear projections end up learning largely the same thing, give or take some noise?
- The third paragraph of Section 2.1.1 indicates that 8-bit linear PCM was used. This is in contrast to Wavenet, for which an 8-bit mu-law encoding was used, and this supposedly improves the audio fidelity of the samples. Did you try this as well?
- Section 2.1 mentions the discretisation of the input and the use of a softmax to model this discretised input, without any reference to prior work that made the same observation. A reference is given in 2.1.1, but it should probably be moved up a bit to avoid giving the impression that this is a novel observation.
Dear AnonReviewer3,
Thank you for the detailed review and comments.
I will try to answer your points one by one in the following sections:
> "a subsequence length of 512 samples (~32 ms) is sufficient to get good results". This is an interesting and somewhat unintuitive result that I think warrants a bit more discussion.
- To make sure that the model can properly model larger context it is necessary to carry the information from the past. In this model this is mostly due to passing the relevant information (e.g. phoneme-like sounds) from one 32ms chunk to the next one (stateful RNN) during Truncated BPTT. It is that some part of the memory of each tier is responsible to keep track of the broader context.
> Lacking details e.g. which value of "r" is used for the different tiers, how many units per layer, ...
- Added to the recent version. Please see the recent comment for changelog.
Specifically, frame size values are FS_1 = FS_2 = 2 and  FS_3 = 8. 1024 GRU units per layer were showed the best performance and each tier has one layer RNN.
> "I think a comparison in terms of computational cost, training time and number of parameters would also be very informative."
- Comparison: (Our re-implementation of WaveNet) vs. (SampleRNN)
Number of parameters (1M) vs. (21M) *
Approximated training time memory footprint (10GB) vs. (5GB)
Mini-batch size (8) vs (128)
Training time (7 days) vs. (3 days)
* We made sure to make the WaveNet model as big as possible while still fitting in one (large) GPU per model.
> "Surprisingly, Table 1 shows a vanilla RNN (LSTM) substantially outperforming this model [WaveNet] in terms of likelihood, which is quite suspicious as LSTMs tend to have effective receptive fields of a few hundred timesteps at best. One would expect the much larger receptive field of the Wavenet model to be reflected in the likelihood scores to some extent.
Similarly, Figure 3 shows the vanilla RNN outperforming the Wavenet reimplementation in human evaluation on the Blizzard dataset. This raises questions about the implementation of the latter. Some discussion about this result and whether the authors expected it or not would be very welcome."
- When we were replicating their results, although we were trying our best to replicate them, we had no frame of reference to compare to except for the unconditional samples. The best speculation as why LSTM is getting better numbers is similar to what I have stated in response to your first question: it is a combination of TBPTT and stateful RNN when training. This seems to be a great approximation of training on really long sequences in audio data.
> Why 2-tier is outperforming the 3-tier model for music?
- We did not expect that, but for any dataset and architecture structure, there is an optimal depth. Considering that this is a deep RNN (which introduces a form of recurrent depth, here very large) and the hypothesis that it is difficult to train such architectures in the first place, it is possible that alternative training procedures could yield better results with a deeper model.
> "This choice of upsampling method is not motivated. Why not just use linear interpolation or nearest neighbour upsampling? What is the advantage of learning this operation? Don't the r linear projections end up learning largely the same thing, give or take some noise?"
- The same argument is applicable to upsampling in CNNs [1] but it works better there.
So we decided to borrow the same technique. There should be no problem using other methods.
However if this module helps the final model, even slightly, and takes away the burden of upsampling from each individual tier, there is no reason not to use it.
In addition, for small upsampling ratio this will learn the same thing give or take some noise but as the ratio grows there will be more structure which can be captured by a learned operation.
> The third paragraph of Section 2.1.1 indicates that 8-bit linear PCM was used. This is in contrast to Wavenet, for which an 8-bit mu-law encoding was used, and this supposedly improves the audio fidelity of the samples. Did you try this as well?
- Mu-law was recently tested. This is a harder representation to model and takes more time to train on. We can get good results that you can find in (https://soundcloud.com/samplernn/sets/mu-law). By the time WaveNet came out we were done by most of the experiments so due to time constraints we did not attempt to re-do everything.
> Section 2.1 mentions the discretisation of the input and the use of a softmax to model this discretised input, without any reference to prior work that made the same observation. A reference is given in 2.1.1, but it should probably be moved up a bit to avoid giving the impression that this is a novel observation.
- Citation was moved up. Thanks for noting.
[1] Dosovitskiy, Alexey, Jost Springenberg, Maxim Tatarchenko, and Thomas Brox. "Learning to Generate Chairs, Tables and Cars with Convolutional Networks." (2016).
The paper proposed a novel SampleRNN to directly model waveform signals and achieved better performance both in terms of objective test NLL and subjective A/B tests.
As mentioned in the discussions, the current status of the paper lack plenty of details in describing their model. Hopefully, this will be addressed in the final version.
The authors attempted to compare with wavenet model, but they didn't manage to get a model better than the baseline LSTM-RNN, which makes all the comparisons to wavenets less convincing.
Hence, instead of wasting time and space comparing to wavenet, detailing the proposed model would be better.
Thanks for reviewing our paper. It is much appreciated.
Please find the top latest comment for the changelog of the recent revision.
Modifications to the last revision:
- Section 2, SampleRNN model. Better model description, changed order of sub-sections, explained the upsampling method more clearly.
- Added a paragraph (before Section 3.1) detailing the training procedure and hyper-parameters.
- Added subsection 3.3 Quantifying information retention. (See authors' response to AnonReviewer3 on Dec. 2 titled "time horizon")
- Minor modifications (typo, citation, rephrasing, etc.)

Meta-review:
role: PC
data: comment: The reviewers were unanimous in their agreement about accepting this paper.
 Pros 
 - novel formulation that don't require sample by sample prediction
 - interesting results
 
 Cons
 - lack of details / explanation in the mathematical formulation / motivations for the model.

Decision: Accept (Poster)