Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Overall, this is an interesting alternative the wavenet. 

In addition to other comments made so far -
It's not clear to me why up-sampling is needed: Why can't the high-frequency module just read the same (repeated) state of low frequency module?
And if you want to have a transformation there, why make it r linear projections? This seems both a redundant and weak modeling choice.  

Thanks.


review_title: upsampling

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper proposed a novel SampleRNN to directly model waveform signals and achieved better performance both in terms of objective test NLL and subjective A/B tests. 

As mentioned in the discussions, the current status of the paper lack plenty of details in describing their model. Hopefully, this will be addressed in the final version.

The authors attempted to compare with wavenet model, but they didn't manage to get a model better than the baseline LSTM-RNN, which makes all the comparisons to wavenets less convincing. Hence, instead of wasting time and space comparing to wavenet, detailing the proposed model would be better. 
review_title: null

final_score: rating: null
review_content: 1) What's the exact RNN used in the result comparison? In section 3, GRU and LSTM are mentioned as two variants of RNNs used. Which one produces the numbers presented in the results section?

2) The results show RNN is consistently better than Wavenet, and some cases by a large margin, which is contrary to what the Wavenet paper reports. The paper did mention in 3.1 the implementation is probably different for the original paper. But if wavenet numbers are not better than simple RNN, the experimental comparisons between SampleRNN and wavenet are hence untrustworthy. It leads to questions whether the theoretical benefits "In contrast to WaveNets, we have the ability to use information from arbitrary past by using stateful RNNs" is useful or not.

3) Reproducibility. The paper complained in section 3.1 "but owing to missing details of architecture and hyperparameters" that they cannot reproduce the wavenet model. In the paper, what recurrent unit (GRU or LSTM) and what's the configuration of those units (cell size etc) are used in the final SampleRNN that generated results? More details would be great. 
review_title: Wavenet vs. RNN comparison are completely different from the Wavenet paper

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper introduces SampleRNN, a hierarchical recurrent neural network model of raw audio. The model is trained end-to-end and evaluated using log-likelihood and by human judgement of unconditional samples, on three different datasets covering speech and music. This evaluation shows the proposed model to compare favourably to the baselines.

It is shown that the subsequence length used for truncated BPTT affects performance significantly, but interestingly, a subsequence length of 512 samples (~32 ms) is sufficient to get good results, even though the features of the data that are modelled span much longer timescales. This is an interesting and somewhat unintuitive result that I think warrants a bit more discussion.

The authors have attempted to reimplement WaveNet, an alternative model of raw audio that is fully convolutional. They were unable to reproduce the exact model architecture from the original paper, but have attempted to build an instance of the model with a receptive field of about 250ms that could be trained in a reasonable time using their computational resources, which is commendable.

The architecture of the Wavenet model is described in detail, but it found it challenging to find the same details for the proposed SampleRNN architecture (e.g. which value of "r" is used for the different tiers, how many units per layer, ...). I think a comparison in terms of computational cost, training time and number of parameters would also be very informative.

Surprisingly, Table 1 shows a vanilla RNN (LSTM) substantially outperforming this model in terms of likelihood, which is quite suspicious as LSTMs tend to have effective receptive fields of a few hundred timesteps at best. One would expect the much larger receptive field of the Wavenet model to be reflected in the likelihood scores to some extent. Similarly, Figure 3 shows the vanilla RNN outperforming the Wavenet reimplementation in human evaluation on the Blizzard dataset. This raises questions about the implementation of the latter. Some discussion about this result and whether the authors expected it or not would be very welcome.

Table 1 and Figure 4 also show the 2-tier SampleRNN outperforming the 3-tier model in terms of likelihood and human rating respectively, which is very counterintuitive as one would expect longer-range temporal correlations to be even more relevant for music than for speech. This is not discussed at all, I think it would be useful to comment on why this could be happening.

Overall, this an interesting attempt to tackle modelling very long sequences with long-range temporal correlations and the results are quite convincing, even if the same can't always be said of the comparison with the baselines. It would be interesting to see how the model performs for conditional generation, seeing as it can be more easily be objectively compared to models like Wavenet in that domain.



Other remarks:

- upsampling the output of the models is done with r separate linear projections. This choice of upsampling method is not motivated. Why not just use linear interpolation or nearest neighbour upsampling? What is the advantage of learning this operation? Don't the r linear projections end up learning largely the same thing, give or take some noise?

- The third paragraph of Section 2.1.1 indicates that 8-bit linear PCM was used. This is in contrast to Wavenet, for which an 8-bit mu-law encoding was used, and this supposedly improves the audio fidelity of the samples. Did you try this as well?

- Section 2.1 mentions the discretisation of the input and the use of a softmax to model this discretised input, without any reference to prior work that made the same observation. A reference is given in 2.1.1, but it should probably be moved up a bit to avoid giving the impression that this is a novel observation.

review_title: null

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Pros:
The authors are presenting an RNN-based alternative to wavenet, for generating audio a sample at a time.
RNNs are a natural candidate for this task so this is an interesting alternative. Furthermore the authors claim to make significant improvement in the quality of the produces samples.
Another novelty here is that they use a quantitative likelihood-based measure to assess them model, in addition to the AB human comparisons used in the wavenet work.

Cons:
The paper is lacking equations that detail the model. This can be remedied in the camera-ready version.
The paper is lacking detailed explanations of the modeling choices:
- It's not clear why an MLP is used in the bottom layer instead of (another) RNN.
- It's not clear why r linear projections are used for up-sampling, instead of feeding the same state to all r samples, or use a more powerful type of transformation. 
As the authors admit, their wavenet implementation is probably not as good as the original one, which makes the comparisons questionable. 

Despite the cons and given that more modeling details are provided, I think this paper will be a valuable contribution. 

review_title: Promising work, paper lacking details

final_score: rating: null
review_content: 1) For the onomatopoeia dataset, it is mentioned that the data is extremely unbalanced. What is the relevance of this given that the model is not trained to do classification? Did you train conditional models?

2) In the last part of Section 4 it is pointed out that the proposed model can use information from arbitrarily far back, but this conflicts with this statement from the introduction: "in practice it is a known problem that [RNNs do] not scale well at such a high temporal resolution as is found when generating acoustic signals one sample at a time." The multiscale structure mitigates this problem to an extent, but does not solve it completely: in practice the models will still have a finite memory horizon. Could the authors clarify these statements and resolve this apparent conflict?


review_title: time horizon

Meta-review: The reviewers were unanimous in their agreement about accepting this paper.
 Pros 
 - novel formulation that don't require sample by sample prediction
 - interesting results
 
 Cons
 - lack of details / explanation in the mathematical formulation / motivations for the model.
Decision: Accept (Poster)