Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents HybridNet, a neural speech (and other audio) synthesis system (vocoder) that combines the popular and effective WaveNet model with an LSTM with the goal of offering a model with faster inference-time audio generation.

Summary: The proposed model, HybridNet is a fairly straightforward variation of WaveNet and thus the paper offers a relatively low novelty. There is also a lack of detail regarding the human judgement experiments that make the significance of the results difficult to interpret. 

Low novelty of approach / impact assessment:
The proposed model is based closely on WaveNet, an existing state-of-the-art vocoder model. The proposal here is to extend WaveNet to include an LSTM that will generate samples between WaveNet samples -- thus allowing WaveNet to sample at a lower sample frequency. WaveNet is known for being relatively slow at test-time generation time, thus allowing it to run at a lower sample frequency should decrease generation time. The introduction of a local LSTM is perhaps not a sufficiently significant innovation. 

Another issue that lowers the assessment of the likely impact of this paper is that there are already a number of alternative mechanism to deal with the sampling speed of WaveNet. In particular, the cited method of Ramachandran et al (2017) uses caching and other tricks to achieve a speed up of 21 times over WaveNet (compared to the 2-4 times speed up of the proposed method). The authors suggest that these are orthogonal strategies that can be combined, but the combination is not attempted in this paper. There are also other methods such as sampleRNN (Mehri et al. 2017) that are faster than WaveNet at inference time. The authors do not compare to this model.

Inappropriate evaluation:
While the model is motivated by the need to reduce the generation of WaveNet sampling, the evaluation is largely based on the quality of the sampling rather than the speed of sampling. The results are roughly calibrated to demonstrate that HybridNet produces higher quality samples when (roughly) adjusted for sampling time. The more appropriate basis of comparison is to compare sample time as a function of sample quality. 

Experiments:
Few details are provided regarding the human judgment experiments with Mechanical Turkers. As a result it is difficulty to assess the appropriateness of the evaluation and therefore the significance of the findings. I would also be much more comfortable with this quality assessment if I was able to hear the samples for myself and compare the quality of the WaveNet samples with HybridNet samples. I will also like to compare the WaveNet samples generated by the authors' implementation with the WaveNet samples posted by van den Oord et al (2017).  


Minor comments / questions:

How, specifically, is validation error defined in the experiments? 

There are a few language glitches distributed throughout the paper.  

review_title: Right name. Low innovation. Samples please!

final_score: rating: 6: Marginally above acceptance threshold
review_content: TL;DR of paper: for sequential prediction, in order to scale up the model size without increasing inference time, use a model that predicts multiple timesteps at once. In this case, use an LSTM on top of a Wavenet for audio synthesis, where the LSTM predicts N steps for every Wavenet forward pass. The main result is being able to train bigger models, by increasing Wavenet depth, without increasing inference time.

The idea is simple and intuitive. I'm interested in seeing how well this approach can generalize to other sequential prediction domains. I suspect that it's easier in the waveform case because neighboring samples are highly correlated. I am surprised by how much an improvement 

However, there are a number of important design decisions that are glossed over in the paper. Here are a few that I am wondering about:
* How well do other multi-step decoders do? For example, another natural choice is using transposed convolutions to upsample multiple timesteps. Fully connected layers? How does changing the number of LSTM layers affect performance?
* Why does the Wavenet output a single timestep? Why not just have the multi-step decoder output all the timesteps?
* How much of a boost does the separate training give over joint training? If you used the idea suggested in the previous point, you wouldn't need this separate training scheme.
* How does performance vary over changing the number of steps the multi-step decoder outputs?

The paper also reads like it was hastily written, so please go back and fix the rough edges.

Right now, the paper feels too coupled to the existing Deep Voice 2 system. As a research paper, it is lacking important ablations. I'll be happy to increase my score if more experiments and results are provided.
review_title: Good results but lacking details about design decisions

final_score: rating: 4: Ok but not good enough - rejection
review_content: By generating multiple samples at once with the LSTM, the model is introducing some independence assumptions between samples that are from neighbouring windows and are not conditionally independent given the context produced by Wavenet. This reduces significantly the generality of the proposed technique.

Pros:
- Attempting to solve the important problem of speeding up autoregressive generation.
- Clarity of the write-up is OK, although it could use some polishing in some parts.
- The work is in the right direction, but the paucity of results and lack of thoroughness reduces somewhat the work's overall significance.

Cons:
- The proposed technique is not particularly novel and it is not clear whether the technique can be used to get speed-ups beyond 2x - something that is important for real-world deployment of Wavenet.
- The amount of innovation is on the low side, as it involves mostly just fairly minor architectural changes.
- The absolute results are not that great (MOS ~3.8 is not close to the SOTA of 4.4 - 4.5)



review_title: Right choice of problem. Introduces significant independence assumptions.

Meta-review: The paper presents a hybrid architecture which combines WaveNet and LSTM for speeding-up raw audio generation. The novelty of the method is limited, as it’s a simple combination of existing techniques. The practical impact of the approach is rather questionable since the generated audio has significantly lower MOS scores than the state-of-the-art WaveNet model.
Decision: Reject