Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a variant of stochastic sequence neural network, the family of VRNN and SRNN. This paper adopts the CW-VAE framework and completes the optimization process under the stochastic sequence neural network framework. The authors test it on the speech domain. The experiments show that it outperforms VRNN and SRNN in the benchmark datasets. 
main_review: Strength:
1. The proposed method is clear. The experiment results support that the proposed method is better than the VRNN and SRNN methods.

Weakness:
1. The novelty is limited. The main ideas about clock-wise RNN network and CWVAE ideas have been proposed. 
2. The hierarchy latent variable idea has been proposed in Stochastic WaveNet (https://arxiv.org/pdf/1806.06116.pdf) and STCN (https://arxiv.org/pdf/1902.06568.pdf). This paper didn’t compare the proposed method with these two methods. 
3. In the paper, https://arxiv.org/pdf/1902.01388.pdf, the authors point out the evaluation problem of the stochastic sequence neural network. They found the stochastic sequence neural network has an unfair advantage over the deterministic model when s!=1. And with some tricks, the deterministic model can catch up the stochastic model's performance. But the authors didn’t discuss and address this issue in this paper. 
summary_of_the_review: The proposed idea is reasonable but didn't conduct a comprehensive study comparing related works.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to put various models under the same experimental setting and compare their rate at compressing speech. The models of choice are vanila LSTMs, variational RNNs, stochastic RNNs, Clockwork VAEs, and WaveNets. The results are also compared against regular compression algorithms, for example, FLAC.
main_review: The paper is well motivated. I do think benchmarking different latent models is worth doing, and reporting the compression rate is the right metric.

The experiments themselves are fine, but the evaluation metric is a little confusing. All models, except vanilla LSTM and WaveNet, have hidden variables to marginalize, so I'm not entirely sure how the likelihoods are computed. Marginalization is difficult as the paper argued. It's unclear whether, for example, the numbers in Table 1 are simply the values of the variational lower bound, or if any approximation is done to marginalize the hidden variables.

The paper also attempts to answer why one model can be better than the others. The paper looks into phonemes and speaker genders, but the message is not clear.

The presentation is fine. The majority of the paper is spent reviewing the models. I have mixed feelings visualizing the models the way it is done in Figure 1. Figure 2 is a much better representation, laying out the conditional assumptions for both the encoders and decoders. I do understand it would take up a lot of space, but it might be worth putting a figure in the appendix. It's also worth talking about the independence assumptions and where uncertainties are baked in.
summary_of_the_review: The paper is well motivated. The experimental design is fine, but it's unclear how the evaluation metric, the likelihood, is computed without marginalizing the hidden vectors. The presentation is fine.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an exploration of the use of latent variable models as generative models of speech.  Noting that such models work well in the image space, but not so much in the speech space, the authors move on to adapt the Clockwork VAE (a video LVM) as a speech model.  In the process the authors present a series of useful technical solutions to various issues that arise in this domain transition.  This, and other generative models of speech are later compared in the experiments section.  The results show that this approach is potentially viable.  The performance of the proposed speech LVM is good, albeit it comes with increased computational complexity (hopefully something to solve in the future).  In addition, it is shown that the resulting latent representation is correlated with phonetic structure, which is a pleasant bonus that other speech generative models (e.g. WaveNet) lack.
main_review: Strengths:  This paper addresses a series of issues that need to be resolved to apply an LV model like clockwork VAE to data like speech.  I find that sequence of steps to be instructive, and the overall target to be one worthy of exploration.  I think that collectively all the engineering described here is a significant amount of work and I appreciate it all being in one place.

Weaknesses: This paper seems quite detached from the community that would find it most interesting.  Speech generation is something that has been studied for a very long time, so I would have expected to at least get some sense of how this approach would compare (on various aspects) with modern practice.  Of course, WaveNet is one model that can serve as a well-recognized benchmark, but there is a lot more to compare with here. I am also uncomfortable with the use of bps as a performance measure of the generative power of these models.  I would have liked to hear some examples and understand how these models differ in their outputs.
summary_of_the_review: This paper provides an insightful exploration of how one can use an LVM as a speech generative model.  Although not completely achieved here, I feel that this paper shows some intriguing progress towards that goal, and towards speech models with semantically meaningful latent states.  I think many researchers in this area will find interest in all the engineering that was put to work in this paper, which might also be useful outside of this particular problem.  On the downside, this paper doesn't feel like it addresses any deep scientific questions (although it touches on some near the end), and it mostly reads like a todo list to get clockwork VAE to work with speech.
review_title: null

Meta-review: This paper presents the application of the hierarchical latent variable model, CW-VAE which is originally developed in the vision community, to the speech domain with meaningful modifications, and provide empirical analysis of the likelihood as well as discussions on the likelihood metrics. The reviewers tend to agree that it is a promising direction to study hierarchically structured LVMs for speech, and the introduction/adaptation of CW-VAE is useful. There were some discussion on the suitability of the likelihood evaluation, and it appears a fair comparison with wavenet shall take place at s=1 (single sample), a resolution level the proposed method does not yet scale up to. On the other hand, an important potential use case of the model is representation learning for speech, as it is a common belief that at suitable resolution the features shall discover units like phoneme. But I find the current evaluation of latent representations by LDA and KNN to be somewhat limited, and in fact there is no comparison with suitable baselines in Sec 3.2 in terms of feature quality. A task closer to modern speech recognition (e.g., with end-to-end models) would be preferred.
Decision: Reject