Multimodal Transitions for Generative Stochastic Networks
Authors: Sherjil Ozair, Li Yao, Yoshua Bengio
ICLR 2014

Abstract: Generative Stochastic Networks (GSNs) have been recently introduced as an alternative to traditional probabilistic modeling: instead of parametrizing the data distribution directly, one parametrizes a transition operator for a Markov chain whose stationary distribution is an estimator of the data generating distribution. The result of training is therefore a machine that generates samples through this Markov chain. However, the previously introduced GSN consistency theorems suggest that in order to capture a wide class of distributions, the transition operator in general should be multimodal, something that has not been done before this paper. We introduce for the first time multimodal transition distributions for GSNs, in particular using models in the NADE family (Neural Autoregressive Density Estimator) as output distributions of the transition operator. A NADE model is related to an RBM (and can thus model multimodal distributions) but its likelihood (and likelihood gradient) can be computed easily. The parameters of the NADE are obtained as a learned function of the previous state of the learned Markov chain. Experiments clearly illustrate the advantage of such multimodal transition distributions over unimodal GSNs.


Peer review:
[Review]:
The authors present work that tackles learning data distributions indirectly, by learning a neural-network based transition operator (a Generative Stochastic Neural Network, GSN), whose stationary distribution is the desired distribution. Specifically, they introduce a GSN where the transition distribution is multimodal. The proposed model is evaluated in two experiments (spiral distribution and generating MNIST) and shown to produce samples that better capture the data distribution.

I'm not enough of an expert to give an in-depth analysis of the proposed method, however, I review the paper in general terms.


Pros: addressing sampling issues is worthwhile, and I would like to learn more about the proposed approach.
Cons: the initial part of the paper is way too wordy. At the same time, details and explanations regarding the model and experimental procedures are missing. There's few actual results; this should have been written as a short workshop paper.


The introduction/motivation section is very long and has some content that is not directly relevant to the paper, or detail that is unnecessary. The authors write over a page to motivate unsupervised learning in general, and to describe well-known facts (Boltzmann machines are difficult to handle; the existence of many, separated modes makes MCMC difficult). This strikes me as too general for the paper and could have been one or two short paragraphs instead. Similarly, what is the benefit of spelling out in full Theorem 1, from an earlier reference? Etc. 

The writing should be made more succinct and be structured better. In Section 1, the general introduction of the topic and overview of the paper should be separated. Wording could be improved (e.g., '[...] MAP approximations could all fall on their face.'; or 'In a sense, this is something that we are directly testing in this paper.'--in a sense?). 

In Section 5, I take it that the NADE factorization is now over pixels rather than frames? The notation should make this clearer (x_{} is used for both). RNADE was not defined (should be with ref [12]). A figure would have been helpful for understanding the GSN-NADE architecture (and the other models).

Results are not presented until Section 6, more than halfway through the paper. Here, the methods used are not properly explained and not enough details are given. 

In the first experiment, what are the model parameters and training procedure? For the second experiment, the authors write 'The training of model uses the same procedure as in [13]', with no further explanation. Similarly: 'To evaluate the quality of trained GSNs as generative models, we adopt the Conservative Sampling-based Log-likelihood (CSL) estimator proposed in [9]' and 'The GSN-1-w is [...] trained with the walkback procedure proposed in Bengio et al. [11].' No explanation is given for any of this. The references are all very recent (2013), so it's not like the general reader can be expected to be familiar with these methods. Again, model parameters are missing ('For the details of models being compared, please refer to [9].').


In conclusion, if the authors were to expand the experimental section and better explain the methods used, and cut out unnecessary content, this could make for a workshop track paper. But I don't see enough content for a conference paper.

Further notes:

Page 3: 'Monte-Carlo Markov Chain (MCMC) methods' -> Markov chain Monte Carlo (MCMC) methods
Page 4: 'Figure 2' should be Figure 1?

It looks like the paper was updated after the review period had started (24 Jan.). It would be good in such cases to notify reviewers by posting a comment here, to make sure they are reviewing the latest version.