Multimodal Transitions for Generative Stochastic Networks
Authors: Sherjil Ozair, Li Yao, Yoshua Bengio
ICLR 2014

Abstract: Generative Stochastic Networks (GSNs) have been recently introduced as an alternative to traditional probabilistic modeling: instead of parametrizing the data distribution directly, one parametrizes a transition operator for a Markov chain whose stationary distribution is an estimator of the data generating distribution. The result of training is therefore a machine that generates samples through this Markov chain. However, the previously introduced GSN consistency theorems suggest that in order to capture a wide class of distributions, the transition operator in general should be multimodal, something that has not been done before this paper. We introduce for the first time multimodal transition distributions for GSNs, in particular using models in the NADE family (Neural Autoregressive Density Estimator) as output distributions of the transition operator. A NADE model is related to an RBM (and can thus model multimodal distributions) but its likelihood (and likelihood gradient) can be computed easily. The parameters of the NADE are obtained as a learned function of the previous state of the learned Markov chain. Experiments clearly illustrate the advantage of such multimodal transition distributions over unimodal GSNs.


Peer review:
[Review]:
Generative Stochastic Networks have recently been introduced as a new generative model formalism. Instead of directly parameterizing the data distribution, GSNs learn the transition distribution for a Markov chain which generates samples from the learned distribution. Previous results on GSNs, namely using the denoising criterion, have used a transition distribution that is factorized or unimodal. This makes it easier to learn the GSN, but limits the expressivity of the model. This paper proposes a type of GSN which uses conditional NADE to represent multi-modal output distributions for the transition operator. Results show it performing better than the uni-modal approach on MNIST and a toy 2d dataset.

The paper is novel. The GSN is a relatively recent formalism, with essentially all work coming from the same research group. This is the first work to propose a multi-modal transition operator. It's also an interesting use of NADE and bears some similarity to the recent use of NADE as an output distribution for recurrent neural networks. It is an interesting paper: well motivated and well written.

Pros:
* In my opinion, GSNs are a fascinating framework and this paper addresses one of the limitations of previous work
* The paper is clear and gives a great deal of background; in summarizing the work to-date on GSNs, it's complimentary to other published work

Cons:
* Experiments are limited to a toy dataset (2d - a few hundred (?) examples) and MNIST; the former is ok for visualization and the later is used to compare methods based on an approximation to test log likelihood proposed by the same group
* The comparisons aren't very extensive - essentially the model without NADE and the baseline model using 'walkback' training
* Although the background and review of GSNs are welcome, more than half of the paper is devoted to background. I feel that the paper makes a good contribution, but not as substantial as other ICLR papers. It could certainly be strengthened by exploring other types of output distribution models beyond NADE or experiments beyond MNIST.

Comments
--------

In the experiments section, I suggest giving a pointer that discussion re: walkback not being applied to GSN-NADE will be addressed later (i.e. in next section).

Section 4 refers to a Figure 2, but I believe it should be Figure 1.

Section 6 'fatorial' -> 'factorial'

Even though the CSL method is referenced, you could give a short description of it in the experiments section.

Is R-NADE being used for the 2d dataset and regular NADE being used for MNIST? I didn't see this explicitly stated.

First sentence on page 7 reads 'On the contrary, GSN-NADE with the walkback training alleviates this issue.' This is confusing, GSN-NADE didn't use walkback training. Should it say 'GSN-NADE without the walkback training?'