Multimodal Transitions for Generative Stochastic Networks
Authors: Sherjil Ozair, Li Yao, Yoshua Bengio
ICLR 2014

Abstract: Generative Stochastic Networks (GSNs) have been recently introduced as an alternative to traditional probabilistic modeling: instead of parametrizing the data distribution directly, one parametrizes a transition operator for a Markov chain whose stationary distribution is an estimator of the data generating distribution. The result of training is therefore a machine that generates samples through this Markov chain. However, the previously introduced GSN consistency theorems suggest that in order to capture a wide class of distributions, the transition operator in general should be multimodal, something that has not been done before this paper. We introduce for the first time multimodal transition distributions for GSNs, in particular using models in the NADE family (Neural Autoregressive Density Estimator) as output distributions of the transition operator. A NADE model is related to an RBM (and can thus model multimodal distributions) but its likelihood (and likelihood gradient) can be computed easily. The parameters of the NADE are obtained as a learned function of the previous state of the learned Markov chain. Experiments clearly illustrate the advantage of such multimodal transition distributions over unimodal GSNs.


Peer review:
[Review]:
Recently, (Bengio et al. 2013b,2013c) proposed a novel way to estimate a data distribution indirectly, by training the transition operator T(x_{t}|x_{t-1}) of a Markov chain instead of trying to estimate the data distribution directly. These Generative Stochastic Networks (GSN) have several theoretical benefits, mainly avoiding the costly process of sampling in a highly multi-modal distribution (either when trying to sample from the model to estimate the so called 'negative gradient update', or when trying to perform MAP inference) as is traditionally required when training highly multi-modal models.

This paper proposes using a Neural Auto-Regressive Density Estimator (NADE) to learn a *multimodal* transitition operator, thus hopefully improving results compared to the above works in which the transition operator is unimodal.

The authors make a very good review of recent advances related to GSNs and argue that using a multi-modal transition operator could allow for better mixing of the markov chain and deal more gracefully with huge numbers of modes.

Two sets of experiments are performed, the first trying to estimate a 'spiral' distribution in 2D (a 2D slice of the traditional Swiss roll seen in manifold learning studies), the second on the Mnist dataset of handwritten digits. These experiments compare GSNs using the proposed multi-modal NADE transition operator with GSNs based on uni-modal transition operators which were used in previous works.

The results show that GSN-NADE fare much better at representing the spiral dataset, but fall a little short of expectations on the Mnist dataset where the walkback trick proposed in (Bengio et al. 2013c) allows a GSN with a uni-modal transition operator to fare better than the proposed GSN-NADE. In other words, the results support the hypothesis that a multi-modal NADE transition operator can be helpful, but using the walkback trick to deal with spurious modes appears to be even more helpful. As a possible future work, the authors propose to try and find a way to combine both benefits since walkback is not trivially applicable to the proposed GSN-NADE.

This study presents a new approach to a problem clearly identified in an earlier study, gives a very good summary of the state-of-the-art, clearly identifies key issues to overcome, and honestly presents its own limitations. Although the results fall a little short of expectations, they do indeed point to the fact that a multi-modal transition operator can be helpful in GSNs.
Regrettably the experiments do not try to assess the quality of the representations obtained with GSN-NADE in a classification setting and on more complex datasets than Mnist.