lisa-lab/pylearn2 — pull request #1093 by vdumoulin (2014-08-22 23:24:09+00:00)
Title: Add variational autoencoder framework

This is a clean subset of the variational autoencoder framework I've been working on lately. I'm opening the pull request now to get feedback, even though there are no tests written yet.

In order to keep the code modular, I'm re-using the MLP framework for both the encoder and the decoder networks, which has proven to be quite useful in my experiments.

One design decision I made (and for which I'd like to know your thoughts) is using multiple inheritance. The `BaseVAE` base class defines an internal interface, which I broke down in two distinct parts that are handled by separate subclasses of the base class:

1. Input-related methods, such as sampling from the conditional `p(x | z)` and computing `log p(x | z)`, are handled by subclasses in the `visible` module.
2. Latent-variable-related methods, such as sampling from the posterior `q(z | x)` and the prior `p(z)`, computing the KL-divergence between posterior and the prior, and computing their respective log-probabilities, are handled by the `latent` module.

A helper method allows dynamical inheritance, so that the user can mix-and-match conditional distributions `p(x | z)` and posterior/prior distribution on latent variables (e.g. having real-valued inputs vs. binary inputs while keeping the same gaussian prior/posterior).