pymc-devs/pymc — pull request #547 by kadeng (2014-06-06 15:58:02+00:00)
Title: Efficient sampling from mixture models

The changes in this pull request are the following

#### Metropolis Hastings Step Method ===
A new step method, called MetropolisHastings which allows arbitrary (non-symmetric) proposal distributions. Implemented wrappers for the existing Proposals as well as explicit support for Categorical proposal distributions. This can easily be extended to cover more proposals (how about HamiltonianMC Proposals, or Swendsen-Wang ?).

The implementation could be made more efficient (some potentially complex terms get added to the acceptance logp, and then subtracted again later on), but it's hard to do that in a general manner. Maybe one could teach Theano to optimize this out.

#### Efficient sampling from Mixture Models
Using that Metropolis Hastings step method, it's easy to create an efficient sampler for Mixture models. So I wrote a first implementation for a Multivariate Normal Mixture distribution. Seems to work well. This could easily be extended to allow flexible mixture kernels.

#### Small Examples
For both, the new Step Method and the Mixture Model, I provided examples.

#### Small fixes

I noticed that the "bound" function from distributions/dist_math.py fails to work correctly in some cases. Especially in the logp of the Categorical distribution, it failed not to evaluate at disallowed values, which
could then lead to index out of bounds exceptions. I fixed that by introducing a new "bound_scalar" function
which uses theano.ifelse.ifelse(..), which has the advantage over theano.switch(..) of not evaluating inactive parts of the expression
graph.

Another small fix had to do with TransformedDistributions. In one example I used a transformed Dirichlet distribution where one of the hyperparameters was a 1. This led to an invalid mode for the distribution. That's no problem, unless you transform that Dirichlet, in which case the mode becomes the starting point for the sampler. Since it's invalid (contains NaN values), that leads to problems.

#### Potential uses
The GMMs can be used as flexible prior densities. They can approximate any function.

Another use case for efficient GMMs is to use them to represent posterior densities. Since it's possible to
efficiently sample from products of mixtures of gaussians, these can be used to enable distributed scalable MCMC, both in the form of embarassingly parallel MCMC, as well as in distributed Nonparametric Belief Propagation (NBP) inference algorithms. I plan to write more on that later. See also my PyPGMc project on GitHub.