Ref: Gal & Zoubin @ Big Learning 2013 on pitfalls of parallel MCMC for DPs

Abstract
Recent work done by Lovell, Adams, and Mansingka [2012] and Williamson, Dubey,
and Xing [2013] has suggested an alternative parametrisation for the Dirichlet
process in order to derive non-approximate parallel MCMC inference for it. This
approach to parallelisation has been picked-up and implemented in several
different fields [Chahuneau et al., 2013, Pan et al., 2013]. In this paper we
show that the approach suggested is impractical due to an extremely unbalanced
distribution of the data. We characterise the requirements of efficient parallel
inference for the Dirichlet process and show that the proposed inference fails
most of these conditions (while approximate approaches often satisfy most of
them). We present both theoretical and experimental evidence of this, analysing
the load balance for the inference showing that it is independent of the size of
the dataset and the number of nodes available in the parallel implementation,
and end with preliminary suggestions of alternative paths of research for
efficient non-approximate parallel inference for the Dirichlet process.
Merge branch 'master' of github.com:mattjj/bayesbib