Seeger & Bouchard AISTATS 2012 Fast VBI for Non-Conjugate Matrix Factorization

Probabilistic matrix factorization methods aim to extract meaningful correlation
structure from an incomplete data matrix by postulating low rank constraints.
Recently, variational Bayesian (VB) inference techniques have successfully been
applied to such large scale bilinear models. However, current algorithms are of
the alternate updating or stochastic gradient descent type, slow to converge and
prone to getting stuck in shallow local minima. While for MAP or maximum margin
estimation, singular value shrinkage algorithms have been proposed which can far
outperform alternate updating, this methodological avenue remains unexplored for
Bayesian techniques. In this paper, we show how to combine a recent singular
value shrinkage characterization of fully observed spherical Gaussian VB matrix
factorization with local variational bounding in order to obtain efficient VB
inference for general MF models with non-conjugate likelihood potentials. In
particular, we show how to handle Poisson and Bernoulli potentials, far more
suited for most MF applications than Gaussian likelihoods. Our algorithm can be
run even for very large models and is easily implemented in Matlab. It exhibits
significantly better prediction performance than MAP estimation on a range of
real-world datasets
More Matthias Seeger references

Sparse linear models:
Many problems of low-level computer vision and image processing, such as
denoising, deconvolution, tomographic reconstruction or superresolution, can be
addressed by maximizing the posterior distribution of a sparse linear model (SLM).
We show how higher-order Bayesian decision-making problems, such as optimizing
image acquisition in magnetic resonance scanners, can be addressed by querying
the SLM posterior covariance, unrelated to the density's mode. We propose a
scalable algorithmic framework, with which SLM posteriors over full,
high-resolution images can be approximated for the first time, solving a
variational optimization problem which is convex if and only if posterior mode
finding is convex.

Gaussian covariance:
We analyze computational aspects of variational approximate inference techniques
for sparse linear models, which have to be understood to allow for large scale
applications. Gaussian covariances play a key role, whose approximation is
computationally hard. While most previous methods gain scalability by not even
representing most posterior dependencies, harmful factorization assumptions can
be avoided by employing data-dependent low-rank approximations instead. We
provide theoretical and empirical insights into algorithmic and statistical
consequences of low-rank covariance approximation errors on decision outcomes in
nonlinear sequential Bayesian experimental design.

GLMs:
We show how variational Bayesian inference can be implemented for very large
generalized linear models. Our relaxation is proven to be a convex problem for
any log-concave model. We provide a generic double loop algorithm for solving
this relaxation on models with arbitrary super-Gaussian potentials. By
iteratively decoupling the criterion, most of the work can be done by solving
large linear systems, rendering our algorithm orders of magnitude faster than
previously proposed solvers for the same problem. We evaluate our method on
problems of Bayesian active learning for large binary classification models, and
show how to address settings with many candidates and sequential inclusion steps.