Challis & Barber AISTATS 2011 Concave Gaussian Variational Approximations

Two popular approaches to forming bounds in approximate Bayesian inference are
local variational methods and minimal Kullback-Leibler divergence methods. For a
large class of models we explicitly relate the two approaches, showing that the
local variational method is equivalent to a weakened form of Kullback-Leibler
Gaussian approximation. This gives a strong motivation to develop efficient
methods for KL minimisation. An important and previously unproven property of
the KL variational Gaussian bound is that it is a concave function in the
parameters of the Gaussian for log concave sites. This observation, along with
compact concave parametrisations of the covariance, enables us to develop fast
scalable optimisation procedures to obtain lower bounds on the marginal
likelihood in large scale Bayesian linear models.