Introduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Generalization bounds for variational
inference
Pierre AlquierJouy-en-Josas
May 6, 2019
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkNotations
Assume that we observe X1 , . . . , Xn i.i.d from Pθ0 in a modelθ{Pθ , θ ∈ Θ} dominated by Q : dP
= pθ . Prior π on Θ.
dQ

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkNotations
Assume that we observe X1 , . . . , Xn i.i.d from Pθ0 in a modelθ{Pθ , θ ∈ Θ} dominated by Q : dP
= pθ . Prior π on Θ.
dQ
The likelihood
Ln (θ) =nYpθ (Xi )

i=1

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkNotations
Assume that we observe X1 , . . . , Xn i.i.d from Pθ0 in a modelθ{Pθ , θ ∈ Θ} dominated by Q : dP
= pθ . Prior π on Θ.
dQ
The likelihood
Ln (θ) =nYpθ (Xi )i=1

The posterior
πn (dθ) ∝ Ln (θ)π(dθ).

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkNotations
Assume that we observe X1 , . . . , Xn i.i.d from Pθ0 in a modelθ{Pθ , θ ∈ Θ} dominated by Q : dP
= pθ . Prior π on Θ.
dQ
The likelihood
Ln (θ) =nYpθ (Xi )i=1

The posterior
πn (dθ) ∝ Ln (θ)π(dθ).
The tempered posterior - 0 < α < 1πn,α (dθ) ∝ [Ln (θ)]α π(dθ).
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkComputation of the posterior
explicit form (conjugate models),

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkComputation of the posterior
explicit form (conjugate models),
MCMC algorithms : Metropolis-Hastings, Gibbs sampler,
Langevin Monte Carlo...

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkComputation of the posterior
explicit form (conjugate models),
MCMC algorithms : Metropolis-Hastings, Gibbs sampler,
Langevin Monte Carlo...
But...
when the dimension is large, the convergence of MCMC
can be extremely slow,

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkComputation of the posterior
explicit form (conjugate models),
MCMC algorithms : Metropolis-Hastings, Gibbs sampler,
Langevin Monte Carlo...
But...
when the dimension is large, the convergence of MCMC
can be extremely slow,
when the model is complex or when the sample size islarge, each evaluation of πn,α (θ) can be expensive.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkComputation of the posterior
explicit form (conjugate models),
MCMC algorithms : Metropolis-Hastings, Gibbs sampler,
Langevin Monte Carlo...
But...
when the dimension is large, the convergence of MCMC
can be extremely slow,
when the model is complex or when the sample size islarge, each evaluation of πn,α (θ) can be expensive.
For these reasons, in the past 20 years, many methods
targeting an approximation of πn,α became popular : ABC, EP
algorithm, variational inference, approximate MCMC ...
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talk

Variational approximations : definitions
Idea of VB : chose a family F of probability distributions on Θ
and approximate πn,α by a distribution in F :Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talk

Variational approximations : definitions
Idea of VB : chose a family F of probability distributions on Θ
and approximate πn,α by a distribution in F :π̃n,α := arg min K(ρ, πn,α ).
ρ∈F

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talk

Variational approximations : definitions
Idea of VB : chose a family F of probability distributions on Θ
and approximate πn,α by a distribution in F :π̃n,α := arg min K(ρ, πn,α ).
ρ∈F

Examples :parametric approximation

F = N (µ, Σ) : µ ∈ Rd , Σ ∈ Sd+ .

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talk

Variational approximations : definitions
Idea of VB : chose a family F of probability distributions on Θ
and approximate πn,α by a distribution in F :π̃n,α := arg min K(ρ, πn,α ).
ρ∈F

Examples :parametric approximation

F = N (µ, Σ) : µ ∈ Rd , Σ ∈ Sd+ .
mean-field approximation, Θ = Θ1 × Θ2 andF : {ρ : ρ(dθ) = ρ1 (dθ1 ) × ρ2 (dθ2 )} .
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkEmpirical lower bound (ELBO)
Note that :π̃n,α = arg min K(ρ, πn,α )
ρ∈F
)
(
Z
n
X
1
= arg min −αlog pθ (Xi )ρ(dθ) + K(ρ, π) .
ρ∈F
n i=1
{z
}
|
−ELBO(ρ)

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talkEmpirical lower bound (ELBO)
Note that :π̃n,α = arg min K(ρ, πn,α )
ρ∈F
)
(
Z
n
X
1
= arg min −αlog pθ (Xi )ρ(dθ) + K(ρ, π) .
ρ∈F
n i=1
{z
}
|
−ELBO(ρ)

So we have the equivalent definition :π̃n,α := arg max ELBO(ρ).
ρ∈F

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talk

Outline of the talkAfter this introduction :Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talk

Outline of the talkAfter this introduction :Section 2 will address the following question :What are the conditions ensuring that π̃n,α leads to good
estimators ?

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talk

Outline of the talkAfter this introduction :Section 2 will address the following question :What are the conditions ensuring that π̃n,α leads to good
estimators ?
We will show general conditions, and many examples.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talk

Outline of the talkAfter this introduction :Section 2 will address the following question :What are the conditions ensuring that π̃n,α leads to good
estimators ?
We will show general conditions, and many examples.
Section 3 will address the following question :Are there efficient algorithms to (provably) compute π̃n,α ?

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talk

Outline of the talkAfter this introduction :Section 2 will address the following question :What are the conditions ensuring that π̃n,α leads to good
estimators ?
We will show general conditions, and many examples.
Section 3 will address the following question :Are there efficient algorithms to (provably) compute π̃n,α ?
We will see that fast algorithms from sequential optimization
can be used in some cases. This also allows to do variational
inference on a data stream that cannot be stored.
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Bayesian inference
Definition of variational approximations
Outline of the talk

Outline of the talk1

Introduction : variational Bayesian inference
Bayesian inference
Definition of variational approximations
Outline of the talk2

Concentration of variational approximations of the posterior
Theoretical results
Applications
Extensions

3

Online variational inference
Sequential estimation problem
Online variational inference
Simulations
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Tools for the consistency of VB
The α-Rényi divergence for α ∈ (0, 1)Z1Dα (P, R) =log (dP)α (dR)1−α .
α−1Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Tools for the consistency of VB
The α-Rényi divergence for α ∈ (0, 1)Z1Dα (P, R) =log (dP)α (dR)1−α .
α−1All the properties derived in :T. Van Erven & P. Harremos. Rényi divergence and Kullback-Leibler divergence. IEEE
Transactions on Information Theory, 2014.

Among others, for 1/2 ≤ α, link with Hellinger and Kullback :H2 (P, R) ≤ Dα (P, R) −−→ K(P, R).
α%1

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

What do we know about πn,α ?
B(r ) = {θ ∈ Θ : K(Pθ0 , Pθ ) ≤ r } .
Theorem, variant of (Bhattacharya, Pati & Yang)
For any sequence (rn ) such that− log π[B(rn )] ≤ nrn
we haveZE1+αDα (Pθ , Pθ0 )πn,α (dθ) ≤
rn .
1−α

A. Bhattacharya, D. Pati & Y. Yang. Bayesian fractional posteriors. The Annals of Statistics,
2019.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Extension of previous result to VB
Theorem (A. & Ridgway)
If there is ρn ∈ F and (rn ) such that
 R
 K(Pθ0 , Pθ )ρn (dθ) ≤ rn ,andK(ρn , π) ≤ nrn ,then, for any α ∈ (0, 1),Z
1+αDα (Pθ , Pθ0 )π̃n,α (dθ) ≤
E
rn .
1−α

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Extension of previous result to VB
Theorem (A. & Ridgway)
If there is ρn ∈ F and (rn ) such that
 R
 K(Pθ0 , Pθ )ρn (dθ) ≤ rn ,andK(ρn , π) ≤ nrn ,then, for any α ∈ (0, 1),Z
1+αDα (Pθ , Pθ0 )π̃n,α (dθ) ≤
E
rn .
1−α
P. Alquier & J. Ridgway. Concentration of tempered posteriors and of their variational
approximations. The Annals of Statistics, to appear.
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Misspecified case
Assume now that X1 , . . . , Xn i.i.d ∼ Q ∈/ {Pθ , θ ∈ Θ}. Put :θ∗ := arg min K(Q, Pθ ).
θ∈Θ

Theorem (A. and Ridgway)
Assume that there is ρn ∈ F such that

Z 
dPθ∗
E log
ρn (dθ) ≤ rn and K(ρn , π) ≤ nrn ,dPθthen, for any α ∈ (0, 1),ZE
Dα (Pθ , Q)π̃n,α (dθ) ≤Pierre Alquier1+ααK(Q, Pθ∗ ) +
rn .
1−α
1−α
Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

First example : nonparametric regression
Nonparametric regression
Yi = f (Xi ) + ξi ,Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

First example : nonparametric regression
Nonparametric regression
Yi = f (Xi ) + ξi ,ξi ∼ N (0, σ 2 ),Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

First example : nonparametric regression
Nonparametric regression
Yi = f (Xi ) + ξi ,ξi ∼ N (0, σ 2 ),f is s-smooth with s unknown,

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

First example : nonparametric regression
Nonparametric regression
Yi = f (Xi ) + ξi ,ξi ∼ N (0, σ 2 ),f is s-smooth with s unknown,Pprior : f (·) = Kj=1 βj φj (·), random K and βj ’s, (ϕj )basis...

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

First example : nonparametric regression
Nonparametric regression
Yi = f (Xi ) + ξi ,ξi ∼ N (0, σ 2 ),f is s-smooth with s unknown,Pprior : f (·) = Kj=1 βj φj (·), random K and βj ’s, (ϕj )basis...
variational approx : βj mutually independent...
2s 2s+1
Under suitable assumptions, rn ∼ log(n).
nPierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

More examples covered in the paper
1logistic regression,

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

More examples covered in the paper
12logistic regression,
matrix completion : we prove that the approx. in
Y. J. Lim & Y. W. Teh. Variational Bayesian approach to movie rating prediction. Proceedings of
KDD cup and workshop, 2007.

leads to minimax-optimal estimation.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

An important example : mixture models
Mixture modelsPθ = Pp,θ1 ,...,θK =PK

j=1 pj qθj ,Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

An important example : mixture models
Mixture models
PPθ = Pp,θ1 ,...,θK = Kj=1 pj qθj ,prior π : p = (p1 , . . . , pK ) ∼ πp = D(α1 , . . . , αK ) and theθj ’s are independent from πθ .

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

An important example : mixture models
Mixture models
PPθ = Pp,θ1 ,...,θK = Kj=1 pj qθj ,prior π : p = (p1 , . . . , pK ) ∼ πp = D(α1 , . . . , αK ) and theθj ’s are independent from πθ .
Tempered posterior :αLn (θ) π(θ) ∝

n X
K
Y

!α
pj qθj (Xi )i=1 j=1

Pierre Alquierπp (p)KY

πθ (θj ).

j=1

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

An important example : mixture models
Mixture models
PPθ = Pp,θ1 ,...,θK = Kj=1 pj qθj ,prior π : p = (p1 , . . . , pK ) ∼ πp = D(α1 , . . . , αK ) and theθj ’s are independent from πθ .
Tempered posterior :αLn (θ) π(θ) ∝

n X
K
Y

!α
pj qθj (Xi )πp (p)

i=1 j=1KYπθ (θj ).

j=1

Variational approximation :π̃n,α (p, θ) = ρp (p)KYρj (θj ).

j=1
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

ELBO maximization for mixtures
Optimization program

X

n Z
K
X
min
−α
log
pj qθj (Xi ) ρ(dθ)ρ=(ρp ,ρ1 ,...,ρK )

i=1

j=1



+ K ρp , πp +KXK ρj , πj




j=1

− log

X
K
j=1


 X
K
pj qθj (Xi ) = min −ωji log(pj qθj (Xi ))ω i ∈SK

j=1

+

K
X



ωji log(ωji )j=1
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Coordinate Descent algorithm

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Numerical example on Gaussian mixtures

P
Gaussian mixture 3j=1 pj N (θj , 1) and Gaussian prior on θj .
Sample size n = 1000, we report the MAE over 10 replications.

Algo.
VBα=0.5
VBα=1EM

p0.03 (0.02)
0.03 (0.02)
0.03 (0.02)

θ1
0.14 (0.30)
0.14 (0.21)
0.14 (0.22)

Pierre Alquierθ20.38 (1.11)
0.36 (0.97)
0.36 (0.97)

θ3
0.05 (0.05)
0.06 (0.04)
0.06 (0.05)

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Mixture models : convergence rates
Theorem (Chérief-Abdellatif, A.)
Chose K2 ≤ αj ≤ 1 and assume that
estimation in (qθ ) (without mixture) at
rate rn .

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Mixture models : convergence rates
Theorem (Chérief-Abdellatif, A.)
Chose K2 ≤ αj ≤ 1 and assume that
estimation in (qθ ) (without mixture) at
rate rn . ThenZEDα (Pp,θ1 ,...,θK , Pp0 ,θ10 ,...,θK0 )π̃n,α (dθ)≤

1+αcst.Krn .
1−αPierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Mixture models : convergence rates
Theorem (Chérief-Abdellatif, A.)
Chose K2 ≤ αj ≤ 1 and assume that
estimation in (qθ ) (without mixture) at
rate rn . ThenZEDα (Pp,θ1 ,...,θK , Pp0 ,θ10 ,...,θK0 )π̃n,α (dθ)≤

1+αcst.Krn .
1−αB.-E. Chérief-Abdellatif, P. Alquier. Consistency of
Variational Bayes Inference for Estimation and Model
Selection in Mixtures. Electronic Journal of Statistics, 2018.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Model selection
D. Blei, A. Kucukelbir & J. McAuliffe. Variational inference : A review for statisticians. JASA,
2017.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Model selection
D. Blei, A. Kucukelbir & J. McAuliffe. Variational inference : A review for statisticians. JASA,
2017.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Model selectionkAssume that we have K models, define π̃n,αa variational
approximation of the tempered posterior in model k, and rnk its
convergence rate if model k is correct. Put :kk̂ = arg max ELBO(π̃n,α).
kPierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Model selectionkAssume that we have K models, define π̃n,αa variational
approximation of the tempered posterior in model k, and rnk its
convergence rate if model k is correct. Put :kk̂ = arg max ELBO(π̃n,α).
kTheorem (Chérief-Abdellatif)
If the true model is actually k0 ,Z
1 + α k0log(K )0nk̂EDα (Pθ , P )π̃n,α (dθ|X1 ) ≤
rn +
.
1−α
n(1 − α)Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

Model selectionkAssume that we have K models, define π̃n,αa variational
approximation of the tempered posterior in model k, and rnk its
convergence rate if model k is correct. Put :kk̂ = arg max ELBO(π̃n,α).
kTheorem (Chérief-Abdellatif)
If the true model is actually k0 ,Z
1 + α k0log(K )0nk̂EDα (Pθ , P )π̃n,α (dθ|X1 ) ≤
rn +
.
1−α
n(1 − α)B.-E. Chérief-Abdellatif. Consistency of ELBO maximization for model selection. Proceedings of
AABI 2018.
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

More extensions
1

more general models with latent variables :Y. Yang, D. Pati & A. Bhattacharya. α-Variational Inference with Statistical Guarantees. The
Annals of Statistics, to appear.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

More extensions
1

more general models with latent variables :Y. Yang, D. Pati & A. Bhattacharya. α-Variational Inference with Statistical Guarantees. The
Annals of Statistics, to appear.

2

case α = 1, i.e approximation of the “usual” posterior :F. Zhang & C. Gao. Convergence Rates of Variational Posterior Distributions. Preprint arXiv,
2017.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Theoretical results
Applications
Extensions

More extensions
1

more general models with latent variables :Y. Yang, D. Pati & A. Bhattacharya. α-Variational Inference with Statistical Guarantees. The
Annals of Statistics, to appear.

2

case α = 1, i.e approximation of the “usual” posterior :F. Zhang & C. Gao. Convergence Rates of Variational Posterior Distributions. Preprint arXiv,
2017.

3

approximation based on another distance, for example :π̃n,α := arg min W(ρ, πn,α ) (Wasserstein distance),ρ∈F

J. Huggins, T. Campbell, M. Kasprzak & T. Broderick. Practical bounds on the error of Bayesianposterior approximations : a nonasymptotic approach. Preprint arXiv, 2018.
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Outline of the talk1

Introduction : variational Bayesian inference
Bayesian inference
Definition of variational approximations
Outline of the talk2

Concentration of variational approximations of the posterior
Theoretical results
Applications
Extensions

3

Online variational inference
Sequential estimation problem
Online variational inference
Simulations
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1

initialize θ1 ,Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2

initialize θ1 ,x1 revealed,

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )update θ2 → θ3 ,Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1
2

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )update θ2 → θ3 ,x3 revealed,

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1
2
3

4

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )update θ2 → θ3 ,x3 revealed,
incur loss
− log pθ3 (x3 )...
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1
2
3

4

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )Objective :update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )update θ2 → θ3 ,x3 revealed,
incur loss
− log pθ3 (x3 )...
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1
2
3

4

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )Objective : make sure that
we learn to predict well as fast
as possible.

update θ2 → θ3 ,x3 revealed,
incur loss
− log pθ3 (x3 )...
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1
2
3

4

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )update θ2 → θ3 ,x3 revealed,
incur loss
− log pθ3 (x3 )Objective : make sure that
we learn to predict well as fast
as possible. KeepTX[− log pθt (xt )]t=1

as small as possible for any T ,without stochastic
assumptions on the data.

...
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Reference

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Reference

The regret :R(T ) =

TX[− log pθt (xt )]
t=1

− inf

θ∈Θ

Pierre AlquierTX[− log pθ (xt )].

t=1

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

EWA strategy / multipicative update...

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

EWA strategy / multipicative update...
learning rate α > 0.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

EWA strategy / multipicative update...
learning rate α > 0.
initialize p1 = π (the prior).

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

EWA strategy / multipicative update...
learning rate α > 0.
initialize p1 = π (the prior).
Algorithm 2 Exponentially Weighted Aggregation
1: for t = 1, 2, . . . do2:
θt = Eθ∼pt [θ],α3:
xt revealed, update pt+1 (dθ) = R[p[pθϑ(x(xt t)])]αppt t(dθ).
(dϑ)
4: end for

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

EWA strategy / multipicative update...
learning rate α > 0.
initialize p1 = π (the prior).
Algorithm 2 Exponentially Weighted Aggregation
1: for t = 1, 2, . . . do2:
θt = Eθ∼pt [θ],α3:
xt revealed, update pt+1 (dθ) = R[p[pθϑ(x(xt t)])]αppt t(dθ).
(dϑ)
4: end forNote that pt = πn,α the tempered posterior, so problem : how
can we compute θt ?
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for EWA
From now, θ 7→ [− log pθ (xt )] is convex + bounded : | · | ≤ C .

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for EWA
From now, θ 7→ [− log pθ (xt )] is convex + bounded : | · | ≤ C .
TheoremTX
t=1[− log pθt (xt )] ≤ inf
p

" T
X

Eθ∼p [− log pθ (xt )]t=1#
αC 2 T
K(p, π)++.
2
αPierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for EWA
From now, θ 7→ [− log pθ (xt )] is convex + bounded : | · | ≤ C .
TheoremTX[− log pθt (xt )] ≤ inf
p

t=1

" T
X

Eθ∼p [− log pθ (xt )]t=1#
αC 2 T
K(p, π)++.
2αUnder similar assumptions than in the batch case, √that is, theprior gives enough mass to relevant θ, and α ∼ 1/ T ,
T
T
X
X
√
[− log pθt (xt )] ≤ inf[− log pθ (xt )] + cst. T
t=1

θ∈Θ

Pierre Alquier

t=1
Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

T
T
X
X
√
[− log pθt (xt )] ≤ inf[− log pθ (xt )] + cst. T
t=1

θ∈Θ

Pierre Alquier

t=1

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

T
T
X
X
√
[− log pθt (xt )] ≤ inf[− log pθ (xt )] + cst. T
t=1

θ∈Θ

t=1

T
T
1 X
q(xt )1 X
q(xt )
cst
log
log
+√ .
≤ inf
T t=1
pθt (xt ) θ∈Θ T t=1pθ (xt )T

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

T
T
X
X
√
[− log pθt (xt )] ≤ inf[− log pθ (xt )] + cst. T
t=1

θ∈Θ

t=1

T
T
1 X
q(xt )1 X
q(xt )
cst
log
log
+√ .
≤ inf
T t=1
pθt (xt ) θ∈Θ T t=1pθ (xt )T

Assuming that x1 , . . . , xT are actually i.i.d from Q, withdensity q, define
T
1 X
θ̂T =
θT ,
T t=1
we have (“online-to-batch” conversion) :cst
E K Q, Pθ̂T ≤ inf K (Q, Pθ ) + √ .
θ∈ΘTPierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Variational approximations of EWA

B.-E. Chérief-Abdellatif, P. Alquier & M. E. Khan. A Generalization Bound for Online Variational
Inference. Preprint arXiv, 2018.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Variational approximations of EWA

B.-E. Chérief-Abdellatif, P. Alquier & M. E. Khan. A Generalization Bound for Online Variational
Inference. Preprint arXiv, 2018.

Parametric variational approximation :F = {qµ , µ ∈ M} .
Objective : propose a way to update µt → µt+1 so that qµt leads
to similar performances as pt in
EWA...
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

SVA and SVB strategies
Algorithm 3 SVA (Sequential Variational Approximation)
1: for t = 1, 2, . . . do2:
θt = Eθ∼qµt [θ],3:
xt revealed, update
#
"
t
X
K(q
,
π)
µ
.
µt+1 = arg min µT ∇µ
Eθ∼qµ [− log pθ (xi )] +
µ∈M
α
i=1
4: end for

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

SVA and SVB strategies
Algorithm 3 SVA (Sequential Variational Approximation)
1: for t = 1, 2, . . . do2:
θt = Eθ∼qµt [θ],3:
xt revealed, update
#
"
t
X
K(q
,
π)
µ
.
µt+1 = arg min µT ∇µ
Eθ∼qµ [− log pθ (xi )] +
µ∈M
α
i=1
4: end for

SVB (Streaming Variational Bayes) has update
K(qµ , qµt )Tµt+1 = arg min µ ∇µ Eθ∼qµ [− log pθ (xt )] +.
µ∈MαPierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

NGVI strategy
NGVI (Natural Gradient Variational Inference) : fix someβ > 0,
µt+1
"

#
K(q
,
π)
K(q
,
q
)
µ
µ
µ
t
= arg min µT ∇µ Eθ∼qµ [− log pθ (xt )]+
+
.
µ∈M
α
β

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

NGVI strategy
NGVI (Natural Gradient Variational Inference) : fix someβ > 0,
µt+1
"

#
K(q
,
π)
K(q
,
q
)
µ
µ
µ
t
= arg min µT ∇µ Eθ∼qµ [− log pθ (xt )]+
+
.
µ∈M
α
β

M. E. Khan & W. Lin. Conjugate-computation variational inference : Converting variational
inference in non-conjugate models to inferences in conjugate models. AISTAT, 2017.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

An example : SVB with Gaussian approximations
As an example, assume that θ ∈ Rd , the prior is
π = N (0, s 2 I ) and that 
we use the variationalapproximation

σ12 . . . 0 family : qµ = qm,σ = N m,  ... . . . ...  .
0 . . . σd2

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

An example : SVB with Gaussian approximations
As an example, assume that θ ∈ Rd , the prior is
π = N (0, s 2 I ) and that 
we use the variationalapproximation

σ12 . . . 0 family : qµ = qm,σ = N m,  ... . . . ...  .
0 . . . σd2
In this case, the update in SVB is :mt+1 = mt − ασt2 ∇m=mt Eθ∼qm,σt [− log pθ (xt )]
ασt ∇σ=σt Eθ∼qmt ,σ [− log pθ (xt )]σt+1 = σt h2where √means “componentwise multiplication” andh(x) = 1 + x 2 − x is also applied componentwise.
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

An example : SVB with Gaussian approximations
As an example, assume that θ ∈ Rd , the prior is
π = N (0, s 2 I ) and that 
we use the variationalapproximation

σ12 . . . 0 family : qµ = qm,σ = N m,  ... . . . ...  .
0 . . . σd2
In this case, the update in SVB is :mt+1 = mt − ασt2 ∇m=mt Eθ∼qm,σt [− log pθ (xt )]
ασt ∇σ=σt Eθ∼qmt ,σ [− log pθ (xt )]σt+1 = σt h2where √means “componentwise multiplication” andh(x) = 1 + x 2 − x is also applied componentwise. We also
have explicit formulas for SVA and NGVI (see the paper).
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for SVA
Theorem (Chérief-Abdellatif, A. & Khan)
Assume that µ 7→ Eθ∼qµ [− log pθ (xt )] is L-Lipschitz andconvex.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for SVA
Theorem (Chérief-Abdellatif, A. & Khan)
Assume that µ 7→ Eθ∼qµ [− log pθ (xt )] is L-Lipschitz andconvex. (this is for example the case as soon as the
log-likelihood is concave in θ and L-Lipschitz, and µ is alocation-scale parameter).

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for SVA
Theorem (Chérief-Abdellatif, A. & Khan)
Assume that µ 7→ Eθ∼qµ [− log pθ (xt )] is L-Lipschitz andconvex. Assume that µ 7→ K(pµ , π) is γ-strongly convex.
Then SVA satisfies :TX[− log pθt (xt )]

t=1

)
" T
#
X
αL2 T
K(qµ , π)+.
Eθ∼qµ[− log pθ (xt )] +
γ
α
t=1

(
≤ inf

µ∈M

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for SVA
Theorem (Chérief-Abdellatif, A. & Khan)
Assume that µ 7→ Eθ∼qµ [− log pθ (xt )] is L-Lipschitz andconvex. Assume that µ 7→ K(pµ , π) is γ-strongly convex.
Then SVA satisfies :TX[− log pθt (xt )]

t=1

)
" T
#
X
αL2 T
K(qµ , π)+.
Eθ∼qµ[− log pθ (xt )] +
γ
α
t=1

(
≤ inf

µ∈M

For SVB : some results in the Gaussian case.
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for SVA
Theorem (Chérief-Abdellatif, A. & Khan)
Assume that µ 7→ Eθ∼qµ [− log pθ (xt )] is L-Lipschitz andconvex. Assume that µ 7→ K(pµ , π) is γ-strongly convex.
Then SVA satisfies :TX[− log pθt (xt )]

t=1

)
" T
#
X
αL2 T
K(qµ , π)+.
Eθ∼qµ[− log pθ (xt )] +
γ
α
t=1

(
≤ inf

µ∈M

For SVB : some results in the Gaussian case. For NGVI : wewere not able to derive regret bounds until now.
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Test on a simulated dataset

Figure – Average cumulative losses on different datasets forclassification and regression tasks with OGA (yellow), OGA-EL(red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Test on the Breast dataset

Figure – Average cumulative losses on different datasets forclassification and regression tasks with OGA (yellow), OGA-EL(red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Test on the Pima Indians dataset

Figure – Average cumulative losses on different datasets forclassification and regression tasks with OGA (yellow), OGA-EL(red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Test on the Boston Housing dataset

Figure – Average cumulative losses on different datasets forclassification and regression tasks with OGA (yellow), OGA-EL(red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Test on the Forest Cover Type dataset

Figure – Average cumulative losses on different datasets forclassification and regression tasks with OGA (yellow), OGA-EL(red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Conclusions
1

Using online-to-batch conversion, we now have algorithms
for variational inference with provable statisticalproperties after a finite number of steps.

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Conclusions
1

2

Using online-to-batch conversion, we now have algorithms
for variational inference with provable statisticalproperties after a finite number of steps.
SVA, SVB competitive with OGA (online gradientalgorithm, “non-Bayesian”).

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Conclusions
1

2

3

Using online-to-batch conversion, we now have algorithms
for variational inference with provable statisticalproperties after a finite number of steps.
SVA, SVB competitive with OGA (online gradientalgorithm, “non-Bayesian”).
NGVI is the best method on all datasets. Its theoreticalanalysis is thus an important open problem. Cannot bedone with our current techniques (using naturalparameters in exponential models lead to non-convex
objectives).

Pierre Alquier

Generalization bounds for variational inferenceIntroduction : variational Bayesian inference
Concentration of variational approximations of the posterior
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Thank you !

Pierre Alquier

Generalization bounds for variational inference