Introduction : variational Bayesian inference
Online variational inference

Generalization bounds for variational
inference
Pierre Alquier

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Notations
Assume that we observe X1 , . . . , Xn i.i.d from Pθ0 in a modelθ{Pθ , θ ∈ Θ} dominated by Q : dP
= pθ . Prior π on Θ.
dQ

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Notations
Assume that we observe X1 , . . . , Xn i.i.d from Pθ0 in a modelθ{Pθ , θ ∈ Θ} dominated by Q : dP
= pθ . Prior π on Θ.
dQ
The likelihood
Ln (θ) =nYpθ (Xi )

i=1

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Notations
Assume that we observe X1 , . . . , Xn i.i.d from Pθ0 in a modelθ{Pθ , θ ∈ Θ} dominated by Q : dP
= pθ . Prior π on Θ.
dQ
The likelihood
Ln (θ) =nYpθ (Xi )i=1

The posterior
πn (dθ) ∝ Ln (θ)π(dθ).

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Notations
Assume that we observe X1 , . . . , Xn i.i.d from Pθ0 in a modelθ{Pθ , θ ∈ Θ} dominated by Q : dP
= pθ . Prior π on Θ.
dQ
The likelihood
Ln (θ) =nYpθ (Xi )i=1

The posterior
πn (dθ) ∝ Ln (θ)π(dθ).
The tempered posterior - 0 < α ≤ 1πn,α (dθ) ∝ [Ln (θ)]α π(dθ).
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Computation of the posterior
explicit form (conjugate models),

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Computation of the posterior
explicit form (conjugate models),
MCMC algorithms : Metropolis-Hastings, Gibbs sampler,
Langevin Monte Carlo...

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Computation of the posterior
explicit form (conjugate models),
MCMC algorithms : Metropolis-Hastings, Gibbs sampler,
Langevin Monte Carlo...
But...
when the dimension is large, the convergence of MCMC
can be extremely slow,

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Computation of the posterior
explicit form (conjugate models),
MCMC algorithms : Metropolis-Hastings, Gibbs sampler,
Langevin Monte Carlo...
But...
when the dimension is large, the convergence of MCMC
can be extremely slow,
when the model is complex or when the sample size islarge, each evaluation of πn,α (θ) can be expensive.

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Computation of the posterior
explicit form (conjugate models),
MCMC algorithms : Metropolis-Hastings, Gibbs sampler,
Langevin Monte Carlo...
But...
when the dimension is large, the convergence of MCMC
can be extremely slow,
when the model is complex or when the sample size islarge, each evaluation of πn,α (θ) can be expensive.
For these reasons, in the past 20 years, many methods
targeting an approximation of πn,α became popular : ABC, EP
algorithm, variational inference, approximate MCMC ...
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Variational approximations : definitions
Idea of VB : chose a family F of probability distributions on Θ
and approximate πn,α by a distribution in F :Pierre Alquier

Generalization bounds for online variational inferenceBayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Introduction : variational Bayesian inference
Online variational inference

Variational approximations : definitions
Idea of VB : chose a family F of probability distributions on Θ
and approximate πn,α by a distribution in F :π̃n,α := arg min K(ρ, πn,α ).
ρ∈F

Pierre Alquier

Generalization bounds for online variational inferenceBayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Introduction : variational Bayesian inference
Online variational inference

Variational approximations : definitions
Idea of VB : chose a family F of probability distributions on Θ
and approximate πn,α by a distribution in F :π̃n,α := arg min K(ρ, πn,α ).
ρ∈F

Examples :parametric approximation

F = N (µ, Σ) : µ ∈ Rd , Σ ∈ Sd+ .

Pierre Alquier

Generalization bounds for online variational inferenceBayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Introduction : variational Bayesian inference
Online variational inference

Variational approximations : definitions
Idea of VB : chose a family F of probability distributions on Θ
and approximate πn,α by a distribution in F :π̃n,α := arg min K(ρ, πn,α ).
ρ∈F

Examples :parametric approximation

F = N (µ, Σ) : µ ∈ Rd , Σ ∈ Sd+ .
mean-field approximation, Θ = Θ1 × Θ2 andF : {ρ : ρ(dθ) = ρ1 (dθ1 ) × ρ2 (dθ2 )} .
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Empirical lower bound (ELBO)
Note that :π̃n,α = arg min K(ρ, πn,α )
ρ∈F
)
(
Z
n
X
1
= arg min −αlog pθ (Xi )ρ(dθ) + K(ρ, π) .
ρ∈F
n i=1
{z
}
|
−ELBO(ρ)

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Empirical lower bound (ELBO)
Note that :π̃n,α = arg min K(ρ, πn,α )
ρ∈F
)
(
Z
n
X
1
= arg min −αlog pθ (Xi )ρ(dθ) + K(ρ, π) .
ρ∈F
n i=1
{z
}
|
−ELBO(ρ)

So we have the equivalent definition :π̃n,α := arg max ELBO(ρ).
ρ∈F

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Consistency results
3 papers (2017) :1

consistency and rates of convergence for α < 1 :P. Alquier & J. Ridgway. Concentration of tempered posteriors and of their variational
approximations. To appear in The Annals of Statistics.

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Consistency results
3 papers (2017) :1

consistency and rates of convergence for α < 1 :P. Alquier & J. Ridgway. Concentration of tempered posteriors and of their variational
approximations. To appear in The Annals of Statistics.

2

extension to models with hidden variables :A. Bhattacharya, D. Pati & Y. Yang. Bayesian fractional posteriors. The Annals of Statistics,
2019.

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Bayesian inference
Definition of variational approximations
Concentration of variational approximations of the posterior

Consistency results
3 papers (2017) :1

consistency and rates of convergence for α < 1 :P. Alquier & J. Ridgway. Concentration of tempered posteriors and of their variational
approximations. To appear in The Annals of Statistics.

2

extension to models with hidden variables :A. Bhattacharya, D. Pati & Y. Yang. Bayesian fractional posteriors. The Annals of Statistics,
2019.

3

extension to α = 1 :F. Zhang & C. Gao. Convergence Rates of Variational Posterior Distributions. Preprint arXiv,
2017.

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1

initialize θ1 ,Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2

initialize θ1 ,x1 revealed,

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )update θ2 → θ3 ,Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1
2

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )update θ2 → θ3 ,x3 revealed,

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1
2
3

4

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )update θ2 → θ3 ,x3 revealed,
incur loss
− log pθ3 (x3 )...
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1
2
3

4

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )Objective :update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )update θ2 → θ3 ,x3 revealed,
incur loss
− log pθ3 (x3 )...
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1
2
3

4

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )Objective : make sure that
we learn to predict well as fast
as possible.

update θ2 → θ3 ,x3 revealed,
incur loss
− log pθ3 (x3 )...
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Sequential estimation problem
1

1
2
3

2

1
2
3

3

1
2
3

4

initialize θ1 ,x1 revealed,
incur loss
− log pθ1 (x1 )update θ1 → θ2 ,x2 revealed,
incur loss
− log pθ2 (x2 )update θ2 → θ3 ,x3 revealed,
incur loss
− log pθ3 (x3 )Objective : make sure that
we learn to predict well as fast
as possible. KeepTX[− log pθt (xt )]t=1

as small as possible for any T ,without stochastic
assumptions on the data.

...
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Reference

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Reference

The regret :R(T ) =

TX[− log pθt (xt )]
t=1

− inf

θ∈Θ

Pierre AlquierTX[− log pθ (xt )].

t=1

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

EWA strategy / multipicative update...

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

EWA strategy / multipicative update...
learning rate α > 0.

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

EWA strategy / multipicative update...
learning rate α > 0.
initialize p1 = π (the prior).

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

EWA strategy / multipicative update...
learning rate α > 0.
initialize p1 = π (the prior).
Algorithm 2 Exponentially Weighted Aggregation
1: for t = 1, 2, . . . do2:
θt = Eθ∼pt [θ],α3:
xt revealed, update pt+1 (dθ) = R[p[pθϑ(x(xt t)])]αppt t(dθ).
(dϑ)
4: end for

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

EWA strategy / multipicative update...
learning rate α > 0.
initialize p1 = π (the prior).
Algorithm 2 Exponentially Weighted Aggregation
1: for t = 1, 2, . . . do2:
θt = Eθ∼pt [θ],α3:
xt revealed, update pt+1 (dθ) = R[p[pθϑ(x(xt t)])]αppt t(dθ).
(dϑ)
4: end forNote that pt = πn,α the tempered posterior, so problem : how
can we compute θt ?
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for EWA
From now, θ 7→ [− log pθ (xt )] is convex + bounded : | · | ≤ C .

Pierre Alquier

Generalization bounds for online variational inferenceSequential estimation problem
Online variational inference
Simulations

Introduction : variational Bayesian inference
Online variational inference

A regret bound for EWA
From now, θ 7→ [− log pθ (xt )] is convex + bounded : | · | ≤ C .
TheoremTX
t=1[− log pθt (xt )] ≤ inf
p

" T
X

Eθ∼p [− log pθ (xt )]t=1#
αC 2 T
K(p, π)++.
2
αPierre Alquier

Generalization bounds for online variational inferenceSequential estimation problem
Online variational inference
Simulations

Introduction : variational Bayesian inference
Online variational inference

A regret bound for EWA
From now, θ 7→ [− log pθ (xt )] is convex + bounded : | · | ≤ C .
TheoremTX[− log pθt (xt )] ≤ inf
p

t=1

" T
X

Eθ∼p [− log pθ (xt )]t=1#
αC 2 T
K(p, π)++.
2αUnder similar assumptions than in the batch case, √that is, theprior gives enough mass to relevant θ, and α ∼ 1/ T ,
T
T
X
X
√
[− log pθt (xt )] ≤ inf[− log pθ (xt )] + cst. T
t=1

θ∈Θ

Pierre Alquier

t=1
Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

T
T
X
X
√
[− log pθt (xt )] ≤ inf[− log pθ (xt )] + cst. T
t=1

θ∈Θ

Pierre Alquier

t=1

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

T
T
X
X
√
[− log pθt (xt )] ≤ inf[− log pθ (xt )] + cst. T
t=1

θ∈Θ

t=1

T
T
1 X
q(xt )1 X
q(xt )
cst
log
log
+√ .
≤ inf
T t=1
pθt (xt ) θ∈Θ T t=1pθ (xt )T

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

T
T
X
X
√
[− log pθt (xt )] ≤ inf[− log pθ (xt )] + cst. T
t=1

θ∈Θ

t=1

T
T
1 X
q(xt )1 X
q(xt )
cst
log
log
+√ .
≤ inf
T t=1
pθt (xt ) θ∈Θ T t=1pθ (xt )T

Assuming that x1 , . . . , xT are actually i.i.d from Q, withdensity q, define
T
1 X
θ̂T =
θT ,
T t=1
we have (“online-to-batch” conversion) :cst
E K Q, Pθ̂T ≤ inf K (Q, Pθ ) + √ .
θ∈ΘTPierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Variational approximations of EWA
B.-E. Chérief-Abdellatif, P. Alquier & M. E. Khan. A Generalization Bound for Online Variational
Inference. Preprint arXiv, 2019.

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Variational approximations of EWA
B.-E. Chérief-Abdellatif, P. Alquier & M. E. Khan. A Generalization Bound for Online Variational
Inference. Preprint arXiv, 2019.

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Variational approximations of EWA
B.-E. Chérief-Abdellatif, P. Alquier & M. E. Khan. A Generalization Bound for Online Variational
Inference. Preprint arXiv, 2019.

Parametric variational approximation : F = {qµ , µ ∈ M}.
Objective : propose a way to update µt → µt+1 so that qµt
leads to similar performances as pt in EWA...
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

SVA and SVB strategies
Algorithm 3 SVA (Sequential Variational Approximation)
1: for t = 1, 2, . . . do2:
θt = Eθ∼qµt [θ],3:
xt revealed, update
#
"
t
X
K(q
,
π)
µ
.
µt+1 = arg min µT ∇µ
Eθ∼qµ [− log pθ (xi )] +
µ∈M
α
i=1
4: end for

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

SVA and SVB strategies
Algorithm 3 SVA (Sequential Variational Approximation)
1: for t = 1, 2, . . . do2:
θt = Eθ∼qµt [θ],3:
xt revealed, update
#
"
t
X
K(q
,
π)
µ
.
µt+1 = arg min µT ∇µ
Eθ∼qµ [− log pθ (xi )] +
µ∈M
α
i=1
4: end for

SVB (Streaming Variational Bayes) has update
K(qµ , qµt )Tµt+1 = arg min µ ∇µ Eθ∼qµ [− log pθ (xt )] +.
µ∈MαPierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

NGVI strategy
NGVI (Natural Gradient Variational Inference) : fix someβ > 0,
µt+1
"

#
K(q
,
π)
K(q
,
q
)
µ
µ
µ
t
= arg min µT ∇µ Eθ∼qµ [− log pθ (xt )]+
+
.
µ∈M
α
β

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

NGVI strategy
NGVI (Natural Gradient Variational Inference) : fix someβ > 0,
µt+1
"

#
K(q
,
π)
K(q
,
q
)
µ
µ
µ
t
= arg min µT ∇µ Eθ∼qµ [− log pθ (xt )]+
+
.
µ∈M
α
β

M. E. Khan & W. Lin. Conjugate-computation variational inference : Converting variational
inference in non-conjugate models to inferences in conjugate models. AISTAT, 2017.

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

An example : SVB with Gaussian approximations
As an example, assume that θ ∈ Rd , the prior is
π = N (0, s 2 I ) and that 
we use the variationalapproximation

σ12 . . . 0 family : qµ = qm,σ = N m,  ... . . . ...  .
0 . . . σd2

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

An example : SVB with Gaussian approximations
As an example, assume that θ ∈ Rd , the prior is
π = N (0, s 2 I ) and that 
we use the variationalapproximation

σ12 . . . 0 family : qµ = qm,σ = N m,  ... . . . ...  .
0 . . . σd2
In this case, the update in SVB is :mt+1 = mt − ασt2 ∇m=mt Eθ∼qm,σt [− log pθ (xt )]
ασt ∇σ=σt Eθ∼qmt ,σ [− log pθ (xt )]σt+1 = σt h2where √means “componentwise multiplication” andh(x) = 1 + x 2 − x is also applied componentwise.
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

An example : SVB with Gaussian approximations
As an example, assume that θ ∈ Rd , the prior is
π = N (0, s 2 I ) and that 
we use the variationalapproximation

σ12 . . . 0 family : qµ = qm,σ = N m,  ... . . . ...  .
0 . . . σd2
In this case, the update in SVB is :mt+1 = mt − ασt2 ∇m=mt Eθ∼qm,σt [− log pθ (xt )]
ασt ∇σ=σt Eθ∼qmt ,σ [− log pθ (xt )]σt+1 = σt h2where √means “componentwise multiplication” andh(x) = 1 + x 2 − x is also applied componentwise. We also
have explicit formulas for SVA and NGVI (see the paper).
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for SVA
Theorem (Chérief-Abdellatif, A. & Khan)
Assume that µ 7→ Eθ∼qµ [− log pθ (xt )] is L-Lipschitz andconvex.

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for SVA
Theorem (Chérief-Abdellatif, A. & Khan)
Assume that µ 7→ Eθ∼qµ [− log pθ (xt )] is L-Lipschitz andconvex. (this is for example the case as soon as the
log-likelihood is concave in θ and L-Lipschitz, and µ is alocation-scale parameter).

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for SVA
Theorem (Chérief-Abdellatif, A. & Khan)
Assume that µ 7→ Eθ∼qµ [− log pθ (xt )] is L-Lipschitz andconvex. Assume that µ 7→ K(pµ , π) is γ-strongly convex.
Then SVA satisfies :TX[− log pθt (xt )]

t=1

)
" T
#
X
αL2 T
K(qµ , π)+.
Eθ∼qµ[− log pθ (xt )] +
γ
α
t=1

(
≤ inf

µ∈M

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for SVA
Theorem (Chérief-Abdellatif, A. & Khan)
Assume that µ 7→ Eθ∼qµ [− log pθ (xt )] is L-Lipschitz andconvex. Assume that µ 7→ K(pµ , π) is γ-strongly convex.
Then SVA satisfies :TX[− log pθt (xt )]

t=1

)
" T
#
X
αL2 T
K(qµ , π)+.
Eθ∼qµ[− log pθ (xt )] +
γ
α
t=1

(
≤ inf

µ∈M

For SVB : some results in the Gaussian case.
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

A regret bound for SVA
Theorem (Chérief-Abdellatif, A. & Khan)
Assume that µ 7→ Eθ∼qµ [− log pθ (xt )] is L-Lipschitz andconvex. Assume that µ 7→ K(pµ , π) is γ-strongly convex.
Then SVA satisfies :TX[− log pθt (xt )]

t=1

)
" T
#
X
αL2 T
K(qµ , π)+.
Eθ∼qµ[− log pθ (xt )] +
γ
α
t=1

(
≤ inf

µ∈M

For SVB : some results in the Gaussian case. For NGVI : wewere not able to derive regret bounds until now.
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Test on a simulated dataset

Figure – Average cumulative losses on different datasets forclassification and regression tasks with OGA (yellow), OGA-EL(red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Test on the Breast dataset

Figure – Average cumulative losses on different datasets forclassification and regression tasks with OGA (yellow), OGA-EL(red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Test on the Pima Indians dataset

Figure – Average cumulative losses on different datasets forclassification and regression tasks with OGA (yellow), OGA-EL(red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Test on the Boston Housing dataset

Figure – Average cumulative losses on different datasets forclassification and regression tasks with OGA (yellow), OGA-EL(red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Test on the Forest Cover Type dataset

Figure – Average cumulative losses on different datasets forclassification and regression tasks with OGA (yellow), OGA-EL(red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Conclusions
1

Using online-to-batch conversion, we now have algorithms
for variational inference with provable statisticalproperties after a finite number of steps.

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Conclusions
1

2

Using online-to-batch conversion, we now have algorithms
for variational inference with provable statisticalproperties after a finite number of steps.
SVA, SVB competitive with OGA (online gradientalgorithm, “non-Bayesian”).

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Conclusions
1

2

3

Using online-to-batch conversion, we now have algorithms
for variational inference with provable statisticalproperties after a finite number of steps.
SVA, SVB competitive with OGA (online gradientalgorithm, “non-Bayesian”).
NGVI is the best method on all datasets. Its theoreticalanalysis is thus an important open problem. Cannot bedone with our current techniques (using naturalparameters in exponential models lead to non-convex
objectives).

Pierre Alquier

Generalization bounds for online variational inferenceIntroduction : variational Bayesian inference
Online variational inference

Sequential estimation problem
Online variational inference
Simulations

Thank you !

Pierre Alquier

Generalization bounds for online variational inference