Concentration of the posterior
Concentration of variational approximations
Further results

Lectures on Variational Inference2) Statistical Analysis of Variational Approximations

Pierre AlquierHeilbronn Institute – University of Bristol – Nov. 27, 2019

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

Lecture 2
1

Concentration of the posteriorReminder on the notations
Theorems on the concentration of the posterior

2Concentration of variational approximations
Theorem for variational approximation
Proof
Applications

3

Further results
Further results in statistical estimation
Further results in machine learning

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posteriorReminder on the notations
1

Concentration of the posteriorReminder on the notations
Theorems on the concentration of the posterior

2Concentration of variational approximations
Theorem for variational approximation
Proof
Applications

3

Further results
Further results in statistical estimation
Further results in machine learning

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posteriorReminder on the notations
From now, and until Section 3, we will focus on parametricestimation in statistics.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posteriorReminder on the notations
From now, and until Section 3, we will focus on parametricestimation in statistics.
Assume that we observe X1 , . . . , Xn i.i.d from P 0 in a model{Pθ , θ ∈ Θ} with p.d.f pθ . Prior π on Θ.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posteriorReminder on the notations
From now, and until Section 3, we will focus on parametricestimation in statistics.
Assume that we observe X1 , . . . , Xn i.i.d from P 0 in a model{Pθ , θ ∈ Θ} with p.d.f pθ . Prior π on Θ. Until Section 3, we
also assume a correct specification : P 0 = Pθ0 .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posteriorReminder on the notations
From now, and until Section 3, we will focus on parametricestimation in statistics.
Assume that we observe X1 , . . . , Xn i.i.d from P 0 in a model{Pθ , θ ∈ Θ} with p.d.f pθ . Prior π on Θ. Until Section 3, we
also assume a correct specification : P 0 = Pθ0 .
The likelihood
Ln (θ) =nYpθ (Xi )

i=1

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posteriorReminder on the notations
From now, and until Section 3, we will focus on parametricestimation in statistics.
Assume that we observe X1 , . . . , Xn i.i.d from P 0 in a model{Pθ , θ ∈ Θ} with p.d.f pθ . Prior π on Θ. Until Section 3, we
also assume a correct specification : P 0 = Pθ0 .
The likelihood
Ln (θ) =nYpθ (Xi )i=1

The tempered posterior - 0 < α < 1πn,α (dθ) ∝ [Ln (θ)]α π(dθ).
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

Concentration of the posterior

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

Concentration of the posterior

Do we have for some d, ∀t > 0hiPθ∼πn,α d(θ, θ0 ) ≥ t −−−→ 0 ?
n→∞Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

Concentration of the posterior

Do we have for some d, ∀t > 0hi?
Pθ∼πn,α d(θ, θ0 ) ≥ t −−−→ 0 ?
n→∞Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

Concentration of the posterior

Do we have for some d, ∀t > 0hi?
Pθ∼πn,α d(θ, θ0 ) ≥ t −−−→ 0 ?
n→∞Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

A simpler result
We will state a simpler results than the concentration
theorems in this book, by following ideas from
A. Bhattacharya, D. Pati & Y. Yang (2019). Bayesian fractional posteriors. The Annals of
Statistics.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

A simpler result
We will state a simpler results than the concentration
theorems in this book, by following ideas from
A. Bhattacharya, D. Pati & Y. Yang (2019). Bayesian fractional posteriors. The Annals of
Statistics.

Definition - Rényi divergence
Assume that P and Q have respective densities p and q withrespect to a measure µ, define, for 0 < α < 1,Z1
Dα (P, R) =log q(x)1−α p(x)α µ(dx).
α−1Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

A simpler result
We will state a simpler results than the concentration
theorems in this book, by following ideas from
A. Bhattacharya, D. Pati & Y. Yang (2019). Bayesian fractional posteriors. The Annals of
Statistics.

Definition - Rényi divergence
Assume that P and Q have respective densities p and q withrespect to a measure µ, define, for 0 < α < 1,Z1
Dα (P, R) =log q(x)1−α p(x)α µ(dx).
α−1
We use d(θ, θ0 ) = Dα (Pθ , Pθ0 ) to measure the concentration.
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

Properties of the Rényi divergence
It is important to note that Dα (P, Q) does not depend on the
choice of µ as can be “seen” fromZ
1
Dα (P, R) =log (dQ)1−α (dP)α .
α−1

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

Properties of the Rényi divergence
It is important to note that Dα (P, Q) does not depend on the
choice of µ as can be “seen” fromZ
1
Dα (P, R) =log (dQ)1−α (dP)α .
α−1
Many properties derived in :T. Van Erven & P. Harremos. Rényi divergence and Kullback-Leibler divergence. IEEE
Transactions on Information Theory, 2014.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

Properties of the Rényi divergence
It is important to note that Dα (P, Q) does not depend on the
choice of µ as can be “seen” fromZ
1
Dα (P, R) =log (dQ)1−α (dP)α .
α−1
Many properties derived in :T. Van Erven & P. Harremos. Rényi divergence and Kullback-Leibler divergence. IEEE
Transactions on Information Theory, 2014.

Among others, for 1/2 ≤ α, link with Hellinger and Kullback :H2 (P, R) ≤ Dα (P, R) −−→ K(P, R).
α%1

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

A theorem in expectation
DefineB(r ) = {θ ∈ Θ : K(Pθ0 , Pθ ) ≤ r } .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

A theorem in expectation
DefineB(r ) = {θ ∈ Θ : K(Pθ0 , Pθ ) ≤ r } .
Theorem (BPY 19, simplified version)
For any sequence (rn ) such that rn ≥ 0 and− log π[B(rn )] ≤ nrn
we have(
EX1 ,...,XnhiEθ∼πn,α Dα (Pθ , Pθ0 )Pierre Alquier, RIKEN AIP

)
≤

1+α
rn .
1−α

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

From expectation to concentration
From the previous result, we can recover concentration results
by a simple application of Markov inequality :Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

From expectation to concentration
From the previous result, we can recover concentration results
by a simple application of Markov inequality :hihi Eθ∼πn,α Dα (Pθ , Pθ0 )Pθ∼πn,α Dα (Pθ , Pθ0 ) ≥ t ≤,tPierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

From expectation to concentration
From the previous result, we can recover concentration results
by a simple application of Markov inequality :hihi Eθ∼πn,α Dα (Pθ , Pθ0 )Pθ∼πn,α Dα (Pθ , Pθ0 ) ≥ t ≤,t(
EX1 ,...,XnhiPθ∼πn,α Dα (Pθ , Pθ0 ) ≥ t

Pierre Alquier, RIKEN AIP)≤(1 + α)rn −−−→ 0,(1 − α)t n→∞

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further resultsReminder on the notations
Theorems on the concentration of the posterior

From expectation to concentration
From the previous result, we can recover concentration results
by a simple application of Markov inequality :hihi Eθ∼πn,α Dα (Pθ , Pθ0 )Pθ∼πn,α Dα (Pθ , Pθ0 ) ≥ t ≤,t(
EX1 ,...,XnhiPθ∼πn,α Dα (Pθ , Pθ0 ) ≥ t)≤(1 + α)rn −−−→ 0,(1 − α)t n→∞hiproba.
Pθ∼πn,α Dα (Pθ , Pθ0 ) ≥ t −−−→ 0.
n→∞Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Generalization to variational approximations
1

Concentration of the posteriorReminder on the notations
Theorems on the concentration of the posterior

2Concentration of variational approximations
Theorem for variational approximation
Proof
Applications

3

Further results
Further results in statistical estimation
Further results in machine learning

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Generalization to variational approximations
In the following paper, we extended BPY’s approach tovariational approximations.
P. Alquier & J. Ridgway (2017). Concentration of tempered posteriors and of their variational
approximations. The Annals of Statistics (to appear), preprint arxiv :1706.09293.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Generalization to variational approximations
In the following paper, we extended BPY’s approach tovariational approximations.
P. Alquier & J. Ridgway (2017). Concentration of tempered posteriors and of their variational
approximations. The Annals of Statistics (to appear), preprint arxiv :1706.09293.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Generalization to variational approximations
In the following paper, we extended BPY’s approach tovariational approximations.
P. Alquier & J. Ridgway (2017). Concentration of tempered posteriors and of their variational
approximations. The Annals of Statistics (to appear), preprint arxiv :1706.09293.

Reminder :π̃n,α = arg min K(ρ, πn,α ).
ρ∈F

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Concentration of VB
Theorem - AR 17
Assume that there is (rn ) and ρn ∈ F such thathiEθ∼ρn K(Pθ0 , Pθ ) ≤ rn
andK(ρn , π) ≤ nrn .
Then, for any α ∈ (0, 1),(
EX1 ,...,XnhiEθ∼π̃n,α Dα (Pθ , Pθ0 )Pierre Alquier, RIKEN AIP

)
≤

1+α
rn .
1−α

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (1/2)
Define the log-likelihood ratio : rn (θ, θ0 ) =

n
X
i=1

Pierre Alquier, RIKEN AIP

logpθ0 (Xi ).
pθ (Xi )Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (1/2)
Define the log-likelihood ratio : rn (θ, θ0 ) =

n
X

log

i=1
1

pθ0 (Xi ).
pθ (Xi )Its expectation :EX1 ,...,Xn [rn (θ, θ0 )] = nK(Pθ0 , Pθ ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (1/2)
Define the log-likelihood ratio : rn (θ, θ0 ) =

n
X

log

i=1
1

pθ0 (Xi ).
pθ (Xi )Its expectation :EX1 ,...,Xn [rn (θ, θ0 )] = nK(Pθ0 , Pθ ).

2

Its exponential moment :EX1 ,...,Xn {exp [−αrn (θ, θ0 )]} =

α
n Z 
Y
pθ (xi )
i=1

=

n
Y

pθ0 (xi )pθ0 (xi )dxi

 Z

α
1−α
exp log pθ (xi ) pθ0 (xi ) dxi

i=1= exp [−n(1 − α)Dα (Pθ , Pθ0 )] .
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (2/2)

Start from the exponential moment :EX1 ,...,Xn {exp [−αrn (θ, θ0 ) + n(1 − α)Dα (Pθ , Pθ0 )]} = 1.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (2/2)

Start from the exponential moment :EX1 ,...,Xn {exp [−αrn (θ, θ0 ) + n(1 − α)Dα (Pθ , Pθ0 )]} = 1.
Expectation w.r.t θ ∼ π and Fubini :EX1 ,...,Xn Eθ∼π {exp [−αrn (θ, θ0 ) + n(1 − α)Dα (Pθ , Pθ0 )]} = 1.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (2/2)
Start from the exponential moment :EX1 ,...,Xn {exp [−αrn (θ, θ0 ) + n(1 − α)Dα (Pθ , Pθ0 )]} = 1.
Expectation w.r.t θ ∼ π and Fubini :EX1 ,...,Xn Eθ∼π {exp [−αrn (θ, θ0 ) + n(1 − α)Dα (Pθ , Pθ0 )]} = 1.
Then :
EX1 ,...,Xn Eθ∼π̃n,αh
exp −αrn (θ, θ0 ) + n(1 − α)Dα (Pθ , Pθ0 ) π(θ) i
+ log
= 1.
π̃n,α (θ)

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (2/2)
Then :
EX1 ,...,Xn Eθ∼π̃n,αh
exp −αrn (θ, θ0 ) + n(1 − α)Dα (Pθ , Pθ0 ) π(θ) i
= 1.
+ log
π̃n,α (θ)

Jensen’s inequality :hexp EX1 ,...,Xn Eθ∼π̃n,α −αrn (θ, θ0 ) + n(1 − α)Dα (Pθ , Pθ0 ) π(θ) i
+ log
≤ 1.
π̃n,α (θ)
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (2/2)
Jensen’s inequality :hexp EX1 ,...,Xn Eθ∼π̃n,α −αrn (θ, θ0 ) + n(1 − α)Dα (Pθ , Pθ0 ) π(θ) i
≤ 1.
+ log
π̃n,α (θ)
Rearranging :EX1 ,...,Xn Eθ∼π̃n,α [Dα (Pθ , Pθ0 ]()
1
EX ,...,X αEθ∼π̃n,α [rn (θ, θ0 )] + K(π̃n,α , π)≤n(1 − α) 1 n |{z}Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (2/2)
Jensen’s inequality :hexp EX1 ,...,Xn Eθ∼π̃n,α −αrn (θ, θ0 ) + n(1 − α)Dα (Pθ , Pθ0 ) π(θ) i
≤ 1.
+ log
π̃n,α (θ)
Rearranging :EX1 ,...,Xn Eθ∼π̃n,α [Dα (Pθ , Pθ0 ]()
1≤
EX ,...,X αEθ∼π̃n,α [rn (θ, θ0 )] + K(π̃n,α , π)n(1 − α) 1 n |{z}=−αELBO(π̃n,α )+ constant

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (2/2)
Rearranging :EX1 ,...,Xn Eθ∼π̃n,α [Dα (Pθ , Pθ0 ])(1
EX ,...,X αEθ∼π̃n,α [rn (θ, θ0 )] + K(π̃n,α , π)≤n(1 − α) 1 n |{z}=−αELBO(π̃n,α )+ constantAs πn,α minimizes the ELBO :EX1 ,...,Xn Eθ∼π̃n,α [Dα (Pθ , Pθ0 ]
 h
i
1
≤
EX ,...,X inf αEθ∼ρ [rn (θ, θ0 )] + K(ρ, π)n(1 − α) 1 n ρ∈F


h
i
1
≤
inf nαEθ∼ρ K(Pθ0 , Pθ ) + K(ρ, π) .
n(1 − α) ρ∈F
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Proof (2/2)
As πn,α minimizes the ELBO :EX1 ,...,Xn Eθ∼π̃n,α [Dα (Pθ , Pθ0 ] hi
1EX ,...,X inf αEθ∼ρ [rn (θ, θ0 )] + K(ρ, π)≤n(1 − α) 1 n ρ∈F


h
i
1
≤
inf nαEθ∼ρ K(Pθ0 , Pθ ) + K(ρ, π) .
n(1 − α) ρ∈F
We end the proof by using the assumption that there is aρ ∈ F such that Eθ∼ρ [K(Pθ0 , Pθ )] ≤ rn and K(ρ, π) ≤ nrn :hi
EX1 ,...,Xn Eθ∼π̃n,α Dα (Pθ , Pθ0 ) ≤Pierre Alquier, RIKEN AIP

11+α[αnrn + nrn ] =rn .
n(1 − α)1−αLectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

A toy example : Gaussian variables
Toy example : assume that X1 , . . . , Xn are i.i.d fromPθ0 = N (θ0 , 1).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

A toy example : Gaussian variables
Toy example : assume that X1 , . . . , Xn are i.i.d fromPθ0 = N (θ0 , 1). Cauchy prior θ ∼ π = C(0, 1).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

A toy example : Gaussian variables
Toy example : assume that X1 , . . . , Xn are i.i.d fromPθ0 = N (θ0 , 1). Cauchy prior θ ∼ π = C(0, 1).
Gaussian approximation of the posterior :
F = N (m, σ 2 ), m ∈ R, σ 2 > 0 .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

A toy example : Gaussian variables
Toy example : assume that X1 , . . . , Xn are i.i.d fromPθ0 = N (θ0 , 1). Cauchy prior θ ∼ π = C(0, 1).
Gaussian approximation of the posterior :
F = N (m, σ 2 ), m ∈ R, σ 2 > 0 .
Note that K(Pθ0 , Pθ ) = |θ − θ0 |2 /2 and sohi |m − θ |2 + σ 20Eθ∼N (m,σ2 ) K(Pθ0 , Pθ ) =2

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

A toy example : Gaussian variables
Toy example : assume that X1 , . . . , Xn are i.i.d fromPθ0 = N (θ0 , 1). Cauchy prior θ ∼ π = C(0, 1).
Gaussian approximation of the posterior :
F = N (m, σ 2 ), m ∈ R, σ 2 > 0 .
Note that K(Pθ0 , Pθ ) = |θ − θ0 |2 /2 and sohi |m − θ |2 + σ 20Eθ∼N (m,σ2 ) K(Pθ0 , Pθ ) =2
Moreover, rough upper bounds lead to
r
r

π
2σ 2
2
2
K(N (m, σ ), π) ≤ log+log(1+2m)+
2σ 2π
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

A toy example : Gaussian variables
Toy example : assume that X1 , . . . , Xn are i.i.d fromPθ0 = N (θ0 , 1). Cauchy prior θ ∼ π = C(0, 1).
Gaussian approximation of the posterior :
F = N (m, σ 2 ), m ∈ R, σ 2 > 0 .
Note that K(Pθ0 , Pθ ) = |θ − θ0 |2 /2 and sohi |m − θ |2 + σ 20Eθ∼N (m,σ2 ) K(Pθ0 , Pθ ) =≤ rn ?
2
Moreover, rough upper bounds lead to
r
r

π
2σ 2
2
2
≤ nrn ?
K(N (m, σ ), π) ≤ log+log(1+2m)+
2σ 2π
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Gaussian example (continued)|m − θ0 |2 + σ 2
≤ rn
2r
r


π
2σ 2
2
+
log(1
+
2m
)
+
≤ nrn
log
2σ 2
π

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Gaussian example (continued)|m − θ0 |2 + σ 2
≤ rn
2r
r


π
2σ 2
2
+
log(1
+
2m
)
+
≤ nrn
log
2σ 2
π
For example satisfied by m = θ0 , σ 2 = 1/n andp1
log nπ+ log(1 + 2θ02 ) + π2
2
2
rn =
.
n

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Gaussian example (continued)|m − θ0 |2 + σ 2
≤ rn
2r
r


π
2σ 2
2
+
log(1
+
2m
)
+
≤ nrn
log
2σ 2
π
For example satisfied by m = θ0 , σ 2 = 1/n andp1
log nπ+ log(1 + 2θ02 ) + π2
2
2
rn =
.
n
We can apply our theorem :
(
)
h
i
1+α
rn .
EX1 ,...,Xn Eθ∼π̃n,α Dα (Pθ , Pθ0 )≤1−αPierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Gaussian example (continued)(hEX1 ,...,Xn Eθ∼π̃n,α Dα (Pθ , Pθ0 )Pierre Alquier, RIKEN AIP

)
i

≤

1+α
rn .
1−α

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Gaussian example (continued)
(

h

)
i

Dα (Pθ , Pθ0 ) =|θ − θ0 |2.
2α

EX1 ,...,Xn Eθ∼π̃n,α Dα (Pθ , Pθ0 )

≤

1+α
rn .
1−α

Here, we have actually

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Gaussian example (continued)
(

h

)
i

Dα (Pθ , Pθ0 ) =|θ − θ0 |2.
2α

EX1 ,...,Xn Eθ∼π̃n,α Dα (Pθ , Pθ0 )

≤

1+α
rn .
1−α

Here, we have actually)ihEθ∼π̃n,α |θ − θ0 |2(
EX1 ,...,Xn≤α(1 + α)
1−α

1
2

log

Pierre Alquier, RIKEN AIP

nπ
2



+ log(1 + 2θ02 ) +nLectures on Variational Inference - 2

pπ
2

.

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Second example : matrix completion

ClaireNial
BrendonAndrew
AdrianDamien
..
.

4
?
?
?
1
?
..
.
Pierre Alquier, RIKEN AIP

?
4
5
4
?
1
..
.

3
?
4
?
?
?
..
.

...
...
...
...
...
...
...

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Matrix completion (continued)
Reminder on matrix completion :M=kXU·,` (V·,` )T is p × m`=1

with prior π given byU·,` , V·,` ∼ N (0, γ` I ),1
∼ Gamma(a, b).
γ`

Mean-field variational approximation with Gaussianand inverse gamma disitributions on U, V and γ` respectively.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Theorem for variational approximation
Proof
Applications

Matrix completion : rate of convergence(
EX1 ,...,Xn)hiEθ∼π̃n,α Dα (PM , PM0 )=Orank(M0 )(m + p) + log(nmp)nPierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2.

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Generalization to variational approximations
1

Concentration of the posteriorReminder on the notations
Theorems on the concentration of the posterior

2Concentration of variational approximations
Theorem for variational approximation
Proof
Applications

3

Further results
Further results in statistical estimation
Further results in machine learning

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Misspecified case

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Misspecified case
Assume we observe X1 , . . . , Xn i.i.d from P 0 and use a model
M = {Pθ , θ ∈ Θ}, but it is possible that P 0 ∈/ M.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Misspecified case
Assume we observe X1 , . . . , Xn i.i.d from P 0 and use a model
M = {Pθ , θ ∈ Θ}, but it is possible that P 0 ∈/ M.
Theorem - AR 17
Assume that there is (rn ) and ρn ∈ F such that

 

pθ∗ (x)Eθ∼ρ EX ∼P 0 log≤ rnpθ (x)and K(ρn , π) ≤ nrn . Then()
hi
EX1 ,...,Xn Eθ∼π̃n,α Dα (Pθ , P 0 ) ≤Pierre Alquier, RIKEN AIP

1+αα
K(Pθ∗ , P 0 )+
rn .
1−α
1−α

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Model selectionkAssume that we have K models, define π̃n,αa variational(k)approximation of the tempered posterior in model k, and rn
its convergence rate if model k is correct. Put :(k)).
k̂ = arg max ELBO(π̃n,αkPierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Model selectionkAssume that we have K models, define π̃n,αa variational(k)approximation of the tempered posterior in model k, and rn
its convergence rate if model k is correct. Put :(k)).
k̂ = arg max ELBO(π̃n,αkTheoremIf the true model is actually k0 ,Z
1 + α (k0 )log(K )0k̂E
Dα (Pθ , P )π̃n,α (dθ) ≤
rn +
.
1−α
n(1 − α)Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Model selection
TheoremIf the true model is actually k0 ,Z

1 + α (k0 )log(K )0k̂E
Dα (Pθ , P )π̃n,α (dθ) ≤
rn +
.
1−α
n(1 − α)This result is actually due to
my PhD student :B.-E. Chérief-Abdellatif (2018).
Consistency of ELBO maximization formodel selection. AABI.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Models with hidden variables (1/2)
The results presented so far do not include approximations inmodels with hidden variables.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Models with hidden variables (1/2)
The results presented so far do not include approximations inmodels with hidden variables.
VB for mixtures
P
Pp,θ = Ki=1 pi qθi ,Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Models with hidden variables (1/2)
The results presented so far do not include approximations inmodels with hidden variables.
VB for mixtures
P
Pp,θ = Ki=1 pi qθi ,VB approximation : the θi ’s are mutually independent andindependent from (p1 , . . . , pK ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Models with hidden variables (1/2)
The results presented so far do not include approximations inmodels with hidden variables.
VB for mixtures
P
Pp,θ = Ki=1 pi qθi ,VB approximation : the θi ’s are mutually independent andindependent from (p1 , . . . , pK ).
Under suitable assumptions, rn ∼ K log(n).
nPierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Models with hidden variables (1/2)
The results presented so far do not include approximations inmodels with hidden variables.
VB for mixtures
P
Pp,θ = Ki=1 pi qθi ,VB approximation : the θi ’s are mutually independent andindependent from (p1 , . . . , pK ).
Under suitable assumptions, rn ∼ K log(n).
nB.-E. Chérief-Abdellatif, P. Alquier (2018). Consistency of Variational Bayes Inference for
Estimation and Model Selection in Mixtures. Electronic Journal of Statistics.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Models with hidden variables (2/2)
For a general approach for models with hidden variables,
including1
mixture models,2
hidden Markov chains,
3
...
see :
Y. Yang, D. Pati & A. Bhattacharya (2017). α-Variational Inference with Statistical Guarantees.
The Annals of Statistics (to appear), preprint arXiv :1712.08983.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

The case α = 1
Our paper does not cover the case α = 1 :Case α = 1[Ln (θ)]α π(dθ) = Ln (θ)π(dθ)

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

The case α = 1
Our paper does not cover the case α = 1 :Case α = 1[Ln (θ)]α π(dθ) = Ln (θ)π(dθ)
Covered in the following paper – note that this case requires
much stronger assumptions :F. Zhang & C. Gao (2017). Convergence Rates of Variational Posterior Distributions. Preprint
arxiv :1712.02519.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

More general machine learning problem
Reminder of the context of machine learning :1
X1 , . . . , Xn i.i.d from P 0 ,2
R(θ) = EX ∼P 0 [`(θ, X )].
P
3
Rn (θ) = n1 ni=1 `(θ, Xi ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

More general machine learning problem
Reminder of the context of machine learning :1
X1 , . . . , Xn i.i.d from P 0 ,2
R(θ) = EX ∼P 0 [`(θ, X )].
P
3
Rn (θ) = n1 ni=1 `(θ, Xi ).
Gibbs posterior, EWA, ...
πn,α (θ) ∝ exp[−αnRn (θ)]π(θ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

More general machine learning problem
Reminder of the context of machine learning :1
X1 , . . . , Xn i.i.d from P 0 ,2
R(θ) = EX ∼P 0 [`(θ, X )].
P
3
Rn (θ) = n1 ni=1 `(θ, Xi ).
Gibbs posterior, EWA, ...
πn,α (θ) ∝ exp[−αnRn (θ)]π(θ).
We define a variational approximation of the Gibbs posterior :π̃n,α (θ) = arg min K(ρ, πn,α ).
ρ∈F

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Variational approximation of Gibbs posteriors
Bounds on the generalization error of the variational
approximation of the Gibbs posteriorEθ∼π̃n,α [R(θ)]
provided in the paper :P. Alquier, J. Ridgway , N. Chopin (2016). On the Properties of Variational Approximations ofGibbs Posteriors. JMLR.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Neural networks

Prior π : independent(`)

θi,j ∼ N (0, σ`2 ).
Variational approximation :independent
(`)

(`)

(`)

θi,j ∼ N (mi,j , (σi,j )2 ).
Source : Wikipedia.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Neural networks for non parametric regression
Badr-Eddine Chérief-Abdellatif proved that, in regression withquadratic loss, suitable neural networks estimate β-Hölder
functions at rate :log(n)2O.
2β
n 2β+1

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Neural networks for non parametric regression
Badr-Eddine Chérief-Abdellatif proved that, in regression withquadratic loss, suitable neural networks estimate β-Hölder
functions at rate :log(n)2O.
2β
n 2β+1

B.-E. Chérief-Abdellatif (2018). Convergence Rates of Variational Inference in Sparse Deep
Learning. Preprint arXiv :1908 :04847.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2

Concentration of the posterior
Concentration of variational approximations
Further results

Further results in statistical estimation
Further results in machine learning

Thank you !

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 2