Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Lectures on Variational Inference
1) Approximate Bayesian Inference in Machine Learning

Pierre AlquierHeilbronn Institute – University of Bristol – Nov. 27, 2019

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Statistical learning problem
1

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Statistical learning problem
1
2

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X .
We have a loss function` : Θ × X → R+ .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Statistical learning problem
1
2

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X .
We have a loss function` : Θ × X → R+ .

3

Objective : base on the sample, learn θ0 ∈ Θ which
minimizes the riskR(θ) = EX ∼P 0 [`(θ, X )].

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 1 : supervised classification
1

We observe X1 = (Z1 , Y1 ), . . . , Xn = (Zn , Yn ) i.i.d fromP 0 unknown in Rd × {0, 1}.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 1 : supervised classification
1

2

We observe X1 = (Z1 , Y1 ), . . . , Xn = (Zn , Yn ) i.i.d fromP 0 unknown in Rd × {0, 1}.
Consider a set of predictors (fθ , θ ∈ Θ) withfθ : Rd → {0, 1} and
0 if y = fθ (z),`(θ, (z, y )) = 1y 6=fθ (z) =1 if y 6= fθ (z).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 1 : supervised classification
1

2

3

We observe X1 = (Z1 , Y1 ), . . . , Xn = (Zn , Yn ) i.i.d fromP 0 unknown in Rd × {0, 1}.
Consider a set of predictors (fθ , θ ∈ Θ) withfθ : Rd → {0, 1} and
0 if y = fθ (z),`(θ, (z, y )) = 1y 6=fθ (z) =1 if y 6= fθ (z).
Objective : learn θ0 ∈ Θ which minimizes theclassification error
R(θ) = EX ∼P 0 [`(θ, X )] = P(Z ,Y )∼P 0 [Y 6= fθ (Z )].
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : parametric estimation
1

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X , with
p.d.f p 0 .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : parametric estimation
1

2

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X , with
p.d.f p 0 .
Consider a parametric family of probability distributions :(Pθ , θ ∈ Θ) with p.d.f pθ and
1`(x, θ) = − log pθ (x) = log.
pθ (x)

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : parametric estimation
1

2

3

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X , with
p.d.f p 0 .
Consider a parametric family of probability distributions :(Pθ , θ ∈ Θ) with p.d.f pθ and
1`(x, θ) = − log pθ (x) = log.
pθ (x)
Objective : learn θ0 ∈ Θ which minimizesR(θ) = EX ∼P 0 log
Pierre Alquier, RIKEN AIP1pθ (X )



Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : parametric estimation
1

2

3

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X , with
p.d.f p 0 .
Consider a parametric family of probability distributions :(Pθ , θ ∈ Θ) with p.d.f pθ and
1`(x, θ) = − log pθ (x) = log.
pθ (x)
Objective : learn θ0 ∈ Θ which minimizesR(θ) = EX ∼P 0

 0



p (X )
log− EX ∼P 0 log p 0 (X )pθ (X )

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : parametric estimation
1

2

3

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X , with
p.d.f p 0 .
Consider a parametric family of probability distributions :(Pθ , θ ∈ Θ) with p.d.f pθ and
1`(x, θ) = − log pθ (x) = log.
pθ (x)
Objective : learn θ0 ∈ Θ which minimizes the Kullback
divergence
R(θ) = K(P0 , Pθ ) − constant.
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : MLE and Bayesian inferenceR(θ) = −EX ∼P 0 [log (pθ (X ))] .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : MLE and Bayesian inferenceR(θ) = −EX ∼P 0 [log (pθ (X ))] .
nEstimator of R(θ) : Rn (θ) = −Pierre Alquier, RIKEN AIPlog(Ln (θ))
1X
log (pθ (Xi )) = −
.
n i=1
n

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : MLE and Bayesian inferenceR(θ) = −EX ∼P 0 [log (pθ (X ))] .
nEstimator of R(θ) : Rn (θ) = −MLE

Pierre Alquier, RIKEN AIPlog(Ln (θ))
1X
log (pθ (Xi )) = −
.
n i=1
n
Bayesian inference

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : MLE and Bayesian inferenceR(θ) = −EX ∼P 0 [log (pθ (X ))] .
nEstimator of R(θ) : Rn (θ) = −MLElog(Ln (θ))
1X
log (pθ (Xi )) = −
.
n i=1
n
Bayesian inferenceθ̂ = arg minθ∈Θ Rn (θ)

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : MLE and Bayesian inferenceR(θ) = −EX ∼P 0 [log (pθ (X ))] .
nEstimator of R(θ) : Rn (θ) = −MLEθ̂ = arg minθ∈Θ Rn (θ)

log(Ln (θ))
1X
log (pθ (Xi )) = −
.
n i=1
n

Bayesian inference
prior π(θ)(θ)π(θ)π(θ|X1 , . . . , Xn ) = R LLnn(θ)π(θ)dθ

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Example 2 : MLE and Bayesian inferenceR(θ) = −EX ∼P 0 [log (pθ (X ))] .
nEstimator of R(θ) : Rn (θ) = −MLEθ̂ = arg minθ∈Θ Rn (θ)

log(Ln (θ))
1X
log (pθ (Xi )) = −
.
n i=1
n

Bayesian inference
prior π(θ)(θ)π(θ)π(θ|X1 , . . . , Xn ) = R LLnn(θ)π(θ)dθThe posteriorπ(θ|X1 , . . . , Xn ) ∝ exp[−nRn (θ)]π(θ).
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

General solution : Gibbs posterior
For the general machine learning problem :1
X1 , . . . , Xn i.i.d from P 0 ,2
R(θ) = EX ∼P 0 [`(θ, X )].

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

General solution : Gibbs posterior
For the general machine learning problem :1
X1 , . . . , Xn i.i.d from P 0 ,2
R(θ) = EX ∼P 0 [`(θ, X )].
Definen
1X`(θ, Xi ).
Rn (θ) =n i=1

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

General solution : Gibbs posterior
For the general machine learning problem :1
X1 , . . . , Xn i.i.d from P 0 ,2
R(θ) = EX ∼P 0 [`(θ, X )].
Definen
1X`(θ, Xi ).
Rn (θ) =n i=1
Gibbs posterior, EWA, ...
πn,α (θ) ∝ exp[−αnRn (θ)]π(θ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Back to Example 2 : tempered posteriors
Note : in the case of parametric estimation,"
πn,α (θ) ∝ exp[−αnRn (θ)]π(θ) = exp αnX#log pθ (Xi ) π(θ).

i=1Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Back to Example 2 : tempered posteriors
Note : in the case of parametric estimation,"
πn,α (θ) ∝ exp[−αnRn (θ)]π(θ) = exp αnX#log pθ (Xi ) π(θ).

i=1

The tempered posterior
πn,α (θ) ∝ [Ln (θ)]α π(dθ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Back to Example 2 : tempered posteriors
Note : in the case of parametric estimation,"
πn,α (θ) ∝ exp[−αnRn (θ)]π(θ) = exp αnX#log pθ (Xi ) π(θ).

i=1

The tempered posterior
πn,α (θ) ∝ [Ln (θ)]α π(dθ).
Tempered posteriors are actually very useful for statisticalinference.
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Back to Example 2 : tempered posteriors
easier to sample from.
R.M. Neal. (1996). Sampling from multimodal distributions using tempered transitions. Statistics
and Computing.
G. Behrens, N. Friel & M. Hurn. (2012). Tuning tempered transitions. Statistics and Computing.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Back to Example 2 : tempered posteriors
easier to sample from.
R.M. Neal. (1996). Sampling from multimodal distributions using tempered transitions. Statistics
and Computing.
G. Behrens, N. Friel & M. Hurn. (2012). Tuning tempered transitions. Statistics and Computing.

more robust to model misspecification.
P. Grünwald and T. Van Ommen (2017). Inconsistency of Bayesian inference for misspecified
linear models, and a proposal for repairing it. Bayesian Analysis.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Back to Example 2 : tempered posteriors
easier to sample from.
R.M. Neal. (1996). Sampling from multimodal distributions using tempered transitions. Statistics
and Computing.
G. Behrens, N. Friel & M. Hurn. (2012). Tuning tempered transitions. Statistics and Computing.

more robust to model misspecification.
P. Grünwald and T. Van Ommen (2017). Inconsistency of Bayesian inference for misspecified
linear models, and a proposal for repairing it. Bayesian Analysis.

theoretical analysis simpler.
A. Bhattacharya, D. Pati & Y. Yang (2019). Bayesian fractional posteriors. The Annals of
Statistics.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Notation – summary
1

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Notation – summary
1
2

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X .
Loss function `.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Notation – summary
1
2
3

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X .
Loss function `.
Minimize R(θ) = EX ∼P 0 [`(θ, X )].

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Notation – summary
1
2
3
4

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X .
Loss function `.
Minimize R(θ) = EX ∼P 0 [`(θ, X )].
Empirical riskn1X`(θ, Xi ).
Rn (θ) =n i=1

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Notation – summary
1
2
3
4

We observe X1 , . . . , Xn i.i.d from P 0 unknown in X .
Loss function `.
Minimize R(θ) = EX ∼P 0 [`(θ, X )].
Empirical riskn1X`(θ, Xi ).
Rn (θ) =n i=1
5Temperature α > 0, Gibbs or tempered posterior
πn,α (θ) ∝ exp[−αnRn (θ)]π(θ).
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Computational problem
Apart from a few classical examples, πn,α is intractable.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Computational problem
Apart from a few classical examples, πn,α is intractable.
Popular methods to compute / sample from the (tempered)posterior :Monte-Carlo methods : MCMC (Gibbs Sampler,
Metropolis-Hastings), SMC, Langevin Monte-Carlo, ABC
etc.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Computational problem
Apart from a few classical examples, πn,α is intractable.
Popular methods to compute / sample from the (tempered)posterior :Monte-Carlo methods : MCMC (Gibbs Sampler,
Metropolis-Hastings), SMC, Langevin Monte-Carlo, ABC
etc.
optimization methods : variational approximations orvariational inference (VI) and expectation-propagation(EP).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Roadmap

I) Lecture 1 : approximate Bayesian inference in ML.
12
3

Introduction.
Definition of VI.
Examples of VI.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Roadmap

I) Lecture 1 : approximate Bayesian inference in ML.
12
3

Introduction.
Definition of VI.
Examples of VI.

II) Lecture 2 : statistical analysis of VI.
With theorems and all !

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Roadmap

I) Lecture 1 : approximate Bayesian inference in ML.
12
3

Introduction.
Definition of VI.
Examples of VI.

II) Lecture 2 : statistical analysis of VI.
With theorems and all !

III) Seminar (Thursday next week) : online VI.
For data streams or large-scale learning.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Bayesian learning
Computational issues
Roadmap

Lecture 1
1

Introduction : computational issues in Bayesian learning
Bayesian learning
Computational issues
Roadmap

2

Variational Approximations : Definition
Definition of VI
The ELBO
Strategies for ELBO maximization
Examples of Variational Approximations in Machine
Learning
Recommender systems and matrix completion
Deep learning

3

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Variational Approximations : Definition
1

Introduction : computational issues in Bayesian learning
Bayesian learning
Computational issues
Roadmap

2

Variational Approximations : Definition
Definition of VI
The ELBO
Strategies for ELBO maximization
Examples of Variational Approximations in Machine
Learning
Recommender systems and matrix completion
Deep learning

3

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Reminder on Kullback divergence
Definition – Kullback divergence
Let P and Q be two probability distributions with p.d.f p andq respectively. Then :
 

 Z

dP
p(u)
K(P, Q) = EU∼P log
p(u)du.
(U)
= log
dQ
q(u)

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Reminder on Kullback divergence
Definition – Kullback divergenceK(P, Q) = EU∼P log



 Z


dP
p(u)
(U)
= log
p(u)du.
dQq(u)TheoremK(P, Q) ≥ 0 and K(P, Q) = 0 ⇔ P = Q.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Reminder on Kullback divergence
Definition – Kullback divergenceK(P, Q) = EU∼P log



 Z


dP
p(u)
(U)
= log
p(u)du.
dQq(u)TheoremK(P, Q) ≥ 0 and K(P, Q) = 0 ⇔ P = Q.
Proof :
q(u)p(u)duK(P, Q) = − log
p(u)
Z 


q(u)
≥ − logp(u)du ≥ log(1) = 0.
p(u)
Z



Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Definition of VI
1

Chose a tractable family F of probability distributions on
the parameter θ,

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Definition of VI
1

2

Chose a tractable family F of probability distributions on
the parameter θ,Defineπ̃n,α = arg min K(ρ, πn,α ).
ρ∈F

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Definition of VI
1

2

Chose a tractable family F of probability distributions on
the parameter θ,Defineπ̃n,α = arg min K(ρ, πn,α ).
ρ∈F

Examples of F :parametric approximation

F = N (µ, Σ) : µ ∈ Rd , Σ ∈ Sd+ .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Definition of VI
1

2

Chose a tractable family F of probability distributions on
the parameter θ,Defineπ̃n,α = arg min K(ρ, πn,α ).
ρ∈F

Examples of F :parametric approximation

F = N (µ, Σ) : µ ∈ Rd , Σ ∈ Sd+ .
mean-field approximation, θ = (θ1 , θ2 ) ∈ Θ = Θ1 × Θ2 ,F : {ρ : ρ(dθ) = ρ1 (dθ1 ) ⊗ ρ2 (dθ2 )} .
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

The ELBO
0 ≤ K(ρ, πn,α )
 

dρ
= Eθ∼ρ log
(θ)
dπn,α

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

The ELBO
0 ≤ K(ρ, πn,α )
 

dρ
= Eθ∼ρ log
(θ)
dπn,α
 

dρ Eθ∼π [exp(−nαRn (θ))]= Eθ∼ρ log(θ)
dπ
exp(−nαRn (θ))

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

The ELBO
0 ≤ K(ρ, πn,α )
 

dρ
= Eθ∼ρ log
(θ)
dπn,α
 

dρ Eθ∼π [exp(−nαRn (θ))]= Eθ∼ρ log(θ)
dπ
exp(−nαRn (θ))= nαEθ∼ρ [Rn (θ)] + K(ρ, π) + log Eθ∼π [exp(−nαRn (θ))] .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

The ELBO
0 ≤ K(ρ, πn,α )
 

dρ
= Eθ∼ρ log
(θ)
dπn,α
 

dρ Eθ∼π [exp(−nαRn (θ))]= Eθ∼ρ log(θ)
dπ
exp(−nαRn (θ))= nαEθ∼ρ [Rn (θ)] + K(ρ, π) + log Eθ∼π [exp(−nαRn (θ))] .
That is,Evidence = log Eθ∼π [exp(−nαRn (θ))]≥ −nαEθ∼ρ [Rn (θ)] − K(ρ, π)= ELBO(ρ) (Evidence Lower Bound).
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Alternative definition of VI with ELBO
We end up with two definitions of VI.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Alternative definition of VI with ELBO
We end up with two definitions of VI.
1

best approximation of the posteriorπ̃n,α = arg min K(ρ, πn,α ),ρ∈F

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Alternative definition of VI with ELBO
We end up with two definitions of VI.
1

best approximation of the posteriorπ̃n,α = arg min K(ρ, πn,α ),ρ∈F

2maximization of the evidence lower boundπ̃n,α = arg max ELBO(ρ)ρ∈F= arg max {−nαEθ∼ρ [Rn (θ)] − K(ρ, π)} .
ρ∈F

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Donsker and Varadhan’s variational inequality
Remark : from the above inequality
0 ≤ K(ρ, πn,α ) = −ELBO(ρ) + evidence,

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Donsker and Varadhan’s variational inequality
Remark : from the above inequality
0 ≤ K(ρ, πn,α ) = −ELBO(ρ) + evidence,
it is clear that without the constraint ρ ∈ F, the ELBO ismaximized by ρ = πn,α .

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Donsker and Varadhan’s variational inequality
Remark : from the above inequality
0 ≤ K(ρ, πn,α ) = −ELBO(ρ) + evidence,
it is clear that without the constraint ρ ∈ F, the ELBO ismaximized by ρ = πn,α .
Theorem : Donsker and Varadhan’s variational inequality
K(πn,α , π)K(ρ, π)Eθ∼πn,α [Rn (θ)] += inf Eθ∼ρ [Rn (θ)] +
.
ρ
nα
nα

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

How to maximize the ELBO ?

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

How to maximize the ELBO ?
Parametric variational inference : F = {qλ , λ ∈ Λ}.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

How to maximize the ELBO ?
Parametric variational inference : F = {qλ , λ ∈ Λ}.
Gradient algorithm
λt+1 = λt + η∇ELBO(qλt ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

How to maximize the ELBO ?
Parametric variational inference : F = {qλ , λ ∈ Λ}.
Gradient algorithm
λt+1 = λt + η∇ELBO(qλt ).
Usually, the gradient is not available in closed-form but often itˆis possible to build an unbiased estimate of it : ∇ELBO(qλt ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

How to maximize the ELBO ?
Parametric variational inference : F = {qλ , λ ∈ Λ}.
Gradient algorithm
λt+1 = λt + η∇ELBO(qλt ).
Usually, the gradient is not available in closed-form but often itˆis possible to build an unbiased estimate of it : ∇ELBO(qλt ).
Stochastic gradient algorithmˆ
λt+1 = λt + η ∇ELBO(qλt ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Stochastic gradient of the ELBO
Ex : qλ = N (µ, σ 2 I ), λ = (µ, σ) ∈ Rd × R∗+ , π = N (0, I ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Stochastic gradient of the ELBO
Ex : qλ = N (µ, σ 2 I ), λ = (µ, σ) ∈ Rd × R∗+ , π = N (0, I ).
ELBO(qλ )= −Eθ∼qλ [nαRn (θ)] − K(qλ , π)
= −Eθ∼N (0,I ) [nαRn (µ + σθ)] −kµk2 + k(σ 2 − log(σ 2 ) − 1)2and so, under for a smooth Rn ,
∇ELBO(qλ ) = −Eθ∼N (0,I ) [nα∇λ Rn (µ + σθ)]−Pierre Alquier, RIKEN AIP

k
2

µ 
σ − σ2

Lectures on Variational Inference - 1,Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Stochastic gradient of the ELBO
Ex : qλ = N (µ, σ 2 I ), λ = (µ, σ) ∈ Rd × R∗+ , π = N (0, I ).
ELBO(qλ )= −Eθ∼qλ [nαRn (θ)] − K(qλ , π)
= −Eθ∼N (0,I ) [nαRn (µ + σθ)] −kµk2 + k(σ 2 − log(σ 2 ) − 1)2and so, under for a smooth Rn ,
∇ELBO(qλ ) = −Eθ∼N (0,I ) [nα∇λ Rn (µ + σθ)]−ˆ∇ELBO(qλ) = −k2µ σ − σ2



Pm

nα
∇R
(µ
+
σθ
)
+
µ
n
j
j=1
 .
Pmm
nα
k
2
j=1 θj ∇Rn (µ + σθj ) + 2 σ − σmPierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1,

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Mean-field approximation
Mean-field variational approximation :F : {ρ : ρ(dθ) = ρ1 (dθ1 ) ⊗ ρ2 (dθ2 )} .
Alternate optimization
ρt+1= arg max ELBO(ρ1 × ρt2 )
1

ρ1
t+1
× ρ2 )
ρ2 = arg max ELBO(ρt+11
ρ2

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Mean-field approximation : explicit formulaρt+1= arg max ELBO(ρ1 ⊗ ρt2 )1
ρ1

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Mean-field approximation : explicit formulaρt+1= arg max ELBO(ρ1 ⊗ ρt2 )1
ρ1

Assume π = π1 × π2 .
max αnEθ1 ∼ρ1 Eθ2 ∼ρt2 [Rn (θ1 , θ2 )] + K(ρ1 , π1 ) + K(ρt2 , π2 ) .
ρ1Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Mean-field approximation : explicit formulaρt+1= arg max ELBO(ρ1 ⊗ ρt2 )1
ρ1

Assume π = π1 × π2 .
max αnEθ1 ∼ρ1 Eθ2 ∼ρt2 [Rn (θ1 , θ2 )] + K(ρ1 , π1 ) + K(ρt2 , π2 ) .
ρ1

Use Donsker and Varadhan’s variational formula.
ρt+1
1 (θ1 ) ∝ exp −nαEθ2 ∼ρt2 [Rn (θ1 , θ2 )|θ1 ] π1 (θ1 ).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Definition of VI
The ELBO
Strategies for ELBO maximization

Further reading

Recent survey on variational inference :D. M. Blei, A. Kucukelbir & J. D. McAuliffe (2017). Variational inference : A review forstatisticians. Journal of the American Statistical Association.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Examples1

Introduction : computational issues in Bayesian learning
Bayesian learning
Computational issues
Roadmap

2

Variational Approximations : Definition
Definition of VI
The ELBO
Strategies for ELBO maximization
Examples of Variational Approximations in Machine
Learning
Recommender systems and matrix completion
Deep learning

3

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Example 1 : recommendation via matrix completion

ClaireNial
BrendonAndrew
AdrianDamien
..
.

4
?
?
?
1
?
..
.
Pierre Alquier, RIKEN AIP

?
4
5
4
?
1
..
.

3
?
4
?
?
?
..
.

...
...
...
...
...
...
...

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

The Netflix challenge

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Matrix completion : notations
The parameter θ is a matrix M 0 ∈ Rm×p , with m, p ≥ 1.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Matrix completion : notations
The parameter θ is a matrix M 0 ∈ Rm×p , with m, p ≥ 1.
Under PM , the observations are random entries of this matrix
with possible noise :Yi = Mi0k ,jk + εkwhere the (ik , jk ) are i.i.d U({1, . . . , m} × {1, . . . , p}).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Matrix completion : notations
The parameter θ is a matrix M 0 ∈ Rm×p , with m, p ≥ 1.
Under PM , the observations are random entries of this matrix
with possible noise :Yi = Mi0k ,jk + εkwhere the (ik , jk ) are i.i.d U({1, . . . , m} × {1, . . . , p}). Assume
that the εk are i.i.d N (0, σ 2 ), σ 2 known.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Matrix completion : notations
The parameter θ is a matrix M 0 ∈ Rm×p , with m, p ≥ 1.
Under PM , the observations are random entries of this matrix
with possible noise :Yi = Mi0k ,jk + εkwhere the (ik , jk ) are i.i.d U({1, . . . , m} × {1, . . . , p}). Assume
that the εk are i.i.d N (0, σ 2 ), σ 2 known. We havemp

1 X X (Mi,j − Ni,j )2
kM − Nk2F
K(PM , PN ) ==.
mp i=1 j=1
2σ 2
2σ 2 mp

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Matrix completion : notations
The parameter θ is a matrix M 0 ∈ Rm×p , with m, p ≥ 1.
Under PM , the observations are random entries of this matrix
with possible noise :Yi = Mi0k ,jk + εkwhere the (ik , jk ) are i.i.d U({1, . . . , m} × {1, . . . , p}). Assume
that the εk are i.i.d N (0, σ 2 ), σ 2 known. We havemp

1 X X (Mi,j − Ni,j )2
kM − Nk2F
K(PM , PN ) ==.
mp i=1 j=1
2σ 2
2σ 2 mp
Usual assumption : M 0 is low-rank.
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Prior specification - main idea
Define :M = |{z}U |{z}
VT .
|{z}
p×m

Pierre Alquier, RIKEN AIP

p×k k×m

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Prior specification - main idea
Define :M = |{z}U |{z}
VT .
|{z}
p×m

p×k k×m

Let U·,` ∼ N (0, γI ) denote the `-th column of M, we have :M=

kXU·,` (V·,` )T⇒rank(M) ≤ k.

`=1

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Prior specification - adaptation
R. Salakhutdinov & A. Mnih (2008). Bayesian probabilistic matrix factorization using MCMC.
Proceedings of ICML.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Prior specification - adaptation
R. Salakhutdinov & A. Mnih (2008). Bayesian probabilistic matrix factorization using MCMC.
Proceedings of ICML.

M=

k
X

U·,` (V·,` )T`=1with k large - e.g. k = min(p, m).

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Prior specification - adaptation
R. Salakhutdinov & A. Mnih (2008). Bayesian probabilistic matrix factorization using MCMC.
Proceedings of ICML.

M=

k
X

U·,` (V·,` )T`=1with k large - e.g. k = min(p, m).
Definition of π :U·,` , V·,` ∼ N (0, γ` I ),γ` is itself random, such that most of the γ` ' 0
1∼ Gamma(a, b).
γ`
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Variational approximation
Y. J. Lim & Y. W. Teh (2007). Variational Bayesian approach to movie rating prediction.
Proceedings of KDD cup and workshop.

Mean-field approximation, F given by :ρ(dU, dV , dγ) =mOρUi (dUi,· )i=1Pierre Alquier, RIKEN AIP

p
O
j=1

ρVj (dVj,· )KOργk (γk ).

k=1

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Variational approximation
Y. J. Lim & Y. W. Teh (2007). Variational Bayesian approach to movie rating prediction.
Proceedings of KDD cup and workshop.

Mean-field approximation, F given by :ρ(dU, dV , dγ) =mOρUi (dUi,· )

i=1

p
O
j=1

ρVj (dVj,· )KOργk (γk ).

k=1

It can be shown that1
ρUi is N (mTi,· , Vi ),2
ρVj is N (nTj,· , Wj ),3
ργk is inverse-Γ(a + (m1 + m2 )/2, βk ),for some m × K matrix m whose rows are denoted by mi,· ,some p × K matrix n and some vector β = (β1 , . . . , βK ).
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

The VB algorithm
The parameters are updated iteratively through the formulae
1

moments of U :2αTmi,· :=

−1

Vi

:=

n

Vi

X

T
k

Yi ,j nj ,·k k

k:ik =ii m1 + m2
2α X hT−1
Wj + nj ,· nj ,· + a +diag(β)kkk
n k:i =i2k2moments of V :Tnj,· :=

−1

Wj

:=

2α
n

Wj

X

T
k

Yi ,j mi ,·k k

k:jk =ji 
2α X hm1 + m2
T−1
Vi + mi ,· mi ,· + a +diag(β)kkk
n k:j =j2
k3

moments of γ :

βk :=

1
2





m1 
m2 

 X
X
2
2
mi,k + (Vi )k,k +nj,k + (Vj )k,k  .
i=1

Pierre Alquier, RIKEN AIP

j=1

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Example 2 : deep learning

Source : Wikipedia.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Example 2 : deep learning
Neural network, recursive
definition :f0,θ (x) = x,(i)f`+1,θ (x) = ϕ!
sX̀(`) (j)
θi,j f`,θ (x) ,
j=1

!
sL
X
(L) (j)fθ (x) = ψ
θi,j f`,θ (x) .
j=1
Source : Wikipedia.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Example 2 : deep learning
Neural network, recursive
definition :f0,θ (x) = x,(i)f`+1,θ (x) = ϕ!
sX̀(`) (j)
θi,j f`,θ (x) ,
j=1

!
sL
X
(L) (j)fθ (x) = ψ
θi,j f`,θ (x) .
j=1
Source : Wikipedia.

Prior π : independent(`)

θi,j ∼ N (0, σ`2 ).
Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Example 2 : deep learning
The posterior is extremely complicated.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Example 2 : deep learning
The posterior is extremely complicated.
(`)

Using a mean-field variational approximation where all the θi,j(`)(`)
are independent N (mi,j , (σi,j )2 ) a posteriori, the authors of :K. Osawa, S. Swaroop, A. Jain, R. Eschenhagen, R. E. Turner, R. Yokota & M. E. Khan (2019).
Practical Deep Learning with Bayesian Principles. NeurIPS.

proposed a refined stochastic gradient algorithm and reachedstate-of-the-art performances on large datasets such as
CIFAR-10 and ImageNet.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1

Introduction : computational issues in Bayesian learning
Variational Approximations : Definition
Examples of Variational Approximations in Machine Learning

Recommender systems and matrix completion
Deep learning

Example 2 : deep learning

K. Osawa, S. Swaroop, A. Jain, R. Eschenhagen, R. E. Turner, R. Yokota & M. E. Khan (2019).
Practical Deep Learning with Bayesian Principles. NeurIPS.

Picture : Roman Bachmann.

Pierre Alquier, RIKEN AIP

Lectures on Variational Inference - 1