Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Generalization bounds for variational inference Pierre Alquier Elements de maths. pour l'IA May 15, 2019
Pierre Alquier
Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Bayesian inference Definition of variational approximations Outline of the talk Notations Assume that we observe X 1 , . . , X n i.i.d from P 60 in a model {P 0 , 0 ∈ Θ} dominated by Q : d Q = p 0 . Prior π on Θ.
Pierre Alquier
Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Bayesian interence Definition of variational approximations Outline of the talk Notations Assume that we observe X 1 , . . , X n i.i.d from P 60 in a model {P 0 , 0 ∈ Θ} dominated by Q : d Q = p 0 . Prior π on Θ. The likelihood Introduction : variational Bar Concentration of variational approximations of the posterior Online variational inference Bayesian interence Definition of variation Outline of the talk Notations Assume that we observe X 1 , . . , X n i.i.d from P 60 in a model {P 0 , 0 ∈ Θ} dominated by Q : d Q = p 0 . Prior π on Θ. The likelihood The posterior π n (dθ) å L n (θ)π (dθ). n Pierre Alquier Generalization bounds for variational inference Introduction : variation Concentration of variational approximations of the posterior Online variational inference Bayesian interenc Definition of variatio Outline of the tall Notations Assume that we observe X 1 , . . , X n i.i.d from P b , in a model {P 0 , 0 ∈ Θ} dominated by Q : d Q = p 0 . Prior π on Θ. The likelihood The posterior π n (dθ) å L n (θ) π (dθ). The tempered posterior - 0 < α < 1 πρα(dθ) å [Ln(θ)[åπ(dθ). Pierre Alquier Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Bayesian inference Definition of variational approximations Outline of the talk Computation of the posterior ● explicit form (conjugate models), Generalization bounds for variational inference Concentration of variational approximations of the posterior Online variational inference Bayesian inference Definition of variational approximations Outline of the talk Computation of the posterior ● explicit form (conjugate models), ● MCMC algorithms : Metropolis-Hastings, Gibbs sampler, Langevin Monte Carlo...
Pierre Alquier
nition of variational approximations line of the tal
● explicit form (conjugate models), ● MCMC algorithms : Metropolis-Hastings, Gibbs sampler, Langevin Monte Carlo... But… ● when the dimension is large, the convergence of MCMC can be extremely slow,
● explicit form (conjugate models), ● MCMC algorithms : Metropolis-Hastings, Gibbs sampler, Langevin Monte Carlo... But… ● when the dimension is large, the convergence of MCMC can be extremely slow, ● when the model is complex or when the sample size is large, each evaluation of π n,α(θ) can be expensive.
Pierre Alquier
Computation of the posterior
● explicit form (conjugate models), ● MCMC algorithms : Metropolis-Hastings, Gibbs sampler, Langevin Monte Carlo... But… ● when the dimension is large, the convergence of MCMC can be extremely slow, ✿ when the model is complex or when the sample size is large, each evaluation of π n,α(θ) can be expensive. For these reasons, in the past 20 years, many methods targeting an approximation of π n,α became popular : ABC, EP algorithm, variational inference, approximate MCMO Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Bayesian inference Definition of variational approximations Outline of the talk Variational approximations : definitions Idea of VB : chose a family F of probability distributions on Θ and approximate π n,α by a distribution in F : Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Bayesian interence Definition of variational approximations Outline of the talk Variational approximations : definitions Idea of VB : chose a family F of probability distributions on Θ and approximate π n,α by a distribution in F :
Pierre Alquier
Concentration of variational approximations of the poster Online variational inter Preimition of variational approximations Variational approximations : definitions Idea of VB : chose a family F of probability distributions on Θ and approximate π n,α by a distribution in F : Examples : ● parametric approximation F = {N(μ,Σ) : μ ∈ Rd,Σ ∈ St} .
Pierre Alquier
Concentration of variational approximations of the poster Online variational infer tinkion of variatio Variational approximations : definitions Idea of VB : chose a family F of probability distributions on Θ and approximate π n,α by a distribution in F : Examples : ● parametric approximation ● mean-field approximation, Θ = Θ 1 × Θ 2 and F : {ρ : ρ(dθ) = ρ1(dθ1) × ρ2(dθ2)} .
Pierre Alquier
Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Bayesian inference Definition of variational approximations Outline of the talk
Note that :
−ELBO( ρ ) Introduction : variational Concentration of variational approximations of the poster Online variational infer Bayesian interence Definition of variational approximations Outline of the talk Empirical lower bound (ELBO) Note that :
-ELBO( ρ ) So we have the equivalent definition :
Pierre Alquier
Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Bayesian inference Definition of variational approximations Outline of the talk Outline of the talk After this introduction : Pierre Alquier Generalization bounds for variational inference Introduction : variational Save Concentration of variational approximations of the posterior Online variational inference Bayesian inference Definition of variational approximations Outline of the talk Outline of the talk After this introduction : Section 2 will address the following question : What are the conditions ensuring that π n,α leads to good estimators ?
Pierre Alquier
Introduction : variation Concentration of variational approximations of the posterior Online variational inference Bayesian interence nition of variational approximations Outline of the talk Outline of the talk After this introduction : Section 2 will address the following question : What are the conditions ensuring that π n,α leads to good estimators ? We will show general conditions, and many examples.
Pierre Alquier
Concentration of variational approximations of the posterior Online variational inference ition of variational approximations Outline of the talk Outline of the talk After this introduction : Section 2 will address the following question : What are the conditions ensuring that π n,α leads to good estimators ? We will show general conditions, and many examples. Section 3 will address the following question : Are there efficient algorithms to (provably) compute π n.g ?
Pierre Alquier
Generalization bounds for variational inference
Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Definition of variational approximations Outline of the talk
After this introduction : Section 2 will address the following question : What are the conditions ensuring that  π n,α leads to good estimators ? We will show general conditions, and many examples. Section 3 will address the following question : Are there efficient algorithms to (provably) compute π n.g ? We will see that fast algorithms from sequential optimization can be used in some cases. This also allows to do variational inference on a data stream that cannot be stored.
Pierre Alquier
Generalization bounds for variational inference
Outline of the talk
Introduction : variational Bayesian inference ● Bayesian inference ● Definition of variational approximations ● Outline of the talk 2) Concentration of variational approximations of the posterior ● Theoretical results ● Applications 3) Online variational inference ● Sequential estimation problem ● Online variational inference ● Simulations Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Theoretical results Applications Tools for the consistency of VB The α-Rényi divergence for α ∈ (0,1) Dα( Concentration of variational approximations of the posteri Online variational infer Theoretical resu pplications Tools for the consistency of VB The α-Rényi divergence for α ∈ (0, 1) All the properties derived in : Van Erven & P. Harremos. Rényi divergence and Kullback-Leibler divergence. IEEE ansactions on Information Theory, 2014. Among others, for 1/2 ≤ α , link with Hellinger and Kullback : H²(P, R) ≤ D (P, R) —→ K(P, R).
Pierre Alquier
What do we know about π n,α B(r) = {θ ∈ Θ : K(Poo, Po) ≤ r} . Theorem, variant of (Bhattacharya, Pati & Yang) For any sequence ( r n ) such that — log π[B(rn)] ≤ nrn we have narya, D. Pati & Y. Yang. Bayesian fractional posteriors. The Annals of S 2019.
Pierre Alquier
Generalization bounds for variational inference Introduction : variation Concentration of variational approximations of the poster Online variational info Theoretical re Extension of previous result to VB Theorem (A. & Ridgway) If there is ρ n ∈ F and (r n ) such that K(Poo, Po)pn(d0) < rn, and then, for any α ∈ (0, 1), Introduction : variational Concentration of variational approximations of the posterior Online variational inference Theoretical results Applications Extension of previous result to VB Theorem (A. & Ridgway) If there is ρ n ∈ F and ( r n ) such that K(Poo, Po)pn(d0) < rn, and (Pn, π) ≤ nrn, then, for any α ∈ (0, 1), E P. Alquier & J. Ridgway. Concentration of tempered posteriors and of their variational approximations. The Annals of Statistics, to apper Generalization bounds for variational inference Concentration of variational approximations of the posterior Online variational infere Thooretical res Application Misspecified case Assume now that X 1 , . . . , X n i.i.d ~ Q f {P o , θ ∈ Θ}. Put : K(Q,Pg). Theorem (A. and Ridgway) Assume that there is ρ n ∈ F such that ρn(dθ) ≤ rn and K(ρn,π) ≤ then, for any α ∈ (0,1), E Pierre Alquier Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Applications First example : nonparametric regression Nonparametric regression ● Y; = f(X;) + ξi, Pierre Alquier Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Theoretical results Applications First example : nonparametric regression Nonparametric regression ● Y; = f(X;) + ξ;, Generalization bounds for variational inference Online variational inference First example : nonparametric regression Nonparametric regression ● Y; = f(X;) + ξi, ● ξ; ~ N (0,σ²), ● f is s-smooth with s unknown, Generalization bounds for variational inference Online variational inferenc First example : nonparametric regression Nonparametric regression ●  Y; = f(X;) + ξ;, ● ξ; ~ N (0,σ²), ● f is s-smooth with s unknown, ● prior : f(:) = ∑ j=1 βjφj(:), random K and βj's, (φj) basis…
Nonparametric regression ● Y; = f(X;) + ξ;, ● ξ; ~ N (0,σ²), ● f is s-smooth with s unknown, ● prior : f(:) = ∑ j=1 βjφj(:), random K and βj's, (ϕj) basis… Under suitable assumptions, r n ~
Pierre Alquier
Generalization bounds for variational inference
Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Theoretical results Applications More examples covered in the paper ● logistic regression, Pierre Alquier Generalization bounds for variational inference Concentration of variational approximation
● logistic regression, matrix completion : we prove that the approx. in Y. J. Lim & Y. W. Teh. Variational Bayesian approach to movie rating prediction. Proceedings of KDD cup and workshop, 2007. leads to minimax-optimal estimation.
Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Extensions Theoretical results Applications ● case α = 1, i.e approximation of the "usual" posterior : F. Zhang & C. Gao. Convergence Rates of Variational Posterior Distributions. Preprint arXiv, 2017. Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Extensions case α = 1, i.e approximation of the “usual” posterior : F. Zhang & C. Gao. Convergence Rates of Variational Posterior Distributions. Preprint arXiv, 2017. approximation based on another distance, for example : (2) arg min W(ρ, π n,α) (Wasserstein distance), J. Huggins, T. Campbell, M. Kasprzak & T. Broderick. Practical bounds on the error of Bayesian posterior approximations : a nonasymptotic approach. Preprint arXiv, 2018.
Pierre Alquier

Introduction : variational Bayesian inference ● Bayesian inference ● Definition of variational approximations ● Outline of the talk 2) Concentration of variational approximations of the posterior ● Theoretical results ● Applications 3) Online variational inference ● Sequential estimation problem ● Online variational inference ● Simulations Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations Sequential estimation problem Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations Sequential estimation problem ● initialize θι, Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations Sequential estimation problem ● initialize θι, ● ×1 revealed, Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations Sequential estimation problem ● initialize θι, ● ×1 revealed, ● incur loss — log Po1 (×1) Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations Sequential estimation problem ● initialize θι, ● ×1 revealed, ● incur loss — log Po1 (×1) Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational interence Sequential estimation problem Online variational inference Simulations Sequential estimation problem O update θ₁ → θ₂, ● initialize θι, ● ×1 revealed, ● incur loss — log Poi (×1) Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inferen Sequential estimation problem Online variational inference Simulations Sequential estimation problem ● initialize θι, ● ×1 revealed, ● incur loss — log Poi (×1) O update θ₁ → θ₂, ● x2 revealed, ●  incur loss — log po, (×2) Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational interes Sequential estimation problem Online variational inference Simulations Sequential estimation problem ● initialize θι, ● ×1 revealed, ● incur loss — log Poi (×1) O update θ₁ → θ₂, ● x2 revealed, ● incur loss — log po, (×2) O update θ₂ → θ₃, Generalization bounds for variational inference Concentration of variational approximations of the posteri- Online variational interes Sequential estimation problem Online variational inference Sequential estimation problem ● initialize θι, ● ×1 revealed, ● incur loss — log Poi (×1) O update θ₁ → θ₂, ●  ×2  revealed, ● incur loss — log po, (×2) O update θ₂ → θ₃, ● x3 revealed, Concentration of variational approximat in ine variational inte Online variational inferen Sequential estimation problem ● initialize θι, ● ×1 revealed, ● incur loss — log Poi (×1) O update θ₁ → θ₂, ● ×2 revealed, ● incur loss — log po, (×2) O update θ2 → θ3, ● x3 revealed, ● incur loss — log po, (x3) Generalization bounds for variational inference Concentration of variational approxi ine variational inferen Sequential estimation problem ● initialize θι, ● ×1 revealed, ● incur loss — log Poi (×1) Objective : O update θ₁ → θ₂, ● ×2 revealed, ● incur loss — log po, (×2) O update θ2 → θ3, ● x3 revealed, ● incur loss — log po, (x3) Sequential estimation problem ● initialize θι, ● ×1 revealed, ● incur loss — log Po1 (×1) O update θ₁ → θ₂, ● ×2 revealed, Q incur loss — log po, (×2) O update θ₂ → θ₃, ● x3 revealed, ● incur loss — log po, (x3) Objective : make sure that we learn to predict well as fast as possible. Sequential estimation problem ● initialize θι, ● ×1 revealed, ● incur loss — log Po1 (×1) Q update θ₁ → θ₂, ● x2 revealed, ● incur loss — log po, (x2) O update θ₂ → θ₃, ● x3 revealed, ● incur loss — log po, (x3) Objective : make sure that we learn to predict well as fast as small as possible for any T, without stochastic assumptions on the data.
Pierre Alquier
Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Reference Sequential estimation problem Online variational inference Simulations Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Reference Sequential estimation problem Online variational inference Simulations The regret : R( T xt )] t=1 Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations EWA strategy / multipicative update... Pierre Alquier Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations EWA strategy / multipicative update... ● learning rate α > 0. Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations EWA strategy / multipicative update... ● learning rate α > 0. ● initialize ρ 1 = π (the prior). Generalization bounds for variational inference Introduction : variation Online variational interes Unline vanational inferen EWA strategy / multipicative update... ● learning rate α > 0. ● initialize ρ 1 = π (the prior). Algorithm 2 Exponentially Weighted Aggregation 1: for t = 1,2,... do 2: θ t = E θ ~ ρ [θ], x, revealed, update pt+1(dθ) 3: 4:  end for
Pierre Alquier
Generalization bounds for variational inference
On ine variational intera EWA strategy / multipicative update... ● learning rate α > 0. ● initialize ρ 1 = π (the prior). Algorithm 2 Exponentially Weighted Aggregation 1: for t = 1,2,... do 2: θ t = Eo~pt [θ], 3: x, revealed, update pt+1(dθ) 4:  end for Note that Pt = π n,α the tempered posterior, so problem : how can we compute θ t ?
Pierre Alquier
Generalization bounds for variational inference
Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations A regret bound for EWA From now, θ → [— log ρθ(xt)] is convex + bounded : | · | ≤ C. Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations A regret bound for EWA From now, θ → [— log ρθ(xt)] is convex + bounded : | · | ≤ C. Theorem A regret bound for EWA From now, θ ∪ [ — log ρθ(xt)] is convex + bounded : | · | ≤ I heorem Under similar assumptions than in the batch case, that is, the prior gives enough mass to relevant θ, and α ～ 1/√T, Pierre Alquier Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations T T log ρθ(xt)] + cst.√ T t=1 t=1
Pierre Alquier
Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations T t=1 1 T t=1 T

Pierre Alquier
Concentration of variational Online variational info Online variational in

Pierre Alquier
Generalization bounds for variational inference
Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations Variational approximations of EWA B.-E. Chérief-Abdellatif, P. Alquier & M. E. Khan. A Generalization Bound for Online Variational Inference. Preprint arXiv, 2019. Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations
B.-E. Chérief-Abdellatif, P. Alquier & M. E. Khan. A Generalization Bound for Online Variational Inference. Preprint arXiv, 2019. Generalization bounds for variational inference Variational approximations of EWA odellatif, P. Alquier & M. E. Khan. A Generalization Bound for Online Variational trence. Preprint arXiv, 2019. Parametric variational approximation : F = {q,, μ ∈ M}. Objective : propose a way to update µt → µt+1 so that qµt leads to similar performances as pt in EWA...
Pierre Alquier
Generalization bounds for variational inference Introduction : variational Concentration of variational approximations of the posterior Online variational interent Online variational inferen Simulations SVA and SVB strategies Algorithm 3 SVA (Sequential Variational Approximation) 1: for t = 1,2,... do 2:
3:
4:  end for SVA and SVB strategies Algorithm 3 SVA (Sequential Variational Approximation) 1: for t 1.2 do 2: 3:
4:  end for

Pierre Alquier
Generalization bounds for variational inference Introduction : variational Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations NGVI strategy NGVI (Natural Gradient Variational Inference) : fix some β > 0, ||t+1

Pierre Alquier
Introduction : variational Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations NGVI strategy NGVI (Natural Gradient Variational Inference) : fix some β > 0, flt+1
M. E. Khan & W. Lin. Conjugate-computation variational inference : Converting variational inference in non-conjugate models to inferences in conjugate models. AISTAT, 2017.
Pierre Alquier
Generalization bounds for variational inference Concentration of variational approximal Online variational inferen Simulations An example : SVB with Gaussian approximations As an example, assume that θ ∈ R d , the prior is π = N(0, s 2 /) and that we use the variational approximation

Pierre Alquier
An example : SVB with Gaussian approximations
where ☺ means “componentwise multiplication” and h(x) = √1 + x² — x is also applied componentwise.
An example : SVB with Gaussian approximations
As an example, assume that θ ∈ R d , the prior is
where ☺ means “componentwise multiplication” and h(x) = √1 + x² — x is also applied componentwise. We also have explicit formulas for SVA and NGVI (see the paper). Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations A regret bound for SVA Theorem (Chérief-Abdellatif, A. & Khan) Assume that μ → E θ~g μ [ − log ρ θ ( x ε )] is L-Lipschitz and convex. Generalization bounds for variational inference Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational interence Sequential estimation problem Online variational inference Simulations A regret bound for SVA Theorem (Chérief-Abdellatif, A. & Khan) Assume that μ → E o~q μ [ − log ρ θ (x t )] is L-Lipschitz and convex. (this is for example the case as soon as the log-likelihood is concave in θ and L-Lipschitz, and μ is a location-scale parameter).
Pierre Alquier
Generalization bounds for variational inference Concentration of variational A regret bound for SVA Theorem (Chérief-Abdellatif, A. & Khan) Assume that μ → Eo~q, [— log ρ θ ( x t )] is L-Lipschitz and convex.  Assume that μ → K(pμ, π) is γ-strongly convex. Then SVA satisfies :
Pierre Alquier
Concentration of varia A regret bound for SVA Theorem (Chérief-Abdellatif, A. & Khan) Assume that μ → Eo~q,[— log ρ θ (x t )] is L-Lipschitz and convex.  Assume that μ → K(pμ, π) is γ-strongly convex. Then SVA satisfies : For SVB : some results in the Gaussian case. Generalization bounds for variational inference A regret bound for SVA Theorem (Chérief-Abdellatif, A. & Khan) Assume that μ → Eo~q,[— log ρ θ (x t )] is L-Lipschitz and convex.  Assume that μ → K(pμ, π) is γ-strongly convex. Then SVA satisfies : For SVB : some results in the Gaussian case. For NGVI : we were not able to derive regret bounds until now.
Pierre Alquier
Generalization bounds for variational inference Introduction : varia Concentration of variational
Figure – Average cumulative losses on different datasets for classification and regression tasks with OGA (yellow), OGA-EL (red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier
Generalization bounds for variational inference
Introduction : V Concentration of variational
Figure – Average cumulative losses on different datasets for classification and regression tasks with OGA (yellow), OGA-EL (red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier
Generalization bounds for variational inference

Figure – Average cumulative losses on different datasets for classification and regression tasks with OGA (yellow), OGA-EL (red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier
Generalization bounds for variational inference

Figure – Average cumulative losses on different datasets for classification and regression tasks with OGA (yellow), OGA-EL (red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier
Generalization bounds for variational inference
Figure – Average cumulative losses on different datasets for classification and regression tasks with OGA (yellow), OGA-EL (red), SVA (blue), SVB (purple) and NGVI (green).
Pierre Alquier
Generalization bounds for variational inference
Introduction : variational Concentration of variational approximations of the posterior Online variational interence Sequential estimation problem Online variational inference Simulations Conclusions ● Using online-to-batch conversion, we now have algorithms for variational inference with provable statistical properties after a finite number of steps.
Pierre Alquier
Sequential estimation problem line variational inference Online variational interence Conclusions ● Using online-to-batch conversion, we now have algorithms for variational inference with provable statistical properties after a finite number of steps. ● SVA, SVB competitive with OGA (online gradient algorithm, ''non-Bayesian'').
Pierre Alquier
Conclusions
● Using online-to-batch conversion, we now have algorithms for variational inference with provable statistical properties after a finite number of steps. ● SVA, SVB competitive with OGA (online gradient algorithm, ''non-Bayesian''). ● NGVI is the best method on all datasets. Its theoretical analysis is thus an important open problem. Cannot be done with our current techniques (using natural parameters in exponential models lead to non-convex objectives).
Pierre Alquier
Generalization bounds for variational inference
Introduction : variational Bayesian inference Concentration of variational approximations of the posterior Online variational inference Sequential estimation problem Online variational inference Simulations Thank you! Pierre Alquier Generalization bounds for variational inference