MMD-Bayes Robust Bayesian Estimation via Maximum Mean Discrepancy Pierre AlquierつIK≡H Advanced Intelligence Project AIP PI Seminar – January 10, 2020 Introduction : so Approximate Bayesian Inference team (ABI), lead by Emtiyaz Khan Please visit the team website https ://emtiyaz.github.io/
Pierre Alquier, RIKEN AIP
MMD-Bayes
Approximate Bayesian Inference team (ABI), lead by Emtiyaz Khan Please visit the team website https ://emtiyaz.github.io/ One of the recurring idea in the team's work : π(θ|X) å π(X|θ) π(θ) → π(θ|X) å exp (−αL(X elihood pr
Pierre Alquier, RIKEN AIP
MMD-Bayes
Introduction : some problem Approximate Bayesian Inference team (ABI), lead by Emtiyaz Khan Please visit the team website https ://emtiyaz.github.io/ One of the recurring idea in the team's work : π(θ|Χ) å π(Χ|θ) π(θ) → π(θ|Χ) å exp (- dihood pri er, N. Chopin and J. Ridgway (2016). On the Properties of Variational Approximations of iors. Journal of Machine Learning Research. S. Swaroop, A. Jain, R. Eschenhagen, R. E. Turner, R. Yokota, M. E. Khan (20 Practical Deep Learning with Bayesian Principles. NeurIPS.
Pierre Alquier, RIKEN AIP
MMD-Bayes Today : a loss function leading to robust estimation. Joint works with : Badr-Eddine Chérief-Abdellatif. hérief-Abdellatif, P. Alquier (2019). MMD-Bayes : Robust Bayesian Estimation via mum Mean Discrepancy. AABI 2019. -E. Chérief-Abdellatif, P. Alquier (2019). Finite Sample Properties of Parametric MMD Estimation : Robustness to Misspecification and Dependence. Preprint arxiv :1912.05737. Pierre Alquier, RIKEN AIP MMD-Bayes troduction : some problems with the likelihood � Kernels and MMD distance ● Kernels and RKHS ● Maximum Mean Discrepancy (MMD) ● MMD-based estimation
MMD-Bayes
● Bayesian estimator based on MMD ● Consistency and rate of convergence ● Simulation study
Pierre Alquier, RIKEN AIP
MMD-Bayes
The Maximum Likelihood Estimator (MLE) Let X 1 , . . . , X n be i.i.d in X from a probability distribution P 0 .
Pierre Alquier, RIKEN AIP

Let X 1 , . . . , X n be i.i.d in X from a probability distribution P 0 . Statistical inference : ● propose a model ( P g , θ ∈ Θ), assume P g = P g . ● compute θ n = θ n (X 1 , . . . , X n ).
Pierre Alquier, RIKEN AIP
MMD-Bayes

Let X 1 , . . . , X n be i.i.d in X from a probability distribution P 0 . Statistical inference : ● propose a model ( P g , θ ∈ Θ), assume P g = P g . ● compute Ô n = Ô n (X 1 , . . . ,X n ). Letting pg denote the density of Pg, then

Pierre Alquier, RIKEN AIP
MMD-Bayes

Let X 1 , . . . , X n be i.i.d in X from a probability distribution P 0 . Statistical inference : ● propose a model ( P g , θ ∈ Θ), assume P g = P g . ● compute θ n = θ n (X 1 , . . . ,X n ). Letting p o denote the density of P o , then

Pierre Alquier, RIKEN AIP
MMD-Bayes Kernels and MMD distance MMD-Bayes
Example :
Example :
exp ( L(θ) = (2√π)" | |;=
Pierre Alquier, RIKEN AIP
MMD-Bayes MMD-Bayes MLE fails in the presence of outliers What is an outlier ? MMD-Bayes
What is an outlier ? Huber proposed the contamination model : with probability ε, X; is not drawn from P θ o but from Q that can be anything : Po = (1 − ε)Poo + εQ.
Pierre Alquier, RIKEN AIP
MMD-Bayes

What is an outlier ? Huber proposed the contamination model : with probability ε, Xi is not drawn from Po but from Q that can be anything : Po = (1 − ε)Poo + εQ. Example : P g = U nif [0, θ], then
Pierre Alquier, RIKEN AIP
MMD-Bayes

What is an outlier ? Huber proposed the contamination model : with probability ε, Xi is not drawn from Po but from Q that can be anything :

Pierre Alquier, RIKEN AIP
MMD-Bayes
MLE fails in the presence of outliers
What is an outlier ? Huber proposed the contamination model : with probability ε, Xi is not drawn from Poo but from Q that can be anything :
In the case of the following contamination, the MLE is extremely far from the truth :

Pierre Alquier, RIKEN AIP
MMD-Bayes
Requirements for a “good” estimator A universal estimator f n must be such that, for some distance d on probability distributions,
Pierre Alquier, RIKEN AIP
MMD-Bayes

A universal estimator f n must be such that, for some distance d on probability distributions, ● when the model is well specified, that is, Po = Pon,

Pierre Alquier, RIKEN AIP
MMD-Bayes

A universal estimator θ n must be such that, for some distance d on probability distributions, ● when the model is well specified, that is, Po = Pog,
● in the misspecified case P o = (1 − ε ) P o + εQ, for any Q,

Pierre Alquier, RIKEN AIP
MMD-Bayes

A universal estimator θ n must be such that, for some distance d on probability distributions, ● when the model is well specified, that is, Po = Poo,
The MLE does not satisfy these requirements.
Pierre Alquier, RIKEN AIP
MMD-Bayes
s and MMD disance Some examples Yatracos’ skeleton estimate θ , :
dim(Θ) n where d τν ( P, Q) = sup | P ( E ) = Q ( E ) | . E Y. G. (1985). Rates of convergence of minimum distance estimators and Kolmogorov's entropy. Annals of Statistics.
Pierre Alquier, RIKEN AIP
MMD-Bayes
and MMD distance Some examples Yatracos’ skeleton estimate θ γ :
dim(甲) n where d TV ( P, Q) = sup | P ( E ) — Q ( E ) | . Y. G. (1985). Rates of convergence of minimum distance estimators and Kolmogorov's entropy. Annals of Statistics. More recent work with the Hellinger distance : Baraud, Y., Birgé, L., & Sart, M. (2017). A new method for estimation and model selection : ρ -estimation. Inventiones mathematicae. Pierre Alquier, RIKEN AIP MMD-Bayes Introduction : some problems with the likelihood Kernels and MMD distance MMD-Bayes Pierre Alquier, RIKEN AIP MMD-Bayes But… Problem with the aforementioned estimators : they cannot be computed in practice.
Pierre Alquier, RIKEN AIP
MMD-Bayes
But… Problem with the aforementioned estimators : they cannot be computed in practice. Additional requirement : an estimator must be computable!!!
Pierre Alquier, RIKEN AIP
MMD-Bayes
troduction : some problems with the likelihood � Kernels and MMD distance ● Kernels and RKHS ● Maximum Mean Discrepancy (MMD) ● MMD-based estimation
/MD-Bayes
● Bayesian estimator based on MMD ● Consistency and rate of convergence ● Simulation study Kernels and RKIN A PAN  DJ IMD-based estimation Reminder : kernels Let H be a Hilbert space and any continuous function Φ : X → H. The function K(x, y) = ⟨Φ(x), Φ(y)⟩ H is called a kernel.
Pierre Alquier, RIKEN AIP
MMD-Bayes
Reminder : kernels
Let H be a Hilbert space and any continuous function Φ : X → H. The function K(x, y) = ⟨Φ(x), Φ(y)⟩ H is called a kernel. Conversely : Mercer's theorem Let K(x, y) be a continuous function such that for any ( x 1 , . . . , x n ) ∈ X n and ( c 1 , . . . , c n )̸= (0 , . . . , 0 ) ∈ R n , c;K ( x;, x;) > 0, then there is H and Φ such that K(x,y) = (Φ(x),Φ(y)\ジェ
Pierre Alquier, RIKEN A
MMD-Bayes
MMD-Bayes Kernels and RKHS Maximum Mean Discrepancy (MMD) MMD-based estimation Reminder : MMD Assume that the kernel is bounded : 0 ≤ K(x,y) ≤ 1. MMD-Bayes Kernels and RKHN Maximum Mean Discrepancy (MMD) MMD-based estimation Reminder : MMD Assume that the kernel is bounded : 0 ≤ K(x,y) ≤ 1. Consider, for any probability distribution P on X, μκ(P) = Ex~P [Φ(x)] .
Pierre Alquier, RIKEN AIP
MMD-Bayes
Maximum Mean Discrepa MD-based estimation Reminder : MMD Assume that the kernel is bounded : 0 ≤ K(x,y) ≤ 1. Consider, for any probability distribution P on X, μκ(P) = Ex~P [Φ(x)] . The kernel K is said to be characteristic if μκ(P) = μκ(Q) ⇒ P = Q.
Pierre Alquier, RIKEN AIP
MMD-Bayes
dmym Mean Discre d-based estimation Reminder : MMD Assume that the kernel is bounded : 0 ≤ K(x,y) ≤ 1. Consider, for any probability distribution P on X, μκ(P) = Ex~P [Φ(x)] . The kernel K is said to be characteristic if µK(P) = µK(Q) ⇒ P = Q. Theorem |×− Y !! ) are char. kernels. K(x, y) = exp(
Pierre Alquier, RIKEN AIP
MMD-Bayes
dmum Mean Disc )-based estimatio Reminder : MMD Assume that the kernel is bounded : 0 ≤ K(x,y) ≤ 1. Consider, for any probability distribution P on X, μκ(P) = Ex~P [Φ(x)] . The kernel K is said to be characteristic if µK(P) = µK(Q) ⇒ P = Q. Theorem _ _ ll×-Yll) are char. kernels. K(x,y) ) and Definition : the MMD distance DK(P, Q) = ‖μκ(P) − μκ(Q)‖ H Pierre Alquier, RIKEN AIP MMD-Bayes MMD-based estimator Reminder of the context : X 1 , . . . , X n be i.i.d in X from a probability distribution P 0 , model (P 0 , θ ∈ Θ). Definition - MMD based estimator
Pierre Alquier, RIKEN AIP
MMD-Bayes
b hased estimation MMD-based estimator Reminder of the context : X 1 , . . . , X n be i.i.d in X from a probability distribution Po, model (Po, θ ∈ Θ). Definition - MMD based estimator Even though this idea was sometimes used before, the first theoretical study : Briol, F. X., Barp, A., Duncan, A. B., & Girolami, M. (2019). Statistical Inference for Generative Models with Maximum Mean Discrepancy. Preprint arXiv :1906.05944.
Pierre Alquier, RIKEN AIP
MMD-Bayes
Maximum Mean De MMD-based estimation Universal estimation with MMD Theorem - Chérief-Abdellatif + PA For any Po, when X 1 , . . . , Xn are i.i.d from Po, ed estima Universal estimation with MMD Theorem - Chérief-Abdellatif + PA For any Po, when X 1 , . . . , Xn are i.i.d from Po, Corollary - Huber contamination model When X 1 , . . . , X n are i.i.d from (1 − ε)P 6 , + εQ,
Pierre Alquier, RIKEN AIP
MMD-Bayes
Introduction c problems with the lik smels and MMD disons Kernels and RKHS Maximum Mean Discrep MMD-based estimation How to compute θ " We actually have

Pierre Alquier, RIKEN AIP
MMD-Bayes
hased estimati How to compute
that can be approximated by sampling from Po.
Pierre Alquier, RIKEN AIP
MMD-Bayes Maximum Mean Discrepancy (MMD) MMD-based estimation Example : Gaussian mean estimation Example :  the model is given by ρ θ = N (θ,σ 2 Ι) for θ ∈ R d .
Pierre Alquier, RIKEN AIP
MMD-Bayes
Example : Gaussian mean estimation Example :  the model is given by po = N(0,021) for θ ∈ Rd. Using a Gaussian kernel k(x,y) = exp(-||x - y 2 ||/ γ 2 ), from the previous theorem and from the equality
we obtain
Pierre Alquier, RIKEN AIP
MMD-Bayes
Example : Gaussian mean estimation Example :  the model is given by ρθ = N (θ,σ 2 I) for θ ∈ R d . Using a Gaussian kernel k(x,y) = exp(-||x - y 2 ||/ γ 2 ), from the previous theorem and from the equality
we obtain

Pierre Alquier, RIKEN AIP
MMD-Bayes
Example : Gaussian mean estimation Example :  the model is given by ρ θ = N (θ,σ 2 Ι) for θ ∈ R d . Using a Gaussian kernel k(x, y) = exp(-||x - y 2 ||/ γ 2 ), from the previous theorem and from the equality
we obtain
Pierre Alquier, RIKEN AIP
MMD-Bayes
Example : Gaussian mean estimation, simulations Model : N (θ, 1), and X 1 , . . . , X n i.i.d N (θ o, 1), n = 100 and we repeat the experiment 200 times.

Pierre Alquier, RIKEN AIP
MMD-Bayes
Example : Gaussian mean estimation, simulations Model : N (θ, 1), and X 1 , . . . , X n i.i.d N (θ o , 1), n = 100 and we repeat the experiment 200 times.
Now, ε = 2% of the observations drawn from a Cauchy. 0.2349 0.0953 mean absolute error
Pierre Alquier, RIKEN AIP
MMD-Bayes
Example : Gaussian mean estimation, simulations Model : N ( θ, 1), and X 1 , . . . , X n i.i.d N ( θ o, 1), n = 100 and we repeat the experiment 200 times.
Now, ε = 2% of the observations drawn from a Cauchy. 0.2349 0.0953 mean absolute error Now, ε = 1% are replaced by 1,000. mean absolute error 10.018 0.0903
MMD-Bayes
Introduction : some problems with the likelihood Kernels and MMD distance MMD-Bayes Kernels and RKHS Maximum Mean Discrepancy (MMD) MMD-based estimation
Dziugaite, G. K., Roy, D. M., & Ghahramani, Z. (2015). Training generative neural networks via maximum mean discrepancy optimization. UAI 2015. Li, Y., Swersky, K. & Zemel, R. (2015). Generative moment matching networks. ICML 2015. From the first reference :
Pierre Alquier, RIKEN AIP
MMD-Bayes
MMD-Bayes Bayesian estimator based on MMD Consistency and rate of convergence Simulation study MMD-Bayes Given a prior π(θ) we propose the following pseudo-posterior : πα(θ|Χ1, . . . , Χn) α e−αDix(Ρα,Ρα),
Pierre Alquier, RIKEN AIP
MMD-Bayes
ncy and rate of convergence MMD-Bayes Given a prior π(θ) we propose the following pseudo-posterior : πα(θ|X1, . . . ,Xn) α e-αD2(Po,Pn)π(θ). We also define a variational approximation for this. Given a set F of probability distributions, πα(θ) = arg min K[q, πα( · X 1 , . . . , Xn)].
Pierre Alquier, RIKEN AIP
MMD-Bayes
Kernels and MMD distance Bayesian estimator based on MMD Consistency and rate of convergence Simulation study Bayesian MMD-based estimation Theorem - Chérief-Abdellatif + PA Let B = {θ ∈ Θ/Dκ (Poo, Po) ≤ 1/√n}. Assume (f satisfies the prior mass condition : π(B) > e-α/√η E π e (dθ)
Pierre Alquier, RIKEN AIP
MMD-Bayes
Consistency and rate of conversence Bayesian MMD-based estimation le A similar result holds for the variational approximation.
Pierre Alquier, RIKEN AIP
MMD-Bayes
Kernels and MMD distance MMD-Bayes Bayesian estimator based on MMD Consistency and rate of convergence Simulation study
Introduction : some problems with the likelihood Kernels and MMD distance MMD-Bayes Bayesian estimator based on MMD Consistency and rate of convergence Simulation study Thank you! Pierre Alquier, RIKEN AIP MMD-Bayes