Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

The nice properties of MMD for statisticalestimation
Pierre AlquierMAD-stat seminar, Toulouse School of Economics. – Nov. 10, 2022

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)The Maximum Likelihood Estimator (MLE)
Let X1 , . . . , Xn be i.i.d in X from a probability distribution P0 .
Statistical inference :propose a model (Pθ , θ ∈ Θ), assume P0 = Pθ0 .
compute θ̂n = θ̂n (X1 , . . . , Xn ).
Letting pθ denote the density of Pθ , thenθ̂nMLE = arg max Ln (θ), where Ln (θ) =
θ∈Θ

n
Y

pθ (Xi ).

i=1

Example : P(m,σ) = N (m, σ 2 ) then
n
n
1X
1X
2
Xi and σ̂ =(Xi − m̂)2 .
m̂ =
n i=1
n i=1
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

MLE not unique / not consistent
Example :pθ (x) =exp(−|x − θ|)p,
2 π|x − θ|

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

MLE not unique / not consistent
Example :pθ (x) =exp(−|x − θ|)p,
2 π|x − θ|P
exp (− ni=1 |Xi − θ|).
Ln (θ) = √Q p(2 π)n ni=1 |Xi − θ|

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

MLE fails in the presence of outliers
What is an outlier ?
Huber proposed the contamination model : with probability ε,
Xi is not drawn from Pθ0 but from Q that can be anything :P0 = (1 − ε)Pθ0 + εQ.
Example : Pθ = Unif [0, θ], thenn

1 Y
Ln (θ) = n1{0≤Xi ≤θ} ⇒ θ̂ = max Xi .
1≤i≤n
θ i=1
In the case of the following contamination, the MLE isextremely far from the truth :P0 = (1 − ε).Unif [0, 1] + ε.N (1010 , 1)...
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

Contents
1

Some problems with the likelihood and how to fix themSome problems with the likelihood
Minimum Distance Estimation (MDE)

2

Minimum MMD estimation
Refinement of the bounds
Applications and extensions

3

Approximate Bayesian Computation and MMD
Discrepancy-based ABC
Contraction of discrepancy-based ABC

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

Minimum Distance Estimation
n

1X
δX .
Empirical distribution : P̂n :=n i=1 i
Minimum Distance Estimation (MDE)
Let d(·, ·) be a metric on probability distributions.
θ̂d := arg min d Pθ , P̂n .
θ∈Θ
Wolfowitz, J. (1957). The minimum distance method. The Annals of Mathematical Statistics.

Idea : MDE with an adequate d leads to robust estimation.
Bickel, P. J. (1976). Another look at robustness : a review of reviews and some newdevelopments. Scandinavian Journal of Statistics. Discussion by Sture Holm.
Parr, W. C. & Schucany, W. R. (1980). Minimum distance and robust estimation. JASA.
Yatracos, Y. G. (1985). Rates of convergence of minimum distance estimators and Kolmogorov’s
entropy. Annals of Statistics.
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

Integral Probability Semimetrics
Integral Probability Semimetrics (IPS)Let F be a set of real-valued, measurable functions and put
dF (P, Q) = sup EX ∼P [f (X )] − EX ∼Q [f (X )] .
f ∈FMüller, A. (1997). Integral probability metrics and their generating classes of functions. Applied
Probability.

assumptions required in order to ensure thatdF (P, Q) = 0 ⇒ P = Q (that is, dF is a metric).
assumptions required in order to ensure that dF < +∞.
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

Non-asymptotic bound for MDE
Theorem 1
X1 , . . . , Xn i.i.d from P0 ,for any f ∈ F, supx∈X |f (x)| ≤ 1.
i
Then hE dF (Pθ̂d , P0 ) ≤ inf dF (Pθ , P0 ) + 4.Radn (F).
Fθ∈Θ

Rademacher complexity"
Radn (F) := sup EY1 ,...,Yn ∼P Eϵ1 ,...,ϵn supP

n1X

f ∈F ni=1where ϵ1 , . . . , ϵn are i.i.d Rademacher variables :P(ϵ1 = 1) = P(ϵ1 = −1) = 1/2.
Pierre Alquier, RIKEN AIP

MMD estimation#
ϵi f (Yi ) .

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

Example 1 : set of indicators
1A (x) =1 if x ∈ A,
0 if x ∈/ A.

Image from Wikipedia.

Reminder - Vapnik-Chervonenkis dimension
Assume that F = {1A , A ∈ A} for some A ⊆ P(X ),SF (x1 , . . . , xn ) := {(f (x1 ), . . . , f (xn )), f ∈ F},
VC(F) := max {n : ∃x1 , . . . , xn , |SF (x1 , . . . , xn )| = 2n } .
Theorem (Bartlett and Mendelson)r
2.VC(F) log(n + 1)
Radn (F) ≤.
nBartlett, P. L. & Mendelson, S. (2002). Rademacher and Gaussian complexities : Risk bounds andstructural results. JMLR.
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

Example 1 : KS and TV distances
Two classical examples :A = {all measurable sets in X }, then dF (·, ·) is the total
variation distance TV(·, ·).
VC(F) = +∞ when |X | = +∞,in general, Radn (F) ↛ 0.

X = R, A = {(−∞, x] , x ∈ R}, then dF (·, ·) is the
Kolmogorov-Smirnov distance KS(·, ·).
KS distance was actually proposed by S. Holm for robust
estimation,
VC(F) = 1, so :r

2 log(n + 1).
E KS(Pθ̂KS , P0 ) ≤ inf KS(Pθ , P0 ) + 4.
θ∈ΘnPierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

Example 2 : Maximum Mean Discrepancy (MMD)
RKHS (H, ⟨·, ·⟩H ) with kernel k(x, y ) = ⟨ϕ(x), ϕ(y )⟩H .
If ∥ϕ(x)∥H = k(x, x) ≤ 1 then EX ∼µ [ϕ(X )] is well-defined .
The map µ 7→ EX ∼µ [ϕ(X )] is one-to-one if k is characteristic.
Gaussian kernel k(x, y ) = exp(−∥x − y ∥2 /γ 2 ) satisfies theseassumption.

F = {f ∈ H : ∥f ∥H ≤ 1}.
Dk (P, Q) := dF (P, Q) = sup EX ∼P [f (X )] − EX ∼Q [f (X )]f ∈F= EX ∼P [ϕ(X )] − EX ∼Q [ϕ(X )]Pierre Alquier, RIKEN AIP

MMD estimation

H.

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

Example 2 : MMD
Theorem (Bartlett and Mendelson)r
F = {f ∈ H : ∥f ∥H ≤ 1} ⇒ Radn (F) ≤supx k(x, x).
nCorollaryhri

E Dk (Pθ̂D , P0 ) ≤ inf Dk (Pθ , P0 ) + 4kθ∈ΘPierre Alquier, RIKEN AIP

MMD estimation

supx k(x, x).
nSome problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

Example 2 : MMD
We actually haven

D2k (Pθ , P̂n ) = EX ,X ′ ∼Pθ [k(X , X ′ )] −and so2X
EX ∼Pθ [k(Xi , X )]
n i=1
1 X
+ 2
k(Xi , Xj )n 1≤i,j≤n∇θ D2k (Pθ , P̂n )("= 2EX ,X ′ ∼Pθ

#
)
n
X
1
k(X , X ′ ) −k(Xi , X ) ∇θ [log pθ (X )]n i=1that can be approximated by sampling from Pθ .
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

Example 3 : Wasserstein
Another classical metric belongs to the IPS family :Wδ (P, Q) =sup

EX ∼P [f (X )] − EX ∼Q [f (X )]f :X →R
Lip(f ) ≤ 1

where Lip(f ) := supx̸=y |f (x) − f (y )|/δ(x, y ).
Bound on the Rademacher complexity when X is bounded :Sriperumbudur, B.K., Fukumizu, K., Gretton, A., Schölkopf, B., Lanckriet, G.R. (2010).
Non-parametric estimation of integral probability metrics. IEEE International Symposium on
Information Theory.

Minimum Wasserstein estimation studied in :Bernton, E., Jacob, P. E., Gerber, M. & Robert, C. P. (2019). On parameter estimation with the
Wasserstein distance. Information and Inference : A Journal of the IMA.
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

MDE and robustness
Reminderhi
E dF (Pθ̂d , P0 ) ≤ inf dF (Pθ , P0 ) + 4.Radn (F).
θ∈ΘF

Huber’s contamination model : P0 = (1 − ε)Pθ0 + εQ.
dF (Pθ0 , P0 ) = sup EX ∼Pθ0 f (X ) − (1 − ε)EX ∼Pθ0 f (X ) − εEX ∼Q f (X )f ∈F

= sup εEX ∼Pθ0 f (X ) − εEX ∼Q f (X )f ∈F

= ε.dF (Pθ0 , Q) ≤ 2εif for any f ∈ F, sup |f (x)| ≤ 1xCorollary - in Huber’s contamination modelhi
E dF (Pθ̂d , Pθ0 ) ≤ 4ε + 4.Radn (F).
F

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Some problems with the likelihood
Minimum Distance Estimation (MDE)

MDE and robustness : toy experiment
Model : N (θ, 1), X1 , . . . , Xn i.i.d N (θ0 , 1), n = 100 and werepeat the exp. 200 times. Kernel k(x, y ) = exp(−|x − y |).

mean abs. error

θ̂MLE θ̂MMDk θ̂KS
0.081 0.094 0.088

Now, ε = 2% of the observations drawn from a Cauchy.
mean abs. error

0.276 0.095 0.088

Now, ε = 1% are replaced by 1, 000.
mean abs. error

Pierre Alquier, RIKEN AIP

10.008 0.088 0.082

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Contents
1

Some problems with the likelihood and how to fix themSome problems with the likelihood
Minimum Distance Estimation (MDE)

2

Minimum MMD estimation
Refinement of the bounds
Applications and extensions

3

Approximate Bayesian Computation and MMD
Discrepancy-based ABC
Contraction of discrepancy-based ABC

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Improving the constant
From now, we assume that supx k(x, x) ≤ 1. We know :hi4
E Dk (Pθ̂D , P0 ) ≤ inf Dk (Pθ , P0 ) + √ .
kθ∈ΘnWe will now prove a better result without using theRademacher complexity :Theoremhi2
E Dk (Pθ̂D , P0 ) ≤ inf Dk (Pθ , P0 ) + √ .
kθ∈ΘnPierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Proof of the theorem : preliminary lemma
Lemma
For any P0 , when X1 , . . . , Xn are i.i.d from P0 ,h

E Dk P̂n , P

0

i

1
≤√ .
n

io2
i
h 
n h 
E Dk P̂n , P 0≤ E D2k P̂n , P 0X2= E (1/n)(µ(δXi ) − µ(P0 ))Hhi= (1/n)E ∥µ(δX1 ) − µ(P0 )∥2H
≤ 1/n.
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Proof of the theorem






∀θ, Dk Pθ̂ , P 0 ≤ Dk Pθ̂ , P̂n + Dk P̂n , P 0≤ Dk Pθ , P̂n + Dk P̂n , P 0



0
0
≤ Dk Pθ , P + 2Dk P̂n , P
2E [Dk (Pθ̂ , P0 )] ≤ inf Dk (Pθ , P0 ) + √ .
θ∈ΘnPierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

A bound in probability
Thanks to McDiarmid’s inequality :Theorem
For any P0 , when X1 , . . . , Xn are i.i.d from P0 , with probability
at least 1 − δ,q

2 + 2 2 log 1δ√
Dk Pθ̂ , P 0 ≤ inf Dk Pθ , P 0 +.
θ∈ΘnJoint work with Badr-Eddine Chérief-Abdellatif (CNRS).

Chérief-Abdellatif, B.-E. and Alquier, P. Finite Sample
Properties of Parametric MMD Estimation : Robustness
to Misspecification and Dependence. Bernoulli, 2022.

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Example : Gaussian mean estimation
Example : Pθ = N (θ, σ 2 I ) for θ ∈ Rd .

Using a Gaussian kernel k(x, y ) = exp(−∥x − y 2 ∥/γ 2 ),
 d2 γ2∥θ − θ′ ∥2
D2k (Pθ , Pθ′ ) = 2
1
−
exp
−
.
4σ 2 + γ 2
4σ 2 + γ 2
Together with the previous result, this gives :∥θ̂nMMD − θ0 ∥2"≤ −(4σ 2 + γ 2 ) log 1 − 4

(1 +

p

d #
2 log 1/δ)2 4σ 2 + γ 2 2.
nγ2

γ = 2dσ 2 ⇒8e(1 +∥θ̂nMMD − θ0 ∥2 ≤ dσ 2
Pierre Alquier, RIKEN AIPp

2 log 1/δ)2(1 + o(1)).
n

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Variance-aware bounds (1/2)
n h 
h 
i
io2
E Dk P̂n , P 0≤ E D2k P̂n , P 0X2= E (1/n)(µ(δXi ) − µ(P0 ))Hhi2= (1/n)E ∥µ(δX1 ) − µ(P0 )∥H|{z}=:vk (P0 )Lemma - variance-aware version
r
r
h 
i
vk (P0 )1
0
E Dk P̂n , P
≤
≤
.
n
n
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Variance-aware bounds (2/2)
Theorem – bound in expectationr
E [Dk (Pθ̂ , P0 )] ≤ inf Dk (Pθ , P0 ) + 2
θ∈Θ

vk (P0 ).
n

Theorem – bound in probability
With probability at least 1 − δ,s
Dk Pθ̂ , P 0 ≤ inf Dk Pθ , P 0 + 2θ∈Θ

vk (P0 )2 log 1δ
8 log 1δ+.
n
3n

Joint work with Geoffrey Wolfer (RIKEN AIP).

Wolfer, G. and Alquier, P. Variance-Aware Estimation of Kernel
Mean Embedding. Preprint arXiv :2210.06672.

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Upper-bounding the variance vk (P0)In the case of the Gaussian kernelk(x, y ) = exp(−∥x − y ∥2 /γ 2 )we have ( 2Tr(VarP (X ))0
2Tr(VarP0 (X ))γ2
vk (P0 ) ≤ 1 − exp −
≤
γ2
1.


Example : Gaussian mean estimation (continued).
Using the variance aware boundγ = γn → +∞ ⇒ ∥θ̂nMMD − θ0 ∥2 ≤ dσ 2
Pierre Alquier, RIKEN AIP

4 log 1/δ
(1 + o(1)).
n

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Empirical boundIn practice, we can estimate vk (P0 ) by
!
n
n
1X
1 X
k(Xi , Xi ) −v̂k :=k(Xi , Xj ) .
n − 1 i=1
n j=1
We have E(v̂k ) = vk (P0 ), andTheorem – bound with empirical variance
Assume that k(x, y ) = ψ(x − y ) ∈ [a, b]. Then, withprobability at least 1 − δ,
s
√
1


v̂
2
log
32
b − a log 1δ
k00δ
Dk Pθ̂ , P ≤ inf Dk Pθ , P +2
+
.
θ∈Θ
n
3n
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Contents
1

Some problems with the likelihood and how to fix themSome problems with the likelihood
Minimum Distance Estimation (MDE)

2

Minimum MMD estimation
Refinement of the bounds
Applications and extensions

3

Approximate Bayesian Computation and MMD
Discrepancy-based ABC
Contraction of discrepancy-based ABC

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Generative Adversarial Networks (GAN, 1/2)
Generative model X ∼ Pθ :U ∼ Unif[0, 1]d ,X = Fθ (U) where Fθ is someNN with weights θ.

Dziugaite, G. K., Roy, D. M. & Ghahramani, Z. (2015). Training generative neural networks via
maximum mean discrepancy optimization. UAI.
Li, Y., Swersky, K. & Zemel, R. (2015). Generative Moment Matching Networks. ICML.

→ proposed to minimize the MMD to learn θ.
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

GAN (2/2)
Results from Dziugaite et al. (2015).

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Inference for Systems of SDEs (1/2)
This paper developped the asymptotic theory of MMD :Briol, F. X., Barp, A., Duncan, A. B., & Girolami, M. (2019). Statistical Inference for Generative
Models with Maximum Mean Discrepancy. Preprint arXiv :1906.05944.

They also applied the method to inference in SDEs :dXt = b(Xt , θ1 )dt + σ(Xt , θ2 )dWt
easy to sample from the model with a given θ = (θ1 , θ2 ),they propose a method to approximate the gradient of
the MMD criterion.

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Inference for Systems of SDEs (2/2)
Example in a (stochastic) Lotka-Volterra model.

Results from Briol et al. (2019) : compare MMD minimization
to Wasserstein minimization.

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Regression
problem with regression : we want to specify and estimate
a parametric model Pθ(X ) for Y |X . MMD requires tospecify a model for (X , Y ).
naturalPn idea : estimate the distribution of X by1
i=1 δXi and use the MMD procedure on Pθ(X ) .
nthe previous theory shows directly that we estimate the
distribution of (X , Y ) consistently.
it is far more difficult to prove that we estimate the
distribution of Y |X .
Joint work with M. Gerber (Bristol).
Alquier, P. and Gerber, M. (2020). Universal Robust Regression via
Maximum Mean Discrepancy. Preprint arXiv.

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Copulasanother semi-parametricmodel : copulas.
asymptotic theory + R
package.
With B.-E. Chérief-Abdellatif (CNRS), J.-D. Fermanian (ENSAE Paris), A. Derumigny (TU Delft).
Alquier, P., Chérief-Abdellatif, B.-E., Derumigny, A. and Fermanian, J.-D. Estimation of copulas
via Maximum Mean Discrepancy. JASA, to appear.

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Example : Gaussian copulas

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Refinement of the bounds
Applications and extensions

Example : other models

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Contents
1

Some problems with the likelihood and how to fix themSome problems with the likelihood
Minimum Distance Estimation (MDE)

2

Minimum MMD estimation
Refinement of the bounds
Applications and extensions

3

Approximate Bayesian Computation and MMD
Discrepancy-based ABC
Contraction of discrepancy-based ABC

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Co-authors and paper
S. Legramanti, D. Durante & P. Alquier (2022). Concentration and robustness of
discrepancy–based ABC via Rademacher complexity. Preprint arXiv :2206.06991.

Sirio Legramanti(University of Bergamo)

Pierre Alquier, RIKEN AIP

Daniele Durante(Bocconi University, Milan)

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Estimators, randomized estimators and Bayes ruleY1:n = Y1 , . . . , Yn i.i.d from µ∗ ,model : (µθ , θ ∈ Θ),
estimator : θ̂ = θ̂(Y1:n ),randomized estimator : ρ̂(·) = ρ̂(Y1:n )(·) probability
measure on Θ.
Examples of randomized estimators :posterior : ρ̂(θ) = π(θ|Y1:n ) ∝ L(θ; Y1:n )π(θ),| {z }|{z}likelihood

prior

fractional/tempered posterior : ρ̂(θ) ∝ [L(θ; Y1:n )]α π(θ),
Gibbs estimator : ρ̂(θ) ∝ exp[−η R(θ; Y1:n )]π(θ).
| {z }
loss

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Evaluating randomized estimators
Assume in this slide that µ∗ = µθ0 : “the truth is in the model”.
Statistical performance of an estimator :consistency : d(θ̂, θ0 ) −−−→ 0 ( in proba., a.s., ...) ?
n→∞rate of convergence : EY1:n [d(θ̂, θ0 )] ≤ rn −−−→ 0 ?
n→∞...
For a randomized estimator :contraction rate :Pθ∼ρ̂ [d(θ, θ0 ) ≥ rn ] −−−→ 0 ( in proba., a.s., ...) ?
n→∞hiaverage risk : EY1:n Eθ∼ρ̂ [d(θ, θ0 )] ≤ rn ?
...
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Approximate Bayesian InferenceWell-known conditions to prove contraction of theposterior,
tools from ML for randomized estimators : PAC-Bayes
bounds.
Given a “non-exact” algorithm targetting ρ̂ instead ofπ(·|Y1:n ) : variational approximations, ABC, etc., we canquantify how well ρ̂ approximates π(·|Y1:n ) ?
study ρ̂ as a randomized estimator and study its
contraction/convergence.

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Reminder on ABC
Approximate Bayesian Computation (ABC)
INPUT : sample Y1:n = (Y1 , . . . , Yn ), model (µθ , θ ∈ Θ), priorπ, statistic S, metric δ and threshold ϵ.
(i) sample θ ∼ π,(ii) sample Z1:n = (Z1 , . . . , Zn ) i.i.d. from µθ :if δ(S(Y1:n ), S(Z1:n )) ≤ ϵ return θ,else goto (i).

OUTPUT : ϑ ∼ ρ̂.
discrete sample space, if S =identity and ϵ = 0, ABC isactually exact : ρ̂(·) = π(·|Y1:n ).
general case : ABC not exact, we can ask two questions :12

is ρ̂(·) a good approximation of π(·|Y1:n ) ?
is ρ̂ a good randomized estimator ?
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Discrepancy-based ABC
Approximate Bayesian Computation (ABC)
INPUT : sample Y1:n , model (µθ , θ ∈ Θ), prior π, IPM dF andthreshold ϵ.
(i) sample θ ∼ π,(ii) sample Z1:n i.i.d. from µθ :if dF (µ̂Y1:n , µ̂Z1:n ) ≤ ϵ return θ,else goto (i).

OUTPUT : ϑ ∼ ρ̂ϵ .
Rermark : when dF is the MMD with kernel k,XXX
dF (µ̂Y1:n , µ̂Z1:n ) =k(Yi , Yj ) − 2k(Yi , Zj ) +k(Zi , Zj ).
i,jPierre Alquier, RIKEN AIPi,j
MMD estimationi,jSome problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Approximation of the posterior
Bernton, E., Jacob, P. E., Gerber, M. & Robert, C. P. (2019). Approximate Bayesian
Computation with the Wasserstein distance. JRSS-B.

Contains a general result that can be applied here.
TheoremAssumeµθ has a continuous density fθ and for
Q some neighborhood
V of Y1:n we have supθ∈Θ supv1:n ∈V ni=1 fθ (vi ) < +∞.
v1:n 7→ dF (µ̂Y1:n , µ̂v1:n ) is continuous.
Then∀ measurable set A, ρ̂ϵ (A) −−→ π(A|Y1:n ).
ϵ→0Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Assumptions for contraction(C1) Y-valued Y1:n = (Y1 , . . . , Yn ) i.i.d from µ∗ , put :ϵ∗ := inf dF (µθ , µ∗ ).
θ∈Θ(C2) prior mass condition : there is c > 0, L ≥ 1 such thatπ θ ∈ Θ : dF (µθ , µ∗ ) − ϵ∗ ≤ ϵ ≥ cϵL(C3) functions in F are bounded :sup sup |f (y )| ≤ b.
f ∈F y ∈Y(C4) the Rademacher complexity Rn (F) satisfies
Rn (F) −−−→ 0.
n→∞Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Contraction of discrepancy-based ABC
Theorem 1
Under (C1)-(C4), with ϵ := ϵn = ϵ∗ + ϵ̄n with ϵ̄n → 0,nϵ̄2n → ∞ and ϵ̄n /Rn (F) → ∞. Then, for any sequence
Mn > 1, 2 · 3Lθ ∈ Θ : dF (µθ , µ∗ ) > ϵ∗ + rn≤
cMns
2 log( Mε̄Ln )4ϵ̄nnwhere rn =+ 2Rn (F) + b
,
3
n

ρ̂ϵn



with probability → 1 with respect to the sample Y1:n .
Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Examplesp
Assume Rn (F) ≤ c 1/np (MMD, Kolmogorov...).
Take Mn = n and ϵ̄n = log(n)/n to get 2 · 3Lθ ∈ Θ : dF (µθ , µ∗ ) > ϵ∗ + rn
≤
cn
p

where rn = O
log(n)/n .

ρ̂ϵnLarger Rn (F) will lead to slower rates.

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Removing (C3)-(C4)if we remove (C3)-(C4), we cannot use classicalconcentration results on dF (µ∗ , µ̂Y1:n ) and dF (µθ , µ̂Z1:n ).
we can still provide a result under the assumption that“some concentration holds”, as
Bernton, E., Jacob, P. E., Gerber, M. & Robert, C. P. (2019). Approximate Bayesian
Computation with the Wasserstein distance. JRSS-B.

for the Wasserstein distance.
however, this will impose assumptions on µ∗ , {µθ , θ ∈ Θ}and might lead to slower contraction rates. In our paper,
we illustrate this with MMD with unbounded kernels :rsupy ∈Y k(y , y )Rn (F) ≤= +∞.
nPierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Example : MMD-ABC with unbounded kernel
Theorem 2
Under (C1)-(C2), and(C5) EY ∼µ∗ [k(Y , Y )] < +∞,(C6) supθ∈Θ EZ ∼µθ [k(Z , Z )] < +∞,ϵn = ϵ∗ + ϵ̄n with ϵ̄n → 0. Then, for some C > 0, for any
sequence Mn > 1, with proba. → 1,
ρ̂ϵn



θ ∈ Θ : dF (µθ , µ∗ ) > ϵ∗ + rn

where rn =For example Mn =4ϵ̄n
M2
+ 2 n2L .
3
n ϵ̄



≤

C
Mn

√
n we can get rn = O(1/n2L+1 ).

Pierre Alquier, RIKEN AIP

MMD estimation

Some problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

Experiments in the Gaussian case
2.5

2.0

α
0.00

1.5

0.05
0.10
1.0

0.15

0.5

0.0

MMD

Wasserstein

KL

Pierre Alquier, RIKEN AIP

γ -divergence

MMD estimation``Exact`` GaussianSome problems with the likelihood and how to fix them
Minimum MMD estimation
Approximate Bayesian Computation and MMD

Discrepancy-based ABC
Contraction of discrepancy-based ABC

La fin終わり
ありがとう ございます。

Pierre Alquier, RIKEN AIP

MMD estimation