Minimum Distance Estimation and MMD
Applications
Algorithms

Minimum Distance Estimation with MMD
Pierre AlquierUCD Dublin, School of Mathematics and Statistics
Statistics and Actuarial Science seminar
Nov. 30, 2023

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

1

Minimum Distance Estimation and MMD

2Applications

3

Algorithms

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Let X1 , . . . , Xn be i.i.d in X from a probability distribution P ∗ .
Statistical inference :propose a model (Pθ , θ ∈ Θ), often assume P ∗ = Pθ∗ ,compute θ̂n = θ̂n (X1 , . . . , Xn ).
Standard methods :Maximum Likelihood Estimators (MLE),
Bayesian inference,
method of moments...
Theory of Bayes/MLE strongly relies on P ∗ = Pθ∗ , method ofmoments unstable if the Xi ’s are heavy-tailed...
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Example :pθ (x) =exp(−|x − θ|)p,
2 π|x − θ|

Pierre Alquier, ESSEC Business SchoolEstimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Example :pθ (x) =exp(−|x − θ|)p,
2 π|x − θ|Pexp (− ni=1 |Xi − θ|).
L(θ) = √Q p
(2 π)n ni=1 |Xi − θ|

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

What is an outlier ?
Huber proposed the contamination model : with probability ε,
Xi is not drawn from Pθ∗ but from Q that can be anything :P ∗ = (1 − ε)Pθ∗ + εQ.
Example : Pθ = Unif [0, θ], thenn

1 YL(θ) = n1{0≤Xi ≤θ} ⇒ θ̂ = max Xi .
1≤i≤n
θ i=1
In the case of the following contamination, the MLE isextremely far from the truth :P0 = 0.99 · Unif [0, 1] + 0.01 · Unif [1000, 2000].
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Minimum Distance Estimation (MDE)
Let d(·, ·) be a metric on probability distributions, we put :n1Xθ̂d := arg min d Pθ , P̂n where P̂n :=δX .
n i=1 iθ∈Θ
MDE with well-chosen d leads to robust estimation.
Wolfowitz, J. (1957). The minimum distance method. The Annals of Mathematical Statistics.
Bickel, P. J. (1976). Another look at robustness : a review of reviews and some newdevelopments. Scandinavian Journal of Statistics. Discussion by Sture Holm.
Parr, W. C. & Schucany, W. R. (1980). Minimum distance and robust estimation. JASA.

Kolmogorov-Smirnov (KS) suggested by Sture Holm.
Wasserstein distance studied recently (but not so robust).
This talk : Maximum Mean Discrepancy (MMD).
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Let H be a Hilbert space and any continuous function
Φ : X → H. The function
K (x, y ) = ⟨Φ(x), Φ(y )⟩His called a kernel.
Mercer’s theorem
Let K (x, y ) be a continuous function such that for any(x1 , . . . , xn ) ∈ X n and (c1 , . . . , cn ) ̸= (0, . . . , 0) ∈ Rn ,n XnX

ci cj K (xi , xj ) > 0,i=1 j=1

then there is H and Φ such that K (x, y ) = ⟨Φ(x), Φ(y )⟩H .
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.
Consider, for any probability distribution P on X ,µK (P) = EX ∼P [Φ(X )] .
The kernel K is said to be characteristic ifµK (P) = µK (Q) ⇒ P = Q.
Examples2∥∥
K (x, y ) = exp(− ∥x−y) and exp(− ∥x−y) are char. kernels.
γ2γ

Definition : the MMD distance
DK (P, Q) = ∥µK (P) − µK (Q)∥H .
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Reminder of the context :1X1 , . . . , Xn be i.i.d in X from a probability distribution P ∗ ,2model (Pθ , θ ∈ Θ).
Definition - MMD Minimum Distance Estimatorn
1Xθ̂K := arg min DK Pθ , P̂n where P̂n =δX .
n i=1 iθ∈Θ

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Theorem
For any P ∗ , when X1 , . . . , Xn are i.i.d from P ∗ ,
2E DK Pθ̂K , P ∗ ≤ inf DK (Pθ , P ∗ ) + √ .
θ∈ΘnChérief-Abdellatif, B.-E. and Alquier, P. (2022). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Bernoulli.

If the model is well-specified, that is P ∗ = Pθ∗ ,2
E DK Pθ̂K , Pθ∗ ≤ √ .
nIn Huber contamination model P ∗ = (1 − ε)Pθ∗ + εQ,using the triangle inequality a few times :2E DK Pθ̂K , Pθ∗ ≤ 4ε + √ .
nPierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Corollary (informally stated)
If the model and the kernel K satisfy, for θ′ → θ,
DK (Pθ , Pθ′ ) ∼ c · ∥θ − θ′ ∥pthen2∥θ̂K − θ ∥ ∼ √c n∗ p
4ε+.
cFor example, if Pθ = N (θ, Id ), d2 γ2∥θ − θ′ ∥22
DK (Pθ , Pθ′ ) = 2
1 − exp − 24σ 2 + γ 24σ + γ 2and thus∥θ̂K − θ∗ ∥2 ≲16dσ 2nPierre Alquier, ESSEC Business School(+16dε).

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Setting : we repeat the exp. 200 times,Pθ = N (θ, 1),
X1 , . . . , Xn i.i.d Pθ∗ , n = 100.

mean abs. error

MLE MMD KS
0.081 0.094 0.088

Now, ε = 2% of the observations drawn from a Cauchy.
mean abs. error

0.276 0.095 0.088

Now, ε = 1% are replaced by 1, 000.
mean abs. error

Pierre Alquier, ESSEC Business School10.008 0.088 0.082

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

1

Minimum Distance Estimation and MMD

2Applications

3

Algorithms

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

1

Minimum Distance Estimation and MMD

2Applications

3

Algorithms

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Application 1 : Generative AI.
Generative model X ∼ Pθ :U ∼ Unif[0, 1]d ,X = Fθ (U) where Fθ is someneural network with weights θ.

Dziugaite, G. K., Roy, D. M. and Ghahramani, Z. (2015). Training generative neural networks via
maximum mean discrepancy optimization. UAI.
Li, Y., Swersky, K. and Zemel, R. (2015). Generative Moment Matching Networks. ICML.

→ proposed to minimize the MMD to learn θ.
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Results from Dziugaite et al. (2015).

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Application 2 : Stochastic Differential Equations.
Briol, F. X., Barp, A., Duncan, A. B., & Girolami, M. (2019). Statistical Inference for Generative
Models with Maximum Mean Discrepancy. Preprint arXiv :1906.05944.

Stochastic version of Lotka-Volterra model.

Comparison of MMD // Wasserstein minimization.

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Application 3 : Copulas.
Cθ (x1 , x2 ) = Pθ (X1 ≤ F1−1 (x1 ), X2 ≤ F2−1 (x2 )).
Alquier, P., Chérief-Abdellatif, B.-E., Derumigny, A. and Fermanian, J.-D. (2023). Estimation ofcopulas via Maximum Mean Discrepancy. JASA.

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Application 4 : quantization / data compression.
Teymur, O., Gorham, J., Riabiz, M. and Oates, C. (2021). Optimal quantisation of probability
measures using maximum mean discrepancy. AISTAT.

Idea : compress a sample X1 , . . . , Xn to x1 , . . . , xm withm ≪ n, by!
nm1X1 Xmin DKδX ,δx .
θ=(x1 ,...,xm )n i=1 i m j=1 j

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Application/extension 5 : time series.
Theorem from a stationary time series withWhen X1 , . . . , Xn are i.i.dmarginal distribution P ∗ ,pP
2 1 + 2 s>0 ρs∗∗√
E DK Pθ̂K , P≤ inf DK (Pθ , P ) +θ∈Θ
nwhereρs = |EK (X0 , Xs ) − EK (X0 , X0′ )|.
Chérief-Abdellatif, B.-E. and Alquier, P. (2022). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Bernoulli.

For classical processes (MarkovP chains and hidden Markovchains, autoregressions etc), s>0 ρs < +∞.
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Application/extension 6 : regression.
Assume we observe (X1 , Y1 ), . . . , (Xn , Yn ).
A direct application of the above allows to estimate thejoint distribution of (X , Y ).
But we are interested in Y |X , and we don’t want tospecify a model for X !
Feasible, but far more complicated as the theory ofconditional mean embeddings is difficult...
Alquier, P. and Gerber, M. (2024). Universal Robust Regression via Maximum Mean Discrepancy.
To appear in Biometrika.

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

1

Minimum Distance Estimation and MMD

2Applications

3

Algorithms

Pierre Alquier, ESSEC Business School

Estimation with MMDMinimum Distance Estimation and MMD
Applications
Algorithms

ReminderµK (P) = EX ∼P [Φ(X )]K (u, v ) = ⟨Φ(u), Φ(v )⟩H ,

D2K (P, Q) = ∥µK (P) − µK (Q)∥2H .
D2K (P, Q) = ∥µK (P)∥2H − 2 ⟨µK (P), µK (Q)⟩H + ∥µK (Q)∥2H .
For example,⟨µK (P), µK (Q)⟩H = ⟨EX ∼P [Φ(X )], EX ′ ∼Q [Φ(X ′ )]⟩H= EX ∼P,X ′ ∼Q ⟨Φ(X ), Φ(X ′ )⟩H= EX ∼P,X ′ ∼Q K (X , X ′ ).
D2K (P, Q) = EX ∼P,Y ∼P K (X , Y ) − 2EX ∼P,X ′ ∼Q K (X , X ′ )+ EX ′ ∼Q,Y ′ ∼Q K (X , X ′ ).
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Reminderθ̂K := arg min DK Pθ , P̂n
θ∈Θ



n

1X
where P̂n =δX .
n i=1 i2
D2K (Pθ , P̂n ) = EX ∼Pθ ,Y ∼Pθ K (X , Y ) −n
XEX ∼Pθ K (X , Xi )n i=11 X
(X+ 2 Ki , Xj ).
n 1≤i,j≤n
Exact evaluation sometimes possible if we have a close
formula for EX ∼Pθ ...
Monte-Carlo approx. only requires to sample X ∼ Pθ .
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Theorem : gradient of MMD
Assume Pθ has a differentiable density pθ , then∇θ D2K (Pθ , P̂n )()nX
1= 2EX ,X ′ ∼PθK (X , X ′ ) −K (Xi , X ) ∇θ log pθ (X ) .
n i=1
If we sample X , X ′ ∼ Pθ , thenn
1X′ˆ∇ := K (X , X ) −K (Xi , X ) ∇θ log pθ (X )n i=1

is an unbiased estimator of ∇θ D2K (Pθ , P̂n ). We can thus
minimize by stochastic gradient.
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

R package (joint work with Mathieu Gerber) available soon.
Will contain stochastic optimization to compute θ̂K in 15-20
classical parametric models + regression, logistic regression,etc.

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms“Bayesian style” estimators (and algorithms) based on MMD ?
1) MMD-Bayeshiπ(θ|X1 , . . . , Xn ) ∝ π(θ) exp −β · DK Pθ , P̂nπ(θ|X1 , . . . , Xn ) approximated by variational inference(using stochastic gradient again)
Chérief-Abdellatif, B.-E. and Alquier, P. (2020). MMD-Bayes : Robust Bayesian Estimation via
Maximum Mean Discrepancy. Proceedings of AABI.

pseudo-marginal MCMC / noisy MCMC :Pacchiardi, L. and Dutta, R. (2021). Generalized Bayesian likelihood-free inference using scoringrules estimators. ArXiv preprint arXiv :2104.03889.

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMDApplications
Algorithms2) MMD-ABC
INPUT : data X1 , . . . , Xn , model (Pθ , θ ∈ Θ), prior π andthreshold ϵ.
(i) sample θ ∼ π,(ii) sample Z1 , . . . , Zn i.i.d. from Pθ ,PP
if DK ( n1 ni=1 δZi , n1 ni=1 δXi ) ≤ ϵ return θ,else goto (i).

OUTPUT : ϑ.
The distribution πMMD-ABC (ϑ) of the output ϑ convergesto the (usual) posterior when ϵ → 0,
ϵ > 0 computationally more efficient AND robust...
S. Legramanti, D. Durante & P. Alquier (2022). Concentration and robustness of
discrepancy–based ABC via Rademacher complexity. Preprint arXiv :2206.06991.
Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Open questions :faster algorithms, especially when likelihood not available,
general results to connect ∥θ1 − θ2 ∥ to DK (Pθ1 , Pθ2 ) in
non-standard models,data-driven kernel choice ? Does it always lead to optimal
rates ?
Much tighter bounds on DK Pθ̂K , P ∗ in the following paper.
Helps to understand the role of the kernel K :Wolfer, G. and Alquier, P. (2022). Variance-Aware Estimation of Kernel Mean Embedding.
Preprint arXiv :2210.06672.

Pierre Alquier, ESSEC Business School

Estimation with MMD

Minimum Distance Estimation and MMD
Applications
Algorithms

Thank you !

Pierre Alquier, ESSEC Business School

Estimation with MMD