Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Parametric estimation via MMD
optimization
Pierre Alquier

Séminaire de Statistique du CREST et du CMAP
Feb. 1, 2021

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Co-authors

Badr-Eddine Chérief-Abdellatif

Alexis Derumigny

Univ. of Oxford

Univ. of Twente

Mathieu GerberJean-David FermanianUniv. of Bristol

ENSAE Paris

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

The Maximum Likelihood Estimator (MLE)
Let X1 , . . . , Xn be i.i.d in X from a probability distribution P0 .

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

The Maximum Likelihood Estimator (MLE)
Let X1 , . . . , Xn be i.i.d in X from a probability distribution P0 .
Statistical inference :propose a model (Pθ , θ ∈ Θ), assume P0 = Pθ0 .
compute θ̂n = θ̂n (X1 , . . . , Xn ).

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

The Maximum Likelihood Estimator (MLE)
Let X1 , . . . , Xn be i.i.d in X from a probability distribution P0 .
Statistical inference :propose a model (Pθ , θ ∈ Θ), assume P0 = Pθ0 .
compute θ̂n = θ̂n (X1 , . . . , Xn ).
Letting pθ denote the density of Pθ , thenθ̂nMLE = arg max L(θ), where L(θ) =θ∈ΘPierre Alquier, RIKEN AIPnYpθ (Xi ).

i=1

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

The Maximum Likelihood Estimator (MLE)
Let X1 , . . . , Xn be i.i.d in X from a probability distribution P0 .
Statistical inference :propose a model (Pθ , θ ∈ Θ), assume P0 = Pθ0 .
compute θ̂n = θ̂n (X1 , . . . , Xn ).
Letting pθ denote the density of Pθ , thenθ̂nMLE = arg max L(θ), where L(θ) =
θ∈Θ

n
Y

pθ (Xi ).

i=1

2

Example : P(m,σ) = N (m, σ ) then
n
n
1X
1X
2
m̂ =
Xi and σ̂ =(Xi − m̂)2 .
n i=1
n i=1
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

MLE not unique / not consistent
Example :pθ (x) =exp(−|x − θ|)p,
2 π|x − θ|

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

MLE not unique / not consistent
Example :pθ (x) =exp(−|x − θ|)p,
2 π|x − θ|P
exp (− ni=1 |Xi − θ|).
L(θ) = √Q p(2 π)n ni=1 |Xi − θ|

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

MLE fails in the presence of outliers
What is an outlier ?

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

MLE fails in the presence of outliers
What is an outlier ?
Huber proposed the contamination model : with probability ε,
Xi is not drawn from Pθ0 but from Q that can be anything :P0 = (1 − ε)Pθ0 + εQ.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

MLE fails in the presence of outliers
What is an outlier ?
Huber proposed the contamination model : with probability ε,
Xi is not drawn from Pθ0 but from Q that can be anything :P0 = (1 − ε)Pθ0 + εQ.
Example : Pθ = Unif [0, θ], then

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

MLE fails in the presence of outliers
What is an outlier ?
Huber proposed the contamination model : with probability ε,
Xi is not drawn from Pθ0 but from Q that can be anything :P0 = (1 − ε)Pθ0 + εQ.
Example : Pθ = Unif [0, θ], thenn

1 Y
L(θ) = n1{0≤Xi ≤θ} ⇒ θ̂ = max Xi .
1≤i≤n
θ i=1

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

MLE fails in the presence of outliers
What is an outlier ?
Huber proposed the contamination model : with probability ε,
Xi is not drawn from Pθ0 but from Q that can be anything :P0 = (1 − ε)Pθ0 + εQ.
Example : Pθ = Unif [0, θ], thenn

1 Y
L(θ) = n1{0≤Xi ≤θ} ⇒ θ̂ = max Xi .
1≤i≤n
θ i=1
In the case of the following contamination, the MLE isextremely far from the truth :P0 = (1 − ε).Unif [0, 1] + ε.N (1010 , 1)...
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Some examples
Yatracos’ skeleton estimate θ̂nY :hi
E dTV (Pθ̂nY , P0 ) ≤ 3dTV (P0 , Pθ0 ) + C .

rdim(Θ)nwheredTV (P, Q) = sup |P(E ) − Q(E )|.
E

Yatracos, Y. G. (1985). Rates of convergence of minimum distance estimators and Kolmogorov’s
entropy. Annals of Statistics.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Some examples
Yatracos’ skeleton estimate θ̂nY :hi
E dTV (Pθ̂nY , P0 ) ≤ 3dTV (P0 , Pθ0 ) + C .

rdim(Θ)nwheredTV (P, Q) = sup |P(E ) − Q(E )|.
E

Yatracos, Y. G. (1985). Rates of convergence of minimum distance estimators and Kolmogorov’s
entropy. Annals of Statistics.

More recent work with the Hellinger distance :Baraud, Y., Birgé, L., & Sart, M. (2017). A new method for estimation and model selection :ρ-estimation. Inventiones mathematicae.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

But...

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

But...

Problem with the aforementioned estimators : they cannot becomputed in practice.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

But...

Problem with the aforementioned estimators : they cannot becomputed in practice.

Additional requirement : an estimator must be computable ! ! !

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Overview of the talk1

Estimation via MMD optimization
Definition of the estimator
Basic properties
References and further works

2

Robustness to outliers and to dependence
Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

3

Further topics
Semi-parametric models
CopulasMMD-Bayes
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

1

Estimation via MMD optimization
Definition of the estimator
Basic properties
References and further works

2

Robustness to outliers and to dependence
Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

3

Further topics
Semi-parametric models
CopulasMMD-Bayes

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Reminder : kernels
Let H be a Hilbert space and any continuous function
Φ : X → H. The function
K (x, y ) = hΦ(x), Φ(y )iHis called a kernel.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Reminder : kernels
Let H be a Hilbert space and any continuous function
Φ : X → H. The function
K (x, y ) = hΦ(x), Φ(y )iHis called a kernel. Conversely :Mercer’s theorem
Let K (x, y ) be a continuous function such that for any(x1 , . . . , xn ) ∈ X n and (c1 , . . . , cn ) 6= (0, . . . , 0) ∈ Rn ,n XnXci cj K (xi , xj ) > 0,i=1 j=1

then there is H and Φ such that K (x, y ) = hΦ(x), Φ(y )iH .
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Reminder : MMD
Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Reminder : MMD
Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.
Consider, for any probability distribution P on X ,µK (P) = EX ∼P [Φ(x)] .

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Reminder : MMD
Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.
Consider, for any probability distribution P on X ,µK (P) = EX ∼P [Φ(x)] .
The kernel K is said to be characteristic ifµK (P) = µK (Q) ⇒ P = Q.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Reminder : MMD
Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.
Consider, for any probability distribution P on X ,µK (P) = EX ∼P [Φ(x)] .
The kernel K is said to be characteristic ifµK (P) = µK (Q) ⇒ P = Q.
Theorem2kk
K (x, y ) = exp(− kx−y
) and exp(− kx−y
) are char. kernels.
γ2γ

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Reminder : MMD
Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.
Consider, for any probability distribution P on X ,µK (P) = EX ∼P [Φ(x)] .
The kernel K is said to be characteristic ifµK (P) = µK (Q) ⇒ P = Q.
Theorem2kk
K (x, y ) = exp(− kx−y
) and exp(− kx−y
) are char. kernels.
γ2γ

Definition : the MMD distance
DK (P, Q) = kµK (P) − µK (Q)kH .
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

MMD-based estimator

Reminder of the context :1
X1 , . . . , Xn be i.i.d in X from a probability distribution P0 ,2model (Pθ , θ ∈ Θ).

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

MMD-based estimator

Reminder of the context :1
X1 , . . . , Xn be i.i.d in X from a probability distribution P0 ,2model (Pθ , θ ∈ Θ).
Definition - MMD based estimator
n


1X
MMD
θ̂n
= arg min DK Pθ , P̂n where P̂n =δX .
n i=1 iθ∈Θ

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

A bound in expectation

Theorem
For any P0 , when X1 , . . . , Xn are i.i.d from P0 ,hi2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Proof of the theorem : preliminary lemma
Lemma
For any P0 , when X1 , . . . , Xn are i.i.d from P0 ,i
1
0E DK P̂n , P
≤√ .
n
h

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Proof of the theorem : preliminary lemma
Lemma
For any P0 , when X1 , . . . , Xn are i.i.d from P0 ,i
1
0E DK P̂n , P
≤√ .
n
h

n h

io2
h

i
E DK P̂n , P 0≤ E D2K P̂n , P 0X2= E (1/n)(µ(δXi ) − µ(P0 ))Hhi2= (1/n)E kµ(δX1 ) − µ(P0 )kH≤ 1/n.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Proof of the theorem∀θ, DK Pθ̂nMMD , P

0









≤ DK Pθ̂nMMD , P̂n + DK P̂n , P≤ DK Pθ , P̂n + DK P̂n , P 0≤ DK Pθ , P 0 + 2DK P̂n , P 0

Pierre Alquier, RIKEN AIP

0

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Proof of the theorem∀θ, DK Pθ̂nMMD , P

0









≤ DK Pθ̂nMMD , P̂n + DK P̂n , P≤ DK Pθ , P̂n + DK P̂n , P 0≤ DK Pθ , P 0 + 2DK P̂n , P 0

0



h

i
2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

A bound in probability

We can replace the control on the expectation of DK P̂n , P
by a bound that holds with large probability, thanks to
McDiarmid’s inequality.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization0Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

A bound in probability

We can replace the control on the expectation of DK P̂n , P
by a bound that holds with large probability, thanks to
McDiarmid’s inequality.

0

Theorem
For any P0 , when X1 , . . . , Xn are i.i.d from P0 , with probability
at least 1 − δ,q

2 + 2 2 log 1δ
√
DK Pθ̂n , P 0 ≤ inf DK Pθ , P 0 +.
θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

How to compute θ̂nMMD ?
We actually have2
D2K (Pθ , P̂n ) = EX ,X 0 ∼Pθ [K (X , X 0 )] −

n
X

n i=1

+

Pierre Alquier, RIKEN AIP

EX ∼Pθ [K (Xi , X )]1 X
K (Xi , Xj )n2 1≤i,j≤nParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

How to compute θ̂nMMD ?
We actually have2
D2K (Pθ , P̂n ) = EX ,X 0 ∼Pθ [K (X , X 0 )] −

n
X

n i=1

+

and soEX ∼Pθ [K (Xi , X )]1 X
K (Xi , Xj )n2 1≤i,j≤n∇θ D2K (Pθ , P̂n )("= 2EX ,X 0 ∼Pθ

#
)
n
X
1
K (X , X 0 ) −K (Xi , X ) ∇θ [log pθ (X )]n i=1that can be approximated by sampling from Pθ .
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Short bibliography
Dziugaite, G. K., Roy, D. M., & Ghahramani, Z. (2015). Training generative neural networks via
maximum mean discrepancy optimization. UAI 2015.

define the estimator and used it to train GANs.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Short bibliography
Dziugaite, G. K., Roy, D. M., & Ghahramani, Z. (2015). Training generative neural networks via
maximum mean discrepancy optimization. UAI 2015.

define the estimator and used it to train GANs.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Short bibliography
Dziugaite, G. K., Roy, D. M., & Ghahramani, Z. (2015). Training generative neural networks via
maximum mean discrepancy optimization. UAI 2015.

define the estimator and used it to train GANs.

Briol, F. X., Barp, A., Duncan, A. B., & Girolami, M. (2019). Statistical Inference for Generative
Models with Maximum Mean Discrepancy. Preprint arXiv :1906.05944.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Definition of the estimator
Basic properties
References and further works

Short bibliography
Dziugaite, G. K., Roy, D. M., & Ghahramani, Z. (2015). Training generative neural networks via
maximum mean discrepancy optimization. UAI 2015.

define the estimator and used it to train GANs.

Briol, F. X., Barp, A., Duncan, A. B., & Girolami, M. (2019). Statistical Inference for Generative
Models with Maximum Mean Discrepancy. Preprint arXiv :1906.05944.

provided the first theoretical study : asymptotic distribution.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

1

Estimation via MMD optimization
Definition of the estimator
Basic properties
References and further works

2

Robustness to outliers and to dependence
Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

3

Further topics
Semi-parametric models
CopulasMMD-Bayes

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Huber contamination model

In this section, I will present the results of the following
preprint.
Chérief-Abdellatif, B.-E. and Alquier, P. (2019). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Preprint arxiv :1912.05737.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Huber contamination model

Reminderhi2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Huber contamination model

Reminderhi2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈Θn
Huber contamination model : P0 = (1 − ε)Pθ0 + εQ.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Huber contamination model
Reminderhi2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈Θn
Huber contamination model : P0 = (1 − ε)Pθ0 + εQ.
DK (Pθ0 , P0 ) = kPθ0 − [(1 − ε)Pθ0 + εQ]kH≤ εkPθ0 kH + εkQkH= 2ε.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Huber contamination model
Reminderhi2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈Θn
Huber contamination model : P0 = (1 − ε)Pθ0 + εQ.
DK (Pθ0 , P0 ) ≤ 2ε.
Corollary
When X1 , . . . , Xn are i.i.d from (1 − ε)Pθ0 + εQ,hi2
E DK Pθ̂nMMD , Pθ0 ≤ 4ε + √ .
nPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Example : Gaussian mean estimation
Example : the model is given by pθ = N (θ, σ 2 I ) for θ ∈ Rd .

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Example : Gaussian mean estimation
Example : the model is given by pθ = N (θ, σ 2 I ) for θ ∈ Rd .
Using a Gaussian kernel K (x, y ) = exp(−kx − y 2 k/γ 2 ), fromthe previous theorem and from the equality
 d2 2γkθ − θ0 k2
2
DK (Pθ , Pθ0 ) = 2
1 − exp − 2
4σ 2 + γ 2
4σ + γ 2
we obtainhi
MMD2− θ0 kE kθ̂n"≤ −(4σ 2 + γ 2 ) log 1 − 4
Pierre Alquier, RIKEN AIP



1
+ ε2
n



4σ 2 + γ 2
γ2

 d2 #

Parametric estimation via MMD optimization.

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Example : Gaussian mean estimation
Example : the model is given by pθ = N (θ, σ 2 I ) for θ ∈ Rd .
Using a Gaussian kernel K (x, y ) = exp(−kx − y 2 k/γ 2 ), fromthe previous theorem and from the equality
 d2 2γkθ − θ0 k2
2
DK (Pθ , Pθ0 ) = 2
1 − exp − 2
4σ 2 + γ 2
4σ + γ 2
we obtainhi
MMD2− θ0 k take γ = 2dσ 2E kθ̂n
"

 2
d #
2 2
1
4σ
+
γ
≤ −(4σ 2 + γ 2 ) log 1 − 4
+ ε2
.
n
γ2
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Example : Gaussian mean estimation
Example : the model is given by pθ = N (θ, σ 2 I ) for θ ∈ Rd .
Using a Gaussian kernel K (x, y ) = exp(−kx − y 2 k/γ 2 ), fromthe previous theorem and from the equality
D2K (Pθ , Pθ0 ) = 2

γ2
4σ 2 + γ 2

 d2 kθ − θ0 k2
1 − exp − 2
4σ + γ 2

we obtainhEkθ̂nMMD − θ0 k2

Pierre Alquier, RIKEN AIP

i

. dσ

2




1
2
+ε .
n

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Example : Gaussian mean estimation, simulations
Model : N (θ, 1), and X1 , . . . , Xn i.i.d N (θ0 , 1), n = 100 and
we repeat the experiment 200 times.

mean absolute error

Pierre Alquier, RIKEN AIPθ̂nMLE θ̂nMMD
0.0722 0.0838

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Example : Gaussian mean estimation, simulations
Model : N (θ, 1), and X1 , . . . , Xn i.i.d N (θ0 , 1), n = 100 and
we repeat the experiment 200 times.

mean absolute errorθ̂nMLE θ̂nMMD
0.0722 0.0838

Now, ε = 2% of the observations drawn from a Cauchy.
mean absolute error

Pierre Alquier, RIKEN AIP

0.2349 0.0953

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Example : Gaussian mean estimation, simulations
Model : N (θ, 1), and X1 , . . . , Xn i.i.d N (θ0 , 1), n = 100 and
we repeat the experiment 200 times.

mean absolute errorθ̂nMLE θ̂nMMD
0.0722 0.0838

Now, ε = 2% of the observations drawn from a Cauchy.
mean absolute error

0.2349 0.0953

Now, ε = 1% are replaced by 1, 000.
mean absolute error
Pierre Alquier, RIKEN AIP

10.018 0.0903
Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

And now, non-independent observations
Lemma
When X1 , . . . , Xn are identically distributed from P0 ,hi0E DK P̂n , P≤?

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

And now, non-independent observations
Lemma
When X1 , . . . , Xn are identically distributed from P0 ,hi0E DK P̂n , P
≤?

i
h
E D2K P̂n , P 0X2= E (1/n)(µ(δXi ) − µ(P0 ))
H

=

1
2
+ 2
E µ(δXi ) − µ(P0 ), µ(δXj ) − µ(P0 ) H
n n 1≤i<j≤nX

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Mesure of dependence via covariance in H
Definition
When (X1 , . . . , Xn , . . . ) is a stationary process with marginal
distribution P0 , we put :%h = E µ(δXt+h ) − µ(P0 ), µ(δXt ) − µ(P0 ) H .

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Mesure of dependence via covariance in H
Definition
When (X1 , . . . , Xn , . . . ) is a stationary process with marginal
distribution P0 , we put :%h = E µ(δXt+h ) − µ(P0 ), µ(δXt ) − µ(P0 ) H .
Lemma - dependent case
When X1 , . . . , Xn are identically distributed from P0 ,
#
"
n
h

i 1
X
0
E DK P̂n , P
≤
1+
%h
n
h=1
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Mesure of dependence via covariance in H
Theorem - dependent case
When (X1 , . . . , Xn , . . . ) is a stationary process with marginal
distribution P0Phi
2 + 2 nh=1 %h√.
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) +θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Mesure of dependence via covariance in H
Theorem - dependent case
When (X1 , . . . , Xn , . . . ) is a stationary process with marginal
distribution P0Phi
2 + 2 nh=1 %h√.
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) +
θ∈Θ
n
P
1
assume that ∞h=1 %h = Σ < +∞ thenhi
2 + 2Σ
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

Mesure of dependence via covariance in H
Theorem - dependent case
When (X1 , . . . , Xn , . . . ) is a stationary process with marginal
distribution P0Phi
2 + 2 nh=1 %h√.
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) +
θ∈Θ
n
P
1
assume that ∞h=1 %h = Σ < +∞ thenhi
2 + 2Σ
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈Θn2

we also have a bound in probability, based on Rio’s versionof Hoeffding’s inequality ; it requires more assumptions.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

An example : auto-regressive processes
Proposition
Assume that Xt takes values in Rd and that
K (x, y ) = F (kx − y k) where F is an L-Lipschitz function.
Assume that
Xt+1 = AXt + εt+1
where the (εt ) are i.i.d with Ekε0 k < ∞, and A is a matrix
with kAk = supkxk=1 kAxk < 1.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

An example : auto-regressive processes
Proposition
Assume that Xt takes values in Rd and that
K (x, y ) = F (kx − y k) where F is an L-Lipschitz function.
Assume that
Xt+1 = AXt + εt+1
where the (εt ) are i.i.d with Ekε0 k < ∞, and A is a matrix
with kAk = supkxk=1 kAxk < 1.
Thent 2LEkε0 k%t ≤ kAk

1 − kAkand Σ =

Pierre Alquier, RIKEN AIP

∞
X
t=1

%t =

2kAkLEkε0 k.
(1 − kAk)2

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

A non-mixing process with Σ < +∞Example : consider X0 ∼ U([0, 1]), ηt i.i.d Be(1/2) and
Xt+1 =

Pierre Alquier, RIKEN AIP

Xt + ηt+1.
2

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

A non-mixing process with Σ < +∞Example : consider X0 ∼ U([0, 1]), ηt i.i.d Be(1/2) and
Xt + ηt+1.
2
It satisfies the assumptions of the previous proposition, we
have %t ≤ L/2t and Σ = 2L.
Xt+1 =

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

A non-mixing process with Σ < +∞Example : consider X0 ∼ U([0, 1]), ηt i.i.d Be(1/2) and
Xt + ηt+1.
2
It satisfies the assumptions of the previous proposition, we
have %t ≤ L/2t and Σ = 2L.
Xt+1 =

Note however that this process is known to be non-mixing.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

A non-mixing process with Σ < +∞Example : consider X0 ∼ U([0, 1]), ηt i.i.d Be(1/2) and
Xt + ηt+1.
2
It satisfies the assumptions of the previous proposition, we
have %t ≤ L/2t and Σ = 2L.
Xt+1 =

Note however that this process is known to be non-mixing.
More generally, we prove the following result :Proposition
Under some (non-restrictive) assumption on the kernel K ,%t ≤ cK .βt (the β-mixing coef.)
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

1

Estimation via MMD optimization
Definition of the estimator
Basic properties
References and further works

2

Robustness to outliers and to dependence
Robustness to outliers and to dependence
Example : estimation of the mean of a Gaussian
Robustness to dependence

3

Further topics
Semi-parametric models
CopulasMMD-Bayes

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Problem with semi-parametric models
The method, as explained so far, must describe completely the
distribution of the observations.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Problem with semi-parametric models
The method, as explained so far, must describe completely the
distribution of the observations. But this is often not the casein practice !

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Problem with semi-parametric models
The method, as explained so far, must describe completely the
distribution of the observations. But this is often not the casein practice ! For example
in regression, we observe (X , Y ) and we only want tomodelize PY |X , but not PX .

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Problem with semi-parametric models
The method, as explained so far, must describe completely the
distribution of the observations. But this is often not the casein practice ! For example
in regression, we observe (X , Y ) and we only want tomodelize PY |X , but not PX .
in copulas, we observe (V1 , V2 ) and we want to modelize
their copula, that is the c.d.f of (FV1 (V1 ), FV2 (V2 )) on[0, 1]2 but we are not interested in FV1 nor FV2 .

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Problem with semi-parametric models
The method, as explained so far, must describe completely the
distribution of the observations. But this is often not the casein practice ! For example
in regression, we observe (X , Y ) and we only want tomodelize PY |X , but not PX .
in copulas, we observe (V1 , V2 ) and we want to modelize
their copula, that is the c.d.f of (FV1 (V1 ), FV2 (V2 )) on[0, 1]2 but we are not interested in FV1 nor FV2 .
Alquier, P. and Gerber, M. (2020). Universal Robust Regression via Maximum Mean Discrepancy.
Preprint arxiv :2006.00840.
Alquier, P., Chérief-Abdellatif, B.-E., Derumigny, A. and Fermanian, J.-D. (2020). Estimation of
copulas via Maximum Mean Discrepancy. Preprint arXiv : 2010.00408.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Regression with MMD
0
0
0
P(X
,Y ) = PY |X PX

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Regression with MMD
0
0
0
P(X
,Y ) = PY |X PX
0

PY |X estimated by Qg (µ̂,X )T

Ex : Qg (µ,X ) = N (µ X , 1).

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Regression with MMD
0
0
0
P(X
,Y ) = PY |X PX
0

PY |X estimated by Qg (µ̂,X )0T

Ex : Qg (µ,X ) = N (µ X , 1).

Pierre Alquier, RIKEN AIP

PX estimated by P̂X =

n
1 X

n i=1

δX .
i

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Regression with MMD
0
0
0
P(X
,Y ) = PY |X PX
0

PY |X estimated by Qg (µ̂,X )0T

Ex : Qg (µ,X ) = N (µ X , 1).

PX estimated by P̂X =

n
1 X

n i=1

δX .
i

Product kernel : K ((x, y ), (x 0 , y 0 )) = KX (x, x 0 )KY (y , y 0 ).

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Regression with MMD
0
0
0
P(X
,Y ) = PY |X PX
0

PY |X estimated by Qg (µ̂,X )0T

Ex : Qg (µ,X ) = N (µ X , 1).

PX estimated by P̂X =

n
1 X

n i=1

δX .
i

Product kernel : K ((x, y ), (x 0 , y 0 )) = KX (x, x 0 )KY (y , y 0 ).
DK (Qg (µ̂,X ) P̂X , PY0 |X P̂X ) ≤ . . . straightforward !

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Regression with MMD
0
0
0
P(X
,Y ) = PY |X PX
0

PY |X estimated by Qg (µ̂,X )0T

Ex : Qg (µ,X ) = N (µ X , 1).

PX estimated by P̂X =

n
1 X

n i=1

δX .
i

Product kernel : K ((x, y ), (x 0 , y 0 )) = KX (x, x 0 )KY (y , y 0 ).
DK (Qg (µ̂,X ) P̂X , PY0 |X P̂X ) ≤ . . . straightforward !
DK (Qg (µ̂,X ) PX0 , PY0 |X PX0 ) ≤ . . . quite difficult.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Regression with MMD
0
0
0
P(X
,Y ) = PY |X PX
0

PY |X estimated by Qg (µ̂,X )0T

Ex : Qg (µ,X ) = N (µ X , 1).

PX estimated by P̂X =

n
1 X

n i=1

δX .
i

Product kernel : K ((x, y ), (x 0 , y 0 )) = KX (x, x 0 )KY (y , y 0 ).
DK (Qg (µ̂,X ) P̂X , PY0 |X P̂X ) ≤ . . . straightforward !
DK (Qg (µ̂,X ) PX0 , PY0 |X PX0 ) ≤ . . . quite difficult.
Requires : f (x, y ) ∈ HK ⇒ EY ∼Qg (θ,x) [f (x, Y )] ∈ HKX .
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Estimation of copulas with MMD
Estimation of copulas : we observe (V1,i , V2,i )1≤i≤n .
estimate FV1 by the empirical c.d.f Fˆ1 ,estimate FV2 by the empirical c.d.f Fˆ2 ,standard MMD procedure with kernel KU on(U1,i , U2,i ) = (Fˆ1 (V1,i ), Fˆ2 (V2,i )).

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Estimation of copulas with MMD
Estimation of copulas : we observe (V1,i , V2,i )1≤i≤n .
estimate FV1 by the empirical c.d.f Fˆ1 ,estimate FV2 by the empirical c.d.f Fˆ2 ,standard MMD procedure with kernel KU on(U1,i , U2,i ) = (Fˆ1 (V1,i ), Fˆ2 (V2,i )).
TheoremWith probability larger than 1 − δ − ν ∈ (0, 1),s  #r "8
1
DKU (Pθ̂n , P0 ) ≤ inf DKU (Pθ , P0 ) +1 + log
θ∈Θ
n
δ
s
 
8 (2)
2
+
k∂ KU k∞ log.
nν
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Estimation of copulas with MMD
TheoremWith probability larger than 1 − δ − ν ∈ (0, 1),s  #r "8
1
1 + log
DKU (Pθ̂n , P0 ) ≤ inf DKU (Pθ , P0 ) +
θ∈Θ
n
δ
s
 
8 (2)
2
k∂ KU k∞ log.
+nν
The paper also :studies the asymptotic normality of θ̂n under someassumptions on the model.
introduces the R package MMDCopula
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

The package MMDCopula

Uses a stochastic gradient algorithm to compute the MMD
estimator of the parameter(s) :1of the main copulas models : Gaussian, Frank, Clayton,
Gumbel, Student, etc.
2
using various kernels : Gaussian, Laplace, etc.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Example : Gaussian copulas

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Example : other models

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

MMD-Bayes

We also studied a “pseudo-Bayesian” version of the estimator :p(θ|X1 , . . . , Xn ) ∝ exp −βD2K Pθ , P̂n π(θ).

Chérief-Abdellatif, B.-E. and Alquier, P. (2020). MMD-Bayes : Robust Bayesian Estimation via
Maximum Mean Discrepancy. Proceedings of AABI.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers and to dependence
Further topics

Semi-parametric models
CopulasMMD-Bayes

Thank you !

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization