Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Parametric estimation via MMD
optimization: robustness to outliers and todependence
Pierre Alquier

CDT in Modern Statistics and Statistical Machine Learning
Imperial College London and University of Oxford
August 18, 2020
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

RIKEN AIP : ABI teamApproximate BayesianInference team (ABI), leadby Emtiyaz Khan
Please visit the team website
https ://emtiyaz.github.io/

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Co-authors

Badr-Eddine Chérief-Abdellatif

Alexis Derumigny

ENSAE Paris → Oxford

Univ. of Twente

Mathieu GerberJean-David FermanianUniv. of Bristol

ENSAE Paris

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

The Maximum Likelihood Estimator (MLE)
Let X1 , . . . , Xn be i.i.d in X from a probability distribution P0 .

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

The Maximum Likelihood Estimator (MLE)
Let X1 , . . . , Xn be i.i.d in X from a probability distribution P0 .
Statistical inference :propose a model (Pθ , θ ∈ Θ), assume P0 = Pθ0 .
compute θ̂n = θ̂n (X1 , . . . , Xn ).

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

The Maximum Likelihood Estimator (MLE)
Let X1 , . . . , Xn be i.i.d in X from a probability distribution P0 .
Statistical inference :propose a model (Pθ , θ ∈ Θ), assume P0 = Pθ0 .
compute θ̂n = θ̂n (X1 , . . . , Xn ).
Letting pθ denote the density of Pθ , thenθ̂nMLE = arg max L(θ), where L(θ) =θ∈ΘPierre Alquier, RIKEN AIPnYpθ (Xi ).

i=1

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

The Maximum Likelihood Estimator (MLE)
Let X1 , . . . , Xn be i.i.d in X from a probability distribution P0 .
Statistical inference :propose a model (Pθ , θ ∈ Θ), assume P0 = Pθ0 .
compute θ̂n = θ̂n (X1 , . . . , Xn ).
Letting pθ denote the density of Pθ , thenθ̂nMLE = arg max L(θ), where L(θ) =
θ∈Θ

n
Y

pθ (Xi ).

i=1

2

Example : P(m,σ) = N (m, σ ) then
n
n
1X
1X
2
m̂ =
Xi and σ̂ =(Xi − m̂)2 .
n i=1
n i=1
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

MLE not unique / not consistent
Example :pθ (x) =exp(−|x − θ|)p,
2 π|x − θ|

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

MLE not unique / not consistent
Example :pθ (x) =exp(−|x − θ|)p,
2 π|x − θ|P
exp (− ni=1 |Xi − θ|).
L(θ) = √Q p(2 π)n ni=1 |Xi − θ|

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

MLE fails in the presence of outliers
What is an outlier ?

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

MLE fails in the presence of outliers
What is an outlier ?
Huber proposed the contamination model : with probability ε,
Xi is not drawn from Pθ0 but from Q that can be anything :P0 = (1 − ε)Pθ0 + εQ.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

MLE fails in the presence of outliers
What is an outlier ?
Huber proposed the contamination model : with probability ε,
Xi is not drawn from Pθ0 but from Q that can be anything :P0 = (1 − ε)Pθ0 + εQ.
Example : Pθ = Unif [0, θ], then

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

MLE fails in the presence of outliers
What is an outlier ?
Huber proposed the contamination model : with probability ε,
Xi is not drawn from Pθ0 but from Q that can be anything :P0 = (1 − ε)Pθ0 + εQ.
Example : Pθ = Unif [0, θ], thenn

1 Y
L(θ) = n1{0≤Xi ≤θ} ⇒ θ̂ = max Xi .
1≤i≤n
θ i=1

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

MLE fails in the presence of outliers
What is an outlier ?
Huber proposed the contamination model : with probability ε,
Xi is not drawn from Pθ0 but from Q that can be anything :P0 = (1 − ε)Pθ0 + εQ.
Example : Pθ = Unif [0, θ], thenn

1 Y
L(θ) = n1{0≤Xi ≤θ} ⇒ θ̂ = max Xi .
1≤i≤n
θ i=1
In the case of the following contamination, the MLE isextremely far from the truth :P0 = (1 − ε).Unif [0, 1] + ε.N (1010 , 1)...
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Requirements for a “good” estimator
A universal estimator θ̂n must be such that, for some distanced on probability distributions,

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Requirements for a “good” estimator
A universal estimator θ̂n must be such that, for some distanced on probability distributions,1
when the model is well specified, that is, P0 = Pθ0 ,E d(Pθ̂n , P0 ) ≤ rn (Θ) −−−→ 0.
n→∞Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Requirements for a “good” estimator
A universal estimator θ̂n must be such that, for some distanced on probability distributions,1
when the model is well specified, that is, P0 = Pθ0 ,E d(Pθ̂n , P0 ) ≤ rn (Θ) −−−→ 0.
n→∞2

in the misspecified case P0 = (1 − ε)Pθ0 + εQ, for any Q,
E d(Pθ̂n , P0 ) ≤ d(P0 , Pθ0 ) + rn (Θ) .
| {z } | {z }
−ε→∞
−−→0
−n→∞
−−→0

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Requirements for a “good” estimator
A universal estimator θ̂n must be such that, for some distanced on probability distributions,1
when the model is well specified, that is, P0 = Pθ0 ,E d(Pθ̂n , P0 ) ≤ rn (Θ) −−−→ 0.
n→∞2

in the misspecified case P0 = (1 − ε)Pθ0 + εQ, for any Q,
E d(Pθ̂n , P0 ) ≤ d(P0 , Pθ0 ) + rn (Θ) .
| {z } | {z }
−ε→∞
−−→0
−n→∞
−−→0

The MLE does not satisfy these requirements.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Some examples
Yatracos’ skeleton estimate θ̂nY :hi
E dTV (Pθ̂nY , P0 ) ≤ 3dTV (P0 , Pθ0 ) + C .

rdim(Θ)nwheredTV (P, Q) = sup |P(E ) − Q(E )|.
E

Yatracos, Y. G. (1985). Rates of convergence of minimum distance estimators and Kolmogorov’s
entropy. Annals of Statistics.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Some examples
Yatracos’ skeleton estimate θ̂nY :hi
E dTV (Pθ̂nY , P0 ) ≤ 3dTV (P0 , Pθ0 ) + C .

rdim(Θ)nwheredTV (P, Q) = sup |P(E ) − Q(E )|.
E

Yatracos, Y. G. (1985). Rates of convergence of minimum distance estimators and Kolmogorov’s
entropy. Annals of Statistics.

More recent work with the Hellinger distance :Baraud, Y., Birgé, L., & Sart, M. (2017). A new method for estimation and model selection :ρ-estimation. Inventiones mathematicae.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

But...

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

But...

Problem with the aforementioned estimators : they cannot becomputed in practice.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

But...

Problem with the aforementioned estimators : they cannot becomputed in practice.

Additional requirement : an estimator must be computable ! ! !

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Overview of the talk1

Estimation via MMD optimization
Definition of the estimator
Basic properties
References and further works

2

Robustness to outliers
Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

3

Robustness to dependence
Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

1

Estimation via MMD optimization
Definition of the estimator
Basic properties
References and further works

2

Robustness to outliers
Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

3

Robustness to dependence
Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Reminder : kernels
Let H be a Hilbert space and any continuous function
Φ : X → H. The function
K (x, y ) = hΦ(x), Φ(y )iHis called a kernel.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Reminder : kernels
Let H be a Hilbert space and any continuous function
Φ : X → H. The function
K (x, y ) = hΦ(x), Φ(y )iHis called a kernel. Conversely :Mercer’s theorem
Let K (x, y ) be a continuous function such that for any(x1 , . . . , xn ) ∈ X n and (c1 , . . . , cn ) 6= (0, . . . , 0) ∈ Rn ,n XnXci cj K (xi , xj ) > 0,i=1 j=1

then there is H and Φ such that K (x, y ) = hΦ(x), Φ(y )iH .
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Reminder : MMD
Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Reminder : MMD
Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.
Consider, for any probability distribution P on X ,µK (P) = EX ∼P [Φ(x)] .

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Reminder : MMD
Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.
Consider, for any probability distribution P on X ,µK (P) = EX ∼P [Φ(x)] .
The kernel K is said to be characteristic ifµK (P) = µK (Q) ⇒ P = Q.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Reminder : MMD
Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.
Consider, for any probability distribution P on X ,µK (P) = EX ∼P [Φ(x)] .
The kernel K is said to be characteristic ifµK (P) = µK (Q) ⇒ P = Q.
Theorem2kk
K (x, y ) = exp(− kx−y
) and exp(− kx−y
) are char. kernels.
γ2γ

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Reminder : MMD
Assume that the kernel is bounded : 0 ≤ K (x, y ) ≤ 1.
Consider, for any probability distribution P on X ,µK (P) = EX ∼P [Φ(x)] .
The kernel K is said to be characteristic ifµK (P) = µK (Q) ⇒ P = Q.
Theorem2kk
K (x, y ) = exp(− kx−y
) and exp(− kx−y
) are char. kernels.
γ2γ

Definition : the MMD distance
DK (P, Q) = kµK (P) − µK (Q)kH .
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

MMD-based estimator

Reminder of the context :1
X1 , . . . , Xn be i.i.d in X from a probability distribution P0 ,2model (Pθ , θ ∈ Θ).

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

MMD-based estimator

Reminder of the context :1
X1 , . . . , Xn be i.i.d in X from a probability distribution P0 ,2model (Pθ , θ ∈ Θ).
Definition - MMD based estimator
n


1X
MMD
θ̂n
= arg min DK Pθ , P̂n where P̂n =δX .
n i=1 iθ∈Θ

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

A preliminary lemma
Lemma
For any P0 , when X1 , . . . , Xn are i.i.d from P0 ,i
1
0E DK P̂n , P
≤√ .
n
h

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

A preliminary lemma
Lemma
For any P0 , when X1 , . . . , Xn are i.i.d from P0 ,i
1
0E DK P̂n , P
≤√ .
n
h

n h

io2
h

i
E DK P̂n , P 0≤ E D2K P̂n , P 0X2= E (1/n)(µ(δXi ) − µ(P0 ))Hhi2= (1/n)E kµ(δX1 ) − µ(P0 )kH≤ 1/n.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

A bound in expectation






∀θ, DK Pθ̂nMMD , P 0 ≤ DK Pθ̂nMMD , P̂n + DK P̂n , P 0≤ DK Pθ , P̂n + DK P̂n , P 0



0
0
≤ DK Pθ , P + 2DK P̂n , P

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

A bound in expectation






∀θ, DK Pθ̂nMMD , P 0 ≤ DK Pθ̂nMMD , P̂n + DK P̂n , P 0≤ DK Pθ , P̂n + DK P̂n , P 0



0
0
≤ DK Pθ , P + 2DK P̂n , P
Theorem
For any P0 , when X1 , . . . , Xn are i.i.d from P0 ,hi2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

A bound in probability

We can replace the control on the expectation of DK P̂n , P
by a bound that holds with large probability, thanks to
McDiarmid’s inequality.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization0Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

A bound in probability

We can replace the control on the expectation of DK P̂n , P
by a bound that holds with large probability, thanks to
McDiarmid’s inequality.

0

Theorem
For any P0 , when X1 , . . . , Xn are i.i.d from P0 , with probability
at least 1 − δ,q

2 + 2 2 log 1δ
√
DK Pθ̂n , P 0 ≤ inf DK Pθ , P 0 +.
θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

How to compute θ̂nMMD ?
We actually have2
D2K (Pθ , P̂n ) = EX ,X 0 ∼Pθ [K (X , X 0 )] −

n
X

n i=1

+

Pierre Alquier, RIKEN AIP

EX ∼Pθ [K (Xi , X )]1 X
K (Xi , Xj )n2 1≤i,j≤nParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

How to compute θ̂nMMD ?
We actually have2
D2K (Pθ , P̂n ) = EX ,X 0 ∼Pθ [K (X , X 0 )] −

n
X

n i=1

+

and soEX ∼Pθ [K (Xi , X )]1 X
K (Xi , Xj )n2 1≤i,j≤n∇θ D2K (Pθ , P̂n )("= 2EX ,X 0 ∼Pθ

#
)
n
X
1
K (X , X 0 ) −K (Xi , X ) ∇θ [log pθ (X )]n i=1that can be approximated by sampling from Pθ .
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Short bibliography
Dziugaite, G. K., Roy, D. M., & Ghahramani, Z. (2015). Training generative neural networks via
maximum mean discrepancy optimization. UAI 2015.

define the estimator and used it to train GANs.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Short bibliography
Dziugaite, G. K., Roy, D. M., & Ghahramani, Z. (2015). Training generative neural networks via
maximum mean discrepancy optimization. UAI 2015.

define the estimator and used it to train GANs.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Short bibliography
Dziugaite, G. K., Roy, D. M., & Ghahramani, Z. (2015). Training generative neural networks via
maximum mean discrepancy optimization. UAI 2015.

define the estimator and used it to train GANs.

Briol, F. X., Barp, A., Duncan, A. B., & Girolami, M. (2019). Statistical Inference for Generative
Models with Maximum Mean Discrepancy. Preprint arXiv :1906.05944.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Short bibliography
Dziugaite, G. K., Roy, D. M., & Ghahramani, Z. (2015). Training generative neural networks via
maximum mean discrepancy optimization. UAI 2015.

define the estimator and used it to train GANs.

Briol, F. X., Barp, A., Duncan, A. B., & Girolami, M. (2019). Statistical Inference for Generative
Models with Maximum Mean Discrepancy. Preprint arXiv :1906.05944.

provided the first theoretical study : asymptotic distribution.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Chérief-Abdellatif, B.-E. and Alquier, P. (2019). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Preprint arxiv :1912.05737.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Chérief-Abdellatif, B.-E. and Alquier, P. (2019). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Preprint arxiv :1912.05737.

the results I will present in the following sections.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Chérief-Abdellatif, B.-E. and Alquier, P. (2019). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Preprint arxiv :1912.05737.

the results I will present in the following sections.
Chérie-Abdellatif, B.-E. and Alquier, P. (2020). MMD-Bayes : Robust Bayesian Estimation via
Maximum Mean Discrepancy. Proceedings of AABI.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Chérief-Abdellatif, B.-E. and Alquier, P. (2019). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Preprint arxiv :1912.05737.

the results I will present in the following sections.
Chérie-Abdellatif, B.-E. and Alquier, P. (2020). MMD-Bayes : Robust Bayesian Estimation via
Maximum Mean Discrepancy. Proceedings of AABI.

studies a Bayesian version of this estimator.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Chérief-Abdellatif, B.-E. and Alquier, P. (2019). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Preprint arxiv :1912.05737.

the results I will present in the following sections.
Chérie-Abdellatif, B.-E. and Alquier, P. (2020). MMD-Bayes : Robust Bayesian Estimation via
Maximum Mean Discrepancy. Proceedings of AABI.

studies a Bayesian version of this estimator.
Alquier, P. and Gerber, M. (2020). Universal Robust Regression via Maximum Mean Discrepancy.
Preprint arxiv :2006.00840.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Chérief-Abdellatif, B.-E. and Alquier, P. (2019). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Preprint arxiv :1912.05737.

the results I will present in the following sections.
Chérie-Abdellatif, B.-E. and Alquier, P. (2020). MMD-Bayes : Robust Bayesian Estimation via
Maximum Mean Discrepancy. Proceedings of AABI.

studies a Bayesian version of this estimator.
Alquier, P. and Gerber, M. (2020). Universal Robust Regression via Maximum Mean Discrepancy.
Preprint arxiv :2006.00840.

appication to (linear, logistic, Poisson) regression.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Chérief-Abdellatif, B.-E. and Alquier, P. (2019). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Preprint arxiv :1912.05737.

the results I will present in the following sections.
Chérie-Abdellatif, B.-E. and Alquier, P. (2020). MMD-Bayes : Robust Bayesian Estimation via
Maximum Mean Discrepancy. Proceedings of AABI.

studies a Bayesian version of this estimator.
Alquier, P. and Gerber, M. (2020). Universal Robust Regression via Maximum Mean Discrepancy.
Preprint arxiv :2006.00840.

appication to (linear, logistic, Poisson) regression.
Alquier, P., Chérief-Abdellatif, B.-E., Derumigny, A. and Fermanian, J.-D. (2020). Estimation of
copulas via Maximum Mean Discrepancy. Coming soon !

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Definition of the estimator
Basic properties
References and further works

Chérief-Abdellatif, B.-E. and Alquier, P. (2019). Finite Sample Properties of Parametric MMD
Estimation : Robustness to Misspecification and Dependence. Preprint arxiv :1912.05737.

the results I will present in the following sections.
Chérie-Abdellatif, B.-E. and Alquier, P. (2020). MMD-Bayes : Robust Bayesian Estimation via
Maximum Mean Discrepancy. Proceedings of AABI.

studies a Bayesian version of this estimator.
Alquier, P. and Gerber, M. (2020). Universal Robust Regression via Maximum Mean Discrepancy.
Preprint arxiv :2006.00840.

appication to (linear, logistic, Poisson) regression.
Alquier, P., Chérief-Abdellatif, B.-E., Derumigny, A. and Fermanian, J.-D. (2020). Estimation of
copulas via Maximum Mean Discrepancy. Coming soon !

application to the estimation of copulas.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

1

Estimation via MMD optimization
Definition of the estimator
Basic properties
References and further works

2

Robustness to outliers
Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

3

Robustness to dependence
Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Huber contamination model

Reminderhi2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Huber contamination model

Reminderhi2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈Θn
Huber contamination model : P0 = (1 − ε)Pθ0 + εQ.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Huber contamination model
Reminderhi2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈Θn
Huber contamination model : P0 = (1 − ε)Pθ0 + εQ.
DK (Pθ0 , P0 ) = kPθ0 − [(1 − ε)Pθ0 + εQ]kH≤ εkPθ0 kH + εkQkH= 2ε.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Huber contamination model
Reminderhi2
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈Θn
Huber contamination model : P0 = (1 − ε)Pθ0 + εQ.
DK (Pθ0 , P0 ) ≤ 2ε.
Corollary
When X1 , . . . , Xn are i.i.d from (1 − ε)Pθ0 + εQ,hi2
E DK Pθ̂nMMD , Pθ0 ≤ 4ε + √ .
nPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Example : Gaussian mean estimation
Example : the model is given by pθ = N (θ, σ 2 I ) for θ ∈ Rd .

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Example : Gaussian mean estimation
Example : the model is given by pθ = N (θ, σ 2 I ) for θ ∈ Rd .
Using a Gaussian kernel K (x, y ) = exp(−kx − y 2 k/γ 2 ), fromthe previous theorem and from the equality
 d2 2γkθ − θ0 k2
2
DK (Pθ , Pθ0 ) = 2
1 − exp − 2
4σ 2 + γ 2
4σ + γ 2
we obtainhi
MMD2− θ0 kE kθ̂n"≤ −(4σ 2 + γ 2 ) log 1 − 4
Pierre Alquier, RIKEN AIP



1
+ ε2
n



4σ 2 + γ 2
γ2

 d2 #

Parametric estimation via MMD optimization.

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Example : Gaussian mean estimation
Example : the model is given by pθ = N (θ, σ 2 I ) for θ ∈ Rd .
Using a Gaussian kernel K (x, y ) = exp(−kx − y 2 k/γ 2 ), fromthe previous theorem and from the equality
 d2 2γkθ − θ0 k2
2
DK (Pθ , Pθ0 ) = 2
1 − exp − 2
4σ 2 + γ 2
4σ + γ 2
we obtainhi
MMD2− θ0 k take γ = 2dσ 2E kθ̂n
"

 2
d #
2 2
1
4σ
+
γ
≤ −(4σ 2 + γ 2 ) log 1 − 4
+ ε2
.
n
γ2
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Example : Gaussian mean estimation
Example : the model is given by pθ = N (θ, σ 2 I ) for θ ∈ Rd .
Using a Gaussian kernel K (x, y ) = exp(−kx − y 2 k/γ 2 ), fromthe previous theorem and from the equality
D2K (Pθ , Pθ0 ) = 2

γ2
4σ 2 + γ 2

 d2 kθ − θ0 k2
1 − exp − 2
4σ + γ 2

we obtainhEkθ̂nMMD − θ0 k2

Pierre Alquier, RIKEN AIP

i

. dσ

2




1
2
+ε .
n

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Example : Gaussian mean estimation, simulations
Model : N (θ, 1), and X1 , . . . , Xn i.i.d N (θ0 , 1), n = 100 and
we repeat the experiment 200 times.

mean absolute error

Pierre Alquier, RIKEN AIPθ̂nMLE θ̂nMMD
0.0722 0.0838

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Example : Gaussian mean estimation, simulations
Model : N (θ, 1), and X1 , . . . , Xn i.i.d N (θ0 , 1), n = 100 and
we repeat the experiment 200 times.

mean absolute errorθ̂nMLE θ̂nMMD
0.0722 0.0838

Now, ε = 2% of the observations drawn from a Cauchy.
mean absolute error

Pierre Alquier, RIKEN AIP

0.2349 0.0953

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

Example : Gaussian mean estimation, simulations
Model : N (θ, 1), and X1 , . . . , Xn i.i.d N (θ0 , 1), n = 100 and
we repeat the experiment 200 times.

mean absolute errorθ̂nMLE θ̂nMMD
0.0722 0.0838

Now, ε = 2% of the observations drawn from a Cauchy.
mean absolute error

0.2349 0.0953

Now, ε = 1% are replaced by 1, 000.
mean absolute error
Pierre Alquier, RIKEN AIP

10.018 0.0903
Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

1

Estimation via MMD optimization
Definition of the estimator
Basic properties
References and further works

2

Robustness to outliers
Application to Huber contamination model
Example : estimation of the mean of a Gaussian
Numerical experiments

3

Robustness to dependence
Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

And now, non-independent observations
Lemma
When X1 , . . . , Xn are identically distributed from P0 ,hi0E DK P̂n , P≤?

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

And now, non-independent observations
Lemma
When X1 , . . . , Xn are identically distributed from P0 ,hi0E DK P̂n , P
≤?

i
h
E D2K P̂n , P 0X2= E (1/n)(µ(δXi ) − µ(P0 ))
H

=

1
2
+ 2
E µ(δXi ) − µ(P0 ), µ(δXj ) − µ(P0 ) H
n n 1≤i<j≤nX

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

Mesure of dependence via covariance in H
Definition
When (X1 , . . . , Xn , . . . ) is a stationary process with marginal
distribution P0 , we put :%h = E µ(δXt+h ) − µ(P0 ), µ(δXt ) − µ(P0 ) H .

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

Mesure of dependence via covariance in H
Definition
When (X1 , . . . , Xn , . . . ) is a stationary process with marginal
distribution P0 , we put :%h = E µ(δXt+h ) − µ(P0 ), µ(δXt ) − µ(P0 ) H .
Lemma - dependent case
When X1 , . . . , Xn are identically distributed from P0 ,
#
"
n
h

i 1
X
0
E DK P̂n , P
≤
1+
%h
n
h=1
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

Mesure of dependence via covariance in H
Theorem - dependent case
When (X1 , . . . , Xn , . . . ) is a stationary process with marginal
distribution P0Phi
2 + 2 nh=1 %h√.
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) +θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

Mesure of dependence via covariance in H
Theorem - dependent case
When (X1 , . . . , Xn , . . . ) is a stationary process with marginal
distribution P0Phi
2 + 2 nh=1 %h√.
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) +
θ∈Θ
n
P
1
assume that ∞h=1 %h = Σ < +∞ thenhi
2 + 2Σ
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈ΘnPierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

Mesure of dependence via covariance in H
Theorem - dependent case
When (X1 , . . . , Xn , . . . ) is a stationary process with marginal
distribution P0Phi
2 + 2 nh=1 %h√.
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) +
θ∈Θ
n
P
1
assume that ∞h=1 %h = Σ < +∞ thenhi
2 + 2Σ
E DK Pθ̂nMMD , P0 ≤ inf DK (Pθ , P0 ) + √ .
θ∈Θn2

we also have a bound in probability, based on Rio’s versionof Hoeffding’s inequality ; it requires more assumptions.
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

An example : auto-regressive processes
Proposition
Assume that Xt takes values in Rd and that
K (x, y ) = F (kx − y k) where F is an L-Lipschitz function.
Assume that
Xt+1 = AXt + εt+1
where the (εt ) are i.i.d with Ekε0 k < ∞, and A is a matrix
with kAk = supkxk=1 kAxk < 1.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

An example : auto-regressive processes
Proposition
Assume that Xt takes values in Rd and that
K (x, y ) = F (kx − y k) where F is an L-Lipschitz function.
Assume that
Xt+1 = AXt + εt+1
where the (εt ) are i.i.d with Ekε0 k < ∞, and A is a matrix
with kAk = supkxk=1 kAxk < 1.
Thent 2LEkε0 k%t ≤ kAk

1 − kAkand Σ =

Pierre Alquier, RIKEN AIP

∞
X
t=1

%t =

2kAkLEkε0 k.
(1 − kAk)2

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

A non-mixing process with Σ < +∞Example : consider X0 ∼ U([0, 1]), ηt i.i.d Be(1/2) and
Xt+1 =

Pierre Alquier, RIKEN AIP

Xt + ηt+1.
2

Parametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

A non-mixing process with Σ < +∞Example : consider X0 ∼ U([0, 1]), ηt i.i.d Be(1/2) and
Xt + ηt+1.
2
It satisfies the assumptions of the previous proposition, we
have %t ≤ L/2t and Σ = 2L.
Xt+1 =

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

A non-mixing process with Σ < +∞Example : consider X0 ∼ U([0, 1]), ηt i.i.d Be(1/2) and
Xt + ηt+1.
2
It satisfies the assumptions of the previous proposition, we
have %t ≤ L/2t and Σ = 2L.
Xt+1 =

Note however that this process is known to be non-mixing.

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

A non-mixing process with Σ < +∞Example : consider X0 ∼ U([0, 1]), ηt i.i.d Be(1/2) and
Xt + ηt+1.
2
It satisfies the assumptions of the previous proposition, we
have %t ≤ L/2t and Σ = 2L.
Xt+1 =

Note however that this process is known to be non-mixing.
More generally, we prove the following result :Proposition
Under some (non-restrictive) assumption on the kernel K ,%t ≤ cK .βt (the β-mixing coef.)
Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization

Estimation via MMD optimization
Robustness to outliers
Robustness to dependence

Extension to non-independent observations
A (new ?) dependence coefficient
Example : auto-regressive observations

Thank you !

Pierre Alquier, RIKEN AIPParametric estimation via MMD optimization