# Dealing with missing data in model-based clustering through a MNAR model

Christophe Biernacki, Claire Boyer, Gilles Celeux, Julie Josse, Fabien

Laporte, Matthieu Marbac Lourdelle, Aude Sportisse

► To cite this version:

Christophe Biernacki, Claire Boyer, Gilles Celeux, Julie Josse, Fabien Laporte, et al. Dealing with missing data in model-based clustering through a MNAR model. The 14th Professor Aleksander Zeliaś International Conference on Modelling and Forecasting of Socio-Economic Phenomena, May 2021, Zakopane, Poland. *hal-03505659*

HAL Id: hal-03505659

https://hal.science/hal-03505659

Submitted on 31 Dec 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

# Dealing with missing data in model-based clustering through a MNAR model

Christophe Biernacki, Claire Boyer, Gilles Celeux, Julie Josse, Fabien Laporte, Matthieu Marbac Lourdelle, Aude Sportisse

THE 14TH PROFESSOR ALEKSANDER ZELIAS INTERNATIONAL CONFERENCE ON MODELLING AND FORECASTING OF SOCIO-ECONOMIC PHENOMENA ON-LINE CONFERENCE MAY 11-13, 2021, ZAKOPANE, POLAND

<figure><img src="image_8.png" /></figure>

Université de Lille

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

CORBONNE UNIVERSITÉ

<figure><img src="image_11.png" /></figure>

ENSAI

<figure><img src="image_12.png" /></figure>

Ecole nationale de la statistique et de l'analyse de l'information
Take home message

1 The missing data pattern may convey some information on clustering

2 Embed the missingness mechanism directly within the clustering modeling step
<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

# Outline

1. Introduction

2. A model-based MNAR clustering approach

3. Identifiability

4. Inference procedures

5. Medical study illustration

6. Concluding remarks
<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

Missing data: an inevitable event

The larger the datasets, the more missing data may appear...

Two traditional solutions (for obtaining a filled dataset)

■ Discard individuals with missing data: more variance or a biased subset

■ Impute missing data: possible bias and underestimation of the variability

General guidelines

■ Obtaining a complete dataset is not the final goal

■ Missing data management should take into account the initial analysis target

Our analysis target: model-based clustering

Embed missing data management into this paradigm...
<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

Missing data: notations

■ $Y = \{y_1|...|y_n\}^T$: full dataset with $n$ individuals

■ $y_i = (y_{i1},...,y_{id}) \in \mathcal{Y}$, depending on the data type: individual $i \in \{1,...,n\}$

■ **continuous data**: $\mathcal{Y} = \mathbb{R}^d$

■ **categorical data**: $\mathcal{Y} = \{0,1\}^{\ell_1} \times ... \times \{0,1\}^{\ell_d}$ where $\ell_j$ is the number of levels for $y_{ij} = (y_{ij}^1, ..., y_{ij}^{\ell_j})$, where $y_{ij}^\ell = 1$ if $y_{ij}$ takes the level $\ell$, 0 otherwise.

■ **mixed data**: combination of continuous and categorical data.

■ $C = \{c_1|...|c_n\}^T \in \{0,1\}^{n \times d}$: pattern of missing data for the full dataset

■ $c_i = (c_{i1},...,c_{id}) \in \{0,1\}^d$: pattern of missing data for individual $i \in \{1,...,n\}$

$$c_{ij} = 1 \Leftrightarrow y_{ij} \text{ is missing}$$

■ $y_i^{\text{obs}}$: the observed variables values for individual $i$

■ $y_i^{\text{mis}}$: the missing variables values for individual $i$
<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

Missing data: typology of the missing mechanisms

■ Missing completely at random (MCAR):

$$P(c|y; \psi) = P(c; \psi) \quad \forall y$$

■ Missing at random (MAR):

$$P(c|y; \psi) = P(c|y^{\text{obs}}, \psi) \quad \forall y^{\text{mis}}$$

■ Missing not at random (MNAR): the mechanism is not MCAR nor MAR

Example of MNAR data

The probability to have a missing value on income depends on the value of income (rich people less inclined to reveal their income).
<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

Ignorable vs. non ignorable model

A missing mechanism is ignorable if likelihoods can be decomposed as

$$L(\theta, \psi; \underbrace{y^{\text{obs}}, c}_{\text{observed data}}) = L(\psi; c|y^{\text{obs}}) \times L(\theta; y^{\text{obs}})$$

Some simple algebra show that this occurs when missing mechanism is not MNAR

Inference of $\theta$

"If the missing mechanism is ignorable then likelihood-based inferences for $\theta$ from $L(\theta; y^{\text{obs}})$ will be the same as likelihood based inference for $\theta$ from $L(\theta, \psi; y^{\text{obs}}, c)$." ([Little and Rubin, 2002] Section 6.2)

- M(C)AR is ignorable

- MNAR is not ignorable
<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

Clustering: model-based approach

■ Partition with K clusters: $Z = (z_1 | \dots | z_n)^T \in \{0, 1\}^{n \times K}$ where

$z_i = (z_{i1}, \dots, z_{iK}) \in \{0, 1\}^K$

$z_{ik} = 1$ if $y_i$ belongs to cluster $k$, $z_{ik} = 0$ otherwise

■ Mixture model: $y_1, \dots, y_n$ are i.i.d. from the mixture

$$f(y_i; \pi, \theta) = \sum_{k=1}^{K} \pi_k f_k(y_i; \theta_k)$$

■ $\pi_k = P(z_{ik} = 1)$, $\pi = (\pi_1, \dots, \pi_K)$

■ $f_k(\cdot; \theta_k)$: pdf of the k-th component parametrized by $\theta_k$, $\theta = (\theta_1, \dots, \theta_K)$

■ continuous data: $f_k(\cdot; \theta_k) = \phi(\cdot; \mu_k, \Sigma_k)$ is the d-variate Gaussian distribution with mean vector $\mu_k$ and covariance matrix $\Sigma_k$

■ categorical data: the features are independent conditionally to the group membership i.e.

$$ \begin{align*} f_k(\cdot; \theta_k) &= \prod_{j=1}^{J} f_{kj}(\cdot; \theta_{kj}), \text{ where } f_{kj} = \prod_{\ell=1}^{\ell_j} (\theta_{kj}^{\ell})^{y_{ij}^{\ell}} \text{ is the multinomial distribution with} \\ \theta_{kj} &= (\theta_{kj}^{\ell} = P(y_{ij}^{\ell} = 1 | z_{ik} = 1))_{\ell=1, \dots, \ell_j} \end{align*} $$

■ mixed data: the features are independent conditionally to the group membership, $f_k(\cdot; \theta_k)$ is the product of univariate Gaussian and multinomial distributions

■ Can also be extended to other cases (count data with Poisson distributions for instance)

Question we address in this work

Which distribution $P(c|y, z; \psi)$ to propose in this clustering context?
<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

# Outline

1. Introduction

2. A model-based MNAR clustering approach

3. Identifiability

4. Inference procedures

5. Medical study illustration

6. Concluding remarks
<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

Proposed zoology of MNAR models in clustering

$$ \mathbb{P}(c_i | y_i, z_{ik} = 1; \psi) = \prod_{j=1}^{d} \mathbb{P}(c_{ij} | y_i, z_{ik} = 1; \psi) $$

* $\text{MNAR}_{y^k} z^j$, with $\psi = (\alpha, \beta)$ where $\alpha = (\alpha_{11}, \dots, \alpha_{1d}, \dots, \alpha_{K1}, \dots, \alpha_{Kd})^T \in \mathbb{R}^{Kd}$ and $\beta = (\beta_{11}, \dots, \beta_{1d}, \dots, \beta_{K1}, \dots, \beta_{Kd})^T \in \mathbb{R}^{Kd}$

$$ \mathbb{P}(c_{ij} = 1 | y_i, z_{ik} = 1; \psi) = \rho(\alpha_{kj} + \beta_{kj} y_{ij}), $$

with $\rho$ the cdf of any continuous distribution (logit, probit)

* $\text{MNAR}_{yz}$, $\text{MNAR}_{yz}^k$, $\text{MNAR}_{yz} z^j$

* $\text{MNAR}_y$; $\text{MNAR}_y^k$; $\text{MNAR}_y z^j$; $\psi = (\beta_{11}, \dots, \beta_{1d}, \dots, \beta_{K1}, \dots, \beta_{Kd})^T$; $\mathbb{P}(c_{ij} = 1 | y_i, z_{ik} = 1; \psi) = \rho(\beta_{kj} y_{ij})$

* $\text{MNAR}_z$, $\text{MNAR}_z^j$; $\psi = (\alpha_{11}, \dots, \alpha_{1d}, \dots, \alpha_{K1}, \dots, \alpha_{Kd})^T$; $\mathbb{P}(c_{ij} = 1 | y_i, z_{ik} = 1; \psi) = \rho(\alpha_{kj})$

<figure><img src="image_61.png" /></figure>
Overview of the proposed MNAR models

<table><thead><tr><th rowspan="2"></th><th colspan="2">Effect on the variable j</th><th colspan="2">Effect on the class membership k</th><th colspan="2">Nb parameters</th></tr><tr><th>Depends on j</th><th>Depends on k</th><th>Depends on j</th><th>Depends on k</th><th>Continuous</th><th>Categorical</th></tr></thead><tbody><tr><td>MNARz<sup>i</sup>y<sup>k</sup></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>2K<sup>d</sup></td><td>K(d + &sum;<sup>d</sup><sub>j=1</sub>(l<sub>j</sub> - 1))</td></tr><tr><td>MNARy<sup>j</sup></td><td>✓</td><td><b>X</b></td><td>✓</td><td>✓</td><td>(K + 1)d</td><td>Kd + &sum;<sup>d</sup><sub>j=1</sub>(l<sub>j</sub> - 1)</td></tr><tr><td>MNARy<sup>k</sup>z</td><td>✓</td><td>✓</td><td><b>X</b></td><td>✓</td><td>K(d + 1)</td><td>K(1 + &sum;<sup>d</sup><sub>j=1</sub>(l<sub>j</sub> - 1))</td></tr><tr><td>MNARyz</td><td>✓</td><td><b>X</b></td><td><b>X</b></td><td>✓</td><td>(K + d)</td><td>K + &sum;<sup>d</sup><sub>j=1</sub>(l<sub>j</sub> - 1)</td></tr><tr><td>MNARy</td><td>✓</td><td><b>X</b></td><td><b>X</b></td><td><b>X</b></td><td>d</td><td>&sum;<sup>d</sup><sub>j=1</sub>(l<sub>j</sub> - 1)</td></tr><tr><td>MNARy<sup>k</sup></td><td>✓</td><td>✓</td><td><b>X</b></td><td><b>X</b></td><td>Kd</td><td>K &sum;<sup>d</sup><sub>j=1</sub>(l<sub>j</sub> - 1)</td></tr><tr><td>MNARz</td><td><b>X</b></td><td><b>X</b></td><td><b>X</b></td><td>✓</td><td>K</td><td>K</td></tr><tr><td>MNARz<sup>i</sup></td><td><b>X</b></td><td><b>X</b></td><td>✓</td><td>✓</td><td>Kd</td><td>Kd</td></tr></tbody></table>

Terminology in the sequel:

* MNARz, MNARz<sup>i</sup>: the only effect of missingness is on the class membership

* MNARy*: all the other models which considers the effect of the missingness depending on the variable

* MNAR*: all the models
MNARz analysis: it depends on y through z!

$$P(c_{ij} = 1 | y_i; \theta, \psi) = \sum_{k=1}^{K} P(c_{ij} = 1 | y_i, z_{ik} = 1; \psi) P(z_{ik} = 1 | y_i; \theta)$$

Example of a univariate Gaussian model with the three components

$$0.2N(\cdot; 0, 1) + 0.3N(\cdot; 1, 2) + 0.5N(\cdot; 2, 3)$$

and with parameters of the logit expression: $\alpha_0 = 1$, $\beta_1 = 1$, $\beta_2 = -1$, $\beta_3 = 1$

<figure><img src="image_62.png" /></figure>
<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>

<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

MNARz analysis: pattern c gives information on partition z!

Draw Bayes error of a MNARz model with two components and 20% of missing data

$\pi_k = 0.5, ||\mu_2 - \mu_1||$ varies, $\Sigma_1 = \Sigma_2 = I, |\beta_2 - \beta_1|$ varies

<figure><img src="image_69.png" /></figure>

Both $\mu_k$ and $\beta_k$ act on the Bayes error
# Reinterpretation of the MNARz and MNARz<sup>j</sup> models as MAR

Commonly used in Machine Learning
[Jones, 1996, Little and Rubin, 2002, Josse et al., 2019]

Mixture model for $Y^{\text{obs}}$ and Bernoulli distribution for $C$
$\Leftrightarrow$ MAR mixture model for $\tilde{Y}^{\text{obs}} = (Y^{\text{obs}}|C)$

For example,

$$Y^{\text{obs}} = \begin{pmatrix} ? & 2.6 & 5 \\ \text{blue} & 1.9 & 4 \\ \text{red} & 2.3 & ? \end{pmatrix}, \quad C = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}$$

then $\tilde{Y}^{\text{obs}}$ is expressed as

$$\tilde{Y}^{\text{obs}} = \begin{pmatrix} ? & 2.6 & 5 & 1 & 0 & 0 \\ \text{blue} & 1.9 & 4 & 0 & 0 & 0 \\ \text{red} & 2.3 & ? & 0 & 0 & 1 \end{pmatrix}.$$

## Proposition 1: in terms of maximum likelihood

The maximum likelihood estimate associated to the dataset $\tilde{Y}^{\text{obs}}$ under MAR model is the one associated to the dataset $Y^{\text{obs}}$ under MNARz or MNARz<sup>j</sup> models.
$\Rightarrow$ can be extended to other estimation strategies
<figure><img src="image_70.png" /></figure>

<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

<figure><img src="image_75.png" /></figure>

# Outline

1. Introduction

2. A model-based MNAR clustering approach

3. Identifiability

4. Inference procedures

5. Medical study illustration

6. Concluding remarks
<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

<figure><img src="image_78.png" /></figure>

<figure><img src="image_79.png" /></figure>

<figure><img src="image_80.png" /></figure>

<figure><img src="image_81.png" /></figure>

Continuous and count data (1)

Previous works: [Teicher, 1963] (without NA), [Miao et al., 2016] (for MNAR data)

Identifiability for a mixture model with MNAR data

⇔ Mixture/MNAR parameters are uniquely determined from available information

Proposition 2: identifiability for continuous and count data

Assume that

1. The marginal mixture $\sum_{k=1}^{K} \pi_k f_k(y_j; \theta_k)$ is identifiable

2. There exists a total ordering $\preceq$ of $\mathcal{F}_j \times \mathcal{R}_j$, for $j \in \{1, \dots, d\}$ fixed, where $\mathcal{F}_j = \{f_{1j}, \dots, f_{Kj}\}$ and $\mathcal{R} = \{\rho_1, \dots, \rho_K\} = \{\rho(\cdot; \psi_1), \dots, \rho(\cdot; \psi_K)\}$. The total ordering is s.t. $\forall k < \ell$, $F_k = \rho_k f_{kj} \preceq F_\ell = \rho_\ell f_{\ell j}$ implies

$$ \lim_{u \to +\infty} \frac{\rho_{\ell}(u) f_{\ell j}(u)}{\rho_{K}(u) f_{Kj}(u)} = 0 $$

Then the mixture model with one of the MNAR* mechanisms is identifiable up to label swapping
Continuous and count data (2)

Is the total ordering checked for classical distributions ?

<table><thead><tr><th rowspan="2">f<sub>k</sub></th><th colspan="2">Gaussian</th><th colspan="2">Poisson</th></tr><tr><th>Probit</th><th>Logit</th><th>Probit</th><th>Logit</th></tr></thead><tbody><tr><td>ρ<sub>k</sub></td><td></td><td></td><td></td><td></td></tr><tr><td>MNAR<sub>k</sub><sup>j</sup>y<sup>k</sup></td><td>✓</td><td>generic idenfiability</td><td>✓</td><td>generic idenfiability</td></tr><tr><td>MNAR<sub>k</sub><sup>j</sup>z</td><td></td><td></td><td></td><td></td></tr><tr><td>MNAR<sub>k</sub><sup>j</sup>y<sup>k</sup></td><td></td><td></td><td></td><td></td></tr><tr><td>MNAR<sub>k</sub><sup>j</sup>z<sup>j</sup></td><td></td><td></td><td></td><td></td></tr><tr><td>MNAR<sub>k</sub><sup>j</sup>yz</td><td></td><td></td><td></td><td></td></tr><tr><td>MNAR<sub>k</sub><sup>j</sup>yz</td><td></td><td></td><td></td><td></td></tr><tr><td>MNAR<sub>k</sub><sup>j</sup>y</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>MNAR<sub>k</sub><sup>j</sup>z</td><td></td><td></td><td></td><td></td></tr><tr><td>MNAR<sub>k</sub><sup>j</sup>z<sup>j</sup></td><td></td><td></td><td></td><td></td></tr></tbody></table>

Generic identifiability: all not-identifiable parameter choices lie within a proper subvariety, and thus form a set of Lebesgue zero measure
Categorical data

Previous work: [Allman et al., 2009] (without NA)

Recall that for categorical data: conditional independence of the features given the group membership holds i.e. $f_k(\cdot; \theta_k) = \prod_{j=1}^{d} f_{kj}(\cdot; \theta_{kj})$

**Proposition 3: identifiability for categorical data**

Assume that $d \ge 2[\log_2 K] + 1$ and $f_k(\cdot; \theta_k) = \prod_{j=1}^{d} f_{kj}(\cdot; \theta_{kj})$

✓ Then the mixture model with *MNARz* or *MNARzj* mechanism is identifiable up to label swapping

X The mixture model with one of the *MNARy\* mechanisms is not identifiable*
<figure><img src="image_82.png" /></figure>

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /></figure>

<figure><img src="image_85.png" /></figure>

<figure><img src="image_86.png" /></figure>

<figure><img src="image_87.png" /></figure>

Mixed data

$f_k(\cdot; \theta_k) = \prod_{j=1}^{d} f_{kj}(\cdot; \theta_{kj})$, thus identifiability of mixed data directly follows from Proposition 2 for continuous variables and from Proposition 3 for categorical variables
<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>

<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /></figure>

# Outline

1. Introduction

2. A model-based MNAR clustering approach

3. Identifiability

4. Inference procedures

5. Medical study illustration

6. Concluding remarks
EM algorithm: looks simple

The expected complete log-likelihood knowing the observed data and a current value of the parameters can be decomposed into two parts

$$
\begin{align*}
Q(\theta, \psi; \theta^r, \psi^r, \pi^r) &= \mathbb{E}[\ell_{\text{comp}}(\theta, \psi; \pi; y, z, c) | y_i^{\text{obs}}, c_i; \theta^r, \psi^r, \pi^r] \\
&= Q_{\psi}(\theta, \pi; \theta^r, \psi^r, \pi^r) + Q_{c}(\psi; \theta^r, \psi^r, \pi^r)
\end{align*}
$$

$$
Q_{\psi}(\theta, \pi; \theta^r, \psi^r, \pi^r) = \sum_{i=1}^{n} \sum_{k=1}^{K} (\tau_{ik})^r \log(\pi_k) + \sum_{i=1}^{n} \sum_{k=1}^{K} (\tau_{ik})^r E_{\psi i}^r(\theta)
$$

$$
Q_c(\psi; \theta^r, \psi^r, \pi^r) = \sum_{i=1}^{n} \sum_{k=1}^{K} (\tau_{ik})^r E_{ic}^r(\psi)
$$

where for $i = 1, \dots, n$ and $k = 1, \dots, K$,

$$
E_{iy}^{r}(\theta) = \mathbb{E} \left[ \log(f_k(y_i; \theta_k)) \mid y_i^{\text{obs}}, z_{ik} = 1, c_i; \theta^r, \psi^r \right]
$$

$$
E_{ic}^{r}(\psi) = \mathbb{E} \left[ \log \left( \mathbb{P}(c_i | y_i, z_{ik} = 1; \psi) \right) \middle| y_i^{\text{obs}}, z_{ik} = 1, c_i; \theta^r, \psi^r \right]
$$

$$
(\tau_{ik})^r = \mathbb{P}(z_{ik} = 1 | y_i^{\text{obs}}, c_i; \theta^r, \psi^r, \pi^r) \propto \pi_k^r f_k(y_i^{\text{obs}}; \theta_k^r) \mathbb{P}(c_i | y_i^{\text{obs}}, z_{ik} = 1; \psi^r)
$$
<figure><img src="image_94.png" /></figure>

<figure><img src="image_95.png" /></figure>

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /></figure>

<figure><img src="image_98.png" /></figure>

<figure><img src="image_99.png" /></figure>

EM algorithm for MNARz and MNARz<sup>j</sup>

MNARz, MNARz<sup>j</sup> : needs some computations but still simple.

$$
\mathbb{P}(c_{ij} = 1 | y_i, z_{ik} = 1; \psi) = \rho(\alpha_{kj}) \quad \text{(independent of } y_i \text{)} \quad (\Delta)
$$

■ Gaussian data : $(y_i|z_{ik}=1; \theta^r) \sim \mathcal{N}(\mu_k, \Sigma_k)$

■ $\mathbb{P}(y_i^{\text{mis}} | y_i^{\text{obs}}, z_{ik} = 1, c_i; \theta^r, \psi^r) = \mathbb{P}(y_i^{\text{mis}} | y_i^{\text{obs}}, z_{ik} = 1; \theta^r) \text{ using } (\Delta)$ and

$$
(y_i^{\text{mis}} | y_i^{\text{obs}}, z_{ik} = 1; \theta^r) \sim \mathcal{N} \left( (\hat{\mu}_{ik}^{\text{mis}})^r, (\hat{\Sigma}_{ik}^{\text{mis}})^r \right)
$$

where $(\hat{\mu}_{ik}^{\text{mis}})^r$ and $(\hat{\Sigma}_{ik}^{\text{mis}})^r$ only depend on $\mu_i^r, \Sigma^r$ and $y_i^{\text{obs}}$

$$
\Rightarrow E_{ij}^{r}(\theta) = \mathbb{E} \left[ \log(f_k(y_i; \theta^k)) \mid y_i^{\text{obs}}, z_{ik} = 1; \theta^r \right] \text{ easy to compute (classical formulae)}
$$

■ Using (Δ)

$$
E_{ic}^{r}(\psi) = \log(\mathbb{P}(c_i | z_{ik} = 1; \psi)) = \sum_{j=1}^{d} c_{ij} \log \rho(\alpha_{kj}) + (1 - c_{ij}) \log(1 - \rho(\alpha_{kj}))
$$

■ Using (Δ)

$$
(\tau_{ik})^r \propto \pi_k^r \phi(y_i^{\text{obs}}, (\mu_{ik}^{\text{obs}})^r, (\Sigma_{ik}^{\text{obs,obs}})^r) \prod_{j=1}^{d} \rho(\alpha_{kj}^r)^{c_{ij}} (1 - \rho(\alpha_{kj}^r))^{1-c_{ij}}
$$
<figure><img src="image_100.png" /></figure>

<figure><img src="image_101.png" /></figure>

<figure><img src="image_102.png" /></figure>

<figure><img src="image_103.png" /></figure>

<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /></figure>

EM algorithm for MNARz and MNARz<sup>j</sup>

Recall that: $(y_i^{\text{mis}} | y_i^{\text{obs}}, z_{ik} = 1; \theta^r) \sim \mathcal{N}((\tilde{\mu}_{ik}^{\text{mis}})^r, (\tilde{\Sigma}_{ik}^{\text{mis}})^r)$.

* **E-step**: for $k = 1, \dots, K$ and $i = 1, \dots, n$, compute $(\tilde{\mu}_{ik}^{\text{mis}})^r, (\tilde{\Sigma}_{ik}^{\text{mis}})^r, (\tau_{ik})^r$ and

$$ (\tilde{y}_{i,k})^r = (y_i^{\text{obs}}, (\tilde{\mu}_{ik}^{\text{mis}})^r) \qquad \tilde{\Sigma}_{ik}^r = \begin{pmatrix} 0_i^{\text{obs,obs}} & 0_i^{\text{obs,mis}} \\ 0_i^{\text{mis,obs}} & (\tilde{\Sigma}_{ik}^{\text{mis}})^r \end{pmatrix} $$

* **M-step**: for $k = 1, \dots, K$, compute

$$ \begin{align*} \pi_k^{r+1} &= \frac{1}{n} \sum_{i=1}^{n} (\tau_{ik})^r & \mu_k^{r+1} &= \frac{\sum_{i=1}^{n} (\tau_{ik})^r (\tilde{y}_{k,i})^r}{\sum_{i=1}^{n} (\tau_{ik})^r} \\ \Sigma_k^{r+1} &= \frac{\sum_{i=1}^{n} [(\tau_{ik})^r ((\tilde{y}_{i,k})^r - \mu_k^{r+1}) ((\tilde{y}_{i,k})^r - \mu_k^{r+1})^T + \tilde{\Sigma}_{ik}^r]}{\sum_{i=1}^{n} (\tau_{ik})^r} \end{align*} $$

For $\psi^{r+1}$: maximization of $Q_c(\psi; \theta^r, \psi^r, \pi^r)$ over $\psi$ with a **Newton-Raphson algorithm** (classical procedure for link functions of interest)

An EM algorithm can also be **easily derived for categorical data**
<figure><img src="image_106.png" /></figure>

<figure><img src="image_107.png" /></figure>

<figure><img src="image_108.png" /></figure>

<figure><img src="image_109.png" /></figure>

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

SEM algorithm for MNARy*

■ **SE-step**: draw the missing data $((y_i^{\text{mis}})^{r+1}, z_i^{r+1}) \sim (\cdot | y_i^{\text{obs}}, c_i; \theta^r, \psi^r, \pi^r)$

Use of One-Gibbs sampling:

■ $(y_i^{\text{mis}})^{r+1} \sim (\cdot | y_i^{\text{obs}}, z_i^r, c_i; \theta^r, \psi^r)$:

X not classical if $\rho$ is Logit,

truncated Gaussian distribution if $\rho$ is Probit

■ $z_k^{r+1} \sim (\cdot | y_k^{r+1}, c_k; \theta^r, \psi^r, \pi^r)$: draw the membership $k$ of $z_k^{r+1}$ from the multinomial distribution with probabilities

$$ \mathbb{P}(z_{ik} = 1 | y_k^{r+1}, c_k; \theta^r, \psi^r, \pi^r) = \frac{\mathbb{P}(c_i | y_k^{r+1}, z_{ik} = 1; \psi^r) \mathbb{P}(y_k^{r+1} | z_{ik} = 1; \theta^r) \pi_k^r}{\sum_{h=1}^K \mathbb{P}(c_i | y_k^{r+1}, z_{ih} = 1; \psi^r) \mathbb{P}(y_k^{r+1} | z_{ih} = 1; \theta^r) \pi_h^r} $$

Let $Y^{r+1} = (Y_1^{r+1} | \dots | Y_n^{r+1})$, $Z^{r+1} = (Z_1^{r+1} | \dots | Z_n^{r+1})$ be the imputed matrix and the partition

■ **M-step**: for $k = 1, \dots, K$, compute

■ $\pi^{r+1}_k$ with the proportion of rows of $Y^{r+1}$ belonging to class $k$

■ $\mu_k^{r+1}, \Sigma_k^{r+1}$ with the mean and covariance matrix of rows of $Y^{r+1}$ belonging to class $k$

■ $\psi^{r+1}$ with a Newton-Raphson algorithm
<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>

<figure><img src="image_115.png" /></figure>

<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

## Outline

1. Introduction

2. A model-based MNAR clustering approach

3. Identifiability

4. Inference procedures

5. Medical study illustration

6. Concluding remarks
Hospital Data: continuous features case

■ Number of patients: $n = 5146$

■ Number of features: $d = 7$

■ Age

■ Size

■ Weight

■ Cardiac frequency

■ Hemoglobin concentration

■ Temperature

■ Minimum Diastolic and Systolic Blood Pressure

■ Percentage of missing data: 6.4%
<figure><img src="image_118.png" /></figure>

<figure><img src="image_119.png" /></figure>

<figure><img src="image_120.png" /></figure>

<figure><img src="image_121.png" /></figure>

<figure><img src="image_122.png" /></figure>

<figure><img src="image_123.png" /></figure>

ICL comparison

<figure><img src="image_124.png" /></figure>

■ MCAR, MNARy and MNARz are equivalent until $K = 3$

■ MNARz and MNARyz clearly indicate presence of an additional cluster ($K = 4$)

It seems to be an illustration of the effect of *c* through MNARz and MNARyz
Missing Pattern

<figure><img src="image_125.png" /></figure>

It seems that MNARz modelling leads to a missing free cluster
Hospital data: mixed features case

■ Number of patients: $n = 5146$

■ Number of features: $d = 15$ (7 continuous and 8 categorical)

■ Percentage of missing data: ~ 4%

Model

Conditional independence of the variables knowing the cluster
<figure><img src="image_126.png" /></figure>

<figure><img src="image_127.png" /></figure>

<figure><img src="image_128.png" /></figure>

<figure><img src="image_129.png" /></figure>

<figure><img src="image_130.png" /></figure>

<figure><img src="image_131.png" /></figure>

ICL comparison

<figure><img src="image_132.png" /></figure>

- MCAR and MNARz are equivalent

- MNARyz seems really inappropriate

- Seems to miss the previous latent structure: requires a specific exploration...
Exploration 1: local independence is not relevant for this data set

<figure><img src="image_133.png" /></figure>

Not accounting for possible conditional dependencies between the continuous
variables is inappropriate for this dataset.
Results from exploration 2

<figure><img src="image_134.png" /></figure>

■ The mixture model bias can not be compensated by the (unbiased) missing mechanism modeling

■ It illustrates again the fact that information (on the latent partition) conveys by data is much more important than information conveys by the pattern
<figure><img src="image_135.png" /></figure>

<figure><img src="image_136.png" /></figure>

<figure><img src="image_137.png" /></figure>

<figure><img src="image_138.png" /></figure>

<figure><img src="image_139.png" /></figure>

<figure><img src="image_140.png" /></figure>

## Outline

1. Introduction

2. A model-based MNAR clustering approach

3. Identifiability

4. Inference procedures

5. Medical study illustration

6. Concluding remarks
<figure><img src="image_141.png" /></figure>

<figure><img src="image_142.png" /></figure>

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /></figure>

<figure><img src="image_145.png" /></figure>

<figure><img src="image_146.png" /></figure>

## Summary

* Interest to put a model on c

* Interest of the simple but meaningful model MNARz

* Link between our models and usual methods

## Ongoing works

* Deeper analysis of the previous results with doctors...

* Implement the proposed models/algo. in the Mixmod softwareª

* Address the trade-off between biased mixture model and biased missingness mechanism in particular for the mixed data case

<footnote>ªhttp://www.mixmod.org</footnote>
<figure><img src="image_147.png" /></figure>

<figure><img src="image_148.png" /></figure>

<figure><img src="image_149.png" /></figure>

<figure><img src="image_150.png" /></figure>

<figure><img src="image_151.png" /></figure>

<figure><img src="image_152.png" /></figure>

References

<figure><img src="image_153.png" /></figure>

Allman, E. S., Matias, C., Rhodes, J. A., et al. (2009).

Identifiability of parameters in latent structure models with many observed variables.

The Annals of Statistics, 37(6A):3099-3132.

<figure><img src="image_154.png" /></figure>

Jones, M. P. (1996).

Indicator and stratification methods for missing explanatory variables in multiple linear regression.

Journal of the American statistical association, 91(433):222-230.

<figure><img src="image_155.png" /></figure>

Josse, J., Prost, N., Scornet, E., and Varoquaux, G. (2019).

On the consistency of supervised learning with missing values.

arXiv preprint arXiv:1902.06931.

<figure><img src="image_156.png" /></figure>

Little, R. J. and Rubin, D. B. (2002).

Statistical Analysis with Missing Data.

Wiley.

<figure><img src="image_157.png" /></figure>

Miao, W., Ding, P., and Geng, Z. (2016).

Identifiability of normal and normal mixture models with nonignorable missing data.

Journal of the American Statistical Association, 111(516):1673-1683.

<figure><img src="image_158.png" /></figure>

Pirjol, D. (2013).

The logistic-normal integral and its generalizations.

Journal of Computational and Applied Mathematics, 237(1):460-469.