# Levels Merging in the Latent Class Model

Christophe Biernacki

► To cite this version:

Christophe Biernacki. Levels Merging in the Latent Class Model. Statistical Learning Sustainability and Impact Evaluation, Jun 2023, Ancona (IT), Italy. hal-04370900

HAL Id: hal-04370900

https://inria.hal.science/hal-04370900v1

Submitted on 3 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Levels Merging in the Latent Class Model

C. Biernacki

Statistical Learning Sustainability and Impact Evaluation, June 21-23 2023, Ancona (Italy)

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

Université
de Lille

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>
Take home message

<figure><img src="image_14.png" /></figure>

potentially numerous

Level merging could increase
clustering accuracy!

<figure><img src="image_15.png" /></figure>
Limit of the Latent Class Model (LCM) in case of numerous levels

■ Cluster analysis is one of the main data analysis method. It aims at estimating a $K$ groups partition $z = (z_1, ..., z_n)$ of a data set $x = (x_1, ..., x_n) \in \mathcal{X}^d$

■ Model-based clustering allows to reformulate cluster analysis as a well-posed estimation problem both for the partition $z$ and for the number $K$ of groups [8]

■ Model-based clustering has led practical successes, even in some challenging situations, as the high dimensional context ("large" $d$ value) [3]

■ When $\mathcal{X}$ corresponds to $d$ categorical variables (frequent situation), the $j$th one having $m_j$ response levels, the standard latent class model (LCM, [5]) is used

$$p(\mathbf{x}_i; \theta) = \sum_{k=1}^{K} \pi_k \prod_{j=1}^{d} \prod_{h=1}^{m_j} (\alpha_k^{j,h})^{x_{ij}^h}, \quad \text{where } \theta = (\pi, \alpha), \pi = (\pi_k), \alpha = (\alpha_k)$$

■ When some $m_{js}$ are large another kind of dimensionality curse appear. Ex.: US states, French depts, professional social categories, medical nomenclature...

The standard latent class model is affected by such a curse of dimension paradigm since the number of free parameters if equal to $K - 1 + K \sum_{j=1}^{d} (m_j - 1)$

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>
LCM-LM: a LCM extension for the case of numerous levels

■ **Idea:** constrain some levels to share the same parameter value (level partitioning)

■ **Notation:** for each variable $j$, $L_j$ clusters of levels $\mathbf{w}_j = (\mathbf{w}_j^1, \dots, \mathbf{w}_j^{m_j})$

■ **New model LCM-LM (Latent Class Model by Levels Merging):** from the full level clustering $\mathbf{w} = (\mathbf{w}_1, \dots, \mathbf{w}_d)$, the new LCM is defined by

$$ \mathbf{p}(\mathbf{x}_i | \mathbf{w}; \vartheta) = \sum_{k=1}^{K} \pi_k \prod_{j=1}^{d} \prod_{\ell=1}^{L_j} \left( \frac{\beta_k^{j\ell}}{\sum_{h=1}^{m_j} w_h^{j\ell}} \right)^{x_h^j} $$

with $\beta = (\beta_1, \dots, \beta_K)$, $\beta_k = (\beta_k^{j\ell}; j = 1, \dots, d; \ell = 1, \dots, L_j)$ and $\vartheta = (\pi, \beta)$

■ **Remark:** LCM-LM is an extension a simpler model in [6]

■ **An extreme case:** if $L_j = 1$ then LCM-LM acts as a variable selection (no effect of variable $j$ on the clustering)

LCM-LM is parsimonious with regards to the number of levels since its number of parameters is equal to $K - 1 + K \sum_{j=1}^{d} (L_j - 1)$

<figure><img src="image_32.png" /></figure>
# LCM-LM: parameter estimation and model selection

## Hint

Possible to reformulating the log-likelihood of LCM-LM as a classical LCM...

* LCM-LM ⇔ LCM with the following new data $y_w = (y_{w,1}, \dots, y_{w,n})$

* Parameter estimation: $L(\vartheta|\mathbf{w}) = L(\vartheta; y_w) + c_w$, thus use a classical LCM EM

* Model selection:

* ICL criterion [1, 2]: just written as a penalization of the classical LCM ICL

$$ \mathrm{ICL}_{\mathrm{w}}^{\mathrm{LCM-LM}} = \ln p(\mathbf{x}, \hat{\mathbf{z}}_{\mathrm{w}} | \mathbf{w}) = \mathrm{ICL}_{\mathrm{w}}^{\mathrm{LCM}} + \underbrace{c_{\mathrm{w}}}_{\gamma} $$

* MICL criterion [7]: MICL (Maximum Integrated Complete-data Likelihood) is a variant of ICL avoiding multiple parameter estimation

$$ \mathrm{MICL}_{\mathrm{w}}^{\mathrm{LCM-LM}} = \ln p(\mathbf{x}, \mathbf{z}_{\mathrm{w}}^* | \mathbf{w}) \quad \text{with} \quad \mathbf{z}_{\mathrm{w}}^* = \arg \max_{z \in \mathcal{Z}} \ln p(\mathbf{x}, \mathbf{z}|\mathbf{w}). $$

Then, the model $\mathbf{w}^*$ maximizing $\mathrm{MICL}_w^{\mathrm{LCM-LM}}$ is retained: $\mathbf{w}^* = \arg \max_{\mathbf{w} \in \mathcal{W}} \mathrm{MICL}_w$. Finally a specific two step algorithm derived from [7] can be used for performing this optimization (not implemented yet).

<figure><img src="image_33.png" /></figure>
# Numerical experiments and conclusion

## ■ Numerical experiments

* **Data set:** 506 patients on the basis of petrial variates alone for the prostate cancer clinical trial data of [4], described by four categorical variables with various numbers of levels: performance rating (so-called PF, with 4 levels), cardiovascular disease history (HX, 2 levels), electrocardiogram code (EKG, 7 levels) and bone metastases (BM, 2 levels). There are also 62 missing values, so about 1% of the whole sample. There exists two stages of the disease (Stage 3 and Stage 4).

* **EM-like package:** RMixtcomp<fnref n="1" /> for LCM categorical data and also dealing with missing values)

* **LCM-LM proposal:** since level 4 of variable PF is under-represented (only 2 individuals), we propose to merge it with level 3 of the same variable

* **Results with $K = 2$:**

<table><thead><tr><th></th><th>LCM</th><th>LCM-LM</th></tr></thead><tbody><tr><td>ICL</td><td>-1728.4</td><td>-1618.4</td></tr><tr><td>error</td><td>49.2%</td><td>28.8%</td></tr></tbody></table>

## ■ Conclusion

* Very promising preliminary numerical results

* Need to still implement MICL

<figure><img src="image_34.png" /></figure>

<footnote n="1">¹https://cran.r-project.org/web/packages/RMixtComp/index.html</footnote>
<figure><img src="image_35.png" /></figure>

C. Biernacki, G. Celeux, and G. Govaert.

Assessing a mixture model for clustering with the integrated completed likelihood.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 22(7):719–725, 2000.

<figure><img src="image_36.png" /></figure>

C. Biernacki, G. Celeux, and G. Govaert.

Exact and Monte Carlo calculations of integrated likelihoods for the latent class model.
*Journal of Statistical Planning and Inference*, 140(11):2991–3002, 2011.

<figure><img src="image_37.png" /></figure>

Christophe Biernacki and Cathy Maugis.

High-dimensional clustering.
In J-J. Droesbeke, G. Saporta, and C. Thomas-Agnan, editors, *Choix de modèles et agrégation*. Technip, September 2017.

<figure><img src="image_38.png" /></figure>

D.P. Byar and S.B. Green.

The choice of treatment for cancer patients based on covariate information: Application to prostate cancer.
*Bulletin du Cancer*, 67:477–490, 1980.

<figure><img src="image_39.png" /></figure>

L. A. Goodman.

Exploratory latent structure models using both identifiable and unidentifiable models.
*Biometrika*, 61:215–231, 1974.

<figure><img src="image_40.png" /></figure>

R. Lebret, S. Iovleff, F. Langrognet, C. Biernacki, G. Celeux, and G. Govaert.

Rmixmod: The r package of the model-based unsupervised, supervised and semi-supervised classification mixmod library.
*Journal of Statistical Software*, in press, 2015.

<figure><img src="image_41.png" /></figure>

M. Marbac and M. Sedki.

Variable selection for model-based clustering using the integrated complete-data likelihood.
*Statistics and Computing*, 27:1049–1063, 2017.

<figure><img src="image_42.png" /></figure>

G.J. McLachlan and K.E. Basford.

Mixture Models: Inference and Applications to Clustering.
Marcel Dekker, New York, 1988.

<figure><img src="image_43.png" /></figure>