# Clustering: from modeling to visualizing Mapping clusters as spherical Gaussians

Christophe Biernacki, Matthieu Marbac, Vincent Vandewalle

► To cite this version:

Christophe Biernacki, Matthieu Marbac, Vincent Vandewalle. Clustering: from modeling to visualizing Mapping clusters as spherical Gaussians. Working Group on Model-Based Clustering, Jul 2023, Pittsburg, Etats-Unis, United States. hal-04370893

HAL Id: hal-04370893

https://inria.hal.science/hal-04370893v1

Submitted on 4 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Gaussian Based Visualization of Gaussian and Non-Gaussian Based Clustering

C. Biernacki

with M. Marbac-Lourdelle and V. Vandewalle

Working Group on Model-Based Clustering

July 17-21 2023, Pittsburgh, USA

Université de Lille

CNRS

Inria

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

Take home message

Traditionally: spaces for visualizing clusters are fixed for their user-convenience

Natural extension: models for visualizing clusters should follow the same principle!

<figure><img src="image_7.png" /></figure>
<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

# Outline

1. Clustering: from modeling to visualizing

2. Mapping clusters as spherical Gaussians

3. Numerical illustrations for complex data

4. Axes interpretability

5. Discussion
<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

Model-based clustering: pitch<fnref n="1" />

■ Data set: $\mathbf{x} = (\mathbf{x}_1, \dots, \mathbf{x}_n)$, each $\mathbf{x}_i \in \mathcal{X}$ with $d_X$ variables

■ Partition (unknown): $\mathbf{z} = (\mathbf{z}_1, \dots, \mathbf{z}_n)$ with binary notation $\mathbf{z}_i = (z_{i1}, \dots, z_{iK})$

■ Statistical model: couples $(\mathbf{x}_i, \mathbf{z}_i)$ independently arise from the parametrized pdf

$$ \underbrace{f(\mathbf{x}_i, \mathbf{z}_i)}_{\in \mathcal{F}} = \prod_{k=1}^{K} [\pi_k f_k(\mathbf{x}_i)]^{z_{ik}} $$

■ Estimating $\mathbf{f}$: implement the MLE principle through an EM-like algorithm

■ Estimating $K$: use some information criteria as BIC, ICL,...

■ Estimating $\mathbf{z}$: use the MAP principle $\hat{z}_{ik} = 1$ if $k = \arg \max_{\ell} t_{i\ell}(\hat{f})$ where

$$ t_{ik}(\mathbf{f}) = p(z_{ik} = 1 | \mathbf{x}_i; \mathbf{f}) = \frac{\pi_k f_k(\mathbf{x}_i)}{\sum_{\ell=1}^{K} \pi_\ell f_\ell(\mathbf{x}_i)} $$

<footnote n="1">¹See for instance [McLachlan & Peel 2004], [Biernacki 2017]</footnote>
<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

Model-based clustering: flexibility of $\mathcal{F}$ for complex $\mathcal{X}$

- Continuous data ($\mathcal{X} = \mathbb{R}^d$): multivariate Gaussian/t distrib. [McNicholas 2016]

- Categorical data: product of multinomial distributions [Goodman 1974]

- Mixing cont./cat.: product Gaussian/multinomial [Moustaki & Papageorgiou 2005]

- Functional data: the discriminative functional mixture [Bouveyron et al. 2015]

- Network data: the Erdös Rényi mixture [Zanghi et al. 2008]

- Other kinds of data, missing data, high dimension,...
<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

Model-based clustering: poor user-friendly understanding

■ *n* or *K* large: poor overview of partition ̂*z*

■ *d*<sub>*X*</sub> large: too many parameters to embrace as a whole in ̂*f*<sub>*k*</sub>

■ Complex *X*: specific and non trivial parameters involved in ̂f_k

Visualization procedures

Aim at proposing user-friendly understanding of the mathematical clustering results
<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

# Overview of clustering visualization: mapping vs. drawing

Visualization is the achievement of two different successive steps:

* **The mapping step:**

* Performs a transformation, typically space dimension reduction of a data set or of a pdf

* It produces no **graphical output** at all (deliver just a mathematical object)

* **The drawing step:**

* Provides the final **graphical display** from the output of the previous mapping step

* Usually involves classical graphical toolboxes and tunes any graphical parameters

Mathematician is first concerned by the more challenging mapping step
<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>

Overview of clustering visualization: individual mapping

■ Aims at visualizing simultaneously the data set x and its estimated partition ŷ

■ Transforms x, defined on X, into y = (y₁, ..., yₙ), defined on a new space Y

$$
M^{\text{ind}} \in \mathcal{M}^{\text{ind}} : \mathbf{x} \in \mathcal{X}^n \mapsto \mathbf{y} = M^{\text{ind}}(\mathbf{x}) \in \mathcal{Y}^n
$$

■ Many methods, depending on X definition: PCA, MCA, MFA, FPCA, MDS...

■ Some of them use ŷ in M<sup>ind</sup>: LDA, mixture entropy preservation [Scrucca 2010]

■ Nearly always, $\mathcal{Y} = \mathbb{R}^2$

Model ŷ is not taken into account through this approach which is focused on x
<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

Overview of clustering visualization: pdf mapping

■ Aims at displaying information relative to the mapping of the *f* distribution

■ Transforms $f = \sum_k \pi_k f_k \in \mathcal{F}$, into a new mixture $g = \sum_k \pi_k g_k \in \mathcal{G}$

$$ M^{\text{pdf}} \in \mathcal{M}^{\text{pdf}} : f \in \mathcal{F} \mapsto g = M^{\text{pdf}}(f) \in \mathcal{G} $$

■ $\mathcal{G}$ is a pdf family defined on the space $\mathcal{Y}$

■ $M^{\text{pdf}}$ is often obtained as a by product of $M^{\text{ind}}$ (tedious outside linear mappings)

■ For large *n*, $M^{\text{ind}}$ finally displays $M^{\text{pdf}}$

■ Often, both *y* and *g* are overlaid
<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

Summary of traditional visualization strategies<fnref n="2" />

Controlling the mapping family $\mathcal{M}^{\text{pdf}}$

$$ \boxed{\text{Strategy}_{\mathcal{M}} : \quad \mathcal{G}(\mathcal{M}^{\text{pdf}}) = \left\{ g : g = \mathcal{M}^{\text{pdf}}(f), f \in \mathcal{F}, \mathcal{M}^{\text{pdf}} \in \underbrace{\mathcal{M}^{\text{pdf}}}_{\text{controlled}} \right\}} $$

* Nature of $\mathcal{G}$ can dramatically depend on the choice of $\mathcal{M}^{\text{pdf}}$

* It can potentially lead to very different cluster shapes!

* Arguments for traditional $\mathcal{M}^{\text{pdf}}$: user-friendly, easy-to-compute

* Examples: linear mappings in all PCA-like methods

<footnote n="2">²Similar thinking with $\mathcal{M}^{\text{ind}}$</footnote>
<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

New visualization strategy

Controlling the pdf family $\mathcal{G}$

$$ \boxed{\text{Strategy}_{\mathcal{G}}} : \quad \underbrace{\mathcal{M}^{\text{pdf}}(\mathcal{G})}_{\text{uncontrolled}} = \left\{ \mathcal{M}^{\text{pdf}} : g = \mathcal{M}^{\text{pdf}}(f), f \in \mathcal{F}, g \in \underbrace{\mathcal{G}}_{\text{controlled}} \right\} $$

- It is the reversed situation where $\mathcal{G}$ is defined instead of $\mathcal{M}^{\text{pdf}}$

- Offer opportunity to impose directly $\mathcal{G}$ to be a user-friendly mixture family

- Strategy_M and Strategy_G are both valid but Strategy_G is rarely explored!

This work: explore Strategy_G
<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

# Outline

1. Clustering: from modeling to visualizing

2. Mapping clusters as spherical Gaussians

3. Numerical illustrations for complex data

4. Axes interpretability

5. Discussion
<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

<figure><img src="image_61.png" /></figure>

Spherical Gaussians as candidates

■ Users are usually familiar with multivariate spherical Gaussians on $\mathcal{Y} = \mathbb{R}^{d_Y}$

■ Thus a simple and “user-friendly” candidate $g$ is a mixture of spherical Gaussians

$$g(\mathbf{y}; \boldsymbol{\mu}) = \sum_{k=1}^{K} \underbrace{\pi_k}_{\text{from } f} \phi_{d_Y}(\mathbf{y}; \underbrace{\boldsymbol{\mu}_k}_{?}, \mathbf{I})$$

where $\boldsymbol{\mu} = (\mu_1, \dots, \mu_K)$ and $\phi_{d_Y}(\cdot; \mu_k, \mathbf{I})$ the pdf of the Gaussian distribution

■ with mean $\mu_k = (\mu_{k1}, \dots, \mu_{kd_Y}) \in \mathbb{R}^{d_Y}$

■ with covariance matrix equal to identity $\mathbf{I}$

$g(\cdot; \boldsymbol{\mu})$ should be then linked with $f$ in order to define a sensible $\mathcal{G}$

$$\mathcal{G} = \{ g : g(\cdot; \boldsymbol{\mu}), \boldsymbol{\mu} \in \arg\min \delta(f, g(\cdot; \boldsymbol{\mu})), f \in \mathcal{F} \}$$
<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>

<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /></figure>

g as the “clustering twin” of f

Question: how to choose $\delta$ since generally $\mathcal{X} \neq \mathcal{Y}$?

Answer: in our clustering context, $\delta$ should measure the clustering ability difference

Kullback-Leibler divergence of clustering ability between both f and $g(\cdot; \mu)^3$

$$ \delta_{\text{KL}}(f, g(\cdot; \mu)) = \int_{\mathcal{T}} p_f(t) \ln \frac{p_f(t)}{p_g(t; \mu)} dt $$

where

■ $p_f$: pdf of proba. of classification $t(f) = (t_i(f))^n_{i=1}$, with $t_i(f) = (t_{ik}(f))^K_{k=1}$

■ $p_g(\cdot; \mu)$: pdf of proba. of classif. $t(g) = (t_i(g))^n_{i=1}$, with $t_i(g) = (t_{ik}(g))^K_{k=1}$

■ $\mathcal{T} = \{t: t = (t_1, \dots, t_{K-1}), t_k > 0, \sum_k t_k < 1\}$

<footnote n="3">³$\bar{p}_f$ is the reference measure</footnote>
<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>

<figure><img src="image_71.png" /></figure>

G reduced to a unique distribution

■ A natural requirement: $p_g(\cdot; \mu)$ and $g$ should be linked by a one-to-one mapping

■ Currently not true since rotations and/or translations are possible

■ It means: for one distribution $f$, there is a unique optimal distribution $g(\cdot; \mu)$

■ Additional constraints on $g(\cdot; \mu)$: $d_Y = K - 1$, $\mu_K = 0$, $\mu_{kh} = 0$ ($h > k$), $\mu_{kk} \ge 0$
<figure><img src="image_72.png" /></figure>

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

Estimating the Gaussian centers (pitch)

■ The Kullback-Leibler divergence $\delta_{KL}$ has generally no closed-form

■ Estimate it by the following consistent (in $S$) Monte-Carlo expression

$$ \hat{\delta}_{KL}(f, g(\cdot; \mu)) = \frac{1}{S} \sum_{s=1}^{S} \ln p_g(t^{(s)}; \mu) + \text{cst} \underbrace{\quad L(\mu; t)}_{\quad} $$

with $S$ independent draws of conditional proba. $t = (t^{(1)}, \dots, t^{(S)})$ from $p_f$

■ It is the normalized (observed-data) log-likelihood function of a mixture model

■ But, by construction, all the conditional probabilities are fixed in this mixture

■ Thus, just maximize the normalized complete-data log-likelihood $L_{comp}(\mu; t)$:

■ $K = 2$: this maximization is straightforward

■ $K > 2$: use a standard **Quasi-Newton algorithm** with different random initializations, for avoiding possible local optima
<figure><img src="image_77.png" /></figure>

<figure><img src="image_78.png" /></figure>

<figure><img src="image_79.png" /></figure>

<figure><img src="image_80.png" /></figure>

From a multivariate to a bivariate Gaussian mixture

■ g is defined on $\mathbb{R}^{K-1}$ but it is more convenient to be on $\mathbb{R}^2$

■ Just apply LDA on g to display this distribution on its most discriminative map

■ It leads to the bivariate spherical Gaussian mixture $\tilde{g}$

$$ \tilde{g}(\tilde{y}; \tilde{\mu}) = \sum_{k=1}^{K} \pi_k \phi_2(\tilde{y}; \tilde{\mu}_k, l), $$

where $\tilde{y} \in \mathbb{R}^2$, $\tilde{\mu} = (\tilde{\mu}_1, \dots, \tilde{\mu}_K)$ and $\tilde{\mu}_k \in \mathbb{R}^2$

■ Use the % of inertia of LDA to measure the quality of the mapping from g to $\tilde{g}$

Remark

If $\mathcal{X} = \mathbb{R}^d$ and f is a Gaussian mixture with isotropic covariance matrices, then the proposed mapping is equivalent to applying a LDA to the centers of f
<figure><img src="image_81.png" /></figure>

<figure><img src="image_82.png" /></figure>

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /></figure>

<figure><img src="image_85.png" /></figure>

Overall accuracy of the mapping between $f$ and $\tilde{g}$

Use the following difference between the normalized entropies of $f$ and $\tilde{g}$

$$ \delta_E(f, \tilde{g}) = -\frac{1}{\ln K} \sum_{k=1}^{K} \left\{ \int_X t_k(\mathbf{x}; f) \ln t_k(\mathbf{x}; f) d\mathbf{x} - \int_{\mathbb{R}^2} t_k(\tilde{\mathbf{y}}; \tilde{g}) \ln t_k(\tilde{\mathbf{y}}; \tilde{g}) d\tilde{\mathbf{y}} \right\} $$

* Such a quantity can be easily estimated by empirical values

* Its meaning is particularly relevant:

* $\delta_E(f, \tilde{g}) \approx 0$: the component overlap conveyed by $\tilde{g}$ (over $f$) is accurate

* $\delta_E(f, \tilde{g}) \approx 1$: $\tilde{g}$ strongly underestimates the component overlap of $f$

* $\delta_E(f, \tilde{g}) \approx -1$: $\tilde{g}$ strongly overestimates the component overlap of $f$

$\delta_E(f, \tilde{g})$ permits to evaluate the bias of the visualization
<figure><img src="image_86.png" /></figure>

<figure><img src="image_87.png" /></figure>

<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /></figure>

Drawing $\tilde{g}$

- **Cluster centers:** the locations of $\tilde{\mu}_1, \dots, \tilde{\mu}_K$ are materialized by vectors

- **Cluster spread:** the 95% confidence level displayed by a black border

- **Cluster overlap:** iso-probability curves of the MAP classification for different levels

- **Mapping accuracy:** $\delta_E(f, \tilde{g})$ and also % of inertia by axis
<figure><img src="image_91.png" /></figure>

<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /></figure>

<figure><img src="image_94.png" /></figure>

<figure><img src="image_95.png" /></figure>

## Outline

1. Clustering: from modeling to visualizing

2. Mapping clusters as spherical Gaussians

3. Numerical illustrations for complex data

4. Axes interpretability

5. Discussion
House of Representatives Congressmen: data<fnref n="4" /> and model

■ Votes of the $n = 435$ U.S. Congressmen on the $d_X = 16$ key votes

■ Categorical data: for each vote, three levels are considered (yea, nay, ?)

■ Data clustered by a mixture of product of multinomial distributions [Goodman 1974]

■ $K = 4$ selected by BIC [Schwarz 1974]

■ Use the R package `Rmixmod` [Lebret et al. 2015]

■ Complex output: 435 individual memberships, $192 = 16 \times 3 \times 4$ parameters

<footnote n="4">⁴[Schlimmer (1987)]</footnote>
# House of Representatives Congressmen: standard MCA visualization

First map of the MCA (R package FactoMineR [Lê et al. 2008]): *difficult to interpret*

<figure><img src="image_96.png" /></figure>
<figure><img src="image_97.png" /></figure>

<figure><img src="image_98.png" /></figure>

<figure><img src="image_99.png" /></figure>

<figure><img src="image_100.png" /></figure>

<figure><img src="image_101.png" /></figure>

## House of Representatives Congressmen: Gaussian visualization

<figure><img src="image_102.png" /></figure>

Mapping of $f$ on this graph is accurate because $\delta_E(f, \tilde{g}) = 0.01$
Contraceptive method choice: data<fnref n="5" /> and model

- Subset of the 1987 National Indonesia Contraceptive Prevalence Survey

- Mixed data: 1473 Indian women with two numerical variables (age and number of children) and eight categorical variables (education level, education level of the husband, religion, occupation, occupation of the husband, standard-of-living index and media exposure)

- Clustered by a mixture *f* assuming that variables are independent within components

- Model selection is done by the BIC criterion which detects six components

- Use the R package *Rmixmod* [Lebret et al. 2015]

<footnote n="5">⁵[Lim et al. 2000]</footnote>
Contraceptive method choice: estimated parameters

<table><caption>Table: Parameters of the continuous variables for the Contraceptive method choice.</caption><thead><tr><th rowspan="2"></th><th colspan="2">Age</th><th colspan="2">Number of children</th></tr><tr><th>Mean</th><th>Variance</th><th>Mean</th><th>Variance</th></tr></thead><tbody><tr><td>Component 1</td><td>35</td><td>30</td><td>4</td><td>4</td></tr><tr><td>Component 2</td><td>35</td><td>22</td><td>3</td><td>2</td></tr><tr><td>Component 3</td><td>40</td><td>42</td><td>5</td><td>9</td></tr><tr><td>Component 4</td><td>25</td><td>10</td><td>1</td><td>1</td></tr><tr><td>Component 5</td><td>24</td><td>13</td><td>2</td><td>1</td></tr><tr><td>Component 6</td><td>45</td><td>7</td><td>5</td><td>8</td></tr></tbody></table>

<table><caption>Table: Modes of the categorical variables for the Contraceptive method choice.</caption><thead><tr><th></th><th>education level</th><th>husband's education level</th><th>religion</th><th>occupation</th><th>husband's occupation</th><th>standard-of-living index</th><th>media exposure</th></tr></thead><tbody><tr><td>Component 1</td><td>3</td><td>3</td><td>2</td><td>2</td><td>3</td><td>4</td><td>1</td></tr><tr><td>Component 2</td><td>4</td><td>4</td><td>2</td><td>2</td><td>1</td><td>4</td><td>1</td></tr><tr><td>Component 3</td><td>1</td><td>2</td><td>2</td><td>2</td><td>3</td><td>3</td><td>1</td></tr><tr><td>Component 4</td><td>4</td><td>4</td><td>2</td><td>2</td><td>1</td><td>4</td><td>1</td></tr><tr><td>Component 5</td><td>3</td><td>3</td><td>2</td><td>2</td><td>3</td><td>3</td><td>1</td></tr><tr><td>Component 6</td><td>4</td><td>4</td><td>2</td><td>2</td><td>1</td><td>4</td><td>1</td></tr></tbody></table>
<figure><img src="image_103.png" /></figure>

<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /></figure>

<figure><img src="image_106.png" /></figure>

<figure><img src="image_107.png" /></figure>

Contraceptive method choice: Gaussian visualization

<figure><img src="image_108.png" /></figure>

Mapping of *f* on this graph is accurate because $\delta_E(f, \tilde{g}) = 0.04$
<figure><img src="image_109.png" /></figure>

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>

Bike sharing system: data<fnref n="6" /> and model

* Station occupancy data collected over the course of one month on the bike sharing system in Paris

* Data collected over 5 weeks, between February, 24 and March, 30, 2014, on 1189 bike stations

* Functional data: station status information (available bikes/docks) downloaded every hour from the open-data APIs of JCDecaux company

* The final data set contains 1189 loading profiles, one per station, sampled at 1448 time points

* Model: profiles of the stations were projected on a basis of 25 Fourier functions

* Model-based clustering of these functional data [Bouveyron et al. 2015] with the R package FunFEM [Bouveyron 2015]

* Retain 10 clusters

<footnote n="6">⁶[Bouveyron et al. (2015)]</footnote>
<figure><img src="image_114.png" /></figure>

<figure><img src="image_115.png" /></figure>

<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>

Bike sharing system: cluster of curves visualization

<figure><img src="image_119.png" /></figure>
<figure><img src="image_120.png" /></figure>

<figure><img src="image_121.png" /></figure>

<figure><img src="image_122.png" /></figure>

<figure><img src="image_123.png" /></figure>

<figure><img src="image_124.png" /></figure>

<figure><img src="image_125.png" /></figure>

Mapping of *f* on this graph is accurate because $\delta_E(f, \tilde{g}) = -0.03$
French political blogosphere: data<fnref n="7" /> and model

* **Not oriented network data:** a single day snapshot of over 1100 political blogs automatically extracted the October, 14th, 2006 and manually classified by the "Observatoire Présidentielle" project.

* Nodes represent hostnames (= a set of pages) and edges represent hyperlinks between different hostnames

* Gather different communities organization due to the existence of several political parties and commentators

* **Assumption:** authors of these blogs tend to link, by political affinities, blogs with similar political positions

* Use the graph clustering via Erdös–Rényi mixture proposed by [Zanghi et al. 2008]

* Use the R package mixer

* As proposed by these authors, we consider $K = 6$ components

<footnote n="7">⁷[Zanghi et al. 2008]</footnote>
French political blogosphere: confusion matrix

<table><caption>Table: Confusion matrix between the component memberships and the political party memberships.</caption><thead><tr><th></th><th>Comp. 1</th><th>Comp. 2</th><th>Comp. 3</th><th>Comp. 4</th><th>Comp. 5</th><th>Comp. 6</th></tr></thead><tbody><tr><td>Cap21</td><td>2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Commentateurs Analystes</td><td>10</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>FN - MNR - MPF</td><td>2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Les Verts</td><td>7</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>PCF - LCR</td><td>7</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>PS</td><td>31</td><td>0</td><td>0</td><td>0</td><td>26</td><td>0</td></tr><tr><td>Parti Radical de Gauche</td><td>11</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>UDF</td><td>1</td><td>1</td><td>0</td><td>30</td><td>0</td><td>0</td></tr><tr><td>UMP</td><td>2</td><td>25</td><td>11</td><td>2</td><td>0</td><td>0</td></tr><tr><td>liberaux</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>24</td></tr></tbody></table>
<figure><img src="image_126.png" /></figure>

<figure><img src="image_127.png" /></figure>

<figure><img src="image_128.png" /></figure>

<figure><img src="image_129.png" /></figure>

<figure><img src="image_130.png" /></figure>

French political blogosphere: Gaussian visualization

<figure><img src="image_131.png" /></figure>

The graph slightly over-represents the component overlaps: $\delta_E(f, \tilde{g}) = -0.216$
<figure><img src="image_132.png" /></figure>

<figure><img src="image_133.png" /></figure>

<figure><img src="image_134.png" /></figure>

<figure><img src="image_135.png" /></figure>

<figure><img src="image_136.png" /></figure>

## Outline

1. Clustering: from modeling to visualizing

2. Mapping clusters as spherical Gaussians

3. Numerical illustrations for complex data

4. Axes interpretability

5. Discussion
<figure><img src="image_137.png" /></figure>

<figure><img src="image_138.png" /></figure>

<figure><img src="image_139.png" /></figure>

<figure><img src="image_140.png" /></figure>

<figure><img src="image_141.png" /></figure>

Prostate cancer data<fnref n="10" />: definition

- **Individuals**: 506 patients with prostatic cancer grouped on clinical criteria into two Stages 3 and 4 of the disease

- **Variables**: *d* = 12 pre-trial variates were measured on each patient, composed by eight continuous variables (age, weight, systolic blood pressure, diastolic blood pressure, serum haemoglobin, size of primary tumour, index of tumour stage and histolic grade, serum prostatic acid phosphatase) and four categorical variables with various numbers of levels (performance rating, cardiovascular disease history, electrocardiogram code, bone metastases)

- **Some missing data**: 62 missing values (≈ 1%)

Use the Rmixtcomp package<fnref n="9" /> to estimate the mixture...

<footnote n="9">⁹https://cran.r-project.org/web/packages/RMixtComp/index.html</footnote>

<footnote n="10">¹⁰Byar DP, Green SB (1980): Bulletin Cancer, Paris 67:477-488</footnote>
<figure><img src="image_142.png" /></figure>

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /></figure>

<figure><img src="image_145.png" /></figure>

<figure><img src="image_146.png" /></figure>

Prostate cancer data: visualization<fnref n="11" />

<figure><img src="image_147.png" /></figure>

<footnote n="11">¹¹Biernacki, C., Marbac, M., Vandewalle, V. Gaussian-Based Visualization of Gaussian and Non-Gaussian-Based Clustering. J Classif (2020).</footnote>
<figure><img src="image_148.png" /></figure>

<figure><img src="image_149.png" /></figure>

<figure><img src="image_150.png" /></figure>

<figure><img src="image_151.png" /></figure>

<figure><img src="image_152.png" /></figure>

Prostate cancer data: axis 1 interpretation

<figure><img src="image_153.png" /></figure>

■ Features SG (and AP) are the most discriminant features

■ Thus axis 1 can be interpreted as a "SG/AP axis"
<figure><img src="image_154.png" /></figure>

<figure><img src="image_155.png" /></figure>

<figure><img src="image_156.png" /></figure>

<figure><img src="image_157.png" /></figure>

<figure><img src="image_158.png" /></figure>

Prostate cancer data: validation of axis 1 interpretation

In the Rmixcomp package, there exists an estimation of the feature importance (which is independent of the Gaussian-based visualization)

<figure><img src="image_159.png" /></figure>

We remark that Rmixcomp feature importance and visualization interpretation are
in accordance...
<figure><img src="image_160.png" /></figure>

<figure><img src="image_161.png" /></figure>

<figure><img src="image_162.png" /></figure>

<figure><img src="image_163.png" /></figure>

<figure><img src="image_164.png" /></figure>

## Outline

1. Clustering: from modeling to visualizing

2. Mapping clusters as spherical Gaussians

3. Numerical illustrations for complex data

4. Axes interpretability

5. Discussion
<figure><img src="image_165.png" /></figure>

<figure><img src="image_166.png" /></figure>

<figure><img src="image_167.png" /></figure>

<figure><img src="image_168.png" /></figure>

<figure><img src="image_169.png" /></figure>

Conclusion

■ Generic method for visualizing the results of any model-based clustering result

■ Very easy to understand output since “Gaussian-like”

■ Permit visualization for any type of data, because only based on proba. of classif.

■ Can be used after any existing package of model-based clustering

■ The overall accuracy of the visualization is also provided

R package on the CRAN: ClusVis
<figure><img src="image_170.png" /></figure>

<figure><img src="image_171.png" /></figure>

<figure><img src="image_172.png" /></figure>

<figure><img src="image_173.png" /></figure>

<figure><img src="image_174.png" /></figure>

Extensions

■ Possibility to explore other pdf visualizations than Gaussians

■ However, should keep in mind simple visualizations are targeted

■ Possibility to compare/select pdf candidates through $\delta_{KL}$ or $\delta_E$
<figure><img src="image_175.png" /></figure>

<figure><img src="image_176.png" /></figure>

<figure><img src="image_177.png" /></figure>

<figure><img src="image_178.png" /></figure>

<figure><img src="image_179.png" /></figure>

## About individual visualization

* Theoretically, impossible to obtain individual visualization from pdf visualization

* However, we can propose a pseudo scatter plot of x as follows

$$x_i \mapsto t_i(f) = t_i(g) \xrightarrow{\text{bijection}} y_i \in \mathbb{R}^{K-1} \xrightarrow{\text{LDA}} \tilde{y}_i \in \mathbb{R}^2$$

* $\tilde{y}$ allows only to visualize the classification position of x

* Example for the congressmen data set

<figure><img src="image_180.png" /></figure>

* **Caution:** do not overlay pdf and individual plots since $\tilde{y} = (\tilde{y}_1, \dots, \tilde{y}_n)$ is not necessarily drawn from a Gaussian mixture