<figure><img src="image_1.png" /></figure>

HAL
open science

# Guided Clustering Variational Autoencoder

Violaine Courrier, Christophe Biernacki

► To cite this version:

Violaine Courrier, Christophe Biernacki. Guided Clustering Variational Autoencoder. 2025. hal-05442698

HAL Id: hal-05442698

https://inria.hal.science/hal-05442698v1

Submitted on 5 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Guided Clustering Variational Autoencoder

Violaine Courrier, Christophe Biernacki

Withings & Inria Lille

March 11, 2025
## 1. Model

1.1 Context

1.2 Modelisation

1.3 ELBO

## 2. Examples

2.1 Withings' sleep dataset

2.2 MNIST/SVHN
Lack of context-awareness : same variables, different goals.

<figure><img src="image_3.png" /></figure>

Strategies :

• Variable selection: determine which features are relevant to each context.

• Feature weighting: assign different weights based on their importance to a context.

• Constraint-based clustering: "these points must be in the same cluster".

• Add a context variable: add an extra dimension that describes the context.
To respond to the problem, we want clusters that are generative of the variables of the input but also a guiding variable $y$.

<figure><img src="image_4.png" /><figcaption>Figure: Illustration of the model. Diamond-shaped nodes denote latent variables, while round ones denote observations</figcaption></figure>
* Choose a cluster c: $p_{\pi}(c) = \text{Cat}(c; \pi)$.

* Generate a latent vector z conditioned on the cluster c: $p_{\mu_c, \sigma_c}(z|c) = \mathcal{N}(z; \mu_c, \sigma_c^2 I)$.

<figure><img src="image_5.png" /><figcaption>Figure: A graphical representation of the GMM.</figcaption></figure>

Here $\pi = (\pi_1, ..., \pi_K) \in [0, 1]^K$, $\sum_{c=1}^K \pi_c = 1$, $\mu_c$ and $\sigma_c^2$ the mean and the diagonal covariance of the multivariate normal distribution corresponding to cluster c.
* Generate the variable $x$ from the latent vector $z$: $p_{\theta_x}(x|z) = \mathcal{N}(x; f_{\theta_x}(z), I)$.

* Generate the variable $y$ from the latent vector $z$: $p_{\theta_y}(y|z) = \mathcal{N}(y; f_{\theta_y}(z), I)$.

<figure><img src="image_6.png" /><figcaption>Figure: A graphical representation of the decoders.</figcaption></figure>

Here $I$ is an identity matrix, and $f_{\theta_a}(z)$ with $a \in \{x, y\}$ are networks with input $z$ and parametrized by $\theta_a$.
* Probabilistic model is a joint distribution of hidden variables $(z, c)$ and observed variables $(x, y)$

$$p_{\Theta}(z, c, x, y)$$

* Inference about the unknown is through the posterior

$$p(z, c|x, y) = \frac{p_{\Theta}(z, c, x, y)}{p(x, y)}$$

* Denominator not tractable → approximate posterior inference
<figure><img src="image_7.png" /><figcaption>Figure: Illustration inspired by [Blei et al., ].</figcaption></figure>

• Fit the variational parameters $\phi$ to be close in KL to the exact posterior.
* Mean-field approximation: $q_{\phi}(z, c|x) = q_{\phi}(z|x)q(c|x)$ with

$$q_{\phi}(z|x) = \mathcal{N}(z; \tilde{\mu}, \tilde{\sigma}^2 I)$$
$$[\tilde{\mu}, \log \tilde{\sigma}] = g_{\phi}(x)$$

<figure><img src="image_8.png" /><figcaption>Figure: A graphical representation of the inference model.</figcaption></figure>
$$
\begin{align*}
& \mathrm{KL}[q_{\phi}(z, c|x) || p_{\Theta}(z, c|x, y)] \\
&= -\mathbb{E}_{q_{\phi}(z, c|x)} \left[ \log \frac{p_{\Theta}(z, c|x, y)}{q_{\phi}(z, c|x)} \right] \\
&= -\mathbb{E}_{q_{\phi}(z, c|x)} \left[ \log \frac{p_{\Theta}(z, c, x, y)}{q_{\phi}(z, c|x)} \right] + \log p(x, y) \\
&\Rightarrow \log p(x, y) \geq \mathbb{E}_{q_{\phi}(z, c|x)} \left[ \log \frac{p_{\Theta}(z, c, x, y)}{q_{\phi}(z, c|x)} \right].
\end{align*}
$$

The objective is to maximize the ELBO:

$$
\begin{align*}
& \arg\max_{\Theta, \phi} \mathrm{ELBO}(x, y) \\
&= \arg\max_{\Theta, \phi} \mathbb{E}_{q_{\phi}(z, c|x)} \left[ \log \frac{p_{\Theta}(z, c, x, y)}{q_{\phi}(z, c|x)} \right].
\end{align*}
$$
$$ \text{ELBO}(x, y) = \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\Theta}(y|z)] + \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\Theta}(x|z)] - \text{KL}[q_{\phi}(z, c|x)||p(z|c)p(c)] $$
$$ \text{ELBO}(x, y) = \underbrace{\mathbb{E}_{q_{\phi}(z|x)}[\log p_{\Theta}(y|z)]}_{\text{Reconstruction of } y} + \underbrace{\mathbb{E}_{q_{\phi}(z|x)}[\log p_{\Theta}(x|z)]}_{\text{Reconstruction of } x} - \underbrace{\text{KL}[q_{\phi}(z, c|x) || p(z|c)p(c)]}_{\text{Structure latent space}} $$
* In *x*, data available with the watch and the sleep analyzer

* Sleep duration (+ light/deep sleep duration)

* Number of sleep interruptions

* BMI

* Age

* In *y*, the AHI score, available only with the sleep analyzer

⇒ Goal: Find clusters that are interpretable in the sense of their AHI score without using the information as input.
# Impact of the guiding variable

* Comparison of the clustering with and without the AHI as a guiding variable.

<figure><img src="image_9.png" /><figcaption>Figure: Kernel density plots comparing per-feature distributions across the two clusters in the test dataset. (Up) Model without any guiding variable. (Down) Model with AHI as the guiding variable.</figcaption></figure>
Impact of the guiding variable

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

Figure: Kernel density plot illustrating the distribution of AHI values across each cluster in the test dataset. (Left) Model without any guiding variable. (Right) Model with AHI as the guiding variable.
• x: SVHN images

• y: MNIST images

<figure><img src="image_12.png" /></figure>

⇒ Goal: Find clusters that are generative both of SVHN and MNIST images, therefore that have a meaning in both domains.
Visualization of the clusters during the training

<figure><img src="image_13.png" /><figcaption>Figure: t-SNE visualisations at different epoch during the training of 5,000 training images.</figcaption></figure>
# Comparison of the performance on standard benchmarks

<table><caption>Table: ACC on standard clustering benchmarks.</caption><thead><tr><th>MODEL</th><th>ACC</th></tr></thead><tbody><tr><td colspan="2"><em>Clustering models with image-specific transformations</em></td></tr><tr><td>DTI K-MEANS [MONNIER ET AL., 2020]</td><td>44.5%</td></tr><tr><td>SCAE [KOSIOREK ET AL., 2019]</td><td>55.3%</td></tr><tr><td>DTI GMM [MONNIER ET AL., 2020]</td><td>57.4%</td></tr><tr><td>ACOL-GAR [KILINC AND UYSAL, 2018]</td><td>76.8%</td></tr><tr><td colspan="2"><em>Clustering models with domain-agnostic designs</em></td></tr><tr><td>GMM [DEMPSTER ET AL., 1977]</td><td>11.6%</td></tr><tr><td>DEC [XIE ET AL., 2016]</td><td>11.9%</td></tr><tr><td>K-MEANS [MACQUEEN, 1967]</td><td>12.2%</td></tr><tr><td>VADE [JIANG ET AL., 2017]</td><td>30.8%</td></tr><tr><td>MFCVAE [FALCK ET AL., 2021]</td><td>56.3%</td></tr><tr><td>IMSAT [HU ET AL., 2017]</td><td>57.3%</td></tr><tr><td><strong>GCVAE</strong></td><td>64.2%</td></tr></tbody></table>
* The generative aspect - key for the interpretation

<figure><img src="image_14.png" /><figcaption>Figure: Four examples of generated SVHN images (left) and a generated MNIST images (right) of five clusters.</figcaption></figure>
* Contextually guided segmentation: introduction of a guiding variable *y*, preserving the full richness of the original dataset *x*.

* Adaptability to different contexts: adapt to different contexts by changing the guiding variable *y*, allowing for flexibility to adjust the clustering objective to a new context.

* Flexible generative architecture: can generate both input data *x* and guiding variables *y* along with the clustering, enhancing cluster interpretability.

* Inference independence: *y* is used only in the generative model, leaving the inference process to rely solely on *x*. The model remains applicable even when *y* is unavailable at prediction time.
<figure><img src="image_15.png" /></figure>

Blei, D., Ranganath, R., and Mohamed, S.

Variational Inference: Foundations and Modern Methods.

<figure><img src="image_16.png" /></figure>

Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977).

Maximum Likelihood from Incomplete Data Via the EM Algorithm.

Journal of the Royal Statistical Society Series B: Statistical Methodology, 39(1):1–22.

<figure><img src="image_17.png" /></figure>

Falck, F., Zhang, H., Willetts, M., Nicholson, G., Yau, C., and Holmes, C. (2021).

Multi-Facet Clustering Variational Autoencoders.

arXiv:2106.05241 [stat].

<figure><img src="image_18.png" /></figure>

Hu, W., Miyato, T., Tokui, S., Matsumoto, E., and Sugiyama, M. (2017).

Learning Discrete Representations via Information Maximizing Self-Augmented Training.

arXiv:1702.08720 [stat].
<figure><img src="image_19.png" /></figure>

Jiang, Z., Zheng, Y., Tan, H., Tang, B., and Zhou, H. (2017).

Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering.

arXiv:1611.05148 [cs].

<figure><img src="image_20.png" /></figure>

Kilinc, O. and Uysal, I. (2018).

Learning Latent Representations in Neural Networks for Clustering through Pseudo Supervision and
Graph-based Activity Regularization.

arXiv:1802.03063 [cs].

<figure><img src="image_21.png" /></figure>

Kosiorek, A. R., Sabour, S., Teh, Y. W., and Hinton, G. E. (2019).

Stacked Capsule Autoencoders.

arXiv:1906.06818 [stat].

<figure><img src="image_22.png" /></figure>

MacQueen, J. (1967).

Some methods for classification and analysis of multivariate observations.

In *Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics*, volume 5.1, pages 281–298. University of California Press.
<figure><img src="image_23.png" /></figure>

Monnier, T., Groueix, T., and Aubry, M. (2020).
Deep Transformation-Invariant Clustering.
arXiv:2006.11132 [cs].

<figure><img src="image_24.png" /></figure>

Xie, J., Girshick, R., and Farhadi, A. (2016).
Unsupervised Deep Embedding for Clustering Analysis.
arXiv:1511.06335 [cs].
Thank you !