<figure><img src="image_1.png" /></figure>

HAL
open science

# Guided Clustering Variational Autoencoder

Courrier Violaine, Biernacki Christophe

► To cite this version:

Courrier Violaine, Biernacki Christophe. Guided Clustering Variational Autoencoder. JdS'2025- 56es Journées de Statistique, Jun 2025, Marseille, France. hal-05442682

HAL Id: hal-05442682

https://inria.hal.science/hal-05442682v1

Submitted on 5 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Guided Clustering Variational Autoencoder

Violaine Courrier, Christophe Biernacki

Withings & Inria Lille

May 26, 2025
## 1. Model

1.1 Context

1.2 Modeling

1.3 ELBO

## 2. Examples

2.1 Withings' sleep dataset

2.2 MNIST/SVHN
* Clustering in practice: rarely purely unsupervised

* Iterative human-in-the-loop refinements & visual analytics are common.

* Formal guidance methods aim to make clustering more relevant:

* Input-Driven: Variable/view selection targets the most statistically distinct partition.

* Constraints: Incorporate known relationships (must-link/cannot-link).

* Goal-Oriented Clustering (GOC): Steer towards externally useful partitions.
Lack of context-awareness: same variables, different goals.

<figure><img src="image_3.png" /></figure>

* Our goal: *x* clustering serving analytical goals (not just best statistical separation).

* Our Method: Guided by a guiding variable *y*, it helps identify and favor a target partition of *x*. This can reveal important but less dominant structures in *x*.

<figure><img src="image_4.png" /><figcaption>Figure: Illustration 2 dimensions.</figcaption></figure>
To respond to the problem, we construct clusters that are generative of the variables of the input but also a **guiding variable** $y$.

<figure><img src="image_5.png" /><figcaption>Figure: Illustration of the model. Diamond-shaped nodes denote latent variables, while round ones denote observations.</figcaption></figure>

* Goal: Insightful $x$ clustering, helped by $y$ guidance.

* Asymmetry of guidance: $y$ informs model learning; model autonomous on new $x$.
* Choose a cluster $c$: $p_{\pi}(c) = Cat(c; \pi)$.

* Generate a latent vector $z$ conditioned on the cluster $c$: $p_{\mu_c, \sigma_c}(z|c) = N(z; \mu_c, \sigma_c^2 I)$.

<figure><img src="image_6.png" /><figcaption>Figure: A graphical representation of the GMM.</figcaption></figure>

Here $\pi = (\pi_1, ..., \pi_K) \in [0, 1]^K$, $\sum_{c=1}^K \pi_c = 1$, $\mu_c$ and $\sigma_c^2$ the mean and the diagonal covariance of the multivariate normal distribution corresponding to cluster $c$.
* Generate the variable $x$ from the latent vector $z$: $p_{\theta_x}(x|z) = \mathcal{N}(x; f_{\theta_x}(z), I)$.

* Generate the variable $y$ from the latent vector $z$: $p_{\theta_y}(y|z) = \mathcal{N}(y; f_{\theta_y}(z), I)$.

<figure><img src="image_7.png" /><figcaption>Figure: A graphical representation of the decoders.</figcaption></figure>

Here $I$ is the identity matrix, and $f_{\theta_a}(z)$ with $a \in \{x, y\}$ are networks with input $z$ and parametrized by $\theta_a$.
* Probabilistic model is a joint distribution of hidden variables $(z, c)$ and observed variables $(x, y)$:

$$p_{\Theta}(z, c, x, y)$$

* Inference about the unknown is performed through the posterior:

$$p(z, c|x, y) = \frac{p_{\Theta}(z, c, x, y)}{p(x, y)}$$

* Denominator not tractable → approximate posterior inference
<figure><img src="image_8.png" /><figcaption>Figure: Illustration inspired by [Blei et al., ].</figcaption></figure>

* Fit the variational parameters $\phi$ to be close in KL to the exact posterior.
* Mean-field approximation: $q_{\phi}(z, c|x) = q_{\phi}(z|x)q(c|x)$, with

$$q_{\phi}(z|x) = \mathcal{N}(z; \tilde{\mu}, \tilde{\sigma}^2 I)$$
$$[\tilde{\mu}, \log \tilde{\sigma}] = g_{\phi}(x)$$

<figure><img src="image_9.png" /><figcaption>Figure: A graphical representation of the inference model.</figcaption></figure>
The ELBO is a lower bound of the observed likelihood:

$$ 
\begin{aligned}
& \mathrm{KL}[q_{\phi}(z, c|x) || p_{\Theta}(z, c|x, y)] \\
& = -\mathbb{E}_{q_{\phi}(z, c|x)} \left[ \log \frac{p_{\Theta}(z, c|x, y)}{q_{\phi}(z, c|x)} \right] \\
& = -\mathbb{E}_{q_{\phi}(z, c|x)} \left[ \log \frac{p_{\Theta}(z, c, x, y)}{q_{\phi}(z, c|x)} \right] + \log p(x, y) \\
& \Rightarrow \log p(x, y) \geq \mathbb{E}_{q_{\phi}(z, c|x)} \left[ \log \frac{p_{\Theta}(z, c, x, y)}{q_{\phi}(z, c|x)} \right].
\end{aligned}
$$

The objective is to maximize the ELBO:

$$ 
\begin{aligned}
& \arg\max_{\Theta, \phi} \mathrm{ELBO}(x, y) \\
& = \arg\max_{\Theta, \phi} \mathbb{E}_{q_{\phi}(z, c|x)} \left[ \log \frac{p_{\Theta}(z, c, x, y)}{q_{\phi}(z, c|x)} \right].
\end{aligned}
$$
$$ \text{ELBO}(x, y) = \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\Theta}(y|z)] + \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\Theta}(x|z)] - \text{KL}[q_{\phi}(z, c|x)||p(z|c)p(c)] $$
$$ \text{ELBO}(x, y) = \underbrace{\mathbb{E}_{q_{\phi}(z|x)}[\log p_{\Theta}(y|z)]}_{\text{Reconstruction of } y} + \underbrace{\mathbb{E}_{q_{\phi}(z|x)}[\log p_{\Theta}(x|z)]}_{\text{Reconstruction of } x} - \underbrace{\text{KL}[q_{\phi}(z, c|x) || p(z|c)p(c)]}_{\text{Structure latent space}} $$
Example 1: Withings' sleep dataset guided by AHI

• In *x*, data available with the watch and the sleep analyzer

• Sleep duration (+ light/deep sleep duration)

• Number of sleep interruptions

• BMI

• Age

• In *y*, the AHI score, available only with the sleep analyzer

⇒ Goal: Find clusters that are interpretable in the sense of their AHI score without using the information as input.
# Impact of the guiding variable

* Comparison of the clustering with and without the AHI as a guiding variable.

<figure><img src="image_10.png" /><figcaption>Figure: Kernel density plots comparing per-feature distributions across the two clusters in the test dataset. (Up) Model without any guiding variable. (Down) Model with AHI as the guiding variable.</figcaption></figure>
Impact of the guiding variable

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

Figure: Kernel density plot illustrating the distribution of AHI values across each cluster in the test dataset. (Left) Model without any guiding variable. (Right) Model with AHI as the guiding variable.
Example 2: MNIST/SVHN

• x: SVHN images

• y: MNIST images

<figure><img src="image_13.png" /></figure>

⇒ Goal: Find clusters that are generative both of SVHN and MNIST images, therefore that have a meaning in both domains.
Visualization of the clusters during the training

<figure><img src="image_14.png" /><figcaption>Figure: t-SNE visualisations at different epoch during the training of 5,000 training images.</figcaption></figure>
Comparison of the performance on standard benchmarks

<table><caption>Table: ACC on standard clustering benchmarks.</caption><thead><tr><th>MODEL</th><th>ACC</th></tr></thead><tbody><tr><td colspan="2"><em>Clustering models with image-specific transformations</em></td></tr><tr><td>DTI K-MEANS [MONNIER ET AL., 2020]</td><td>44.5%</td></tr><tr><td>SCAE [KOSIOREK ET AL., 2019]</td><td>55.3%</td></tr><tr><td>DTI GMM [MONNIER ET AL., 2020]</td><td>57.4%</td></tr><tr><td>ACOL-GAR [KILINC AND UYSAL, 2018]</td><td><strong>76.8%</strong></td></tr><tr><td colspan="2"><em>Clustering models with domain-agnostic designs</em></td></tr><tr><td>GMM [DEMPSTER ET AL., 1977]</td><td>11.6%</td></tr><tr><td>DEC [XIE ET AL., 2016]</td><td>11.9%</td></tr><tr><td>K-MEANS [MACQUEEN, 1967]</td><td>12.2%</td></tr><tr><td>VADE [JIANG ET AL., 2017]</td><td>30.8%</td></tr><tr><td>MFCVAE [FALCK ET AL., 2021]</td><td>56.3%</td></tr><tr><td>IMSAT [HU ET AL., 2017]</td><td>57.3%</td></tr><tr><td><strong>GCVAE</strong></td><td><strong>64.2%</strong></td></tr></tbody></table>
* The generative aspect - key for the interpretation

<figure><img src="image_15.png" /><figcaption>Figure: Four examples of generated SVHN images (left) and a generated MNIST images (right) of five clusters.</figcaption></figure>
* **Contextually guided segmentation:** introduction of a guiding variable *y*, preserving the full richness of the original dataset *x*.

* **Adaptability to different contexts:** adapt to different contexts by changing the guiding variable *y*, allowing for flexibility to adjust the clustering objective to a new context.

* **Generative architecture for interpretability:** can generate both input data *x* and guiding variables *y* along with the clustering, enhancing cluster interpretability.

* **Inference independence:** *y* is used only in the generative model, leaving the inference process to rely solely on *x*. The model remains applicable even when *y* is unavailable at prediction time.
**Dynamic clustering:** Extend the model for time series data to allow individual cluster assignments to evolve over time.

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>
<figure><img src="image_21.png" /></figure>

Blei, D., Ranganath, R., and Mohamed, S.
Variational Inference: Foundations and Modern Methods.

<figure><img src="image_22.png" /></figure>

Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977).
Maximum Likelihood from Incomplete Data Via the EM Algorithm.
*Journal of the Royal Statistical Society Series B: Statistical Methodology*, 39(1):1–22.

<figure><img src="image_23.png" /></figure>

Falck, F., Zhang, H., Willetts, M., Nicholson, G., Yau, C., and Holmes, C. (2021).
Multi-Facet Clustering Variational Autoencoders.
arXiv:2106.05241 [stat].

<figure><img src="image_24.png" /></figure>

Hu, W., Miyato, T., Tokui, S., Matsumoto, E., and Sugiyama, M. (2017).
Learning Discrete Representations via Information Maximizing Self-Augmented Training.
arXiv:1702.08720 [stat].
<figure><img src="image_25.png" /></figure>

Jiang, Z., Zheng, Y., Tan, H., Tang, B., and Zhou, H. (2017).

Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering.

arXiv:1611.05148 [cs].

<figure><img src="image_26.png" /></figure>

Kilinc, O. and Uysal, I. (2018).

Learning Latent Representations in Neural Networks for Clustering through Pseudo Supervision and Graph-based Activity Regularization.

arXiv:1802.03063 [cs].

<figure><img src="image_27.png" /></figure>

Kosiorek, A. R., Sabour, S., Teh, Y. W., and Hinton, G. E. (2019).

Stacked Capsule Autoencoders.

arXiv:1906.06818 [stat].

<figure><img src="image_28.png" /></figure>

MacQueen, J. (1967).

Some methods for classification and analysis of multivariate observations.

In *Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics*, volume 5.1, pages 281–298. University of California Press.
<figure><img src="image_29.png" /></figure>

Monnier, T., Groueix, T., and Aubry, M. (2020).
Deep Transformation-Invariant Clustering.
arXiv:2006.11132 [cs].

<figure><img src="image_30.png" /></figure>

Xie, J., Girshick, R., and Farhadi, A. (2016).
Unsupervised Deep Embedding for Clustering Analysis.
arXiv:1511.06335 [cs].
Thank you !
$$
\begin{align*}
\mathrm{ELBO}(x, y) &= \mathbb{E}_{q_{\phi}(z, c|x)} \left[ \log \frac{p_{\Theta}(z, c, x, y)}{q_{\phi}(z, c|x)} \right] \\
&= \int \sum_{c=1}^{K} q_{\phi}(z|x) q(c|x) \log \frac{p_{\Theta}(y|z) p_{\Theta}(x|z) p_{\Theta}(z|c) p_{\Theta}(c)}{q_{\phi}(z|x) q(c|x)} dz \\
&= \int q_{\phi}(z|x) \log \frac{p_{\Theta}(y|z) p_{\Theta}(x|z) p_{\Theta}(z)}{q_{\phi}(z|x)} dz - \int q_{\phi}(z|x) \mathrm{KL}[q(c|x) || p_{\Theta}(c|x)] dz
\end{align*}
$$

The 1st term does not depend on $c$ and the 2nd term is non-negative.

⇒ Maximizing the lower bound ELBO with respect to $q(c|x)$ requires that $\mathrm{KL}[q(c|x)||p_{\Theta}(c|x)] = 0$. With $\nu$ a constant, we have:

$$ \frac{q(c|x)}{p_{\Theta}(c|x)} = \nu. $$
Since $\sum_c q(c|x) = 1$ and $\sum_c p_\Theta(c|z) = 1$, we have:

$$ \frac{q(c|x)}{p_\Theta(c|z)} = 1. $$

Taking the expectation on both sides, we can obtain:

$$ q(c|x) = \mathbb{E}_{q_{\phi}(z|x)}[p_{\Theta}(c|z)]. $$

We will approximate $q(c|x)$ using the SGVB estimator:

$$ q(c|x) = \mathbb{E}_{q_{\phi}(z|x)}[p(c|z)] \approx \frac{1}{L} \sum_{l=1}^{L} \frac{p_{\Theta}(z^{(l)}|c)p_{\Theta}(c)}{\sum_{c'} p_{\Theta}(z^{(l)}|c')p_{\Theta}(c')}. $$
$$
\begin{aligned}
\text{ELBO}(x, y) = & -\frac{1}{L} \sum_{l=1}^{L} ||y - f_{\theta_y}(z^{(l)})||_2^2 - \frac{1}{L} \sum_{l=1}^{L} ||x - f_{\theta_x}(z^{(l)})||_2^2 \\
& - \frac{1}{2} \sum_{c=1}^{K} q(c|x) \sum_{j=1}^{J} \left( \log \sigma_{cj}^2 + \frac{\tilde{\sigma}_j^2}{\sigma_{cj}^2} + \frac{(\tilde{\mu}_j - \mu_{cj})^2}{\sigma_{cj}^2} \right) \\
& + \sum_{c=1}^{K} q(c|x) \log \pi_c + \frac{1}{2} \sum_{j=1}^{J} (1 + \log \tilde{\sigma}_j^2) - \sum_{c=1}^{K} q(c|x) \log q(c|x)
\end{aligned}
$$

with $z^{(l)} \sim \mathcal{N}(\tilde{\mu}, \tilde{\sigma}^2 I)$, and $[\tilde{\mu}, \log \tilde{\sigma}^2] = g_\phi(x)$. L is the number of Monte Carlo samples in the Stochastic Gradient Variational Bayes estimator and J is the dimension of z.
* Pre-training is used to initialize GMM parameters $(\mu, \sigma)$.

* We introduce the weight $\alpha$ in the loss function to balance the reconstruction of $x$, $y$ and the structure of the latent space without the clusters

$$ \text{ELBO}_{\alpha, \beta}(x, y) = \alpha \times \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\Theta}(y|z)] + \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\Theta}(x|z)] \\ - \beta \times \text{KL}[q_{\phi}(z|x) || \mathcal{N}(z; 0_J, I)] $$

with $0_J$ a vector null of dimension $J$.

* By prioritizing the reconstruction of $y$ with $\alpha > 1$, we encourage the model to align the latent space with the guiding variable.
* CNNs for both the encoder and the decoder of SVHN, and a MLP for the decoder of MNIST.

* Latent space dimension: 20.

* $\beta = 3$.

* Pretraining: $\alpha = 10$, learning rate = 0.0001.

* Training: learning rate of 0.001 for the parameters of the encoder and decoders, and 0.0001 for the parameters of the GMM, and set the number of clusters to 10.
* MLPs in the encoder and both the decoders.

* Latent space dimension: 5.

* $\beta = 0.03$.

* Pretraining: $\alpha = 10$, learning rate = 0.00005.

* Training: learning rate of 0.0001 for the parameters of the encoder and decoders, and 0.00001 for the parameters of the GMM, and set the number of clusters to 2.
# Withings' dataset details

* 50,000 individuals - 1 night per individual

* Recorded by the Withings Sleep Analyzer<fnref n="1" />

* Equal number of users across the three categories based on the AHI

<table><caption>Table: Descriptive statistics of the variables</caption><thead><tr><td>Variable (unit)</td><td>Range</td><td>Mean</td><td>Std. Dev.</td></tr></thead><tbody><tr><td>sleep_duration (seconds)</td><td>14880 - 36000</td><td>26224</td><td>4224</td></tr><tr><td>light_sleep_duration (seconds)</td><td>3600 - 31860</td><td>15747</td><td>4651</td></tr><tr><td>deep_sleep_duration (seconds)</td><td>3600 - 32220</td><td>10472</td><td>4022</td></tr><tr><td>nb_sleep_interruptions</td><td>0 - 20</td><td>2.74</td><td>2.34</td></tr><tr><td>avg_night_hr (bpm)</td><td>40 - 111</td><td>62.49</td><td>8.57</td></tr><tr><td>bmi (kg/m²)</td><td>16 - 50</td><td>27.53</td><td>5.14</td></tr><tr><td>age (years)</td><td>18 - 80</td><td>50</td><td>12.67</td></tr><tr><td>apnea_hypopnea_index</td><td>0 - 40</td><td>18.14</td><td>13.47</td></tr></tbody></table>

<footnote n="1">¹https://www.withings.com/us/en/sleep</footnote>