# Guided Clustering Variational Autoencoder (GCVA): A Predictive Clustering Approach with a VAE

Violaine Courrier, Christophe Biernacki

► To cite this version:

Violaine Courrier, Christophe Biernacki. Guided Clustering Variational Autoencoder (GCVA): A Predictive Clustering Approach with a VAE. 15-th Scientific Meeting Classification and Data Analysis Group (CLADAG), Sep 2025, Naples, Italy. 10.1080/10618600.2021.2000872.url: . hal-05442684

HAL Id: hal-05442684

https://inria.hal.science/hal-05442684v1

Submitted on 5 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Guided Clustering Variational Autoencoder (GCVA)
A Predictive Clustering Approach with a VAE

Violaine Courrier¹, Christophe Biernacki²

¹Withings, ¹,²Inria, ¹,²Université de Lille, ¹,²CNRS

15-th Scientific Meeting Classification and Data Analysis Group (CLADAG)
Naples 8-10 September 2025
1. Motivation for predictive clustering

2. The proposed predictive clustering approach: GCVA

3. Estimation of the model parameters/weights

4. Real numerical example

5. Final remarks
Some (classical) methods aim to make clustering more relevant.

* **Human methods:** Iterative human-in-the-loop refinements & visual analytics are common.

* **Formal guidance methods:**

* Input-Driven: Variable/view selection targets the most statistically distinct partition.

* Constraints: Incorporate known relationships (must-link/cannot-link).

* Goal-Oriented Clustering (GOC) / Predictive Clustering: Steer towards externally useful partitions (useful for a given predefined goal).
Classic vs. predictive clustering

* Classic clustering: $p(c|x)$ for discovering clusters $c$ within the dataset $x$.

* Predictive clustering: $p(c|x)$ and $p(y|c)$ for discovering $c$ within $x$, guided by a guiding variable $y$ which helps identify and favor a target partition of $x$.

<figure><img src="image_2.png" /></figure>

This can reveal important but less dominant structures in $x$.

<figure><img src="image_3.png" /></figure>

Figure: Illustration of $x$ in 2 dimensions: If $y$ is related to Axis 2, the corresponding less dominant cluster $c$ can be preferred.
* **Two-step baselines** (cluster inputs $x$, then train predictors for $y$): the resulting clusters are often irrelevant to the predictive task.

* **Predictive Clustering Trees (PCTs)**: greedy, top-down partitioning of the feature space, which can lead to suboptimal and unstable cluster structures.

* **Simultaneous Clustering-Regression Models**: assume simple linear relationships within clusters and can struggle with the curse of dimensionality in high-dimensional data.

## Our model

Use a variational information bottleneck to simultaneously learn a non-linear latent space where clusters correspond to true predictive patterns, rather than being the result of greedy algorithms or restrictive linear assumptions.

Tomaž Stepišnik et al. (2021). "Oblique predictive clustering trees". In: *Knowledge-Based Systems* 227, p. 107228

Matthieu Marbac et al. (2022). "Simultaneous semi-parametric estimation of clustering and regression". In: *Journal of Computational and Graphical Statistics* 31.2, pp. 477-485. DOI: 10.1080/10618600.2021.2000872. URL: https://inria.hal.science/hal-03090573
1. Motivation for predictive clustering

2. The proposed predictive clustering approach: GCVA

3. Estimation of the model parameters/weights

4. Real numerical example

5. Final remarks
<figure><img src="image_4.png" /></figure>

Our model is composed by standard statistical "building blocks":

* Part 1: A **Variational Encoder** that projects **x** into a latent space (variable **z**),

* Part 2: A **Gaussian Mixture Model** that structures this latent space into clusters,

* Part 3: A **Predictor** that aligns these clusters by predicting the guide **y**.
Inference model (Part 1): Variational Encoder

Since $p(z, c|x)$ is unknown/untractable, it is classical to use the mean-field approximation:

$$q_{\phi}(z, c|x) = q_{\phi}(z|x) \underbrace{q(c|x)}_{\text{see later}} \text{ with } q_{\phi}(z|x) = \mathcal{N}(z; \tilde{\mu}, \tilde{\sigma}^2 I) \text{ and } [\tilde{\mu}, \log \tilde{\sigma}] = g_{\phi}(x)$$

<figure><img src="image_5.png" /><figcaption>Figure: A graphical representation of the Variational Encoder.</figcaption></figure>

<footnote>Rémi Boutin et al. (May 2024). “The Deep Latent Position Block Model”. In: sciencesconf.org. Recueil des résumés longs de la SFdS 2024. Bordeaux, France, pp. 1218–1226. URL: https://hal.science/hal-04910241</footnote>
# Generative model (Part 2): Gaussian Mixture Model (GMM)

* Choose a cluster c: $p_{\pi}(c) = \text{Cat}(c; \pi)$.

* Generate a latent vector z conditioned on the cluster c: $p_{\mu_c, \sigma_c}(z|c) = N(z; \mu_c, \sigma_c^2 I)$.

<figure><img src="image_6.png" /><figcaption>Figure: A graphical representation of the GMM.</figcaption></figure>

* $\pi = (\pi_1, ..., \pi_K) \in [0, 1]^K, \sum_{c=1}^K \pi_c = 1,$

* $\mu_c$ and $\sigma_c^2$ the mean and the variance of the multivariate normal distribution corresponding to cluster c.

<footnote>Geoffrey J McLachlan et al. (2019). "Finite mixture models". In: *Annual review of statistics and its application* 6.1, pp. 355-378</footnote>
# Generative model (Part 3): Predictive network (it is a Decoder also)

* Generate the variable *y* from the latent vector *z*: $p_{\theta_y}(y|z) = \mathcal{N}(y; f_{\theta_y}(z), I)$.

<figure><img src="image_7.png" /><figcaption>Figure: A graphical representation of the Predictor.</figcaption></figure>

* *I* is the identity matrix,

* $f_{\theta_y}(z)$ is a network with input *z* and parametrized by $\theta_y$.

<footnote>Diederik P Kingma et al. (2022). *Auto-Encoding Variational Bayes*. arXiv: 1312.6114 [stat.ML]. URL: https://arxiv.org/abs/1312.6114</footnote>
1. Motivation for predictive clustering

2. The proposed predictive clustering approach: GCVA

3. Estimation of the model parameters/weights

4. Real numerical example

5. Final remarks
In a standard Variational Autoencoder (replace $\tilde{y}$ by $\tilde{x}$), the observed log-likelihood $\ell(\theta, \phi; x)$ is untractable thus the ELBO is maximized with $\theta = (\theta_y, \pi, \mu_c, \sigma_c^2)$

$$ \mathrm{ELBO}_{\theta,\phi}(x) = \mathbb{E}_{q_{\phi}(z,c|x)}[\log p_{\theta}(x|z, c)] $$

subject to the constraint $\mathrm{KL}[q_{\phi}(z, c|x) || p_{\theta}(z, c)] < \epsilon_{\beta}$ with $\beta$ a regularisation coefficient. Therefore, with $\beta$ a regularisation coefficient, it is equivalent to maximize

$$ \max_{\theta,\phi} \mathbb{E}_{q_{\phi}(z,c|x)}[\log p_{\theta}(x|z, c)] - \beta \times \mathrm{KL}[q_{\phi}(z, c|x) || p_{\theta}(z, c)]. $$

According to Surendran et al. (2024), the sequence of parameter estimates $(\theta_n, \phi_n)$ is expected to converge to a stationary point of the objective function at a rate of $O(\log n/\sqrt{n})$, where $n$ is the number of optimization iterations.

<footnote>Irina Higgins et al. (2017). "beta-vae: Learning basic visual concepts with a constrained variational framework". In: International conference on learning representations</footnote>

<footnote>Sobiran Surendran et al. (2024). "Theoretical convergence guarantees for variational autoencoders". In: arXiv preprint arXiv:2410.16750</footnote>
In our proposed VAE, we replace the output $\tilde{x}$ by $\tilde{y}$ (the input $x$ is unchanged), leading to the following adaptation of the ELBO

$$ \mathrm{ELBO}_{\theta, \phi}(x, \underline{y}) = \mathbb{E}_{q_{\phi}(z, c|x)}[\log p_{\theta}(\underline{y} \mid z, c)] $$

subject to the (unchanged) constraint $\mathrm{KL}[q_{\phi}(z, c|x) || p_{\theta}(z, c)] < \epsilon_{\beta}$.

Therefore, with $\beta$ a regularisation coefficient, it is equivalent to maximize

$$ \max_{\theta, \phi} \mathbb{E}_{q_{\phi}(z, c|x)}[\log p_{\theta}(\underline{y} \mid z, c)] - \beta \times \mathrm{KL}[q_{\phi}(z, c|x) || p_{\theta}(z, c)]. $$
* Coded in Pytorch

* Hyperparameters:
  * Architectures of the encoder and decoder: MLPs, CNNs, RNNs...
  * Size of the networks
  * Number of clusters
  * Regularization parameter $\beta$
  * Optimizer (Adam)
  * Learning rate
  * Choosing $q(c|x)$
* $\beta$ parameter is a well-established tool for controlling VAE behavior.

* In our case, a small $\beta$ ($\beta < 1$) is empirically optimal because it creates a more tractable optimization problem:

* Prevents the complex GMM-based KL term from destabilizing training,

* Avoids posterior collapse by reducing regularization pressure.

* Strategies to find a suitable $\beta$ value:

* Standard hyperparameter tuning (grid search, Bayesian optimization).

* KL annealing: $\beta$ is gradually increased during training. This allows the model to first learn to reconstruct the data before the regularization is applied, preventing common training issues.

<footnote>Christopher P. Burgess et al. (2018). *Understanding disentangling in $\beta$-VAE*. arXiv: 1804.03599 [stat.ML]. URL: https://arxiv.org/abs/1804.03599</footnote>

<footnote>Hao Fu et al. (2019). *Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing*. arXiv: 1903.10145 [cs.LG]. URL: https://arxiv.org/abs/1903.10145</footnote>
**CAUTION**

If the networks (encoders) are too complex:

* Information on clusters *c* is "hidden" in millions of network parameters,

* The latent space becomes a simple passthrough.

By keeping the networks simple, the model is forced to encode information in the clusters.

To learn robust clusters, we design our networks to act as an information bottleneck.

* By intentionally constraining networks capacity by reducing their sizes, we force the model to find an efficient representation.

* It must compress the most important data features into the latent space *z*.

<footnote>Naftali Tishby et al. (2000). *The information bottleneck method*. arXiv: physics/0004057 [physics.data-an]. URL: https://arxiv.org/abs/physics/0004057</footnote>
Strategies to balance the VAE and GMM components in literature:

*   **GMVAE:** Pretrain VAE without a GMM, then a single learning rate is sufficient to fine-tune the VAE and train the GMM parameters simultaneously.

*   **VaDE:** The VAE parameters are updated via backpropagation using a single learning rate. The GMM parameters are re-estimated at each epoch using the posteriors computed from the model's forward pass (analogous to the M-step of an EM algo.).

Our model:

*   **We use differential learning rates,** which is a more general, modern technique for training complex, end-to-end models.

Nat Dilokthanakul et al. (2017). *Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders*. arXiv: 1611.02648 [cs.LG]. URL: https://arxiv.org/abs/1611.02648

Zhuxi Jiang et al. (June 2017). *Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering*. arXiv:1611.05148 [cs]. URL: http://arxiv.org/abs/1611.05148 (visited on 03/03/2025)
We can establish that $q(c|x) = \mathbb{E}_{q_{\phi}(z|x)}[p_{\theta}(c|z)]$ and then we approximate $q(c|x)$ using the Stochastic Gradient Variational Bayes (SGVB) estimator:

$$q(c|x) \approx \frac{1}{L} \sum_{l=1}^{L} \frac{p_{\theta}(z^{(l)}|c)p_{\theta}(c)}{\sum_{c'} p_{\theta}(z^{(l)}|c')p_{\theta}(c')}. $$
1. Motivation for predictive clustering

2. The proposed predictive clustering approach: GCVA

3. Estimation of the model parameters/weights

4. Real numerical example

5. Final remarks
* x: Street View House Numbers images (SVHN)

* y: handwritten digits images (MNIST)

Globally, thousands of images.

<figure><img src="image_8.png" /></figure>

⇒ Goal: Find clusters in SVHN that are generative of MNIST images.

<footnote>Yuge Shi et al. (2019). “Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models”. In: *Advances in Neural Information Processing Systems*. Ed. by H. Wallach et al. Vol. 32. Curran Associates, Inc. URL: https://proceedings.neurips.cc/paper_files/paper/2019/file/0ae775a8cb3b499ad1fca944e6f5c836-Paper.pdf</footnote>
* CNN for encoding SVHN and MLP (Multilayer Perceptron) for decoding MNIST.

* Latent space dimension: 20.

* $\hat{\beta} = 0.1$.

* Pretraining:

* Learning rate for the parameters of the encoder and decoder: 0.0001,

* Epochs: 5.

* Training:

* Learning rate for the parameters of the encoder and decoder: 0.0005,

* Learning rate for the parameters of the GMM: 0.0005,

* Number of clusters: 10,

* Epochs: 50,

* Training time: 14 minutes.
Visualization of the clusters during the training

<figure><img src="image_9.png" /><figcaption>Figure: t-SNE visualisations at different epochs during the training of 5,000 training images.</figcaption></figure>
Comparison of the performance on standard benchmarks

<table><caption>Table: ACC (accuracy) on standard clustering benchmarks.</caption><thead><tr><th>MODEL</th><th>ACC</th></tr></thead><tbody><tr><td colspan="2"><em>Clustering models with image-specific transformations</em></td></tr><tr><td>DTI K-MEANS - MONNIER ET AL. (2020)</td><td>44.5%</td></tr><tr><td>SCAE - KOSIOREK ET AL. (2019)</td><td>55.3%</td></tr><tr><td>DTI GMM - MONNIER ET AL. (2020)</td><td>57.4%</td></tr><tr><td>ACOL-GAR - KILINC ET AL. (2018)</td><td><strong>76.8%</strong></td></tr><tr><td colspan="2"><em>Clustering models with domain-agnostic designs</em></td></tr><tr><td>GMM - DEMPSTER ET AL. (1977)</td><td>11.6%</td></tr><tr><td>DEC - XIE ET AL. (2016)</td><td>11.9%</td></tr><tr><td>K-MEANS - MACQUEEN (1967)</td><td>12.2%</td></tr><tr><td>VADE - JIANG ET AL. (2017)</td><td>30.8%</td></tr><tr><td>MFCVAE - FALCK ET AL. (2021)</td><td>56.3%</td></tr><tr><td>IMSAT - HU ET AL. (2017)</td><td>57.3%</td></tr><tr><td><strong>GCVAE</strong></td><td><strong>62.1%</strong></td></tr></tbody></table>
1. Motivation for predictive clustering

2. The proposed predictive clustering approach: GCVA

3. Estimation of the model parameters/weights

4. Real numerical example

5. Final remarks
* **Contextually guided clustering**: introduction of a guiding variable *y*, preserving the full richness of the original dataset *x*.

* **Adaptability to different contexts**: adapt to different contexts by changing the guiding variable *y*, allowing for flexibility to adjust the clustering objective to a new context.

* **Adaptability to different types of variables**: the proposed modeling is valid for different types of variables in *x* and in *y*.

* **Inference independence**: *y* is used only in the generative model, leaving the inference process to rely solely on *x*. The model remains applicable even when *y* is unavailable at prediction time.
* Release the model as an open-source package.

* Apply the model using data from connected medical devices (Withings Cie datasets).

* Enhance model interpretability by training a separate decoder post-hoc. This decoder will map the learned latent space back to the input space ($z \mapsto x$), enabling the visualization of cluster prototypes and the generation of synthetic data.
Thank you !