Disentanglement of Variations with Multimodal Generative Modeling

## Abstract

Abstract Multimodal data are prevalent across various domains, and learning robust representations of such data is paramount to enhancing generation quality and downstream task performance.
To handle heterogeneity and interconnections among different modalities, recent multimodal generative models extract shared and private (modality-specific) information with two separate variables.
Despite attempts to enforce disentanglement between these two variables, these methods struggle with challenging datasets where the likelihood model is insufficient.
In this paper, we propose Information-disentangled Multimodal VAE (IDMVAE) to explicitly address this issue, with rigorous mutual information-based regularizations, including cross-view mutual information maximization for extracting shared variables, and a cycle-consistency style loss for redundancy removal using generative augmentations.
We further introduce diffusion models to improve the capacity of latent priors.
These newly proposed components are complementary to each other. Compared to existing approaches, IDMVAE shows a clean separation between shared and private information, demonstrating superior generation quality and semantic coherence on challenging datasets.

## 1 Introduction

Most real-world data are inherently multimodal or multi-view(^1^11We use “modality” and “view” interchangeably as they both appear in the literature.). Videos contain both visual scenes and sounds (Zhao et al., 2018; Owens & Efros, 2018; Chen et al., 2020a; Gong et al., 2023; Kim et al., 2024); robots can see and feel via sensors (Lee et al., 2019); images are often accompanied by captions (Radford et al., 2021; Jia et al., 2021); and heterogeneous human, animal, and environmental data are collected for health improvements (Adisasmito et al., 2022).
In addition to these naturally occurring data, synthetic multi-view data constructed from semantically similar input components or via augmentation are also widely used to learn useful representations for downstream tasks (Veličković et al., 2019; Chen et al., 2020b; Caron et al., 2020; Tian et al., 2020a; Bardes et al., 2022). Despite the abundance of such data, leveraging them is nontrivial even with naturally aligned modalities due to their diversity and complex correlations. Therefore, a core challenge is to integrate information across views to learn universal, transferrable representations.

Variational autoencoders (VAEs, Kingma & Welling, 2014) and their multmodal extensions have emerged as a powerful paradigm to tackle this problem (Wang et al., 2016; Suzuki et al., 2016). They can extract useful shared information in data with missing modalities (Wu & Goodman, 2018) and noise (Shi et al., 2021). While early works have assumed that a single latent space can capture all relevant information and data variations (Shi et al., 2019; Sutter et al., 2021), recent approaches have recognized the existence of both shared and modality-specific (private) information in real-world datasets (Daunhawer et al., 2022; Lee & Pavlovic, 2021; Palumbo et al., 2023; 2024). However, modeling shared and private components naturally exposes a challenge:

> How can we achieve maximal disentanglement between shared and private variables so that learned representations are complete and non-redundant?

Without a clean separation, shared information leaks into private encodings and vice versa, causing weak coherence across modalities, wasted model capacity, and inadequate generative quality.

#### Our Contributions.

We propose Information-disentangled Multimodal VAE (IDMVAE), a novel framework for unsupervised multimodal representation learning hailing from theoretical stringency and practical performance.
(1) Different from prior works which use capacity-based (Wang et al., 2016) or shortcut-preventing (Palumbo et al., 2023) heuristics, we employ mutual information (MI)-based regularizations to ensure disentanglement. In particular, we use cross-view MI to extract common factors that likelihood models fail to fully capture, thereby enhancing cross-modal coherence. Additionally, a cycle-consistency-style loss removes redundancy between shared and private latents using samples generated by the model itself, eliminating the need for domain-specific augmentations.
(2) To overcome limitations of simple Gaussian priors, we leverage diffusion-based priors (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) that capture the richness of multimodal latent spaces, leading to greater representational capacities.
(3) Across multiple complex datasets spanning image, text and multi-omics data, IDMVAE performs consistently better than state-of-the-art methods in terms of cross-modal generation and coherence, showing a synergy between MI-based objectives and diffusion priors, which leads to improved performance.

## 2 Method

Figure: Figure 1: Our approach with two modalities.
Refer to caption: figures/Schematic_noHat.png

Given a set of $M$ modalities ${\mathbf{X}}=\{{\mathbf{x}}_{1},\dots,{\mathbf{x}}_{M}\}$, we would like to learn a factorized latent space for each modality $m$, separating information shared across modalities, captured by latent variable $\mathbf{z}$, from modality-specific private information, captured by ${\mathbf{w}}_{m}$.
As illustrated in Figure [1](https://arxiv.org/html/2509.23548v1#S2.F1), we assume that, $\forall m$, ${\mathbf{x}}_{m}$ is generated by ${\mathbf{z}}_{m}$ and ${\mathbf{w}}_{m}$ jointly with $p({\mathbf{x}}_{m}|{\mathbf{z}}_{m},{\mathbf{w}}_{m})$, and the latent variables have independent priors $p({\mathbf{z}},\{{\mathbf{w}}_{m}\}_{m=1}^{M})=p({\mathbf{z}})\prod_{m=1}^{M}p({\mathbf{w}}_{m})$.
We perform variational inference and parameterize the approximate posteriors with a factorized form: $q({\mathbf{z}},{\mathbf{w}}_{m}|{\mathbf{x}}_{m})=q({\mathbf{z}}|{\mathbf{x}}_{m})\cdot q({\mathbf{w}}_{m}|{\mathbf{x}}_{m})$. Learned $q({\mathbf{z}}|{\mathbf{x}}_{m})$ and $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m})$ provide representations of original inputs which can be used in downstream tasks.
Our method consists of three major components.

### 2.1 Likelihood modeling with Multimodal VAE

We begin with the Evidence Lower Bound (ELBO) of the MMVAE+ (Palumbo et al., 2023) model as our foundation. Let $q(\mathbf{z}|{\mathbf{x}}_{m})$ be the posterior of shared variable derived from modality $m$ (with distribution parameters modeled by an encoder), and $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m})$ be the posterior of private variable from modality $m$ (modeled by another encoder). One can define a global posterior of ${\mathbf{z}}$ by aggregating information from view-specific posteriors, e.g., using the mixture-of-experts (MoE) scheme $q(\mathbf{z}|{\mathbf{X}})=\frac{1}{M}\sum_{m=1}^{M}q(\mathbf{z}|{\mathbf{x}}_{m})$. Together with generative distributions $p({\mathbf{x}}_{m}|{\mathbf{z}},{\mathbf{w}}_{m})$ modeled by decoders of each view, an ELBO for ${\mathbf{X}}$ can be derived with variational inference (Kingma & Welling, 2014), which involves reconstructing ${\mathbf{x}}_{n}$ using samples of $q({\mathbf{w}}_{n}|{\mathbf{x}}_{n})$ and $q(\mathbf{z}|{\mathbf{x}}_{m})$, for $n,m=1,\dots,M$.

Since the posterior $q({\mathbf{w}}_{n}|{\mathbf{x}}_{n})$ is dependent on ${\mathbf{x}}_{n}$ and potentially retains shared information, to keep decoder $p({\mathbf{x}}_{n}|{\mathbf{z}},{\mathbf{w}}_{n})$ from taking the “shortcut” to use leaked shared information, MMVAE+ uses sample of $\tilde{{\mathbf{w}}}_{n}$ from an auxiliary prior $r(\tilde{{\mathbf{w}}}_{n})$ instead
for cross-view reconstruction, where ${\mathbf{z}}$ is derived from another view $m\neq n$; avoiding such shortcut enforces shared information to come from $q({\mathbf{z}}|{\mathbf{x}}_{m})$.
This design leads to the following minimizing objective:

$$ $\displaystyle\mathcal{L}_{\text{MMVAE+}}=-\frac{1}{M}\sum_{m=1}^{M}\mathbb{E}_{\begin{subarray}{c}{\mathbf{z}}\sim q({\mathbf{z}}|{\mathbf{x}}_{m})\\ {\mathbf{w}}_{m}\sim q({\mathbf{w}}_{m}|{\mathbf{x}}_{m})\\ \{\tilde{{\mathbf{w}}}_{n}\sim r(\tilde{{\mathbf{w}}}_{n})\}_{n\neq m}\end{subarray}}\left[\log\left(\frac{p({\mathbf{x}}_{m}|\mathbf{z},{\mathbf{w}}_{m})p(\mathbf{z})p({\mathbf{w}}_{m})}{q(\mathbf{z}|{\mathbf{X}})q({\mathbf{w}}_{m}|{\mathbf{x}}_{m})}\prod_{n\neq m}p({\mathbf{x}}_{n}|\mathbf{z},\tilde{{\mathbf{w}}}_{n})\right)\right].$ $$

Within the expectation, $p({\mathbf{x}}_{m}|\mathbf{z},{\mathbf{w}}_{m})$ and $p({\mathbf{x}}_{n}|\mathbf{z},\tilde{{\mathbf{w}}}_{n})$ are conditional likelihood of self- and cross-reconstructions, respectively.  Palumbo et al. (2023) shows that $-\mathcal{L}_{\text{MMVAE+}}({\mathbf{x}}_{1:M})\leq\log({\mathbf{x}}_{1:M})$ remains a valid ELBO. Compared with previous multimodal VAE using MoE parameterization (Shi et al., 2019), MMVAE+ achieves superior performance for extracting shared information, although the result is somewhat sensitive to the relative capacity (dimensionality) of ${\mathbf{z}}$ and ${\mathbf{w}}$.
In the rest of this section, we will show that regularizing the generative model with mutual information (MI) is more effective at achieving disentanglement.
We emphasize that MMVAE+ is one option for multimodal VAE, and our improvement below can also be applied to other models, e.g., PoE (Wu & Goodman, 2018) or MoPoE (Sutter et al., 2021).

### 2.2 Shared Variable Extraction with Cross-view MI Maximization

While general (per-dimension) disentanglement of variations is theoretically challenging (Locatello et al., 2019), the underlying structure of our setup that inputs of different modalities share a common cause facilitate (variable-level) disentanglement of shared versus private
information.

To extract the shared information, it is natural to enforce the shared representation of modality $m$, denoted ${\mathbf{z}}_{m}$ (with distribution $q({\mathbf{z}}|{\mathbf{x}}_{m}$)) to have high mutual information (MI) with ${\mathbf{x}}_{n}$ for $n\neq m$.
Note this is partially pursued by $\mathcal{L}_{\text{MMVAE+}}$ through cross-view reconstruction. In light of the decomposition $I({\mathbf{z}}_{m},{\mathbf{w}}_{n};{\mathbf{x}}_{n})=I({\mathbf{z}}_{m};{\mathbf{x}}_{n})+I({\mathbf{w}}_{n};{\mathbf{x}}_{n}|{\mathbf{z}}_{m})$, we can maximize $I({\mathbf{z}}_{m},{\mathbf{w}}_{n};{\mathbf{x}}_{n})$ while minimizing $I({\mathbf{w}}_{n};{\mathbf{x}}_{n}|{\mathbf{z}}_{m})$ to maximize $I({\mathbf{z}}_{m};{\mathbf{x}}_{n})$. Focusing on the first term,
since $I({\mathbf{z}}_{m},{\mathbf{w}}_{n};{\mathbf{x}}_{n})=H({\mathbf{x}}_{n})-H({\mathbf{x}}_{n}|{\mathbf{z}}_{m},{\mathbf{w}}_{n})$ where the entropy $H({\mathbf{x}}_{n})$ is a constant,
minimizing the conditional entropy
$H({\mathbf{x}}_{n}|{\mathbf{z}}_{m},{\mathbf{w}}_{n})=\mathbb{E}_{{\mathbf{x}}_{n},{\mathbf{z}}_{m},{\mathbf{w}}_{n}}[-\log p({\mathbf{x}}_{n}|{\mathbf{z}}_{m},{\mathbf{w}}_{n})]$
is equivalent to maximizing conditional likelihood. However, maximizing this upper bound does not ensure maximal $I({\mathbf{z}}_{m};{\mathbf{x}}_{n})$ due to the gap $I({\mathbf{w}}_{n};{\mathbf{x}}_{n}|{\mathbf{z}}_{m})$.
Therefore, likelihood maximization alone does not ensure disentanglement.

We thus take the alternative approach to maximize $I({\mathbf{z}}_{m},{\mathbf{z}}_{n})$ which is a lower bound of $I({\mathbf{z}}_{m},{\mathbf{x}}_{n})$:
$I({\mathbf{z}}_{m};{\mathbf{x}}_{n})=I({\mathbf{z}}_{m};{\mathbf{z}}_{n},{\mathbf{x}}_{n})-I({\mathbf{z}}_{m};{\mathbf{z}}_{n}|{\mathbf{x}}_{n})=I({\mathbf{z}}_{m};{\mathbf{z}}_{n},{\mathbf{x}}_{n})=I({\mathbf{z}}_{m};{\mathbf{z}}_{n})+I({\mathbf{z}}_{m};{\mathbf{x}}_{n}|{\mathbf{z}}_{n})\geq I({\mathbf{z}}_{m};{\mathbf{z}}_{n})$,
where $I({\mathbf{z}}_{m};{\mathbf{z}}_{n}|{\mathbf{x}}_{n})=0$ in the first step due to variability of ${\mathbf{z}}_{n}$ coming from ${\mathbf{x}}_{n}$ only (Federici et al., 2020).
In this work, we use the contrastive estimate of MI (Oord et al., 2018):

$$ $\displaystyle I({\mathbf{z}}_{m};{\mathbf{z}}_{n})\approx Contrast({\mathbf{z}}_{m},{\mathbf{z}}_{n}):=\mathbb{E}_{{\mathbf{z}}_{m},{\mathbf{z}}_{n}}\log\left[\frac{\phi({\mathbf{z}}_{m},{\mathbf{z}}_{n})}{\phi({\mathbf{z}}_{m},{\mathbf{z}}_{n})+\sum_{j=1}^{k}\phi({\mathbf{z}}_{m},\bar{{\mathbf{z}}}_{n}^{j})}\right]$ (1) $$

where $\phi({\mathbf{z}}_{m},{\mathbf{z}}_{n})=\exp\left(\frac{{\mathbf{z}}_{m}^{\top}{\mathbf{z}}_{n}}{\left\lVert{\mathbf{z}}_{m}\right\rVert\cdot\left\lVert{\mathbf{z}}_{n}\right\rVert}\right)$ is the affinity function, and $\{\bar{{\mathbf{z}}}_{n}^{j}\}_{j=1}^{k}$ are $k$ negative examples randomly sampled from the minibatch not aligned with ${\mathbf{z}}_{m}$.
Since we have $M$ modalities, we compute the average of cross-modality MIs as our regularization for extracting shared information:

$$ $\displaystyle\mathcal{L}_{\text{CrossMI}}=-\frac{2}{M(M-1)}\sum\nolimits_{m<n}Contrast({\mathbf{z}}_{m},{\mathbf{z}}_{n}).$ $$

### 2.3 Disentanglement with Generative Augmentation

While equation [1](https://arxiv.org/html/2509.23548v1#S2.E1) encourages ${\mathbf{z}}$ to capture shared information across views, it does not guarantee that the learned ${\mathbf{z}}_{m}$ contains no private information which should be modeled by ${\mathbf{w}}_{m}$. Similarly, even if ${\mathbf{z}}_{m}$ contains no private information and the self-reconstruction term in $\mathcal{L}_{\text{MMVAE+}}$ encourages $({\mathbf{z}}_{m},{\mathbf{w}}_{m})$ to jointly capture all information about ${\mathbf{x}}_{m}$, the learned ${\mathbf{w}}_{m}$ can still retain shared information. Thus, we need additional regularization to remove redundancy between ${\mathbf{z}}_{m}$ and ${\mathbf{w}}_{m}$.

To motivate our method, consider the desired scenario where ${\mathbf{z}}_{m}$ and ${\mathbf{w}}_{m}$ are disentangled, so that they each can be varied independently to generate new samples of ${\mathbf{x}}_{m}$ using the decoder.
Let ${\mathbf{x}}_{m}$ and ${\mathbf{x}}_{m}^{\prime}$ be two input samples, and let $({\mathbf{z}}_{m},{\mathbf{w}}_{m})$ be a pair of samples drawn from the posteriors $q({\mathbf{z}}|{\mathbf{x}}_{m})$ and $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m})$, respectively, and similarly $({\mathbf{z}}_{m}^{\prime},{\mathbf{w}}_{m}^{\prime})$ be a pair of samples drawn from conditional posteriors for ${\mathbf{x}}_{m}^{\prime}$.
With disentanglement and a good likelihood model, a sample ${\mathbf{x}}_{m}^{+}\sim p({\mathbf{x}}_{m}|{\mathbf{z}}_{m},{\mathbf{w}}_{m}^{\prime})$ would share the same ${\mathbf{z}}$ with ${\mathbf{x}}_{m}$. In turn, when we map ${\mathbf{x}}_{m}^{+}$ back to the latent space, $q({\mathbf{z}}|{\mathbf{x}}_{m}^{+})$ and $q({\mathbf{z}}|{\mathbf{x}}_{m})$ should be similar.
Likewise, $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{+})$ and $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{\prime})$ should be similar.

More formally, assume that ${\mathbf{z}}_{m}$ is sufficient for ${\mathbf{x}}_{n}$, meaning that it captures all shared information, i.e., $I({\mathbf{z}}_{m};{\mathbf{x}}_{n})=I({\mathbf{x}}_{m};{\mathbf{x}}_{n})$ as encouraged by $\mathcal{L}_{\text{CrossMI}}$. Then in view of $I({\mathbf{z}}_{m};{\mathbf{x}}_{n})=H({\mathbf{z}}_{m})-H({\mathbf{z}}_{m}|{\mathbf{x}}_{n})$, we would like to find the minimal ${\mathbf{z}}_{m}$ (with lowest $H({\mathbf{z}}_{m})$) by minimizing

$$ $\displaystyle H({\mathbf{z}}_{m}|{\mathbf{x}}_{n})=\mathbb{E}_{{\mathbf{z}}_{m},{\mathbf{x}}_{n}}[-\log p({\mathbf{z}}_{m}|{\mathbf{x}}_{n})]\approx\mathbb{E}_{{\mathbf{X}}\sim p({\mathbf{X}}),{\mathbf{z}}_{m}\sim q({\mathbf{z}}|{\mathbf{x}}_{m})}[-\log q({\mathbf{z}}={\mathbf{z}}_{m}|{\mathbf{x}}_{n})].$ $$

Similar approaches have been used by Federici et al. (2020, symmetric KL for minimizing $I({\mathbf{x}}_{m},{\mathbf{z}}_{m}|{\mathbf{x}}_{n})$) and Tsai et al. (2019, inverse prediction) for learning minimally sufficient shared variable.
Essentially, for extracting shared information, ${\mathbf{x}}_{m}$ and ${\mathbf{x}}_{n}$ indeed constitute two views that are mutually redundant, satisfying $I({\mathbf{x}}_{m};{\mathbf{x}}_{n}|{\mathbf{z}})=0$, so that the IB principle naturally applies.

Note however, we do not have multiple natural views sharing ${\mathbf{w}}_{m}$ to carry out the above idea. This challenge motivates us to synthesize the view ${\mathbf{x}}_{m}^{+}\sim p({\mathbf{x}}_{m}|{\mathbf{z}}_{m},{\mathbf{w}}_{m}^{\prime})$, with ${\mathbf{w}}_{m}^{\prime}\sim q({\mathbf{w}}|{\mathbf{x}}_{m}^{\prime})$ from another sample ${\mathbf{x}}_{m}^{\prime}$, to reduce the redundancy of learned ${\mathbf{w}}_{m}$ by approximately minimizing $H({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{\prime})$ with the following loss

$$ $\displaystyle\mathbb{E}_{{\mathbf{x}}_{m}\sim p({\mathbf{x}}_{m}),{\mathbf{x}}_{m}^{\prime}\sim p({\mathbf{x}}_{m}),{\mathbf{z}}_{m}\sim q({\mathbf{z}}|{\mathbf{x}}_{m}),{\mathbf{w}}_{m}^{\prime}\sim q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{\prime}),{\mathbf{x}}_{m}^{+}\sim p({\mathbf{x}}_{m}|{\mathbf{z}}_{m},{\mathbf{w}}_{m}^{\prime})}[-\log q({\mathbf{w}}_{m}^{\prime}|{\mathbf{x}}_{m}^{+})].$ $$

Assuming that posteriors are parameterized Gaussians, this loss reduces to $\ell_{2}$ loss for matching means of posteriors.
In practice, we find it more stable to use a contrastive loss for matching, i.e.,

$$ $\displaystyle{\mathcal{L}}_{\text{GenAug},{\mathbf{w}}_{m}}:=-Contrast({\mathbf{w}}_{m}^{\prime\prime},{\mathbf{w}}_{m}^{\prime})\qquad\text{where}\quad{\mathbf{x}}_{m}^{+}\sim p({\mathbf{x}}_{m}|{\mathbf{z}}_{m},{\mathbf{w}}_{m}^{\prime}),\;{\mathbf{w}}_{m}^{\prime\prime}\sim q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{+}).$ $$

We show results obtained with contrastive estimation in the main paper, and provide empirical comparisons of the two implementations in Appendix [A](https://arxiv.org/html/2509.23548v1#A1).
We also define ${\mathcal{L}}_{\text{GenAug},{\mathbf{z}}_{m}}$ similarly by switching the role of ${\mathbf{z}}_{m}$ and ${\mathbf{w}}_{m}$.
The total redundancy removal regularization is defined as

$$ $\displaystyle{\mathcal{L}}_{\text{GenAug}}=\frac{1}{2M}\sum\nolimits_{m=1}^{M}\left({\mathcal{L}}_{\text{GenAug},{\mathbf{z}}_{m}}+{\mathcal{L}}_{\text{GenAug},{\mathbf{w}}_{m}}\right).$ $$

Although we do not have the reconstruction target for ${\mathbf{x}}_{m}^{+}$, matching $q({\mathbf{z}}|{\mathbf{x}}_{m}^{+})$ with $q({\mathbf{z}}|{\mathbf{x}}_{m})$, and matching $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{+})$ with $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{\prime})$ implement a form of *cycle-consistency* (Zhu et al., 2017), and provide learning signals for both the encoder and the decoder.
Previously, Bai et al. (2021) derived an ELBO of sequence data for disentangling static versus dynamic components, which involved mutual information terms based on data augmentation, similar to ${\mathcal{L}}_{\text{GenAug}}$. However, their augmentation requires strong domain knowledge (e.g., shuffling the frame order does not alter the static component, and color change applied to all frames does not alter the dynamic component).
In contrast, our augmentations require no domain knowledge and are produced by the model itself.

### 2.4 The Final IDMVAE Objective

We define our objective of Information-Disentangled Multimodal VAE (IDMVAE) as

$$ $\min\mathcal{L}_{\text{IDMVAE}}:=\mathcal{L}_{\text{MMVAE+}}+\lambda_{1}\mathcal{L}_{\text{CrossMI}}+\lambda_{2}\mathcal{L}_{\text{GenAug}}$ (2) $$

where $\lambda_{1}$ and $\lambda_{2}$ are user parameters tuned on the validation set.

#### Diffusion Priors

In most multimodal VAEs, the prior distributions are chosen to be simple and easy to sample from, e.g., Gaussian for continuous data.

However, such unstructured priors may not be ideal for representation learning, whose purpose is to discover useful structure of data for supervised downstream tasks. As an example, a representation containing rich label information most likely have a clustering structure where data of different classes are separated far apart, and will not have a uni-modal distribution like Gaussian.
We use diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) to overcome this limitation by parameterizing $p({\mathbf{z}})$ as a denoising process started with pure noise. To naturally introduce diffusion models into our loss,we decompose the KL divergence inside $\mathcal{L}_{\text{MMVAE+}}$ (which we minimize) as (Vahdat et al., 2021):

$$ $\displaystyle D_{\mathrm{KL}}(q({\mathbf{z}}|{\mathbf{x}})||p({\mathbf{z}}))=E_{q({\mathbf{z}}|{\mathbf{x}})}\left[\log q({\mathbf{z}}|{\mathbf{x}})\right]+E_{q({\mathbf{z}}|{\mathbf{x}})}\left[-\log p({\mathbf{z}})\right].$ (3) $$

The first term maximizes the entropy of the approximate posterior $q({\mathbf{z}}|{\mathbf{x}})$. The second term maximizes the likelihood of samples from $q({\mathbf{z}}|{\mathbf{x}})$ under $p({\mathbf{z}})$, which we model with diffusion models. We can treat ${\mathbf{z}}\sim q({\mathbf{z}}|{\mathbf{x}})$ as “data”, and destroy its structure by gradually adding noise to it, resulting in pure noise after a number of steps. With repeated applications of a denoising network, diffusion models gradually reverse the noising process, and recover the original data from pure noise. Diffusion models have well-defined ELBO objectives which *lower bound* $\log p({\mathbf{z}})$, and plugging them into equation [3](https://arxiv.org/html/2509.23548v1#S2.E3) yields *valid upper bounds* of the KL divergence. Since the latent variables are of low dimensionality, we parameterize the diffusion backward process with a simple feedforward network.
In practice, we introduce additional loss weight for $E_{q({\mathbf{z}}|{\mathbf{x}})}\left[-\log p({\mathbf{z}})\right]$, and model the mean of $q({\mathbf{z}}|{\mathbf{x}})$ with the DDPM parameterization (Ho et al., 2020).
We optimize over all modules (encoders, decoders, diffusion networks) jointly in an *end-to-end* manner.
A recent work (Palumbo et al., 2024) proposed a two-step approach which first learns the representations with MMVAE+, and then learns diffusion models in the input space, conditioned on VAE reconstructions. Note our use of diffusion model has a different motivation, and we jointly train it during representation learning.

## 3 Related Work

#### Disentanglement in VAEs.

To achieve disentangled latent representations in VAEs, researchers have commonly used mutual information (MI) based regularization, and employed various metrics to assess the results (Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2018). However, it has been shown that, without supervision or inductive bias in the model, it is theoretically challenging to recover (per-dimension) disentanglement (Locatello et al., 2019).

#### Contrastive and self-supervised learning (SSL).

SSL is a paradigm that aims at learning useful representations from large amounts of unlabeled data, by creating *artificial targets* that are generally correlated with downstream tasks. SSL is often applied to a single modality, with artificial views created based on the structures of data (Oord et al., 2018; Logeswaran & Lee, 2018; Hjelm et al., 2019; Bachman et al., 2019; Chen et al., 2020b; Caron et al., 2020; Tian et al., 2020a; Bardes et al., 2022; Zbontar et al., 2021),
as well as multimodal data (Radford et al., 2021; Jia et al., 2021; Elizalde et al., 2023)
and many methods are motivated by the classical infomax principle (Linsker, 1988) and they implement neural estimation of mutual information, with contrastive loss being the most popular variant.
Recent works have proposed theoretical interpretations of SSL and contrastive learning (Wang & Isola, 2020; Zimmermann et al., 2021; Hyvärinen et al., 2019; Tian et al., 2020b; Tosh et al., 2021; Chen et al., 2021; Zhai et al., 2024), with the focus of providing guarantees for extracting the shared variable, without considering the private variables.

A few works took private variations into consideration.
von Kügelgen et al. (2021) proposed a generative model in which the latent space is divided into “content” and “style”; importantly, data augmentations were assumed to preserve content while altering dimensions within style.  Tsai et al. (2021) studied self-supervised learning from a multi-view perspective and with the *multi-view redundancy* assumption (Chaudhuri et al., 2009; Tosh et al., 2021) that the private variable of each view contains little information for the downstream task, they focused on extracting the shared variable
with combinations of several multi-view losses.
Realizing the limitation of this assumption, Liang et al. (2023) studied the scenario where the private variables contain significant useful information, and proposed a contrastive learning algorithm for extracting it. Their algorithm required sophisticated data augmentation procedures designed for downstream task.
Lyu et al. (2022) proposed a model for understanding SSL, assuming a data generation process similar to ours. They extracted shared variable with CCA loss, and private variable by MI minimization between shared and private variables. The generation quality of their model is suboptimal, due to it being fully deterministic.

#### Information bottleneck (IB) and mutual-information regularization.

Another set of probabilistic models were motivated by the IB method (Tishby et al., 1999; Tishby & Zaslavsky, 2015; Achille & Soatto, 2018).
Alemi et al. (2017) proposed a variational IB method to extract ${\mathbf{z}}$ from ${\mathbf{x}}_{1}$ which has high MI with ${\mathbf{x}}_{2}$ (estimated with conditional likelihood), so that it captures the shared information, and at the same time has low MI with ${\mathbf{x}}_{1}$ so that it contains little nuisance factors/private information. Federici et al. (2020) leveraged the multi-view redundancy assumption that all the information ${\mathbf{x}}_{1}$ contains about an unobserved label is also contained in ${\mathbf{x}}_{2}$, and showed that that if the learned representation ${\mathbf{z}}$ is sufficient, in the sense that $I({\mathbf{x}}_{1},{\mathbf{x}}_{2}|{\mathbf{z}})=0$, then ${\mathbf{z}}$ has all the predictive power from $({\mathbf{x}}_{1},{\mathbf{x}}_{2})$ for label.
Remarkably, their objective did not involve any reconstruction paths, and the authors considered this to be an advantage, given that density modeling for high dimensional data is difficult.
Wang et al. (2025) extended Federici et al. (2020) and proposed a two-step approach to first extract shared and then the private variables with guarantees, again without generative modeling.
We argue that, with the development of powerful generative models, likelihood modeling becomes feasible and provides the additional benefit of (controllable) generation.

## 4 Experiments

We compare our method, IDMVAE, and its variant with diffusion priors, against several baselines.

MMVAE (Shi et al., 2019): uses a MoE inference network to combine information from different modalities. It only models the shared variable ${\mathbf{z}}$ with ELBO.

MoPoE-VAE (Sutter et al., 2021): uses a mixture-of-products-of-experts inference network for ${\mathbf{z}}$.

DMVAE (Lee & Pavlovic, 2021): performs PoE inference for ${\mathbf{z}}$, and models ${\mathbf{w}}_{m}$ within ELBO.

MMVAE+ (Palumbo et al., 2023): performs separation of shared versus private information with the help of auxiliary prior variables. It is a special case of IDMVAE (w.o. diffusion) with $\lambda_{1}=\lambda_{2}=0$.

DisentangledSSL (Wang et al., 2025): performs extraction of shared variable (using the method of Federici et al., 2020) and private variable in two sequential steps. It is a state-of-the-art disentanglement method without likelihood modeling, but it can only be applied to two views currently.

### 4.1 Results on PolyMNIST-Quadrant

Figure: Figure 2: PolyMNIST-Quadrant dataset. Digits (0-9) are placed in one of the four quadrants randomly. Each column contains one multimodal sample. Each modality has a different background scheme. Digit label is shared across all modalities, while quadrant label is private to each modality.
Refer to caption: figures/PolyMNIST/PolyMNIST-Quadrant--4x-train-total-sample.png

PolyMNIST (Sutter et al., 2021) is a benchmark for multimodal representation learning, consisting of MNIST (LeCun et al., 1998) digits overlaid on complex backgrounds. We make the dataset more challenging, by taking each MNIST digit and placing a 32x32
scaled version of it into one of four quadrants of a 64x64 canvas;
see Figure [2](https://arxiv.org/html/2509.23548v1#S4.F2) for an illustration. This modification introduces the private latent variable which captures the quadrant position (with ground truth label) and background for each modality, allowing for nuanced evaluation of disentanglement and generation.
Our training/validation/test sets contain 220,000/5,000/10,000 samples.
We use the deep residual network (He et al., 2015) architecture as the backbone of encoders and decoders for all methods.
The dimensionality is set to 32 for ${\mathbf{z}}$ and 128 for ${\mathbf{w}}_{m}$.

**Table 1: Latent classification on PolyMNIST-Quadrant. Accuracies are averaged over 5 modalities.**
| Model | $z\to Digit\uparrow$ | $z\to Quad\downarrow$ | $w\to Quad\uparrow$ | $w\to Digit\downarrow$ |
| --- | --- | --- | --- | --- |
| MMVAE | 0.492 | 0.798 | — | — |
| MoPoE-VAE | 0.536 | 0.751 | — | — |
| DMVAE | 0.157 | 0.254 | 0.710 | 0.179 |
| MMVAE+ | 0.382 | 0.355 | 0.999 | 0.341 |
| IDMVAE (ours) | 0.983 | 0.271 | 0.999 | 0.162 |
| – ${\mathcal{L}}_{\text{CrossMI}}$<br>($\lambda_{1}=0$) | 0.111 | 0.267 | 0.999 | 0.356 |
| – ${\mathcal{L}}_{\text{GenAug}}$<br>($\lambda_{2}=0$) | 0.977 | 0.277 | 0.999 | 0.202 |
| + Diffusion prior | 0.982 | 0.267 | 0.999 | 0.143 |

#### Latent Classification.

For evaluation, we perform linear classification on the *samples* of posterior distributions (samples reflect both mean and variance of posteriors). Multi-class logistic regression models are trained on the posterior samples of training set and applied to posterior samples of the test set.
We perform two types of classifications: (1) predicting shared label from the shared variable (${\mathbf{z}}$) and private label from the private variable (${\mathbf{w}}_{m}$), where high accuracy is better, indicating the desired variation is captured; and (2) cross-classification, where we predict shared label from ${\mathbf{w}}_{m}$ and predict quadrant label from ${\mathbf{z}}$. Ideally, with successful disentanglement, cross-classification accuracies should approach the performance of a random classifier (e.g., 10% for predicting digits from ${\mathbf{w}}_{m}$, 25% for predicting quadrants from ${\mathbf{z}}$).
We present results of different methods in Table [1](https://arxiv.org/html/2509.23548v1#S4.T1), as well as performance of our method when either ${\mathcal{L}}_{\text{CrossMI}}$ or ${\mathcal{L}}_{\text{GenAug}}$ is removed from our loss.
Clearly, our method achieves superior performance. ${\mathcal{L}}_{\text{CrossMI}}$ is critical for extracting the shared variable, and this is because the digits occupy a small number of pixels and pure likelihood modeling may ignore them.
${\mathcal{L}}_{\text{GenAug}}$ helps remove redundant information, so that cross-classification accuracy is reduced.
Adding diffusion in latent space (last row of table) leads to small gain.

**Table 2: Generative coherence, averaged over 5 views, on PolyMNIST-Quadrant. We use subscript $q$ to indicate samples from posteriors and subscript $p$ to indicate samples from priors. For generated images, digit label is determined by ${\mathbf{z}}_{s,q}$ or otherwise random (with target accuracy 10%), quadrant label is determined by ${\mathbf{w}}_{s,q}$ or otherwise random (with target accuracy 25%).**
| Model | Self Gen ($s=t$) | Cross Gen ($s\neq t$) | Uncond. |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $Gen({\mathbf{z}}_{s,q},{\mathbf{w}}_{t,p})$ | $Gen({\mathbf{z}}_{t,p},{\mathbf{w}}_{s,q})$ | $Gen({\mathbf{z}}_{s,q},{\mathbf{w}}_{t,p})$ | $Gen({\mathbf{z}}_{p},{\mathbf{w}}_{p})$ |  |  |  |  |
| $Digit\uparrow$ | $Quad\downarrow$ | $Digit\downarrow$ | $Quad\uparrow$ | $Digit\uparrow$ | $Quad\downarrow$ | $Digit\uparrow$ |  |
| MMVAE | 0.879 | 0.998 | — | — | 0.170 | 0.248 | 0.041 |
| MoPoE-VAE | 0.861 | 0.999 | — | — | 0.173 | 0.250 | 0.029 |
| DMVAE | 0.297 | 0.252 | 0.532 | 0.999 | 0.161 | 0.249 | 0.005 |
| MMVAE+ | 0.120 | 0.251 | 0.915 | 0.999 | 0.119 | 0.250 | 0.000 |
| IDMVAE (ours) | 0.898 | 0.249 | 0.162 | 0.999 | 0.881 | 0.250 | 0.070 |
| – ${\mathcal{L}}_{\text{CrossMI}}$ ($\lambda_{1}=0$) | 0.101 | 0.252 | 0.926 | 0.999 | 0.100 | 0.250 | 0.000 |
| – ${\mathcal{L}}_{\text{GenAug}}$ ($\lambda_{2}=0$) | 0.670 | 0.250 | 0.370 | 0.999 | 0.671 | 0.250 | 0.008 |
| + Diffusion prior | 0.942 | 0.251 | 0.106 | 0.999 | 0.887 | 0.251 | 0.664 |

Figure: Figure 3: Unconditional generations on PolyMNIST-Quadrant. Each row is a multimodal sample generated with a prior sample of ${\mathbf{z}}$, so images in the same row ideally have the same digit identity.
Refer to caption: figures/PolyMNIST/uncond_gen/8x5/MMVAE__09-09_T0__06-23_15__8x5.png

#### Conditional Coherence.

This metric evaluates the model’s ability to generate consistent samples across modalities. We assess this for both self-reconstruction and cross-modal generation.
Formally, we combine either posterior ${\mathbf{z}}_{s,q}\sim q({\mathbf{z}}|{\mathbf{x}}_{s})$ or prior ${\mathbf{z}}_{s,p}\sim p({\mathbf{z}})$ (using diffusion prior if available) of a modality $s$, with
the posterior ${\mathbf{w}}_{t,q}\sim q({\mathbf{w}}|{\mathbf{x}}_{t})$ or the prior
${\mathbf{w}}_{t,p}\sim p({\mathbf{w}}_{t})$ (using diffusion prior if available) of modality $t$, and apply $q({\mathbf{x}}_{t}|{\mathbf{z}},{\mathbf{w}}_{t})$ to generate a new sample of modality $t$. This sample should have the same digit label as ${\mathbf{x}}_{s}$ if posterior of ${\mathbf{z}}$ is used, and random digit label if prior is used.
Similarly, the quadrant label can be determined based on whether posterior or prior is used for ${\mathbf{w}}_{t}$.
We then use ResNet classifiers
trained on original images to predict corresponding labels of generated images, and the averaged accuracy across modalities is referred to as *coherence*.
We provide conditional generative coherence in Table [2](https://arxiv.org/html/2509.23548v1#S4.T2) (left panel for self generation where $s=t$, and middle panel for cross generation $s\neq t$); see Appendix [B.1](https://arxiv.org/html/2509.23548v1#A2.SS1) for sample generations. The results are consistent with those of latent classification, and diffusion priors significantly boost coherence.

#### Unconditional Coherence.

This metric further assesses the consistency of the shared information in unconditionally generated samples.
We first sample a shared latent code ${\mathbf{z}}_{p}\sim p({\mathbf{z}})$ (using diffusion prior when available).
For each modality $m$, we then sample an independent private ${\mathbf{w}}_{m,p}\sim p({\mathbf{w}}_{m})$ and generate a sample $\hat{{\mathbf{x}}}_{m}$ from the combined latent code $({\mathbf{z}}_{p},{\mathbf{w}}_{m,p})$. The generated multimodal sample $\{\hat{x}_{1},...,\hat{x}_{M}\}$ are then passed to their respective digit classifiers (ResNet) trained on original training images, to predict the shared label. A sample set is considered coherent if *all* classifiers agree on the same shared label. We report the percentage of coherent sets as unconditional coherence, shown in Table [2](https://arxiv.org/html/2509.23548v1#S4.T2) (right panel). Most methods obtain close to zero unconditional coherence, indicating the difficulty of matching prior and posterior distributions for latent variables.
However, with diffusion prior our method achieves significantly better coherence, thanks to its flexibility. We show generations in Figure [3](https://arxiv.org/html/2509.23548v1#S4.F3), and 2D visualizations of latent codes in Appendix [B.3](https://arxiv.org/html/2509.23548v1#A2.SS3).

Figure: Figure 4: Augmentations.
Refer to caption: figures/PolyMNIST/Ours/Normal/Conditional-Generations-Grid-m2-Qz-Qw--m2-Qw-Qz--05-19-3--07-13-02-6x6.png

#### Generative augmentation.

Recall that in ${\mathcal{L}}_{\text{GenAug}}$ we mix and match posteriors of ${\mathbf{z}}$ and ${\mathbf{w}}_{m}$ from different samples to generate new samples in modality $m$. We provide illustration of such samples from our trained model in Figure [4](https://arxiv.org/html/2509.23548v1#S4.F4). The first row and first column contain images for which we extract posterior samples of ${\mathbf{z}}$ and ${\mathbf{w}}_{m}$ respectively. And the rest of the grid contain generate images using samples of ${\mathbf{w}}_{m}$ of the corresponding row and ${\mathbf{z}}$ of the corresponding column. We observe that images in each column share the same digit, while images in each row share share the same quadrant, as desired. Generated images are of high quality, showing that we can independently vary shared and private variables to obtain controllable generations.

### 4.2 Results on CUB

The CUB-200-2011 dataset (Wah et al., 2011; Reed et al., 2016; Shi et al., 2019; Palumbo et al., 2023; 2024)
is a widely used benchmark for fine-grained visual categorization, containing 64x64 RGB images of 200 bird species. Each image is paired with 10 textual descriptions.
Following Palumbo et al. (2024), we group 22 categories of species from the 200 bird species into 8 super-categories, yielding 1-of-8 class labels for these species. Data with category label is split into training/validation/test with 80%/10%/10% portions. The rest 178 species are added to the training set for representation learning. The training/validation/test sets contain 115,240/1,280/1,360 samples, respectively.
See more details on data generation in Appendix [C](https://arxiv.org/html/2509.23548v1#A3).

For this dataset, the two modalities (image, text) share rich information about bird category, since the text describes the color of different parts of the bird. To evaluate the quality of private information, we note that the horizontal direction of the bird (with direction inferred from the original CUB attributes, see Appendix [C](https://arxiv.org/html/2509.23548v1#A3) for details) can only be inferred from the image. Therefore, we consider the direction as private label for the image modality.

**Table 3: Latent classification on CUB, using posterior means. ${\mathbf{z}}_{1}$ and ${\mathbf{w}}_{1}$ refers to image latents.**
| Model | ${\mathbf{z}}\to Cat.\uparrow$ | ${\mathbf{z}}_{1}\to Dir.\downarrow$ | ${\mathbf{w}}_{1}\to Dir.\uparrow$ | ${\mathbf{w}}\to Cat.\downarrow$ |
| --- | --- | --- | --- | --- |
| MMVAE | 0.685 | 0.820 | — | — |
| MoPoE-VAE | 0.731 | 0.837 | — | — |
| DMVAE | 0.418 | 0.771 | 0.843 | 0.400 |
| MMVAE+ | 0.725 | 0.692 | 0.612 | 0.323 |
| DisentangledSSL | 0.831 | 0.557 | 0.592 | 0.179 |
| IDMVAE (ours) | 0.815 | 0.501 | 0.720 | 0.200 |
| –<br>${\mathcal{L}}_{\text{CrossMI}}$<br>($\lambda_{1}=0$) | 0.759 | 0.767 | 0.635 | 0.292 |
| –<br>${\mathcal{L}}_{\text{GenAug}}$<br>($\lambda_{2}=0$) | 0.810 | 0.493 | 0.698 | 0.230 |
| + Diffusion prior | 0.840 | 0.526 | 0.667 | 0.321 |

We use ResNet as encoders and decoders for images, while convolution network as those for texts (using one-hot representation of text with a vocabulary of 1,590 words).
And the dimensionality is set to 48 for ${\mathbf{z}}$ and 16 for ${\mathbf{w}}_{m}$, following Palumbo et al. (2023).
After representation learning, we perform latent linear classification similar to the previous section. With disentangled latent representations, the target (random) classification accuracy is 50% for predicting direction from ${\mathbf{z}}$, and 12.5% for predicting category from ${\mathbf{w}}_{1}$ (derived from image). The results of latent classification are given in Table [3](https://arxiv.org/html/2509.23548v1#S4.T3). Again, cross-view mutual information maximization is critical for recovering ${\mathbf{z}}$, when we do not have a very strong likelihood model (due to limited image data). On the other hand, generative augmentation still helps reduce redundancy in latent space.
In Figure [5](https://arxiv.org/html/2509.23548v1#S4.F5), we provide examples of cross-modality generations and our method achieves more coherent generation than MMVAE+;
additional conditional generations are given in Appendix [C.3](https://arxiv.org/html/2509.23548v1#A3.SS3).
We note that DisentangledSSL performs well for extracting ${\mathbf{z}}$ (their first step has a objective that similarly maximizes mutual information across views), but failed to retain private information in its second step. In contrast, our model keeps the most useful information in the latent space with generative modeling.

Figure: Figure 5: Cross-modality generation on CUB. We combine posterior sample ${\mathbf{z}}$ of the modality being conditioned on, with prior sample ${\mathbf{w}}$ of the other modality for generation. Top row: image-to-text. Bottom row: text-to-image. The top image is the original image paired with text, while the rest three are different samples. Note our generations better match the conditional input in color.
Refer to caption: figures/CUB/cross-gen-img2cap/No3-Idx2995-Left-pudgy-blue.png

### 4.3 Results on The Cancer Genome Atlas (TCGA)

TCGA dataset(^2^22https://www.cancer.gov/tcga) is a real-world multi-omics dataset that is by nature multimodal. Using the same data processing procedure from Lee & van der Schaar (2021), we obtain a dataset of 10,960 samples (of which 9,477 are labeled) with 5 views (each of 100 dimensions), each representing a molecular modality and labels (see Appendix [D.1](https://arxiv.org/html/2509.23548v1#A4.SS1) for details). The binary label represents 1-year mortality of a patient-sample. We selected 2 views (mRNA and miRNA) which had 9,874 samples out of all possible combinations after filtering out samples with missing values. After adding data with missing labels to the training set, a 90%/5%/5% split was performed with 5 different seeds. Due to the complex nature of biological data, private information may be predictive as well.

As shown in Table [4](https://arxiv.org/html/2509.23548v1#S4.T4), our method in general performs better than baseline methods in terms of accuracy (see Appendix [D.3](https://arxiv.org/html/2509.23548v1#A4.SS3) for AUROC results), both learned shared and private latent spaces are predictive, and combining ${\mathbf{z}}$ and ${\mathbf{w}}$ achieves the best performance. This is likely because clean disentanglement separates predictive information between shared and private latent variables, making predictions based on combined latent space more robust. In particular, ${\mathcal{L}}_{\text{CrossMI}}$ contributed most to the performance, and adding diffusion priors in latent space consistently improves performance.

**Table 4: Prediction accuracy on TCGA, averaged over 2 modalities and 5 splits.**
| Model | ${\mathbf{z}}\uparrow$ | ${\mathbf{w}}$ | ${\mathbf{z}}+{\mathbf{w}}\uparrow$ |
| --- | --- | --- | --- |
| MMVAE | 0.695$\pm$0.010 | — | — |
| MoPoE-VAE | 0.695$\pm$0.014 | — | — |
| DMVAE | 0.688$\pm$0.018 | 0.691$\pm$0.014 | 0.697$\pm$0.016 |
| MMVAE+ | 0.692$\pm$0.010 | 0.690$\pm$0.012 | 0.690$\pm$0.011 |
| DisentangledSSL | 0.691$\pm$0.011 | 0.691$\pm$0.012 | 0.690$\pm$0.011 |
| IDMVAE (ours) | 0.707$\pm$0.016 | 0.708$\pm$0.013 | 0.718$\pm$0.017 |
| – ${\mathcal{L}}_{\text{CrossMI}}$ ($\lambda_{1}=0$) | 0.691$\pm$0.014 | 0.689$\pm$0.010 | 0.691$\pm$0.014 |
| – ${\mathcal{L}}_{\text{GenAug}}$ ($\lambda_{2}=0$) | 0.701$\pm$0.015 | 0.706$\pm$0.019 | 0.723$\pm$0.013 |
| + Diffusion prior | 0.714$\pm$0.009 | 0.719$\pm$0.024 | 0.731$\pm$0.019 |

## 5 Conclusions

We have proposed IDMVAE, a generative model for learning disentangled representation from multimodal data.
Our innovations include the incorporation of cross-view mutual information maximization for shared variable extraction, redundancy removal based on generative augmentation, and flexible latent priors with diffusion models. These components are complimentary to each other and jointly overcome the limitations of pure likelihood modeling, resulting in superior performance than existing state-of-the-art multimodal VAEs as well as non-generative disentanglement method.

In the future, we would like to extend the model to handle missing modalities, leveraging the controllable generation capability of our model. On the other hand, for the CUB dataset, we were not able to generate very high fidelity samples of images, perhaps due to limited data volume and capacity of the decoder. We would like to introduce (possibly pre-trained) diffusion models in the input space to produce high quality samples, which may be more useful for generative augmentation.

## References

- Achille & Soatto (2018)
Alessandro Achille and Stefano Soatto.
Emergence of invariance and disentanglement in deep representations.
*Journal of Machine Learning Research*, 19(1):1947–1980, 2018.
- Adisasmito et al. (2022)
Wiku B. Adisasmito, Salama Almuhairi, Casey Barton Behravesh, Pépé Bilivogui, Salome A. Bukachi, Natalia Casas, Natalia Cediel Becerra, Dominique F. Charron, Abhishek Chaudhary, Janice R. Ciacci Zanella, Andrew A. Cunningham, Osman Dar, Nitish Debnath, Baptiste Dungu, Elmoubasher Farag, George F. Gao, David T. S. Hayman, Margaret Khaitsa, Marion P. G. Koopmans, Catherine Machalaba, John S. Mackenzie, Wanda Markotter, Thomas C. Mettenleiter, Serge Morand, Vyacheslav Smolenskiy, and Lei Zhou.
One health: A new definition for a sustainable and healthy future.
18(6):e1010537, 2022.
ISSN 1553-7374.
doi: 10.1371/journal.ppat.1010537.
- Alemi et al. (2017)
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy.
Deep variational information bottleneck.
In *ICLR*, 2017.
- Bachman et al. (2019)
Philip Bachman, R Devon Hjelm, and William Buchwalter.
Learning representations by maximizing mutual information across views.
In *NeurIPS*, 2019.
- Bai et al. (2021)
Junwen Bai, Weiran Wang, and Carla Gomes.
Contrastively disentangled sequential variational autoencoder.
In *NeurIPS*, 2021.
- Bardes et al. (2022)
Adrien Bardes, Jean Ponce, and Yann LeCun.
VICReg: Variance-invariance-covariance regularization for self-supervised learning.
In *ICLR*, 2022.
- Caron et al. (2020)
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments.
In *NeurIPS*, 2020.
- Chaudhuri et al. (2009)
Kamalika Chaudhuri, Sham M. Kakade, Karen Livescu, and Karthik Sridharan.
Multi-view clustering via canonical correlation analysis.
In *ICML*, 2009.
- Chen et al. (2020a)
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman.
Vggsound: A large-scale audio-visual dataset.
In *ICASSP*, 2020a.
- Chen et al. (2021)
Jeff Z. Hao Chen, Colin Wei, Adrien Gaidon, and Tengyu Ma.
Provable guarantees for self-supervised deep learning with spectral contrastive loss.
In *NeurIPS*, 2021.
- Chen et al. (2018)
Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud.
Isolating sources of disentanglement in vaes.
In *NeurIPS*, 2018.
- Chen et al. (2020b)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.
A simple framework for contrastive learning of visual representations.
In *ICML*, 2020b.
- Daunhawer et al. (2022)
Imant Daunhawer, Thomas M. Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E. Vogt.
On the limitations of multimodal VAEs.
In *ICLR*, 2022.
- Elizalde et al. (2023)
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang.
CLAP: Learning audio concepts from natural language supervision.
In *ICASSP*, 2023.
- Federici et al. (2020)
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata.
Learning robust representations via multi-view information bottleneck.
In *ICLR*, 2020.
- Gong et al. (2023)
Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass.
Contrastive audio-visual masked autoencoder.
In *ICLR*, 2023.
- He et al. (2015)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition, 2015.
- Higgins et al. (2017)
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.
beta-VAE: Learning basic visual concepts with a constrained variational framework.
In *ICLR*, 2017.
- Hjelm et al. (2019)
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.
Learning deep representations by mutual information estimation and maximization.
In *ICLR*, 2019.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In *NeurIPS*, 2020.
- Hyvärinen et al. (2019)
Aapo Hyvärinen, Hiroaki Sasaki, and Richard E. Turner.
Nonlinear ICA using auxiliary variables and generalized contrastive learning.
In *AISTATS*, 2019.
- Jia et al. (2021)
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.
Scaling up visual and vision-language representation learning with noisy text supervision.
In *ICML*, 2021.
- Kim & Mnih (2018)
Hyunjik Kim and Andriy Mnih.
Disentangling by factorising.
In *ICML*, 2018.
- Kim et al. (2024)
Jongsuk Kim, Hyeongkeun Lee, Kyeongha Rho, Junmo Kim, and Joon Son Chung.
EquiAV: Leveraging equivariance for audio-visual contrastive learning.
In *ICML*, 2024.
- Kingma & Welling (2014)
Diederik P. Kingma and Max Welling.
Auto-encoding variational Bayes.
In *ICLR*, 2014.
- Kumar et al. (2018)
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan.
Variational inference of disentangled latent concepts from unlabeled observations.
In *ICLR*, 2018.
- LeCun et al. (1998)
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.
Gradient-based learning applied to document recognition.
*Proc. IEEE*, 86(11):2278–2324, 1998.
- Lee & van der Schaar (2021)
Changhee Lee and Mihaela van der Schaar.
A variational information bottleneck approach to multi-omics data integration.
In *AISTATS*, 2021.
- Lee et al. (2019)
Michelle A. Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg.
Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks.
In *International Conference on Robotics and Automation (ICRA)*, 2019.
doi: 10.1109/ICRA.2019.8793485.
- Lee & Pavlovic (2021)
Mihee Lee and Vladimir Pavlovic.
Private-shared disentangled multimodal VAE for learning of latent representations.
In *2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)*, 2021.
- Liang et al. (2023)
Paul Pu Liang, Zihao Deng, Martin Q. Ma, James Zou, Louis-Philippe Morency, and Russ Salakhutdinov.
Factorized contrastive learning: Going beyond multi-view redundancy.
In *NeurIPS*, 2023.
- Linsker (1988)
R. Linsker.
Self-organization in a perceptual network.
*Computer*, 21(3):105–117, 1988.
doi: 10.1109/2.36.
- Locatello et al. (2019)
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Scholkopf, and Olivier Bachem.
Challenging common assumptions in the unsupervised learning of disentangled representations.
In *ICML*, 2019.
- Logeswaran & Lee (2018)
Lajanugen Logeswaran and Honglak Lee.
An efficient framework for learning sentence representations.
In *ICLR*, 2018.
- Lyu et al. (2022)
Qi Lyu, Xiao Fu, Weiran Wang, and Songtao Lu.
Understanding latent correlation-based multiview learning and self-supervision: An identifiability perspective.
In *ICLR*, 2022.
- Oord et al. (2018)
Aaron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding.
arXiv:1807.03748, 2018.
- Owens & Efros (2018)
Andrew Owens and Alexei A. Efros.
Audio-visual scene analysis with self-supervised multisensory features.
In *ECCV*, 2018.
- Palumbo et al. (2023)
Emanuele Palumbo, Imant Daunhawer, and Julia E. Vogt.
MMVAE+: Enhancing the generative quality of multimodal VAEs without compromises.
In *ICLR*, 2023.
- Palumbo et al. (2024)
Emanuele Palumbo, Laura Manduchi, Sonia Laguna, Daphné Chopard, and Julia E Vogt.
Deep generative clustering with multimodal diffusion variational autoencoders.
In *ICLR*, 2024.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision.
In *ICML*, 2021.
- Reed et al. (2016)
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele.
Learning deep representations of fine-grained visual descriptions.
In *CVPR*, pp. 49–58, 2016.
- Shi et al. (2019)
Yuge Shi, N. Siddharth, Brooks Paige, and Philip H. S. Torr.
Variational mixture-of-experts autoencoders for multi-modal deep generative models.
In *NeurIPS*, 2019.
- Shi et al. (2021)
Yuge Shi, Brooks Paige, Philip Torr, and Siddharth N.
Relating by contrasting: A data-efficient framework for multimodal generative models.
In *ICLR*, 2021.
- Sohl-Dickstein et al. (2015)
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In *ICML*, 2015.
- Song et al. (2021)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
In *ICLR*, 2021.
- Sutter et al. (2021)
Thomas M. Sutter, Imant Daunhawer, and Julia E. Vogt.
Generalized multimodal ELBO.
In *ICLR*, 2021.
- Suzuki et al. (2016)
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo.
Joint multimodal learning with deep generative models.
*arXiv:1611.01891*, 2016.
- Tian et al. (2020a)
Yonglong Tian, Dilip Krishnan, and Phillip Isola.
Contrastive multiview coding.
In *ECCV*, 2020a.
- Tian et al. (2020b)
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.
What makes for good views for contrastive learning?
In *NeurIPS*, 2020b.
- Tishby & Zaslavsky (2015)
Naftali Tishby and Noga Zaslavsky.
Deep learning and the information bottleneck principle.
In *2015 IEEE Information Theory Workshop (ITW)*, 2015.
- Tishby et al. (1999)
Naftali Tishby, Fernando C. Pereira, and William Bialek.
The information bottleneck method.
In *37th Annual Allerton Conference on Communication, Control and Computing*, 1999.
- Tosh et al. (2021)
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu.
Contrastive learning, multi-view redundancy, and linear models.
In *Algorithmic Learning Theory*, 2021.
- Tsai et al. (2019)
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov.
Learning factorized multimodal representations.
In *ICLR*, 2019.
- Tsai et al. (2021)
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency.
Self-supervised learning from a multi-view perspective.
In *ICLR*, 2021.
- Vahdat et al. (2021)
Arash Vahdat, Karsten Kreis, and Jan Kautz.
Score-based generative modeling in latent space.
In *NeurIPS*, 2021.
- Veličković et al. (2019)
Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm.
Deep graph infomax.
In *ICLR*, 2019.
- von Kügelgen et al. (2021)
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello.
Self-supervised learning with data augmentations provably isolates content from style.
In *ICLR*, 2021.
- Wah et al. (2011)
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.
The caltech-ucsd birds-200-2011 dataset.
Technical report, Caltech, 2011.
- Wang et al. (2025)
Chenyu Wang, Sharut Gupta, Xinyi Zhang, Sana Tonekaboni, Stefanie Jegelka, Tommi Jaakkola, and Caroline Uhler.
An information criterion for controlled disentanglement of multimodal data.
In *ICLR*, 2025.
- Wang & Isola (2020)
Tongzhou Wang and Phillip Isola.
Understanding contrastive representation learning through alignment and uniformity on the hypersphere.
In *ICML*, 2020.
- Wang et al. (2016)
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu.
Deep variational canonical correlation analysis.
*arXiv:1610.03454*, 2016.
- Wu & Goodman (2018)
Mike Wu and Noah Goodman.
Multimodal generative models for scalable weakly-supervised learning.
In *NeurIPS*, 2018.
- Zbontar et al. (2021)
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.
Barlow twins: Self-supervised learning via redundancy reduction.
In *ICML*, 2021.
- Zhai et al. (2024)
Runtian Zhai, Bingbin Liu, Andrej Risteski, J Zico Kolter, and Pradeep Kumar Ravikumar.
Understanding augmentation-based self-supervised representation learning via RKHS approximation and regression.
In *ICLR*, 2024.
- Zhao et al. (2018)
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba.
The sound of pixels.
In *ECCV*, 2018.
- Zhu et al. (2017)
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.
Unpaired image-to-image translation using cycle-consistent adversarial networks.
In *ICCV*, 2017.
- Zimmermann et al. (2021)
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel.
Contrastive learning inverts the data generating process.
In *ICML*, 2021.

## Appendix A Different Implementations for Generative Augmentation

In section [2.3](https://arxiv.org/html/2509.23548v1#S2.SS3), we have discussed two implementations of generative augmentation for redundancy removal.
Here we provide a detailed comparison of them.

Recall that ${\mathbf{x}}_{m}$ and ${\mathbf{x}}_{m}^{\prime}$ are two input samples, $({\mathbf{z}}_{m},{\mathbf{w}}_{m})$ is pair of samples drawn from the posteriors $q({\mathbf{z}}|{\mathbf{x}}_{m})$ and $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m})$ respectively, and similarly $({\mathbf{z}}_{m}^{\prime},{\mathbf{w}}_{m}^{\prime})$ is a pair of samples drawn from conditional posteriors for ${\mathbf{x}}_{m}^{\prime}$.
With disentanglement and a good generative model, we could independently vary one variable while keeping the other the same to obtain a new sample.
In particular, a sample ${\mathbf{x}}_{m}^{+}\sim p({\mathbf{x}}_{m}|{\mathbf{z}}_{m},{\mathbf{w}}_{m}^{\prime})$ would share the same ${\mathbf{z}}$ with ${\mathbf{x}}_{m}$. In turn, when we map ${\mathbf{x}}_{m}^{+}$ back to the latent space, $q({\mathbf{z}}|{\mathbf{x}}_{m}^{+})$ and $q({\mathbf{z}}|{\mathbf{x}}_{m})$ should be similar.
Likewise, $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{+})$ and $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{\prime})$ should be similar.

#### Least squares matching.

In the first implementation, we would like to minimize $I({\mathbf{z}}_{m};{\mathbf{x}}_{n})$ by approximately minimizing $H({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{\prime})$:

$$ $\displaystyle\mathbb{E}_{{\mathbf{x}}_{m}\sim p({\mathbf{x}}_{m}),{\mathbf{x}}_{m}^{\prime}\sim p({\mathbf{x}}_{m}),{\mathbf{z}}_{m}\sim q({\mathbf{z}}|{\mathbf{x}}_{m}),{\mathbf{w}}_{m}^{\prime}\sim p({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{\prime}),{\mathbf{x}}_{m}^{+}\sim p({\mathbf{x}}_{m}|{\mathbf{z}}_{m},{\mathbf{w}}_{m}^{\prime})}[-\log q({\mathbf{w}}_{m}^{\prime}|{\mathbf{x}}_{m}^{+})].$ $$

Assuming that posteriors are parameterized Gaussians, ${\mathcal{L}}_{\text{GenAug},{\mathbf{w}}_{m}}$ reduces to $\ell_{2}$ loss for matching means of posteriors, and we implement it as

$$ $\displaystyle{\mathcal{L}}_{\text{GenAug}}^{lsq}=\mathbb{E}_{{\mathbf{x}}_{m}\sim p({\mathbf{x}}_{m}),{\mathbf{x}}_{m}^{\prime}\sim p({\mathbf{x}}_{m}),{\mathbf{z}}_{m}\sim q({\mathbf{z}}|{\mathbf{x}}_{m}),{\mathbf{w}}_{m}^{\prime}\sim q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{\prime}),{\mathbf{x}}_{m}^{+}\sim p({\mathbf{x}}_{m}|{\mathbf{z}}_{m},{\mathbf{w}}_{m}^{\prime})}\left\lVert\overline{{\mathbf{w}}}_{m}^{\prime}-\overline{{\mathbf{w}}}_{m}^{\prime\prime}\right\rVert^{2}$ $$

where $\overline{{\mathbf{w}}}_{m}^{\prime}$ is the posterior mean of $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{\prime})$ while $\overline{{\mathbf{w}}}_{m}^{\prime\prime}$ is the posterior mean of $q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{+})$.

#### Contrastive matching.

In practice, we find it more stable to use a contrastive loss for matching, i.e.,

$$ $\displaystyle{\mathcal{L}}_{\text{GenAug}}^{contrast}:=-Contrast({\mathbf{w}}_{m}^{\prime\prime},{\mathbf{w}}_{m}^{\prime})\qquad\text{where}\quad{\mathbf{x}}_{m}^{+}\sim p({\mathbf{x}}_{m}|{\mathbf{z}}_{m},{\mathbf{w}}_{m}^{\prime}),\;{\mathbf{w}}_{m}^{\prime\prime}\sim q({\mathbf{w}}_{m}|{\mathbf{x}}_{m}^{+}).$ $$

We plug in the two different implementations into our loss. In Table [5](https://arxiv.org/html/2509.23548v1#A1.T5) and  [6](https://arxiv.org/html/2509.23548v1#A1.T6), we provide the comparison of the two on PolyMNIST-Quadrant, each with its loss coefficient tuned on the validation set.
We find the best coefficients to be $\lambda_{1}$=80 and $\lambda_{2}^{lsq}$=0.75 for ${\mathcal{L}}_{\text{GenAug}}^{lsq}$, and $\lambda_{1}$=80 and $\lambda_{2}^{contrast}$=20 for ${\mathcal{L}}_{\text{GenAug}}^{contrast}$; diffusion prior loss has a coefficient of $1.0$ when incorporated.
We observe that both implementations improve the disentanglement compared with using ${\mathcal{L}}_{\text{CrossMI}}$ only, with ${\mathcal{L}}_{\text{GenAug}}^{contrast}$ outperforming ${\mathcal{L}}_{\text{GenAug}}^{lsq}$.

**Table 5: Comparison of ${\mathcal{L}}_{\text{GenAug}}^{lsq}$ and ${\mathcal{L}}_{\text{GenAug}}^{contrast}$ for generative augmentation regularization in latent linear classification on PolyMNIST-Quadrant. Accuracies are averaged over 5 modalities.**
| Our Models | $z\to Digit\uparrow$ | $z\to Quad\downarrow$ | $w\to Quad\uparrow$ | $w\to Digit\downarrow$ |
| --- | --- | --- | --- | --- |
| ${\mathcal{L}}_{\text{CrossMI}}$ Only ($\lambda_{2}=0$) | 0.977 | 0.277 | 0.999 | 0.202 |
| ${\mathcal{L}}_{\text{CrossMI}}$ +<br>${\mathcal{L}}_{\text{GenAug}}^{lsq}$ | 0.972 | 0.267 | 0.999 | 0.186 |
| + diffusion prior | 0.980 | 0.263 | 0.999 | 0.154 |
| ${\mathcal{L}}_{\text{CrossMI}}$ +<br>${\mathcal{L}}_{\text{GenAug}}^{contrast}$ | 0.983 | 0.271 | 0.999 | 0.162 |
| + diffusion prior | 0.982 | 0.267 | 0.999 | 0.143 |

**Table 6: Comparison of ${\mathcal{L}}_{\text{GenAug}}^{lsq}$ and ${\mathcal{L}}_{\text{GenAug}}^{contrast}$ for generative augmentation regularization in generative coherence, averaged over 5 views, on PolyMNIST-Quadrant. We use subscript $q$ to indicate samples from posteriors and subscript $p$ to indicate samples from priors. For generated images, digit label is determined by ${\mathbf{z}}_{s,q}$ or otherwise random (with target accuracy 10%), quadrant label is determined by ${\mathbf{w}}_{s,q}$ or otherwise random (with target accuracy 25%).**
| Our Models | Self Gen ($s=t$) | Cross Gen ($s\neq t$) | Uncond. |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $Gen({\mathbf{z}}_{s,q},{\mathbf{w}}_{t,p})$ | $Gen({\mathbf{z}}_{t,p},{\mathbf{w}}_{s,q})$ | $Gen({\mathbf{z}}_{s,q},{\mathbf{w}}_{t,p})$ | $Gen({\mathbf{z}}_{p},{\mathbf{w}}_{p})$ |  |  |  |  |
| $Digit\uparrow$ | $Quad\downarrow$ | $Digit\downarrow$ | $Quad\uparrow$ | $Digit\uparrow$ | $Quad\downarrow$ | $Digit\uparrow$ |  |
| ${\mathcal{L}}_{\text{CrossMI}}$ Only ($\lambda_{2}=0$) | 0.670 | 0.250 | 0.370 | 0.999 | 0.671 | 0.250 | 0.008 |
| ${\mathcal{L}}_{\text{CrossMI}}$ +<br>${\mathcal{L}}_{\text{GenAug}}^{lsq}$ | 0.817 | 0.250 | 0.219 | 0.999 | 0.812 | 0.250 | 0.044 |
| + Diffusion prior | 0.917 | 0.249 | 0.109 | 0.999 | 0.875 | 0.250 | 0.668 |
| ${\mathcal{L}}_{\text{CrossMI}}$ +<br>${\mathcal{L}}_{\text{GenAug}}^{contrast}$ | 0.898 | 0.249 | 0.162 | 0.999 | 0.881 | 0.250 | 0.070 |
| + Diffusion prior | 0.942 | 0.251 | 0.106 | 0.999 | 0.887 | 0.251 | 0.664 |

## Appendix B Details and Additional Results on PolyMNIST-Quadrant

### B.1 Conditional Generation

In Figure [6](https://arxiv.org/html/2509.23548v1#A2.F6) and Figure [7](https://arxiv.org/html/2509.23548v1#A2.F7), we provide additional results on conditional generations, where one latent variable is sampled from the posterior, while the other is sampled from the prior.
In Figure [8](https://arxiv.org/html/2509.23548v1#A2.F8), we provide conditional generations for which both ${\mathbf{z}}$ and ${\mathbf{w}}$ are sampled from posteriors; this simulates the samples we use in ${\mathcal{L}}_{\text{GenAug}}$.
In all cases, our method provides the most coherent generations, consistent with the quantitative results in Section [4.1](https://arxiv.org/html/2509.23548v1#S4.SS1).

Figure: Figure 6: Conditional generations on PolyMNIST-Quadrant, with conditioning on ${\mathbf{z}}$. The top row shows the samples (from modality 1) we condition on. We sample ${\mathbf{z}}\sim q({\mathbf{z}}|{\mathbf{x}}_{1})$, sample the private variable from the corresponding prior ${\mathbf{w}}_{m}\sim p({\mathbf{w}}_{m}|{\mathbf{x}}_{m})$, and generate a new sample from $p({\mathbf{x}}_{m}|{\mathbf{z}},{\mathbf{w}}_{m})$. Row 2 to row 6 are generated samples for modalities 1 to 5. Note for well-disentangled latent variables, each column shall contain the same digit ${\mathbf{z}}$. For each row we used the same prior sample of ${\mathbf{w}}$, so images in the same row shall have the same quadrant, writing style, and background.
Refer to caption: figures/PolyMNIST/MMVAE/Conditional-Generations-Grid-m0-Qz-Pw--06-23-15--07-22-03--6x5.png

Figure: Figure 7: Conditional generations on PolyMNIST-Quadrant, with conditioning on ${\mathbf{w}}_{m}$. The top row shows the samples (from modality 1) we condition on. We sample ${\mathbf{w}}\sim q({\mathbf{w}}|{\mathbf{x}}_{1})$, and sample ${\mathbf{z}}\sim p({\mathbf{z}})$, and generate a new sample from $p({\mathbf{x}}_{1}|{\mathbf{z}},{\mathbf{w}}_{1})$. Row 2 to row 6 are generated samples. Note for well-disentangled latent variables, each column shall have the same quadrant position and background. For each row we used the same prior sample of ${\mathbf{z}}$, so images in the same row shall have the same digit.
Refer to caption: figures/PolyMNIST/con_gen_Qw_Pz/DMVAE__09-09_T0__08-31_16__6x5.png

Figure: Figure 8: Conditional generation on PolyMNIST-Quadrant. The first row and first column contain images for which we extract posterior samples of ${\mathbf{z}}$ and ${\mathbf{w}}_{m}$, respectively. And the rest of the grid contains generated images using latent samples of the corresponding row and column. This figure illustrates the samples we use in generative augmentation.
Refer to caption: figures/PolyMNIST/MMVAEplus/Conditional-Generations-Grid-m2-Qz-Qw--m2-Qw-Qz--05-23-9--07-13-03-11x11.png

### B.2 Implementation Details

We utilize a deep residual network (ResNet) architecture of $3$ residual blocks, with the number of filters doubling from 64 to up to 512 after each block for the encoder,
and the number of filters halving after each decoder for the decoder, for all five modalities. And each modality’s information is factorized in the latent space into a shared latent dimension of 32 and a private latent dimension of 128. Models are trained for 100 epochs using the Adam optimizer with a learning rate of $5e^{-4}$ and a batch size of 128, and use the other default hyperparameters of MMVAE+ baseline, including the KL divergence coefficient $\beta$ of 2.5.
We performed a grid search over the coefficients to tune the regularization terms, $\lambda_{1}$ and $\lambda_{2}$, after training for 100 epochs.
We search them in the range $[0.01,100]$.
We tune $\lambda_{1}$ individually first to find the best general performance in latent classification for ${\mathcal{L}}_{\text{CrossMI}}$, and fix the $\lambda_{1}$, then combine with ${\mathcal{L}}_{\text{GenAug}}^{contrast}$, and find the best combination of $\lambda_{1}$=80 and $\lambda_{2}^{contrast}$=20. Finally, we tune the diffusion prior weight to 1.0 out of {0.01, 0.1, 1.0, 10.0}, which optimizes the final general performance at the 100th epoch.

### B.3 Latent Visualization

In Figure [9](https://arxiv.org/html/2509.23548v1#A2.F9), we provide 2D visualizations of the latent representation $q({\mathbf{z}}|{\mathbf{x}}_{1})$ on the PolyMNIST-Quadrant test set.
Observe that, without diffusion prior (left panel), there exists a gap between the posterior and the Gaussian prior, whereas the capacity of diffusion prior is strong enough to ensure good overlap between the two distributions (right panel).

Figure: Figure 9: 2D Latent Visualization (UMAP) of learned shared latent representation ${\mathbf{z}}$ by our methods on the test set. We color each representation according to its ground truth digit label. Black points are samples from the prior.
Refer to caption: figures/PolyMNIST/t_SNE_or_UMAP/UMAP/z_digit_with_prior/OurNorm__09-08_PT0__05-19_3__m1__Mcrop.png

## Appendix C Details and Additional Results on CUB

### C.1 Dataset

The 8 super-categories, namely Blackbird, Gull, Jay, Oriole, Tanager, Tern, Warbler, and Wren, are created following the same grouping method introduced by  Palumbo et al. (2024), as shown in Figure [10](https://arxiv.org/html/2509.23548v1#A3.F10).

In addition, we introduce a private binary label representing the bird’s horizontal direction, determined by the part location annotations provided in the original dataset (Wah et al., 2011), as illustrated in Figure [11](https://arxiv.org/html/2509.23548v1#A3.F11).
Specifically, we compare the average horizontal position of the group of the bird’s ‘head’ parts with the average horizontal position of the group of its ‘body’ parts. If the head is positioned to the left of the body, the direction label is ‘left’ (label 0); otherwise, it is ‘right’ (label 1). This creates a modality-specific (private) label for the image that cannot be inferred from the text captions. At the same time, as shown in Figure [11](https://arxiv.org/html/2509.23548v1#A3.F11) (b) and (c), a very small fraction of the images have
invisible ‘head’ or ‘body’ location annotations, or the locations are too close, in which case they are not assigned the direction label.
Direction labels of validation and test images are verified by human.

Figure: Figure 10: CUB category labels: dividing 22 species into 8 super-categories (Palumbo et al., 2024).
Refer to caption: figures/CUB/intro_snapshot/CUB_dataset_structure_from_CMVAE.png

Figure: Figure 11: The collected parts in the original dataset (Wah et al., 2011), and sample images with invisible ‘body’ or ‘head’ location annotations or ambiguous horizontal direction.
Refer to caption: figures/CUB/intro_snapshot/CUB_Collected_Parts.png

### C.2 Implementation Details

For the image modality, we utilize a deep residual network (ResNet) architecture of $5$ residual blocks, with the number of filters doubling from 64 to up to 1024 after each block for the encoder, and the number of filters halving after each block for the decoder.
For text modality, we utilize a convolutional neural network, CNN-based encoder and decoder with one-hot encoded captions from a vocabulary size of 1590 words.
The shared variable has a dimensionality of 48, and the private variable has a dimensionality of 16.
Models are trained for 150 epochs using the Adam optimizer with a learning rate of $10^{-3}$ and a batch size of 128, and other default hyperparameters of the MMVAE+ baseline, including the KL divergence coefficient $\beta=1.0$.

During training, we apply horizontal flip augmentation to the image modality, with a flip probability of $0.5$.
Then we tune the ${\mathcal{L}}_{\text{CrossMI}}$, ${\mathcal{L}}_{\text{GenAug}}^{contrast}$, and diffusion prior similarly to [B.2](https://arxiv.org/html/2509.23548v1#A2.SS2), and obtain the optimal coefficients $\lambda_{1}=40$, $\lambda_{2}^{contrast}=0.05$, and diffusion loss weight $0.1$.

### C.3 Conditional Generation

In Figure [12](https://arxiv.org/html/2509.23548v1#A3.F12),  [13](https://arxiv.org/html/2509.23548v1#A3.F13), [14](https://arxiv.org/html/2509.23548v1#A3.F14), we provide conditional generation of competitive methods.

Figure: Figure 12: Text-to-image generation on CUB. We combine posterior sample ${\mathbf{z}}$ of the text modality (top row), with prior sample ${\mathbf{w}}$ of the image modality (shared by each row) for generation.
Refer to caption: figures/CUB/cross-gen-cap2img-table/MMVAEplus--Cap2Img--qzm0-pwm0--150--08-19-T0-8x10.png

Figure: Figure 13: Image-to-text generation on CUB. We combine posterior sample ${\mathbf{z}}$ of the image modality (top row), with prior sample ${\mathbf{w}}$ of the text modality (shared by each row) for generation.
Refer to caption: figures/CUB/cross-gen-img2cap-table/MMVAEplus--Img2Cap--qzm0-pwm0--150--08-19-T0-8x10.png

Figure: Figure 14: Image-to-image generation on CUB. We combine the posterior sample ${\mathbf{z}}$ of the image modality (top row), with the prior sample ${\mathbf{w}}$ of the image modality (shared by each row) for generation.
Refer to caption: figures/CUB/self-gen-img2img-table/MMVAEplus--Img2Img--qzm0-pwm0--150--08-19-T0-8x10.png

## Appendix D Details and Additional results on TCGA

### D.1 Data Preparation

TCGA dataset (2016 version) is processed by combining multi-omics data across different cancer types and different patients before features are selected and kernel PCA was performed to reduce the dimensionality to 100. Because clinical trials usually have small sample sizes, high dimensionalities, and complex dependencies, this dataset is ideal to test the robustness of our method. One significant outcome of cancer multi-omics data is the days of survival after samples were collected. Because patients may not show up for checkups, it is possible that a sample is censored (unlabeled). The days of survival are then converted to a binary 1-year mortality indicator. The dataset itself has missing views but contains predictive information and high correlations among different views, making it suitable for our task. Further notice that this dataset does not contain private ground truth.

### D.2 Implementation

A 2-layer MLP with 128 as hidden dimensions was used for encoding and decoding with 48 latent dimensions (16 for ${\mathbf{z}}$ and 32 for ${\mathbf{w}}$). Evaluation for all methods regarding this dataset is done by averaging logits from each view. 50 epochs were run to train the model. For TCGA dataset, baseline methods are performed with default hyperparameter, which gives KL divergence a coefficient of 2.5. For DisentangledSSL baseline in particular, step 1 coefficient was set to 0 since we use posterior mean instead of zsample to match other methods and step 2 coefficient was set to 0.01. To tune our method, we performed a grid search with coefficients $\{0.001,0.01,0.1,1,10,100\}$ and chose the best combination on validation set, before recording the performance on test set. $\lambda_{1}=10,\lambda_{2}=0.001$ were chosen to be the best combination at 40 epochs. For ablation studies, we set one coefficient to be 0 while keeping the other one optimal in a combined setting. For the optimal coefficients combination, we used the model at epoch 40; $\lambda_{1}=0$, at epoch 50; and $\lambda_{2}=0$, at epoch 35. For adding a diffusion prior, we tuned the diffusion weight to be 0.1 out of 0.1, 1, 10 while keeping $\lambda_{1},\lambda_{2}$ same as the optimal combination and chose the best performance at validation set at epoch 40.

### D.3 Prediction AUROC

In Table [7](https://arxiv.org/html/2509.23548v1#A4.T7), we provide the linear classification AUROC of different methods using latent representations. The relative merits of different methods are consistent with that observed with the accuracy metric in Table [4](https://arxiv.org/html/2509.23548v1#S4.T4).

**Table 7: Prediction AUROC Performance with ablation on TCGA dataset, averaged over 2 modalities and 5 splits. Tuning reported in Appendix [D.2](https://arxiv.org/html/2509.23548v1#A4.SS2).**
| Model | ${\mathbf{z}}\uparrow$ | ${\mathbf{w}}$ | ${\mathbf{z}}+{\mathbf{w}}\uparrow$ |
| --- | --- | --- | --- |
| MMVAE | 0.653$\pm$0.033 | — | — |
| MoPoE-VAE | 0.660$\pm$0.024 | — | — |
| DMVAE | 0.609$\pm$0.030 | 0.636$\pm$0.037 | 0.643$\pm$0.032 |
| MMVAE+ | 0.586$\pm$ 0.027 | 0.581$\pm$0.033 | 0.585$\pm$0.033 |
| DisentangledSSL | 0.693$\pm$0.046 | 0.551$\pm$0.019 | 0.699$\pm$0.045 |
| IDMVAE (ours) | 0.740$\pm$0.025 | 0.740$\pm$0.022 | 0.767$\pm$0.026 |
| – ${\mathcal{L}}_{\text{CrossMI}}$ ($\lambda_{1}=0$) | 0.549$\pm$0.017 | 0.545$\pm$0.026 | 0.548$\pm$0.026 |
| – ${\mathcal{L}}_{\text{GenAug}}$ ($\lambda_{2}=0$) | 0.740$\pm$0.019 | 0.746$\pm$0.022 | 0.771$\pm$0.021 |
| + Diffusion prior | 0.745$\pm$0.024 | 0.751$\pm$0.029 | 0.772$\pm$0.022 |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	Is the diffusion prior applied only to the shared latent z, or also to private w_m? If only to z, why is a simple Gaussian sufficient for the private components?
2.	How do the MI and cycle constraints correspond to orthogonality or covariance restrictions in linear Bayesian joint additive factor models such as [5]?
3.	The paper does not mention a β-VAE coefficient. Is β fixed at 1, and do $\lambda_{\text{MI}}$, $\lambda_{\text{GenAug}}$ effectively play this role? How do these hyperparameters influence the average posterior entropies $H[q(z|x_m)] $ and $H[q(w_m|x_m)] $ over training.
4.	Some VAE models e.g. [6] address the inference-consistency problem in multimodal VAEs, i.e. the inability to obtain coherent subset posteriors such as $p(w_2 \mid x_1)$ via a tighter ELBO and flexible encoder aggregation. Does IDMVAE provide an alternative route to this goal by enforcing independence between $w_m$ and the shared latent z, thereby avoiding aggregation bias or looser bounds for modalities with high conditional mutual information?
5.	The CrossMI loss is motivated by encouraging $I(z_m;x_n)\approx I(x_m;x_n)$, implying that the shared latent fully explains cross-modal dependence. Would this not render private latents redundant? How is this balanced to prevent collapse of private variation, and does this depend on dataset redundancy?
6.	Integrating a diffusion prior end-to-end into a multimodal VAE can introduce instability, as seen in recent works like REPA-E [7] or uses some annealing tricks [8]. The paper does not discuss training schedules, warm-up strategies, or whether gradients from the diffusion loss are stopped for encoder updates.

Refs:
[5] Anceschi, Niccolo, et al. "Bayesian joint additive factor models for multiview learning." arXiv preprint arXiv:2406.00778(2024).\
[6] Hirt, Marcel, et al. "Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives." Transactions on Machine Learning Research.\
[7] Leng, Xingjian, et al. "Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers." arXiv preprint arXiv:2504.10483 (2025).\
[8] Vahdat, Arash, Karsten Kreis, and Jan Kautz. "Score-based generative modeling in latent space." Advances in neural information processing systems 34 (2021): 11287-11302.
soundness: 3
strengths: Originality:\
Introduces a principled formulation of multimodal disentanglement by combining cross-view mutual-information maximization with generative augmentation, directly targeting the separation of shared and private factors.\
Conceptually unifies ideas from contrastive learning, information bottleneck, and generative modeling in a single probabilistic framework.

Quality:\
Methodologically robust: each loss term is clearly motivated by known weaknesses of multimodal VAEs (e.g., posterior collapse, shared–private leakage).\
The generative augmentation mechanism leverages the model’s own decoder for cycle consistency, avoiding handcrafted data augmentations\
Comprehensive experiments across synthetic (PolyMNIST-Quadrant) and real (CUB, TCGA) datasets demonstrate consistent improvements in disentanglement, coherence, and generation quality.\
Ablation studies isolate the contribution of each component (CrossMI, GenAug, diffusion prior) and show interpretable, consistent improvements.\

Clarity:\
The paper is well-structured and clearly written, with each component (CrossMI, GenAug, diffusion prior) well motivated and described.\
Mathematical notation and objective formulation are concise and consistent.

Significance:\
Addresses a challenge in multimodal learning of separating  shared vs. private information with a practical, modular solution applicable to existing architectures.\
Demonstrates that learned latent priors (diffusion models) can substantially improve multimodal VAEs, opening a new direction for hybrid diffusion–VAE research. However, it lacks a clear comparison with other multi-modal (latent) diffusion models, such as [1,2].

Refs: 
[1] Bounoua, Mustapha, Giulio Franzese, and Pietro Michiardi. "Multi-modal latent diffusion." Entropy 26.4 (2024): 320. \
[2] Chen, Changyou, et al. "Diffusion models for multi-modal generative modeling." arXiv preprint arXiv:2407.17571 (2024).
summary: The paper “Disentanglement of Variations with Multi-Modal Generative Modeling (IDMVAE)” proposes a new multimodal variational autoencoder designed to cleanly separate shared and private sources of variation across data modalities. Standard multimodal VAEs, such as MMVAE and MoPoE, often suffer from latent leakage, where modality-specific information contaminates the shared latent space, causing incoherent cross-modal generation and poor disentanglement. IDMVAE introduces three complementary mechanisms to address this.

1.	A cross-view mutual information maximization term aligns shared representations by encouraging different modality-specific encoders to capture the same underlying latent factors.
2.	A generative augmentation cycle removes redundancy between shared and private latents by decoding “mixed” latent pairs into synthetic samples and enforcing that shared latents remain invariant while private latents retain modality-specific structure.
3.	A latent diffusion prior replaces the standard Gaussian prior with a learned diffusion model that can represent complex, multimodal latent distributions, reducing prior–posterior mismatch and improving unconditional generation quality.

The resulting objective extends the MMVAE+ ELBO with information-theoretic and generative-consistency regularizers, all trainable end-to-end. Experiments on PolyMNIST-Quadrant, CUB (image–text), and TCGA (multi-omics) show that IDMVAE achieves substantially cleaner separation of shared vs. private factors, higher conditional and unconditional coherence, and improved latent alignment compared to strong baselines. Ablation studies demonstrate that cross-view MI alignment improves cross-modal reconstruction, the generative augmentation reduces leakage, and the diffusion prior enhances sample quality.
weaknesses: The connection between maximizing $I(z_m; z_n)$ and achieving conditional sufficiency $I(x_m;x_n|z)=0$ remains heuristic. Under what assumptions (e.g., conditional independence or additive latent structure) do the suggested losses approximate or minimal-sufficiency conditions.\
The InfoNCE estimator can be sensitive to negative-sample size, temperature, and batch composition. The paper fixes these hyperparameters without ablation or analysis.\
Adding contrastive, augmentation, and diffusion losses likely increases computational cost. The paper lacks a discussion of these computation costs.\
You compare to MMVAE variants but not to contrastive approaches such as [3-4], or as mentioned above, latent diffusion models [1-2]. \
The generative augmentation term is described as reducing redundancy between z and w_m, but the paper does not analyze how effectively this enforces $I(z; w_m) \to 0$ in the experiments.



Refs: \
[3] Liang, Paul Pu, et al. "Factorized contrastive learning: Going beyond multi-view redundancy." Advances in Neural Information Processing Systems 36 (2023): 32971-32998.\
[4] Dufumier, Benoit, et al. "What to align in multimodal contrastive learning?." arXiv preprint arXiv:2409.07402 (2024).

---

confidence: 5
contribution: 2
initial_rating: 2
presentation: 3
questions: Please review the Weaknesses section and attempt to address the points raised there.
soundness: 2
strengths: The paper is well-written and well-presented. The paper also seems to have strong theoretical grounds for the idea they presented in this work. Their MI-based regularization methods could be useful for multimodal VAE works in general. The proposed losses are also useful for learning a shared representation, which is important for downstream tasks. Their proposed idea of using practical contrastive loss for cross-view MI maximization and their disentanglement learning with generative augmentation losses is a good contribution. They propose using diffusion models for learning complex latents, but this has already been proposed and implemented in other works (look at the weaknesses part).
summary: This paper introduces the Information-disentangled Multimodal VAE (IDMVAE), a new model for learning from data with multiple modalities. IDMVAE achieves this improved disentanglement by using Mutual Information (MI) regularizations and latent diffusion priors. They try to learn a common latent z that captures all the shared information encouraged by the added new regularization losses introduced in this work. In addition, they utilize diffusion models to create more complex and capable latent priors, thereby moving beyond simple Gaussian priors and enhancing the model's representational capacity. The authors demonstrate that IDMVAE achieves better shared and private information, resulting in superior generation quality and enhanced semantic coherence on some simple datasets compared to existing methods.
weaknesses: Even though the paper is well presented, there are considerable weaknesses in this paper that render the contribution marginal.


1. $ \textbf{Used datasets are toy datasets} $  

The datasets used in this work are very small and not significant enough to show the practicality of the presented method. I encourage the authors to look at [1], which is also a work done on multimodal VAEs using a score-based model that used the CelebAMask-HQ multimodal dataset, which is larger than the datasets used in this paper, as a starting point. The paper is also not cited and not compared as a baseline. The authors can also add other larger datasets than this to prove their method is still useful in these settings.

2. $\textbf{Latent diffusion proposal}$

 Using a latent diffusion/score-based model to learn complex latents for multimodal VAEs is also presented in [2]. But it hasn't been cited in this work and presented as a new thing.

3. $\textbf{Limited baselines}$

 The baselines compared in this work are limited. Additional recent baselines should also be added to the work. E.g., [1,2,3,4]. 

4. $\textbf{Evaluation of Quality}$

 The paper doesn't use FID for evaluating the quality of generated unconditional and conditional images as in [1]. I believe a quantifiable measure of quality is important in multimodal VAE works.

5. $\textbf{Evaluation of method of PoE Multimodal Models}$

 The paper proposes that the method can be used in PoE/MoPoE multimodal VAE models but those haven't been explored as ablations. Can the authors try their method in at least one setup of these multimodal VAEs as a proof of concept or ablation study.


References
1. Wesego, Daniel, and Amirmohammad Rooshenas. "Score-based multimodal autoencoders." arXiv preprint arXiv:2305.15708 (2023).
2. Wesego, Daniel, and Pedram Rooshenas. "Multimodal ELBO with Diffusion Decoders." arXiv preprint arXiv:2408.16883 (2024).
3. Bounoua, Mustapha, Giulio Franzese, and Pietro Michiardi. "Multi-modal latent diffusion." Entropy 26, no. 4 (2024): 320.
4. Palumbo, Emanuele, Laura Manduchi, Sonia Laguna, Daphné Chopard, and Julia E. Vogt. "Deep generative clustering with multimodal diffusion variational autoencoders." In The Twelfth International Conference on Learning Representations. 2024.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. Can you provide a way to visualize the learned shared latent variable and modality-specific variables? Comparing the visualizations for the MMVAE+ and the proposed method could be straightforward.

2. Can this method scale up to a large-scale Multimodal dataset (e.g., MSCOCO)? 

3. How does the proposed method compare to other diffusion models in terms of, for example, CLIP score?

4. How do you handle the text input? Any pre-trained embedding models?
soundness: 3
strengths: 1. The cross-view regularization for disentangling the shared latent variable z and modality-specific w is well-motivated.

2. The experiments consider cross-domain data, and the ablation studies are complementary.

3. Using diffusion to model latent priors results in better alignment between prior and posterior distributions, leading to improved unconditional generation and coherence.
summary: This paper introduces Information-Disentangled Multimodal VAE (IDMVAE), a new multimodal generative framework designed to improve disentanglement between shared and modality-specific representations.
weaknesses: 1. Compared to multimodal VAEs, the novelty is based on the cross-view regularization. For the diffusion part, CMVAE also integrates the diffusion model for improved performance. The conceptual advance is relatively modest.

2. The paper does not provide training efficiency comparisons or runtime analyses (especially for the diffusion part). It unclear whether the performance comes from the model complexity.

3. While figures illustrate disentanglement, generated samples on CUB are of relatively low fidelity, suggesting the model’s generative capacity is still limited.


Decision: Accept (Poster)