Disentanglement of Variations with Multimodal Generative Modeling
Authors: Yijie Zhang
Yiyang Shen
Weiran Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Variational Autoencoder
Disentanglement
Multi-view Information Bottleneck
Diffusion Models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Multimodal data are prevalent across various domains, and learning robust representations of such data is paramount to enhancing generation quality and downstream task performance. To handle heterogeneity and interconnections among different modalities, recent multimodal generative models extract shared and private (modality-specific) information with two separate variables. Despite attempts to enforce disentanglement between these two variables, these methods struggle with challenging datasets where the likelihood model is insufficient. In this paper, we propose Information-Disentangled Multimodal VAE (IDMVAE) to explicitly address this issue, with rigorous mutual information-based regularizations, including cross-view mutual information maximization for extracting shared variables, and a cycle-consistency style loss for redundancy removal using generative augmentations. We further introduce diffusion models to improve the capacity of latent priors. These newly proposed components are complementary to each other. Compared to existing approaches, IDMVAE shows a clean separation between shared and private information, demonstrating superior generation quality and semantic coherence on challenging datasets.

Published as a conference paper at ICLR 2026
DISENTANGLEMENT OF VARIATIONS WITH MULTI- MODAL GENERATIVE MODELING
Yijie Zhang, Yiyang Shen, Weiran Wang Department of Computer Science
University of Iowa
lowa City, IA 52242, USA
Zyanie-zhang, y shenn wei wereennweerannnnnne weirann-ang//euoow. edu
ABSTRACT
Multimodal data are prevalent across various domains, and learning robust repre- sentations of such data is paramount to enhancing generation quality and down- stream task performance. To handle heterogeneity and interconnections among different modalities, recent multimodal generative models extract shared and pri- vate (modality-speciffe information with two separate variables. Despite attempts enforce disentanglement between these two variables, these methods struggle with challenging datasets where the likelihood model is insufficient. In this paper, we propose Information-Disentee Disentangled Multimodal VAE (IDMVAE) explicitly address this issue, with rigorous mutual information-basee regularizations, includ- ing cross-view mutual information maximization for extracting shared variables, and cycle-consistency style loss for redundancy removal using generative aug- mentations. We further introduce diffusion models to improve the capacity of la- tent priors. These newly proposed components are complementary to each other. Compared to existing approaches, IDMVAE shows a clean separation between shared and private information, demonstrating superior generation quality and se- mantic coherence on challenging datasets.
INTRODUCTION
Most real-world data are inherently multimodal multi-view and sounds (Zhao et al. 2018 Owens Eiros 2018 Chen et al. 2020a Gong et al. 2023 Kim et al. 2024h robots nied by captions (Radford et al.) 2021: Jia et al.] 2021); and heterogeneous human, animal, and environmental data are collected for health improvements (Adisasmito et al. (022). In addition to these naturally occurring data, synthetic multi-view data constructed from semantically similar input components or via augmentation are also widely used learn useful representations for downstream
Videos contain both visual scenes
and feel via sensors (Lee et al. 2019); images often accompa-
tasks /Veličković et al.) 2019 Chen et al.) 2020b; Caron et al. 2020 Tian et al. 2020aa] Bardes et al., 2022). Despite the abundance of such data, leveraging them nontrivial even with naturally aligned modalities due to their diversity and complex correlations. Therefore, core challenge is to integrate information across views to learn universal, transferable representations.
Variational autoencoders (VAEs, Kingma & Welling 2014) and their multimodal extensions have emerged as a powerful paradigm to tackle this problem (Wang et al. 2016 Suzuki et al. 2016). They can extract useful shared information in data with missing modalities (Wu & Goodman, 2018) and
noise (Shi et al.| 2021). While early works have assumed that a single latent space can capture all relevant information and data variations (Shi et al.) 2019; Sutter et al. 2021), recent approaches have recognized the existence of both shared and prodality-speciffe (private) information real-world
datasets (Daunhawer et al. 2022 Lee & Pavlovic 2021 Palumbo et al., 2023 2024). However, modeling shared and private components naturally exposes a challenge:
How can we achieve maximal disentanglement between shared and private vari- ables so that learned representations are complete and non-redundant?
We "modality" and "view" interchangeably they both appear in the literature.

---

Published as a conference paper at ICLR 2026
Without clean separation, shared information leaks into private encodings and vice versa, causing weak coherence across modalities, wasted model capacity, and inadequate generation quality.
Our Contributions. We propose Information- Disenntanledd Multimodal VAE (IDMVAH), a novel framework for unsupervised multimodal representation learning hailing from theoretical stringency and practical performance. (1) Different from prior works which use capacity- 2016) or shortcut- shorteut-preventing (Palumbo et al. 2023) heuristics, we employ mutual information (MI)- based regularizations to ensure disentanglement. In particular, we use cross-view MI to extract common factors that likelihood models fail fully capture, thereby enhancing moss-modal coher- ence. Additionally, a sycle-consistencyystye loss removes redundancy between shared and private latents using samples generated by the model itself, eliminating the need for domain-spectie speciifi aug- mentations. (2) To overcome limitations of simple Gaussian priors, we leverage diffusion- based priors (Sohl-Dickstein al.) 2015 Ho et al., 2020; Song al., 2021) that capture the richness of multimodal latent spaces, leading to greater representational capacities. (3) Across multiple com- plex datasets spanning image, text, and multi-omis data, IDMVAE performs consistently better than state-of thrt methods in terms of cross moosl generation and coherence, showing synergy between MI-based objectives and diffusion priors, which leads to improved performance.
(Wang et al.)
2
METHOD
Given a set of M modalities X fx .......,,y XMr) } we would like to learn factorized latent space for each modality m, separating information shared across modalities, captured by latent variable 2, from modality-specific private information, cap- tured by Wm- As illustrated in Figure [] sume that, Vm, Xm is generated by Zm and w.. jointly with p(xm Zm, Wm). and the latent vari- ables have independent priors, i.e., 2 II. Wm. and p(z, {w, JM = p(z) IT p(wm). We perform variational inference and parameterize the approximate posteriors with factorized form: q(( (z, (m|(xm) = q(z)xm) q(wm X..). Learned q (()) and @ xmm provide representations of original inputs which can be used in downstream tasks. Our method consists of three components.
Category
Orientation
X1
X2
I
I
H2
W2
bright orange wings
X1
X2
Figure 1: Our approach with two modalities.
2.1 LIKELIHOOD MODELING WITH MULTIMODAL VAE
We begin with the objective of the MMVAE+ model (Palumbo et al. 2023) our foundation. Let q(z[Xm) be the posterior of shared variable derived from modality m (with distribution parame- ters modeled by encoder), and of (w (x,.) be the posterior of private variable from modality m (modeled by another encoder). One can define a global posterior by aggregating view-specific (MoE) scheme qzz[X) M
posteriors, c.g., using the mixture-of-expeer gether with generative distributions pxxm zz, W.. modeled by decoders of each view, an Evidence Lower Bound (ELBO) for X can be derived with variational inference (Kingma & Welling 2014), which involves reconstructing Xn, using samples of q( (,,)x,) and ((z) x,n), for n, m I,...,
1 Σm= ((z)xm). To-
Since the posterior q(w,,x,) is dependent X,, and potentially retains shared information, to keep decoder p(xn |z, w,. from taking the "shortcut" to use leaked shared information, MMVAB+ uses sample of W.. from auxiliary prior r r(w,.. instead for cross-view reconstruction, where derived from another view m o nı; avoiding such shorteut enforces shared information to come from (m)) This design leads to the following minimizing objective:
M Σ
ppxxm (z, Wm Wmmp((z) (p())p((/)))) Wm)
II p D(Xx,12; wa) }
LMMVAE+ M
E
(e (2) xm.) wm. rig( (g( wm sem) [w., (wr w.ettm
log
q ( ())g((ve (Win Xm))
Within the expectation, p(xm Iz, W. and p(x, z. w are conditional likelihood of self- and cross- reconstructions, respectively. Palumbo al. (0023) shows that -LXMVAES (x1:M))((X1:)) S log(x1:M)
2

---

Published as a conference paper at ICLR 2026
remains valid ELBO. Compared with previous multimodal VAE using the MoE parameteriza- tion (Shi et al. 2019), MMVAE+ achieves superior performance for extracting shared information, although the result Is somewhat sensitive the relative capacity (dimensionality) of z and w. In the rest of this section, we will show that regularizing the generative model with mutual information (MI) is more effective achieving disentanglement. We emphasize that MMVAE+ is one option for multimodal VAE, and our improvement below can also be applied to other models, c.g., PoE (Wu & Goodman 2018) or MoPoE (uttter et al. (021).
2.2 SHARED VARIABLE EXTRACTION WITH CROSS-VIEW MI MAXIMIZATION
While general (per-dimension) disentanglement of variations is theoretically challenging (occteell et al. 2019), the underlying structure of our setup that inputs of different modalities share common cause facilitates (variable-level) disentanglement of shared versus private information.
To extract the shared information, it is natural to enforce the shared representation of modality m, denoted 7m.. (with distribution q(2 xm)) )) to have high mutual information (MI) with x, for # m. We note this is partially pursued by LMMVAE+, indirectly through the cross-view recon- struction: in light of the decomposition I (Zm, Wn; Xn) = [(zm; x..) + (Wxxi Zm). we can maximize I (zm; Xx, by maximizing I(zm: Wn; x,) while minimizing I ([w,; Xn, (m)). Focusing on [Z., Wa; X.. H(x)) the conditional entropy H (x,, Zm, W..) Ex.. Nm...,..... [- log pl (x,, Imm wn)] is equivalent to max- imizing conditional likelihood, as done by MMVAE+. However, maximizing this upper bound I( W,; x,) alone does not ensure maximal (zm x,), due to the gap [(w,; (m)) Therefore, likelihood maximization alone does not ensure the extraction of a shared variable.
(x,, Zm, Wn) where the entropy H Hx,,) is a constant, minimizing
We thus take the alternative approach to maximize 1 (Zmm Zn) which is a lower bound of 1 [(zmi Xn):: 1 [Ziii x,,) I iZ Z,, x.,) 1 (Zmi i Z,, (x,) [Zz Z,; x,)) 1(zmi Z,,) + I (z X,, Z,) 2 I [zm: 2n), where [z. Zn xx)) = in the first step due to variability of z, coming from X., only (Federici al. 2020 In this work, use the contrastive estimate of MI (Oord et al. 2018):
Ó(Zm: Zn))
[(zmi Zn) xe Contrast(Zm Zn) == Ez 2n log
Ólmm, Zn) + > 8 (2m,, Zc)
is the affinity function, and (z) ( are i negative examples randomly sampled from the minibatch not aligned with Zm- Since we have M modalities, we compute the average of cross- modoalty MIs as our regularization for extracting shared information:
where ((zm. 2n) = exp
2n
2m
LCmssMI = M(M - 1) Σ mnn Contrast(Zms zn))
2.3 DISENTANGLEMENT WITH GENERATIVE AUGMENTATION
While equation I encourages z to capture shared information across views, it does not guarantee that the learned Zm contains no private information which should be modeled by Wm- Similarly, even if Zm contains no private information and the self-reconstruction term in LMMVAE+ encour- ages (Zm, Wm) to jointly capture all information about xm, the learned Wm can still retain shared information. Thus, we need additional regularization to remove redundancy between Zm and wm.
To motivate our method, consider the desired scenario where Zm and wm are disentangled, so that they each can be varied independently to generate new samples of xm. using the decoder. Let Xm and x/,, be two input samples, and let (zm, ) be a pair of samples drawn from the posteriors q(z(Xm) and q( ((),) Xx,), respectively, and similarly (z w'r)) ) be pair of samples drawn from conditional posteriors for x',,- With disentanglement and a good likelihood model, a sample x ~ p(xm Iz,, w'n would share the same with X,. In turn, when we map x back to the latent space, ((((xt)) and (())x..) should be similar. Likewise, q( (xt)) and g( ((wml) g(wmlx..) should be similar.
More formally, assume that Zm is sufficient for x,,, meaning that it captures all shared information, ie., [(zm: x,) = [(xm x,) as encouraged by LCiossM-- Then in view of [[zm; (()m ) = H Hzz)) H (zm)) Xx),, ), we would like find the minimal Zmm (with lowest H (Z..)) by minimizing
H (Z = Ee x., |- p(zm x,.] xe Exxp(()) am q(x/xx..)) ..) 1- log q(z Zm x..]
3

---

Published as a conference paper at ICLR 2026
Similar approaches have been used by Federici et al. (2020 symmetric KL for minimizing [Xm, Zm X,,) and Tsai et al. (2019) inverse prediction) for learning minimally sufficient shared variable. Essentially, for extracting shared information, Xm and x, indeed constitute two views that are mutually redundant, satisfying 7(xm; x,|2z) = 0, so that the IB principle naturally applies.
Note, however, do not have multiple natural views sharing Wm. carry out the above idea. This challenge motivates us to synthesize the view x ~ p(Xmm Zms w'). with w'n ~ +((w) x from another sample xm to reduce the redundancy of learned Wm by approximately minimizing H ([wm] ([m[Xmm) with the following loss
Xx xm==p(xXm))s (xpim)m (amaamm))mmmrppp [K.) ).n... (wg(Wm (. (p pppppppmmmmm) (en///////mm.e w | log q( ((ww("' [=]x)
Assuming that posterions are parameterized Gaussians, this loss reduces to l loss for matching means of posteriors. In practice, we find it more effective to use a contrastive loss for matching, i.c.,
LGenAug.w. == -Contrast(ww) W(n where X ~ p(xm Zm, w'n). w" ~ (.
We show results obtained with contrastive estimation in the main paper, and provide empirical com- parisons of the two implementations in Appendix [A] We also define LienAug.za similarly by switch- ing the role of Zm and W.. The total redundancy removal regularization is defined as
1
Σ
M
LGenAug
m= (CGenAug.... + LGenAug ww w..)
2M
Although we do not have the reconstruction target for x matching ((((xt)) with g(2) xm and matching ((wm|xx) with g( (mm(x',,)) implement form of cycle-consistency (Zhu et al. 2017). and provide learning signals for both the encoder and the decoder. Previously, Bai et al. (2021) derived an ELBO of sequence data for disentangling static versus dynamic components, which in- volved mutual information terms based on data augmentation, similar to LeenAugg However, their augmentation requires strong domain knowledge (e.g., shuffling the frame order does not alter the static component, and color change applied to all frames does not alter the dynamic component). In
contrast, our augmentations require no domain knowledge and are produced by the model itself.
2.4 THE FINAL IDMVAE OBJECTIVE
We define our objective of Information- Dissnttion--Dseettngeeddd Multimodal VAE (IDMVAE) as min LIDMVAE == LAMVAE+ X CCrssoMII + X2 AGnnnnu
where X and X2 are user parameters tuned on the validation set.
Diffusion Priors In most multimodal VAEs, the prior distributions are chosen to be simple and easy sample from, e.g., Gaussian for continuous data. However, such unstructured priors may not be ideal for representation learning, whose purpose is to discover the useful structure of data for supervised downstream tasks. As an example, representation containing rich label information most likely have a clustering structure where data of different classes are separated far apart, and will have a uni-modal distribution like the Gaussian. We use diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020; (oog et al. [022] to overcome this limitation by parameterizing p ((z) as a denoising process started with pure noise. To naturally introduce diffusion models into our loss,we decompose the KI. divergence inside LAMVAE+ (which we minimize) as (Vahdat et al. 2021):
Bg(x(x) [logg(z)>x) g(z)x)] + Ex(x [- log)(())
The first term maximizes the entropy of the approximate posterior q x). The second term max- imizes the likelihood of samples from qzllx) under p(z), which we model with diffusion models. We can treat z ~ q(z[x) as "data", and destroy its structure by gradually adding noise to it, resulting in pure noise after a number of steps. With repeated applications of a denoising network, diffu- sion models gradually reverse the noising process and recover the original data from pure noise. Diffusion models have well-defined ELBO objectives which lower bound log p(z), and plugging them into equation 3 yields valid upper bounds of the KL divergence. Since the latent variables are of low dimensionality, we parameterize the diffusion backward process with a simple feedforward network. In practice, mean of q(z(x) with the DDPM parameterization (Ho et al. 2020). We optimize over all modules (encoders, decoders, diffusion networks) jointly in an end-to- endd-t--ndd manner. A recent work (Palumbo) et al. 2024) proposed two-step approach which first learns the representations with MMVAE+. and then learns diffusion models in the input space, conditioned on VAE reconstructions. Note our use of diffusion model has different motivation, and we jointly train it during representation learning.
introduce additional loss weight for [g(e)x) llg p(2)], and model the
4

---

Published as a conference paper at ICLR 2026
3 RELATED WORKS
Disentanglement in VA To achieve disentangled latent representations in VAEs, researchers have commonly used mutual information (MI) based regularization, and employed various metrics to assess the results (Higgins et al., [017] Kim & Mnih] 2018;; Chen et al.) [2018] Kumar et al.) 2018). However, it has been shown that, without supervision inductive bias in the model, it is theoretically challenging to recover (per-dimension) disentanglement (Locatello et al. 2019).
Contrastive and self-supervised learning (SSL). SSL is applied to a single modality, with artifi- cial views created based on the structures of data (Oord et al. [0188] Logeswaran & Lee [2018] Hjelm
et al.
2019 Bachman et al. 2019, Chen et al.) 2020b; Caron al. 2020 Tian et al. 2020a Bardes
ct al.
2022 Zbontar et al. 2021). as well as multimodal data (Radford et al., 2021 Jia et al. 2021
Elizalde et al.) 2023). and many methods are motivated by the classical infomax principle (Linsker
1988) and they implement neural estimation of mutual information, with contrastive loss being the
most popular variant. Recent works have proposed theoretical interpretations of SSL and contrastive
learning (Wang & Isola 2020 Zimmermann et al. 2021 Hyvärinen et al.) 2019 Tian et al. 2020b]
Tosh et al., 2021yCher et al.
2021 Zhai al. 2024), with the focus of providing guarantees for
extracting the shared variable, without considering the private variables.
A few works took private variations into consideration. von Kügelgen et al. (021)) proposed generative model in which the latent space is divided into "content" " and "style", importantly, data augmentations were assumed to preserve content while altering dimensions within style. Tsai et al. (202I) studied self-supervised learning from multi-view perspective and with the multi-view re- dundancy assumption (Chaudhuri al. 2009 Tosh et all]]220) that the private variable of each view contains little information for the downstream task, they focused on extracting the shared variable with combinations of several multi-view losses. Realizing the limitation of this assumption, Liang cet al. (2023) studied the scenario where the private variables contain significant useful information, and proposed a contrastive learning algorithm for extracting it. Their algorithm required sophisti- cated data augmentation procedures designed for the downstream task. Lyu et al. (2022) proposed a model for understanding SSL, assuming a data generation process similar to ours. They extracted the shared variable with CCA loss and the private variable by MI minimization.
Information bottleneck (IB) and mutual-inffrrmation regularization. Another set of proba- bilistic models was motivated by the IB method (Tishby et al.) 1999, (Tishby & Zaslavsky 2015] Achille & Soatto, 2018) 8). Alemi et al.] (2017) proposed a variational IB method to extract z from which has high MI with X2 (estimated with conditional likelihood), so that it captures the shared in- formation, and at the same time has low MI with X1 that it contains little nuisance factors/private information. Federici et al. (2020) leveraged the multi-view redundancy assumption that all the in- formation x₃ contains about unobserved label is also contained in X2, and showed that that if the learned representation z is sufficient, in the sense that [(x1, X² (z) = O, then z has all the predictive power from (x1, X2) for label. Remarkably, their objective did not involve any reconstruction paths, and the authors considered this to be an advantage, given that density modeling for high dimensional data difficult. |Wang al. (222)) extended Federici et al./ (2020) and proposed two-step approach to first extract shared and then private variables with guarantees, again without generative modeling. We argue that, with the development of powerful generative models, likelihood modeling becomes feasible and provides the additional benefit of (controllable) generation.
Multimodal latent diffusions. Recent works have proposed to first learn independent VAEs for each modality, and then couple their latent representation with diffusion models in second training stage (Bounoua et al. 2023 Wesego & Rooshenas] 2024). A key limitation of this approach is that separately trained VAES may fail to capture cross-modal correlations, leading a loss of information that the diffusion model cannot subsequently recover. In contrast, our latent prior is jointly trained with the rest of the objective and regularization terms, with the gradient of the latent diffusion loss propagated to the encoders. More importantly, these works do not perform disentanglement of the shared versus private variables, and therefore do not offer the same level of control as we do, e.g., we can combine and from different data points or samples from the prior to generate new inputs.
EXPERIMENTS
We compare method, IDMVAE, and its variant with diffusion priors, against several baselines.
5

---

Published as a conference paper at ICLR 2026
Table 1: Latent classification on PolyMNIST-Quadrant, Accuracies are averaged over modalities. For methods with single latent variable in each view, evaluation results are collected under 2.
Model
2 + Digit 1
- Quad +
w Quad
- Digit 4
MMVAE
0.492
0.798
MoPoE- VAE
0,536
0.751
DMVAE
0.157
0.254
0.710
0.179
MMVAE+
0.382
0.355
0.999
0.341
SBM-VAE
0.263
0.995
IDMVAE (ours)
0.983
0.271
0,999
0.162
CrossMi (λ1 0)
0.111
0.267
0,999
0.356
LeenAug (Ag = 0)
0.977
0.277
0.999
0.202
+ Diffusion prior
0.982
0.267
0.999
0.143
MMVAE (Shi et al. 2019): uses a MoE inference network to combine information from different modalities. only models the shared variable z with ELBO.
MoPoE- VAE (Sutter et al.
2021):
uses a mixture-of -exxuure-f--rrouuctt--ff pow--pcsooooooooooooee -xperts inference network for z.
DMVAE LLe & Pavlovic
2021
performs PoE inference for z, and models Wm within ELBO,
MMVAE+ (Palumbo et al.
2023) : performs separation of shared versus private information with the
help of auxiliary prior variables. It is special case of IDMVAE (w.o. diffusion) with À X 0.
DisentangledSSL (Wang et al. 2025): performs extraction of shared variable (using the method of Federici et al. 2020) and private variable in two sequential steps. It a state-of-the-ar disentan- glement method without likelihood modeling, but it can only be applied to two views currently.
SBM-VAE (Wesego & Rooshenas 2024): first trains individual VAEs for each modality with a sin- gle latent variable, and then couples modalities with diffusion modeling on the joint representations.
4.1 RESULTS ON POLYMNIST POLYMNIST-QUADRANT
PolyMNIST (Sutter et al., 2021 is benchmark for multimodal representation learning, consisting of MNIST (LeCun et al. 1998) digits overlaid on com- plex backgrounds. We make the dataset more chal- lenging by taking each MNIST digit and placing 32x32 scaled version of it into one of four quad- rants of 64x64 canvas; Figure for for an illustra- tion. This modification introduces the private latent variable which captures the quadrant position (with ground truth label) and background for each modality, allowing for nuanced evaluation of disentanglement and generation. Our training/valie saining/validationnees sets con- tain
Figure 2: DolyMNNISTQuadrrat dataset. Digits (0-9) are placed in one of the four quadrants randomly. Each column contains one multimodal sample. Each modality has different background scheme. The digit label is shared across all modalities, while quadrant label is private to each modality.
0/00055000/10,000 samples. We use the deep residual network (He et al.] (015) architecture as the backbone of encoders and decoders for all methods. The dimensionality is set to 32 for and 128 for wm.
Latent Classification. For evaluation, we perform linear classification on the samples of posterior distributions (samples reflect both mean and variance of posteriors). Multi-class logistic regression models are trained on the posterior samples of the training set and applied to the posterior samples of the test set. We perform two types of classifications: (1) predicting shared label from the shared variable and private label from the private variable Wm. where high accuracy is better, indicating the desired variation is captured; and (2) cross-classifictioon, where we predict shared label from Wm and predict private label from z. Ideally, with successful disentanglement, cross-classification random classifier (e.g.. 10% for predicting digits from Wm. 25% for predicting quadrants from z). We present results of different methods in Table as well as performance of our method when either LomaMI or LGenAug is removed from our loss. Clearly, our method achieves superior performance. LCrosoMI is critical for extracting the shared
accuracies should approach the performance

---

Published as a conference paper at ICLR 2026
Table Generative coherence, averaged over 5 views, on PolyMIISSTQQuadratt We use subscript to indicate samples from posteriors and subscript to indicate samples from priors. For generated images, digit label is determined by zz.q or otherwise random (with target accuracy 10%). quadrant label is determined by Wa,9 or otherwise random (with target accuracy 25%).
Self Gen (s =
Cross Gen (s #
Uncond.
Model
Gen Ka.q: Wep Gen Wa.g)
Gen GZsgz Wt.p)
Gem Sp. wpl
Digit Quad 4
Digit + Quad
Digit 1
Digit
MMVAE
0.170
0.041
MoPoE- VAE
0.173
0.029
DMVAE
0.297
0.252
0.532
0.999
0.161
0,005
MMVAE+
0.120
0.251
0.915
0.999
0.119
0.000
SBM-VAE
0.158
0.007
IDMVAE (ours)
0.898
0.249
0.162
0.999
0.881
0.070
CrosssMi (A1 = 0)
0.101
0.252
0.926
0.999
0.100
0.000
LeenAng (A2 0)
0.670
0.250
0.370
0.999
0,671
0.008
+ Diffusion prior
0.942
0.251
0.106
0.999
0.887
0.664
MMVAE
MoPoE-VAE
DMVAE
MMVAE+
IDMVAE Diff.
Figure 3: Unconditional generations on SAMMIIST-Quadran Each row is multimodal sample generated with a prior sample of z, images in the same ideally have the same digit identity.
variable, and this is because the digits occupy a small number of pixels and pure likelihood modeling may ignore them. GGenAug helps remove redundant information, so that cross-llassffication accuracy is reduced. Adding diffusion in latent space (last row of table) leads to small gain.
Conditional Coherence. This metric evaluates the model's ability generate consistent samples across modalities. We assess this for both self-reconee relf-rrcoostruucion and cross-modal generation. Formally, we combine either posterior 2s.g ~ q(z|x,, or prior Zs.p ~ p(z) (using diffusion prior if available) of a modality with the posterior Wt.q ~ q(w] (x,) or the prior Wt.p p(w:)) (using diffusion prior available) of modality f, and apply p(x; z, We) to generate a new sample modality t. This sample should have the same digit label as x, if the posterior of z is used, and random digit label if the prior is used. Similarly, the quadrant label can be determined based on whether posterior or prior used for w,. We then ResNet classifiers trained original images to predict corresponding labels of generated images, and the averaged accuracy across modalities is referred to as coherence. We provide conditional generative coherence in Table (left panel for self generation where and middle panel for cross generation & f); see Appendix B.2 for sample generations. The results are consistent with those of latent classification, and diffusion priors significantly boost coherence.
Unconditional Coherence. This metric further assesses the consistency of the shared informa- tion in unconditionally generated samples. We first sample a shared latent code Zp p(z) (us- ing diffusion prior when available). For each modality m, we then sample an independent private Wm.p ~ p( (wm) and generate a sample xm. from the combined latent code (Zp, Wm.p). The gen- erated multimodal sample ( are then passed to their respective digit classifiers (ResNet) trained on original training images, to predict the shared label. A sample set is considered coherent if all classifiers agree the same shared label. We report the percentage of coherent sets uncon- ditional coherence, shown in Table (right panel). Most methods obtain close to zero unconditional coherence, indicating the difficulty of matching prior and posterior distributions for latent variables.

---

Published as a conference paper at ICLR 2026
Table Latent classification on CUB-HQ. using posterior means, ž and W1 refers image latents.
Model
Z + Cat. 1
21 7 Dir.
WI 7 Dir. 1
w - Cat.
MMVAE
0.602
0.727
RoPoE-VAB
0.615
0.777
DMVAE
0.511
0.658
0.732
0.394
MMVAE+
0.633
0.787
0.567
0,486
SBM- VAE
0.452
0.799
DisentangledSSL.
0.672
0.538
0.567
0.173
IDMVAE (ours)
0.764
0.592
0,677
0.322
CroasMI (λ1 0)
0.614
0.792
0.557
0.379
LeenAug (A2 0)
0.777
0.498
0.792
0.395
Diffusion prior
0.752
0.526
0.797
0.311
However, with diffusion prior, our method achieves significantly better coherence, thanks its flex- ibility. We show generations in Figure 3 and 2D visualizations of latent codes in Appendix B.3
Generative augmentation. Recall that LGenAug we mix and match posteriors of and wm from different samples to generate new samples in modality m. We provide illustration of such sam- ples from our trained model in Figure 4 The first row and first column contain images for which we extract posterior samples of z and Wm respectively. And the rest of the grid contain generate images using samples of Wm of the corresponding row and of the corresponding column. We observe that images in each column share the same digit, while images in each row share the same quad- rant, as desired. Generated images are of high quality, showing that we can independently vary shared and private variables to obtain controllable generations.
3
3
3
2
3
Figure 4: Augmentations.
4.2 RESULTS ON HIGIT-QUALTYY CUB (CUB-HQ)
The CUB-200-2011 dataset (Wah et al. 2011 Reed et al. 2016 Shi et al. 2019 Palumbo et al.) 2023 2024) is widely used benchmark for fine-graine one-grained visual categorization, containing RGB im- ages of 200 bird species. Each image is paired with 10 textual descriptions. Following Palumbo et al. 2024), we group 22 categories of species from the 200 bird species into super-categories, yield- ing I-of-8 class labels for these species. Data with category label is split into vraiiig/valldatioon// with 70%/15%/15% portions. The rest 178 species are added to the training set for representation learning. The wraniin/validattion/eess sets contain 113,900/1,944//2.044 samples, respectively.
In this experiment, we use the 256x256 resolution version of the original CUB dataset (Wah et al, 2011) without cropping images using bounding boxes for birds. Compared the cropped version used in prior works (Shi et al. 2019 Palumbo et al. 2023). in our setup, each image contains richer background and is more challenging for representation learning. For this dataset, the two modalities (image, text) share rich information about bird category, since the text describes the color of different parts of the bird. To evaluate the quality of private information. we note that the horizontal direction of the bird (with direction inferred from the original CUB attributes) can only be inferred from the image. Therefore, we consider the direction as private label for the image modality. We refer to this dataset as high-quality CUB (CUB-HQ); see more details data generation in Appendix C.I
We pre-process the image modality with pretrained- -AE encode similarly used by Peebles & Xiel (2022). converting each original RGB images to 4 x 32 x 32 tensor, which then serves as the model input. We use ResNet as encoders and decoders for pre-processed input images, while convolution network those for texts (using one-hot representation of text with vocabulary of 1,590 words). The dimensionality is set to 256 for both z and w..-
Representation quality We perform linear classification on the latent representations z and w to exam their information content; the results are given Table Note that with disentangled latent representations, the target classification accuracy is 50% (i.e., random) for predicting direction from
hhans//huuginngfcccccoaahhhhhiip
8

---

Published as a conference paper at ICLR 2026
Table Generative coherence, as measured by FID and CLIPscores on the CUB-HQ test set. As reference, the CLIPScore between ground truth test images and text is 0.762. For image-to-image generation, we use the posterior of and the prior of w.
Text-to- Img
Img-to-text
Img-to-Img
Model
FID 4
CLIP
CLIP
FID |
CLIP
DMVAE
104.167
0.665
0.683
70.534
0.707
MMVAE+
70.157
0.691
0.693
62.528
0.712
3BM-VAE
79.900
0.684
0.687
IDMVAE (ours)
64,435
0.718
0,736
58.065
0.721
CrosssMi (A1
72.166
0.694
0.692
62,938
0.710
LeenAng (22 = 0)
66,291
0.709
0.719
69.988
0.702
Diffusion prior
60.549
0.721
0.737
59.700
0.716
MMVAE+
IDMVAE (Ours)
+ diffusion prior
Shd itt AL.
Hetil lec
Hi
Figure 5: Top: Text-to-image generation on CUB-HQ, with the DiT denoiser. We combine the posterior sample z of the text (top row) with the prior sample w the image modality (shared by each row) for generation. Ideally, images in the same column are of similar bird category and color, and images in the same row have the same bird orientation. Bottom: Image-to- soge-too-mmaee generation. We combine the posterior sample of the image (top row) with the prior sample of the image modality (shared by each row) for generation. Ideally, images in the same column are similar bird category and color, while images in the same have the same bird orientation.
z, and 12.5% for predicting category from WT (derived from image). We observe that cross-view MI is critical for recovering when we do not have very strong likelihood model (due to limited image data). On the other hand, generative augmentation significantly reduces redundancy in the latent space by removing the directional information from z and capturing it in w. Adding diffusion prior leads to the best performance overall, demonstrating effectiveness of a flexible latent prior. Our methods outperform both generative and deterministic baselines by clear margin.
Generative coherence After training, our model can generate images channels and 32 x 32 resolution, which can then be decoded using the pretrained- VAE decoder into 256 256 RGB

---

Published as a conference paper at ICLR 2026
Table 5: Prediction accuracy on TCGA, averaged over modalities and splits.
Model
t
1
MMVAE
0.695+000110
MoPoE-VAL
0.695+0.014
DMVAE
0.6881100118
0.691
0.697
MMVAE+
0.692+000110
0.690#0.012
0.690=0.011
DisentangledSSL
0.691+0.011
0.691#0.012
0.690=0.011
IDMVAE (ours)
0.707
0.718=0.017 0.708
LCrosMMI (A1 0)
0.691
0.689#0.010
0.691=0.014
GGenAug (A2 = 0)
0.701
0.706=00019
0.723=0.013
+ Diffusion prior
0.714++.009
0.719+
0.731
images. However, given the relatively small training set size, the generated images tend to be blurry. Following Pandey et al.] (2022) and Palumbo et al. (2024), we train a diffusion model to generate high-quality images conditioned
our model's generations, using DiT architecture (Peebles & Xie| 2022). This DiT acts as a denoiser for our model; see Appendix details. To measure the generative coherence of models, we perform image-to-text, text-to-image, and image-to-image generations. We measure the quality of generated high-resolution images with
for more implementation
FID (Heusel et al. 2017). and the coherence between text and images using CLIPScore (Hessel et al. 2021). These metrics are given in Table 4 and our model's generation performance exceeds the baselines. We provide samples of text-to- imoge and image-to-ine to-image generations in Figure
In Appendix C.3 we provide more generations with and without the DiT denoiser. The success of our approach shows that LGeaAug is effective even when the generative augmentations are not of very high quality. While the IDMVAE samples produced without DIT exhibit blurriness, they success- fully capture core features (color and orientation of the bird), thereby facilitating disentanglement. The DiT denoiser, finetuned separately, can then fill in the details for high-quality generation.
4.3 RESULTS ON THE CANCER GENOME ATLAS (TCGA)
TCGA datase real-world multi-omics dataset that is by nature multimodal. Using the same data processing procedure from Lee & van der Schaar 2021), we obtain a dataset of 10,960 samples (of which 9,477 are labeled) with 5 views (each of 100 dimensions), each representing a molecular modality and labels (see Appendix D.I for details). The binary label represents 1-year mortality of a patient-sample. We selected 2 views (mRNA and miRNA) which had 9,874 samples out of all possible combinations after filtering out samples with missing values. After adding data with missing labels to the training set, 90%/5%/5% split was performed with 5 different seeds. Due to the complex nature of biological data, private information may be predictive as well.
As shown in Table 5 our method in general performs better than baseline methods terms of ac- curacy (see Appendix D.3 for AUROC results), both learned shared and private latent spaces are predictive, and combining z and w achieves the best performance. This is likely because clean dis- entanglement separates predictive information between shared and private latent variables, making predictions based on combined latent space more robust. In particular, LcrosM contributed most to the performance, and adding diffusion priors in latent space consistently improves performance.
CONCLUSIONS
We have proposed IDMVAE, a generative model for learning disentangled representations from multimodal data. Our innovations include the incorporation of cross-view mutual information maxi- mization for shared variable extraction, redundancy removal based on generative augmentation, and flexible latent priors with diffusion models. These components are complimentary to each other and jointly overcome the limitations of pure likelihood modeling, resulting in superior performance than existing otate-of-the-art multimodal VAEs as well as non-generative disentanglement methods. In the future, we would like to extend the model to handle missing modalities (Lee & van der Schaar) 2021], leveraging the controllable generation capability of our model, and to scenarios where infor- mation is shared locally instead of globally among the views (Liu et al. 2023).
hhtps//www.caneergovhggg
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. Journal of Machine Learning Research, 19(1):1947-1980 2018.
Wiku B. Adisasmito, Salama Almuhairi, Casey Barton Behravesh, Pépé Bilivogui, Salome A. Bukachi, Natalia Casas, Natalia Cediel Becerra, Dominique F. Charron, Abhishek Chaudhary, Janice R. Ciacci Zanella, Andrew A. Cunningham, Osman Dar, Nitish Debnath, Baptiste Dungu, Elmoubasher Farag, George F. Gao, David T. S. Hayman, Margaret Khaitsa, Marion P. G. Koopmans, Catherine Machalaba, John S. Mackenzie, Wanda Markotter, Thomas C. Metten- leiter, Serge Morand, Vyacheslav Smolenskiy, and Lei Zhou. One health: new defini- tion for a sustainable and healthy future. 18(6)ee010537, 2022. ISSN 1553-7374. doi: http:///jooaapppp.111100000
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. In ICLR, 2017.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurlPS, 2019.
Junwen Bai, Weiran Wang, and Carla Gomes. autoencoder. NewrIPS, 2021.
Contrastively disentangled sequential variational
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varinne-imvaiinn--ovviinc regulariza- tion for self-supervieed learning. In ICLR, 2022.
Mustapha Bounoua, Giulio Franzese, and Pietro Michiardi.
Multi-modal latent diffusion.
arXiv:2306.0444 044454 [cs.LG], 2023.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
Kamalika Chaudhuri, Sham M. Kakade, Karen Livescu, and Karthik Sridharan. Multi-view cluster- ing via canonical correlation analysis. In ICML, 2009.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP. 2020a.
Jeff Z. Hao Chen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In NeurlPS, 2021.
Ricky T. Q Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentan- glement in vaes. In NeurlPS, 2018.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In /CMI, 2020b.
Imant Daunhawer, Thomas M. Sutter, Kieran Chin-Cheong, Emanuele Palumbo, and Julia E. Vogt. On the limitations of multimodal VAEs. ICLR, 2022.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. CLAP: Learning audio concepts from natural language supervision. In ICASSP, 2023.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In ICLR, 2020.
Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass. Contrastive audio-visual masked autoencoder. In ICLR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference-free evaluation metric for image captioning. In EMNLP, 2021.

---

Published as a conference paper at ICLR 2026
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurlPS, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAB: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Karan Grewal, Phil Bachman, Adam
Jonathan Ho, Ajay Jain, and Picter Abbeel, Denoising diffusion probabilistic models. In NeurIPS, 2020.
Aapo Hyvärinen. Hiroaki Sasaki, and Richard E. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. AISTATS, 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. ICML. 2021.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. ICML, 2018.
Jongsuk Kim, Hyeongkeun Lee, Kyeongha Rho, Junmo Kim, and Joon Son Chung. EquiAV: Lever- aging equivariance for audio-visual contrastive learning. In ICML, 2024.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. In ICLR, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEK, 86(1 0122278-2244 1998.
Changhee Lee and Mihaela van der Schaar. A variational information bottleneck approach multi- omics data integration. In AISTATS, 2021.
Michelle A. Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Ani- mesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In International Conference on Robotics and Automation (ICRA). 2019. doi: 11111///CAAA221..89995....
Mihee Lee and Vladimir Pavlovic. Private-shared disentangled multimodal VAE for learning of la- tent representations. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021.
Paul Liang, Zihao Deng, Martin Q. Ma, James Zou, Louis Phili-P Morency, and Russ Salakhut- dinov. Factorized contrastive learning: Going beyond multi-view redundancy. In NeurIPS, 2023.
R. Linsker. Self-organization in a perceptual network. Computer, 21:3):105-111 1988. doi: 10. 1109/2.36.
Tennison Liu, Jeroen Berrevoets. Zhaozhi Qian, and Mihaela Van Der Schaar. Learning representa- tions without compositional assumptions. In ICML, 2023.
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Scholkopf, and Olivier Bachem. Challenging common assumptions the unsupervised learning of disentangled repre- sentations. In ICML, 2019.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representa- tions. In ICLR, 2018.
Qi Lyu, Xiao Fu, Weiran Wang, and Songtao Lu. Understanding latent correlation-banl based multiview learning and self-supervision: An identifiability perspective. In ICLR, 2022.
12

---

Published as a conference paper at ICLR 2026
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv:1807.085 .03748, 2018.
Andrew Owens and Alexei A. Efros. Audio-visual scene analysis with self- supervuee multisensory features. In ECCV, 2018.
Emanuele Palumbo, Imant Daunhawer, and Julia E. Vogt. MMVAE+: Enhancing the generative quality of multimodal VAEs without compromises. In ICLR, 2023.
Emanuele Palumbo, Laura Manduchi, Sonia Laguna, Daphné Chopard, and Julia Vogt. Deep generative clustering with multimodal diffusion variational autoencoders. In ICLR, 2024.
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffuse Efficient, controllable and high-fidelity generation from low-dimensional latents. Transactions on Machine Learning Research, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine- grained visual descriptions. In CVPR, pp. 49-58, 2016.
Yuge Shi, N. Siddharth, Brooks Paige, and Philip H. S. Torr. Variational mixture-of offereeo-t--e-tttt-e- autoen- coders for multi-modal deep generative models. In NeurlPS, 2019.
Yuge Shi, Brooks Paige, Philip Torr, and Siddharth N. Relating by contrasting: A data-efficient framework for multimodal generative models. In ICLR, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In (CML,2015,
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Thomas M. Sutter, Imant Daunhawer, and Julia E. Vogt. Generalized multimodal ELBO. In ICLR, 2021.
Masahiro Suzuki, Kotaro Nakayama. and Yutaka Matsuo. Joint multimodal learning with deep generative models. orXiv:1611.0189// 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NeurlPS, 2020b.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), 2015.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In 37th Anmual Allerton Conference on Communication, Control and Computing. 1999.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, 2021.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis Phili-Ph Morency, and Ruslan Salakhut- dinov. Learning factorized multimodal representations. In ICLR, 2019.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from multi-view perspective. In ICI.R, 2021.
13

---

generative modeling in latent space. In Published as a conference paper at ICLR 2026
Arash Vahdat, Karsten Kreis, and Jan Kautz. NeurIPS, 2021.
Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR, 2019.
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In ICLR, 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-uesd birds-200-2011 dataset. Technical report, Caltech, 2011.
Chenyu Wang, Sharut Gupta, Xinyi Zhang, Sana Tonekaboni, Stefanie Jegelka, Tommi Jaakkola, and Caroline Uhler. An information criterion for controlled disentanglement of multimodal data. In ICLR, 2025.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In ICMI., 2020.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Liveseu. Deep variational canonical correla- tion analysis. arXiv: 1610.03444 2016.
Daniel Wesego and Pedram Rooshenas. Score-based multimodal autoencoder. In Transactions on Machine Learning Research, 2024.
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learn- ing. In NeurIPS, 2018
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In JCML, 2021.
Runtian Zhai, Bingbin Liu, Andrej Risteski, J Zico Kolter, and Pradeep Kumar Ravikumar. Under- standing augmentation-based self-supervised representation learning via RKHS approximation and regression. In ICLR, 2024.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, 2018.
Jun-Yan Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthis Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In ICML, 2021.
14

---

Published as a conference paper at ICLR 2026
A DIFFERENT IMPLEMENTATIONS FOR GENERATIVE AUGMENTATION
In section 2.3
we have discussed two implementations of generative augmentation for redundancy removal. Here we provide detailed comparison of them.
Recall that Xm and x are two input samples, (Zm, Wm) is pair of samples drawn from the posteri- g(z/xm) and q( (m (xm) respectively, and similarly (z' w(.) is pair of samples drawn from conditional posteriors for xm With disentanglement and a good generative model, we could inde- pendently vary one variable while keeping the other the same obtain new sample. In particular, sample X p(x, Imm, w'n) would share the same with Xm- In turn, when we map x back the latent space, (z[[x() (m)) and q(z(xm) should be similar. Likewise, ((w, (x and q( ((w,, x should be similar.
Least squares matching. In the first implementation, we would like to minimize [zzm xn) by approximately minimizing ((wm x
Xmx p(x ), >p( - x...) (w )orth ~p(x.. Im.. (... wwn) log a( (
Assuming that posteriors are parameterized Gaussians, LeenAug,w.. reduces to lz loss for matching means of posteriors, and we implement it as
plog GenAug
E x.. (x....))))))))).))...... 3.x% (pp( Ao (x/ w. wi (wmi (x ppp(xm w/ || w" ||
where W is the posterior mean of (Wwm/ (xm)) while wht the posterior mean of q(wm (xx))).
Contrastive matching. In practice, we find it more stable to use contrastive loss for matching,
i.e.,
-=ooonlras(yve (w", w where x p(xm Immy wh)), w g(w (x
GenAug
We plug in the two different implementations into our loss. In and 7 we provide the compar- ison of the two on COAMINIST-Quadrran each with its loss coefficient tuned on the validation set. We find the best coefficients to be =1= and =0775 for Llag GenAug and =1=80 and Xpontrost_22 for GenAug diffusion prior loss has coefficient of 1.0 when incorporated. We observe that both implementations improve the disentanglement compared with using LCsssMii only, with Ceontroat cloq outperforming GenAug
Table 6: Comparison of clog and Conntrast for generative augmentation regularization in latent GenAug linear classification on PolyMNiST- Quaarratt. Accuracies are averaged over 5 modalities.
Our Models
Digit f
Quad
Quad 1
Digit
CCossMi Only (A2 = 0)
0.977
0.277
0.999
0.202
LCauMM GendAug
0.972
0.267
0.999
0.186
diffusion prior
0.980
0.263
0.999
0.154
CrossMI + GenAug
0.983
0.271
0.999
0.162
diffusion prior
0.982
0.267
0.999
0.143
15

---

Published as a conference paper at ICLR 2026
Table 7: Comparison of GenAug and Ceontrast for generative augmentation regularization gen- We use subscript q to indicate
clog
GenAug
erative coherence, averaged over views, on PolyMNISTTQuadde samples from posteriors and subscript to indicate samples from priors. For generated images, digit label determined by Za.q or otherwise random (with target accuracy 10%), quadrant label is deter- mined by Wa,q otherwise random (with target accuracy 25%).
Self Gen (s= t)
Cross Gen (s + f)
Uncond.
Our Models
Gen bs.g Wt.))
Gen(Zr. Wa.g
Gen(Zz.q Wt.p)
Gen(Zp: Wp
Digit
Quad
Digil + Quad
Digit f
Digit 1
ErooSMII Only (A2 0)
0.670
0.250
0.370
0.999
0.671
0.008
CC GenAug
0.817
0.250
0.219
0.999
0.812
0.044
+ Diffusion prior
0.917
0.249
0.109
0.999
0.875
0.668
CrossMI GenAug
Acontrast
0.898
0.249
0.162
0.999
0,881
0.070
+ Diffusion prior
0.942
0.251
0.106
0.999
0.887
0.664
16

---

Published as a conference paper at ICLR 2026
B DETAILS AND ADDITIONAL RESULTS ON POLYMNIST-QUADRANN
B.1 IMPLEMENTATION DETAILS
We utilize a deep residual network (ResNet) architecture of residual blocks, with the number of filters doubling from 64 to up to 512 after each block for the encoder, for all five modalities. Our model has a total of 201M parameters without diffusion prior and 206.5M Parameters with diffusion prior. And each modality's information is factorized in the latent space into a shared latent dimension using the Adam optimizer with a learning rate of 5e and batch size of 128, and use the other default hyperparameters of MMVAE+ baseline, including the KL divergence coefficient 3 of 2.5. We performed grid search over the coefficients to tune the regularization terms, Å and A2. after training for 100 epochs. We search them in the range [0.01, 100). We tune Å individually first to find the best general performance in latent classification for LCresMi and fix the X1, then combine with Ceontrast and find the best combination of = and Agontrast=20. Finally, we tune the diffusion GenAAu
and private latent dimension of 128. Models are trained for 100 epochs
prior weight to 1.0 out of (0.01, 0.1, 1.0, 10.0), which optimizes the final general performance at the 100th epoch.
B.2 CONDITIONAL GENERATION
In Figure 6 and Figure 7 provide additional results conditional generations, where one latent variable is sampled from the posterior, while the other is sampled from the prior. Figure g we provide conditional generations for which both and the samples we use LennAug- In Table g we give a quantitative measure the generation results using FID (Heusel et al., 2017). In all cases, our method provides the most coherent generations, consistent with the quantitative results in Section 4.1
sampled from posteriors; this simulates
Table 8: Generative quality, as measured by FID, on CIAMMIISSTQQuadaaa test set.
Cross Conditional
Uncondidtional
Model
FID +
FID
DMVAE
100.817
79.646
MMVAE+
86.091
87.008
SBM-VAE
128.7
128.8
IDMVAE (ours)
84.528
87.108
LCrosMi (Aı = 0)
85.589
85,563
LGenAug (A2 = 0)
84.698
85,596
Diffusion prior
73.186
73.681
17

---

Published as a conference paper at ICLR 2026
3
a
2
S
2
3
MMVAE
MoPoE-VAE
DMVAE
2
3
2
D
3
0
2
3
4
0
9
4
0
2
3
4
MMVAE+
IDMVAE
diffusion prior
Figure 6: Conditional generations on DolyMNiST-Quadraan with conditioning on z. The top row shows the samples (from modality 1) we condition on. We sample ~ (()xx)). ), sample the pri- vate variable from the corresponding prior Wm ~ p(wm) Xm). and generate new sample from p(xmle lz, Wm). Row 2 to row 6 generated samples for modalities to 5. Note for well- disentangled latent variables, each column shall contain the same digit z. For each row, we used the same prior sample of w, so images in the same row shall have the same quadrant, writing style, and background.
2
2
2
3
2
4
DMVAE
MMVAE+
IDMVAE
diffusion prior
Figure 7: Conditional generations PuuyMattT-Quadrrnt,, with conditioning Wm- The top row shows the samples (from modality 1) we condition on. We sample w ~ q(w [x])) and sample sample from p(x; Iz, w1). Row 2 to 6 are generated samples.
p(z), and generate Note for well-disentangled latent variables, each column shall have the same quadrant position and background. For each row we used the same prior sample of z, so images in the same row shall have the same digit.
18

---

Published as a conference paper at ICLR 2026
2
MMVAE+
IDMVAE
Figure 8: Conditional generation on PolyMNIST-Quadrant. The first row and first column contain images for which extract posterior samples of and w.. respectively. And the rest of the grid contains generated images using latent samples of the corresponding row and column. This figure illustrates the samples we in generative augmentation.
19

---

Published as a conference paper at ICLR 2026
B.3 LATENT VISUALIZATION
In Figure 9 and Figure 10 we provide 2D visualizations of latent representations of view 2 (i.e.. samples of ((4|x2) and q! (||x)), respectively) on the PolyMNIST-Quadran test set. Our method leads to improved separation of digit or quadrant classes in the latent spaces, as also shown in the quantitative analysis (Table [ Observe that, without a diffusion prior, there exists gap between the posterior and the Gaussian prior, whereas the capacity of the diffusion prior is strong enough ensure good overlap between the two distributions, and therefore leads to superior unconditional generation.
MMVAE+
IDMVAE
class
class
class
class
class
class
class
class
class
class
class
class
class
class
class
IDMVAE w. standard prior
IDMVAE w. diffusion prior
class
class
class
class
class
class
class
class
Figure 9: 2D visualization (by UMAP) of learned Z on the test set. We color each point according its ground truth digit label, and black markers correspond samples from the prior.
20

---

Published as a conference paper at ICLR 2026
MMVAE+
IDMVAE
class class
class
class
class
class
IDMVAE w. standard prior
IDMVAE w. diffusion prior
prior
class
class
class
Figure 10: 2D visualization (by UMAP) of learned wa on the We color each point according to its ground truth quadrant label, and black markers correspond to samples from the prior.
21

---

Published as a conference paper at ICLR 2026
C RESULTS ON HIGH-QUALITY CUB (CUB-HQ)
C.I DATASET
The super-categories. namely Blackbird, Gull, Jay, Oriole, Tanager, Tern, Warbler, and Wren, are created following the same grouping method introduced by Palumbo et al. (2024), as shown in Figure [I In addition, introduce private binary label representing the bird's horizontal direction, determined by the part location annotations provided in the original dataset (Wah et al. 2011) as illustrated in Figure T2 Specifically, group of the bird's 'head' parts with the average horizontal position the group of its 'body' parts. If the head is positioned to the left of the body, the direction label is 'left' (label 0); otherwise, it is "right' (label 1). This creates a modality-specificc (private) label for the image that cannot be inferred from the text captions. At the same time, as shown in Figure T2 (b) and (c), very small fraction of the images have invisible 'head" or "body" location annotations, or the locations are too close, in which case they are not assigned the direction label. Direction labels of validation and test images are verified by human.
compare the average horizontal position of the
Gull
Jay
Oriole
Rursty Blackbird
Red-winged Blackaird
Herring
Blue Jay
Florida Jay
Scott's Orlole
Hooded Oriole
Baltimore Oriole
Blackbird
Tanager
Tern
Warbler
Wren
Scarlet Tanager
Summer Tanager
Elegant Torm
Caspian Tarn
Pina Warbler
Vellow Warbler
House Wren
Garoline Wren
Artic Teem
Kentucky Warbler
Marsh Wren
Figure 11: CUB category labels: dividing 22 species into super-categories PPllumo et al. 2024
(Right Wing not tin)
(Righ
NOY
Ut
Terid
a. Collected Parts
Invisible
c. Ambiguous
The collected parts in the original dataset (Wah et al. 2011). and sample images with invisible "body" or 'head' location annotations or ambiguous horizontal direction.
Figure 12:
In this experiment, we use the 256x256 resolution version of the original CUB dataset Wah et al. 2011) without cropping images using bounding boxes for birds. Compared to the CUB dataset used in the main paper, the CUB-HQ setup contains richer background and more challenging for representation learning.
22

---

Published as a conference paper at ICLR 2026
C.2 IMPLEMENTATION DETAILS
Pre-trained VAE for data pre-processing We pre-process the image modality with a pretrained- VAE e similarly used by Peebles & Xie (2022), converting each original RGB images to 4 x 32 x 32 tensor, which then serves as the model input. We can then apply the corresponding pre-trained decoder on samples of our model to generate high resolution images.
Architecture of IDMVAE We use deep residual network (ResNet) for the image modality. Both the encoder and decoder have five residual blocks. For the encoder, the number of filters doubles after each block, starting at 64 and ending at 1024. The decoder mirrors this, with the number of filters halving after each block. For the text modality, we utilize a CNN-based encoder and decoder on one-hot encoded captions with vocabulary size of 1,590 words. Our model has total of 137M parameters without diffusion prior and 140M Parameters with diffusion prior. During training, we apply horizontal flip augmentation to the image modality, with flip probability of 0.5.
Hyperparameterss Both the shared variable and the private variable have 256 dimensions. Models are trained for 50 epochs using Adam optimizer with learning rate of 10 40-4 and batch size of 256, with the rest of the hyperparameters being the default of the MMVAE+ baseline, including the KI. divergence coefficient
1.0.
Then we tune the coefficients of LC Leenuug and diffusion prior similarly to B.1 and obtain the optimal coefficients A = 40, A2 = 10, and diffusion loss weight 0.1.
High resolution image generation After training, model can generate images of 4 channels and 32 x 32 resolution, which can then be decoded using the pretrained- VAE decoder into 256 x 256 RGB images. However, given the relatively small training set size of CUB, the generated images tend to be blurry. Following Pandey et al. (2022) and Palumbo et al. (2024), we train a diffusion model to generate high-quality images conditioned
our model's 's generations. Specifically, for each model, we extract its reconstructions of the training data by the image modality architecture (i.e.. applying two encoders obtain (zo, wo). followed by the image decoder), and train DiT model (Peebles & Xie 2022) to generate the ground-trute truth training images from pure Gaussian noise, conditioned on model reconstructions. We use the pre-trained checkpoint of the XI.-2 architecture, and add an additional patch embedding layer to map conditioning features into representations that are later incorporated into the AdaLN module of each DiT block. We finetune all parameters of the resulting model, with minibatches of 32 images for 70000 updates.
C.3 RESULTS
Generative coherence To measure the generative coherence of models, we perform image-to- text, text-to-image, and image-to-image generations. We measure the quality of generated high- resolution images with FID (Heusel et al. 2017). and the coherence between text and images using CLIPScore (Hessel et al., 2021]. These metries are given in Table For reference, the CLIPScore between ground truth test images and text is 0.762. The generation performance of our model is superior to those of the baselines. We provide samples of text-to-image generation Figure [3 samples of image-to-text generation in Figure 14 and samples of image-to-image generation in and Figure 16
Figure and
Generative augmentation To understand how generative augmentation and LGenAug help with disentanglement, we visualize such generations in Figure 17 The first row and first column contain images for which we extract posterior samples of and w, respectively. And the rest of the grid contain generated images using samples of w of the corresponding row and z of the corresponding column. We observe that images in each column mostly share the same bird color, while images in each row mostly share the same orientation as desired. It is important to note that, different from Poly the generated images with our ResNet architecture are not of high resolution, but they already capture essential information regarding shared and private labels. Conditional generations by DiTs are of high quality, showing that we can independently vary shared and private variables to obtain controllable generations for complex modalities.
23

---

Published as a conference paper at ICLR 2026
MMVAE+
IDMVAE (Ours)
diffusion prior
ihht l: 1r1.
MMVAE+
IDMVAE (Ours)
+ diffusion prior
FaE lE: At.
A. 2a
A. del
J.rdee
Figure 13: Text-to-imant generation CUB-HQ. without DiT conditional generation (top panel) and with DiT conditional generation (bottom panel). We combine the posterior sample of the text modality (top row) with the prior sample w of the image modality (shared by each row) for generation. Ideally, images in the same column are of similar bird category and color, and images in the same row shall have the same bird orientation.
MMVAE+
IDMVAE (Ours)
+ diffusion prior
K222 Ettt
f ef il.re
fi B. HH ffll
fl.... Fa.iiff
AA... fl. A1.3
Binll 12r
hil
H1. Had.
1118.
ShinlLLLE H.. ihst. is
H.t Estoct zea MI.. olaa
1a. la. ia
Wattl
il.. hid.. rit
St.tte of....
4 il. mimiii...
dE. fife 1.
3t. .lll..r
An titt E. 6 fift Sttl.
HR
Rebbur... schal
1al 33,52
Aifall sth
at.1 FOR Hil
Auala. Nh. Tiex
Ra. l
dlcll. !.
Figure 14: Image-to-text generation on CUB-HQ. We combine the posterior sample the image
modality (top row) with the prior sample of the text modality (shared by each row) for generation.
24

---

Published as a conference paper at ICLR 2026
MMVAE+
IDMVAE (Ours)
diffusion prior
MMVAE+
IDMVAE (Ours)
+ diffuusion prior
Figure 15: Image-to- im-ee generation on CUB-HQ, without DiT conditional generation (top panel) and with DiT conditional generation (bottom panel). We combine the posterior sample of the image modality (top row) with the prior sample w of the image modality (shared by each row) for generation. Ideally, images in the same column are of similar bird category and color, while images in the same row shall have the same bird orientation.
25

---

Published as a conference paper at ICLR 2026
MMVAE+
IDMVAE (Ours)
diffusion prior
MMVAE+
IDMVAE (Ours)
+ diffusion prior
Figure 16: image-to-image generation on CUB-HQ, without DiT conditional generation (top panel) and with DiT conditional generation (bottom panel). We combine the posterior sample of the image modality (top row), with the prior sample z of the image modality (shared by each row) for generation. Ideally images in the same column are of the same bird orientation, while images in the same row shall have similar bird category and color.
26

---

Published as a conference paper at ICLR 2026
MMVAE+
IDMVAE (Ours)
diffusion prior
MMVAE+
IDMVAE (Ours)
+ diffusion prior
Figure 17: Generative augmentations on CUB-HQ, without DiT conditional generation (top panel) and with DiT conditional generation (bottom panel). The first row and first column contain images for which we extract posterior samples of z and w, respectively. And the rest of the grid contain generated images using samples of of the corresponding row and z of the corresponding column. Ideally, images in the same column are of the same bird category and color, while images in the same row shall have the same bird orientation.
27

---

Published as a conference paper at ICLR 2026
Latent Visualization In Figure 18 and Figure 19 provide visualization of the latent repre- sentations of shared variables. Consistent with latent classification results (Table 3 our method leads to better separation of classes. Furthermore, diffusion priors flexible enough to model the structured latents.
MMVAE+
IDMVAE
class
class
class
class
class
class
class
class
class
class
IDMVAE w. diffusion prior
IDMVAE w. diffusion prior
class
class
class
class
class
dest
2D visualization (by UMAP) of Z1, i.e., learned shared representation of the image modality, on the validation set. We color each representation according to its ground truth bird category cluster label, and black markers correspond to samples from the prior.
Figure 18:
28

---

Published as a conference paper at ICLR 2026
MMVAE+
IDMVAE
class
class
class
class
class
class
class
class
class
class
Class
class
class
IDMVAE w. standard prior
IDMVAE w. diffusion prior
prior
class
class
class
class
Figure 19: 2D visualization (by UMAP) Z2, i.c., learned shared representation of the text modal- ity, on the validation set. We color each representation according to its ground truth bird category cluster label, and the black markers correspond to samples from the prior.
29

---

Published as a conference paper at ICLR 2026
D DETAILS AND ADDITIONAL RESULTS ON TCGA
D.1 DATA PREPARATION
TCGA dataset (2016 version) is processed by combining multi-omics data across different cancer types and different patients before features are selected and kernel PCA was performed to reduce the dimensionality 100. Because clinical trials usually have small sample sizes, high dimensionalities, and complex dependencies, this dataset is ideal to test the robustness of our method. One significant outcome of cancer multi-omics data the days of survival after samples were collected. Because patients may not show up for checkups, is possible that sample is censored (unlabeled). The days of survival are then converted to a binary 1-year mortality indicator. The dataset itself has missing views but contains predictive information and high correlations among different views, making suitable for our task. Further notice that this dataset does not contain private ground truth.
D.2 IMPLEMENTATION
2-layer MLP with 128 as hidden dimensions was used for encoding and decoding with 48 latent dimensions (16 for z and 32 for w). Evaluation for all methods regarding this dataset is done by averaging logits from each view. 50 epochs were run to train the model. For TCGA dataset, base- line methods are performed with default hyperparameter, which gives KL divergence coefficient of 2.5. For DisentangledSSL. baseline particular, step coefficient was set to since we use posterior mean instead of zsample to match other methods and step 2 coefficient To tune our method, we performed a grid search with coefficients {0.001, 0.01, 1, 10, 100} and chose the best combination
set to 0.01.
validation set, before recording the performance test set. A = 10, X2 = 0.001 were chosen to be the best combination at 40 epochs. For ablation studies, set coefficient to be O while keeping the other optimal coefficients combination, we used the model at epoch 40: λ = O, at epoch 50; and A2 = 0, epoch 35. For adding diffusion prior, we tuned the diffusion weight be 0.1 of 0.1, 1, 10 while keeping Å1, A2 same as the optimal combination and chose the best performance at validation set at epoch 40.
optimal in combined setting. For the
D.3 PREDICTION AUROC
In Table 9. provide the linear classification AUROC of different methods using latent represen- tations. The relative merits of different methods are consistent with that observed with the accuracy metric in T
Table 9: Prediction AUROC Performance with ablation on TCGA dataset, averaged over 2 modali- ties and 5 splits. Tuning reported in Appendix D.2
Model
t
Z + W
MMVAE
0.653-10.033
MoPoE-VAE
0.660+000024
DMVAE
0.609+00.030
0.63670.037
0.643+-0.032
MMVAE+
0.586+ 0.027
0.5013
0.585+0.033
DisentangledSSL.
0.693100:06
0.551
0.69910.045
IDMVAE (ours)
0.740-L0.025
0.740:10.022
0.76710.026
LCrossMI (A1 0)
0.549-10.017
0.545-10.02
0.548
LuenAug (A2 = 0)
0.740+10.019
0.746:
0.771
Diffusion prior
0.745-10.024
0.751110.029
0.772+0.022
30


Reviews:
[{"review_id": "VWd5X953d5", "reviewer": "Reviewer_1dPy", "summary": "The paper “Disentanglement of Variations with Multi-Modal Generative Modeling (IDMVAE)” proposes a new multimodal variational autoencoder designed to cleanly separate shared and private sources of variation across data modalities. Standard multimodal VAEs, such as MMVAE and MoPoE, often suffer from latent leakage, where modality-specific information contaminates the shared latent space, causing incoherent cross-modal generation and poor disentanglement. IDMVAE introduces three complementary mechanisms to address this.\n\n1.\tA cross-view mutual information maximization term aligns shared representations by encouraging different modality-specific encoders to capture the same underlying latent factors.\n2.\tA generative augmentation cycle removes redundancy between shared and private latents by decoding “mixed” latent pairs into synthetic samples and enforcing that shared latents remain invariant while private latents retain modality-specific structure.\n3.\tA latent diffusion prior replaces the standard Gaussian prior with a learned diffusion model that can represent complex, multimodal latent distributions, reducing prior–posterior mismatch and improving unconditional generation quality.\n\nThe resulting objective extends the MMVAE+ ELBO with information-theoretic and generative-consistency regularizers, all trainable end-to-end. Experiments on PolyMNIST-Quadrant, CUB (image–text), and TCGA (multi-omics) show that IDMVAE achieves substantially cleaner separation of shared vs. private factors, higher conditional and unconditional coherence, and improved latent alignment compared to strong baselines. Ablation studies demonstrate that cross-view MI alignment improves cross-modal reconstruction, the generative augmentation reduces leakage, and the diffusion prior enhances sample quality.", "questions": "1.\tIs the diffusion prior applied only to the shared latent z, or also to private w_m? If only to z, why is a simple Gaussian sufficient for the private components?\n2.\tHow do the MI and cycle constraints correspond to orthogonality or covariance restrictions in linear Bayesian joint additive factor models such as [5]?\n3.\tThe paper does not mention a β-VAE coefficient. Is β fixed at 1, and do $\\lambda_{\\text{MI}}$, $\\lambda_{\\text{GenAug}}$ effectively play this role? How do these hyperparameters influence the average posterior entropies $H[q(z|x_m)] $ and $H[q(w_m|x_m)] $ over training.\n4.\tSome VAE models e.g. [6] address the inference-consistency problem in multimodal VAEs, i.e. the inability to obtain coherent subset posteriors such as $p(w_2 \\mid x_1)$ via a tighter ELBO and flexible encoder aggregation. Does IDMVAE provide an alternative route to this goal by enforcing independence between $w_m$ and the shared latent z, thereby avoiding aggregation bias or looser bounds for modalities with high conditional mutual information?\n5.\tThe CrossMI loss is motivated by encouraging $I(z_m;x_n)\\approx I(x_m;x_n)$, implying that the shared latent fully explains cross-modal dependence. Would this not render private latents redundant? How is this balanced to prevent collapse of private variation, and does this depend on dataset redundancy?\n6.\tIntegrating a diffusion prior end-to-end into a multimodal VAE can introduce instability, as seen in recent works like REPA-E [7] or uses some annealing tricks [8]. The paper does not discuss training schedules, warm-up strategies, or whether gradients from the diffusion loss are stopped for encoder updates.\n\nRefs:\n[5] Anceschi, Niccolo, et al. \"Bayesian joint additive factor models for multiview learning.\" arXiv preprint arXiv:2406.00778(2024).\\\n[6] Hirt, Marcel, et al. \"Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives.\" Transactions on Machine Learning Research.\\\n[7] Leng, Xingjian, et al. \"Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers.\" arXiv preprint arXiv:2504.10483 (2025).\\\n[8] Vahdat, Arash, Karsten Kreis, and Jan Kautz. \"Score-based generative modeling in latent space.\" Advances in neural information processing systems 34 (2021): 11287-11302.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\\\nIntroduces a principled formulation of multimodal disentanglement by combining cross-view mutual-information maximization with generative augmentation, directly targeting the separation of shared and private factors.\\\nConceptually unifies ideas from contrastive learning, information bottleneck, and generative modeling in a single probabilistic framework.\n\nQuality:\\\nMethodologically robust: each loss term is clearly motivated by known weaknesses of multimodal VAEs (e.g., posterior collapse, shared–private leakage).\\\nThe generative augmentation mechanism leverages the model’s own decoder for cycle consistency, avoiding handcrafted data augmentations\\\nComprehensive experiments across synthetic (PolyMNIST-Quadrant) and real (CUB, TCGA) datasets demonstrate consistent improvements in disentanglement, coherence, and generation quality.\\\nAblation studies isolate the contribution of each component (CrossMI, GenAug, diffusion prior) and show interpretable, consistent improvements.\\\n\nClarity:\\\nThe paper is well-structured and clearly written, with each component (CrossMI, GenAug, diffusion prior) well motivated and described.\\\nMathematical notation and objective formulation are concise and consistent.\n\nSignificance:\\\nAddresses a challenge in multimodal learning of separating  shared vs. private information with a practical, modular solution applicable to existing architectures.\\\nDemonstrates that learned latent priors (diffusion models) can substantially improve multimodal VAEs, opening a new direction for hybrid diffusion–VAE research. However, it lacks a clear comparison with other multi-modal (latent) diffusion models, such as [1,2].\n\nRefs: \n[1] Bounoua, Mustapha, Giulio Franzese, and Pietro Michiardi. \"Multi-modal latent diffusion.\" Entropy 26.4 (2024): 320. \\\n[2] Chen, Changyou, et al. \"Diffusion models for multi-modal generative modeling.\" arXiv preprint arXiv:2407.17571 (2024).", "weaknesses": "The connection between maximizing $I(z_m; z_n)$ and achieving conditional sufficiency $I(x_m;x_n|z)=0$ remains heuristic. Under what assumptions (e.g., conditional independence or additive latent structure) do the suggested losses approximate or minimal-sufficiency conditions.\\\nThe InfoNCE estimator can be sensitive to negative-sample size, temperature, and batch composition. The paper fixes these hyperparameters without ablation or analysis.\\\nAdding contrastive, augmentation, and diffusion losses likely increases computational cost. The paper lacks a discussion of these computation costs.\\\nYou compare to MMVAE variants but not to contrastive approaches such as [3-4], or as mentioned above, latent diffusion models [1-2]. \\\nThe generative augmentation term is described as reducing redundancy between z and w_m, but the paper does not analyze how effectively this enforces $I(z; w_m) \\to 0$ in the experiments.\n\n\n\nRefs: \\\n[3] Liang, Paul Pu, et al. \"Factorized contrastive learning: Going beyond multi-view redundancy.\" Advances in Neural Information Processing Systems 36 (2023): 32971-32998.\\\n[4] Dufumier, Benoit, et al. \"What to align in multimodal contrastive learning?.\" arXiv preprint arXiv:2409.07402 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m19zYntSzr", "reviewer": "Reviewer_bE3u", "summary": "This paper introduces the Information-disentangled Multimodal VAE (IDMVAE), a new model for learning from data with multiple modalities. IDMVAE achieves this improved disentanglement by using Mutual Information (MI) regularizations and latent diffusion priors. They try to learn a common latent z that captures all the shared information encouraged by the added new regularization losses introduced in this work. In addition, they utilize diffusion models to create more complex and capable latent priors, thereby moving beyond simple Gaussian priors and enhancing the model's representational capacity. The authors demonstrate that IDMVAE achieves better shared and private information, resulting in superior generation quality and enhanced semantic coherence on some simple datasets compared to existing methods.", "questions": "Please review the Weaknesses section and attempt to address the points raised there.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and well-presented. The paper also seems to have strong theoretical grounds for the idea they presented in this work. Their MI-based regularization methods could be useful for multimodal VAE works in general. The proposed losses are also useful for learning a shared representation, which is important for downstream tasks. Their proposed idea of using practical contrastive loss for cross-view MI maximization and their disentanglement learning with generative augmentation losses is a good contribution. They propose using diffusion models for learning complex latents, but this has already been proposed and implemented in other works (look at the weaknesses part).", "weaknesses": "Even though the paper is well presented, there are considerable weaknesses in this paper that render the contribution marginal.\n\n\n1. $ \\textbf{Used datasets are toy datasets} $  \n\nThe datasets used in this work are very small and not significant enough to show the practicality of the presented method. I encourage the authors to look at [1], which is also a work done on multimodal VAEs using a score-based model that used the CelebAMask-HQ multimodal dataset, which is larger than the datasets used in this paper, as a starting point. The paper is also not cited and not compared as a baseline. The authors can also add other larger datasets than this to prove their method is still useful in these settings.\n\n2. $\\textbf{Latent diffusion proposal}$\n\n Using a latent diffusion/score-based model to learn complex latents for multimodal VAEs is also presented in [2]. But it hasn't been cited in this work and presented as a new thing.\n\n3. $\\textbf{Limited baselines}$\n\n The baselines compared in this work are limited. Additional recent baselines should also be added to the work. E.g., [1,2,3,4]. \n\n4. $\\textbf{Evaluation of Quality}$\n\n The paper doesn't use FID for evaluating the quality of generated unconditional and conditional images as in [1]. I believe a quantifiable measure of quality is important in multimodal VAE works.\n\n5. $\\textbf{Evaluation of method of PoE Multimodal Models}$\n\n The paper proposes that the method can be used in PoE/MoPoE multimodal VAE models but those haven't been explored as ablations. Can the authors try their method in at least one setup of these multimodal VAEs as a proof of concept or ablation study.\n\n\nReferences\n1. Wesego, Daniel, and Amirmohammad Rooshenas. \"Score-based multimodal autoencoders.\" arXiv preprint arXiv:2305.15708 (2023).\n2. Wesego, Daniel, and Pedram Rooshenas. \"Multimodal ELBO with Diffusion Decoders.\" arXiv preprint arXiv:2408.16883 (2024).\n3. Bounoua, Mustapha, Giulio Franzese, and Pietro Michiardi. \"Multi-modal latent diffusion.\" Entropy 26, no. 4 (2024): 320.\n4. Palumbo, Emanuele, Laura Manduchi, Sonia Laguna, Daphné Chopard, and Julia E. Vogt. \"Deep generative clustering with multimodal diffusion variational autoencoders.\" In The Twelfth International Conference on Learning Representations. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8qlKlRSIB3", "reviewer": "Reviewer_wgHU", "summary": "This paper introduces Information-Disentangled Multimodal VAE (IDMVAE), a new multimodal generative framework designed to improve disentanglement between shared and modality-specific representations.", "questions": "1. Can you provide a way to visualize the learned shared latent variable and modality-specific variables? Comparing the visualizations for the MMVAE+ and the proposed method could be straightforward.\n\n2. Can this method scale up to a large-scale Multimodal dataset (e.g., MSCOCO)? \n\n3. How does the proposed method compare to other diffusion models in terms of, for example, CLIP score?\n\n4. How do you handle the text input? Any pre-trained embedding models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The cross-view regularization for disentangling the shared latent variable z and modality-specific w is well-motivated.\n\n2. The experiments consider cross-domain data, and the ablation studies are complementary.\n\n3. Using diffusion to model latent priors results in better alignment between prior and posterior distributions, leading to improved unconditional generation and coherence.", "weaknesses": "1. Compared to multimodal VAEs, the novelty is based on the cross-view regularization. For the diffusion part, CMVAE also integrates the diffusion model for improved performance. The conceptual advance is relatively modest.\n\n2. The paper does not provide training efficiency comparisons or runtime analyses (especially for the diffusion part). It unclear whether the performance comes from the model complexity.\n\n3. While figures illustrate disentanglement, generated samples on CUB are of relatively low fidelity, suggesting the model’s generative capacity is still limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)